Introduction
We develop ridgelet analysis for neural networks with infinite-dimensional inputs. The construction connects explicit integral representations to finite-width approximation and uniform approximation on compact sets.
From function inputs to network coefficients
Operator learning seeks to approximate nonlinear maps whose inputs are functions and whose outputs may also be functions. Examples include solution maps of differential equations and layers that combine an integral operator with a pointwise nonlinearity [54 , 64 , 67 ] . On a separable real Hilbert space H \mathcal{H} H , a basic model with values in a Hilbert space Y \mathcal{Y} Y is
f N ( x ) = ∑ j = 1 N v j σ ( ⟨ a j , x ⟩ − b j ) , a j ∈ H , b j ∈ R , v j ∈ Y . f_N(x) = \sum_{j=1}^{N} v_j \sigma(\langle a_j, x\rangle- b_j), \qquad a_j \in\mathcal{H},\ b_j \in\mathbb{R},\ v_j \in\mathcal{Y}. f N ( x ) = j = 1 ∑ N v j σ (⟨ a j , x ⟩ − b j ) , a j ∈ H , b j ∈ R , v j ∈ Y .
Each direction a j a_j a j defines a linear measurement of the input. Beyond asking whether these networks can approximate a target, we ask how to choose their directions, biases, and output coefficients from that target.
An integral representation addresses this question by replacing the finite sum with a superposition of neurons,
f ( x ) = S σ [ Γ ] ( x ) : = ∫ H × R σ ( ⟨ a , x ⟩ − b ) Γ ( d a , d b ) . f(x) = S_\sigma[\Gamma](x) := \int_{\mathcal{H}\times\mathbb{R}} \sigma(\langle a, x\rangle- b)\,\Gamma(\mathrm{d}a, \mathrm{d}b). f ( x ) = S σ [ Γ ] ( x ) := ∫ H × R σ (⟨ a , x ⟩ − b ) Γ ( d a , d b ) .
Here Γ \Gamma Γ is a finite scalar or vector coefficient measure. Its total variation measures the size of the representation. Once Γ \Gamma Γ is known, its normalized variation supplies a probability law for sampling neurons; its polar density supplies the output weights. Barron’s argument then relates the finite-width approximation error to the variation and moments of this measure [4 , 5 ] .
The analytic tool used here is a ridgelet transform: an integral operator that sends a target to coefficients indexed by directions and biases. It is related to the Fourier, Radon, and wavelet transforms [16 , 74 , 87 ] , but we construct and explain it directly on H \mathcal{H} H . No knowledge of ridgelet analysis is needed to read the definitions or the proofs below.
Universality of such networks is usually proved by existence arguments: by reduction to a finite-dimensional approximation theorem, or by Stone–Weierstrass-type density arguments on the infinite-dimensional space [9 , 23 , 46 ] ; random features give rates on Hilbert inputs [36 ] . These results establish existence or sampling bounds without an analysis operator. In this paper, we supply that operator: an explicit coefficient formula, a norm identity, and reconstruction, for the full class of continuous non-polynomial activations of polynomial growth, from which universality and finite-width rates follow.
The main technical obstacle is that an infinite-dimensional Hilbert space has no locally finite, translation-invariant analogue of Lebesgue measure, so the finite-dimensional ridgelet integrals cannot be carried over directly. We integrate against a Gaussian input measure and a homogeneous mixture of Gaussian measures on the direction space; the homogeneity parameter of the mixture replaces the dimension-dependent factor of the finite-dimensional admissibility condition. Section 3 constructs these measures and the transform.
Main results
Our main contributions connect transform identities to constructive neural-network approximation; the universality statement itself is of a type already known, and the novelty lies in how the approximants are obtained. The precise assumptions are stated with each theorem.
Transform and reconstruction. We construct a Gaussian-weighted ridgelet transform with a Plancherel identity: the norm of the coefficients equals a constant times the target norm in a Hilbert space defined by weighted Fourier analysis (Theorem 3.14). The transform is injective and has closed range. Reconstruction uses a frame operator , an invertible correction induced by the Gaussian weights, which can be undone after synthesis or before analysis (Theorem 4.8).
Explicit network coefficients. For targets given by a Fourier integral, we derive an explicit coefficient formula and an integral-network representation (Theorem 4.5). Smoothness of the Fourier density along lines through the origin, together with weighted bounds on its derivatives, makes the coefficient measure finite and gives the parameter moments needed for approximation (Theorem 6.6). The representation allows continuous activations of polynomial growth; non-polynomiality permits a nonzero reconstruction constant. In particular, ReLU is admissible when paired with a suitable analysis filter (Corollary 5.5). Different admissible analysis filters give different coefficients for the same target, so the representation is a family indexed by the filter.
Approximation rates and universality. Monte Carlo approximation of the integral representation gives finite-width networks. For globally Lipschitz activations, their expected uniform error on a compact set is of order N − 1 / 2 N^{-1/2} N − 1/2 , where N N N is the width, with no explicit input or output dimension factor (Theorem 6.5). The constant depends on the coefficient variation, parameter moments, and compact set. Density of the represented targets then proves cc-universality , meaning uniform approximation of every continuous target on each compact subset of the input space (Theorem 6.8). These results cover Hilbert-valued outputs, including function-valued operators.
Examples and numerical evidence. We give Gaussian targets, Gaussian-parameter networks, and neural-operator layers built from periodic convolution and a Dirichlet solution operator, with explicit representations (Section 7). Three experiments illustrate the predicted approximation rates across input resolutions, check an explicit transform formula, and separate finite-width error from numerical discretization error (Section 8).
Scope. The coefficient formula is explicit once a spectral density is given; for a general continuous target, the universality proof first chooses a spectral approximation, and the constants V V V and M 2 M_{2} M 2 depend on that choice. We do not obtain a uniform width–accuracy relation for all continuous targets, a coefficient-computation algorithm, or a statistical learning guarantee. Universality is proved in the compact-open topology, not globally on weighted non-compact spaces, and the N − 1 / 2 N^{-1/2} N − 1/2 rate is a Barron-type statement for targets whose coefficient has finite variation and moments, consistent with the lower bounds for Lipschitz operator classes (Section 1.3). Every finite-width network factors through finitely many measurements ⟨ a j , x ⟩ \langle a_j,x\rangle ⟨ a j , x ⟩ ; representing the directions themselves with finitely many numbers is a further approximation problem. Tempered reconstruction is an identity in a dual space, and general nonlinear operator activations are outside the reconstruction theorem.
Related work
Ridgelet analysis originates with Murata [74 ] , Candès [16 ] , and Candès and Donoho [17 ] as a transform adapted to ridge functions. Its distributional extension admits unbounded activations, including ReLU [87 ] ; weighted Sobolev activation spaces are developed in Sonoda et al. [90 ] ; and the Fourier-slice method yields transforms for group convolutional and non-Euclidean networks [88 , 89 ] , where the Hilbert-space construction of Sonoda et al. [89 Section 4] integrates over a finite-dimensional subspace. Complexity bounds derived from integral representations are surveyed in Kůrková [59 ] .
Integration over an infinite-dimensional space with respect to a Gaussian measure is classical: Fourier–Wiener transforms [14 , 60 ] , Gaussian Radon transforms [7 , 39 , 42 , 43 , 73 ] , and a ridgelet transform for Wiener functionals [68 ] , whose atoms are generated by a functional on the Cameron–Martin space and whose basic identity relates two Gaussian spaces of different variance rather than reconstructing the target. In this paper, the atoms are neurons, the direction measure is homogeneous, and the Plancherel identity holds on a single space; Appendix F makes these comparisons explicit, including the relation of our analysis transform to the Gaussian Radon transform.
Universal approximation with infinite-dimensional inputs goes back to the approximation of functionals [19 , 20 , 81 , 91 ] . Compact-open universality is known on Fréchet spaces with operator activations [9 , 32 ] , in L p ( μ ) L^p(\mu) L p ( μ ) [33 ] , for scalar ridge neurons on locally convex spaces with every continuous non-polynomial activation [46 ] , and globally on weighted spaces [23 , 82 ] ; neural operators [53 , 54 , 64 , 65 , 67 ] motivate the examples in § 7 . These are existence theorems; here the approximants are obtained from an explicit coefficient, one for each admissible analysis filter.
Finite-width bounds by sampling a parameter distribution follow Barron [5 ] , Bach [4 ] , and E, Ma, and Wu [28 ] , with Hilbert-input versions in Gonon et al. [36 , 37 ] , Banach-valued versions in Korolev [52 ] , Neufeld and Schmocker [78 ] , and encoder–decoder versions in Yang and Shi [99 ] . In this paper, the sampling measure is the homogeneous measure ν α \nu_\alpha ν α , the coefficient is characterized by an analysis operator with a Plancherel identity and a closed-range projection, and the uniform rate covers Hilbert-valued outputs. Lower bounds [2 , 62 , 63 ] exclude such rates for all Lipschitz operators; ours concerns a Barron-type class. Continuous frames [30 , 41 ] , Gaussian measures [12 , 24 , 80 ] , and Wiener chaos [48 ] supply the general background.
Notation and organization
Unless stated otherwise, H \mathcal{H} H is an infinite-dimensional separable real Hilbert space with inner product ⟨ ⋅ , ⋅ ⟩ \langle\cdot,\cdot\rangle ⟨ ⋅ , ⋅ ⟩ , B ( H ) \mathcal{B}(\mathcal{H}) B ( H ) its Borel sigma-algebra, and μ Q \mu_Q μ Q the centred Gaussian measure with covariance Q Q Q , that is, the Borel probability measure with ∫ H e i ⟨ x , ξ ⟩ μ Q ( d x ) = e − ⟨ Q ξ , ξ ⟩ / 2 \int_{\mathcal{H}} e^{i\langle x,\xi\rangle}\,\mu_Q(dx)=e^{-\langle Q\xi,\xi\rangle/2} ∫ H e i ⟨ x , ξ ⟩ μ Q ( d x ) = e − ⟨ Q ξ , ξ ⟩ /2 . Throughout, P P P and Q Q Q are injective, positive, self-adjoint, trace-class operators on H \mathcal{H} H , and D ω a : = ω a D_{\omega}a:=\omega a D ω a := ω a . Functions on H \mathcal{H} H are complex valued unless stated otherwise; Y \mathcal{Y} Y denotes a separable complex Hilbert space of outputs. On R \mathbb{R} R we use
h ♯ ( ω ) = ∫ R h ( t ) e − i t ω d t , h ( t ) = 1 2 π ∫ R h ♯ ( ω ) e i t ω d ω . (1) h^\sharp(\omega)=\int_{\mathbb{R}}h(t)e^{-it\omega}\,dt,\qquad h(t)=\frac{1}{2\pi}\int_{\mathbb{R}}h^\sharp(\omega)e^{it\omega}\,d\omega. \tag*{(1)} h ♯ ( ω ) = ∫ R h ( t ) e − i t ω d t , h ( t ) = 2 π 1 ∫ R h ♯ ( ω ) e i t ω d ω . (1)
We use ♯ \sharp ♯ for one-dimensional Fourier transforms, including tempered distributions and the partial transform in the bias:
γ ♯ ( a , ω ) : = ∫ R γ ( a , b ) e − i b ω d b . \gamma^\sharp(a,\omega):=\int_{\mathbb{R}}\gamma(a,b)e^{-ib\omega}\,db. γ ♯ ( a , ω ) := ∫ R γ ( a , b ) e − ibω d b .
A hat denotes the spatial Fourier transform, with the same negative exponential convention. For a finite measure M M M on H \mathcal{H} H ,
M ^ ( ξ ) : = ∫ H e − i ⟨ x , ξ ⟩ M ( d x ) . \widehat{M}(\xi):=\int_{\mathcal{H}}e^{-i\langle x,\xi\rangle}\,M(dx). M ( ξ ) := ∫ H e − i ⟨ x , ξ ⟩ M ( d x ) .
For square-integrable coefficients the partial transform is understood in L 2 L^2 L 2 ; see Lemma 3.6 . The letter ρ \rho ρ always denotes the analysis filter, σ \sigma σ a scalar synthesis activation; ω \omega ω is a frequency or dilation variable, ξ \xi ξ a spectral variable, a a a a direction, and b b b a bias. Distributional pairings ⟨ u , φ ⟩ \langle u,\varphi\rangle ⟨ u , φ ⟩ are bilinear.
We write μ C \mu_C μ C for a centred Gaussian law with covariance C C C , on the ambient Hilbert space specified in context; in one dimension μ v \mu_v μ v denotes the Gaussian law of variance v v v . When operator notation is needed, F R F_{\mathbb{R}} F R denotes the one-dimensional Fourier transform and F b F_b F b its partial version in the bias; F Q F_Q F Q denotes the Gaussian-weighted spatial transform. The inverse one-dimensional transform is denoted by h ˇ \check{h} h ˇ . Spectral densities are denoted by g g g , parameter densities by γ \gamma γ , and finite coefficient measures by Γ \Gamma Γ .
§ 2 fixes the network classes. § 3 constructs the transform and proves the Plancherel identity (Theorem 3.14 ). § 4 proves the representation and reconstruction theorems (Theorems 4.5 and 4.8 ), describes the frame operator, and records the vector-valued extension. § 5 gives reconstruction formulas for activations that are tempered distributions, including ReLU. § 6 contains the approximation rates and the universality theorem (Theorems 6.6 and 6.8 ). § 7 presents explicit examples, § 8 reports numerical experiments, and § 9 discusses open questions. The supporting lemmas and proofs needed for these results appear in the main text before their use. The appendices give additional kernel and projection formulas, supplementary approximation bounds, the finite-dimensional comparison and dilation obstruction, numerical methods, and an extended overview of related work (Appendix § 15 ). The experiment code is supplied as supplementary material.
Every theorem, proposition, lemma, corollary, and example of this paper has been formalized and machine-checked in Lean 4 with Mathlib. The blueprint at https://shosonoda.github.io/lean-operator-ridgelet/ links each item to the Lean statement that carries it and to its proof. Two clauses are stated there in a weaker form than here: the L 2 L^2 L 2 clause in the conclusion of Theorem 5.10 , and the closed forms of the reconstruction constants in Proposition 5.12 , which the Lean statements carry as the Sobolev pairing itself. Nothing in the present paper depends on the formalization; it is offered as an independent check.
Neural networks with Hilbert-space inputs
We define finite-width networks and integral networks with scalar activations and Hilbert-valued outputs. These definitions fix the coefficient measures and the compact-open topology used in the representation and approximation results.
Finite-width and integral networks
Definition 2.1 (Finite-width network). Let σ : R → C \sigma:\mathbb{R}\to\mathbb{C} σ : R → C be continuous. A width-N N N network on H \mathcal{H} H with values in Y \mathcal{Y} Y is
f N ( x ) = ∑ j = 1 N v j σ ( ⟨ a j , x ⟩ − b j ) , v j ∈ Y , ( a j , b j ) ∈ H × R . (2) f_N(x)=\sum_{j=1}^{N}v_j\,\sigma\left(\langle a_j,x\rangle-b_j\right),\qquad v_j\in\mathcal{Y},\qquad(a_j,b_j)\in\mathcal{H}\times\mathbb{R}. \tag*{(2)} f N ( x ) = j = 1 ∑ N v j σ ( ⟨ a j , x ⟩ − b j ) , v j ∈ Y , ( a j , b j ) ∈ H × R . (2)
The scalar case is Y = C \mathcal{Y}=\mathbb{C} Y = C .
We measure approximation in the compact-open topology of C ( H ; Y ) C(\mathcal{H};\mathcal{Y}) C ( H ; Y ) , generated by the seminorms
∥ f ∥ C ( K ; Y ) : = sup x ∈ K ∥ f ( x ) ∥ Y , K ⊂ H compact , \|f\|_{C(K;\mathcal{Y})}:=\sup_{x\in K}\|f(x)\|_{\mathcal{Y}},\qquad K\subset\mathcal{H}\text{ compact}, ∥ f ∥ C ( K ; Y ) := x ∈ K sup ∥ f ( x ) ∥ Y , K ⊂ H compact ,
and write C ( K ) C(K) C ( K ) when Y = C \mathcal{Y}=\mathbb{C} Y = C . Compact-open approximation does not truncate the input: it asks for uniform approximation on each fixed compact subset of H \mathcal{H} H . Finite-rank projections converge uniformly on a compact set, but a target need not factor through any one of them, and the examples in § 7 do not.
A vector measure assigns a vector to each Borel set and is countably additive in the output norm. Its variation is the positive measure ∣ Γ ∣ ( E ) = sup P ∑ j ∥ Γ ( E j ) ∥ Y |\Gamma|(E) = \sup_{\mathcal{P}} \sum_j \|\Gamma(E_j)\|_{\mathcal{Y}} ∣Γ∣ ( E ) = sup P ∑ j ∥Γ ( E j ) ∥ Y , where the supremum is over finite Borel partitions of E E E ; bounded variation means that its mass on the whole parameter space is finite. Since Y \mathcal{Y} Y is a Hilbert space, such a measure has a density h h h with respect to its variation: Γ ( E ) = ∫ E h d ∣ Γ ∣ \Gamma(E) = \int_E h\,\mathrm{d}|\Gamma| Γ ( E ) = ∫ E h d ∣Γ∣ , with ∥ h ∥ Y = 1 \|h\|_{\mathcal{Y}} = 1 ∥ h ∥ Y = 1 almost everywhere. Accordingly, an integral ∫ u d Γ \int u\,\mathrm{d}\Gamma ∫ u d Γ of a scalar function means the Bochner integral ∫ u h d ∣ Γ ∣ \int uh\,\mathrm{d}|\Gamma| ∫ u h d ∣Γ∣ . A vector-valued function is Bochner integrable when it is strongly measurable (an almost-everywhere limit of measurable simple functions) and its norm is integrable; its integral is obtained by integrating simple functions that approximate it in L 1 L^1 L 1 . We refer to [27 ] for more details.
Definition 2.2 (Integral network). Let Γ \Gamma Γ be a Y \mathcal{Y} Y -valued Borel measure of bounded variation on Θ : = H × R \Theta:= \mathcal{H} \times\mathbb{R} Θ := H × R , with variation measure ∣ Γ ∣ |\Gamma| ∣Γ∣ and total variation ∥ Γ ∥ T V : = ∣ Γ ∣ ( Θ ) \|\Gamma\|_{\mathrm{TV}} := |\Gamma|(\Theta) ∥Γ ∥ TV := ∣Γ∣ ( Θ ) . Whenever the Bochner integral exists, define
S σ [ Γ ] ( x ) : = ∫ H × R σ ( ⟨ a , x ⟩ − b ) Γ ( d a , d b ) . (3) S_\sigma[\Gamma](x) := \int_{\mathcal{H}\times\mathbb{R}} \sigma(\langle a,x\rangle-b)\,\Gamma(\mathrm{d}a,\mathrm{d}b). \tag*{(3)} S σ [ Γ ] ( x ) := ∫ H × R σ (⟨ a , x ⟩ − b ) Γ ( d a , d b ) . (3) If σ \sigma σ is globally Lipschitz, it suffices that
∫ H × R ( 1 + ∥ a ∥ + ∣ b ∣ ) d ∣ Γ ∣ ( a , b ) < ∞ . (4) \int_{\mathcal{H}\times\mathbb{R}} (1+\|a\|+|b|)\,\mathrm{d}|\Gamma|(a,b) < \infty. \tag*{(4)} ∫ H × R ( 1 + ∥ a ∥ + ∣ b ∣ ) d ∣Γ∣ ( a , b ) < ∞. (4) When Γ = γ λ \Gamma= \gamma\lambda Γ = γ λ has a density with respect to a sigma-finite reference measure λ \lambda λ on Θ \Theta Θ , we also write S σ [ γ ] S_\sigma[\gamma] S σ [ γ ] , and we say that the network has coefficient γ \gamma γ .
Finite-width networks are the syntheses of finite atomic measures. The total variation ∥ Γ ∥ T V \|\Gamma\|_{\mathrm{TV}} ∥Γ ∥ TV is the infinite-width analogue of the ℓ 1 \ell^1 ℓ 1 -norm of the outer weights, and it is the quantity that enters Barron’s discretization in § 6 . Scalar coefficients are complex; a real network is recovered by taking real parts.
The Gaussian-weighted ridgelet transform
We construct the transform using a Gaussian input measure and a homogeneous mixture of Gaussian direction measures. The Fourier-slice and Plancherel identities establish its analytic properties, and explicit filters show how the admissibility assumptions can be met.
The input and direction measures
Recall that P P P and Q Q Q are injective, positive, self-adjoint, trace-class operators on H \mathcal{H} H ; μ Q \mu_Q μ Q is the input Gaussian law. Gaussian weighting makes the input integral available, but a single Gaussian law on directions is not homogeneous under dilation. In finite-dimensional ridgelet analysis, the dilation Jacobian ∣ ω ∣ − m |\omega|^{-m} ∣ ω ∣ − m of Lebesgue measure on R m \mathbb{R}^m R m enters the admissibility condition. The following mixture replaces m m m by a freely chosen homogeneity parameter α \alpha α , so the corresponding condition remains meaningful on H \mathcal{H} H . For α > 0 \alpha> 0 α > 0 , define a Borel measure on the direction space by
ν α ( E ) : = ∫ 0 ∞ μ 2 s P ( E ) s α / 2 − 1 d s , E ∈ B ( H ) . (5) \nu_\alpha(E) := \int_0^\infty\mu_{2sP}(E)\,s^{\alpha/2-1}\,\mathrm{d}s,\qquad E \in\mathcal{B}(\mathcal{H}). \tag*{(5)} ν α ( E ) := ∫ 0 ∞ μ 2 s P ( E ) s α /2 − 1 d s , E ∈ B ( H ) . (5)
We first realize the Gaussian components on a common probability space and justify integration against the mixture.
The Gaussian measures can be realized on a common probability space ( Ω , F , P ) (\Omega,\mathcal{F},\mathbb{P}) ( Ω , F , P ) . Let P e j = p j e j Pe_j=p_je_j P e j = p j e j with p j > 0 p_j>0 p j > 0 , ∑ j p j < ∞ \sum_j p_j<\infty ∑ j p j < ∞ , and let Z j Z_j Z j be independent standard normal variables. The series
X = ∑ j ≥ 1 p j Z j e j , X s = 2 s X X=\sum_{j\geq1}\sqrt{p_j}Z_je_j,\qquad X_s=\sqrt{2s}\,X X = j ≥ 1 ∑ p j Z j e j , X s = 2 s X
converges in L 2 ( Ω ; H ) L^2(\Omega;\mathcal{H}) L 2 ( Ω ; H ) and almost surely: orthogonality gives the L 2 L^2 L 2 tail estimate, and E ∑ j p j Z j 2 = tr P < ∞ \mathbb{E}\sum_jp_jZ_j^2=\operatorname{tr}P<\infty E ∑ j p j Z j 2 = tr P < ∞ gives almost-sure square summability. Its finite-dimensional coordinates show that X ∼ μ P X\sim\mu_P X ∼ μ P and X s ∼ μ 2 s P X_s\sim\mu_{2sP} X s ∼ μ 2 s P ; see Prévôt and Röckner [80 Chapter 2] and Bogachev [12 Chapter 2] . All moments of ∥ X ∥ \lVert X\rVert ∥ X ∥ are finite: by Jensen’s inequality applied to the probability weights p j / tr P p_j/\operatorname{tr}P p j / tr P ,
E ∥ X ∥ 2 m = E ( ∑ j p j Z j 2 ) m ≤ ( tr P ) m E Z 1 2 m = ( tr P ) m ( 2 m − 1 ) ! ! . (6) \mathbb{E}\lVert X\rVert^{2m}=\mathbb{E}\left(\sum_jp_jZ_j^2\right)^m\leq(\operatorname{tr}P)^m\mathbb{E}Z_1^{2m}=(\operatorname{tr}P)^m(2m-1)!!. \tag*{(6)} E ∥ X ∥ 2 m = E ( j ∑ p j Z j 2 ) m ≤ ( tr P ) m E Z 1 2 m = ( tr P ) m ( 2 m − 1 )!! . (6)
The same construction with Q Q Q in place of P P P realizes μ Q \mu_Q μ Q .
Lemma 3.1 (Measurability and integration of the mixture). For every E ∈ B ( H ) E\in\mathcal{B}(\mathcal{H}) E ∈ B ( H ) , the map s ↦ μ 2 s P ( E ) s\mapsto\mu_{2sP}(E) s ↦ μ 2 s P ( E ) is Borel measurable. The mixture (5) defines a countably additive Borel measure, and for every nonnegative Borel function F : H → [ 0 , ∞ ] F:\mathcal{H}\to[0,\infty] F : H → [ 0 , ∞ ] ,
∫ H F ( ξ ) ν α ( d ξ ) = ∫ 0 ∞ ∫ H F ( ξ ) μ 2 s P ( d ξ ) s α / 2 − 1 d s . (7) \int_{\mathcal{H}}F(\xi)\,\nu_\alpha(\mathrm{d}\xi)=\int_0^\infty\int_{\mathcal{H}}F(\xi)\,\mu_{2sP}(\mathrm{d}\xi)\,s^{\alpha/2-1}\,\mathrm{d}s. \tag*{(7)} ∫ H F ( ξ ) ν α ( d ξ ) = ∫ 0 ∞ ∫ H F ( ξ ) μ 2 s P ( d ξ ) s α /2 − 1 d s . (7) Both sides may be infinite. For complex F F F the identity holds when ∫ H ∣ F ∣ d ν α < ∞ \int_{\mathcal{H}}|F|\,\mathrm{d}\nu_\alpha<\infty ∫ H ∣ F ∣ d ν α < ∞ .
Proof. The map ( s , x ) ↦ 2 s x (s,x)\mapsto\sqrt{2s}\,x ( s , x ) ↦ 2 s x is continuous on ( 0 , ∞ ) × H (0,\infty)\times\mathcal{H} ( 0 , ∞ ) × H , and μ 2 s P ( E ) = ∫ H 1 E ( 2 s x ) μ P ( d x ) \mu_{2sP}(E)=\int_{\mathcal{H}}\mathbf{1}_E(\sqrt{2s}\,x)\,\mu_P(\mathrm{d}x) μ 2 s P ( E ) = ∫ H 1 E ( 2 s x ) μ P ( d x ) . The integrand is jointly Borel, so its integral is a Borel function of s s s . Monotone convergence applied to disjoint unions proves countable additivity. The integral identity holds for indicators by definition, then for nonnegative simple functions, and finally for all nonnegative Borel F F F by monotone convergence; the complex case follows by linearity.
Lemma 3.2 (Homogeneous Gaussian mixture). For every α > 0 \alpha>0 α > 0 , the measure ν α \nu_\alpha ν α is infinite on H \mathcal{H} H and has full support, and for ω ≠ 0 \omega\neq0 ω = 0 ,
( D ω ) # ν α = ∣ ω ∣ − α ν α . (8) (D_\omega)_\#\nu_\alpha=|\omega|^{-\alpha}\nu_\alpha. \tag*{(8)} ( D ω ) # ν α = ∣ ω ∣ − α ν α . (8) equivalently, for every nonnegative Borel F F F on H \mathcal{H} H ,
∫ H F ( ω a ) ν α ( d a ) = ∣ ω ∣ − α ∫ H F ( ξ ) ν α ( d ξ ) . (9) \int_{\mathcal{H}}F(\omega a)\,\nu_\alpha(\mathrm{d}a)=|\omega|^{-\alpha}\int_{\mathcal{H}}F(\xi)\,\nu_\alpha(\mathrm{d}\xi). \tag*{(9)} ∫ H F ( ω a ) ν α ( d a ) = ∣ ω ∣ − α ∫ H F ( ξ ) ν α ( d ξ ) . (9) The measure ν α \nu_\alpha ν α is moreover sigma-finite and finite on bounded Borel sets.
Proof. Let p j > 0 p_j>0 p j > 0 be the eigenvalues of P P P . For a ∼ μ 2 s P a\sim\mu_{2sP} a ∼ μ 2 s P , any k ≥ 1 k\geq1 k ≥ 1 , and the ball B r = { a ∈ H : ∥ a ∥ < r } B_r=\{a\in\mathcal{H}:\lVert a\rVert<r\} B r = { a ∈ H : ∥ a ∥ < r } , the first k k k coordinates give
μ 2 s P ( B r ) ≤ r k ( π s ) − k / 2 ∏ j ≤ k p j − 1 / 2 , \mu_{2sP}(B_r)\leq r^k(\pi s)^{-k/2}\prod_{j\leq k}p_j^{-1/2}, μ 2 s P ( B r ) ≤ r k ( π s ) − k /2 j ≤ k ∏ p j − 1/2 ,
because the density of each coordinate is bounded by ( 4 π s p j ) − 1 / 2 (4\pi sp_j)^{-1/2} ( 4 π s p j ) − 1/2 and the coordinates lie in an interval of length 2 r 2r 2 r . Choose k > α k>\alpha k > α , which is possible because dim H = ∞ \dim\mathcal{H}=\infty dim H = ∞ . The part s ≤ 1 s\leq1 s ≤ 1 of (5) is integrable because its integrand is at most s α / 2 − 1 s^{\alpha/2-1} s α /2 − 1 ; the small-ball bound treats s ≥ 1 s\geq1 s ≥ 1 . Thus bounded sets have finite mass. Balls exhaust H \mathcal{H} H , whereas ν α ( H ) = ∫ 0 ∞ s α / 2 − 1 d s = ∞ \nu_{\alpha}(\mathcal{H})=\int_{0}^{\infty}s^{\alpha/2-1}\,ds=\infty ν α ( H ) = ∫ 0 ∞ s α /2 − 1 d s = ∞ , proving sigma-finiteness and infinitude. Every Gaussian component has full support because P P P is injective, and so does the mixture.
The law of ω a \omega a ω a under μ 2 s P \mu_{2sP} μ 2 s P is μ 2 s ω 2 P \mu_{2s\omega^{2}P} μ 2 s ω 2 P ; the substitution u = s ω 2 u=s\omega^{2} u = s ω 2 in (5) proves (8) for both signs of ω \omega ω , and (9) is its integrated form by Lemma 3.1 .
In finite dimension the mixture is explicit: on H = R m \mathcal{H}=\mathbb{R}^{m} H = R m with P = I P=I P = I and 0 < α < m 0<\alpha<m 0 < α < m , ν α ( d a ) = c m , α ∥ a ∥ α − m d a \nu_{\alpha}(da)=c_{m,\alpha}\lVert a\rVert^{\alpha-m}da ν α ( d a ) = c m , α ∥ a ∥ α − m d a (Appendix § 13 .§ 2 ). Thus α \alpha α is the homogeneity degree of the direction measure, not a dimension, and in infinite dimension every α > 0 \alpha>0 α > 0 is allowed.
Admissible filters, the transform, and the Fourier slice
Definition 3.3 (Admissible analysis filter). A real function ρ ∈ S ( R ) \rho\in\mathcal{S}(\mathbb{R}) ρ ∈ S ( R ) is α \alpha α -admissible if
0 < ( ( ρ , ρ ) ) α : = 1 2 π ∫ R ∣ ρ ♯ ( ω ) ∣ 2 ∣ ω ∣ − α d ω < ∞ . (10) 0<((\rho,\rho))_{\alpha}:=\frac{1}{2\pi}\int_{\mathbb{R}}\left|\rho^{\sharp}(\omega)\right|^{2}|\omega|^{-\alpha}\,d\omega<\infty. \tag*{(10)} 0 < (( ρ , ρ ) ) α := 2 π 1 ∫ R ρ ♯ ( ω ) 2 ∣ ω ∣ − α d ω < ∞. (10) We call it a band-pass filter if moreover
ρ ♯ ∈ C c ∞ ( R ∖ { 0 } ) , (11) \rho^{\sharp}\in C^{\infty}_{c}(\mathbb{R}\setminus\{0\}), \tag*{(11)} ρ ♯ ∈ C c ∞ ( R ∖ { 0 }) , (11) which specifies the compact Fourier support away from zero required by the term band-pass here. Such a filter is α \alpha α -admissible for every α > 0 \alpha>0 α > 0 ; since ρ ♯ \rho^{\sharp} ρ ♯ vanishes near the origin, the integral in (10) is in fact finite and positive for every real α \alpha α . For two admissible filters put
( ( ρ 1 , ρ 2 ) ) α : = 1 2 π ∫ R ρ 1 ♯ ( ω ) ρ 2 ♯ ( ω ) ‾ ∣ ω ∣ − α d ω . (12) ((\rho_{1},\rho_{2}))_{\alpha}:=\frac{1}{2\pi}\int_{\mathbb{R}}\rho_{1}^{\sharp}(\omega)\overline{\rho_{2}^{\sharp}(\omega)}|\omega|^{-\alpha}\,d\omega. \tag*{(12)} (( ρ 1 , ρ 2 ) ) α := 2 π 1 ∫ R ρ 1 ♯ ( ω ) ρ 2 ♯ ( ω ) ∣ ω ∣ − α d ω . (12) The self-admissibility constant (10) is the case ρ 1 = ρ 2 = ρ \rho_{1}=\rho_{2}=\rho ρ 1 = ρ 2 = ρ of this pairing, since ρ \rho ρ is real. Explicit band-pass filters, and the Mexican hat as an example of a non-band-pass admissible filter, are given in Examples 3.15 and 3.16.
Definition 3.4 (Gaussian-weighted ridgelet transform). For f ∈ L 1 ( H , μ Q ) f\in L^{1}(\mathcal{H},\mu_{Q}) f ∈ L 1 ( H , μ Q ) and ρ ∈ S ( R ) \rho\in\mathcal{S}(\mathbb{R}) ρ ∈ S ( R ) , define
R ρ f ( a , b ) : = ∫ H f ( x ) ρ ( ⟨ a , x ⟩ − b ) μ Q ( d x ) , ( a , b ) ∈ H × R , (13) R_{\rho}f(a,b):=\int_{\mathcal{H}}f(x)\rho(\langle a,x\rangle-b)\,\mu_{Q}(dx),\qquad(a,b)\in\mathcal{H}\times\mathbb{R}, \tag*{(13)} R ρ f ( a , b ) := ∫ H f ( x ) ρ (⟨ a , x ⟩ − b ) μ Q ( d x ) , ( a , b ) ∈ H × R , (13) and let λ α : = ν α ⊗ d b \lambda_{\alpha}:=\nu_{\alpha}\otimes db λ α := ν α ⊗ d b be the parameter measure on H × R \mathcal{H}\times\mathbb{R} H × R .
The Gaussian input weight makes the defining integral meaningful on H \mathcal{H} H . The associated Fourier operator is the Fourier transform of the finite measure f μ Q f\mu_{Q} f μ Q ,
F Q f ( ξ ) : = ∫ H f ( x ) e − i ⟨ x , ξ ⟩ μ Q ( d x ) = f μ Q ^ ( ξ ) , f ∈ L 1 ( H , μ Q ) . (14) F_{Q}f(\xi):=\int_{\mathcal{H}}f(x)e^{-i\langle x,\xi\rangle}\,\mu_{Q}(dx)=\widehat{f\mu_{Q}}(\xi),\qquad f\in L^{1}(\mathcal{H},\mu_{Q}). \tag*{(14)} F Q f ( ξ ) := ∫ H f ( x ) e − i ⟨ x , ξ ⟩ μ Q ( d x ) = f μ Q ( ξ ) , f ∈ L 1 ( H , μ Q ) . (14)
Because μ Q \mu_{Q} μ Q is a probability measure, L 2 ( μ Q ) ⊂ L 1 ( μ Q ) L^{2}(\mu_{Q})\subset L^{1}(\mu_{Q}) L 2 ( μ Q ) ⊂ L 1 ( μ Q ) , and F Q f F_{Q}f F Q f is bounded and continuous.
Lemma 3.5 (Fourier-slice identity). Let f ∈ L 1 ( H , μ Q ) f\in L^{1}(\mathcal{H},\mu_{Q}) f ∈ L 1 ( H , μ Q ) and ρ ∈ S ( R ) \rho\in\mathcal{S}(\mathbb{R}) ρ ∈ S ( R ) . Then R ρ f R_{\rho}f R ρ f is bounded and jointly continuous on H × R \mathcal{H}\times\mathbb{R} H × R , and for every a ∈ H a\in\mathcal{H} a ∈ H
∥ R ρ f ( a , ⋅ ) ∥ L 1 ( R ) ≤ ∥ f ∥ L 1 ( μ Q ) ∥ ρ ∥ L 1 ( R ) , \lVert R_{\rho}f(a,\cdot)\rVert_{L^{1}(\mathbb{R})}\leq\lVert f\rVert_{L^{1}(\mu_{Q})}\lVert\rho\rVert_{L^{1}(\mathbb{R})}, ∥ R ρ f ( a , ⋅ ) ∥ L 1 ( R ) ≤ ∥ f ∥ L 1 ( μ Q ) ∥ ρ ∥ L 1 ( R ) , ∥ R ρ f ( a , ⋅ ) ∥ L 2 ( R ) 2 ≤ ∥ f ∥ L 2 ( μ Q ) 2 ∥ ρ ∥ L 2 ( R ) 2 if f ∈ L 2 ( μ Q ) . \lVert R_{\rho}f(a,\cdot)\rVert_{L^{2}(\mathbb{R})}^{2}\leq\lVert f\rVert_{L^{2}(\mu_{Q})}^{2}\lVert\rho\rVert_{L^{2}(\mathbb{R})}^{2}\qquad\text{if }f\in L^{2}(\mu_{Q}). ∥ R ρ f ( a , ⋅ ) ∥ L 2 ( R ) 2 ≤ ∥ f ∥ L 2 ( μ Q ) 2 ∥ ρ ∥ L 2 ( R ) 2 if f ∈ L 2 ( μ Q ) . For every a ∈ H a\in\mathcal{H} a ∈ H and ω ∈ R \omega\in\mathbb{R} ω ∈ R ,
( R ρ f ) ♯ ( a , ω ) : = ∫ R R ρ f ( a , b ) e − i ω b d b = ρ ♯ ( − ω ) F Q f ( ω a ) . (15) (R_{\rho}f)^{\sharp}(a,\omega):=\int_{\mathbb{R}}R_{\rho}f(a,b)e^{-i\omega b}\,db=\rho^{\sharp}(-\omega)F_{Q}f(\omega a). \tag*{(15)} ( R ρ f ) ♯ ( a , ω ) := ∫ R R ρ f ( a , b ) e − iω b d b = ρ ♯ ( − ω ) F Q f ( ω a ) . (15)
Proof. Boundedness follows from ∣ R ρ f ( a , b ) ∣ ≤ ∥ f ∥ 1 ∥ ρ ∥ ∞ |R_\rho f(a,b)| \le\|f\|_1\|\rho\|_\infty ∣ R ρ f ( a , b ) ∣ ≤ ∥ f ∥ 1 ∥ ρ ∥ ∞ , and joint continuity from dominated convergence. Cauchy–Schwarz in the probability measure μ Q \mu_Q μ Q and Tonelli in b b b give the L 2 L^2 L 2 bound. The L 1 L^1 L 1 bound and Fubini follow from ∫ H ∫ R ∣ f ( x ) ∣ ∣ ρ ( ⟨ a , x ⟩ − b ) ∣ d b μ Q ( d a ) = ∥ f ∥ L 1 ( μ Q ) ∥ ρ ∥ L 1 \int_{\mathcal H}\int_{\mathbb R}|f(x)||\rho(\langle a,x\rangle-b)|\,db\,\mu_Q(da)=\|f\|_{L^1(\mu_Q)}\|\rho\|_{L^1} ∫ H ∫ R ∣ f ( x ) ∣∣ ρ (⟨ a , x ⟩ − b ) ∣ d b μ Q ( d a ) = ∥ f ∥ L 1 ( μ Q ) ∥ ρ ∥ L 1 . With u = ⟨ a , x ⟩ − b u=\langle a,x\rangle-b u = ⟨ a , x ⟩ − b , the inner Fourier transform equals e − i ω ⟨ a , x ⟩ ρ ♯ ( − ω ) e^{-i\omega\langle a,x\rangle}\rho^\sharp(-\omega) e − iω ⟨ a , x ⟩ ρ ♯ ( − ω ) , which is (15) . □
The Fourier-slice identity determines the coefficient from the spectral side alone. For square-integrable coefficients we use the partial Fourier transform in the bias; its precise construction and jointly measurable representatives are given in Lemma 3.6.
We first construct the partial Fourier transform on the coefficient space. This lemma, and several auxiliary results below, are stated for coefficients with values in a separable complex Hilbert space Y \mathcal Y Y ; the scalar case is Y = C \mathcal Y=\mathbb C Y = C . Vector- and function-valued outputs arise for neural operators and for compositions of network layers (§ 4.3 and § 7 ), and stating these results once for Y \mathcal Y Y settles measurability, the interchange of integrals, and Plancherel for every later use, with Bochner integrals in place of scalar integrals.
Lemma 3.6 (Partial Fourier transform of a square-integrable coefficient). Let ν \nu ν be a sigma-finite Borel measure on H \mathcal H H , and let Y \mathcal Y Y be a separable complex Hilbert space. The Fourier transform in the bias defines a unitary map
F b : L 2 ( ν ⊗ d b ; Y ) ⟶ L 2 ( ν ⊗ d ω 2 π ; Y ) . \mathcal F_b:L^2(\nu\otimes db;\mathcal Y)\longrightarrow L^2\left(\nu\otimes\frac{d\omega}{2\pi};\mathcal Y\right). F b : L 2 ( ν ⊗ d b ; Y ) ⟶ L 2 ( ν ⊗ 2 π d ω ; Y ) . Its image has a jointly strongly measurable representative γ ♯ ( a , ω ) \gamma^\sharp(a,\omega) γ ♯ ( a , ω ) whose section, for ν \nu ν -almost every a a a , is the L 2 L^2 L 2 Fourier transform of γ ( a , ⋅ ) \gamma(a,\cdot) γ ( a , ⋅ ) . These representatives are equal almost everywhere in the product measure, and
∫ H ∫ R ∥ γ ♯ ( a , ω ) ∥ Y 2 d ω 2 π ν ( d a ) = ∥ γ ∥ L 2 ( ν ⊗ d b ; Y ) 2 . \int_{\mathcal H}\int_{\mathbb R}\|\gamma^\sharp(a,\omega)\|_{\mathcal Y}^2\frac{d\omega}{2\pi}\,\nu(da)=\|\gamma\|_{L^2(\nu\otimes db;\mathcal Y)}^2. ∫ H ∫ R ∥ γ ♯ ( a , ω ) ∥ Y 2 2 π d ω ν ( d a ) = ∥ γ ∥ L 2 ( ν ⊗ d b ; Y ) 2 . Proof. Fubini identifies the two spaces with the Bochner spaces
L 2 ( ν ; L 2 ( d b ; Y ) ) , L 2 ( ν ; L 2 ( d ω 2 π ; Y ) ) . L^2\left(\nu;L^2(db;\mathcal Y)\right),\qquad L^2\left(\nu;L^2\left(\frac{d\omega}{2\pi};\mathcal Y\right)\right). L 2 ( ν ; L 2 ( d b ; Y ) ) , L 2 ( ν ; L 2 ( 2 π d ω ; Y ) ) . Apply the Hilbert-valued one-dimensional Plancherel unitary to each section. This preserves strong measurability by approximation with simple functions, and preserves the squared norm by Tonelli. The inverse operation is the sectionwise inverse Fourier transform. The inverse Fubini identification supplies a jointly measurable representative; sectionwise uniqueness and Fubini give product almost-everywhere uniqueness. A square-integrable coefficient need not be integrable in the bias, so its Fourier transform cannot in general be computed by an absolutely convergent integral. The transform and its inverse are defined here through convergence in the L 2 L^2 L 2 norm; no pointwise convergence of a Fourier integral is asserted. □
This partial transform yields the following coefficient operator. Its scaled-isometry property is proved immediately below.
Definition 3.7 (The coefficient operator). Let ρ \rho ρ be α \alpha α -admissible and let g : H → C g:\mathcal H\to\mathbb C g : H → C be Borel with g ∈ L 2 ( ν α ) g\in L^2(\nu_\alpha) g ∈ L 2 ( ν α ) . The coefficient W ρ g ∈ L 2 ( λ α ) W_{\rho}g\in L^2(\lambda_\alpha) W ρ g ∈ L 2 ( λ α ) is the function whose partial Fourier transform in the bias is
( W ρ g ) ♯ ( a , ω ) = ρ ♯ ( − ω ) g ( ω a ) . (16) (W_{\rho}g)^\sharp(a,\omega)=\rho^\sharp(-\omega)g(\omega a). \tag*{(16)} ( W ρ g ) ♯ ( a , ω ) = ρ ♯ ( − ω ) g ( ω a ) . (16) For every such g g g , the following inverse Fourier integral converges absolutely for ν α \nu_\alpha ν α -almost every a a a , and defines a representative for all b ∈ R b\in\mathbb R b ∈ R :
γ g ( a , b ) : = W ρ g ( a , b ) = 1 2 π ∫ R ρ ♯ ( − ω ) g ( ω a ) e i ω b d ω . (17) \gamma_g(a,b):=W_{\rho}g(a,b)=\frac{1}{2\pi}\int_{\mathbb R}\rho^\sharp(-\omega)g(\omega a)e^{i\omega b}\,d\omega. \tag*{(17)} γ g ( a , b ) := W ρ g ( a , b ) = 2 π 1 ∫ R ρ ♯ ( − ω ) g ( ω a ) e iω b d ω . (17)
Lemma 3.8 (The coefficient operator is a scaled isometry). W ρ : L 2 ( ν α ) → L 2 ( λ α ) W_\rho: L^2(\nu_\alpha) \to L^2(\lambda_\alpha) W ρ : L 2 ( ν α ) → L 2 ( λ α ) is well defined, independent of the Borel representative of g g g , and ∥ W ρ g ∥ L 2 ( λ α ) 2 = ( ( ρ , ρ ) ) α ∥ g ∥ L 2 ( ν α ) 2 \|W_\rho g\|_{L^2(\lambda_\alpha)}^2 = ((\rho,\rho))_\alpha\|g\|_{L^2(\nu_\alpha)}^2 ∥ W ρ g ∥ L 2 ( λ α ) 2 = (( ρ , ρ ) ) α ∥ g ∥ L 2 ( ν α ) 2 . For every g ∈ L 2 ( ν α ) g \in L^2(\nu_\alpha) g ∈ L 2 ( ν α ) , ρ † ( − ⋅ ) g ( ⋅ a ) ∈ L 1 ( R ) \rho^\dagger(-\cdot)g(\cdot\,a) \in L^1(\mathbb{R}) ρ † ( − ⋅ ) g ( ⋅ a ) ∈ L 1 ( R ) for ν α \nu_\alpha ν α -almost every a a a , so (17) holds without an additional L 1 ( ν α ) L^1(\nu_\alpha) L 1 ( ν α ) hypothesis.
Proof. The map ( a , ω ) ↦ g ( ω a ) (a,\omega) \mapsto g(\omega a) ( a , ω ) ↦ g ( ω a ) is Borel, and by (9) and Tonelli,
1 2 π ∫ R ∫ H ∣ ρ † ( ω ) ∣ 2 ∣ g ( ω a ) ∣ 2 ν α ( d a ) d ω = ( ( ρ , ρ ) ) α ∥ g ∥ L 2 ( ν α ) 2 . \frac{1}{2\pi}\int_{\mathbb{R}}\int_{\mathcal{H}}|\rho^\dagger(\omega)|^2|g(\omega a)|^2\,\nu_\alpha(da)\,d\omega= ((\rho,\rho))_\alpha\|g\|_{L^2(\nu_\alpha)}^2. 2 π 1 ∫ R ∫ H ∣ ρ † ( ω ) ∣ 2 ∣ g ( ω a ) ∣ 2 ν α ( d a ) d ω = (( ρ , ρ ) ) α ∥ g ∥ L 2 ( ν α ) 2 .
This also proves representative independence by applying the identity to the difference of two representatives, and Lemma 3.6 gives the coefficient. To justify its integral formula, put
A ρ 2 = ∫ R ∣ ρ † ( ω ) ∣ 2 ∣ ω ∣ − α ( 1 + ω 2 ) d ω < ∞ , B g ( a ) 2 = ∫ R ∣ g ( ω a ) ∣ 2 ∣ ω ∣ α 1 + ω 2 d ω . A_\rho^2=\int_{\mathbb{R}}|\rho^\dagger(\omega)|^2|\omega|^{-\alpha}(1+\omega^2)\,d\omega<\infty,\qquad B_g(a)^2=\int_{\mathbb{R}}|g(\omega a)|^2\frac{|\omega|^\alpha}{1+\omega^2}\,d\omega. A ρ 2 = ∫ R ∣ ρ † ( ω ) ∣ 2 ∣ ω ∣ − α ( 1 + ω 2 ) d ω < ∞ , B g ( a ) 2 = ∫ R ∣ g ( ω a ) ∣ 2 1 + ω 2 ∣ ω ∣ α d ω .
The first integral is finite by admissibility near zero and Schwartz decay at infinity. Homogeneity and Tonelli give ∫ H B g ( a ) 2 ν α ( d a ) = π ∥ g ∥ 2 2 \int_{\mathcal{H}}B_g(a)^2\nu_\alpha(da)=\pi\|g\|_2^2 ∫ H B g ( a ) 2 ν α ( d a ) = π ∥ g ∥ 2 2 . Cauchy–Schwarz bounds ∫ ∣ ρ † ( − ω ) g ( ω a ) ∣ d ω \int|\rho^\dagger(-\omega)g(\omega a)|d\omega ∫ ∣ ρ † ( − ω ) g ( ω a ) ∣ d ω by A ρ B g ( a ) < ∞ A_\rho B_g(a)<\infty A ρ B g ( a ) < ∞ for almost every a a a . Fourier inversion for an L 1 ∩ L 2 L^1\cap L^2 L 1 ∩ L 2 frequency section proves the formula. □ \square □
The construction and norm identity require only ( ( ρ , ρ ) ) α < ∞ ((\rho,\rho))_\alpha<\infty (( ρ , ρ ) ) α < ∞ ; its positivity makes the scaled isometry injective. The additional condition g ∈ L 1 ( ν α ) g\in L^1(\nu_\alpha) g ∈ L 1 ( ν α ) enters the representation theorem, where it defines a bounded continuous target and justifies synthesis.
Lemma 3.9 (The adjoint coefficient operator). For an α \alpha α -admissible ρ \rho ρ , let
W ρ ∗ : L 2 ( λ α ) ⟶ L 2 ( ν α ) W_\rho^*:L^2(\lambda_\alpha)\longrightarrow L^2(\nu_\alpha) W ρ ∗ : L 2 ( λ α ) ⟶ L 2 ( ν α ) be the Hilbert adjoint of the coefficient operator W ρ W_\rho W ρ . Writing C = ( ( ρ , ρ ) ) α C=((\rho,\rho))_\alpha C = (( ρ , ρ ) ) α , one has, for every γ ∈ L 2 ( λ α ) \gamma\in L^2(\lambda_\alpha) γ ∈ L 2 ( λ α ) ,
W ρ ∗ W ρ = C I d , ∥ W ρ ∗ ∥ L 2 ( ν α ) ≤ C ∥ γ ∥ L 2 ( λ α ) . W_\rho^*W_\rho=C\,\mathrm{Id},\qquad\|W_\rho^*\|_{L^2(\nu_\alpha)}\le\sqrt{C}\|\gamma\|_{L^2(\lambda_\alpha)}. W ρ ∗ W ρ = C Id , ∥ W ρ ∗ ∥ L 2 ( ν α ) ≤ C ∥ γ ∥ L 2 ( λ α ) . For any jointly measurable representative γ ♯ \gamma^\sharp γ ♯ of the partial Fourier transform supplied by Lemma 3.6 , its explicit formula is
W ρ ∗ γ ( ξ ) = 1 2 π ∫ R ∖ { 0 } ρ † ( − ω ) ‾ ∣ ω ∣ − α γ ♯ ( ξ / ω , ω ) d ω . (18) W_\rho^*\gamma(\xi)=\frac{1}{2\pi}\int_{\mathbb{R}\setminus\{0\}}\overline{\rho^\dagger(-\omega)}|\omega|^{-\alpha}\gamma^\sharp(\xi/\omega,\omega)\,d\omega. \tag*{(18)} W ρ ∗ γ ( ξ ) = 2 π 1 ∫ R ∖ { 0 } ρ † ( − ω ) ∣ ω ∣ − α γ ♯ ( ξ / ω , ω ) d ω . (18) where integration is with respect to Lebesgue measure on R ∖ { 0 } \mathbb{R}\setminus\{0\} R ∖ { 0 } . The integral converges absolutely for ν α \nu_\alpha ν α -almost every ξ \xi ξ , and its L 2 L^2 L 2 class is independent of that representative.
Proof. Polarizing Lemma 3.8 gives W ρ ∗ W ρ = C I d W_\rho^*W_\rho=C\,\mathrm{Id} W ρ ∗ W ρ = C Id , and the adjoint has the same norm as the operator. For the formula, homogeneity and Lemma 3.6 give
1 2 π ∫ H ∫ R ∖ { 0 } ∣ ω ∣ − α ∣ γ ♯ ( ξ / ω , ω ) ∣ 2 d ω ν α ( d ξ ) = ∥ γ ∥ 2 2 . \frac{1}{2\pi}\int_{\mathcal{H}}\int_{\mathbb{R}\setminus\{0\}}|\omega|^{-\alpha}|\gamma^\sharp(\xi/\omega,\omega)|^2\,d\omega\,\nu_\alpha(d\xi)=\|\gamma\|_2^2. 2 π 1 ∫ H ∫ R ∖ { 0 } ∣ ω ∣ − α ∣ γ ♯ ( ξ / ω , ω ) ∣ 2 d ω ν α ( d ξ ) = ∥ γ ∥ 2 2 .
Weighted Cauchy–Schwarz proves absolute convergence and the same norm bound for the formula; the change of variables also preserves product null sets. Fubini and ξ = ω a \xi=\omega a ξ = ω a then give ⟨ W ρ ∗ γ , g ⟩ = ⟨ γ , W ρ g ⟩ \langle W_\rho^*\gamma,g\rangle=\langle\gamma,W_\rho g\rangle ⟨ W ρ ∗ γ , g ⟩ = ⟨ γ , W ρ g ⟩ , with absolute integral bounded by C ∥ γ ∥ 2 ∥ g ∥ 2 \sqrt{C}\|\gamma\|_2\|g\|_2 C ∥ γ ∥ 2 ∥ g ∥ 2 . Thus the formula is the adjoint. □ \square □
Following the terminology for adjoint transforms in tomography [75 ] , we call W ρ ∗ W_\rho^* W ρ ∗ the backprojection operator . Here it combines coefficient data along lines through the origin and takes values in the frequency domain.
In this notation the Fourier-slice identity reads
R ρ f = W ρ F Q f , f ∈ L 1 ( μ Q ) , F Q f ∈ L 2 ( ν α ) : (19) R_\rho f = W_\rho F_Q f,\qquad f \in L^1(\mu_Q),\ F_Q f \in L^2(\nu_\alpha): \tag*{(19)} R ρ f = W ρ F Q f , f ∈ L 1 ( μ Q ) , F Q f ∈ L 2 ( ν α ) : (19)
the ridgelet transform of f f f is the coefficient of its Gaussian-weighted Fourier transform.
The Hilbert space E α E_\alpha E α
Definition 3.10 (The Hilbert space E α E_\alpha E α ). Set
D α : = { f ∈ L 2 ( H , μ Q ) : F Q f ∈ L 2 ( H , ν α ) } , (20) D_\alpha:= \{f \in L^2(\mathcal{H},\mu_Q): F_Q f \in L^2(\mathcal{H},\nu_\alpha)\}, \tag*{(20)} D α := { f ∈ L 2 ( H , μ Q ) : F Q f ∈ L 2 ( H , ν α )} , (20) ⟨ f , g ⟩ E α : = ∫ H F Q f ( ξ ) F Q g ( ξ ) ‾ ν α ( d ξ ) , f , g ∈ D α . (21) \langle f,g\rangle_{E_\alpha} := \int_{\mathcal{H}} F_Q f(\xi)\overline{F_Q g(\xi)}\,\nu_\alpha(d\xi),\qquad f,g \in D_\alpha. \tag*{(21)} ⟨ f , g ⟩ E α := ∫ H F Q f ( ξ ) F Q g ( ξ ) ν α ( d ξ ) , f , g ∈ D α . (21) The Hilbert space E α E_\alpha E α is the completion of D α D_\alpha D α in the norm induced by (21), and
K α : = F Q ( D α ) ‾ L 2 ( ν α ) (22) K_\alpha:= \overline{F_Q(D_\alpha)}^{L^2(\nu_\alpha)} \tag*{(22)} K α := F Q ( D α ) L 2 ( ν α ) (22) is the closure of the range of F Q F_Q F Q on D α D_\alpha D α .
Lemma 3.11 (Positivity and the unitary extension). The form (21) is positive definite on D α D_\alpha D α . The restriction of F Q F_Q F Q to D α D_\alpha D α extends uniquely to a unitary map F Q : E α → K α F_Q:E_\alpha\to K_\alpha F Q : E α → K α , denoted by the same symbol.
Proof. If the norm of f ∈ D α f \in D_\alpha f ∈ D α vanishes, then F Q f = 0 F_Q f = 0 F Q f = 0 ν α \nu_\alpha ν α -almost everywhere; continuity of F Q f F_Q f F Q f and full support of ν α \nu_\alpha ν α make it zero everywhere, and the Fourier transform determines finite complex Borel measures on a separable Hilbert space (finite-dimensional projections determine all cylinder sets, which generate B ( H ) \mathcal{B}(\mathcal{H}) B ( H ) ), so f μ Q = 0 f\mu_Q = 0 f μ Q = 0 and f = 0 f = 0 f = 0 in L 2 ( μ Q ) L^2(\mu_Q) L 2 ( μ Q ) . Thus F Q F_Q F Q is an isometry from the pre-Hilbert space D α D_\alpha D α onto a dense subspace of K α K_\alpha K α , and it extends to the completion. □
The completion matters: the norm of E α E_\alpha E α need not control the L 2 ( μ Q ) L^2(\mu_Q) L 2 ( μ Q ) -norm, so elements of E α E_\alpha E α are abstract limits, while elements of the dense subspace D α D_\alpha D α are functions. On this core the extended F Q F_Q F Q agrees with f μ Q ^ \widehat{f\mu_Q} f μ Q as an L 2 ( ν α ) L^2(\nu_\alpha) L 2 ( ν α ) equivalence class. For a general element of E α E_\alpha E α , the extension supplies such a class; no pointwise Gaussian integral representation is assumed.
Lemma 3.12 (Gaussian decay with polynomial weights). For every t > 0 t > 0 t > 0 , α > 0 \alpha> 0 α > 0 , and integer m ≥ 0 m \ge0 m ≥ 0 ,
∫ H ∥ ξ ∥ 2 m e − t ⟨ Q ξ , ξ ⟩ ν α ( d ξ ) < ∞ . (23) \int_{\mathcal{H}} \|\xi\|^{2m}e^{-t\langle Q\xi,\xi\rangle}\,\nu_\alpha(d\xi) < \infty. \tag*{(23)} ∫ H ∥ ξ ∥ 2 m e − t ⟨ Q ξ , ξ ⟩ ν α ( d ξ ) < ∞. (23) Consequently, if f ∈ L 2 ( μ Q ) f \in L^2(\mu_Q) f ∈ L 2 ( μ Q ) and ∣ F Q f ( ξ ) ∣ ≤ C ( 1 + ∥ ξ ∥ ) p e − t ⟨ Q ξ , ξ ⟩ / 2 |F_Q f(\xi)| \le C(1+\|\xi\|)^p e^{-t\langle Q\xi,\xi\rangle/2} ∣ F Q f ( ξ ) ∣ ≤ C ( 1 + ∥ ξ ∥ ) p e − t ⟨ Q ξ , ξ ⟩ /2 for some C , p , t > 0 C,p,t > 0 C , p , t > 0 , then f ∈ D α f \in D_\alpha f ∈ D α for every α > 0 \alpha> 0 α > 0 .
Proof. Let θ j > 0 \theta_j > 0 θ j > 0 be the eigenvalues of the injective trace-class operator P 1 / 2 Q P 1 / 2 P^{1/2}QP^{1/2} P 1/2 Q P 1/2 . Gaussian integration gives, for u > 0 u > 0 u > 0 ,
∫ H e − u ⟨ Q ξ , ξ ⟩ μ 2 s P ( d ξ ) = ∏ j ( 1 + 4 s u θ j ) − 1 / 2 ; (24) \int_{\mathcal{H}} e^{-u\langle Q\xi,\xi\rangle}\,\mu_{2sP}(d\xi) = \prod_j(1+4su\theta_j)^{-1/2}; \tag*{(24)} ∫ H e − u ⟨ Q ξ , ξ ⟩ μ 2 s P ( d ξ ) = j ∏ ( 1 + 4 s u θ j ) − 1/2 ; (24)
no commutativity of P P P and Q Q Q is required, since under μ 2 s P \mu_{2sP} μ 2 s P the quadratic form ⟨ Q ξ , ξ ⟩ \langle Q\xi,\xi\rangle ⟨ Q ξ , ξ ⟩ has the law of 2 s ∑ j θ j Z j 2 2s\sum_j\theta_j Z_j^2 2 s ∑ j θ j Z j 2 . By (7) it suffices to bound ∫ H ∥ ξ ∥ 2 m e − t ⟨ Q ξ , ξ ⟩ μ 2 s P ( d ξ ) \int_{\mathcal{H}}\|\xi\|^{2m}e^{-t\langle Q\xi,\xi\rangle}\,\mu_{2sP}(d\xi) ∫ H ∥ ξ ∥ 2 m e − t ⟨ Q ξ , ξ ⟩ μ 2 s P ( d ξ ) . Cauchy–Schwarz, (6) with ξ = 2 s X \xi=\sqrt{2s}X ξ = 2 s X , and (24) with u = 2 t u=2t u = 2 t give
∫ H ∥ ξ ∥ 2 m e − t ⟨ Q ξ , ξ ⟩ μ 2 s P ( d ξ ) ≤ ( 2 s ) m ( ( tr P ) 2 m ( 4 m − 1 ) ! ! ) 1 / 2 ∏ j ( 1 + 8 s t θ j ) − 1 / 4 . \int_{\mathcal{H}}\|\xi\|^{2m}e^{-t\langle Q\xi,\xi\rangle}\,\mu_{2sP}(d\xi)\leq(2s)^m\bigl((\operatorname{tr}P)^{2m}(4m-1)!!\bigr)^{1/2}\prod_j(1+8st\theta_j)^{-1/4}. ∫ H ∥ ξ ∥ 2 m e − t ⟨ Q ξ , ξ ⟩ μ 2 s P ( d ξ ) ≤ ( 2 s ) m ( ( tr P ) 2 m ( 4 m − 1 )!! ) 1/2 j ∏ ( 1 + 8 s t θ j ) − 1/4 .
For s ≤ 1 s\leq1 s ≤ 1 the product is at most one and the bound is C s m Cs^m C s m . For s ≥ 1 s\geq1 s ≥ 1 , retaining k k k factors gives ∏ j ≤ k ( 8 s t θ j ) − 1 / 4 = C k s − k / 4 \prod_{j\leq k}(8st\theta_j)^{-1/4}=C_ks^{-k/4} ∏ j ≤ k ( 8 s t θ j ) − 1/4 = C k s − k /4 . Choosing k > 4 m + 2 α k>4m+2\alpha k > 4 m + 2 α , both parts are integrable against s α / 2 − 1 d s s^{\alpha/2-1}ds s α /2 − 1 d s , proving (23) . For the consequence, ∣ F Q f ∣ 2 ≤ C 2 ( 1 + ∥ ξ ∥ ) 2 p e − t ⟨ Q ξ , ξ ⟩ |F_Qf|^2\leq C^2(1+\|\xi\|)^{2p}e^{-t\langle Q\xi,\xi\rangle} ∣ F Q f ∣ 2 ≤ C 2 ( 1 + ∥ ξ ∥ ) 2 p e − t ⟨ Q ξ , ξ ⟩ is integrable by the first part. □
Example 3.13 (Elements of D α \mathcal{D}_\alpha D α ). The constant function 1 1 1 has F Q 1 ( ξ ) = e − ⟨ Q ξ , ξ ⟩ / 2 F_Q1(\xi)=e^{-\langle Q\xi,\xi\rangle/2} F Q 1 ( ξ ) = e − ⟨ Q ξ , ξ ⟩ /2 , so 1 ∈ D α 1\in\mathcal{D}_\alpha 1 ∈ D α for every α > 0 \alpha>0 α > 0 and E α ≠ { 0 } \mathcal{E}_\alpha\ne\{0\} E α = { 0 } .
Proof. The Gaussian characteristic functional gives F Q 1 F_Q1 F Q 1 , and Lemma 3.12 with m = 0 m=0 m = 0 , p = 0 p=0 p = 0 , t = 1 t=1 t = 1 gives 1 ∈ D α 1\in\mathcal{D}_\alpha 1 ∈ D α . □
Further examples of elements of D α \mathcal{D}_\alpha D α will be the non-cylindrical Gaussian targets and Gaussian-activation operator layers treated in Section 7.
Plancherel identity, closed range, and injectivity
The following theorem is the first of the main results: it is the Plancherel identity of the ridgelet transform in infinite dimension.
Theorem 3.14 (Plancherel identity and injectivity). Let α > 0 \alpha>0 α > 0 .
(i) For f , g ∈ D α f,g\in\mathcal{D}_\alpha f , g ∈ D α and α \alpha α -admissible ρ 1 , ρ 2 \rho_1,\rho_2 ρ 1 , ρ 2 , the transforms R ρ 1 f R_{\rho_1}f R ρ 1 f and R ρ 2 g R_{\rho_2}g R ρ 2 g belong to L 2 ( λ α ) L^2(\lambda_\alpha) L 2 ( λ α ) , and
⟨ R ρ 1 f , R ρ 2 g ⟩ L 2 ( λ α ) = ( ( ρ 1 , ρ 2 ) ) α ⟨ f , g ⟩ E α . (25) \langle R_{\rho_1}f,R_{\rho_2}g\rangle_{L^2(\lambda_\alpha)}=((\rho_1,\rho_2))_\alpha\langle f,g\rangle_{\mathcal{E}_\alpha}. \tag*{(25)} ⟨ R ρ 1 f , R ρ 2 g ⟩ L 2 ( λ α ) = (( ρ 1 , ρ 2 ) ) α ⟨ f , g ⟩ E α . (25) (ii) An α \alpha α -admissible ρ \rho ρ determines a unique bounded extension R ρ : E α → L 2 ( λ α ) R_\rho:\mathcal{E}_\alpha\to L^2(\lambda_\alpha) R ρ : E α → L 2 ( λ α ) with ∥ R ρ f ∥ 2 = ( ( ρ , ρ ) ) α ∥ f ∥ E α 2 \|R_\rho f\|^2=((\rho,\rho))_\alpha\|f\|_{\mathcal{E}_\alpha}^2 ∥ R ρ f ∥ 2 = (( ρ , ρ ) ) α ∥ f ∥ E α 2 ; its range is closed, and R ρ = W ρ F Q R_\rho=W_\rho F_Q R ρ = W ρ F Q .
(iii) If ρ \rho ρ is α \alpha α -admissible and f ∈ L 1 ( H , μ Q ) f\in L^1(\mathcal{H},\mu_Q) f ∈ L 1 ( H , μ Q ) , then R ρ f = 0 R_\rho f=0 R ρ f = 0 λ α \lambda_\alpha λ α -almost everywhere implies f = 0 f=0 f = 0 μ Q \mu_Q μ Q -almost everywhere.
Proof. By Lemma 3.5, R ρ j f ( a , ⋅ ) ∈ L 2 ( R ) R_{\rho_j}f(a,\cdot)\in L^2(\mathbb{R}) R ρ j f ( a , ⋅ ) ∈ L 2 ( R ) for every a a a and j = 1 , 2 j=1,2 j = 1 , 2 , so the one-dimensional Plancherel identity ∥ u ∥ L 2 ( R ) 2 = 1 2 π ∥ u ^ ∥ L 2 ( R ) 2 \|u\|_{L^2(\mathbb{R})}^2=\frac{1}{2\pi}\|\widehat{u}\|_{L^2(\mathbb{R})}^2 ∥ u ∥ L 2 ( R ) 2 = 2 π 1 ∥ u ∥ L 2 ( R ) 2 applies in the bias. Tonelli, (15) , and (9) give, for j = 1 , 2 j=1,2 j = 1 , 2 ,
∥ R ρ j f ∥ L 2 ( λ α ) 2 = 1 2 π ∫ R ∣ ρ j † ( ω ) ∣ 2 ∫ H ∣ F Q f ( ω a ) ∣ 2 ν α ( d a ) d ω = ( ( ρ j , ρ j ) ) α ∥ f ∥ E α 2 < ∞ , \|R_{\rho_j}f\|_{L^2(\lambda_\alpha)}^2=\frac{1}{2\pi}\int_{\mathbb{R}}|\rho_j^\dagger(\omega)|^2\int_{\mathcal{H}}|F_Qf(\omega a)|^2\,\nu_\alpha(da)\,d\omega=((\rho_j,\rho_j))_\alpha\|f\|_{\mathcal{E}_\alpha}^2<\infty, ∥ R ρ j f ∥ L 2 ( λ α ) 2 = 2 π 1 ∫ R ∣ ρ j † ( ω ) ∣ 2 ∫ H ∣ F Q f ( ω a ) ∣ 2 ν α ( d a ) d ω = (( ρ j , ρ j ) ) α ∥ f ∥ E α 2 < ∞ , where the factor 1 2 π \frac{1}{2\pi} 2 π 1 is the Plancherel normalization absorbed in (12); the same holds for g g g . Weighted Cauchy–Schwarz bounds 1 2 π ∫ R ∣ ρ 1 † ρ 2 † ∣ ∣ ω ∣ − α d ω \frac{1}{2\pi}\int_{\mathbb{R}}|\rho_1^\dagger\rho_2^\dagger||\omega|^{-\alpha}d\omega 2 π 1 ∫ R ∣ ρ 1 † ρ 2 † ∣∣ ω ∣ − α d ω by ( ( ( ρ 1 , ρ 1 ) ) α ( ( ρ 2 , ρ 2 ) ) α ) 1 / 2 (((\rho_1,\rho_1))_\alpha((\rho_2,\rho_2))_\alpha)^{1/2} ((( ρ 1 , ρ 1 ) ) α (( ρ 2 , ρ 2 ) ) α ) 1/2 , and ∫ H ∣ F Q f F Q g ∣ d ν α ≤ ∥ f ∥ E α ∥ g ∥ E α \int_{\mathcal{H}}|F_QfF_Qg|d\nu_\alpha\leq\|f\|_{\mathcal{E}_\alpha}\|g\|_{\mathcal{E}_\alpha} ∫ H ∣ F Q f F Q g ∣ d ν α ≤ ∥ f ∥ E α ∥ g ∥ E α . These estimates justify Fubini in the cross identity and separate the frequency integral from the inner product of E α \mathcal{E}_\alpha E α , proving (25) . The norm identity extends R ρ R_\rho R ρ from the dense core D α \mathcal{D}_\alpha D α to its completion, an isometry up to a nonzero scalar has closed range, and R ρ = W ρ F Q R_\rho=W_\rho F_Q R ρ = W ρ F Q holds on the core by (19) and extends by continuity of both sides (Lemma 3.8).
(iii). For f ∈ L 1 ( μ Q ) f\in L^1(\mu_Q) f ∈ L 1 ( μ Q ) , Lemma 3.5 gives R ρ f ( a , ⋅ ) ∈ L 1 ( R ) R_\rho f(a,\cdot)\in L^1(\mathbb{R}) R ρ f ( a , ⋅ ) ∈ L 1 ( R ) for every a a a . Tonelli applied to the hypothesis shows that, for ν α \nu_\alpha ν α -almost every a a a , this bias function vanishes almost everywhere, so its
Fourier transform is zero at every frequency. Choose ω 0 ≠ 0 \omega_0 \ne0 ω 0 = 0 with ρ ♯ ( ω 0 ) ≠ 0 \rho^\sharp(\omega_0) \ne0 ρ ♯ ( ω 0 ) = 0 ; then (15) and (9) imply F Q f = 0 F_Qf=0 F Q f = 0 ν α \nu_\alpha ν α -almost everywhere. No finite spectral-norm assumption on f f f is used. Since F Q f F_Qf F Q f is continuous and supp ν α = H \operatorname{supp}\nu_\alpha=\mathcal{H} supp ν α = H , it vanishes everywhere, and Fourier uniqueness for finite measures, as in the proof of Lemma 3.11 , gives f μ Q = 0 f\mu_Q=0 f μ Q = 0 . □
Admissibility in Theorem 3.14 is a condition on the analysis filter. It imposes nothing on a synthesis activation such as ReLU, which is paired with the filter through the distributional constant of § 5 .
Examples of analysis filters
The analysis filter and the synthesis activation play different roles. ReLU is a synthesis activation; its Fourier transform is distributional and it does not satisfy (10) . The following Schwartz filters satisfy the self-admissibility condition and illustrate the distinction between admissibility and band-pass support.
Example 3.15 (A band-pass filter for every α > 0 \alpha>0 α > 0 ). Let η ( u ) = exp ( − 1 / ( 1 − u 2 ) ) \eta(u)=\exp(-1/(1-u^2)) η ( u ) = exp ( − 1/ ( 1 − u 2 )) for ∣ u ∣ < 1 |u|<1 ∣ u ∣ < 1 and η ( u ) = 0 \eta(u)=0 η ( u ) = 0 otherwise, and define a real even ρ b p \rho_{\mathrm{bp}} ρ bp by ρ b p ♯ ( ω ) : = − η ( 2 ∣ ω ∣ − 3 ) \rho_{\mathrm{bp}}^\sharp(\omega):=-\eta(2|\omega|-3) ρ bp ♯ ( ω ) := − η ( 2∣ ω ∣ − 3 ) . Its Fourier transform is smooth, nonpositive, nonzero, and supported in { 1 ≤ ∣ ω ∣ ≤ 2 } \{1\leq|\omega|\leq2\} { 1 ≤ ∣ ω ∣ ≤ 2 } , so ρ b p ∈ S ( R ) \rho_{\mathrm{bp}}\in\mathcal{S}(\mathbb{R}) ρ bp ∈ S ( R ) is real and even, satisfies (11) , and is α \alpha α -admissible for every α > 0 \alpha>0 α > 0 ; the sign makes it admissible for ReLU synthesis in the sense of Corollary 5.5. Multiplying by ( ( ρ b p , ρ b p ) ) α − 1 / 2 ((\rho_{\mathrm{bp}},\rho_{\mathrm{bp}}))_\alpha^{-1/2} (( ρ bp , ρ bp ) ) α − 1/2 normalizes the admissibility constant to one.
Proof. The bump η \eta η and all of its derivatives vanish at ∣ u ∣ = 1 |u|=1 ∣ u ∣ = 1 , and the support stays away from ω = 0 \omega=0 ω = 0 , so the absolute value causes no loss of smoothness. Fourier inversion maps C ∞ C^\infty C ∞ into S \mathcal{S} S , and even real Fourier data give an even real inverse. On the compact support, ∣ ω ∣ − α |\omega|^{-\alpha} ∣ ω ∣ − α is bounded above and below, so (10) is finite and positive. □
Example 3.16 (Mexican hat). The Schwartz function ρ M H ( t ) : = ( 1 − t 2 / 2 ) e − t 2 / 2 \rho_{\mathrm{MH}}(t):=(1-t^2/2)e^{-t^2/2} ρ MH ( t ) := ( 1 − t 2 /2 ) e − t 2 /2 has ρ M H ♯ ( ω ) = 2 π ω 2 e − ω 2 / 2 \rho_{\mathrm{MH}}^\sharp(\omega)=\sqrt{2\pi}\omega^2e^{-\omega^2/2} ρ MH ♯ ( ω ) = 2 π ω 2 e − ω 2 /2 . For α > 0 \alpha>0 α > 0 it is α \alpha α -admissible in the sense of (10) exactly when α < 5 \alpha<5 α < 5 , with ( ( ρ M H , ρ M H ) ) α = Γ ( ( 5 − α ) / 2 ) ((\rho_{\mathrm{MH}},\rho_{\mathrm{MH}}))_\alpha=\Gamma((5-\alpha)/2) (( ρ MH , ρ MH ) ) α = Γ (( 5 − α ) /2 ) ; in particular ( ( ρ M H , ρ M H ) ) 1 = 1 ((\rho_{\mathrm{MH}},\rho_{\mathrm{MH}}))_1=1 (( ρ MH , ρ MH ) ) 1 = 1 . It is not band pass, so Theorems 3.14 and 4.8 and Theorem 4.5 (ii) apply when α < 5 \alpha<5 α < 5 . The band-pass statements Theorem 4.5 (iii) and Theorem 5.4 do not apply to this filter.
Proof. If g ( t ) = e − t 2 / 2 g(t)=e^{-t^2/2} g ( t ) = e − t 2 /2 , then ρ M H = − g ′ ′ \rho_{\mathrm{MH}}=-g'' ρ MH = − g ′′ , and the differentiation rule gives the Fourier transform. Then ( ( ρ M H , ρ M H ) ) α = ∫ R ∣ ω ∣ 4 − α e − ω 2 d ω = Γ ( ( 5 − α ) / 2 ) ((\rho_{\mathrm{MH}},\rho_{\mathrm{MH}}))_\alpha=\int_{\mathbb{R}}|\omega|^{4-\alpha}e^{-\omega^2}\,d\omega=\Gamma((5-\alpha)/2) (( ρ MH , ρ MH ) ) α = ∫ R ∣ ω ∣ 4 − α e − ω 2 d ω = Γ (( 5 − α ) /2 ) , convergent at zero exactly when 4 − α > − 1 4-\alpha>-1 4 − α > − 1 , that is, when α < 5 \alpha<5 α < 5 . Here α > 0 \alpha>0 α > 0 is the standing restriction under which ν α \nu_\alpha ν α is defined; for α ≤ 0 \alpha\leq0 α ≤ 0 the integral converges as well. □
More generally, if ρ ∈ S ( R ) \rho\in\mathcal{S}(\mathbb{R}) ρ ∈ S ( R ) has k k k vanishing moments, then ρ ♯ ( ω ) = O ( ∣ ω ∣ k ) \rho^\sharp(\omega)=O(|\omega|^k) ρ ♯ ( ω ) = O ( ∣ ω ∣ k ) at zero and (10) is finite whenever α < 2 k + 1 \alpha<2k+1 α < 2 k + 1 . The band-pass condition (11) supplies arbitrarily many vanishing moments at once and is useful for the smooth criterion. The Sobolev criterion below permits filters whose Fourier support meets the origin.
Representation and reconstruction
We construct coefficients for targets with spectral densities and derive integral representations under the stated integrability conditions. We then identify the correction induced by the weights and give reconstruction formulas, including the vector-valued extension.
Targets with a spectral density
The Fourier-slice identity, read backwards, produces a network from a spectral density. For a Borel function g ∈ L 1 ( ν α ) g \in L^1(\nu_\alpha) g ∈ L 1 ( ν α ) put
f g ( x ) : = ∫ H e i ⟨ x , ξ ⟩ g ( ξ ) ν α ( d ξ ) , x ∈ H . (26) f_g(x) := \int_{\mathcal{H}} e^{i\langle x,\xi\rangle}g(\xi)\,\nu_\alpha(d\xi), \qquad x \in\mathcal{H}. \tag*{(26)} f g ( x ) := ∫ H e i ⟨ x , ξ ⟩ g ( ξ ) ν α ( d ξ ) , x ∈ H . (26)
Thus f g ( x ) = g ν α ‾ ( − x ) f_g(x)=\overline{g\nu_\alpha}(-x) f g ( x ) = g ν α ( − x ) . We call g ν α g\nu_\alpha g ν α the spectral measure of f g f_g f g , and g g g its spectral density with respect to ν α \nu_\alpha ν α . The class of such targets is the analogue, for the reference measure ν α \nu_\alpha ν α , of the class of functions with an integrable spectral measure used by Barron [5 ] . Its density in C ( H ) C(\mathcal{H}) C ( H ) , proved in Theorem 6.8 , uses full support and finiteness of ν α \nu_\alpha ν α on bounded sets. The elementary facts about an individual spectral target require neither condition.
For the band-pass representation in part (iii) of the next theorem, write g a ( ω ) : = g ( ω a ) g_a(\omega):=g(\omega a) g a ( ω ) := g ( ω a ) . For a ≠ 0 a\ne0 a = 0 , this is the restriction of g g g to the line { ω a : ω ∈ R } \{\omega a:\omega\in\mathbb{R}\} { ω a : ω ∈ R } through the origin. We impose smoothness on g a g_a g a and weighted integrability on its derivatives. For a band-pass ρ \rho ρ , fix a symmetric compact set I ⊂ R ∖ { 0 } I\subset\mathbb{R}\setminus\{0\} I ⊂ R ∖ { 0 } containing supp ρ \operatorname{supp}\rho supp ρ . The other parts of the theorem do not require this regularity condition.
Lemma 4.1 (An integrable spectral density defines a unique continuous target). Let ν \nu ν be a sigma-finite Borel measure on H \mathcal{H} H , let Y \mathcal{Y} Y be a separable complex Hilbert space, and let g ∈ L 1 ( ν ; Y ) g\in L^1(\nu;\mathcal{Y}) g ∈ L 1 ( ν ; Y ) . Then f g ( x ) : = ∫ H e i ⟨ x , ξ ⟩ g ( ξ ) ν ( d ξ ) f_g(x):=\int_{\mathcal{H}}e^{i\langle x,\xi\rangle}g(\xi)\nu(d\xi) f g ( x ) := ∫ H e i ⟨ x , ξ ⟩ g ( ξ ) ν ( d ξ ) belongs to C b ( H ; Y ) C_b(\mathcal{H};\mathcal{Y}) C b ( H ; Y ) , satisfies ∥ f g ∥ ∞ ≤ ∥ g ∥ L 1 ( ν ; Y ) \|f_g\|_\infty\le\|g\|_{L^1(\nu;\mathcal{Y})} ∥ f g ∥ ∞ ≤ ∥ g ∥ L 1 ( ν ; Y ) , and determines g g g uniquely up to ν \nu ν -almost-everywhere equality. Only the Fourier transform of the finite vector measure g ν g\nu g ν enters: boundedness, continuity, and uniqueness follow from g ∈ L 1 ( ν ; Y ) g\in L^1(\nu;\mathcal{Y}) g ∈ L 1 ( ν ; Y ) alone, and neither the Gaussian input measure μ Q \mu_Q μ Q nor the analysis filter ρ \rho ρ plays a role.
Proof. The norm bound is the integral triangle inequality, and continuity follows by dominated convergence with majorant ∥ g ( ξ ) ∥ Y \|g(\xi)\|_{\mathcal{Y}} ∥ g ( ξ ) ∥ Y . If f g = 0 f_g=0 f g = 0 , pair the finite vector measure g ν g\nu g ν with any vector of a countable dense subset of Y \mathcal{Y} Y . Each resulting finite complex measure has zero Fourier transform and hence is zero: finite-dimensional Fourier uniqueness determines its cylinder sets, which generate the Borel sigma-algebra of the separable Hilbert space H \mathcal{H} H . The corresponding scalar densities vanish almost everywhere. Intersecting the countably many full-measure sets and using continuity of the pairing gives g = 0 g=0 g = 0 almost everywhere. Apply this to a difference for uniqueness. □ \square □
Definition 4.2 (Regularity along rays). A bounded Borel function g : H → C g:\mathcal{H}\to\mathbb{C} g : H → C is regular along rays if for every a ∈ H a\in\mathcal{H} a ∈ H its restriction g a ( ω ) = g ( ω a ) g_a(\omega)=g(\omega a) g a ( ω ) = g ( ω a ) is C ∞ C^\infty C ∞ on a neighbourhood of I I I , and for every integer $m\ge 0
M m ( g ) : = ∫ H ( 1 + ∥ a ∥ ) m + 2 max k ≤ m sup ω ∈ I ∣ ∂ ω k g a ( ω ) ∣ ν α ( d a ) < ∞ . (27) M_m(g):=\int_{\mathcal{H}}(1+\|a\|)^{m+2}\max_{k\le m}\sup_{\omega\in I}\left|\partial_\omega^k g_a(\omega)\right|\,\nu_\alpha(da)<\infty. \tag*{(27)} M m ( g ) := ∫ H ( 1 + ∥ a ∥ ) m + 2 k ≤ m max ω ∈ I sup ∂ ω k g a ( ω ) ν α ( d a ) < ∞. (27)
The next lemma is stated for a density with values in a separable complex Hilbert space Y \mathcal{Y} Y , as needed for the vector-valued extension in § 4.3 ; the scalar case is Y = C \mathcal{Y}=\mathbb{C} Y = C . The coefficient formula (17) applies verbatim to such g g g : the scalar factor ρ ♯ ( − ω ) e i ω b \rho^\sharp(-\omega)e^{i\omega b} ρ ♯ ( − ω ) e iω b multiplies the vector g ( ω a ) g(\omega a) g ( ω a ) , and the integral is a Bochner integral,
γ g ( a , b ) = 1 2 π ∫ R ρ ♯ ( − ω ) e i ω b g ( ω a ) d ω ∈ Y . \gamma_g(a,b)=\frac{1}{2\pi}\int_{\mathbb{R}}\rho^\sharp(-\omega)e^{i\omega b}g(\omega a)\,d\omega\in\mathcal{Y}. γ g ( a , b ) = 2 π 1 ∫ R ρ ♯ ( − ω ) e iω b g ( ω a ) d ω ∈ Y .
Accordingly W ρ W_\rho W ρ acts on L 2 ( ν α ; Y ) L^2(\nu_\alpha;\mathcal{Y}) L 2 ( ν α ; Y ) through the Y \mathcal{Y} Y -valued partial Fourier transform of Lemma 3.6 , and the proof of Lemma 3.8 applies with ∥ ⋅ ∥ Y \|\cdot\|_{\mathcal{Y}} ∥ ⋅ ∥ Y in place of the absolute value.
Lemma 4.3 (Finite-order coefficient decay and moments). Fix a band-pass ρ \rho ρ , a symmetric compact I ⊂ R ∖ { 0 } I \subset\mathbb{R} \setminus\{0\} I ⊂ R ∖ { 0 } containing supp ρ ♯ \operatorname{supp}\rho^\sharp supp ρ ♯ , and an integer r ≥ 0 r \ge0 r ≥ 0 . Let Y \mathcal{Y} Y be a separable complex Hilbert space and let g : H → Y g:\mathcal{H}\to\mathcal{Y} g : H → Y be bounded and Borel, with ω ↦ g ( ω a ) \omega\mapsto g(\omega a) ω ↦ g ( ω a ) of class C r + 2 C^{r+2} C r + 2 near I I I for every a a a . Set
D m ( a ) : = max k ≤ m sup ω ∈ I ∥ ∂ ω k g ( ω a ) ∥ Y , A m , r ( g ) : = ∫ H ( 1 + ∥ a ∥ ) r D m ( a ) ν α ( d a ) . D_m(a):=\max_{k\le m}\sup_{\omega\in I}\|\partial_\omega^k g(\omega a)\|_{\mathcal{Y}},\qquad A_{m,r}(g):=\int_{\mathcal{H}}(1+\|a\|)^rD_m(a)\,\nu_\alpha(da). D m ( a ) := k ≤ m max ω ∈ I sup ∥ ∂ ω k g ( ω a ) ∥ Y , A m , r ( g ) := ∫ H ( 1 + ∥ a ∥ ) r D m ( a ) ν α ( d a ) . If A r + 2 , r ( g ) < ∞ A_{r+2,r}(g)<\infty A r + 2 , r ( g ) < ∞ , then g ∈ L 1 ∩ L 2 ( ν α ; Y ) g\in L^1\cap L^2(\nu_\alpha;\mathcal{Y}) g ∈ L 1 ∩ L 2 ( ν α ; Y ) , the inverse integral (17) defines a jointly measurable representative of W ρ g W_\rho g W ρ g , and
( 1 + ∣ b ∣ ) r + 2 ∥ γ g ( a , b ) ∥ Y ≤ C ρ , I , r D r + 2 ( a ) , (28) (1+|b|)^{r+2}\|\gamma_g(a,b)\|_{\mathcal{Y}}\le C_{\rho,I,r}D_{r+2}(a), \tag*{(28)} ( 1 + ∣ b ∣ ) r + 2 ∥ γ g ( a , b ) ∥ Y ≤ C ρ , I , r D r + 2 ( a ) , (28) ∫ H × R ( 1 + ∥ a ∥ + ∣ b ∣ ) r ∥ γ g ( a , b ) ∥ Y d λ α ≤ c ρ , I , r A r + 2 , r ( g ) . (29) \int_{\mathcal{H}\times\mathbb{R}}(1+\|a\|+|b|)^r\|\gamma_g(a,b)\|_{\mathcal{Y}}\,d\lambda_\alpha\le c_{\rho,I,r}A_{r+2,r}(g). \tag*{(29)} ∫ H × R ( 1 + ∥ a ∥ + ∣ b ∣ ) r ∥ γ g ( a , b ) ∥ Y d λ α ≤ c ρ , I , r A r + 2 , r ( g ) . (29) The constants are independent of g g g and of the dimension of Y \mathcal{Y} Y .
Proof. The functions under the suprema are jointly Borel, since derivatives are limits of difference quotients of g ( ω a ) g(\omega a) g ( ω a ) . Continuity in ω \omega ω lets us take the suprema over a countable dense subset of I I I , so D m D_m D m is measurable. Fix any ω 0 ∈ I \omega_0\in I ω 0 ∈ I . Homogeneity bounds ∫ ∥ g ∥ d ν α \int\|g\|\,d\nu_\alpha ∫ ∥ g ∥ d ν α by ∣ ω 0 ∣ α ∫ D 0 d ν α < ∞ |\omega_0|^\alpha\int D_0\,d\nu_\alpha<\infty ∣ ω 0 ∣ α ∫ D 0 d ν α < ∞ ; boundedness of g g g gives its square integrability.
Put m = r + 2 m=r+2 m = r + 2 and h a ( ω ) = ρ ♯ ( − ω ) g ( ω a ) h_a(\omega)=\rho^\sharp(-\omega)g(\omega a) h a ( ω ) = ρ ♯ ( − ω ) g ( ω a ) . For each a a a this is a compactly supported C m C^m C m , Y \mathcal{Y} Y -valued function. Extension by zero outside the neighbourhood of I I I is C m C^m C m , because all derivatives of ρ ♯ \rho^\sharp ρ ♯ vanish at the boundary of its support. Leibniz’ rule and symmetry of I I I give ∥ h a ∥ L 1 + ∥ h a ( m ) ∥ L 1 ≤ C ρ , I , m ′ D m ( a ) \|h_a\|_{L^1}+\|h_a^{(m)}\|_{L^1}\le C'_{\rho,I,m}D_m(a) ∥ h a ∥ L 1 + ∥ h a ( m ) ∥ L 1 ≤ C ρ , I , m ′ D m ( a ) . The inverse Fourier integral is jointly measurable by parameter integration and is the coefficient by Lemma 3.8. For ∣ b ∣ ≤ 1 |b|\le1 ∣ b ∣ ≤ 1 bound it by ( 2 π ) − 1 ∥ h a ∥ 1 (2\pi)^{-1}\|h_a\|_1 ( 2 π ) − 1 ∥ h a ∥ 1 . For ∣ b ∣ > 1 |b|>1 ∣ b ∣ > 1 , integration by parts m m m times, with vanishing boundary terms, bounds it by ( 2 π ) − 1 ∣ b ∣ − m ∥ h a ( m ) ∥ 1 (2\pi)^{-1}|b|^{-m}\|h_a^{(m)}\|_1 ( 2 π ) − 1 ∣ b ∣ − m ∥ h a ( m ) ∥ 1 . These inequalities prove (28). Finally, 1 + ∥ a ∥ + ∣ b ∣ ≤ ( 1 + ∥ a ∥ ) ( 1 + ∣ b ∣ ) 1+\|a\|+|b|\le(1+\|a\|)(1+|b|) 1 + ∥ a ∥ + ∣ b ∣ ≤ ( 1 + ∥ a ∥ ) ( 1 + ∣ b ∣ ) and ∫ R ( 1 + ∣ b ∣ ) − 2 d b = 2 \int_{\mathbb{R}}(1+|b|)^{-2}\,db=2 ∫ R ( 1 + ∣ b ∣ ) − 2 d b = 2 give (29) by Tonelli.
In particular, C 4 C^4 C 4 regularity of g a ( ω ) = g ( ω a ) g_a(\omega)=g(\omega a) g a ( ω ) = g ( ω a ) and A 4 , 2 ( g ) < ∞ A_{4,2}(g)<\infty A 4 , 2 ( g ) < ∞ suffice for the second parameter moment. This finite-order estimate does not by itself establish an integral representation with a continuous activation of polynomial growth. For a density satisfying Definition 4.2, A r + 2 , r ( g ) ≤ M r + 2 ( g ) A_{r+2,r}(g)\le M_{r+2}(g) A r + 2 , r ( g ) ≤ M r + 2 ( g ) , so every parameter moment is finite.
Lemma 4.4 (Examples satisfying the spectral regularity condition). The following functions satisfy Definition 4.2 for every band-pass ρ \rho ρ .
(a) g ( ξ ) = q ( ξ ) e − κ ( ξ ) / 2 g(\xi)=q(\xi)e^{-\kappa(\xi)/2} g ( ξ ) = q ( ξ ) e − κ ( ξ ) /2 , where κ ( ξ ) = ⟨ S ξ , ξ ⟩ \kappa(\xi)=\langle S\xi,\xi\rangle κ ( ξ ) = ⟨ S ξ , ξ ⟩ for a bounded positive operator S S S with S ≥ θ Q S\ge\theta Q S ≥ θ Q for some θ > 0 \theta>0 θ > 0 , and where q q q is a polynomial in κ ( ξ ) \kappa(\xi) κ ( ξ ) and in finitely many bounded linear functionals ℓ 1 , … , ℓ k \ell_1,\ldots,\ell_k ℓ 1 , … , ℓ k of ξ \xi ξ , each satisfying ∣ ℓ i ( ξ ) ∣ 2 ≤ C i κ ( ξ ) |\ell_i(\xi)|^2\le C_i\kappa(\xi) ∣ ℓ i ( ξ ) ∣ 2 ≤ C i κ ( ξ ) for all ξ \xi ξ , equivalently ℓ i = ⟨ S 1 / 2 v i , ⋅ ⟩ \ell_i=\langle S^{1/2}v_i,\cdot\rangle ℓ i = ⟨ S 1/2 v i , ⋅ ⟩ for some v i ∈ H v_i\in\mathcal{H} v i ∈ H . A polynomial in κ \kappa κ alone, in particular a constant q q q , needs no further condition.
(b) g ( ξ ) = φ ( ∥ ξ − ξ 0 ∥ 2 ) g(\xi)=\varphi(\|\xi-\xi_0\|^2) g ( ξ ) = φ ( ∥ ξ − ξ 0 ∥ 2 ) with ξ 0 ∈ H \xi_0\in\mathcal{H} ξ 0 ∈ H and φ ∈ C c ∞ ( R ) \varphi\in C^\infty_c(\mathbb{R}) φ ∈ C c ∞ ( R ) ; more generally any bounded g g g that vanishes outside a bounded set, whose restrictions g a ( ω ) = g ( ω a ) g_a(\omega)=g(\omega a) g a ( ω ) = g ( ω a ) are C ∞ C^\infty C ∞ near I I I , and for which
sup ω ∈ I ∣ ∂ ω k g ( ω a ) ∣ ≤ C k ( 1 + ∥ a ∥ ) p k for all k . \sup_{\omega\in I}|\partial_\omega^k g(\omega a)|\le C_k(1+\|a\|)^{p_k}\quad\text{for all }k. ω ∈ I sup ∣ ∂ ω k g ( ω a ) ∣ ≤ C k ( 1 + ∥ a ∥ ) p k for all k . (c) Finite linear combinations of functions satisfying Definition 4.2, and Bochner integrals
∫ Ω g y m ( d y ) \int_\Omega g_y\,m(dy) ∫ Ω g y m ( d y ) of a uniformly bounded measurable family ( g y ) y ∈ Ω (g_y)_{y\in\Omega} ( g y ) y ∈ Ω of such functions over a finite measure m m m on $\Omega whose bounds in (27) are uniform in y y y on a neighbourhood of I I I , in the
following sense: there are an open set U ⊇ I U \supseteq I U ⊇ I , on which ω ↦ g y ( ω a ) \omega\mapsto g_y(\omega a) ω ↦ g y ( ω a ) is C ∞ C^\infty C ∞ for every y , a y,a y , a , and finite-valued Borel functions h k : H → [ 0 , ∞ ) h_k:\mathcal{H}\to[0,\infty) h k : H → [ 0 , ∞ ) , k ≥ 0 k\geq0 k ≥ 0 , with
sup ω ∈ U ∣ ∂ ω k g y ( ω a ) ∣ ≤ h k ( a ) for every y ∈ Ω , a ∈ H , k ≥ 0 , \sup_{\omega\in U}\left|\partial_\omega^k g_y(\omega a)\right|\leq h_k(a)\qquad\text{for every }y\in\Omega,\ a\in\mathcal{H},\ k\geq0, ω ∈ U sup ∂ ω k g y ( ω a ) ≤ h k ( a ) for every y ∈ Ω , a ∈ H , k ≥ 0 ,
∫ H ( 1 + ∥ a ∥ ) m + 2 max k ≤ m h k ( a ) ν α ( d a ) < ∞ for every integer m ≥ 0. \int_{\mathcal{H}}(1+\lVert a\rVert)^{m+2}\max_{k\leq m}h_k(a)\,\nu_\alpha(da)<\infty\qquad\text{for every integer }m\geq0. ∫ H ( 1 + ∥ a ∥ ) m + 2 k ≤ m max h k ( a ) ν α ( d a ) < ∞ for every integer m ≥ 0.
The supremum must be taken over the neighbourhood U U U , and the bound must hold for every a a a , not only ν α \nu_\alpha ν α -almost everywhere.
Proof. For (a), the domination ∣ ℓ i ( ξ ) ∣ 2 ≤ C i κ ( ξ ) |\ell_i(\xi)|^2\leq C_i\kappa(\xi) ∣ ℓ i ( ξ ) ∣ 2 ≤ C i κ ( ξ ) gives ∣ q ( ξ ) ∣ ≤ C ( 1 + κ ( ξ ) ) p |q(\xi)|\leq C(1+\kappa(\xi))^p ∣ q ( ξ ) ∣ ≤ C ( 1 + κ ( ξ ) ) p for some C C C and p p p , hence
∣ g ( ξ ) ∣ ≤ C sup t ≥ 0 ( 1 + t ) p e − t / 2 , |g(\xi)|\leq C\sup_{t\geq0}(1+t)^p e^{-t/2}, ∣ g ( ξ ) ∣ ≤ C t ≥ 0 sup ( 1 + t ) p e − t /2 ,
and g g g is bounded; without the domination g g g need not be bounded, because κ \kappa κ may be small in directions where ℓ i \ell_i ℓ i is not. The two forms of the condition are equivalent because ∣ ℓ i ( ξ ) ∣ 2 ≤ C i ∥ S 1 / 2 ξ ∥ 2 |\ell_i(\xi)|^2\leq C_i\lVert S^{1/2}\xi\rVert^2 ∣ ℓ i ( ξ ) ∣ 2 ≤ C i ∥ S 1/2 ξ ∥ 2 says that ℓ i \ell_i ℓ i factors through S 1 / 2 S^{1/2} S 1/2 as a bounded functional on the closure of its range, which the Riesz representation writes as ⟨ v i , S 1 / 2 ξ ⟩ \langle v_i,S^{1/2}\xi\rangle ⟨ v i , S 1/2 ξ ⟩ . Since κ ( ω a ) = ω 2 κ ( a ) \kappa(\omega a)=\omega^2\kappa(a) κ ( ω a ) = ω 2 κ ( a ) and ℓ i ( ω a ) = ω ℓ i ( a ) \ell_i(\omega a)=\omega\ell_i(a) ℓ i ( ω a ) = ω ℓ i ( a ) , ∂ ω k [ q ( ω a ) e − ω 2 κ ( a ) / 2 ] \partial_\omega^k[q(\omega a)e^{-\omega^2\kappa(a)/2}] ∂ ω k [ q ( ω a ) e − ω 2 κ ( a ) /2 ] is a polynomial in ω \omega ω , κ ( a ) \kappa(a) κ ( a ) , and ℓ i ( a ) \ell_i(a) ℓ i ( a ) , times e − ω 2 κ ( a ) / 2 e^{-\omega^2\kappa(a)/2} e − ω 2 κ ( a ) /2 ; on I ⊂ { r ≤ ∣ ω ∣ ≤ R } I\subset\{r\leq|\omega|\leq R\} I ⊂ { r ≤ ∣ ω ∣ ≤ R } it is bounded by C k ( 1 + ∥ a ∥ ) p k e − r 2 θ ⟨ Q a , a ⟩ / 2 C_k(1+\lVert a\rVert)^{p_k}e^{-r^2\theta\langle Qa,a\rangle/2} C k ( 1 + ∥ a ∥ ) p k e − r 2 θ ⟨ Q a , a ⟩ /2 , and Lemma 3.12 integrates this against ( 1 + ∥ a ∥ ) m + 2 ν α ( d a ) (1+\lVert a\rVert)^{m+2}\nu_\alpha(da) ( 1 + ∥ a ∥ ) m + 2 ν α ( d a ) . The same bound holds with I I I replaced by any compact set of frequencies, in particular by a compact neighbourhood of I I I , with r r r the smallest ∣ ω ∣ |\omega| ∣ ω ∣ on that set. For (b), if g g g vanishes outside the ball of radius R 0 R_0 R 0 , then ∂ ω k g ( ω a ) \partial_\omega^k g(\omega a) ∂ ω k g ( ω a ) vanishes for all ω ∈ I \omega\in I ω ∈ I unless ∥ a ∥ ≤ R 0 / r \lVert a\rVert\leq R_0/r ∥ a ∥ ≤ R 0 / r , and ν α \nu_\alpha ν α is finite on bounded sets; the polynomial bound is integrable there. For (b), a finite linear combination is handled by the triangle inequality in (27) . For the Bochner integral g : = ∫ Ω g y m ( d y ) g:=\int_\Omega g_y\,m(dy) g := ∫ Ω g y m ( d y ) , which is bounded by m ( Ω ) sup y ∥ g y ∥ ∞ m(\Omega)\sup_y\lVert g_y\rVert_\infty m ( Ω ) sup y ∥ g y ∥ ∞ , fix a ∈ H a\in\mathcal{H} a ∈ H . The dominating bounds sup ω ∈ U ∣ ∂ ω k g y ( ω a ) ∣ ≤ h k ( a ) \sup_{\omega\in U}|\partial_\omega^k g_y(\omega a)|\leq h_k(a) sup ω ∈ U ∣ ∂ ω k g y ( ω a ) ∣ ≤ h k ( a ) are finite constants in y y y , hence m m m -integrable, so differentiation under the integral sign applies on U U U : the function ω ↦ g ( ω a ) \omega\mapsto g(\omega a) ω ↦ g ( ω a ) is C ∞ C^\infty C ∞ on the neighbourhood U U U of I I I , with
∂ ω k g ( ω a ) = ∫ Ω ∂ ω k g y ( ω a ) m ( d y ) . \partial_\omega^k g(\omega a)=\int_\Omega\partial_\omega^k g_y(\omega a)\,m(dy). ∂ ω k g ( ω a ) = ∫ Ω ∂ ω k g y ( ω a ) m ( d y ) .
This is exactly what dominating bounds on a neighbourhood provide; suprema over I I I alone, or bounds valid only for ν α \nu_\alpha ν α -almost every a a a , control no difference quotient and leave ω ↦ g ( ω a ) \omega\mapsto g(\omega a) ω ↦ g ( ω a ) possibly nowhere differentiable. Taking suprema over I ⊂ U I\subset U I ⊂ U and integrating in a a a , Tonelli gives
M m ( g ) ≤ m ( Ω ) ∫ H ( 1 + ∥ a ∥ ) m + 2 max k ≤ m h k ( a ) ν α ( d a ) < ∞ M_m(g)\leq m(\Omega)\int_{\mathcal{H}}(1+\lVert a\rVert)^{m+2}\max_{k\leq m}h_k(a)\,\nu_\alpha(da)<\infty M m ( g ) ≤ m ( Ω ) ∫ H ( 1 + ∥ a ∥ ) m + 2 k ≤ m max h k ( a ) ν α ( d a ) < ∞
for every m ≥ 0. □ m\geq0. \quad\square m ≥ 0. □
Homogeneity (9) at a fixed ω ∈ I \omega\in I ω ∈ I shows that Definition 4.2 implies g ∈ L 1 ( ν α ) ∩ L 2 ( ν α ) g\in L^1(\nu_\alpha)\cap L^2(\nu_\alpha) g ∈ L 1 ( ν α ) ∩ L 2 ( ν α ) . The preceding lemma covers Gaussian-type densities and smooth densities with bounded support. The coefficient estimate above supplies integrability before any synthesis identity is invoked.
More precisely, the independent decay estimate of Lemma 4.3 shows that C 4 C^4 C 4 regularity of g a g_a g a and the weighted integral A 4 , 2 ( g ) < ∞ A_{4,2}(g)<\infty A 4 , 2 ( g ) < ∞ defined there suffice for the second parameter moment. The full condition in Definition 4.2 supplies moments of every order and will also justify the integral representation with continuous activations of polynomial growth.
The next theorem is the representation theorem announced in the introduction, the first main result of this section.
Theorem 4.5 (Integral representation of targets with a spectral density). Let α > 0 \alpha>0 α > 0 . In part (ii), let β \beta β be an α \alpha α -admissible Schwartz filter; in part (iii), let it be band pass.
For g ∈ L 1 ( ν α ) g\in L^1(\nu_\alpha) g ∈ L 1 ( ν α ) , the function f g f_g f g is bounded and continuous, ∥ f g ∥ ∞ ≤ ∥ g ∥ L 1 ( ν α ) \lVert f_g\rVert_\infty\leq\lVert g\rVert_{L^1(\nu_\alpha)} ∥ f g ∥ ∞ ≤ ∥ g ∥ L 1 ( ν α ) , and f g = 0 f_g=0 f g = 0 only if g = 0 g=0 g = 0 ν α \nu_\alpha ν α -almost everywhere.
(ii) For g ∈ L 1 ( ν α ) ∩ L 2 ( ν α ) g \in L^{1}(\nu_{\alpha}) \cap L^{2}(\nu_{\alpha}) g ∈ L 1 ( ν α ) ∩ L 2 ( ν α ) and every x ∈ H x \in\mathcal{H} x ∈ H , the iterated integral
∫ H [ ∫ R γ g ( a , b ) ρ ( ⟨ a , x ⟩ − b ) d b ] ν α ( d a ) = ( ( ρ , ρ ) ) α f g ( x ) (30) \int_{\mathcal{H}}\left[\int_{\mathbb{R}}\gamma_g(a,b)\rho(\langle a,x\rangle-b)\,db\right]\nu_{\alpha}(da)=((\rho,\rho))_{\alpha}f_g(x) \tag*{(30)} ∫ H [ ∫ R γ g ( a , b ) ρ (⟨ a , x ⟩ − b ) d b ] ν α ( d a ) = (( ρ , ρ ) ) α f g ( x ) (30)
converges absolutely. If moreover γ g ∈ L 1 ( λ α ) \gamma_g \in L^{1}(\lambda_{\alpha}) γ g ∈ L 1 ( λ α ) , the left side is the integral network S ρ [ γ g λ α ] ( x ) S_{\rho}[\gamma_g\lambda_{\alpha}](x) S ρ [ γ g λ α ] ( x ) of Definition 2.2.
(iii) Let σ ∈ S ′ ( R ) \sigma\in\mathcal{S}'(\mathbb{R}) σ ∈ S ′ ( R ) be a continuous function of polynomial growth, and let g g g satisfy Definition 4.2. For every x ∈ H x \in\mathcal{H} x ∈ H , γ g ( a , b ) σ ( ⟨ a , x ⟩ − b ) \gamma_g(a,b)\sigma(\langle a,x\rangle-b) γ g ( a , b ) σ (⟨ a , x ⟩ − b ) is absolutely integrable in the product measure λ α \lambda_{\alpha} λ α . Thus the following is an ordinary integral network with the finite coefficient measure γ g λ α \gamma_g\lambda_{\alpha} γ g λ α :
∫ H [ ∫ R γ g ( a , b ) σ ( ⟨ a , x ⟩ − b ) d b ] ν α ( d a ) = ( ( σ , ρ ) ) α f g ( x ) , (31) \int_{\mathcal{H}}\left[\int_{\mathbb{R}}\gamma_g(a,b)\sigma(\langle a,x\rangle-b)\,db\right]\nu_{\alpha}(da)=((\sigma,\rho))_{\alpha}f_g(x), \tag*{(31)} ∫ H [ ∫ R γ g ( a , b ) σ (⟨ a , x ⟩ − b ) d b ] ν α ( d a ) = (( σ , ρ ) ) α f g ( x ) , (31)
( ( σ , ρ ) ) α : = 1 2 π ⟨ σ ♯ , ρ ♯ ( − ⋅ ) ∣ ⋅ ∣ − α ⟩ . ((\sigma,\rho))_{\alpha}:=\frac{1}{2\pi}\langle\sigma^{\sharp},\rho^{\sharp}(-\cdot)|\cdot|^{-\alpha}\rangle. (( σ , ρ ) ) α := 2 π 1 ⟨ σ ♯ , ρ ♯ ( − ⋅ ) ∣ ⋅ ∣ − α ⟩ .
The constant is a pairing with a test function supported in − supp ρ ♯ -\operatorname{supp}\rho^{\sharp} − supp ρ ♯ . The identity allows ( ( σ , ρ ) ) α = 0 ((\sigma,\rho))_{\alpha}=0 (( σ , ρ ) ) α = 0 . If σ \sigma σ is not a polynomial, a band-pass ρ \rho ρ with nonzero constant exists; this separate condition is needed for normalization and universality.
Proof. (i). Apply Lemma 4.1 with ν = ν α \nu=\nu_{\alpha} ν = ν α and Y = C \mathcal{Y}=\mathbb{C} Y = C .
(ii). Fix x x x and write u = ⟨ a , x ⟩ u=\langle a,x\rangle u = ⟨ a , x ⟩ . By Lemma 3.8, for ν α \nu_{\alpha} ν α -almost every a a a the function γ g ( a , ⋅ ) \gamma_g(a,\cdot) γ g ( a , ⋅ ) belongs to L 2 ( R ) L^{2}(\mathbb{R}) L 2 ( R ) with Fourier transform h a ( ω ) = ρ ♯ ( − ω ) g ( ω a ) h_a(\omega)=\rho^{\sharp}(-\omega)g(\omega a) h a ( ω ) = ρ ♯ ( − ω ) g ( ω a ) , which is integrable and square integrable; compact support is not needed. Parseval’s identity, applied to γ g ( a , ⋅ ) \gamma_g(a,\cdot) γ g ( a , ⋅ ) and to b ↦ ρ ( u − b ) b\mapsto\rho(u-b) b ↦ ρ ( u − b ) , whose Fourier transform is e − i ω u ρ ♯ ( − ω ) e^{-i\omega u}\rho^{\sharp}(-\omega) e − iω u ρ ♯ ( − ω ) , gives
∫ R γ g ( a , b ) ρ ( u − b ) d b = 1 2 π ∫ R ρ ♯ ( − ω ) g ( ω a ) ρ ♯ ( ω ) e i ω u d ω . (32) \int_{\mathbb{R}}\gamma_g(a,b)\rho(u-b)\,db=\frac{1}{2\pi}\int_{\mathbb{R}}\rho^{\sharp}(-\omega)g(\omega a)\rho^{\sharp}(\omega)e^{i\omega u}\,d\omega. \tag*{(32)} ∫ R γ g ( a , b ) ρ ( u − b ) d b = 2 π 1 ∫ R ρ ♯ ( − ω ) g ( ω a ) ρ ♯ ( ω ) e iω u d ω . (32)
where we used that ρ \rho ρ is real, so that the conjugate of the Fourier transform of b ↦ ρ ( u − b ) b\mapsto\rho(u-b) b ↦ ρ ( u − b ) at ω \omega ω equals e i ω u ρ ♯ ( ω ) e^{i\omega u}\rho^{\sharp}(\omega) e iω u ρ ♯ ( ω ) . The right side is bounded by 1 2 π ∫ ∣ ρ ♯ ( ω ) ∣ 2 ∣ g ( ω a ) ∣ d ω \frac{1}{2\pi}\int|\rho^{\sharp}(\omega)|^{2}|g(\omega a)|\,d\omega 2 π 1 ∫ ∣ ρ ♯ ( ω ) ∣ 2 ∣ g ( ω a ) ∣ d ω , whose ν α \nu_{\alpha} ν α -integral equals 1 2 π ∫ ∣ ρ ♯ ( ω ) ∣ 2 ∣ ω ∣ − α d ω ∥ g ∥ L 1 ( ν α ) < ∞ \frac{1}{2\pi}\int|\rho^{\sharp}(\omega)|^{2}|\omega|^{-\alpha}\,d\omega\,\|g\|_{L^{1}(\nu_{\alpha})}<\infty 2 π 1 ∫ ∣ ρ ♯ ( ω ) ∣ 2 ∣ ω ∣ − α d ω ∥ g ∥ L 1 ( ν α ) < ∞ by (9). This proves absolute convergence of the iterated integral and justifies Fubini in
∫ H ∫ R γ g ( a , b ) ρ ( u − b ) d b ν α ( d a ) = 1 2 π ∫ R ρ ♯ ( ω ) ρ ♯ ( − ω ) [ ∫ H g ( ω a ) e i ω ⟨ a , x ⟩ ν α ( d a ) ] d ω . \int_{\mathcal{H}}\int_{\mathbb{R}}\gamma_g(a,b)\rho(u-b)\,db\,\nu_{\alpha}(da)=\frac{1}{2\pi}\int_{\mathbb{R}}\rho^{\sharp}(\omega)\rho^{\sharp}(-\omega)\left[\int_{\mathcal{H}}g(\omega a)e^{i\omega\langle a,x\rangle}\,\nu_{\alpha}(da)\right]d\omega. ∫ H ∫ R γ g ( a , b ) ρ ( u − b ) d b ν α ( d a ) = 2 π 1 ∫ R ρ ♯ ( ω ) ρ ♯ ( − ω ) [ ∫ H g ( ω a ) e iω ⟨ a , x ⟩ ν α ( d a ) ] d ω .
With ξ = ω a \xi=\omega a ξ = ω a , homogeneity turns the bracket into ∣ ω ∣ − α f g ( x ) |\omega|^{-\alpha}f_g(x) ∣ ω ∣ − α f g ( x ) . The remaining frequency integral is
1 2 π ∫ R ρ ♯ ( ω ) ρ ♯ ( − ω ) ∣ ω ∣ − α d ω = ( ( ρ , ρ ) ) α , \frac{1}{2\pi}\int_{\mathbb{R}}\rho^{\sharp}(\omega)\rho^{\sharp}(-\omega)|\omega|^{-\alpha}\,d\omega=((\rho,\rho))_{\alpha}, 2 π 1 ∫ R ρ ♯ ( ω ) ρ ♯ ( − ω ) ∣ ω ∣ − α d ω = (( ρ , ρ ) ) α ,
because ρ \rho ρ is real. If γ g ∈ L 1 ( λ α ) \gamma_g\in L^{1}(\lambda_{\alpha}) γ g ∈ L 1 ( λ α ) , the double integral converges absolutely since ρ \rho ρ is bounded, and the iterated integral is the integral network.
(iii). Let g g g satisfy Definition 4.2. Choose an integer r r r at least as large as a polynomial growth order of σ \sigma σ . By Lemma 4.3,
∫ H × R ∣ γ g ( a , b ) ∣ ∣ σ ( ⟨ a , x ⟩ − b ) ∣ d λ α ≤ C σ , x , r ∫ ( 1 + ∥ a ∥ + ∣ b ∣ ) r ∣ γ g ∣ d λ α < ∞ . \int_{\mathcal{H}\times\mathbb{R}}|\gamma_g(a,b)||\sigma(\langle a,x\rangle-b)|\,d\lambda_{\alpha}\leq C_{\sigma,x,r}\int(1+\|a\|+|b|)^{r}|\gamma_g|\,d\lambda_{\alpha}<\infty. ∫ H × R ∣ γ g ( a , b ) ∣∣ σ (⟨ a , x ⟩ − b ) ∣ d λ α ≤ C σ , x , r ∫ ( 1 + ∥ a ∥ + ∣ b ∣ ) r ∣ γ g ∣ d λ α < ∞.
Thus the ordinary integral network already exists, independently of its identification below. For every a a a , the function ω ↦ h a ( ω ) = ρ ♯ ( − ω ) g ( ω a ) \omega\mapsto h_a(\omega)=\rho^{\sharp}(-\omega)g(\omega a) ω ↦ h a ( ω ) = ρ ♯ ( − ω ) g ( ω a ) is C c ∞ C^{\infty}_{c} C c ∞ with support in I I I , so γ g ( a , ⋅ ) ∈ S ( R ) \gamma_g(a,\cdot)\in\mathcal{S}(\mathbb{R}) γ g ( a , ⋅ ) ∈ S ( R ) for every a a a , and ∫ ∣ γ g ( a , b ) ∣ ∣ σ ( u − b ) ∣ d b < ∞ \int|\gamma_g(a,b)||\sigma(u-b)|\,db<\infty ∫ ∣ γ g ( a , b ) ∣∣ σ ( u − b ) ∣ d b < ∞ because σ \sigma σ has polynomial growth. For γ ∈ S ( R ) \gamma\in\mathcal{S}(\mathbb{R}) γ ∈ S ( R ) and σ ∈ S ′ ( R ) \sigma\in\mathcal{S}'(\mathbb{R}) σ ∈ S ′ ( R ) , the translation identity γ ( u − t ) = 1 2 π ∫ γ ♯ ( ω ) e i ω u e − i ω t d ω \gamma(u-t)=\frac{1}{2\pi}\int\gamma^\sharp(\omega)e^{i\omega u}e^{-i\omega t}\,d\omega γ ( u − t ) = 2 π 1 ∫ γ ♯ ( ω ) e iω u e − iω t d ω and the definition of the distributional Fourier transform give
∫ R γ ( b ) σ ( u − b ) d b = 1 2 π ⟨ σ ♯ , ω ↦ γ ♯ ( ω ) e i ω u ⟩ . (33) \int_{\mathbb{R}}\gamma(b)\sigma(u-b)\,db=\frac{1}{2\pi}\langle\sigma^\sharp,\omega\mapsto\gamma^\sharp(\omega)e^{i\omega u}\rangle. \tag*{(33)} ∫ R γ ( b ) σ ( u − b ) d b = 2 π 1 ⟨ σ ♯ , ω ↦ γ ♯ ( ω ) e iω u ⟩ . (33)
Applied to γ = γ g ( a , ⋅ ) \gamma=\gamma_g(a,\cdot) γ = γ g ( a , ⋅ ) , the test function is φ a ( ω ) : = ρ ♯ ( − ω ) g ( ω a ) e i ω ⟨ a , x ⟩ \varphi_a(\omega):=\rho^\sharp(-\omega)g(\omega a)e^{i\omega\langle a,x\rangle} φ a ( ω ) := ρ ♯ ( − ω ) g ( ω a ) e iω ⟨ a , x ⟩ , supported in − supp ρ ♯ ⊂ I -\operatorname{supp}\rho^\sharp\subset I − supp ρ ♯ ⊂ I . A tempered distribution has finite order: there are m m m and C C C with ∣ ⟨ σ ♯ , φ ⟩ ∣ ≤ C ∥ φ ∥ C m |\langle\sigma^\sharp,\varphi\rangle|\le C\|\varphi\|_{C^m} ∣ ⟨ σ ♯ , φ ⟩ ∣ ≤ C ∥ φ ∥ C m for every φ ∈ C c ∞ ( R ) \varphi\in C^\infty_c(\mathbb{R}) φ ∈ C c ∞ ( R ) supported in the fixed compact set I I I . Let I ′ I' I ′ be a compact neighbourhood of I I I in R ∖ { 0 } \mathbb{R}\setminus\{0\} R ∖ { 0 } , and let X X X be the closure in C m ( R ) C^m(\mathbb{R}) C m ( R ) of C c ∞ ( int I ′ ) C^\infty_c(\operatorname{int}I') C c ∞ ( int I ′ ) . Then X X X is a separable Banach space containing every C m C^m C m function supported in I I I (mollify), the pairing with σ ♯ \sigma^\sharp σ ♯ extends to a bounded functional on X X X , and evaluation at each ω \omega ω is bounded on X X X . The map a ↦ φ a ∈ X a\mapsto\varphi_a\in X a ↦ φ a ∈ X is weakly measurable, because ( a , ω ) ↦ ∂ ω k g ( ω a ) (a,\omega)\mapsto\partial_\omega^k g(\omega a) ( a , ω ) ↦ ∂ ω k g ( ω a ) is Borel as a pointwise limit of difference quotients of the Borel function ( a , ω ) ↦ g ( ω a ) (a,\omega)\mapsto g(\omega a) ( a , ω ) ↦ g ( ω a ) ; by Pettis’ theorem it is strongly measurable. Leibniz’ rule and ∣ ⟨ a , x ⟩ ∣ ≤ ∥ a ∥ ∥ x ∥ |\langle a,x\rangle|\le\|a\|\|x\| ∣ ⟨ a , x ⟩ ∣ ≤ ∥ a ∥∥ x ∥ give
∥ φ a ∥ C m ≤ C ρ , m ( 1 + ∥ x ∥ ) m ( 1 + ∥ a ∥ ) m max k ≤ m sup ω ∈ I ∣ ∂ ω k g ( ω a ) ∣ , \|\varphi_a\|_{C^m}\le C_{\rho,m}(1+\|x\|)^m(1+\|a\|)^m\max_{k\le m}\sup_{\omega\in I}|\partial_\omega^k g(\omega a)|, ∥ φ a ∥ C m ≤ C ρ , m ( 1 + ∥ x ∥ ) m ( 1 + ∥ a ∥ ) m k ≤ m max ω ∈ I sup ∣ ∂ ω k g ( ω a ) ∣ ,
which is ν α \nu_\alpha ν α -integrable by (27). Hence a ↦ φ a a\mapsto\varphi_a a ↦ φ a is Bochner integrable in X X X , and the bounded functional ⟨ σ ♯ , ⋅ ⟩ \langle\sigma^\sharp,\cdot\rangle ⟨ σ ♯ , ⋅ ⟩ commutes with the integral:
∫ H ∫ R γ g ( a , b ) σ ( ⟨ a , x ⟩ − b ) d b ν α ( d a ) = 1 2 π ⟨ σ ♯ , ∫ H φ a ν α ( d a ) ⟩ . \int_{\mathcal{H}}\int_{\mathbb{R}}\gamma_g(a,b)\sigma(\langle a,x\rangle-b)\,db\,\nu_\alpha(da)=\frac{1}{2\pi}\left\langle\sigma^\sharp,\int_{\mathcal{H}}\varphi_a\,\nu_\alpha(da)\right\rangle. ∫ H ∫ R γ g ( a , b ) σ (⟨ a , x ⟩ − b ) d b ν α ( d a ) = 2 π 1 ⟨ σ ♯ , ∫ H φ a ν α ( d a ) ⟩ .
Evaluating the Bochner integral at ω \omega ω and substituting ξ = ω a \xi=\omega a ξ = ω a in (9),
( ∫ H φ a ν α ( d a ) ) ( ω ) = ρ ♯ ( − ω ) ∫ H g ( ω a ) e i ⟨ ω a , x ⟩ ν α ( d a ) = ρ ♯ ( − ω ) ∣ ω ∣ − α f g ( x ) , \left(\int_{\mathcal{H}}\varphi_a\,\nu_\alpha(da)\right)(\omega)=\rho^\sharp(-\omega)\int_{\mathcal{H}}g(\omega a)e^{i\langle\omega a,x\rangle}\,\nu_\alpha(da)=\rho^\sharp(-\omega)|\omega|^{-\alpha}f_g(x), ( ∫ H φ a ν α ( d a ) ) ( ω ) = ρ ♯ ( − ω ) ∫ H g ( ω a ) e i ⟨ ω a , x ⟩ ν α ( d a ) = ρ ♯ ( − ω ) ∣ ω ∣ − α f g ( x ) ,
so the right side equals f g ( x ) 1 2 π ⟨ σ ♯ , ρ ♯ ( − ⋅ ) ∣ ⋅ ∣ − α ⟩ = ( ( σ , ρ ) ) α f g ( x ) f_g(x)\frac{1}{2\pi}\langle\sigma^\sharp,\rho^\sharp(-\cdot)|\cdot|^{-\alpha}\rangle=((\sigma,\rho))_\alpha f_g(x) f g ( x ) 2 π 1 ⟨ σ ♯ , ρ ♯ ( − ⋅ ) ∣ ⋅ ∣ − α ⟩ = (( σ , ρ ) ) α f g ( x ) . Absolute convergence of the outer integral follows from the same bound on ∥ φ a ∥ C m \|\varphi_a\|_{C^m} ∥ φ a ∥ C m .
Finally, if ⟨ σ ♯ , ρ ♯ ( − ⋅ ) ∣ ⋅ ∣ − α ⟩ = 0 \langle\sigma^\sharp,\rho^\sharp(-\cdot)|\cdot|^{-\alpha}\rangle=0 ⟨ σ ♯ , ρ ♯ ( − ⋅ ) ∣ ⋅ ∣ − α ⟩ = 0 for every real band-pass ρ \rho ρ , then, since every φ ∈ C c ∞ ( R ∖ { 0 } ) \varphi\in C^\infty_c(\mathbb{R}\setminus\{0\}) φ ∈ C c ∞ ( R ∖ { 0 }) is a combination φ 1 + i φ 2 \varphi_1+i\varphi_2 φ 1 + i φ 2 of Hermitian-symmetric functions φ j \varphi_j φ j and each φ j ( − ω ) ∣ ω ∣ α \varphi_j(-\omega)|\omega|^\alpha φ j ( − ω ) ∣ ω ∣ α is the Fourier transform of a real band-pass filter, σ ♯ \sigma^\sharp σ ♯ vanishes on C c ∞ ( R ∖ { 0 } ) C^\infty_c(\mathbb{R}\setminus\{0\}) C c ∞ ( R ∖ { 0 }) . Then supp σ ♯ ⊂ { 0 } \operatorname{supp}\sigma^\sharp\subset\{0\} supp σ ♯ ⊂ { 0 } , so σ ♯ \sigma^\sharp σ ♯ is a finite combination of derivatives of δ 0 \delta_0 δ 0 and σ \sigma σ is a polynomial. □
No Gaussian input measure enters Theorem 4.5: the representation uses only the sigma-finiteness and homogeneity of ν α \nu_\alpha ν α . The input measure is needed to compute a spectral density from a given function f f f , which is the content of the ridgelet transform and of the next subsection. If σ \sigma σ is a polynomial, its Fourier transform is supported at zero, so the band-pass constant in (iii) vanishes; no nontrivial reconstruction is then claimed.
The frame operator and the reconstruction formula
Let E α c E_\alpha^c E α c be the continuous anti-dual of E α E_\alpha E α . Define the frame operator T α T_\alpha T α as the Riesz isomorphism, and define the anti-dual transpose of the extended F Q F_Q F Q by
T α f [ g ] : = ⟨ f , g ⟩ E α , f , g ∈ E α , (34) T_\alpha f[g]:=\langle f,g\rangle_{E_\alpha},\qquad f,g\in E_\alpha, \tag*{(34)} T α f [ g ] := ⟨ f , g ⟩ E α , f , g ∈ E α , (34)
F Q ′ G [ g ] : = ⟨ G , F Q g ⟩ L 2 ( ν α ) , G ∈ L 2 ( ν α ) , g ∈ E α . (35) F'_QG[g]:=\langle G,F_Qg\rangle_{L^2(\nu_\alpha)},\qquad G\in L^2(\nu_\alpha),\ g\in E_\alpha. \tag*{(35)} F Q ′ G [ g ] := ⟨ G , F Q g ⟩ L 2 ( ν α ) , G ∈ L 2 ( ν α ) , g ∈ E α . (35)
Thus F Q ′ : L 2 ( ν α ) → E α ′ F_Q':L^2(\nu_\alpha)\to E_\alpha' F Q ′ : L 2 ( ν α ) → E α ′ ; the prime denotes the anti-dual transpose formed using the completed input space. The Hilbert adjoint of the unitary map F Q : E α → K α F_Q:E_\alpha\to K_\alpha F Q : E α → K α takes values in E α E_\alpha E α . These constructions assert no everywhere-defined adjoint into the ambient L 2 ( μ Q ) L^2(\mu_Q) L 2 ( μ Q ) . Synthesis with the analysis filter is the transpose of the transform,
S ρ : = R ρ ′ : L 2 ( λ α ) → E α ′ , ( S ρ γ ) [ g ] : = ⟨ γ , R ρ g ⟩ L 2 ( λ α ) . (36) S_\rho:=R_\rho':L^2(\lambda_\alpha)\to E_\alpha',\qquad(S_\rho\gamma)[g]:=\langle\gamma,R_\rho g\rangle_{L^2(\lambda_\alpha)}. \tag*{(36)} S ρ := R ρ ′ : L 2 ( λ α ) → E α ′ , ( S ρ γ ) [ g ] := ⟨ γ , R ρ g ⟩ L 2 ( λ α ) . (36)
When γ λ α \gamma\lambda_\alpha γ λ α is a finite measure with the moment (4), S ρ γ S_\rho\gamma S ρ γ is the integral network S ρ [ γ λ α ] S_\rho[\gamma\lambda_\alpha] S ρ [ γ λ α ] of Definition 2.2 paired with g g g through μ Q \mu_Q μ Q (Lemma 4.6); the notation is therefore consistent.
The term frame operator refers to the dual-space map T α : E α → E α ′ T_\alpha:E_\alpha\to E_\alpha' T α : E α → E α ′ . If R ρ ∗ R_\rho^* R ρ ∗ denotes the Hilbert adjoint into E α E_\alpha E α , then
R ρ ∗ R ρ = C Id E α , R ρ ′ R ρ = C T α , C = ( ( ρ , ρ ) ) α , R_\rho^*R_\rho=C\operatorname{Id}_{E_\alpha},\qquad R_\rho'R_\rho=CT_\alpha,\qquad C=((\rho,\rho))_\alpha, R ρ ∗ R ρ = C Id E α , R ρ ′ R ρ = C T α , C = (( ρ , ρ ) ) α ,
by Theorem 3.14 and the definition of the transpose. This distinguishes the Hilbert adjoint from synthesis taking values in E α ′ E_\alpha' E α ′ . We first establish two facts used in reconstruction: the agreement of weak and integral synthesis, and recovery from Hermite coefficients.
Lemma 4.6 (Synthesis of an integrable coefficient is the integral network). Let ρ ∈ S ( R ) \rho\in\mathcal{S}(\mathbb{R}) ρ ∈ S ( R ) be real and let γ ∈ L 1 ( λ α ) ∩ L 2 ( λ α ) \gamma\in L^1(\lambda_\alpha)\cap L^2(\lambda_\alpha) γ ∈ L 1 ( λ α ) ∩ L 2 ( λ α ) . Then the integral network S ρ [ γ λ α ] S_\rho[\gamma\lambda_\alpha] S ρ [ γ λ α ] of Definition 2.2 is a bounded Borel function on H \mathcal{H} H , and for every g ∈ D α g\in D_\alpha g ∈ D α
( S ρ γ ) [ g ] = ∫ H S ρ [ γ λ α ] ( x ) g ( x ) ‾ μ Q ( d x ) , (S_\rho\gamma)[g]=\int_{\mathcal{H}}S_\rho[\gamma\lambda_\alpha](x)\overline{g(x)}\,\mu_Q(dx), ( S ρ γ ) [ g ] = ∫ H S ρ [ γ λ α ] ( x ) g ( x ) μ Q ( d x ) , with S ρ γ S_\rho\gamma S ρ γ the synthesis operator (36).
Proof. Boundedness follows from ∣ S ρ [ γ λ α ] ∣ ≤ ∥ γ ∥ 1 ∥ ρ ∥ ∞ |S_\rho[\gamma\lambda_\alpha]|\leq\|\gamma\|_1\|\rho\|_\infty ∣ S ρ [ γ λ α ] ∣ ≤ ∥ γ ∥ 1 ∥ ρ ∥ ∞ . The absolute integral of the product in the synthesis pairing is at most ∥ γ ∥ 1 ∥ ρ ∥ ∞ ∥ g ∥ L 1 ( μ Q ) \|\gamma\|_1\|\rho\|_\infty\|g\|_{L^1(\mu_Q)} ∥ γ ∥ 1 ∥ ρ ∥ ∞ ∥ g ∥ L 1 ( μ Q ) , which is finite. Since ρ \rho ρ is real, Fubini therefore gives
⟨ γ , R ρ g ⟩ L 2 ( λ α ) = ∫ H × R γ ( a , b ) ∫ H g ( x ) ρ ( ⟨ a , x ⟩ − b ) μ Q ( d x ) λ α ( d a , d b ) = ∫ H g ( x ) ‾ S ρ [ γ λ α ] ( x ) μ Q ( d x ) . \begin{aligned}
\langle\gamma,R_\rho g\rangle_{L^2(\lambda_\alpha)}
=\int_{\mathcal{H}\times\mathbb{R}}\gamma(a,b)\int_{\mathcal{H}}g(x)\rho(\langle a,x\rangle-b)\,\mu_Q(dx)\,\lambda_\alpha(da,db) \\
&=\int_{\mathcal{H}}\overline{g(x)}S_\rho[\gamma\lambda_\alpha](x)\,\mu_Q(dx).
\end{aligned} ⟨ γ , R ρ g ⟩ L 2 ( λ α ) = ∫ H × R γ ( a , b ) ∫ H g ( x ) ρ (⟨ a , x ⟩ − b ) μ Q ( d x ) λ α ( d a , d b ) = ∫ H g ( x ) S ρ [ γ λ α ] ( x ) μ Q ( d x ) .
Lemma 4.7 (Entire extension and totality of the Hermite coefficients). For f ∈ L 2 ( μ Q ) f\in L^2(\mu_Q) f ∈ L 2 ( μ Q ) and ξ ≠ 0 \xi\neq0 ξ = 0 , put τ ( ξ ) = ⟨ Q ξ , ξ ⟩ 1 / 2 \tau(\xi)=\langle Q\xi,\xi\rangle^{1/2} τ ( ξ ) = ⟨ Q ξ , ξ ⟩ 1/2 and E f ( z ξ ) : = e z 2 τ ( ξ ) 2 / 2 F Q f ( z ξ ) E_f(z\xi):=e^{z^2\tau(\xi)^2/2}F_Qf(z\xi) E f ( z ξ ) := e z 2 τ ( ξ ) 2 /2 F Q f ( z ξ ) for z ∈ C z\in\mathbb{C} z ∈ C . Then z ↦ E f ( z ξ ) z\mapsto E_f(z\xi) z ↦ E f ( z ξ ) is entire,
E f ( z ξ ) = ∑ n = 0 ∞ ( − i z τ ( ξ ) ) n n ! E μ Q [ f He n ( ⟨ x , ξ ⟩ τ ( ξ ) ) ] , E_f(z\xi)=\sum_{n=0}^{\infty}\frac{(-iz\tau(\xi))^n}{n!}\mathbb{E}_{\mu_Q}\left[f\operatorname{He}_n\left(\frac{\langle x,\xi\rangle}{\tau(\xi)}\right)\right], E f ( z ξ ) = n = 0 ∑ ∞ n ! ( − i z τ ( ξ ) ) n E μ Q [ f He n ( τ ( ξ ) ⟨ x , ξ ⟩ ) ] , the series converges locally uniformly, ∣ E f ( z ξ ) ∣ ≤ ∥ f ∥ L 2 ( μ Q ) e ∣ z ∣ 2 τ ( ξ ) 2 / 2 |E_f(z\xi)|\leq\|f\|_{L^2(\mu_Q)}e^{|z|^2\tau(\xi)^2/2} ∣ E f ( z ξ ) ∣ ≤ ∥ f ∥ L 2 ( μ Q ) e ∣ z ∣ 2 τ ( ξ ) 2 /2 , and the following inversion formula holds:
E μ Q [ f ( x ) He n ( ⟨ x , ξ ⟩ τ ( ξ ) ) ] = i n τ ( ξ ) n d n d t n ( e t 2 τ ( ξ ) 2 / 2 F Q f ( t ξ ) ) ∣ t = 0 , τ ( ξ ) : = ⟨ Q ξ , ξ ⟩ 1 / 2 . (37) \mathbb{E}_{\mu_Q}\left[f(x)\operatorname{He}_n\left(\frac{\langle x,\xi\rangle}{\tau(\xi)}\right)\right]
=\frac{i^n}{\tau(\xi)^n}\frac{d^n}{dt^n}\left(e^{t^2\tau(\xi)^2/2}F_Qf(t\xi)\right)\bigg|_{t=0},\qquad\tau(\xi):=\langle Q\xi,\xi\rangle^{1/2}. \tag*{(37)} E μ Q [ f ( x ) He n ( τ ( ξ ) ⟨ x , ξ ⟩ ) ] = τ ( ξ ) n i n d t n d n ( e t 2 τ ( ξ ) 2 /2 F Q f ( t ξ ) ) t = 0 , τ ( ξ ) := ⟨ Q ξ , ξ ⟩ 1/2 . (37) The coefficients, over all ξ \xi ξ and n n n , determine f f f in L 2 ( μ Q ) L^2(\mu_Q) L 2 ( μ Q ) .
Proof. For a standard normal Z Z Z , the generating function e t Z − t 2 / 2 = ∑ n He n ( Z ) t n / n ! e^{tZ-t^{2}/2}=\sum_n \operatorname{He}_n(Z)t^n/n! e tZ − t 2 /2 = ∑ n He n ( Z ) t n / n ! converges in L 2 L^2 L 2 for each complex t t t , locally uniformly, since orthogonality gives the squared norm sum ∑ n ∣ t ∣ 2 n / n ! = e ∣ t ∣ 2 \sum_n |t|^{2n}/n!=e^{|t|^2} ∑ n ∣ t ∣ 2 n / n ! = e ∣ t ∣ 2 . Take Z = ⟨ x , ξ ⟩ / τ ( ξ ) Z=\langle x,\xi\rangle/\tau(\xi) Z = ⟨ x , ξ ⟩ / τ ( ξ ) and t = − i z τ ( ξ ) t=-iz\tau(\xi) t = − i z τ ( ξ ) ; pairing with f f f gives the series and entireness, Cauchy–Schwarz gives the bound, and termwise differentiation proves (37) .
For totality, take an eigenbasis Q e j = q j e j Qe_j=q_je_j Q e j = q j e j , q j > 0 q_j>0 q j > 0 , and the independent coordinates Z j = ⟨ x , e j ⟩ / q j Z_j=\langle x,e_j\rangle/\sqrt{q_j} Z j = ⟨ x , e j ⟩ / q j . Finite products ∏ j He n j ( Z j ) \prod_j\operatorname{He}_{n_j}(Z_j) ∏ j He n j ( Z j ) form a complete orthogonal system in L 2 ( μ Q ) L^2(\mu_Q) L 2 ( μ Q ) : they are complete at each finite-dimensional Gaussian level, and conditional expectations onto the first m m m coordinates converge in L 2 L^2 L 2 [48 Chapter 2] . For every fixed total degree, polarization of the Wick power of ∑ j t j Z j \sum_j t_jZ_j ∑ j t j Z j expresses these products as linear combinations of the directional Wick powers He n ( ⟨ x , ξ ⟩ / τ ( ξ ) ) \operatorname{He}_n(\langle x,\xi\rangle/\tau(\xi)) He n (⟨ x , ξ ⟩ / τ ( ξ )) with ξ = ∑ j t j q j − 1 / 2 e j \xi=\sum_j t_jq_j^{-1/2}e_j ξ = ∑ j t j q j − 1/2 e j of finite support. Thus all orthogonal expansion coefficients of f f f are recovered.
We can now state the reconstruction theorem, a central result of the paper.
Theorem 4.8 (Reconstruction and the frame operator). Let α > 0 \alpha>0 α > 0 and let ρ \rho ρ be an α \alpha α -admissible Schwartz filter.
(i) Frame identity. The frame operator satisfies T α = F Q ′ F Q T_\alpha=F'_QF_Q T α = F Q ′ F Q ; it is an isometric bijection E α → E α ′ \mathcal{E}_\alpha\to\mathcal{E}'_\alpha E α → E α ′ , and
S ρ R ρ f = ( ( ρ , ρ ) ) α T α f , f ∈ E α . (38) S_\rho R_\rho f=((\rho,\rho))_\alpha T_\alpha f,\qquad f\in\mathcal{E}_\alpha. \tag*{(38)} S ρ R ρ f = (( ρ , ρ ) ) α T α f , f ∈ E α . (38) (ii) Two reconstruction formulas. For f ∈ E α f\in\mathcal{E}_\alpha f ∈ E α and g ∈ E α ′ g\in\mathcal{E}'_\alpha g ∈ E α ′ ,
f = ( ( ρ , ρ ) ) α − 1 T α − 1 S ρ R ρ f , g = ( ( ρ , ρ ) ) α − 1 S ρ ( R ρ T α − 1 g ) . (39) f=((\rho,\rho))_\alpha^{-1}T_\alpha^{-1}S_\rho R_\rho f,\qquad g=((\rho,\rho))_\alpha^{-1}S_\rho(R_\rho T_\alpha^{-1}g). \tag*{(39)} f = (( ρ , ρ ) ) α − 1 T α − 1 S ρ R ρ f , g = (( ρ , ρ ) ) α − 1 S ρ ( R ρ T α − 1 g ) . (39) (iii) The frame operator in terms of the spectral density. If f ∈ D α f\in\mathcal{D}_\alpha f ∈ D α and F Q f ∈ L 1 ( ν α ) F_Qf\in L^1(\nu_\alpha) F Q f ∈ L 1 ( ν α ) , then T α f T_\alpha f T α f is represented by the bounded continuous function f F Q f f_{F_Qf} f F Q f , in the sense that T α f [ v ] = ∫ H f F Q f ( x ) v ( x ) ‾ μ Q ( d x ) T_\alpha f[v]=\int_{\mathcal{H}}f_{F_Qf}(x)\overline{v(x)}\,\mu_Q(dx) T α f [ v ] = ∫ H f F Q f ( x ) v ( x ) μ Q ( d x ) for v ∈ D α v\in\mathcal{D}_\alpha v ∈ D α . Conversely, if g ∈ K α g\in\mathcal{K}_\alpha g ∈ K α , then F Q ′ g ∈ E α ′ F'_Qg\in\mathcal{E}'_\alpha F Q ′ g ∈ E α ′ and R ρ T α − 1 F Q ′ g = W ρ g R_\rho T_\alpha^{-1}F'_Qg=W_\rho g R ρ T α − 1 F Q ′ g = W ρ g ; when g ∈ L 1 ( ν α ) g\in L^1(\nu_\alpha) g ∈ L 1 ( ν α ) , F Q ′ g F'_Qg F Q ′ g is represented by f g f_g f g and the second formula in (39) is (30) .
(iv) Reconstruction by backprojection. The bounded adjoint W ρ ∗ W_\rho^* W ρ ∗ of Lemma 3.9, given by (18) , satisfies W ρ ∗ R ρ f = ( ( ρ , ρ ) ) α F Q f W_\rho^*R_\rho f=((\rho,\rho))_\alpha F_Qf W ρ ∗ R ρ f = (( ρ , ρ ) ) α F Q f in L 2 ( ν α ) L^2(\nu_\alpha) L 2 ( ν α ) for every f ∈ E α f\in\mathcal{E}_\alpha f ∈ E α . For f ∈ D α f\in\mathcal{D}_\alpha f ∈ D α , using the continuous Fourier-slice representative makes this identity pointwise in ξ \xi ξ . The following final step uses the Gaussian input specifically: f f f is recovered from F Q f F_Qf F Q f by the Hermite coefficients in (37) , which determine f f f in L 2 ( μ Q ) L^2(\mu_Q) L 2 ( μ Q ) . We write Δ Q \Delta_Q Δ Q for this inverse of F Q F_Q F Q on its range on D α \mathcal{D}_\alpha D α , so that
f = Δ Q [ ( ( ρ , ρ ) ) α − 1 W ρ ∗ R ρ f ] . f=\Delta_Q\left[((\rho,\rho))_\alpha^{-1}W_\rho^*R_\rho f\right]. f = Δ Q [ (( ρ , ρ ) ) α − 1 W ρ ∗ R ρ f ] .
Proof. Since F Q F_Q F Q is unitary onto K α \mathcal{K}_\alpha K α , F Q ′ F Q f [ g ] = ⟨ F Q f , F Q g ⟩ = ⟨ f , g ⟩ E α = T α f [ g ] F'_QF_Qf[g]=\langle F_Qf,F_Qg\rangle=\langle f,g\rangle_{\mathcal{E}_\alpha}=T_\alpha f[g] F Q ′ F Q f [ g ] = ⟨ F Q f , F Q g ⟩ = ⟨ f , g ⟩ E α = T α f [ g ] , and the Riesz representation theorem makes T α T_\alpha T α an isometric bijection. By (36) and (25) , ( S ρ R ρ f ) [ g ] = ⟨ R ρ f , R ρ g ⟩ = ( ( ρ , ρ ) ) α T α f [ g ] (S_\rho R_\rho f)[g]=\langle R_\rho f,R_\rho g\rangle=((\rho,\rho))_\alpha T_\alpha f[g] ( S ρ R ρ f ) [ g ] = ⟨ R ρ f , R ρ g ⟩ = (( ρ , ρ ) ) α T α f [ g ] , which is (i); (ii) follows by applying T α − 1 T_\alpha^{-1} T α − 1 , respectively by substituting f = T α − 1 g f=T_\alpha^{-1}g f = T α − 1 g .
(iii) Let f ∈ D α f\in\mathcal{D}_\alpha f ∈ D α , put g = F Q f ∈ L 1 ( ν α ) g=F_Qf\in L^1(\nu_\alpha) g = F Q f ∈ L 1 ( ν α ) , and take v ∈ D α v\in\mathcal{D}_\alpha v ∈ D α . The product ∣ g ( ξ ) ∣ ∣ v ( x ) ∣ |g(\xi)||v(x)| ∣ g ( ξ ) ∣∣ v ( x ) ∣ is integrable against ν α ⊗ μ Q \nu_\alpha\otimes\mu_Q ν α ⊗ μ Q , so Fubini gives
∫ H f g ( x ) v ( x ) ‾ μ Q ( d x ) = ∫ H g ( ξ ) F Q v ( ξ ) ‾ ν α ( d ξ ) = ⟨ f , v ⟩ E α = T α f [ v ] . \int_{\mathcal{H}}f_g(x)\overline{v(x)}\,\mu_Q(dx)=\int_{\mathcal{H}}g(\xi)\overline{F_Qv(\xi)}\,\nu_\alpha(d\xi)=\langle f,v\rangle_{\mathcal{E}_\alpha}=T_\alpha f[v]. ∫ H f g ( x ) v ( x ) μ Q ( d x ) = ∫ H g ( ξ ) F Q v ( ξ ) ν α ( d ξ ) = ⟨ f , v ⟩ E α = T α f [ v ] .
Conversely, for g ∈ K α g\in\mathcal{K}_\alpha g ∈ K α set u = T α − 1 F Q ′ g u=T_\alpha^{-1}F'_Qg u = T α − 1 F Q ′ g . For every v ∈ E α v\in\mathcal{E}_\alpha v ∈ E α ,
⟨ F Q u , F Q v ⟩ = ⟨ u , v ⟩ E α = F Q ′ g [ v ] = ⟨ g , F Q v ⟩ . \langle F_Qu,F_Qv\rangle=\langle u,v\rangle_{\mathcal{E}_\alpha}=F'_Qg[v]=\langle g,F_Qv\rangle. ⟨ F Q u , F Q v ⟩ = ⟨ u , v ⟩ E α = F Q ′ g [ v ] = ⟨ g , F Q v ⟩ .
Unitarity onto K α \mathcal{K}_{\alpha} K α implies F Q u = g F_Q u = g F Q u = g , and hence R ρ T α − 1 F Q ′ g = W ρ g R_{\rho}T_{\alpha}^{-1}F_Q'g = W_{\rho}g R ρ T α − 1 F Q ′ g = W ρ g . If also g ∈ L 1 ( ν α ) g \in L^1(\nu_{\alpha}) g ∈ L 1 ( ν α ) , the same Fubini computation identifies F Q ′ g F_Q'g F Q ′ g with the functional represented by f g f_g f g . Thus F Q ′ g = C − 1 S ρ W ρ g F_Q'g = C^{-1}S_{\rho}W_{\rho}g F Q ′ g = C − 1 S ρ W ρ g ; when γ g ∈ L 1 ( λ α ) \gamma_g \in L^1(\lambda_{\alpha}) γ g ∈ L 1 ( λ α ) , Lemma 4.6 identifies this with ordinary synthesis paired against v v v .
(iv). Boundedness of W ρ ∗ W_{\rho}^{*} W ρ ∗ and the identity W ρ ∗ W ρ = C Id W_{\rho}^{*}W_{\rho}=C\operatorname{Id} W ρ ∗ W ρ = C Id are proved in Lemma 3.9 . For f ∈ D α f \in\mathcal{D}_{\alpha} f ∈ D α , the Fourier-slice representative (15) is continuous, and ( R ρ f ) # ( ξ / ω , ω ) = ρ ∗ ( − ω ) F Q f ( ξ ) (R_{\rho}f)^{\#}(\xi/\omega,\omega)=\rho^{*}(-\omega)F_Qf(\xi) ( R ρ f ) # ( ξ / ω , ω ) = ρ ∗ ( − ω ) F Q f ( ξ ) for every ξ \xi ξ and ω ≠ 0 \omega\ne0 ω = 0 ; inserting this into (18) gives W ρ ∗ R ρ f ( ξ ) = 1 2 π ∫ R ∖ { 0 } ∣ ρ # ( ω ) ∣ 2 ∣ ω ∣ − α d ω F Q f ( ξ ) = ( ( ρ , ρ ) ) α F Q f ( ξ ) W_{\rho}^{*}R_{\rho}f(\xi)=\frac{1}{2\pi}\int_{\mathbb{R}\setminus\{0\}}|\rho^{\#}(\omega)|^{2}|\omega|^{-\alpha}\,d\omega\,F_Qf(\xi)=((\rho,\rho))_{\alpha}F_Qf(\xi) W ρ ∗ R ρ f ( ξ ) = 2 π 1 ∫ R ∖ { 0 } ∣ ρ # ( ω ) ∣ 2 ∣ ω ∣ − α d ω F Q f ( ξ ) = (( ρ , ρ ) ) α F Q f ( ξ ) pointwise. The Hermite formula and the totality statement are Lemma 4.7 .
Corollary 4.9 (Stability under coefficient perturbations). Let ρ \rho ρ be α \alpha α -admissible, write C = ( ( ρ , ρ ) ) α C=((\rho,\rho))_{\alpha} C = (( ρ , ρ ) ) α , and define the bounded decoder D ρ : = C − 1 T α − 1 S ρ D_{\rho}:=C^{-1}T_{\alpha}^{-1}S_{\rho} D ρ := C − 1 T α − 1 S ρ . Then D ρ R ρ = Id D_{\rho}R_{\rho}=\operatorname{Id} D ρ R ρ = Id and ∥ D ρ ∥ ≤ C − 1 / 2 \|D_{\rho}\|\le C^{-1/2} ∥ D ρ ∥ ≤ C − 1/2 . In particular, if f ∈ E α f\in\mathcal{E}_{\alpha} f ∈ E α , γ δ ∈ L 2 ( λ α ) \gamma_{\delta}\in L^{2}(\lambda_{\alpha}) γ δ ∈ L 2 ( λ α ) , and ∥ γ δ − R ρ f ∥ 2 ≤ δ \|\gamma_{\delta}-R_{\rho}f\|_{2}\le\delta ∥ γ δ − R ρ f ∥ 2 ≤ δ , then
∥ D ρ γ δ − f ∥ E α ≤ δ / C . \|D_{\rho}\gamma_{\delta}-f\|_{\mathcal{E}_{\alpha}}\le\delta/\sqrt{C}. ∥ D ρ γ δ − f ∥ E α ≤ δ / C .
Proof. The left-inverse identity is Theorem 4.8 (ii). The transpose S ρ = R ρ ′ S_{\rho}=R_{\rho}' S ρ = R ρ ′ has norm at most C \sqrt{C} C , and T α − 1 T_{\alpha}^{-1} T α − 1 is an isometry, so ∥ D ρ ∥ ≤ C − 1 C \|D_{\rho}\|\le C^{-1}\sqrt{C} ∥ D ρ ∥ ≤ C − 1 C . Apply this bound to γ δ − R ρ f \gamma_{\delta}-R_{\rho}f γ δ − R ρ f .
This stability estimate is in the norm of E α \mathcal{E}_{\alpha} E α ; it does not give an ambient L 2 ( μ Q ) L^{2}(\mu_Q) L 2 ( μ Q ) or pointwise error estimate.
The frame correction can therefore be undone after synthesis, before analysis, or through the coefficients using backprojection and Hermite expansion. For approximation we use the density formulation directly: W ρ g W_{\rho}g W ρ g represents the spectral target f g f_g f g . The finite-dimensional weighted case, where the correction is a familiar Fourier multiplier, is recorded separately in Corollary D.2 .
Remark 4.10 (Which inverse is bounded). T α − 1 : E α ′ → E α T_{\alpha}^{-1}:\mathcal{E}'_{\alpha}\to\mathcal{E}_{\alpha} T α − 1 : E α ′ → E α is bounded with norm one. No bounded inverse in the ambient L 2 ( μ Q ) L^{2}(\mu_Q) L 2 ( μ Q ) -norm is asserted; the two statements differ because E α \mathcal{E}_{\alpha} E α is completed in the spectral norm. On f ∈ D α f\in\mathcal{D}_{\alpha} f ∈ D α the inverse factorizes as T α f ↦ F Q f ↦ f μ Q ↦ f T_{\alpha}f\mapsto F_Qf\mapsto f_{\mu_Q}\mapsto f T α f ↦ F Q f ↦ f μ Q ↦ f , through the unitary coordinate, Fourier–Stieltjes uniqueness, and the Radon–Nikodym derivative; the Hermite expansion (37) implements the last two steps constructively.
Vector-valued targets
Let Y \mathcal{Y} Y be a separable complex Hilbert space. All objects above have Y \mathcal{Y} Y -valued versions: L 2 ( μ Q ; Y ) L^{2}(\mu_Q;\mathcal{Y}) L 2 ( μ Q ; Y ) , F Q f ( ξ ) = ∫ f ( x ) e − i ⟨ x , ξ ⟩ μ Q ( d x ) ∈ Y F_Qf(\xi)=\int f(x)e^{-i\langle x,\xi\rangle}\,\mu_Q(dx)\in\mathcal{Y} F Q f ( ξ ) = ∫ f ( x ) e − i ⟨ x , ξ ⟩ μ Q ( d x ) ∈ Y as a Bochner integral, the core D α ( Y ) \mathcal{D}_{\alpha}(\mathcal{Y}) D α ( Y ) , the completion E α ( Y ) \mathcal{E}_{\alpha}(\mathcal{Y}) E α ( Y ) with inner product ∫ ⟨ F Q f , F Q g ⟩ Y d ν α \int\langle F_Qf,F_Qg\rangle_{\mathcal{Y}}\,d\nu_{\alpha} ∫ ⟨ F Q f , F Q g ⟩ Y d ν α , the transform R ρ f ∈ L 2 ( λ α ; Y ) R_{\rho}f\in L^{2}(\lambda_{\alpha};\mathcal{Y}) R ρ f ∈ L 2 ( λ α ; Y ) , the coefficient W ρ g W_{\rho}g W ρ g of a density g ∈ L 2 ( ν α ; Y ) g\in L^{2}(\nu_{\alpha};\mathcal{Y}) g ∈ L 2 ( ν α ; Y ) , the target f g f_g f g , and the backprojection.
Theorem 4.11 (Vector-valued extension). Theorems 3.14 , 4.5 and 4.8 hold for Y \mathcal{Y} Y -valued targets, with the same constants, with absolute values replaced by norms in Y \mathcal{Y} Y , scalar integrals by Bochner integrals, and L 2 L^{2} L 2 spaces by their Y \mathcal{Y} Y -valued counterparts. The Riesz map and its inverse remain isometries; their operator norms are one if Y ≠ { 0 } \mathcal{Y}\ne\{0\} Y = { 0 } and zero if Y = { 0 } \mathcal{Y}=\{0\} Y = { 0 } .
Proof. Use Lemma 3.6 for bias Fourier representatives and Plancherel, Lemma 4.1 for spectral synthesis and uniqueness, and Lemma 4.3 for joint absolute integrability and coefficient moments. The proof of Lemma 3.9 applies with Hilbert norms and Bochner integrals. These facts justify the Fubini and Parseval steps of Theorems 3.14 and 4.5 with the same constants. Completing the vector core and applying Riesz representation on E α ( Y ) \mathcal{E}_{\alpha}(\mathcal{Y}) E α ( Y ) proves Theorem 4.8 ; its Gaussian Hermite formula is applied componentwise. If Y ≠ { 0 } \mathcal{Y}\ne\{0\} Y = { 0 } , any nonzero constant Y \mathcal{Y} Y -valued function belongs to the core by Lemma 3.12 , so the isometries have norm one; if Y = { 0 } \mathcal{Y}=\{0\} Y = { 0 } , both spaces and operator norms are zero. The regularity condition in Definition 4.2 uses ∥ ⋅ ∥ Y \|\cdot\|_{\mathcal{Y}} ∥ ⋅ ∥ Y in place of absolute value. For the integral representation with a continuous activation of polynomial growth, the fixed-support C m C^{m} C m Bochner argument of the proof of Theorem 4.5 works in Y \mathcal{Y} Y -valued test functions, with the distributional pairing tensored with the identity of Y \mathcal{Y} Y .
For Y \mathcal{Y} Y -valued networks the coefficient W ρ g W_{\rho}g W ρ g takes values in Y \mathcal{Y} Y ; sampling it is discussed in Section 6.4.
Reconstruction formulas and activation functions
We extend reconstruction to tempered activations and give conditions under which synthesis is an absolutely convergent network integral. ReLU and explicit filters connect these formulas to the network approximation results that follow.
The analysis filter ρ \rho ρ is a Schwartz function, but the activation that synthesizes a network may be unbounded. Following the distributional ridgelet theory of Sonoda and Murata [87 ] and the weighted Sobolev formulation of Sonoda et al. [90 ] , we take σ ∈ S ′ ( R ) \sigma\in\mathcal{S}'(\mathbb{R}) σ ∈ S ′ ( R ) real valued for synthesis and pair its Fourier transform with the band-pass filter. Since ρ ♯ \rho^{\sharp} ρ ♯ vanishes near the origin, the pairing
( ( σ , ρ ) ) α = 1 2 π ⟨ σ ♯ , ρ ♯ ( − ⋅ ) ∣ ⋅ ∣ − α ⟩ (40) ((\sigma,\rho))_{\alpha}=\frac{1}{2\pi}\left\langle\sigma^{\sharp},\rho^{\sharp}(-\cdot)|\cdot|^{-\alpha}\right\rangle \tag*{(40)} (( σ , ρ ) ) α = 2 π 1 ⟨ σ ♯ , ρ ♯ ( − ⋅ ) ∣ ⋅ ∣ − α ⟩ (40)
of Theorem 4.5 (iii) is well defined for every tempered σ \sigma σ , and no value of σ ♯ \sigma^{\sharp} σ ♯ at the singular point 0 is used. The weighted Sobolev activation spaces A s , t = ⟨ ⋅ ⟩ t H s ( R ) A_{s,t}=\langle\cdot\rangle^{t}H^{s}(\mathbb{R}) A s , t = ⟨ ⋅ ⟩ t H s ( R ) of Sonoda et al. [90 ] , in which this pairing is continuous, are defined below.
Weighted Sobolev activation spaces
Write ⟨ u ⟩ = ( 1 + u 2 ) 1 / 2 \langle u\rangle=(1+u^{2})^{1/2} ⟨ u ⟩ = ( 1 + u 2 ) 1/2 and define the Bessel operator on the frequency variable by B q v : = F R [ ⟨ ⋅ ⟩ q F R − 1 v ] B^{q}v:=\mathcal{F}_{\mathbb{R}}[\langle\cdot\rangle^{q}\mathcal{F}_{\mathbb{R}}^{-1}v] B q v := F R [⟨ ⋅ ⟩ q F R − 1 v ] . For s ∈ R s\in\mathbb{R} s ∈ R , t ≥ 0 t\geq0 t ≥ 0 , the weighted Sobolev activation space of Sonoda et al. [90 ] is
A s , t : = ⟨ ⋅ ⟩ t H s ( R ) ⊂ S ′ ( R ) , ∥ σ ∥ A s , t : = ∥ ⟨ ω ⟩ s B − t σ ♯ ∥ L 2 , (41) A_{s,t}:=\langle\cdot\rangle^{t}H^{s}(\mathbb{R})\subset\mathcal{S}'(\mathbb{R}),\qquad\|\sigma\|_{A_{s,t}}:=\|\langle\omega\rangle^{s}B^{-t}\sigma^{\sharp}\|_{L^{2}}, \tag*{(41)} A s , t := ⟨ ⋅ ⟩ t H s ( R ) ⊂ S ′ ( R ) , ∥ σ ∥ A s , t := ∥ ⟨ ω ⟩ s B − t σ ♯ ∥ L 2 , (41)
and the dual test norm is ∥ r ∥ T s , t ♯ : = ∥ ⟨ ω ⟩ − s B t r ∥ L 2 \|r\|_{\mathcal{T}^{\sharp}_{s,t}}:=\|\langle\omega\rangle^{-s}B^{t}r\|_{L^{2}} ∥ r ∥ T s , t ♯ := ∥ ⟨ ω ⟩ − s B t r ∥ L 2 for r ∈ S ( R ) r\in\mathcal{S}(\mathbb{R}) r ∈ S ( R ) . Membership in A s , t A_{s,t} A s , t controls the distributional pairing; it does not by itself provide a continuous pointwise representative.
Lemma 5.1 (Hilbert structure and continuous activation pairing). The map σ ↦ ⟨ ω ⟩ s B − t σ ♯ \sigma\mapsto\langle\omega\rangle^{s}B^{-t}\sigma^{\sharp} σ ↦ ⟨ ω ⟩ s B − t σ ♯ is an isometric isomorphism A s , t → L 2 ( R ) A_{s,t}\to L^{2}(\mathbb{R}) A s , t → L 2 ( R ) , and
∣ 1 2 π ⟨ σ ♯ , r ⟩ ∣ ≤ 1 2 π ∥ σ ∥ A s , t ∥ r ∥ T s , t ♯ . (42) \left|\frac{1}{2\pi}\langle\sigma^{\sharp},r\rangle\right|\leq\frac{1}{2\pi}\|\sigma\|_{A_{s,t}}\|r\|_{\mathcal{T}^{\sharp}_{s,t}}. \tag*{(42)} 2 π 1 ⟨ σ ♯ , r ⟩ ≤ 2 π 1 ∥ σ ∥ A s , t ∥ r ∥ T s , t ♯ . (42) Thus the pairing extends to the completion of the test filters in T s , t ♯ \mathcal{T}^{\sharp}_{s,t} T s , t ♯ .
Proof. For g ∈ L 2 g\in L^{2} g ∈ L 2 , σ = ⟨ ⋅ ⟩ t F R − 1 [ ⟨ ω ⟩ − s g ] \sigma=\langle\cdot\rangle^{t}\mathcal{F}_{\mathbb{R}}^{-1}[\langle\omega\rangle^{-s}g] σ = ⟨ ⋅ ⟩ t F R − 1 [⟨ ω ⟩ − s g ] is a preimage, proving surjectivity and completeness. The multiplier ⟨ u ⟩ t \langle u\rangle^{t} ⟨ u ⟩ t is real and even, so B t B^{t} B t is symmetric for the bilinear pairing, and for Schwartz σ , r \sigma,r σ , r
⟨ σ ♯ , r ⟩ = ∫ ( ⟨ ω ⟩ s B − t σ ♯ ) ( ⟨ ω ⟩ − s B t r ) d ω . \langle\sigma^{\sharp},r\rangle=\int(\langle\omega\rangle^{s}B^{-t}\sigma^{\sharp})(\langle\omega\rangle^{-s}B^{t}r)\,d\omega. ⟨ σ ♯ , r ⟩ = ∫ (⟨ ω ⟩ s B − t σ ♯ ) (⟨ ω ⟩ − s B t r ) d ω .
Schwartz functions are dense in A s , t A_{s,t} A s , t by the isomorphism, and convergence there implies convergence in S ′ \mathcal{S}' S ′ ; the identity extends by density and Cauchy–Schwarz proves the bound.
Lemma 5.2 (Standard activations and admissible test filters). ReLU, tanh, the Gaussian distribution function, and e − u 2 / 2 e^{-u^{2}/2} e − u 2 /2 belong to A 0 , 2 A_{0,2} A 0 , 2 , are globally Lipschitz, and are not polynomials. For every non-polynomial real σ ∈ S ′ \sigma\in\mathcal{S}' σ ∈ S ′ there is a real band-pass ρ \rho ρ with ( ( σ , ρ ) ) α = 1 ((\sigma,\rho))_{\alpha}=1 (( σ , ρ ) ) α = 1 .
Proof. Membership in A 0 , 2 \mathcal{A}_{0,2} A 0 , 2 means ⟨ u ⟩ − 2 σ ∈ L 2 \langle u\rangle^{-2}\sigma\in L^2 ⟨ u ⟩ − 2 σ ∈ L 2 ; three of the functions are bounded, and ReLU satisfies ∫ R u 2 ( 1 + u 2 ) − 2 d u < ∞ \int_{\mathbb{R}} u^2(1+u^2)^{-2}\,du < \infty ∫ R u 2 ( 1 + u 2 ) − 2 d u < ∞ . Their derivatives are bounded wherever defined. The bounded functions are nonconstant and ReLU is not smooth at zero, so none is a polynomial. The last statement is the final part of the proof of Theorem 4.5 , followed by rescaling ρ \rho ρ .
Reconstruction with tempered activations
We now use band-pass regularizations to define synthesis in the dual space.
Definition 5.3 (Regularized synthesis). Let σ ∈ S ′ ( R ) \sigma\in\mathcal{S}'(\mathbb{R}) σ ∈ S ′ ( R ) be real and let ρ \rho ρ be a band-pass filter. Choose an even χ ∈ C c ∞ ( R ∖ { 0 } ) \chi\in C^\infty_c(\mathbb{R}\setminus\{0\}) χ ∈ C c ∞ ( R ∖ { 0 }) equal to one on a neighbourhood of supp ρ ♯ \operatorname{supp}\rho^\sharp supp ρ ♯ , and an even, compactly supported, smooth approximate identity ( η ε ) ε > 0 (\eta_\varepsilon)_{\varepsilon>0} ( η ε ) ε > 0 . Define the real Schwartz functions σ ε \sigma_\varepsilon σ ε by
σ ε ♯ : = χ ( σ ♯ ∗ η ε ) ∈ C c ∞ ( R ∖ { 0 } ) , (43) \sigma_\varepsilon^\sharp:= \chi(\sigma^\sharp* \eta_\varepsilon) \in C^\infty_c(\mathbb{R}\setminus\{0\}), \tag*{(43)} σ ε ♯ := χ ( σ ♯ ∗ η ε ) ∈ C c ∞ ( R ∖ { 0 }) , (43) and, for γ ∈ Ran R ρ ⊂ L 2 ( λ α ) \gamma\in\operatorname{Ran} R_\rho\subset L^2(\lambda_\alpha) γ ∈ Ran R ρ ⊂ L 2 ( λ α ) , the regularized synthesis S σ ε γ : = R σ ε ′ γ ∈ E α ′ S_{\sigma_\varepsilon}\gamma:= R_{\sigma_\varepsilon}'\gamma\in\mathcal{E}'_\alpha S σ ε γ := R σ ε ′ γ ∈ E α ′ as in (36). The synthesis with σ \sigma σ is
S σ γ : = lim ε ↓ 0 S σ ε γ in E α ′ , (44) S_\sigma\gamma:= \lim_{\varepsilon\downarrow0} S_{\sigma_\varepsilon}\gamma\quad\text{in } \mathcal{E}'_\alpha, \tag*{(44)} S σ γ := ε ↓ 0 lim S σ ε γ in E α ′ , (44) whenever the limit exists.
Theorem 5.4 (Reconstruction with a tempered activation). Let σ ∈ S ′ ( R ) \sigma\in\mathcal{S}'(\mathbb{R}) σ ∈ S ′ ( R ) be real and let ρ \rho ρ be a band-pass filter. For every f ∈ E α f \in\mathcal{E}_\alpha f ∈ E α the limit (44) exists for γ = R ρ f \gamma= R_\rho f γ = R ρ f , does not depend on χ \chi χ or ( η ε ) (\eta_\varepsilon) ( η ε ) , and
S σ R ρ f = ( ( σ , ρ ) ) α T α f . (45) S_\sigma R_\rho f = ((\sigma,\rho))_\alpha T_\alpha f. \tag*{(45)} S σ R ρ f = (( σ , ρ ) ) α T α f . (45) If ( ( σ , ρ ) ) α ≠ 0 ((\sigma,\rho))_\alpha\ne0 (( σ , ρ ) ) α = 0 , then
f = ( ( σ , ρ ) ) α − 1 T α − 1 S σ R ρ f , g = ( ( σ , ρ ) ) α − 1 S σ ( R ρ T α − 1 g ) , f ∈ E α , g ∈ E α ′ . (46) f = ((\sigma,\rho))_\alpha^{-1}T_\alpha^{-1}S_\sigma R_\rho f,\qquad
g = ((\sigma,\rho))_\alpha^{-1}S_\sigma(R_\rho T_\alpha^{-1}g),\qquad
f \in\mathcal{E}_\alpha,\ g \in\mathcal{E}'_\alpha. \tag*{(46)} f = (( σ , ρ ) ) α − 1 T α − 1 S σ R ρ f , g = (( σ , ρ ) ) α − 1 S σ ( R ρ T α − 1 g ) , f ∈ E α , g ∈ E α ′ . (46) If σ \sigma σ is not a polynomial, equivalently σ ≠ 0 \sigma\ne0 σ = 0 in S ′ / P \mathcal{S}'/\mathcal{P} S ′ / P with P \mathcal{P} P the polynomials, then a band-pass ρ \rho ρ with ( ( σ , ρ ) ) α ≠ 0 ((\sigma,\rho))_\alpha\ne0 (( σ , ρ ) ) α = 0 exists.
Proof. Since σ \sigma σ is real, σ ♯ \sigma^\sharp σ ♯ is Hermitian symmetric, σ ♯ ( − ω ) = σ ♯ ( ω ) ‾ \sigma^\sharp(-\omega)=\overline{\sigma^\sharp(\omega)} σ ♯ ( − ω ) = σ ♯ ( ω ) in the distributional sense; as χ \chi χ and η ε \eta_\varepsilon η ε are even and real, σ ε ♯ = χ ( σ ♯ ∗ η ε ) \sigma_\varepsilon^\sharp=\chi(\sigma^\sharp*\eta_\varepsilon) σ ε ♯ = χ ( σ ♯ ∗ η ε ) is Hermitian symmetric, so σ ε \sigma_\varepsilon σ ε is a real Schwartz function with σ ε ♯ ∈ C c ∞ ( R ∖ { 0 } ) \sigma_\varepsilon^\sharp\in C^\infty_c(\mathbb{R}\setminus\{0\}) σ ε ♯ ∈ C c ∞ ( R ∖ { 0 }) . Hence σ ε \sigma_\varepsilon σ ε is α \alpha α -admissible whenever it is nonzero, and Theorem 3.14 (i) applies to the pair ( ρ , σ ε ) (\rho,\sigma_\varepsilon) ( ρ , σ ε ) . With S σ ε = R σ ε ′ S_{\sigma_\varepsilon}=R_{\sigma_\varepsilon}' S σ ε = R σ ε ′ as in (36), for f ∈ D α f\in\mathcal{D}_\alpha f ∈ D α and g ∈ E α g\in\mathcal{E}_\alpha g ∈ E α ,
( S σ ε R ρ f ) [ g ] = ⟨ R ρ f , R σ ε g ⟩ L 2 ( λ α ) = 1 2 π ∫ R ρ ♯ ( ω ) σ ε ♯ ( ω ) ‾ ∣ ω ∣ − α d ω ⟨ f , g ⟩ E α = ( ( σ ε , ρ ) ) α T α f [ g ] , (S_{\sigma_\varepsilon}R_\rho f)[g]
= \langle R_\rho f,R_{\sigma_\varepsilon}g\rangle_{L^2(\lambda_\alpha)}
= \frac{1}{2\pi}\int_{\mathbb{R}}\rho^\sharp(\omega)\overline{\sigma_\varepsilon^\sharp(\omega)}|\omega|^{-\alpha}\,d\omega\,\langle f,g\rangle_{\mathcal{E}_\alpha}
= ((\sigma_\varepsilon,\rho))_\alpha T_\alpha f[g], ( S σ ε R ρ f ) [ g ] = ⟨ R ρ f , R σ ε g ⟩ L 2 ( λ α ) = 2 π 1 ∫ R ρ ♯ ( ω ) σ ε ♯ ( ω ) ∣ ω ∣ − α d ω ⟨ f , g ⟩ E α = (( σ ε , ρ ) ) α T α f [ g ] ,
where the last equality uses Hermitian symmetry, σ ε ♯ ( ω ) = σ ε ♯ ( − ω ) ‾ \sigma_\varepsilon^\sharp(\omega)=\overline{\sigma_\varepsilon^\sharp(-\omega)} σ ε ♯ ( ω ) = σ ε ♯ ( − ω ) , and the substitution ω ↦ − ω \omega\mapsto-\omega ω ↦ − ω . Both sides are continuous in f ∈ E α f\in\mathcal{E}_\alpha f ∈ E α , so the identity holds on E α \mathcal{E}_\alpha E α . Since χ = 1 \chi=1 χ = 1 on a neighbourhood of supp ρ ♯ \operatorname{supp}\rho^\sharp supp ρ ♯ , the pairing defining ( ( σ ε , ρ ) ) α ((\sigma_\varepsilon,\rho))_\alpha (( σ ε , ρ ) ) α equals 1 2 π ⟨ σ ♯ ∗ η ε , ρ ♯ ( − ⋅ ) ∣ ⋅ ∣ − α ⟩ \frac{1}{2\pi}\langle\sigma^\sharp*\eta_\varepsilon,\rho^\sharp(-\cdot)|\cdot|^{-\alpha}\rangle 2 π 1 ⟨ σ ♯ ∗ η ε , ρ ♯ ( − ⋅ ) ∣ ⋅ ∣ − α ⟩ , and distributional convergence σ ♯ ∗ η ε → σ ♯ \sigma^\sharp*\eta_\varepsilon\to\sigma^\sharp σ ♯ ∗ η ε → σ ♯ against the fixed test function ρ ♯ ( − ω ) ∣ ω ∣ − α \rho^\sharp(-\omega)|\omega|^{-\alpha} ρ ♯ ( − ω ) ∣ ω ∣ − α gives ( ( σ ε , ρ ) ) α → ( ( σ , ρ ) ) α ((\sigma_\varepsilon,\rho))_\alpha\to((\sigma,\rho))_\alpha (( σ ε , ρ ) ) α → (( σ , ρ ) ) α . Since T α T_\alpha T α is an isometry,
∥ S σ ε R ρ f − ( ( σ , ρ ) ) α T α f ∥ E α ′ = ∣ ( ( σ ε , ρ ) ) α − ( ( σ , ρ ) ) α ∣ ∥ f ∥ E α ⟶ 0. \lVert S_{\sigma_\varepsilon}R_\rho f-((\sigma,\rho))_\alpha T_\alpha f\rVert_{\mathcal{E}'_\alpha}
=|((\sigma_\varepsilon,\rho))_\alpha-((\sigma,\rho))_\alpha|\lVert f\rVert_{\mathcal{E}_\alpha}\longrightarrow0. ∥ S σ ε R ρ f − (( σ , ρ ) ) α T α f ∥ E α ′ = ∣ (( σ ε , ρ ) ) α − (( σ , ρ ) ) α ∣ ∥ f ∥ E α ⟶ 0.
The limit depends only on the pairing (40), hence not on χ \chi χ or ( η ε ) (\eta_\varepsilon) ( η ε ) . Applying T α − 1 T_\alpha^{-1} T α − 1 , respectively substituting f = T α − 1 g f=T_\alpha^{-1}g f = T α − 1 g , proves (46). The existence of a band-pass ρ \rho ρ with nonzero constant for non-polynomial σ \sigma σ was proved at the end of the proof of Theorem 4.5 .
The argument uses the direction measure only through the homogeneity (9) and the finiteness of the admissibility constants of ρ \rho ρ and of the regularizations σ ε \sigma_{\varepsilon} σ ε ; neither the positivity of α \alpha α nor the full support of ν α \nu_{\alpha} ν α enters (45) or the independence of χ \chi χ and ( η ε ) (\eta_{\varepsilon}) ( η ε ) . Full support is used earlier, in Theorem 3.14 , to make R ρ R_{\rho} R ρ injective.
Here a synthesis activation is called admissible if it can be paired with a real band-pass analysis filter to give a nonzero reconstruction constant ( ( σ , ρ ) ) α ((\sigma,\rho))_{\alpha} (( σ , ρ ) ) α . This is a condition on the pair ( σ , ρ ) (\sigma,\rho) ( σ , ρ ) ; the self-admissibility condition (10) concerns the analysis filter alone.
Corollary 5.5 (ReLU is admissible). Let σ ( t ) = ReLU ( t ) = max ( t , 0 ) \sigma(t)=\operatorname{ReLU}(t)=\max(t,0) σ ( t ) = ReLU ( t ) = max ( t , 0 ) . Under the convention (1),
ReLU ♯ = − fp ( ω − 2 ) + i π δ 0 ′ , (47) \operatorname{ReLU}^{\sharp}=-\operatorname{fp}(\omega^{-2})+i\pi\delta'_0, \tag*{(47)} ReLU ♯ = − fp ( ω − 2 ) + iπ δ 0 ′ , (47) which equals − ω − 2 -\omega^{-2} − ω − 2 away from the origin. If ρ ♯ ∈ C c ∞ ( R ∖ { 0 } ) \rho^{\sharp}\in C^\infty_c(\mathbb{R}\setminus\{0\}) ρ ♯ ∈ C c ∞ ( R ∖ { 0 }) is nonzero, even, and nonpositive, then
( ( ReLU , ρ ) ) α = − 1 2 π ∫ R ρ ♯ ( ω ) ∣ ω ∣ − α − 2 d ω > 0. (48) ((\operatorname{ReLU},\rho))_{\alpha}=-\frac{1}{2\pi}\int_{\mathbb{R}}\rho^{\sharp}(\omega)|\omega|^{-\alpha-2}\,\mathrm{d}\omega>0. \tag*{(48)} (( ReLU , ρ ) ) α = − 2 π 1 ∫ R ρ ♯ ( ω ) ∣ ω ∣ − α − 2 d ω > 0. (48) After rescaling ρ \rho ρ the constant is one, and (46) and Theorem 4.5 (iii) hold with ReLU synthesis for every α > 0 \alpha>0 α > 0 .
Proof. From ReLU ( t ) = ( ∣ t ∣ + t ) / 2 \operatorname{ReLU}(t)=(|t|+t)/2 ReLU ( t ) = ( ∣ t ∣ + t ) /2 , the identities ( ∣ t ∣ ) ♯ = − 2 fp ( ω − 2 ) (|t|)^{\sharp}=-2\operatorname{fp}(\omega^{-2}) ( ∣ t ∣ ) ♯ = − 2 fp ( ω − 2 ) and t ♯ = 2 π i δ 0 ′ t^{\sharp}=2\pi i\delta'_0 t ♯ = 2 π i δ 0 ′ give (47). The test function in (40) is supported away from zero, so the δ 0 ′ \delta'_0 δ 0 ′ term vanishes and the finite part is ordinary multiplication by ω − 2 \omega^{-2} ω − 2 ; evenness of ρ ♯ \rho^{\sharp} ρ ♯ and its sign give (48).
Example 5.6 (Standard activations). ReLU, tanh, the Gaussian distribution function, and the Gaussian e − u 2 / 2 e^{-u^2/2} e − u 2 /2 are globally Lipschitz, are not polynomials, and belong to A 0 , 2 = ⟨ ⋅ ⟩ 2 L 2 ( R ) A_{0,2}=\langle\cdot\rangle^2L^2(\mathbb{R}) A 0 , 2 = ⟨ ⋅ ⟩ 2 L 2 ( R ) ; see Lemma 5.2 . Each of them is therefore covered by Theorem 5.4 , by Theorem 4.5 (iii), and by the finite-width bounds of § 6 .
Proof. The three properties are verified in Lemma 5.2 ; a globally Lipschitz function has polynomial growth, so Theorem 4.5 (iii) applies. The activation hypotheses of the finite-width results below are therefore satisfied.
Remark 5.7 (Identity in the dual space versus integral network). (45) is an identity in E α ′ E'_{\alpha} E α ′ ; it does not by itself produce a finite coefficient measure on H × R \mathcal{H}\times\mathbb{R} H × R . When the target has a spectral density g g g satisfying Definition 4.2 , Theorem 4.5 (iii) upgrades the second formula in (46) to an absolutely convergent integral network with coefficient γ g \gamma_g γ g , and Theorem 6.6 shows that this coefficient measure has finite total variation and moments. The examples of § 7 with ReLU synthesis are of this kind.
Integral representation under Sobolev conditions
Band-pass support avoids the singular frequency at zero, but it is not necessary for an absolutely convergent network integral. The next theorem replaces smoothness of g ( ω ) = g ( ω a ) g(\omega)=g(\omega a) g ( ω ) = g ( ω a ) by a condition on the inverse Fourier transform of h a ( ω ) : = ρ ♯ ( − ω ) g a ( ω ) h_a(\omega):=\rho^{\sharp}(-\omega)g_a(\omega) h a ( ω ) := ρ ♯ ( − ω ) g a ( ω ) . This condition also allows non-band-pass filters. We first give the weighted inverse Fourier estimates and the dual pairing used in the proof. The direction measure may be ν α \nu_{\alpha} ν α , or any homogeneous sigma-finite measure as specified below. The Fourier pairing in the statement is its bounded Sobolev-dual extension, defined precisely in Lemma 5.9 .
Write ⟨ t ⟩ = ( 1 + t 2 ) 1 / 2 \langle t\rangle=(1+t^2)^{1/2} ⟨ t ⟩ = ( 1 + t 2 ) 1/2 , and use the angular Fourier convention
f ♯ ( ω ) = ∫ R f ( t ) e − i t ω d t , h ~ ( t ) = 1 2 π ∫ R h ( ω ) e i t ω d ω . f^{\sharp}(\omega)=\int_{\mathbb{R}}f(t)e^{-it\omega}\,\mathrm{d}t,\qquad\widetilde{h}(t)=\frac{1}{2\pi}\int_{\mathbb{R}}h(\omega)e^{it\omega}\,\mathrm{d}\omega. f ♯ ( ω ) = ∫ R f ( t ) e − i t ω d t , h ( t ) = 2 π 1 ∫ R h ( ω ) e i t ω d ω .
Here Y \mathcal{Y} Y is the standing separable complex Hilbert space. The inverse transform is understood in L 2 L^2 L 2 when the integral need not exist pointwise. For s ≥ 0 s \ge0 s ≥ 0 , define the Bessel-potential space in the frequency variable by
H ω s ( R ; Y ) = { h ∈ L 2 ( R ; Y ) : ⟨ t ⟩ s h ~ ( t ) ∈ L 2 } , ∥ h ∥ H ω s 2 = 2 π ∫ R ⟨ t ⟩ 2 s ∥ h ~ ( t ) ∥ Y 2 d t . (49) H^s_\omega(\mathbb{R};\mathcal{Y})=\{h\in L^2(\mathbb{R};\mathcal{Y}):\langle t\rangle^s\widetilde{h}(t)\in L^2\},\qquad\|h\|_{H^s_\omega}^2=2\pi\int_{\mathbb{R}}\langle t\rangle^{2s}\|\widetilde{h}(t)\|_{\mathcal{Y}}^2\,dt. \tag*{(49)} H ω s ( R ; Y ) = { h ∈ L 2 ( R ; Y ) : ⟨ t ⟩ s h ( t ) ∈ L 2 } , ∥ h ∥ H ω s 2 = 2 π ∫ R ⟨ t ⟩ 2 s ∥ h ( t ) ∥ Y 2 d t . (49)
Thus ∥ h ∥ H ω 0 = ∥ h ∥ 2 \|h\|_{H^0_\omega}=\|h\|_2 ∥ h ∥ H ω 0 = ∥ h ∥ 2 by Plancherel. The displayed norm is the Bessel-potential norm for this Fourier normalization. It is a separable Hilbert space: the map h ↦ 2 π ⟨ ⋅ ⟩ s h ~ h\mapsto\sqrt{2\pi}\langle\cdot\rangle^s\widetilde{h} h ↦ 2 π ⟨ ⋅ ⟩ s h is an isometric isomorphism onto L 2 ( R ; Y ) L^2(\mathbb{R};\mathcal{Y}) L 2 ( R ; Y ) . For Banach-valued data in Theorem 5.10 , only the weighted inverse-transform norm in (49) is used. The weighted L 1 L^1 L 1 estimate below follows directly from Cauchy–Schwarz for that norm, without a Fourier isometry.
Lemma 5.8 (Weighted inverse Fourier estimates and modulation). Let s > 1 / 2 s>1/2 s > 1/2 , 0 ≤ r < s − 1 / 2 0\le r<s-1/2 0 ≤ r < s − 1/2 , and put
A s , r : = 1 2 π ( ∫ R ( 1 + t 2 ) − ( s − r ) d t ) 1 / 2 . A_{s,r}:=\frac{1}{\sqrt{2\pi}}\left(\int_{\mathbb{R}}(1+t^2)^{-(s-r)}\,dt\right)^{1/2}. A s , r := 2 π 1 ( ∫ R ( 1 + t 2 ) − ( s − r ) d t ) 1/2 . For every h ∈ H ω s ( R ; Y ) h\in H^s_\omega(\mathbb{R};\mathcal{Y}) h ∈ H ω s ( R ; Y ) ,
∫ R ⟨ t ⟩ r ∥ h ~ ( t ) ∥ Y d t ≤ A s , r ∥ h ∥ H ω s . (50) \int_{\mathbb{R}}\langle t\rangle^r\|\widetilde{h}(t)\|_{\mathcal{Y}}\,dt\le A_{s,r}\|h\|_{H^s_\omega}. \tag*{(50)} ∫ R ⟨ t ⟩ r ∥ h ( t ) ∥ Y d t ≤ A s , r ∥ h ∥ H ω s . (50) Reflection R h ( ω ) = h ( − ω ) Rh(\omega)=h(-\omega) R h ( ω ) = h ( − ω ) is an isometry on this space. For M u h ( ω ) = e i u ω h ( ω ) M_uh(\omega)=e^{iu\omega}h(\omega) M u h ( ω ) = e i u ω h ( ω ) ,
∥ M u h ∥ H ω s ≤ ( 1 + ∣ u ∣ ) s ∥ h ∥ H ω s . (51) \|M_uh\|_{H^s_\omega}\le(1+|u|)^s\|h\|_{H^s_\omega}. \tag*{(51)} ∥ M u h ∥ H ω s ≤ ( 1 + ∣ u ∣ ) s ∥ h ∥ H ω s . (51) The map ( u , h ) ↦ M u h (u,h)\mapsto M_uh ( u , h ) ↦ M u h is jointly continuous.
Proof. Cauchy–Schwarz, applied to ⟨ t ⟩ − ( s − r ) \langle t\rangle^{-(s-r)} ⟨ t ⟩ − ( s − r ) and ⟨ t ⟩ s ∥ h ~ ( t ) ∥ Y \langle t\rangle^s\|\widetilde{h}(t)\|_{\mathcal{Y}} ⟨ t ⟩ s ∥ h ( t ) ∥ Y , proves (50) . The scalar integral is finite precisely when s − r > 1 / 2 s-r>1/2 s − r > 1/2 . Fourier inversion in L 2 L^2 L 2 gives F R − 1 ( R h ) ( t ) = h ~ ( − t ) \mathcal{F}_{\mathbb{R}}^{-1}(Rh)(t)=\widetilde{h}(-t) F R − 1 ( R h ) ( t ) = h ( − t ) and F R − 1 ( M u h ) ( t ) = h ~ ( t + u ) \mathcal{F}_{\mathbb{R}}^{-1}(M_uh)(t)=\widetilde{h}(t+u) F R − 1 ( M u h ) ( t ) = h ( t + u ) . Since ⟨ t − u ⟩ ≤ ⟨ t ⟩ + ∣ u ∣ ≤ ( 1 + ∣ u ∣ ) ⟨ t ⟩ \langle t-u\rangle\le\langle t\rangle+|u|\le(1+|u|)\langle t\rangle ⟨ t − u ⟩ ≤ ⟨ t ⟩ + ∣ u ∣ ≤ ( 1 + ∣ u ∣ ) ⟨ t ⟩ , changing variables proves the reflection and modulation estimates. Translations are strongly continuous in L 2 ( R , ⟨ t ⟩ 2 s d t ; Y ) L^2(\mathbb{R},\langle t\rangle^{2s}dt;\mathcal{Y}) L 2 ( R , ⟨ t ⟩ 2 s d t ; Y ) : this holds by dominated convergence for continuous compactly supported functions with values in a finite-dimensional subspace of Y \mathcal{Y} Y ; these functions are dense, and the displayed translation bound is uniform for u u u in a bounded interval. Approximation therefore proves strong continuity for every element. Finally, for u n → u u_n\to u u n → u and h n → h h_n\to h h n → h , split M u n h n − M u h M_{u_n}h_n-M_uh M u n h n − M u h into M u n ( h n − h ) + ( M u n − M u ) h M_{u_n}(h_n-h)+(M_{u_n}-M_u)h M u n ( h n − h ) + ( M u n − M u ) h , and apply the local uniform bound and strong continuity.
Only the weighted L 1 L^1 L 1 estimate (50) and the reflection isometry are used for Theorem 5.10 ; the modulation bound (51) and the joint continuity are the mapping properties that the alternative route through an H ω s H^s_\omega H ω s -valued Bochner integral would need, and are recorded because they identify the bias translation (53) as a modulation.
Lemma 5.9 (The bilinear Sobolev pairing and its translation formula). Suppose σ : R → C \sigma:\mathbb{R}\to\mathbb{C} σ : R → C is continuous and ∣ σ ( t ) ∣ ≤ C σ ( 1 + ∣ t ∣ ) p |\sigma(t)|\le C_\sigma(1+|t|)^p ∣ σ ( t ) ∣ ≤ C σ ( 1 + ∣ t ∣ ) p , where p ≥ 0 p\ge0 p ≥ 0 , and s > p + 1 / 2 s>p+1/2 s > p + 1/2 . Set b σ , s = ∥ ⟨ ⋅ ⟩ − s σ ∥ 2 < ∞ b_{\sigma,s}=\|\langle\cdot\rangle^{-s}\sigma\|_2<\infty b σ , s = ∥ ⟨ ⋅ ⟩ − s σ ∥ 2 < ∞ . Its Fourier transform defines an element of the bilinear dual H ω − s H^{-s}_\omega H ω − s , with the following normalization:
1 2 π ⟨ σ # , q ⟩ = ∫ R σ ( t ) q ~ ( − t ) d t , q ∈ H ω s ( R ) . (52) \frac{1}{2\pi}\langle\sigma^\#,q\rangle=\int_{\mathbb{R}}\sigma(t)\widetilde{q}(-t)\,dt,\qquad q\in H^s_\omega(\mathbb{R}). \tag*{(52)} 2 π 1 ⟨ σ # , q ⟩ = ∫ R σ ( t ) q ( − t ) d t , q ∈ H ω s ( R ) . (52) and the norm of this functional is at most ( 2 π ) − 1 / 2 b σ , s (2\pi)^{-1/2}b_{\sigma,s} ( 2 π ) − 1/2 b σ , s . For Y \mathcal{Y} Y -valued tests the right side defines a bounded linear map L σ Y : H ω s ( R ; Y ) → Y L^\mathcal{Y}_\sigma:H^s_\omega(\mathbb{R};\mathcal{Y})\to\mathcal{Y} L σ Y : H ω s ( R ; Y ) → Y with the same bound. If h ∈ H ω s ( R ; Y ) h\in H^s_\omega(\mathbb{R};\mathcal{Y}) h ∈ H ω s ( R ; Y ) , γ = h ~ \gamma=\widetilde{h} γ = h , and u ∈ R u\in\mathbb{R} u ∈ R , then the Bochner integral below is absolute and
∫ R σ ( u − b ) γ ( b ) d b = L σ Y ( M u h ) . (53) \int_{\mathbb{R}}\sigma(u-b)\gamma(b)\,db=L^\mathcal{Y}_\sigma(M_uh). \tag*{(53)} ∫ R σ ( u − b ) γ ( b ) d b = L σ Y ( M u h ) . (53)
Proof. The growth assumption makes b σ , s b_{\sigma,s} b σ , s finite. Weighted Cauchy–Schwarz proves
∫ R ∣ σ ( t ) ∣ ∥ q ˇ ( − t ) ∥ Y d t ≤ b σ , s 2 π ∥ q ∥ H ω s , \int_{\mathbb{R}}|\sigma(t)|\|\check{q}(-t)\|_{\mathcal{Y}}\,dt \leq\frac{b_{\sigma,s}}{\sqrt{2\pi}}\|q\|_{H^s_\omega}, ∫ R ∣ σ ( t ) ∣∥ q ˇ ( − t ) ∥ Y d t ≤ 2 π b σ , s ∥ q ∥ H ω s ,
so the claimed integrals and bounded maps exist. For a scalar Schwartz test q q q , the definition of the distributional Fourier transform and q ♭ ^ ( t ) = 2 π q ˇ ( − t ) \widehat{q^\flat}(t)=2\pi\check{q}(-t) q ♭ ( t ) = 2 π q ˇ ( − t ) give (52). Scalar Schwartz functions are dense in H ω s H^s_\omega H ω s : under the isomorphism following (49), the inverse Fourier transforms of Schwartz functions are dense in weighted L 2 L^2 L 2 , as follows by truncation and mollification. Thus the bounded extension agrees with the Fourier distribution, establishing the stated dual interpretation. The vector-valued construction uses the same absolute Bochner integral and does not require a scalar choice of coordinates.
Now F R − 1 ( M u h ) ( t ) = γ ( t + u ) \mathcal{F}_{\mathbb{R}}^{-1}(M_u h)(t)=\gamma(t+u) F R − 1 ( M u h ) ( t ) = γ ( t + u ) , so F R − 1 ( M u h ) ( − t ) = γ ( u − t ) \mathcal{F}_{\mathbb{R}}^{-1}(M_u h)(-t)=\gamma(u-t) F R − 1 ( M u h ) ( − t ) = γ ( u − t ) . Substitution b = u − t b=u-t b = u − t in its absolutely convergent pairing proves (53). In particular, the test is e i u ω h ( ω ) e^{iu\omega}h(\omega) e i u ω h ( ω ) , with positive modulation and no reflection, corresponding to the minus sign in the neuron bias. □
Theorem 5.10 (Integral representation under Sobolev conditions). Let H \mathcal{H} H be a real inner-product space with its Borel structure and let ν \nu ν be a sigma-finite Borel measure on H \mathcal{H} H such that, for every t ≠ 0 t\neq0 t = 0 , ( D t ) # ν = ∣ t ∣ − α ν (D_t)_\#\nu=|t|^{-\alpha}\nu ( D t ) # ν = ∣ t ∣ − α ν , where D t a = t a D_ta=ta D t a = t a and α > 0 \alpha>0 α > 0 . Let Y \mathcal{Y} Y be a complex Banach space, let ρ ∈ S ( R ; R ) \rho\in\mathcal{S}(\mathbb{R};\mathbb{R}) ρ ∈ S ( R ; R ) be nonzero, let σ : R → C \sigma:\mathbb{R}\to\mathbb{C} σ : R → C be continuous with ∣ σ ( t ) ∣ ≤ C σ ( 1 + ∣ t ∣ ) p |\sigma(t)|\leq C_\sigma(1+|t|)^p ∣ σ ( t ) ∣ ≤ C σ ( 1 + ∣ t ∣ ) p , where p ≥ 0 p\geq0 p ≥ 0 and s > p + 1 / 2 s>p+1/2 s > p + 1/2 , and let g : H → Y g:\mathcal{H}\to\mathcal{Y} g : H → Y be strongly measurable. Write g a ( ω ) : = g ( ω a ) g_a(\omega):=g(\omega a) g a ( ω ) := g ( ω a ) and h a ( ω ) : = ρ ♭ ( − ω ) g a ( ω ) h_a(\omega):=\rho^\flat(-\omega)g_a(\omega) h a ( ω ) := ρ ♭ ( − ω ) g a ( ω ) , and assume the following.
There is a jointly measurable γ g : H × R → Y \gamma_g:\mathcal{H}\times\mathbb{R}\to\mathcal{Y} γ g : H × R → Y such that, for ν \nu ν -almost every a a a , the function b ↦ γ g ( a , b ) b\mapsto\gamma_g(a,b) b ↦ γ g ( a , b ) has the weighted integrability and Fourier transform
∫ R ⟨ b ⟩ 2 s ∥ γ g ( a , b ) ∥ Y 2 d b < ∞ and ∫ R γ g ( a , b ) e − i ω b d b = h a ( ω ) for every ω ∈ R . \int_{\mathbb{R}}\langle b\rangle^{2s}\|\gamma_g(a,b)\|_{\mathcal{Y}}^2\,db<\infty
\qquad\text{and}\qquad
\int_{\mathbb{R}}\gamma_g(a,b)e^{-i\omega b}\,db=h_a(\omega)\quad\text{for every }\omega\in\mathbb{R}. ∫ R ⟨ b ⟩ 2 s ∥ γ g ( a , b ) ∥ Y 2 d b < ∞ and ∫ R γ g ( a , b ) e − iω b d b = h a ( ω ) for every ω ∈ R . When Y \mathcal{Y} Y is a Hilbert space, this is equivalently h a ∈ H ω s ( R ; Y ) h_a\in H^s_\omega(\mathbb{R};\mathcal{Y}) h a ∈ H ω s ( R ; Y ) with h ˇ a = γ g ( a , ⋅ ) \check{h}_a=\gamma_g(a,\cdot) h ˇ a = γ g ( a , ⋅ ) .
B s ( ρ , g ) : = ∫ H ( 1 + ∥ a ∥ ) s ∥ h a ∥ H ω s ν ( d a ) < ∞ \mathcal{B}_s(\rho,g):=\int_{\mathcal{H}}(1+\|a\|)^s\|h_a\|_{H^s_\omega}\,\nu(da)<\infty B s ( ρ , g ) := ∫ H ( 1 + ∥ a ∥ ) s ∥ h a ∥ H ω s ν ( d a ) < ∞ .
The Lebesgue class q α , ρ ( ω ) = ρ ♭ ( − ω ) ∣ ω ∣ − α q_{\alpha,\rho}(\omega)=\rho^\flat(-\omega)|\omega|^{-\alpha} q α , ρ ( ω ) = ρ ♭ ( − ω ) ∣ ω ∣ − α belongs to H ω s ( R ) H^s_\omega(\mathbb{R}) H ω s ( R ) . No value at ω = 0 \omega=0 ω = 0 is prescribed.
Then g ∈ L 1 ( ν ; Y ) g\in L^1(\nu;\mathcal{Y}) g ∈ L 1 ( ν ; Y ) , so that f g ( x ) = ∫ H e i ⟨ x , ξ ⟩ g ( ξ ) ν ( d ξ ) f_g(x)=\int_{\mathcal{H}}e^{i\langle x,\xi\rangle}g(\xi)\,\nu(d\xi) f g ( x ) = ∫ H e i ⟨ x , ξ ⟩ g ( ξ ) ν ( d ξ ) is defined, and γ g ∈ L 1 ( ν ⊗ d b ; Y ) \gamma_g\in L^1(\nu\otimes db;\mathcal{Y}) γ g ∈ L 1 ( ν ⊗ d b ; Y ) ; for every 0 ≤ r < s − 1 / 2 0\leq r<s-1/2 0 ≤ r < s − 1/2 ,
∫ H × R ( 1 + ∥ a ∥ + ∣ b ∣ ) r ∥ γ g ( a , b ) ∥ Y ν ( d a ) d b ≤ 2 r / 2 A s , r B s ( ρ , g ) , (54) \int_{\mathcal{H}\times\mathbb{R}}(1+\|a\|+|b|)^r\|\gamma_g(a,b)\|_{\mathcal{Y}}\,\nu(da)\,db\leq2^{r/2}A_{s,r}\mathcal{B}_s(\rho,g), \tag*{(54)} ∫ H × R ( 1 + ∥ a ∥ + ∣ b ∣ ) r ∥ γ g ( a , b ) ∥ Y ν ( d a ) d b ≤ 2 r /2 A s , r B s ( ρ , g ) , (54) so the vector measure Γ g = γ g ( ν ⊗ d b ) \Gamma_g=\gamma_g(\nu\otimes db) Γ g = γ g ( ν ⊗ d b ) is finite. For every x ∈ H x\in\mathcal{H} x ∈ H the direction average
Ψ x ( t ) : = ∫ H γ g ( a , ⟨ a , x ⟩ − t ) ν ( d a ) \Psi_x(t):=\int_{\mathcal{H}}\gamma_g(a,\langle a,x\rangle-t)\,\nu(da) Ψ x ( t ) := ∫ H γ g ( a , ⟨ a , x ⟩ − t ) ν ( d a ) converges absolutely for almost every t t t , belongs to L 1 ( R ; Y ) L^1(\mathbb{R};\mathcal{Y}) L 1 ( R ; Y ) , and
Ψ x ( t ) = q ˇ α , ρ ( − t ) f g ( x ) for almost every t ∈ R . (55) \Psi_x(t)=\check{q}_{\alpha,\rho}(-t)f_g(x)\qquad\text{for almost every }t\in\mathbb{R}. \tag*{(55)} Ψ x ( t ) = q ˇ α , ρ ( − t ) f g ( x ) for almost every t ∈ R . (55) Consequently the ordinary, absolutely convergent synthesis satisfies, for every x ∈ H x\in\mathcal{H} x ∈ H ,
S σ [ Γ g ] ( x ) : = ∫ H × R σ ( ⟨ a , x ⟩ − b ) γ g ( a , b ) ν ( d a ) d b = ( ( σ , ρ ) ) α f g ( x ) , ( ( σ , ρ ) ) α = 1 2 π ⟨ σ ♭ , q α , ρ ⟩ . (56) S_\sigma[\Gamma_g](x):=\int_{\mathcal{H}\times\mathbb{R}}\sigma(\langle a,x\rangle-b)\gamma_g(a,b)\,\nu(da)\,db=((\sigma,\rho))_\alpha f_g(x),\qquad((\sigma,\rho))_\alpha=\frac{1}{2\pi}\langle\sigma^\flat,q_{\alpha,\rho}\rangle. \tag*{(56)} S σ [ Γ g ] ( x ) := ∫ H × R σ (⟨ a , x ⟩ − b ) γ g ( a , b ) ν ( d a ) d b = (( σ , ρ ) ) α f g ( x ) , (( σ , ρ ) ) α = 2 π 1 ⟨ σ ♭ , q α , ρ ⟩ . (56)
The absolute convergence is uniform on bounded input sets, and the synthesis is continuous. No non-polynomial assumption is needed for this identity; normalization to reproduce f g f_g f g requires the separate condition ( σ , ρ ) α ≠ 0 (\sigma,\rho)_\alpha\ne0 ( σ , ρ ) α = 0 . If, in addition, Y \mathcal{Y} Y is a separable complex Hilbert space, ( ρ , ρ ) α < ∞ (\rho,\rho)_\alpha< \infty ( ρ , ρ ) α < ∞ , and g ∈ L 2 ( ν ; Y ) g \in L^2(\nu;\mathcal{Y}) g ∈ L 2 ( ν ; Y ) , then γ g ∈ L 2 ( ν ⊗ d b ; Y ) \gamma_g \in L^2(\nu\otimes\mathrm{d}b;\mathcal{Y}) γ g ∈ L 2 ( ν ⊗ d b ; Y ) with
∫ H × R ∥ γ g ( a , b ) ∥ Y 2 ν ( d a ) d b = ( ρ , ρ ) α ∥ g ∥ L 2 ( ν ; Y ) 2 . (57) \int_{\mathcal{H}\times\mathbb{R}} \|\gamma_g(a,b)\|_{\mathcal{Y}}^2 \nu(\mathrm{d}a)\,\mathrm{d}b = (\rho,\rho)_\alpha\|g\|_{L^2(\nu;\mathcal{Y})}^2. \tag*{(57)} ∫ H × R ∥ γ g ( a , b ) ∥ Y 2 ν ( d a ) d b = ( ρ , ρ ) α ∥ g ∥ L 2 ( ν ; Y ) 2 . (57)
Proof. Throughout, γ g \gamma_g γ g is the jointly measurable coefficient of hypothesis 1, and h a h_a h a is the Fourier transform of b ↦ γ g ( a , b ) b \mapsto\gamma_g(a,b) b ↦ γ g ( a , b ) , so that ∥ h a ∥ H ω s 2 = 2 π ∫ R ⟨ b ⟩ 2 s ∥ γ g ( a , b ) ∥ Y 2 d b \|h_a\|_{H^s_\omega}^2 = 2\pi\int_{\mathbb{R}} \langle b\rangle^{2s}\|\gamma_g(a,b)\|_{\mathcal{Y}}^2\,\mathrm{d}b ∥ h a ∥ H ω s 2 = 2 π ∫ R ⟨ b ⟩ 2 s ∥ γ g ( a , b ) ∥ Y 2 d b . Measurable versions may be chosen Borel, since the base measures are sigma-finite Borel measures; completion does not change the resulting density measure.
Moments. Since r < s r < s r < s ,
( 1 + ∥ a ∥ + ∣ b ∣ ) r ≤ ( 1 + ∥ a ∥ ) r ( 1 + ∣ b ∣ ) r ≤ 2 r / 2 ( 1 + ∥ a ∥ ) s ⟨ b ⟩ r , (1+\|a\|+|b|)^r \le(1+\|a\|)^r(1+|b|)^r \le2^{r/2}(1+\|a\|)^s\langle b\rangle^r, ( 1 + ∥ a ∥ + ∣ b ∣ ) r ≤ ( 1 + ∥ a ∥ ) r ( 1 + ∣ b ∣ ) r ≤ 2 r /2 ( 1 + ∥ a ∥ ) s ⟨ b ⟩ r ,
so integrating the weighted inverse Fourier estimate (50) over ν \nu ν proves (54) . In particular Γ g \Gamma_g Γ g has finite variation, with ∣ Γ g ∣ = ∥ γ g ∥ Y ( ν ⊗ d b ) |\Gamma_g|=\|\gamma_g\|_{\mathcal{Y}}(\nu\otimes\mathrm{d}b) ∣ Γ g ∣ = ∥ γ g ∥ Y ( ν ⊗ d b ) , and to see that g ∈ L 1 ( ν ; Y ) g\in L^1(\nu;\mathcal{Y}) g ∈ L 1 ( ν ; Y ) , choose ω 0 ≠ 0 \omega_0\ne0 ω 0 = 0 with ρ ♯ ( − ω 0 ) ≠ 0 \rho^\sharp(-\omega_0)\ne0 ρ ♯ ( − ω 0 ) = 0 . Hypothesis 1 and the weighted inverse Fourier estimate with r = 0 r=0 r = 0 give
∥ g ∥ L 1 ( ν ; Y ) ≤ ∣ ω 0 ∣ α ∣ ρ ♯ ( − ω 0 ) ∣ − 1 A s , 0 B s ( ρ , g ) < ∞ . \|g\|_{L^1(\nu;\mathcal{Y})}\le|\omega_0|^\alpha|\rho^\sharp(-\omega_0)|^{-1}A_{s,0}\mathcal{B}_s(\rho,g)<\infty. ∥ g ∥ L 1 ( ν ; Y ) ≤ ∣ ω 0 ∣ α ∣ ρ ♯ ( − ω 0 ) ∣ − 1 A s , 0 B s ( ρ , g ) < ∞.
Here we integrated ∥ h a ( ω 0 ) ∥ ≤ A s , 0 ∥ h a ∥ H ω s \|h_a(\omega_0)\|\le A_{s,0}\|h_a\|_{H^s_\omega} ∥ h a ( ω 0 ) ∥ ≤ A s , 0 ∥ h a ∥ H ω s and used homogeneity. Thus f g f_g f g is defined, bounded and continuous.
A majorant. If ∥ x ∥ ≤ R \|x\|\le R ∥ x ∥ ≤ R , then
∣ σ ( ⟨ a , x ⟩ − b ) ∣ ∥ γ g ( a , b ) ∥ Y ≤ C σ max { 1 , R } p ( 1 + ∥ a ∥ + ∣ b ∣ ) p ∥ γ g ( a , b ) ∥ Y , |\sigma(\langle a,x\rangle-b)|\|\gamma_g(a,b)\|_{\mathcal{Y}}\le C_\sigma\max\{1,R\}^p(1+\|a\|+|b|)^p\|\gamma_g(a,b)\|_{\mathcal{Y}}, ∣ σ (⟨ a , x ⟩ − b ) ∣∥ γ g ( a , b ) ∥ Y ≤ C σ max { 1 , R } p ( 1 + ∥ a ∥ + ∣ b ∣ ) p ∥ γ g ( a , b ) ∥ Y ,
whose right side is integrable by (54) with r = p r=p r = p , which is admissible because p < s − 1 / 2 p<s-1/2 p < s − 1/2 . This is a common absolute majorant on the whole ball ∥ x ∥ ≤ R \|x\|\le R ∥ x ∥ ≤ R ; it makes the synthesis absolutely convergent, uniformly there, justifies the direction–bias Fubini interchange below, and proves continuity of the synthesis by dominated convergence and continuity of σ \sigma σ .
The direction average. Fix x ∈ H x\in\mathcal{H} x ∈ H and write u a = ⟨ a , x ⟩ u_a=\langle a,x\rangle u a = ⟨ a , x ⟩ . For each a a a the map b ↦ u a − b b\mapsto u_a-b b ↦ u a − b preserves Lebesgue measure, and ( a , b ) ↦ u a − b (a,b)\mapsto u_a-b ( a , b ) ↦ u a − b is jointly measurable, so (54) with r = 0 r=0 r = 0 shows that ( a , t ) ↦ γ g ( a , u a − t ) (a,t)\mapsto\gamma_g(a,u_a-t) ( a , t ) ↦ γ g ( a , u a − t ) is integrable for ν ⊗ d t \nu\otimes\mathrm{d}t ν ⊗ d t . Hence Ψ x \Psi_x Ψ x is defined for almost every t t t , lies in L 1 ( R ; Y ) L^1(\mathbb{R};\mathcal{Y}) L 1 ( R ; Y ) , and Fubini gives
S σ [ Γ g ] ( x ) = ∫ R σ ( t ) Ψ x ( t ) d t . (58) S_\sigma[\Gamma_g](x)=\int_{\mathbb{R}}\sigma(t)\Psi_x(t)\,\mathrm{d}t. \tag*{(58)} S σ [ Γ g ] ( x ) = ∫ R σ ( t ) Ψ x ( t ) d t . (58)
Identification of Ψ x \Psi_x Ψ x . Both Ψ x \Psi_x Ψ x and Θ x : = q ˇ α , ρ ( − ⋅ ) f g ( x ) \Theta_x:=\check q_{\alpha,\rho}(-\cdot)f_g(x) Θ x := q ˇ α , ρ ( − ⋅ ) f g ( x ) are integrable, the latter by (50) applied to q α , ρ q_{\alpha,\rho} q α , ρ , so both have continuous Fourier transforms. Let ω ≠ 0 \omega\ne0 ω = 0 . Fubini in ( a , t ) (a,t) ( a , t ) , the substitution b = u a − t b=u_a-t b = u a − t , hypothesis 1 and homogeneity give
Ψ x ^ ( ω ) = ∫ H e − i ω u a h a ( − ω ) ν ( d a ) = ρ ♯ ( ω ) ∫ H e i ⟨ x , − ω a ⟩ g ( − ω a ) ν ( d a ) = ρ ♯ ( ω ) ∣ ω ∣ − α f g ( x ) = q α , ρ ( − ω ) f g ( x ) = Θ x ^ ( ω ) . \begin{aligned}
\widehat{\Psi_x}(\omega)=\int_{\mathcal{H}}e^{-i\omega u_a}h_a(-\omega)\,\nu(\mathrm{d}a)=\rho^\sharp(\omega)\int_{\mathcal{H}}e^{i\langle x,-\omega a\rangle}g(-\omega a)\,\nu(\mathrm{d}a) \\
&=\rho^\sharp(\omega)|\omega|^{-\alpha}f_g(x)=q_{\alpha,\rho}(-\omega)f_g(x)=\widehat{\Theta_x}(\omega).
\end{aligned} Ψ x ( ω ) = ∫ H e − iω u a h a ( − ω ) ν ( d a ) = ρ ♯ ( ω ) ∫ H e i ⟨ x , − ω a ⟩ g ( − ω a ) ν ( d a ) = ρ ♯ ( ω ) ∣ ω ∣ − α f g ( x ) = q α , ρ ( − ω ) f g ( x ) = Θ x ( ω ) .
Two continuous functions that agree off the origin agree everywhere, so Ψ x ^ = Θ x ^ \widehat{\Psi_x}=\widehat{\Theta_x} Ψ x = Θ x , and uniqueness of the Fourier transform on L 1 ( R ; Y ) L^1(\mathbb{R};\mathcal{Y}) L 1 ( R ; Y ) proves (55) . The argument uses no value of q α , ρ q_{\alpha,\rho} q α , ρ at the origin, and no evaluation of a Sobolev class at a prescribed frequency.
The identity. By (58) , (55) and (52) ,
S σ [ Γ g ] ( x ) = ∫ R σ ( t ) q ^ α , ρ ( − t ) d t f g ( x ) = 1 2 π ⟨ σ ♯ , q α , ρ ⟩ f g ( x ) = ( ( σ , ρ ) ) α f g ( x ) , S_{\sigma}[\Gamma_g](x)=\int_{\mathbb{R}}\sigma(t)\widehat{q}_{\alpha,\rho}(-t)\,\mathrm{d}t\,f_g(x)=\frac{1}{2\pi}\langle\sigma^{\sharp},q_{\alpha,\rho}\rangle f_g(x)=((\sigma,\rho))_{\alpha}f_g(x), S σ [ Γ g ] ( x ) = ∫ R σ ( t ) q α , ρ ( − t ) d t f g ( x ) = 2 π 1 ⟨ σ ♯ , q α , ρ ⟩ f g ( x ) = (( σ , ρ ) ) α f g ( x ) ,
the first integral being absolutely convergent by Lemma 5.9 . This proves (56) .
The L 2 L^2 L 2 statement. Assume now that Y \mathcal{Y} Y is a separable complex Hilbert space, that ( ( ρ , ρ ) ) α < ∞ ((\rho,\rho))_{\alpha}<\infty (( ρ , ρ ) ) α < ∞ and that g ∈ L 2 ( ν ; Y ) g\in L^2(\nu;\mathcal{Y}) g ∈ L 2 ( ν ; Y ) . Plancherel in the bias for almost every a a a , Tonelli, and homogeneity give
∫ H × R ∥ γ g ( a , b ) ∥ Y 2 ν ( d a ) d b = 1 2 π ∫ R ∣ ρ ♯ ( ω ) ∣ 2 ∫ H ∥ g ( ω a ) ∥ Y 2 ν ( d a ) d ω = ( ( ρ , ρ ) ) α ∥ g ∥ L 2 ( ν ; Y ) 2 , \int_{\mathcal{H}\times\mathbb{R}}\|\gamma_g(a,b)\|_{\mathcal{Y}}^2\,\nu(\mathrm{d}a)\,\mathrm{d}b=\frac{1}{2\pi}\int_{\mathbb{R}}|\rho^{\sharp}(\omega)|^2\int_{\mathcal{H}}\|g(\omega a)\|_{\mathcal{Y}}^2\,\nu(\mathrm{d}a)\,\mathrm{d}\omega
=\left((\rho,\rho)\right)_{\alpha}\|g\|_{L^2(\nu;\mathcal{Y})}^2, ∫ H × R ∥ γ g ( a , b ) ∥ Y 2 ν ( d a ) d b = 2 π 1 ∫ R ∣ ρ ♯ ( ω ) ∣ 2 ∫ H ∥ g ( ω a ) ∥ Y 2 ν ( d a ) d ω = ( ( ρ , ρ ) ) α ∥ g ∥ L 2 ( ν ; Y ) 2 ,
the omitted frequency ω = 0 \omega=0 ω = 0 being a Lebesgue null set. This also identifies the coefficient with W ρ g W_{\rho}g W ρ g , the L 2 L^2 L 2 coefficient of Definition 3.7 .
Remark 5.11 (What the synthesis identity uses). The identity (56) uses no property of H \mathcal{H} H beyond its inner product and its Borel structure, and α > 0 \alpha>0 α > 0 only through the existence of homogeneous measures of that degree. It uses no property of Y \mathcal{Y} Y beyond completeness of its norm: the Hilbert structure and the separability of the standing output space enter only in (57) , through Plancherel in the bias variable. Admissibility of ρ \rho ρ is not assumed either: positivity of ( ( ρ , ρ ) ) α ((\rho,\rho))_{\alpha} (( ρ , ρ ) ) α is never used, and its finiteness only in (57) . Finally g ∈ L 1 ( ν ; Y ) g\in L^1(\nu;\mathcal{Y}) g ∈ L 1 ( ν ; Y ) need not be assumed, because hypotheses 1 and 2 give it: choose ω 0 ≠ 0 \omega_0\ne0 ω 0 = 0 with ρ ♯ ( − ω 0 ) ≠ 0 \rho^{\sharp}(-\omega_0)\ne0 ρ ♯ ( − ω 0 ) = 0 , so that (50) with r = 0 r=0 r = 0 bounds ∣ ρ ♯ ( − ω 0 ) ∣ ∥ g ( ω 0 a ) ∥ Y = ∥ h a ( ω 0 ) ∥ Y |\rho^{\sharp}(-\omega_0)|\|g(\omega_0a)\|_{\mathcal{Y}}=\|h_a(\omega_0)\|_{\mathcal{Y}} ∣ ρ ♯ ( − ω 0 ) ∣∥ g ( ω 0 a ) ∥ Y = ∥ h a ( ω 0 ) ∥ Y by A s , 0 ∥ h a ∥ H s A_{s,0}\|h_a\|_{H^s} A s , 0 ∥ h a ∥ H s ; integrating in a a a and using homogeneity,
∥ g ∥ L 1 ( ν ; Y ) ≤ ∣ ω 0 ∣ α ∣ ρ ♯ ( − ω 0 ) ∣ − 1 A s , 0 B s ( ρ , g ) . \|g\|_{L^1(\nu;\mathcal{Y})}\leq|\omega_0|^{\alpha}|\rho^{\sharp}(-\omega_0)|^{-1}A_{s,0}\mathcal{B}_s(\rho,g). ∥ g ∥ L 1 ( ν ; Y ) ≤ ∣ ω 0 ∣ α ∣ ρ ♯ ( − ω 0 ) ∣ − 1 A s , 0 B s ( ρ , g ) .
For reference, A s , r = ( 2 π ) − 1 / 2 ( ∫ R ( 1 + t 2 ) − ( s − r ) d t ) 1 / 2 A_{s,r}=(2\pi)^{-1/2}\left(\int_{\mathbb{R}}(1+t^2)^{-(s-r)}\,\mathrm{d}t\right)^{1/2} A s , r = ( 2 π ) − 1/2 ( ∫ R ( 1 + t 2 ) − ( s − r ) d t ) 1/2 . The distinction between synthesis and sampling matters: a second parameter moment requires s > 5 / 2 s>5/2 s > 5/2 . With this extra condition and a real globally Lipschitz activation, Theorem 6.5 applies; reproducing f g f_g f g requires division of the coefficient by a nonzero ( ( σ , ρ ) ) α ((\sigma,\rho))_{\alpha} (( σ , ρ ) ) α .
These hypotheses allow concrete non-band-pass filters. For ρ ♯ ( ω ) = ω 2 k e − ω 2 \rho^{\sharp}(\omega)=\omega^{2k}e^{-\omega^2} ρ ♯ ( ω ) = ω 2 k e − ω 2 and g ( ξ ) = e − ∥ ξ ∥ 2 v g(\xi)=e^{-\|\xi\|^2}v g ( ξ ) = e − ∥ ξ ∥ 2 v , it suffices that 2 k > α + 2 s − 1 2 2k>\alpha+2s-\frac{1}{2} 2 k > α + 2 s − 2 1 . For example, α = 1 \alpha=1 α = 1 , s = 3 s=3 s = 3 , k = 4 k=4 k = 4 permits a second moment and nonzero reconstruction constants for both Gaussian and ReLU activations. The following proposition states these conditions and constants, followed by their verification.
Proposition 5.12 (Non-band-pass Gaussian-derivative filters). Let ν \nu ν be a sigma-finite Borel measure on H \mathcal{H} H satisfying ( D ω ) # ν = ∣ ω ∣ − α ν (D_\omega)_\#\nu=|\omega|^{-\alpha}\nu ( D ω ) # ν = ∣ ω ∣ − α ν for ω ≠ 0 \omega\ne0 ω = 0 , with α > 0 \alpha>0 α > 0 , and suppose ν ( { a : ∥ a ∥ ≤ 1 } ) < ∞ \nu(\{a:\|a\|\leq1\})<\infty ν ({ a : ∥ a ∥ ≤ 1 }) < ∞ . Fix s > 1 / 2 s>1/2 s > 1/2 , choose an integer k ≥ 1 k\geq1 k ≥ 1 with
2 k > α + 2 s − 1 2 , (59) 2k>\alpha+2s-\frac{1}{2}, \tag*{(59)} 2 k > α + 2 s − 2 1 , (59) and define
ρ ♯ ( ω ) = ω 2 k e − ω 2 , g ( ξ ) = e − ∥ ξ ∥ 2 v , v ∈ Y . \rho^{\sharp}(\omega)=\omega^{2k}e^{-\omega^2},\qquad g(\xi)=e^{-\|\xi\|^2}v,\qquad v\in\mathcal{Y}. ρ ♯ ( ω ) = ω 2 k e − ω 2 , g ( ξ ) = e − ∥ ξ ∥ 2 v , v ∈ Y . Then ρ \rho ρ is a real Schwartz filter, is not band-pass, and satisfies all filter and density hypotheses of Theorem 5.10 for this s s s . In particular this gives an absolutely convergent network integral for every continuous activation of growth order p < s − 1 2 p<s-\frac{1}{2} p < s − 2 1 .
Put δ = 2 k − α \delta=2k-\alpha δ = 2 k − α . For σ ( t ) = e − t 2 / 2 \sigma(t)=e^{-t^2/2} σ ( t ) = e − t 2 /2 , the reconstruction constant is
( ( σ , ρ ) ) α = Γ ( ( δ + 1 ) / 2 ) 2 π ( 3 / 2 ) − ( δ + 1 ) / 2 > 0. ((\sigma,\rho))_{\alpha}=\frac{\Gamma((\delta+1)/2)}{\sqrt{2\pi}}(3/2)^{-(\delta+1)/2}>0. (( σ , ρ ) ) α = 2 π Γ (( δ + 1 ) /2 ) ( 3/2 ) − ( δ + 1 ) /2 > 0.
If s > 3 / 2 s > 3/2 s > 3/2 , ReLU also satisfies the hypotheses, and
( ( ReLU , ρ ) ) α = − 1 2 π Γ ( ( δ − 1 ) / 2 ) ≠ 0. ((\operatorname{ReLU},\rho))_{\alpha}=-\frac{1}{2\pi}\Gamma((\delta-1)/2)\ne0. (( ReLU , ρ ) ) α = − 2 π 1 Γ (( δ − 1 ) /2 ) = 0.
The local finiteness assumption holds for the homogeneous Gaussian mixtures used in this paper on an infinite-dimensional Hilbert space.
Proof. Homogeneity gives ν ( B R ) = R α ν ( B 1 ) \nu(B_R)=R^\alpha\nu(B_1) ν ( B R ) = R α ν ( B 1 ) for closed balls centered at zero and R > 0 R>0 R > 0 . Splitting into B 1 B_1 B 1 and the annuli 2 j < ∥ a ∥ ≤ 2 j + 1 2^j<\lVert a\rVert\le2^{j+1} 2 j < ∥ a ∥ ≤ 2 j + 1 , j ≥ 0 j\ge0 j ≥ 0 , shows that
∫ H ( 1 + ∥ a ∥ 2 ) − d / 2 ν ( d a ) < ∞ whenever d > α . (60) \int_{\mathcal{H}}(1+\lVert a\rVert^2)^{-d/2}\nu(da)<\infty\qquad\text{whenever }d>\alpha. \tag*{(60)} ∫ H ( 1 + ∥ a ∥ 2 ) − d /2 ν ( d a ) < ∞ whenever d > α . (60)
Indeed the annular integrals are bounded by ν ( B 1 ) 2 α 2 − j ( d − α ) \nu(B_1)2^{\alpha}2^{-j(d-\alpha)} ν ( B 1 ) 2 α 2 − j ( d − α ) , a summable geometric series. Gaussian decay therefore gives g ∈ L 1 ( ν ; Y ) ∩ L 2 ( ν ; Y ) g\in L^1(\nu;Y)\cap L^2(\nu;Y) g ∈ L 1 ( ν ; Y ) ∩ L 2 ( ν ; Y ) ; the first is automatic by Remark 5.11, and the second supplies (57) .
The inverse transform of the real even Schwartz function ω 2 k e − ω 2 \omega^{2k}e^{-\omega^2} ω 2 k e − ω 2 is real and Schwartz. Its Fourier transform is nonzero at every ω ≠ 0 \omega\ne0 ω = 0 , so it is not band-pass. Self-admissibility is equivalent here to 4 k − α > − 1 4k-\alpha>-1 4 k − α > − 1 , since the squared Fourier transform is ∣ ω ∣ 4 k e − 2 ω 2 |\omega|^{4k}e^{-2\omega^2} ∣ ω ∣ 4 k e − 2 ω 2 ; this follows from (59) . The integral is positive.
Set A = ( 1 + ∥ a ∥ 2 ) 1 / 2 A=(1+\lVert a\rVert^2)^{1/2} A = ( 1 + ∥ a ∥ 2 ) 1/2 , h 0 ( ω ) = ω 2 k e − ω 2 h_0(\omega)=\omega^{2k}e^{-\omega^2} h 0 ( ω ) = ω 2 k e − ω 2 , and γ 0 = h ˇ 0 \gamma_0=\check{h}_0 γ 0 = h ˇ 0 . The functions h a h_a h a and their inverse transforms satisfy
h a ( ω ) = A − 2 k h 0 ( A ω ) v , h ˇ a ( b ) = A − 2 k − 1 γ 0 ( b / A ) v . h_a(\omega)=A^{-2k}h_0(A\omega)v,\qquad\check{h}_a(b)=A^{-2k-1}\gamma_0(b/A)v. h a ( ω ) = A − 2 k h 0 ( A ω ) v , h ˇ a ( b ) = A − 2 k − 1 γ 0 ( b / A ) v .
Since A ≥ 1 A\ge1 A ≥ 1 , substitution b = A t b=At b = A t gives
∥ h a ∥ H ω s ≤ ∥ v ∥ Y ∥ h 0 ∥ H ω s A s − 2 k − 1 / 2 . \lVert h_a\rVert_{H^s_\omega}\le\lVert v\rVert_Y\lVert h_0\rVert_{H^s_\omega}A^{s-2k-1/2}. ∥ h a ∥ H ω s ≤ ∥ v ∥ Y ∥ h 0 ∥ H ω s A s − 2 k − 1/2 .
The coefficient ( a , b ) ↦ A − 2 k − 1 γ 0 ( b / A ) v (a,b)\mapsto A^{-2k-1}\gamma_0(b/A)v ( a , b ) ↦ A − 2 k − 1 γ 0 ( b / A ) v is jointly continuous, because a ↦ A a\mapsto A a ↦ A is continuous with A ≥ 1 A\ge1 A ≥ 1 and γ 0 \gamma_0 γ 0 is Schwartz; for each a a a , its restriction to the bias variable is h ˇ a \check{h}_a h ˇ a . This is hypothesis 1 of Theorem 5.10 . Moreover,
( 1 + ∥ a ∥ ) s ∥ h a ∥ H s ≤ 2 s / 2 ∥ v ∥ Y ∥ h 0 ∥ H s ( 1 + ∥ a ∥ 2 ) ( 2 s − 2 k − 1 / 2 ) / 2 . (1+\lVert a\rVert)^s\lVert h_a\rVert_{H^s}\le2^{s/2}\lVert v\rVert_Y\lVert h_0\rVert_{H^s}(1+\lVert a\rVert^2)^{(2s-2k-1/2)/2}. ( 1 + ∥ a ∥ ) s ∥ h a ∥ H s ≤ 2 s /2 ∥ v ∥ Y ∥ h 0 ∥ H s ( 1 + ∥ a ∥ 2 ) ( 2 s − 2 k − 1/2 ) /2 .
This is integrable by (60) and (59) , which say 2 k + 1 / 2 − 2 s > α 2k+1/2-2s>\alpha 2 k + 1/2 − 2 s > α . Thus B s < ∞ \mathcal{B}_s<\infty B s < ∞ .
The Sobolev membership of q α , ρ ( ω ) = ∣ ω ∣ δ e − ω 2 q_{\alpha,\rho}(\omega)=|\omega|^\delta e^{-\omega^2} q α , ρ ( ω ) = ∣ ω ∣ δ e − ω 2 follows from the same dilates, by the subordination formula
∣ ω ∣ − α = 1 Γ ( α / 2 ) ∫ 0 ∞ u α / 2 − 1 e − u ω 2 d u , ω ≠ 0. (61) |\omega|^{-\alpha}=\frac{1}{\Gamma(\alpha/2)}\int_0^\infty u^{\alpha/2-1}e^{-u\omega^2}\,du,\qquad\omega\ne0. \tag*{(61)} ∣ ω ∣ − α = Γ ( α /2 ) 1 ∫ 0 ∞ u α /2 − 1 e − u ω 2 d u , ω = 0. (61)
It writes q α , ρ q_{\alpha,\rho} q α , ρ as the superposition
q α , ρ ( ω ) = 1 Γ ( α / 2 ) ∫ 0 ∞ u α / 2 − 1 τ − 2 k h 0 ( τ ω ) d u , τ = τ ( u ) = ( 1 + u ) 1 / 2 , q_{\alpha,\rho}(\omega)=\frac{1}{\Gamma(\alpha/2)}\int_0^\infty u^{\alpha/2-1}\tau^{-2k}h_0(\tau\omega)\,du,\qquad\tau=\tau(u)=(1+u)^{1/2}, q α , ρ ( ω ) = Γ ( α /2 ) 1 ∫ 0 ∞ u α /2 − 1 τ − 2 k h 0 ( τ ω ) d u , τ = τ ( u ) = ( 1 + u ) 1/2 ,
of the rescaled functions τ − 2 k h 0 ( τ ⋅ ) \tau^{-2k}h_0(\tau\cdot) τ − 2 k h 0 ( τ ⋅ ) , whose inverse Fourier transforms are the dilates τ − 2 k − 1 γ 0 ( ⋅ / τ ) \tau^{-2k-1}\gamma_0(\cdot/\tau) τ − 2 k − 1 γ 0 ( ⋅ / τ ) . Since ∥ τ − 2 k − 1 γ 0 ( ⋅ / τ ) ∥ L 1 = τ − 2 k ∥ γ 0 ∥ L 1 \lVert\tau^{-2k-1}\gamma_0(\cdot/\tau)\rVert_{L^1}=\tau^{-2k}\lVert\gamma_0\rVert_{L^1} ∥ τ − 2 k − 1 γ 0 ( ⋅ / τ ) ∥ L 1 = τ − 2 k ∥ γ 0 ∥ L 1 and ∫ 0 ∞ u α / 2 − 1 ( 1 + u ) − k d u < ∞ \int_0^\infty u^{\alpha/2-1}(1+u)^{-k}\,du<\infty ∫ 0 ∞ u α /2 − 1 ( 1 + u ) − k d u < ∞ for 0 < α < 2 k 0<\alpha<2k 0 < α < 2 k , Fubini gives the coefficient
q ˇ α , ρ ( b ) = 1 Γ ( α / 2 ) ∫ 0 ∞ u α / 2 − 1 τ − 2 k − 1 γ 0 ( b / τ ) d u . \check{q}_{\alpha,\rho}(b)=\frac{1}{\Gamma(\alpha/2)}\int_0^\infty u^{\alpha/2-1}\tau^{-2k-1}\gamma_0(b/\tau)\,du. q ˇ α , ρ ( b ) = Γ ( α /2 ) 1 ∫ 0 ∞ u α /2 − 1 τ − 2 k − 1 γ 0 ( b / τ ) d u .
For the norm, weighted Cauchy–Schwarz in u u u with the weights w ( u ) = u α / 2 − 1 w(u) = u^{\alpha/2-1} w ( u ) = u α /2 − 1 and m ( u ) = τ ( u ) s − 2 k − 1 / 2 m(u) = \tau(u)^{s-2k-1/2} m ( u ) = τ ( u ) s − 2 k − 1/2 , followed by Tonelli in ( u , b ) (u,b) ( u , b ) , bounds ∥ q α , ρ ∥ H ω s 2 \|q_{\alpha,\rho}\|_{H^s_\omega}^2 ∥ q α , ρ ∥ H ω s 2 by
1 Γ ( α / 2 ) 2 ( ∫ 0 ∞ w m d u ) ∫ 0 ∞ w m ∥ τ − 2 k h 0 ( τ ⋅ ) ∥ H ω s 2 d u ≤ ∥ h 0 ∥ H ω s 2 Γ ( α / 2 ) 2 ( ∫ 0 ∞ u α / 2 − 1 ( 1 + u ) ( s − 2 k − 1 / 2 ) / 2 d u ) 2 . \frac{1}{\Gamma(\alpha/2)^2}\left(\int_0^\infty wm\,du\right)\int_0^\infty\frac{w}{m}\|\tau^{-2k}h_0(\tau\cdot)\|_{H^s_\omega}^2\,du
\le\frac{\|h_0\|_{H^s_\omega}^2}{\Gamma(\alpha/2)^2}\left(\int_0^\infty u^{\alpha/2-1}(1+u)^{(s-2k-1/2)/2}\,du\right)^2. Γ ( α /2 ) 2 1 ( ∫ 0 ∞ w m d u ) ∫ 0 ∞ m w ∥ τ − 2 k h 0 ( τ ⋅ ) ∥ H ω s 2 d u ≤ Γ ( α /2 ) 2 ∥ h 0 ∥ H ω s 2 ( ∫ 0 ∞ u α /2 − 1 ( 1 + u ) ( s − 2 k − 1/2 ) /2 d u ) 2 .
by the dilation estimate already displayed. The Beta-type integral converges at the origin because α > 0 \alpha> 0 α > 0 and at infinity because 2 k > α + s − 1 / 2 2k > \alpha+ s - 1/2 2 k > α + s − 1/2 , which is weaker than (59) . Hence q α , ρ ∈ H ω s ( R ) q_{\alpha,\rho} \in H^s_\omega(\mathbb{R}) q α , ρ ∈ H ω s ( R ) , with no discussion of weak derivatives or of boundary terms at the origin.
For the Gaussian activation, σ ♯ ( ω ) = 2 π e − ω 2 / 2 \sigma^\sharp(\omega) = \sqrt{2\pi}e^{-\omega^2/2} σ ♯ ( ω ) = 2 π e − ω 2 /2 . The ordinary Fourier pairing and substitution z = ( 3 / 2 ) ω 2 z = (3/2)\omega^2 z = ( 3/2 ) ω 2 give the first displayed constant. For ReLU use ReLU ♯ = − Pf ( ω − 2 ) + i π δ 0 ′ \operatorname{ReLU}^\sharp= -\operatorname{Pf}(\omega^{-2}) + i\pi\delta'_0 ReLU ♯ = − Pf ( ω − 2 ) + iπ δ 0 ′ , and pair it with the superposition (61) rather than with q α , ρ q_{\alpha,\rho} q α , ρ directly. Each test ω ↦ τ − 2 k h 0 ( τ ω ) \omega\mapsto\tau^{-2k}h_0(\tau\omega) ω ↦ τ − 2 k h 0 ( τ ω ) is a Schwartz function vanishing to order 2 k ≥ 2 2k \ge2 2 k ≥ 2 at the origin, so δ 0 ′ \delta'_0 δ 0 ′ contributes nothing and the finite part is the absolutely convergent integral − ∫ R ω 2 k − 2 e − τ 2 ω 2 d ω -\int_{\mathbb{R}}\omega^{2k-2}e^{-\tau^2\omega^2}\,d\omega − ∫ R ω 2 k − 2 e − τ 2 ω 2 d ω ; no cutoff sequence is needed. Fubini in ( u , ω ) (u,\omega) ( u , ω ) , licensed by ∫ 0 ∞ u α / 2 − 1 ( 1 + u ) − ( 2 k − 1 ) / 2 d u < ∞ \int_0^\infty u^{\alpha/2-1}(1+u)^{-(2k-1)/2}\,du < \infty ∫ 0 ∞ u α /2 − 1 ( 1 + u ) − ( 2 k − 1 ) /2 d u < ∞ for α < 2 k − 1 \alpha< 2k-1 α < 2 k − 1 , which follows from (59) and s > 3 / 2 s > 3/2 s > 3/2 , then returns the subordination integral in the ω \omega ω variable and gives
( ( ReLU , ρ ) ) α = − 1 2 π ∫ R ∣ ω ∣ δ − 2 e − ω 2 d ω = − 1 2 π Γ ( ( δ − 1 ) / 2 ) . ((\operatorname{ReLU},\rho))_\alpha=-\frac{1}{2\pi}\int_{\mathbb{R}}|\omega|^{\delta-2}e^{-\omega^2}\,d\omega=-\frac{1}{2\pi}\Gamma((\delta-1)/2). (( ReLU , ρ ) ) α = − 2 π 1 ∫ R ∣ ω ∣ δ − 2 e − ω 2 d ω = − 2 π 1 Γ (( δ − 1 ) /2 ) .
Finally, for the Gaussian mixture write Z ∼ μ P Z \sim\mu_P Z ∼ μ P . Tonelli and the substitution implicit in ∥ 2 t Z ∥ ≤ R \|\sqrt{2t}Z\| \le R ∥ 2 t Z ∥ ≤ R give
ν α ( B R ) = E ∫ 0 R 2 / ( 2 ∥ Z ∥ 2 ) t α / 2 − 1 d t = 2 1 − α / 2 α R α E ∥ Z ∥ − α < ∞ . \nu_\alpha(B_R)=\mathbb{E}\int_0^{R^2/(2\|Z\|^2)}t^{\alpha/2-1}\,dt=\frac{2^{1-\alpha/2}}{\alpha}R^\alpha\mathbb{E}\|Z\|^{-\alpha}<\infty. ν α ( B R ) = E ∫ 0 R 2 / ( 2∥ Z ∥ 2 ) t α /2 − 1 d t = α 2 1 − α /2 R α E ∥ Z ∥ − α < ∞.
For finiteness choose an integer n > α n > \alpha n > α and n n n positive eigenvalues of the injective covariance P P P . If their minimum is λ ∗ > 0 \lambda_* > 0 λ ∗ > 0 , then ∥ Z ∥ 2 ≥ λ ∗ ∑ j = 1 n Z j 2 \|Z\|^2 \ge\lambda_* \sum_{j=1}^n Z_j^2 ∥ Z ∥ 2 ≥ λ ∗ ∑ j = 1 n Z j 2 , with independent standard real Gaussians Z j Z_j Z j . Polar integration in R n \mathbb{R}^n R n gives a finite negative moment of order α < n \alpha< n α < n . This proves the claimed local finiteness and completes the example.
Approximation by finite-width networks
We obtain finite-width networks by Monte Carlo approximation of the coefficient integral and estimate their error uniformly on compact sets. Conditions on the spectral density provide the required coefficient moments, and density of these targets yields cc-universality. The construction does not require a projection of the input space or the direction parameter.
Approximation rates
Let Γ \Gamma Γ be a finite complex measure on H × R \mathcal{H} \times\mathbb{R} H × R with polar decomposition Γ = h ∣ Γ ∣ \Gamma= h|\Gamma| Γ = h ∣Γ∣ , ∣ h ∣ = 1 |h| = 1 ∣ h ∣ = 1 ∣ Γ ∣ |\Gamma| ∣Γ∣ -almost everywhere. If Γ = 0 \Gamma= 0 Γ = 0 take the zero network. Otherwise write
V : = ∥ Γ ∥ T V , p : = ∣ Γ ∣ / V , (62) V := \|\Gamma\|_{\mathrm{TV}}, \qquad p := |\Gamma|/V, \tag*{(62)} V := ∥Γ ∥ TV , p := ∣Γ∣/ V , (62)
sample independent θ j = ( a j , b j ) ∼ p \theta_j = (a_j,b_j) \sim p θ j = ( a j , b j ) ∼ p , and form the sampled network
f N ( x ) : = V N ∑ j = 1 N h ( θ j ) σ ( ⟨ a j , x ⟩ − b j ) , (63) f_N(x) := \frac{V}{N}\sum_{j=1}^N h(\theta_j)\,\sigma(\langle a_j,x\rangle-b_j), \tag*{(63)} f N ( x ) := N V j = 1 ∑ N h ( θ j ) σ (⟨ a j , x ⟩ − b j ) , (63)
which is unbiased for f = S σ [ Γ ] f = S_\sigma[\Gamma] f = S σ [ Γ ] .
Definition 6.1 (Activation-dependent Rademacher complexity). For a compact K ⊂ H K \subset\mathcal{H} K ⊂ H , define
R N ( K ; p , σ ) : = E θ , ε sup x ∈ K ∣ 1 N ∑ j = 1 N ε j h ( θ j ) σ ( ⟨ a j , x ⟩ − b j ) ∣ . (64) \mathcal{R}_N(K;p,\sigma) := \mathbb{E}_{\theta,\varepsilon}\sup_{x\in K}\left|\frac{1}{N}\sum_{j=1}^{N}\varepsilon_j h(\theta_j)\sigma(\langle a_j,x\rangle-b_j)\right| . \tag*{(64)} R N ( K ; p , σ ) := E θ , ε x ∈ K sup N 1 j = 1 ∑ N ε j h ( θ j ) σ (⟨ a j , x ⟩ − b j ) . (64) where ε j \varepsilon_j ε j are independent Rademacher signs.
Lemma 6.2 (Approximation by averages of integrable Banach-valued atoms). Let X X X be a separable Banach space, p p p a probability measure, and Φ ∈ L 1 ( p ; X ) \Phi\in L^1(p;X) Φ ∈ L 1 ( p ; X ) . For independent samples θ j ∼ p \theta_j \sim p θ j ∼ p and independent Rademacher signs ε j \varepsilon_j ε j ,
E ∥ 1 N ∑ j = 1 N ε j Φ ( θ j ) ∥ X ⟶ 0. \mathbb{E}\left\|\frac{1}{N}\sum_{j=1}^{N}\varepsilon_j\Phi(\theta_j)\right\|_X \longrightarrow0. E N 1 j = 1 ∑ N ε j Φ ( θ j ) X ⟶ 0. Moreover, with m = ∫ Φ d p m=\int\Phi\,dp m = ∫ Φ d p ,
E ∥ 1 N ∑ j = 1 N Φ ( θ j ) − m ∥ X ≤ 2 E ∥ 1 N ∑ j = 1 N ε j Φ ( θ j ) ∥ X . \mathbb{E}\left\|\frac{1}{N}\sum_{j=1}^{N}\Phi(\theta_j)-m\right\|_X \leq2\mathbb{E}\left\|\frac{1}{N}\sum_{j=1}^{N}\varepsilon_j\Phi(\theta_j)\right\|_X. E N 1 j = 1 ∑ N Φ ( θ j ) − m X ≤ 2 E N 1 j = 1 ∑ N ε j Φ ( θ j ) X .
Proof. Approximate Φ \Phi Φ in L 1 ( p ; X ) L^1(p;X) L 1 ( p ; X ) by a simple function Ψ = ∑ k = 1 J x k 1 E k \Psi=\sum_{k=1}^{J}x_k\mathbf{1}_{E_k} Ψ = ∑ k = 1 J x k 1 E k . The expected norm of the signed-average error is at most ∥ Φ − Ψ ∥ L 1 \|\Phi-\Psi\|_{L^1} ∥Φ − Ψ ∥ L 1 . For the simple function it is at most ∑ k ∥ x k ∥ p ( E k ) / N \sum_k\|x_k\|\sqrt{p(E_k)/N} ∑ k ∥ x k ∥ p ( E k ) / N , by the scalar second-moment identity and Cauchy–Schwarz. First let N → ∞ N\to\infty N → ∞ , then let the approximation error tend to zero. For the second assertion introduce independent copies θ j ′ \theta'_j θ j ′ , apply Jensen to the ghost sample, and use exchangeability of each pair to insert signs into Φ ( θ j ) − Φ ( θ j ′ ) \Phi(\theta_j)-\Phi(\theta'_j) Φ ( θ j ) − Φ ( θ j ′ ) . The triangle inequality bounds the result by two copies of the signed average. Every integral is justified by Φ ∈ L 1 \Phi\in L^1 Φ ∈ L 1 . □
Theorem 6.3 (Compact-open approximation bound). Whenever the atoms x ↦ h ( θ ) σ ( ⟨ a , x ⟩ − b ) x\mapsto h(\theta)\sigma(\langle a,x\rangle-b) x ↦ h ( θ ) σ (⟨ a , x ⟩ − b ) are strongly measurable and integrable in C ( K ) C(K) C ( K ) , the sampled network satisfies
E ∥ f N − f ∥ C ( K ) ≤ 2 V R N ( K ; p , σ ) . (65) \mathbb{E}\|f_N-f\|_{C(K)}\leq2V\mathcal{R}_N(K;p,\sigma). \tag*{(65)} E ∥ f N − f ∥ C ( K ) ≤ 2 V R N ( K ; p , σ ) . (65)
Proof. Apply Lemma 6.2 with X = C ( K ) X=C(K) X = C ( K ) and multiply by V V V . This also proves convergence to zero. No Lipschitz or second-moment assumption enters this step. □
Lemma 6.4 (A two-coordinate Rademacher comparison). Let S S S be a nonempty set, let N ≥ 1 N\geq1 N ≥ 1 , and let ψ i , u i , v i : S → R \psi_i,u_i,v_i:S\to\mathbb{R} ψ i , u i , v i : S → R be bounded functions, for 1 ≤ i ≤ N 1\leq i\leq N 1 ≤ i ≤ N . Suppose that
∣ ψ i ( s ) − ψ i ( t ) ∣ ≤ ∣ u i ( s ) − u i ( t ) ∣ + ∣ v i ( s ) − v i ( t ) ∣ ( s , t ∈ S ) . |\psi_i(s)-\psi_i(t)|\leq|u_i(s)-u_i(t)|+|v_i(s)-v_i(t)|\qquad(s,t\in S). ∣ ψ i ( s ) − ψ i ( t ) ∣ ≤ ∣ u i ( s ) − u i ( t ) ∣ + ∣ v i ( s ) − v i ( t ) ∣ ( s , t ∈ S ) . For independent Rademacher signs ε i , ε i 1 , ε i 2 \varepsilon_i,\varepsilon_{i1},\varepsilon_{i2} ε i , ε i 1 , ε i 2 , one has
E sup s ∈ S ∑ i = 1 N ε i ψ i ( s ) ≤ 2 E sup s ∈ S ∑ i = 1 N ( ε i 1 u i ( s ) + ε i 2 v i ( s ) ) . (66) \mathbb{E}\sup_{s\in S}\sum_{i=1}^{N}\varepsilon_i\psi_i(s)\leq2\mathbb{E}\sup_{s\in S}\sum_{i=1}^{N}(\varepsilon_{i1}u_i(s)+\varepsilon_{i2}v_i(s)). \tag*{(66)} E s ∈ S sup i = 1 ∑ N ε i ψ i ( s ) ≤ 2 E s ∈ S sup i = 1 ∑ N ( ε i 1 u i ( s ) + ε i 2 v i ( s )) . (66)
Proof. We first prove a one-sign comparison. Let F , ψ , u , v : S → R F,\psi,u,v:S\to\mathbb{R} F , ψ , u , v : S → R be bounded, with ψ , u , v \psi,u,v ψ , u , v satisfying the stated increment condition, so that every supremum below is finite. Fix δ > 0 \delta>0 δ > 0 and choose s + ∈ S s_{+} \in S s + ∈ S with F ( s + ) + ψ ( s + ) > sup s ∈ S ( F + ψ ) − δ / 2 F(s_{+})+\psi(s_{+})>\sup_{s\in S}(F+\psi)-\delta/2 F ( s + ) + ψ ( s + ) > sup s ∈ S ( F + ψ ) − δ /2 and F ( s − ) − ψ ( s − ) > sup s ∈ S ( F − ψ ) − δ / 2 F(s_{-})-\psi(s_{-})>\sup_{s\in S}(F-\psi)-\delta/2 F ( s − ) − ψ ( s − ) > sup s ∈ S ( F − ψ ) − δ /2 ; no maximizer need exist. Put
A = F ( s + ) − F ( s − ) , r = u ( s + ) − u ( s − ) , q = v ( s + ) − v ( s − ) . A=F(s_{+})-F(s_{-}),\qquad r=u(s_{+})-u(s_{-}),\qquad q=v(s_{+})-v(s_{-}). A = F ( s + ) − F ( s − ) , r = u ( s + ) − u ( s − ) , q = v ( s + ) − v ( s − ) .
For two independent signs η 1 , η 2 \eta_{1},\eta_{2} η 1 , η 2 , the variable D = 2 ( η 1 r + η 2 q ) D=2(\eta_{1}r+\eta_{2}q) D = 2 ( η 1 r + η 2 q ) is symmetric. Consequently
E ∣ A + D ∣ = 1 2 E ( ∣ A + D ∣ + ∣ A − D ∣ ) ≥ E ∣ D ∣ . \mathbb{E}|A+D|=\frac{1}{2}\mathbb{E}\left(|A+D|+|A-D|\right)\geq\mathbb{E}|D|. E ∣ A + D ∣ = 2 1 E ( ∣ A + D ∣ + ∣ A − D ∣ ) ≥ E ∣ D ∣.
Direct enumeration of the four sign pairs gives
E ∣ η 1 r + η 2 q ∣ = ∣ r + q ∣ + ∣ r − q ∣ 2 = max { ∣ r ∣ , ∣ q ∣ } ≥ ∣ r ∣ + ∣ q ∣ 2 . \mathbb{E}|\eta_{1}r+\eta_{2}q|=\frac{|r+q|+|r-q|}{2}=\max\{|r|,|q|\}\geq\frac{|r|+|q|}{2}. E ∣ η 1 r + η 2 q ∣ = 2 ∣ r + q ∣ + ∣ r − q ∣ = max { ∣ r ∣ , ∣ q ∣ } ≥ 2 ∣ r ∣ + ∣ q ∣ .
Using only the two indices s + , s − s_{+},s_{-} s + , s − to bound a supremum from below, and using max { a , b } = ( a + b + ∣ a − b ∣ ) / 2 \max\{a,b\}=(a+b+|a-b|)/2 max { a , b } = ( a + b + ∣ a − b ∣ ) /2 , we therefore obtain
E η 1 , η 2 sup s ∈ S ( F ( s ) + 2 η 1 u ( s ) + 2 η 2 v ( s ) ) ≥ F ( s + ) + F ( s − ) 2 + 1 2 E ∣ A + D ∣ ≥ F ( s + ) + F ( s − ) 2 + ∣ r ∣ + ∣ q ∣ 2 ≥ F ( s + ) + F ( s − ) + ψ ( s + ) − ψ ( s − ) 2 ≥ E ε sup s ∈ S ( F ( s ) + ε ψ ( s ) ) − δ 2 . \begin{aligned}
\mathbb{E}_{\eta_{1},\eta_{2}}\sup_{s\in S}\left(F(s)+2\eta_{1}u(s)+2\eta_{2}v(s)\right)
&\geq\frac{F(s_{+})+F(s_{-})}{2}+\frac{1}{2}\mathbb{E}|A+D|\\
&\geq\frac{F(s_{+})+F(s_{-})}{2}+\frac{|r|+|q|}{2}\\
&\geq\frac{F(s_{+})+F(s_{-})+\psi(s_{+})-\psi(s_{-})}{2}\\
&\geq\mathbb{E}_{\varepsilon}\sup_{s\in S}\left(F(s)+\varepsilon\psi(s)\right)-\frac{\delta}{2}.
\end{aligned} E η 1 , η 2 s ∈ S sup ( F ( s ) + 2 η 1 u ( s ) + 2 η 2 v ( s ) ) ≥ 2 F ( s + ) + F ( s − ) + 2 1 E ∣ A + D ∣ ≥ 2 F ( s + ) + F ( s − ) + 2 ∣ r ∣ + ∣ q ∣ ≥ 2 F ( s + ) + F ( s − ) + ψ ( s + ) − ψ ( s − ) ≥ E ε s ∈ S sup ( F ( s ) + ε ψ ( s ) ) − 2 δ .
Letting δ ↓ 0 \delta\downarrow0 δ ↓ 0 proves the one-sign inequality, which allows an arbitrary bounded offset F F F . Condition on all signs except ε i \varepsilon_i ε i , apply it to replace ε i ψ i \varepsilon_i\psi_i ε i ψ i by 2 ε i 1 u i + 2 ε i 2 v i 2\varepsilon_{i1}u_i+2\varepsilon_{i2}v_i 2 ε i 1 u i + 2 ε i 2 v i , and then average over the conditioned signs. Repeating this for i = 1 , … , N i=1,\ldots,N i = 1 , … , N proves (66) .
Both sides are finite averages of bounded suprema over the 2 N 2^N 2 N , respectively 4 N 4^N 4 N , sign patterns, so no countability, measurability or separability of S S S enters, and no assumption that any of the functions vanish at a common index is needed. □
Theorem 6.5 (Dimension-free uniform Hilbert-valued Barron bound). Let Γ \Gamma Γ be a finite-variation Y \mathcal{Y} Y -valued Borel measure, σ : R → R \sigma:\mathbb{R}\to\mathbb{R} σ : R → R globally Lipschitz, and N ≥ 1 N\geq1 N ≥ 1 . For V > 0 V>0 V > 0 , write Γ = h ∣ Γ ∣ \Gamma=h|\Gamma| Γ = h ∣Γ∣ , ∥ h ∥ Y = 1 \|h\|_{\mathcal{Y}}=1 ∥ h ∥ Y = 1 almost everywhere, and p = ∣ Γ ∣ / V p=|\Gamma|/V p = ∣Γ∣/ V . Suppose
M 2 2 : = ∫ H × R ( ∥ a ∥ 2 + ∣ b ∣ 2 ) p ( d a , d b ) < ∞ . (67) M_{2}^{2}:=\int_{\mathcal{H}\times\mathbb{R}}\left(\|a\|^{2}+|b|^{2}\right)p(da,db)<\infty. \tag*{(67)} M 2 2 := ∫ H × R ( ∥ a ∥ 2 + ∣ b ∣ 2 ) p ( d a , d b ) < ∞. (67) For a nonempty compact K ⊂ H K\subset\mathcal{H} K ⊂ H , put R K = sup x ∈ K ∥ x ∥ 2 + 1 R_{K}=\sup_{x\in K}\sqrt{\|x\|^{2}+1} R K = sup x ∈ K ∥ x ∥ 2 + 1 . The sampled network (63) , now with Y \mathcal{Y} Y -valued weights, satisfies
E ∥ f N − f ∥ C ( K ; Y ) ≤ V N ( 4 ∣ σ ( 0 ) ∣ + 8 Lip ( σ ) R K M 2 ) ≤ 8 V N ( ∣ σ ( 0 ) ∣ + Lip ( σ ) R K M 2 ) . (68) \mathbb{E}\|f_{N}-f\|_{C(K;\mathcal{Y})}\leq\frac{V}{\sqrt{N}}\left(4|\sigma(0)|+8\operatorname{Lip}(\sigma)R_{K}M_{2}\right)
\leq\frac{8V}{\sqrt{N}}\left(|\sigma(0)|+\operatorname{Lip}(\sigma)R_{K}M_{2}\right). \tag*{(68)} E ∥ f N − f ∥ C ( K ; Y ) ≤ N V ( 4∣ σ ( 0 ) ∣ + 8 Lip ( σ ) R K M 2 ) ≤ N 8 V ( ∣ σ ( 0 ) ∣ + Lip ( σ ) R K M 2 ) . (68) At least one deterministic width-N N N realization satisfies the first bound. For V = 0 V=0 V = 0 take the zero network; for empty K K K the error is zero.
Proof. Put L = Lip ( σ ) L = \operatorname{Lip}(\sigma) L = Lip ( σ ) and b 0 = ∣ σ ( 0 ) ∣ b_0 = |\sigma(0)| b 0 = ∣ σ ( 0 ) ∣ . We may assume V > 0 V > 0 V > 0 and K ≠ ∅ K \ne\emptyset K = ∅ . The Radon–Nikodym property of a Hilbert space gives a strongly ∣ Γ ∣ |\Gamma| ∣Γ∣ -measurable polar density h h h with ∥ h ∥ Y = 1 \lVert h\rVert_Y = 1 ∥ h ∥ Y = 1 ∣ Γ ∣ |\Gamma| ∣Γ∣ -almost everywhere. We use a Borel representative and change it to zero on a Borel null set so that ∥ h ( θ ) ∥ Y ≤ 1 \lVert h(\theta)\rVert_Y \le1 ∥ h ( θ ) ∥ Y ≤ 1 everywhere. Such a representative exists: a strongly measurable function is an almost-everywhere limit of simple measurable functions; if the measure has been completed, replace the finitely many sets in each of those simple functions by Borel sets modulo null sets, and take the limit outside the resulting Borel null set. These modifications do not change Γ \Gamma Γ , f f f , or the law of the sampled network.
Equip H ~ = H ⊕ R \widetilde{H} = H \oplus\mathbb{R} H = H ⊕ R with its Hilbert norm, and set
t = ( a , b ) , z x = ( x , − 1 ) , Φ ( t ) ( x ) = h ( t ) σ ( ⟨ t , z x ⟩ ) . t = (a,b), \qquad z_x = (x,-1), \qquad\Phi(t)(x) = h(t)\sigma(\langle t,z_x\rangle). t = ( a , b ) , z x = ( x , − 1 ) , Φ ( t ) ( x ) = h ( t ) σ (⟨ t , z x ⟩) .
For every t t t , the function Φ ( t ) \Phi(t) Φ ( t ) belongs to C ( K ; Y ) C(K;Y) C ( K ; Y ) , and
∥ Φ ( t ) ∥ C ( K ; Y ) ≤ b 0 + L R K ∥ t ∥ . (69) \lVert\Phi(t)\rVert_{C(K;Y)} \le b_0 + LR_K\lVert t\rVert. \tag*{(69)} ∥ Φ ( t ) ∥ C ( K ; Y ) ≤ b 0 + L R K ∥ t ∥ . (69)
We verify its strong measurability, rather than presupposing measurability of an uncountable supremum. The map
( t , y ) ⟼ [ x ⟼ y σ ( ⟨ t , z x ⟩ ) ] from H ~ × Y to C ( K ; Y ) (t,y) \longmapsto\left[x \longmapsto y\sigma(\langle t,z_x\rangle)\right] \text{ from } \widetilde{H} \times Y \text{ to } C(K;Y) ( t , y ) ⟼ [ x ⟼ y σ (⟨ t , z x ⟩) ] from H × Y to C ( K ; Y )
is continuous: the norm of the difference at ( t , y ) (t,y) ( t , y ) and ( t ′ , y ′ ) (t',y') ( t ′ , y ′ ) is at most
L R K ∥ t − t ′ ∥ ∥ y ∥ + ( b 0 + L R K ∥ t ′ ∥ ) ∥ y − y ′ ∥ . LR_K\lVert t-t'\rVert\lVert y\rVert+ (b_0 + LR_K\lVert t'\rVert)\lVert y-y'\rVert. L R K ∥ t − t ′ ∥ ∥ y ∥ + ( b 0 + L R K ∥ t ′ ∥) ∥ y − y ′ ∥ .
Its domain is separable, so its range is separable. Composing it with the Borel map t ↦ ( t , h ( t ) ) t \mapsto(t,h(t)) t ↦ ( t , h ( t )) gives a Borel, separably valued C ( K ; Y ) C(K;Y) C ( K ; Y ) -valued map, which is strongly measurable. Cauchy–Schwarz gives ∫ ∥ t ∥ d p ≤ M 2 \int\lVert t\rVert\,dp \le M_2 ∫ ∥ t ∥ d p ≤ M 2 ; hence (69) makes Φ \Phi Φ Bochner integrable. Evaluation at any x ∈ K x \in K x ∈ K is continuous linear, so f ∣ K = V ∫ Φ d p f|_K = V\int\Phi\,dp f ∣ K = V ∫ Φ d p in C ( K ; Y ) C(K;Y) C ( K ; Y ) . The same envelope also shows that all random uniform norms used below are measurable and integrable.
We first estimate the Rademacher sum conditionally on arbitrary fixed samples t 1 , … , t N t_1,\ldots,t_N t 1 , … , t N . Put h i = h ( t i ) h_i = h(t_i) h i = h ( t i ) and B i = b 0 + L R K ∥ t i ∥ B_i = b_0 + LR_K\lVert t_i\rVert B i = b 0 + L R K ∥ t i ∥ . Regard Y Y Y as a real Hilbert space with inner product ( y , w ) R = Re ⟨ y , w ⟩ Y (y,w)_{\mathbb{R}} = \operatorname{Re}\langle y,w\rangle_Y ( y , w ) R = Re ⟨ y , w ⟩ Y ; its norm is unchanged. For y y y in its closed unit ball B Y B_Y B Y , put q i ( y ) = ( h i , y ) R q_i(y) = (h_i,y)_{\mathbb{R}} q i ( y ) = ( h i , y ) R . For every w ∈ Y w \in Y w ∈ Y ,
∥ w ∥ Y = sup y ∈ B Y ( w , y ) R . \lVert w\rVert_Y = \sup_{y\in B_Y}(w,y)_{\mathbb{R}}. ∥ w ∥ Y = y ∈ B Y sup ( w , y ) R .
This follows by Cauchy–Schwarz and, when w ≠ 0 w \ne0 w = 0 , by taking y = w / ∥ w ∥ Y y = w/\lVert w\rVert_Y y = w / ∥ w ∥ Y ; for w = 0 w = 0 w = 0 both sides vanish.
Choose countable norm-dense subsets D K ⊂ K D_K \subset K D K ⊂ K and D Y ⊂ B Y D_Y \subset B_Y D Y ⊂ B Y , with 0 ∈ D Y 0 \in D_Y 0 ∈ D Y . All processes below are continuous in ( x , y ) (x,y) ( x , y ) , so their suprema on K × B Y K \times B_Y K × B Y equal their suprema on the countable set S = D K × D Y S = D_K \times D_Y S = D K × D Y . On S S S define
ψ i ( x , y ) = σ ( ⟨ t i , z x ⟩ ) q i ( y ) , u i ( x , y ) = L ⟨ t i , z x ⟩ , v i ( x , y ) = B i q i ( y ) . \psi_i(x,y) = \sigma(\langle t_i,z_x\rangle)q_i(y), \qquad u_i(x,y) = L\langle t_i,z_x\rangle, \qquad v_i(x,y) = B_iq_i(y). ψ i ( x , y ) = σ (⟨ t i , z x ⟩) q i ( y ) , u i ( x , y ) = L ⟨ t i , z x ⟩ , v i ( x , y ) = B i q i ( y ) .
These functions are bounded. Since ∣ q i ( y ) ∣ ≤ 1 |q_i(y)| \le1 ∣ q i ( y ) ∣ ≤ 1 and ∣ σ ( ⟨ t i , z x ⟩ ) ∣ ≤ B i |\sigma(\langle t_i,z_x\rangle)| \le B_i ∣ σ (⟨ t i , z x ⟩) ∣ ≤ B i , splitting the difference of the two products gives
∣ ψ i ( x , y ) − ψ i ( x ′ , y ′ ) ∣ ≤ L ∣ ⟨ t i , z x − z x ′ ⟩ ∣ + B i ∣ q i ( y ) − q i ( y ′ ) ∣ = ∣ u i ( x , y ) − u i ( x ′ , y ′ ) ∣ + ∣ v i ( x , y ) − v i ( x ′ , y ′ ) ∣ . \begin{aligned}
|\psi_i(x,y)-\psi_i(x',y')| &\le L|\langle t_i,z_x-z_{x'}\rangle| + B_i|q_i(y)-q_i(y')| \\
&= |u_i(x,y)-u_i(x',y')| + |v_i(x,y)-v_i(x',y')|.
\end{aligned} ∣ ψ i ( x , y ) − ψ i ( x ′ , y ′ ) ∣ ≤ L ∣ ⟨ t i , z x − z x ′ ⟩ ∣ + B i ∣ q i ( y ) − q i ( y ′ ) ∣ = ∣ u i ( x , y ) − u i ( x ′ , y ′ ) ∣ + ∣ v i ( x , y ) − v i ( x ′ , y ′ ) ∣.
Lemma 6.4, followed by real Hilbert duality, therefore yields
E ε sup x ∈ K ∥ ∑ i = 1 N ε i h i σ ( ⟨ t i , z x ⟩ ) ∥ Y = E ε sup ( x , y ) ∈ S ∑ i = 1 N ε i ψ i ( x , y ) \mathbb{E}_{\varepsilon}\sup_{x\in K}\left\|\sum_{i=1}^{N}\varepsilon_i h_i\sigma\left(\langle t_i,z_x\rangle\right)\right\|_Y
= \mathbb{E}_{\varepsilon}\sup_{(x,y)\in S}\sum_{i=1}^{N}\varepsilon_i\psi_i(x,y) E ε x ∈ K sup i = 1 ∑ N ε i h i σ ( ⟨ t i , z x ⟩ ) Y = E ε ( x , y ) ∈ S sup i = 1 ∑ N ε i ψ i ( x , y )
≤ 2 E ε i 1 , ε i 2 sup x ∈ K , y ∈ B Y ( L ⟨ ∑ i = 1 N ε i 1 t i , z x ⟩ + ⟨ ∑ i = 1 N ε i 2 B i h i , y ⟩ ) R \le2\mathbb{E}_{\varepsilon_{i_1},\varepsilon_{i_2}}\sup_{x\in K,\ y\in B_Y}\left(L\left\langle\sum_{i=1}^{N}\varepsilon_{i_1}t_i,z_x\right\rangle+\left\langle\sum_{i=1}^{N}\varepsilon_{i_2}B_i h_i,y\right\rangle\right)_{\mathbb{R}} ≤ 2 E ε i 1 , ε i 2 x ∈ K , y ∈ B Y sup ( L ⟨ i = 1 ∑ N ε i 1 t i , z x ⟩ + ⟨ i = 1 ∑ N ε i 2 B i h i , y ⟩ ) R
≤ 2 L R K E ∥ ∑ i = 1 N ε i 1 t i ∥ + 2 E ∥ ∑ i = 1 N ε i 2 B i h i ∥ Y \le2LR_K\mathbb{E}\left\|\sum_{i=1}^{N}\varepsilon_{i_1}t_i\right\|+2\mathbb{E}\left\|\sum_{i=1}^{N}\varepsilon_{i_2}B_i h_i\right\|_Y ≤ 2 L R K E i = 1 ∑ N ε i 1 t i + 2 E i = 1 ∑ N ε i 2 B i h i Y
≤ 2 L R K ( ∑ i = 1 N ∥ t i ∥ 2 ) 1 / 2 + 2 ( ∑ i = 1 N B i 2 ) 1 / 2 \le2LR_K\left(\sum_{i=1}^{N}\|t_i\|^2\right)^{1/2}+2\left(\sum_{i=1}^{N}B_i^2\right)^{1/2} ≤ 2 L R K ( i = 1 ∑ N ∥ t i ∥ 2 ) 1/2 + 2 ( i = 1 ∑ N B i 2 ) 1/2
≤ 2 b 0 N + 4 L R K ( ∑ i = 1 N ∥ t i ∥ 2 ) 1 / 2 . \le2b_0\sqrt{N}+4LR_K\left(\sum_{i=1}^{N}\|t_i\|^2\right)^{1/2}. ≤ 2 b 0 N + 4 L R K ( i = 1 ∑ N ∥ t i ∥ 2 ) 1/2 .
Here the penultimate inequality follows by expanding the squared Hilbert norms: mixed terms have zero sign expectation, and ∥ h i ∥ Y ≤ 1 \|h_i\|_Y\le1 ∥ h i ∥ Y ≤ 1 . The last inequality is the triangle inequality in R N \mathbb{R}^N R N applied to ( B i ) i = ( b 0 ) i + L R K ( ∥ t i ∥ ) i (B_i)_i=(b_0)_i+LR_K(\|t_i\|)_i ( B i ) i = ( b 0 ) i + L R K ( ∥ t i ∥ ) i . These arguments also cover L = 0 L=0 L = 0 or M 2 = 0 M_2=0 M 2 = 0 without division by either quantity. Jensen’s inequality now gives
E t , ε ∥ 1 N ∑ i = 1 N ε i Φ ( t i ) ∥ C ( K ; Y ) ≤ 2 b 0 + 4 L R K M 2 N , \mathbb{E}_{t,\varepsilon}\left\|\frac{1}{N}\sum_{i=1}^{N}\varepsilon_i\Phi(t_i)\right\|_{C(K;Y)}
\le\frac{2b_0+4LR_KM_2}{\sqrt{N}}, E t , ε N 1 i = 1 ∑ N ε i Φ ( t i ) C ( K ; Y ) ≤ N 2 b 0 + 4 L R K M 2 ,
because E ( ∑ i ∥ t i ∥ 2 ) 1 / 2 ≤ N M 2 \mathbb{E}\left(\sum_i\|t_i\|^2\right)^{1/2}\le\sqrt{N}M_2 E ( ∑ i ∥ t i ∥ 2 ) 1/2 ≤ N M 2 .
For completeness, symmetrization here needs no Hilbert structure in C ( K ; Y ) C(K;Y) C ( K ; Y ) . Let Z i = Φ ( t i ) Z_i=\Phi(t_i) Z i = Φ ( t i ) , let Z i ′ Z_i' Z i ′ be independent copies, and let m = E Z 1 m=\mathbb{E}Z_1 m = E Z 1 . Conditional Jensen, exchangeability of each independent pair ( Z i , Z i ′ ) (Z_i,Z_i') ( Z i , Z i ′ ) , and the triangle inequality imply
E ∥ 1 N ∑ i = 1 N Z i − m ∥ C ( K ; Y ) ≤ E ∥ 1 N ∑ i = 1 N ( Z i − Z i ′ ) ∥ C ( K ; Y ) = E ∥ 1 N ∑ i = 1 N ε i ( Z i − Z i ′ ) ∥ C ( K ; Y ) \begin{aligned}
\mathbb{E}\left\|\frac{1}{N}\sum_{i=1}^{N}Z_i-m\right\|_{C(K;Y)}
\le\mathbb{E}\left\|\frac{1}{N}\sum_{i=1}^{N}(Z_i-Z_i')\right\|_{C(K;Y)} \\
&=\mathbb{E}\left\|\frac{1}{N}\sum_{i=1}^{N}\varepsilon_i(Z_i-Z_i')\right\|_{C(K;Y)}
\end{aligned} E N 1 i = 1 ∑ N Z i − m C ( K ; Y ) ≤ E N 1 i = 1 ∑ N ( Z i − Z i ′ ) C ( K ; Y ) = E N 1 i = 1 ∑ N ε i ( Z i − Z i ′ ) C ( K ; Y )
≤ 2 E ∥ 1 N ∑ i = 1 N ε i Z i ∥ C ( K ; Y ) . \le2\mathbb{E}\left\|\frac{1}{N}\sum_{i=1}^{N}\varepsilon_iZ_i\right\|_{C(K;Y)}. ≤ 2 E N 1 i = 1 ∑ N ε i Z i C ( K ; Y ) .
Multiplying by V V V proves the first inequality in (68) ; the second uses 4 b 0 ≤ 8 b 0 4b_0\le8b_0 4 b 0 ≤ 8 b 0 . Finally, a nonnegative integrable random variable with expectation at most A A A cannot exceed A A A almost surely: otherwise its difference from A A A would be strictly positive almost surely and have positive integral. Thus some sample satisfies the first displayed bound, which proves the deterministic assertion. □
The Hilbert-valued variance identity is given in § 6.4 . Concentration for bounded parameters and the separation of input truncation from finite-width approximation error are additional consequences recorded in Appendix § 12 .
Finite variation from the spectral density
The coefficient γ g \gamma_g γ g of Theorem 4.5 lies in L 2 ( λ α ) L^2(\lambda_\alpha) L 2 ( λ α ) , and λ α \lambda_\alpha λ α is an infinite measure, so finite total variation and moments of the coefficient measure require additional hypotheses. The condition in Definition 4.2 supplies them; this is the content of the following theorem, which together with Theorem 6.8 below is the main result of this section.
Theorem 6.6 (Finite total variation and moments of the coefficient measure). Let ρ \rho ρ be a band-pass filter, fix the frequency window I I I in Definition 4.2 , and let g g g satisfy that definition. For every integer r ≥ 0 r \ge0 r ≥ 0 there is a constant c ρ , I , r c_{\rho,I,r} c ρ , I , r , independent of g g g , such that
∫ H × R ( 1 + ∥ a ∥ + ∣ b ∣ ) r ∣ γ g ( a , b ) ∣ d λ α ≤ c ρ , I , r M r + 2 ( g ) < ∞ . (70) \int_{\mathcal{H}\times\mathbb{R}} (1+\lVert a\rVert+|b|)^r|\gamma_g(a,b)|\,\mathrm{d}\lambda_\alpha\le c_{\rho,I,r}M_{r+2}(g)<\infty. \tag*{(70)} ∫ H × R ( 1 + ∥ a ∥ + ∣ b ∣ ) r ∣ γ g ( a , b ) ∣ d λ α ≤ c ρ , I , r M r + 2 ( g ) < ∞. (70) The same estimate holds for Y \mathcal{Y} Y -valued densities, with norms and a constant independent of the dimension of Y \mathcal{Y} Y . In particular,
∫ H × R ( 1 + ∥ a ∥ 2 + ∣ b ∣ 2 ) ∣ γ g ( a , b ) ∣ λ α ( d a , d b ) ≤ c ρ , I , 2 M 4 ( g ) < ∞ . (71) \int_{\mathcal{H}\times\mathbb{R}} (1+\lVert a\rVert^2+|b|^2)|\gamma_g(a,b)|\,\lambda_\alpha(\mathrm{d}a,\mathrm{d}b)\le c_{\rho,I,2}M_4(g)<\infty. \tag*{(71)} ∫ H × R ( 1 + ∥ a ∥ 2 + ∣ b ∣ 2 ) ∣ γ g ( a , b ) ∣ λ α ( d a , d b ) ≤ c ρ , I , 2 M 4 ( g ) < ∞. (71) Consequently, for every real globally Lipschitz σ \sigma σ , the target ( ( σ , ρ ) ) α f g ((\sigma,\rho))_\alpha f_g (( σ , ρ ) ) α f g is the integral network S σ [ γ g λ α ] S_\sigma[\gamma_g\lambda_\alpha] S σ [ γ g λ α ] , and its sampled network (63) satisfies, with V = ∥ γ g ∥ L 1 ( λ α ) V=\lVert\gamma_g\rVert_{L^1(\lambda_\alpha)} V = ∥ γ g ∥ L 1 ( λ α ) and M 2 M_2 M 2 as in (67) for p = ∣ γ g ∣ λ α / V p=|\gamma_g|\lambda_\alpha/V p = ∣ γ g ∣ λ α / V when V > 0 V>0 V > 0 (take the zero network when V = 0 V=0 V = 0 ),
E ∥ f N − ( ( σ , ρ ) ) α f g ∥ C ( K ) ≤ 8 V N ( ∣ σ ( 0 ) ∣ + Lip ( σ ) R K M 2 ) for every compact K ⊂ H . (72) \mathbb{E}\lVert f_N-((\sigma,\rho))_\alpha f_g\rVert_{C(K)}
\le\frac{8V}{\sqrt{N}}\bigl(|\sigma(0)|+\operatorname{Lip}(\sigma)R_KM_2\bigr)
\qquad\text{for every compact }K\subset\mathcal{H}. \tag*{(72)} E ∥ f N − (( σ , ρ ) ) α f g ∥ C ( K ) ≤ N 8 V ( ∣ σ ( 0 ) ∣ + Lip ( σ ) R K M 2 ) for every compact K ⊂ H . (72)
Proof. For each integer r ≥ 0 r\ge0 r ≥ 0 , Lemma 4.3 gives
∫ ( 1 + ∥ a ∥ + ∣ b ∣ ) r ∥ γ g ( a , b ) ∥ Y d λ α ≤ c ρ , I , r A r + 2 , r ( g ) ≤ c ρ , I , r M r + 2 ( g ) . \int(1+\lVert a\rVert+|b|)^r\lVert\gamma_g(a,b)\rVert_{\mathcal{Y}}\,\mathrm{d}\lambda_\alpha
\le c_{\rho,I,r}A_{r+2,r}(g)\le c_{\rho,I,r}M_{r+2}(g). ∫ ( 1 + ∥ a ∥ + ∣ b ∣ ) r ∥ γ g ( a , b ) ∥ Y d λ α ≤ c ρ , I , r A r + 2 , r ( g ) ≤ c ρ , I , r M r + 2 ( g ) .
This proves all moments of the coefficient measure, including finite total variation at r = 0 r=0 r = 0 and (71) at r = 2 r=2 r = 2 , independently of synthesis. For globally Lipschitz real σ \sigma σ , ∣ σ ( ⟨ a , x ⟩ − b ) ∣ ≤ ∣ σ ( 0 ) ∣ + Lip ( σ ) ( ∥ a ∥ ∥ x ∥ + ∣ b ∣ ) |\sigma(\langle a,x\rangle-b)|\le|\sigma(0)|+\operatorname{Lip}(\sigma)(\lVert a\rVert\lVert x\rVert+|b|) ∣ σ (⟨ a , x ⟩ − b ) ∣ ≤ ∣ σ ( 0 ) ∣ + Lip ( σ ) (∥ a ∥ ∥ x ∥ + ∣ b ∣ ) . The first moment therefore makes the synthesis absolutely integrable, and Theorem 4.5 (iii) identifies it with ( ( σ , ρ ) ) α f g ((\sigma,\rho))_\alpha f_g (( σ , ρ ) ) α f g , including when the constant is zero. If V > 0 V>0 V > 0 , the second moment gives (67) for the sampling distribution p = ∣ γ g ∣ λ α / V p=|\gamma_g|\lambda_\alpha/V p = ∣ γ g ∣ λ α / V , and Theorem 6.5 proves (72). If V = 0 V=0 V = 0 , the coefficient and its synthesis vanish and the zero network suffices.
Constructive universal approximation
We now state the approximation theorem that is the main result of the paper from the point of view of approximation theory.
The following qualitative result supplies discretization when no explicit Lipschitz rate is required. Use the polar decomposition (62) with V > 0 V>0 V > 0 ; the zero measure is represented by the zero network.
Lemma 6.7 (Qualitative finite-atomic approximation). Let σ : R → C \sigma:\mathbb{R}\to\mathbb{C} σ : R → C be continuous. For compact K ⊂ H K\subset\mathcal{H} K ⊂ H , assume ∫ ∥ σ ( ⟨ a , ⋅ ⟩ − b ) ∥ C ( K ) d ∣ Γ ∣ ( a , b ) < ∞ \int\lVert\sigma(\langle a,\cdot\rangle-b)\rVert_{C(K)}\,\mathrm{d}|\Gamma|(a,b)<\infty ∫ ∥ σ (⟨ a , ⋅ ⟩ − b ) ∥ C ( K ) d ∣Γ∣ ( a , b ) < ∞ . Then for every ε > 0 \varepsilon>0 ε > 0 there is a finite atomic complex measure Γ ε \Gamma_\varepsilon Γ ε with ∥ S σ Γ ε − S σ Γ ∥ C ( K ) < ε \lVert S_\sigma\Gamma_\varepsilon-S_\sigma\Gamma\rVert_{C(K)}<\varepsilon ∥ S σ Γ ε − S σ Γ ∥ C ( K ) < ε .
Proof. The atom map ( a , b ) ↦ σ ( ⟨ a , ⋅ ⟩ − b ) ∣ K (a,b) \mapsto\sigma(\langle a,\cdot\rangle-b)|_K ( a , b ) ↦ σ (⟨ a , ⋅ ⟩ − b ) ∣ K is continuous into the separable Banach space C ( K ) C(K) C ( K ) : convergent parameters have uniformly bounded preactivations on K K K , where continuity of σ \sigma σ is uniform. Thus V ( a , b ) = h ( a , b ) σ ( ⟨ a , ⋅ ⟩ − b ) ∣ K \mathcal{V}(a,b)=h(a,b)\sigma(\langle a,\cdot\rangle-b)|_K V ( a , b ) = h ( a , b ) σ (⟨ a , ⋅ ⟩ − b ) ∣ K is Bochner integrable, and its mean belongs to the closed convex hull of its essential range by Hahn–Banach separation. Approximate the mean by a finite convex combination of essential-range values, then each value by an actual V ( a j , b j ) \mathcal{V}(a_j,b_j) V ( a j , b j ) , and multiply by V V V .
Theorem 6.8 (Constructive universal approximation with approximation rates). Let σ : R → R \sigma:\mathbb{R}\to\mathbb{R} σ : R → R be continuous, of polynomial growth, and not a polynomial, and let ρ \rho ρ be a band-pass filter with ( ( σ , ρ ) ) α = 1 ((\sigma,\rho))_\alpha=1 (( σ , ρ ) ) α = 1 , which exists by Theorem 5.4 . Let f : H → C f:\mathcal{H}\to\mathbb{C} f : H → C be continuous, let K ⊂ H K\subset\mathcal{H} K ⊂ H be compact, and let ε > 0 \varepsilon>0 ε > 0 . Then there is a smooth spectral density g g g , vanishing outside a bounded set and satisfying Definition 4.2 , such that:
(i) ∥ f − f g ∥ C ( K ) < ε \|f-f_g\|_{C(K)}<\varepsilon ∥ f − f g ∥ C ( K ) < ε ;
(ii) f g = S σ [ γ g λ α ] f_g=S_\sigma[\gamma_g\lambda_\alpha] f g = S σ [ γ g λ α ] is an integral network with activation σ \sigma σ whose coefficient measure γ g λ α \gamma_g\lambda_\alpha γ g λ α is finite and has finite moments of all orders;
(iii) if σ \sigma σ is globally Lipschitz, the sampled network f N f_N f N of γ g λ α \gamma_g\lambda_\alpha γ g λ α satisfies
E ∥ f − f N ∥ C ( K ) ≤ ε + 8 V N ( ∣ σ ( 0 ) ∣ + Lip ( σ ) R K M 2 ) , (73) \mathbb{E}\|f-f_N\|_{C(K)}\leq\varepsilon+\frac{8V}{\sqrt{N}}\left(|\sigma(0)|+\operatorname{Lip}(\sigma)R_KM_2\right), \tag*{(73)} E ∥ f − f N ∥ C ( K ) ≤ ε + N 8 V ( ∣ σ ( 0 ) ∣ + Lip ( σ ) R K M 2 ) , (73) with V = ∥ γ g ∥ L 1 ( λ α ) V=\|\gamma_g\|_{L^1(\lambda_\alpha)} V = ∥ γ g ∥ L 1 ( λ α ) and M 2 M_2 M 2 the second moment of ∣ γ g ∣ λ α / V |\gamma_g|\lambda_\alpha/V ∣ γ g ∣ λ α / V ; at least one deterministic width-N N N network satisfies the same bound.
In particular the finite-width networks with activation σ \sigma σ are dense in C ( H ) C(\mathcal{H}) C ( H ) for the compact-open topology. The same statements hold for continuous f : H → Y f:\mathcal{H}\to\mathcal{Y} f : H → Y , with the norms in (i) and (iii) replaced by C ( K ; Y ) C(K;\mathcal{Y}) C ( K ; Y ) ; the same explicit bound follows from Theorem 6.5 .
Proof. Step 1: characters are dense. The functions χ ξ ( x ) : = e i ⟨ x , ξ ⟩ \chi_\xi(x):=e^{i\langle x,\xi\rangle} χ ξ ( x ) := e i ⟨ x , ξ ⟩ , ξ ∈ H \xi\in\mathcal{H} ξ ∈ H , restricted to K K K , satisfy χ ξ χ ξ ′ = χ ξ + ξ ′ \chi_\xi\chi_{\xi'}=\chi_{\xi+\xi'} χ ξ χ ξ ′ = χ ξ + ξ ′ , χ ξ ‾ = χ − ξ \overline{\chi_\xi}=\chi_{-\xi} χ ξ = χ − ξ , and χ 0 = 1 \chi_0=1 χ 0 = 1 , so their finite linear combinations form a self-conjugate subalgebra of C ( K ) C(K) C ( K ) containing the constants. If x ≠ x ′ x\ne x' x = x ′ in K K K , then ξ = t ( x − x ′ ) \xi=t(x-x') ξ = t ( x − x ′ ) with 0 < t ∥ x − x ′ ∥ 2 < 2 π 0<t\|x-x'\|^2<2\pi 0 < t ∥ x − x ′ ∥ 2 < 2 π gives χ ξ ( x ) ≠ χ ξ ( x ′ ) \chi_\xi(x)\ne\chi_\xi(x') χ ξ ( x ) = χ ξ ( x ′ ) . By the Stone–Weierstrass theorem there are w 1 , … , w J ∈ C w_1,\ldots,w_J\in\mathbb{C} w 1 , … , w J ∈ C and ξ 1 , … , ξ J ∈ H \xi_1,\ldots,\xi_J\in\mathcal{H} ξ 1 , … , ξ J ∈ H with ∥ f − ∑ j w j χ ξ j ∥ C ( K ) < ε / 2 \|f-\sum_j w_j\chi_{\xi_j}\|_{C(K)}<\varepsilon/2 ∥ f − ∑ j w j χ ξ j ∥ C ( K ) < ε /2 .
Step 2: characters have spectral densities. Let φ ∈ C ∞ ( R ) \varphi\in C^\infty(\mathbb{R}) φ ∈ C ∞ ( R ) be nonnegative, positive on ( − 1 , 1 ) (-1,1) ( − 1 , 1 ) , and zero outside [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] , and for δ > 0 \delta>0 δ > 0 put g j ( ξ ) : = φ ( ∥ ξ − ξ j ∥ 2 / δ 2 ) / Z j g_j(\xi):=\varphi(\|\xi-\xi_j\|^2/\delta^2)/Z_j g j ( ξ ) := φ ( ∥ ξ − ξ j ∥ 2 / δ 2 ) / Z j , where Z j : = ∫ φ ( ∥ ξ − ξ j ∥ 2 / δ 2 ) ν α ( d ξ ) Z_j:=\int\varphi(\|\xi-\xi_j\|^2/\delta^2)\nu_\alpha(d\xi) Z j := ∫ φ ( ∥ ξ − ξ j ∥ 2 / δ 2 ) ν α ( d ξ ) is positive because ν α \nu_\alpha ν α has full support and is finite on bounded sets. Then g j ≥ 0 g_j\geq0 g j ≥ 0 , ∫ g j d ν α = 1 \int g_jd\nu_\alpha=1 ∫ g j d ν α = 1 , and g j g_j g j vanishes outside the ball of radius δ \delta δ around ξ j \xi_j ξ j . For x ∈ K x\in K x ∈ K ,
∣ χ ξ j ( x ) − f g j ( x ) ∣ = ∣ ∫ H ( e i ⟨ x , ξ j ⟩ − e i ⟨ x , ξ ⟩ ) g j ( ξ ) ν α ( d ξ ) ∣ ≤ sup ∥ ξ − ξ j ∥ ≤ δ ∣ ⟨ x , ξ j − ξ ⟩ ∣ ≤ r K δ , |\chi_{\xi_j}(x)-f_{g_j}(x)|=\left|\int_{\mathcal{H}}\left(e^{i\langle x,\xi_j\rangle}-e^{i\langle x,\xi\rangle}\right)g_j(\xi)\nu_\alpha(d\xi)\right|\leq\sup_{\|\xi-\xi_j\|\leq\delta}|\langle x,\xi_j-\xi\rangle|\leq r_K\delta, ∣ χ ξ j ( x ) − f g j ( x ) ∣ = ∫ H ( e i ⟨ x , ξ j ⟩ − e i ⟨ x , ξ ⟩ ) g j ( ξ ) ν α ( d ξ ) ≤ ∥ ξ − ξ j ∥ ≤ δ sup ∣ ⟨ x , ξ j − ξ ⟩ ∣ ≤ r K δ ,
with r K = sup K ∥ x ∥ r_K=\sup_K\|x\| r K = sup K ∥ x ∥ . Choose δ \delta δ with r K δ ∑ j ∣ w j ∣ < ε / 2 r_K\delta\sum_j|w_j|<\varepsilon/2 r K δ ∑ j ∣ w j ∣ < ε /2 and put g : = ∑ j w j g j g:=\sum_jw_jg_j g := ∑ j w j g j . Then ∥ f − f g ∥ C ( K ) < ε \|f-f_g\|_{C(K)}<\varepsilon ∥ f − f g ∥ C ( K ) < ε , which is (i).
Step 3: regularity. Each g j g_j g j is bounded and vanishes outside a bounded set. Its restrictions ω ↦ g j ( ω a ) \omega\mapsto g_j(\omega a) ω ↦ g j ( ω a ) are C ∞ C^\infty C ∞ , because ω ↦ ∥ ω a − ξ j ∥ 2 \omega\mapsto\|\omega a-\xi_j\|^2 ω ↦ ∥ ω a − ξ j ∥ 2 is a polynomial, and their derivatives on I I I are bounded by polynomials in ∥ a ∥ \|a\| ∥ a ∥ by the chain rule. Hence g g g satisfies Definition 4.2 by Lemma 4.4(b) and the finite-combination clause of (c). Theorem 4.5 (iii) with ( ( σ , ρ ) ) α = 1 ((\sigma,\rho))_\alpha=1 (( σ , ρ ) ) α = 1 gives f g = ∫ γ g σ ( ⟨ a , ⋅ ⟩ − b ) d λ α f_g=\int\gamma_g\sigma(\langle a,\cdot\rangle-b)d\lambda_\alpha f g = ∫ γ g σ (⟨ a , ⋅ ⟩ − b ) d λ α , and the moment bound of Theorem 6.6 , with an arbitrary integer order, shows that γ g λ α \gamma_g\lambda_\alpha γ g λ α is finite with finite moments of all orders; this is (ii). Part (iii) is Theorem 6.5 applied to f g f_g f g , together with the triangle inequality and (i). Density of finite-width networks follows from (iii), or from (ii) and Lemma 6.7 when σ \sigma σ is merely continuous.
Vector-valued case. For f ∈ C ( H ; Y ) f \in C(\mathcal{H};\mathcal{Y}) f ∈ C ( H ; Y ) and compact K K K , f f f is uniformly continuous on K K K ; given ε \varepsilon ε , choose a finite cover of K K K by balls of radius δ ′ \delta' δ ′ on which f f f varies by less than ε / 4 \varepsilon/4 ε /4 , a continuous partition of unity ( π i ) (\pi_i) ( π i ) subordinate to it, and points x i x_i x i in the balls. Then ∥ f − ∑ i π i f ( x i ) ∥ C ( K ; Y ) < ε / 4 \left\|f-\sum_i\pi_i f(x_i)\right\|_{C(K;\mathcal{Y})}<\varepsilon/4 ∥ f − ∑ i π i f ( x i ) ∥ C ( K ; Y ) < ε /4 , and each scalar π i \pi_i π i is approximated by a trigonometric sum as in Step 1. The resulting approximant is ∑ j w j χ ξ j \sum_j w_j\chi_{\xi_j} ∑ j w j χ ξ j with w j ∈ Y w_j\in\mathcal{Y} w j ∈ Y , and Steps 2 and 3 apply with Y \mathcal{Y} Y -valued g = ∑ j w j g j g=\sum_j w_jg_j g = ∑ j w j g j , using Theorem 4.11. The uniform rate for the resulting vector coefficient measure follows from Theorem 6.5 .
The coefficient is constructed on the full direction space H \mathcal{H} H , without choosing an input truncation first. A sampled network still depends on only finitely many linear measurements of its input. The constants V V V and M 2 M_2 M 2 depend on the spectral approximant, and hence on f f f , K K K , ε \varepsilon ε ; the theorem does not quantify this first approximation step. If f f f already has a density satisfying Theorem 6.6 , sampling applies directly with no preliminary error. The weaker Sobolev route of Theorem 5.10 gives the same conclusion for its class when s > 5 / 2 s>5/2 s > 5/2 and the reconstruction constant is nonzero.
Vector-valued approximation
Let Γ \Gamma Γ be a Y \mathcal{Y} Y -valued measure of bounded variation on H × R \mathcal{H}\times\mathbb{R} H × R . Since Y \mathcal{Y} Y is a Hilbert space, Γ = h ∣ Γ ∣ \Gamma=h|\Gamma| Γ = h ∣Γ∣ with ∥ h ∥ Y = 1 \|h\|_{\mathcal{Y}}=1 ∥ h ∥ Y = 1 ∣ Γ ∣ |\Gamma| ∣Γ∣ -almost everywhere, and the sampled network (63) is defined with Y \mathcal{Y} Y -valued weights.
Lemma 6.9 (Mean-square error identity for Hilbert-valued averages). Let X X X be a separable Hilbert space and Y ∈ L 2 ( p ; X ) \mathcal{Y}\in L^2(p;X) Y ∈ L 2 ( p ; X ) . For independent copies Y j Y_j Y j , put f = V E Y f=V\mathbb{E}Y f = V E Y and f N = V N − 1 ∑ j Y j f_N=VN^{-1}\sum_jY_j f N = V N − 1 ∑ j Y j . Then
E ∥ f N − f ∥ X 2 = V 2 N ( E ∥ Y ∥ X 2 − ∥ E Y ∥ X 2 ) ≤ V 2 N E ∥ Y ∥ X 2 , (74) \mathbb{E}\|f_N-f\|_X^2=\frac{V^2}{N}\left(\mathbb{E}\|Y\|_X^2-\|\mathbb{E}Y\|_X^2\right)\leq\frac{V^2}{N}\mathbb{E}\|Y\|_X^2, \tag*{(74)} E ∥ f N − f ∥ X 2 = N V 2 ( E ∥ Y ∥ X 2 − ∥ E Y ∥ X 2 ) ≤ N V 2 E ∥ Y ∥ X 2 , (74) and a deterministic sample satisfies the same upper bound.
Proof. Write Z j = Y j − E Y Z_j=Y_j-\mathbb{E}Y Z j = Y j − E Y . Independence and zero means make E ⟨ Z j , Z k ⟩ = 0 \mathbb{E}\langle Z_j,Z_k\rangle=0 E ⟨ Z j , Z k ⟩ = 0 for j ≠ k j\ne k j = k , so the expanded squared norm leaves N E ∥ Z 1 ∥ 2 N\mathbb{E}\|Z_1\|^2 N E ∥ Z 1 ∥ 2 .
Corollary 6.10 (Vector-valued rates). Let σ \sigma σ be real and globally Lipschitz, and let N ≥ 1 N\geq1 N ≥ 1 . If V = 0 V=0 V = 0 , all conclusions hold for the zero network. Otherwise put p = ∣ Γ ∣ / V p=|\Gamma|/V p = ∣Γ∣/ V .
(i) Suppose (67) holds. For every Borel probability measure ζ \zeta ζ on H \mathcal{H} H with ∫ ∥ x ∥ 2 ζ ( d x ) < ∞ \int\|x\|^2\zeta(dx)<\infty ∫ ∥ x ∥ 2 ζ ( d x ) < ∞ ,
E ∥ f N − f ∥ L 2 ( ζ ; Y ) 2 = 1 N ( V 2 ∫ H × R ∥ σ ( ⟨ a , ⋅ ⟩ − b ) ∥ L 2 ( ζ ) 2 p ( d a , d b ) − ∥ f ∥ L 2 ( ζ ; Y ) 2 ) . (75) \mathbb{E}\|f_N-f\|_{L^2(\zeta;\mathcal{Y})}^2=\frac{1}{N}\left(V^2\int_{\mathcal{H}\times\mathbb{R}}\|\sigma(\langle a,\cdot\rangle-b)\|_{L^2(\zeta)}^2p(da,db)-\|f\|_{L^2(\zeta;\mathcal{Y})}^2\right). \tag*{(75)} E ∥ f N − f ∥ L 2 ( ζ ; Y ) 2 = N 1 ( V 2 ∫ H × R ∥ σ (⟨ a , ⋅ ⟩ − b ) ∥ L 2 ( ζ ) 2 p ( d a , d b ) − ∥ f ∥ L 2 ( ζ ; Y ) 2 ) . (75) Consequently,
E ∥ f N − f ∥ L 2 ( ζ ; Y ) 2 ≤ V 2 N ∫ H × R ∥ σ ( ⟨ a , ⋅ ⟩ − b ) ∥ L 2 ( ζ ) 2 p ( d a , d b ) \mathbb{E}\|f_N-f\|_{L^2(\zeta;\mathcal{Y})}^2\leq\frac{V^2}{N}\int_{\mathcal{H}\times\mathbb{R}}\|\sigma(\langle a,\cdot\rangle-b)\|_{L^2(\zeta)}^2p(da,db) E ∥ f N − f ∥ L 2 ( ζ ; Y ) 2 ≤ N V 2 ∫ H × R ∥ σ (⟨ a , ⋅ ⟩ − b ) ∥ L 2 ( ζ ) 2 p ( d a , d b ) ≤ 2 V 2 N ( ∣ σ ( 0 ) ∣ 2 + Lip ( σ ) 2 ( 1 + ∫ ∥ x ∥ 2 ζ ( d x ) ) M 2 2 ) . \leq\frac{2V^2}{N}\left(|\sigma(0)|^2+\operatorname{Lip}(\sigma)^2\left(1+\int\|x\|^2\zeta(dx)\right)M_2^2\right). ≤ N 2 V 2 ( ∣ σ ( 0 ) ∣ 2 + Lip ( σ ) 2 ( 1 + ∫ ∥ x ∥ 2 ζ ( d x ) ) M 2 2 ) . (ii) For this part it suffices that ∫ ( ∥ a ∥ + ∣ b ∣ ) d p < ∞ \int(\|a\|+|b|)\,dp<\infty ∫ ( ∥ a ∥ + ∣ b ∣ ) d p < ∞ . For every compact K ⊂ H K\subset\mathcal{H} K ⊂ H , E ∥ f N − f ∥ C ( K ; Y ) ≤ 2 V R N Y ( K ; p , σ ) \mathbb{E}\|f_N-f\|_{C(K;\mathcal{Y})}\leq2V\mathcal{R}_N^{\mathcal{Y}}(K;p,\sigma) E ∥ f N − f ∥ C ( K ; Y ) ≤ 2 V R N Y ( K ; p , σ ) , where R N Y \mathcal{R}_N^{\mathcal{Y}} R N Y is (64) with the absolute value replaced by the norm of Y \mathcal{Y} Y , and R N Y ( K ; p , σ ) → 0 \mathcal{R}_N^{\mathcal{Y}}(K;p,\sigma)\to0 R N Y ( K ; p , σ ) → 0 as N → ∞ N\to\infty N → ∞ .
Proof. Part (i) is the Hilbert-valued variance identity of Lemma 6.9 applied in X = L 2 ( ζ ; Y ) X=L^{2}(\zeta;\mathcal{Y}) X = L 2 ( ζ ; Y ) . The atom has squared norm V 2 ∥ σ ( ⟨ a , ⋅ ⟩ − b ) ∥ L 2 ( ζ ) 2 V^{2}\|\sigma(\langle a,\cdot\rangle-b)\|_{L^{2}(\zeta)}^{2} V 2 ∥ σ (⟨ a , ⋅ ⟩ − b ) ∥ L 2 ( ζ ) 2 because ∥ h ∥ Y = 1 \|h\|_{\mathcal{Y}}=1 ∥ h ∥ Y = 1 almost everywhere. Its second moment is finite by ∣ σ ( u ) ∣ 2 ≤ 2 ∣ σ ( 0 ) ∣ 2 + 2 Lip ( σ ) 2 u 2 |\sigma(u)|^{2}\le2|\sigma(0)|^{2}+2\operatorname{Lip}(\sigma)^{2}u^{2} ∣ σ ( u ) ∣ 2 ≤ 2∣ σ ( 0 ) ∣ 2 + 2 Lip ( σ ) 2 u 2 and u 2 ≤ ( ∥ a ∥ 2 + b 2 ) ( ∥ x ∥ 2 + 1 ) u^{2}\le(\|a\|^{2}+b^{2})(\|x\|^{2}+1) u 2 ≤ ( ∥ a ∥ 2 + b 2 ) ( ∥ x ∥ 2 + 1 ) , which also give the bound. For (ii), the map ( a , b , y ) ↦ [ x ↦ y σ ( ⟨ a , x ⟩ − b ) ] (a,b,y)\mapsto[x\mapsto y\sigma(\langle a,x\rangle-b)] ( a , b , y ) ↦ [ x ↦ y σ (⟨ a , x ⟩ − b )] is continuous into C ( K ; Y ) C(K;\mathcal{Y}) C ( K ; Y ) ; composing with the measurable polar density gives a strongly measurable atom. Its norm is at most ∣ σ ( 0 ) ∣ + Lip ( σ ) R K ∥ a ∥ 2 + b 2 |\sigma(0)|+\operatorname{Lip}(\sigma)R_{K}\sqrt{\|a\|^{2}+b^{2}} ∣ σ ( 0 ) ∣ + Lip ( σ ) R K ∥ a ∥ 2 + b 2 , which is integrable under the first-moment assumption. Apply Lemma 6.2 in the separable Banach space C ( K ; Y ) C(K;\mathcal{Y}) C ( K ; Y ) .
Under a second moment, Theorem 6.5 already gives an explicit uniform rate for the whole Hilbert-valued output. The variance formula records the corresponding exact L 2 L^{2} L 2 identity. The first-moment conclusion requires less integrability and extends, through Lemma 6.2 , to integrable atoms in any separable Banach space.
Examples on infinite-dimensional spaces
We apply the representation and approximation results to explicit targets on infinite-dimensional spaces. Gaussian targets and nonlinear layers built from convolution and a Dirichlet solution operator demonstrate how the coefficients and transform formulas can be computed.
A function f f f on H \mathcal{H} H is cylindrical if f = f ~ ∘ L f=\widetilde{f}\circ L f = f ∘ L for a finite-rank linear map L L L , continuous or not. The examples below identify explicit conditions under which the exact targets are non-cylindrical. Their finite-width approximants use finitely many linear measurements, with directions selected from the coefficient law. We write φ v \varphi_{v} φ v for the centred one-dimensional Gaussian density of variance v > 0 v>0 v > 0 , with φ 0 = δ 0 \varphi_{0}=\delta_{0} φ 0 = δ 0 , and σ G a u s s ( u ) : = e − u 2 / 2 \sigma_{\mathrm{Gauss}}(u):=e^{-u^{2}/2} σ Gauss ( u ) := e − u 2 /2 for the Gaussian activation.
A Gaussian target with a closed-form transform
Lemma 7.1 (Gaussian integral of a quadratic exponential). Let Ξ \Xi Ξ be a positive self-adjoint trace-class operator, S S S a bounded positive self-adjoint operator, and x ∈ H x\in\mathcal{H} x ∈ H . Put M : = Ξ 1 / 2 S Ξ 1 / 2 M:=\Xi^{1/2}S\Xi^{1/2} M := Ξ 1/2 S Ξ 1/2 , which is trace class. Then
∫ H e i ⟨ x , ξ ⟩ − ⟨ S ξ , ξ ⟩ / 2 μ Ξ ( d ξ ) = det ( I + M ) − 1 / 2 exp ( − 1 2 ⟨ Ξ 1 / 2 ( I + M ) − 1 Ξ 1 / 2 x , x ⟩ ) . \int_{\mathcal{H}}e^{i\langle x,\xi\rangle-\langle S\xi,\xi\rangle/2}\,\mu_{\Xi}(d\xi)=\det(I+M)^{-1/2}\exp\left(-\frac{1}{2}\langle\Xi^{1/2}(I+M)^{-1}\Xi^{1/2}x,x\rangle\right). ∫ H e i ⟨ x , ξ ⟩ − ⟨ S ξ , ξ ⟩ /2 μ Ξ ( d ξ ) = det ( I + M ) − 1/2 exp ( − 2 1 ⟨ Ξ 1/2 ( I + M ) − 1 Ξ 1/2 x , x ⟩ ) . Proof. Let ( u j ) (u_{j}) ( u j ) be an orthonormal eigenbasis of M M M with eigenvalues m j ≥ 0 m_{j}\ge0 m j ≥ 0 , and let η j \eta_{j} η j be independent standard normal variables. The series ξ : = ∑ j η j Ξ 1 / 2 u j \xi:=\sum_{j}\eta_{j}\Xi^{1/2}u_{j} ξ := ∑ j η j Ξ 1/2 u j converges in L 2 ( Ω ; H ) L^{2}(\Omega;\mathcal{H}) L 2 ( Ω ; H ) because ∑ j ∥ Ξ 1 / 2 u j ∥ 2 = tr Ξ < ∞ \sum_{j}\|\Xi^{1/2}u_{j}\|^{2}=\operatorname{tr}\Xi<\infty ∑ j ∥ Ξ 1/2 u j ∥ 2 = tr Ξ < ∞ , and its characteristic functional is exp ( − 1 2 ∑ j ⟨ Ξ 1 / 2 u j , z ⟩ 2 ) = e − ⟨ Ξ z , z ⟩ / 2 \exp(-\frac{1}{2}\sum_{j}\langle\Xi^{1/2}u_{j},z\rangle^{2})=e^{-\langle\Xi z,z\rangle/2} exp ( − 2 1 ∑ j ⟨ Ξ 1/2 u j , z ⟩ 2 ) = e − ⟨ Ξ z , z ⟩ /2 , so ξ ∼ μ Ξ \xi\sim\mu_{\Xi} ξ ∼ μ Ξ . For the partial sums ξ n \xi_{n} ξ n , ⟨ S ξ n , ξ n ⟩ = ∑ j , k ≤ n η j η k ⟨ M u j , u k ⟩ = ∑ j ≤ n m j η j 2 \langle S\xi_{n},\xi_{n}\rangle=\sum_{j,k\le n}\eta_{j}\eta_{k}\langle Mu_{j},u_{k}\rangle=\sum_{j\le n}m_{j}\eta_{j}^{2} ⟨ S ξ n , ξ n ⟩ = ∑ j , k ≤ n η j η k ⟨ M u j , u k ⟩ = ∑ j ≤ n m j η j 2 , which converges almost surely to ∑ j m j η j 2 \sum_{j}m_{j}\eta_{j}^{2} ∑ j m j η j 2 since ∑ j m j < ∞ \sum_{j}m_{j}<\infty ∑ j m j < ∞ ; continuity of the quadratic form along an almost surely convergent subsequence gives ⟨ S ξ , ξ ⟩ = ∑ j m j η j 2 \langle S\xi,\xi\rangle=\sum_{j}m_{j}\eta_{j}^{2} ⟨ S ξ , ξ ⟩ = ∑ j m j η j 2 . Likewise ⟨ x , ξ ⟩ = ∑ j η j x j \langle x,\xi\rangle=\sum_{j}\eta_{j}x_{j} ⟨ x , ξ ⟩ = ∑ j η j x j with x j : = ⟨ Ξ 1 / 2 x , u j ⟩ x_{j}:=\langle\Xi^{1/2}x,u_{j}\rangle x j := ⟨ Ξ 1/2 x , u j ⟩ . By independence and dominated convergence,
E e i ⟨ x , ξ ⟩ − ⟨ S ξ , ξ ⟩ / 2 = ∏ j E e i η j x j − m j η j 2 / 2 = ∏ j ( 1 + m j ) − 1 / 2 e − x j 2 / ( 2 ( 1 + m j ) ) , \mathbb{E}e^{i\langle x,\xi\rangle-\langle S\xi,\xi\rangle/2}=\prod_{j}\mathbb{E}e^{i\eta_{j}x_{j}-m_{j}\eta_{j}^{2}/2}=\prod_{j}(1+m_{j})^{-1/2}e^{-x_{j}^{2}/(2(1+m_{j}))}, E e i ⟨ x , ξ ⟩ − ⟨ S ξ , ξ ⟩ /2 = j ∏ E e i η j x j − m j η j 2 /2 = j ∏ ( 1 + m j ) − 1/2 e − x j 2 / ( 2 ( 1 + m j )) , and ∑ j x j 2 / ( 1 + m j ) = ⟨ ( I + M ) − 1 Ξ 1 / 2 x , Ξ 1 / 2 x ⟩ \sum_{j}x_{j}^{2}/(1+m_{j})=\langle(I+M)^{-1}\Xi^{1/2}x,\Xi^{1/2}x\rangle ∑ j x j 2 / ( 1 + m j ) = ⟨( I + M ) − 1 Ξ 1/2 x , Ξ 1/2 x ⟩ .
Example 7.2 (Closed-form transform and integral representation of T α f W T_{\alpha}f_{W} T α f W ). Let W W W be a bounded, positive, injective, self-adjoint operator such that M : = Q 1 / 2 W Q 1 / 2 M:=Q^{1/2}WQ^{1/2} M := Q 1/2 W Q 1/2 is trace class, and let
f W ( x ) : = e − ⟨ W x , x ⟩ / 2 , D : = det ( I + M ) , κ W ( ξ ) : = ⟨ Q 1 / 2 ( I + M ) − 1 Q 1 / 2 ξ , ξ ⟩ . (76) f_{W}(x):=e^{-\langle Wx,x\rangle/2},\qquad D:=\det(I+M),\qquad\kappa_{W}(\xi):=\langle Q^{1/2}(I+M)^{-1}Q^{1/2}\xi,\xi\rangle. \tag*{(76)} f W ( x ) := e − ⟨ W x , x ⟩ /2 , D := det ( I + M ) , κ W ( ξ ) := ⟨ Q 1/2 ( I + M ) − 1 Q 1/2 ξ , ξ ⟩ . (76)
Then, for every α > 0 \alpha> 0 α > 0 and every band-pass ρ \rho ρ :
F Q f W ( ξ ) = D − 1 / 2 e − κ W ( ξ ) / 2 F_Qf_W(\xi)=D^{-1/2}e^{-\kappa_W(\xi)/2} F Q f W ( ξ ) = D − 1/2 e − κ W ( ξ ) /2 and R ρ f W ( a , b ) = D − 1 / 2 ( ρ ∗ ϕ κ W ( a ) ) ( − b ) R_\rho f_W(a,b)=D^{-1/2}(\rho*\phi_{\kappa_W(a)})(-b) R ρ f W ( a , b ) = D − 1/2 ( ρ ∗ ϕ κ W ( a ) ) ( − b ) ; in particular f W ∈ D α f_W\in\mathcal{D}_\alpha f W ∈ D α , and f W f_W f W is not cylindrical when W W W has infinite rank.
g : = F Q f W g:=F_Qf_W g := F Q f W satisfies Definition 4.2 , and the image T α f W T_\alpha f_W T α f W is represented by
f g ( x ) = D − 1 / 2 ∫ 0 ∞ det ( I + 2 s P 1 / 2 S W P 1 / 2 ) − 1 / 2 exp ( − 1 2 ⟨ Ξ s x , x ⟩ ) s α / 2 − 1 d s . (77) f_g(x)=D^{-1/2}\int_0^\infty\det\left(I+2sP^{1/2}S_WP^{1/2}\right)^{-1/2}\exp\left(-\frac{1}{2}\langle\Xi_sx,x\rangle\right)s^{\alpha/2-1}\,\mathrm{d}s. \tag*{(77)} f g ( x ) = D − 1/2 ∫ 0 ∞ det ( I + 2 s P 1/2 S W P 1/2 ) − 1/2 exp ( − 2 1 ⟨ Ξ s x , x ⟩ ) s α /2 − 1 d s . (77)
where S W : = Q 1 / 2 ( I + M ) − 1 Q 1 / 2 S_W:=Q^{1/2}(I+M)^{-1}Q^{1/2} S W := Q 1/2 ( I + M ) − 1 Q 1/2 and Ξ s : = 2 s P 1 / 2 ( I + 2 s P 1 / 2 S W P 1 / 2 ) − 1 P 1 / 2 \Xi_s:=2sP^{1/2}\left(I+2sP^{1/2}S_WP^{1/2}\right)^{-1}P^{1/2} Ξ s := 2 s P 1/2 ( I + 2 s P 1/2 S W P 1/2 ) − 1 P 1/2 .
For every real, globally Lipschitz, non-polynomial σ \sigma σ , including ReLU, the coefficient measure R ρ f W λ α = γ g λ α R_\rho f_W\lambda_\alpha=\gamma_g\lambda_\alpha R ρ f W λ α = γ g λ α has finite total variation and second moment, the integral network S σ [ R ρ f W λ α ] S_\sigma[R_\rho f_W\lambda_\alpha] S σ [ R ρ f W λ α ] equals ( ( σ , ρ ) ) α f g ((\sigma,\rho))_\alpha f_g (( σ , ρ ) ) α f g , and its sampled network converges at the rate N − 1 / 2 N^{-1/2} N − 1/2 in C ( K ) C(K) C ( K ) as in (72) .
Proof. (i). Lemma 7.1 with Ξ = Q \Xi=Q Ξ = Q , S = W S=W S = W , and x x x replaced by − ξ -\xi − ξ gives F Q f W ( ξ ) = D − 1 / 2 e − κ W ( ξ ) / 2 F_Qf_W(\xi)=D^{-1/2}e^{-\kappa_W(\xi)/2} F Q f W ( ξ ) = D − 1/2 e − κ W ( ξ ) /2 . By (15) , the bias Fourier transform of R ρ f W ( a , ⋅ ) R_\rho f_W(a,\cdot) R ρ f W ( a , ⋅ ) is D − 1 / 2 ρ ^ ( − ω ) e − ω 2 κ W ( a ) / 2 D^{-1/2}\widehat{\rho}(-\omega)e^{-\omega^2\kappa_W(a)/2} D − 1/2 ρ ( − ω ) e − ω 2 κ W ( a ) /2 , whose inverse Fourier transform is D − 1 / 2 ( ρ ∗ ϕ κ W ( a ) ) ( − b ) D^{-1/2}(\rho*\phi_{\kappa_W(a)})(-b) D − 1/2 ( ρ ∗ ϕ κ W ( a ) ) ( − b ) . Since ( I + M ) − 1 ≥ ( 1 + ∥ M ∥ ) − 1 I (I+M)^{-1}\ge(1+\lVert M\rVert)^{-1}I ( I + M ) − 1 ≥ ( 1 + ∥ M ∥ ) − 1 I , we have κ W ( ξ ) ≥ ( 1 + ∥ M ∥ ) − 1 ⟨ Q ξ , ξ ⟩ \kappa_W(\xi)\ge(1+\lVert M\rVert)^{-1}\langle Q\xi,\xi\rangle κ W ( ξ ) ≥ ( 1 + ∥ M ∥ ) − 1 ⟨ Q ξ , ξ ⟩ , and Lemma 3.12 gives f W ∈ D α f_W\in\mathcal{D}_\alpha f W ∈ D α . If f W = F ~ ∘ L f_W=\widetilde{F}\circ L f W = F ∘ L with L L L of finite rank and W W W of infinite rank, take 0 ≠ x ∈ ker L 0\ne x\in\ker L 0 = x ∈ ker L ; then f W ( x ) = f W ( 0 ) = 1 f_W(x)=f_W(0)=1 f W ( x ) = f W ( 0 ) = 1 , while ⟨ W x , x ⟩ = ∥ W 1 / 2 x ∥ 2 > 0 \langle W x,x\rangle=\lVert W^{1/2}x\rVert^2>0 ⟨ W x , x ⟩ = ∥ W 1/2 x ∥ 2 > 0 gives f W ( x ) < 1 f_W(x)<1 f W ( x ) < 1 .
(ii). g = F Q f W ∈ L 1 ( ν α ) g=F_Qf_W\in L^1(\nu_\alpha) g = F Q f W ∈ L 1 ( ν α ) by (i), so Theorem 4.8 (iii) represents T α f W T_\alpha f_W T α f W by f g f_g f g . By Lemma 3.1 , applied to the absolutely integrable complex integrand,
f g ( x ) = D − 1 / 2 ∫ 0 ∞ [ ∫ H e i ⟨ x , ξ ⟩ − ⟨ S W ξ , ξ ⟩ / 2 μ 2 s P ( d ξ ) ] s α / 2 − 1 d s . f_g(x)=D^{-1/2}\int_0^\infty\left[\int_{\mathcal{H}}e^{i\langle x,\xi\rangle-\langle S_W\xi,\xi\rangle/2}\,\mu_{2sP}(\mathrm{d}\xi)\right]s^{\alpha/2-1}\,\mathrm{d}s. f g ( x ) = D − 1/2 ∫ 0 ∞ [ ∫ H e i ⟨ x , ξ ⟩ − ⟨ S W ξ , ξ ⟩ /2 μ 2 s P ( d ξ ) ] s α /2 − 1 d s .
Applying Lemma 7.1 with Ξ = 2 s P \Xi=2sP Ξ = 2 s P and S = S W S=S_W S = S W evaluates the bracket and gives (77) .
(iii). g g g has the form of Lemma 4.4 (a) with q = D − 1 / 2 q=D^{-1/2} q = D − 1/2 and S = S W ≥ ( 1 + ∥ M ∥ ) − 1 Q S=S_W\ge(1+\lVert M\rVert)^{-1}Q S = S W ≥ ( 1 + ∥ M ∥ ) − 1 Q , so it satisfies Definition 4.2 . (19) gives R ρ f W = W ρ g = γ g R_\rho f_W=W_\rho g=\gamma_g R ρ f W = W ρ g = γ g , and Theorem 6.6 gives finite total variation of the coefficient measure, the identity S σ [ γ g λ α ] = ( ( σ , ρ ) ) α f g S_\sigma[\gamma_g\lambda_\alpha]=((\sigma,\rho))_\alpha f_g S σ [ γ g λ α ] = (( σ , ρ ) ) α f g , and the rate.
Of the standing hypotheses on W W W , the transform and convolution formulas of part (i) and the membership f W ∈ D α f_W\in\mathcal{D}_\alpha f W ∈ D α use only that W W W is bounded, positive and self-adjoint with M M M trace class; injectivity of W W W is needed for the non-cylindricity of f W f_W f W , and the analysis filter does not appear in part (ii) at all.
Formula (77) displays the frame operator concretely: the Gaussian target and its image T α f W T_\alpha f_W T α f W are both explicit, and the ridgelet coefficient of the former is the network coefficient of the latter.
Gaussian-parameter networks on l2
Let H = ℓ 2 ( N ) \mathcal{H}=\ell^2(\mathbb{N}) H = ℓ 2 ( N ) with standard basis ( e j ) (e_j) ( e j ) , and let Q e j = q j e j Qe_j=q_je_j Q e j = q j e j with q j > 0 q_j>0 q j > 0 and ∑ j q j < ∞ \sum_jq_j<\infty ∑ j q j < ∞ , so that Q Q Q is injective, trace class, and of infinite rank.
Lemma 7.3 (Absolute hinge representation of the Gaussian). For ϕ ( u ) = e − u 2 / 2 \phi(u)=e^{-u^2/2} ϕ ( u ) = e − u 2 /2 , ϕ ( u ) = ∫ R ( u − b ) + ϕ ′ ′ ( b ) d b \phi(u)=\int_{\mathbb{R}}(u-b)_+\phi''(b)\,\mathrm{d}b ϕ ( u ) = ∫ R ( u − b ) + ϕ ′′ ( b ) d b absolutely for each u u u , and ∫ R ( 1 + ∣ b ∣ k ) ∣ ϕ ′ ′ ( b ) ∣ d b < ∞ \int_{\mathbb{R}}(1+|b|^k)|\phi''(b)|\mathrm{d}b<\infty ∫ R ( 1 + ∣ b ∣ k ) ∣ ϕ ′′ ( b ) ∣ d b < ∞ for every k ≥ 0 k\ge0 k ≥ 0 .
Proof. Since ϕ ′ ′ ( b ) = ( b 2 − 1 ) e − b 2 / 2 \phi''(b)=(b^2-1)e^{-b^2/2} ϕ ′′ ( b ) = ( b 2 − 1 ) e − b 2 /2 , all polynomially weighted absolute integrals are finite. Integrating by parts on ( − L , u ) (-L,u) ( − L , u ) gives ∫ − L u ( u − b ) ϕ ′ ′ ( b ) d b = ϕ ( u ) − ϕ ( − L ) − ( u + L ) ϕ ′ ( − L ) \int_{-L}^{u}(u-b)\phi''(b)\,\mathrm{d}b=\phi(u)-\phi(-L)-(u+L)\phi'(-L) ∫ − L u ( u − b ) ϕ ′′ ( b ) d b = ϕ ( u ) − ϕ ( − L ) − ( u + L ) ϕ ′ ( − L ) , which tends to ϕ ( u ) \phi(u) ϕ ( u ) as L → ∞ L\to\infty L → ∞ .
Example 7.4 (ReLU and Gaussian networks with Gaussian parameters). The integral networks
f R e L U , Q ( x ) : = ∫ H ReLU ( ⟨ a , x ⟩ ) μ Q ( d a ) , f G a u s s , Q ( x ) : = ∫ H σ G a u s s ( ⟨ a , x ⟩ ) μ Q ( d a ) (78) f_{\mathrm{ReLU},Q}(x) := \int_{\mathcal{H}} \operatorname{ReLU}(\langle a,x\rangle)\,\mu_Q(\mathrm{d}a),\qquad f_{\mathrm{Gauss},Q}(x) := \int_{\mathcal{H}} \sigma_{\mathrm{Gauss}}(\langle a,x\rangle)\,\mu_Q(\mathrm{d}a) \tag*{(78)} f ReLU , Q ( x ) := ∫ H ReLU (⟨ a , x ⟩) μ Q ( d a ) , f Gauss , Q ( x ) := ∫ H σ Gauss (⟨ a , x ⟩) μ Q ( d a ) (78) have the closed forms
f R e L U , Q ( x ) = ⟨ Q x , x ⟩ 2 π , f G a u s s , Q ( x ) = 1 1 + ⟨ Q x , x ⟩ , f G a u s s , Q ( x ) = ∫ H × R ReLU ( ⟨ a , x ⟩ − b ) ϕ ′ ′ ( b ) μ Q ( d a ) d b . (79) \begin{aligned}
f_{\mathrm{ReLU},Q}(x) &= \sqrt{\frac{\langle Qx,x\rangle}{2\pi}},\qquad f_{\mathrm{Gauss},Q}(x) = \frac{1}{\sqrt{1+\langle Qx,x\rangle}},\\
f_{\mathrm{Gauss},Q}(x) &= \int_{\mathcal{H}\times\mathbb{R}} \operatorname{ReLU}(\langle a,x\rangle-b)\,\phi^{\prime\prime}(b)\,\mu_Q(\mathrm{d}a)\,\mathrm{d}b. \tag*{(79)}
\end{aligned} f ReLU , Q ( x ) f Gauss , Q ( x ) = 2 π ⟨ Q x , x ⟩ , f Gauss , Q ( x ) = 1 + ⟨ Q x , x ⟩ 1 , = ∫ H × R ReLU (⟨ a , x ⟩ − b ) ϕ ′′ ( b ) μ Q ( d a ) d b . (79) where ϕ ( u ) = e − u 2 / 2 \phi(u)=e^{-u^2/2} ϕ ( u ) = e − u 2 /2 and ϕ ′ ′ ( b ) = ( b 2 − 1 ) e − b 2 / 2 \phi^{\prime\prime}(b)=(b^2-1)e^{-b^2/2} ϕ ′′ ( b ) = ( b 2 − 1 ) e − b 2 /2 . Neither target is cylindrical. The second identity shows that a Gaussian-activation network with finite coefficient measure is also a ReLU network with finite coefficient measure, with all parameter moments finite.
Proof. For fixed x x x , ⟨ a , x ⟩ \langle a,x\rangle ⟨ a , x ⟩ is centred Gaussian with variance v = ⟨ Q x , x ⟩ v=\langle Qx,x\rangle v = ⟨ Q x , x ⟩ ; if Z ∼ μ v Z\sim\mu_v Z ∼ μ v , then E Z + = v / 2 π \mathbb{E}Z_+=\sqrt{v/2\pi} E Z + = v /2 π and E e − Z 2 / 2 = ( 1 + v ) − 1 / 2 \mathbb{E}e^{-Z^2/2}=(1+v)^{-1/2} E e − Z 2 /2 = ( 1 + v ) − 1/2 . Integrating ϕ ′ ′ \phi^{\prime\prime} ϕ ′′ twice from − ∞ -\infty − ∞ gives ϕ ( u ) = ∫ R ( u − b ) + ϕ ′ ′ ( b ) d b \phi(u)=\int_{\mathbb{R}}(u-b)_+\phi^{\prime\prime}(b)\,\mathrm{d}b ϕ ( u ) = ∫ R ( u − b ) + ϕ ′′ ( b ) d b (Lemma 7.3), and Fubini is justified by ∫ ∣ ϕ ′ ′ ∣ < ∞ \int|\phi^{\prime\prime}|<\infty ∫ ∣ ϕ ′′ ∣ < ∞ and finite Gaussian first moments. If a target factored through a finite-rank map, a nonzero vector in its kernel would give the same value as x = 0 x=0 x = 0 , contradicting injectivity of Q Q Q .
Corollary 7.5 (Discretization of the Gaussian-parameter ReLU network). Let a 1 , … , a N a_1,\ldots,a_N a 1 , … , a N be independent with law μ Q \mu_Q μ Q and f R e L U , Q , N ( x ) : = 1 N ∑ j = 1 N ReLU ( ⟨ a j , x ⟩ ) f_{\mathrm{ReLU},Q,N}(x):=\frac{1}{N}\sum_{j=1}^N\operatorname{ReLU}(\langle a_j,x\rangle) f ReLU , Q , N ( x ) := N 1 ∑ j = 1 N ReLU (⟨ a j , x ⟩) . Then, for every compact K ⊂ H K\subset\mathcal{H} K ⊂ H , E ∥ f R e L U , Q , N − f R e L U , Q ∥ C ( K ) ≤ 8 R K tr Q / N \mathbb{E}\|f_{\mathrm{ReLU},Q,N}-f_{\mathrm{ReLU},Q}\|_{C(K)}\leq8R_K\sqrt{\operatorname{tr}Q}/\sqrt{N} E ∥ f ReLU , Q , N − f ReLU , Q ∥ C ( K ) ≤ 8 R K tr Q / N .
Proof. Apply Theorem 6.5 with V = 1 V=1 V = 1 , b = 0 b=0 b = 0 , σ ( 0 ) = 0 \sigma(0)=0 σ ( 0 ) = 0 , Lip ( σ ) = 1 \operatorname{Lip}(\sigma)=1 Lip ( σ ) = 1 , and M 2 2 = E ∥ a ∥ 2 = tr Q M_2^2=\mathbb{E}\|a\|^2=\operatorname{tr}Q M 2 2 = E ∥ a ∥ 2 = tr Q .
Neural-operator layers
The layers of neural operators [54 ] compose an integral operator, a pointwise nonlinearity, and another integral operator. We show that such a layer is an integral network in the sense of Definition 2.2, with an explicit finite parameter measure, and that with the Gaussian activation its ridgelet transform is explicit and the whole theory of Sections 4 and 6 applies.
Let ( Ω , m ) (\Omega,m) ( Ω , m ) be a finite measure space, let y ↦ a y ∈ H y\mapsto a_y\in\mathcal{H} y ↦ a y ∈ H be Borel and bounded, ∥ A ∥ ∞ : = sup y ∥ a y ∥ < ∞ \|A\|_\infty:=\sup_y\|a_y\|<\infty ∥ A ∥ ∞ := sup y ∥ a y ∥ < ∞ , and let y ↦ v y ∈ Y y\mapsto v_y\in\mathcal{Y} y ↦ v y ∈ Y be Borel with ∫ Ω ∥ v y ∥ Y m ( d y ) < ∞ \int_\Omega\|v_y\|_{\mathcal{Y}}m(\mathrm{d}y)<\infty ∫ Ω ∥ v y ∥ Y m ( d y ) < ∞ . Define the integral operators
( A x ) ( y ) : = ⟨ a y , x ⟩ , B u : = ∫ Ω v y u ( y ) m ( d y ) , (Ax)(y):=\langle a_y,x\rangle,\qquad Bu:=\int_\Omega v_yu(y)m(\mathrm{d}y), ( A x ) ( y ) := ⟨ a y , x ⟩ , B u := ∫ Ω v y u ( y ) m ( d y ) ,
and, for a continuous real activation σ \sigma σ of polynomial growth, the layer
F ( x ) : = B σ ( A x ) = ∫ Ω v y σ ( ⟨ a y , x ⟩ ) m ( d y ) , x ∈ H . (80) F(x):=B\sigma(Ax)=\int_\Omega v_y\sigma(\langle a_y,x\rangle)m(\mathrm{d}y),\qquad x\in\mathcal{H}. \tag*{(80)} F ( x ) := B σ ( A x ) = ∫ Ω v y σ (⟨ a y , x ⟩) m ( d y ) , x ∈ H . (80)
For φ ∈ Y \varphi\in\mathcal{Y} φ ∈ Y the scalar observable is F φ ( x ) : = ⟨ F ( x ) , φ ⟩ Y F_\varphi(x):=\langle F(x),\varphi\rangle_{\mathcal{Y}} F φ ( x ) := ⟨ F ( x ) , φ ⟩ Y , and w φ ( y ) : = ⟨ v y , φ ⟩ Y w_\varphi(y):=\langle v_y,\varphi\rangle_{\mathcal{Y}} w φ ( y ) := ⟨ v y , φ ⟩ Y .
Example 7.6 (Neural-operator layer as an integral network). (i) Network structure. F = S σ [ Γ ] F=S_\sigma[\Gamma] F = S σ [ Γ ] with the Y \mathcal{Y} Y -valued measure Γ = ι # ( v y m ( d y ) ) \Gamma=\iota_\#(v_y m(\mathrm{d}y)) Γ = ι # ( v y m ( d y )) , ι ( y ) = ( a y , 0 ) \iota(y)=(a_y,0) ι ( y ) = ( a y , 0 ) . Put B 1 = ∫ Ω ∥ v y ∥ Y m ( d y ) B_1=\int_\Omega\|v_y\|_{\mathcal{Y}}m(\mathrm{d}y) B 1 = ∫ Ω ∥ v y ∥ Y m ( d y ) . Its variation satisfies V ≤ B 1 V\leq B_1 V ≤ B 1 , and its normalized second parameter moment is at most ∥ A ∥ ∞ 2 \|A\|_\infty^2 ∥ A ∥ ∞ 2 when V > 0 V>0 V > 0 . If σ \sigma σ is real and globally Lipschitz, Theorem 6.5 yields
E ∥ F N − F ∥ C ( K ; Y ) ≤ B 1 N ( 4 ∣ σ ( 0 ) ∣ + 8 Lip ( σ ) R K ∥ A ∥ ∞ ) . (81) \mathbb{E}\|F_N-F\|_{C(K;\mathcal{Y})}\leq\frac{B_1}{\sqrt{N}}\left(4|\sigma(0)|+8\operatorname{Lip}(\sigma)R_K\|A\|_\infty\right). \tag*{(81)} E ∥ F N − F ∥ C ( K ; Y ) ≤ N B 1 ( 4∣ σ ( 0 ) ∣ + 8 Lip ( σ ) R K ∥ A ∥ ∞ ) . (81)
Thus the bound controls the entire output function. The scalar observable has the same bound with B 1 B_1 B 1 replaced by ∥ w φ ∥ L 1 ( m ) \lVert w_\varphi\rVert_{L^1(m)} ∥ w φ ∥ L 1 ( m ) . The L 2 ( ζ ; Y ) L^2(\zeta; \mathcal{Y}) L 2 ( ζ ; Y ) variance formula of Corollary 6.10 also applies when ζ \zeta ζ has a finite second input moment.
(ii) Gaussian activation: explicit transform. Let σ = σ G a u s s \sigma=\sigma_{\mathrm{Gauss}} σ = σ Gauss . For ξ ∈ H \xi\in\mathcal{H} ξ ∈ H put v y 2 : = ⟨ Q a y , a y ⟩ v_y^2:=\langle Q a_y,a_y\rangle v y 2 := ⟨ Q a y , a y ⟩ and S y : = Q − ( 1 + v y 2 ) − 1 ( Q a y ) ⊗ ( Q a y ) S_y:=Q-(1+v_y^2)^{-1}(Q a_y)\otimes(Q a_y) S y := Q − ( 1 + v y 2 ) − 1 ( Q a y ) ⊗ ( Q a y ) . Then F φ ∈ D α F_\varphi\in D_\alpha F φ ∈ D α for every α > 0 \alpha>0 α > 0 , with
F Q F φ ( ξ ) = ∫ Ω w φ ( y ) ( 1 + v y 2 ) − 1 / 2 e − ⟨ S y ξ , ξ ⟩ / 2 m ( d y ) , (82) F_QF_\varphi(\xi)=\int_\Omega w_\varphi(y)(1+v_y^2)^{-1/2}e^{-\langle S_y\xi,\xi\rangle/2}\,m(dy), \tag*{(82)} F Q F φ ( ξ ) = ∫ Ω w φ ( y ) ( 1 + v y 2 ) − 1/2 e − ⟨ S y ξ , ξ ⟩ /2 m ( d y ) , (82)
R ρ F φ ( a , b ) = ∫ Ω w φ ( y ) ( 1 + v y 2 ) − 1 / 2 ( ρ ∗ ϕ ⟨ S y a , a ⟩ ) ( − b ) m ( d y ) , (83) R_\rho F_\varphi(a,b)=\int_\Omega w_\varphi(y)(1+v_y^2)^{-1/2}(\rho*\phi_{\langle S_y a,a\rangle})(-b)\,m(dy), \tag*{(83)} R ρ F φ ( a , b ) = ∫ Ω w φ ( y ) ( 1 + v y 2 ) − 1/2 ( ρ ∗ ϕ ⟨ S y a , a ⟩ ) ( − b ) m ( d y ) , (83)
These formulas are mixtures of Gaussian Fourier factors and one-dimensional convolutions, respectively, and S y ≥ ( 1 + ∥ Q ∥ ∥ A ∥ ∞ 2 ) − 1 Q S_y\geq(1+\lVert Q\rVert\lVert A\rVert_\infty^2)^{-1}Q S y ≥ ( 1 + ∥ Q ∥ ∥ A ∥ ∞ 2 ) − 1 Q . Consequently F Q F φ F_QF_\varphi F Q F φ satisfies Definition 4.2 , the reconstruction formulas of Theorem 4.8 hold for F φ F_\varphi F φ , and the ridgelet coefficient R ρ F φ R_\rho F_\varphi R ρ F φ synthesizes, with any real Lipschitz non-polynomial σ ′ \sigma' σ ′ , the target ( ( σ ′ , ρ ) ) α T α F φ ((\sigma',\rho))_\alpha T_\alpha F_\varphi (( σ ′ , ρ ) ) α T α F φ with the finite-width rate of (72) . The same holds for F F F itself as a Y \mathcal{Y} Y -valued target.
(iii) ReLU form. By (79) , the Gaussian-activation layer is also the ReLU network F ( x ) = ∫ Ω × R v y ϕ ′ ′ ( b ) ReLU ( ⟨ a y , x ⟩ − b ) m ( d y ) d b F(x)=\int_{\Omega\times\mathbb{R}}v_y\phi''(b)\operatorname{ReLU}(\langle a_y,x\rangle-b)\,m(dy)\,db F ( x ) = ∫ Ω × R v y ϕ ′′ ( b ) ReLU (⟨ a y , x ⟩ − b ) m ( d y ) d b , whose coefficient measure is finite with all moments finite.
(iv) Non-cylindricity. If A A A has infinite rank, σ = σ G a u s s \sigma=\sigma_{\mathrm{Gauss}} σ = σ Gauss , and w φ > 0 w_\varphi>0 w φ > 0 m m m -almost everywhere, then F φ F_\varphi F φ is not cylindrical.
Proof. (i). Since ∥ a y ∥ ≤ ∥ A ∥ ∞ \lVert a_y\rVert\leq\lVert A\rVert_\infty ∥ a y ∥ ≤ ∥ A ∥ ∞ and σ \sigma σ has polynomial growth, y ↦ v y σ ( ⟨ a y , x ⟩ ) y\mapsto v_y\sigma(\langle a_y,x\rangle) y ↦ v y σ (⟨ a y , x ⟩) is Bochner integrable for each x x x , and the change of variables for the pushforward measure Γ = ι # ( v y m ( d y ) ) \Gamma=\iota_\#(v_y m(dy)) Γ = ι # ( v y m ( d y )) gives S σ [ Γ ] ( x ) = ∫ Ω v y σ ( ⟨ a y , x ⟩ ) m ( d y ) = F ( x ) S_\sigma[\Gamma](x)=\int_\Omega v_y\sigma(\langle a_y,x\rangle)m(dy)=F(x) S σ [ Γ ] ( x ) = ∫ Ω v y σ (⟨ a y , x ⟩) m ( d y ) = F ( x ) . The variation of a pushforward is at most the pushforward of the variation, so ∥ Γ ∥ T V ≤ ∫ ∣ v y ∣ m ( d y ) \lVert\Gamma\rVert_{\mathrm{TV}}\leq\int\lvert v_y\rvert m(dy) ∥ Γ ∥ TV ≤ ∫ ∣ v y ∣ m ( d y ) , and ∫ ( ∥ a ∥ 2 + b 2 ) d ∣ Γ ∣ ≤ ∥ A ∥ ∞ 2 ∥ Γ ∥ T V \int(\lVert a\rVert^2+b^2)d\lvert\Gamma\rvert\leq\lVert A\rVert_\infty^2\lVert\Gamma\rVert_{\mathrm{TV}} ∫ (∥ a ∥ 2 + b 2 ) d ∣ Γ ∣ ≤ ∥ A ∥ ∞ 2 ∥ Γ ∥ TV . The scalar observable has coefficient measure ι # ( w φ m ) \iota_\#(w_\varphi m) ι # ( w φ m ) with total variation ∥ w φ ∥ L 1 ( m ) \lVert w_\varphi\rVert_{L^1(m)} ∥ w φ ∥ L 1 ( m ) , and Theorem 6.5 gives the displayed bound.
(ii). Let ( Z , W ) (Z,W) ( Z , W ) be a centred Gaussian pair with variances v 2 , τ 2 v^2,\tau^2 v 2 , τ 2 and covariance r r r , and assume v > 0 v>0 v > 0 . Given Z Z Z , W W W is Gaussian with mean r Z / v 2 rZ/v^2 r Z / v 2 and variance τ 2 − r 2 / v 2 \tau^2-r^2/v^2 τ 2 − r 2 / v 2 , so E [ e − i W ∣ Z ] = e − i r Z / v 2 e − ( τ 2 − r 2 / v 2 ) / 2 \mathbb{E}[e^{-iW}\mid Z]=e^{-irZ/v^2}e^{-(\tau^2-r^2/v^2)/2} E [ e − iW ∣ Z ] = e − i r Z / v 2 e − ( τ 2 − r 2 / v 2 ) /2 , and
E [ e − Z 2 / 2 e − i r Z / v 2 ] = ∫ R e − z 2 / 2 − i r z / v 2 e − z 2 / ( 2 v 2 ) 2 π v 2 d z = ( 1 + v 2 ) − 1 / 2 exp ( − r 2 2 v 2 ( 1 + v 2 ) ) , \mathbb{E}\left[e^{-Z^2/2}e^{-irZ/v^2}\right]=\int_{\mathbb{R}}e^{-z^2/2-irz/v^2}\frac{e^{-z^2/(2v^2)}}{\sqrt{2\pi v^2}}\,dz=(1+v^2)^{-1/2}\exp\left(-\frac{r^2}{2v^2(1+v^2)}\right), E [ e − Z 2 /2 e − i r Z / v 2 ] = ∫ R e − z 2 /2 − i r z / v 2 2 π v 2 e − z 2 / ( 2 v 2 ) d z = ( 1 + v 2 ) − 1/2 exp ( − 2 v 2 ( 1 + v 2 ) r 2 ) ,
since the tilted density is Gaussian with variance v 2 / ( 1 + v 2 ) v^2/(1+v^2) v 2 / ( 1 + v 2 ) . Multiplying, E [ e − Z 2 / 2 e − i W ] = ( 1 + v 2 ) − 1 / 2 exp ( − τ 2 / 2 + r 2 / ( 2 ( 1 + v 2 ) ) ) \mathbb{E}[e^{-Z^2/2}e^{-iW}]=(1+v^2)^{-1/2}\exp(-\tau^2/2+r^2/(2(1+v^2))) E [ e − Z 2 /2 e − iW ] = ( 1 + v 2 ) − 1/2 exp ( − τ 2 /2 + r 2 / ( 2 ( 1 + v 2 ))) ; the case v = 0 v=0 v = 0 is the limit. Under μ Q \mu_Q μ Q , the pair ( ⟨ a y , x ⟩ , ⟨ x , ξ ⟩ ) (\langle a_y,x\rangle,\langle x,\xi\rangle) (⟨ a y , x ⟩ , ⟨ x , ξ ⟩) is centred Gaussian with v 2 = v y 2 v^2=v_y^2 v 2 = v y 2 , τ 2 = ⟨ Q ξ , ξ ⟩ \tau^2=\langle Q\xi,\xi\rangle τ 2 = ⟨ Q ξ , ξ ⟩ , and r = ⟨ Q a y , ξ ⟩ r=\langle Q a_y,\xi\rangle r = ⟨ Q a y , ξ ⟩ , which gives F Q [ σ G a u s s ( ⟨ a y , ⋅ ⟩ ) ] ( ξ ) = ( 1 + v y 2 ) − 1 / 2 e − ⟨ S y ξ , ξ ⟩ / 2 F_Q[\sigma_{\mathrm{Gauss}}(\langle a_y,\cdot\rangle)](\xi)=(1+v_y^2)^{-1/2}e^{-\langle S_y\xi,\xi\rangle/2} F Q [ σ Gauss (⟨ a y , ⋅ ⟩)] ( ξ ) = ( 1 + v y 2 ) − 1/2 e − ⟨ S y ξ , ξ ⟩ /2 . Since σ G a u s s \sigma_{\mathrm{Gauss}} σ Gauss is bounded and w φ ∈ L 1 ( m ) w_\varphi\in L^1(m) w φ ∈ L 1 ( m ) , Fubini gives the first formula in (83) , and the Fourier slice with the inverse Fourier transform of ω ↦ e − ω 2 ⟨ S y a , a ⟩ / 2 \omega\mapsto e^{-\omega^2\langle S_y a,a\rangle/2} ω ↦ e − ω 2 ⟨ S y a , a ⟩ /2 gives the second. Cauchy–Schwarz for the positive form Q Q Q gives ⟨ Q a y , ξ ⟩ 2 ≤ v y 2 ⟨ Q ξ , ξ ⟩ \langle Q a_y,\xi\rangle^2\leq v_y^2\langle Q\xi,\xi\rangle ⟨ Q a y , ξ ⟩ 2 ≤ v y 2 ⟨ Q ξ , ξ ⟩ , hence ⟨ S y ξ , ξ ⟩ ≥ ( 1 + v y 2 ) − 1 ⟨ Q ξ , ξ ⟩ ≥ ( 1 + ∥ Q ∥ ∥ A ∥ ∞ 2 ) − 1 ⟨ Q ξ , ξ ⟩ \langle S_y\xi,\xi\rangle\geq(1+v_y^2)^{-1}\langle Q\xi,\xi\rangle\geq(1+\lVert Q\rVert\lVert A\rVert_\infty^2)^{-1}\langle Q\xi,\xi\rangle ⟨ S y ξ , ξ ⟩ ≥ ( 1 + v y 2 ) − 1 ⟨ Q ξ , ξ ⟩ ≥ ( 1 + ∥ Q ∥ ∥ A ∥ ∞ 2 ) − 1 ⟨ Q ξ , ξ ⟩ ; in particular F φ ∈ D α F_\varphi\in D_\alpha F φ ∈ D α by Lemma 3.12 . Each g y ( ξ ) : = ( 1 + v y 2 ) − 1 / 2 e − ⟨ S y ξ , ξ ⟩ / 2 g_y(\xi):=(1+v_y^2)^{-1/2}e^{-\langle S_y\xi,\xi\rangle/2} g y ( ξ ) := ( 1 + v y 2 ) − 1/2 e − ⟨ S y ξ , ξ ⟩ /2 is of the form of Lemma 4.4 (a) with ∥ S y ∥ ≤ ∥ Q ∥ \lVert S_y\rVert\leq\lVert Q\rVert ∥ S y ∥ ≤ ∥ Q ∥ and the uniform lower bound S y ≥ θ Q S_y\geq\theta Q S y ≥ θ Q , θ : = ( 1 + ∥ Q ∥ ∥ A ∥ ∞ 2 ) − 1 \theta:=(1+\lVert Q\rVert\lVert A\rVert_\infty^2)^{-1} θ := ( 1 + ∥ Q ∥ ∥ A ∥ ∞ 2 ) − 1 , just proved, and the family is bounded by 1 1 1 because ( 1 + v y 2 ) − 1 / 2 ≤ 1 (1+v_y^2)^{-1/2}\leq1 ( 1 + v y 2 ) − 1/2 ≤ 1 . Fix a compact symmetric neighbourhood of I I I inside R ∖ { 0 } \mathbb{R}\setminus\{0\} R ∖ { 0 } , let U U U be its interior and r ′ > 0 r' > 0 r ′ > 0 the smallest ∣ ω ∣ |\omega| ∣ ω ∣ on it. Since ∂ ω k [ ( 1 + υ y 2 ) − 1 / 2 e − ω 2 ⟨ S y , a , a ⟩ / 2 ] \partial_\omega^k\left[(1+\upsilon_y^2)^{-1/2}e^{-\omega^2\langle S_{y,a},a\rangle/2}\right] ∂ ω k [ ( 1 + υ y 2 ) − 1/2 e − ω 2 ⟨ S y , a , a ⟩ /2 ] is a polynomial in ω \omega ω and ⟨ S y , a , a ⟩ \langle S_{y,a},a\rangle ⟨ S y , a , a ⟩ times e − ω 2 ⟨ S y , a , a ⟩ / 2 e^{-\omega^2\langle S_{y,a},a\rangle/2} e − ω 2 ⟨ S y , a , a ⟩ /2 , and ⟨ S y , a , a ⟩ ≤ ∥ Q ∥ ∥ a ∥ 2 \langle S_{y,a},a\rangle\leq\|Q\|\|a\|^2 ⟨ S y , a , a ⟩ ≤ ∥ Q ∥∥ a ∥ 2 , the proof of part (a) gives
sup ω ∈ U ∣ ∂ ω k g y ( ω a ) ∣ ≤ C k ( 1 + ∥ a ∥ ) 2 k e − r ′ 2 θ ⟨ Q a , a ⟩ / 2 = : h k ( a ) \sup_{\omega\in U}|\partial_\omega^k g_y(\omega a)| \leq C_k(1+\|a\|)^{2k}e^{-r'^2\theta\langle Qa,a\rangle/2}=:h_k(a) ω ∈ U sup ∣ ∂ ω k g y ( ω a ) ∣ ≤ C k ( 1 + ∥ a ∥ ) 2 k e − r ′2 θ ⟨ Q a , a ⟩ /2 =: h k ( a )
for every y y y and every a ∈ H a\in\mathcal{H} a ∈ H , with C k C_k C k independent of y y y , and Lemma 3.12 makes the weights h k h_k h k satisfy the moment bounds of Lemma 4.4(c). Part (c) of that lemma, applied with the finite measure ∣ w φ ∣ m |w_\varphi|m ∣ w φ ∣ m and the family ( w φ / ∣ w φ ∣ ) g y (w_\varphi/|w_\varphi|)g_y ( w φ /∣ w φ ∣ ) g y , which obeys the same bounds because the extra factor has modulus at most one, shows that F Q F φ = ∫ w φ ( y ) g y m ( d y ) F_QF_\varphi=\int w_\varphi(y)g_y\,m(dy) F Q F φ = ∫ w φ ( y ) g y m ( d y ) satisfies Definition 4.2. The remaining claims are Theorem 4.8 and Theorem 6.6, and the Y \mathcal{Y} Y -valued statement is the same computation with Bochner integrals (Theorem 4.11).
(iii). By Lemma 7.3, σ G a u s s ( ⟨ a y , x ⟩ ) = ∫ R ReLU ( ⟨ a y , x ⟩ − b ) φ ′ ′ ( b ) d b \sigma_{\mathrm{Gauss}}(\langle a_y,x\rangle)=\int_{\mathbb{R}}\operatorname{ReLU}(\langle a_y,x\rangle-b)\varphi''(b)\,db σ Gauss (⟨ a y , x ⟩) = ∫ R ReLU (⟨ a y , x ⟩ − b ) φ ′′ ( b ) d b , and Fubini is justified by ∫ ∥ v y ∥ ∫ ( 1 + ∥ a y ∥ ∥ x ∥ + ∣ b ∣ ) ∣ φ ′ ′ ( b ) ∣ d b m ( d y ) < ∞ \int\|v_y\|\int(1+\|a_y\|\|x\|+|b|)|\varphi''(b)|\,db\,m(dy)<\infty ∫ ∥ v y ∥ ∫ ( 1 + ∥ a y ∥∥ x ∥ + ∣ b ∣ ) ∣ φ ′′ ( b ) ∣ d b m ( d y ) < ∞ .
(iv). If F φ = F ~ ∘ L F_\varphi=\widetilde{F}\circ L F φ = F ∘ L with L L L of finite rank, then ker L \ker L ker L has finite codimension while ker A \ker A ker A has infinite codimension, since H / ker A \mathcal{H}/\ker A H / ker A is isomorphic to Ran A \operatorname{Ran} A Ran A ; hence there is x ∈ ker L ∖ ker A x\in\ker L\setminus\ker A x ∈ ker L ∖ ker A , and F φ ( t x ) = F ~ ( 0 ) = F φ ( 0 ) = ∫ w φ d m F_\varphi(tx)=\widetilde{F}(0)=F_\varphi(0)=\int w_\varphi\,dm F φ ( t x ) = F ( 0 ) = F φ ( 0 ) = ∫ w φ d m for all t ∈ R t\in\mathbb{R} t ∈ R . On the other hand σ G a u s s ( t ⟨ a y , x ⟩ ) → 1 { ⟨ a y , x ⟩ = 0 } \sigma_{\mathrm{Gauss}}(t\langle a_y,x\rangle)\to\mathbf{1}_{\{\langle a_y,x\rangle=0\}} σ Gauss ( t ⟨ a y , x ⟩) → 1 {⟨ a y , x ⟩ = 0 } as t → ∞ t\to\infty t → ∞ , boundedly, so F φ ( t x ) → ∫ w φ 1 { A x = 0 } d m F_\varphi(tx)\to\int w_\varphi\mathbf{1}_{\{Ax=0\}}\,dm F φ ( t x ) → ∫ w φ 1 { A x = 0 } d m , which is strictly smaller than ∫ w φ d m \int w_\varphi\,dm ∫ w φ d m because A x ≠ 0 Ax\neq0 A x = 0 on a set of positive measure and w φ > 0 w_\varphi>0 w φ > 0 almost everywhere. □ \square □
Remark 7.7. (Equivariance of the transform). If U U U is a unitary operator on H \mathcal{H} H with U Q U ∗ = Q UQU^*=Q U Q U ∗ = Q , then U # μ Q = μ Q U_{\#}\mu_Q=\mu_Q U # μ Q = μ Q and, for f ∈ L 1 ( μ Q ) f\in L^1(\mu_Q) f ∈ L 1 ( μ Q ) , R ρ [ f ∘ U ] ( a , b ) = R ρ f ( U a , b ) R_\rho[f\circ U](a,b)=R_\rho f(Ua,b) R ρ [ f ∘ U ] ( a , b ) = R ρ f ( U a , b ) ; if also U P U ∗ = P UPU^*=P U P U ∗ = P , then ν α \nu_\alpha ν α and λ α \lambda_\alpha λ α are U U U -invariant and the Plancherel identity, the frame operator, and the reconstruction formulas commute with the action of U U U . This is the infinite-dimensional analogue of the group equivariance of ridgelet transforms studied in Sonoda et al. [88 ] . On R d \mathbb{R}^d R d , unlike on T d \mathbb{T}^d T d , no translation-invariant Gaussian measure with trace-class covariance exists, so a Gaussian input weight on L 2 ( R d ) L^2(\mathbb{R}^d) L 2 ( R d ) necessarily breaks translation invariance; the network structure (i) of Example 7.6 does not involve the input weight and is unaffected.
Example 7.8. (Periodic convolution layer). Let Ω = T d = ( R / 2 π Z ) d \Omega=\mathbb{T}^d=(\mathbb{R}/2\pi\mathbb{Z})^d Ω = T d = ( R /2 π Z ) d with normalized Lebesgue measure, H = L 2 ( T d ; R ) \mathcal{H}=L^2(\mathbb{T}^d;\mathbb{R}) H = L 2 ( T d ; R ) , Y = L 2 ( T d ) \mathcal{Y}=L^2(\mathbb{T}^d) Y = L 2 ( T d ) , and let k , ψ ∈ L 2 ( T d ; R ) k,\psi\in L^2(\mathbb{T}^d;\mathbb{R}) k , ψ ∈ L 2 ( T d ; R ) . For a kernel u u u , its Fourier coefficients are u ^ ( n ) : = ∫ T d u ( y ) e − i ⟨ n , y ⟩ d y \widehat{u}(n):=\int_{\mathbb{T}^d}u(y)e^{-i\langle n,y\rangle}\,dy u ( n ) := ∫ T d u ( y ) e − i ⟨ n , y ⟩ d y , using normalized Haar measure. With a y : = k ( y − ⋅ ) a_y:=k(y-\cdot) a y := k ( y − ⋅ ) and v y : = ψ ( ⋅ − y ) v_y:=\psi(\cdot-y) v y := ψ ( ⋅ − y ) , the layer (80) is F ( x ) = ψ ∗ σ ( k ∗ x ) F(x)=\psi*\sigma(k*x) F ( x ) = ψ ∗ σ ( k ∗ x ) , a convolutional layer followed by a smoothing convolution. Here ∥ A ∥ ∞ = ∥ k ∥ 2 \|A\|_\infty=\|k\|_2 ∥ A ∥ ∞ = ∥ k ∥ 2 , ∫ ∥ v y ∥ d y = ∥ ψ ∥ 2 \int\|v_y\|\,dy=\|\psi\|_2 ∫ ∥ v y ∥ d y = ∥ ψ ∥ 2 , and F F F commutes with all translations of T d \mathbb{T}^d T d , and with every isometry of T d \mathbb{T}^d T d that fixes k k k and ψ \psi ψ . If k ^ ( n ) ≠ 0 \widehat{k}(n)\neq0 k ( n ) = 0 for infinitely many n ∈ Z d n\in\mathbb{Z}^d n ∈ Z d , then A A A has infinite rank. If additionally σ = σ G a u s s \sigma=\sigma_{\mathrm{Gauss}} σ = σ Gauss , then with φ ≡ 1 \varphi\equiv1 φ ≡ 1 and ψ ^ ( 0 ) ≠ 0 \widehat{\psi}(0)\neq0 ψ ( 0 ) = 0 the observable F 1 ( x ) = ψ ^ ( 0 ) ∫ T d σ ( ( k ∗ x ) ( y ) ) d y F_1(x)=\widehat{\psi}(0)\int_{\mathbb{T}^d}\sigma((k*x)(y))\,dy F 1 ( x ) = ψ ( 0 ) ∫ T d σ (( k ∗ x ) ( y )) d y is not cylindrical. Choosing Q = P = ( I − Δ ) − s Q=P=(I-\Delta)^{-s} Q = P = ( I − Δ ) − s with s > d / 2 s>d/2 s > d /2 , which is injective, trace class, and translation invariant, the transform is equivariant as well: R ρ [ f ∘ τ z ] ( a , b ) = R ρ f ( τ z a , b ) R_\rho[f\circ\tau_z](a,b)=R_\rho f(\tau_z a,b) R ρ [ f ∘ τ z ] ( a , b ) = R ρ f ( τ z a , b ) for every translation τ z \tau_z τ z (Remark 7.7).
Proof. With normalized Haar measure, ∥ a y ∥ = ∥ k ∥ 2 \|a_y\|=\|k\|_2 ∥ a y ∥ = ∥ k ∥ 2 and ∥ v y ∥ = ∥ ψ ∥ 2 \|v_y\|=\|\psi\|_2 ∥ v y ∥ = ∥ ψ ∥ 2 for every y y y , and y ↦ a y y\mapsto a_y y ↦ a y , y ↦ v y y\mapsto v_y y ↦ v y are continuous into L 2 ( T d ) L^2(\mathbb{T}^d) L 2 ( T d ) . Convolutions commute with translations and with isometries fixing their kernels, and σ \sigma σ acts pointwise, so F F F is equivariant. In the Fourier basis A A A is diagonal with entries k ^ ( n ) \widehat{k}(n) k ( n ) , so its rank is the number of nonzero coefficients; with φ ≡ 1 \varphi\equiv1 φ ≡ 1 , w 1 ( y ) = ∫ ψ = ψ ^ ( 0 ) w_1(y)=\int\psi=\widehat{\psi}(0) w 1 ( y ) = ∫ ψ = ψ ( 0 ) is a nonzero constant. For the Gaussian activation, Example 7.6(iv) applies after reversing the sign of the observable if this constant is negative. The operator ( I − Δ ) − s (I-\Delta)^{-s} ( I − Δ ) − s has eigenvalues ( 1 + ∣ m ∣ 2 ) − s (1+|m|^2)^{-s} ( 1 + ∣ m ∣ 2 ) − s , summable for s > d / 2 s>d/2 s > d /2 , is injective, and commutes with translations, so Remark 7.7 applies. □ \square □
Example 7.9 (Dirichlet solution operator with a pointwise nonlinearity). Let Ω = ( 0 , 1 ) \Omega=(0,1) Ω = ( 0 , 1 ) with Lebesgue measure, H = Y = L 2 ( 0 , 1 ) \mathcal{H}=\mathcal{Y}=L^2(0,1) H = Y = L 2 ( 0 , 1 ) , and let L D − 1 : = ( I − ∂ t 2 ) − 1 L_D^{-1}:=(I-\partial_t^2)^{-1} L D − 1 := ( I − ∂ t 2 ) − 1 with Dirichlet boundary conditions,
( L D − 1 x ) ( y ) = ∫ 0 1 g ( y , t ) x ( t ) d t , g ( y , t ) = sinh ( min ( y , t ) ) sinh ( 1 − max ( y , t ) ) sinh 1 , (L_D^{-1}x)(y)=\int_0^1 g(y,t)x(t)\,\mathrm{d}t,\qquad g(y,t)=\frac{\sinh(\min(y,t))\sinh(1-\max(y,t))}{\sinh1}, ( L D − 1 x ) ( y ) = ∫ 0 1 g ( y , t ) x ( t ) d t , g ( y , t ) = sinh 1 sinh ( min ( y , t )) sinh ( 1 − max ( y , t )) , so that u = L D − 1 x u=L_D^{-1}x u = L D − 1 x solves − u ′ ′ + u = x -u^{\prime\prime}+u=x − u ′′ + u = x , u ( 0 ) = u ( 1 ) = 0 u(0)=u(1)=0 u ( 0 ) = u ( 1 ) = 0 . With a y = v y = g ( y , ⋅ ) a_y=v_y=g(y,\cdot) a y = v y = g ( y , ⋅ ) , the layer (80) is F ( x ) = L D − 1 σ ( L D − 1 x ) F(x)=L_D^{-1}\sigma(L_D^{-1}x) F ( x ) = L D − 1 σ ( L D − 1 x ) : the solution of the linear problem whose source is a pointwise nonlinearity of the solution with source x x x , which is the nonlinear correction in the Picard update u 1 = L D − 1 x + F ( x ) u_1=L_D^{-1}x+F(x) u 1 = L D − 1 x + F ( x ) starting from u 0 = L D − 1 x u_0=L_D^{-1}x u 0 = L D − 1 x for − u ′ ′ + u = σ ( u ) + x -u^{\prime\prime}+u=\sigma(u)+x − u ′′ + u = σ ( u ) + x . The operator L D − 1 L_D^{-1} L D − 1 has eigenvalues λ n = ( 1 + π 2 n 2 ) − 1 \lambda_n=(1+\pi^2n^2)^{-1} λ n = ( 1 + π 2 n 2 ) − 1 with eigenfunctions e n ( t ) = 2 sin ( n π t ) e_n(t)=\sqrt{2}\sin(n\pi t) e n ( t ) = 2 sin ( nπ t ) , so it is injective, trace class, and of infinite rank, and ∥ A ∥ ∞ ≤ sup y ∥ g ( y , ⋅ ) ∥ 2 < ∞ \lVert A\rVert_\infty\leq\sup_y\lVert g(y,\cdot)\rVert_2<\infty ∥ A ∥ ∞ ≤ sup y ∥ g ( y , ⋅ ) ∥ 2 < ∞ . Example 7.6 applies, and with Q = P = L D − 1 Q=P=L_D^{-1} Q = P = L D − 1 the input measure is a Gaussian field with covariance equal to the inverse Dirichlet operator. Applying L D − 1 L_D^{-1} L D − 1 to white noise instead would give covariance L D − 2 L_D^{-2} L D − 2 . The linear operator L D − 1 L_D^{-1} L D − 1 itself is an exact ReLU network,
L D − 1 x = ∑ n λ n e n [ ReLU ( ⟨ e n , x ⟩ ) − ReLU ( − ⟨ e n , x ⟩ ) ] , L_D^{-1}x=\sum_n\lambda_ne_n[\operatorname{ReLU}(\langle e_n,x\rangle)-\operatorname{ReLU}(-\langle e_n,x\rangle)], L D − 1 x = n ∑ λ n e n [ ReLU (⟨ e n , x ⟩) − ReLU ( − ⟨ e n , x ⟩)] , whose 2 N 2N 2 N -neuron truncation has error at most λ N + 1 ∥ x ∥ = O ( N − 2 ) \lambda_{N+1}\lVert x\rVert=O(N^{-2}) λ N + 1 ∥ x ∥ = O ( N − 2 ) uniformly on bounded sets. This is the linear spectral-truncation estimate. For the nonlinear layer, (81) supplies a separate uniform approximation estimate for the full output.
Proof. Direct differentiation shows that g ( y , ⋅ ) g(y,\cdot) g ( y , ⋅ ) solves the boundary value problem with a unit jump of the derivative at t = y t=y t = y , so L D − 1 L_D^{-1} L D − 1 is the integral operator with kernel g g g ; the eigenpairs are checked by − e n ′ ′ + e n = ( 1 + π 2 n 2 ) e n -e_n^{\prime\prime}+e_n=(1+\pi^2n^2)e_n − e n ′′ + e n = ( 1 + π 2 n 2 ) e n . The kernel is continuous and bounded by sinh 2 ( 1 / 2 ) / sinh 1 \sinh^2(1/2)/\sinh1 sinh 2 ( 1/2 ) / sinh 1 , so y ↦ g ( y , ⋅ ) y\mapsto g(y,\cdot) y ↦ g ( y , ⋅ ) is continuous and bounded into L 2 ( 0 , 1 ) L^2(0,1) L 2 ( 0 , 1 ) , and m = d y m=\mathrm{d}y m = d y is finite; hence Example 7.6 applies. The ReLU expansion is the spectral theorem with u = ReLU ( u ) − ReLU ( − u ) u=\operatorname{ReLU}(u)-\operatorname{ReLU}(-u) u = ReLU ( u ) − ReLU ( − u ) , and orthogonality gives
∥ L D − 1 x − ( L D − 1 ) N x ∥ 2 = ∑ n > N λ n 2 ⟨ e n , x ⟩ 2 ≤ λ N + 1 2 ∥ x ∥ 2 . \lVert L_D^{-1}x-(L_D^{-1})_Nx\rVert^2=\sum_{n>N}\lambda_n^2\langle e_n,x\rangle^2\leq\lambda_{N+1}^2\lVert x\rVert^2. ∥ L D − 1 x − ( L D − 1 ) N x ∥ 2 = n > N ∑ λ n 2 ⟨ e n , x ⟩ 2 ≤ λ N + 1 2 ∥ x ∥ 2 .
Numerical experiments
We compare the observed finite-width approximation errors with the predicted rates and check an explicit transform formula numerically. Varying the input resolution distinguishes approximation in network width from errors caused by numerical discretization. The three experiments illustrate Theorems 6.6 and 6.8 and Example 7.6 on ℓ 2 \ell^2 ℓ 2 or L 2 ( 0 , 1 ) L^2(0,1) L 2 ( 0 , 1 ) . For computation the input is truncated to d d d coordinates, with several values of d d d . These finite computations do not establish a uniform-in-dimension theorem. Errors are suprema on finite test sets, averaged over independent trials; fitted slopes use logarithmic coordinates and widths N = 4 , 8 , … , 4096 N=4,8,\ldots,4096 N = 4 , 8 , … , 4096 . Appendix E gives the parameter-drawing procedures, quadrature, and reproduction details.
Experiment 1: ReLU network with Gaussian random directions on ℓ 2 \ell^2 ℓ 2 . This is Corollary 7.5 . Let Q = diag ( j − 2 ) Q=\operatorname{diag}(j^{-2}) Q = diag ( j − 2 ) on ℓ 2 \ell^2 ℓ 2 , truncated to d ∈ { 10 , 10 2 , 10 3 , 10 4 } d\in\{10,10^2,10^3,10^4\} d ∈ { 10 , 1 0 2 , 1 0 3 , 1 0 4 } coordinates, and let K K K consist of 200 points with coordinates x j = y j / j x_j=y_j/j x j = y j / j , y j y_j y j uniform on [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] . The target f R e L U , Q ( x ) = ⟨ Q x , x ⟩ / 2 π f_{\mathrm{ReLU},Q}(x)=\sqrt{\langle Qx,x\rangle/2\pi} f ReLU , Q ( x ) = ⟨ Q x , x ⟩ /2 π is compared with f R e L U , Q , N ( x ) = N − 1 ∑ j ≤ N ReLU ( ⟨ a j , x ⟩ ) f_{\mathrm{ReLU},Q,N}(x)=N^{-1}\sum_{j\leq N}\operatorname{ReLU}(\langle a_j,x\rangle) f ReLU , Q , N ( x ) = N − 1 ∑ j ≤ N ReLU (⟨ a j , x ⟩) , a j ∼ μ Q a_j\sim\mu_Q a j ∼ μ Q , over 20 trials. Table 1 and figure 1 (upper left) show that the mean supremum error decays like N − 1 / 2 N^{-1/2} N − 1/2 and is similar across the tested d d d ; the bound of Corollary 7.5 , which equals 0.24 0.24 0.24 at N = 4096 N=4096 N = 4096 , is about twenty times larger than the observed error. Thus the experiment checks the rate rather than sharpness of the bound.
Experiment 2: the ridgelet coefficient of f W f_W f W synthesizes C T α f W CT_\alpha f_W C T α f W . This is Example 7.2 (iii) with α = 1 \alpha=1 α = 1 , P = Q = diag ( j − 2 ) P=Q=\operatorname{diag}(j^{-2}) P = Q = diag ( j − 2 ) , W = I W=I W = I , and the analysis filter ρ ( t ) = He 4 ( t ) e − t 2 / 2 \rho(t)=\operatorname{He}_4(t)e^{-t^2/2} ρ ( t ) = He 4 ( t ) e − t 2 /2 , whose Fourier transform is 2 π ω 4 e − ω 2 / 2 \sqrt{2\pi}\,\omega^4e^{-\omega^2/2} 2 π ω 4 e − ω 2 /2 . This filter has four vanishing moments and admissibility constant ( ( ρ , ρ ) ) 1 = 6 ((\rho,\rho))_1 = 6 (( ρ , ρ ) ) 1 = 6 ; it is not band pass, but Theorem 4.5 (ii) uses only admissibility, and the finiteness of the total variation and of the second moment of γ g = R ρ f W \gamma_g = R_\rho f_W γ g = R ρ f W is verified directly from ∫ R ∣ γ g ( a , b ) ∣ d b = D − 1 / 2 2 π m 4 ( 1 + κ W ( a ) ) − 2 \int_{\mathbb{R}} |\gamma_g(a,b)|\,\mathrm{d}b = D^{-1/2}\sqrt{2\pi}\,m_4\left(1+\kappa_W(a)\right)^{-2} ∫ R ∣ γ g ( a , b ) ∣ d b = D − 1/2 2 π m 4 ( 1 + κ W ( a ) ) − 2 , m 4 = E ∣ He 4 ( Z ) ∣ m_4 = \mathbb{E}|\operatorname{He}_4(Z)| m 4 = E ∣ He 4 ( Z ) ∣ , and Lemma 3.12 . The coefficient is explicit, γ g ( a , b ) = D − 1 / 2 2 π ϕ u ( 4 ) ( b ) \gamma_g(a,b) = D^{-1/2}\sqrt{2\pi}\,\phi_u^{(4)}(b) γ g ( a , b ) = D − 1/2 2 π ϕ u ( 4 ) ( b ) with u = 1 + κ W ( a ) u = 1+\kappa_W(a) u = 1 + κ W ( a ) , and the target ( ( ρ , ρ ) ) 1 f g ((\rho,\rho))_1 f_g (( ρ , ρ ) ) 1 f g is computed from (77) by quadrature. Parameters are drawn by rejection sampling for the bias and importance resampling for the direction; these numerical procedures approximate the coefficient law ∣ γ g ∣ λ 1 / V |\gamma_g|\lambda_1/V ∣ γ g ∣ λ 1 / V , as detailed in Appendix E. Over 10 trials, with d ∈ { 10 , 10 2 , 10 3 } d \in\{10,10^2,10^3\} d ∈ { 10 , 1 0 2 , 1 0 3 } and K K K consisting of 100 points drawn by the rule of Experiment 1, the mean supremum error of the sampled network f N = V N ∑ j sgn γ g ( θ j ) ρ ( ⟨ a j , ⋅ ⟩ − b j ) f_N = \frac{V}{N}\sum_j \operatorname{sgn}\gamma_g(\theta_j)\rho(\langle a_j,\cdot\rangle-b_j) f N = N V ∑ j sgn γ g ( θ j ) ρ (⟨ a j , ⋅ ⟩ − b j ) against 6 f g 6f_g 6 f g , whose supremum on K K K is about 12.5, decays approximately like N − 1 / 2 N^{-1/2} N − 1/2 across the tested dimensions (table 1, figure 1(upper right)).
Experiment 3: the Dirichlet operator layer on L 2 ( 0 , 1 ) L^2(0,1) L 2 ( 0 , 1 ) . This is Example 7.9 with the Gaussian activation σ G a u s s \sigma_{\mathrm{Gauss}} σ Gauss and the observable φ ≡ 1 \varphi\equiv1 φ ≡ 1 , for which w ( y ) = ∫ 0 1 g ( y , t ) d t = 1 − cosh ( y − 1 2 ) / cosh 1 2 > 0 w(y)=\int_0^1 g(y,t)\,\mathrm{d}t=1-\cosh\left(y-\frac{1}{2}\right)/\cosh\frac{1}{2}>0 w ( y ) = ∫ 0 1 g ( y , t ) d t = 1 − cosh ( y − 2 1 ) / cosh 2 1 > 0 and V = ∥ w ∥ L 1 = 1 − 2 tanh 1 2 ≈ 0.0758 V=\|w\|_{L^1}=1-2\tanh\frac{1}{2}\approx0.0758 V = ∥ w ∥ L 1 = 1 − 2 tanh 2 1 ≈ 0.0758 . The interval is discretized by d ∈ { 64 , 512 , 4096 } d\in\{64,512,4096\} d ∈ { 64 , 512 , 4096 } midpoints, and the test set consists of 100 inputs x = ∑ n ≤ 30 ( u n / n ) e n x=\sum_{n\le30}(u_n/n)e_n x = ∑ n ≤ 30 ( u n / n ) e n with u n u_n u n uniform in [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] . (a) The sampled network f N ( x ) = V N ∑ j σ G a u s s ( ⟨ g ( y j , ⋅ ) , x ⟩ ) f_N(x)=\frac{V}{N}\sum_j\sigma_{\mathrm{Gauss}}(\langle g(y_j,\cdot),x\rangle) f N ( x ) = N V ∑ j σ Gauss (⟨ g ( y j , ⋅ ) , x ⟩) , y j ∼ w / V y_j\sim w/V y j ∼ w / V , is compared with F 1 ( x ) = ∫ 0 1 w ( y ) σ G a u s s ( ( L D − 1 x ) ( y ) ) d y F_1(x)=\int_0^1w(y)\sigma_{\mathrm{Gauss}}((L_D^{-1}x)(y))\,\mathrm{d}y F 1 ( x ) = ∫ 0 1 w ( y ) σ Gauss (( L D − 1 x ) ( y )) d y over 20 trials; the error decays like N − 1 / 2 N^{-1/2} N − 1/2 for d = 512 d=512 d = 512 and d = 4096 d=4096 d = 4096 , while for d = 64 d=64 d = 64 it levels off near 10 − 5 10^{-5} 1 0 − 5 , which is the quadrature error of the 64-point reference value. (b) The closed-form transform (83) , with the filter of Experiment 2, is compared at a = 2 sin ( π t ) + 1 2 2 sin ( 3 π t ) a=\sqrt{2}\sin(\pi t)+\frac{1}{2}\sqrt{2}\sin(3\pi t) a = 2 sin ( π t ) + 2 1 2 sin ( 3 π t ) and b ∈ { 0 , − 1 2 , − 1 } b\in\{0,-\frac{1}{2},-1\} b ∈ { 0 , − 2 1 , − 1 } with a Monte Carlo estimate of ∫ F 1 ( x ) ρ ( ⟨ a , x ⟩ − b ) μ L D ( d x ) \int F_1(x)\rho(\langle a,x\rangle-b)\,\mu_{L_D}(\mathrm{d}x) ∫ F 1 ( x ) ρ (⟨ a , x ⟩ − b ) μ L D ( d x ) using 2 × 10 5 2\times10^5 2 × 1 0 5 Gaussian samples and d ′ ∈ { 8 , 64 , 512 } d'\in\{8,64,512\} d ′ ∈ { 8 , 64 , 512 } modes of the Karhunen–Loève expansion; the discretization is specified in Appendix E. The closed-form values are ( 0.18119 , 0.09062 , − 0.06298 ) (0.18119, 0.09062, -0.06298) ( 0.18119 , 0.09062 , − 0.06298 ) ; the Monte Carlo estimates are ( 0.18111 , 0.09107 , − 0.06258 ) (0.18111, 0.09107, -0.06258) ( 0.18111 , 0.09107 , − 0.06258 ) for d ′ = 8 d'=8 d ′ = 8 , ( 0.18106 , 0.09041 , − 0.06303 ) (0.18106, 0.09041, -0.06303) ( 0.18106 , 0.09041 , − 0.06303 ) for d ′ = 64 d'=64 d ′ = 64 , and ( 0.18077 , 0.09072 , − 0.06258 ) (0.18077, 0.09072, -0.06258) ( 0.18077 , 0.09072 , − 0.06258 ) for d ′ = 512 d'=512 d ′ = 512 , with standard errors ( 0.00013 , 0.00023 , 0.00018 ) (0.00013, 0.00023, 0.00018) ( 0.00013 , 0.00023 , 0.00018 ) : all agree within about three standard errors.
The experiments illustrate the approximation estimates rather than the computational cost, which grows with d d d through the inner products. Across the tested resolutions, the errors and slopes are similar once the discretization error is small enough. Finite test sets, quadrature, and importance resampling limit what these computations alone establish.
Discussion
We discuss what the analysis, representation, and approximation results establish and identify the remaining limitations. These include the preliminary approximation of a general continuous target, the numerical representation of directions, and extensions of the measure construction.
Main results. With a Gaussian input weight and a homogeneous Gaussian mixture on the direction parameter, the ridgelet transform on an infinite-dimensional Hilbert space keeps the structure of the finite-dimensional theory. The admissibility constant is a one-dimensional integral in which no dimension appears, the transform is injective with a Plancherel identity on E α E_\alpha E α , and synthesis returns the target up to the frame operator T α T_\alpha T α , the correction induced by the weights, which is undone after synthesis, before analysis, or on the coefficients. Targets with a spectral density are integral networks with explicit coefficients; every continuous target is a compact-open limit of such networks; and sampling the coefficient gives finite-width networks at the rate N − 1 / 2 N^{-1/2} N − 1/2 when the density satisfies Definition 4.2 and the activation is globally Lipschitz. Weak reconstruction covers every non-polynomial tempered activation, and ReLU, tanh, the Gaussian distribution function, and the Gaussian activation satisfy all of these conditions. The neural-operator layers of the examples are integral networks with explicit parameter measures to which the finite-width bounds apply.
Approximation rates and numerical implementation. The Sobolev criterion of Theorem 5.10 permits non-band-pass filters and a finite range of coefficient moments. For s > 5 / 2 s > 5/2 s > 5/2 , it connects to the Hilbert-valued uniform approximation theorem just as the smooth criterion does. The experiments support the predicted approximation rates on finite test sets and check an explicit transform independently; they also expose a discretization floor. They do not remove the need to account for numerical integration and finite representations of directions in an implementation.
Position among related results. The universality statement is of a type already available as an existence theorem: compact-open density of networks with scalar ridge neurons holds on locally convex spaces for every continuous non-polynomial activation [46 ] , and global density on weighted spaces is proved in Cuchiero et al. [23 ] . What the ridgelet transform adds is the construction of the approximants: an explicit coefficient for every target with a spectral density and every admissible analysis filter, a Plancherel identity with an identified frame operator, and a sampling argument that yields the finite-width rate and applies to the examples. The rate is a Barron-type statement for a restricted class, consistent with the lower bounds for Lipschitz operator classes [2 , 62 , 63 ] . The transform differs from the ridgelet transform for Wiener functionals of Martias [68 ] in its atoms, its parameter measure, and its Plancherel identity on a single space; Appendix § 15 records these comparisons.
Open problems. Natural next steps are quantitative control of the spectral approximation for regular target classes, the cost of representing infinite-dimensional directions in a finite computation, and extensions to operator activations and to input measures beyond the Gaussian construction.
A kernel interpretation of the frame operator
This appendix gives a regularized characteristic functional and an optional kernel interpretation. Neither calculation is a prerequisite of the main proofs.
The regularized characteristic functional of the mixture
The following calculation is used only in Remark A.2.
Lemma A.1 (Truncation of the scale integral). For z ≠ 0 z \ne0 z = 0 and α > 0 \alpha> 0 α > 0 , put q = ⟨ P z , z ⟩ > 0 q = \langle Pz,z\rangle> 0 q = ⟨ P z , z ⟩ > 0 and ν α ε , M : = ∫ ε M μ 2 s P s α / 2 − 1 d s \nu_{\alpha}^{\varepsilon,M} := \int_{\varepsilon}^{M} \mu_{2sP}s^{\alpha/2-1}\,\mathrm{d}s ν α ε , M := ∫ ε M μ 2 s P s α /2 − 1 d s for 0 < ε < M < ∞ 0 < \varepsilon< M < \infty 0 < ε < M < ∞ . Then
lim ε ↓ 0 , M ↑ ∞ ∫ H e i ⟨ z , ξ ⟩ ν α ε , M ( d ξ ) = ∫ 0 ∞ e − s q s α / 2 − 1 d s = Γ ( α / 2 ) q − α / 2 . \lim_{\varepsilon\downarrow0,\ M\uparrow\infty}\int_{\mathcal{H}}e^{i\langle z,\xi\rangle}\nu_{\alpha}^{\varepsilon,M}(\mathrm{d}\xi)=\int_{0}^{\infty}e^{-sq}s^{\alpha/2-1}\,\mathrm{d}s=\Gamma(\alpha/2)q^{-\alpha/2}. ε ↓ 0 , M ↑ ∞ lim ∫ H e i ⟨ z , ξ ⟩ ν α ε , M ( d ξ ) = ∫ 0 ∞ e − s q s α /2 − 1 d s = Γ ( α /2 ) q − α /2 .
In contrast , ∫ H e i ⟨ z , ξ ⟩ ν α ( d ξ ) = ∞ \int_{\mathcal{H}}e^{i\langle z,\xi\rangle}\nu_{\alpha}(\mathrm{d}\xi)=\infty ∫ H e i ⟨ z , ξ ⟩ ν α ( d ξ ) = ∞ , so the limit is not a Lebesgue integral against ν α \nu_{\alpha} ν α .
Proof. The truncated mixture is finite, so Fubini and the characteristic functional of μ 2 s P \mu_{2sP} μ 2 s P give ∫ H e i ⟨ z , ξ ⟩ ν α ε , M ( d ξ ) = ∫ ε M e − s q s α / 2 − 1 d s \int_{\mathcal{H}}e^{i\langle z,\xi\rangle}\nu_{\alpha}^{\varepsilon,M}(\mathrm{d}\xi)=\int_{\varepsilon}^{M}e^{-sq}s^{\alpha/2-1}\,\mathrm{d}s ∫ H e i ⟨ z , ξ ⟩ ν α ε , M ( d ξ ) = ∫ ε M e − s q s α /2 − 1 d s . Monotone convergence and the substitution u = s q u=sq u = s q finish the proof.
Remark A.2 (A Riesz-type kernel for the frame operator). The regularized characteristic functional of ν α \nu_\alpha ν α is Γ ( α / 2 ) ⟨ P z , z ⟩ − α / 2 \Gamma(\alpha/2)\langle Pz,z\rangle^{-\alpha/2} Γ ( α /2 ) ⟨ P z , z ⟩ − α /2 (Lemma A.1 ). Interchanging integrals formally in (iii) gives
T α f ( x ) ‘ ‘ = ‘ ‘ Γ ( α / 2 ) ∫ H f ( y ) ⟨ P ( x − y ) , x − y ⟩ − α / 2 μ Q ( d y ) , T_\alpha f(x) \mathrel{``=``} \Gamma(\alpha/2)\int_{\mathcal{H}} f(y)\langle P(x-y),x-y\rangle^{-\alpha/2}\,\mu_Q(dy), T α f ( x ) ‘‘ = ‘‘ Γ ( α /2 ) ∫ H f ( y ) ⟨ P ( x − y ) , x − y ⟩ − α /2 μ Q ( d y ) ,
a formal integral operator with the Riesz-type kernel ⟨ P ( x − y ) , x − y ⟩ − α / 2 \langle P(x-y),x-y\rangle^{-\alpha/2} ⟨ P ( x − y ) , x − y ⟩ − α /2 . On R m \mathbb{R}^m R m with P = I P=I P = I and 0 < α < m 0<\alpha<m 0 < α < m , the kernel gives a Riesz potential of order m − α m-\alpha m − α , namely k m , α ( − Δ ) − ( m − α ) / 2 ( f p Q ) k_{m,\alpha}(-\Delta)^{-(m-\alpha)/2}(f p_Q) k m , α ( − Δ ) − ( m − α ) /2 ( f p Q ) , in agreement with Corollary D.2 . The interchange is not justified for an infinite measure and an oscillatory integrand, and the identity is not used.
Coefficient projections and minimum-norm solutions
This appendix derives additional range and minimum-norm formulas from the coefficient adjoint and the reconstruction theorem proved in the main text.
Coefficient projection
Let C = L 2 ( H × R , λ α ) \mathcal{C}=L^2(\mathcal{H}\times\mathbb{R},\lambda_\alpha) C = L 2 ( H × R , λ α ) , normed through the partial Fourier transform in the bias, ∥ γ ∥ C 2 = 1 2 π ∫ ∣ γ ^ ( a , ω ) ∣ 2 ν α ( d a ) d ω \|\gamma\|_{\mathcal{C}}^2=\frac{1}{2\pi}\int|\widehat{\gamma}(a,\omega)|^2\nu_\alpha(da)d\omega ∥ γ ∥ C 2 = 2 π 1 ∫ ∣ γ ( a , ω ) ∣ 2 ν α ( d a ) d ω , and let ρ \rho ρ be α \alpha α -admissible with C = ( ( ρ , ρ ) ) α \mathcal{C}=((\rho,\rho))_\alpha C = (( ρ , ρ ) ) α .
Proposition B.1 (Bounded backprojection and orthogonal range projection). The Hilbert adjoint W ρ ∗ W_\rho^* W ρ ∗ of W ρ : L 2 ( ν α ) → C W_\rho:L^2(\nu_\alpha)\to\mathcal{C} W ρ : L 2 ( ν α ) → C , called backprojection, is given by (18): the integral there converges absolutely for ν α \nu_\alpha ν α -almost every ξ \xi ξ , is independent, as an L 2 L^2 L 2 class, of the jointly measurable Fourier representative of γ \gamma γ , and satisfies ∥ W ρ ∗ γ ∥ L 2 ( ν α ) ≤ C ∥ γ ∥ C \|W_\rho^*\gamma\|_{L^2(\nu_\alpha)}\le\sqrt{\mathcal{C}}\|\gamma\|_{\mathcal{C}} ∥ W ρ ∗ γ ∥ L 2 ( ν α ) ≤ C ∥ γ ∥ C and W ρ ∗ W ρ = C Id W_\rho^*W_\rho=\mathcal{C}\operatorname{Id} W ρ ∗ W ρ = C Id . With P K α P_{\mathcal{K}_\alpha} P K α the orthogonal projection onto (22),
Π ρ = C − 1 W ρ P K α W ρ ∗ (84) \Pi_\rho=\mathcal{C}^{-1}W_\rho P_{\mathcal{K}_\alpha}W_\rho^* \tag*{(84)} Π ρ = C − 1 W ρ P K α W ρ ∗ (84)
is the orthogonal projection onto Ran R ρ \operatorname{Ran}R_\rho Ran R ρ . The minimum-norm solution of S ρ γ = F ∈ E α ′ S_\rho\gamma=F\in\mathcal{E}'_\alpha S ρ γ = F ∈ E α ′ is C − 1 R ρ T α − 1 F \mathcal{C}^{-1}R_\rho T_\alpha^{-1}F C − 1 R ρ T α − 1 F , and all solutions differ from it by an element of ( Ran R ρ ) ⊥ (\operatorname{Ran}R_\rho)^\perp ( Ran R ρ ) ⊥ .
Proof. The adjoint formula, representative independence and norm bound are proved in Lemma 3.9 . Since R ρ = W ρ F Q R_\rho=W_\rho F_Q R ρ = W ρ F Q and F Q F_Q F Q is unitary onto K α \mathcal{K}_\alpha K α , the operator C − 1 / 2 W ρ ∣ K α \mathcal{C}^{-1/2}W_\rho|_{\mathcal{K}_\alpha} C − 1/2 W ρ ∣ K α is an isometry with closed image; its product with its adjoint is (84). Finally R ρ ′ R ρ = C T α R_\rho'R_\rho=\mathcal{C}T_\alpha R ρ ′ R ρ = C T α by (38), and the kernel of the transpose R ρ ′ R_\rho' R ρ ′ is ( Ran R ρ ) ⊥ (\operatorname{Ran}R_\rho)^\perp ( Ran R ρ ) ⊥ ; orthogonality proves minimum norm and the description of the solution set.
The projection formula uses the closed spectral subspace K α \mathcal{K}_\alpha K α explicitly. The factor P K α P_{\mathcal{K}_\alpha} P K α distinguishes projection onto the analysis range from projection onto W ρ L 2 ( ν α ) W_\rho L^2(\nu_\alpha) W ρ L 2 ( ν α ) .
Remark B.2 (Range of the analysis operator). By Theorem 3.14 , Ran R ρ = W ρ ( K α ) ⊂ W ρ ( L 2 ( ν α ) ) \operatorname{Ran}R_\rho=W_\rho(\mathcal{K}_\alpha)\subset W_\rho(L^2(\nu_\alpha)) Ran R ρ = W ρ ( K α ) ⊂ W ρ ( L 2 ( ν α )) , and both are closed. The projection onto W ρ L 2 ( ν α ) W_\rho L^2(\nu_\alpha) W ρ L 2 ( ν α ) is C − 1 W ρ W ρ ∗ \mathcal{C}^{-1}W_\rho W_\rho^* C − 1 W ρ W ρ ∗ . The projection onto Ran R ρ \operatorname{Ran}R_\rho Ran R ρ is C − 1 W ρ P K α W ρ ∗ \mathcal{C}^{-1}W_\rho P_{\mathcal{K}_\alpha}W_\rho^* C − 1 W ρ P K α W ρ ∗ , the minimum-norm solution of S ρ γ = F S_\rho\gamma=F S ρ γ = F is C − 1 R ρ T α − 1 F \mathcal{C}^{-1}R_\rho T_\alpha^{-1}F C − 1 R ρ T α − 1 F , and all solutions differ by elements of ( Ran R ρ ) ⊥ (\operatorname{Ran}R_\rho)^\perp ( Ran R ρ ) ⊥ (Proposition B.1). These formulas retain the closed spectral subspace explicitly and separate the reconstruction of analyzed targets from arbitrary coefficient synthesis.
Additional approximation bounds
This appendix gives concentration for bounded parameters and separates input truncation from finite-width approximation. These are consequences of the main approximation results, rather than ingredients of their proofs. Use the polar decomposition (62) with V > 0 V>0 V > 0 ; the zero measure is represented by the zero network.
Corollary C.1 (Concentration for bounded parameters). Under Theorem 6.5 , suppose ∥ a ∥ 2 + ∣ b ∣ 2 ≤ B 2 \|a\|^2+|b|^2\le B^2 ∥ a ∥ 2 + ∣ b ∣ 2 ≤ B 2 almost surely for some B ≥ 0 B\ge0 B ≥ 0 , and put M K = ∣ σ ( 0 ) ∣ + Lip ( σ ) R K B M_K=|\sigma(0)|+\operatorname{Lip}(\sigma)R_KB M K = ∣ σ ( 0 ) ∣ + Lip ( σ ) R K B . With probability at least 1 − δ 1-\delta 1 − δ ,
∥ f N − f ∥ C ( K ) ≤ 8 V N ( ∣ σ ( 0 ) ∣ + Lip ( σ ) R K M 2 ) + V M K 2 log ( 1 / δ ) N . \|f_N-f\|_{C(K)}\le\frac{8V}{\sqrt{N}}\left(|\sigma(0)|+\operatorname{Lip}(\sigma)R_KM_2\right)+VM_K\sqrt{\frac{2\log(1/\delta)}{N}}. ∥ f N − f ∥ C ( K ) ≤ N 8 V ( ∣ σ ( 0 ) ∣ + Lip ( σ ) R K M 2 ) + V M K N 2 log ( 1/ δ ) . Proof. Each atom has norm at most M K M_K M K , so replacing one sample changes ∥ f N − f ∥ C ( K ) \|f_N-f\|_{C(K)} ∥ f N − f ∥ C ( K ) by at most 2 V M K / N 2VM_K/N 2 V M K / N . The bounded-difference inequality bounds the excess over the expectation, and Theorem 6.5 bounds the expectation.
Corollary C.2 (Input truncation and finite-width approximation errors). Let Π m \Pi_m Π m be finite-rank orthogonal projections converging strongly to the identity. For f ∈ C ( H ) f\in C(\mathcal{H}) f ∈ C ( H ) and compact K K K , ∥ f − f ∘ Π m ∥ C ( K ) → 0 \|f-f\circ\Pi_m\|_{C(K)}\to0 ∥ f − f ∘ Π m ∥ C ( K ) → 0 . If f = S σ Γ f=S_\sigma\Gamma f = S σ Γ satisfies the hypotheses of Theorem 6.5 and the same samples are used with directions Π m a j \Pi_m a_j Π m a j , then
E ∥ f − f m , N ∥ C ( K ) ≤ Lip ( σ ) ( ∫ ∥ a ∥ d ∣ Γ ∣ ) sup K ∥ x − Π m x ∥ + 8 V N ( ∣ σ ( 0 ) ∣ + Lip ( σ ) R K M 2 ) . \mathbb{E}\|f-f_{m,N}\|_{C(K)}
\le\operatorname{Lip}(\sigma)\left(\int\|a\|\,\mathrm{d}|\Gamma|\right)\sup_K\|x-\Pi_mx\|
+\frac{8V}{\sqrt{N}}\left(|\sigma(0)|+\operatorname{Lip}(\sigma)R_KM_2\right). E ∥ f − f m , N ∥ C ( K ) ≤ Lip ( σ ) ( ∫ ∥ a ∥ d ∣Γ∣ ) K sup ∥ x − Π m x ∥ + N 8 V ( ∣ σ ( 0 ) ∣ + Lip ( σ ) R K M 2 ) . Proof. A finite-net argument gives sup K ∥ x − Π m x ∥ → 0 \sup_K\|x-\Pi_mx\|\to0 sup K ∥ x − Π m x ∥ → 0 . If f ∘ Π m f\circ\Pi_m f ∘ Π m did not converge uniformly on K K K , a sequence x m ∈ K x_m\in K x m ∈ K with ∣ f ( Π m x m ) − f ( x m ) ∣ ≥ ε |f(\Pi_mx_m)-f(x_m)|\ge\varepsilon ∣ f ( Π m x m ) − f ( x m ) ∣ ≥ ε would have a convergent subsequence x m → x x_m\to x x m → x , along which Π m x m → x \Pi_mx_m\to x Π m x m → x as well, contradicting continuity of f f f at x x x . The triangle inequality separates truncation from sampling; in the integral representation the Lipschitz bound gives the truncation term, projecting directions does not increase their second moment, and the sampling proof applies on the original parameter probability space.
The finite-dimensional case and the dilation obstruction
This appendix compares the weighted construction with finite-dimensional ridgelet analysis. It also explains why a construction based on dilating a Gaussian spectral measure cannot produce a finite coefficient measure in infinite dimension.
A Fourier calculation for comparison
For comparison only, take Lebesgue input and direction measures on R m \mathbb{R}^m R m , and write
R ρ L e b f ( a , b ) = ∫ R m f ( x ) ρ ( ⟨ a , x ⟩ − b ) d x . (85) R_{\rho}^{\mathrm{Leb}}f(a,b)=\int_{\mathbb{R}^m}f(x)\rho(\langle a,x\rangle-b)\,\mathrm{d}x. \tag*{(85)} R ρ Leb f ( a , b ) = ∫ R m f ( x ) ρ (⟨ a , x ⟩ − b ) d x . (85)
For Schwartz f , ρ f,\rho f , ρ , Fubini and a translation in the bias give
( R ρ L e b f ) ♯ ( a , ω ) = ρ ♯ ( − ω ) f ^ ( ω a ) . (86) \left(R_{\rho}^{\mathrm{Leb}}f\right)^\sharp(a,\omega)=\rho^\sharp(-\omega)\widehat{f}(\omega a). \tag*{(86)} ( R ρ Leb f ) ♯ ( a , ω ) = ρ ♯ ( − ω ) f ( ω a ) . (86)
If σ \sigma σ is Schwartz and ρ \rho ρ is band pass, Parseval in the bias followed by ξ = ω a \xi=\omega a ξ = ω a gives the iterated synthesis identity
S σ L e b R ρ L e b f = K σ , ρ f , K σ , ρ = ( 2 π ) m − 1 ∫ R σ ♯ ( ω ) ρ ♯ ( − ω ) ∣ ω ∣ − m d ω . (87) S_{\sigma}^{\mathrm{Leb}}R_{\rho}^{\mathrm{Leb}}f=K_{\sigma,\rho}f,\qquad
K_{\sigma,\rho}=(2\pi)^{m-1}\int_{\mathbb{R}}\sigma^\sharp(\omega)\rho^\sharp(-\omega)|\omega|^{-m}\,\mathrm{d}\omega. \tag*{(87)} S σ Leb R ρ Leb f = K σ , ρ f , K σ , ρ = ( 2 π ) m − 1 ∫ R σ ♯ ( ω ) ρ ♯ ( − ω ) ∣ ω ∣ − m d ω . (87)
The transformed integrals are absolute: the frequency factor is supported away from zero and f ^ \widehat{f} f is integrable. This calculation explains the scaling factor, but the Hilbert-space construction in the main text is defined independently of it.
The finite-dimensional reference measure
On H = R m \mathcal{H} = \mathbb{R}^{m} H = R m with P = I P = I P = I and 0 < α < m 0 < \alpha< m 0 < α < m , Tonelli and the Gaussian density give, for a ≠ 0 a \ne0 a = 0 ,
d ν α d a ( a ) = ( 4 π ) − m / 2 ∫ 0 ∞ s ( α − m ) / 2 − 1 e − ∥ a ∥ 2 / ( 4 s ) d s = ( 4 π ) − m / 2 ( ∥ a ∥ 2 4 ) ( α − m ) / 2 Γ ( m − α 2 ) . \frac{d\nu_{\alpha}}{da}(a) = (4\pi)^{-m/2}\int_{0}^{\infty}s^{(\alpha-m)/2-1}e^{-\lVert a\rVert^{2}/(4s)}\,ds
= (4\pi)^{-m/2}\left(\frac{\lVert a\rVert^{2}}{4}\right)^{(\alpha-m)/2}\Gamma\left(\frac{m-\alpha}{2}\right). d a d ν α ( a ) = ( 4 π ) − m /2 ∫ 0 ∞ s ( α − m ) /2 − 1 e − ∥ a ∥ 2 / ( 4 s ) d s = ( 4 π ) − m /2 ( 4 ∥ a ∥ 2 ) ( α − m ) /2 Γ ( 2 m − α ) .
by the substitution u = ∥ a ∥ 2 / ( 4 s ) u=\lVert a\rVert^{2}/(4s) u = ∥ a ∥ 2 / ( 4 s ) . Hence
ν α ( d a ) = c m , α ∥ a ∥ α − m d a , c m , α = 2 − α π − m / 2 Γ ( m − α 2 ) . (88) \nu_{\alpha}(da)=c_{m,\alpha}\lVert a\rVert^{\alpha-m}\,da,\qquad c_{m,\alpha}=2^{-\alpha}\pi^{-m/2}\Gamma\left(\frac{m-\alpha}{2}\right). \tag*{(88)} ν α ( d a ) = c m , α ∥ a ∥ α − m d a , c m , α = 2 − α π − m /2 Γ ( 2 m − α ) . (88)
The density is locally integrable at zero because its radial integral is proportional to ∫ 0 r t α − 1 d t \int_{0}^{r}t^{\alpha-1}dt ∫ 0 r t α − 1 d t . The restriction α < m \alpha<m α < m is needed for convergence at s = ∞ s=\infty s = ∞ ; at α = m \alpha=m α = m the mixture diverges logarithmically, and Lebesgue measure, the classical choice of direction measure, must be chosen separately rather than obtained by substituting α = m \alpha=m α = m . In infinite dimension no such restriction arises, which is the content of Lemma 3.2 .
Remark D.1 (Comparison with familiar spaces). In finite dimension, (88) gives
∥ f ∥ E α 2 = c m , α ∫ R m ∣ f p Q ^ ( ξ ) ∣ 2 ∥ ξ ∥ α − m d ξ , 0 < α < m , \lVert f\rVert_{\mathcal{E}_{\alpha}}^{2}=c_{m,\alpha}\int_{\mathbb{R}^{m}}\left|\widehat{fp_{Q}}(\xi)\right|^{2}\lVert\xi\rVert^{\alpha-m}\,d\xi,\qquad0<\alpha<m, ∥ f ∥ E α 2 = c m , α ∫ R m f p Q ( ξ ) 2 ∥ ξ ∥ α − m d ξ , 0 < α < m ,
where p Q p_{Q} p Q is the Gaussian density: a homogeneous Sobolev norm of order ( α − m ) / 2 (\alpha-m)/2 ( α − m ) /2 of the weighted density f p Q fp_{Q} f p Q . In infinite dimension there is no Lebesgue measure to remove the weight, and we do not identify E α \mathcal{E}_{\alpha} E α with a Sobolev space. Likewise K α \mathcal{K}_{\alpha} K α is a closed subspace of L 2 ( ν α ) L^{2}(\nu_{\alpha}) L 2 ( ν α ) but is not asserted to be a reproducing kernel Hilbert space.
Filtered backprojection
Corollary D.2 (The frame operator in finite dimension). Let H = R m \mathcal{H}=\mathbb{R}^{m} H = R m , 0 < α < m 0<\alpha<m 0 < α < m , ν α \nu_{\alpha} ν α as in (88) , and let p p p be a nondegenerate Gaussian density. Assume f ∈ L 2 ( p d x ) f\in L^{2}(pdx) f ∈ L 2 ( p d x ) with g : = f p ∈ S ( R m ) g:=fp\in\mathcal{S}(\mathbb{R}^{m}) g := f p ∈ S ( R m ) . Then f ∈ D α f\in\mathcal{D}_{\alpha} f ∈ D α , and the frame operator, written as a distribution against the pivot measure p d x pdx p d x , has the representative
t f = ∫ R m e i ⟨ x , ξ ⟩ g ^ ( ξ ) ν α ( d ξ ) = k m , α ( − Δ ) − ( m − α ) / 2 g , k m , α = ( 2 π ) m c m , α . t_{f}=\int_{\mathbb{R}^{m}}e^{i\langle x,\xi\rangle}\widehat{g}(\xi)\,\nu_{\alpha}(d\xi)=k_{m,\alpha}(-\Delta)^{-(m-\alpha)/2}g,\qquad k_{m,\alpha}=(2\pi)^{m}c_{m,\alpha}. t f = ∫ R m e i ⟨ x , ξ ⟩ g ( ξ ) ν α ( d ξ ) = k m , α ( − Δ ) − ( m − α ) /2 g , k m , α = ( 2 π ) m c m , α . For a band-pass ρ \rho ρ , the synthesis S ρ R ρ f S_{\rho}R_{\rho}f S ρ R ρ f is represented by ( ρ , ρ ) α t f (\rho,\rho)_{\alpha}t_{f} ( ρ , ρ ) α t f , and, distributionally,
f = p − 1 k m , α ( ( ρ , ρ ) ) α ( − Δ ) ( m − α ) / 2 S ρ R ρ f . f=\frac{p^{-1}}{k_{m,\alpha}((\rho,\rho))_{\alpha}}(-\Delta)^{(m-\alpha)/2}S_{\rho}R_{\rho}f. f = k m , α (( ρ , ρ ) ) α p − 1 ( − Δ ) ( m − α ) /2 S ρ R ρ f . If instead the direction measure is Lebesgue measure and α = m \alpha=m α = m , the multiplier is one and k = ( 2 π ) m k=(2\pi)^{m} k = ( 2 π ) m , so that f = ( 2 π ) − m ( ( ρ , ρ ) m ) − 1 p − 1 S ρ R ρ f f=(2\pi)^{-m}((\rho,\rho)_{m})^{-1}p^{-1}S_{\rho}R_{\rho}f f = ( 2 π ) − m (( ρ , ρ ) m ) − 1 p − 1 S ρ R ρ f .
Proof. Here F Q f = g ^ F_{Q}f=\widehat{g} F Q f = g . The weight ∥ ξ ∥ α − m \lVert\xi\rVert^{\alpha-m} ∥ ξ ∥ α − m is locally integrable and g ^ \widehat{g} g is Schwartz, so g ^ ∈ L 1 ∩ L 2 ( ν α ) \widehat{g}\in L^{1}\cap L^{2}(\nu_{\alpha}) g ∈ L 1 ∩ L 2 ( ν α ) and Fourier inversion gives t f ^ = ( 2 π ) m c m , α ∥ ξ ∥ α − m g ^ \widehat{t_{f}}=(2\pi)^{m}c_{m,\alpha}\lVert\xi\rVert^{\alpha-m}\widehat{g} t f = ( 2 π ) m c m , α ∥ ξ ∥ α − m g . For smooth compactly supported h h h , Fubini identifies ∫ t f h p d x \int t_{f}h\,pdx ∫ t f h p d x with ⟨ f , h ⟩ E α \langle f,h\rangle_{\mathcal{E}_{\alpha}} ⟨ f , h ⟩ E α , which is Theorem 4.8 (iii). The multiplier ∥ ξ ∥ m − α \lVert\xi\rVert^{m-\alpha} ∥ ξ ∥ m − α cancels the locally integrable multiplier, proving the formula in distributions. For Lebesgue direction measure, direct Fourier inversion gives t f = ( 2 π ) m f p t_{f}=(2\pi)^{m}fp t f = ( 2 π ) m f p .
The two weights are visible in the formula: division by p p p removes the input weight, and ( − Δ ) ( m − α ) / 2 (-\Delta)^{(m-\alpha)/2} ( − Δ ) ( m − α ) /2 removes the homogeneous direction weight. With Lebesgue measure on both sides the filter is a constant, which is the classical statement (87) .
Why the spectral measure cannot be dilated
In finite dimension, the Fourier-slice identity (86) says that the bias Fourier slice of the ridgelet coefficient at frequency ω \omega ω is ρ † ( − ω ) \rho^{\dagger}(-\omega) ρ † ( − ω ) times the spectral measure of f f f dilated by 1 / ω 1/\omega 1/ ω . One might try to define a coefficient in infinite dimension by this prescription, starting from a target with a finite spectral measure. The following proposition shows that this is impossible for a Gaussian spectral measure, that is, for the target f W f_W f W of Example 7.2 .
Proposition D.3 (Dilation obstruction). Let dim H = ∞ \dim\mathcal{H} = \infty dim H = ∞ and let W W W be injective, positive, self-adjoint, and trace class, with eigenvectors e j e_j e j and eigenvalues w j > 0 w_j > 0 w j > 0 .
(i) For t > 0 t > 0 t > 0 , the Borel sets E t : = { x : lim n → ∞ 1 n ∑ j = 1 n ⟨ x , e j ⟩ 2 / w j = t } E_t := \{x : \lim_{n \to\infty} \frac{1}{n}\sum_{j=1}^{n} \langle x,e_j\rangle^2/w_j = t\} E t := { x : lim n → ∞ n 1 ∑ j = 1 n ⟨ x , e j ⟩ 2 / w j = t } are pairwise disjoint and μ W t ( E t ) = 1 \mu_W^t(E_t) = 1 μ W t ( E t ) = 1 . Consequently a sigma-finite measure dominates μ W t \mu_W^t μ W t for at most countably many t t t .
(ii) Let r r r be a bounded Borel function on R \mathbb{R} R with { r ≠ 0 } \{r \ne0\} { r = 0 } of positive Lebesgue measure. There is no finite complex Borel measure Γ \Gamma Γ on H × R \mathcal{H} \times\mathbb{R} H × R whose bias slices satisfy
Γ ω + ( E ) : = ∫ E × R e i ω b Γ ( d a , d b ) = r ( ω ) ( D 1 / ω ) # μ W ( E ) \Gamma^+_\omega(E) := \int_{E \times\mathbb{R}} e^{i\omega b}\,\Gamma(\mathrm{d}a,\mathrm{d}b)
= r(\omega) (D_{1/\omega})_{\#}\mu_W(E) Γ ω + ( E ) := ∫ E × R e iω b Γ ( d a , d b ) = r ( ω ) ( D 1/ ω ) # μ W ( E ) for almost every ω ≠ 0 \omega\ne0 ω = 0 , E ∈ B ( H ) E \in\mathcal{B}(\mathcal{H}) E ∈ B ( H ) .
Proof. (i) The strong law of large numbers for the independent normalized Gaussian coordinates gives μ W t ( E t ) = 1 \mu_W^t(E_t) = 1 μ W t ( E t ) = 1 , and the sets are disjoint for distinct t t t . If a sigma-finite measure ν \nu ν dominated μ W t \mu_W^t μ W t for uncountably many t t t , then ν ( E t ) > 0 \nu(E_t) > 0 ν ( E t ) > 0 for uncountably many disjoint sets; intersecting with an exhaustion by sets of finite ν \nu ν -measure and using positive rational thresholds shows that a sigma-finite measure has at most countably many disjoint sets of positive measure, a contradiction. (ii) If Γ \Gamma Γ existed, the finite measure μ ( E ) : = ∣ Γ ∣ ( E × R ) \mu(E) := |\Gamma|(E \times\mathbb{R}) μ ( E ) := ∣Γ∣ ( E × R ) would satisfy ∣ Γ ω + ( E ) ∣ ≤ μ ( E ) |\Gamma^+_\omega(E)| \le\mu(E) ∣ Γ ω + ( E ) ∣ ≤ μ ( E ) for every ω \omega ω , hence would dominate ( D 1 / ω ) # μ W = μ W / ω 2 (D_{1/\omega})_{\#}\mu_W = \mu_W/\omega^2 ( D 1/ ω ) # μ W = μ W / ω 2 for almost every ω \omega ω with r ( ω ) ≠ 0 r(\omega) \ne0 r ( ω ) = 0 . These are uncountably many distinct values of t = ω − 2 t = \omega^{-2} t = ω − 2 , contradicting (i).
The finite-variation hypothesis is essential: the marginal of a sigma-finite measure need not be sigma-finite, and the proposition makes no claim about coefficient objects that are not measures. The construction of § 3 avoids the obstruction because ν α \nu_\alpha ν α enters as an integrator in the parameter space and is never required to dominate the dilates of a spectral measure; the targets it represents exactly are those with a spectral density with respect to ν α \nu_\alpha ν α (Theorem 4.5 ), and Gaussian targets are reached through the frame operator (Example 7.2 ) or by approximation (Theorem 6.8 ).
Numerical methods and additional results
This appendix describes the numerical procedures behind § 8 . It records how parameters are drawn, how integrals and function-space inputs are discretized, and how the reported errors are measured. The figures and principal results are in figure 1 and table 1.
Reproduction and error measurement
The supplementary Python script experiments.py reproduces all numbers and figures with the fixed seed 20260908. The full run uses widths N = 4 , 8 , … , 4096 N = 4, 8, \ldots, 4096 N = 4 , 8 , … , 4096 , and the same sequence of sampled neurons is accumulated as the width increases within each trial. The test set is fixed within each resolution and experiment. At each width we record the largest absolute error on that finite set and average over trials; the slopes in table 1 are least-squares slopes of log ( mean error ) \log(\text{mean error}) log ( mean error ) against log N \log N log N . Experiments 1 and 3(a) use 20 trials, and Experiment 2 uses 10. The errors measure finite test sets rather than the supremum over the underlying infinite compact family of inputs.
Gaussian directions and coefficient measures
For Experiment 1, independent coordinates of a direction have the form a j = Z j / j a_j = Z_j/j a j = Z j / j , with standard normal Z j Z_j Z j , for 1 ≤ j ≤ d 1 \le j \le d 1 ≤ j ≤ d . The 200 test inputs have coordinates x j = u j / j x_j = u_j/j x j = u j / j , with independent u j u_j u j uniform in [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] . The target is evaluated from its exact finite-dimensional formula ∑ j = 1 d j − 2 x j 2 / ( 2 π ) \sqrt{\sum_{j=1}^{d}j^{-2}x_j^2/(2\pi)} ∑ j = 1 d j − 2 x j 2 / ( 2 π ) .
For Experiment 2, the same rule draws 100 test inputs. The matrices are diagonal, with p j = q j = j − 2 p_j=q_j=j^{-2} p j = q j = j − 2 , w j = 1 w_j=1 w j = 1 , and ( S W ) j j = q j / ( 1 + q j ) (S_W)_{jj}=q_j/(1+q_j) ( S W ) j j = q j / ( 1 + q j ) ; determinants are evaluated by summing logarithms. The target 6 f g 6f_g 6 f g is evaluated by the one-dimensional quadrature in (77) . In the mixture integral ∫ 0 ∞ h ( s ) s − 1 / 2 d s \int_0^\infty h(s)s^{-1/2}\,ds ∫ 0 ∞ h ( s ) s − 1/2 d s , the substitution s = r 2 s=r^2 s = r 2 on ( 0 , 1 ) (0,1) ( 0 , 1 ) removes the endpoint singularity, and the remaining integral is evaluated on ( 1 , ∞ ) (1,\infty) ( 1 , ∞ ) .
Parameters are drawn from ∣ γ g ∣ λ 1 / V |\gamma_g|\lambda_1/V ∣ γ g ∣ λ 1 / V as follows: the bias given a a a is sampled exactly by rejection from a mixture of Gaussian moment densities; the direction is sampled by sampling–importance resampling from a pool of 2 × 10 5 2 \times10^5 2 × 1 0 5 proposals ( s , a ) (s,a) ( s , a ) with s ∼ θ 0 BetaPrime ( 1 / 2 , 3 / 2 ) s \sim\theta_0\operatorname{BetaPrime}(1/2,3/2) s ∼ θ 0 BetaPrime ( 1/2 , 3/2 ) , a ∼ μ 2 s P a \sim\mu_{2sP} a ∼ μ 2 s P , and weight ( 1 + κ W ( a ) ) − 2 ( 1 + s / θ 0 ) 2 (1+\kappa_W(a))^{-2}(1+s/\theta_0)^2 ( 1 + κ W ( a ) ) − 2 ( 1 + s / θ 0 ) 2 , where θ 0 = ( 2 tr P S W ) − 1 \theta_0=(2\operatorname{tr}PS_W)^{-1} θ 0 = ( 2 tr P S W ) − 1 (effective sample sizes between 2.8 × 10 3 2.8 \times10^3 2.8 × 1 0 3 and 9.7 × 10 3 9.7 \times10^3 9.7 × 1 0 3 ). The total variation V V V is computed by quadrature from (24) .
The conditional bias law can be made explicit. Writing u = 1 + κ W ( a ) u=1+\kappa_W(a) u = 1 + κ W ( a ) and b = u z b=\sqrt{u}\,z b = u z , the density of z z z is ∣ He 4 ( z ) ∣ φ 1 ( z ) / m 4 |\operatorname{He}_4(z)|\varphi_1(z)/m_4 ∣ He 4 ( z ) ∣ φ 1 ( z ) / m 4 . Rejection sampling uses the proposal
1 4 φ 1 ( z ) + 1 2 z 2 φ 1 ( z ) + 1 12 z 4 φ 1 ( z ) = z 4 + 6 z 2 + 3 12 φ 1 ( z ) \frac{1}{4}\varphi_1(z)+\frac{1}{2}z^2\varphi_1(z)+\frac{1}{12}z^4\varphi_1(z)=\frac{z^4+6z^2+3}{12}\varphi_1(z) 4 1 φ 1 ( z ) + 2 1 z 2 φ 1 ( z ) + 12 1 z 4 φ 1 ( z ) = 12 z 4 + 6 z 2 + 3 φ 1 ( z )
and accepts with probability ∣ He 4 ( z ) ∣ / ( z 4 + 6 z 2 + 3 ) |\operatorname{He}_4(z)|/(z^4+6z^2+3) ∣ He 4 ( z ) ∣/ ( z 4 + 6 z 2 + 3 ) . This gives the exact conditional law. Directions are drawn with replacement from the weighted proposal pool, so this part approximates the desired continuous direction law. The pool is reused across trials at each resolution; the reported trial variation is conditional on that pool and does not measure its approximation error. The code uses the equivalent bias convention ⟨ a , x ⟩ + c \langle a,x\rangle+c ⟨ a , x ⟩ + c with c = − b c=-b c = − b ; the even bias density and its even sign factor give the same synthesis integral.
The variation integral is evaluated using
∫ ( 1 + κ W ( a ) ) − 2 ν 1 ( d a ) = ∫ 0 ∞ ∫ 0 ∞ t e − t ∏ j = 1 d ( 1 + 4 t s p j ( S W ) j j ) − 1 / 2 d t s − 1 / 2 d s . \int(1+\kappa_W(a))^{-2}\,\nu_1(da)=\int_0^\infty\int_0^\infty t e^{-t}\prod_{j=1}^{d}(1+4tsp_j(S_W)_{jj})^{-1/2}\,dt\,s^{-1/2}\,ds. ∫ ( 1 + κ W ( a ) ) − 2 ν 1 ( d a ) = ∫ 0 ∞ ∫ 0 ∞ t e − t j = 1 ∏ d ( 1 + 4 t s p j ( S W ) j j ) − 1/2 d t s − 1/2 d s .
Multiplication by D − 1 / 2 2 π m 4 D^{-1/2}\sqrt{2\pi}\,m_4 D − 1/2 2 π m 4 gives V V V . The value m 4 = E ∣ He 4 ( Z ) ∣ m_4=\mathbb{E}|\operatorname{He}_4(Z)| m 4 = E ∣ He 4 ( Z ) ∣ is computed by quadrature, split at the four real roots of He 4 \operatorname{He}_4 He 4 ; the numerical integral is restricted to [ − 12 , 12 ] [-12,12] [ − 12 , 12 ] .
For Experiment 3(a), the interval grid is t i = ( i − 1 / 2 ) / d t_i=(i-1/2)/d t i = ( i − 1/2 ) / d , 1 ≤ i ≤ d 1\le i\le d 1 ≤ i ≤ d , and inner products and integrals use the midpoint rule with weight 1 / d 1/d 1/ d . Each of the 100 test inputs is a sum x = ∑ n = 1 30 ( u n / n ) e n x=\sum_{n=1}^{30}(u_n/n)e_n x = ∑ n = 1 30 ( u n / n ) e n , with independent uniform u n ∈ [ − 1 , 1 ] u_n\in[-1,1] u n ∈ [ − 1 , 1 ] and e n ( t ) = 2 sin ( π n t ) e_n(t)=\sqrt{2}\sin(\pi nt) e n ( t ) = 2 sin ( π n t ) . The reference value of F 1 F_1 F 1 uses the Green matrix g ( t i , t j ) g(t_i,t_j) g ( t i , t j ) on the same grid. Draws from w / V w/V w / V use inversion of a cumulative distribution computed by the trapezoidal rule on 20001 uniform grid points, with linear interpolation. The direction at a drawn point y y y is the Green function g ( y , ⋅ ) g(y,\cdot) g ( y , ⋅ ) , evaluated on the input grid. The plateau at d = 64 d=64 d = 64 therefore includes the error in the quadrature reference, rather than measuring a failure of the finite-width approximation rate.
For Experiment 3(b), the closed-form transform uses 2048 midpoints for inner products and 400 midpoints for the outer integral in y y y . The direct Monte Carlo calculation uses 2 × 10 5 2 \times10^{5} 2 × 1 0 5 independent draws
x = ∑ n = 1 d ′ λ n 1 / 2 ζ n e n , λ n = ( 1 + π 2 n 2 ) − 1 , d ′ ∈ { 8 , 64 , 512 } , x=\sum_{n=1}^{d'}\lambda_n^{1/2}\zeta_n e_n,\qquad\lambda_n=(1+\pi^2 n^2)^{-1},\qquad d'\in\{8,64,512\}, x = n = 1 ∑ d ′ λ n 1/2 ζ n e n , λ n = ( 1 + π 2 n 2 ) − 1 , d ′ ∈ { 8 , 64 , 512 } ,
with independent standard normal ζ n \zeta_n ζ n . For each draw, F 1 ( x ) F_1(x) F 1 ( x ) and ⟨ a , x ⟩ \langle a,x\rangle ⟨ a , x ⟩ are evaluated on the 2048-point grid. The reported standard error is the empirical standard deviation of the integrand divided by the square root of the number of draws. Agreement with the closed-form values is a check at the specified direction and biases; it does not eliminate quadrature error or the truncation of the Gaussian input expansion.
Extended overview of related work
This appendix expands § 1.3 . It places the Gaussian-weighted ridgelet transform among four bodies of work: harmonic analysis on infinite-dimensional spaces, universal approximation by networks with infinite-dimensional inputs, approximation and complexity theory for neural operators, and the analysis of Gaussian measures and functional data. The contributions of this paper are the explicit coefficient W ρ G W_{\rho}G W ρ G for targets with a ν α \nu_{\alpha} ν α -spectral density, the Plancherel identity and the frame operator T α T_{\alpha} T α on the Hilbert space E α \mathcal{E}_{\alpha} E α , weak reconstruction for tempered non-polynomial activations, the dimension-free N − 1 / 2 N^{-1/2} N − 1/2 rate for Lipschitz activations, and the three infinite-dimensional examples. We do not claim priority for integrating over an infinite-dimensional space with a Gaussian measure, for transforms of ridgelet or wavelet type on Wiener space, or for universal approximation with infinite-dimensional inputs; each has a literature summarized below.
Ridgelet transforms and integral representations
The ridgelet transform of Murata [74 ] , Candès [16 ] , and Candès and Donoho [17 ] represents a function on R m \mathbb{R}^{m} R m as a superposition of ridge functions x ↦ σ ( ⟨ a , x ⟩ − b ) x\mapsto\sigma(\langle a,x\rangle-b) x ↦ σ (⟨ a , x ⟩ − b ) , and it is the continuous counterpart of a depth-two network with a coefficient measure on the parameter space. Its distributional extension admits unbounded activations such as ReLU [87 ] ; weighted Sobolev activation spaces and the geometry of the synthesis null space are developed in Sonoda et al. [90 ] , where the infinite-dimensional null space refers to the coefficient space over a finite-dimensional input; the Fourier-slice method yields transforms for group convolutional networks and non-Euclidean inputs [88 , 89 ] ; and complexity bounds derived from integral representations are surveyed in Kůrková [59 ] . The Hilbert-space construction in Sonoda et al. [89 Section 4] integrates over a finite-dimensional subspace with Lebesgue measure. In this paper, we replace that integral by the Gaussian input measure μ Q \mu_Q μ Q and the homogeneous direction measure ν α \nu_{\alpha} ν α on the whole Hilbert space, and we use the transform as a tool for networks: it yields the coefficients, the universality theorem, the Barron-type rate, and the examples.
Harmonic analysis on infinite-dimensional spaces
Fourier transforms of measures. The Fourier transform of a finite Borel measure ν \nu ν on a Hilbert space H H H is its characteristic functional ν ^ ( ξ ) = ∫ H e − i ⟨ x , ξ ⟩ d ν ( x ) \widehat{\nu}(\xi)=\int_H e^{-i\langle x,\xi\rangle}\,d\nu(x) ν ( ξ ) = ∫ H e − i ⟨ x , ξ ⟩ d ν ( x ) . Gaussian measures are characterized by this transform, and the theorems of Minlos and Sazonov describe which positive-definite functions are transforms of Radon measures on the dual of a nuclear space or on a Hilbert space [12 , 24 ] . Bochner’s theorem fails in infinite dimensions: ξ ↦ exp ( − ∥ ξ ∥ 2 / 2 ) \xi\mapsto\exp(-\lVert\xi\rVert^2/2) ξ ↦ exp ( − ∥ ξ ∥ 2 /2 ) is not the transform of a finite Borel measure on an infinite-dimensional Hilbert space, a fact also relevant to kernels on function spaces [98 ] . In this paper, we use the characteristic functional of the finite measure μ Q \mu_Q μ Q only along lines through the origin, ω ↦ ω a \omega\mapsto\omega a ω ↦ ω a for a fixed direction a a a (rays, in the terminology of tomography), and relate it to the one-dimensional Fourier transform in the bias variable. The transform of the measure is therefore used only through its restriction to such lines, and this is why coefficient measures cannot be obtained from a weak Fourier-side identity alone.
Fourier–Wiener, Segal–Bargmann, and white-noise transforms. Cameron and Martin [14 , 15 ] introduced the Fourier–Wiener transform of analytic and of square-integrable functionals on Wiener space. Segal [83 ] identified the Gaussian L 2 L^2 L 2 space with the symmetric Fock space, the setting of the infinite-dimensional Segal–Bargmann transform, and Gross [38 ] developed potential theory on abstract Wiener space. In white-noise analysis, Kuo [60 ] defined a Fourier transform of generalized functionals on the Gel’fand triple ( S ) ⊂ L 2 ( μ ) ⊂ ( S ) ∗ (S) \subset L^2(\mu) \subset(S)^* ( S ) ⊂ L 2 ( μ ) ⊂ ( S ) ∗ and related it to the Gross and Volterra Laplacians; Lee [66 ] characterized measures through this transform; the monograph of Hida et al. [40 ] collects the Laplacians and the Fourier and Fourier–Mehler transforms. Jorgensen et al. [49 ] construct transforms between Gaussian fields and reproducing kernel Hilbert spaces and derive from them a Fourier transform for general Gaussian processes. These transforms act on functionals of a Gaussian variable, and the Fourier–Wiener transform also enters the inversion of the Gaussian Radon transform by Holmes [42 ] . In this paper, we use only the one-dimensional Fourier transform and the Gaussian characteristic functional, and the Hilbert space E α E_\alpha E α is defined by a spectral norm rather than through a Fock decomposition.
Gaussian harmonic analysis. Harmonic analysis with respect to a Gaussian measure is organized around the Hermite expansion, the Ornstein–Uhlenbeck semigroup, and Meyer’s inequalities for Gaussian Sobolev norms; see the survey of Bogachev [13 ] for the infinite-dimensional theory and Urbina-Romero [94 ] for Gaussian analogues of Littlewood–Paley and maximal functions. Wiener chaos and the totality of Hermite polynomials [48 ] underlie the density arguments of this paper. The Ornstein–Uhlenbeck semigroup and the Fourier–Mehler transform play, for Gaussian measures, the role that dilations play for Lebesgue measure. In this paper, we do not use a dilation group on H \mathcal{H} H ; the scale of a direction a a a enters only through the homogeneity of ν α \nu_\alpha ν α .
Gaussian Radon transforms. Gaussian Radon transforms provide a related approach to integration over infinite-dimensional hyperplanes. Early work on Gaussian surface measures is due to Hertle [39 ] ; Mihai and Sengupta [73 ] develop the Radon–Gauss transform and an inversion procedure. The abstract Wiener space construction of Holmes and Sengupta [43 ] associates Gaussian probability measures with closed affine Cameron–Martin subspaces. Support theorems are established in Becnel and Sengupta [8 ] and Holmes and Sengupta [43 ] , while Holmes [42 ] uses disintegration and the Segal–Bargmann transform for inversion. Becnel [7 ] gives another inversion procedure by taking limits of finite-dimensional Radon inversions under continuity and growth assumptions. Holmes and Sengupta [44 ] study Brownian conditioning and Fock space decompositions in classical Wiener space, and Holmes and Sengupta [45 ] interpret kernel ridge regression through the Gaussian Radon transform.
The relation to our analysis transform can be made explicit. For X ∼ μ Q X \sim\mu_Q X ∼ μ Q , a ≠ 0 a \ne0 a = 0 , and bounded continuous f f f , set ν a = ⟨ Q a , a ⟩ \nu_a = \langle Qa,a\rangle ν a = ⟨ Q a , a ⟩ , let p ν a p_{\nu_a} p ν a be the density of N ( 0 , ν a ) N(0,\nu_a) N ( 0 , ν a ) , and write $\mathcal{G}_Q f(a,t) = \mathbb{E}[f(X)\mid ⟨ a , X ⟩ = t \langle a, X\rangle= t ⟨ a , X ⟩ = t , using the Gaussian conditional kernel. Disintegration gives
R ρ f ( a , b ) = ∫ R ρ ( t − b ) p v a ( t ) G Q f ( a , t ) d t . R_{\rho}f(a,b)=\int_{\mathbb{R}}\rho(t-b)\,p_{v_a}(t)\,G_Qf(a,t)\,dt. R ρ f ( a , b ) = ∫ R ρ ( t − b ) p v a ( t ) G Q f ( a , t ) d t .
Thus R ρ R_{\rho} R ρ filters the Gaussian marginal-density-weighted Radon transform in the bias variable. Formally taking ρ = δ \rho=\delta ρ = δ recovers p v a ( b ) G Q f ( a , b ) p_{v_a}(b)G_Qf(a,b) p v a ( b ) G Q f ( a , b ) , with this identity understood distributionally in b b b . The Dirac filter, however, has infinite self-admissibility integral ∫ R ∣ ω ∣ − α d ω \int_{\mathbb{R}}|\omega|^{-\alpha}\,d\omega ∫ R ∣ ω ∣ − α d ω for every α > 0 \alpha>0 α > 0 , so it lies outside our Plancherel theorem. As a synthesis activation, σ = δ \sigma=\delta σ = δ is covered by Theorem 5.4 with a band-pass analysis filter, in the regularized dual-space sense. The comparison above concerns continuous ambient linear measurements; the Banach-space and general Cameron–Martin formulations have a broader geometric scope. In this paper, our contribution concerns the homogeneous direction measure, the resulting ridgelet frame identities, and their connection to finite-width neural approximation, rather than a generalization of all Gaussian Radon results.
Wavelets and ridgelets on Wiener space. Transforms of wavelet and ridgelet type on the classical Wiener space W = C 0 ( [ 0 , 1 ] ; R d ) W=C_0([0,1];\mathbb{R}^d) W = C 0 ([ 0 , 1 ] ; R d ) with Cameron–Martin space H C M \mathcal{H}_{\mathrm{CM}} H CM have been constructed by Martias. In Martias [68 ] , a profile Ψ ∈ L 2 ( μ ) \Psi\in L^2(\mu) Ψ ∈ L 2 ( μ ) of the form Ψ ( ω ) = η ( ∥ ω ∥ H C M ) \Psi(\omega)=\eta(\|\omega\|_{\mathcal{H}_{\mathrm{CM}}}) Ψ ( ω ) = η ( ∥ ω ∥ H CM ) with ∫ 0 ∞ t 3 ∣ η ( t ) ∣ 2 d t < ∞ \int_0^\infty t^3|\eta(t)|^2\,dt<\infty ∫ 0 ∞ t 3 ∣ η ( t ) ∣ 2 d t < ∞ generates the atoms Ψ a , ω 0 , ω 1 ( ω ) = a − 1 / 2 Ψ ( a − 1 ⟨ ω , ω 0 ⟩ H C M ω 0 − ω 1 ) \Psi_{a,\omega_0,\omega_1}(\omega)=a^{-1/2}\Psi\left(a^{-1}\langle\omega,\omega_0\rangle_{\mathcal{H}_{\mathrm{CM}}}\omega_0-\omega_1\right) Ψ a , ω 0 , ω 1 ( ω ) = a − 1/2 Ψ ( a − 1 ⟨ ω , ω 0 ⟩ H CM ω 0 − ω 1 ) with scale a > 0 a>0 a > 0 , orientation ω 0 \omega_0 ω 0 , and location ω 1 \omega_1 ω 1 ; the transform of f ∈ L 2 ( μ δ ) f\in L^2(\mu_\delta) f ∈ L 2 ( μ δ ) is the pairing with these atoms, and the parameters are integrated against a − 2 d a a^{-2}\,da a − 2 d a , a Wiener measure with a variance parameter in ω 0 \omega_0 ω 0 , and the Wiener measure in ω 1 \omega_1 ω 1 . The resulting identity equates the integrated product of two transforms with a constant multiple of the inner product in L 2 ( μ δ / 2 ) L^2(\mu_{\delta/\sqrt{2}}) L 2 ( μ δ / 2 ) ; they note that it is not a resolution of the identity: the pairing is taken in L 2 ( μ δ ) L^2(\mu_\delta) L 2 ( μ δ ) while the identity returns the inner product of a Gaussian space with a different variance, so synthesizing the coefficients with the same atoms does not reproduce f f f , and they replace reconstruction by an approximation that converges as δ → 0 \delta\to0 δ → 0 . They also derive a characterization of Hölder regularity from the decay of the transform in the scale. The companion works treat a wavelet transform for Wiener functionals with applications to densities of diffusions and backward stochastic differential equations [70 ] , tail properties of the ridgelet transform [69 ] , and multiresolution analyses of L 2 ( μ ) L^2(\mu) L 2 ( μ ) built from integer translations along one Cameron–Martin direction and dyadic dilations of the argument, with orthonormal wavelet bases obtained through the Fourier–Wiener transform [71 ] . In this paper, the construction differs in four respects. The atom is a neuron σ ( ⟨ a , x ⟩ − b ) \sigma(\langle a,x\rangle-b) σ (⟨ a , x ⟩ − b ) whose profile is a function of one real variable, so that synthesis produces a network; the parameter measure is the homogeneous measure ν α \nu_{\alpha} ν α on H \mathcal{H} H in the direction and Lebesgue measure in the bias, rather than a product of a scale measure and Gaussian measures in orientation and location; the Plancherel identity holds on a single space E α \mathcal{E}_{\alpha} E α , with the correction induced by the weights identified as the frame operator T α T_{\alpha} T α and undone explicitly; and the admissibility conditions are formulated on filter pairs in weighted Sobolev spaces, which admits tempered non-polynomial activations. We therefore regard Martias [68 ] as the closest precedent for a ridgelet transform on an infinite-dimensional Gaussian space, and the present construction as a different transform designed for network synthesis.
Functionals. Approximation of nonlinear continuous functionals on compact subsets of C [ a , b ] C[a,b] C [ a , b ] or L p L^p L p by one-hidden-layer networks goes back to Chen and Chen [19 ] , with the operator version in Chen and Chen [20 ] and a radial-basis version in Chen and Chen [21 ] . Mhaskar and Hahm [72 ] obtain lower bounds on the achievable order and nearly matching constructions for functionals on L p L^p L p and C ( [ − 1 , 1 ] s ) C([-1,1]^s) C ([ − 1 , 1 ] s ) ; Stinchcombe [91 ] characterizes activations through compactifications and points out the abstract structure needed for infinite-dimensional inputs; and Rossi and Conan-Guez [81 ] prove universality and consistency for functional multi-layer perceptrons whose neurons are ρ ( ∫ w x + b ) \rho(\int wx+b) ρ ( ∫ w x + b ) , the architecture of functional data analysis closest to the one studied here. Rates for functional deep ReLU networks are given in Song et al. [85 ] and Song et al. [86 ] , and for functionals on reproducing kernel Hilbert spaces, with kernel-dependent rates, in Zhou et al. [103 ] . These rates are obtained through finite-dimensional discretizations and depend on the discretization dimension; in this paper, the rate depends on the activation and on the total variation and moments of the ridgelet coefficient instead.
Fréchet, topological, and non-Euclidean inputs. Benth et al. [9 ] prove compact-open universality for networks x ↦ ⟨ ℓ , σ ( A x + b ) ⟩ x\mapsto\langle\ell,\sigma(Ax+b)\rangle x ↦ ⟨ ℓ , σ ( A x + b )⟩ with operator-valued activations on Fréchet spaces and study finite-dimensional implementability; Galimberti [32 ] extends the setting to non-metric spaces, and Galimberti [33 ] proves density in L p ( μ ) L^p(\mu) L p ( μ ) , 1 ≤ p < ∞ 1\le p<\infty 1 ≤ p < ∞ , for every Radon probability measure μ \mu μ under a bounded separating activation, with finite-parameter and Banach-valued versions. Bilokopytov and Xanthos [11 ] give infinite-dimensional versions of the characterization of non-polynomial activations and strengthen the density result of Benth et al. [9 ] . Kratsios and Bilokopytov [57 ] describe feature and readout maps that preserve universality. Ismailov [46 ] proves density of ∑ i c i σ ( w i f i ( x ) − θ i ) \sum_i c_i\sigma(w_if_i(x)-\theta_i) ∑ i c i σ ( w i f i ( x ) − θ i ) on compact subsets of a topological space for Tauber–Wiener activations and feature families with a separation property (Theorem 2.1) and, on locally convex spaces with f i f_i f i in the dual and any continuous non-polynomial σ \sigma σ (Theorem 2.2), which is the compact-open statement for the scalar ridge architecture; Ismailov [47 ] extends this to deep and vector-valued networks. Moreover, Galimberti et al. [34 ] and Acciaio et al. [1 ] construct universal causal models for sequences in Banach and metric spaces. Signatures of paths provide a linear universal family on path spaces; Cox et al. [22 ] prove universality of linear functionals of the signature for infinite-dimensional rough paths in the norm topology and, on norm-bounded sets, in a weak-* topology.
Weighted and global universality. Cuchiero et al. [23 ] introduce functional input neural networks on weighted spaces: an additive family of real-valued maps forms the hidden layer, a scalar activation is applied to each neuron, and linear readouts take values in a Banach space. Their Theorem 4.13 proves density in the weighted space B ψ ( X ; Y ) \mathcal{B}_{\psi}(X;Y) B ψ ( X ; Y ) , which yields global approximation beyond compact sets, from weighted Stone–Weierstrass theorems; Proposition 4.4 gives sufficient conditions on the activation, including a Fourier condition (A3) in the spirit of the non-polynomial criterion; Corollary 4.17 treats non-anticipative functionals; Theorem 5.4 gives a global result for linear functions of the signature; and Theorem 6.5 identifies the reproducing kernel Hilbert space of a signature kernel with the Cameron–Martin space of a Gaussian process with values in B ψ ( X ) \mathcal{B}_{\psi}(X) B ψ ( X ) . Their first example, φ ( x ) = ∑ n = 1 N y n φ ( ∫ S x d ν n + b n ) \varphi(x)=\sum_{n=1}^{N}y_n\varphi(\int_Sx\,d\nu_n+b_n) φ ( x ) = ∑ n = 1 N y n φ ( ∫ S x d ν n + b n ) with finite signed Radon measures ν n \nu_n ν n , is a finite integral network in the terminology of this paper, with a discrete coefficient measure. Schmocker and Teichmann [82 ] extend this to differentiable maps on weighted infinite-dimensional manifolds, approximating directional derivatives through a weighted Nachbin theorem, and Neufeld and Schmocker [79 ] prove the one-dimensional weighted results, including dimension-independent rates for targets with integrable Fourier transforms, on which the lifting relies; see also van Nuland [96 ] for uniform approximation on non-compact domains. Universality for operators and their Fréchet derivatives is studied by Yao et al. [100 ] for Fourier neural operators and by de Feo [25 ] , who shows that it fails in Fréchet compact-open topologies and Fréchet–Sobolev norms and holds in Bastiani topologies and weighted Bastiani–Sobolev spaces. In this paper, the universality theorem is of compact-open type, with a coefficient measure chosen for each compact set and each tolerance, and it is proved through targets with a spectral density; the global weighted statements above are stronger in topology but do not identify the coefficients. Conversely, the explicit coefficient W ρ G W_{\rho}G W ρ G , the Plancherel identity, and the finite-width rate of this paper have no counterpart in these existence theorems, and the admissible analysis filters form a family, each giving a different representation of the same target.
Neural operators: universality, rates, and complexity
Universal approximation for DeepONets [64 , 67 ] , Fourier neural operators [53 ] , PCA-Net [10 , 61 ] , and the general neural-operator framework [54 ] is reviewed in Kovachki et al. [56 ] . Lanthaler et al. [65 ] show that nonlocality and nonlinearity together imply universality, with an averaging neural operator as the minimal example; Gödeke and Fernsel [35 ] prove a compact-open universality theorem for encoder–decoder architectures between normed or metric spaces and distinguish architectures that can be chosen independently of the compact set; Zappala and Bagherian [102 ] treat transformers and neural integral operators between Banach spaces; and Chen et al. [18 ] obtain universality for continuous-depth neural flows.
Quantitative results for solution operators of partial differential equations include the error estimates of Lanthaler et al. [64 ] and, for nonlinear parabolic equations, the Picard-iteration construction of Furuya et al. [31 ] , whose Theorem 1 achieves accuracy ε \varepsilon ε on a ball of initial data with depth O ( ( log ε − 1 ) 2 ) O((\log\varepsilon^{-1})^2) O (( log ε − 1 ) 2 ) and width O ( ε − 1 ( log ε − 1 ) 2 ) O(\varepsilon^{-1}(\log\varepsilon^{-1})^2) O ( ε − 1 ( log ε − 1 ) 2 ) , the nonlinearity being approximated as a function of one variable; Taniguchi and Sonoda [92 ] derive generalization bounds for the corresponding Picard-type learning problem in which the entropy term does not grow with the Picard depth. These rates exploit the structure of the equation. For operator classes defined by regularity alone, lower bounds show that no such rates exist: Lanthaler and Stuart [63 ] establish a curse of parametric complexity for C r C^r C r and Lipschitz operators that applies to PCA-Net, DeepONet, and FNO, Lanthaler [62 ] gives metric-entropy lower bounds that hold for every activation, Kovachki et al. [55 ] prove a corresponding curse of data complexity through n n n -widths, and Adcock et al. [2 ] show that the minimax risk for Lipschitz operators between Hilbert spaces cannot decay algebraically in the sample size, Hölder smoothness giving no improvement. Kratsios et al. [58 ] reduce the active complexity by routing inputs to one of several experts. In this paper, the rate is consistent with these lower bounds because it applies to targets whose ridgelet coefficient has finite total variation and moments, a Barron-type class determined by smoothness of the spectral density along lines through the origin (regularity along rays), and not to all Lipschitz operators.
Summary tables
Tables 2 and 3 list universal approximation theorems with infinite-dimensional inputs, and tables 4 to 6 list quantitative upper bounds and lower bounds, each with the class of targets, the form of the approximant, and the norm or topology in which approximation is measured. “One hidden layer” means a finite sum of neurons with trained output weights; “random features” means that the hidden parameters are sampled and only the readout is chosen; cc means uniform convergence on compact subsets of the input space.
Finite-width bounds and random features
Finite-width bounds by sampling a parameter distribution follow Barron [5 ] and the variation-norm viewpoint of Bach [4 ] and E, Ma, and Wu [28 ] . Gonon et al. [36 ] prove random ReLU approximation on separable Hilbert input spaces with an expected squared L 2 L^2 L 2 error of order N − 1 N^{-1} N − 1 under a Fourier–Stieltjes representation with moment conditions and absolute continuity with respect to a sampling measure, and they treat infinite input histories in Gonon et al. [37 ] . Korolev [52 ] studies two-layer
Table 2. Universal approximation with infinite-dimensional inputs: functionals, Fréchet and topological inputs, weighted spaces, and signatures.
Table 3. Universal approximation with infinite-dimensional inputs: neural operators, derivatives, and this paper.
Table 4. Quantitative upper bounds for functionals and for networks with Hilbert or Banach inputs, as rates in the width N N N .
Table 5. Quantitative upper bounds for neural operators and surrogates of operators.
Table 6. Lower bounds: no algebraic rate exists for the class.
networks between Banach spaces with separable preduals whose nonlinearity is the lattice positive part, proving direct and inverse theorems with Monte Carlo rates for the associated variation space. Neufeld and Schmocker [78 ] prove universality and rates for Banach-space-valued random feature models in Bochner norm; their rates for random networks in weighted Sobolev spaces on R m \mathbb{R}^{m} R m are derived from the finite-dimensional ridgelet reconstruction with an admissible filter pair, which is the finite-dimensional counterpart of the sampling argument used here. Further, Yang and Shi [99 ] obtain dimension-independent bounds for encoder–decoder neural operators in Bochner L q L^{q} L q . Kernel and random-feature surrogates for operators are developed by Nelsen and Stuart [76 , 77 ] and Batlle et al. [6 ] . In this paper, the sampled network is a random feature model whose sampling density is proportional to the modulus of the explicit ridgelet coefficient; the additional content is that the coefficient is characterized by an analysis operator with a Plancherel identity and a closed-range projection, that the rate is uniform on compact sets for Lipschitz activations, and that it extends to Hilbert-valued outputs in a Bochner norm. Continuous frames and coorbit discretization [30 , 41 ] provide the general framework in which the coefficient space could be studied further; we do not assume their hypotheses.
Gaussian measures, functional data, and statistical learning
Gaussian measures on Hilbert spaces and their Cameron–Martin spaces are treated in Bogachev [12 ] , Da Prato and Zabczyk [24 ] , and Prévôt and Röckner [80 ] . The same objects appear in functional data analysis, where observations are modeled as realizations of a Hilbert-space-valued process and expanded in Karhunen–Loève form [29 , 97 ] , and in Gaussian process regression on function spaces [84 ] , whose posterior contraction rates depend on the position of the target relative to the Cameron–Martin space [95 ] ; the equivalence between Gaussian process regression and kernel ridge regression is reviewed in Kanagawa et al. [51 ] . Characteristic kernels on separable Hilbert spaces and the explicit reproducing kernel Hilbert space of Gauss-type kernels are constructed in Wynne and Duncan [98 ] , and operator-valued kernels for functional responses in Kadri et al. [50 ] . Neural networks with functional inputs have been developed in statistics by Rossi and Conan-Guez [81 ] , Yao et al. [101 ] , and Thind et al. [93 ] . On the learning side, convergence rates for linear operators from noisy data are obtained in de Hoop et al. [26 ] , minimax rates for nonlinear operators in Adcock et al. [2 ] , and near-optimal learning of Gaussian Sobolev operators through Hermite expansions and principal component analysis in Adcock et al. [3 ] . In this paper, we develop an approximation theory, not a learning theory: the input measure μ Q \mu_{Q} μ Q is the Gaussian measure of the functional data model, the spectral density G G G is a Fourier-side description of the target with respect to this measure, and the sampled network is a Monte Carlo construction rather than an estimator from data. Statistical rates are not claimed.
Declarations
Funding. This work was partially supported by JSPS KAKENHI 24K21316, 25H01453, 25K24910, JST BOOST JPMJBY24F2, JST CREST JPMJCR2515, JPMJCR24Q1, JST ERATO JPMJER2601, JST ARiSE JPMJAR2633.
Competing interests. The authors declare that they have no competing interests.
Data availability. No external datasets were used. The numerical results supporting § 8 , including the data in Table 1 and Figure 1 , are provided as results.json and figure files in the supplementary archive python-code.zip. The computational setup is described in Appendix § 14 and in the archive’s README.
Table 1. Mean supremum error on K K K of the sampled networks at N = 4 N=4 N = 4 and N = 4096 N=4096 N = 4096 , and the fitted slope of the error against N N N in logarithmic coordinates, for each truncation dimension d d d .
Figure 1. Mean supremum error on K K K against the width N N N for Experiments 1 (upper left), 2 (upper right), and 3(a) (bottom), for several truncation dimensions d d d ; the dashed line is proportional to N − 1 / 2 N^{-1/2} N − 1/2 .
Code availability. The Lean formalization snapshot is supplied in lean-code.zip; the Python code for § 8 is supplied in python-code.zip. Both supplementary archives include instructions for reproducing the results.