All models are wrong, but some are useful.
George E.P. Box
Deep learning models [1 , 6 , 7 , 8 , 15 ] have a layered structure. A feedforward deep neural network with L L L layers can be defined as a function
f : R n 0 → R n L f : \mathbb{R}^{n_0} \to\mathbb{R}^{n_L} f : R n 0 → R n L
constructed by composing affine transformations and nonlinear activations:
f ( x ) = f ( L ) ∘ f ( L − 1 ) ∘ ⋯ ∘ f ( 1 ) ( x ) , f(x) = f^{(L)} \circ f^{(L-1)} \circ\cdots\circ f^{(1)}(x), f ( x ) = f ( L ) ∘ f ( L − 1 ) ∘ ⋯ ∘ f ( 1 ) ( x ) ,
For each layer l = 1 , … , L l = 1, \ldots, L l = 1 , … , L ,
f ( l ) ( x ) = σ ( l ) ( W ( l ) x + b ( l ) ) . f^{(l)}(x) = \sigma^{(l)}\left(W^{(l)}x + b^{(l)}\right). f ( l ) ( x ) = σ ( l ) ( W ( l ) x + b ( l ) ) .
where x ∈ R n 0 x \in\mathbb{R}^{n_0} x ∈ R n 0 is an input vector, n l n_l n l is the number of units (width) in layer l l l , W ( l ) ∈ R n l × n l − 1 W^{(l)} \in\mathbb{R}^{n_l \times n_{l-1}} W ( l ) ∈ R n l × n l − 1 is the weight matrix of layer l l l , b ( l ) ∈ R n l b^{(l)} \in\mathbb{R}^{n_l} b ( l ) ∈ R n l is the bias vector of layer l l l , and σ ( l ) : R → R \sigma^{(l)} : \mathbb{R} \to\mathbb{R} σ ( l ) : R → R is the activation function of layer l l l , which is applied element-wise.
The depth of the network is L L L , and the parameters of the deep learning model are the collection of all weights and biases:
θ = { W ( l ) , b ( l ) } l = 1 L . \theta= \left\{W^{(l)}, b^{(l)}\right\}_{l=1}^{L}. θ = { W ( l ) , b ( l ) } l = 1 L .
Training a deep learning model consists of finding θ \theta θ that minimizes a loss function L ( f ( x ) , y ) \mathcal{L}(f(x), y) L ( f ( x ) , y ) over a dataset { ( x i , y i ) } \{(x_i, y_i)\} {( x i , y i )} .
Formally, a deep learning model is just a composition of affine maps and nonlinearities parameterized by weights and biases, which are optimized via loss minimization. As we will see, multilayered neural networks are universal approximators [10 , 12 , 20 , 28 , 30 ] .
Universal Approximators
Let us consider the problem of representing general functions of an n n n -dimensional real variable, x ∈ R n x \in\mathbb{R}^n x ∈ R n , by finite linear combinations:
f : R n → R f : \mathbb{R}^n \to\mathbb{R} f : R n → R
f ( x ) = ∑ j = 1 h α j σ ( ∑ i = 1 n w i j x i + b j ) f(x) = \sum_{j=1}^{h} \alpha_j \sigma\left(\sum_{i=1}^{n} w_{ij}x_i + b_j\right) f ( x ) = j = 1 ∑ h α j σ ( i = 1 ∑ n w ij x i + b j )
which corresponds to a feed-forward neural network with n n n inputs, h h h hidden neurons, and a single output y = f ( x ) y = f(x) y = f ( x ) . W = [ w i j ] W = [w_{ij}] W = [ w ij ] is the n × h n \times h n × h weight matrix for the hidden layer, ( b j ) (b_j) ( b j ) is the h × 1 h \times1 h × 1 vector of biases for the hidden neurons, and ( α j ) (\alpha_j) ( α j ) is the h × 1 h \times1 h × 1 weight vector of the output neuron. Finally, σ \sigma σ is the univariate activation function of the hidden units.
Cybenko's Universal Approximation Theorem
If σ ( z ) \sigma(z) σ ( z ) is a sigmoidal function, i.e. lim z → + ∞ σ ( z ) = 1 \lim_{z\to+\infty}\sigma(z) = 1 lim z → + ∞ σ ( z ) = 1 and lim z → − ∞ σ ( z ) = 0 \lim_{z\to-\infty}\sigma(z) = 0 lim z → − ∞ σ ( z ) = 0 (as in the logistic function), George Cybenko [10 ] proved that functions defined as f ( x ) f(x) f ( x ) are dense in the space of continuous functions on the unit cube if σ \sigma σ is any continuous sigmoidal function, such as the logistic function. Cybenko, then at the University of Illinois at Urbana-Champaign, proved the universal approximation theorem for artificial neural networks with sigmoid activation functions:
Figure 1. Main results discussed in this technical report, according to Google Nano Banana.
Theorem I.1: Universal approximation theorem for artificial neural networks with sigmoid activation functions Let σ \sigma σ be any continuous discriminatory function. The finite sums of the form
G ( x ) = ∑ j = 1 h α j σ ( ∑ i = 1 n w i j x i + b j ) G(x) = \sum_{j=1}^{h} \alpha_j \sigma\left(\sum_{i=1}^{n} w_{ij}x_i+b_j\right) G ( x ) = j = 1 ∑ h α j σ ( i = 1 ∑ n w ij x i + b j ) are dense in C ( I n ) C(I_n) C ( I n ) . In other words, given any f ∈ C ( I n ) f \in C(I_n) f ∈ C ( I n ) and ϵ > 0 \epsilon> 0 ϵ > 0 , there is a sum, G ( x ) G(x) G ( x ) , of the above form, for which
∣ G ( x ) − f ( x ) ∣ < ϵ ∀ x ∈ I n |G(x)-f(x)| < \epsilon\quad\forall x \in I_n ∣ G ( x ) − f ( x ) ∣ < ϵ ∀ x ∈ I n
I n I_n I n is the n-dimensional unit hypercube, whereas C ( I n ) C(I_n) C ( I n ) is the space of continuous functions in I n I_n I n . The theorem above states that finite superpositions of a fixed, univariate function that is discriminatory can uniformly approximate any continuous function of n n n real variables with support in the unit hypercube.
Given that sigmoid activation functions are continuous and discriminatory (lemma B.1), any continuous function can be uniformly approximated by a continuous neural network having only one hidden sigmoid layer with an arbitrary continuous sigmoidal nonlinearity. Therefore, a feed-forward neural network with a single hidden sigmoid layer, acting as a universal approximator, can represent any continuous function, provided that we have enough hidden units.
Funahashi's Universal Approximation Theorem
Independently from Cybenko, Ken-ichi Funahashi [12 ] also proved that any continuous mapping can be approximately realized multilayer neural networks with at least one hidden layer with sigmoid activation functions.
Theorem I.2: Universal approximation theorem for artificial neural networks with bounded, monotonically increasing activation functions Let ϕ ( x ) \phi(x) ϕ ( x ) be a non-constant, bounded, and monotonically increasing continuous function. Let K K K be a compact subset (bounded closed subset) of R n \mathbb{R}^{n} R n and f f f a real valued continuous function on K K K . Then, for an arbitrary ϵ > 0 \epsilon> 0 ϵ > 0 , there exists an integer N N N and real constants α i \alpha_i α i , b i b_i b i , and w i j w_{ij} w ij such that
f ~ ( x ) = ∑ j = 1 h α j ϕ ( ∑ i = 1 n w i j x i + b j ) \widetilde{f}(x) = \sum_{j=1}^{h} \alpha_j \phi\left(\sum_{i=1}^{n} w_{ij}x_i+b_j\right) f ( x ) = j = 1 ∑ h α j ϕ ( i = 1 ∑ n w ij x i + b j ) satisfies max k ∈ K ∣ f ( x ) − f ~ ( x ) ∣ < ϵ \max_{k \in K}|f(x)-\widetilde{f}(x)| < \epsilon max k ∈ K ∣ f ( x ) − f ( x ) ∣ < ϵ
The above theorem proves that, for an arbitrary ϵ > 0 \epsilon> 0 ϵ > 0 , there exists a three-layer network, with a ϕ ( x ) \phi(x) ϕ ( x ) activation function in the hidden layer and a linear output layer, whose output function f ~ ( x ) \widetilde{f}(x) f ( x ) approximates f ( x ) f(x) f ( x ) as close as needed. Obviously, the sigmoid function σ ( z ) = 1 / ( 1 − e − z ) \sigma(z)=1/(1-e^{-z}) σ ( z ) = 1/ ( 1 − e − z ) typically used in neural networks satisfies the requirements of ϕ ( x ) \phi(x) ϕ ( x ) , i.e. non-constant, bounded, and monotonically increasing.
Remark. Funahashi’s theorem uses bounded functions. Rectified linear units, R e L U ( z ) = max { 0 , z } ReLU(z)=\max\{0,z\} R e LU ( z ) = max { 0 , z } , are not bounded. However, the linear combination ϕ ( z ) = \phi(z)= ϕ ( z ) = R e L U ( z − 1 ) − R e L U ( z ) ReLU(z-1)-ReLU(z) R e LU ( z − 1 ) − R e LU ( z ) falls within the scope of Funahashi’s theorem, so its conclusion is also valid for ϕ ( z ) = R e L U ( z ) \phi(z)=ReLU(z) ϕ ( z ) = R e LU ( z ) .
Table 1. Common activation functions in neural networks.
While Cybenko’s and Funahashi’s techniques differs, both support the same conclusion for sigmoidal activations. Funahashi’s proof guarantees the existence of parameters approximating arbitrary spectra, paralleling Cybenko’s measure-separation proof. Both are non-constructive in the sense of parameter selection.
Alternative Activation Functions
Apart from sigmoid functions, such as the logistic function, and the hyperbolic tangent, other activation functions are common in neural networks (see Table I). Cybenko and Funahashi proved the universal approximation capabilities of multi-layered neural networks when using sigmoid activation functions in the hidden layer. What happens with other activation functions?
Hornik's Approximation Theorems
The Hornik theorems [19 , 20 ] formalize the universal approximation capabilities of multilayer feedforward neural networks for bounded and continuous activation functions:
Theorem II.1: Universal approximation theorem for artificial neural networks with bounded and continuous activation functions Let σ : R → R \sigma:\mathbb{R}\to\mathbb{R} σ : R → R be a bounded, non-constant, and continuous activation function. Then, for any compact set K ⊂ R n K\subset\mathbb{R}^n K ⊂ R n , the set of functions of the form
H ( x ) = ∑ j = 1 h α j σ ( ∑ i = 1 n w j i x i + b j ) H(x)=\sum_{j=1}^{h}\alpha_j\sigma\left(\sum_{i=1}^{n}w_{ji}x_i+b_j\right) H ( x ) = j = 1 ∑ h α j σ ( i = 1 ∑ n w j i x i + b j ) is dense in C ( K ) C(K) C ( K ) , the space of continuous functions on K K K with the uniform norm.
Hornik’s results [20 ] build on the Stone–Weierstrass theorem, which states that certain algebraic structures (like neural networks with nonlinear activations) can approximate any continuous function. The neural network acts as a flexible basis expansion, where the hidden units serve as nonlinear basis functions.
A feedforward neural network with a single hidden layer using a non-constant, bounded, and continuous activation function can approximate any continuous function on a compact subset K ⊂ R n K\subset\mathbb{R}^n K ⊂ R n to arbitrary precision. Not only can such networks approximate functions, but they can also approximate their derivatives [19 ] :
Theorem II.2: Universal approximation theorem for derivatives Let σ : R → R \sigma:\mathbb{R}\to\mathbb{R} σ : R → R be a bounded, non-constant, and continuous activation function. Then, for any f ∈ C k ( K ) f\in C^k(K) f ∈ C k ( K ) , the space of k k k -times continuously differentiable functions on compact K ⊂ R n K\subset\mathbb{R}^n K ⊂ R n , there exists a neural network f ~ N \tilde{f}_N f ~ N such that
∥ f ( x ) − f ~ N ( x ) ∥ C k ( K ) < ϵ \lVert f(x)-\tilde{f}_N(x)\rVert_{C^k(K)}<\epsilon ∥ f ( x ) − f ~ N ( x ) ∥ C k ( K ) < ϵ
for any ϵ > 0 \epsilon>0 ϵ > 0 .
Leshno's Approximation Theorem
Hornik’s approximation theorems are sufficient: they tells us when universal approximation is possible. Leshno’s approximation theorem is necessary and sufficient: it tells us exactly which activation functions work and which don’t. So while Hornik opened the door to understanding neural networks as universal approximators, Leshno sharpened the criteria and closed the door on polynomial activations [28 ] :
Theorem II.3: Leshno’s theorem: Universal approximation theorem for artificial neural networks with non-polynomial activation functions Let σ : R → R \sigma:\mathbb{R}\to\mathbb{R} σ : R → R be a continuous activation function . Then the set of functions defined by
L ( x ) = ∑ j = 1 h α j σ ( ∑ i = 1 n w i j x i + b j ) L(x)=\sum_{j=1}^{h}\alpha_j\sigma\left(\sum_{i=1}^{n}w_{ij}x_i+b_j\right) L ( x ) = j = 1 ∑ h α j σ ( i = 1 ∑ n w ij x i + b j ) is dense in C ( K ) C(K) C ( K ) for every compact K ⊂ R n K\subset\mathbb{R}^{n} K ⊂ R n if and only if σ \sigma σ is not a polynomial almost everywhere.
Leshno used algebraic and functional analysis to prove non-polynomiality is essential. The universal approximation property holds for a wide class of activation functions, not just sigmoids. The key requirement is that the activation function must be non-polynomial. If σ \sigma σ is a polynomial (e.g., σ ( x ) = x 2 \sigma(x)=x^2 σ ( x ) = x 2 ), then the multilayered network cannot universally approximate. If σ \sigma σ is non-polynomial, even if unbounded or discontinuous in places (e.g., ReLU), it can be a universal approximator.
As a consequence of Leshno’s theorem, ReLU, softplus, and many others activation functions qualify. Since they satisfy the non-polynomial constraint, they can be used as universal approximators.
Barron’s Approximation Error Bounds
Andrew Barron, also from the University of Illinois at Urbana Champaign, provided explicit convergence rates for approximation in L 2 L^2 L 2 , unlike previous results, which are existence theorems.
Theorem II.4: Barron’s Theorem Let f : R n → R f:\mathbb{R}^{n}\to\mathbb{R} f : R n → R be a function with Fourier transform F \mathcal{F} F such that its Barron norm is finite, i.e.
∫ R n ∥ ω ∥ ∣ F ( ω ) ∣ d ω < ∞ . \int_{\mathbb{R}^{n}}\|\omega\||\mathcal{F}(\omega)|\,d\omega<\infty. ∫ R n ∥ ω ∥∣ F ( ω ) ∣ d ω < ∞. Then there exists a single-hidden-layer neural network f H f_H f H with H H H hidden units such that
∥ f − f H ∥ L 2 ≤ C H , \|f-f_H\|_{L^2}\leq\frac{C}{\sqrt{H}}, ∥ f − f H ∥ L 2 ≤ H C , where C C C depends on the Barron norm of f f f .
Cybenko, Funahashi, Hornik, and Leshno proved that neural networks are universal approximators, but did not quantify how fast the approximation improves with more neurons. Barron [5 ] shows that, for a specific class of functions (Barron space), the approximation error decays as O ( 1 / H ) O(1/\sqrt{H}) O ( 1/ H ) ,
Leshno’s condition (non-polynomial activation) is still necessary for Barron’s bounds to hold (polynomial activations cannot generate the required Fourier richness). The Barron space is a strict subset of L 2 L^2 L 2 , characterized by Fourier decay and low-frequency dominance. Functions in Barron space can be approximated efficiently, even in high dimensions, breaking the curse of dimensionality.
The Power of Deep Networks
The universal approximation properties of neural networks are powerful, but focus only on existence. An important question is “how many neurons are required to yield an approximation of a given quality?” Funahashi [12 ] already suggested that “it should be studied theoretically studied in the future that the possibility of k > 3 k>3 k > 3 -layer networks can realize a given mapping with less costs (number of units or connections) than three-layer networks, within error ϵ \epsilon ϵ .”
Guido Montúfar [30 ] studied the expressive power of deep feedforward neural networks, particularly focusing on piecewise linear activations like ReLU.
Consider a feedforward neural network with input dimension n 0 n_0 n 0 and L L L hidden layers with widths n 1 … n L n_1\ldots n_L n 1 … n L using the ReLU activation function (i.e. σ ( z ) = max { 0 , z } \sigma(z)=\max\{0,z\} σ ( z ) = max { 0 , z } ). The network computes a piecewise linear function f : R n 0 → R f:\mathbb{R}^{n_0}\to\mathbb{R} f : R n 0 → R , partitioning the input space into linear regions where f f f is affine. (i.e. linear regions correspond to distinct affine pieces of the function computed by the network).
Theorem III.1: Expressivity via Linear Regions Let N N N be a feedforward ReLU network with input dimension n 0 n_0 n 0 , L L L hidden layers, and widths n 1 … n L n_1\ldots n_L n 1 … n L , n l ≥ n 0 ∀ l n_l\geq n_0\ \forall l n l ≥ n 0 ∀ l . Then, the maximal number of linear regions of the function computed by the neural network N N N is lower bounded by
( ∏ l = 1 L − 1 ⌊ n l n 0 ⌋ ) ∑ j = 0 n 0 ( n L j ) \left(\prod_{l=1}^{L-1}\left\lfloor\frac{n_l}{n_0}\right\rfloor\right)\sum_{j=0}^{n_0}\binom{n_L}{j} ( l = 1 ∏ L − 1 ⌊ n 0 n l ⌋ ) j = 0 ∑ n 0 ( j n L )
Assuming n 0 ∈ O ( 1 ) n_0\in O(1) n 0 ∈ O ( 1 ) and n i = n ∀ i ≥ 1 n_i=n\ \forall i\geq1 n i = n ∀ i ≥ 1 , the number of regions of a single hidden layer model with L n Ln L n hidden units behaves as O ( L n 0 n n 0 ) O(L^{n_0}n^{n_0}) O ( L n 0 n n 0 ) . A deep ReLU network can compute functions that have Ω ( ( n / n 0 ) ( L − 1 ) n 0 n n 0 ) \Omega((n/n_0)^{(L-1)n_0}n^{n_0}) Ω (( n / n 0 ) ( L − 1 ) n 0 n n 0 ) linear regions.
Montúfar’s lower bound shows that depth increases expressivity exponentially, while width contributes polynomially. The number of regions grows exponentially with depth, confirming that, using the same number of parameters, deep networks are exponentially more expressive than shallow ones. The number of linear regions of deep models grows exponentially in L L L and polynomially in n n n , which is much faster than that of shallow models with n L nL n L hidden units ((n / n 0 ) ( L − 1 ) n 0 n/n_0)^{(L-1)n_0} n / n 0 ) ( L − 1 ) n 0 vs. L n 0 L^{n_0} L n 0 ).
Depth is not just convenient, it is formally necessary for efficient representation of certain functions: some functions can be represented efficiently by deep networks but require exponentially more units in shallow networks [38 ] .
[36 ] provides tighter upper and lower bounds on the number of linear regions achievable by ReLU networks. Montúfar quantified expressivity via combinatorial complexity (number of linear regions), but there are alternative measures of expressivity beyond linear region counting [34 ] . Depth increases expressivity in a way consistent with Montúfar’s exponential linear region separation.
Conclusion
Universal approximation theorems (Cybenko, Funahashi, Hornik & Leshno) provide the mathematical justification for why neural networks are powerful enough to model complex, nonlinear phenomena. Shallow networks can approximate any continuous function on compact sets. Leshno’s theorem clarified that non-polynomial activations are necessary for universality, which explains why ReLU, logistic, hyperbolic tangent, and other common activations work (but pure polynomials do not).
However, approximation theorems provide no efficiency, training, nor generalization guarantees. They do not indicate how many neurons are needed for a given accuracy, whether gradient descent will find the right parameters, or whether networks trained on finite data will generalize well.
Barron provided quantitative error bounds for particular function classes (approximation rates for functions with bounded Fourier norm), whereas Montúfar was the first to show that expressivity grows exponentially with depth (in terms of the number of linear regions). Deep networks are strictly more expressive than shallow ones with the same number of parameters. Depth is fundamentally more powerful than width, providing a rigorous mathematical explanation for the use of deep learning models.
Mathematical Background
Definition A.1. The n n n -dimensional unit hypercube is I n = [ 0 , 1 ] n I_n = [0,1]^n I n = [ 0 , 1 ] n . More precisely,
I n = [ 0 , 1 ] n = { ( x 1 , … , x n ) ∈ R n ∣ 0 ≤ x i ≤ 1 , i = 1 , … , n } I_n = [0,1]^n
= \{(x_1,\ldots,x_n) \in\mathbb{R}^n \mid0 \le x_i \le1,\ i=1,\ldots,n\} I n = [ 0 , 1 ] n = {( x 1 , … , x n ) ∈ R n ∣ 0 ≤ x i ≤ 1 , i = 1 , … , n }
Definition A.2. Let K \mathbb{K} K denote the field of scalars, typically R \mathbb{R} R or C \mathbb{C} C . The space of continuous functions on I n I_n I n is
C ( I n ) = { f : I n → K ∣ f is continuous } . C(I_n) = \{f:I_n \to\mathbb{K} \mid f\text{ is continuous}\}. C ( I n ) = { f : I n → K ∣ f is continuous } . Under point-wise addition and scalar multiplication, C ( I n ) C(I_n) C ( I n ) is a vector space.
Measures
Remark. C ( R n ) C(\mathbb{R}^n) C ( R n ) is the set of all continuous functions on R n \mathbb{R}^n R n , whereas C ∞ ( R n ) C^\infty(\mathbb{R}^n) C ∞ ( R n ) is the set of all infinitely differentiable functions on R n \mathbb{R}^n R n .
In computer science, a functional usually refers to a higher-order function, i.e. a function that takes one or more functions as arguments or returns a function as a result.
In mathematics, a functional is a map whose codomain is the field of scalars,
F : X → K , F:X \to\mathbb{K}, F : X → K ,
where X X X is typically a space of functions. The input elements of X X X are often themselves functions, which explains the similarity between the mathematical and computer-science uses of the term.
Definition A.3. Let V V V and W W W be vector spaces over the same field K \mathbb{K} K . A map
is said to be linear if for all f , g ∈ V f,g \in V f , g ∈ V and all scalars α , β ∈ K \alpha,\beta\in\mathbb{K} α , β ∈ K ,
L ( α f + β g ) = α L ( f ) + β L ( g ) . L(\alpha f+\beta g)=\alpha L(f)+\beta L(g). L ( α f + β g ) = α L ( f ) + β L ( g ) .
A functional need not be linear. In linear algebra, a linear form (also known as a linear functional, a one-form, or a covector) is a linear map from a vector space to its field of scalars. In functional analysis, it refers to a mapping from a space X X X into the field of real or complex numbers. A linear functional on C ( I n ) C(I_n) C ( I n ) is a scalar-valued linear map defined on C ( I n ) C(I_n) C ( I n ) , in concordance with the computer science interpretation of functional.
Definition A.4. A linear functional on C ( I n ) C(I_n) C ( I n ) is a linear map
L : C ( I n ) → K . L:C(I_n) \to\mathbb{K}. L : C ( I n ) → K . That is,
L ( α f + β g ) = α L ( f ) + β L ( g ) L(\alpha f+\beta g)=\alpha L(f)+\beta L(g) L ( α f + β g ) = α L ( f ) + β L ( g ) for all f , g ∈ C ( I n ) f,g \in C(I_n) f , g ∈ C ( I n ) and all α , β ∈ K \alpha,\beta\in\mathbb{K} α , β ∈ K .
A measure is a generalization and formalization of geometrical measures (length, area, volume) and other common notions, such as magnitude, mass, and probability of events. In mathematical analysis and in probability theory, a σ \sigma σ -algebra is part of the formalism for defining sets that can be measured (also σ \sigma σ -field, where the σ \sigma σ comes from the German “Summe,” meaning “sum”):
Definition A.5. A σ \sigma σ -algebra on a set X X X is a nonempty collection Σ \Sigma Σ of subsets of X X X closed under complement, countable unions, and countable intersections.1
Σ ⊂ P ( X ) \Sigma\subset\mathcal{P}(X) Σ ⊂ P ( X ) such that
X ∈ Σ X \in\Sigma X ∈ Σ ,
E ∈ Σ ⇒ E c ∈ Σ E \in\Sigma\Rightarrow E^{c} \in\Sigma E ∈ Σ ⇒ E c ∈ Σ ,
If E 1 … E n ∈ Σ E_1 \ldots E_n \in\Sigma E 1 … E n ∈ Σ , then ⋃ k E k ∈ Σ \bigcup_k E_k \in\Sigma ⋃ k E k ∈ Σ .
Countable intersections then follow automatically.
Definition A.6. The ordered pair ( X , Σ ) (X,\Sigma) ( X , Σ ) is called a measurable space.
Definition A.7. Any element of the chosen σ \sigma σ -algebra Σ \Sigma Σ is a measurable set.
Definition A.8. Let X X X be a set and Σ \Sigma Σ a σ \sigma σ -algebra over X X X , defining subsets of X X X that are measurable. A measure μ : Σ → [ 0 , + ∞ ) \mu:\Sigma\to[0,+\infty) μ : Σ → [ 0 , + ∞ ) is a set function from the σ \sigma σ -algebra that satisfies the following conditions:
μ ( ∅ ) = 0 \mu(\varnothing)=0 μ ( ∅ ) = 0 .
Non-negativity: For all E ∈ Σ E \in\Sigma E ∈ Σ , μ ( E ) ≥ 0 \mu(E) \ge0 μ ( E ) ≥ 0 .
Countable additivity (or σ \sigma σ -additivity): For all countable collections { E k } k = 1 ∞ \{E_k\}_{k=1}^{\infty} { E k } k = 1 ∞ of pairwise disjoint sets in Σ \Sigma Σ , μ ( ⋃ k = 1 ∞ E k ) = ∑ k = 1 ∞ μ ( E k ) \mu\left(\bigcup_{k=1}^{\infty}E_k\right)=\sum_{k=1}^{\infty}\mu(E_k) μ ( ⋃ k = 1 ∞ E k ) = ∑ k = 1 ∞ μ ( E k ) .
Definition A.9. Borel set. Let X X X be a topological space. The Borel σ \sigma σ -algebra of X X X , denoted by B ( X ) \mathcal{B}(X) B ( X ) , is the smallest σ \sigma σ -algebra containing all open subsets of X X X . The elements of B ( X ) \mathcal{B}(X) B ( X ) are called Borel sets.
Definition A.10. Borel measure. Let X X X be a topological space and let B ( X ) \mathcal{B}(X) B ( X ) denote its Borel σ \sigma σ -algebra. A Borel measure on X X X is a measure μ : B ( X ) → [ 0 , ∞ ] \mu:\mathcal{B}(X)\to[0,\infty] μ : B ( X ) → [ 0 , ∞ ] defined on the Borel σ \sigma σ -algebra.
Measures on topological spaces are usually defined on the Borel σ \sigma σ -algebra because open and closed sets have geometric meaning, the Borel σ \sigma σ -algebra is rich enough for most applications (e.g. analysis, probability, and statistics), and continuous functions interact naturally with Borel sets (the preimage of a Borel set under a continuous function is again a Borel set).
Definition A.11. Signed measure. Let ( X , Σ ) (X,\Sigma) ( X , Σ ) be a measurable space. A signed measure is a countably additive set function μ : Σ → [ − ∞ , ∞ ] \mu:\Sigma\to[-\infty,\infty] μ : Σ → [ − ∞ , ∞ ] such that μ ( ∅ ) = 0 \mu(\varnothing)=0 μ ( ∅ ) = 0 and at most one of the values + ∞ +\infty + ∞ and − ∞ -\infty − ∞ is attained. Countable additivity means that for every sequence { E k } k = 1 ∞ \{E_k\}_{k=1}^{\infty} { E k } k = 1 ∞ of pairwise disjoint measurable sets,
μ ( ⋃ k = 1 ∞ E k ) = ∑ k = 1 ∞ μ ( E k ) . \mu\left(\bigcup_{k=1}^{\infty}E_k\right)=\sum_{k=1}^{\infty}\mu(E_k). μ ( k = 1 ⋃ ∞ E k ) = k = 1 ∑ ∞ μ ( E k ) . Unlike ordinary measures, a signed measure may take both positive and negative values.
In mathematics, specifically general topology, compactness is a property that seeks to generalize the notion of a closed and bounded subset of Euclidean space. Roughly speaking, compact sets are those that can be covered by arbitrary collections of open sets, while still admitting a finite subcover.2
Definition A.12. Regular Borel measure. Let X X X be a topological space. A Borel measure μ \mu μ on X X X is regular if every Borel set E ∈ B ( X ) E \in\mathcal{B}(X) E ∈ B ( X ) can be approximated from inside by compact sets and from outside by open sets:
μ ( E ) = sup { μ ( K ) : K ⊆ E , K compact } , \mu(E)=\sup\{\mu(K):K\subseteq E,\ K\text{ compact}\}, μ ( E ) = sup { μ ( K ) : K ⊆ E , K compact } , and
μ ( E ) = inf { μ ( U ) : E ⊆ U , U open } . \mu(E)=\inf\{\mu(U):E\subseteq U,\ U\text{ open}\}. μ ( E ) = inf { μ ( U ) : E ⊆ U , U open } . The first property is called inner regularity and the second property is called outer regularity.
Definition A.13. The space of finite signed regular Borel measures on the compact space I n = [ 0 , 1 ] n I_n=[0,1]^n I n = [ 0 , 1 ] n is denoted by M ( I n ) M(I_n) M ( I n ) .
That is, M ( I n ) = { μ : B ( I n ) → R } M(I_n)=\{\mu:\mathcal{B}(I_n)\to\mathbb{R}\} M ( I n ) = { μ : B ( I n ) → R } , where μ \mu μ is a finite signed regular Borel measure and finiteness means ∣ μ ∣ ( I n ) < ∞ |\mu|(I_n)<\infty ∣ μ ∣ ( I n ) < ∞ , with ∣ μ ∣ |\mu| ∣ μ ∣ denoting the total variation measure of μ \mu μ .
Since the measure is finite, i.e. μ ( I n ) < ∞ \mu(I_n)<\infty μ ( I n ) < ∞ , local finiteness is automatic on compact subsets: μ ( K ) < ∞ \mu(K)<\infty μ ( K ) < ∞ for every compact K K K , since μ ( I n ) < ∞ \mu(I_n)<\infty μ ( I n ) < ∞ and every compact set satisfies K ⊆ I n K\subseteq I_n K ⊆ I n , so μ ( K ) ≤ μ ( I n ) < ∞ \mu(K)\leq\mu(I_n)<\infty μ ( K ) ≤ μ ( I n ) < ∞ .
Definition A.14. Measurable function. A measurable function f : X → R f:X\to\mathbb{R} f : X → R satisfies f − 1 ( B ) ∈ Σ f^{-1}(B)\in\Sigma f − 1 ( B ) ∈ Σ for every Borel set B ⊆ R B\subseteq\mathbb{R} B ⊆ R .
A function f : X → R f:X\to\mathbb{R} f : X → R is measurable if it is compatible with the underlying notion of measure, allowing quantities such as ∫ X f d μ \int_X f\,d\mu ∫ X f d μ to be defined whenever f f f is integrable.
Definition A.15: L L L -infinity, L ∞ ( X ) = L ∞ ( X , Σ , μ ) L^\infty(X)=L^\infty(X,\Sigma,\mu) L ∞ ( X ) = L ∞ ( X , Σ , μ ) , is the vector space of essentially bounded measurable functions with the essential supremum norm. L ∞ ( X ) L^\infty(X) L ∞ ( X ) depends on Σ \Sigma Σ and μ \mu μ , yet we omit them for notational convenience.
A function f ∈ L ∞ ( X ) f\in L^\infty(X) f ∈ L ∞ ( X ) is essentially bounded when there is a constant c < ∞ c<\infty c < ∞ such that ∣ f ( x ) ∣ ≤ c |f(x)|\le c ∣ f ( x ) ∣ ≤ c except on a set of measure zero.
The essential supremum norm3 is defined as ∥ f ∥ ∞ = inf { c ≥ 0 : ∣ f ( x ) ∣ ≤ c for almost every x ∈ X } \|f\|_\infty=\inf\{c\ge0: |f(x)|\le c\text{ for almost every }x\in X\} ∥ f ∥ ∞ = inf { c ≥ 0 : ∣ f ( x ) ∣ ≤ c for almost every x ∈ X } .
L ∞ ( X ) L^\infty(X) L ∞ ( X ) is a Banach space under the norm ∥ ⋅ ∥ ∞ \|\cdot\|_\infty ∥ ⋅ ∥ ∞ . The vector space L ∞ ( X ) L^\infty(X) L ∞ ( X ) consists of equivalence classes of essentially bounded measurable functions, where two functions are identified if they are equal almost everywhere (functions that differ only on a set of measure zero regarded as equivalent).
Definition A.16: A simple function is a measurable function that takes only finitely many values.
Theorem A.1: Simple functions are dense in L ∞ ( X ) L^\infty(X) L ∞ ( X ) with respect to the essential-supremum norm for any measure space ( X , Σ , μ ) (X,\Sigma,\mu) ( X , Σ , μ ) .
Proof. To prove that simple functions are dense in L ∞ ( X ) L^\infty(X) L ∞ ( X ) with respect to the essential-supremum norm, we need to show that for every essentially bounded measurable function f ∈ L ∞ ( X ) f\in L^\infty(X) f ∈ L ∞ ( X ) and every ε > 0 \varepsilon>0 ε > 0 , there exists a simple function that ∥ f − s ∥ ∞ < ε \|f-s\|_\infty<\varepsilon ∥ f − s ∥ ∞ < ε .
Let [ f ] ∈ L ∞ ( X ) [f]\in L^\infty(X) [ f ] ∈ L ∞ ( X ) , and choose a measurable representative f f f of the equivalence class. Since f f f is essentially bounded, there exists c > 0 c>0 c > 0 such that ∣ f ( x ) ∣ ≤ c |f(x)|\le c ∣ f ( x ) ∣ ≤ c almost everywhere (modifying on a null set, i.e. a set of measure zero, if needed) only because L ∞ L^\infty L ∞ consists of equivalence classes modulo almost-everywhere equality. Replacing f f f by an equivalent representative if necessary, we may assume that ∣ f ( x ) ∣ ≤ c |f(x)|\le c ∣ f ( x ) ∣ ≤ c for all x ∈ X x\in X x ∈ X .
Given ε > 0 \varepsilon>0 ε > 0 , we can partition the interval [ − c , c ] [-c,c] [ − c , c ] into finitely many subintervals of length less than ε > 0 \varepsilon>0 ε > 0 . For example, I k = [ a k , a k + 1 ) I_k=[a_k,a_{k+1}) I k = [ a k , a k + 1 ) , with k = 1.. N − 1 k=1..N-1 k = 1.. N − 1 and I N = [ a N , a N + 1 ] I_N=[a_N,a_{N+1}] I N = [ a N , a N + 1 ] , where a k = − c + ( k − 1 ) δ a_k=-c+(k-1)\delta a k = − c + ( k − 1 ) δ , and δ < ε \delta<\varepsilon δ < ε .
Now, we can define the simple function s ( x ) = a k s(x)=a_k s ( x ) = a k whenever f ( x ) ∈ I k f(x)\in I_k f ( x ) ∈ I k (i.e. a representative value a k ∈ I k a_k\in I_k a k ∈ I k for each interval I k I_k I k ). This function takes only finitely many values { a 1 . . a N } \{a_1..a_N\} { a 1 .. a N } , and each level set { x ∈ X : f ( x ) ∈ I k } \{x\in X:f(x)\in I_k\} { x ∈ X : f ( x ) ∈ I k } is measurable, so s s s is a simple measurable function.
For every x ∈ X x\in X x ∈ X , we have: ∣ f ( x ) − s ( x ) ∣ < δ < ε |f(x)-s(x)|<\delta<\varepsilon ∣ f ( x ) − s ( x ) ∣ < δ < ε . Hence, ∣ f − s ∣ < ε |f-s|<\varepsilon ∣ f − s ∣ < ε for almost every x ∈ X x\in X x ∈ X . By the definition of the essential supremum norm, ∥ f − s ∥ L ∞ = ess sup x ∈ X ∣ f ( x ) − s ( x ) ∣ < ε \|f-s\|_{L^\infty}=\operatorname*{ess\,sup}_{x\in X}|f(x)-s(x)|<\varepsilon ∥ f − s ∥ L ∞ = ess sup x ∈ X ∣ f ( x ) − s ( x ) ∣ < ε . Therefore, ∥ f − s ∥ L ∞ < ε \|f-s\|_{L^\infty}<\varepsilon ∥ f − s ∥ L ∞ < ε . Since ε > 0 \varepsilon>0 ε > 0 was arbitrary, simple functions are dense in L ∞ ( X ) L^\infty(X) L ∞ ( X ) with respect to the essential-supremum norm.
QED.
The Riesz Representation Theorem is a fundamental result in functional analysis that characterizes the dual of certain function spaces. There are several versions depending on the context. In measure theory, the Riesz–Markov–Kakutani representation theorem relates linear functionals on spaces of continuous functions on a locally compact space to measures in measure theory.
Definition A.17: Compact support Let X X X be a topological space and let f : X → R f:X\to\mathbb{R} f : X → R . The support of f f f is the closure of the set { x ∈ X : f ( x ) ≠ 0 } \{x\in X:f(x)\ne0\} { x ∈ X : f ( x ) = 0 } . The function f f f is said to have compact support if its support is compact.
Definition A.18: Positive linear functional. Let X X X be a set and let V V V be a vector space of real-valued functions. A linear functional L : V → R L:V\to\mathbb{R} L : V → R is positive if, for every f ∈ V f\in V f ∈ V , f ( x ) ≥ 0 ∀ x ∈ X f(x)\ge0\ \forall x\in X f ( x ) ≥ 0 ∀ x ∈ X implies L ( f ) ≥ 0 L(f)\ge0 L ( f ) ≥ 0 .
Definition A.19: Locally compact space A topological space X X X is locally compact if every point x ∈ X x\in X x ∈ X has an open neighborhood whose closure is compact. Intuitively, every point has a small neighborhood that behaves like a compact set.
Definition A.20: Hausdorff space A topological space X X X is Hausdorff if for every pair of distinct points x , y ∈ X x,y\in X x , y ∈ X , there exist disjoint open sets U U U and V V V such that x ∈ U , y ∈ V x\in U,y\in V x ∈ U , y ∈ V .
In a Hausdorff space, points can be separated by open neighborhoods. This ensures that limits of sequences (or nets) are unique when they exist. Most familiar spaces, such as the real line R \mathbb{R} R or metric spaces, are Hausdorff spaces.
Definition A.21: Radon measure Let X X X be a locally compact Hausdorff space. A Radon measure on X X X is a Borel measure that is
locally finite, i.e. every point x ∈ X x\in X x ∈ X has an open neighborhood U U U such that μ ( U ) < ∞ \mu(U)<\infty μ ( U ) < ∞ (on locally compact Hausdorff spaces, this is equivalent to finite measure on compact sets, i.e. μ ( K ) < ∞ \mu(K)<\infty μ ( K ) < ∞ for every compact set K K K );
inner regular, i.e. for every Borel set E ∈ B ( X ) E\in\mathcal{B}(X) E ∈ B ( X ) :
μ ( E ) = sup { μ ( K ) : K ⊆ E , K compact } \mu(E)=\sup\{\mu(K):K\subseteq E,\ K\text{ compact}\} μ ( E ) = sup { μ ( K ) : K ⊆ E , K compact } ;
outer regular, i.e. for every Borel set E ∈ B ( X ) E\in\mathcal{B}(X) E ∈ B ( X ) :
μ ( E ) = inf { μ ( U ) : E ⊆ U , U open } \mu(E)=\inf\{\mu(U):E\subseteq U,\ U\text{ open}\} μ ( E ) = inf { μ ( U ) : E ⊆ U , U open } .
Informally, Radon measures can be viewed as well-behaved Borel measures. Their regularity properties allow the measure of complicated Borel sets to be approximated using compact subsets from within and open supersets from outside.
On a locally compact Hausdorff space, every finite regular Borel measure is a Radon measure.
Theorem A.2: Riesz–Markov–Kakutani Representation Theorem (measure theory) Let K K K be
a locally compact Hausdorff space, and let C c ( K ) C_c(K) C c ( K ) be the space of continuous functions with compact support. Then every positive linear functional L : C c ( K ) → R L:C_c(K)\to\mathbb{R} L : C c ( K ) → R is represented by a unique Radon measure μ \mu μ such that:
L ( f ) = ∫ K f d μ ∀ f ∈ C c ( K ) . L(f)=\int_K f\,d\mu\quad\forall f\in C_c(K). L ( f ) = ∫ K f d μ ∀ f ∈ C c ( K ) .
The Riesz-Markov-Kakutani theorem4 establishes a one-to-one correspondence between positive linear functionals on continuous functions and Radon measures. In other words, every positive linear functional can be represented as integration against a unique measure.
Since I n = [ 0 , 1 ] n I_n=[0,1]^n I n = [ 0 , 1 ] n is compact, every continuous function on I n I_n I n has compact support and, therefore, C c ( I n ) = C ( I n ) C_c(I_n)=C(I_n) C c ( I n ) = C ( I n ) . For the compact space I n I_n I n , the Riesz-Markov-Kakutani Representation Theorem states that every positive linear functional L : C ( I n ) → R L:C(I_n)\to\mathbb{R} L : C ( I n ) → R can be represented uniquely as
L ( f ) = ∫ I n f d μ , L(f)=\int_{I_n} f\,d\mu, L ( f ) = ∫ I n f d μ ,
for some finite regular Borel measure μ ∈ M ( I n ) \mu\in M(I_n) μ ∈ M ( I n ) .
Since I n = [ 0 , 1 ] n I_n=[0,1]^n I n = [ 0 , 1 ] n is a compact metric space, every finite Borel measure on I n I_n I n is automatically regular. Regularity guarantees that the measure of any Borel set can be approximated arbitrarily well using compact subsets from within and open supersets from outside. Consequently, every finite Borel measure on I n I_n I n is a Radon measure.
Swapping limits and integrals
The Lebesgue Bounded Convergence Theorem [27 ] gives mathematicians a safe way to swap limits and integrals, a powerful tool when analyzing functions that change over time. The Lebesgue Bounded Convergence Theorem is a special case of the Dominated Convergence Theorem, where the dominating function is the constant function g ( x ) = c g(x)=c g ( x ) = c .
Let E ⊂ R d E\subset\mathbb{R}^d E ⊂ R d be a measurable set and let μ \mu μ denote Lebesgue measure (length in one dimension, area in two dimensions, volume in three dimensions, and their higher-dimensional analogues).
∥ g ∥ 1 = ∫ E ∣ g ( x ) ∣ d μ ( x ) . \lVert g\rVert_1=\int_E |g(x)|\,d\mu(x). ∥ g ∥ 1 = ∫ E ∣ g ( x ) ∣ d μ ( x ) .
Theorem A.3: Lebesgue Bounded Convergence Theorem Let ( f n ) n ∈ N (f_n)_{n\in\mathbb{N}} ( f n ) n ∈ N be a sequence of measurable functions defined on a measurable set E ⊂ R d E\subset\mathbb{R}^d E ⊂ R d with finite measure μ ( E ) < ∞ \mu(E)<\infty μ ( E ) < ∞ . Suppose that f n ( x ) → f ( x ) f_n(x)\to f(x) f n ( x ) → f ( x ) pointwise almost everywhere on E E E and also that there exists a constant c > 0 c>0 c > 0 such that ∣ f n ( x ) ∣ ≤ c |f_n(x)|\le c ∣ f n ( x ) ∣ ≤ c for all n n n and almost every x ∈ E x\in E x ∈ E . Then the limit function f f f is measurable and integrable,
lim n → ∞ ∫ E f n ( x ) d μ ( x ) = ∫ E f ( x ) d μ ( x ) , \lim_{n\to\infty}\int_E f_n(x)\,d\mu(x)=\int_E f(x)\,d\mu(x), n → ∞ lim ∫ E f n ( x ) d μ ( x ) = ∫ E f ( x ) d μ ( x ) , and
lim n → ∞ ∥ f n − f ∥ 1 = 0. \lim_{n\to\infty}\lVert f_n-f\rVert_1=0. n → ∞ lim ∥ f n − f ∥ 1 = 0. Since ∣ f n ( x ) ∣ ≤ c |f_n(x)|\le c ∣ f n ( x ) ∣ ≤ c almost everywhere and f n ( x ) → f ( x ) f_n(x)\to f(x) f n ( x ) → f ( x ) almost everywhere, it follows that ∣ f ( x ) ∣ ≤ c |f(x)|\le c ∣ f ( x ) ∣ ≤ c almost everywhere. Because μ ( E ) < ∞ \mu(E)<\infty μ ( E ) < ∞ ,
∫ E ∣ f ( x ) ∣ d μ ( x ) ≤ c μ ( E ) < ∞ , \int_E |f(x)|\,d\mu(x)\le c\mu(E)<\infty, ∫ E ∣ f ( x ) ∣ d μ ( x ) ≤ c μ ( E ) < ∞ , so f f f is integrable.
If you have a sequence of functions that get closer and closer to a final function (pointwise convergence), they are all uniformly bounded (none of them goes off to infinity), and you are working over a space with finite size (i.e. a domain whose total measure is finite, such as a bounded interval, a bounded region in the plane, or any measurable set with finite volume), then the integrals (which measure total size or area under the curve) of those functions also get closer to the integral of the final function. The theorem provides conditions under which a limiting operation may be moved inside an integral. If the functions f n f_n f n converge pointwise almost everywhere to a limiting function f f f , none of the functions becomes arbitrarily large (they are uniformly bounded by the same constant c c c ), and the domain E E E has finite measure (finite length, area, or volume), then the total accumulated quantity represented by the integral behaves continuously under the limit. Moreover, the entire error between f n f_n f n and f f f , measured by the total absolute deviation ∫ E ∣ f n − f ∣ d μ \int_E|f_n-f|\,d\mu ∫ E ∣ f n − f ∣ d μ (i.e. measured by the L 1 L^1 L 1 distance), tends to zero. Not only do the function values converge almost everywhere, but the cumulative discrepancy over the entire domain also vanishes.
For engineers, this theorem justifies exchanging a limit and an integral when approximating signals, density fields, temperature distributions, or numerical solutions by uniformly bounded approximations over a finite domain. The boundedness condition prevents small regions from developing arbitrarily large spikes that could otherwise invalidate the interchange of limit and integration.
Separating points by hyperplanes
The Hahn–Banach Extension Theorem5 is one of the foundational results in functional analysis. One of its most important geometric consequences is the Hahn–Banach Separation Theorem, often called the point-set separation theorem, which provides a powerful separation principle in convex analysis.
Definition A.22: Hyperplane. A hyperplane in a real vector space X X X is a set of the form
H = { x ∈ X : L ( x ) = α } H = \{x \in X : L(x) = \alpha\} H = { x ∈ X : L ( x ) = α } where L : X → R L : X \to\mathbb{R} L : X → R is a nonzero linear functional and α ∈ R \alpha\in\mathbb{R} α ∈ R .
The sets H − = { x ∈ X : L ( x ) ≤ α } H^{-} = \{x \in X : L(x) \le\alpha\} H − = { x ∈ X : L ( x ) ≤ α } and H + = { x ∈ X : L ( x ) ≥ α } H^{+} = \{x \in X : L(x) \ge\alpha\} H + = { x ∈ X : L ( x ) ≥ α } are called the associated half-spaces.
Definition A.23: Convexity. A set Y ⊆ X Y \subseteq X Y ⊆ X is convex if for every y 1 , y 2 ∈ Y y_1, y_2 \in Y y 1 , y 2 ∈ Y and every t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] ,
t y 1 + ( 1 − t ) y 2 ∈ Y ty_1 + (1-t)y_2 \in Y t y 1 + ( 1 − t ) y 2 ∈ Y Geometrically, every line segment joining two points of Y Y Y remains entirely inside Y Y Y .
Definition A.24: Closed set. A subset Y ⊆ X Y \subseteq X Y ⊆ X is closed if it contains all of its limit points. Equivalently, whenever a sequence ( y n ) ⊆ Y (y_n) \subseteq Y ( y n ) ⊆ Y converges to a point y ∈ X y \in X y ∈ X , the limit y y y also belongs to Y Y Y .
Definition A.25: Normed vector space. A normed vector space is a vector space X X X equipped with a function
∥ ⋅ ∥ : X → [ 0 , ∞ ) \lVert\cdot\rVert: X \to[0,\infty) ∥ ⋅ ∥ : X → [ 0 , ∞ ) called a norm, satisfying
∥ x ∥ = 0 ⟺ x = 0 , \lVert x\rVert= 0 \Longleftrightarrow x = 0, ∥ x ∥ = 0 ⟺ x = 0 , ∥ λ x ∥ = ∣ λ ∣ ∥ x ∥ , \lVert\lambda x\rVert= \lvert\lambda\rvert\lVert x\rVert, ∥ λ x ∥ = ∣ λ ∣ ∥ x ∥ , and
∥ x + y ∥ ≤ ∥ x ∥ + ∥ y ∥ \lVert x+y\rVert\le\lVert x\rVert+\lVert y\rVert ∥ x + y ∥ ≤ ∥ x ∥ + ∥ y ∥ for all x , y ∈ X x,y \in X x , y ∈ X and all scalars λ \lambda λ . The norm measures the size or length of vectors in X X X .
Definition A.26: Continuity of linear functionals. A linear functional is a linear map L : X → R L : X \to\mathbb{R} L : X → R . It is continuous if it is continuous with respect to the norm topology of X X X . Intuitively, small changes in x x x produce small changes in L ( x ) L(x) L ( x ) . In normed spaces, continuity is equivalent to boundedness for some constant C > 0 C > 0 C > 0 :
∣ L ( x ) ∣ ≤ C ∥ x ∥ \lvert L(x)\rvert\le C\lVert x\rVert ∣ L ( x )∣ ≤ C ∥ x ∥ for some constant C > 0 C > 0 C > 0 and all x ∈ X x \in X x ∈ X .
Theorem A.4: Hahn–Banach Separation Theorem. Let X X X be a normed vector space and let M ⊂ X M \subset X M ⊂ X be a linear subspace. If M M M is not dense in X X X , then there exists a nonzero continuous linear functional L : X → R L : X \to\mathbb{R} L : X → R such that
L ( m ) = 0 ∀ m ∈ M . L(m) = 0 \qquad\forall m \in M. L ( m ) = 0 ∀ m ∈ M . Equivalently,
M ⊥ = { L ∈ X ∗ : L ( m ) = 0 ∀ m ∈ M } M^{\perp} = \{L \in X^{*}: L(m) = 0 \ \forall m \in M\} M ⊥ = { L ∈ X ∗ : L ( m ) = 0 ∀ m ∈ M } contains a nonzero element.
The notation ⊥ \perp ⊥ is analogous to orthogonality in Euclidean space: M ⊥ M^{\perp} M ⊥ denotes the annihilator of M M M in the dual space X ∗ X^{*} X ∗ (the continuous dual space of the normed vector space X X X , i.e., the space of all continuous linear functionals L : X → R L : X \to\mathbb{R} L : X → R ).
Corollary A.5. Let X X X be a normed vector space and M ⊂ X M \subset X M ⊂ X a linear subspace. If the only continuous linear functional vanishing on M M M is the zero functional, then M ‾ = X \overline{M} = X M = X . In other words, M M M is dense in X X X .
Remark. Geometrically, if M M M fails to be dense, then its closure M ‾ \overline{M} M is a proper closed subspace of X X X . By the Hahn–Banach Separation Theorem, any point x 0 ∉ M ‾ x_0 \notin\overline{M} x 0 ∈ / M can be separated from M ‾ \overline{M} M by a continuous hyperplane. The separating hyperplane is given by a nonzero continuous linear functional L L L satisfying L ( x ) = 0 L(x) = 0 L ( x ) = 0 , x ∈ M ‾ x \in\overline{M} x ∈ M , and hence L ( x ) = 0 L(x) = 0 L ( x ) = 0 for every x ∈ M x \in M x ∈ M . This yields the fundamental dual characterization of density:
M ‾ = X ⟺ M ⊥ = { 0 } \overline{M} = X \quad\Longleftrightarrow\quad M^{\perp} = \{0\} M = X ⟺ M ⊥ = { 0 }
A linear subspace is dense if and only if the only continuous linear functional that vanishes on it is the zero functional.
Theorem A.6: Hahn–Banach Separation Theorem (Geometric Form). Let X X X be a real normed vector space, Y ⊆ X Y \subseteq X Y ⊆ X be a non-empty closed convex set, and x 0 ∈ X ∖ Y x_0 \in X \setminus Y x 0 ∈ X ∖ Y (i.e., a point not in Y Y Y ). Then there exists a nonzero continuous linear functional L : X → R L : X \to\mathbb{R} L : X → R , and a real number α ∈ R \alpha\in\mathbb{R} α ∈ R such that:
L ( x 0 ) > α ≥ L ( y ) ∀ y ∈ Y . L(x_0) > \alpha\ge L(y) \qquad\forall y \in Y. L ( x 0 ) > α ≥ L ( y ) ∀ y ∈ Y . Equivalently, the hyperplane
H = { x ∈ X : L ( x ) = α } H = \{x \in X : L(x) = \alpha\} H = { x ∈ X : L ( x ) = α }
strictly separates6 the point x 0 x_0 x 0 from the convex set Y Y Y . The theorem, therefore, guarantees that convex sets can be separated from external points by hyperplanes.
Proof. Embed the convex set Y Y Y into the convex and closed set C = Y × { 1 } ⊂ X × R C = Y \times\{1\} \subset X \times\mathbb{R} C = Y × { 1 } ⊂ X × R . The origin ( 0 , 0 ) ∈ X × R (0,0) \in X \times\mathbb{R} ( 0 , 0 ) ∈ X × R , yet ( 0 , 0 ) ∉ C (0,0) \notin C ( 0 , 0 ) ∈ / C .
The convex cone generated by C C C is K = { t ( y , 1 ) : y ∈ Y , t ≥ 0 } K = \{t(y,1) : y \in Y,\ t \ge0\} K = { t ( y , 1 ) : y ∈ Y , t ≥ 0 } . For x 0 ∉ Y x_0 \notin Y x 0 ∈ / Y , ( x 0 , 1 ) ∉ K (x_0,1) \notin K ( x 0 , 1 ) ∈ / K . If ( x 0 , 1 ) (x_0,1) ( x 0 , 1 ) belonged to K K K , there would exist t n ≥ 0 t_n \ge0 t n ≥ 0 and y 0 ∈ Y y_0 \in Y y 0 ∈ Y such that ( t n y n , t n ) → ( x 0 , 1 ) (t_n y_n,t_n) \to(x_0,1) ( t n y n , t n ) → ( x 0 , 1 ) . Its second coordinate implies that t n → 1 t_n \to1 t n → 1 , so y n → x 0 y_n \to x_0 y n → x 0 (a contradiction, x 0 x_0 x 0 would belong to Y Y Y ).
Let E = X × R E = X \times\mathbb{R} E = X × R and consider the linear span M = span ( K ) ⊂ E M = \operatorname{span}(K) \subset E M = span ( K ) ⊂ E . since ( x 0 , 1 ) ∉ K (x_0,1) \notin K ( x 0 , 1 ) ∈ / K , we have ( x 0 , 1 ) ∉ M ‾ (x_0,1) \notin\overline{M} ( x 0 , 1 ) ∈ / M . Consequently M M M is a proper closed subspace of the space M ‾ + R ( x 0 , 1 ) \overline{M} + \mathbb{R}(x_0,1) M + R ( x 0 , 1 ) . Hence M M M is not dense there..
By the corollary, there exists a nonzero continuous linear functional F : E → R F : E \to\mathbb{R} F : E → R such that F ( u ) = 0 F(u) = 0 F ( u ) = 0 , u ∈ M u \in M u ∈ M . Since K ⊂ M K \subset M K ⊂ M , F ( t ( y , 1 ) ) = 0 F(t(y,1)) = 0 F ( t ( y , 1 )) = 0 (y ∈ Y y \in Y y ∈ Y , t ≥ 0 t \ge0 t ≥ 0 ). Therefore F ( y , 1 ) = 0 F(y,1) = 0 F ( y , 1 ) = 0 , ∀ y ∈ Y \forall y \in Y ∀ y ∈ Y .
Every continuous linear functional on X × R X \times\mathbb{R} X × R has the form F ( x , t ) = L ( x ) − α t F(x,t) = L(x) - \alpha t F ( x , t ) = L ( x ) − α t for some L ∈ X ∗ L \in X^* L ∈ X ∗ and α ∈ R \alpha\in\mathbb{R} α ∈ R . Since F ( y , 1 ) = 0 F(y,1) = 0 F ( y , 1 ) = 0 for all y ∈ Y y \in Y y ∈ Y , L ( y ) = α ∀ y ∈ Y L(y) = \alpha\ \forall y \in Y L ( y ) = α ∀ y ∈ Y . The functional is nonzero, so L L L is not identically zero.
Furthermore, F ( x 0 , 1 ) ≠ 0 F(x_0,1) \ne0 F ( x 0 , 1 ) = 0 , because ( x 0 , 1 ) ∉ M ‾ (x_0,1) \notin\overline{M} ( x 0 , 1 ) ∈ / M . Replacing F F F by − F -F − F if necessary, we may assume F ( x 0 , 1 ) > 0 F(x_0,1) > 0 F ( x 0 , 1 ) > 0 and, therefore, L ( x 0 ) − α > 0 L(x_0) - \alpha> 0 L ( x 0 ) − α > 0 , i.e. L ( x 0 ) > α L(x_0) > \alpha L ( x 0 ) > α .
Combining this with L ( y ) = α L(y) = \alpha L ( y ) = α for all y ∈ Y y \in Y y ∈ Y , we obtain the desired result:
L ( x 0 ) > α ≥ L ( y ) ∀ y ∈ Y L(x_0) > \alpha\ge L(y) \qquad\forall y \in Y L ( x 0 ) > α ≥ L ( y ) ∀ y ∈ Y
Thus the theorem follows from the annihilator corollary by lifting the problem to X × R X \times\mathbb{R} X × R and applying the non-dense-subspace result there.
The geometric Hahn-Banach separation theorem asserts that every point lying outside a closed convex set can be distinguished from that set by a continuous linear measurement L L L . The values of L L L on the set Y Y Y are all bounded above by α \alpha α , whereas the value of L L L at x 0 x_0 x 0 is strictly larger than α \alpha α . Consequently, Y Y Y and x 0 x_0 x 0 lie on opposite sides of the hyperplane L ( x ) = α L(x) = \alpha L ( x ) = α .
Remark. The condition L ≠ 0 L \ne0 L = 0 is essention and cannot be omitted. Otherwise, the statement becomes vacuous because the zero functional cannot provide a separating hyperplane.
Remark. Likewise, if Y Y Y is not closed, strict separation can fail. The closedness assumption ensures a positive geometric gap between x 0 x_0 x 0 and the set; i.e. it guarantees that the distance from x 0 x_0 x 0 to Y Y Y is strictly positive: d ( x 0 , Y ) = inf y ∈ Y ∥ x 0 − y ∥ > 0 d(x_0,Y) = \inf_{y \in Y} \lVert x_0 - y \rVert> 0 d ( x 0 , Y ) = inf y ∈ Y ∥ x 0 − y ∥ > 0 when Y Y Y is closed and x 0 ∉ Y x_0 \notin Y x 0 ∈ / Y in metric spaces.
Fourier analysis
Definition A.27: Fourier Transform Let f ∈ L 1 ( R n ) f \in L^1(\mathbb{R}^n) f ∈ L 1 ( R n ) . The (unnormalized) Fourier transform of f f f is defined by
f ^ ( ω ) = F [ f ] ( ω ) = ∫ R n f ( x ) e − i x ⋅ ω d x , ω ∈ R n , \hat{f}(\omega) = \mathcal{F}[f](\omega) = \int_{\mathbb{R}^n} f(x)e^{-i x\cdot\omega}\,dx,\qquad\omega\in\mathbb{R}^n, f ^ ( ω ) = F [ f ] ( ω ) = ∫ R n f ( x ) e − i x ⋅ ω d x , ω ∈ R n , where x ⋅ ω = ∑ k = 1 n x k ω k x\cdot\omega= \sum_{k=1}^{n} x_k\omega_k x ⋅ ω = ∑ k = 1 n x k ω k denotes the Euclidean inner product.
Definition A.28: Inverse Fourier Transform For a sufficiently regular function F F F , the inverse Fourier transform is given by
f ( x ) = F − 1 [ F ] ( x ) = 1 ( 2 π ) n ∫ R n F ( ω ) e i x ⋅ ω d ω , x ∈ R n . f(x) = \mathcal{F}^{-1}[F](x) = \frac{1}{(2\pi)^n}\int_{\mathbb{R}^n} F(\omega)e^{i x\cdot\omega}\,d\omega,\qquad x \in\mathbb{R}^n. f ( x ) = F − 1 [ F ] ( x ) = ( 2 π ) n 1 ∫ R n F ( ω ) e i x ⋅ ω d ω , x ∈ R n .
Remark. Under this convention, the inverse transform contains the factor ( 2 π ) − n (2\pi)^{-n} ( 2 π ) − n , which is unnecessary in the normalized Fourier transform ∫ R n f ( x ) e − 2 π i x ⋅ ω d x \int_{\mathbb{R}^n} f(x)e^{-2\pi i x\cdot\omega}\,dx ∫ R n f ( x ) e − 2 π i x ⋅ ω d x .
These operators extend to L 2 ( R n ) L^2(\mathbb{R}^n) L 2 ( R n ) , tempered distributions, and other function spaces by density and continuity. They also satisfy the well-known property:
F − 1 [ F f ] ( x ) = f ( x ) \mathcal{F}^{-1}[\mathcal{F}f](x) = f(x) F − 1 [ F f ] ( x ) = f ( x )
Definition A.29: Support Let f : R n → R f : \mathbb{R}^n \to\mathbb{R} f : R n → R be a function. The support of f f f is the closure of the set where f f f is nonzero:
supp ( f ) = { x ∈ R n : f ( x ) ≠ 0 } ‾ . \operatorname{supp}(f) = \overline{\{x \in\mathbb{R}^n : f(x) \ne0\}}. supp ( f ) = { x ∈ R n : f ( x ) = 0 } .
A function is said to be compactly supported if its support is a compact subset of R n \mathbb{R}^n R n .
Definition A.30: Band-Limited Function A function f : R n → C f : \mathbb{R}^n \to\mathbb{C} f : R n → C is called band-limited if its Fourier transform vanishes outside some bounded set. Equivalently, there exists R > 0 R > 0 R > 0 such that
supp ( f ^ ) ⊆ B R ( 0 ) = { ξ ∈ R n : ∥ ξ ∥ ≤ R } . \operatorname{supp}(\hat{f}) \subseteq B_R(0) = \{\xi\in\mathbb{R}^n : \lVert\xi\rVert\le R\}. supp ( f ^ ) ⊆ B R ( 0 ) = { ξ ∈ R n : ∥ ξ ∥ ≤ R } .
Definition A.31: Convolution Let f , g : R n → R f,g : \mathbb{R}^n \to\mathbb{R} f , g : R n → R be functions such that the integral below is well defined. Their convolution is the function
( f ∗ g ) ( x ) = ∫ R n f ( x − y ) g ( y ) d y . (f \ast g)(x) = \int_{\mathbb{R}^n} f(x-y)g(y)\,dy. ( f ∗ g ) ( x ) = ∫ R n f ( x − y ) g ( y ) d y . Convolution averages the values of one function using the other as a kernel.
Definition A.32: Schwartz Function A function ϕ : R n → C \phi: \mathbb{R}^n \to\mathbb{C} ϕ : R n → C is called a Schwartz function if it is infinitely differentiable and all of its derivatives decay at infinity faster than any polynomial. Precisely, for every pair of multi-indices α , β \alpha,\beta α , β ,
sup x ∈ R n ∣ x α D β ϕ ( x ) ∣ < ∞ . \sup_{x \in\mathbb{R}^n} \lvert x^\alpha D^\beta\phi(x) \rvert< \infty. x ∈ R n sup ∣ x α D β ϕ ( x )∣ < ∞.
The set of all Schwartz functions is denoted by S ( R n ) \mathcal{S}(\mathbb{R}^n) S ( R n ) .
::: Remark. Schwartz functions are smooth functions whose values and all derivatives tend to zero at infinity faster than any inverse power of ∥ x ∥ \lVert x\rVert ∥ x ∥ . Typical examples include the Gaussian ϕ ( x ) = e − ∥ x ∥ 2 \phi(x)=e^{-\lVert x\rVert^2} ϕ ( x ) = e − ∥ x ∥ 2 . Schwartz functions are particularly important in Fourier analysis because the Fourier transform maps S ( R n ) \mathcal{S}(\mathbb{R}^n) S ( R n ) onto itself.
Definition A.33: Entire Function. A function
F : C n → C F:\mathbb{C}^n\to\mathbb{C} F : C n → C is called entire if it is holomorphic in every complex variable on all of C n \mathbb{C}^n C n . In other words, F F F is complex-differentiable with respect to every complex variable throughout all of C n \mathbb{C}^n C n . Equivalently, it admits a convergent power-series expansion about every point of C n \mathbb{C}^n C n ; i.e. it can be represented locally by a convergent power series in the complex variables z 1 , … , z n z_1,\ldots,z_n z 1 , … , z n at every point of C n \mathbb{C}^n C n .
Definition A.34: Support Function. Let K ⊂ R n K\subset\mathbb{R}^n K ⊂ R n be a compact convex set. The support function of K K K is
h K ( y ) = sup x ∈ K x ⋅ y , y ∈ R n . h_K(y)=\sup_{x\in K}x\cdot y,\qquad y\in\mathbb{R}^n. h K ( y ) = x ∈ K sup x ⋅ y , y ∈ R n . It measures the maximal extent of K K K in the direction y y y .
The Paley–Wiener theorem7 connects support properties of a function’s Fourier transform with analytic continuation and growth conditions of the function itself: compact support in the time domain corresponds to a entire function of exponential type in the frequency domain.
Theorem A.7: Paley–Wiener theorem (one-dimensional). A function F : C → C F:\mathbb{C}\to\mathbb{C} F : C → C is the Fourier transform of an L 2 L^2 L 2 -function f f f supported in [ − R , R ] ⊂ R [-R,R]\subset\mathbb{R} [ − R , R ] ⊂ R if and only if:
F F F is entire (holomorphic on all of C \mathbb{C} C ), and
Growth bound: There exists C > 0 C>0 C > 0 such that for all z ∈ C z\in\mathbb{C} z ∈ C ,
∣ F ( z ) ∣ ≤ C ( 1 + ∥ z ∥ ) N e R ∣ ℑ z ∣ , |F(z)|\le C(1+\lVert z\rVert)^N e^{R|\Im z|}, ∣ F ( z ) ∣ ≤ C ( 1 + ∥ z ∥ ) N e R ∣ℑ z ∣ , for some integer N N N .
Theorem A.8: Paley–Wiener theorem (multidimensional). Let K ⊂ R n K\subset\mathbb{R}^n K ⊂ R n be a compact convex set and let f ∈ L 2 ( R n ) f\in L^2(\mathbb{R}^n) f ∈ L 2 ( R n ) satisfy supp ( f ) ⊆ K \operatorname{supp}(f)\subseteq K supp ( f ) ⊆ K . Then its Fourier transform F = f ^ F=\widehat{f} F = f extends uniquely to an entire function on C n \mathbb{C}^n C n satisfying the growth estimate
∣ F ( z ) ∣ ≤ C ( 1 + ∥ z ∥ ) N e h K ( ℑ z ) |F(z)|\le C(1+\lVert z\rVert)^N e^{h_K(\Im z)} ∣ F ( z ) ∣ ≤ C ( 1 + ∥ z ∥ ) N e h K ( ℑ z ) for suitable constants C , N > 0 C,N>0 C , N > 0 .
Conversely, every entire function satisfying such a growth condition and the appropriate L 2 L^2 L 2 -integrability on R n \mathbb{R}^n R n arises as the Fourier transform of an L 2 L^2 L 2 -function supported in K K K .
The key intuition is that entire functions are exceptionally smooth. They can be extended from real variables x ∈ R n x\in\mathbb{R}^n x ∈ R n to complex variables z ∈ C n z\in\mathbb{C}^n z ∈ C n without encountering singularities.
Corollary A.9: Band-Limited Functions are Entire. If f ∈ L 2 ( R n ) f\in L^2(\mathbb{R}^n) f ∈ L 2 ( R n ) is band-limited, i.e., supp ( f ^ ) ⊆ K \operatorname{supp}(\widehat{f})\subseteq K supp ( f ) ⊆ K for some compact set K ⊂ R n K\subset\mathbb{R}^n K ⊂ R n , then f f f extends to an entire function on C n \mathbb{C}^n C n . Moreover, its growth is controlled by the support function h K h_K h K :
∣ f ( z ) ∣ ≤ C ( 1 + ∥ z ∥ ) N e h K ( ℑ z ) . |f(z)|\le C(1+\lVert z\rVert)^N e^{h_K(\Im z)}. ∣ f ( z ) ∣ ≤ C ( 1 + ∥ z ∥ ) N e h K ( ℑ z ) .
Lemma A.10: Uniform Approximation on Compact Sets by Band-Limited Functions. For every compact K ⊂ R n K\subset\mathbb{R}^n K ⊂ R n , every f ∈ C ( K ) f\in C(K) f ∈ C ( K ) , and every δ > 0 \delta>0 δ > 0 , there exists a smooth band-limited function F ∈ C ∞ ( R n ) F\in C^\infty(\mathbb{R}^n) F ∈ C ∞ ( R n ) such that
supp ( F ^ ) ⊂ B R ( 0 ) \operatorname{supp}(\widehat{F})\subset B_R(0) supp ( F ) ⊂ B R ( 0 ) for some R > 0 R>0 R > 0 , and
sup x ∈ K ∣ F ( x ) − f ( x ) ∣ < δ . \sup_{x\in K}|F(x)-f(x)|<\delta. x ∈ K sup ∣ F ( x ) − f ( x ) ∣ < δ .
Cybenko's Universal Approximation Theorem
Cybenko’s theorem is stated on the compact unit cube I n = [ 0 , 1 ] n ⊂ R n I_n=[0,1]^n\subset\mathbb{R}^n I n = [ 0 , 1 ] n ⊂ R n because the proof uses C ( I n ) C(I_n) C ( I n ) as a Banach space (definition A.2) and the Riesz Representation Theorem (theorem A.2). Given the space of finite signed measures on I n I_n I n , M ( I n ) M(I_n) M ( I n ) (definition A.13), the Riesz Representation Theorem identifies C ( I n ) ∗ ≃ M ( I n ) C(I_n)^*\simeq M(I_n) C ( I n ) ∗ ≃ M ( I n ) .
Definition B.1: A function σ : R → R \sigma:\mathbb{R}\to\mathbb{R} σ : R → R is discriminatory if, whenever a finite signed measure μ ∈ M ( I n ) \mu\in M(I_n) μ ∈ M ( I n ) satisfies
∫ I n σ ( w T x + b ) d μ ( x ) = 0 \int_{I_n}\sigma(w^\mathsf{T}x+b)\,d\mu(x)=0 ∫ I n σ ( w T x + b ) d μ ( x ) = 0 for all w ∈ R n w\in\mathbb{R}^n w ∈ R n and b ∈ R b\in\mathbb{R} b ∈ R , then μ = 0 \mu=0 μ = 0 .
Definition B.2: We say that σ : R → R \sigma:\mathbb{R}\to\mathbb{R} σ : R → R is sigmoidal if lim z → + ∞ σ ( z ) = 1 \lim_{z\to+\infty}\sigma(z)=1 lim z → + ∞ σ ( z ) = 1 and lim z → − ∞ σ ( z ) = 0 \lim_{z\to-\infty}\sigma(z)=0 lim z → − ∞ σ ( z ) = 0
Definition B.3: The continuous function space C ( I n ) = { f : I n → R : f is continuous } C(I_n)=\{f:I_n\to\mathbb{R}: f\text{ is continuous}\} C ( I n ) = { f : I n → R : f is continuous } is equipped with the uniform norm ∥ f ∥ ∞ = sup x ∈ I n ∣ f ( x ) ∣ \lVert f\rVert_\infty=\sup_{x\in I_n}|f(x)| ∥ f ∥ ∞ = sup x ∈ I n ∣ f ( x ) ∣ . A subset S ⊂ C ( I n ) S\subset C(I_n) S ⊂ C ( I n ) is dense if any continuous function can be approximated arbitrarily well in this norm:
∀ f ∈ C ( I n ) , ∀ ϵ > 0 , ∃ g ∈ S such that ∥ f − g ∥ ∞ < ϵ \forall f\in C(I_n),\ \forall\epsilon>0,\ \exists g\in S\text{ such that }\lVert f-g\rVert_\infty<\epsilon ∀ f ∈ C ( I n ) , ∀ ϵ > 0 , ∃ g ∈ S such that ∥ f − g ∥ ∞ < ϵ
Theorem I.1: Universal approximation theorem for artificial neural networks with sigmoid activation functions. Let σ \sigma σ be any continuous discriminatory function. The finite sums of the form
G ( x ) = ∑ j = 1 h α j σ ( ∑ i = 1 n w i j x i + b j ) G(x)=\sum_{j=1}^{h}\alpha_j\sigma\left(\sum_{i=1}^{n}w_{ij}x_i+b_j\right) G ( x ) = j = 1 ∑ h α j σ ( i = 1 ∑ n w ij x i + b j ) are dense in C ( I n ) C(I_n) C ( I n ) . In other words, given any f ∈ C ( I n ) f\in C(I_n) f ∈ C ( I n ) and ϵ > 0 \epsilon>0 ϵ > 0 , there is a sum, G ( x ) G(x) G ( x ) , of the above form, for which
∣ G ( x ) − f ( x ) ∣ < ϵ ∀ x ∈ I n \lvert G(x)-f(x)\rvert<\epsilon\quad\forall x\in I_n ∣ G ( x ) − f ( x )∣ < ϵ ∀ x ∈ I n
Remark. The class G ( x ) = ∑ j = 1 h α j σ ( w j T x + b j ) G(x)=\sum_{j=1}^{h}\alpha_j\sigma(w_j^{\mathsf{T}}x+b_j) G ( x ) = ∑ j = 1 h α j σ ( w j T x + b j ) is exactly the set of functions computed by a neural network with n n n inputs, one hidden layer containing h h h neurons, activation function σ \sigma σ , and a linear output layer.
Proof. Let S ⊂ C ( I n ) S\subset C(I_n) S ⊂ C ( I n ) be the set of functions of the form G ( x ) G(x) G ( x ) .8 Clearly, S S S is a linear subspace of C ( I n ) C(I_n) C ( I n ) . We prove that the closure of S S S is C ( I n ) C(I_n) C ( I n ) by contradiction, i.e. S ‾ = C ( I n ) \overline{S}=C(I_n) S = C ( I n ) .
Assume that the closure of S S S is not all of C ( I n ) C(I_n) C ( I n ) . Then, the closure of S S S , say R R R , is a closed proper subspace of C ( I n ) C(I_n) C ( I n ) . Since R R R is a proper closed subspace of the Banach space C ( I n ) C(I_n) C ( I n ) , the Hahn–Banach separation theorem (theorem A.6) implies the existence of a nonzero continuous linear functional L L L vanishing on R R R : L ≠ 0 L\ne0 L = 0 and L ( R ) = L ( S ) = 0 L(R)=L(S)=0 L ( R ) = L ( S ) = 0 .
By the Riesz Representation Theorem (theorem A.2), this bounded linear functional, L L L , is of the form
L ( h ) = ∫ I n h ( x ) d μ ( x ) ≠ 0 L(h)=\int_{I_n}h(x)d\mu(x)\ne0 L ( h ) = ∫ I n h ( x ) d μ ( x ) = 0
for some μ ∈ M ( I n ) \mu\in M(I_n) μ ∈ M ( I n ) , for all h ∈ C ( I n ) h\in C(I_n) h ∈ C ( I n ) . In particular, since σ ( w T x + b ) \sigma(w^{\mathsf{T}}x+b) σ ( w T x + b ) is in R R R for all w w w and b b b ,9 we must have that
∫ I n σ ( w T x + b ) d μ ( x ) = 0 \int_{I_n}\sigma(w^{\mathsf{T}}x+b)d\mu(x)=0 ∫ I n σ ( w T x + b ) d μ ( x ) = 0
for all w w w and b b b .
However, we assumed that σ \sigma σ was discriminatory so that this condition implies that μ = 0 \mu=0 μ = 0 contradicting our assumption. Hence, the subspace S S S must be dense in C ( I n ) C(I_n) C ( I n ) . □ \square □
This demonstrates that sums of the form G ( x ) G(x) G ( x ) are dense in C ( I n ) C(I_n) C ( I n ) provided that σ \sigma σ is continuous and discriminatory. We can connect this result with the sigmoid activation functions used by neural networks:
Lemma B.1: Any bounded, measurable sigmoidal function, σ \sigma σ , is discriminatory. In particular, any continuous sigmoidal function is discriminatory.
Proof. Based on (10)[pp.307-308], with correction [11 ]
σ ( λ ( w T x + b ) + φ ) = { 1 for w T x + b > 0 as λ → ∞ 0 for w T x + b < 0 as λ → ∞ σ ( φ ) for w T x + b = 0 ∀ λ \sigma(\lambda(w^{\mathsf{T}}x+b)+\varphi)=
\begin{cases}
1 & \text{for }w^{\mathsf{T}}x+b>0\text{ as }\lambda\to\infty\\
0 & \text{for }w^{\mathsf{T}}x+b<0\text{ as }\lambda\to\infty\\
\sigma(\varphi) & \text{for }w^{\mathsf{T}}x+b=0\ \forall\lambda
\end{cases} σ ( λ ( w T x + b ) + φ ) = ⎩ ⎨ ⎧ 1 0 σ ( φ ) for w T x + b > 0 as λ → ∞ for w T x + b < 0 as λ → ∞ for w T x + b = 0 ∀ λ
Therefore, σ λ ( x ) = σ ( λ ( w T x + b ) + φ ) \sigma_\lambda(x)=\sigma(\lambda(w^{\mathsf{T}}x+b)+\varphi) σ λ ( x ) = σ ( λ ( w T x + b ) + φ ) converges pointwise and boundedly to
γ ( x ) = { 1 for w T x + b > 0 0 for w T x + b < 0 σ ( φ ) for w T x + b = 0 \gamma(x)=
\begin{cases}
1 & \text{for }w^{\mathsf{T}}x+b>0\\
0 & \text{for }w^{\mathsf{T}}x+b<0\\
\sigma(\varphi) & \text{for }w^{\mathsf{T}}x+b=0
\end{cases} γ ( x ) = ⎩ ⎨ ⎧ 1 0 σ ( φ ) for w T x + b > 0 for w T x + b < 0 for w T x + b = 0
Let Π w , b \Pi_{w,b} Π w , b be the hyperplane defined by { x ∣ w T x + b = 0 } \{x\mid w^{\mathsf{T}}x+b=0\} { x ∣ w T x + b = 0 } and let H w , b H_{w,b} H w , b be the open half-space defined by { x ∣ w T x + b > 0 } \{x\mid w^{\mathsf{T}}x+b>0\} { x ∣ w T x + b > 0 } . Then, by the Lebesgue Bounded Convergence Theorem (theorem A.3), for all φ \varphi φ , w w w , and b b b , we have10
0 = ∫ I n σ λ ( x ) d μ ( x ) ∀ λ 0=\int_{I_n}\sigma_\lambda(x)d\mu(x)\quad\forall\lambda 0 = ∫ I n σ λ ( x ) d μ ( x ) ∀ λ
0 = lim λ → ∞ ∫ I n σ λ ( x ) d μ ( x ) = ∫ I n γ ( x ) d μ ( x ) = σ ( φ ) μ ( Π w , b ) + μ ( H w , b ) \begin{aligned}
0=\lim_{\lambda\to\infty}\int_{I_n}\sigma_\lambda(x)d\mu(x) \\
&=\int_{I_n}\gamma(x)d\mu(x) \\
&=\sigma(\varphi)\mu(\Pi_{w,b})+\mu(H_{w,b})
\end{aligned} 0 = λ → ∞ lim ∫ I n σ λ ( x ) d μ ( x ) = ∫ I n γ ( x ) d μ ( x ) = σ ( φ ) μ ( Π w , b ) + μ ( H w , b )
We now show that the measure of all half-spaces being 0 0 0 implies that the measure μ \mu μ itself must be 0 0 0 . This would be trivial if μ \mu μ were a positive measure but here it is not. Why is the measure of all half-spaces equal to 0 0 0 ? Since σ ( φ ) μ ( Π w , b ) + μ ( H w , b ) = 0 \sigma(\varphi)\mu(\Pi_{w,b})+\mu(H_{w,b})=0 σ ( φ ) μ ( Π w , b ) + μ ( H w , b ) = 0 for every φ \varphi φ :
σ ( φ 1 ) μ ( Π w , b ) + μ ( H w , b ) = 0 \sigma(\varphi_1)\mu(\Pi_{w,b})+\mu(H_{w,b})=0 σ ( φ 1 ) μ ( Π w , b ) + μ ( H w , b ) = 0
σ ( φ 2 ) μ ( Π w , b ) + μ ( H w , b ) = 0 \sigma(\varphi_2)\mu(\Pi_{w,b})+\mu(H_{w,b})=0 σ ( φ 2 ) μ ( Π w , b ) + μ ( H w , b ) = 0
Because a sigmoidal function is non-constant, choose φ 1 , φ 2 \varphi_1,\varphi_2 φ 1 , φ 2 with σ ( φ 1 ) ≠ σ ( φ 2 ) \sigma(\varphi_1)\ne\sigma(\varphi_2) σ ( φ 1 ) = σ ( φ 2 ) . Subtracting gives
μ ( Π w , b ) = 0 \mu(\Pi_{w,b})=0 μ ( Π w , b ) = 0
Substituting back gives
μ ( H w , b ) = 0 \mu(H_{w,b})=0 μ ( H w , b ) = 0
Thus every hyperplane and every open half-space has μ \mu μ -measure zero. Now we will prove that the measure μ \mu μ itself must be 0 0 0 . Fix w w w . For a bounded measurable function, h h h , define the linear functional, F : L ∞ ( J ) → C F:L^\infty(J)\to\mathbb{C} F : L ∞ ( J ) → C where J = { w T x : x ∈ I n } J=\{w^\mathsf{T}x:x\in I_n\} J = { w T x : x ∈ I n } , according to
F ( h ) = ∫ I n h ( w T x ) d μ ( x ) F(h)=\int_{I_n}h(w^\mathsf{T}x)d\mu(x) F ( h ) = ∫ I n h ( w T x ) d μ ( x )
and note that F F F is a bounded functional on L ∞ ( J ) L^\infty(J) L ∞ ( J ) since μ \mu μ is a finite signed measure. Let h h h be the indicator function of the half-closed interval [ b , ∞ ) [b,\infty) [ b , ∞ ) :
1 [ b , ∞ ) ( u ) = { 1 , u ≥ b , 0 , u < b . \mathbf{1}_{[b,\infty)}(u)=
\begin{cases}
1, & u\ge b,\\
0, & u<b.
\end{cases} 1 [ b , ∞ ) ( u ) = { 1 , 0 , u ≥ b , u < b .
Then
{ x : h ( w T x ) = 1 } = { x : w T x ≥ b } = Π w , − b ∪ H w , − b \{x:h(w^\mathsf{T}x)=1\}=\{x:w^\mathsf{T}x\ge b\}=\Pi_{w,-b}\cup H_{w,-b} { x : h ( w T x ) = 1 } = { x : w T x ≥ b } = Π w , − b ∪ H w , − b
where
Π w , − b = { x : w T x = b } , H w , − b = { x : w T x > b } \Pi_{w,-b}=\{x:w^\mathsf{T}x=b\},\qquad H_{w,-b}=\{x:w^\mathsf{T}x>b\} Π w , − b = { x : w T x = b } , H w , − b = { x : w T x > b }
Therefore
F ( h ) = ∫ I n h ( w T x ) d μ ( x ) = μ ( { x : w T x ≥ b } ) = μ ( Π w , − b ) + μ ( H w , − b ) . \begin{aligned}
F(h)&=\int_{I_n}h(w^\mathsf{T}x)d\mu(x)\\
&=\mu(\{x:w^\mathsf{T}x\ge b\})\\
&=\mu(\Pi_{w,-b})+\mu(H_{w,-b}).
\end{aligned} F ( h ) = ∫ I n h ( w T x ) d μ ( x ) = μ ({ x : w T x ≥ b }) = μ ( Π w , − b ) + μ ( H w , − b ) .
Similarly, F ( h ) = 0 F(h)=0 F ( h ) = 0 if h h h is the indicator function of the open interval ( b , ∞ ) (b,\infty) ( b , ∞ ) . By linearity, F ( h ) = 0 F(h)=0 F ( h ) = 0 for the indicator function of any interval and hence for any simple function (that is, sum of indicator functions of intervals). F F F vanishes on every simple function. Now let h ∈ L ∞ ( J ) h\in L^\infty(J) h ∈ L ∞ ( J ) be bounded and measurable. There exists a sequence of simple functions { h k } \{h_k\} { h k } such that h k → h h_k\to h h k → h pointwise and ∣ h k ∣ ≤ ∥ h ∥ ∞ |h_k|\le\lVert h\rVert_\infty ∣ h k ∣ ≤ ∥ h ∥ ∞ . Since μ \mu μ is finite, the Bounded Convergence Theorem (theorem A.3) gives F ( h ) = lim k → ∞ F ( h k ) F(h)=\lim_{k\to\infty}F(h_k) F ( h ) = lim k → ∞ F ( h k ) . Strictly speaking, since μ \mu μ is a finite signed measure, the bounded convergence theorem applies to its Jordan decomposition μ = μ + − μ − \mu=\mu^+-\mu^- μ = μ + − μ − . Because F ( h k ) = 0 F(h_k)=0 F ( h k ) = 0 for every k k k , it follows that F ( h ) = 0 F(h)=0 F ( h ) = 0 . Hence F ≡ 0 F\equiv0 F ≡ 0 on L ∞ ( J ) L^\infty(J) L ∞ ( J ) . 11 ^{11} 11
In particular, the bounded measurable functions s ( u ) = sin ( u ) s(u)=\sin(u) s ( u ) = sin ( u ) and c ( u ) = cos ( u ) c(u)=\cos(u) c ( u ) = cos ( u ) give
F ( s + i c ) = ∫ I n cos ( w T x ) + i sin ( w T x ) d μ ( x ) = ∫ I n e i ω T x d μ ( x ) = 0 \begin{aligned}
F(s+ic)=\int_{I_n}\cos(w^\mathsf{T}x)+i\sin(w^\mathsf{T}x)d\mu(x) \\
&=\int_{I_n}e^{i\omega^\mathsf{T}x}d\mu(x)=0
\end{aligned} F ( s + i c ) = ∫ I n cos ( w T x ) + i sin ( w T x ) d μ ( x ) = ∫ I n e i ω T x d μ ( x ) = 0
Since the preceding argument holds for every ω ∈ R n \omega\in\mathbb{R}^n ω ∈ R n , we obtain μ ^ ( ω ) = ∫ I n e i ω T x d μ ( x ) = 0 ∀ ω ∈ R n \widehat{\mu}(\omega)=\int_{I_n}e^{i\omega^\mathsf{T}x}d\mu(x)=0\ \forall\omega\in\mathbb{R}^n μ ( ω ) = ∫ I n e i ω T x d μ ( x ) = 0 ∀ ω ∈ R n . Thus the Fourier transform of μ \mu μ vanishes identically. When the Fourier transform of μ \mu μ is 0, μ \mu μ must be zero as well: If μ ^ ( ω ) = ∫ e i ω T x d μ ( x ) \widehat{\mu}(\omega)=\int e^{i\omega^\mathsf{T}x}d\mu(x) μ ( ω ) = ∫ e i ω T x d μ ( x ) vanishes for all ω ∈ R n \omega\in\mathbb{R}^n ω ∈ R n , then μ = 0 \mu=0 μ = 0 by the uniqueness theorem for Fourier transforms of finite measures. Hence, σ \sigma σ is discriminatory. □
Remark. The Fourier transform of the measure μ \mu μ must be zero because the μ \mu μ -measure of every half-space is zero. The complete logic argument is: all half-spaces have μ \mu μ -measure 0 ⇒ F = 0 ⇒ μ ^ = 0 ⇒ μ = 0 0\Rightarrow F=0\Rightarrow\widehat{\mu}=0\Rightarrow\mu=0 0 ⇒ F = 0 ⇒ μ = 0 ⇒ μ = 0 .
Remark. The theorem actually proves that, if σ \sigma σ is continuous and discriminatory, then single-hidden-layer networks using σ \sigma σ are dense in C ( I n ) C(I_n) C ( I n ) . Since every bounded measurable sigmoidal function is discriminatory, as a corollary, every continuous sigmoidal activation function satisfies the universal approximation property.
Funahashi’s Universal Approximation Theorem
Mollifiers (also known as approximations to the identity) are smooth functions used to create sequences of smooth functions approximating non-smooth (generalized) functions, via convolution. Intuitively, given a (generalized) function, convolving it with a mollifier “mollifies” it: its sharp features are smoothed, while still remaining close to the original.
Definition B.4: Approximation to the Identity
A family of functions { ρ ε } ε > 0 \{\rho_\varepsilon\}_{\varepsilon>0} { ρ ε } ε > 0 is called an approximation to the identity if
ρ ε ( x ) ≥ 0 , ∫ R n ρ ε ( x ) d x = 1 , \rho_\varepsilon(x)\ge0,\qquad\int_{\mathbb{R}^n}\rho_\varepsilon(x)\,dx=1, ρ ε ( x ) ≥ 0 , ∫ R n ρ ε ( x ) d x = 1 ,
and the mass of ρ ε \rho_\varepsilon ρ ε concentrates near the origin as ε → 0 \varepsilon\to0 ε → 0 . Equivalently, for every δ > 0 \delta>0 δ > 0 ,
∫ ∥ x ∥ > δ ρ ε ( x ) d x ⟶ 0 ( ε → 0 ) . \int_{\lVert x\rVert>\delta}\rho_\varepsilon(x)\,dx\longrightarrow0\qquad(\varepsilon\to0). ∫ ∥ x ∥ > δ ρ ε ( x ) d x ⟶ 0 ( ε → 0 ) .
A mollifier is a smooth approximation to the identity.
Definition B.5: Mollifier Let ρ ( x ) \rho(x) ρ ( x ) be a function on R n \mathbb{R}^n R n which satisfies the following conditions:
ρ ( x ) ≥ 0 \rho(x)\ge0 ρ ( x ) ≥ 0
ρ ( x ) \rho(x) ρ ( x ) has continuous partial derivatives of all orders.
The support of ρ ( x ) \rho(x) ρ ( x ) is contained in the unit ball ∥ x ∥ ≤ 1 \lVert x\rVert\le1 ∥ x ∥ ≤ 1 .
∫ R n ρ ( x ) d x = 1 \int_{\mathbb{R}^n}\rho(x)dx=1 ∫ R n ρ ( x ) d x = 1
Then, for ε > 0 \varepsilon>0 ε > 0 , ρ ε ( x ) = ( 1 / ε ) n ρ ( x / ε ) \rho_\varepsilon(x)=(1/\varepsilon)^n\rho(x/\varepsilon) ρ ε ( x ) = ( 1/ ε ) n ρ ( x / ε ) . If u ( x ) u(x) u ( x ) is locally summable, u ∈ L l o c 1 ( R n ) u\in L^1_{\mathrm{loc}}(\mathbb{R}^n) u ∈ L loc 1 ( R n ) , define the ρ ε ∗ \rho_\varepsilon\ast ρ ε ∗ operator, called a mollifier, as
ρ ε ∗ u ( x ) = ∫ R n ρ ε ( x − y ) u ( y ) d y \rho_\varepsilon\ast u(x)=\int_{\mathbb{R}^n}\rho_\varepsilon(x-y)u(y)dy ρ ε ∗ u ( x ) = ∫ R n ρ ε ( x − y ) u ( y ) d y
ρ ε ∗ u ( x ) \rho_\varepsilon\ast u(x) ρ ε ∗ u ( x ) has continuous partial derivatives of all orders, i.e. ρ ε ∗ u ( x ) ∈ C ∞ \rho_\varepsilon\ast u(x) \in C^\infty ρ ε ∗ u ( x ) ∈ C ∞ .
Proposition 1: Differentiation of a Mollified Function. Let u ∈ L l o c 1 ( R n ) u \in L^1_{\mathrm{loc}}(\mathbb{R}^n) u ∈ L loc 1 ( R n ) and let ρ ε \rho_\varepsilon ρ ε be a mollifier. Then
ρ ε ∗ u ∈ C ∞ ( R n ) , \rho_\varepsilon\ast u \in C^\infty(\mathbb{R}^n), ρ ε ∗ u ∈ C ∞ ( R n ) , and for every multi-index α \alpha α ,
D α ( ρ ε ∗ u ) = ( D α ρ ε ) ∗ u . D^\alpha(\rho_\varepsilon\ast u) = (D^\alpha\rho_\varepsilon) \ast u. D α ( ρ ε ∗ u ) = ( D α ρ ε ) ∗ u . Equivalently,
∂ ∣ α ∣ ∂ x α ( ∫ R n ρ ε ( x − y ) u ( y ) d y ) = ∫ R n D α ρ ε ( x − y ) u ( y ) d y . \frac{\partial^{|\alpha|}}{\partial x^\alpha}\left(\int_{\mathbb{R}^n} \rho_\varepsilon(x-y)u(y)\,dy\right)=\int_{\mathbb{R}^n}D^\alpha\rho_\varepsilon(x-y)u(y)\,dy. ∂ x α ∂ ∣ α ∣ ( ∫ R n ρ ε ( x − y ) u ( y ) d y ) = ∫ R n D α ρ ε ( x − y ) u ( y ) d y . Hence all derivatives of ρ ε ∗ u \rho_\varepsilon\ast u ρ ε ∗ u exist and are continuous.
The support of ρ ε ∗ u ( x ) \rho_\varepsilon\ast u(x) ρ ε ∗ u ( x ) is contained in the ε \varepsilon ε neighborhood of support of u ( x ) u(x) u ( x ) when u u u has a well-defined support (e.g. u ∈ L l o c 1 u \in L^1_{\mathrm{loc}} u ∈ L loc 1 ).
Proposition 2: Support of a Mollified Function. If u ∈ L l o c 1 ( R n ) u \in L^1_{\mathrm{loc}}(\mathbb{R}^n) u ∈ L loc 1 ( R n ) , then
supp ( ρ ε ∗ u ) ⊆ supp ( u ) + B ε ( 0 ) ‾ . \operatorname{supp}(\rho_\varepsilon\ast u) \subseteq\operatorname{supp}(u)+\overline{B_\varepsilon(0)}. supp ( ρ ε ∗ u ) ⊆ supp ( u ) + B ε ( 0 ) . where
A + B = { a + b : a ∈ A , b ∈ B } A+B=\{a+b:a\in A,\ b\in B\} A + B = { a + b : a ∈ A , b ∈ B } denotes the Minkowski sum of two sets.
If u ( x ) u(x) u ( x ) is a continuous function with compact support, then ρ ε ∗ u ( x ) → u ( x ) \rho_\varepsilon\ast u(x) \to u(x) ρ ε ∗ u ( x ) → u ( x ) uniformly on R n \mathbb{R}^n R n as ε → 0 \varepsilon\to0 ε → 0 . If u ( x ) ∈ L p ( R n ) u(x) \in L^p(\mathbb{R}^n) u ( x ) ∈ L p ( R n ) (p ≥ 1 p \ge1 p ≥ 1 ), then ρ ε ∗ u ( x ) → u ( x ) \rho_\varepsilon\ast u(x) \to u(x) ρ ε ∗ u ( x ) → u ( x ) in L p L^p L p as ε → 0 \varepsilon\to0 ε → 0 .
In topology, the Tietze extension theorem (also known as the Tietze–Urysohn–Brouwer extension theorem or Urysohn-Brouwer lemma)12 states that any real-valued, continuous function on a closed subset of a normal topological space can be extended to the entire space, preserving boundedness if necessary.
Theorem B.2: Tietze Extension Theorem. Let X X X be a normal topological space and let A ⊂ X A \subset X A ⊂ X be a closed subset. Every continuous function f : A → R f:A\to\mathbb{R} f : A → R admits a continuous extension f ~ : X → R \widetilde{f}:X\to\mathbb{R} f : X → R such that f ~ ∣ A = f \widetilde{f}|_A=f f ∣ A = f , satisfying that supp f \operatorname{supp}f supp f is compact. Moreover, if f f f is bounded, then f ~ \widetilde{f} f may be chosen so that ∥ f ~ ∥ ∞ = ∥ f ∥ ∞ \|\widetilde{f}\|_\infty=\|f\|_\infty ∥ f ∥ ∞ = ∥ f ∥ ∞ .
Corollary B.3. Let K ⊂ R n K\subset\mathbb{R}^n K ⊂ R n be compact and let f ∈ C ( K ) f\in C(K) f ∈ C ( K ) . Then there exists a function f ~ ∈ C ( R n ) \widetilde{f}\in C(\mathbb{R}^n) f ∈ C ( R n ) such that
f ~ ( x ) = f ( x ) , x ∈ K . \widetilde{f}(x)=f(x),\quad x\in K. f ( x ) = f ( x ) , x ∈ K .
Proof sketch. Based on (12) and (23).
Extension: Extend f f f to a continuous function f ~ ∈ C ( R n ) \widetilde{f}\in C(\mathbb{R}^n) f ∈ C ( R n ) using Tietze’s extension theorem, so that we pass from f ∈ C ( K ) f\in C(K) f ∈ C ( K ) to a continuous function f ~ ∈ C ( R n ) \widetilde{f}\in C(\mathbb{R}^n) f ∈ C ( R n ) that can then be convolved with the Schwartz approximate identity.
Choose a function ϕ ∈ S ( R n ) \phi\in\mathcal{S}(\mathbb{R}^n) ϕ ∈ S ( R n ) whose Fourier transform ϕ ^ ∈ C ∞ ( R n ) \widehat{\phi}\in C^\infty(\mathbb{R}^n) ϕ ∈ C ∞ ( R n ) satisfies ϕ ^ ( 0 ) = 1 \widehat{\phi}(0)=1 ϕ ( 0 ) = 1 .
For ε > 0 \varepsilon>0 ε > 0 , define
ϕ ε ( x ) = ε − n ϕ ( x / ε ) . \phi_\varepsilon(x)=\varepsilon^{-n}\phi(x/\varepsilon). ϕ ε ( x ) = ε − n ϕ ( x / ε ) .
Then
ϕ ε ^ ( ξ ) = ϕ ^ ( ε ξ ) , \widehat{\phi_\varepsilon}(\xi)=\widehat{\phi}(\varepsilon\xi), ϕ ε ( ξ ) = ϕ ( ε ξ ) ,
so ϕ ε ^ \widehat{\phi_\varepsilon} ϕ ε has compact support and { ϕ ε } ε > 0 \{\phi_\varepsilon\}_{\varepsilon>0} { ϕ ε } ε > 0 is an approximate identity.
F ε = f ~ ∗ ϕ ε . F_\varepsilon=\widetilde{f}\ast\phi_\varepsilon. F ε = f ∗ ϕ ε .
Since ϕ ε \phi_\varepsilon ϕ ε is a Schwartz function, F ε ∈ C ∞ ( R n ) F_\varepsilon\in C^\infty(\mathbb{R}^n) F ε ∈ C ∞ ( R n ) . Moreover, F [ F ε ] = F [ f ~ ] F [ ϕ ε ] \mathcal{F}[F_\varepsilon]=\mathcal{F}[\widetilde{f}]\mathcal{F}[\phi_\varepsilon] F [ F ε ] = F [ f ] F [ ϕ ε ] . Because F [ ϕ ε ] \mathcal{F}[\phi_\varepsilon] F [ ϕ ε ] is compactly supported, F [ F ε ] \mathcal{F}[F_\varepsilon] F [ F ε ] is compactly supported as well. Hence F ε F_\varepsilon F ε is band-limited.
F ε = f ~ ∗ ϕ ε ⟶ f ~ F_\varepsilon=\widetilde{f}\ast\phi_\varepsilon\longrightarrow\widetilde{f} F ε = f ∗ ϕ ε ⟶ f
uniformly on K K K as ε → 0 \varepsilon\to0 ε → 0 .
Therefore, for every δ > 0 \delta>0 δ > 0 , there exists ε > 0 \varepsilon>0 ε > 0 such that
sup x ∈ K ∣ F ε ( x ) − f ( x ) ∣ < δ . \sup_{x\in K}|F_\varepsilon(x)-f(x)|<\delta. x ∈ K sup ∣ F ε ( x ) − f ( x ) ∣ < δ .
Thus f f f can be uniformly approximated on K K K by smooth band-limited functions.
Lemma B.4: Information on ϕ ( z ) \phi(z) ϕ ( z ) . Let ϕ ( z ) \phi(z) ϕ ( z ) be a non-constant, bounded, and monotonically increasing continuous function. Then there exist constants δ , α > 0 \delta,\alpha>0 δ , α > 0 such that ψ ( z ) \psi(z) ψ ( z ) is integrable (i.e. ψ ( z ) ∈ L 1 ( R ) \psi(z)\in L^1(\mathbb{R}) ψ ( z ) ∈ L 1 ( R ) ) and F [ ψ ] ( 1 ) ≠ 0 \mathcal{F}[\psi](1)\ne0 F [ ψ ] ( 1 ) = 0 , where ψ ( z ) = ϕ ( z / δ + α ) − ϕ ( z / δ + α ) \psi(z)=\phi(z/ \delta+\alpha)-\phi(z/ \delta+\alpha) ψ ( z ) = ϕ ( z / δ + α ) − ϕ ( z / δ + α ) . In particular, ψ ( z ) \psi(z) ψ ( z ) is real-valued because ϕ ( z ) \phi(z) ϕ ( z ) is real-valued.
Proof. Let L , L ′ > 0 L,L' > 0 L , L ′ > 0 be arbitrary large numbers. Then ψ ( z ) \psi(z) ψ ( z ) is integrable:
∫ − L ′ L ψ ( z ) d z = δ ∫ − L ′ / δ + α L / δ + α ϕ ( s ) d s − δ ∫ − L ′ / δ − α L / δ − α ϕ ( s ) d s = δ ∫ L / δ − α L / δ + α ϕ ( s ) d s − δ ∫ − L ′ / δ − α − L ′ / δ + α ϕ ( s ) d s \begin{aligned}
\int_{-L'}^{L}\psi(z)\,dz=\delta\int_{-L'/\delta+\alpha}^{L/\delta+\alpha}\phi(s)\,ds-\delta\int_{-L'/\delta-\alpha}^{L/\delta-\alpha}\phi(s)\,ds \\
&=\delta\int_{L/\delta-\alpha}^{L/\delta+\alpha}\phi(s)\,ds-\delta\int_{-L'/\delta-\alpha}^{-L'/\delta+\alpha}\phi(s)\,ds
\end{aligned} ∫ − L ′ L ψ ( z ) d z = δ ∫ − L ′ / δ + α L / δ + α ϕ ( s ) d s − δ ∫ − L ′ / δ − α L / δ − α ϕ ( s ) d s = δ ∫ L / δ − α L / δ + α ϕ ( s ) d s − δ ∫ − L ′ / δ − α − L ′ / δ + α ϕ ( s ) d s
∈ [ 0 , 4 δ α sup ∣ ϕ ∣ ] \in[0,4\delta\alpha\sup|\phi|] ∈ [ 0 , 4 δ α sup ∣ ϕ ∣ ]
If F [ ψ ] ( 1 ) = 0 \mathcal{F}[\psi](1)=0 F [ ψ ] ( 1 ) = 0 for all δ > 0 \delta>0 δ > 0 , then F [ ϕ ( ⋅ + α ) − ϕ ( ⋅ − α ) ] = 0 \mathcal{F}[\phi(\cdot+\alpha)-\phi(\cdot-\alpha)]=0 F [ ϕ ( ⋅ + α ) − ϕ ( ⋅ − α )] = 0 , so ϕ ( z + α ) = ϕ ( z − α ) \phi(z+\alpha)=\phi(z-\alpha) ϕ ( z + α ) = ϕ ( z − α ) . If u = z − α u=z-\alpha u = z − α , when ϕ ( u ) = ϕ ( u + 2 α ) \phi(u)=\phi(u+2\alpha) ϕ ( u ) = ϕ ( u + 2 α ) . ϕ ( z ) \phi(z) ϕ ( z ) would be a periodic function with period 2 α 2\alpha 2 α . Combining this periodicity with the assumption that ϕ \phi ϕ is increasing, ϕ \phi ϕ would be constant on [ 0 , 2 α ] [0,2\alpha] [ 0 , 2 α ] and, again from its periodicity, ϕ ( z ) \phi(z) ϕ ( z ) would be constant on R \mathbb{R} R . But this contradicts the assumption that ϕ \phi ϕ is not constant. Therefore, F [ ψ ] ( 1 ) ≠ 0 \mathcal{F}[\psi](1)\ne0 F [ ψ ] ( 1 ) = 0 .
Funahashi applies the continuous Fourier transform to prove a discrete theorem, so we need to transform continuous representations into discrete ones:
Lemma B.5: From continuous to discrete Let A > 0 A>0 A > 0 , K ⊂ R n K\subset\mathbb{R}^{n} K ⊂ R n a compact set, and h ( w , x ) h(w,x) h ( w , x ) a continuous function on [ − A , A ] n × K [-A,A]^{n}\times K [ − A , A ] n × K . Let us define the functions H ( x ) H(x) H ( x ) and H N ( x ) H_N(x) H N ( x ) , with N ∈ N N\in\mathbb{N} N ∈ N on K K K :
H ( x ) = ∫ [ − A , A ] n h ( w , x ) d w , H(x)=\int_{[-A,A]^{n}}h(w,x)\,dw, H ( x ) = ∫ [ − A , A ] n h ( w , x ) d w , H N ( x ) = κ n ∑ k 1 , … , k n = 0 N − 1 h ( − A + κ k 1 , … , − A + κ k n , x ) H_N(x)=\kappa^n\sum_{k_1,\ldots,k_n=0}^{N-1}h(-A+\kappa k_1,\ldots,-A+\kappa k_n,x) H N ( x ) = κ n k 1 , … , k n = 0 ∑ N − 1 h ( − A + κ k 1 , … , − A + κ k n , x ) with κ = 2 A / N \kappa=2A/N κ = 2 A / N . Then, for all ϵ > 0 \epsilon>0 ϵ > 0 , there exists N 0 ∈ N N_0\in\mathbb{N} N 0 ∈ N such that max x ∈ K ∣ H ( x ) − H N ( x ) ∣ < ϵ \max_{x\in K}|H(x)-H_N(x)|<\epsilon max x ∈ K ∣ H ( x ) − H N ( x ) ∣ < ϵ for all N ≥ N 0 N\ge N_0 N ≥ N 0 .
Proof. Let 1 = ( 1 , 1 , … , 1 ) ∈ R n \mathbf{1}=(1,1,\ldots,1)\in\mathbb{R}^{n} 1 = ( 1 , 1 , … , 1 ) ∈ R n and k ∈ { 1 , 2 , … , N − 1 } n \mathbf{k}\in\{1,2,\ldots,N-1\}^{n} k ∈ { 1 , 2 , … , N − 1 } n meaning that k = ( k 1 , k 2 , … , k n ) \mathbf{k}=(k_1,k_2,\ldots,k_n) k = ( k 1 , k 2 , … , k n ) for every integer k j ∈ { 1 , 2 , … , N − 1 } k_j\in\{1,2,\ldots,N-1\} k j ∈ { 1 , 2 , … , N − 1 } , j = 1.. n j=1..n j = 1.. n . Then
∑ k ∈ { 1 , 2 , … , N − 1 } n = ∑ k 1 … k n = 0 N − 1 \sum_{\mathbf{k}\in\{1,2,\ldots,N-1\}^{n}}=\sum_{k_1\ldots k_n=0}^{N-1} k ∈ { 1 , 2 , … , N − 1 } n ∑ = k 1 … k n = 0 ∑ N − 1
For any k \mathbf{k} k :
( − A + κ k 1 , … , − A + κ k n , x ) = − A 1 + κ k = − A 1 + 2 A N k (-A+\kappa k_1,\ldots,-A+\kappa k_n,x)=-A\mathbf{1}+\kappa\mathbf{k}=-A\mathbf{1}+\frac{2A}{N}\mathbf{k} ( − A + κ k 1 , … , − A + κ k n , x ) = − A 1 + κ k = − A 1 + N 2 A k
Putting H N ( x ) H_N(x) H N ( x ) in vector notation:
H N ( x ) = κ N ∑ k ∈ { 1 , 2 , … , N − 1 } n h ( − A 1 + κ k , x ) H_N(x)=\kappa^N\sum_{\mathbf{k}\in\{1,2,\ldots,N-1\}^{n}}h(-A\mathbf{1}+\kappa\mathbf{k},x) H N ( x ) = κ N k ∈ { 1 , 2 , … , N − 1 } n ∑ h ( − A 1 + κ k , x )
We evaluate ∣ H ( x ) − H N ( x ) ∣ |H(x)-H_N(x)| ∣ H ( x ) − H N ( x ) ∣ and, by the uniform continuity of h ( w , x ) h(w,x) h ( w , x ) , for any ϵ > 0 \epsilon>0 ϵ > 0 , there exists δ > 0 \delta>0 δ > 0 such that
∣ h ( w , x ) − h ( w ′ , x ) ∣ < ϵ ( 2 A ) n |h(w,x)-h(w',x)|<\frac{\epsilon}{(2A)^n} ∣ h ( w , x ) − h ( w ′ , x ) ∣ < ( 2 A ) n ϵ
for any w , w ′ ∈ R n w,w'\in\mathbb{R}^{n} w , w ′ ∈ R n satisfying ∣ w − w ′ ∣ < δ |w-w'|<\delta ∣ w − w ′ ∣ < δ . Then, we fix N 0 ∈ N N_0\in\mathbb{N} N 0 ∈ N so that 2 A N 0 N < δ \frac{2A}{N_0}\sqrt{N}<\delta N 0 2 A N < δ and let N > N 0 N>N_0 N > N 0 :
∣ w − ( − A 1 + 2 A N k ) ∣ < 2 A N 0 N < δ \left|w-\left(-A\mathbf{1}+\frac{2A}{N}\mathbf{k}\right)\right|<\frac{2A}{N_0}\sqrt{N}<\delta w − ( − A 1 + N 2 A k ) < N 0 2 A N < δ
w ∈ ∏ j = 1 n [ − A + 2 k j A N , − A + 2 ( k j + 1 ) A N ] w\in\prod_{j=1}^{n}\left[-A+\frac{2k_jA}{N},-A+\frac{2(k_j+1)A}{N}\right] w ∈ j = 1 ∏ n [ − A + N 2 k j A , − A + N 2 ( k j + 1 ) A ]
So we obtain the following approximation
∣ h ( w , x ) − h ( − A 1 + 2 A N k , x ) ∣ < ϵ ( 2 A ) n \left|h(w,x)-h\left(-A\mathbf{1}+\frac{2A}{N}\mathbf{k},x\right)\right|<\frac{\epsilon}{(2A)^n} h ( w , x ) − h ( − A 1 + N 2 A k , x ) < ( 2 A ) n ϵ
For each k ∈ { 1 , 2 , … , N − 1 } n \mathbf{k}\in\{1,2,\ldots,N-1\}^{n} k ∈ { 1 , 2 , … , N − 1 } n , we define
C ( k ) = ∏ j = 1 n [ − A + 2 k j A N , − A + 2 ( k j + 1 ) A N ] C(\mathbf{k})=\prod_{j=1}^{n}\left[-A+\frac{2k_jA}{N},-A+\frac{2(k_j+1)A}{N}\right] C ( k ) = j = 1 ∏ n [ − A + N 2 k j A , − A + N 2 ( k j + 1 ) A ]
Then, by the definition of H N ( x ) H_N(x) H N ( x ) in vector notation and the above approximation:
∣ H ( x ) − H N ( x ) ∣ ≤ ∑ k ∣ ∫ C ( k ) h ( w , x ) d w − ∫ C ( k ) h ( − A 1 + 2 A N k , x ) d w ∣ = ∑ k ∣ ∫ C ( k ) ( h ( w , x ) − h ( − A 1 + 2 A N k , x ) ) d w ∣ ≤ ∑ k ϵ ( 2 A ) n ( 2 A N ) n ≤ ϵ \begin{aligned}
|H(x)-H_N(x)|
\le\sum_{\mathbf{k}}\left|\int_{C(\mathbf{k})}h(w,x)\,dw-\int_{C(\mathbf{k})}h\left(-A\mathbf{1}+\frac{2A}{N}\mathbf{k},x\right)\,dw\right| \\
&=\sum_{\mathbf{k}}\left|\int_{C(\mathbf{k})}\left(h(w,x)-h\left(-A\mathbf{1}+\frac{2A}{N}\mathbf{k},x\right)\right)\,dw\right| \\
&\le\sum_{\mathbf{k}}\frac{\epsilon}{(2A)^n}\left(\frac{2A}{N}\right)^n \\
&\le\epsilon
\end{aligned} ∣ H ( x ) − H N ( x ) ∣ ≤ k ∑ ∫ C ( k ) h ( w , x ) d w − ∫ C ( k ) h ( − A 1 + N 2 A k , x ) d w = k ∑ ∫ C ( k ) ( h ( w , x ) − h ( − A 1 + N 2 A k , x ) ) d w ≤ k ∑ ( 2 A ) n ϵ ( N 2 A ) n ≤ ϵ
By the use of the Fourier transform in the real line, we can approximate the Fourier inverse transform of the Fourier transform (23). Let us represent the n n n -dimensional hypercube centered at the origin with a side length of 2 A 2A 2 A (i.e. a closed ball under the infinity norm) as I A n = B ∞ [ 0 , A ] = [ − A , A ] n I_A^n=B_\infty[0,A]=[-A,A]^n I A n = B ∞ [ 0 , A ] = [ − A , A ] n and define the following functions;
M [ f ] ( w ) = ( 1 2 π ) n F [ f ] ( w ) F [ ψ ] ( 1 ) M[f](w)=\left(\frac{1}{2\pi}\right)^n\frac{\mathcal{F}[f](w)}{\mathcal{F}[\psi](1)} M [ f ] ( w ) = ( 2 π 1 ) n F [ ψ ] ( 1 ) F [ f ] ( w )
I A ′ , A [ f ] ( x ) = ∫ I A n ∫ − A ′ A ′ ψ ( x ⋅ w − w 0 ) ⋅ M [ f ] ( w ) e i w 0 d w 0 d w I_{A',A}[f](x)=\int_{I_A^n}\int_{-A'}^{A'}\psi(x\cdot w-w_0)\cdot M[f](w)e^{iw_0}\,dw_0\,dw I A ′ , A [ f ] ( x ) = ∫ I A n ∫ − A ′ A ′ ψ ( x ⋅ w − w 0 ) ⋅ M [ f ] ( w ) e i w 0 d w 0 d w
I ∞ , A [ f ] ( x ) = lim A ′ → ∞ I A ′ , A [ f ] ( x ) I_{\infty,A}[f](x)=\lim_{A'\to\infty}I_{A',A}[f](x) I ∞ , A [ f ] ( x ) = A ′ → ∞ lim I A ′ , A [ f ] ( x )
J A [ f ] ( x ) = ( 1 2 π ) n ∫ I A n F [ f ] ( w ) e i w ⋅ x d w J_A[f](x)=\left(\frac{1}{2\pi}\right)^n\int_{I_A^n}\mathcal{F}[f](w)e^{iw\cdot x}\,dw J A [ f ] ( x ) = ( 2 π 1 ) n ∫ I A n F [ f ] ( w ) e i w ⋅ x d w
Lemma B.6: Convergence of I I I and J J J . Assuming that f ( x ) ∈ L 1 ( R n ) f(x) \in L^{1}(\mathbb{R}^{n}) f ( x ) ∈ L 1 ( R n ) satisfies F [ f ] ( w ) ∈ L 1 ( R n ) \mathcal{F}[f](w) \in L^{1}(\mathbb{R}^{n}) F [ f ] ( w ) ∈ L 1 ( R n ) (i.e. both f ( x ) f(x) f ( x ) and its Fourier transform are integrable), for all 0 < A < ∞ 0<A<\infty 0 < A < ∞ and x ∈ R n x\in\mathbb{R}^{n} x ∈ R n ,
I ∞ , A [ f ] ( x ) = J A [ f ] ( x ) I_{\infty,A}[f](x)=J_{A}[f](x) I ∞ , A [ f ] ( x ) = J A [ f ] ( x ) In addition, both { I ∞ , A [ f ] ( x ) } A > 0 \{I_{\infty,A}[f](x)\}_{A>0} { I ∞ , A [ f ] ( x ) } A > 0 and { J A [ f ] ( x ) } A > 0 \{J_{A}[f](x)\}_{A>0} { J A [ f ] ( x ) } A > 0 converge uniformly in R n \mathbb{R}^{n} R n .
Proof. Given ψ ( z ) \psi(z) ψ ( z ) defined in Lemma Lemma B.4, we can define
Ψ A ′ ( x ) = ∫ − ∞ ∞ ψ ( z ) e − i z χ [ x ⋅ w − A ′ , x ⋅ w + A ′ ] ( z ) d z \Psi_{A'}(x)=\int_{-\infty}^{\infty}\psi(z)e^{-iz\chi_{[x\cdot w-A',x\cdot w+A']}(z)}\,dz Ψ A ′ ( x ) = ∫ − ∞ ∞ ψ ( z ) e − i z χ [ x ⋅ w − A ′ , x ⋅ w + A ′ ] ( z ) d z where χ K \chi_{K} χ K be a smooth cutoff that equals 1 1 1 on K K K and decays outside a slightly larger compact set. By the Lebesgue dominated convergence theorem (Theorem A.3),
lim A ′ → ∞ Ψ A ′ ( x ) = F [ ψ ] ( 1 ) \lim_{A'\to\infty}\Psi_{A'}(x)=\mathcal{F}[\psi](1) A ′ → ∞ lim Ψ A ′ ( x ) = F [ ψ ] ( 1 ) To prove that I ∞ , A [ f ] ( x ) = J A [ f ] ( x ) I_{\infty,A}[f](x)=J_{A}[f](x) I ∞ , A [ f ] ( x ) = J A [ f ] ( x ) , it suffices to prove that
lim A ′ → ∞ ∫ I A n F [ f ] ( w ) e i w ⋅ x Ψ A ′ ( x ) d w = ∫ I A n F [ f ] ( w ) e i w ⋅ x F [ ψ ] ( 1 ) d w \begin{aligned}
\lim_{A'\to\infty}\int_{I_{A}^{n}}\mathcal{F}[f](w)e^{iw\cdot x}\Psi_{A'}(x)\,dw
&=\int_{I_{A}^{n}}\mathcal{F}[f](w)e^{iw\cdot x}\mathcal{F}[\psi](1)\,dw
\end{aligned} A ′ → ∞ lim ∫ I A n F [ f ] ( w ) e i w ⋅ x Ψ A ′ ( x ) d w = ∫ I A n F [ f ] ( w ) e i w ⋅ x F [ ψ ] ( 1 ) d w We can observe that
∣ F [ f ] ( w ) e i w ⋅ x Ψ A ′ ( x ) ∣ ≤ ∣ F [ f ] ( w ) ∣ ⋅ ∥ ψ ∥ L 1 ( R ) \left|\mathcal{F}[f](w)e^{iw\cdot x}\Psi_{A'}(x)\right|\leq|\mathcal{F}[f](w)|\cdot\|\psi\|_{L^{1}(\mathbb{R})} F [ f ] ( w ) e i w ⋅ x Ψ A ′ ( x ) ≤ ∣ F [ f ] ( w ) ∣ ⋅ ∥ ψ ∥ L 1 ( R ) and that bound is independent of A ′ A' A ′ and integrable on [ − A , A ] n [-A,A]^{n} [ − A , A ] n . Therefore, applying the Lebesgue dominated convergence theorem again, we obtain the desired result, so I ∞ , A [ f ] ( x ) = J A [ f ] ( x ) I_{\infty,A}[f](x)=J_{A}[f](x) I ∞ , A [ f ] ( x ) = J A [ f ] ( x ) .
Moreover, { J A [ f ] ( x ) } A > 0 \{J_{A}[f](x)\}_{A>0} { J A [ f ] ( x ) } A > 0 converges to F − 1 [ F [ f ] ] ( x ) \mathcal{F}^{-1}[\mathcal{F}[f]](x) F − 1 [ F [ f ]] ( x ) uniformly in R n \mathbb{R}^{n} R n . Since F [ f ] ( w ) \mathcal{F}[f](w) F [ f ] ( w ) is integrable:
sup x ∈ R n ∣ F − 1 [ F [ f ] ] ( x ) − J A [ f ] ( x ) ∣ = ( 1 2 π ) n sup x ∈ R n ∣ ∫ R n F [ f ] ( w ) e i w ⋅ x ( 1 − χ I A n ( w ) ) d w ∣ ≤ ( 1 2 π ) n ∫ R n ∣ F [ f ] ( w ) ∣ ( 1 − χ I A n ( w ) ) d w → 0 ( A → ∞ ) \begin{aligned}
\sup_{x\in\mathbb{R}^{n}}\left|\mathcal{F}^{-1}[\mathcal{F}[f]](x)-J_{A}[f](x)\right|
&=\left(\frac{1}{2\pi}\right)^{n}\sup_{x\in\mathbb{R}^{n}}\left|\int_{\mathbb{R}^{n}}\mathcal{F}[f](w)e^{iw\cdot x}(1-\chi_{I_{A}^{n}}(w))\,dw\right|\\
&\leq\left(\frac{1}{2\pi}\right)^{n}\int_{\mathbb{R}^{n}}|\mathcal{F}[f](w)|(1-\chi_{I_{A}^{n}}(w))\,dw\\
&\to0\quad(A\to\infty)
\end{aligned} x ∈ R n sup F − 1 [ F [ f ]] ( x ) − J A [ f ] ( x ) = ( 2 π 1 ) n x ∈ R n sup ∫ R n F [ f ] ( w ) e i w ⋅ x ( 1 − χ I A n ( w )) d w ≤ ( 2 π 1 ) n ∫ R n ∣ F [ f ] ( w ) ∣ ( 1 − χ I A n ( w )) d w → 0 ( A → ∞ )
Theorem I.2: Universal approximation theorem for artificial neural networks with bounded, monotonically increasing activation functions. Let ϕ ( x ) \phi(x) ϕ ( x ) be a non-constant, bounded, and monotonically increasing continuous function. Let K K K be a compact subset (bounded closed subset) of R n \mathbb{R}^{n} R n and f f f a real valued continuous function on K K K . Then, for an arbitrary ϵ > 0 \epsilon>0 ϵ > 0 , there exists an integer N N N and real constants α i \alpha_i α i , b i b_i b i , and w i j w_{ij} w ij such that
f ~ ( x ) = ∑ j = 1 h α j ϕ ( ∑ i = 1 n w i j x i + b j ) \tilde{f}(x)=\sum_{j=1}^{h}\alpha_j\phi\left(\sum_{i=1}^{n}w_{ij}x_i+b_j\right) f ~ ( x ) = j = 1 ∑ h α j ϕ ( i = 1 ∑ n w ij x i + b j ) satisfies max k ∈ K ∣ f ( x ) − f ~ ( x ) ∣ < ϵ \max_{k\in K}|f(x)-\tilde{f}(x)|<\epsilon max k ∈ K ∣ f ( x ) − f ~ ( x ) ∣ < ϵ .
Remark. As in Cybenko’s universal approximation theorem, f ~ ( x ) \tilde{f}(x) f ~ ( x ) belongs to the space of standard three-layer feedforward neural networks: n n n inputs, one hidden layer containing h h h neurons with activation function ϕ \phi ϕ , and a single linear output neuron. The universal approximation theorem states that a multilayer network composed of at least one hidden layer can approximate any continuous function if the number of hidden units is large enough and the parameters (weights and thresholds) are appropriately adjusted.
Proof. (23) We know that I ∞ , A [ f ] ( x ) = J A [ f ] ( x ) I_{\infty,A}[f](x)=J_{A}[f](x) I ∞ , A [ f ] ( x ) = J A [ f ] ( x ) for all 0 < A < ∞ 0<A<\infty 0 < A < ∞ and x ∈ R n x\in\mathbb{R}^{n} x ∈ R n (Lemma B.6). Let us suppose that f ∈ C c ∞ ( R n ) ⊂ S ( R n ) f\in C_{c}^{\infty}(\mathbb{R}^{n})\subset S(\mathbb{R}^{n}) f ∈ C c ∞ ( R n ) ⊂ S ( R n ) ,
f ( x ) = F − 1 [ F [ f ] ] ( x ) = lim A → ∞ J A [ f ] ( x ) = lim A → ∞ I ∞ , A [ f ] ( x ) f(x)=\mathcal{F}^{-1}[\mathcal{F}[f]](x)=\lim_{A\to\infty}J_{A}[f](x)=\lim_{A\to\infty}I_{\infty,A}[f](x) f ( x ) = F − 1 [ F [ f ]] ( x ) = A → ∞ lim J A [ f ] ( x ) = A → ∞ lim I ∞ , A [ f ] ( x ) which converges uniformly in R n \mathbb{R}^{n} R n . Therefore, there exists A 0 > 0 A_{0}>0 A 0 > 0 such that for all A > A 0 A>A_{0} A > A 0 :
max x ∈ R n ∣ f ( x ) − I ∞ , A [ f ] ( x ) ∣ < ϵ 3 \max_{x\in\mathbb{R}^{n}}|f(x)-I_{\infty,A}[f](x)|<\frac{\epsilon}{3} x ∈ R n max ∣ f ( x ) − I ∞ , A [ f ] ( x ) ∣ < 3 ϵ We can approximate I ∞ , A [ f ] ( x ) I_{\infty,A}[f](x) I ∞ , A [ f ] ( x ) on K K K using I A ′ , A [ f ] ( x ) I_{A',A}[f](x) I A ′ , A [ f ] ( x ) with 0 < A ′ < ∞ 0<A'<\infty 0 < A ′ < ∞ . For a particular x ∈ K x\in K x ∈ K :
∣ I ∞ , A [ f ] ( x ) − I A ′ , A [ f ] ( x ) ∣ \left|I_{\infty,A}[f](x)-I_{A',A}[f](x)\right| ∣ I ∞ , A [ f ] ( x ) − I A ′ , A [ f ] ( x ) ∣ ≤ 1 ( 2 π ) n ∣ ψ ^ ( 1 ) ∣ ∫ I A n ∣ f ^ ( w ) ∣ ∫ R ∖ [ − A , A ] ∣ ψ ( x ⋅ w − w 0 ) ∣ d w 0 d w \leq\frac{1}{(2\pi)^{n}|\hat{\psi}(1)|}\int_{I_{A}^{n}}|\hat{f}(w)|\int_{\mathbb{R}\setminus[-A,A]}|\psi(x\cdot w-w_{0})|\,dw_{0}\,dw ≤ ( 2 π ) n ∣ ψ ^ ( 1 ) ∣ 1 ∫ I A n ∣ f ^ ( w ) ∣ ∫ R ∖ [ − A , A ] ∣ ψ ( x ⋅ w − w 0 ) ∣ d w 0 d w = 1 ( 2 π ) n ∣ ψ ^ ( 1 ) ∣ ∫ I A n ∣ f ^ ( w ) ∣ ∫ − ∞ ∞ ∣ ψ ( t ) ∣ χ D ( t ) d t d w =\frac{1}{(2\pi)^{n}|\hat{\psi}(1)|}\int_{I_{A}^{n}}|\hat{f}(w)|\int_{-\infty}^{\infty}|\psi(t)|\chi_{D}(t)\,dt\,dw = ( 2 π ) n ∣ ψ ^ ( 1 ) ∣ 1 ∫ I A n ∣ f ^ ( w ) ∣ ∫ − ∞ ∞ ∣ ψ ( t ) ∣ χ D ( t ) d t d w where ψ ^ = F [ ψ ] \hat{\psi}=\mathcal{F}[\psi] ψ ^ = F [ ψ ] , f ^ = F [ f ] \hat{f}=\mathcal{F}[f] f ^ = F [ f ] , and D = R ∖ [ x ⋅ w − A ′ , x ⋅ w + A ′ ] D=\mathbb{R}\setminus[x\cdot w-A',x\cdot w+A'] D = R ∖ [ x ⋅ w − A ′ , x ⋅ w + A ′ ] . Given that K K K is bounded, there exists R > 0 R>0 R > 0 with K ⊂ B ( R ) K\subset B(R) K ⊂ B ( R ) . For w ∈ I A n = [ − A , A ] n w\in I_{A}^{n}=[-A,A]^{n} w ∈ I A n = [ − A , A ] n , ∣ x ⋅ w ∣ ≤ ∥ x ∥ ∥ w ∥ ≤ R ⋅ n A |x\cdot w|\leq\|x\|\|w\|\leq R\cdot\sqrt{n}A ∣ x ⋅ w ∣ ≤ ∥ x ∥∥ w ∥ ≤ R ⋅ n A :
D = R ∖ [ x ⋅ w − A ′ , x ⋅ w + A ′ ] = ( − ∞ , x ⋅ w − A ′ ) ∪ ( x ⋅ w + A ′ , + ∞ ) ⊂ ( − ∞ , n R A − A ′ ) ∪ ( − n R A + A ′ , + ∞ ) = : E \begin{aligned}
D&=\mathbb{R}\setminus[x\cdot w-A',x\cdot w+A']\\
&=(-\infty,x\cdot w-A')\cup(x\cdot w+A',+\infty)\\
&\subset(-\infty,\sqrt{n}RA-A')\cup(-\sqrt{n}RA+A',+\infty)=:E
\end{aligned} D = R ∖ [ x ⋅ w − A ′ , x ⋅ w + A ′ ] = ( − ∞ , x ⋅ w − A ′ ) ∪ ( x ⋅ w + A ′ , + ∞ ) ⊂ ( − ∞ , n R A − A ′ ) ∪ ( − n R A + A ′ , + ∞ ) =: E The set E E E is independent of x x x and w w w . Hence:
( 2 π ) n ∣ ψ ^ ( 1 ) ∣ max x ∈ K ∣ I ∞ , A [ f ] ( x ) − I A ′ , A [ f ] ( x ) ∣ ≤ ∫ I A n ∣ f ^ ( w ) ∣ d w ⋅ max x ∈ K , w ∈ I A n ∫ − ∞ ∞ ∣ ψ ( t ) ∣ χ D ( t ) d t ≤ ∫ I A n ∣ f ^ ( w ) ∣ d w ⋅ ∫ − ∞ ∞ ∣ ψ ( t ) ∣ χ E ( t ) d t \begin{aligned}
(2\pi)^{n}|\hat{\psi}(1)|\max_{x\in K}|I_{\infty,A}[f](x)-I_{A',A}[f](x)|
&\leq\int_{I_{A}^{n}}|\hat{f}(w)|\,dw\cdot\max_{\substack{x\in K,w\in I_{A}^{n}}}\int_{-\infty}^{\infty}|\psi(t)|\chi_{D}(t)\,dt\\
&\leq\int_{I_{A}^{n}}|\hat{f}(w)|\,dw\cdot\int_{-\infty}^{\infty}|\psi(t)|\chi_{E}(t)\,dt
\end{aligned} ( 2 π ) n ∣ ψ ^ ( 1 ) ∣ x ∈ K max ∣ I ∞ , A [ f ] ( x ) − I A ′ , A [ f ] ( x ) ∣ ≤ ∫ I A n ∣ f ^ ( w ) ∣ d w ⋅ x ∈ K , w ∈ I A n max ∫ − ∞ ∞ ∣ ψ ( t ) ∣ χ D ( t ) d t ≤ ∫ I A n ∣ f ^ ( w ) ∣ d w ⋅ ∫ − ∞ ∞ ∣ ψ ( t ) ∣ χ E ( t ) d t
We can observe that lim A ′ → ∞ ∣ ψ ( t ) ∣ χ E ( t ) = 0 \lim_{A'\to\infty}|\psi(t)|\chi_{E}(t)=0 lim A ′ → ∞ ∣ ψ ( t ) ∣ χ E ( t ) = 0 , ∣ ψ ( t ) ∣ ∈ L 1 ( R ) |\psi(t)|\in L^{1}(\mathbb{R}) ∣ ψ ( t ) ∣ ∈ L 1 ( R ) , and ∣ ψ ( t ) ∣ χ E ( t ) ≤ ∣ ψ ( t ) ∣ |\psi(t)|\chi_{E}(t)\leq|\psi(t)| ∣ ψ ( t ) ∣ χ E ( t ) ≤ ∣ ψ ( t ) ∣ . Using the Lebesgue dominated convergence theorem once more:
lim A ′ → ∞ ∫ − ∞ ∞ ∣ ψ ( t ) ∣ χ E ( t ) d t = 0. \lim_{A'\to\infty}\int_{-\infty}^{\infty}|\psi(t)|\chi_{E}(t)\,\mathrm{d}t=0. A ′ → ∞ lim ∫ − ∞ ∞ ∣ ψ ( t ) ∣ χ E ( t ) d t = 0.
In consequence, there exists A 0 ′ > 0 A'_0>0 A 0 ′ > 0 so that for all A ′ > A 0 ′ A'>A'_0 A ′ > A 0 ′ :
max x ∈ K ∣ I ∞ , A [ f ] ( x ) − I A ′ , A [ f ] ( x ) ∣ < ϵ 3 \max_{x\in K}|I_{\infty,A}[f](x)-I_{A',A}[f](x)|<\frac{\epsilon}{3} x ∈ K max ∣ I ∞ , A [ f ] ( x ) − I A ′ , A [ f ] ( x ) ∣ < 3 ϵ
Combining this with our previous approximation of f ( x ) f(x) f ( x ) and I ∞ , A [ f ] ( x ) I_{\infty,A}[f](x) I ∞ , A [ f ] ( x ) , we obtain
max x ∈ K ∣ f ( x ) − I A ′ , A [ f ] ( x ) ∣ < 2 ϵ 3 \max_{x\in K}|f(x)-I_{A',A}[f](x)|<\frac{2\epsilon}{3} x ∈ K max ∣ f ( x ) − I A ′ , A [ f ] ( x ) ∣ < 3 2 ϵ
This result holds for f ∈ C c ∞ ( R n ) f\in C^\infty_c(\mathbb{R}^n) f ∈ C c ∞ ( R n ) . Let us now consider the general case, where f f f is a continuous function defined over K K K .
Extension: Let us now choose a real-valued extension g ( x ) ∈ C c ( R n ) g(x)\in C_c(\mathbb{R}^n) g ( x ) ∈ C c ( R n ) of f ( x ) f(x) f ( x ) , which is valid due to the Tietze extension theorem (Theorem Theorem B.2).
Let ρ ( x ) ∈ C c ∞ ( R n ) \rho(x)\in C^\infty_c(\mathbb{R}^n) ρ ( x ) ∈ C c ∞ ( R n ) be such that 0 ≤ ρ ( x ) ≤ χ B ( 1 ) ( x ) 0\leq\rho(x)\leq\chi_{B_{(1)}}(x) 0 ≤ ρ ( x ) ≤ χ B ( 1 ) ( x ) for all x ∈ R n x\in\mathbb{R}^n x ∈ R n and ∥ ρ ∥ L 1 ( R n ) = 1 \|\rho\|_{L^1(\mathbb{R}^n)}=1 ∥ ρ ∥ L 1 ( R n ) = 1 . Applying a mollifier ρ β ( x ) = β − n ρ ( β − 1 x ) \rho_\beta(x)=\beta^{-n}\rho(\beta^{-1}x) ρ β ( x ) = β − n ρ ( β − 1 x ) (Definition Definition B.5) to g ( x ) g(x) g ( x ) , we find β ∈ ( 0 , 1 ) \beta\in(0,1) β ∈ ( 0 , 1 ) such that
∥ g − ρ β ∗ g ∥ L ∞ ( R n ) < ϵ 3 \|g-\rho_\beta*g\|_{L^\infty(\mathbb{R}^n)}<\frac{\epsilon}{3} ∥ g − ρ β ∗ g ∥ L ∞ ( R n ) < 3 ϵ
Geometrically, supp ( g ) ⊂ supp ( g ∗ ρ β ) \operatorname{supp}(g)\subset\operatorname{supp}(g*\rho_\beta) supp ( g ) ⊂ supp ( g ∗ ρ β ) and supp ( ρ β ∗ g ) \operatorname{supp}(\rho_\beta*g) supp ( ρ β ∗ g ) is in a compact set L L L . Since ρ β ∗ g ( x ) ∈ C c ∞ ( R n ) \rho_\beta*g(x)\in C^\infty_c(\mathbb{R}^n) ρ β ∗ g ( x ) ∈ C c ∞ ( R n ) , we can apply our previous approximation result for infinitely differentiable functions in R n \mathbb{R}^n R n :
max x ∈ supp ( g ∗ ρ β ) ∣ ρ β ∗ g ( x ) − I A ′ , A [ ρ β ∗ g ] ( x ) ∣ < 2 ϵ 3 \max_{x\in\operatorname{supp}(g*\rho_\beta)}|\rho_\beta*g(x)-I_{A',A}[\rho_\beta*g](x)|<\frac{2\epsilon}{3} x ∈ supp ( g ∗ ρ β ) max ∣ ρ β ∗ g ( x ) − I A ′ , A [ ρ β ∗ g ] ( x ) ∣ < 3 2 ϵ
Since g ( x ) g(x) g ( x ) is an extension of f ( x ) f(x) f ( x ) :
max x ∈ K ∣ f ( x ) − I A ′ , A [ ρ β ∗ g ] ( x ) ∣ = max x ∈ K ∣ ρ β ∗ g ( x ) − I A ′ , A [ ρ β ∗ g ] ( x ) ∣ \max_{x\in K}|f(x)-I_{A',A}[\rho_\beta*g](x)|
=\max_{x\in K}|\rho_\beta*g(x)-I_{A',A}[\rho_\beta*g](x)| x ∈ K max ∣ f ( x ) − I A ′ , A [ ρ β ∗ g ] ( x ) ∣ = x ∈ K max ∣ ρ β ∗ g ( x ) − I A ′ , A [ ρ β ∗ g ] ( x ) ∣
Therefore,
max x ∈ K ∣ f ( x ) − I A ′ , A [ ρ β ∗ g ] ( x ) ∣ ≤ max x ∈ supp ( g ) ∣ g ( x ) − I A ′ , A [ ρ β ∗ g ] ( x ) ∣ ≤ max x ∈ supp ( g ) ∣ g ( x ) − ρ β ∗ g ( x ) ∣ + max x ∈ supp ( g ) ∣ ρ β ∗ g ( x ) − I A ′ , A [ ρ β ∗ g ] ( x ) ∣ ≤ ∥ g ( x ) − ρ β ∗ g ( x ) ∥ L ∞ ( R n ) + max x ∈ supp ( g ) ∣ ρ β ∗ g ( x ) − I A ′ , A [ ρ β ∗ g ] ( x ) ∣ < ϵ \begin{aligned}
\max_{x\in K}|f(x)-I_{A',A}[\rho_\beta*g](x)|
&\leq\max_{x\in\operatorname{supp}(g)}|g(x)-I_{A',A}[\rho_\beta*g](x)|\\
&\leq\max_{x\in\operatorname{supp}(g)}|g(x)-\rho_\beta*g(x)|\\
&\quad+\max_{x\in\operatorname{supp}(g)}|\rho_\beta*g(x)-I_{A',A}[\rho_\beta*g](x)|\\
&\leq\|g(x)-\rho_\beta*g(x)\|_{L^\infty(\mathbb{R}^n)}\\
&\quad+\max_{x\in\operatorname{supp}(g)}|\rho_\beta*g(x)-I_{A',A}[\rho_\beta*g](x)|\\
&<\epsilon
\end{aligned} x ∈ K max ∣ f ( x ) − I A ′ , A [ ρ β ∗ g ] ( x ) ∣ ≤ x ∈ supp ( g ) max ∣ g ( x ) − I A ′ , A [ ρ β ∗ g ] ( x ) ∣ ≤ x ∈ supp ( g ) max ∣ g ( x ) − ρ β ∗ g ( x ) ∣ + x ∈ supp ( g ) max ∣ ρ β ∗ g ( x ) − I A ′ , A [ ρ β ∗ g ] ( x ) ∣ ≤ ∥ g ( x ) − ρ β ∗ g ( x ) ∥ L ∞ ( R n ) + x ∈ supp ( g ) max ∣ ρ β ∗ g ( x ) − I A ′ , A [ ρ β ∗ g ] ( x ) ∣ < ϵ
f ( x ) f(x) f ( x ) is real-valued but I A ′ , A [ ρ β ∗ g ] ( x ) I_{A',A}[\rho_\beta*g](x) I A ′ , A [ ρ β ∗ g ] ( x ) is complex-valued. Its real part, H ( x ) H(x) H ( x ) , can be used to approximate f ( x ) f(x) f ( x ) :
∣ f ( x ) − I A ′ , A [ ρ β ∗ g ] ( x ) ∣ ≥ ∣ Re ( f ( x ) − I A ′ , A [ ρ β ∗ g ] ( x ) ) ∣ = ∣ f ( x ) − H ( x ) ∣ \begin{aligned}
|f(x)-I_{A',A}[\rho_\beta*g](x)|
&\geq|\operatorname{Re}(f(x)-I_{A',A}[\rho_\beta*g](x))|\\
&=|f(x)-H(x)|
\end{aligned} ∣ f ( x ) − I A ′ , A [ ρ β ∗ g ] ( x ) ∣ ≥ ∣ Re ( f ( x ) − I A ′ , A [ ρ β ∗ g ] ( x )) ∣ = ∣ f ( x ) − H ( x ) ∣
i.e.
max x ∈ K ∣ f ( x ) − H ( x ) ∣ < ϵ \max_{x\in K}|f(x)-H(x)|<\epsilon x ∈ K max ∣ f ( x ) − H ( x ) ∣ < ϵ
Applying Lemma B.5 to H ( x ) H(x) H ( x ) :
max x ∈ K ∣ H ( x ) − H N [ f ] ( x ) ∣ < ϵ \max_{x\in K}|H(x)-H_N[f](x)|<\epsilon x ∈ K max ∣ H ( x ) − H N [ f ] ( x ) ∣ < ϵ
where
H N [ f ] ( x ) = ( 2 A N ) n ∑ k ∈ { 1 , 2 , … , N − 1 } n h ( − A 1 + 2 A N k , x ) H_N[f](x)=\left(\frac{2A}{N}\right)^n\sum_{k\in\{1,2,\ldots,N-1\}^n}h\left(-A\mathbf{1}+\frac{2A}{N}k,x\right) H N [ f ] ( x ) = ( N 2 A ) n k ∈ { 1 , 2 , … , N − 1 } n ∑ h ( − A 1 + N 2 A k , x )
h ( w , x ) = ∫ − A ′ A ′ ψ ( x ⋅ w − w 0 ) γ ( w , w 0 ) d w 0 h(w,x)=\int_{-A'}^{A'}\psi(x\cdot w-w_0)\gamma(w,w_0)\,\mathrm{d}w_0 h ( w , x ) = ∫ − A ′ A ′ ψ ( x ⋅ w − w 0 ) γ ( w , w 0 ) d w 0
γ ( w , w 0 ) = Re ( ( 1 2 π ) n F [ ρ β ∗ g ] ( w ) F [ ψ ] ( 1 ) e i w 0 ) \gamma(w,w_0)=\operatorname{Re}\left(\left(\frac{1}{2\pi}\right)^n\frac{\mathcal{F}[\rho_\beta*g](w)}{\mathcal{F}[\psi](1)}e^{iw_0}\right) γ ( w , w 0 ) = Re ( ( 2 π 1 ) n F [ ψ ] ( 1 ) F [ ρ β ∗ g ] ( w ) e i w 0 )
Hence,
max x ∈ K ∣ f ( x ) − H N [ f ] ( x ) ∣ < 2 ϵ \max_{x\in K}|f(x)-H_N[f](x)|<2\epsilon x ∈ K max ∣ f ( x ) − H N [ f ] ( x ) ∣ < 2 ϵ
We apply Lemma B.5 again, now to H N ( x ) H_N(x) H N ( x ) in order to obtain our desired function f ~ ( x ) \widetilde{f}(x) f ( x ) :
( 2 A N ) − n H N [ f ] ( x ) = ∑ k ∈ { 1 , 2 , … , N − 1 } n h ( − A 1 + 2 A N k , x ) = ∫ − A ′ A ′ ∑ k ∈ { 1 , 2 , … , N − 1 } n ψ ( x ⋅ ( − A 1 + 2 A N k ) − w 0 ) ⋅ γ ( ( − A 1 + 2 A N k ) , w 0 ) d w 0 \begin{aligned}
\left(\frac{2A}{N}\right)^{-n}H_N[f](x) \\
&=\sum_{k\in\{1,2,\ldots,N-1\}^n}h\left(-A\mathbf{1}+\frac{2A}{N}k,x\right) \\
&=\int_{-A'}^{A'}\sum_{k\in\{1,2,\ldots,N-1\}^n}\psi\left(x\cdot\left(-A\mathbf{1}+\frac{2A}{N}k\right)-w_0\right)
\cdot\gamma\left(\left(-A\mathbf{1}+\frac{2A}{N}k\right),w_0\right)\,\mathrm{d}w_0
\end{aligned} ( N 2 A ) − n H N [ f ] ( x ) = k ∈ { 1 , 2 , … , N − 1 } n ∑ h ( − A 1 + N 2 A k , x ) = ∫ − A ′ A ′ k ∈ { 1 , 2 , … , N − 1 } n ∑ ψ ( x ⋅ ( − A 1 + N 2 A k ) − w 0 ) ⋅ γ ( ( − A 1 + N 2 A k ) , w 0 ) d w 0
Approximating by a Riemann sum, we get
( 2 A N ) − n H N [ f ] ( x ) ≈ R M [ f ] ( x ) = 2 A ′ M ∑ m = 0 M − 1 ∑ k ∈ { 1 , 2 , … , N − 1 } n ψ ( x ⋅ ( − A 1 + 2 A N k ) − ( − A ′ + 2 m A ′ M ) ) ⋅ γ ( ( − A 1 + 2 A N k ) , − A ′ + 2 m A ′ M ) \begin{aligned}
\left(\frac{2A}{N}\right)^{-n}H_N[f](x)
&\approx R_M[f](x)\\
&=\frac{2A'}{M}\sum_{m=0}^{M-1}\sum_{k\in\{1,2,\ldots,N-1\}^n}
\psi\left(x\cdot\left(-A\mathbf{1}+\frac{2A}{N}k\right)-\left(-A'+\frac{2mA'}{M}\right)\right)\\
&\quad\cdot\gamma\left(\left(-A\mathbf{1}+\frac{2A}{N}k\right),-A'+\frac{2mA'}{M}\right)
\end{aligned} ( N 2 A ) − n H N [ f ] ( x ) ≈ R M [ f ] ( x ) = M 2 A ′ m = 0 ∑ M − 1 k ∈ { 1 , 2 , … , N − 1 } n ∑ ψ ( x ⋅ ( − A 1 + N 2 A k ) − ( − A ′ + M 2 m A ′ ) ) ⋅ γ ( ( − A 1 + N 2 A k ) , − A ′ + M 2 m A ′ )
Using Lemma B.5 , we can find
max x ∈ K ∣ ( 2 A N ) − n H N [ f ] ( x ) − R M [ f ] ( x ) ∣ < ( 2 A N ) − n ϵ \max_{x\in K}\left|\left(\frac{2A}{N}\right)^{-n}H_N[f](x)-R_M[f](x)\right|<\left(\frac{2A}{N}\right)^{-n}\epsilon x ∈ K max ( N 2 A ) − n H N [ f ] ( x ) − R M [ f ] ( x ) < ( N 2 A ) − n ϵ
Combining this estimate with our previous approximation of f ( x ) f(x) f ( x ) to H N [ f ] ( x ) H_N[f](x) H N [ f ] ( x ) , we get
max x ∈ K ∣ f ( x ) − ( 2 A N ) n R M [ f ] ( x ) ∣ < 3 ϵ \max_{x\in K}\left|f(x)-\left(\frac{2A}{N}\right)^nR_M[f](x)\right|<3\epsilon x ∈ K max f ( x ) − ( N 2 A ) n R M [ f ] ( x ) < 3 ϵ
In order to conclude our proof, we just have to show that ( 2 A N ) n R M [ f ] ( x ) \left(\frac{2A}{N}\right)^nR_M[f](x) ( N 2 A ) n R M [ f ] ( x ) is precisely our desired function f ~ ( x ) \tilde{f}(x) f ~ ( x ) . We obtain an equivalent expression for R M [ f ] ( x ) R_M[f](x) R M [ f ] ( x ) as follows:
R M [ f ] ( x ) = 2 A ′ M ∑ m = 0 M − 1 ∑ k ∈ { 1 , 2 , … , N − 1 } n ψ ( ( x , − 1 ) ⋅ ( − A 1 + 2 A N k , − A ′ + 2 m A ′ M ) ) ⋅ γ ( ( − A 1 + 2 A N k , − A ′ + 2 m A ′ M ) ) \begin{aligned}
R_M[f](x)
&=\frac{2A'}{M}\sum_{m=0}^{M-1}\sum_{k\in\{1,2,\ldots,N-1\}^n}
\psi\left((x,-1)\cdot\left(-A1+\frac{2A}{N}k,-A'+\frac{2mA'}{M}\right)\right)\\
&\qquad\cdot\gamma\left(\left(-A1+\frac{2A}{N}k,-A'+\frac{2mA'}{M}\right)\right)
\end{aligned} R M [ f ] ( x ) = M 2 A ′ m = 0 ∑ M − 1 k ∈ { 1 , 2 , … , N − 1 } n ∑ ψ ( ( x , − 1 ) ⋅ ( − A 1 + N 2 A k , − A ′ + M 2 m A ′ ) ) ⋅ γ ( ( − A 1 + N 2 A k , − A ′ + M 2 m A ′ ) )
We define
Ω ( m , k ) = ( − A 1 + 2 A N k , − A ′ + 2 m A ′ M ) ∈ R n + 1 \Omega(m,k)=\left(-A1+\frac{2A}{N}k,-A'+\frac{2mA'}{M}\right)\in\mathbb{R}^{n+1} Ω ( m , k ) = ( − A 1 + N 2 A k , − A ′ + M 2 m A ′ ) ∈ R n + 1
The set { Ω ( m , k ) } \{\Omega(m,k)\} { Ω ( m , k )} contains N n M N^nM N n M vectors in R n + 1 \mathbb{R}^{n+1} R n + 1 . We can refer to them as Ω ( l ) \Omega(l) Ω ( l ) , with l = 1.. N n M l=1..N^nM l = 1.. N n M , so each vector is Ω ( l ) = ( Ω l , 1 , … , Ω l , n + 1 ) \Omega(l)=(\Omega_{l,1},\ldots,\Omega_{l,n+1}) Ω ( l ) = ( Ω l , 1 , … , Ω l , n + 1 ) .
Given the definition of ψ ( x ) \psi(x) ψ ( x ) , we have
R M [ f ] ( x ) = 2 A ′ M ∑ l = 1 N n M ψ ( ( x , − 1 ) ⋅ Ω ( l ) ) γ ( Ω ( l ) ) = 2 A ′ M ∑ l = 1 N n M γ ( Ω ( l ) ) ψ ( ∑ j = 1 n x j Ω l , j − Ω l , n + 1 ) = 2 A ′ M ∑ l = 1 N n M γ ( Ω ( l ) ) ϕ ( ∑ j = 1 n x j Ω l , j δ − ( Ω l , n + 1 δ − α ) ) = 2 A ′ M ∑ l = 1 N n M γ ( Ω ( l ) ) ϕ ( ∑ j = 1 n x j Ω l , j δ − ( Ω l , n + 1 δ + α ) ) \begin{aligned}
R_M[f](x)
&=\frac{2A'}{M}\sum_{l=1}^{N^nM}\psi((x,-1)\cdot\Omega(l))\gamma(\Omega(l))\\
&=\frac{2A'}{M}\sum_{l=1}^{N^nM}\gamma(\Omega(l))\psi\left(\sum_{j=1}^{n}x_j\Omega_{l,j}-\Omega_{l,n+1}\right)\\
&=\frac{2A'}{M}\sum_{l=1}^{N^nM}\gamma(\Omega(l))\phi\left(\sum_{j=1}^{n}\frac{x_j\Omega_{l,j}}{\delta}-\left(\frac{\Omega_{l,n+1}}{\delta}-\alpha\right)\right)\\
&=\frac{2A'}{M}\sum_{l=1}^{N^nM}\gamma(\Omega(l))\phi\left(\sum_{j=1}^{n}\frac{x_j\Omega_{l,j}}{\delta}-\left(\frac{\Omega_{l,n+1}}{\delta}+\alpha\right)\right)
\end{aligned} R M [ f ] ( x ) = M 2 A ′ l = 1 ∑ N n M ψ (( x , − 1 ) ⋅ Ω ( l )) γ ( Ω ( l )) = M 2 A ′ l = 1 ∑ N n M γ ( Ω ( l )) ψ ( j = 1 ∑ n x j Ω l , j − Ω l , n + 1 ) = M 2 A ′ l = 1 ∑ N n M γ ( Ω ( l )) ϕ ( j = 1 ∑ n δ x j Ω l , j − ( δ Ω l , n + 1 − α ) ) = M 2 A ′ l = 1 ∑ N n M γ ( Ω ( l )) ϕ ( j = 1 ∑ n δ x j Ω l , j − ( δ Ω l , n + 1 + α ) )
Rearranging, we can get
( 2 A N ) n R M [ f ] ( x ) = ∑ j = 1 2 N n M α j ϕ ( ∑ i = 1 n w i j x i + b j ) = f ~ \begin{aligned}
\left(\frac{2A}{N}\right)^nR_M[f](x)
&=\sum_{j=1}^{2N^nM}\alpha_j\phi\left(\sum_{i=1}^{n}w_{ij}x_i+b_j\right)\\
&=\tilde{f}
\end{aligned} ( N 2 A ) n R M [ f ] ( x ) = j = 1 ∑ 2 N n M α j ϕ ( i = 1 ∑ n w ij x i + b j ) = f ~
Hence, our approximation of ( 2 A N ) n R M [ f ] ( x ) \left(\frac{2A}{N}\right)^nR_M[f](x) ( N 2 A ) n R M [ f ] ( x ) concludes the proof of the theorem:
max x ∈ K ∣ f ( x ) − f ~ ( x ) ∣ < 3 ϵ \max_{x\in K}\left|f(x)-\tilde{f}(x)\right|<3\epsilon x ∈ K max f ( x ) − f ~ ( x ) < 3 ϵ
Remark. Please, observe how we might need an exponential number of hidden neurons in order to obtain the desired approximation.
Funahashi’s original proof [12 ] is based on an integral representation formula due to Bunpei Irie and Sei Miyake [22 ] , which suggests the realization of functions by three-layer (one-hidden-layer) networks with an infinite number of hidden neurons.13 From this, the general case (for any number of hidden layers) can be proved by induction and, for the two hidden layers case,14 Funahashi also included a simpler proof using the Kolmogorov-Arnold-Sprecher theorem [2 , 25 , 37 ] , which had been already used by Robert Hetch-Nielsen to indicate that any continuous mapping can be represented by a four-layer (two-hidden-layer) neural network [17 ] .
Theorem I.2: Universal approximation theorem for artificial neural networks with bounded, monotonically increasing activation functions Let ϕ ( x ) \phi(x) ϕ ( x ) be a non-constant, bounded, and monotonically increasing continuous function. Let K K K be a compact subset (bounded closed subset) of R n \mathbb{R}^n R n and f f f a real valued continuous function on K K K . Then, for an arbitrary ϵ > 0 \epsilon>0 ϵ > 0 , there exists an integer N N N and real constants α i \alpha_i α i , b i b_i b i , and w i j w_{ij} w ij such that
f ~ ( x ) = ∑ j = 1 h α j ϕ ( ∑ i = 1 n w i j x i + b j ) \tilde{f}(x)=\sum_{j=1}^{h}\alpha_j\phi\left(\sum_{i=1}^{n}w_{ij}x_i+b_j\right) f ~ ( x ) = j = 1 ∑ h α j ϕ ( i = 1 ∑ n w ij x i + b j ) satisfies max x ∈ K ∣ f ( x ) − f ~ ( x ) ∣ < ϵ \max_{x\in K}|f(x)-\tilde{f}(x)|<\epsilon max x ∈ K ∣ f ( x ) − f ~ ( x ) ∣ < ϵ
Sketch of Funahashi’s original proof. The following sketch presents a Fourier-analytic interpretation of why sigmoidal ridge functions are capable of approximating arbitrary continuous functions: Finite superpositions of ridge functions x ↦ σ ( w ⋅ x + b ) x\mapsto\sigma(w\cdot x+b) x ↦ σ ( w ⋅ x + b ) , where w ∈ R n w\in\mathbb{R}^n w ∈ R n and b ∈ R b\in\mathbb{R} b ∈ R , are dense in C ( K ) C(K) C ( K ) for every compact K ⊂ R n K\subset\mathbb{R}^n K ⊂ R n , i.e. they approximate arbitrary continuous functions on compact
Irie and Miyake [22 ] proved that arbitrary functions can be represented by a three-layer network with an infinite number of hidden units, i.e. $\lim_{A\to\infty}|1_{\infty,A}f f ( x ) ∥ L 2 = 0 f(x)\|_{L^2}=0 f ( x ) ∥ L 2 = 0 , provided that the activation function is absolutely integrable (ϕ ( z ) ∈ L 1 ( R ) \phi(z) \in L^1(\mathbb{R}) ϕ ( z ) ∈ L 1 ( R ) ).
Funahashi’s strategy is as follows: Let f ∈ C ( K ) f \in C(K) f ∈ C ( K ) , where K ⊂ R n K \subset\mathbb{R}^n K ⊂ R n is compact. By Lemma A.10, f f f may be uniformly approximated on K K K by a smooth band-limited function F F F . Since F F F is band-limited, the Paley–Wiener theorem implies that F F F extends to an entire function of exponential type. Thus the approximation problem may be viewed in terms of reconstructing a function from compactly-supported frequency data. Ridge functions x ↦ σ ( w ⋅ x + b ) x \mapsto\sigma(w \cdot x + b) x ↦ σ ( w ⋅ x + b ) introduce frequency information along selected directions w w w . By varying the directions and offsets, finite linear combinations of ridge sigmoids generate a rich family of approximating functions. Funahashi establishes that this family is sufficiently rich to be dense in C ( K ) C(K) C ( K ) . Consequently, for every f ∈ C ( K ) f \in C(K) f ∈ C ( K ) and every ε > 0 \varepsilon> 0 ε > 0 , there exists a finite superposition
f ~ ( x ) = ∑ j = 1 h α j σ ( w j ⋅ x + b j ) \tilde{f}(x) = \sum_{j=1}^{h} \alpha_j \sigma(w_j \cdot x + b_j) f ~ ( x ) = j = 1 ∑ h α j σ ( w j ⋅ x + b j )
such that
sup x ∈ K ∣ f ( x ) − f ~ ( x ) ∣ < ε . \sup_{x \in K} |f(x) - \tilde{f}(x)| < \varepsilon. x ∈ K sup ∣ f ( x ) − f ~ ( x ) ∣ < ε .
Reduction to approximation of band-limited functions
Let f ∈ C ( K ) f \in C(K) f ∈ C ( K ) , where K ⊂ R n K \subset\mathbb{R}^n K ⊂ R n is compact, and let ε > 0 \varepsilon> 0 ε > 0 . By Lemma A.10, there exists a smooth band-limited function F ∈ C ∞ ( R n ) F \in C^\infty(\mathbb{R}^n) F ∈ C ∞ ( R n ) such that sup x ∈ K ∣ f ( x ) − F ( x ) ∣ < ε 2 \sup_{x \in K} |f(x) - F(x)| < \frac{\varepsilon}{2} sup x ∈ K ∣ f ( x ) − F ( x ) ∣ < 2 ε . Therefore, it suffices to approximate the band-limited function F F F on K K K .
Since F F F is band-limited, there exists R > 0 R > 0 R > 0 such that supp ( F ^ ) ⊆ B R ( 0 ) \operatorname{supp}(\widehat{F}) \subseteq B_R(0) supp ( F ) ⊆ B R ( 0 ) . By the Paley–Wiener theorem (Corollary A.9), F F F extends to an entire function on C n \mathbb{C}^n C n whose growth is controlled by the bandwidth R R R . Thus F F F possesses a highly structured spectral representation. The Paley–Wiener theorem (Theorem A.8) provides a bridge between band-limited functions and entire functions of exponential type, which motivates the use of Fourier-analytic techniques in the approximation problem.
The family generated by sigmoids has rich frequency content
The key idea is that ridge sigmoids possess sufficiently rich frequency content to approximate band-limited functions. Consider a unit vector u ∈ S n − 1 u \in S^{n-1} u ∈ S n − 1 and functions of the form σ ( λ u ⋅ x + b ) \sigma(\lambda u \cdot x + b) σ ( λ u ⋅ x + b ) , with λ > 0 \lambda> 0 λ > 0 , b ∈ R b \in\mathbb{R} b ∈ R . As functions of the scalar variable t = u ⋅ x t = u \cdot x t = u ⋅ x , these are translations/dilations of σ \sigma σ .
Translations and dilations of a ridge sigmoid alter its location, offset, and characteristic scale. Varying the parameters w w w and b b b produces a rich collection of ridge functions with different spectral characteristics. From the Fourier viewpoint, translations primarily affect phase, while dilations modify the characteristic frequency scale.
Under mild regularity assumptions on σ \sigma σ (sigmoidal, non-polynomial, typically of bounded variation or absolutely integrable derivative), the Fourier transform σ ^ \hat{\sigma} σ ^ of a sigmoid σ \sigma σ is nontrivial and exhibits controlled decay properties. This suggests that finite superpositions of ridge sigmoids may be used to reproduce increasingly complex spectral patterns. In other words, finite superpositions of ridge sigmoids can synthesize spectra by placing energy along chosen directions and phases, allowing the approximation of band-limited targets by finite superpositions.
Approximation in L 2 L^2 L 2 by finite superpositions
A central result underlying Funahashi’s theorem is that finite superpositions of non-polynomial ridge sigmoids form a class rich enough to approximate band-limited functions.
The spectral synthesis consists of constructing linear combinations whose Fourier transforms approximate F ( F ) \mathcal{F}(F) F ( F ) in L 2 L^2 L 2 over the compact support of F ( F ) \mathcal{F}(F) F ( F ) :
F ~ ( x ) = ∑ j = 1 h α j σ ( w j ⋅ x + b j ) \widetilde{\mathcal{F}}(x) = \sum_{j=1}^{h} \alpha_j \sigma(w_j \cdot x + b_j) F ( x ) = j = 1 ∑ h α j σ ( w j ⋅ x + b j )
so that α j \alpha_j α j , w j w_j w j , and b j b_j b j match the amplitudes, directions, and phases of F ( F ) \mathcal{F}(F) F ( F ) .
Since F F F is band-limited, standard uniform bounds transfer this small L 2 L^2 L 2 error to uniform error on K K K after truncation/localization. This is a standard mathematical argument in numerical analysis that describes how to transfer a mean-error guarantee (in the L 2 L^2 L 2 norm) into a pointwise, uniform-error guarantee (in the L ∞ L^\infty L ∞ norm) over a bounded domain K K K .15
Combining the reduction to band-limited functions and the spectral approximation, we obtain our desired result: for any ε > 0 \varepsilon> 0 ε > 0 , there exists a finite superposition f ~ ( x ) \tilde{f}(x) f ~ ( x ) such that sup x ∈ K ∣ f ( x ) − f ~ ( x ) ∣ < ε \sup_{x \in K} |f(x) - \tilde{f}(x)| < \varepsilon sup x ∈ K ∣ f ( x ) − f ~ ( x ) ∣ < ε . Hence, finite sums of the form f ~ ( x ) \tilde{f}(x) f ~ ( x ) are dense in C ( K ) C(K) C ( K ) .
In summary, by showing that the span of shifted/dilated sigmoids is dense in the Paley–Wiener class, Funahashi concluded that they are dense in C ( K ) C(K) C ( K ) .
□ \square □
Proof by contradiction. For simplicity, we present a Fourier-analytic proof under the additional assumption that σ \sigma σ is a non-constant sigmoidal function whose derivative belongs to L 1 ( R ) L^{1}(\mathbb{R}) L 1 ( R ) , i.e. σ ′ ∈ L 1 ( R ) \sigma' \in L^{1}(\mathbb{R}) σ ′ ∈ L 1 ( R ) . Funahashi’s original argument removes this regularity assumption by a more delicate analysis.
This proof is based on a density argument combining functional analysis and Fourier analysis. The key idea is to show that any continuous linear functional that annihilates all ridge functions must vanish identically. By the Hahn–Banach theorem, this implies that the span of ridge functions is dense in C ( K ) C(K) C ( K ) .
Then, by the Hahn–Banach theorem (Theorem A.6), there exists a nonzero continuous linear functional L : C ( K ) → R L : C(K) \to\mathbb{R} L : C ( K ) → R such that L ( σ ( w ⋅ x + b ) ) = 0 L(\sigma(w \cdot x + b)) = 0 L ( σ ( w ⋅ x + b )) = 0 for all w ∈ R n w \in\mathbb{R}^{n} w ∈ R n and b ∈ R b \in\mathbb{R} b ∈ R .
By the Riesz representation theorem (Theorem A.2), there exists a nonzero finite signed Borel measure μ \mu μ on K K K such that L ( f ) = ∫ K f ( x ) d μ ( x ) L(f) = \int_{K} f(x)\,d\mu(x) L ( f ) = ∫ K f ( x ) d μ ( x ) . Hence ∫ K σ ( w ⋅ x + b ) d μ ( x ) = 0 \int_{K} \sigma(w \cdot x + b)\,d\mu(x) = 0 ∫ K σ ( w ⋅ x + b ) d μ ( x ) = 0 for all w ∈ R n w \in\mathbb{R}^{n} w ∈ R n and b ∈ R b \in\mathbb{R} b ∈ R .
F w ′ ( t ) = ∫ K σ ′ ( w ⋅ x + t ) d μ ( x ) . F'_w(t) = \int_{K} \sigma'(w \cdot x + t)\,d\mu(x). F w ′ ( t ) = ∫ K σ ′ ( w ⋅ x + t ) d μ ( x ) .
Since F w ≡ 0 F_w \equiv0 F w ≡ 0 , it follows that F w ′ ≡ 0 F'_w \equiv0 F w ′ ≡ 0 .
F w ′ ^ ( ξ ) = σ ′ ^ ( ξ ) ∫ K e − i ξ w ⋅ x d μ ( x ) . \widehat{F'_w}(\xi) = \widehat{\sigma'}(\xi) \int_{K} e^{-i\xi w\cdot x}\,d\mu(x). F w ′ ( ξ ) = σ ′ ( ξ ) ∫ K e − i ξ w ⋅ x d μ ( x ) .
Since F w ′ ≡ 0 F'_w \equiv0 F w ′ ≡ 0 , we have
0 = σ ′ ^ ( ξ ) ∫ K e − i ξ w ⋅ x d μ ( x ) . 0 = \widehat{\sigma'}(\xi) \int_{K} e^{-i\xi w\cdot x}\,d\mu(x). 0 = σ ′ ( ξ ) ∫ K e − i ξ w ⋅ x d μ ( x ) .
Since σ ′ ∈ L 1 ( R ) \sigma' \in L^{1}(\mathbb{R}) σ ′ ∈ L 1 ( R ) , its Fourier transform σ ′ ^ \widehat{\sigma'} σ ′ is continuous. Because σ \sigma σ is nonconstant, its derivative cannot vanish almost everywhere. Hence σ ′ ≢ 0 \sigma' \not\equiv0 σ ′ ≡ 0 in L 1 ( R ) L^{1}(\mathbb{R}) L 1 ( R ) , and by injectivity of the Fourier transform on L 1 ( R ) L^{1}(\mathbb{R}) L 1 ( R ) , σ ′ ^ ≢ 0 \widehat{\sigma'} \not\equiv0 σ ′ ≡ 0 . Hence there exists ξ 0 ∈ R \xi_{0} \in\mathbb{R} ξ 0 ∈ R such that σ ′ ^ ( ξ 0 ) ≠ 0 \widehat{\sigma'}(\xi_{0}) \ne0 σ ′ ( ξ 0 ) = 0 . By continuity, there exists a nonempty open interval I ⊂ R I \subset\mathbb{R} I ⊂ R containing ξ 0 \xi_{0} ξ 0 for which σ ′ ^ ( ξ ) ≠ 0 \widehat{\sigma'}(\xi) \ne0 σ ′ ( ξ ) = 0 (ξ ∈ I \xi\in I ξ ∈ I ). Hence ∫ K e − i ξ w ⋅ x d μ ( x ) = 0 \int_{K} e^{-i\xi w\cdot x}\,d\mu(x) = 0 ∫ K e − i ξ w ⋅ x d μ ( x ) = 0 for all w w w and all ξ ∈ I \xi\in I ξ ∈ I .
Since I = ( a , b ) I = (a,b) I = ( a , b ) contains some ξ 0 ≠ 0 \xi_{0} \ne0 ξ 0 = 0 , choose a smaller interval J ⊂ I J \subset I J ⊂ I such that 0 ∉ J 0 \notin J 0 ∈ / J . Then
U = { ξ w : ξ ∈ J , w ∈ R n } = R n ∖ { 0 } , U = \{\xi w : \xi\in J,\ w \in\mathbb{R}^{n}\} = \mathbb{R}^{n} \setminus\{0\}, U = { ξ w : ξ ∈ J , w ∈ R n } = R n ∖ { 0 } ,
which is a nonempty open subset of R n \mathbb{R}^{n} R n . Hence μ ^ \widehat{\mu} μ vanishes on a nonempty open set.
μ ^ ( η ) = ∫ K e − i η ⋅ x d μ ( x ) \widehat{\mu}(\eta) = \int_{K} e^{-i\eta\cdot x}\,d\mu(x) μ ( η ) = ∫ K e − i η ⋅ x d μ ( x )
extends to an entire function on C n \mathbb{C}^{n} C n .
Since μ ^ \widehat{\mu} μ extends to an entire function on C n \mathbb{C}^{n} C n and vanishes on a nonempty open subset of R n \mathbb{R}^{n} R n . Therefore, it vanishes on a nonempty open subset of C n \mathbb{C}^{n} C n relative to the real slice. By the identity theorem for holomorphic functions of several variables, μ ^ ≡ 0 \widehat{\mu} \equiv0 μ ≡ 0 .
Therefore, no nonzero annihilating functional exists (the annihilator of the ridge-function span is trivial). By the Hahn–Banach theorem (Corollary A.5),
span { σ ( w ⋅ x + b ) : w ∈ R n , b ∈ R } ‾ = C ( K ) . \overline{\operatorname{span}\{\sigma(w \cdot x + b) : w \in\mathbb{R}^{n},\ b \in\mathbb{R}\}} = C(K). span { σ ( w ⋅ x + b ) : w ∈ R n , b ∈ R } = C ( K ) .
Hence the span of ridge functions is dense in C ( K ) C(K) C ( K ) . □
Remark. The assumption that σ \sigma σ is non-polynomial is essential. If σ \sigma σ is a polynomial of degree m m m , then every finite superposition ∑ j = 1 h α j σ ( w j ⋅ x + b j ) \sum_{j=1}^{h} \alpha_j \sigma(w_j \cdot x + b_j) ∑ j = 1 h α j σ ( w j ⋅ x + b j ) is itself a polynomial of degree at most m m m .
Consequently, the resulting function class is contained in the finite-dimensional vector space of polynomials of degree at most m m m and therefore cannot be dense in C ( K ) C(K) C ( K ) .
A final note is in order: A. Ronald Gallant and Halbert L. White [13 ] provided an earlier proof for the approximation capabilities of artificial neural networks. They introduced a specific, non-periodic activation function (the sigmoid-like “cosine squasher”) and proved that a feed-forward network with a single hidden layer using this activation function could perfectly replicate a “Fourier network,” which yields a Fourier series approximation to a given function as its output. Therefore, an approximation to any desired accuracy can be achieved by a neural network, using sufficiently many hidden units. In this sense, neural networks do not make avoidable mistakes, hence the title of their ICNN paper.
In their 1992 follow-up paper [14 ] , Gallant and White extended their 1988 proof: Neural networks don’t just approximate the underlying target function; they also consistently estimate its partial derivatives (density in Sobolev spaces), a result also obtained by Hornik [21 ] .
Hornik's Universal Approximation Theorems
In a series of papers published in the Neural Networks journal, Kurt Hornik, Maxwell Stinchcombe, and the aforementioned Halbert White established that one-hidden-layer neural networks using arbitrary squashing functions (e.g., logistic, hyperbolic tangent) are capable of approximating any Borel measurable function to any desired degree of accuracy [20 ] , as well as its derivatives [19 , 21 ] .
Hornik, Stinchcombe, and White’s proof of multilayer feedforward networks as a class of universal approximators is based on Gallant and White’s “cosine squasher” result [13 ] and resorts to the Stone-Weierstrass Theorem.
A family A A A of real functions defined on a set E E E is an algebra if A A A is closed under addition, multiplication, and scalar multiplication. A family A A A separates points on E E E if for every x , y ∈ E x,y \in E x , y ∈ E , x ≠ y x \ne y x = y , there exists a function f f f in A A A such that f ( x ) ≠ f ( y ) f(x) \ne f(y) f ( x ) = f ( y ) . The family A A A vanishes at no point of E E E if for each x ∈ E x \in E x ∈ E there exists f ∈ A f \in A f ∈ A such that f ( x ) ≠ 0 f(x) \ne0 f ( x ) = 0 .
Theorem B.7: Stone-Weierstrass Theorem Let A A A be an algebra of real continuous functions on a compact set K K K . If A A A separates points on K K K and if A A A vanishes at no point of K K K , then the uniform closure B B B of A A A consists of all real continuous functions on K K K , i.e., A A A is μ K \mu_K μ K -dense in the space of real continuous functions on $K.
Definition B.6: A squashing function Ψ ( z ) : R → [ 0 , 1 ] \Psi(z):\mathbb{R} \to[0,1] Ψ ( z ) : R → [ 0 , 1 ] is a monotonic (non-decreasing) function with lim z → + ∞ Ψ ( z ) = 1 \lim_{z\to+\infty}\Psi(z)=1 lim z → + ∞ Ψ ( z ) = 1 and $\lim_{z\to-\infty}\Psi(z)=0
Squashing functions are measurable because their discontinuities are at most countable (finite or countable infinity), by Froda’s Theorem, and the set of points where the function is discontinuous, being countable, has a Lebesgue measure zero—they have countable many discontinuities. A function that is continuous almost everywhere (i.e., continuous everywhere except on a set of measure zero like a countable set) is Borel measurable.
Examples of squashing functions include the threshold function Ψ 1 ( z ) = 1 { z ≥ 0 } \Psi_1(z)=1_{\{z\geq0\}} Ψ 1 ( z ) = 1 { z ≥ 0 } , where 1 { ⋅ } 1_{\{\cdot\}} 1 { ⋅ } is the indicator function, the ramp function Ψ r a m p ( z ) = z ⋅ 1 { 0 ≤ z ≤ 1 } + 1 { z > 1 } \Psi_{\mathrm{ramp}}(z)=z\cdot1_{\{0\leq z\leq1\}}+1_{\{z>1\}} Ψ ramp ( z ) = z ⋅ 1 { 0 ≤ z ≤ 1 } + 1 { z > 1 } , and Gallant and White’s cosine squasher Ψ c o s i n e ( z ) = ( 1 + cos ( z + 3 π / 2 ) ) ⋅ 1 { − π / 2 ≤ z ≤ π / 2 } + 1 { z > π / 2 } \Psi_{\mathrm{cosine}}(z)=(1+\cos(z+3\pi/2))\cdot1_{\{-\pi/2\leq z\leq\pi/2\}}+1_{\{z>\pi/2\}} Ψ cosine ( z ) = ( 1 + cos ( z + 3 π /2 )) ⋅ 1 { − π /2 ≤ z ≤ π /2 } + 1 { z > π /2 } ,
Definition B.7: Σ \Sigma Σ networks For any measurable function G : R → R G:\mathbb{R}\to\mathbb{R} G : R → R , Σ n ( G ) \Sigma^n(G) Σ n ( G ) is the class of functions
{ f : R n → R : f ( x ) = ∑ j = 1 q β j G ( A j ( x ) ) } \left\{f:\mathbb{R}^n\to\mathbb{R}:f(x)=\sum_{j=1}^{q}\beta_jG(A_j(x))\right\} { f : R n → R : f ( x ) = j = 1 ∑ q β j G ( A j ( x )) } where A j ( x ) A_j(x) A j ( x ) is an affine function of the form A ( x ) = w ⋅ x + b A(x)=w\cdot x+b A ( x ) = w ⋅ x + b .
Definition B.8: Σ Π \Sigma\Pi ΣΠ networks For any measurable function G : R → R G:\mathbb{R}\to\mathbb{R} G : R → R , Σ Π n ( G ) \Sigma\Pi^n(G) Σ Π n ( G ) is the class of functions
{ f : R n → R : f ( x ) = ∑ j = 1 q β j ∏ k = 1 l j G ( A j k ( x ) ) } \left\{f:\mathbb{R}^n\to\mathbb{R}:f(x)=\sum_{j=1}^{q}\beta_j\prod_{k=1}^{l_j}G(A_{jk}(x))\right\} ⎩ ⎨ ⎧ f : R n → R : f ( x ) = j = 1 ∑ q β j k = 1 ∏ l j G ( A j k ( x )) ⎭ ⎬ ⎫ where A j k ( x ) A_{jk}(x) A j k ( x ) is an affine function of the form A ( x ) = w ⋅ x + b A(x)=w\cdot x+b A ( x ) = w ⋅ x + b .
Σ \Sigma Σ networks are a special case of Σ Π \Sigma\Pi ΣΠ networks, with l j = 1 l_j=1 l j = 1 for all j j j .
The classes Σ n ( G ) \Sigma^n(G) Σ n ( G ) and Σ Π n ( G ) \Sigma\Pi^n(G) Σ Π n ( G ) belong to the set of Borel measurable functions from R n \mathbb{R}^n R n to R \mathbb{R} R for any Borel measurable G G G , i.e. Σ n ( G ) ⊆ Σ Π n ( G ) ⊆ M ( R n ) \Sigma^n(G)\subseteq\Sigma\Pi^n(G)\subseteq\mathcal{M}(\mathbb{R}^n) Σ n ( G ) ⊆ Σ Π n ( G ) ⊆ M ( R n ) . When G G G is continuous, Σ n ( G ) \Sigma^n(G) Σ n ( G ) and Σ Π n ( G ) \Sigma\Pi^n(G) Σ Π n ( G ) belong to the set of continuous functions from R n \mathbb{R}^n R n to R \mathbb{R} R , i.e. Σ n ( G ) ⊆ Σ Π n ( G ) ⊆ C ( R n ) \Sigma^n(G)\subseteq\Sigma\Pi^n(G)\subseteq C(\mathbb{R}^n) Σ n ( G ) ⊆ Σ Π n ( G ) ⊆ C ( R n ) .
Closeness of functions f f f and g g g belonging to C ( R n ) C(\mathbb{R}^n) C ( R n ) or M ( R n ) M(\mathbb{R}^n) M ( R n ) is measured by a metric μ \mu μ . Closeness of one class of functions to another class is described by the concept of denseness:
Definition B.9: Dense. A subset S S S of a metric space ( X , μ ) (X,\mu) ( X , μ ) is μ \mu μ -dense in a subset T T T when, for every ϵ > 0 \epsilon>0 ϵ > 0 and every t ∈ T t\in T t ∈ T , there is an s ∈ S s\in S s ∈ S such that μ ( s , t ) < ϵ \mu(s,t)<\epsilon μ ( s , t ) < ϵ .
An element of S S S can approximate an element of T T T to any desired degree of accuracy
Definition B.10: Uniformly Dense. A subset S S S of C ( R n ) C(\mathbb{R}^n) C ( R n ) is said to be uniformly dense on compacta in C ( R n ) C(\mathbb{R}^n) C ( R n ) , if, for every compact subset K ⊂ R n K\subset\mathbb{R}^n K ⊂ R n , S S S is μ K \mu_K μ K -dense in C ( R n ) C(\mathbb{R}^n) C ( R n ) , where, for f , g ∈ C ( R n ) f,g\in C(\mathbb{R}^n) f , g ∈ C ( R n ) , μ K ( f , g ) = sup x ∈ K ∣ f ( x ) − g ( x ) ∣ \mu_K(f,g)=\sup_{x\in K}|f(x)-g(x)| μ K ( f , g ) = sup x ∈ K ∣ f ( x ) − g ( x ) ∣ .
A sequence of functions { f n } \{f_n\} { f n } converges to a function f f f uniformly on compacta if, for all compact K ∈ R n K\in\mathbb{R}^n K ∈ R n , μ K ( f n , f ) → 0 \mu_K(f_n,f)\to0 μ K ( f n , f ) → 0 as n → ∞ n\to\infty n → ∞ .
Lemma B.8: Σ Π \Sigma\Pi ΣΠ is uniformly dense. Σ Π \Sigma\Pi ΣΠ feedforward networks are capable of arbitrarily accurate approximation to any real-valued continuous function over a compact set: Σ Π n ( G ) \Sigma\Pi^n(G) Σ Π n ( G ) is uniformly dense on compacta in C ( R n ) C(\mathbb{R}^n) C ( R n ) .
Proof. By applying Stone-Weierstrass Theorem (Theorem B.7), for any compact set K ⊂ R n K\subset\mathbb{R}^n K ⊂ R n and any G G G , Σ Π n ( G ) \Sigma\Pi^n(G) Σ Π n ( G ) is an algebra on K K K .
If x , y ∈ K x,y\in K x , y ∈ K , x ≠ y x\ne y x = y , there is an affine function A A A such that G ( A ( x ) ) ≠ G ( A ( y ) ) G(A(x))\ne G(A(y)) G ( A ( x )) = G ( A ( y )) . Just pick a , b ∈ R a,b\in\mathbb{R} a , b ∈ R , a ≠ b a\ne b a = b , such that G ( a ) ≠ G ( b ) G(a)\ne G(b) G ( a ) = G ( b ) , and pick A A A to satisfy A ( x ) = a A(x)=a A ( x ) = a and A ( y ) = b A(y)=b A ( y ) = b . Then G ( A ( x ) ) ≠ G ( A ( y ) ) G(A(x))\ne G(A(y)) G ( A ( x )) = G ( A ( y )) , so Σ Π n ( G ) \Sigma\Pi^n(G) Σ Π n ( G ) is separating on K K K .
There are G ( A ( x ) ) G(A(x)) G ( A ( x )) that are constant and not equal to zero. Just pick b ∈ R b\in\mathbb{R} b ∈ R such that G ( b ) ≠ 0 G(b)\ne0 G ( b ) = 0 and set A ( x ) = 0 ⋅ x + b A(x)=0\cdot x+b A ( x ) = 0 ⋅ x + b . For all x ∈ K x\in K x ∈ K , G ( A ( x ) ) = G ( b ) ≠ 0 G(A(x))=G(b)\ne0 G ( A ( x )) = G ( b ) = 0 , so Σ Π n ( G ) \Sigma\Pi^n(G) Σ Π n ( G ) vanishes at no point of K K K .
So the Stone-Weierstrass Theorem implies that Σ Π n ( G ) \Sigma\Pi^n(G) Σ Π n ( G ) is μ K \mu_K μ K -dense in the space of real continuous functions on $K for any compact set K K K .
Definition B.11: Probability Measure. Let ρ \rho ρ be a probability measure on ( R n , B n ) (\mathbb{R}^{n},\mathcal{B}^{n}) ( R n , B n ) , where B n \mathcal{B}^{n} B n is the Borel σ \sigma σ -field of R n \mathbb{R}^{n} R n . The measure describes the relative frequency of occurrence of input patterns x x x .
If f , g ∈ M ( R n ) f,g\in M(\mathbb{R}^{n}) f , g ∈ M ( R n ) , we say they are ρ \rho ρ -equivalent if ρ { x ∈ R n : f ( x ) = g ( x ) } = 1 \rho\{x\in\mathbb{R}^{n}:f(x)=g(x)\}=1 ρ { x ∈ R n : f ( x ) = g ( x )} = 1 . Functions that are ρ \rho ρ -equivalent differ only on a set of patterns occurring with probability (measure) zero.
Given a probability measure ρ \rho ρ on ( R n , B n ) (\mathbb{R}^{n},\mathcal{B}^{n}) ( R n , B n ) , we define the metric μ ρ : M ( R n ) × M ( R n ) → R + \mu_{\rho}:M(\mathbb{R}^{n})\times M(\mathbb{R}^{n})\to\mathbb{R}^{+} μ ρ : M ( R n ) × M ( R n ) → R + :
μ ρ ( f , g ) = inf { ϵ > 0 : ρ { x : ∣ f ( x ) − g ( x ) ∣ > ϵ } < ϵ } \mu_{\rho}(f,g)=\inf\{\epsilon>0:\rho\{x:|f(x)-g(x)|>\epsilon\}<\epsilon\} μ ρ ( f , g ) = inf { ϵ > 0 : ρ { x : ∣ f ( x ) − g ( x ) ∣ > ϵ } < ϵ } Two functions are close in this metric if and only if there is only a small probability that they differ significantly. In the extreme case, when f f f and g g g are ρ \rho ρ -equivalent, μ ρ ( f , g ) = 0 \mu_{\rho}(f,g)=0 μ ρ ( f , g ) = 0 .
There are different ways to describe that μ ρ ( f n , f ) \mu_{\rho}(f_n,f) μ ρ ( f n , f ) converges to 0 0 0 :
μ ρ ( f n , f ) → 0 \mu_{\rho}(f_n,f)\to0 μ ρ ( f n , f ) → 0 .
For every ϵ > 0 \epsilon>0 ϵ > 0 , ρ { x : ∣ f n ( x ) − f ( x ) ∣ > ϵ } → 0 \rho\{x:|f_n(x)-f(x)|>\epsilon\}\to0 ρ { x : ∣ f n ( x ) − f ( x ) ∣ > ϵ } → 0
∫ min { ∣ f n ( x ) − f ( x ) ∣ , 1 } ρ ( d x ) → 0 \int\min\{|f_n(x)-f(x)|,1\}\rho(dx)\to0 ∫ min { ∣ f n ( x ) − f ( x ) ∣ , 1 } ρ ( d x ) → 0
Lemma B.9: Uniform convergence on compacta and μ ρ \mu_{\rho} μ ρ -convergence. If { f n } \{f_n\} { f n } is a sequence of functions in M ( R n ) M(\mathbb{R}^{n}) M ( R n ) that converges uniformly on compacta to the function f f f , then μ ρ ( f n , f ) → 0 \mu_{\rho}(f_n,f)\to0 μ ρ ( f n , f ) → 0 .
Proof. Let ϵ > 0 \epsilon>0 ϵ > 0 and suppose ρ ( R n ) = 1 \rho(\mathbb{R}^{n})=1 ρ ( R n ) = 1 for convenience. Since R n \mathbb{R}^{n} R n is a locally compact metric space, ρ \rho ρ is a regular measure. Therefore, there is a compact subset K ∈ R n K\in\mathbb{R}^{n} K ∈ R n with ρ ( K ) > 1 − ϵ / 2 \rho(K)>1-\epsilon/2 ρ ( K ) > 1 − ϵ /2 .
It is sufficient to find N ∈ N N\in\mathbb{N} N ∈ N such that, ∀ n ≥ N \forall n\ge N ∀ n ≥ N ,
∫ min { ∣ f n ( x ) − f ( x ) ∣ , 1 } ρ ( d x ) < ϵ . \int\min\{|f_n(x)-f(x)|,1\}\rho(dx)<\epsilon. ∫ min { ∣ f n ( x ) − f ( x ) ∣ , 1 } ρ ( d x ) < ϵ . Picking N N N such that, ∀ n ≥ N \forall n\ge N ∀ n ≥ N , sup x ∈ K ∣ f n ( x ) − f ( x ) ∣ < ϵ / 2 \sup_{x\in K}|f_n(x)-f(x)|<\epsilon/2 sup x ∈ K ∣ f n ( x ) − f ( x ) ∣ < ϵ /2 , we have
∫ R n − K min { ∣ f n ( x ) − f ( x ) ∣ , 1 } ρ ( d x ) + ∫ K min { ∣ f n ( x ) − f ( x ) ∣ , 1 } ρ ( d x ) < ϵ / 2 + ϵ / 2 = ϵ . \int_{\mathbb{R}^{n}-K}\min\{|f_n(x)-f(x)|,1\}\rho(dx)+\int_{K}\min\{|f_n(x)-f(x)|,1\}\rho(dx)<\epsilon/2+\epsilon/2=\epsilon. ∫ R n − K min { ∣ f n ( x ) − f ( x ) ∣ , 1 } ρ ( d x ) + ∫ K min { ∣ f n ( x ) − f ( x ) ∣ , 1 } ρ ( d x ) < ϵ /2 + ϵ /2 = ϵ .
Lemma B.10: Finite measures are μ ρ \mu_{\rho} μ ρ -dense. For any finite measure ρ \rho ρ , C ( R n ) C(\mathbb{R}^{n}) C ( R n ) is μ ρ \mu_{\rho} μ ρ -dense in M ( R n ) M(\mathbb{R}^{n}) M ( R n ) .
Proof. Given ϵ > 0 \epsilon>0 ϵ > 0 and an arbitrary f ∈ M ( R n ) f\in M(\mathbb{R}^{n}) f ∈ M ( R n ) , we must find g ∈ C ( R n ) g\in C(\mathbb{R}^{n}) g ∈ C ( R n ) such that μ ρ ( f , g ) < ϵ \mu_{\rho}(f,g)<\epsilon μ ρ ( f , g ) < ϵ .
For sufficiently large M M M ,
∫ min { ∣ f ⋅ 1 { ∣ f ∣ < M } − f ∣ , 1 } d ρ < ϵ / 2 \int\min\{|f\cdot1_{\{|f|<M\}}-f|,1\}d\rho<\epsilon/2 ∫ min { ∣ f ⋅ 1 { ∣ f ∣ < M } − f ∣ , 1 } d ρ < ϵ /2 In fact, measure theory textbooks show that there exists a continuous function g g g such that
∫ min { ∣ f ⋅ 1 { ∣ f ∣ < M } − g ∣ , 1 } d ρ < ϵ / 2 \int\min\{|f\cdot1_{\{|f|<M\}}-g|,1\}d\rho<\epsilon/2 ∫ min { ∣ f ⋅ 1 { ∣ f ∣ < M } − g ∣ , 1 } d ρ < ϵ /2 Therefore,
∫ min { ∣ f − g ∣ , 1 } d ρ < ϵ \int\min\{|f-g|,1\}d\rho<\epsilon ∫ min { ∣ f − g ∣ , 1 } d ρ < ϵ
Lemma B.11: Σ Π \Sigma\Pi ΣΠ is μ \mu μ -dense. For every G G G and every probability measure ρ \rho ρ on ( R n , B n ) (\mathbb{R}^{n},\mathcal{B}^{n}) ( R n , B n ) , Σ Π n ( G ) \Sigma\Pi^{n}(G) Σ Π n ( G ) is μ ρ \mu_{\rho} μ ρ -dense in M ( R n ) M(\mathbb{R}^{n}) M ( R n ) .
Proof. Given any continuous nonconstant function, it follows from Lemmas B.8 (Σ Π \Sigma\Pi ΣΠ is uniformly dense) and B.9 (connecting uniform convergence to μ ρ \mu_{\rho} μ ρ -convergence) that Σ Π n ( G ) \Sigma\Pi^{n}(G) Σ Π n ( G ) is μ ρ \mu_{\rho} μ ρ -dense in C ( R n ) C(\mathbb{R}^{n}) C ( R n ) .
Since C ( R n ) C(\mathbb{R}^{n}) C ( R n ) is dense in M ( R n ) M(\mathbb{R}^{n}) M ( R n ) by Lemma B.10, it follows that Σ Π n ( G ) \Sigma\Pi^{n}(G) Σ Π n ( G ) is μ ρ \mu_{\rho} μ ρ -dense in M ( R n ) M(\mathbb{R}^{n}) M ( R n ) by the triangle inequality.
Lemma B.12: Continuous squashing functions. Let F F F be a continuous squashing function and Ψ \Psi Ψ an arbitrary squashing function. For every ϵ > 0 \epsilon>0 ϵ > 0 , there is an element H ϵ ∈ Σ 1 ( Ψ ) H_{\epsilon}\in\Sigma^{1}(\Psi) H ϵ ∈ Σ 1 ( Ψ ) such that sup λ ∈ R ∣ F ( λ ) − H ϵ ( λ ) ∣ < ϵ \sup_{\lambda\in\mathbb{R}}|F(\lambda)-H_{\epsilon}(\lambda)|<\epsilon sup λ ∈ R ∣ F ( λ ) − H ϵ ( λ ) ∣ < ϵ .
Proof. For an arbitrary ϵ > 0 \epsilon>0 ϵ > 0 , we must find a collection of constants β j \beta_j β j and affine functions A j A_j A j such that
sup λ ∈ R ∣ F ( λ ) − ∑ j = 1 Q − 1 β j Ψ ( A j ( λ ) ) ∣ < ϵ . \sup_{\lambda\in\mathbb{R}}\left|F(\lambda)-\sum_{j=1}^{Q-1}\beta_j\Psi(A_j(\lambda))\right|<\epsilon. λ ∈ R sup F ( λ ) − j = 1 ∑ Q − 1 β j Ψ ( A j ( λ )) < ϵ . Pick Q Q Q such that 1 / Q < ϵ / 2 1/Q<\epsilon/2 1/ Q < ϵ /2 and set β j = 1 / Q \beta_j=1/Q β j = 1/ Q for j ∈ { 1 … Q − 1 } j\in\{1\ldots Q-1\} j ∈ { 1 … Q − 1 } .
Pick M > 0 M>0 M > 0 such that Ψ ( − M ) < ϵ / 2 Q \Psi(-M)<\epsilon/2Q Ψ ( − M ) < ϵ /2 Q and Ψ ( M ) > 1 − ϵ / 2 Q \Psi(M)>1-\epsilon/2Q Ψ ( M ) > 1 − ϵ /2 Q . Since Ψ \Psi Ψ is a squashing function, such an M M M exists.
Set r j = sup { λ : F ( λ ) = j / Q } r_j=\sup\{\lambda:F(\lambda)=j/Q\} r j = sup { λ : F ( λ ) = j / Q } for j ∈ { 1 … Q − 1 } j\in\{1\ldots Q-1\} j ∈ { 1 … Q − 1 } and r Q = sup { λ : F ( λ ) = 1 − 1 / 2 Q } r_Q=\sup\{\lambda:F(\lambda)=1-1/2Q\} r Q = sup { λ : F ( λ ) = 1 − 1/2 Q } . Since F F F is a continuous squashing function, such r j r_j r j ’s exist.
For any r < s r<s r < s , let A r , s A_{r,s} A r , s be the unique affine function satisfying A r , s ( r ) = M A_{r,s}(r)=M A r , s ( r ) = M and A r , s ( s ) = − M A_{r,s}(s)=-M A r , s ( s ) = − M .
The desired approximation is
H ϵ ( λ ) = ∑ j = 1 Q − 1 β j Ψ ( A r j , r j + 1 ( λ ) ) H_{\epsilon}(\lambda)=\sum_{j=1}^{Q-1}\beta_j\Psi(A_{r_j,r_{j+1}}(\lambda)) H ϵ ( λ ) = j = 1 ∑ Q − 1 β j Ψ ( A r j , r j + 1 ( λ )) For every interval ( − ∞ , r 1 ] , ( r 1 , r 2 ] … ( r Q − 1 , r Q ] , ( r Q , ∞ ) (-\infty,r_1],(r_1,r_2]\ldots(r_{Q-1},r_Q],(r_Q,\infty) ( − ∞ , r 1 ] , ( r 1 , r 2 ] … ( r Q − 1 , r Q ] , ( r Q , ∞ ) ,
∣ F ( λ ) − H ϵ ( λ ) ∣ < ϵ |F(\lambda)-H_{\epsilon}(\lambda)|<\epsilon ∣ F ( λ ) − H ϵ ( λ ) ∣ < ϵ
Lemma B.13: Σ Π \Sigma\Pi ΣΠ for squashing functions. For every squashing function Ψ \Psi Ψ and every probability measure ρ \rho ρ on ( R n , B n ) (\mathbb{R}^{n},\mathcal{B}^{n}) ( R n , B n ) , Σ n ( Ψ ) \Sigma^{n}(\Psi) Σ n ( Ψ ) is uniformly dense on compacta in C ( R n ) C(\mathbb{R}^{n}) C ( R n ) and μ ρ \mu_{\rho} μ ρ -dense in M ( R n ) M(\mathbb{R}^{n}) M ( R n ) .
Proof. By Lemmas B.9 (connecting uniform convergence to μ ρ \mu_{\rho} μ ρ -convergence) and B.11 (Σ Π n ( G ) \Sigma\Pi^{n}(G) Σ Π n ( G ) is μ ρ \mu_{\rho} μ ρ -dense in M ( R n ) M(\mathbb{R}^{n}) M ( R n ) ), it is sufficient to show that Σ Π n ( Ψ ) \Sigma\Pi^{n}(\Psi) Σ Π n ( Ψ ) is uniformly dense on compacta in Σ Π n ( F ) \Sigma\Pi^{n}(F) Σ Π n ( F ) for some continuous squashing function F F F .
For this, it is sufficient to show that every function of the form ∏ k = 1 r F ( A k ( x ) ) \prod_{k=1}^{r}F(A_k(x)) ∏ k = 1 r F ( A k ( x )) can be uniformly approximated by members of Σ Π n ( Ψ ) \Sigma\Pi^{n}(\Psi) Σ Π n ( Ψ ) .
For an arbitrary ϵ > 0 \epsilon>0 ϵ > 0 , because multiplication is continuous and [ 0 , 1 ] r [0,1]^{r} [ 0 , 1 ] r is compact, there is a δ > 0 \delta>0 δ > 0 such that ∣ a k − b k ∣ < δ |a_k-b_k|<\delta ∣ a k − b k ∣ < δ for 0 ≤ a k , b k ≤ 1 0\le a_k,b_k\le1 0 ≤ a k , b k ≤ 1 for k ∈ { 1 … r } k\in\{1\ldots r\} k ∈ { 1 … r } . Therefore,
∣ ∏ k = 1 r a k − ∏ k = 1 r b k ∣ < ϵ . \left|\prod_{k=1}^{r}a_k-\prod_{k=1}^{r}b_k\right|<\epsilon. k = 1 ∏ r a k − k = 1 ∏ r b k < ϵ .
By Lemma B.12, there is a function H δ ( λ ) = ∑ t = 1 T β t Ψ ( A t r ( λ ) ) H_{\delta}(\lambda)=\sum_{t=1}^{T}\beta_t\Psi(A_t^r(\lambda)) H δ ( λ ) = ∑ t = 1 T β t Ψ ( A t r ( λ )) such that sup λ ∈ R ∣ F ( λ ) − H δ ( λ ) ∣ < ϵ \sup_{\lambda\in\mathbb{R}}|F(\lambda)-H_{\delta}(\lambda)|<\epsilon sup λ ∈ R ∣ F ( λ ) − H δ ( λ ) ∣ < ϵ . Then,
sup λ ∈ R n ∣ ∏ k = 1 r F ( A k ( x ) ) − ∏ k = 1 r H δ ( A k ( x ) ) ∣ < ϵ \sup_{\lambda\in\mathbb{R}^n}\left|\prod_{k=1}^{r}F(A_k(x))-\prod_{k=1}^{r}H_{\delta}(A_k(x))\right|<\epsilon λ ∈ R n sup k = 1 ∏ r F ( A k ( x )) − k = 1 ∏ r H δ ( A k ( x )) < ϵ
Since A t r ( A ( x ) ) A_t^r(A(x)) A t r ( A ( x )) is an affine function ∏ k = 1 r H δ ( A k ( x ) ) ∈ Σ Π n ( Ψ ) \prod_{k=1}^{r}H_{\delta}(A_k(x))\in\Sigma\Pi^n(\Psi) ∏ k = 1 r H δ ( A k ( x )) ∈ Σ Π n ( Ψ ) . Therefore, ∏ k = 1 r F ( A k ( x ) ) \prod_{k=1}^{r}F(A_k(x)) ∏ k = 1 r F ( A k ( x )) can be uniformly approximated by elements of Σ Π n ( Ψ ) \Sigma\Pi^n(\Psi) Σ Π n ( Ψ ) .
Lemma B.14. For every squashing function Ψ \Psi Ψ and and every probability measure ρ \rho ρ on ( R n , B n ) (\mathbb{R}^n,\mathcal{B}^n) ( R n , B n ) , Σ n ( Ψ ) \Sigma^n(\Psi) Σ n ( Ψ ) is uniformly dense on compacta in C ( R n ) C(\mathbb{R}^n) C ( R n ) and μ ρ \mu_\rho μ ρ -dense in M ( R n ) M(\mathbb{R}^n) M ( R n ) .
Proof. For every squashing function Ψ \Psi Ψ , ϵ > 0 \epsilon>0 ϵ > 0 , and M > 0 M>0 M > 0 , there is a function cos M ∈ Σ 1 ( Ψ ) \cos_M\in\Sigma^1(\Psi) cos M ∈ Σ 1 ( Ψ ) such that sup λ ∈ [ − M , + M ] ∣ cos M ( λ ) − cos ( λ ) ∣ < ϵ \sup_{\lambda\in[-M,+M]}|\cos_M(\lambda)-\cos(\lambda)|<\epsilon sup λ ∈ [ − M , + M ] ∣ cos M ( λ ) − cos ( λ ) ∣ < ϵ .
Just use Gallant and White’s cosine squasher [13 ] : Ψ c o s i n e ( z ) = ( 1 + cos ( z + 3 π / 2 ) ) ⋅ 1 { − π / 2 ≤ z < π / 2 } + 1 { z ≥ π / 2 } \Psi_{\mathit{cosine}}(z)=(1+\cos(z+3\pi/2))\cdot1_{\{-\pi/2\leq z<\pi/2\}}+1_{\{z\geq\pi/2\}} Ψ cosine ( z ) = ( 1 + cos ( z + 3 π /2 )) ⋅ 1 { − π /2 ≤ z < π /2 } + 1 { z ≥ π /2 } . By adding, subtracting and scaling a finite number of affinely shifted versions of Ψ c o s i n e \Psi_{\mathit{cosine}} Ψ cosine , we can get the cosine function on any interval [ − M , + M ] [-M,+M] [ − M , + M ] . Lemma B.12 and the triangle inequality provide the needed cos M \cos_M cos M function.
Let g ( x ) = ∑ j = 1 Q β j cos ( A j ( x ) ) g(x)=\sum_{j=1}^{Q}\beta_j\cos(A_j(x)) g ( x ) = ∑ j = 1 Q β j cos ( A j ( x )) , where A j A_j A j is an affine function. For arbitrary squashing function Ψ \Psi Ψ , compact K ⊂ R n K\subset\mathbb{R}^n K ⊂ R n , and ϵ > 0 \epsilon>0 ϵ > 0 , there is an f ∈ Σ n ( Ψ ) f\in\Sigma^n(\Psi) f ∈ Σ n ( Ψ ) such that sup x ∈ K ∣ g ( x ) − f ( x ) ∣ < ϵ \sup_{x\in K}|g(x)-f(x)|<\epsilon sup x ∈ K ∣ g ( x ) − f ( x ) ∣ < ϵ .
Pick M > 0 M>0 M > 0 such that A j ( K ) ⊂ [ − M , + M ] A_j(K)\subset[-M,+M] A j ( K ) ⊂ [ − M , + M ] for j ∈ { 1.. Q } j\in\{1..Q\} j ∈ { 1.. Q } . Since Q Q Q is finite, K K K is compact, and the affine functions A j A_j A j are continuous, such an M M M can be found. Let Q ′ = Q ⋅ ∑ j = 1 Q ∣ β j ∣ Q'=Q\cdot\sum_{j=1}^{Q}|\beta_j| Q ′ = Q ⋅ ∑ j = 1 Q ∣ β j ∣ . By the result in the previous paragraph, ∣ ∑ j = 1 Q β j cos M ( A j ( x ) ) − g ( x ) ∣ < ϵ |\sum_{j=1}^{Q}\beta_j\cos_M(A_j(x))-g(x)|<\epsilon ∣ ∑ j = 1 Q β j cos M ( A j ( x )) − g ( x ) ∣ < ϵ for all x ∈ K x\in K x ∈ K . Since cos M ∈ Σ 1 ( Ψ ) \cos_M\in\Sigma^1(\Psi) cos M ∈ Σ 1 ( Ψ ) , f ( x ) = ∑ j = 1 Q cos M ( A j ( x ) ) ∈ Σ n ( Ψ ) f(x)=\sum_{j=1}^{Q}\cos_M(A_j(x))\in\Sigma^n(\Psi) f ( x ) = ∑ j = 1 Q cos M ( A j ( x )) ∈ Σ n ( Ψ )
For every squashing function Ψ \Psi Ψ , Σ n ( Ψ ) \Sigma^n(\Psi) Σ n ( Ψ ) is uniformly dense on compacta in C ( R n ) C(\mathbb{R}^n) C ( R n ) :
By Theorem B.8, Σ Π \Sigma\Pi ΣΠ is uniformly dense. Therefore, the trigonometric polynomials { ∑ j = 1 Q β j ∏ k = 1 r cos ( A j k ( x ) ) } \{\sum_{j=1}^{Q}\beta_j\prod_{k=1}^{r}\cos(A_{jk}(x))\} { ∑ j = 1 Q β j ∏ k = 1 r cos ( A j k ( x ))} are uniformly dense on C ( R n ) C(\mathbb{R}^n) C ( R n ) , where A j k A_{jk} A j k are affine functions. Repeatedly applying the trigonometric identity cos ( a ) cos ( b ) = cos ( a + b ) − cos ( a − b ) \cos(a)\cos(b)=\cos(a+b)-\cos(a-b) cos ( a ) cos ( b ) = cos ( a + b ) − cos ( a − b ) allows us to rewrite every trigonometric polynomial in the form ∑ t = 1 T α t cos ( A t ( x ) ) \sum_{t=1}^{T}\alpha_t\cos(A_t(x)) ∑ t = 1 T α t cos ( A t ( x )) where α t ∈ R \alpha_t\in\mathbb{R} α t ∈ R and A t A_t A t is an affine function. The density result now follows from the approximation above.
As we have just seen, Σ n ( Ψ ) \Sigma^n(\Psi) Σ n ( Ψ ) is uniformly dense on compacta in C ( R n ) C(\mathbb{R}^n) C ( R n ) . By Lemma B.9 (connecting uniform convergence to μ ρ \mu_\rho μ ρ -convergence), Σ n ( Ψ ) \Sigma^n(\Psi) Σ n ( Ψ ) is μ ρ \mu_\rho μ ρ -dense in C ( R n ) C(\mathbb{R}^n) C ( R n ) .
The triangle inequality and Lemma B.10 imply that Σ n ( Ψ ) \Sigma^n(\Psi) Σ n ( Ψ ) is also μ ρ \mu_\rho μ ρ -dense in M ( R n ) M(\mathbb{R}^n) M ( R n ) .
Theorem II.1: Universal approximation theorem for artificial neural networks with bounded and continuous activation functions Let σ : R → R \sigma:\mathbb{R}\to\mathbb{R} σ : R → R be a bounded, non-constant, and continuous activation function. Then, for any compact set K ⊂ R n K\subset\mathbb{R}^n K ⊂ R n , the set of functions of the form
H ( x ) = ∑ j = 1 h α j σ ( ∑ i = 1 n w j i x i + b j ) H(x)=\sum_{j=1}^{h}\alpha_j\sigma\left(\sum_{i=1}^{n}w_{ji}x_i+b_j\right) H ( x ) = j = 1 ∑ h α j σ ( i = 1 ∑ n w j i x i + b j ) is dense in C ( K ) C(K) C ( K ) , the space of continuous functions on K K K with the uniform norm.
Proof. Hornik et al. [20 ] proved their result for Σ Π \Sigma\Pi ΣΠ networks and later restricted it to the particular case of Σ \Sigma Σ networks. They first proved that networks can approximate continuous function, and then extended their result to Borel measurable functions. Let G ( z ) : R → R G(z):\mathbb{R}\to\mathbb{R} G ( z ) : R → R be any continuous nonconstant function:
For every G G G and every probability measure ρ \rho ρ on ( R n , B n ) (\mathbb{R}^n,\mathcal{B}^n) ( R n , B n ) , Σ Π n ( G ) \Sigma\Pi^n(G) Σ Π n ( G ) is μ ρ \mu_\rho μ ρ -dense in M ( R n ) M(\mathbb{R}^n) M ( R n ) (Lemma B.11).
The activation function G G G is not required to be a squashing function. The continuity requirement discards the threshold function Ψ 1 ( z ) = 1 { z ≥ 0 } \Psi_1(z)=1_{\{z\geq0\}} Ψ 1 ( z ) = 1 { z ≥ 0 } .
However, for squashing functions continuity is not necessary. For every squashing function Ψ \Psi Ψ and and every probability measure ρ \rho ρ on ( R n , B n ) (\mathbb{R}^n,\mathcal{B}^n) ( R n , B n ) , Σ Π n ( Ψ ) \Sigma\Pi^n(\Psi) Σ Π n ( Ψ ) is uniformly dense on compacta in C ( R n ) C(\mathbb{R}^n) C ( R n ) and μ ρ \mu_\rho μ ρ -dense in M ( R n ) M(\mathbb{R}^n) M ( R n ) (Lemma B.13).
Because of their simpler structure, Σ \Sigma Σ networks have similar approximation capabilities (Lemma B.14): For every squashing function Ψ \Psi Ψ and and every probability measure ρ \rho ρ on ( R n , B n ) (\mathbb{R}^n,\mathcal{B}^n) ( R n , B n ) , Σ n ( Ψ ) \Sigma^n(\Psi) Σ n ( Ψ ) is uniformly dense on compacta in C ( R n ) C(\mathbb{R}^n) C ( R n ) and μ ρ \mu_\rho μ ρ -dense in M ( R n ) M(\mathbb{R}^n) M ( R n ) .
In other words, standard feedforward networks with a single hidden layer can approximate any continuous function uniformly on any compact set and any measurable function arbitrarily well, regardless of the squashing function Ψ \Psi Ψ (continuous or not). Therefore, Σ \Sigma Σ networks are universal approximators.
Remark. Hornik and his colleagues extended Funahashi’s theorem for non-continuous activation functions.
Neural networks can approximate functions that are not differentiable in the classical sense, but possess only a generalized derivative, as is the case for certain piecewise
differentiable functions [21 ] . Hornik established the general conditions ensuring that networks with sufficiently smooth activation functions are capable of arbitrarily accurate approximation to a function and its derivatives [19 ] .16
For broad classes of multilayer feedforward networks, the set Σ ( G ) \Sigma(G) Σ ( G ) is dense in general spaces of functions where distance between functions is measured taking into account differences between the derivatives of the functions [21 ] . In fact, whenever Ψ \Psi Ψ is continuous, bounded and nonconstant, standard multilayer feedforward networks with activation function Ψ \Psi Ψ can approximate any continuous function on compact subsets X ⊂ R n X \subset\mathbb{R}^{n} X ⊂ R n arbitrarily well with respect to uniform distance ρ μ , X \rho_{\mu,X} ρ μ , X provided that sufficiently many hidden units are available [19 ] .
Whereas [21 ] makes use of a Fourier integral representation for single hidden layer feedforward networks with a continuum of hidden units, as proposed by [22 ] , [19 ] improves their results following Cybenko’s original approach [10 ] : the application of the Hahn-Banach theorem (Theorem A.6) combined with representation theorems (Theorem A.2) for continuous linear functionals on the function spaces under consideration.
The set of all functions implemented by a one-hidden-layer neural network with h h h hidden units and a single output unit, which is assumed to be linear, is defined as
N n ( h ) [ Ψ ] = { f : R n → R | f ( x ) = ∑ j = 1 h β j Ψ ( w j ⋅ x + b j ) } \mathcal{N}^{(h)}_{n}[\Psi] = \left\{ f:\mathbb{R}^{n} \to\mathbb{R}\middle| f(x) = \sum_{j=1}^{h}\beta_j\Psi(w_j\cdot x+b_j) \right\} N n ( h ) [ Ψ ] = { f : R n → R f ( x ) = j = 1 ∑ h β j Ψ ( w j ⋅ x + b j ) }
The set of all functions implemented by such networks with an arbitrary number of hidden units is
N n [ Ψ ] = ⋃ h = 1 ∞ N n ( h ) [ Ψ ] \mathcal{N}_{n}[\Psi] = \bigcup_{h=1}^{\infty}\mathcal{N}^{(h)}_{n}[\Psi] N n [ Ψ ] = h = 1 ⋃ ∞ N n ( h ) [ Ψ ]
For 1 ≤ p < ∞ 1 \le p < \infty 1 ≤ p < ∞ , the L p L^{p} L p norm is defined as
∥ f ∥ p , μ = [ ∫ R n ∣ f ( x ) ∣ p d μ ( x ) ] 1 / p \|f\|_{p,\mu} = \left[\int_{\mathbb{R}^{n}} |f(x)|^{p}d\mu(x)\right]^{1/p} ∥ f ∥ p , μ = [ ∫ R n ∣ f ( x ) ∣ p d μ ( x ) ] 1/ p
The L p L^{p} L p distance (or metric) defines the distance between two functions:
ρ p , μ ( f , g ) = ∥ f − g ∥ p , μ \rho_{p,\mu}(f,g) = \|f-g\|_{p,\mu} ρ p , μ ( f , g ) = ∥ f − g ∥ p , μ
L p ( μ ) L^{p}(\mu) L p ( μ ) is the space of all functions such that ∥ f ∥ p , μ < ∞ \|f\|_{p,\mu}<\infty ∥ f ∥ p , μ < ∞ , i.e. is the set of all measurable functions whose total size (measured by the L p L^{p} L p norm) is a finite number. For a function to belong to L p ( μ ) L^{p}(\mu) L p ( μ ) , its p p p -th power must be integrable over the entire domain, which forces the function to be well-behaved in two main ways: controlled spikes (even with vertical asymptotes, it cannot shoot up to infinity too sharply) and fast decay at infinity (must shrink toward zero quickly enough).
A subset S ⊂ L p ( μ ) S \subset L^{p}(\mu) S ⊂ L p ( μ ) is dense in L p ( μ ) L^{p}(\mu) L p ( μ ) if, for arbitrary f ∈ L p ( μ ) f \in L^{p}(\mu) f ∈ L p ( μ ) and ϵ > 0 \epsilon> 0 ϵ > 0 , there is a function g ∈ S g \in S g ∈ S such that ρ p , μ ( f , g ) < ϵ \rho_{p,\mu}(f,g) < \epsilon ρ p , μ ( f , g ) < ϵ .
Theorem B.15. If Ψ \Psi Ψ is bounded and nonconstant, then N n [ Ψ ] \mathcal{N}_{n}[\Psi] N n [ Ψ ] is dense in L p ( μ ) L^{p}(\mu) L p ( μ ) for all finite measures μ \mu μ on R n \mathbb{R}^{n} R n .
Corollary B.16. As convergence in L p ( μ ) L^{p}(\mu) L p ( μ ) implies convergence in μ \mu μ measure, whenever Ψ \Psi Ψ is bounded and nonconstant, all measurable functions on R n \mathbb{R}^{n} R n can be approximated by functions in N n [ Ψ ] \mathcal{N}_{n}[\Psi] N n [ Ψ ] .
C ( X ) C(X) C ( X ) is the space of all continuous functions on X X X . A subset S ⊂ C ( X ) S \subset C(X) S ⊂ C ( X ) is dense in C ( X ) C(X) C ( X ) if, for arbitrary f ∈ C ( X ) f \in C(X) f ∈ C ( X ) and ϵ > 0 \epsilon> 0 ϵ > 0 , there is a function g ∈ S g \in S g ∈ S such that ρ p , X ( f , g ) < ϵ \rho_{p,X}(f,g) < \epsilon ρ p , X ( f , g ) < ϵ .
Theorem B.17. If Ψ \Psi Ψ is continuous, bounded, and nonconstant, then N n [ Ψ ] \mathcal{N}_{n}[\Psi] N n [ Ψ ] is dense in C ( X ) C(X) C ( X ) for all compact subsets X ⊂ R n X \subset\mathbb{R}^{n} X ⊂ R n .
Proof of Theorems B.15 and B.17. Since Ψ \Psi Ψ is bounded, N n [ Ψ ] \mathcal{N}_{n}[\Psi] N n [ Ψ ] is a linear subspace of L p ( μ ) L^{p}(\mu) L p ( μ ) for all finite measures μ \mu μ on R n \mathbb{R}^{n} R n . If, for some measure μ \mu μ , N n [ Ψ ] \mathcal{N}_{n}[\Psi] N n [ Ψ ] is not dense in L p ( μ ) L^{p}(\mu) L p ( μ ) , there is a nonzero continuous linear functional Λ \Lambda Λ on L p ( μ ) L^{p}(\mu) L p ( μ ) that vanishes on N n [ Ψ ] \mathcal{N}_{n}[\Psi] N n [ Ψ ] .
Λ \Lambda Λ is of the form Λ ( f ) = ∫ R n f g d μ \Lambda(f)=\int_{\mathbb{R}^{n}}fg\,d\mu Λ ( f ) = ∫ R n f g d μ for some g ∈ L q ( μ ) g \in L^{q}(\mu) g ∈ L q ( μ ) where q q q is the conjugate exponent q = p / ( p − 1 ) q=p/(p-1) q = p / ( p − 1 ) .17 By Hölder’s inequality, ∥ f g ∥ 1 ≤ ∥ f ∥ p ∥ g ∥ q \|fg\|_{1}\leq\|f\|_{p}\|g\|_{q} ∥ f g ∥ 1 ≤ ∥ f ∥ p ∥ g ∥ q . Given σ ( B ) = ∫ B g d μ \sigma(B)=\int_{B}g\,d\mu σ ( B ) = ∫ B g d μ :
∣ σ ( B ) ∣ = ∣ ∫ R n 1 B g d μ ∣ ≤ ∥ 1 B ∥ p , μ ∥ g ∥ q , μ ≤ μ ( R n ) p ∥ g ∥ q , μ < ∞ |\sigma(B)|=\left|\int_{\mathbb{R}^{n}}1_{B}g\,d\mu\right|
\leq\|1_{B}\|_{p,\mu}\|g\|_{q,\mu}
\leq\sqrt[p]{\mu(\mathbb{R}^{n})}\|g\|_{q,\mu}<\infty ∣ σ ( B ) ∣ = ∫ R n 1 B g d μ ≤ ∥ 1 B ∥ p , μ ∥ g ∥ q , μ ≤ p μ ( R n ) ∥ g ∥ q , μ < ∞
Therefore, σ \sigma σ is a nonzero finite signed measure on R n \mathbb{R}^{n} R n such that Λ ( f ) = ∫ R n f g d μ = ∫ R n f d σ \Lambda(f)=\int_{\mathbb{R}^{n}}fg\,d\mu=\int_{\mathbb{R}^{n}}f\,d\sigma Λ ( f ) = ∫ R n f g d μ = ∫ R n f d σ . Since Λ \Lambda Λ vanishes on N n [ Ψ ] \mathcal{N}_{n}[\Psi] N n [ Ψ ] , for all w ∈ R n w\in\mathbb{R}^{n} w ∈ R n and b ∈ R b\in\mathbb{R} b ∈ R :
∫ R n Ψ ( w x + b ) d σ ( x ) = 0 \int_{\mathbb{R}^{n}}\Psi(wx+b)\,d\sigma(x)=0 ∫ R n Ψ ( w x + b ) d σ ( x ) = 0
Similarly, suppose that Ψ \Psi Ψ is continuous and, for some compact subset X X X of R n \mathbb{R}^{n} R n , N n [ Ψ ] \mathcal{N}_{n}[\Psi] N n [ Ψ ] is not dense in C ( X ) C(X) C ( X ) . Following Cybenko’s proof (Theorem 1.1), there exists a nonzero finite signed measure σ \sigma σ on R n \mathbb{R}^n R n (σ \sigma σ is actually concentrated on X X X ) such that, for all w ∈ R n w \in\mathbb{R}^n w ∈ R n and b ∈ R b \in\mathbb{R} b ∈ R :
∫ R n Ψ ( w x + b ) d σ ( x ) = 0 \int_{\mathbb{R}^n} \Psi(wx+b)\,d\sigma(x)=0 ∫ R n Ψ ( w x + b ) d σ ( x ) = 0
In both cases, we reduce our problem to answer whether there exists a nonzero finite signed measure σ \sigma σ , i.e. whether Ψ \Psi Ψ is discriminatory or not (Definition B.1). Cybenko [10 ] proved that, when Ψ \Psi Ψ is sigmoidal, Ψ \Psi Ψ is discriminatory. Hornik [19 ] proved a stronger result: whenever Ψ \Psi Ψ is bounded and nonconstant, Ψ \Psi Ψ is discriminatory (see Theorem B.18 below). Combining Hornik’s result with our argument, the proof of Theorems B.15 and B.17 is complete.
Theorem B.18. Whenever Ψ \Psi Ψ is bounded and nonconstant, Ψ \Psi Ψ is discriminatory.
Proof. Hornik’s proof [19 ] is based on Fourier analysis. Suppose that Ψ \Psi Ψ is bounded and nonconstant. Suppose that σ \sigma σ is a finite signed measure σ \sigma σ on R n \mathbb{R}^n R n such that
∫ R n Ψ ( w x + b ) d σ ( x ) = 0 \int_{\mathbb{R}^n}\Psi(wx+b)\,d\sigma(x)=0 ∫ R n Ψ ( w x + b ) d σ ( x ) = 0
for all w ∈ R n w \in\mathbb{R}^n w ∈ R n and b ∈ R b \in\mathbb{R} b ∈ R . Set u ∈ R n u \in\mathbb{R}^n u ∈ R n and define a finite signed measure σ u \sigma_u σ u on R \mathbb{R} R induced by the transformation x ↦ u ⋅ x x \mapsto u \cdot x x ↦ u ⋅ x . For all Borel sets, we have σ u ( B ) = σ { x ∈ R n : u ⋅ x ∈ B } \sigma_u(B)=\sigma\{x\in\mathbb{R}^n:u\cdot x\in B\} σ u ( B ) = σ { x ∈ R n : u ⋅ x ∈ B } .
For all bounded functions χ \chi χ on R \mathbb{R} R :
∫ R n χ ( u x ) d σ ( x ) = ∫ R χ ( t ) d σ u ( t ) \int_{\mathbb{R}^n}\chi(ux)\,d\sigma(x)=\int_{\mathbb{R}}\chi(t)\,d\sigma_u(t) ∫ R n χ ( ux ) d σ ( x ) = ∫ R χ ( t ) d σ u ( t )
Hence, for all w , b ∈ R w,b\in\mathbb{R} w , b ∈ R :
∫ R n Ψ ( w u x + b ) d σ ( x ) = ∫ R Ψ ( w t + b ) d σ u ( t ) = 0 \int_{\mathbb{R}^n}\Psi(wux+b)\,d\sigma(x)=\int_{\mathbb{R}}\Psi(wt+b)\,d\sigma_u(t)=0 ∫ R n Ψ ( w ux + b ) d σ ( x ) = ∫ R Ψ ( w t + b ) d σ u ( t ) = 0
By choosing b b b such that Ψ ( b ) ≠ 0 \Psi(b)\ne0 Ψ ( b ) = 0 and w = 0 w=0 w = 0 , we find that ∫ R d σ u ( t ) = σ ^ u ( 0 ) = 0 \int_{\mathbb{R}}d\sigma_u(t)=\widehat{\sigma}_u(0)=0 ∫ R d σ u ( t ) = σ u ( 0 ) = 0 , where σ ^ u \widehat{\sigma}_u σ u is the Fourier transform of σ u \sigma_u σ u .
For u = 0 u=0 u = 0 , σ 0 \sigma_0 σ 0 is concentrated at t = 0 t=0 t = 0 and σ 0 { 0 } = σ ^ 0 = 0 \sigma_0\{0\}=\widehat{\sigma}_0=0 σ 0 { 0 } = σ 0 = 0 . Hence σ 0 = 0 \sigma_0=0 σ 0 = 0 .
Now suppose u ≠ 0 u\ne0 u = 0 . Choose an integrable function f ∈ L 1 ( R ) f\in L^1(\mathbb{R}) f ∈ L 1 ( R ) whose Fourier transform has no zero, e.g. f ( t ) = e − t 2 f(t)=e^{-t^2} f ( t ) = e − t 2 . As, by Hölder’s inequality,
∫ R ∫ R ∣ Ψ ( w ( s + t ) + b ) ∣ ∣ f ( s ) ∣ d s d ∣ σ u ( t ) ∣ ≤ ∥ f ∥ L 1 ∥ σ u ∥ M sup t ∈ R ∣ Ψ ( t ) ∣ < ∞ \int_{\mathbb{R}}\int_{\mathbb{R}}|\Psi(w(s+t)+b)|\,|f(s)|\,ds\,d|\sigma_u(t)|
\le\|f\|_{L^1}\|\sigma_u\|_M\sup_{t\in\mathbb{R}}|\Psi(t)|<\infty ∫ R ∫ R ∣Ψ ( w ( s + t ) + b ) ∣ ∣ f ( s ) ∣ d s d ∣ σ u ( t ) ∣ ≤ ∥ f ∥ L 1 ∥ σ u ∥ M t ∈ R sup ∣Ψ ( t ) ∣ < ∞
we can apply Fubini’s theorem:
0 = ∫ R [ ∫ R Ψ ( w t + ( b + w s ) ) d σ u ( t ) ] f ( s ) d s = ∫ R ∫ R Ψ ( w ( s + t ) + b ) f ( s ) d s d σ u ( t ) = ∫ R Ψ ( w t + b ) d ( f ∗ σ u ) ( t ) \begin{aligned}
0&=\int_{\mathbb{R}}\left[\int_{\mathbb{R}}\Psi(wt+(b+ws))\,d\sigma_u(t)\right]f(s)\,ds\\
&=\int_{\mathbb{R}}\int_{\mathbb{R}}\Psi(w(s+t)+b)f(s)\,ds\,d\sigma_u(t)\\
&=\int_{\mathbb{R}}\Psi(wt+b)\,d(f*\sigma_u)(t)
\end{aligned} 0 = ∫ R [ ∫ R Ψ ( w t + ( b + w s )) d σ u ( t ) ] f ( s ) d s = ∫ R ∫ R Ψ ( w ( s + t ) + b ) f ( s ) d s d σ u ( t ) = ∫ R Ψ ( w t + b ) d ( f ∗ σ u ) ( t )
where f ∗ σ u f*\sigma_u f ∗ σ u denotes the convolution of f f f and σ u \sigma_u σ u . Since L 1 ( R ) L^1(\mathbb{R}) L 1 ( R ) is a closed ideal in M ( R ) M(\mathbb{R}) M ( R ) , for f ∈ L 1 ( R ) f\in L^1(\mathbb{R}) f ∈ L 1 ( R ) and μ ∈ M ( R ) \mu\in M(\mathbb{R}) μ ∈ M ( R ) , the convolution f ∗ μ = μ ∗ f ∈ L 1 ( R ) f*\mu=\mu*f\in L^1(\mathbb{R}) f ∗ μ = μ ∗ f ∈ L 1 ( R ) . In particular, f ∗ σ u f*\sigma_u f ∗ σ u is absolutely continuous with respect to Lebesgue measure. Let h ∈ L 1 ( R ) h\in L^1(\mathbb{R}) h ∈ L 1 ( R ) be the corresponding Radon-Nikodym derivative. Then, h ^ = f ^ ⋅ σ ^ u \widehat{h}=\widehat{f}\cdot\widehat{\sigma}_u h = f ⋅ σ u and h ^ ( 0 ) = 0 \widehat{h}(0)=0 h ( 0 ) = 0 . The above equation is, therefore, equivalent to
∫ R Ψ ( w t + b ) h ( t ) d t = 0 \int_{\mathbb{R}}\Psi(wt+b)h(t)\,dt=0 ∫ R Ψ ( w t + b ) h ( t ) d t = 0
Let α ≠ 0 \alpha\ne0 α = 0 and γ ∈ R \gamma\in\mathbb{R} γ ∈ R . By replacing w w w by 1 / α 1/\alpha 1/ α and b b b by γ / α \gamma/\alpha γ / α and performing the change of variables t ↦ α t − γ t\mapsto\alpha t-\gamma t ↦ α t − γ :
∫ R Ψ ( t ) h ( α t − γ ) d t = 0 \int_{\mathbb{R}}\Psi(t)h(\alpha t-\gamma)\,dt=0 ∫ R Ψ ( t ) h ( α t − γ ) d t = 0
This equation implies that ∫ R Ψ ( t ) f ( t ) d t \int_{\mathbb{R}}\Psi(t)f(t)\,dt ∫ R Ψ ( t ) f ( t ) d t vanishes for all f f f in the closed translation subspace I I I spanned by the family M α h M_\alpha h M α h , where M α h ( t ) = h ( α t ) M_\alpha h(t)=h(\alpha t) M α h ( t ) = h ( α t ) , for α ≠ 0 \alpha\ne0 α = 0 . As a consequence, I I I is an ideal in L 1 ( R ) L^1(\mathbb{R}) L 1 ( R ) .
Define Z ( f ) Z(f) Z ( f ) as the set of all ω ∈ R \omega\in\mathbb{R} ω ∈ R where the Fourier transform f ^ ( ω ) \widehat{f}(\omega) f ( ω ) vanishes. When I I I is an ideal, Z ( I ) Z(I) Z ( I ) , the zero set of I I I , is the set of ω \omega ω where the Fourier transforms of all functions in I I I vanish.
Suppose that h h h is nonzero. As M α h ( ω ) = h ^ ( ω / α ) α M_\alpha h(\omega)=\widehat{h}(\omega/\alpha)\alpha M α h ( ω ) = h ( ω / α ) α , we find that Z ( I ) = { 0 } Z(I)=\{0\} Z ( I ) = { 0 } and I I I is the set of all integrable functions f f f with ∫ R f ( t ) d t = f ^ ( 0 ) = 0 \int_{\mathbb{R}}f(t)\,dt=\widehat{f}(0)=0 ∫ R f ( t ) d t = f ( 0 ) = 0 .
For all functions f ∈ I f\in I f ∈ I , we trivially have { 0 } = Z ( I ) ⊆ Z ( f ) \{0\}=Z(I)\subseteq Z(f) { 0 } = Z ( I ) ⊆ Z ( f ) . Conversely, suppose that f f f has zero integral. As the intersection of the boundaries of Z ( I ) Z(I) Z ( I ) and Z ( f ) Z(f) Z ( f ) equals { 0 } \{0\} { 0 } and contains no perfect set, f ∈ I f\in I f ∈ I .
When h h h is nonzero, the fact that the integral ∫ R Ψ ( t ) f ( t ) d t \int_{\mathbb{R}}\Psi(t)f(t)\,dt ∫ R Ψ ( t ) f ( t ) d t vanishes for all integrable functions which have zero integral implies that Ψ \Psi Ψ is constant, which was ruled out by our initial assumption. Hence h = 0 h=0 h = 0 and h ^ = f ^ ⋅ σ ^ u \widehat{h}=\widehat{f}\cdot\widehat{\sigma}_u h = f ⋅ σ u is identically zero. In turn, σ ^ u \widehat{\sigma}_u σ u vanishes identically, because f ^ \widehat{f} f . Therefore, σ u = 0 \sigma_u=0 σ u = 0 for all u ∈ R n u\in\mathbb{R}^n u ∈ R n .
Then, the Fourier transform of σ \sigma σ at u u u is
σ ^ ( u ) = ∫ R n e i u x d σ ( x ) = ∫ R e i t d σ u ( x ) = 0 \begin{aligned}
\widehat{\sigma}(u)&=\int_{\mathbb{R}^n}e^{iux}\,d\sigma(x)\\
&=\int_{\mathbb{R}}e^{it}\,d\sigma_u(x)\\
&=0
\end{aligned} σ ( u ) = ∫ R n e i ux d σ ( x ) = ∫ R e i t d σ u ( x ) = 0
Since σ ^ = 0 \widehat{\sigma}=0 σ = 0 , then σ = 0 \sigma=0 σ = 0 and our proof is complete: Ψ \Psi Ψ is discriminatory, i.e. no nonzero finite signed measure σ \sigma σ exists such that ∫ Ψ ( w x + b ) d σ ( x ) = 0 \int\Psi(wx+b)\,d\sigma(x)=0 ∫ Ψ ( w x + b ) d σ ( x ) = 0 .
A multi-index is a k k k -tuple α = ( α 1 , … , α k ) \alpha=(\alpha_1,\ldots,\alpha_k) α = ( α 1 , … , α k ) of non-negative integers. The order of the multi-index is ∣ α ∣ = ∑ i = 1 k α i |\alpha|=\sum_{i=1}^k\alpha_i ∣ α ∣ = ∑ i = 1 k α i and its corresponding partial derivative is
D α f ( x ) = ∂ α 1 ⋯ α k f ∂ ξ 1 α 1 ⋯ ∂ ξ k α k ( x ) D^\alpha f(x)=\frac{\partial^{\alpha_1\cdots\alpha_k}f}{\partial\xi_1^{\alpha_1}\cdots\partial\xi_k^{\alpha_k}}(x) D α f ( x ) = ∂ ξ 1 α 1 ⋯ ∂ ξ k α k ∂ α 1 ⋯ α k f ( x )
for a sufficiently smooth function f f f of x = ( ξ 1 , … , ξ k ) ∈ R k x=(\xi_1,\ldots,\xi_k)\in\mathbb{R}^k x = ( ξ 1 , … , ξ k ) ∈ R k .
C m ( R k ) C^m(\mathbb{R}^k) C m ( R k ) is the space of all functions f f f which, with all their partial derivatives D α f D^\alpha f D α f of order ∣ α ∣ ≥ m |\alpha|\geq m ∣ α ∣ ≥ m , are continuous on R k \mathbb{R}^k R k .
For all subsets X ⊂ R k X\subset\mathbb{R}^k X ⊂ R k and f ∈ C m ( R k ) f\in C^m(\mathbb{R}^k) f ∈ C m ( R k ) :
∥ f ∥ m , μ , X = max ∣ α ∣ ≤ m sup x ∈ X ∣ D α f ( x ) ∣ \|f\|_{m,\mu,X}=\max_{|\alpha|\leq m}\sup_{x\in X}|D^\alpha f(x)| ∥ f ∥ m , μ , X = ∣ α ∣ ≤ m max x ∈ X sup ∣ D α f ( x ) ∣
A subset S S S of C m ( R k ) C^m(\mathbb{R}^k) C m ( R k ) is uniformly dense on compacta in C m ( R k ) C^m(\mathbb{R}^k) C m ( R k ) if, for all compact subsets X X X of R k \mathbb{R}^k R k and all ϵ > 0 \epsilon>0 ϵ > 0 , there is a function g ( f , X , ϵ ) ∈ S g(f,X,\epsilon)\in S g ( f , X , ϵ ) ∈ S such that ∥ f − g ∥ m , μ , X < ϵ \|f-g\|_{m,\mu,X}<\epsilon ∥ f − g ∥ m , μ , X < ϵ .
For f ∈ C m ( R k ) f\in C^m(\mathbb{R}^k) f ∈ C m ( R k ) , μ \mu μ a finite measure on R k \mathbb{R}^k R k , and 1 ≤ p < ∞ 1\leq p<\infty 1 ≤ p < ∞ ,
∥ f ∥ m , p , μ = [ ∑ ∣ α ∣ ≤ m ∫ R k ∣ D α f ∣ p d μ ] 1 / p \|f\|_{m,p,\mu}=\left[\sum_{|\alpha|\leq m}\int_{\mathbb{R}^k}|D^\alpha f|^p\,d\mu\right]^{1/p} ∥ f ∥ m , p , μ = ∣ α ∣ ≤ m ∑ ∫ R k ∣ D α f ∣ p d μ 1/ p
The weighted Sobolev space C m , p ( μ ) C^{m,p}(\mu) C m , p ( μ ) is defined by
C m , p ( μ ) = { f ∈ C m ( R k ) : ∥ f ∥ m , p , μ < ∞ } C^{m,p}(\mu)=\{f\in C^m(\mathbb{R}^k):\|f\|_{m,p,\mu}<\infty\} C m , p ( μ ) = { f ∈ C m ( R k ) : ∥ f ∥ m , p , μ < ∞ }
It should be noted that C m , p ( μ ) = C m ( R k ) C^{m,p}(\mu)=C^m(\mathbb{R}^k) C m , p ( μ ) = C m ( R k ) if μ \mu μ has compact support.
A subset S S S of C m , p ( μ ) C^{m,p}(\mu) C m , p ( μ ) is dense in C m , p ( μ ) C^{m,p}(\mu) C m , p ( μ ) if, for all f ∈ C m , p ( μ ) f\in C^{m,p}(\mu) f ∈ C m , p ( μ ) and ϵ > 0 \epsilon>0 ϵ > 0 , there is a function g ( f , ϵ ) ∈ S g(f,\epsilon)\in S g ( f , ϵ ) ∈ S such that ∥ f − g ∥ m , p , μ < ϵ \|f-g\|_{m,p,\mu}<\epsilon ∥ f − g ∥ m , p , μ < ϵ .
Lemma B.19. Let w w w be the familiar function in C ∞ ( R n ) C^\infty(\mathbb{R}^n) C ∞ ( R n ) with support in the unit sphere given by
w ( x ) = { c e − 1 / ( 1 − ∣ x ∣ 2 ) if ∣ x ∣ < 1 , 0 if ∣ x ∣ ≥ 1 w(x)=
\begin{cases}
c e^{-1/(1-|x|^2)} & \text{if } |x|<1,\\
0 & \text{if } |x|\geq1
\end{cases} w ( x ) = { c e − 1/ ( 1 − ∣ x ∣ 2 ) 0 if ∣ x ∣ < 1 , if ∣ x ∣ ≥ 1 where ∣ x ∣ |x| ∣ x ∣ is the Euclidean length of x x x and c c c a constant chosen so that ∫ R n w ( x ) d x = 1 \int_{\mathbb{R}^n}w(x)\,dx=1 ∫ R n w ( x ) d x = 1 . For ϵ > 0 \epsilon>0 ϵ > 0 ,
w ϵ ( x ) = ϵ − n w ( x / ϵ ) w_\epsilon(x)=\epsilon^{-n}w(x/\epsilon) w ϵ ( x ) = ϵ − n w ( x / ϵ ) Suppose that f f f and σ \sigma σ satisfy one of the two following conditions:
If T y f ( x ) = f ( x + y ) T_yf(x)=f(x+y) T y f ( x ) = f ( x + y ) denotes translation by y y y :
∫ R n f J ϵ σ d x = ∫ R n [ ∫ R n T y f d σ ] w ϵ ( y ) d y \int_{\mathbb{R}^n}fJ_\epsilon\sigma\,dx
=\int_{\mathbb{R}^n}\left[\int_{\mathbb{R}^n}T_yf\,d\sigma\right]w_\epsilon(y)\,dy ∫ R n f J ϵ σ d x = ∫ R n [ ∫ R n T y f d σ ] w ϵ ( y ) d y = ∫ R n J ϵ f d σ =\int_{\mathbb{R}^n}J_\epsilon f\,d\sigma = ∫ R n J ϵ f d σ lim ϵ → 0 ∫ R n f J ϵ σ d x = ∫ R n f d σ \lim_{\epsilon\to0}\int_{\mathbb{R}^n}fJ_\epsilon\sigma\,dx
=\int_{\mathbb{R}^n}f\,d\sigma ϵ → 0 lim ∫ R n f J ϵ σ d x = ∫ R n f d σ where, for a locally finite signed measure σ \sigma σ on R n \mathbb{R}^n R n , J ϵ σ J_\epsilon\sigma J ϵ σ is the convolution w ϵ ∗ σ w_\epsilon*\sigma w ϵ ∗ σ .
Proof. The integral equation is a straightforward application of Fubini’s theorem using the symmetry of w ϵ w_\epsilon w ϵ . The limit follows from Lebesgue’s bounded convergence theorem (Theorem A.3).
Remark. If f f f is a locally integrable function on R n \mathbb{R}^n R n , let J ϵ f J_\epsilon f J ϵ f be the convolution w ϵ ∗ f w_\epsilon*f w ϵ ∗ f :
J ϵ f = w ϵ ∗ f ∈ C ∞ ( R n ) J_\epsilon f=w_\epsilon*f\in C^\infty(\mathbb{R}^n) J ϵ f = w ϵ ∗ f ∈ C ∞ ( R n ) .
∥ J ϵ f ∥ u ≤ ∥ f ∥ u \|J_\epsilon f\|_u\leq\|f\|_u ∥ J ϵ f ∥ u ≤ ∥ f ∥ u , where ∥ f ∥ u = sup R n ∣ f ( x ) ∣ \|f\|_u=\sup_{\mathbb{R}^n}|f(x)| ∥ f ∥ u = sup R n ∣ f ( x ) ∣ .
If f f f is bounded, then J ϵ f J_\epsilon f J ϵ f is uniformly bounded in x x x and ϵ \epsilon ϵ .
If σ \sigma σ is a locally finite signed measure, J ϵ σ = w ϵ ∗ σ ∈ C ∞ ( R n ) J_\epsilon\sigma=w_\epsilon*\sigma\in C^\infty(\mathbb{R}^n) J ϵ σ = w ϵ ∗ σ ∈ C ∞ ( R n ) .
If σ \sigma σ has compact support, J ϵ σ J_\epsilon\sigma J ϵ σ has compact support.
Theorem B.20. If Ψ ∈ C m ( R n ) \Psi\in C^m(\mathbb{R}^n) Ψ ∈ C m ( R n ) is nonconstant and bounded, then N n [ Ψ ] \mathcal{N}_n[\Psi] N n [ Ψ ] is uniformly m m m -dense on compacta in C m , p ( μ ) C^{m,p}(\mu) C m , p ( μ ) for all finite measures μ \mu μ on R n \mathbb{R}^n R n with compact support.
Proof. If N n [ Ψ ] \mathcal{N}_n[\Psi] N n [ Ψ ] is not uniformly m m m -dense on compacta in C m ( R n ) C^m(\mathbb{R}^n) C m ( R n ) , by the dual space argument, there exists a collection σ α \sigma_\alpha σ α , ∣ α ∣ ≤ m |\alpha|\leq m ∣ α ∣ ≤ m , of finite signed measures with support in some compact subset X X X of R n \mathbb{R}^n R n such that the functional
Λ [ f ] = ∑ ∣ α ∣ ≤ m ∫ R n D α f d σ α \Lambda[f]=\sum_{|\alpha|\leq m}\int_{\mathbb{R}^n}D^\alpha f\,d\sigma_\alpha Λ [ f ] = ∣ α ∣ ≤ m ∑ ∫ R n D α f d σ α
vanishes on N n [ Ψ ] \mathcal{N}_n[\Psi] N n [ Ψ ] , but not identically on C m ( R n ) C^m(\mathbb{R}^n) C m ( R n ) . For ϵ > 0 \epsilon>0 ϵ > 0 , let us define the functionals Λ ϵ \Lambda_\epsilon Λ ϵ as
Λ ϵ [ f ] = ∑ ∣ α ∣ ≤ m ∫ R n D α J ϵ σ α d x \Lambda_\epsilon[f]=\sum_{|\alpha|\leq m}\int_{\mathbb{R}^n}D^\alpha J_\epsilon\sigma_\alpha\,dx Λ ϵ [ f ] = ∣ α ∣ ≤ m ∑ ∫ R n D α J ϵ σ α d x
The integrals exist because J ϵ σ α J_\epsilon\sigma_\alpha J ϵ σ α have compact support. For all f ∈ C m ( R n ) f\in C^m(\mathbb{R}^n) f ∈ C m ( R n ) , using the lemma above:
Λ ϵ [ f ] = ∫ R n [ ∑ ∣ α ∣ ≤ m ∫ R n D α T y f d σ α ] w ϵ ( y ) d y = ∫ R n Λ [ T y f ] w ϵ ( y ) d y \begin{aligned}
\Lambda_\epsilon[f]=\int_{\mathbb{R}^n}\left[\sum_{|\alpha|\leq m}\int_{\mathbb{R}^n}D^\alpha T_yf\,d\sigma_\alpha\right]w_\epsilon(y)\,dy \\
&=\int_{\mathbb{R}^n}\Lambda[T_yf]w_\epsilon(y)\,dy
\end{aligned} Λ ϵ [ f ] = ∫ R n ∣ α ∣ ≤ m ∑ ∫ R n D α T y f d σ α w ϵ ( y ) d y = ∫ R n Λ [ T y f ] w ϵ ( y ) d y
and
lim ϵ → 0 Λ ϵ [ f ] = Λ [ f ] \lim_{\epsilon\to0}\Lambda_\epsilon[f]=\Lambda[f] ϵ → 0 lim Λ ϵ [ f ] = Λ [ f ]
Integration by parts yields
Λ ϵ [ f ] = ∫ R n [ ∑ ∣ α ∣ ≤ m ( − 1 ) α D α J ϵ σ α ] f d x \Lambda_\epsilon[f]=\int_{\mathbb{R}^n}\left[\sum_{|\alpha|\leq m}(-1)^\alpha D^\alpha J_\epsilon\sigma_\alpha\right]f\,dx Λ ϵ [ f ] = ∫ R n ∣ α ∣ ≤ m ∑ ( − 1 ) α D α J ϵ σ α f d x
If we define h ϵ = ∑ ∣ α ∣ ≤ m ( − 1 ) α D α J ϵ σ α h_{\epsilon}=\sum_{|\alpha|\leq m}(-1)^{\alpha}D^{\alpha}J_{\epsilon}\sigma_{\alpha} h ϵ = ∑ ∣ α ∣ ≤ m ( − 1 ) α D α J ϵ σ α , then Λ ϵ [ f ] = ∫ R n h ϵ f d x \Lambda_{\epsilon}[f]=\int_{\mathbb{R}^{n}}h_{\epsilon}f\,dx Λ ϵ [ f ] = ∫ R n h ϵ f d x .
Let us write Ψ θ , θ 0 ( x ) = Ψ ( θ ⋅ x + θ 0 ) \Psi_{\theta,\theta_{0}}(x)=\Psi(\theta\cdot x+\theta_{0}) Ψ θ , θ 0 ( x ) = Ψ ( θ ⋅ x + θ 0 ) . Suppose that Λ \Lambda Λ vanishes on N n [ Ψ ] \mathcal{N}_{n}[\Psi] N n [ Ψ ] . Since Ψ θ , θ 0 ∈ N n [ Ψ ] \Psi_{\theta,\theta_{0}}\in\mathcal{N}_{n}[\Psi] Ψ θ , θ 0 ∈ N n [ Ψ ] , we infer Λ [ Ψ θ , θ 0 ] = 0 \Lambda[\Psi_{\theta,\theta_{0}}]=0 Λ [ Ψ θ , θ 0 ] = 0 . Given that T y Ψ θ , θ 0 = Ψ θ , θ 0 − θ ⋅ y T_{y}\Psi_{\theta,\theta_{0}}=\Psi_{\theta,\theta_{0}-\theta\cdot y} T y Ψ θ , θ 0 = Ψ θ , θ 0 − θ ⋅ y , it follows that
∫ R n Ψ θ , θ 0 h ϵ d x = Λ ϵ [ Ψ θ , θ 0 ] = ∫ R n Λ [ T y Ψ θ , θ 0 ] w ϵ ( y ) d y = 0 \int_{\mathbb{R}^{n}}\Psi_{\theta,\theta_{0}}h_{\epsilon}\,dx=\Lambda_{\epsilon}[\Psi_{\theta,\theta_{0}}]
=\int_{\mathbb{R}^{n}}\Lambda[T_{y}\Psi_{\theta,\theta_{0}}]w_{\epsilon}(y)\,dy
=0 ∫ R n Ψ θ , θ 0 h ϵ d x = Λ ϵ [ Ψ θ , θ 0 ] = ∫ R n Λ [ T y Ψ θ , θ 0 ] w ϵ ( y ) d y = 0
By assumption, Ψ \Psi Ψ is bounded and nonconstant, so Theorem B.18 implies that h ϵ ≡ 0 h_{\epsilon}\equiv0 h ϵ ≡ 0 .
As a consequence, for all functions f ∈ C m ( R n ) f\in C^{m}(\mathbb{R}^{n}) f ∈ C m ( R n ) , Λ ϵ [ f ] = ∫ R n h ϵ f d x \Lambda_{\epsilon}[f]=\int_{\mathbb{R}^{n}}h_{\epsilon}f\,dx Λ ϵ [ f ] = ∫ R n h ϵ f d x vanishes and Λ [ f ] = lim ϵ → 0 Λ ϵ [ f ] = 0 \Lambda[f]=\lim_{\epsilon\to0}\Lambda_{\epsilon}[f]=0 Λ [ f ] = lim ϵ → 0 Λ ϵ [ f ] = 0 , which was ruled out by assumption.
Therefore, N n [ Ψ ] \mathcal{N}_{n}[\Psi] N n [ Ψ ] is uniformly m m m -dense on compacta in C m ( R n ) C^{m}(\mathbb{R}^{n}) C m ( R n ) . We must now complete our proof by showing that, for every function f ∈ C m ( R n ) f\in C^{m}(\mathbb{R}^{n}) f ∈ C m ( R n ) and ϵ > 0 \epsilon>0 ϵ > 0 , there is a function g ∈ N n [ Ψ ] g\in\mathcal{N}_{n}[\Psi] g ∈ N n [ Ψ ] such that ∥ f − g ∥ m , p , μ < ϵ \|f-g\|_{m,p,\mu}<\epsilon ∥ f − g ∥ m , p , μ < ϵ .
Let X X X be a compact set containing the support of μ \mu μ . Then we can establish a standard bound showing that a measure-based L p L^{p} L p norm is controlled by a uniform (supremum) norm over a compact set.
∥ f − g ∥ m , p , μ ≤ γ ∥ f − g ∥ m , μ , X \|f-g\|_{m,p,\mu}\leq\gamma\|f-g\|_{m,\mu,X} ∥ f − g ∥ m , p , μ ≤ γ ∥ f − g ∥ m , μ , X
where γ p = μ ( R n ) # { α : ∣ α ∣ ≤ m } \gamma^{p}=\mu(\mathbb{R}^{n})\#\{\alpha:|\alpha|\leq m\} γ p = μ ( R n ) # { α : ∣ α ∣ ≤ m } and # { α : ∣ α ∣ ≤ m } \#\{\alpha:|\alpha|\leq m\} # { α : ∣ α ∣ ≤ m } is the cardinality (total count) of all multi-indices of order at most m m m (i.e. the total number of partial derivatives of f − g f-g f − g from order 0 up to order m m m ).
If we choose g ∈ N n [ Ψ ] g\in\mathcal{N}_{n}[\Psi] g ∈ N n [ Ψ ] such that ∥ f − g ∥ m , μ , X < ϵ / γ \|f-g\|_{m,\mu,X}<\epsilon/\gamma ∥ f − g ∥ m , μ , X < ϵ / γ (something we can do because we just proved that N n [ Ψ ] \mathcal{N}_{n}[\Psi] N n [ Ψ ] is uniformly m m m -dense on compacta), then ∥ f − g ∥ m , p , μ < ϵ \|f-g\|_{m,p,\mu}<\epsilon ∥ f − g ∥ m , p , μ < ϵ . QED.
Theorem B.21. If Ψ ∈ C m ( R n ) \Psi\in C^{m}(\mathbb{R}^{n}) Ψ ∈ C m ( R n ) is nonconstant and all its derivatives up to order m m m are bounded, then N n [ Ψ ] \mathcal{N}_{n}[\Psi] N n [ Ψ ] is dense in C m , p ( μ ) C^{m,p}(\mu) C m , p ( μ ) for all finite measures μ \mu μ on R n \mathbb{R}^{n} R n .
Remark. The conditions imposed on Ψ \Psi Ψ are very general and they are satisfied by all common smooth squashing activation functions.
Proof. The proof of this theorem is parallel to the proof of Theorem B.20. Let C m , μ ( R n ) C^{m,\mu}(\mathbb{R}^{n}) C m , μ ( R n ) be the space of all functions f ∈ C m ( R n ) f\in C^{m}(\mathbb{R}^{n}) f ∈ C m ( R n ) that, with their derivatives up to order m m m , are bounded:
C m , μ ( R n ) = { f ∈ C m ( R n ) : ∥ D α f ∥ μ < ∞ , ∣ α ∣ ≤ m } C^{m,\mu}(\mathbb{R}^{n})=\{f\in C^{m}(\mathbb{R}^{n}):\|D^{\alpha}f\|_{\mu}<\infty,\ |\alpha|\leq m\} C m , μ ( R n ) = { f ∈ C m ( R n ) : ∥ D α f ∥ μ < ∞ , ∣ α ∣ ≤ m }
C m , μ ( R n ) C^{m,\mu}(\mathbb{R}^{n}) C m , μ ( R n ) is a dense subset of C m , p ( μ ) C^{m,p}(\mu) C m , p ( μ ) . Since Ψ ∈ C m ( R n ) \Psi\in C^{m}(\mathbb{R}^{n}) Ψ ∈ C m ( R n ) , then N n [ Ψ ] ⊂ C m , μ ( R n ) ⊂ C m , p ( μ ) \mathcal{N}_{n}[\Psi]\subset C^{m,\mu}(\mathbb{R}^{n})\subset C^{m,p}(\mu) N n [ Ψ ] ⊂ C m , μ ( R n ) ⊂ C m , p ( μ ) .
If N n [ Ψ ] \mathcal{N}_{n}[\Psi] N n [ Ψ ] is not dense in C m , p ( μ ) C^{m,p}(\mu) C m , p ( μ ) , the dual space argument yields the existence of functions g α ∈ L q ( μ ) g_{\alpha}\in L^{q}(\mu) g α ∈ L q ( μ ) , with ∣ α ∣ ≤ m |\alpha|\leq m ∣ α ∣ ≤ m and q = p / ( p − 1 ) q=p/(p-1) q = p / ( p − 1 ) , such that the functional Λ [ f ] = ∑ ∣ α ∣ ≤ m ∫ R n D α f g α d μ \Lambda[f]=\sum_{|\alpha|\leq m}\int_{\mathbb{R}^{n}}D^{\alpha}f g_{\alpha}\,d\mu Λ [ f ] = ∑ ∣ α ∣ ≤ m ∫ R n D α f g α d μ vanishes on N n [ Ψ ] \mathcal{N}_{n}[\Psi] N n [ Ψ ] , but not identically on C m , p ( μ ) C^{m,p}(\mu) C m , p ( μ ) .
To complete the proof, just follow the proof of Theorem B.20 with the finite signed measures σ α \sigma_{\alpha} σ α given by d σ α = g α d μ d\sigma_{\alpha}=g_{\alpha}d\mu d σ α = g α d μ and C m , μ ( R n ) C^{m,\mu}(\mathbb{R}^{n}) C m , μ ( R n ) instead of C m ( R n ) C^{m}(\mathbb{R}^{n}) C m ( R n ) .
Theorem II.2: Universal approximation theorem for derivatives. Let σ : R → R \sigma:\mathbb{R}\to\mathbb{R} σ : R → R be a bounded, nonconstant, and continuous activation function. Then, for any f ∈ C k ( K ) f\in C^{k}(K) f ∈ C k ( K ) , the space of k k k -times continuously differentiable functions on compact K ⊂ R n K\subset\mathbb{R}^{n} K ⊂ R n , there exists a neural network f ~ N \tilde{f}_{N} f ~ N such that
∥ f ( x ) − f ~ N ( x ) ∥ C k ( K ) < ϵ \|f(x)-\tilde{f}_{N}(x)\|_{C^{k}(K)}<\epsilon ∥ f ( x ) − f ~ N ( x ) ∥ C k ( K ) < ϵ for any $\epsilon>0.
Proof. The theorem is essentially an immediate consequence of Theorem B.20 once we identify C k ( K ) C^{k}(K) C k ( K ) with the corresponding C k , p ( μ ) C^{k,p}(\mu) C k , p ( μ ) space for a suitable measure μ \mu μ . A concise argument is as follows:
Let K ⊂ R n K\subset\mathbb{R}^{n} K ⊂ R n be compact and let f ∈ C k ( K ) f\in C^{k}(K) f ∈ C k ( K ) . Choose any finite measure μ \mu μ whose support is exactly K K K ; for example, one may take a finite Borel measure satisfying supp ( μ ) = K \operatorname{supp}(\mu)=K supp ( μ ) = K .
Theorem B.20 states that if Ψ ∈ C k ( R n ) \Psi\in C^{k}(\mathbb{R}^{n}) Ψ ∈ C k ( R n ) is bounded and nonconstant, then the neural-network class N n [ Ψ ] \mathcal{N}_{n}[\Psi] N n [ Ψ ] is uniformly k k k -dense on compacta in C k , p ( μ ) C^{k,p}(\mu) C k , p ( μ ) . By definition of uniform k k k -density, for every f ∈ C k , p f\in C^{k,p} f ∈ C k , p and every ϵ > 0 \epsilon>0 ϵ > 0 , there exists a neural network f N ∈ N n [ Ψ ] f_{N}\in\mathcal{N}_{n}[\Psi] f N ∈ N n [ Ψ ] such that
max ∣ α ∣ ≤ k sup x ∈ K ∣ D α f ( x ) − D α f ~ N ( x ) ∣ < ϵ \max_{|\alpha|\leq k}\sup_{x\in K}|D^{\alpha}f(x)-D^{\alpha}\tilde{f}_{N}(x)|<\epsilon ∣ α ∣ ≤ k max x ∈ K sup ∣ D α f ( x ) − D α f ~ N ( x ) ∣ < ϵ
Since supp ( μ ) = K \operatorname{supp}(\mu)=K supp ( μ ) = K , the space C k , p ( μ ) C^{k,p}(\mu) C k , p ( μ ) may be identified with C k ( K ) C^{k}(K) C k ( K ) , and the quantity on the left is precisely the C k ( K ) C^{k}(K) C k ( K ) -norm of the approximation error:
∥ f − f ~ N ∥ C k ( K ) = max ∣ α ∣ ≤ k sup x ∈ K ∣ D α f ( x ) − D α f ~ N ( x ) ∣ \|f-\tilde{f}_{N}\|_{C^{k}(K)}=\max_{|\alpha|\leq k}\sup_{x\in K}|D^{\alpha}f(x)-D^{\alpha}\tilde{f}_{N}(x)| ∥ f − f ~ N ∥ C k ( K ) = ∣ α ∣ ≤ k max x ∈ K sup ∣ D α f ( x ) − D α f ~ N ( x ) ∣
Therefore, for every f ∈ C k ( K ) f\in C^{k}(K) f ∈ C k ( K ) and every ϵ > 0 \epsilon>0 ϵ > 0 , there exists a neural network f ~ N \tilde{f}_{N} f ~ N such that
∥ f − f ~ N ∥ C k ( K ) < ϵ \|f-\tilde{f}_{N}\|_{C^{k}(K)}<\epsilon ∥ f − f ~ N ∥ C k ( K ) < ϵ
This is exactly the statement of the theorem, which is just a specialization of the density result to the case m = k m=k m = k and a compactly-supported measure whose support is K K K .
Therefore, a single-layer feedforward network can learn the true underlying function and its partial derivatives, making no “avoidable” systematic errors [13 ] .
Leshno's Universal Approximation Theorem
In 1993, Moshe Leshno, Vladimir Ya. Lin, Allan Pinkus, and Shimon Schocken generalized previous universal approximation theorems with the following result: “A standard multilayer feedforward network with a locally bounded piecewise continuous activation function can approximate any continuous function to any degree of accuracy if and only if the network’s activation function is not a polynomial” [28 ] . They also remarked that the threshold or bias is essential, without it their theorem does not hold.
Previous approximation results stated that, whenever activation functions satisfy some assumptions, which varied from paper to paper, multilayer neural networks are universal approximators. Leshno et al. [28 ] established the necessary and sufficient conditions for universal approximation.
The function f : R n → R f : \mathbb{R}^{n} \to\mathbb{R} f : R n → R computed by a single-hidden-layer neural network with n n n inputs, h h h hidden neurons, and a single linear output neuron is given by:
f ( x ) = ∑ j = 1 h α j σ ( ∑ i = 1 n w i j x i + b j ) = ∑ j = 1 h α j σ ( w j ⋅ x + b j ) f(x) = \sum_{j=1}^{h} \alpha_j \sigma\left(\sum_{i=1}^{n} w_{ij}x_i + b_j\right) = \sum_{j=1}^{h} \alpha_j \sigma\left(w_j \cdot x + b_j\right) f ( x ) = j = 1 ∑ h α j σ ( i = 1 ∑ n w ij x i + b j ) = j = 1 ∑ h α j σ ( w j ⋅ x + b j )
The parameters of the hidden layer consist of the number of hidden neurons (h h h ), a set of weights ({ w i j } \{w_{ij}\} { w ij } ), a set of biases/thresholds ({ b j } \{b_j\} { b j } ), and an activation function (σ : R → R \sigma: \mathbb{R} \to\mathbb{R} σ : R → R ). Its parameter space is Λ = ⟨ h , { w i j } , { b j } , σ ⟩ \Lambda= \langle h, \{w_{ij}\}, \{b_j\}, \sigma\rangle Λ = ⟨ h , { w ij } , { b j } , σ ⟩ . A particular configuration of the hidden layer is described by the particular quadruple ω ∈ Λ \omega\in\Lambda ω ∈ Λ . The hidden layer characterized by ω \omega ω is N ω \mathcal{N}_{\omega} N ω , the function computed by N ω \mathcal{N}_{\omega} N ω is f ω : R → R n f_{\omega} : \mathbb{R} \to\mathbb{R}^{n} f ω : R → R n , and the family of such functions is F = { f ω ∣ ω ∈ Λ } \mathcal{F} = \{f_{\omega}|\omega\in\Lambda\} F = { f ω ∣ ω ∈ Λ } .
Discovering all the functions that can be approximated by networks with a hidden layer of the form N ω \mathcal{N}_{\omega} N ω is equivalent to characterizing the closure F ‾ \overline{\mathcal{F}} F of F \mathcal{F} F . This closure is the complete set of functions that a neural network can either produce directly or approximate with arbitrarily small error. Whether a target function belongs to the closure depends entirely on the metric (the rule used to measure the distance or error between two functions from R n \mathbb{R}^{n} R n to R \mathbb{R} R ).
Definition B.12: Metric. A metric on a set S S S is a function d : S × S → R d : S \times S \to\mathbb{R} d : S × S → R that satisfies the following four properties:
Non-negativity: d ( s , t ) ≥ 0 d(s,t) \ge0 d ( s , t ) ≥ 0 .
Identity of indiscernibles: s = t ⇔ d ( s , t ) = 0 s = t \Leftrightarrow d(s,t) = 0 s = t ⇔ d ( s , t ) = 0 .
Symmetry: d ( s , t ) = d ( t , s ) d(s,t) = d(t,s) d ( s , t ) = d ( t , s ) .
Triangle inequality: d ( s , t ) ≤ d ( s , u ) + d ( u , t ) d(s,t) \le d(s,u) + d(u,t) d ( s , t ) ≤ d ( s , u ) + d ( u , t ) .
Definition B.13: Closure. The closure of a set S S S of a metric space ( X , d ) (X,d) ( X , d ) is defined as
S ‾ = { r ∣ ∀ ε > 0 , ∃ s ∈ S , d ( s , t ) < ε } \overline{S} = \{r \mid\forall\varepsilon> 0, \exists s \in S, d(s,t) < \varepsilon\} S = { r ∣ ∀ ε > 0 , ∃ s ∈ S , d ( s , t ) < ε }
A function f ∈ L ∞ ( Ω ) f \in L^{\infty}(\Omega) f ∈ L ∞ ( Ω ) is essentially bounded on a measurable set Ω \Omega Ω with respect to the essential supremum norm ∥ f ∥ L ∞ ( Ω ) \|f\|_{L^{\infty}(\Omega)} ∥ f ∥ L ∞ ( Ω ) . A function f ∈ L l o c ∞ ( Ω ) f \in L^{\infty}_{\mathrm{loc}}(\Omega) f ∈ L loc ∞ ( Ω ) is locally essentially bounded on a domain Ω \Omega Ω (i.e. an open set in R n \mathbb{R}^{n} R n ) if, for every compact set K ⊂ Ω K \subset\Omega K ⊂ Ω , f ∈ L ∞ ( K ) f \in L^{\infty}(K) f ∈ L ∞ ( K ) .
Definition B.14: A set F F F of functions in L l o c ∞ ( Ω ) L^{\infty}_{\mathrm{loc}}(\Omega) L loc ∞ ( Ω ) is dense in C ( R n ) C(\mathbb{R}^{n}) C ( R n ) if, for every function f ∈ C ( R n ) f \in C(\mathbb{R}^{n}) f ∈ C ( R n ) and every compact set K ⊂ R n K \subset\mathbb{R}^{n} K ⊂ R n , there exists a sequence of functions f j ∈ F f_j \in F f j ∈ F such that
A set of functions F F F is dense in C ( R n ) C(\mathbb{R}^{n}) C ( R n ) if, for every continuous function f ∈ C ( R n ) f \in C(\mathbb{R}^{n}) f ∈ C ( R n ) there is a function f ~ ∈ F \tilde{f} \in F f ~ ∈ F that is a good approximation to f f f .
Let M M M be the set of functions in L l o c ∞ ( R ) L^{\infty}_{\mathrm{loc}}(\mathbb{R}) L loc ∞ ( R ) whose closure of the set of points of discontinuity is of zero Lebesgue measure (i.e. they are Riemann integrable on every compact interval). For any σ ∈ M \sigma\in M σ ∈ M , interval [ a , b ] [a,b] [ a , b ] , and ε > 0 \varepsilon> 0 ε > 0 , there is a finite number of intervals, { U i } \{U_i\} { U i } , whose union U U U is of measure δ \delta δ , such that σ \sigma σ is uniformly continuous on [ a , b ] ∖ U [a,b] \setminus U [ a , b ] ∖ U .
Theorem B.22: For σ ∈ M \sigma\in M σ ∈ M , let the span 18 \operatorname{span}^{18} span 18
Σ n = span { σ ( w ⋅ x + b ) : w ∈ R n , b ∈ R } \Sigma_n = \operatorname{span}\{\sigma(w \cdot x + b) : w \in\mathbb{R}^{n}, b \in\mathbb{R}\} Σ n = span { σ ( w ⋅ x + b ) : w ∈ R n , b ∈ R } Σ n \Sigma_n Σ n is dense in C ( R n ) C(\mathbb{R}^{n}) C ( R n ) if and only if σ \sigma σ is not an algebraic polynomial.
Proof. [28 ] If σ \sigma σ is a polynomial, then Σ n \Sigma_n Σ n is not dense in C ( R n ) C(\mathbb{R}^{n}) C ( R n ) .
If σ \sigma σ is a polynomial of degree k k k , then σ ( w ⋅ x + b ) \sigma(w \cdot x + b) σ ( w ⋅ x + b ) is a polynomial of degree k k k . Then, Σ n \Sigma_n Σ n is exactly the set of algebraic polynomials of degree at most k k k . Therefore Σ n \Sigma_n Σ n cannot be dense in C ( R n ) C(\mathbb{R}^{n}) C ( R n ) .19
Hence, we will always assume that σ \sigma σ is not a polynomial. Now, we can reduce our problem to Σ 1 \Sigma_1 Σ 1 : If Σ 1 \Sigma_1 Σ 1 is dense in C ( R ) C(\mathbb{R}) C ( R ) , then Σ n \Sigma_n Σ n is dense in C ( R n ) C(\mathbb{R}^{n}) C ( R n ) .
The space V = span { f ( a ⋅ x ) ∣ a ∈ R n , f ∈ C ( R n ) } V = \operatorname{span}\{f(a \cdot x)|a \in\mathbb{R}^{n}, f \in C(\mathbb{R}^{n})\} V = span { f ( a ⋅ x ) ∣ a ∈ R n , f ∈ C ( R n )} is dense in C ( R n ) C(\mathbb{R}^{n}) C ( R n ) . For any compact set K ⊂ R n K \subset\mathbb{R}^{n} K ⊂ R n , V V V is dense in C ( K ) C(K) C ( K ) . Let g ∈ C ( R n ) g \in C(\mathbb{R}^{n}) g ∈ C ( R n ) . For ε > 0 \varepsilon> 0 ε > 0 , there exists f i ∈ C ( R ) f_i \in C(\mathbb{R}) f i ∈ C ( R ) such that
∣ g ( x ) − ∑ k = i k f i ( a i ⋅ x ) ∣ < ε / 2 \left|g(x) - \sum_{k=i}^{k} f_i(a_i \cdot x)\right| < \varepsilon/2 g ( x ) − k = i ∑ k f i ( a i ⋅ x ) < ε /2
for all x ∈ K x \in K x ∈ K . where a i ∈ R n a_i \in\mathbb{R}^{n} a i ∈ R n . { a i : x ∈ K } ⊆ [ α i , β i ] \{a_i : x \in K\} \subseteq[\alpha_i,\beta_i] { a i : x ∈ K } ⊆ [ α i , β i ] for some finite interval [ α i , β i ] [\alpha_i,\beta_i] [ α i , β i ] . Given that Σ 1 \Sigma_1 Σ 1 is dense in [ α i , β i ] [\alpha_i,\beta_i] [ α i , β i ] , there exist constants c i j , w i j , b i j c_{ij},w_{ij},b_{ij} c ij , w ij , b ij such that
∣ f i ( x ) − ∑ j = i m i c i j σ ( w i j y + b i j ) ∣ < ϵ / 2 k \left|f_i(x)-\sum_{j=i}^{m_i}c_{ij}\sigma(w_{ij}y+b_{ij})\right|<\epsilon/2k f i ( x ) − j = i ∑ m i c ij σ ( w ij y + b ij ) < ϵ /2 k
for all y ∈ [ α i , β i ] y\in[\alpha_i,\beta_i] y ∈ [ α i , β i ] . Therefore,
∣ f ( x ) − ∑ k = i k ∑ j = i m i c i j σ ( w i j ( a i ⋅ x ) + b i j ) ∣ < ϵ \left|f(x)-\sum_{k=i}^{k}\sum_{j=i}^{m_i}c_{ij}\sigma(w_{ij}(a_i\cdot x)+b_{ij})\right|<\epsilon f ( x ) − k = i ∑ k j = i ∑ m i c ij σ ( w ij ( a i ⋅ x ) + b ij ) < ϵ
for all x ∈ K x\in K x ∈ K . Therefore, Σ 1 \Sigma_1 Σ 1 is dense in C ( R ) C(\mathbb{R}) C ( R ) implies that Σ n \Sigma_n Σ n is dense in C ( R n ) C(\mathbb{R}^n) C ( R n ) .
Let us now complete the proof using the following steps:
If σ ∈ C ∞ ( R ) \sigma\in C^\infty(\mathbb{R}) σ ∈ C ∞ ( R ) , then Σ 1 \Sigma_1 Σ 1 is dense in C ( R ) C(\mathbb{R}) C ( R ) .
For each φ ∈ C 0 ( R ) \varphi\in C_0(\mathbb{R}) φ ∈ C 0 ( R ) , σ ∗ φ ∈ Σ 1 ‾ \sigma*\varphi\in\overline{\Sigma_1} σ ∗ φ ∈ Σ 1
If, for some φ ∈ C 0 ∞ ( R ) \varphi\in C^\infty_0(\mathbb{R}) φ ∈ C 0 ∞ ( R ) , σ ∗ φ \sigma*\varphi σ ∗ φ is not a polynomial, then Σ 1 \Sigma_1 Σ 1 is dense in C ( R ) C(\mathbb{R}) C ( R ) .
If, for all φ ∈ C 0 ∞ ( R ) \varphi\in C^\infty_0(\mathbb{R}) φ ∈ C 0 ∞ ( R ) , σ ∗ φ \sigma*\varphi σ ∗ φ is a polynomial, then ∃ d ∈ N \exists d\in\mathbb{N} ∃ d ∈ N such that σ ∗ φ \sigma*\varphi σ ∗ φ is a polynomial of degree at most d d d for all φ ∈ C 0 ∞ ( R ) \varphi\in C^\infty_0(\mathbb{R}) φ ∈ C 0 ∞ ( R ) .
If σ ∗ φ \sigma*\varphi σ ∗ φ is a polynomial of degree at most d d d for all φ ∈ C 0 ∞ ( R ) \varphi\in C^\infty_0(\mathbb{R}) φ ∈ C 0 ∞ ( R ) , then σ \sigma σ is a polynomial of degree at most d d d (a.e.).
Step 1. If σ ∈ C ∞ ( R ) \sigma\in C^\infty(\mathbb{R}) σ ∈ C ∞ ( R ) , then Σ 1 \Sigma_1 Σ 1 is dense in C ( R ) C(\mathbb{R}) C ( R ) .
If σ ∈ C ∞ ( R ) \sigma\in C^\infty(\mathbb{R}) σ ∈ C ∞ ( R ) , ( σ ( ( w + h ) x + b ) − σ ( w x + b ) ) / h ∈ Σ 1 (\sigma((w+h)x+b)-\sigma(wx+b))/h\in\Sigma_1 ( σ (( w + h ) x + b ) − σ ( w x + b )) / h ∈ Σ 1 and, therefore, ( d / d w ) σ ( w x + b ) ∈ Σ 1 ‾ (d/dw)\sigma(wx+b)\in\overline{\Sigma_1} ( d / d w ) σ ( w x + b ) ∈ Σ 1 for every h ≠ 0 h\ne0 h = 0 .
By the same argument, ( d k / d w k ) σ ( w x + b ) ∈ Σ 1 ‾ (d^k/dw^k)\sigma(wx+b)\in\overline{\Sigma_1} ( d k / d w k ) σ ( w x + b ) ∈ Σ 1 for all k ∈ N k\in\mathbb{N} k ∈ N .
( d k / d w k ) σ ( w x + b ) = x k σ ( k ) ( w x + b ) (d^k/dw^k)\sigma(wx+b)=x^k\sigma^{(k)}(wx+b) ( d k / d w k ) σ ( w x + b ) = x k σ ( k ) ( w x + b ) , where σ ( k ) \sigma^{(k)} σ ( k ) is the k k k -th derivative of σ \sigma σ . Since σ \sigma σ is not a polynomial, there exists θ k ∈ R \theta_k\in\mathbb{R} θ k ∈ R such that σ ( k ) ( θ k ) ≠ 0 \sigma^{(k)}(\theta_k)\ne0 σ ( k ) ( θ k ) = 0 . Hence,
x k σ ( k ) ( θ k ) = d k d w k σ ( w x + b ) ∣ w = 0 , b = θ k ∈ Σ 1 ‾ x^k\sigma^{(k)}(\theta_k)=\left.\frac{d^k}{dw^k}\sigma(wx+b)\right|_{w=0,b=\theta_k}\in\overline{\Sigma_1} x k σ ( k ) ( θ k ) = d w k d k σ ( w x + b ) w = 0 , b = θ k ∈ Σ 1
This implies that Σ 1 ‾ \overline{\Sigma_1} Σ 1 contains all polynomials.20 By Weierstrass Approximation Theorem,21 it follows that Σ 1 ‾ \overline{\Sigma_1} Σ 1 contains C ( K ) C(K) C ( K ) for all compact sets K ⊂ R K\subset\mathbb{R} K ⊂ R . In other words, Σ 1 \Sigma_1 Σ 1 is dense in C ( R ) C(\mathbb{R}) C ( R ) .
x k = 1 σ ( k ) ( θ k ) ( x k σ ( k ) ( θ k ) ) ∈ Σ 1 ‾ x^k=\frac{1}{\sigma^{(k)}(\theta_k)}\left(x^k\sigma^{(k)}(\theta_k)\right)\in\overline{\Sigma_1} x k = σ ( k ) ( θ k ) 1 ( x k σ ( k ) ( θ k ) ) ∈ Σ 1
Since Σ 1 ‾ \overline{\Sigma_1} Σ 1 contains 1 , x , x 2 , x 3 , … 1,x,x^2,x^3,\ldots 1 , x , x 2 , x 3 , … for all k ≥ 0 k\ge0 k ≥ 0 , it contains every linear combination ∑ k = 0 m a k x k \sum_{k=0}^{m}a_kx^k ∑ k = 0 m a k x k , i.e. all polynomials.
Step 2. For each φ ∈ C 0 ( R ) \varphi\in C_0(\mathbb{R}) φ ∈ C 0 ( R ) , σ ∗ φ ∈ Σ 1 ‾ \sigma*\varphi\in\overline{\Sigma_1} σ ∗ φ ∈ Σ 1
Assume that supp φ ⊆ [ − α , α ] \operatorname{supp}\varphi\subseteq[-\alpha,\alpha] supp φ ⊆ [ − α , α ] . We will prove that we can uniformly approximate σ ∗ φ \sigma*\varphi σ ∗ φ from Σ 1 ‾ \overline{\Sigma_1} Σ 1 on [ − α , α ] [-\alpha,\alpha] [ − α , α ] :
∑ i = 1 m σ ( x − y i ) φ ( y i ) Δ y i \sum_{i=1}^{m}\sigma(x-y_i)\varphi(y_i)\Delta y_i i = 1 ∑ m σ ( x − y i ) φ ( y i ) Δ y i
uniformly converges to σ ∗ φ \sigma*\varphi σ ∗ φ on [ − α , α ] [-\alpha,\alpha] [ − α , α ] , where y i = − α + 2 i α / m y_i=-\alpha+2i\alpha/m y i = − α + 2 i α / m and Δ y i = 2 α / m \Delta y_i=2\alpha/m Δ y i = 2 α / m with i = 1 … m i=1\ldots m i = 1 … m .
Given ϵ > 0 \epsilon>0 ϵ > 0 , we choose δ > 0 \delta>0 δ > 0 so that
10 δ ∥ σ ∥ L ∞ [ − 2 α , 2 α ] ∥ φ ∥ L ∞ ≤ ϵ . 10\delta\|\sigma\|_{L^\infty[-2\alpha,2\alpha]}\|\varphi\|_{L^\infty}\le\epsilon. 10 δ ∥ σ ∥ L ∞ [ − 2 α , 2 α ] ∥ φ ∥ L ∞ ≤ ϵ .
For this δ > 0 \delta>0 δ > 0 , there is a finite number r ( δ ) r(\delta) r ( δ ) of intervals, the measure of whose union U U U is δ \delta δ , such that σ \sigma σ is uniformly continuous on [ − 2 α , 2 α ] [-2\alpha,2\alpha] [ − 2 α , 2 α ] . We choose m m m sufficiently large, so that m δ < α r ( δ ) m\delta<\alpha r(\delta) m δ < α r ( δ ) .
If ∣ s − t ∣ ≤ 2 α / m |s-t|\le2\alpha/m ∣ s − t ∣ ≤ 2 α / m , by the uniform continuity of φ \varphi φ :
∣ φ ( s ) − φ ( t ) ∣ ≤ ϵ 2 α ∥ σ ∥ L ∞ [ − 2 α , 2 α ] |\varphi(s)-\varphi(t)|\le\frac{\epsilon}{2\alpha\|\sigma\|_{L^\infty[-2\alpha,2\alpha]}} ∣ φ ( s ) − φ ( t ) ∣ ≤ 2 α ∥ σ ∥ L ∞ [ − 2 α , 2 α ] ϵ
If s , t ∈ [ − 2 α , 2 α ] ∖ U s,t\in[-2\alpha,2\alpha]\setminus U s , t ∈ [ − 2 α , 2 α ] ∖ U and ∣ s − t ∣ ≤ 2 α / m |s-t|\le2\alpha/m ∣ s − t ∣ ≤ 2 α / m , by the uniform continuity of σ \sigma σ on [ − 2 α , 2 α ] ∖ U [-2\alpha,2\alpha]\setminus U [ − 2 α , 2 α ] ∖ U :
∣ σ ( s ) − σ ( t ) ∣ ≤ ϵ ∥ φ ∥ L 1 |\sigma(s)-\sigma(t)|\le\frac{\epsilon}{\|\varphi\|_{L^1}} ∣ σ ( s ) − σ ( t ) ∣ ≤ ∥ φ ∥ L 1 ϵ
For x ∈ [ − α , α ] x\in[-\alpha,\alpha] x ∈ [ − α , α ] , set Δ i = [ y i − 1 , y i ] \Delta_i=[y_{i-1},y_i] Δ i = [ y i − 1 , y i ] with y 0 = − α y_0=-\alpha y 0 = − α . Since supp φ ⊆ [ − α , α ] \operatorname{supp}\varphi\subseteq[-\alpha,\alpha] supp φ ⊆ [ − α , α ] :
∣ ∫ σ ( x − y ) φ ( y ) d y − ∑ i = 1 m ∫ Δ i σ ( x − y i ) φ ( y ) d y ∣ ≤ ∑ i = 1 m ∫ Δ i ∣ σ ( x − y ) − σ ( x − y i ) ∣ ∣ φ ( y ) ∣ d y \left|\int\sigma(x-y)\varphi(y)\,dy-\sum_{i=1}^{m}\int_{\Delta_i}\sigma(x-y_i)\varphi(y)\,dy\right|
\le\sum_{i=1}^{m}\int_{\Delta_i}|\sigma(x-y)-\sigma(x-y_i)||\varphi(y)|\,dy ∫ σ ( x − y ) φ ( y ) d y − i = 1 ∑ m ∫ Δ i σ ( x − y i ) φ ( y ) d y ≤ i = 1 ∑ m ∫ Δ i ∣ σ ( x − y ) − σ ( x − y i ) ∣∣ φ ( y ) ∣ d y
If x − Δ i x-\Delta_i x − Δ i does not intersect U U U , from our inequality on ∣ σ ( s ) − σ ( t ) ∣ |\sigma(s)-\sigma(t)| ∣ σ ( s ) − σ ( t ) ∣ :
∫ Δ i ∣ σ ( x − y ) − σ ( x − y i ) ∣ ∣ φ ( y ) ∣ d y ≤ ϵ ∥ φ ∥ L 1 ∫ Δ i ∣ φ ( y ) ∣ d y \int_{\Delta_i}|\sigma(x-y)-\sigma(x-y_i)||\varphi(y)|\,dy
\le\frac{\epsilon}{\|\varphi\|_{L^1}}\int_{\Delta_i}|\varphi(y)|\,dy ∫ Δ i ∣ σ ( x − y ) − σ ( x − y i ) ∣∣ φ ( y ) ∣ d y ≤ ∥ φ ∥ L 1 ϵ ∫ Δ i ∣ φ ( y ) ∣ d y
When ( x − Δ i ) ∩ U ≠ ∅ (x-\Delta_i)\cap U\ne\varnothing ( x − Δ i ) ∩ U = ∅ , we denote such intervals as Δ ~ i \widetilde{\Delta}_i Δ i . U U U has measure δ \delta δ and is composed of r ( δ ) r(\delta) r ( δ ) intervals. The total length of the Δ ~ i \widetilde{\Delta}_i Δ i intervals is, at most, δ + ( 4 α / m ) r ( δ ) \delta+(4\alpha/m)r(\delta) δ + ( 4 α / m ) r ( δ ) . By our choice of m m m , δ + ( 4 α / m ) r ( δ ) ≤ 5 δ \delta+(4\alpha/m)r(\delta)\le5\delta δ + ( 4 α / m ) r ( δ ) ≤ 5 δ . From our choice of δ \delta δ :
∫ Δ ~ i ∣ σ ( x − y ) − σ ( x − y i ) ∣ ∣ φ ( y ) ∣ d y ≤ 2 ∥ σ ∥ L ∞ [ − 2 α , 2 α ] ∥ φ ∥ L ∞ 5 δ ≤ ϵ \int_{\widetilde{\Delta}_i}|\sigma(x-y)-\sigma(x-y_i)||\varphi(y)|\,dy
\le2\|\sigma\|_{L^\infty[-2\alpha,2\alpha]}\|\varphi\|_{L^\infty}5\delta\le\epsilon ∫ Δ i ∣ σ ( x − y ) − σ ( x − y i ) ∣∣ φ ( y ) ∣ d y ≤ 2∥ σ ∥ L ∞ [ − 2 α , 2 α ] ∥ φ ∥ L ∞ 5 δ ≤ ϵ
Finally,
∣ ∑ i = 1 m ∫ Δ i σ ( x − y i ) φ ( y ) d y − ∑ i = 1 m σ ( x − y i ) φ ( y i ) Δ y i ∣ = ∣ ∑ i = 1 m ∫ Δ i σ ( x − y i ) [ φ ( y ) − φ ( y i ) ] d y ∣ ≤ ∑ i = 1 m ∫ Δ i ∣ σ ( x − y i ) ∣ ∣ φ ( y ) − φ ( y i ) ∣ d y ≤ ∑ i = 1 m ∫ Δ i ∣ σ ( x − y i ) ∣ d y [ ϵ 2 α ∥ σ ∥ L ∞ [ − 2 α , 2 α ] ] ≤ ϵ \begin{aligned}
\left|\sum_{i=1}^{m}\int_{\Delta_i}\sigma(x-y_i)\varphi(y)\,dy-\sum_{i=1}^{m}\sigma(x-y_i)\varphi(y_i)\Delta y_i\right| \\
&=\left|\sum_{i=1}^{m}\int_{\Delta_i}\sigma(x-y_i)[\varphi(y)-\varphi(y_i)]\,dy\right| \\
&\le\sum_{i=1}^{m}\int_{\Delta_i}|\sigma(x-y_i)||\varphi(y)-\varphi(y_i)|\,dy \\
&\le\sum_{i=1}^{m}\int_{\Delta_i}|\sigma(x-y_i)|\,dy\left[\frac{\epsilon}{2\alpha\|\sigma\|_{L^\infty[-2\alpha,2\alpha]}}\right]\le\epsilon
\end{aligned} i = 1 ∑ m ∫ Δ i σ ( x − y i ) φ ( y ) d y − i = 1 ∑ m σ ( x − y i ) φ ( y i ) Δ y i = i = 1 ∑ m ∫ Δ i σ ( x − y i ) [ φ ( y ) − φ ( y i )] d y ≤ i = 1 ∑ m ∫ Δ i ∣ σ ( x − y i ) ∣∣ φ ( y ) − φ ( y i ) ∣ d y ≤ i = 1 ∑ m ∫ Δ i ∣ σ ( x − y i ) ∣ d y [ 2 α ∥ σ ∥ L ∞ [ − 2 α , 2 α ] ϵ ] ≤ ϵ
where the last inequality comes from our expression on ∣ φ ( s ) − φ ( t ) ∣ |\varphi(s)-\varphi(t)| ∣ φ ( s ) − φ ( t ) ∣ .
Therefore, we obtain our desired approximation of σ ∗ φ \sigma\ast\varphi σ ∗ φ :
∣ ∫ σ ( x − y ) φ ( y ) d y − ∑ i = 1 m σ ( x − y i ) φ ( y i ) Δ y i ∣ ≤ 3 ϵ \left|\int\sigma(x-y)\varphi(y)\,dy-\sum_{i=1}^{m}\sigma(x-y_i)\varphi(y_i)\Delta y_i\right|\le3\epsilon ∫ σ ( x − y ) φ ( y ) d y − i = 1 ∑ m σ ( x − y i ) φ ( y i ) Δ y i ≤ 3 ϵ
Step 3. If, for some φ ∈ C 0 ∞ ( R ) \varphi\in C^\infty_0(\mathbb{R}) φ ∈ C 0 ∞ ( R ) , σ ∗ φ \sigma\ast\varphi σ ∗ φ is not a polynomial, then Σ 1 \Sigma_1 Σ 1 is dense in C ( R ) C(\mathbb{R}) C ( R ) .
From step 2 above, σ ∗ φ ∈ Σ 1 ‾ \sigma\ast\varphi\in\overline{\Sigma_1} σ ∗ φ ∈ Σ 1 . Then ( σ ∗ φ ) ( w x + b ) ∈ Σ 1 (\sigma\ast\varphi)(wx+b)\in\Sigma_1 ( σ ∗ φ ) ( w x + b ) ∈ Σ 1 . For any φ ∈ C 0 ∞ ( R ) \varphi\in C^\infty_0(\mathbb{R}) φ ∈ C 0 ∞ ( R ) , σ ∗ φ ∈ C ∞ ( R ) \sigma\ast\varphi\in C^\infty(\mathbb{R}) σ ∗ φ ∈ C ∞ ( R ) . From step 1, if σ ∗ φ \sigma\ast\varphi σ ∗ φ is not a polynomial, Σ 1 \Sigma_1 Σ 1 is dense in C ( R ) C(\mathbb{R}) C ( R ) .
We now assume that σ ∗ φ \sigma\ast\varphi σ ∗ φ is a polynomial for all φ ∈ C 0 ∞ ( R ) \varphi\in C^\infty_0(\mathbb{R}) φ ∈ C 0 ∞ ( R ) . We will conclude from this fact that σ \sigma σ is itself a polynomial (a.e.).
Step 4. If, for all φ ∈ C 0 ∞ ( R ) \varphi\in C^\infty_0(\mathbb{R}) φ ∈ C 0 ∞ ( R ) , σ ∗ φ \sigma\ast\varphi σ ∗ φ is a polynomial, then ∃ d ∈ N \exists d\in\mathbb{N} ∃ d ∈ N such that σ ∗ φ \sigma\ast\varphi σ ∗ φ is a polynomial of degree at most d d d for all φ ∈ C 0 ∞ ( R ) \varphi\in C^\infty_0(\mathbb{R}) φ ∈ C 0 ∞ ( R ) .
For any a < b a<b a < b , the set C 0 ∞ [ a , b ] C^\infty_0[a,b] C 0 ∞ [ a , b ] is the set of all C ∞ C^\infty C ∞ functions with support in [ a , b ] [a,b] [ a , b ] .
In the case of φ ∈ C 0 ∞ [ a , b ] \varphi\in C^\infty_0[a,b] φ ∈ C 0 ∞ [ a , b ] , we define a metric ρ \rho ρ on C 0 ∞ [ a , b ] C^\infty_0[a,b] C 0 ∞ [ a , b ] as
ρ ( φ 1 , φ 2 ) = ∑ n = 0 ∞ 2 − n ∥ φ 1 − φ 2 ∥ n 1 + ∥ φ 1 − φ 2 ∥ n \rho(\varphi_1,\varphi_2)=\sum_{n=0}^{\infty}2^{-n}\frac{\|\varphi_1-\varphi_2\|_n}{1+\|\varphi_1-\varphi_2\|_n} ρ ( φ 1 , φ 2 ) = n = 0 ∑ ∞ 2 − n 1 + ∥ φ 1 − φ 2 ∥ n ∥ φ 1 − φ 2 ∥ n
where ∥ φ ∥ n = ∑ j = 0 n sup x ∈ [ a , b ] ∣ φ ( j ) ( x ) ∣ \|\varphi\|_n=\sum_{j=0}^{n}\sup_{x\in[a,b]}|\varphi^{(j)}(x)| ∥ φ ∥ n = ∑ j = 0 n sup x ∈ [ a , b ] ∣ φ ( j ) ( x ) ∣ .
C 0 ∞ [ a , b ] C^\infty_0[a,b] C 0 ∞ [ a , b ] with the metric ρ \rho ρ is a Fréchet space (a complete metric vector space).
By our assumption, σ ∗ φ \sigma\ast\varphi σ ∗ φ is a polynomial for every φ ∈ C 0 ∞ [ a , b ] \varphi\in C^\infty_0[a,b] φ ∈ C 0 ∞ [ a , b ] . We define the space
V k = { φ ∈ C 0 ∞ [ a , b ] ∣ degree ( σ ∗ φ ) ≤ k } V_k=\{\varphi\in C^\infty_0[a,b]\mid\operatorname{degree}(\sigma\ast\varphi)\le k\} V k = { φ ∈ C 0 ∞ [ a , b ] ∣ degree ( σ ∗ φ ) ≤ k }
V k V_k V k is a closed subspace, V k ⊆ V k + 1 V_k\subseteq V_{k+1} V k ⊆ V k + 1 , and ⋃ k = 0 ∞ V k = C 0 ∞ [ a , b ] \bigcup_{k=0}^{\infty}V_k=C^\infty_0[a,b] ⋃ k = 0 ∞ V k = C 0 ∞ [ a , b ] .
Since C 0 ∞ [ a , b ] C^\infty_0[a,b] C 0 ∞ [ a , b ] is a complete metric space, by Baire’s Category Theorem,22 there is an integer d d d such that V d = C 0 ∞ [ a , b ] V_d=C^\infty_0[a,b] V d = C 0 ∞ [ a , b ] . This completes the proof for φ ∈ C 0 ∞ [ a , b ] \varphi\in C^\infty_0[a,b] φ ∈ C 0 ∞ [ a , b ] .
For the general case, the degree d d d does not depend on [ a , b ] [a,b] [ a , b ] yet, by translation, d d d depends at most of the length of the interval. For φ ∈ C 0 ∞ [ A , B ] \varphi\in C^\infty_0[A,B] φ ∈ C 0 ∞ [ A , B ] , we can find φ i ∈ C 0 ∞ [ a i , b i ] \varphi_i\in C^\infty_0[a_i,b_i] φ i ∈ C 0 ∞ [ a i , b i ] such that [ A , B ] ⊆ ⋃ i = 1 k [ a i , b i ] [A,B]\subseteq\bigcup_{i=1}^{k}[a_i,b_i] [ A , B ] ⊆ ⋃ i = 1 k [ a i , b i ] , b i − a i = b − a b_i-a_i=b-a b i − a i = b − a , and φ = ∑ i = 1 k φ i \varphi=\sum_{i=1}^{k}\varphi_i φ = ∑ i = 1 k φ i . Then σ ∗ φ = ∑ i = 1 k σ ∗ φ i \sigma\ast\varphi=\sum_{i=1}^{k}\sigma\ast\varphi_i σ ∗ φ = ∑ i = 1 k σ ∗ φ i and every σ ∗ φ i \sigma\ast\varphi_i σ ∗ φ i is a polynomial of degree ≤ d \le d ≤ d , so degree ( σ ∗ φ ) ≤ d \operatorname{degree}(\sigma\ast\varphi)\le d degree ( σ ∗ φ ) ≤ d .
Step 5. If σ ∗ φ \sigma\ast\varphi σ ∗ φ is a polynomial of degree at most d d d for all φ ∈ C 0 ∞ ( R ) \varphi\in C^\infty_0(\mathbb{R}) φ ∈ C 0 ∞ ( R ) , then σ \sigma σ is a polynomial of degree at most d d d almost everywhere (a.e.).
From step 4, for all φ ∈ C 0 ∞ ( R n ) \varphi\in C^\infty_0(\mathbb{R}^n) φ ∈ C 0 ∞ ( R n ) ,
σ ∗ φ ( d + 1 ) = ∫ σ ( x − y ) φ ( d + 1 ) ( y ) d y = 0 \sigma\ast\varphi^{(d+1)}=\int\sigma(x-y)\varphi^{(d+1)}(y)\,dy=0 σ ∗ φ ( d + 1 ) = ∫ σ ( x − y ) φ ( d + 1 ) ( y ) d y = 0
From standard results in Distribution Theory, if smoothing a function σ \sigma σ always produces a polynomial of degree at most d d d , then σ \sigma σ itself must already be a polynomial of degree at most d d d almost everywhere.23 Therefore, σ \sigma σ is itself a polynomial of degree at most d d d (a.e.).
In neural network theory, this is the key step that extends Universal Approximation Theorems from smooth activations to non-smooth activations like ReLU or step functions. Why? Because, when an activation function σ \sigma σ is not differentiable, we cannot take its derivatives directly. Instead, we convolve σ \sigma σ with a smooth, compactly supported test function φ ∈ C 0 ∞ ( R ) \varphi\in C^\infty_0(\mathbb{R}) φ ∈ C 0 ∞ ( R ) . The convolution σ ∗ φ \sigma\ast\varphi σ ∗ φ averages σ \sigma σ against φ \varphi φ .
If σ \sigma σ is not a polynomial, we want to guarantee that σ ∗ φ \sigma\ast\varphi σ ∗ φ is not a polynomial either. Proving the contrapositive (that σ ∗ φ \sigma* \varphi σ ∗ φ being a polynomial forces σ \sigma σ to be a polynomial) closes the argument and completes our proof.
Remark. As mentioned in (28), an alternative proof of Theorem B.22 can be based on Schwartz’s theory of mean-periodic functions. That theory establishes that, if σ ∈ C ( R ) \sigma\in C(\mathbb{R}) σ ∈ C ( R ) is not a polynomial, then the span of { σ ( x + θ ) : θ ∈ R } \{\sigma(x+\theta):\theta\in\mathbb{R}\} { σ ( x + θ ) : θ ∈ R } contains in its closure a function of the form e λ x cos ν x e^{\lambda x}\cos\nu x e λ x cos ν x for some real ( λ , ν ) ≠ ( 0 , 0 ) (\lambda,\nu)\ne(0,0) ( λ , ν ) = ( 0 , 0 ) . Since those functions are in C ∞ ( R ) C^{\infty}(\mathbb{R}) C ∞ ( R ) , Theorem B.22 follows.
Theorem II.3: Leshno’s theorem: Universal approximation theorem for artificial neural networks with non-polynomial activation functions. Let σ : R → R \sigma:\mathbb{R}\to\mathbb{R} σ : R → R be a continuous activation function. Then the set of functions defined by
L ( x ) = ∑ j = 1 h α j σ ( ∑ i = 1 n w i j x i + b j ) L(x)=\sum_{j=1}^{h}\alpha_j\sigma\left(\sum_{i=1}^{n}w_{ij}x_i+b_j\right) L ( x ) = j = 1 ∑ h α j σ ( i = 1 ∑ n w ij x i + b j ) is dense in C ( K ) C(K) C ( K ) for every compact K ⊂ R n K\subset\mathbb{R}^{n} K ⊂ R n if and only if σ \sigma σ is not a polynomial almost everywhere.
Proof. Leshno’s universal approximation theorem for non-polynomial activation functions can be proved directly from Theorem B.22 just by connecting the topological definitions of C ( R n ) C(\mathbb{R}^{n}) C ( R n ) and C ( K ) C(K) C ( K ) , once we identify the set of functions L ( x ) L(x) L ( x ) with the span Σ n \Sigma_n Σ n and address the continuity of σ \sigma σ :
By definition of linear span, Σ n \Sigma_n Σ n is the set of functions of the form L ( x ) L(x) L ( x ) .
Since σ : R → R \sigma:\mathbb{R}\to\mathbb{R} σ : R → R is continuous, it is automatically bounded on every compact set and has zero discontinuities, so σ ∈ M \sigma\in M σ ∈ M .
Sufficiency (⇐ \Leftarrow ⇐ ): Suppose σ \sigma σ is not a polynomial almost everywhere, so it is certainly not an algebraic polynomial everywhere:
By Theorem B.22 , Σ n \Sigma_n Σ n is dense in C ( R n ) C(\mathbb{R}^{n}) C ( R n ) .
Restrict to K K K : Let K ⊂ R n K\subset\mathbb{R}^{n} K ⊂ R n be an arbitrary compact set and let f : K → R ∈ C ( K ) f:K\to\mathbb{R}\in C(K) f : K → R ∈ C ( K ) .
Extend f f f to R n \mathbb{R}^{n} R n : Using Tietze Extension Theorem (Theorem B.2), the continuous function f ∈ C ( K ) f\in C(K) f ∈ C ( K ) can be extended to a continuous function f ~ ∈ C ( R n ) \tilde{f}\in C(\mathbb{R}^{n}) f ~ ∈ C ( R n ) such that f ~ ( x ) = f ( x ) \tilde{f}(x)=f(x) f ~ ( x ) = f ( x ) for all x ∈ K x\in K x ∈ K .
Approximate on K K K : Because Σ n \Sigma_n Σ n is dense in C ( R n ) C(\mathbb{R}^{n}) C ( R n ) , for f ~ ∈ C ( R n ) \tilde{f}\in C(\mathbb{R}^{n}) f ~ ∈ C ( R n ) , compact set K K K , and ϵ > 0 \epsilon>0 ϵ > 0 , there exists a function L ∈ Σ n L\in\Sigma_n L ∈ Σ n such that sup x ∈ K ∣ L ( x ) − f ~ ( x ) ∣ < ϵ \sup_{x\in K}|L(x)-\tilde{f}(x)|<\epsilon sup x ∈ K ∣ L ( x ) − f ~ ( x ) ∣ < ϵ .
Since f ~ ( x ) = f ( x ) \tilde{f}(x)=f(x) f ~ ( x ) = f ( x ) on K K K and L L L has the exact form given in the theorem, we have ∥ L − f ∥ K < ϵ \|L-f\|_K<\epsilon ∥ L − f ∥ K < ϵ . Thus, the set of functions L ( x ) L(x) L ( x ) is dense in C ( K ) C(K) C ( K ) for every compact K ⊂ R n K\subset\mathbb{R}^{n} K ⊂ R n .
Necessity (⇒ \Rightarrow ⇒ ): Suppose Σ n \Sigma_n Σ n is dense in C ( K ) C(K) C ( K ) for every compact K ⊂ R n K\subset\mathbb{R}^{n} K ⊂ R n .
Density in C ( R n ) C(\mathbb{R}^{n}) C ( R n ) : Let g ∈ C ( R n ) g\in C(\mathbb{R}^{n}) g ∈ C ( R n ) , K ⊂ R n K\subset\mathbb{R}^{n} K ⊂ R n , and ϵ > 0 \epsilon>0 ϵ > 0 . The restriction of g g g to K K K , denoted g ∣ K g|_K g ∣ K , belongs to C ( K ) C(K) C ( K ) . By assumption, Σ n \Sigma_n Σ n is dense in C ( K ) C(K) C ( K ) , so there exists L ∈ Σ n L\in\Sigma_n L ∈ Σ n such that:
sup x ∈ K ∣ L ( x ) − g ( x ) ∣ = ∥ L − g ∣ K ∥ K < ϵ \sup_{x\in K}|L(x)-g(x)|=\|L-g|_K\|_K<\epsilon x ∈ K sup ∣ L ( x ) − g ( x ) ∣ = ∥ L − g ∣ K ∥ K < ϵ
Because this holds for every g ∈ C ( R n ) g\in C(\mathbb{R}^{n}) g ∈ C ( R n ) , every compact K ⊂ R n K\subset\mathbb{R}^{n} K ⊂ R n , and every ϵ > 0 \epsilon>0 ϵ > 0 , it follows that Σ n \Sigma_n Σ n is dense in C ( R n ) C(\mathbb{R}^{n}) C ( R n ) .
Application of Theorem B.22 : Because Σ n \Sigma_n Σ n is dense in C ( R n ) C(\mathbb{R}^{n}) C ( R n ) , σ \sigma σ cannot be an algebraic polynomial.
The “almost everywhere” condition: Suppose, for the sake of contradiction, that σ \sigma σ is equal to an algebraic polynomial P ( t ) P(t) P ( t ) almost everywhere on R \mathbb{R} R .
The set E = { t ∈ R : σ ( t ) ≠ P ( t ) } E=\{t\in\mathbb{R}:\sigma(t)\ne P(t)\} E = { t ∈ R : σ ( t ) = P ( t )} has Lebesgue measure zero.
The complement R ∖ E = { t ∈ R : σ ( t ) = P ( t ) } \mathbb{R}\setminus E=\{t\in\mathbb{R}:\sigma(t)=P(t)\} R ∖ E = { t ∈ R : σ ( t ) = P ( t )} is dense in R \mathbb{R} R , since any non-empty open interval in R \mathbb{R} R has strictly positive Lebesgue measure and cannot be contained in E E E .
Because both σ \sigma σ and the polynomial P P P are continuous functions on R \mathbb{R} R and they agree on a dense subset R ∖ E \mathbb{R}\setminus E R ∖ E , they must agree everywhere on R \mathbb{R} R (i.e., σ ( t ) = P ( t ) \sigma(t)=P(t) σ ( t ) = P ( t ) for all t ∈ R t\in\mathbb{R} t ∈ R ).
This means σ \sigma σ is an algebraic polynomial everywhere, which leads us to a contradiction, so σ \sigma σ is not a polynomial almost everywhere.
In conclusion, feedforward neural network can approximate any continuous function if and only if its activation function is not polynomial.
Barron’s Approximation Error Bounds
Andrew Barron [5 ] studied how the approximation error is related to the number of hidden neurons in a feed-forward neural network. His results hold for approximating functions whose Fourier transform first moment is bounded: an average of the norm of the frequency vector weighted by the Fourier magnitude distribution is used to measure how functions oscillate (i.e. their smoothness).
Definition B.15: Barron norm. Given a function f : R n → R f:\mathbb{R}^{n}\to\mathbb{R} f : R n → R , its Barron norm is given by
C f = ∫ R n ∥ ω ∥ ∣ F [ f ] ( ω ) ∣ d ω = ∫ R n ∥ ω ∥ ∣ f ^ ( ω ) ∣ d ω C_f=\int_{\mathbb{R}^{n}}\|\omega\||\mathcal{F}[f](\omega)|\,d\omega=\int_{\mathbb{R}^{n}}\|\omega\||\hat{f}(\omega)|\,d\omega C f = ∫ R n ∥ ω ∥∣ F [ f ] ( ω ) ∣ d ω = ∫ R n ∥ ω ∥∣ f ^ ( ω ) ∣ d ω
∥ ω ∥ = ∥ ω ∥ L 2 = ω ⋅ ω \|\omega\|=\|\omega\|_{L^2}=\sqrt{\omega\cdot\omega} ∥ ω ∥ = ∥ ω ∥ L 2 = ω ⋅ ω
Definition B.16: Barron function. Γ \Gamma Γ is the class of functions on R n \mathbb{R}^{n} R n for which their Barron norm C f = ∫ ∣ ω ∣ F ( d ω ) C_f = \int|\omega| \mathcal{F}(\mathrm{d}\omega) C f = ∫ ∣ ω ∣ F ( d ω ) is finite.
The finiteness of C f C_f C f imposes that the function f f f has a gradient with an integrable Fourier representation. Functions with finite C f C_f C f are continuously differentiable on R n \mathbb{R}^{n} R n and their gradient holds F [ ∇ f ] ( ω ) = i ω f ^ ( ω ) \mathcal{F}[\nabla f](\omega) = i\omega\widehat{f}(\omega) F [ ∇ f ] ( ω ) = iω f ( ω ) .
Definition B.17: Γ C \Gamma_C Γ C . For each C > 0 C > 0 C > 0 , Γ C \Gamma_C Γ C is the set of functions f : R n → R f : \mathbb{R}^{n} \to\mathbb{R} f : R n → R such that C f ≤ C C_f \leq C C f ≤ C .
Definition B.18: Γ B \Gamma_B Γ B . Analogously to Γ \Gamma Γ , Γ B \Gamma_B Γ B is defined for functions that are approximated on bounded measurable subsets B B B of their domain in R n \mathbb{R}^{n} R n . Γ B \Gamma_B Γ B is the set of functions f : B → R f : B \to\mathbb{R} f : B → R such that C f ≤ C C_f \leq C C f ≤ C .
Linear functions f ( x ) = w ⋅ x f(x) = w \cdot x f ( x ) = w ⋅ x and the class of infinitely differentiable functions on R ∗ n \mathbb{R} * n R ∗ n are not contained in Γ \Gamma Γ but are in Γ B \Gamma_B Γ B for any bounded set B B B . Containment in Γ B \Gamma_B Γ B for every bounded set B B B is denoted by Γ ∗ = ∩ B Γ B \Gamma_* = \cap_B \Gamma_B Γ ∗ = ∩ B Γ B .
Definition B.19: Γ C , B \Gamma_{C,B} Γ C , B . Γ C , B \Gamma_{C,B} Γ C , B is the class of functions f f f in Γ B \Gamma_B Γ B for which C f = ∫ ∣ ω ∣ B F ( d ω ) ≤ C C_f = \int|\omega|_B \mathcal{F}(\mathrm{d}\omega) \leq C C f = ∫ ∣ ω ∣ B F ( d ω ) ≤ C , where ∣ ω ∣ B = sup x ∈ B ∣ ω ⋅ x ∣ |\omega|_B = \sup_{x \in B} |\omega\cdot x| ∣ ω ∣ B = sup x ∈ B ∣ ω ⋅ x ∣ .
For the ball B r = { x : ∣ x ∣ ≤ r } B_r = \{x : |x| \leq r\} B r = { x : ∣ x ∣ ≤ r } , the norm simplifies to ∣ ω ∣ B = r ∣ ω ∣ |\omega|_B = r|\omega| ∣ ω ∣ B = r ∣ ω ∣ .
An upper bound of the approximation error by linear combinations of sigmoidal functions can be obtained if we take into account that, when the function has an integrable representation in terms of parameterized basis functions, then a random sample of the parameters of the basis functions can lead to an accurate approximation.
The approximation error is measured by integrating the squared error with respect to an arbitrary probability measure μ \mu μ on the ball B r = { x : ∣ x ∣ ≤ r } B_r = \{x : |x| \leq r\} B r = { x : ∣ x ∣ ≤ r } of radius r > 0 r > 0 r > 0 .
Lemma B.23: Convex combinations in a Hilbert space. If f f f is the closure of the convex hull of a set G G G in a Hilbert space, with ∥ g ∥ < b \|g\| < b ∥ g ∥ < b for each g ∈ G g \in G g ∈ G , for every h ≥ 1 h \geq1 h ≥ 1 and every c ′ > b 2 − ∥ f ∥ 2 c' > b^2 - \|f\|^2 c ′ > b 2 − ∥ f ∥ 2 , there is an f h f_h f h in the convex hull of h h h points in G G G such that
∥ f − f h ∥ 2 ≤ c ′ h \|f - f_h\|^2 \leq\frac{c'}{h} ∥ f − f h ∥ 2 ≤ h c ′ Proof. Given n ≥ 1 n \geq1 n ≥ 1 and δ > 0 \delta> 0 δ > 0 , let f ∗ f^* f ∗ a point in the convex hull of G G G with ∥ f − f ∗ ∥ ≤ δ / h \|f - f^*\| \leq\delta/h ∥ f − f ∗ ∥ ≤ δ / h . Therefore, f ∗ f^* f ∗ is of the form ∑ k = 1 m γ k g k \sum_{k=1}^{m} \gamma_k g_k ∑ k = 1 m γ k g k , with g k ∈ G g_k \in G g k ∈ G , γ k ≥ 0 \gamma_k \geq0 γ k ≥ 0 , ∑ k = 1 m γ k = 1 \sum_{k=1}^{m} \gamma_k = 1 ∑ k = 1 m γ k = 1 , for some sufficiently large m m m .
Let g g g be randomly drawn from the set { g 1 … g m ∗ } \{g_1 \ldots g_m^*\} { g 1 … g m ∗ } with P ( g = g k ∗ ) = γ k P(g = g^*_k) = \gamma_k P ( g = g k ∗ ) = γ k . Let g 1 … g h g_1 \ldots g_h g 1 … g h be independently drawn from the same distribution as g g g . Their sample average is f h = ( 1 / h ) ∑ i = 1 h g i f_h = (1/h)\sum_{i=1}^{h} g_i f h = ( 1/ h ) ∑ i = 1 h g i . Then E [ f h ] = f ∗ E[f_h] = f^* E [ f h ] = f ∗ . The expected value of the squared norm of the error is E [ ∥ f h − f ∗ ∥ 2 ] = ( 1 / h ) E [ ∥ g − f ∗ ∥ 2 ] = ( 1 / h ) ( E [ ∥ g ∥ 2 ] − ∥ f ∗ ∥ 2 ) E[\|f_h - f^*\|^2] = (1/h)E[\|g - f^*\|^2] = (1/h)(E[\|g\|^2] - \|f^*\|^2) E [ ∥ f h − f ∗ ∥ 2 ] = ( 1/ h ) E [ ∥ g − f ∗ ∥ 2 ] = ( 1/ h ) ( E [ ∥ g ∥ 2 ] − ∥ f ∗ ∥ 2 ) , which is bounded by ( 1 / h ) ( b 2 − ∥ f ∗ ∥ 2 ) (1/h)(b^2 - \|f^*\|^2) ( 1/ h ) ( b 2 − ∥ f ∗ ∥ 2 ) .
Since the expected value is bounded, there must exist g 1 … g h g_1 \ldots g_h g 1 … g h for which ∥ f h − f ∗ ∥ 2 ≤ ( 1 / h ) ( b 2 − ∥ f ∗ ∥ 2 ) \|f_h - f^*\|^2 \leq(1/h)(b^2 - \|f^*\|^2) ∥ f h − f ∗ ∥ 2 ≤ ( 1/ h ) ( b 2 − ∥ f ∗ ∥ 2 ) . The proof of Lemma B.23 is completed using the triangle inequality and ∥ f − f h ∥ 2 ≤ δ / h \|f - f_h\|^2 \leq\delta/h ∥ f − f h ∥ 2 ≤ δ / h for a sufficiently small δ \delta δ .
By combining the approximation of f ˉ \bar{f} f ˉ by f ∗ f^* f ∗ with the probabilistic approximation of f ∗ f^* f ∗ by f h f_h f h :
∥ f h − f ∗ ∥ 2 ≤ 1 h ( b 2 − ∥ f ∗ ∥ 2 ) ≤ b 2 h \|f_h - f^*\|^2 \leq\frac{1}{h}(b^2 - \|f^*\|^2) \leq\frac{b^2}{h} ∥ f h − f ∗ ∥ 2 ≤ h 1 ( b 2 − ∥ f ∗ ∥ 2 ) ≤ h b 2
We apply the triangle inequality:
∥ f ˉ − f h ∥ ≤ ∥ f ˉ − f ∗ ∥ + ∥ f ∗ − f h ∥ ≤ δ h + b h \|\bar{f} - f_h\| \leq\|\bar{f} - f^*\| + \|f^* - f_h\| \leq\frac{\delta}{h} + \frac{b}{\sqrt{h}} ∥ f ˉ − f h ∥ ≤ ∥ f ˉ − f ∗ ∥ + ∥ f ∗ − f h ∥ ≤ h δ + h b
Squaring gives
∥ f ˉ − f h ∥ 2 ≤ ( δ h + b h ) 2 = δ 2 h 2 + 2 b δ h 3 / 2 + b 2 h \|\bar{f} - f_h\|^2 \leq\left(\frac{\delta}{h} + \frac{b}{\sqrt{h}}\right)^2 = \frac{\delta^2}{h^2} + \frac{2b\delta}{h^{3/2}} + \frac{b^2}{h} ∥ f ˉ − f h ∥ 2 ≤ ( h δ + h b ) 2 = h 2 δ 2 + h 3/2 2 b δ + h b 2
Since
δ 2 h 2 + 2 b δ h 3 / 2 ∈ o ( 1 h ) \frac{\delta^2}{h^2} + \frac{2b\delta}{h^{3/2}} \in o\left(\frac{1}{h}\right) h 2 δ 2 + h 3/2 2 b δ ∈ o ( h 1 )
for sufficiently large h h h . Then, for any ϵ > 0 \epsilon> 0 ϵ > 0 :
∥ f ˉ − f h ∥ 2 ≤ b 2 + ϵ h \|\bar{f} - f_h\|^2 \leq\frac{b^2 + \epsilon}{h} ∥ f ˉ − f h ∥ 2 ≤ h b 2 + ϵ
Equivalently, there exists a constant c ′ > 0 c' > 0 c ′ > 0 such that
∥ f ˉ − f h ∥ 2 ≤ c ′ h \|\bar{f} - f_h\|^2 \leq\frac{c'}{h} ∥ f ˉ − f h ∥ 2 ≤ h c ′
Lemma B.24: For every function in Γ C , B \Gamma_{C,B} Γ C , B and every sigmoidal function σ \sigma σ , the function f ˉ ( x ) = f ( x ) − f ( 0 ) \bar{f}(x) = f(x) - f(0) f ˉ ( x ) = f ( x ) − f ( 0 ) is in the closure of the convex hull of G σ G_\sigma G σ , where the closure is taken in L 2 ( μ , B ) L_2(\mu, B) L 2 ( μ , B ) .
Proof. Let F ^ ( d ω ) = e i Θ ( ω ) F ( d ω ) \widehat{F}(\mathrm{d}\omega) = e^{i\Theta(\omega)}F(\mathrm{d}\omega) F ( d ω ) = e i Θ ( ω ) F ( d ω ) be the magnitude and phase decomposition of the Fourier transform of an extension of the function f f f on B B B for which ∫ ∣ ω ∣ B F ( d ω ) ≤ C \int|\omega|_B F(\mathrm{d}\omega) \leq C ∫ ∣ ω ∣ B F ( d ω ) ≤ C . Let Ω = { ω ∈ R n : ω ≠ 0 } \Omega= \{\omega\in\mathbb{R}^{n} : \omega\neq0\} Ω = { ω ∈ R n : ω = 0 } . Since f ( x ) f(x) f ( x ) is real-valued:
f ( x ) − f ( 0 ) = Re ∫ ( e i ω ⋅ x − 1 ) F ^ ( d ω ) f(x) - f(0) = \operatorname{Re}\int(e^{i\omega\cdot x} - 1)\widehat{F}(\mathrm{d}\omega) f ( x ) − f ( 0 ) = Re ∫ ( e iω ⋅ x − 1 ) F ( d ω ) = Re ∫ ( e i ω ⋅ x − 1 ) e i Θ ( ω ) F ( d ω ) = \operatorname{Re}\int(e^{i\omega\cdot x} - 1)e^{i\Theta(\omega)}F(\mathrm{d}\omega) = Re ∫ ( e iω ⋅ x − 1 ) e i Θ ( ω ) F ( d ω ) = ∫ Ω ( cos ( ω ⋅ x + Θ ( ω ) ) − cos ( Θ ( ω ) ) ) F ( d ω ) = \int_{\Omega} (\cos(\omega\cdot x + \Theta(\omega)) - \cos(\Theta(\omega)))F(\mathrm{d}\omega) = ∫ Ω ( cos ( ω ⋅ x + Θ ( ω )) − cos ( Θ ( ω ))) F ( d ω ) = ∫ Ω g ( x , ω ) Λ ( d ω ) = \int_{\Omega} g(x,\omega)\Lambda(\mathrm{d}\omega) = ∫ Ω g ( x , ω ) Λ ( d ω ) for x ∈ B x \in B x ∈ B , where C f , B = ∫ ∣ ω ∣ B F ( d ω ) ≤ C C_{f,B} = \int|\omega|_B F(\mathrm{d}\omega) \leq C C f , B = ∫ ∣ ω ∣ B F ( d ω ) ≤ C , Λ ( d ω ) = ∣ ω ∣ B F ( d ω ) / C f , B \Lambda(\mathrm{d}\omega) = |\omega|_B F(\mathrm{d}\omega)/C_{f,B} Λ ( d ω ) = ∣ ω ∣ B F ( d ω ) / C f , B is a probability distribution, and
g ( x , ω ) = C f , B ∣ ω ∣ B ( cos ( ω ⋅ x + Θ ( ω ) ) − cos ( Θ ( ω ) ) ) g(x,\omega) = \frac{C_{f,B}}{|\omega|_B}(\cos(\omega\cdot x + \Theta(\omega)) - \cos(\Theta(\omega))) g ( x , ω ) = ∣ ω ∣ B C f , B ( cos ( ω ⋅ x + Θ ( ω )) − cos ( Θ ( ω ))) For x ∈ B x \in B x ∈ B and ω ≠ 0 \omega\neq0 ω = 0 , g ( x , ω ) ≤ C ∣ ω ⋅ x ∣ / ∣ ω ∣ B ≤ C g(x,\omega) \leq C|\omega\cdot x|/|\omega|_B \leq C g ( x , ω ) ≤ C ∣ ω ⋅ x ∣/∣ ω ∣ B ≤ C .
The integral above represents f ˉ \bar{f} f ˉ as an infinite convex combination of functions in the class G cos G_{\cos} G c o s :
G cos = { γ ∣ ω ∣ B ( cos ( ω ⋅ x + b ) − cos ( b ) ) : ω ≠ 0 , ∣ γ ∣ ≤ C , b ∈ R } G_{\cos}=\left\{\frac{\gamma}{|\omega|_{B}}\left(\cos(\omega\cdot x+b)-\cos(b)\right):\omega\ne0,|\gamma|\le C,b\in\mathbb{R}\right\} G c o s = { ∣ ω ∣ B γ ( cos ( ω ⋅ x + b ) − cos ( b ) ) : ω = 0 , ∣ γ ∣ ≤ C , b ∈ R }
It follows that f ˉ \bar{f} f ˉ is in the closure of the convex hull G cos G_{\cos} G c o s :
For each f ∈ Γ C , B f\in\Gamma_{C,B} f ∈ Γ C , B , the function f ( x ) − f ( 0 ) f(x)-f(0) f ( x ) − f ( 0 ) is in the closure of the convex hull of G cos G_{\cos} G c o s .
If ω 1 … ω h \omega_1\ldots\omega_h ω 1 … ω h is a random sample of h h h points, independently drawn from the distribution Λ \Lambda Λ , by Fubini’s theorem
E ∫ B r ( f ( x ) − ∑ i = 1 h g ( x , ω i ) ) 2 μ ( d x ) = ∫ B r E ( f ( x ) − ∑ i = 1 h g ( x , ω i ) ) 2 μ ( d x ) = 1 h var ( g ( x , ω ) ) μ ( d x ) ≤ C 2 h \begin{aligned}
E\int_{B_r}\left(f(x)-\sum_{i=1}^{h}g(x,\omega_i)\right)^2\mu(dx) \\
&=\int_{B_r}E\left(f(x)-\sum_{i=1}^{h}g(x,\omega_i)\right)^2\mu(dx) \\
&=\frac{1}{h}\operatorname{var}(g(x,\omega))\mu(dx) \\
&\le\frac{C^2}{h}
\end{aligned} E ∫ B r ( f ( x ) − i = 1 ∑ h g ( x , ω i ) ) 2 μ ( d x ) = ∫ B r E ( f ( x ) − i = 1 ∑ h g ( x , ω i ) ) 2 μ ( d x ) = h 1 var ( g ( x , ω )) μ ( d x ) ≤ h C 2
Therefore, the mean value of the squared L 2 ( μ , B ) L_2(\mu,B) L 2 ( μ , B ) norm of a convex combination in G cos G_{\cos} G c o s converges to zero as h → ∞ h\to\infty h → ∞ , at rate O ( 1 / h ) O(1/h) O ( 1/ h ) in accordance to Lemma B.23 .
G cos G_{\cos} G c o s is in the closure of the convex hull of G s t e p G_{\mathrm{step}} G step .
Each function in G cos G_{\cos} G c o s is a composition of a 1D sinusoidal function g ( z ) = γ ∣ ω ∣ B ( cos ( ∣ ω ∣ B x + b ) − cos ( b ) ) g(z)=\frac{\gamma}{|\omega|_B}(\cos(|\omega|_B x+b)-\cos(b)) g ( z ) = ∣ ω ∣ B γ ( cos ( ∣ ω ∣ B x + b ) − cos ( b )) and a linear function z ( x ) = α ⋅ x z(x)=\alpha\cdot x z ( x ) = α ⋅ x , where α = ω / ∣ ω ∣ B \alpha=\omega/|\omega|_B α = ω /∣ ω ∣ B for some ω ≠ 0 \omega\ne0 ω = 0 . For x ∈ B x\in B x ∈ B , z ( x ) ∈ [ − 1 , 1 ] z(x)\in[-1,1] z ( x ) ∈ [ − 1 , 1 ] . Then, it suffices to analyze the approximation of the sinusoidal function in [ − 1 , 1 ] [-1,1] [ − 1 , 1 ]
Since g g g is uniformly continuous in [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] , it is uniformly well approximated by piecewise constant functions, for any sequence of partitions of [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] into intervals of maximum width tending to 0. Such piecewise functions can be represented as linear combinations of step functions.
Consider g ( z ) g(z) g ( z ) restricted to z ∈ [ 0 , 1 ] z\in[0,1] z ∈ [ 0 , 1 ] , with g ( 0 ) = 0 g(0)=0 g ( 0 ) = 0 . For a partition 0 = t 0 < t 1 < … < t k = 1 0=t_0<t_1<\ldots<t_k=1 0 = t 0 < t 1 < … < t k = 1 ,
g k , + ( z ) = ∑ k = 1 k − 1 ( g ( t i ) − g ( t i − 1 ) ) 1 { z ≥ t i } g_{k,+}(z)=\sum_{k=1}^{k-1}(g(t_i)-g(t_{i-1}))1_{\{z\ge t_i\}} g k , + ( z ) = k = 1 ∑ k − 1 ( g ( t i ) − g ( t i − 1 )) 1 { z ≥ t i }
interpolates the function g g g at the points t i t_i t i as a linear combination of step functions. The derivative of g g g is bounded by ∣ γ ∣ ≤ C |\gamma|\le C ∣ γ ∣ ≤ C , so the sum of the absolute values of the coefficients is also bounded by C C C , i.e. ∑ i ∣ g ( t i ) − g ( t i − 1 ) ∣ ≤ C \sum_i|g(t_i)-g(t_{i-1})|\le C ∑ i ∣ g ( t i ) − g ( t i − 1 ) ∣ ≤ C .
In a similar way,
g k , − ( z ) = ∑ k = 1 k − 1 ( g ( − t i ) − g ( − t i − 1 ) ) 1 { z ≤ − t i } g_{k,-}(z)=\sum_{k=1}^{k-1}(g(-t_i)-g(-t_{i-1}))1_{\{z\le-t_i\}} g k , − ( z ) = k = 1 ∑ k − 1 ( g ( − t i ) − g ( − t i − 1 )) 1 { z ≤ − t i }
Adding g h , + ( z ) + g h , − ( z ) g_{h,+}(z)+g_{h,-}(z) g h , + ( z ) + g h , − ( z ) yields a sequence of piecewise constant functions on [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] that are uniformly close to g ( z ) g(z) g ( z ) . They are linear combinations of step functions and the sum of the absolute values of their coefficients is bounded by 2 C 2C 2 C .
The functions g ( z ) g(z) g ( z ) are, therefore, in the closure of the convex hull of the functions γ step ( z − t ) \gamma\operatorname{step}(z-t) γ step ( z − t ) and γ step ( − z − t ) \gamma\operatorname{step}(-z-t) γ step ( − z − t ) with ∣ γ ∣ ≤ 2 C |\gamma|\le2C ∣ γ ∣ ≤ 2 C and t ≤ 1 t\le1 t ≤ 1 , where step ( z ) = 1 z ≥ 0 \operatorname{step}(z)=1_{z\ge0} step ( z ) = 1 z ≥ 0 denotes the step function:
G s t e p = { γ step ( α ⋅ x − t ) : ∣ γ ∣ ≤ 2 C , ∣ α ∣ B = 1 , ∣ t ∣ ≤ 1 } G_{\mathrm{step}}=\left\{\gamma\operatorname{step}(\alpha\cdot x-t):|\gamma|\le2C,|\alpha|_B=1,|t|\le1\right\} G step = { γ step ( α ⋅ x − t ) : ∣ γ ∣ ≤ 2 C , ∣ α ∣ B = 1 , ∣ t ∣ ≤ 1 }
G cos G_{\cos} G c o s is in the closure of the convex hull of G s t e p μ G_{\mathrm{step}}^\mu G step μ .
The result for G s t e p G_{\mathrm{step}} G step holds if, for each α \alpha α , the parameter t t t is restricted to a subset T α T_\alpha T α that is dense in [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] .
We can restrict t t t to the continuity points of the distribution z = α ⋅ x z=\alpha\cdot x z = α ⋅ x induced by the measure μ \mu μ on R n \mathbb{R}^n R n to obtain the class G s t e p μ G_{\mathrm{step}}^\mu G step μ .
G s t e p μ G_{\mathrm{step}}^\mu G step μ is in the closure of the convex hull of G σ G_\sigma G σ .
The sequence of sigmoidal functions σ ( ∣ a ∣ ( α ⋅ x − t ) ) \sigma(|a|(\alpha\cdot x-t)) σ ( ∣ a ∣ ( α ⋅ x − t )) , with ∣ a ∣ → ∞ |a|\to\infty ∣ a ∣ → ∞ , has pointwise limit equal to step ( α ⋅ x − t ) \operatorname{step}(\alpha\cdot x-t) step ( α ⋅ x − t ) , except possibly for x x x in the set with α ⋅ x − t = 0 \alpha\cdot x-t=0 α ⋅ x − t = 0 , which has measure zero by the restriction imposed on t t t . By the dominated convergence theorem, the limit also holds in L 2 ( μ , B ) L_2(\mu,B) L 2 ( μ , B ) . Functions in G s t e p μ G_{\mathrm{step}}^\mu G step μ are in the closure in L 2 ( μ , B ) L_2(\mu,B) L 2 ( μ , B ) of the class of sigmoidal functions.
Putting together the previous closure results:
Γ C 0 ⊂ G cos ‾ ⊂ G s t e p μ ‾ ⊂ G σ ‾ \Gamma_C^0\subset\overline{G_{\cos}}\subset\overline{G_{\mathrm{step}}^\mu}\subset\overline{G_\sigma} Γ C 0 ⊂ G c o s ⊂ G step μ ⊂ G σ
where G ‾ \overline{G} G denotes the closure in L 2 ( μ , B ) L_2(\mu,B) L 2 ( μ , B ) of the convex hull of G G G and Γ C 0 \Gamma_C^0 Γ C 0 is the set of functions in Γ C \Gamma_C Γ C with f ( 0 ) = 0 f(0)=0 f ( 0 ) = 0 . This completes our proof.
Theorem B.25. For every function f ∈ Γ C , B f\in\Gamma_{C,B} f ∈ Γ C , B , every sigmoidal function σ \sigma σ , every probability measure μ \mu μ , and every h ≥ 1 h\ge1 h ≥ 1 , there exists a linear combination of sigmoidal functions f h ( n ) f_h(n) f h ( n ) such that
∫ B ( f ( x ) − f h ( x ) ) 2 μ ( x ) d x ≤ ( 2 C ) 2 h \int_B(f(x)-f_h(x))^2\mu(x)\,dx\le\frac{(2C)^2}{h} ∫ B ( f ( x ) − f h ( x ) ) 2 μ ( x ) d x ≤ h ( 2 C ) 2
Proof. The proof is based on convex combinations in Hilbert spaces (Lemma B.23 ) and the convex hull of sigmoidal functions (Lemma B.24 ).
Given a measurable set B B B containing the point x = 0 x=0 x = 0 , if, for functions in the class Γ C , B \Gamma_{C,B} Γ C , B , f ‾ ( x ) = f ( x ) − f ( 0 ) \overline{f}(x)=f(x)-f(0) f ( x ) = f ( x ) − f ( 0 ) is in the closure of the convex hull of G σ G_{\sigma} G σ in L 2 ( μ , B ) L_{2}(\mu,B) L 2 ( μ , B ) . Then, by Lemma B.23 , there exists a convex combination of h h h sigmoidal functions such that the square of the L 2 ( μ , B ) L_{2}(\mu,B) L 2 ( μ , B ) norm is bounded by a constant divided by h h h .
The constant can be determined to be ( 2 C ) 2 (2C)^2 ( 2 C ) 2 as follows:
When the sigmoidal function is bounded by 1, the functions in G σ G_{\sigma} G σ are bounded by b = 2 C b=2C b = 2 C . Any c ′ > ( 2 C ) 2 − ∥ f ‾ ∥ 2 c'>(2C)^2-\lVert\overline{f}\rVert^2 c ′ > ( 2 C ) 2 − ∥ f ∥ 2 is valid for Lemma B.23 . There is a convex combination of h h h functions in G σ G_{\sigma} G σ for which the square of the L 2 ( μ , B ) L_{2}(\mu,B) L 2 ( μ , B ) norm of the approximation error is bounded by c ′ / h c'/h c ′ / h .
If the sigmoidal function is not bounded by 1, combine Lemma B.23 and Γ C 0 ⊂ G s t e p μ \Gamma_{C}^{0}\subset G_{\mathrm{step}}^{\mu} Γ C 0 ⊂ G step μ to obtain a convex combination of h h h functions in G s t e p μ G_{\mathrm{step}}^{\mu} G step μ for which the square of the L 2 ( μ , B ) L_{2}(\mu,B) L 2 ( μ , B ) norm of the approximation error is bounded by ( 2 C ) 2 − ( 1 / 2 ) ∥ f ‾ ∥ 2 (2C)^2-(1/2)\lVert\overline{f}\rVert^2 ( 2 C ) 2 − ( 1/2 ) ∥ f ∥ 2 divided by h h h . Then, using G s t e p μ ⊂ G σ G_{\mathrm{step}}^{\mu}\subset G_{\sigma} G step μ ⊂ G σ , by scaling the sigmoid function, sufficiently accurate replacements of the step functions can be obtained so that a convex combination of h h h functions in G σ G_{\sigma} G σ yields a square L 2 ( μ , B ) L_{2}(\mu,B) L 2 ( μ , B ) norm of the approximation error bounded by ( 2 C ) 2 / h (2C)^2/h ( 2 C ) 2 / h .
This completes Barron’s proof [5 ] .
Theorem II.4: Barron’s Theorem. Let f : R n → R f:\mathbb{R}^{n}\to\mathbb{R} f : R n → R be a function with Fourier transform F \mathcal{F} F such that its Barron norm is finite, i.e.
∫ R n ∥ ω ∥ ∣ F ( ω ) ∣ d ω < ∞ . \int_{\mathbb{R}^{n}}\lVert\omega\rVert\lvert\mathcal{F}(\omega)\rvert\,\mathrm{d}\omega<\infty. ∫ R n ∥ ω ∥ ∣ F ( ω )∣ d ω < ∞. Then there exists a single-hidden-layer neural network f H f_{H} f H with H H H hidden units such that
∥ f − f H ∥ L 2 ≤ C H , \lVert f-f_{H}\rVert_{L^{2}}\le\frac{C}{\sqrt{H}}, ∥ f − f H ∥ L 2 ≤ H C , where C C C depends on the Barron norm of f f f .
Proof. The Barron norm is
∥ f ∥ B : = ∫ R n ∥ ω ∥ ∣ F ( ω ) ∣ d ω \lVert f\rVert_{B}:=\int_{\mathbb{R}^{n}}\lVert\omega\rVert\lvert\mathcal{F}(\omega)\rvert\,\mathrm{d}\omega ∥ f ∥ B := ∫ R n ∥ ω ∥ ∣ F ( ω )∣ d ω A Barron function is a function with a finite Barron norm C f = ∥ f ∥ B < ∞ C_{f}=\lVert f\rVert_{B}<\infty C f = ∥ f ∥ B < ∞ . By the definition of the Barron space, f ∈ Γ C f , B f\in\Gamma_{C_{f},B} f ∈ Γ C f , B . Therefore, Barron’s approximation theorem (Theorem B.25) applies: for every h ≥ 1 h\ge1 h ≥ 1 , there exists a linear combination of sigmoidal units f h f_{h} f h such that
∫ B ( f ( x ) − f h ( x ) ) 2 d μ ( x ) ≤ ( 2 C f ) 2 h \int_{B}(f(x)-f_{h}(x))^{2}\,\mathrm{d}\mu(x)\le\frac{(2C_{f})^{2}}{h} ∫ B ( f ( x ) − f h ( x ) ) 2 d μ ( x ) ≤ h ( 2 C f ) 2 Setting h = H h=H h = H , where H H H is the number of hidden units:
∥ f − f H ∥ L 2 ( B , μ ) 2 ≤ 4 C f 2 H \lVert f-f_{H}\rVert_{L^{2}(B,\mu)}^{2}\le\frac{4C_{f}^{2}}{H} ∥ f − f H ∥ L 2 ( B , μ ) 2 ≤ H 4 C f 2 Taking square roots yields
∥ f − f H ∥ L 2 ( B , μ ) ≤ 2 C f H \lVert f-f_{H}\rVert_{L^{2}(B,\mu)}\le\frac{2C_{f}}{\sqrt{H}} ∥ f − f H ∥ L 2 ( B , μ ) ≤ H 2 C f Just by renaming 2 C f 2C_{f} 2 C f as C C C , we obtain the desired result: every Barron function admits approximation by a single-hidden-layer neural network with H H H hidden units at the rate O ( 1 / H ) O(1/\sqrt{H}) O ( 1/ H ) .
Remark. The dimension n n n can appear indirectly through the constant C f C_{f} C f . Dependence of the constant on n n n does not affect the convergence rate, but, when C f C_{f} C f is exponentially large in n n n , an exponentially large value of h h h might be needed for C f 2 / h C_{f}^{2}/h C f 2 / h to be small enough for approximation (an unsurprising result, given that C f C_{f} C f involves an n n n -dimensional integral).
Feedforward neural networks with a single hidden layer of h h h sigmoidal units achieve integrated squared error of order O ( 1 / h ) O(1/h) O ( 1/ h ) . Barron found that surprising when compared to the exponential number of parameters required by polynomial, spline, and trigonometric expansions. The neural network just needs ( n + 2 ) ∗ h + 1 (n+2)*h+1 ( n + 2 ) ∗ h + 1 parameters.24
Montúfar on the Power of Deep Networks
In a NIPS’2004 paper (now NeurIPS), Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio studied the complexity of the functions that are computable by deep feedforward neural networks with piecewise linear activations (e.g. ReLU) in terms of the symmetries and the number of linear regions that they have [30 ] .
Definition B.20. A map F F F identifies two neighborhoods S S S and T T T of its input domain if it maps them to a common subset F ( S ) = F ( T ) F(S)=F(T) F ( S ) = F ( T ) of its output domain. In this case, we say that S S S and T T T are identified by F F F .
Lemma B.26. The maximal number of linear regions of the functions computed by an L L L -layer neural network with piecewise linear activations is at least N L = ∑ R ∈ P L − 1 N R L − 1 \mathcal{N}^{L}=\sum_{R\in P^{L-1}}\mathcal{N}_{R}^{L-1} N L = ∑ R ∈ P L − 1 N R L − 1 , where N R 0 = 1 \mathcal{N}_{R}^{0}=1 N R 0 = 1 for each region R ⊆ R n 0 R\subseteq\mathbb{R}^{n_{0}} R ⊆ R n 0 and P L P^{L} P L denotes the set of neighbourhoods in distinct linear regions of the function computed by the L L L -th layer.
Theorem III.1: Expressivity via Linear Regions. Let N N N be a feedforward ReLU network with input dimension n 0 n_{0} n 0 , L L L hidden layers, and widths n 1 … n L n_{1}\ldots n_{L} n 1 … n L , n l ≥ n 0 ∀ l n_{l}\geq n_{0}\ \forall l n l ≥ n 0 ∀ l . Then, the maximal number of linear regions of the function computed by the neural network N N N is lower bounded by
( ∏ l = 1 L − 1 ⌊ n l n 0 ⌋ n 0 ) ∑ j = 0 n 0 ( n L j ) \left(\prod_{l=1}^{L-1}\left\lfloor\frac{n_{l}}{n_{0}}\right\rfloor^{n_{0}}\right)\sum_{j=0}^{n_{0}}\binom{n_{L}}{j} ( l = 1 ∏ L − 1 ⌊ n 0 n l ⌋ n 0 ) j = 0 ∑ n 0 ( j n L )
Proof. Montúfar’s proof, included in their arXiv technical report [31 ] , counts the number of regions by dividing the first L − 1 L-1 L − 1 network layers into n 0 n_{0} n 0 independent parts, one for each input neuron (i.e. each part is sensitive only to one coordinate of the input-space).
The units of each layer are organized into n 0 n_{0} n 0 nonempty groups of units of sizes p 1 = … = p n 0 = ⌊ n l / n 0 ⌋ p_{1}=\ldots=p_{n_{0}}=\left\lfloor n_{l}/n_{0}\right\rfloor p 1 = … = p n 0 = ⌊ n l / n 0 ⌋ for a layer of width n l n_{l} n l .
The input weights of each group are defined in such a way that the units in that group are sensitive to only one coordinate of the n 0 n_{0} n 0 -dimensional input-space. By choosing the input and bias weights in the right way, the alternating sum of the activations of the p p p units within one group folds their input-space coordinate p p p times into itself.
Consider a layer of n n n rectifiers with n 0 n_{0} n 0 input variables, where n ≥ n 0 n\geq n_{0} n ≥ n 0 . We partition the set of rectifier units into n 0 n_{0} n 0 (non-overlapping) subsets of cardinality p = ⌊ n / n 0 ⌋ p=\lfloor n/n_{0}\rfloor p = ⌊ n / n 0 ⌋ . Weights and biases in each group are chosen as follows:
h 1 ( x ) = max { 0 , w x } h_{1}(x)=\max\{0,wx\} h 1 ( x ) = max { 0 , w x }
h 2 ( x ) = max { 0 , 2 w x − 1 } h_{2}(x)=\max\{0,2wx-1\} h 2 ( x ) = max { 0 , 2 w x − 1 }
h 3 ( x ) = max { 0 , 2 w x − 2 } h_{3}(x)=\max\{0,2wx-2\} h 3 ( x ) = max { 0 , 2 w x − 2 }
h p ( x ) = max { 0 , 2 w x − ( p − 1 ) } h_{p}(x)=\max\{0,2wx-(p-1)\} h p ( x ) = max { 0 , 2 w x − ( p − 1 )}
where w w w is a vector whose j j j -th entry is 1 and all other entries are 0. The inner product w x wx w x selects the j j j -th coordinate of x x x .
Adding these rectifiers with alternating signs, we obtain the scalar function h ~ j ( x ) \tilde{h}_{j}(x) h ~ j ( x ) :
1 a l t = [ 1 , − 1 , 1 , … , ( − 1 ) p − 1 ] \mathbf{1}_{\mathrm{alt}}=[1,-1,1,\ldots,(-1)^{p-1}] 1 alt = [ 1 , − 1 , 1 , … , ( − 1 ) p − 1 ]
h ~ j ( x ) = 1 a l t ⋅ [ h 1 ( x ) , h 2 ( x ) , h 3 ( x ) , … , h p ( x ) ] \tilde{h}_{j}(x)=\mathbf{1}_{\mathrm{alt}}\cdot[h_{1}(x),h_{2}(x),h_{3}(x),\ldots,h_{p}(x)] h ~ j ( x ) = 1 alt ⋅ [ h 1 ( x ) , h 2 ( x ) , h 3 ( x ) , … , h p ( x )]
The function h ~ j ( x ) \tilde{h}_{j}(x) h ~ j ( x ) acts only on the j j j -th coordinate of x x x , so it can be interpreted as a scalar function. This scalar function acts on p p p intervals: ( − ∞ , 0 ] (-\infty,0] ( − ∞ , 0 ] , [ 0 , 1 ] [0,1] [ 0 , 1 ] , [ 1 , 2 ] [1,2] [ 1 , 2 ] ...[ p − 1 , ∞ ) [p-1,\infty) [ p − 1 , ∞ ) . Each of those intervals are mapped by h ~ ( x ) \tilde{h}(x) h ~ ( x ) onto the interval ( 0 , 1 ) (0,1) ( 0 , 1 ) .
If we consider the n 0 n_{0} n 0 sets of rectifiers, the function h ~ = [ h ~ 1 , h ~ 2 , h ~ 3 , … , h ~ p ] \tilde{h}=[\tilde{h}_{1},\tilde{h}_{2},\tilde{h}_{3},\ldots,\tilde{h}_{p}] h ~ = [ h ~ 1 , h ~ 2 , h ~ 3 , … , h ~ p ] is locally symmetric about each hyperplane with a fixed j j j -th coordinate equal to x j = 1 … p x_{j}=1\ldots p x j = 1 … p . A periodic pattern emerges and the function h ~ \tilde{h} h ~ identifies p n 0 p^{n_{0}} p n 0 hypercubes delimited by these hyperplanes.
NOTE. h ~ \tilde{h} h ~ arises from h h h by composition with a linear function 1 a l t \mathbf{1}_{\mathrm{alt}} 1 alt (alternating sums). This linear function can be effectively absorbed in the preactivation function of the next layer. Hence, we can treat h ~ \tilde{h} h ~ as the function computed by the current layer. Computations by deeper layers, as functions of the unit hypercube output of this rectifier layer, are replicated on each of the p n 0 p^{n_{0}} p n 0 input-space hypercubes.
Once the parameters of each group are chosen so that they fold its 1D input-space many times into itself, the alternating sum h ~ = h 1 − h 2 + h 3 − … ± h p \tilde{h}=h_{1}-h_{2}+h_{3}-\ldots\pm h_{p} h ~ = h 1 − h 2 + h 3 − … ± h p of the activations of the p p p units in a group as the activation of a fictitious intermediary unit. The compound output of all these n 0 n_{0} n 0 intermediary units partitions the input-space, R n 0 \mathbb{R}^{n_{0}} R n 0 , into a grid of ∏ i = 1 n 0 p i \prod_{i=1}^{n_{0}}p_{i} ∏ i = 1 n 0 p i regions. Each of these input-space regions is mapped to the n 0 n_{0} n 0 -dimensional unit cube in the output-space of the intermediary layer. This unit cube is the effective input for the next hidden layer. With each layer of the network, the number of identified regions is multiplied by ∏ i = 1 n 0 p i \prod_{i=1}^{n_{0}}p_{i} ∏ i = 1 n 0 p i .
The function h ~ \tilde{h} h ~ computes an alternating sum that folds the input-coordinate j j j times into the interval ( 0 , 1 ) (0,1) ( 0 , 1 ) . The function h ~ j ( x ) \tilde{h}_{j}(x) h ~ j ( x ) identifies the input-space region with j j j -th coordinate x j x_{j} x j restricted to the intervals ( 0 , 1 ) , ( 1 , 2 ) … ( p − 1 , p ) (0,1),(1,2)\ldots(p-1,p) ( 0 , 1 ) , ( 1 , 2 ) … ( p − 1 , p ) . Consider one of those intervals ( k − 1 , k ) (k-1,k) ( k − 1 , k ) . For x ∈ ( k − 1 , k ) x\in(k-1,k) x ∈ ( k − 1 , k ) , h ~ ( x ) = x + 2 ∑ i = 1 k − 1 ( − 1 ) i ( x − i ) = ( − 1 ) k − 1 x − ( ( − 1 ) k − 1 − 1 ) / 2 \tilde{h}(x)=x+2\sum_{i=1}^{k-1}(-1)^{i}(x-i)=(-1)^{k-1}x-((-1)^{k-1}-1)/2 h ~ ( x ) = x + 2 ∑ i = 1 k − 1 ( − 1 ) i ( x − i ) = ( − 1 ) k − 1 x − (( − 1 ) k − 1 − 1 ) /2 . Therefore h ~ ( k − 1 ) = − ( ( − 1 ) k − 1 − 1 ) / 2 \tilde{h}(k-1)=-((-1)^{k-1}-1)/2 h ~ ( k − 1 ) = − (( − 1 ) k − 1 − 1 ) /2 and h ~ ( k ) = ( − 1 ) k − 1 ( x − ( k − 1 ) ) \tilde{h}(k)=(-1)^{k-1}(x-(k-1)) h ~ ( k ) = ( − 1 ) k − 1 ( x − ( k − 1 )) . One of them is always 0, the other is 1, so h ~ ( { k − 1 , k } ) = { 0 , 1 } \tilde{h}(\{k-1,k\})=\{0,1\} h ~ ({ k − 1 , k }) = { 0 , 1 } . Since the function is linear in the interval ( k − 1 , k ) (k-1,k) ( k − 1 , k ) , h ~ \tilde{h} h ~ maps the interval ( k − 1 , k ) (k-1,k) ( k − 1 , k ) onto the interval ( 0 , 1 ) (0,1) ( 0 , 1 ) .
The number of input-space neighborhoods that are mapped by the first L − 1 L-1 L − 1 layers onto the (open) unit hypercube ( 0 , 1 ) n 0 (0,1)^{n_{0}} ( 0 , 1 ) n 0 of the (effective) output space of the ( L − 1 ) (L-1) ( L − 1 ) -th layer is given by
N ( 0 , 1 ) n 0 L − 1 = ∏ l = 0 L − 1 ∏ i = 1 n 0 p l , i \mathcal{N}_{(0,1)^{n_{0}}}^{L-1}=\prod_{l=0}^{L-1}\prod_{i=1}^{n_{0}}p_{l,i} N ( 0 , 1 ) n 0 L − 1 = l = 0 ∏ L − 1 i = 1 ∏ n 0 p l , i
where p l , i p_{l,i} p l , i is the number of units in the i i i -th group of units in the l l l -th layer.
The inputs and bias of the last hidden layer can be chosen in such a way that the function h L h_{L} h L partitions its (effective) input neighborhood ( 0 , 1 ) n 0 (0,1)^{n_0} ( 0 , 1 ) n 0 by an arrangement of n L n_L n L hyperplanes in general position, i.e., into ∑ j = 0 n 0 ( n L j ) \sum_{j=0}^{n_0}\binom{n_L}{j} ∑ j = 0 n 0 ( j n L ) regions.
Rectifier units can either be constant or linear. The boundary between those two behaviors is given by a hyperplane. The collection of hyperplanes derived from all the units in the same layer forms a hyperplane arrangement that splits the input-space into several regions. An arrangement of n n n hyperplanes in R n 0 \mathbb{R}^{n_0} R n 0 has at most ∑ j = 0 n 0 ( n j ) \sum_{j=0}^{n_0}\binom{n}{j} ∑ j = 0 n 0 ( j n ) regions (33).
This implies that shallow rectifier neural networks with n 0 n_0 n 0 inputs and n 1 n_1 n 1 hidden units can only identify ∑ j = 0 n 0 ( n 1 j ) \sum_{j=0}^{n_0}\binom{n_1}{j} ∑ j = 0 n 0 ( j n 1 ) different regions.
Let m l m_l m l denote the remainder of n l n_l n l divided by n 0 n_0 n 0 . Choosing p l , 1 = … = p l , n 0 − m l = ⌊ n l / n 0 ⌋ p_{l,1}=\ldots=p_{l,n_0-m_l}=\lfloor n_l/n_0\rfloor p l , 1 = … = p l , n 0 − m l = ⌊ n l / n 0 ⌋ and p l , n 0 − m l + 1 = … = p l , n 0 = ⌊ n l / n 0 ⌋ + 1 p_{l,n_0-m_l+1}=\ldots=p_{l,n_0}=\lfloor n_l/n_0\rfloor+1 p l , n 0 − m l + 1 = … = p l , n 0 = ⌊ n l / n 0 ⌋ + 1 , we obtain a total of linear regions
( ∏ l = 1 L − 1 ⌊ n l n 0 ⌋ n 0 − m l ( ⌊ n l n 0 ⌋ + 1 ) m l ) ∑ j = 0 n 0 ( n L j ) \left(\prod_{l=1}^{L-1}\left\lfloor\frac{n_l}{n_0}\right\rfloor^{n_0-m_l}\left(\left\lfloor\frac{n_l}{n_0}\right\rfloor+1\right)^{m_l}\right)\sum_{j=0}^{n_0}\binom{n_L}{j} ( l = 1 ∏ L − 1 ⌊ n 0 n l ⌋ n 0 − m l ( ⌊ n 0 n l ⌋ + 1 ) m l ) j = 0 ∑ n 0 ( j n L )
This is equal to the bound given in theorem when all remainders m l = n l − n 0 ⌊ n l / n 0 ⌋ m_l=n_l-n_0\lfloor n_l/n_0\rfloor m l = n l − n 0 ⌊ n l / n 0 ⌋ are zero. Otherwise, it is larger. QED.
The previous result, stated in terms of the maximal number of linear regions of the functions computable by deep neural networks, can be expressed in terms of the number of parameters of neural networks:
O ( L n 0 − 1 n n 0 − 1 ) O\left(L^{n_0-1}n^{n_0-1}\right) O ( L n 0 − 1 n n 0 − 1 )
Ω ( ( n n 0 ) n 0 ( L − 1 ) n n 0 − 2 L ) \Omega\left(\left(\frac{n}{n_0}\right)^{n_0(L-1)}\frac{n^{n_0-2}}{L}\right) Ω ( ( n 0 n ) n 0 ( L − 1 ) L n n 0 − 2 )
Proof. For shallow models:
R ≥ ∑ j = 0 n 0 ( n j ) ∈ Θ ( n n 0 ) R\geq\sum_{j=0}^{n_0}\binom{n}{j}\in\Theta(n^{n_0}) R ≥ j = 0 ∑ n 0 ( j n ) ∈ Θ ( n n 0 )
given that
( n n 0 ) = n n 0 n 0 ! + O ( n n 0 − 1 ) \binom{n}{n_0}=\frac{n^{n_0}}{n_0!}+O(n^{n_0-1}) ( n 0 n ) = n 0 ! n n 0 + O ( n n 0 − 1 )
Number of parameters: A single-hidden-layer network with input dimension n 0 n_0 n 0 , width n n n , and output dimension 1 1 1 has P = ( n 0 + 1 ) n + ( n + 1 ) ∈ Θ ( n ) P=(n_0+1)n+(n+1)\in\Theta(n) P = ( n 0 + 1 ) n + ( n + 1 ) ∈ Θ ( n ) parameters, Θ ( L n ) \Theta(Ln) Θ ( L n ) given that L = 1 L=1 L = 1
R P ∈ Θ ( n n 0 L n ) = Θ ( L − 1 n n 0 − 1 ) \frac{R}{P}\in\Theta\left(\frac{n^{n_0}}{Ln}\right)=\Theta\left(L^{-1}n^{n_0-1}\right) P R ∈ Θ ( L n n n 0 ) = Θ ( L − 1 n n 0 − 1 )
⊂ O ( L − 1 n n 0 − 1 ) \subset O\left(L^{-1}n^{n_0-1}\right) ⊂ O ( L − 1 n n 0 − 1 )
For deep models:
R ≥ ( ⌊ n n 0 ⌋ ) L − 1 ∑ j = 0 n 0 ( n j ) R\geq\left(\left\lfloor\frac{n}{n_0}\right\rfloor\right)^{L-1}\sum_{j=0}^{n_0}\binom{n}{j} R ≥ ( ⌊ n 0 n ⌋ ) L − 1 j = 0 ∑ n 0 ( j n )
Since
⌊ n n 0 ⌋ ∈ Θ ( n n 0 ) \left\lfloor\frac{n}{n_0}\right\rfloor\in\Theta\left(\frac{n}{n_0}\right) ⌊ n 0 n ⌋ ∈ Θ ( n 0 n )
and
∑ j = 0 n 0 ( n j ) ∈ Θ ( n n 0 ) \sum_{j=0}^{n_0}\binom{n}{j}\in\Theta(n^{n_0}) j = 0 ∑ n 0 ( j n ) ∈ Θ ( n n 0 )
then
R ∈ Ω ( ( n n 0 ) n 0 ( L − 1 ) n n 0 ) R\in\Omega\left(\left(\frac{n}{n_0}\right)^{n_0(L-1)}n^{n_0}\right) R ∈ Ω ( ( n 0 n ) n 0 ( L − 1 ) n n 0 )
R P ∈ Ω ( ( n n 0 ) n 0 ( L − 1 ) n n 0 L n 2 ) = Ω ( ( n n 0 ) n 0 ( L − 1 ) n n 0 − 2 L ) \begin{aligned}
\frac{R}{P}\in\Omega\left(\frac{\left(\frac{n}{n_0}\right)^{n_0(L-1)}n^{n_0}}{Ln^2}\right) \\
&=\Omega\left(\left(\frac{n}{n_0}\right)^{n_0(L-1)}\frac{n^{n_0-2}}{L}\right)
\end{aligned} P R ∈ Ω L n 2 ( n 0 n ) n 0 ( L − 1 ) n n 0 = Ω ( ( n 0 n ) n 0 ( L − 1 ) L n n 0 − 2 )
Shallow networks achieve only polynomial growth in the number of linear regions per parameter, whereas deep networks achieve an exponential growth in depth L L L , demonstrating the substantially greater expressivity of deep ReLU architectures.
References