betaFeedback

WorksComputer Science

Projected Neural Additive Models as Universal Approximators

Abstract

Neural additive models (NAMs) offer interpretability but lack expressivity due to their rigid additive structure, whereas multi-layer perceptrons (MLPs) achieve universal approximation at the cost of transparency. We establish the first theoretical foundation for projected neural additive models (PNAMs), which augment NAMs with a learnable linear transformation T\boldsymbol{T} to capture cross-variable interactions. While standard NAMs cannot approximate even the simplest interaction term χ1χ2χ_1 χ_2 because their cross-derivatives are identically zero, we prove that PNAMs are universal approximators and characterize the minimum number of projections required for exact polynomial representation. Furthermore, we derive an approximation rate of O(M−s/(N−1))\mathcal{O}(M^{-s / (N - 1)}) for Cs\mathcal{C}^s functions (i.e., functions with ss continuous derivatives), where MM denotes the number of projections and NN the input dimension; this rate is known to be optimal for ridge function approximation. To recover interpretability from the projected inputs, we introduce structured sparsity regularizers and post-hoc symbolic regression techniques that enable global input ranking, parameter pruning, and the conversion of learned bases into compact analytical expressions. Experiments on knot theory invariants, MNIST, and phase field fracture mechanics demonstrate that PNAMs match or surpass similar architectures in accuracy while discovering salient features and yielding symbolic models with orders of magnitude fewer parameters.

Introduction

While deep neural networks have been widely adopted across diverse tasks due to their expressivity, they inherently lack interpretability [1]. Among the myriad methods introduced to address this issue, Agarwal et al. [2] propose the neural additive model (NAM), an architecture constrained to a linear combination of single-variable functions parameterized by independent multi-layer perceptrons (MLPs). Although this decoupled structure provides feature-wise interpretability, the strictly additive nature of NAMs limits their expressivity; in particular, their cross-derivatives are identically zero, rendering them incapable of representing functions with nontrivial input interactions.

To overcome this limitation, Phan et al. [3] introduce a learnable linear transformation TT preceding the NAM (see Figure 1)—an architecture we refer to as the projected neural additive model (PNAM). A complementary line of work—GA²M [4], NODE-GA²M [5], GAMI-Net [6], SPAM [7], and SIAN [8]—augments generalized additive models (GAMs) [9] and NAMs with explicit interaction terms such as F(χi,χj)\mathcal{F}(\chi_i,\chi_j) acting on pairs of inputs. Interpretability is then pairwise, but the number of interaction terms grows combinatorially with the number of inputs NN, and the interaction order is capped by construction: a model limited to pairwise terms cannot represent, e.g., χ1χ2χ3\chi_1\chi_2\chi_3 and is therefore not a universal approximator. If the ground truth is dominated by a few low-order interactions of raw features, such explicit models are more directly interpretable. The PNAM is instead preferable when interactions are higher-order, dense in a few directions, or unknown a priori, as it captures them along MM learned directions, with MM far below the N(N−1)/2N(N - 1)/2 pairs that a pairwise model enumerates. While the transformation TT empirically enhances expressivity by capturing cross-variable couplings, the PNAM architecture was introduced without a rigorous theoretical foundation or a formal proof of universal approximation.

Architecture of the PNAM

Figure 1. Architecture of the PNAM. The output y^\hat{y} is predicted via a linear combination of single-variable functions, parameterized by independent MLPs, which are evaluated on linear combinations of the inputs χ\chi. The transformation TT and weighting coefficients ζ\zeta (both denoted with dashed lines) can be optimized to yield sparser (and more interpretable) solutions.

Contributions. Our core contributions establish this missing foundation alongside novel mechanisms for interpretability and sparsity. First, we prove that PNAMs are universal approximators (Theorem 2.1)—a fundamental guarantee previously missing for this architecture class. Second, we characterize the projection complexity of TT (Appendix § 8.§ 1) by establishing constructive upper bounds and a parameter-counting lower bound on the minimum number of projections MM required to represent any degree-PP polynomial in NN variables exactly. Third, building on these results, we derive an approximation rate of O(M−s/(N−1))O(M^{-s/(N-1)}) for CsC^s functions (Theorem 2.2), a rate known to be optimal for ridge function approximation [10, 11].

To mitigate the reduction in interpretability induced by TT, we introduce structured regularization techniques that enable global feature ranking, penalize unnecessary input interactions, and promote sparse solutions. While standard NAMs provide feature-level interpretability via their shape functions, our regularized PNAMs offer global input ranking across the full dataset [12, 13]. Moreover, we leverage the modularity of PNAMs to prune nonessential parameters and extract compact symbolic expressions from the learned single-variable bases. Empirically, experiments on knot theory invariants confirm that PNAMs recover accurate analytical expressions comparable to Kolmogorov–Arnold networks (KANs) [14], entirely without prior knowledge of the relevant invariants.

Related work. The PNAM belongs to a growing class of architecture-based models designed to enhance the interpretability of deep neural networks without compromising expressivity. Prominent alternatives include KANs [14], deep polynomial neural networks [15], and graph neural networks with inductive biases [16], alongside related interpretability methods such as SINDy [17], structural pruning [18], and gradient-based attribution [19]. Compared to KANs—perhaps their closest architectural relative3—PNAMs offer a distinct and complementary theoretical profile. While KANs leverage the Kolmogorov–Arnold representation theorem (which necessitates two layers of nonlinearities), PNAMs utilize a single linear transformation followed by 1D nonlinearities. This simpler structure enables the approximation rate of Theorem 2.2.

In addition, we emphasize that although we have equipped PNAMs with post-hoc symbolic regression (SR) capabilities, this technique is primarily employed for pruning. Like most additive architectures, PNAMs are not designed to recover exact physical laws with precise functional forms; their additive nature prevents them from compactly approximating operations such as division,4 and the number of projections required to represent complex interactions can be prohibitively large. Consequently, we reserve the recovery of exact physical laws for dedicated SR algorithms leveraging reinforcement learning [21], physics-inspired strategies [22], and genetic programming [23]. Further discussion on interpretability and optimization constraints is provided in § 4.

Projected neural additive models

Given a training dataset {X,y}={(χν,yν)}ν=1D\{\mathcal{X},y\} = \{(\chi_\nu,y_\nu)\}_{\nu=1}^{D} containing DD input-output pairs, where χ∈RN\chi\in\mathbb{R}^{N} is an input vector of NN independent variables and y∈Ry \in\mathbb{R} is the corresponding scalar label, the goal of supervised learning is to construct an approximating function F:RN→R\mathcal{F} : \mathbb{R}^{N} \to\mathbb{R}. We hypothesize that this multi-variable function can be approximated by a weighted sum of MM single-variable functions {fi}i=1M\{f_i\}_{i=1}^{M} acting on a projected space:

y^=∑i=1Mζifi(∑j=1NTijχj)+ϵ=∑i=1Mgi(zi)+ϵ,(1)\hat{y} = \sum_{i=1}^{M} \zeta_i f_i\left(\sum_{j=1}^{N} T_{ij}\chi_j\right) + \epsilon= \sum_{i=1}^{M} g_i(z_i) + \epsilon, \tag*{(1)}

where y^\hat{y} is the predicted value and ϵ\epsilon represents an additive noise term. The projected variables z∈RMz \in\mathbb{R}^{M} in Eq. (1) result from a linear transformation T:RN→RMT : \mathbb{R}^{N} \to\mathbb{R}^{M} of the inputs, such that z=Tχz = T\chi. For compactness, each feature function fi:R→Rf_i : \mathbb{R} \to\mathbb{R} and its corresponding weighting coefficient ζi\zeta_i are combined into a composite function gi:R→Rg_i : \mathbb{R} \to\mathbb{R}.

The single-variable functions {fi}\{f_i\} can be implemented using various bases (e.g., polynomials or neural networks); here, we parameterize the output as a linear combination of MLPs:

y^=∑i=1MζiMLP⁡i(∑j=1NTijχj;θi)+ϵ,(2)\hat{y} = \sum_{i=1}^{M} \zeta_i \operatorname{MLP}_i\left(\sum_{j=1}^{N} T_{ij}\chi_j;\theta_i\right) + \epsilon, \tag*{(2)}

where each fif_i is an LL-layer MLP with a set of learnable weights and biases θi={Wi(1),…,Wi(L),si(1),…,si(L−1)}\theta_i = \{W_i^{(1)},\ldots,W_i^{(L)},s_i^{(1)},\ldots,s_i^{(L-1)}\} and an element-wise activation function AA:

MLP⁡i(zi;θi)=Wi(L)A(Wi(L−1)A(…A(Wi(1)zi+si(1))…)+si(L−1)).(3)\operatorname{MLP}_i(z_i;\theta_i) = W_i^{(L)} A\left(W_i^{(L-1)} A\left(\ldots A\left(W_i^{(1)}z_i+s_i^{(1)}\right)\ldots\right)+s_i^{(L-1)}\right). \tag*{(3)}

To effectively capture high-frequency components, we leverage a Fourier feature mapping [24, 25] that transforms each projected variable ziz_i into

γi(zi)=[cos⁡(2πBizi)T,sin⁡(2πBizi)T]T,(4)\gamma_i(z_i) = [\cos(2\pi B_i z_i)^{\mathsf{T}},\sin(2\pi B_i z_i)^{\mathsf{T}}]^{\mathsf{T}}, \tag*{(4)}

where entries in BiB_i are sampled from a Gaussian distribution N(0,σ2)\mathcal{N}(0,\sigma^2), with standard deviation σ\sigma, and fixed after initialization. When this mapping is used, the MLP in Eq. (3) is evaluated on γi(zi)\gamma_i(z_i) rather than the scalar ziz_i. As illustrated in Figure 1, the transformation TT, weighting coefficients ζ={ζi}i=1M\zeta= \{\zeta_i\}_{i=1}^{M}, and error term ϵ\epsilon in Eq. (2) are encoded as two weight matrices and a bias term, respectively. This construction formalizes the PNAM architecture, extending the foundational and empirical frameworks established by Agarwal et al. [2] and Phan et al. [3].

Remark 1 (Comparison with NAMs and MLPs). In a PNAM, each basis gig_i in Eq. (1) is a function of a transformed variable ziz_i, rather than a raw input χj\chi_j as in NAMs or other GAMs [9, 26]. The linear transformation TT enables the model to capture cross-variable couplings (e.g., χ1χ2\chi_1\chi_2 in Example A.1) that standard NAMs cannot. Although this projection precludes the direct visualization of input contributions typical of NAMs, we demonstrate in § 2.4 how the static matrix TT can be examined to perform global input ranking and identify critical interactions. This structured interpretability—encompassing feature pruning and dimensionality reduction—is generally absent in black-box, fully connected MLPs.

Universal approximation and approximation rate

By the Stone–Weierstrass theorem [27, 28, 29], polynomials are dense in the space of continuous functions C(D)\mathcal{C}(\mathcal{D}) for a compact domain D⊂RN\mathcal{D} \subset\mathbb{R}^{N}, meaning they can approximate any continuous function to arbitrary precision. Consequently, we can establish universal approximation for PNAMs by demonstrating that they can reproduce arbitrary multi-variable polynomials. The key insight is that the learnable linear transformation TT enables PNAMs to express these polynomials via a superposition of ridge functions—functions of the form gi(∑jTijχj)g_i\left(\sum_j T_{ij}\chi_j\right)—whereas NAMs, restricted to zi=χiz_i=\chi_i (the identity projection), cannot represent even the degree-2 cross-term χ1χ2\chi_1\chi_2 and therefore fail to reproduce arbitrary polynomials. We show in Theorem A.1 that single-variable polynomials of projected inputs can serve as the bases to reproduce any multi-variable polynomial, providing the constructive foundation for the general approximation results that follow.

Alternatively, instead of single-variable polynomials, one can employ a sum of 1D neural networks acting on the projected inputs to achieve universal approximation. Since these networks satisfy the 1D universal approximation property [30, 31, 32], they can readily approximate the required single-variable polynomials used to construct the multi-variable approximation. This, by extension, guarantees the universal approximation property of the resulting multi-dimensional PNAM architecture. This capability is formally stated in the following theorem.

Theorem 2.1 (Universal approximation). Let G\mathcal{G} be a set of single-variable polynomials or 1D neural networks. Given any function F1\mathcal{F}_1 in the set of continuous real-valued functions C(D)\mathcal{C}(\mathcal{D}) for a compact domain D⊂RN\mathcal{D} \subset\mathbb{R}^{N} and any ε>0\varepsilon> 0, there exist a linear transformation T:RN→RMT : \mathbb{R}^{N} \to\mathbb{R}^{M}, where M≥1M \ge1, and a set of functions {gi:R→R}i=1M\{g_i : \mathbb{R} \to\mathbb{R}\}_{i=1}^{M} in G\mathcal{G} such that

∣F1(χ)−∑i=1Mgi(∑j=1NTijχj)∣<ε,(5)\left|\mathcal{F}_1(\chi)-\sum_{i=1}^{M}g_i\left(\sum_{j=1}^{N}T_{ij}\chi_j\right)\right|<\varepsilon, \tag*{(5)}

for all χ∈D\chi\in\mathcal{D}.

Proof. Since D\mathcal{D} is compact, the Stone–Weierstrass theorem applies [27, 28, 29], that is, polynomials are dense in C(D)\mathcal{C}(\mathcal{D}). As a result, there exists a (multi-variable) polynomial F2\mathcal{F}_2 such that

∣F1(χ)−F2(χ)∣<ε/2.(6)|\mathcal{F}_1(\chi)-\mathcal{F}_2(\chi)|<\varepsilon/2. \tag*{(6)}

Then for F2\mathcal{F}_2, by Theorem A.1, we have

F2(χ)=∑i=1Mg~i(∑j=1NTijχj),(7)\mathcal{F}_2(\chi)=\sum_{i=1}^{M}\widetilde{g}_i\left(\sum_{j=1}^{N}T_{ij}\chi_j\right), \tag*{(7)}

with single-variable polynomials {g~i}\{\widetilde{g}_i\}. Note that Theorem A.1 covers any polynomial F2\mathcal{F}_2, establishing the existence of a transformation TT and functions {g~i}\{\widetilde{g}_i\} that reproduce it exactly. If G\mathcal{G} is a set of polynomials, we are done; if G\mathcal{G} represents neural networks, we treat the polynomials {g~i}\{\widetilde{g}_i\} as intermediate target functions. By the universal approximation theorem for neural networks [30, 31, 32], for each ii, there exists a network gig_i with

∥g~i−gi∥C(Di)<ε2M,(8)\lVert\widetilde{g}_i-g_i\rVert_{\mathcal{C}(\mathcal{D}_i)}<\frac{\varepsilon}{2M}, \tag*{(8)}

where ∥⋅∥C(Di)\lVert\cdot\rVert_{\mathcal{C}(\mathcal{D}_i)} denotes the supremum norm over Di\mathcal{D}_i, the image of D\mathcal{D} under the iith projection. Summing overall i=1,…,Mi=1,\ldots,M and applying the triangle inequality yield

∣F2(χ)−∑i=1Mgi(∑j=1NTijχj)∣≤∑i=1M∥g~i−gi∥C(Di)<M⋅ε2M=ε2.\left|\mathcal{F}_2(\chi)-\sum_{i=1}^{M}g_i\left(\sum_{j=1}^{N}T_{ij}\chi_j\right)\right| \leq\sum_{i=1}^{M}\lVert\widetilde{g}_i-g_i\rVert_{\mathcal{C}(\mathcal{D}_i)} < M\cdot\frac{\varepsilon}{2M} =\frac{\varepsilon}{2}.

Combining this result with Eq. (6) via the triangle inequality gives Eq. (5), completing the proof. □\square

Theorem 2.1 establishes that PNAMs can approximate any continuous function to arbitrary precision, but does not quantify the rate at which the approximation error decreases with the number of projections MM. Leveraging the polynomial reproducing property of Theorem A.1 and the classical Jackson theorem [33], the following theorem derives an explicit approximation rate.

Theorem 2.2 (Approximation rate). Let F1∈Cs(D)\mathcal{F}_1 \in C^s(\mathcal{D}) for a compact domain D⊂RN\mathcal{D} \subset\mathbb{R}^N with s≥1s \ge1 and ∥F1∥Cs≤B\|\mathcal{F}_1\|_{C^s} \le B. For any M≥1M \ge1, there exists a PNAM F^M\widehat{\mathcal{F}}_M with MM projections satisfying

∥F1−F^M∥∞≤C1M−s/(N−1),(9)\|\mathcal{F}_1-\widehat{\mathcal{F}}_M\|_\infty\le C_1 M^{-s/(N-1)}, \tag*{(9)}

where C1C_1 depends on D\mathcal{D}, ss, NN, and BB.

Proof. By Jackson’s theorem [33], there exists a polynomial F2\mathcal{F}_2 of degree PP such that

∥F1−F2∥∞≤C2P−s,(10)\|\mathcal{F}_1-\mathcal{F}_2\|_\infty\le C_2 P^{-s}, \tag*{(10)}

where C2C_2 depends on D\mathcal{D}, ss, NN, and BB. By Theorem A.1, F2\mathcal{F}_2 can be represented exactly by a PNAM with M≤(P+1)N−1M \le(P+1)^{N-1} projections. Inverting this relation gives P≥M1/(N−1)−1P \ge M^{1/(N-1)}-1. Thus, for large MM, the asymptotic scaling satisfies P≍M1/(N−1)P \asymp M^{1/(N-1)}. Substituting this result into the Jackson bound from Eq. (10) yields

∥F1−F2∥∞≤C2P−s≤C1M−s/(N−1),\|\mathcal{F}_1-\mathcal{F}_2\|_\infty\le C_2 P^{-s} \le C_1 M^{-s/(N-1)},

where C2C_2 and factors from the inversion are absorbed into C1C_1. Because F2\mathcal{F}_2 is represented exactly by the PNAM F^M\widehat{\mathcal{F}}_M, the approximation error ∥F2−F^M∥∞\|\mathcal{F}_2-\widehat{\mathcal{F}}_M\|_\infty vanishes. Applying the triangle inequality:

∥F1−F^M∥∞≤∥F1−F2∥∞+∥F2−F^M∥∞≤C1M−s/(N−1),\|\mathcal{F}_1-\widehat{\mathcal{F}}_M\|_\infty\le\|\mathcal{F}_1-\mathcal{F}_2\|_\infty+\|\mathcal{F}_2-\widehat{\mathcal{F}}_M\|_\infty\le C_1 M^{-s/(N-1)},

which completes the proof.5

Remark 2 (Choice of projection bound). The approximation rate of O(M−s/(N−1))O(M^{-s/(N-1)}) in Theorem 2.2 follows directly from the bound on MM established in Theorem A.1. Using the alternative bounds from Theorems A.2 or A.3 would produce weaker results: Theorem A.3 gives M≍PN/N!M \asymp P^N/N!, which recovers the classical polynomial approximation rate of O(M−s/N)O(M^{-s/N}) [11, 33], while the exponential scaling of Theorem A.2 yields only a logarithmic rate of O((log⁡M)−s)O((\log M)^{-s}). Theorem A.1 therefore provides the best approximation rate among the three bounds, since it requires the fewest projections for a given polynomial degree PP in the large-PP regime. Moreover, Maiorov [10] proves that O(M−s/(N−1))O(M^{-s/(N-1)}) is the optimal rate for ridge function approximation of Sobolev-class functions, confirming that the bound in Theorem A.1 is tight in this asymptotic sense.

Remark 3 (Parameter complexity). To achieve approximation error ε\varepsilon, Theorem 2.2 requires M≍ε−(N−1)/sM \asymp\varepsilon^{-(N-1)/s} projections. To achieve a matching error, each gig_i approximating a single-variable function in CsC^s is a network with width W≍ε−1/(2s)W \asymp\varepsilon^{-1/(2s)}. Thus, the parameter complexity of PNAMs is

O(MN+MLW2)≍O(ε−(N−1)/s⋅ε−1/s)=O(ε−N/s)O(MN+MLW^2) \asymp O(\varepsilon^{-(N-1)/s}\cdot\varepsilon^{-1/s})=O(\varepsilon^{-N/s})

for fixed depth LL, where the MNMN term is dominated by MLW2MLW^2 as ε→0\varepsilon\to0. This result matches the optimal complexity established for standard MLPs [34].

Extension to multiple outputs

To accommodate multiple outputs, let y={{yiν}ν=1D}i=1K\mathbf{y}=\{\{y_{i\nu}\}_{\nu=1}^{D}\}_{i=1}^{K}, ζ={{ζij}j=1Mj}i=1K\boldsymbol{\zeta}=\{\{\zeta_{ij}\}_{j=1}^{M_j}\}_{i=1}^{K}, and ϵ={ϵi}i=1K\boldsymbol{\epsilon}=\{\epsilon_i\}_{i=1}^{K}, with KK denoting the number of dependent variables. We extend Eq. (1) to represent each output y^i\widehat{y}_i as

y^i=∑j=1Mζijfij(∑k=1NTjkxk)+ϵi=∑j=1Mgij(zj)+ϵi,(11)\widehat{y}_i=\sum_{j=1}^{M}\zeta_{ij}f_{ij}\left(\sum_{k=1}^{N}T_{jk}x_k\right)+\epsilon_i=\sum_{j=1}^{M}g_{ij}(z_j)+\epsilon_i, \tag*{(11)}

where fij:R→Rf_{ij}:\mathbb{R}\to\mathbb{R} is the single-variable function associated with the jthj^{\mathrm{th}} projection for the ithi^{\mathrm{th}} output [35]. To see that Eq. (11) is a natural extension of Eq. (1), consider the case where the bases {gij}\{g_{ij}\} share no common inputs {zj}\{z_j\} across different outputs {y^i}\{\widehat{y}_i\}. In this scenario, the total number of projections is M=∑i=1KmiM=\sum_{i=1}^{K}m_i, with mim_i denoting the number of projections required to approximate each y^i\widehat{y}_i independently. Equivalently, for each index jj, both the coefficients {ζij}\{\zeta_{ij}\} and functions {fij}\{f_{ij}\} are nonzero only for a disjoint subset of indices {j}\{j\} of size mim_i, yielding KK independent basis sets. By extension of Theorem 2.1, MM can be chosen sufficiently large to approximate any KK-tuple of continuous functions on a compact domain.

Learning problem and constraints for sparsity

Let Θ={θj}j=1M\Theta=\{\theta_j\}_{j=1}^{M} denote all learnable parameters of the 1D MLPs. We optimize the parameters of PNAMs by minimizing the following loss function (in parentheses) for DD training samples:

T∗,Θ∗,ζ∗,ϵ∗=arg⁡min⁡T,Θ,ζ,ϵ(L(y,y^(X;T,Θ,ζ,ϵ))+LP+1D(w1ℓ1+w2ℓ2)+1M(w3ℓ3+w4ℓ4+w5ℓ5))(12)T^*,\Theta^*,\zeta^*,\epsilon^*=\underset{T,\Theta,\zeta,\epsilon}{\arg\min}\left(\mathcal{L}(y,\hat{y}(X;T,\Theta,\zeta,\epsilon))+\mathcal{L}_{\mathrm{P}}+\frac{1}{D}(w_1\ell_1+w_2\ell_2)+\frac{1}{M}(w_3\ell_3+w_4\ell_4+w_5\ell_5)\right) \tag*{(12)}

where the empirical loss L\mathcal{L} is defined as

L=−1D∑i=1K∑ν=1Dyiνlog⁡exp⁡(y^i(χν;T,Θ,ζ,ϵ))∑l=1Kexp⁡(y^l(χν;T,Θ,ζ,ϵ))\mathcal{L}=-\frac{1}{D}\sum_{i=1}^{K}\sum_{\nu=1}^{D}y_{i\nu}\log\frac{\exp(\hat{y}_i(\chi_\nu;T,\Theta,\zeta,\epsilon))}{\sum_{l=1}^{K}\exp(\hat{y}_l(\chi_\nu;T,\Theta,\zeta,\epsilon))}

for classification (cross-entropy), or

L=1KD∑i=1K∑ν=1D(yiν−y^i(χν;T,Θ,ζ,ϵ))2\mathcal{L}=\frac{1}{KD}\sum_{i=1}^{K}\sum_{\nu=1}^{D}(y_{i\nu}-\hat{y}_i(\chi_\nu;T,\Theta,\zeta,\epsilon))^2

for regression (mean squared error). The term LP\mathcal{L}_{\mathrm{P}} optionally imposes physical constraints that restrict the space of admissible solutions [36, 37, 38].

To prevent overfitting and produce interpretable solutions, we employ two structured regularizers:

ℓ1=∥Θ∥2,ℓ2=∥{{{gij(χν;T,Θ,ζ)}ν=1D}j=1M}i=1K∥2,(13)\ell_1=\|\Theta\|_2,\qquad\ell_2=\left\|\left\{\left\{\left\{g_{ij}(\chi_\nu;T,\Theta,\zeta)\right\}_{\nu=1}^{D}\right\}_{j=1}^{M}\right\}_{i=1}^{K}\right\|_2, \tag*{(13)}

where ℓ1\ell_1 is the standard L2L_2 regularization of the network weights and biases [39].6 and ℓ2\ell_2 discourages the bases {gij}\{g_{ij}\} from taking on disproportionately large values [2]. Given that the inputs and outputs are typically normalized, this constraint underpins the global input ranking of § 2.4, in which the columns of the linear transformation TT serve as a proxy for predictive importance.

To further promote sparsity and disentanglement, we analyze the singular value decomposition (SVD) of TT:

T=Q1ΣQ2T,T=Q_1\Sigma Q_2^{\mathsf{T}},

where Q1Q_1 and Q2Q_2 are orthonormal matrices of dimensions M×MM\times M and N×NN\times N, respectively, and Σ\Sigma is an M×NM\times N diagonal matrix containing the min⁡(M,N)\min(M,N) singular values. Drawing on the metric

Φ(R1,R2)=∥I−R1R2T∥F=2(3−tr⁡(R1R2T)),R1,R2∈SO(3),\Phi(R_1,R_2)=\left\|I-R_1R_2^{\mathsf{T}}\right\|_{\mathrm{F}}=\sqrt{2\left(3-\operatorname{tr}(R_1R_2^{\mathsf{T}})\right)},\qquad R_1,R_2\in SO(3),

used to measure the distance between two 3D rotations [40], we define a joint penalty for the deviation of Q1Q_1 and Q2Q_2 from the identity matrix II:

ℓ3=Φ(I,Q1)+Φ(I,Q2)=2(M−tr⁡Q1)+2(N−tr⁡Q2).(14)\ell_3=\Phi(I,Q_1)+\Phi(I,Q_2)=\sqrt{2(M-\operatorname{tr}Q_1)}+\sqrt{2(N-\operatorname{tr}Q_2)}. \tag*{(14)}

This term effectively penalizes unnecessary couplings between the inputs; in isolation, ℓ3\ell_3 is minimized when the transformation TT is a rectangular diagonal matrix. This shrinkage toward the axis-aligned NAM subclass—the most interpretable member of the PNAM family—acts as a structured prior: since ℓ3\ell_3 is dominated by the empirical loss L\mathcal{L}, cross-variable couplings are introduced only when they reduce L\mathcal{L} by more than the cost they incur.7 Finally, we apply L1L_1 sparsity constraints on TT and ζ\zeta to drive nonessential interactions and bases to zero [25, 35, 41]:

ℓ4=∥T∥1,ℓ5=∥ζ∥1.(15)\ell_4=\|T\|_1,\qquad\ell_5=\|\zeta\|_1. \tag*{(15)}

Post-processing and symbolic regression

Following training, we apply three post-processing techniques to prune the PNAM and enhance interpretability [1, 42].

The first technique exploits the regularizer ℓ2\ell_2 in Eq. (13). The column-wise mean of the absolute entries in the linear transformation TT—defined as 1M∑j=1M∣Tjk∣\frac{1}{M}\sum_{j=1}^{M}|T_{jk}| and termed the mean absolute coefficients—measures how strongly each input is used in the learned projections. When the bases {gij}\{g_{ij}\} are constrained by ℓ2\ell_2 and the variables are normalized, {gij}\{g_{ij}\} become comparable in magnitude across projections, so a column of TT is large only if this reduces the empirical loss L\mathcal{L}. The mean absolute coefficients then serve as an empirical proxy for predictive importance (validated in Appendix § 8.§ 2) rather than a guaranteed property of the PNAM architecture. We can thus retain only the top n≤Nn \le N features and zero out the columns of TT associated with the (N−n)(N-n) least significant inputs. The second technique applies hard thresholding to the transformation TT and weighting coefficients ζ\zeta, zeroing out entries where ∣Tjk∣<T0|T_{jk}| < T_0 and ∣ζij∣<ζ0|\zeta_{ij}| < \zeta_0 for user-defined thresholds T0T_0 and ζ0\zeta_0.8

Finally, the third technique leverages the single-variable nature of the bases {gij}\{g_{ij}\} to convert them into explicit symbolic expressions. While various SR algorithms are compatible (such as DSR [21] or AI Feynman [22]), we employ PySR [23], which utilizes genetic programming [43, 44], for its efficiency and ease of use. To distill the ithi^{\mathrm{th}} output of the jthj^{\mathrm{th}} MLP into symbolic form, we minimize the following objective for fixed ii and jj:

gij∗=arg⁡min⁡gij(1D∑ν=1D((ζijMLPij(zj;θj)−gij(zj))∣ν)2+w6((ζijdMLPij(zj;θj)dzj−dgij(zj)dzj)∣ν)2)(16)g_{ij}^{\ast}=\underset{g_{ij}}{\arg\min}(\frac{1}{D}\sum_{\nu=1}^{D}((\zeta_{ij}\mathrm{MLP}_{ij}(z_j;\theta_j)-g_{ij}(z_j))|_{\nu})^2+w_6((\zeta_{ij}\frac{\mathrm{d}\mathrm{MLP}_{ij}(z_j;\theta_j)}{\mathrm{d}z_j}-\frac{\mathrm{d}g_{ij}(z_j)}{\mathrm{d}z_j})|_{\nu})^2) \tag*{(16)}

where gijg_{ij} is a mathematical expression of a single variable zjz_j, and w6w_6 controls the weight placed on matching the derivatives of the learned MLP. Summing the resulting {gij∗}\{g_{ij}^{\ast}\} over jj and adding the bias ϵi\epsilon_i recovers the global output y^i\hat{y}_i as defined in Eq. (11). These post-processing steps can reduce tens of thousands of network parameters to as few as tens or hundreds, effectively bypassing the NP-hardness of multi-variable SR [21, 45] while retaining the predictive power of deep neural networks.

Numerical experiments

We illustrate the expressivity and interpretability of PNAMs through a benchmark study on an extensive dataset of mathematical knots from [46], covering (i) multi-class classification and (ii) single-output regression. In both tasks, PNAMs recover the same key invariants identified by previous studies [14, 46] and produce compact symbolic expressions without prior knowledge of the relevant invariants. Additional results for knot theory are provided in Appendices § 8.§ 2 and § 8.§ 3. Experiments on the MNIST dataset [47], provided in Appendix § 8.§ 4, further demonstrate the feature selection and dimensionality reduction capabilities of PNAMs for high-dimensional image classification. Finally, we examine the benign overfitting behavior of PNAMs as the projection dimension MM increases using a phase field simulation of fracture propagation [48] with a limited dataset (Appendix § 8.§ 5).

Unless otherwise stated, we employ an 80/20 train-test split; the training set is further divided into training and validation subsets (80/20). The PNAM is implemented in PyTorch [49] using SiLU9 [50, 51] as the activation function AA in Eq. (3). We initialize the projection dimension to M=min⁡(8,N)M=\min(8,N) and adjust it based on the complexity of the task.

Unless otherwise stated, the regularization weights w1,…,w5w_1,\ldots,w_5 in Eq. (12) are set to 0.01 for classification and 0.001 for regression tasks.10 By default, the optional physical constraint term LP\mathcal{L}_{P} is omitted, and the regularization weight w6w_6 in Eq. (16) is set to zero. We utilize a batch size of 256 samples11 and the Adam optimizer [52] with an initial learning rate of 0.001 that decays by a factor of 0.995 per epoch to train all networks. All training is conducted on a single NVIDIA A100-SXM4-40GB GPU. Once trained, each basis gijg_{ij} is evolved for one minute using 30 populations of 30 expressions with a maximum complexity of 30; all operators and leaf nodes are assigned a complexity of one. Other relevant hyperparameters and training details are reported alongside the results.

Benchmarking with knot theory

Established by a team of researchers mostly from Google DeepMind (DM) [46], the dataset of mathematical knots consists of 243,746 samples, each possessing 17 geometric invariants: adjoint torsion degree, torsion degree, short geodesic (real part), short geodesic (imaginary part), injectivity radius, Chern–Simons invariant, cusp volume, longitudinal translation, meridional translation (imaginary part), meridional translation (real part), volume, and six symmetry groups. The goal is to use these 17 invariants to predict the knot signature, which takes on one of 14 discrete values (even integers ranging from −12 to 14). To assess the versatility of PNAMs, we explore two distinct formulations: (i) a multi-class classification problem with 14 classes and (ii) a single-output regression problem.

Classification. The first two rows of Table 1 are reproduced from [14] and compare the performance of the MLP12 implemented by Davies et al. [46] against that of the KAN. The subsequent rows present our implementations of the MLP, the NAM, and four parameterizations of the PNAM, all utilizing two hidden layers of 64 and 32 neurons.13 We scale all inputs to have zero mean and unit variance and train the models for 50 epochs without early stopping. All six models achieve test accuracies exceeding 90%, with the NAM performing the worst and the PNAM with M=64M = 64 matching our implementation of the MLP. The accuracy of the PNAM increases monotonically with MM (from 8 to 64), consistent with Theorem 2.2. Moreover, the accuracy gain of the PNAM over the NAM for all three values of MM in Table 1 suggests that the penalty ℓ3\ell_3 in Eq. (14) does not collapse the PNAM into a NAM in practice (see, e.g., the learned projection matrix TT in Fig. A.2(b)). Although the accuracy of PNAMs can be improved by increasing MM or by replacing the first hidden layer of the basis MLPs with a Fourier feature mapping (e.g., the last row of Table 1, with Bj∈R32B_j \in\mathbb{R}^{32} and σ=1\sigma= 1 in Eq. (4)), these modifications increase the complexity of the learned functions.

MethodArchitectureParameter countTest acc.
DM’s MLP4 layers: [17, 300, 300, 300, 14]3×1053 \times10^578.0%
KAN2 layers: [17, 1, 14] (G=3G = 3, k=3k = 3)2×1022 \times10^281.6%
Our MLP3 layers: [17, 64, 32, 14]1.2×1041.2 \times10^495.8±0.1%95.8 \pm0.1\%
NAM3 layers: 17×[1,64,32,14]17 \times[1, 64, 32, 14]2.1×1052.1 \times10^592.4±0.2%92.4 \pm0.2\%
PNAM3 layers: 8×[1,64,32,14]8 \times[1, 64, 32, 14]9.8×1049.8 \times10^493.6±0.4%93.6 \pm0.4\%
PNAM3 layers: 17×[1,64,32,14]17 \times[1, 64, 32, 14]2.1×1052.1 \times10^595.0±0.2%95.0 \pm0.2\%
PNAM3 layers: 64×[1,64,32,14]64 \times[1, 64, 32, 14]7.9×1057.9 \times10^595.9±0.1%95.9 \pm0.1\%
PNAM3 layers: 8×[1,2(32),32,14]8 \times[1, 2(32), 32, 14] (σ=1\sigma= 1)9.8×1049.8 \times10^494.3±0.3%94.3 \pm0.3\%

Table 1. Performance comparison of different neural network architectures for the multi-class classification problem of predicting the signature of mathematical knots. The first two rows are reproduced from Table 3 of Liu et al. [14]; see [14] for definitions of GG and kk. For the remaining rows, the mean and standard deviation of the test accuracy are computed over 10 runs. The last row incorporates the Fourier feature mapping described in Eq. (4).

Table 2 summarizes the three most important features identified by the mean absolute coefficients of the linear transformation TT for the PNAM with M=8M = 8 and without the Fourier feature mapping.

RankInput featureSymbolFrequencyTest acc.
1Re(meridional translation)χ10\chi_{10}10/1055.1 ±\pm 3.5%
2Longitudinal translationχ8\chi_{8}10/1074.2 ±\pm 1.5%
3Im(meridional translation)χ9\chi_{9}7/1078.1 ±\pm 2.4%
3Cusp volumeχ7\chi_{7}1/1080.5%
3Im(short geodesic)χ4\chi_{4}1/1073.5%
3Volumeχ11\chi_{11}1/1070.5%

Table 2. Ranking of important features based on the mean absolute coefficients for the PNAM with M=8M = 8 (without the Fourier feature mapping) in Table 1. The mean and standard deviation of the test accuracy (associated with retaining only the top nn features) are computed over the runs in which the top nn features appeared.

Across all 10 runs, the meridional translation (real part) and the longitudinal translation consistently have the largest and second largest mean absolute coefficients, respectively; the meridional translation (imaginary part) ranks third in seven runs. The top two invariants are likewise ranked first and second in 10/10 runs for M=17M = 17 and M=64M = 64, indicating that the feature ranking remains stable as the capacity grows. Retaining only the columns of TT associated with the top n=3n = 3 inputs in Table 2, the PNAM still achieves a test accuracy of 78.1% (refer to Fig. A.1 for additional details). These findings are consistent with Fig. 3 of Davies et al. [46]14 and Fig. 4.3 of Liu et al. [14]. Interestingly, PNAMs recover these key invariants through direct inspection of the projection weights, whereas Davies et al. [46] employ gradient-based attribution [19] and Liu et al. [14] leverage a specific KAN architecture with a hidden dimension of one to extract feature importance.

Regression. Davies et al. [46] and Liu et al. [14] leverage insights from the classification task to construct explicit mathematical expressions for the knot signature, now formulated as a single-output regression problem. In Table A.5, expression A corresponds to the equation handcrafted by Davies et al. [46], while expressions B to F are derived from post-processing steps of KANs trained specifically on the three most relevant invariants [14]. Although Davies et al. [46] originally report a test accuracy between 70 and 80% for expression A, Liu et al. [14] report a test accuracy of 83.1%. Since the accuracy appears sensitive to the random data split employed, we evaluate all candidate expressions on our specific test set and the entire knot dataset (denoted as “Our” and “Total acc.” in Table A.5), in addition to the values reported by Liu et al. [14] for expressions A to F.

We demonstrate that the PNAM with regularization weight w=0.001w = 0.001 in Table A.3—trained on the full set of invariants without prior knowledge of their relevance—can be distilled into symbolic expressions with as few as tens of parameters. We emphasize that expressions G to I in Table A.5 emerge from the PNAM pipeline after pruning and SR; their particular forms depend on the available data and would likely change as more invariants are considered. The key contribution here is the ability of PNAMs to discover relevant relationships as new data and invariants are introduced.

Among the results summarized in Table A.4, expressions H and I are obtained from the run depicted in Fig. A.2, using n=3n = 3 features with the cusp volume15 ranked as the third most significant. Although the cusp volume is not explicitly identified in [14, 46] as a primary invariant for predictive accuracy, the PNAM independently identifies it as an invariant that potentially enhances performance. Moreover, comparing expressions E and G suggests that PNAMs can match KANs in accuracy while utilizing fewer invariants without relying on additional assumptions. For an analysis of how the regularization weights in Eq. (12) influence performance, see Appendix § 8.§ 3.

Limitations

Interpretability scope. Although PNAMs provide global interpretability, they do not offer local interpretability. Because the bases {gij}\{g_{ij}\} operate on the projected variables {zj}\{z_j\} rather than the original inputs {xk}\{x_k\}, examining gijg_{ij} in isolation does not directly reveal the contribution of a specific input χk\chi_k to the output. Furthermore, the mean absolute coefficients of the linear transformation TT measure feature usage in the learned projections, and they serve as a proxy for predictive importance only when the regularizer ℓ2\ell_2 in Eq. (13) is active and appropriately calibrated. In the absence of such a constraint, the resulting ranking may be unstable or erroneous (see, e.g., Fig. A.5(a)). Even with this constraint, the ranking is an empirical property of the regularized models, validated in the settings evaluated here, as opposed to a general guarantee of the architecture. Finally, we emphasize that the PNAM is a predictive, rather than a causal, model. Large mean absolute coefficients indicate a strong predictive influence but do not imply a causal relationship, particularly when the model is trained on highly correlated inputs.

An alternative approach to interpretability combines MLPs with post-hoc attribution, pruning, or symbolic distillation. Attribution explains the predictions and pruning reduces the model size, but the model itself remains a black box. Symbolic distillation, moreover, amounts to the multi-variable SR problem whose intractability motivates our design. The PNAM instead provides an integrated pipeline in which the same trained model supports feature ranking, pruning, and conversion into a closed-form expression, because the additive structure reduces SR to independent 1D problems. However, the GAM/NAM extensions with explicit interaction terms [4, 5, 6, 7, 8] are more directly interpretable when the ground truth is dominated by a few low-order interactions of raw features.

Optimization and training. The universal approximation and rate results of § 2.1 are only existential: they assert that accurate transformations and bases exist, not that gradient-based optimization recovers them—a characteristic shared with classical universal approximation theorems for MLPs [30, 31]. Two observations nevertheless connect the theory to the learned models. First, the transformation achieving an exact representation is highly nonunique (e.g., Example A.1), so the optimization target is a large set of admissible transformations rather than a single isolated construction. Second, performance improves monotonically with the projection dimension MM on average across all our experiments (see Table 1, A.7, and A.9), tracking the capacity scaling predicted by Theorem 2.2.

In practice, the joint optimization over TT, Θ\Theta, ζ\zeta, and ϵ\epsilon in Eq. (12) is non-convex, and performance is sensitive to both MM and the regularization weights w1,…,w5w_1,\ldots,w_5. Over-regularization leads to sparse but inaccurate models, while under-regularization results in dense projections that are significantly harder to interpret. In addition, the SR step in Section 2.4 is run for only one minute per basis in the numerical experiments. While this ensures computational efficiency, it may be insufficient for capturing the nuances of highly complex functions. Consequently, the resulting symbolic expressions are approximations and should be rigorously validated before use in safety-critical applications.

Conclusion

In this work, we establish the first theoretical foundation for PNAMs, proving universal approximation, characterizing the minimum number of projections MM, and deriving an approximation rate of O(M−s/(N−1))\mathcal{O}(M^{-s/(N-1)}) for Cs\mathcal{C}^s functions. Building on this foundation, we introduce structured regularization, input pruning, and symbolic regression techniques to recover sparse and interpretable models. Across a diverse set of experiments, PNAMs consistently match or surpass similar architectures in predictive accuracy while promoting sparsity. In the knot theory benchmark, PNAMs recover the same key invariants identified by prior studies and produce compact symbolic expressions with orders of magnitude fewer parameters than the trained networks. Additional experiments on MNIST and phase field fracture mechanics demonstrate the feature selection capability of PNAMs for high-dimensional image classification and their robustness in the limited-data regime, respectively. As a stand-alone model, the PNAM is not meant to replace the MLP, the NAM, or classical SR. Instead, it offers a principled middle ground: achieving the accuracy of the MLP while retaining the additive structure of the NAM, at the cost of increased memory and training time proportional to MM. Future work includes parallelizing the evaluation of the 1D bases to reduce this overhead and exploring the use of PNAMs as backbones in graph neural networks and transformers, where their modularity could enable structural pruning of network parameters and accelerate inference.

Acknowledgments and Disclosure of Funding

NNP is supported by the SMART Scholarship Program and the SMART SEED Innovation Award, both administered by the U.S. Army DEVCOM Army Research Laboratory. WCS is supported by the Department of Energy (DOE), National Nuclear Security Administration, Predictive Science Academic Alliance Program (PSAAP) under Award Number DE-NA0003962. Additional support is provided for NNP and WCS through the Dynamic Materials and Interactions Program from the Air Force Office of Scientific Research under grant contract FA9550-22-1-0310 and through the Army Research Office under grant contract W911NF-24-1-0116. QD is supported in part by the DOE-ASCR award DE-SC0025347 and the National Science Foundation (NSF) award DMS-2309245. Support is also provided by the NSF under grant contract CMMI-1846875 for the high-performance computing facility. All of this support is gratefully acknowledged. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the sponsors, including the Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes, notwithstanding any copyright notation herein. The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References

[1] W. J. Murdoch, C. Singh, K. Kumbier, R. Abbasi-Asl, and B. Yu. Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences, 116(44):22071–22080, 2019.

[2] R. Agarwal, L. Melnick, N. Frosst, X. Zhang, B. Lengerich, R. Caruana, and G. E. Hinton. Neural additive models: Interpretable machine learning with neural nets. Advances in Neural Information Processing Systems, 34(359):4699–4711, 2021.

[3] N. N. Phan, W.C. Sun, and J. D. Clayton. HYDRA: Symbolic feature engineering of overparameterized Eulerian hyperelasticity models for fast inference time. Computer Methods in Applied Mechanics and Engineering, 437:117792, 2025.

[4] Y. Lou, R. Caruana, J. Gehrke, and G. Hooker. Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 623–631, 2013.

[5] C.-H. Chang, R. Caruana, and A. Goldenberg. NODE-GAM: Neural generalized additive model for interpretable deep learning. arXiv preprint arXiv:2106.01613, 2021.

[6] Z. Yang, A. Zhang, and A. Sudjianto. GAMI-Net: An explainable neural network based on generalized additive models with structured interactions. Pattern Recognition, 120:108192, 2021.

[7] A. Dubey, F. Radenovic, and D. Mahajan. Scalable interpretability via polynomials. Advances in Neural Information Processing Systems, 35:36748–36761, 2022.

[8] J. Enouen and Y. Liu. Sparse interaction additive networks via feature interaction detection and sparse selection. Advances in Neural Information Processing Systems, 35:13908–13920, 2022.

[9] T. Hastie and R. Tibshirani. Generalized additive models. Statistical Science, 1(3):297–310, 1986.

[10] V. E. Maiorov. On best approximation by ridge functions. Journal of Approximation Theory, 99(1):68–94, 1999.

[11] A. Pinkus. Approximation theory of the MLP model in neural networks. Acta Numerica, 8:143–195, 1999.

[12] C. Molnar. Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. Leanpub, 2019.

Appendix

Polynomial reproducing

In this section, we establish the polynomial reproducing property, namely, that any multi-variable polynomial FF of χ∈RN\chi\in\mathbb{R}^{N} can be represented by a superposition of ridge functions. Specifically, we show that there exists a linear transformation TT such that F(χ)=∑i=1Mgi(zi)F(\chi)=\sum_{i=1}^{M}g_i(z_i), where each gig_i is a single-variable polynomial of the projected input zi=∑j=1NTijχjz_i=\sum_{j=1}^{N}T_{ij}\chi_j. This result is proved for arbitrary dimensions via mathematical induction on the input dimension NN, starting with the base case for two dimensions (N=2N=2).

To facilitate the inductive step, we first present a lemma demonstrating the reproducing property for a specific bivariate case. This lemma provides the necessary constructive foundation for the general NN-dimensional result that follows.

Lemma A.1 (Monomial representation). For any bivariate monomial χ1pχ2q\chi_1^p\chi_2^q, there exist a linear transformation T:R2→RMT:\mathbb{R}^{2}\to\mathbb{R}^{M}, where M≤p+q+1M\leq p+q+1, with Ti1=1T_{i1}=1 and Ti2=ciT_{i2}=c_i, and a set of coefficients {ζi}i=1M\{\zeta_i\}_{i=1}^{M} such that

χ1pχ2q=∑i=1Mζi(∑j=12Tijχj)p+q=∑i=1Mζi(χ1+ciχ2)p+q.(17)\chi_1^p\chi_2^q=\sum_{i=1}^{M}\zeta_i\left(\sum_{j=1}^{2}T_{ij}\chi_j\right)^{p+q}=\sum_{i=1}^{M}\zeta_i(\chi_1+c_i\chi_2)^{p+q}. \tag*{(17)}

Proof. Applying the binomial expansion to each term on the right-hand side of Eq. (A.1) and reordering the summations, we obtain

∑i=1Mζi(χ1+ciχ2)p+q=∑i=1Mζi∑m=0p+q(p+qm)χ1m(ciχ2)p+q−m=∑m=0p+q(p+qm)(∑i=1Mζicip+q−m)χ1mχ2p+q−m.\sum_{i=1}^{M}\zeta_i(\chi_1+c_i\chi_2)^{p+q} =\sum_{i=1}^{M}\zeta_i\sum_{m=0}^{p+q}\binom{p+q}{m}\chi_1^m(c_i\chi_2)^{p+q-m} =\sum_{m=0}^{p+q}\binom{p+q}{m}\left(\sum_{i=1}^{M}\zeta_i c_i^{p+q-m}\right)\chi_1^m\chi_2^{p+q-m}.

To satisfy the equality in Eq. (A.1), the coefficients of the resulting polynomial must match the target monomial χ1pχ2q\chi_1^p\chi_2^q. Specifically, the coefficient must be 1 when m=pm=p and 0 otherwise. This leads to the following system of MM linear equations for {ζi}\{\zeta_i\}:

(p+qm)∑i=1Mζicip+q−m=δ~mp,m∈{0,1,…,p+q}.\binom{p+q}{m}\sum_{i=1}^{M}\zeta_i c_i^{p+q-m}=\widetilde{\delta}_{mp},\qquad m\in\{0,1,\ldots,p+q\}.

where δ~mp\widetilde{\delta}_{mp} is the Kronecker delta. Written in matrix form, this system is

[c1p+qc2p+q⋯cMp+qc1p+q−1c2p+q−1⋯cMp+q−1⋮⋮⋱⋮11⋯1][ζ1ζ2⋮ζM]=[0⋮(p+qp)−1⋮0].\begin{bmatrix} c_1^{p+q} & c_2^{p+q} & \cdots& c_M^{p+q}\\ c_1^{p+q-1} & c_2^{p+q-1} & \cdots& c_M^{p+q-1}\\ \vdots& \vdots& \ddots& \vdots\\ 1 & 1 & \cdots& 1 \end{bmatrix} \begin{bmatrix} \zeta_1\\ \zeta_2\\ \vdots\\ \zeta_M \end{bmatrix} = \begin{bmatrix} 0\\ \vdots\\ \binom{p+q}{p}^{-1}\\ \vdots\\ 0 \end{bmatrix}.

We note that the coefficient matrix forms a so-called Vandermonde matrix [53, 54]. It is a standard result that a Vandermonde matrix is invertible if and only if the constants {ci}i=1M\{c_i\}_{i=1}^{M} are distinct. By choosing M=p+q+1M=p+q+1 mutually different values for {ci}\{c_i\}, we guarantee a unique solution for the coefficients {ζi}\{\zeta_i\}. This establishes that M=p+q+1M=p+q+1 projections suffice, thus proving the bound M≤p+q+1M\leq p+q+1.

Even for this simple case, we may observe the nonuniqueness in the construction of the linear transformation TT. Consequently, it is reasonable to expect that for specific machine learning applications, TT may be optimized to achieve effective representations in a reduced-dimensional feature space. This potential for dimensionality reduction is illustrated in the following example.

Example A.1 (Dimensionality reduction). We present instances where the required projection dimension MM is strictly smaller than the upper bound p+q+1p+q+1 established in Lemma A.1. For p=q=1p=q=1, a variation of the polarization identity (see Theorem A.2) gives

χ1χ2=(χ1+14χ2)2−(χ1−14χ2)2,\chi_1\chi_2=\left(\chi_1+\frac{1}{4}\chi_2\right)^2-\left(\chi_1-\frac{1}{4}\chi_2\right)^2,

which corresponds to M=2M=2, whereas the lemma suggests an upper bound of 3. In the trivial case where p=0p=0 or q=0q=0, the monomial is univariate, and we may simply use the identity transformation with M=1M=1. Furthermore, MM can be smaller than the input dimension NN. For example, the quadratic form χ12+2χ1χ2+χ22\chi_1^2+2\chi_1\chi_2+\chi_2^2 can be represented as z12z_1^2 with z1=χ1+χ2z_1=\chi_1+\chi_2, yielding M=1<2=NM=1<2=N. These cases demonstrate the efficacy of utilizing a learned transformation to achieve dimensionality reduction through an appropriate latent space.

Next, we consider the extension of Lemma A.1. Since a product of two single-variable polynomials f1(χ1)f_1(\chi_1) and f2(χ2)f_2(\chi_2) can be expressed as a linear combination of bivariate monomials {χ1pχ2q}\{\chi_1^p\chi_2^q\}, Lemma A.1 can be generalized to represent polynomials with a product form.

Lemma A.2 (Product-form representation). For any bivariate polynomial of (χ1,χ2)∈R2(\chi_1,\chi_2)\in\mathbb{R}^2 having the product form f1(χ1)f2(χ2)f_1(\chi_1)f_2(\chi_2), there exist a linear transformation T:R2→RMT:\mathbb{R}^2\to\mathbb{R}^M, where M≤deg⁡(f1)+deg⁡(f2)+1M\leq\deg(f_1)+\deg(f_2)+1, with deg⁡(⋅)\deg(\cdot) denoting the degree of a polynomial, and a set of single-variable polynomials {gi:R→R}i=1M\{g_i:\mathbb{R}\to\mathbb{R}\}_{i=1}^{M} such that

f1(χ1)f2(χ2)=∑i=1Mgi(Ti1χ1+Ti2χ2).(18)f_1(\chi_1)f_2(\chi_2)=\sum_{i=1}^{M}g_i(T_{i1}\chi_1+T_{i2}\chi_2). \tag*{(18)}

Proof. We begin by expanding the single-variable polynomials in their respective power bases:

f1(χ1)=∑p=0deg⁡(f1)apχ1p,f2(χ2)=∑q=0deg⁡(f2)bqχ2q.f_1(\chi_1) = \sum_{p=0}^{\deg(f_1)} a_p\chi_1^p,\qquad f_2(\chi_2) = \sum_{q=0}^{\deg(f_2)} b_q\chi_2^q.

Multiplying these expressions yields a bivariate polynomial represented as a linear combination of monomials:

f1(χ1)f2(χ2)=∑p=0deg⁡(f1)∑q=0deg⁡(f2)apbqχ1pχ2q.f_1(\chi_1)f_2(\chi_2) = \sum_{p=0}^{\deg(f_1)}\sum_{q=0}^{\deg(f_2)} a_pb_q\chi_1^p\chi_2^q.

To apply Lemma A.1 using a single unified linear transformation TT for all terms, we fix M=deg⁡(f1)+deg⁡(f2)+1M = \deg(f_1) + \deg(f_2) + 1 distinct constants {ci}i=1M\{c_i\}_{i=1}^M. For any given monomial χ1pχ2q\chi_1^p\chi_2^q in the expansion, the total degree satisfies p+q≤deg⁡(f1)+deg⁡(f2)=M−1p + q \le\deg(f_1) + \deg(f_2) = M - 1.

The matching of coefficients for a specific monomial generates a Vandermonde system of p+q+1p + q + 1 equations with MM unknowns {ζpqi}i=1M\{\zeta_{pqi}\}_{i=1}^M. Since p+q+1≤Mp + q + 1 \le M and the constants {ci}\{c_i\} are distinct, the resulting (p+q+1)×M(p + q + 1) \times M Vandermonde matrix has full row rank. This guarantees that the system is consistent and admits at least one valid solution set {ζpqi}i=1M\{\zeta_{pqi}\}_{i=1}^M for each pair (p,q)(p,q). While the solution is nonunique for p+q+1<Mp + q + 1 < M, we may fix any particular solution, such as the minimum-norm solution obtained via the Moore–Penrose pseudoinverse [55] of the Vandermonde matrix.

By solving for {ζpqi}\{\zeta_{pqi}\} using the same MM constants for every monomial, we can employ a global transformation TT with Ti1=1T_{i1} = 1 and Ti2=ciT_{i2} = c_i. Substituting the monomial representations back into the expansion and reordering the summations, we obtain

f1(χ1)f2(χ2)=∑i=1M(∑p=0deg⁡(f1)∑q=0deg⁡(f2)apbqζpqi(χ1+ciχ2)p+q).f_1(\chi_1)f_2(\chi_2) = \sum_{i=1}^{M}\left(\sum_{p=0}^{\deg(f_1)}\sum_{q=0}^{\deg(f_2)}a_pb_q\zeta_{pqi}(\chi_1+c_i\chi_2)^{p+q}\right).

This result matches the form of Eq. (A.2) by defining each single-variable polynomial gig_i as

gi(zi)=∑p=0deg⁡(f1)∑q=0deg⁡(f2)apbqζpqizip+q,g_i(z_i) = \sum_{p=0}^{\deg(f_1)}\sum_{q=0}^{\deg(f_2)}a_pb_q\zeta_{pqi}z_i^{p+q},

where zi=Ti1χ1+Ti2χ2z_i = T_{i1}\chi_1 + T_{i2}\chi_2 represents a projected input. This completes the proof. □

Another consequence is the reproducing property for arbitrary polynomials in two dimensions, which follows from Lemma A.2 by observing that each monomial χ1pχ2q\chi_1^p\chi_2^q is itself a product of two single-variable polynomials.

Lemma A.3 (Bivariate polynomial representation). For any bivariate polynomial FF of (χ1,χ2)∈R2(\chi_1,\chi_2) \in\mathbb{R}^2, there exist a linear transformation T:R2→RMT : \mathbb{R}^2 \to\mathbb{R}^M, where M≤deg⁡(F)+1M \le\deg(F) + 1, and a set of single-variable polynomials {gi:R→R}i=1M\{g_i : \mathbb{R} \to\mathbb{R}\}_{i=1}^M such that

F(χ1,χ2)=∑i=1Mgi(Ti1χ1+Ti2χ2).(19)F(\chi_1,\chi_2) = \sum_{i=1}^{M} g_i(T_{i1}\chi_1 + T_{i2}\chi_2). \tag*{(19)}

Proof. Note that deg⁡(F)\deg(F) refers to the degree of the multi-variable polynomial, defined as the maximum value of p+qp + q among the monomials {χ1pχ2q}\{\chi_1^p\chi_2^q\} appearing in FF with nonzero coefficients. Expanding FF in the monomial basis, we have

F(χ1,χ2)=∑p+q≤deg⁡(F)apqχ1pχ2q.F(\chi_1,\chi_2) = \sum_{p+q\le\deg(F)}a_{pq}\chi_1^p\chi_2^q.

The key observation is that each monomial χ1pχ2q\chi_1^p\chi_2^q is a specific instance of the product form f1(χ1)f2(χ2)f_1(\chi_1)f_2(\chi_2) treated in Lemma A.2, where deg⁡(f1)+deg⁡(f2)=p+q≤deg⁡(F)\deg(f_1) + \deg(f_2) = p + q \le\deg(F). Lemma A.2 therefore applies to each monomial individually, utilizing a unified set of M=deg⁡(F)+1M = \deg(F) + 1 projections shared globally across all monomials. The remainder of the proof follows identically to that of Lemma A.2. □

Finally, we employ mathematical induction to extend the result to any input dimension.

Theorem A.1 (Polynomial representation via induction). For any multi-variable polynomial F\mathcal{F} of (χ1,…,χN)∈RN(\chi_1,\ldots,\chi_N)\in\mathbb{R}^N, there exist a linear transformation T:RN→RMT:\mathbb{R}^N\to\mathbb{R}^M, for some $M that can be chosen to depend only on deg⁡(F)\deg(\mathcal{F}) and NN (e.g., M=(deg⁡(F)+1)N−1M=(\deg(\mathcal{F})+1)^{N-1}), and a set of single-variable polynomials {gi:R→R}i=1M\{g_i:\mathbb{R}\to\mathbb{R}\}_{i=1}^M such that

F(χ1,…,χN)=∑i=1Mgi(∑j=1NTijχj).(20)\mathcal{F}(\chi_1,\ldots,\chi_N)=\sum_{i=1}^{M}g_i\left(\sum_{j=1}^{N}T_{ij}\chi_j\right). \tag*{(20)}

Proof. We proceed by mathematical induction on the input dimension NN. For the base case N=2N=2, Lemma A.3 establishes that a single transformation T:R2→RMT:\mathbb{R}^2\to\mathbb{R}^M suffices for any bivariate polynomial of degree deg⁡(F)\deg(\mathcal{F}), as the choice of projection constants {ci}\{c_i\} depends only on deg⁡(F)\deg(\mathcal{F}).

Now, assume that Theorem A.1 holds uniformly for the space of polynomials of degree deg⁡(F)\deg(\mathcal{F}) in NN variables, that is, there exists a global transformation TN:RN→RMNT_N:\mathbb{R}^N\to\mathbb{R}^{M_N} that can represent any such polynomial by varying only the single-variable functions. We then consider a polynomial F(χ1,…,χN,χN+1)\mathcal{F}(\chi_1,\ldots,\chi_N,\chi_{N+1}) of degree deg⁡(F)\deg(\mathcal{F}) defined on RN+1\mathbb{R}^{N+1}. It can be expanded as

F(χ1,…,χN,χN+1)=∑r=0RFr(χ1,…,χN)χN+1r,\mathcal{F}(\chi_1,\ldots,\chi_N,\chi_{N+1})=\sum_{r=0}^{R}\mathcal{F}_r(\chi_1,\ldots,\chi_N)\chi_{N+1}^{r},

for some polynomials {Fr}r=0R\{\mathcal{F}_r\}_{r=0}^{R}, where R≤deg⁡(F)R\leq\deg(\mathcal{F}). Since F\mathcal{F} has degree deg⁡(F)\deg(\mathcal{F}), each term Fr(χ1,…,χN)χN+1r\mathcal{F}_r(\chi_1,\ldots,\chi_N)\chi_{N+1}^{r} must satisfy deg⁡(Fr)≤deg⁡(F)−r\deg(\mathcal{F}_r)\leq\deg(\mathcal{F})-r. In particular, because deg⁡(Fr)≤deg⁡(F)\deg(\mathcal{F}_r)\leq\deg(\mathcal{F}) for all rr, the inductive hypothesis allows us to use the same global transformation TNT_N for all r∈{0,…,R}r\in\{0,\ldots,R\}. Thus, there exists a set of single-variable polynomials {{g~ri}i=1MN}r=0R\{\{\widetilde{g}_{ri}\}_{i=1}^{M_N}\}_{r=0}^{R} such that

Fr(χ1,…,χN)=∑i=1MNg~ri(zN,i),zN,i=∑j=1NTN,ijχj.(21)\mathcal{F}_r(\chi_1,\ldots,\chi_N)=\sum_{i=1}^{M_N}\widetilde{g}_{ri}(z_{N,i}),\qquad z_{N,i}=\sum_{j=1}^{N}T_{N,ij}\chi_j. \tag*{(21)}

Substituting Eq. (A.5) into the expansion and rearranging the summations, we obtain

F(χ1,…,χN,χN+1)=∑i=1MN(∑r=0Rg~ri(zN,i)χN+1r).\mathcal{F}(\chi_1,\ldots,\chi_N,\chi_{N+1})=\sum_{i=1}^{M_N}\left(\sum_{r=0}^{R}\widetilde{g}_{ri}(z_{N,i})\chi_{N+1}^{r}\right).

Let Hi(zN,i,χN+1)=∑r=0Rg~ri(zN,i)χN+1r\mathcal{H}_i(z_{N,i},\chi_{N+1})=\sum_{r=0}^{R}\widetilde{g}_{ri}(z_{N,i})\chi_{N+1}^{r}. Each Hi\mathcal{H}_i is a bivariate polynomial in the variables zN,iz_{N,i} and χN+1\chi_{N+1} of total degree at most deg⁡(F)\deg(\mathcal{F}). By Lemma A.3, each Hi\mathcal{H}_i admits a representation involving at most deg⁡(F)+1\deg(\mathcal{F})+1 projections of the form zN+1,ik=zN,i+ckχN+1z_{N+1,ik}=z_{N,i}+c_k\chi_{N+1}.

Given that the choice of {ck}\{c_k\} is independent of the specific polynomials {Hi}\{\mathcal{H}_i\}, we can share these deg⁡(F)+1\deg(\mathcal{F})+1 constants across all i∈{1,…,MN}i\in\{1,\ldots,M_N\}, resulting in a global transformation TN+1:RN+1→RMN+1T_{N+1}:\mathbb{R}^{N+1}\to\mathbb{R}^{M_{N+1}} for N+1N+1 dimensions. Therefore, the total number of projections is

MN+1≤MN⋅(deg⁡(F)+1).M_{N+1}\leq M_N\cdot(\deg(\mathcal{F})+1).

Solving this recurrence relation with the base case M2=deg⁡(F)+1M_2=\deg(\mathcal{F})+1 yields

MN≤(deg⁡(F)+1)N−1.M_N\leq(\deg(\mathcal{F})+1)^{N-1}.

As a result, Theorem A.1 holds for the input dimension N+1N+1. This completes the induction process and the proof.

Theorem A.1 shows that to represent a polynomial F\mathcal{F} of degree P=deg⁡(F)P=\deg(\mathcal{F}) in NN variables, the number of required projections satisfies M≤(P+1)N−1M\leq(P+1)^{N-1}; thus, for large PP, the bound scales as M∼PN−1M\sim P^{N-1}. This bound is consistent with classical ridge function approximation theory (see, e.g., [11]), which establishes the same PN−1P^{N-1} scaling. While Theorem A.1 achieves this scaling constructively via mathematical induction using a Vandermonde system of equations, Pinkus [11] leverages a non-constructive dimension argument on the space of homogeneous polynomials to guarantee the existence of suitable directions without specifying them. Alternatively, an estimate on MM can be obtained via a complementary construction based on the polarization identity [56].

Theorem A.2 (Polynomial representation via polarization). Let PN,P\mathcal{P}_{N,P} be the space of multi-variable polynomials in NN variables of degree at most PP. Any F∈PN,P\mathcal{F} \in\mathcal{P}_{N,P} can be represented exactly by a PNAM with MM projections, where MM is bounded by the total number of terms generated by the polarization of each basis monomial, that is,

M≤∑d=1P(N+d−1d)2d−1.M \leq\sum_{d=1}^{P} \binom{N+d-1}{d} 2^{d-1}.

Proof. Any polynomial F(χ)\mathcal{F}(\boldsymbol{\chi}) of degree PP, where χ=(χ1,…,χN)∈RN\boldsymbol{\chi} = (\chi_1,\ldots,\chi_N) \in\mathbb{R}^N, can be written as a linear combination of monomials using multi-index notation:

F(χ)=∑∣α∣≤Paαχα,χα=χ1α1⋯χNαN.\mathcal{F}(\boldsymbol{\chi}) = \sum_{|\boldsymbol{\alpha}|\leq P} a_{\boldsymbol{\alpha}}\boldsymbol{\chi}^{\boldsymbol{\alpha}}, \qquad\boldsymbol{\chi}^{\boldsymbol{\alpha}} = \chi_1^{\alpha_1}\cdots\chi_N^{\alpha_N}.

The number of monomials of degree exactly dd is (N+d−1d)\binom{N+d-1}{d}, and the total dimension of the polynomial space is dim⁡(PN,P)=(N+PP)\dim(\mathcal{P}_{N,P}) = \binom{N+P}{P}. Note that constant monomials (∣α∣=0|\boldsymbol{\alpha}|=0) require no projections, as their contribution can be absorbed into the constant term of any single-variable polynomial in the representation; hence, they do not contribute to MM.

Step 1: Polarization of a single monomial. For each monomial χα\boldsymbol{\chi}^{\boldsymbol{\alpha}} of degree d=∣α∣≥1d=|\boldsymbol{\alpha}|\geq1, we apply the polarization identity. We encode the factored structure of χα=χ1α1⋯χNαN\boldsymbol{\chi}^{\boldsymbol{\alpha}}=\chi_1^{\alpha_1}\cdots\chi_N^{\alpha_N} as a product of dd linear factors by introducing dd vectors v1,…,vd∈RN\mathbf{v}_1,\ldots,\mathbf{v}_d\in\mathbb{R}^N, where αj\alpha_j of them equal the jjth standard basis vector ej\mathbf{e}_j, so that vi⋅χ=χj\mathbf{v}_i\cdot\boldsymbol{\chi}=\chi_j for the corresponding factor. The polarization identity then gives

χα=1d!2d∑δ∈{−1,1}d(∏i=1dδi)(∑i=1dδivi⋅χ)d.\boldsymbol{\chi}^{\boldsymbol{\alpha}}=\frac{1}{d!2^d}\sum_{\boldsymbol{\delta}\in\{-1,1\}^d}\left(\prod_{i=1}^{d}\delta_i\right)\left(\sum_{i=1}^{d}\delta_i\mathbf{v}_i\cdot\boldsymbol{\chi}\right)^d.

Pairing each sign vector δ\boldsymbol{\delta} with its negation −δ-\boldsymbol{\delta} yields equal contributions, since

∏i=1d(−δi)(−∑i=1dδivi⋅χ)d=(−1)d(−1)d∏i=1dδi(∑i=1dδivi⋅χ)d=∏i=1dδi(∑i=1dδivi⋅χ)d.\prod_{i=1}^{d}(-\delta_i)\left(-\sum_{i=1}^{d}\delta_i\mathbf{v}_i\cdot\boldsymbol{\chi}\right)^d =(-1)^d(-1)^d\prod_{i=1}^{d}\delta_i\left(\sum_{i=1}^{d}\delta_i\mathbf{v}_i\cdot\boldsymbol{\chi}\right)^d =\prod_{i=1}^{d}\delta_i\left(\sum_{i=1}^{d}\delta_i\mathbf{v}_i\cdot\boldsymbol{\chi}\right)^d.

As a result, the 2d2^d terms collapse to 2d−12^{d-1} distinct projected power functions of the form ζk(tk⋅χ)d\zeta_k(\mathbf{t}_k\cdot\boldsymbol{\chi})^d, where ζk=∏i=1dδkid!2d−1\zeta_k=\frac{\prod_{i=1}^{d}\delta_{ki}}{d!2^{d-1}} and tk=∑i=1dδkivi\mathbf{t}_k=\sum_{i=1}^{d}\delta_{ki}\mathbf{v}_i are the scalar weight and projection direction associated with the kkth sign vector pair (δk,−δk)(\boldsymbol{\delta}_k,-\boldsymbol{\delta}_k), respectively.

Step 2: Summation over the polynomial. Substituting the polarized form of each monomial back into the polynomial expansion yields

F(χ)=∑∣α∣≤P∣α∣≥1aα(∑k=12∣α∣−1ζαk(tαk⋅χ)∣α∣).\mathcal{F}(\boldsymbol{\chi})=\sum_{\substack{|\boldsymbol{\alpha}|\leq P\\|\boldsymbol{\alpha}|\geq1}}a_{\boldsymbol{\alpha}}\left(\sum_{k=1}^{2^{|\boldsymbol{\alpha}|-1}}\zeta_{\boldsymbol{\alpha}k}(\mathbf{t}_{\boldsymbol{\alpha}k}\cdot\boldsymbol{\chi})^{|\boldsymbol{\alpha}|}\right).

By linearity, rearranging the summations gives

F(χ)=∑α≠0∑kgαk(tαk⋅χ),(22)\mathcal{F}(\boldsymbol{\chi})=\sum_{\boldsymbol{\alpha}\neq\mathbf{0}}\sum_k g_{\boldsymbol{\alpha}k}(\mathbf{t}_{\boldsymbol{\alpha}k}\cdot\boldsymbol{\chi}), \tag*{(22)}

where gαk(z)=aαζαkz∣α∣g_{\boldsymbol{\alpha}k}(z)=a_{\boldsymbol{\alpha}}\zeta_{\boldsymbol{\alpha}k}z^{|\boldsymbol{\alpha}|} is a single-variable polynomial (specifically a monomial).

Step 3: Consolidation. Based on the above construction, the total number of projections MM is bounded by the sum over all monomial degrees of the number of projections per monomial:

M≤∑d=1P(N+d−1d)2d−1.M \leq\sum_{d=1}^{P} \binom{N+d-1}{d}2^{d-1}.

This completes the proof.

Theorems A.1 and A.2 each provide a constructive upper bound for MM, but with complementary limitations: Theorem A.1 scales as PN−1P^{N-1} and is exponential in NN, while Theorem A.2 scales as (2P−1NP)/P!(2^{P-1}N^{P})/P! and is exponential in PP. A bound that is simultaneously polynomial in both NN and PP can be obtained by a non-constructive argument via the Waring rank [56].

Theorem A.3 (Bound via Waring rank). The minimum number of projections MM required to represent any multi-variable polynomial FF in NN variables of degree PP is bounded by the dimension of the polynomial space:

M≤(N+PP).M \leq\binom{N+P}{P}.

Proof. The set of all multi-variable polynomials of degree at most PP in NN variables, PN,P\mathcal{P}_{N,P}, is a vector space with dimension dim⁡(PN,P)=(N+PP)\dim(\mathcal{P}_{N,P}) = \binom{N+P}{P}. The set of projected power functions {(t⋅χ)d∣t∈RN, 1≤d≤P}\{(t \cdot\chi)^d \mid t \in\mathbb{R}^{N},\ 1 \leq d \leq P\} spans PN,P\mathcal{P}_{N,P}, as every monomial can be recovered via the polarization identity in Theorem A.2. Since this spanning set covers the entire space, one can extract a basis of size dim⁡(PN,P)=(N+PP)\dim(\mathcal{P}_{N,P}) = \binom{N+P}{P}. Any F∈PN,PF \in\mathcal{P}_{N,P} can therefore be expressed as a linear combination of at most (N+PP)\binom{N+P}{P} projected functions.

As noted, the Waring bound (N+PP)∼PN/N!\binom{N+P}{P} \sim P^N/N! scales polynomially in both NN and PP, making it tighter than Theorem A.2 for large PP regardless of NN. This bound is looser than Theorem A.1 when PP is large relative to NN, but tighter than Theorem A.1 when NN is large relative to PP. For large PP, the Waring bound can be further refined for generic polynomials via the notion of generic Waring rank, as formalized by the Alexander–Hirschowitz theorem [56, 57]. The following corollary summarizes the resulting bounds on MM, including a parameter-counting lower bound.

Corollary A.4 (Projection complexity). For the space PN,P\mathcal{P}_{N,P} of multi-variable polynomials in NN variables of degree at most PP, the minimum number of projections MM in the exact polynomial representation satisfies

1N+P+1(N+PP)≤M≤(N+PP),\frac{1}{N+P+1}\binom{N+P}{P} \leq M \leq\binom{N+P}{P},

where the lower bound follows from a parameter-counting argument and the upper bound from Theorem A.3. In the exact polynomial representation setting, each of the MM projections contributes NN free parameters from the linear transformation TT and P+1P+1 coefficients from the single-variable polynomial gig_i, giving M(N+P+1)M(N+P+1) total free parameters, which must be at least dim⁡(PN,P)=(N+PP)\dim(\mathcal{P}_{N,P}) = \binom{N+P}{P}. For generic polynomials with large PP, the Alexander–Hirschowitz theorem [56, 57] tightens the upper bound to

Mgeneric=⌈1N(N+PP)⌉,M_{\mathrm{generic}} = \left\lceil\frac{1}{N}\binom{N+P}{P} \right\rceil,

which differs from the lower bound by a factor of N+P+1N\frac{N+P+1}{N}.

The bounds established in this section—from the constructive bound of Theorem A.1 to the parameter-counting lower bound of Corollary A.4—characterize the minimum MM required for exact polynomial representation. In practice, however, PNAMs do not aim to represent a fixed polynomial exactly, but rather to learn an effective approximation of an unknown function from data. As demonstrated in Example A.1, the nonuniqueness of the transformation TT means that the theoretical upper bounds on MM are not tight in general; different constructions can achieve the same representation with fewer projections. This suggests that, through learning and optimization, PNAMs can discover representations that are more compact than the theoretical bounds would predict, adapting TT to the specific structure of the target function rather than to the worst case.

Additional results for knot theory: classification

Figure A.1 shows a complete ranking of the inputs and the associated test accuracy of using the top nn inputs from one of the runs summarized in Table 2.

Figure A.1. One complete instance of the ranking and accuracy reported in Table 2. (a) Relative importance of the input features based on the mean absolute coefficients, and (b) test accuracy associated with using the top nn features in (a).

Validation of the importance ranking. To test whether the ranking by the mean absolute coefficients of the linear transformation TT captures predictive importance as opposed to mere feature usage, we perform four post-training checks on the PNAM with M=8M = 8 and without the Fourier feature mapping in Table 1. First, columns of TT are zeroed out according to the ranking of each run. Table A.1 reveals that removing the single top-ranked feature reduces the test accuracy from 93.6 to 30.9%, whereas removing the three bottom-ranked features has no measurable effect. Second, starting from the transformation TT with all columns zeroed out, a greedy selection reactivates one column at a time, taking the column with the largest gain in accuracy as the next most significant input. This procedure measures predictive importance without reference to the coefficients of TT, and it recovers the same top two invariants as the mean absolute coefficients in 10/10 runs and the same third invariant in 5/10 runs, where the third position varies between the meridional translation (imaginary part) and the cusp volume (which also appears at rank 3 in Table 2). However, the greedy selection requires O(N2)O(N^2) evaluations of the test set, so the ranking by the mean absolute coefficients remains the practical choice when the number of inputs NN is large (e.g., N=784N = 784 for the MNIST dataset).

Remove nnTest acc.
TopBottom
0 (baseline)93.6 ± 0.4%93.6 ± 0.4%
130.9 ± 2.1%93.7 ± 0.4%
228.8 ± 1.2%93.6 ± 0.4%
325.2 ± 2.8%93.5 ± 0.4%

Table A.1. Test accuracy of the PNAM with M=8M = 8 (without the Fourier feature mapping) in Table 1 after zeroing out the columns of the linear transformation TT associated with the top or bottom nn features of each run. The mean and standard deviation are computed over 10 runs without retraining.

Third, permutation importance—the drop in accuracy when the values of one input are shuffled across the test set—correlates with the mean absolute coefficients, yielding a Spearman rank correlation of ρ=0.939±0.023\rho= 0.939 \pm0.023; the two rankings agree on the most important feature in 10/10 runs. Fourth, the gradient sensitivity ∣E[∂L/∂xk]∣|\mathbb{E}[\partial\mathcal{L}/\partial x_k]| over the test set, which incorporates the derivatives of the bases {gij}\{g_{ij}\} and cancellations across projections, gives ρ=0.977±0.014\rho= 0.977 \pm0.014, agreeing on the top-ranked feature in 9/10 runs, where the single disagreement selects the second-ranked feature. These checks complement the accuracies of retaining the top nn features in Table 2 and Fig. A.1(b).

Group-lasso MLP baseline. As an alternative to the ranking by the mean absolute coefficients, we retrain the MLP in Table 1 under an otherwise identical protocol with the added penalty (w′/N)∑k∥Wk(1)∥2(w'/N)\sum_k \lVert\mathbf{W}^{(1)}_k\rVert_2, where Wk(1)\mathbf{W}^{(1)}_k is the kkth column of the first-layer weight matrix W(1)\mathbf{W}^{(1)}, rank the input features by ∥Wk(1)∥2\lVert\mathbf{W}^{(1)}_k\rVert_2, and sweep w′w' over {0,0.001,0.01,0.1,1}\{0, 0.001, 0.01, 0.1, 1\}. At every w′w', this baseline is a weaker feature selector than the PNAM. At the representative value w′=0.01w' = 0.01, it matches the unregularized MLP in accuracy (95.7 ± 0.1%) and its most important feature agrees with that of the PNAM in 10/10 runs, but the features ranked second and third vary across runs and never include the longitudinal translation or the meridional translation (imaginary part). Retaining its top three features produces a test accuracy of 50.2 ± 3.8%, against 78.1 ± 2.4% for the PNAM under the same input pruning (Table 2). This behavior persists for all w′≤0.1w' \le0.1. At w′=1w' = 1, the selection improves, recovering the longitudinal translation among its top three features in 9/10 runs with a top-three accuracy of 75.5 ± 6.3%, but the accuracy with the full feature set falls to 86.8 ± 1.2%, below even that of the NAM. In the MLP, the deeper layers freely recombine the first-layer outputs, so the column norms become informative only once a sufficiently strong penalty concentrates the first-layer weights on a few inputs, by which point the accuracy has already degraded.

Ablation of the projection matrix. Table A.2 isolates the roles of learning and regularizing the linear transformation TT in one matched setting: knot classification with M=17M = 17, where all models use 17 basis MLPs of identical architecture under the protocol of Table 1. First, random mixing is worse than no mixing at all, so the performance gain of PNAMs cannot be attributed to the projection mechanism alone or to the additional parameters of TT. The gain comes from learning the projection itself: the learned transformation with regularization weights w1=⋯=w5=w=0w_1 = \cdots= w_5 = w = 0 outperforms the random transformation fixed at initialization by 6.5% and the identity transformation by 2.9%. Second, regularizing TT costs 0.3% in accuracy but, in exchange, stabilizes the feature ranking. In the regularized models, the three most significant invariants of Table 2 occupy ranks 1, 2, and 3 in 10/10, 10/10, and 8/10 runs, respectively. In the unregularized models, the top-ranked feature is stable, but the second-ranked feature varies across runs. The regularization weight ww thus allows the user to navigate the trade-off between accuracy and ranking reliability discussed in § 4.

TTTest acc.
Random, fixed at initialization88.8 ± 0.5%
Identity (= NAM, Table 1)92.4 ± 0.2%
Learned, regularized (Table 1)95.0 ± 0.2%
Learned, unregularized (w=0w = 0)95.3 ± 0.2%

Table A.2. Ablation of the projection matrix TT on knot classification with M=17M = 17. All models use 17 basis MLPs of identical architecture under the protocol of Table 1; the mean and standard deviation of the test accuracy are computed over 10 runs.

Alternative to the penalty ℓ3\ell_3. To assess the sensitivity of our results to the form of the penalty ℓ3\ell_3, we retrain the PNAM with M=8M = 8 (without the Fourier feature mapping) in Table 1 under an otherwise identical protocol, with ℓ3\ell_3 replaced by ℓ3′\ell'_3 in footnote 7. This substitution yields the same test accuracy (93.6 ± 0.4%) and the same top two invariants in 10/10 runs, with the invariant ranked third slightly more stable under ℓ3\ell_3 (7/10 runs compared to 4/10). Our conclusions are thus insensitive to this design choice.

Additional results for knot theory: regression

In Table A.3, we compare the performance of the MLP and the NAM with 17 bases against that of the PNAM with eight bases under three values of the regularization weights w1=⋯=w5=ww_1 = \cdots= w_5 = w. Both the NAM and the PNAM can be converted into symbolic expressions after training. Without regularization (w=0w = 0), the PNAM is within 3.2% of the MLP. The larger gap at w=0.001w = 0.001 is the price of a sparser model with a more reliable ranking, and this model is chosen for symbolic distillation below. Given the poor performance of the NAM, we proceed with just the PNAM. Similar to Table 2, Table A.4 suggests that the meridional translation (real part) and the longitudinal translation are the most and second most important invariants, respectively. Nevertheless, solely comparing the mean absolute coefficients may not be sufficient, as unimportant features could be incorrectly ranked as important. We can, however, eliminate these features by examining the accuracy associated with including them, since including an unimportant feature does not improve accuracy compared to the corresponding case with one fewer input.

Table A.3: Performance comparison of different neural network architectures for the single-output regression problem of predicting the signature of mathematical knots. The mean and standard deviation of the test accuracy are computed over 10 runs. The three PNAMs use regularization weights w1=⋯=w5=ww_1 = \cdots= w_5 = w; Figs. A.4 and A.5 show their corresponding training and validation histories and the effect of ww on the input ranking, respectively.

MethodArchitectureParameter countTest acc.
MLP3 layers: [17, 64, 32, 1]1.2×1041.2 \times10^494.7±0.2%94.7 \pm0.2\%
NAM3 layers: 17×17 \times [1, 64, 32, 1]2.1×1052.1 \times10^565.0±0.2%65.0 \pm0.2\%
PNAM (w=0w = 0)3 layers: 8×8 \times [1, 64, 32, 1]9.8×1049.8 \times10^491.5±0.4%91.5 \pm0.4\%
PNAM (w=0.001w = 0.001)3 layers: 8×8 \times [1, 64, 32, 1]9.8×1049.8 \times10^485.5±0.3%85.5 \pm0.3\%
PNAM (w=0.01w = 0.01)3 layers: 8×8 \times [1, 64, 32, 1]9.8×1049.8 \times10^469.9±6.4%69.9 \pm6.4\%

Table A.3.

Table A.4: Ranking of important features based on the mean absolute coefficients for the PNAM with w=0.001w = 0.001 in Table A.3. The mean and standard deviation of the test accuracy (associated with retaining only the top nn features) are computed over the runs in which the top nn features appeared.

RankInput featureSymbolFrequencyTest acc.
1Re(meridional translation)χ10\chi_{10}10/1055.7±1.2%55.7 \pm1.2\%
2Longitudinal translationχ8\chi_86/1074.8±1.4%74.8 \pm1.4\%
2Torsion degreeχ2\chi_24/1054.9±0.5%54.9 \pm0.5\%
3Torsion degreeχ2\chi_23/1076.4±0.4%76.4 \pm0.4\%
3Re(short geodesic)χ3\chi_33/1061.8±7.3%61.8 \pm7.3\%
3Longitudinal translationχ8\chi_82/1076.5±0.2%76.5 \pm0.2\%
3Cusp volumeχ7\chi_71/1081.3%81.3\%
3Im(short geodesic)χ4\chi_41/1074.3%74.3\%

Table A.4.

Figure A.2 shows a complete ranking of the inputs and the associated test accuracy of using the top nn inputs from one of the runs summarized in Table A.4. The ranking in Fig. A.2(d) is determined from the mean absolute coefficients of the linear transformation TT in Fig. A.2(b). As intended by the regularization in § 2.3, the learned TT (i) is predominantly diagonal due to the penalty ℓ3\ell_3 in Eq. (14) and (ii) is sparse due to the constraint ℓ4\ell_4 in Eq. (15), while retaining the essential off-diagonal entries. The post-processed transformation TT with n=3n = 3 in Fig. A.2(c) and the weighting coefficients ζ\zeta of just two nonzero bases in Fig. A.2(a) are then leveraged to construct expressions HH and II in Table A.5. Although each basis is evolved for only one minute using a few basic operators, Fig. A.3 demonstrates that SR can consistently approximate the single-variable bases because of their simple 1D nature. Expressions HH and II achieve test accuracies of 81.2 and 81.4%, respectively, comparable to the test accuracy of 81.3% in Fig. A.2(e) for n=3n = 3 inputs. Moreover, Table A.6 reveals that both expressions are more than two times faster for inference than the fully connected MLP.

Recall that the above results correspond to the regularization weight w=0.001w = 0.001, chosen to balance the trade-off between accuracy and sparsity. While increasing ww can make PNAMs sparser, Fig. A.4 indicates that overprioritizing sparsity can lead to unstable training, resulting in poorer performance. Specifically, setting w=0.01w = 0.01 results in such a sparse TT (see the small coefficient values in Fig. A.5(c)) that the accuracy in Fig. A.5(d) is poor regardless of nn. On the other hand, Fig. A.5(b) suggests that setting w=0w = 0—thereby deactivating the regularization constraints—improves the accuracy for large nn, due to the extensive number of coefficients with large values in Fig. A.5(a), but reduces it for small nn, due to the erroneous ranking of the input features. Although the exact ranking may not be correct, this PNAM can still narrow down a set of variables (e.g., n=6n = 6) that is relevant for predictive accuracy while remaining competitive with the MLP in Table A.3.

High-dimensional image classification with the MNIST dataset

To demonstrate that PNAMs can efficiently handle high-dimensional problems with numerous inputs, we evaluate their performance on the MNIST dataset [47], highlighting their dimensionality reduction capability and providing visual insight into their input pruning mechanism. The MNIST dataset contains 70,000 images of handwritten digits from 0 to 9, each of size 28×2828 \times28 pixels, partitioned into

Figure A.2. One complete instance of the ranking and accuracy reported in Table A.4, along with optimized parameters of the PNAM. (a) Weighting coefficients ζ\zeta prior to post-processing, (b) linear transformation TT prior to post-processing, (c) post-processed TT, retaining only the top n=3n = 3 features and zeroing out the rows that correspond to the zero-valued columns of ζ\zeta in (a), (d) relative importance of the input features based on the mean absolute coefficients of TT in (b), and (e) test accuracy associated with using the top nn features in (d).

Figure A.3. Reproducibility of SR for learning 1D functions. Both sets of bases (PNAM) are from the run in Fig. A.2, with (a) the two bases (SR) comprising expression H and (b) the two bases (SR) comprising expression I in Table A.5. The neural network parameterization of the PNAM, expression H, and expression I all use n=3n = 3 features and achieve test accuracies of about 81%.

60,000 images for training/validation and 10,000 images for testing. Each image is reshaped into a 784-dimensional feature vector of raw pixel intensities, resulting in a high-dimensional classification

IDExpressionPC†Discovered byEval. of test acc. ReportedEval. of test acc. OurTotal acc.
Aχ8χ102(χ102+χ92)\frac{\chi_8\chi_{10}}{2(\chi_{10}^{2}+\chi_9^{2})}3Human (DM)83.1%74.5%73.8%
B−0.02sin⁡(4.98χ9+0.85)+0.08∣4.02χ10+6.28∣−0.52−0.04e−0.88(1−0.45χ8)2-0.02\sin(4.98\chi_9+0.85)+0.08\lvert4.02\chi_{10}+6.28\rvert-0.52-\frac{0.04e^{-0.88(1-0.45\chi_8)^2}}{}12[3, 1] KAN62.6%27.0%26.8%
C0.17tan⁡(−1.51+0.1e−1.43(1−0.4χ9)2+0.09e−0.06(1−0.21χ8)2+1.32e−3.18(1−0.43χ10)2)0.17\tan(-1.51+0.1e^{-1.43(1-0.4\chi_9)^2}+0.09e^{-0.06(1-0.21\chi_8)^2}+1.32e^{-3.18(1-0.43\chi_{10})^2})17[3, 1, 1] KAN71.9%41.7%41.5%
D−0.09+1.04exp⁡(−9.59(−0.62sin⁡(0.61χ10+7.26))−0.32tan⁡(0.03χ8−6.59)+1−0.11e−1.77(0.31−χ9)2−1.09e−7.6(0.65(1−0.01χ8)3+0.27arctan⁡(0.53χ9−0.6)+0.09+exp⁡(−2.58(1−0.36χ10)2))-0.09+1.04\exp(-9.59(-0.62\sin(0.61\chi_{10}+7.26))-0.32\tan(0.03\chi_8-6.59)+1-0.11e^{-1.77(0.31-\chi_9)^2}-1.09e^{-7.6(0.65(1-0.01\chi_8)^3+0.27\arctan(0.53\chi_9-0.6)+0.09+\exp(-2.58(1-0.36\chi_{10})^2)})29[3, 2, 1] KAN84.0%––
E4.76χ8χ102(3.09χ9+6.05χ102+3.54χ92)\frac{4.76\chi_8\chi_{10}}{2(3.09\chi_9+6.05\chi_{10}^{2}+3.54\chi_9^{2})}7[3, 2, 1] KAN + Padé approx.82.8%79.3%79.3%
F2.94−2.92(1−0.10χ10)20.32(0.18−χ10)2+5.36(1−0.04χ8)2+0.50\frac{2.94-2.92(1-0.10\chi_{10})^2}{0.32(0.18-\chi_{10})^2+5.36(1-0.04\chi_8)^2+0.50}13[3, 1] KAN [3, 1] KAN77.8%27.0%26.8%
G12.766(0.132(−χ10+0.035χ8+0.157)2+0.592(−0.23χ10+0.008χ8+1)2(0.162χ10−0.006χ8+0.076)−1)4+7.202(0.871χ10+0.029χ8−(0.229χ10+0.008χ8−0.103)(0.229χ10+0.008χ8+0.159(χ10+0.033χ8−0.449)2+0.625)−0.173)2−10.64312.766(0.132(-\chi_{10}+0.035\chi_8+0.157)^2+0.592(-0.23\chi_{10}+0.008\chi_8+1)^2(0.162\chi_{10}-0.006\chi_8+0.076)-1)^4+7.202(0.871\chi_{10}+0.029\chi_8-(0.229\chi_{10}+0.008\chi_8-0.103)(0.229\chi_{10}+0.008\chi_8+0.159(\chi_{10}+0.033\chi_8-0.449)^2+0.625)-0.173)^2-10.643318×[1,64,32,1]8\times[1,64,32,1] PNAM (n=2n=2)–75.9%75.7%
H26((0.096χ10−0.002χ7+0.004χ8−0.237)(0.267χ10−0.004χ7+0.011χ8+0.119(χ10−0.016χ7+0.04χ8−0.316)2−1)2+0.133)(0.734χ10−0.012χ7+0.029χ8+0.418)+2.054(0.589χ10+0.018χ7−0.027χ8+1)2−4.056(χ10+0.031χ7−0.046χ8+0.195)2(0.107χ10+0.003χ7−0.005χ8−0.064(χ10+0.031χ7−0.046χ8+0.087(χ10+0.031χ7−0.046χ8−0.165(χ10+0.031χ7−0.046χ8+0.195)2+0.195)2+0.195)2+1)2−0.89526((0.096\chi_{10}-0.002\chi_7+0.004\chi_8-0.237)(0.267\chi_{10}-0.004\chi_7+0.011\chi_8+0.119(\chi_{10}-0.016\chi_7+0.04\chi_8-0.316)^2-1)^2+0.133)(0.734\chi_{10}-0.012\chi_7+0.029\chi_8+0.418)+2.054(0.589\chi_{10}+0.018\chi_7-0.027\chi_8+1)^2-4.056(\chi_{10}+0.031\chi_7-0.046\chi_8+0.195)^2(0.107\chi_{10}+0.003\chi_7-0.005\chi_8-0.064(\chi_{10}+0.031\chi_7-0.046\chi_8+0.087(\chi_{10}+0.031\chi_7-0.046\chi_8-0.165(\chi_{10}+0.031\chi_7-0.046\chi_8+0.195)^2+0.195)^2+0.195)^2+1)^2-0.895528×[1,64,32,1]8\times[1,64,32,1] PNAM (n=3n=3)–81.2%80.9%
I2.574χ10+0.078χ7−0.13χ8+26(0.233(−χ10+0.016χ7−0.04χ8−0.597)2sin⁡2(0.367χ10−0.006χ7+0.015χ8−0.719)+sin⁡(0.777χ10−0.013χ7+0.031χ8+0.262))(0.168χ10−0.003χ7+0.007χ8−0.053)+15.626(−sin⁡(0.54χ10+0.017χ7−0.025χ8+0.105)−0.049)2(−0.165χ10−0.005χ7+0.008χ8−0.645)+0.5092.574\chi_{10}+0.078\chi_7-0.13\chi_8+26(0.233(-\chi_{10}+0.016\chi_7-0.04\chi_8-0.597)^2\sin^2(0.367\chi_{10}-0.006\chi_7+0.015\chi_8-0.719)+\sin(0.777\chi_{10}-0.013\chi_7+0.031\chi_8+0.262))(0.168\chi_{10}-0.003\chi_7+0.007\chi_8-0.053)+15.626(-\sin(0.54\chi_{10}+0.017\chi_7-0.025\chi_8+0.105)-0.049)^2(-0.165\chi_{10}-0.005\chi_7+0.008\chi_8-0.645)+0.509348×[1,64,32,1]8\times[1,64,32,1] PNAM (n=3n=3)–81.4%81.0%

Table A.5. Mathematical expressions for the knot dataset. Inputs χ7\chi_7, χ8\chi_8, χ9\chi_9, and χ10\chi_{10} are the cusp volume, longitudinal translation, meridional translation (imaginary part), and meridional translation (real part), respectively. Expressions A to F are reproduced from Table 4 of Liu et al. [14]. A factor of 1/21/2 is added to expression A for consistency with DeepMind’s findings [46, 58]. Expression D has missing parentheses, so it cannot be evaluated. A factor of 1/21/2 is added to expression E for consistency with expression A. For expressions G and H, bases {gij}\{g_{ij}\} in Eq. (16) use addition, subtraction, multiplication, and square as operators; they additionally use exponential, sine, and tangent for expression I. Every constant in the expressions is counted as one parameter. See our code (footnote 2) to reproduce these results.

MethodTraining time (s/epoch)Inference time (ns/sample)Test acc.
MLP2.129094.7±0.2%94.7\pm0.2\%
NAM14430065.0±0.2%65.0\pm0.2\%
PNAM8.9210085.5±0.3%85.5\pm0.3\%
Expression G–7875.9%
Expression H–13081.2%
Expression I–12081.4%

Table A.6. Training and inference times of the neural networks in Table A.3, along with inference times of expressions G to I in Table A.5. Each basis comprising expressions G to I is evolved for one minute. Training times are obtained from an NVIDIA A100-SXM4-40GB GPU, while inference times are obtained from an Apple M1 CPU with 8 GB of memory. The test accuracy is repeated to aid comparison.

task where the model must map dense feature vectors to their corresponding digit labels. This experiment illustrates the ability of PNAMs to preserve interpretability while achieving strong predictive accuracy by identifying and utilizing only the most informative inputs, effectively pruning irrelevant features.

Figure A.4. Effects of regularization weights w1=⋯=w5=ww_1 = \cdots= w_5 = w on the convergence of PNAMs for the knot dataset. Total loss corresponds to the loss function in Eq. (12). The three PNAMs use the same projection dimension M=8M = 8, and each basis MLP has two hidden layers of 64 and 32 neurons.

Figure A.5. Effects of regularization weights w1=⋯=w5=ww_1 = \cdots= w_5 = w on the input ranking and test accuracy of PNAMs for the knot dataset. Regularization weight w=0w = 0 for (a) and (b); w=0.01w = 0.01 for (c) and (d). (a) and (c): Relative importance of the input features based on the mean absolute coefficients. (b) and (d): Test accuracy associated with using the top nn features in (a) and (c), respectively. Results for w=0.001w = 0.001 are provided in Fig. A.2(d) and (e).

In Table A.7, we compare PNAMs against a standard NAM and fully connected MLPs, all of which are trained for up to 1000 epochs with an early stopping patience of 50 epochs. First, the linear transformation TT allows us to project the original 784-dimensional vector into a lower-dimensional feature space of size MM. Even with a small M=8M = 8, the resulting PNAM outperforms the standard NAM, despite the latter requiring one basis per input and having nearly 100 times more parameters. Second, increasing MM enhances the expressivity of PNAMs, allowing them to fit the training data perfectly (see Fig. A.6) and achieve a test accuracy of 98.1% for M=64M = 64. Third, although PNAMs perform similarly to MLPs, suggesting comparable expressive power in practice, they retain the key advantage of interpretability. Unlike black-box MLPs, PNAMs offer insight into which variables are crucial for predictive accuracy, enabling model transparency and post-hoc analysis.

MethodArchitectureParameter countTraining acc.Test acc.
MLP3 layers: [784, 64, 32, 10]1.2×1041.2 \times10^{4}99.9%96.1%
MLP3 layers: [784, 128, 64, 10]4.9×1044.9 \times10^{4}99.9%96.3%
MLP4 layers: [784, 128, 64, 32, 10]6.6×1046.6 \times10^{4}99.9%96.7%
MLP4 layers: [784, 256, 128, 64, 10]2.6×1052.6 \times10^{5}99.9%97.1%
NAM3 layers: 784×784 \times [1, 64, 32, 10]9.6×1069.6 \times10^{6}95.0%93.0%
PNAM3 layers: 8×8 \times [1, 64, 32, 10]1.0×1051.0 \times10^{5}96.0%94.7%
PNAM3 layers: 16×16 \times [1, 64, 32, 10]2.1×1052.1 \times10^{5}98.5%96.7%
PNAM3 layers: 32×32 \times [1, 64, 32, 10]4.2×1054.2 \times10^{5}100%97.5%
PNAM3 layers: 64×64 \times [1, 64, 32, 10]8.4×1058.4 \times10^{5}100%98.1%

Table A.7. Performance comparison of different neural network architectures for the multi-class classification problem of predicting handwritten digits from the MNIST dataset. Training and validation histories of the PNAMs are provided in Fig. A.6.

Figure A.6. Effects of the projection dimension MM on the convergence and accuracy of PNAMs for the MNIST dataset. Training and validation histories of (a) the loss function in Eq. (12) and (b) the accuracy. Each basis MLP has two hidden layers of 64 and 32 neurons; all models are trained using regularization weights w1=⋯=w5=0.01w_1 = \cdots= w_5 = 0.01.

Figure A.7 illustrates that PNAMs produce a consistent ranking of feature importance across different values of MM, demonstrating robustness in their learned feature sensitivity. This consistency supports effective pruning of nonessential inputs with minimal impact on accuracy. Specifically, for the PNAM with M=64M = 64, we remove the least important pixels and visualize two progressively smaller subsets in Fig. A.8. Retaining only the top 400 and 200 relevant pixels, this model still achieves test accuracies of 98.0 and 94.0%, respectively. Visual inspection of the remaining pixels reveals that they occupy semantically meaningful regions, allowing human observers to easily identify the underlying digit classes. Further insight is provided by Fig. A.9, which shows that the majority of predictive information is concentrated in a small number of dominant singular values, enabling both dimensionality reduction and feature pruning.

Finally, we examine the impact of the regularization weights w1,…,w5w_1,\ldots,w_5 on the performance of PNAMs. Similar to the trend observed in Fig. A.4, Fig. A.10 indicates that increasing the regularization weights leads to less stable training as the optimization objective shifts from prioritizing predictive accuracy to enforcing sparsity. Nevertheless, as shown in Table A.8, these regularization weights can be tuned to obtain a desired trade-off between accuracy and sparsity, tailored to the specific needs of the user.

Figure A.7. Impact of input pruning on the test accuracy for the PNAMs in Table A.7. Unimportant inputs are pruned according to the mean absolute coefficients (see § 2.4).

Figure A.8. Demonstration of important inputs for the PNAM with M=64M = 64 in Fig. A.7. Shown are the first 10 unique handwritten digits in the MNIST test set, arranged from 0 to 9, using (a) the n=400n = 400 (∼50%\sim50\%) and (b) the n=200n = 200 (∼25%\sim25\%) most relevant pixels, which achieve test accuracies of 98.0 and 94.0%, respectively.

Figure A.9. Singular values of the linear transformation TT for the PNAMs in Table A.7. The linear decay and sudden drop in magnitude of the singular values on a log scale suggest that we can prune coefficients in TT without severely degrading performance.

Phase field theory for fracture of brittle solids

We consider a phase field simulation of fracture in boron carbide (B4C\mathrm{B}_4\mathrm{C}) with isotropic elasticity and isotropic fracture energy from [48], for quasi-static extension up to peak load. The dataset contains 96 samples,17 homogenized from this simulation. These samples are snapshots along a single loading path, and the 80/20 train-test split of § 3 is sampled randomly along this path, so the reported mean squared error (MSE) measures interpolation within the loading path. Given the homogenized values of the axial strain, order parameter, and magnitude of the material gradient of the order parameter, the goal is to predict the average strain energy, phase energy, and axial stress. Among the inputs, the order parameter measures local damage from 0 (undamaged) to 1 (fully fractured), and the magnitude of its material gradient measures how localized the damage is. Physically, the strain energy is the elastic energy stored by deformation, the phase energy is the energy expended in creating fracture surfaces, and the stress is their total rate of change with strain.

Figure A.10. Effects of regularization weights w1=⋯=w5=ww_1 = \cdots= w_5 = w on the convergence and accuracy of PNAMs for the MNIST dataset. Training and validation histories of (a) the loss function in Eq. (12) and (b) the accuracy. The four PNAMs use the same projection dimension M=8M = 8, and each basis MLP has two hidden layers of 64 and 32 neurons.

wwTraining acc.Test acc.Percent of zero TjkT_{jk}
0.00195.6%94.3%14.1%
0.0196.0%94.7%32.8%
0.195.7%94.7%68.9%
190.9%91.3%76.5%

Table A.8. Trade-off between accuracy and sparsity due to regularization weights w1=⋯=w5=ww_1 = \cdots= w_5 = w for the PNAMs in Fig. A.10. An element with ∣Tjk∣<10−4\lvert T_{jk}\rvert< 10^{-4} is considered zero, since setting such an element to zero has little to no effect on the test accuracy.

Since the stress is obtained by differentiating the sum of the two energies, we frame this task as a two-output regression problem. The MSE is employed as L\mathcal{L} in Eq. (12) to predict the strain energy and phase energy; to predict the stress, we use the following form for the physical constraint LP\mathcal{L}_{P}.18

LP=wP1D∑ν=1D((y1,1′+y2,1′)∣ν−∂(y^1(χ;T,Θ,ζ,ϵ)+y^2(χ;T,Θ,ζ,ϵ))∂χ1∣ν)2\mathcal{L}_{P} = \frac{w_{P1}}{D} \sum_{\nu=1}^{D} \left( \left.(y'_{1,1} + y'_{2,1})\right|_{\nu} - \left.\frac{\partial(\hat{y}_{1}(\chi;T,\Theta,\zeta,\epsilon) + \hat{y}_{2}(\chi;T,\Theta,\zeta,\epsilon))}{\partial\chi_{1}}\right|_{\nu} \right)^2

where y1,1′y'_{1,1} and y2,1′y'_{2,1} are the derivatives of the strain energy and phase energy, respectively, with respect to the strain χ1\chi_1. We set the regularization weight wP1=1w_{P1} = 1.19 In addition, every training and validation batch is augmented with one reference point at the undeformed state (zero inputs with zero targets), anchoring the predicted energies and stress at the origin.

All variables are scaled to have zero min and unit max. Table A.9 summarizes the performance of the MLP, the NAM, and four parameterizations of the PNAM, all utilizing three hidden layers of 256 neurons and trained for up to 5000 epochs with an early stopping patience of 50 epochs. On average, the test MSE of PNAMs decreases as the projection dimension MM increases [3]. Nevertheless, a PNAM with M=8M = 8 achieves the lowest validation MSE among all 60 networks underlying Table A.9, and we select this network for symbolic distillation. We then set w6=1w_6 = 1 in Eq. (16) to ensure accurate stress predictions and convert this network with approximately 10610^6 parameters into a symbolic form with roughly 100 parameters. As illustrated in Figs. A.11 and A.12, by employing SiLU as the activation function AA in Eq. (3), all bases {gij}\{g_{ij}\} can be approximated as single-variable polynomials, listed in Table A.10. Predictions of the strain energy, phase energy, and stress for the linear combinations of these polynomials are shown in Fig. A.13, achieving a test MSE of 2.25×10−52.25 \times10^{-5}.

MethodArchitectureParameter countTest MSE
MLP4 layers: [3, 256, 256, 256, 2]2.6×1052.6 \times10^{5}(7.05±2.02)×10−5(7.05 \pm2.02) \times10^{-5}
NAM4 layers: 3×[1,256,256,256,2]3 \times[1, 256, 256, 256, 2]7.9×1057.9 \times10^{5}(1.12±0.62)×10−4(1.12 \pm0.62) \times10^{-4}
PNAM4 layers: 3×[1,256,256,256,2]3 \times[1, 256, 256, 256, 2]7.9×1057.9 \times10^{5}(3.31±2.11)×10−4(3.31 \pm2.11) \times10^{-4}
PNAM4 layers: 8×[1,256,256,256,2]8 \times[1, 256, 256, 256, 2]2.1×1062.1 \times10^{6}(1.34±0.91)×10−4(1.34 \pm0.91) \times10^{-4}
PNAM4 layers: 16×[1,256,256,256,2]16 \times[1, 256, 256, 256, 2]4.2×1064.2 \times10^{6}(8.01±3.76)×10−5(8.01 \pm3.76) \times10^{-5}
PNAM4 layers: 32×[1,256,256,256,2]32 \times[1, 256, 256, 256, 2]8.4×1068.4 \times10^{6}(6.47±2.71)×10−5(6.47 \pm2.71) \times10^{-5}

Table A.9. Performance comparison of different neural network architectures for the two-output regression problem of predicting the strain energy and phase energy, from which the stress is derived. The mean and standard deviation of the combined test MSE for all three variables are computed over 10 runs.

Figure A.11. Bases {g1j}\{g_{1j}\} and their derivatives. The expressions of {g1j}\{g_{1j}\} are presented in Table A.10.

Table A.11 reports training and inference times for the neural networks in Table A.9, alongside the inference time of the symbolic form in Table A.10. As expected, training and inference times increase with MM. However, converting the PNAM into a compact symbolic expression yields faster inference, consistent with the knot theory results in Table A.6. Finally, Fig. A.14 reveals that increasing MM improves performance but may lead to overfitting after saturation.

Figure A.12. Bases {g2j}\{g_{2j}\} and their derivatives. The expressions of {g2j}\{g_{2j}\} are presented in Table A.10.

Bases {gij}\{g_{ij}\}Projected variables {zj}\{z_j\}
BasisExpressionVariableExpression
g1,1g_{1,1}−z13+1.524z12+z1+((−z1+(z1−0.028)2+0.75)2−0.102)2−0.269-z_1^3 + 1.524z_1^2 + z_1 + ((-z_1 + (z_1 - 0.028)^2 + 0.75)^2 - 0.102)^2 - 0.269z1z_1203.741χ1+1.87χ2−3.048χ3203.741\chi_1 + 1.87\chi_2 - 3.048\chi_3
g2,1g_{2,1}z12−(z1−0.597)(z1−0.291)(1.378z1(0.786z1−1)4+0.525z1−0.074)−0.093z_1^2 - (z_1 - 0.597)(z_1 - 0.291)(1.378z_1(0.786z_1 - 1)^4 + 0.525z_1 - 0.074) - 0.093
g1,2g_{1,2}z23(0.147−0.074z2)+0.383z22+0.119z2−0.119z_2^3(0.147 - 0.074z_2) + 0.383z_2^2 + 0.119z_2 - 0.119z2z_2235.816χ1−0.461χ2+2.333χ3235.816\chi_1 - 0.461\chi_2 + 2.333\chi_3
g2,2g_{2,2}−10.229z2(0.003−0.002z2)+0.011-10.229z_2(0.003 - 0.002z_2) + 0.011
g1,3g_{1,3}0.967z3(z3−1.108)(−3.255z32+z3−1.418)−2z3+0.070.967z_3(z_3 - 1.108)(-3.255z_3^2 + z_3 - 1.418) - 2z_3 + 0.07z3z_3−114.828χ1+1.284χ2+12.089χ3-114.828\chi_1 + 1.284\chi_2 + 12.089\chi_3
g2,3g_{2,3}1.046z3+2.265(z3−0.403)2(0.65z32−z3+0.835)2−0.251.046z_3 + 2.265(z_3 - 0.403)^2(0.65z_3^2 - z_3 + 0.835)^2 - 0.25
g1,4g_{1,4}−z4(0.009z4+0.069)+0.005-z_4(0.009z_4 + 0.069) + 0.005z4z_4−185.754χ1+1.866χ2+2.19χ3-185.754\chi_1 + 1.866\chi_2 + 2.19\chi_3
g2,4g_{2,4}00
g1,5g_{1,5}0.118z5(2z5+0.217)(0.033z5(z5+1)−z52+3.239)−0.1160.118z_5(2z_5 + 0.217)(0.033z_5(z_5 + 1) - z_5^2 + 3.239) - 0.116z5z_5190.698χ1+0.282χ2+6.271χ3190.698\chi_1 + 0.282\chi_2 + 6.271\chi_3
g2,5g_{2,5}−z5(−0.03z5+0.003(0.772z5(z5−0.466)2−z5+0.596)2+0.009)−0.002-z_5(-0.03z_5 + 0.003(0.772z_5(z_5 - 0.466)^2 - z_5 + 0.596)^2 + 0.009) - 0.002
g1,6g_{1,6}00z6z_61.503χ21.503\chi_2
g2,6g_{2,6}z6(0.003z62+0.009z6−0.043)+0.016z_6(0.003z_6^2 + 0.009z_6 - 0.043) + 0.016
g1,7g_{1,7}z7((z7(z7+0.917)2−0.014)+0.079z7−0.087)−0.029z_7((z_7(z_7 + 0.917)^2 - 0.014) + 0.079z_7 - 0.087) - 0.029z7z_7−119.681χ1+0.531χ2−5.929χ3-119.681\chi_1 + 0.531\chi_2 - 5.929\chi_3
g2,7g_{2,7}−0.004z73+0.009z72+0.024z7+0.007-0.004z_7^3 + 0.009z_7^2 + 0.024z_7 + 0.007
g1,8g_{1,8}0.003−0.009z80.003 - 0.009z_8z8z_8−27.984χ1−0.769χ2+6.628χ3-27.984\chi_1 - 0.769\chi_2 + 6.628\chi_3
g2,8g_{2,8}00

Table A.10. Mathematical expressions for the phase field dataset. Inputs χ1\chi_1, χ2\chi_2, and χ3\chi_3 are the strain, order parameter, and magnitude of the material gradient of the order parameter, respectively. To convert the PNAM with M=8M = 8 in Table A.9 and T0=ζ0=0.05T_0 = \zeta_0 = 0.05 (see § 2.4) into symbolic form, bases {gij}\{g_{ij}\} in Eq. (16) use addition, subtraction, multiplication, and square as operators. Counting every constant in {gij}\{g_{ij}\} and {zj}\{z_j\} as one parameter, the symbolic form has 112 parameters and achieves a test MSE of 2.25×10−52.25 \times10^{-5}.

Figure A.13. Test predictions for the linear combinations of the polynomials {gij}\{g_{ij}\} in Table A.10. (a) Linear combination of {g1j}\{g_{1j}\}, (b) linear combination of {g2j}\{g_{2j}\}, and (c) derivative of the sum of the linear combinations of {g1j}\{g_{1j}\} and {g2j}\{g_{2j}\} with respect to the strain.

MethodTraining time (s/epoch)Inference time (μ\mus/sample)Test MSE
MLP1.127(7.05±2.02)×10−5(7.05 \pm2.02) \times10^{-5}
NAM3.061(1.12±0.62)×10−4(1.12 \pm0.62) \times10^{-4}
PNAM (M=3M = 3)3.258(3.31±2.11)×10−4(3.31 \pm2.11) \times10^{-4}
PNAM (M=8M = 8)6.8150(1.34±0.91)×10−4(1.34 \pm0.91) \times10^{-4}
PNAM (M=16M = 16)13320(8.01±3.76)×10−5(8.01 \pm3.76) \times10^{-5}
PNAM (M=32M = 32)26610(6.47±2.71)×10−5(6.47 \pm2.71) \times10^{-5}
Expression (Table A.10)–172.25×10−52.25 \times10^{-5}

Table A.11. Training and inference times of the neural networks in Table A.9, along with the inference time of the combined expression (i.e., the sum of the linear combinations of {g1j}\{g_{1j}\} and {g2j}\{g_{2j}\}) in Table A.10. Each nonzero basis gijg_{ij} is evolved for one minute. Training times are obtained from an NVIDIA A100-SXM4-40GB GPU, while inference times are obtained from an Apple M1 CPU with 8 GB of memory. The test MSE is repeated to aid comparison.

Figure A.14. Effects of the projection dimension MM on the convergence of PNAMs for the phase field dataset. Total loss corresponds to the loss function in Eq. (12). Each basis MLP has three hidden layers of 256 neurons; all models are trained using regularization weights w1=⋯=w5=0.001w_1 = \cdots= w_5 = 0.001.

References

  1. [1]W. J. Murdoch, C. Singh, K. Kumbier, R. Abbasi-Asl, and B. Yu. Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences, 116(44):22071–22080, 2019.DOI
  2. [2]R. Agarwal, L. Melnick, N. Frosst, X. Zhang, B. Lengerich, R. Caruana, and G. E. Hinton. Neural additive models: Interpretable machine learning with neural nets. Advances in Neural Information Processing Systems, 34(359):4699–4711, 2021.arxiv.org/abs/2004.13912
  3. [3]N. N. Phan, W.C. Sun, and J. D. Clayton. HYDRA: Symbolic feature engineering of overparameterized Eulerian hyperelasticity models for fast inference time. Computer Methods in Applied Mechanics and Engineering, 437:117792, 2025.DOI
  4. [4]Y. Lou, R. Caruana, J. Gehrke, and G. Hooker. Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 623–631, 2013.DOI
  5. [5]C.-H. Chang, R. Caruana, and A. Goldenberg. NODE-GAM: Neural generalized additive model for interpretable deep learning. arXiv preprint arXiv:2106.01613, 2021.arxiv.org/abs/2106.01613
  6. [6]Z. Yang, A. Zhang, and A. Sudjianto. GAMI-Net: An explainable neural network based on generalized additive models with structured interactions. Pattern Recognition, 120:108192, 2021.arxiv.org/abs/2003.07132
  7. [7]A. Dubey, F. Radenovic, and D. Mahajan. Scalable interpretability via polynomials. Advances in Neural Information Processing Systems, 35:36748–36761, 2022.arxiv.org/abs/2205.14108
  8. [8]J. Enouen and Y. Liu. Sparse interaction additive networks via feature interaction detection and sparse selection. Advances in Neural Information Processing Systems, 35:13908–13920, 2022.arxiv.org/abs/2209.09326
  9. [9]T. Hastie and R. Tibshirani. Generalized additive models. Statistical Science, 1(3):297–310, 1986.
  10. [10]V. E. Maiorov. On best approximation by ridge functions. Journal of Approximation Theory, 99(1):68–94, 1999.DOI
  11. [11]A. Pinkus. Approximation theory of the MLP model in neural networks. Acta Numerica, 8:143–195, 1999.DOI
  12. [12]C. Molnar. Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. Leanpub, 2019.
  13. [13]Rudin, C. Chen, Z. Chen, H. Huang, L. Semenova, and C. Zhong. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistics Surveys, 16:1–85, 2022.arxiv.org/abs/2103.11251
  14. [14]Liu, Y. Wang, S. Vaidya, F. Ruehle, J. Halverson, M. Soljacic, T. Hou, and M. Tegmark. KAN: Kolmogorov–Arnold networks. In International Conference on Learning Representations, pages 70367–70413, 2025.DOI
  15. [15]G. G. Chrysos, S. Moschoglou, G. Bouritsas, J. Deng, Y. Panagakis, and S. Zafeiriou. Deep polynomial neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4021–4034, 2022.DOI
  16. [16]M. Cranmer, A. Sanchez-Gonzalez, P. Battaglia, R. Xu, K. Cranmer, D. Spergel, and S. Ho. Discovering symbolic models from deep learning with inductive biases. Advances in Neural Information Processing Systems, 33(1462):17429–17442, 2020.
  17. [17]S. L. Brunton, J. L. Proctor, and J. N. Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 113(15):3932–3937, 2016.DOI
  18. [18]G. Fang, X. Ma, M. Song, M. B. Mi, and X. Wang. DepGraph: Towards any structural pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16091–16101, 2023.arxiv.org/abs/2301.12900
  19. [19]M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, volume 70, pages 3319–3328. PMLR, 2017.arxiv.org/abs/1703.01365
  20. [20]A. Odrzywolek. All elementary functions from a single binary operator. arXiv preprint arXiv:2603.21852, 2026.
  21. [21]B. K. Petersen, M. Landajuela, T. N. Mundhenk, C. P. Santiago, S. K. Kim, and J. T. Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. arXiv preprint arXiv:1912.04871, 2019.arxiv.org/abs/1912.04871
  22. [22]S.-M. Udrescu and M. Tegmark. AI Feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16):eaay2631, 2020.DOI
  23. [23]M. Cranmer. Interpretable machine learning for science with PySR and SymbolicRegression.jl. arXiv preprint arXiv:2305.01582, 2023.arxiv.org/abs/2305.01582
  24. [24]M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33(632):7537–7547, 2020.arxiv.org/abs/2006.10739
  25. [25]B. Bahmani, H. S. Suh, and W.C. Sun. Discovering interpretable elastoplasticity models via the neural polynomial method enabled symbolic regressions. Computer Methods in Applied Mechanics and Engineering, 422:116827, 2024.DOI
  26. [26]F. Radenovic, A. Dubey, and D. Mahajan. Neural basis models for interpretability. Advances in Neural Information Processing Systems, 35(612):8414–8426, 2022.arxiv.org/abs/2205.14120
  27. [27]M. H. Stone. Applications of the theory of Boolean rings to general topology. Transactions of the American Mathematical Society, 41(3):375–481, 1937.
  28. [28]M. H. Stone. The generalized Weierstrass approximation theorem. Mathematics Magazine, 21 (4):167–184, 1948.
  29. [29]N. E. Cotter. The Stone–Weierstrass theorem and its application to neural networks. IEEE Transactions on Neural Networks, 1(4):290–295, 1990.pubmed.ncbi.nlm.nih.gov/18282849
  30. [30]K. Hornik, M. Stinchcombe, and H. White. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359–366, 1989.DOI
  31. [31]M. Leshno, V. Y. Lin, A. Pinkus, and S. Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks, 6(6): 861–867, 1993.DOI
  32. [32]Z. Lu, H. Pu, F. Wang, Z. Hu, and L. Wang. The expressive power of neural networks: A view from the width. Advances in Neural Information Processing Systems, 30:6232–6240, 2017.arxiv.org/abs/1709.02540
  33. [33]R. A. DeVore and G. G. Lorentz. Constructive Approximation, volume 303 of A Series of Comprehensive Studies in Mathematics. Springer, Berlin, 1993.
  34. [34]D. Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks, 94: 103–114, 2017.DOI
  35. [35]S. Xu, Z. Bu, P. Chaudhari, and I. J. Barnett. Sparse neural additive model: Interpretable deep learning with feature selection via group sparsity. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, volume 3, pages 343–359. Springer, 2023.arxiv.org/abs/2202.12482
  36. [36]W. M. Czarnecki, S. Osindero, M. Jaderberg, G. Swirszcz, and R. Pascanu. Sobolev training for neural networks. Advances in Neural Information Processing Systems, 30:4281–4290, 2017.arxiv.org/abs/1706.04859
  37. [37]M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019.DOI
  38. [38]J.-H. Bastek, W.C. Sun, and D. Kochmann. Physics-informed diffusion models. In International Conference on Learning Representations, pages 3360–3385, 2025.arxiv.org/abs/2403.14404
  39. [39]A. Krogh and J. Hertz. A simple weight decay can improve generalization. Advances in Neural Information Processing Systems, 4:950–957, 1991.
  40. [40]D. Q. Huynh. Metrics for 3D rotations: Comparison and analysis. Journal of Mathematical Imaging and Vision, 35:155–164, 2009.DOI
  41. [41]R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267–288, 1996.
  42. [42]H. Cheng, M. Zhang, and J. Q. Shi. A survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):10558–10578, 2024.DOI
  43. [43]J. H. Holland. Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence. MIT Press, Cambridge, MA, 1992.
  44. [44]J. R. Koza. Genetic programming as a means for programming computers by natural selection. Statistics and Computing, 4:87–112, 1994.DOI
  45. [45]M. Virgolin and S. P. Pissis. Symbolic regression is NP-hard. arXiv preprint arXiv:2207.01018, 2022.arxiv.org/abs/2207.01018
  46. [46]A. Davies, P. Veličković, L. Buesing, S. Blackwell, D. Zheng, N. Tomašev, R. Tanburn, P. Battaglia, C. Blundell, A. Juhász, et al. Advancing mathematics by guiding human intuition with AI. Nature, 600:70–74, 2021.DOI
  47. [47]Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  48. [48]J. D. Clayton, J. Knap, and R. B. Leavy. On rate dependence and anisotropy in phase field modeling of polycrystalline fracture. Mechanics of Materials, 180:104606, 2023.DOI
  49. [49]A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32(721):8026–8037, 2019.arxiv.org/abs/1912.01703
  50. [50]D. Hendrycks and K. Gimpel. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415, 2016.
  51. [51]S. Elfwing, E. Uchibe, and K. Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3–11, 2018.DOI
  52. [52]D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.arxiv.org/abs/1412.6980
  53. [53]N. Macon and A. Spitzbart. Inverses of Vandermonde matrices. The American Mathematical Monthly, 65(2):95–100, 1958.DOI
  54. [54]L. R. Turner. Inverse of the Vandermonde matrix with applications. Technical report, NASA Lewis Research Center, Cleveland, OH (United States), 1966.
  55. [55]J. C. A. Barata and M. S. Hussein. The Moore–Penrose pseudoinverse: A tutorial review of the theory. Brazilian Journal of Physics, 42:146–165, 2012.DOI
  56. [56]J. M. Landsberg. Tensors: Geometry and Applications, volume 128 of Graduate Studies in Mathematics. American Mathematical Society, Providence, RI, 2012.DOI
  57. [57]J. Alexander and A. Hirschowitz. Polynomial interpolation in several variables. Journal of Algebraic Geometry, 4(2):201–222, 1995.
  58. [58]A. Davies, A. Juhász, M. Lackenby, and N. Tomašev. The signature and cusp geometry of hyperbolic knots. Geometry & Topology, 28(5):2313–2343, 2024.DOI
  59. [59]N. Mallinar, J. Simon, A. Abedsoltan, P. Pandit, M. Belkin, and P. Nakkiran. Being, tempered, or catastrophic: A taxonomy of overfitting. Advances in Neural Information Processing Systems, 35(87):1182–1195, 2022.arxiv.org/abs/2207.06569
  60. [60]M. E. Gurtin. Generalized Ginzburg–Landau and Cahn–Hilliard equations based on a microforce balance. Physica D: Nonlinear Phenomena, 92(3-4):178–192, 1996.DOI
  61. [61]C. Miehe, F. Welschinger, and M. Hofacker. Thermodynamically consistent phase-field models of fracture: Variational principles and multi-field FE implementations. International Journal for Numerical Methods in Engineering, 83(10):1273–1311, 2010.DOI

Paper details

vlnc:W_zxdubyu2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents