betaFeedback

WorksComputer Science

CP-MLPs: A Tensor-Rank Theory of Tied and Untied MLP Blocks

Abstract

Transformer feed-forward blocks combine several design choices: squared activations, gating, and tied versus untied parameter sharing, whose separate contributions to expressivity remain poorly understood. We introduce CP-MLPs, a tensor framework in which each multiplicative unit contributes a rank-1 interaction with three roles: detecting an input feature, reading a value, and writing to the output. Tied gated blocks such as ReLU2^2 force the detector and value roles to share the same direction; untied gated blocks such as ReGLU and SwiGLU let them differ. We show that this distinction yields exact, degree-wise tensor-rank separations: untied detector-value units are strictly more width-efficient for indefinite quadratic interactions and role-asymmetric monomials, while tied units are structurally matched to symmetric pure-power targets. The width advantage of untying therefore depends on the target structure. Experiments show loss plateaus below the exact-representation width, but these plateaus alone do not establish a positive approximation gap: arbitrarily accurate approximation is governed by border rank. In trained gated MLPs, untied blocks retain an advantage in selected comparisons at matched input-projection budget. Code is available at https://github.com/rarefin/CP-MLPs.

Introduction

Feed-forward layers account for roughly two-thirds of the parameters in a transformer [20] language model [12], and a wide range of designs have been proposed for them. Common variants include squared activations such as ReLU2\mathrm{ReLU}^{2} [18], gated blocks such as ReGLU and SwiGLU [17], and tied variants of SwiGLU. Bilinear MLPs are less common as training blocks in modern LLMs, but they provide a clean quadratic base case and have received attention in the interpretability literature [15]. These architectures differ not only in their gating nonlinearity, but also in whether the same direction is used both to detect a feature and to read out its value. In this work we ask when untying these two roles improves expressivity, in the sense of reducing the number of channels required to represent a given interaction.

ReLU2\mathrm{ReLU}^{2} and tied SwiGLU use the same direction for the detector and value. Untied blocks such as bilinear MLPs, ReGLU, and SwiGLU allow the two to differ. This decouples two design choices that are typically conflated: the choice of gate, and whether the detector and value branches are tied.

The comparison is naturally made after input symmetrization. A polynomial function evaluates its coefficient tensor on repeated copies of the same input, so only the input-symmetric part of the tensor contributes to the function it represents [9]. Tied and untied blocks therefore parameterize the same space of input-symmetric tensors, but through different families of atoms.

Untying helps when the target interaction is better matched to an untied detector-value atom than to a tied pure-power atom.

Our analysis gives exact width separations at fixed polynomial degree. At degree two, untied bilinear atoms can represent certain indefinite quadratic interactions with strictly fewer channels than tied atoms. At higher degree, role-asymmetric monomials are realized by a single untied detector-value atom but require multiple tied pure-power atoms. For smooth gates such as SwiGLU, these statements transfer degree by degree to the Taylor coefficient tensors. These degree-wise results give necessary, but not generally sufficient, width bounds for the full analytic function. The point is not that tying is uniformly worse: tied atoms are well suited to symmetric pure-power targets, and untied atoms to interactions that separate detection from value.

Our coefficient-space experiments distinguish exact rank, border rank, and numerical recovery (Appendices D.9–D.11). A positive approximation gap requires width below border rank, the minimum width permitting arbitrarily accurate approximation. Below exact rank but at or above border rank, the error infimum is zero despite possible numerical loss plateaus. Both families represent pure-power targets at width one, so differences there concern optimization. The untied advantage also appears in selected trained gated-MLP comparisons under parameter matching.

Contributions.

  1. A unified CP-MLP framework. We express bilinear, ReLU2^{2}, tied SwiGLU, ReGLU, and SwiGLU blocks in a shared atom-based tensor notation with detector, value, and output roles.

  1. Exact atom-rank separations. We prove that untied detector-value atoms achieve strictly lower width on indefinite quadratic interactions and role-asymmetric monomials, while tied atoms remain well matched to pure-power targets.

  1. Empirical link between exact rank, border rank, and loss. We distinguish positive approximation gaps from zero-infimum limits and optimization failures, and evaluate tied and untied gated MLPs under width and parameter-budget controls.

Related Work

Transformer MLP variants and interpretations. Transformer feed-forward designs have diversified considerably. GLU-style activations were introduced for language modeling by Dauphin et al. [ [10], and variants such as ReGLU and SwiGLU were later shown to improve Transformer feed-forward layers [17]. Squared-ReLU activations emerged as effective Transformer primitives through architecture-search studies [18], building on a longer line of activation-function design that includes Swish/SiLU [16]. In parallel, interpretability work has argued that Transformer feed-forward layers behave like structured key-value memories rather than generic nonlinear post-processing [12]. These works motivate studying MLP blocks as structured computational modules, but do not directly isolate the expressivity effect of tying versus untying detector and value branches, which is the focus of this paper.

Tensor, polynomial, and Waring-rank views. Our framework uses canonical polyadic (CP) tensor decomposition, where tensors are written as sums of rank-1 outer products [14]. Recent work on bilinear MLPs also exploits tensor structure to analyze feed-forward blocks, observing that bilinear layers admit a third-order tensor representation [15]. The TT-Net family of Chrysos et al. [ [6, 7] parameterizes polynomial expansions of the input directly through tensor decompositions, demonstrating that such polynomial blocks are competitive with standard activations on a range of tasks. Polynomial networks have also been studied with algebraic and tensor tools, for example through the geometry of polynomial function classes [13]. Our work differs in that we do not propose polynomial networks as an architecture; we use polynomial structure as an analytical lens for comparing existing MLP variants.

For the higher-degree separation we rely on classical Waring rank: a scalar tied degree-kk channel is a pure power (ℓTx)k(\ell^{\mathsf{T}}x)^k, so tied atom rank coincides with Waring rank. The monomial Waring-rank formula of Carlini et al. [ [3], together with the real-rank result of Carlini et al. [ [5], implies that x1k−1x2x_1^{k-1}x_2 requires kk tied atoms but only one untied detector-value atom. This exact-rank separation differs from the approximation threshold: the same monomial has tied border rank two [Comas and and Seiguer, 2011]. Our contribution is to connect these rank facts to practical MLP atom families, showing that tied and untied blocks draw on different atoms inside the same input-symmetric tensor space.

The CP-MLP framework

We use a common tensor notation for the multiplicative feed-forward blocks studied in this paper. Bilinear blocks multiply two linear projections, GLU-style blocks gate one projection by another, and square-style blocks reuse the same projection twice. In every case, each hidden channel produces a product of projected input directions, and the output matrix sums these channel-wise products. At each active degree, the raw coefficient tensor has the structure of a canonical polyadic (CP) decomposition [14], and we call the resulting representation a CP-MLP.

The framework separates two design choices: the gate, which selects the polynomial degrees that appear in a local expansion, and the tying pattern, which selects whether the detector and value directions of a channel are shared. The expressivity analysis in § 4 isolates the effect of the second choice.

Definition 3.1 (CP-MLP block). Fix a scalar gate g:R→Rg : \mathbb{R} \to\mathbb{R} and let G:Rm→RmG : \mathbb{R}^{m} \to\mathbb{R}^{m} be its coordinatewise extension, G(z)=(g(z1),…,g(zm))TG(z) = (g(z_1), \ldots, g(z_m))^{\mathsf{T}}. Given weights U,V∈Rm×dU, V \in\mathbb{R}^{m \times d} (rows uaT,vaTu_a^{\mathsf{T}}, v_a^{\mathsf{T}}) and W∈Rp×mW \in\mathbb{R}^{p \times m} (columns waw_a), a width-mm CP-MLP block is

F(x)=W(G(Ux)⊙Vx)=∑a=1mwag(uaTx)(vaTx).F(x) = W\big(G(Ux) \odot Vx\big) = \sum_{a=1}^{m} w_a g(u_a^{\mathsf{T}}x)(v_a^{\mathsf{T}}x).

We call uau_a the detector direction, vav_a the value direction, and waw_a the output direction of channel aa. A tied block constrains V=UV = U; an untied block parameterizes UU and VV separately, allowing equality as a special case. We consider bias-free blocks throughout.

The detector direction selects the scalar feature passed through the gate, while the value direction supplies the linear signal that the gate modulates. Tying forces both roles onto the same direction; untying lets them differ. Figure 1 shows the corresponding three-leg view of a single channel.

Leg diagram of a single CP-MLP channel

Figure 1. Leg diagram of a single CP-MLP channel. The channel computes ha(x)=g(uaTx)(vaTx)h_a(x) = g(u_a^{\mathsf{T}}x)(v_a^{\mathsf{T}}x) and writes the result through waw_a. At each active polynomial degree, the channel contributes one rank-1 CP atom with output, detector, and value legs.

Degree-wise atoms

Assume the gate is analytic at the origin, g(z)=∑j≥0cjzjg(z) = \sum_{j \ge0} c_j z^j (piecewise-analytic gates such as ReLU are treated region-wise; see Table 1). Substituting z=uaTxz = u_a^{\mathsf{T}}x and multiplying by vaTxv_a^{\mathsf{T}}x, one channel contributes ∑j≥0cj(uaTx)j(vaTx)\sum_{j \ge0} c_j(u_a^{\mathsf{T}}x)^j(v_a^{\mathsf{T}}x); reindexing by total degree k=j+1k = j + 1, the degree-kk term carries the coefficient ck−1c_{k-1}. Writing T[x,…,x]∈RpT[x,\ldots,x] \in\mathbb{R}^{p} for the contraction of the kk input modes of T∈Rp⊗(Rd)⊗kT \in\mathbb{R}^{p} \otimes(\mathbb{R}^{d})^{\otimes k} against xx (output mode free; conventions in Appendix § 8.§ 1), the degree-kk component of the block is F(k)(x)=T(k)[x,…,x]F^{(k)}(x) = T^{(k)}[x,\ldots,x], with raw coefficient tensor

BlockGate g(z)g(z)TyingSignature S(g)S(g)Effective atom
Bilinear [15]zzuntied{2}\{2\}w⊗Sym⁡(u⊗v)w \otimes\operatorname{Sym}(u \otimes v)
ReGLU [17]ReLU⁡(z)\operatorname{ReLU}(z)untied{2}\{2\}{sup}†`region-wise w⊗Sym⁡(u⊗v)w \otimes\operatorname{Sym}(u \otimes v)
SwiGLU [17]SiLU⁡(z)=zσ(z)\operatorname{SiLU}(z)=z\sigma(z)untied{2,3,5,7,…}\{2,3,5,7,\ldots\}w⊗Sym⁡(u⊗(k−1)⊗v)w \otimes\operatorname{Sym}(u^{\otimes(k-1)} \otimes v)
ReLU2 [18]ReLU⁡(z)\operatorname{ReLU}(z)tied{2}\{2\}{sup}†`region-wise w⊗u⊗2w \otimes u^{\otimes2}
Tied SwiGLUSiLU⁡(z)=zσ(z)\operatorname{SiLU}(z)=z\sigma(z)tied{2,3,5,7,…}\{2,3,5,7,\ldots\}w⊗u⊗kw \otimes u^{\otimes k}

†Region-wise: ReLU is piecewise linear, so the channel is a degree-2 polynomial inside each fixed activation region.

Table 1. Feed-forward blocks as CP-MLPs. The gate controls which degrees are active through its polynomial signature; the tying pattern controls whether detector and value directions are shared.

T(k)=ck−1∑a=1mwa⊗ua⊗(k−1)⊗va∈Rp⊗(Rd)⊗k.T^{(k)} = c_{k-1}\sum_{a=1}^{m} w_a \otimes u_a^{\otimes(k-1)} \otimes v_a \in\mathbb{R}^{p} \otimes(\mathbb{R}^{d})^{\otimes k}.

Each active degree thus admits a CP decomposition with at most mm rank-one terms, one per channel. We define S(g)\mathcal{S}(g) for the polynomial signature of the gate, i.e. the set of active degrees.

Definition 3.2 (Polynomial signature). The polynomial signature of gg is S(g):={k≥1:ck−1≠0}S(g):=\{k\ge1:c_{k-1}\ne0\}.

The gate selects which degrees are active through S(g)S(g); the tying pattern selects the atom used at each active degree. When va=uav_a=u_a, the raw degree-kk atom collapses to wa⊗ua⊗kw_a\otimes u_a^{\otimes k}; when vav_a is independent of uau_a, it retains the detector-value form wa⊗ua⊗(k−1)⊗vaw_a\otimes u_a^{\otimes(k-1)}\otimes v_a.

Effective atoms after symmetrization

The tensor above distinguishes detector and value slots, but the function it represents evaluates the tensor on repeated copies of the same input, so input-slot order is not observable. We write Sym⁡(T)\operatorname{Sym}(T) for the symmetrization of TT, the average of TT over all permutations of its input modes (with the output mode, when present, left fixed), then

T[x,…,x]=Sym⁡(T)[x,…,x].T[x,\ldots,x]=\operatorname{Sym}(T)[x,\ldots,x].

The proof is immediate: every permutation of TT receives the same vector xx in every input slot. A full statement, together with the quadratic matrix specialization, is given in Appendix § 9.§ 1.

After symmetrization, each architecture yields a family of input-symmetric atoms; an untied atom can have tensor rank greater than one.

Definition 3.3 (Effective tied and untied CP-MLP atoms). The degree-kk effective untied atom is

Auntied(k)(w,u,v):=w⊗Sym⁡(u⊗(k−1)⊗v)∈Rp⊗Sym⁡k(Rd),\mathcal{A}_{\mathrm{untied}}^{(k)}(w,u,v):=w\otimes\operatorname{Sym}(u^{\otimes(k-1)}\otimes v)\in\mathbb{R}^{p}\otimes\operatorname{Sym}^{k}(\mathbb{R}^{d}),

where Sym⁡k(Rd)\operatorname{Sym}^{k}(\mathbb{R}^{d}) denotes the space of symmetric order-kk tensors (Definition B.1). The degree-kk effective tied atom is the specialization v=uv=u:

Atied(k)(w,u):=w⊗u⊗k∈Rp⊗Sym⁡k(Rd).\mathcal{A}_{\mathrm{tied}}^{(k)}(w,u):=w\otimes u^{\otimes k}\in\mathbb{R}^{p}\otimes\operatorname{Sym}^{k}(\mathbb{R}^{d}).

Tied and untied blocks therefore parameterize tensors in the same symmetric space Rp⊗Sym⁡k(Rd)\mathbb{R}^{p}\otimes\operatorname{Sym}^{k}(\mathbb{R}^{d}), but draw on different atom families: tied blocks use pure-power atoms w⊗u⊗kw\otimes u^{\otimes k}, while untied blocks use symmetrized detector-value atoms w⊗Sym⁡(u⊗(k−1)⊗v)w\otimes\operatorname{Sym}(u^{\otimes(k-1)}\otimes v). Setting v=uv=u recovers the tied family as a special case. The benefit of untying is not that antisymmetric components become visible, but that the symmetrized untied family can represent certain symmetric tensors with strictly fewer atoms.

Table 1 places the blocks studied in this paper along the two axes of the framework: the gate and the tying pattern. Bilinear, ReGLU, and SwiGLU differ only in their gates and share the untied detector-value atom structure. ReLU2 and tied SwiGLU use tied atoms, with the same direction serving both the detector and value roles. Tied SwiGLU sets v=uv=u while retaining the SiLU gate, giving SiLU⁡(z)z=z2σ(z)\operatorname{SiLU}(z)z=z^{2}\sigma(z).

The next section asks how many atoms of each family are required to represent a given target tensor.

Expressivity via CP-MLP atom rank

The atom families of § 3.2 suggest a natural notion of width: the smallest number of atoms needed to express a given coefficient tensor. Fixing an active degree kk, each CP-MLP channel contributes one effective atom in Tk:=Rp⊗Sym⁡k(Rd)\mathcal{T}_k := \mathbb{R}^p \otimes\operatorname{Sym}^k(\mathbb{R}^d), so the question we want to answer is how many such atoms are required to represent a target T∈TkT \in\mathcal{T}_k. We call this minimum the CP-MLP atom rank; since each atom corresponds to one channel at degree kk, it is the degree-wise width required to represent a coefficient tensor.

For polynomial gates, T(k)T^{(k)} is the ordinary degree-kk coefficient tensor; for analytic gates, it is the degree-kk Taylor coefficient tensor, satisfying DkF(0)=k!Sym⁡(T(k))D^kF(0) = k! \operatorname{Sym}(T^{(k)}). Throughout this section kk denotes an active degree of the gate, and any nonzero scalar Taylor coefficient is absorbed into the output factor ww, so the rank comparisons below depend only on the channel atom family.

Atom-rank definitions

Definition 4.1 (Tied CP-MLP atom rank). For T∈TkT \in\mathcal{T}_k, the tied CP-MLP atom rank rtied(T)r_{\mathrm{tied}}(T) is the smallest integer mm such that

T=∑a=1mwa⊗ua⊗k,wa∈Rp,ua∈Rd.T = \sum_{a=1}^{m} w_a \otimes u_a^{\otimes k}, \qquad w_a \in\mathbb{R}^p,\quad u_a \in\mathbb{R}^d.

Definition 4.2 (Untied CP-MLP atom rank). For T∈TkT \in\mathcal{T}_k, the untied CP-MLP atom rank runtied(T)r_{\mathrm{untied}}(T) is the smallest integer mm such that

T=∑a=1mwa⊗Sym⁡(ua⊗(k−1)⊗va),wa∈Rp,ua,va∈Rd.T = \sum_{a=1}^{m} w_a \otimes\operatorname{Sym}\left(u_a^{\otimes(k-1)} \otimes v_a\right), \qquad w_a \in\mathbb{R}^p,\quad u_a,v_a \in\mathbb{R}^d.

Both ranks are minimum-width quantities at a fixed degree, counting the fewest atoms required to express a coefficient tensor in the corresponding family. Since a tied atom is recovered from an untied atom by setting va=uav_a = u_a, the untied family contains the tied family, which immediately implies the following inequality.

Proposition 4.3 (Untied atoms contain tied atoms). For every T∈TkT \in\mathcal{T}_k, runtied(T)≤rtied(T)r_{\mathrm{untied}}(T) \le r_{\mathrm{tied}}(T).

Proof. Any tied representation T=∑awa⊗ua⊗kT = \sum_a w_a \otimes u_a^{\otimes k} is also an untied representation with va=uav_a = u_a, since ua⊗k=Sym⁡(ua⊗(k−1)⊗ua)u_a^{\otimes k} = \operatorname{Sym}(u_a^{\otimes(k-1)} \otimes u_a). Minimizing over tied representations yields the claim.

The remainder of the section asks when this inequality is strict. We first study degree two, where the coefficient tensor is a symmetric matrix, and then prove a higher-degree separation using a role-asymmetric monomial.

Remark 4.4 (Scalar targets suffice). For a scalar target S∈Sym⁡k(Rd)S \in\operatorname{Sym}^k(\mathbb{R}^d), embedding T=eo⊗ST = e_o \otimes S gives rtied(T)=rtied(S)r_{\mathrm{tied}}(T) = r_{\mathrm{tied}}(S) and runtied(T)=runtied(S)r_{\mathrm{untied}}(T) = r_{\mathrm{untied}}(S): output projection cannot increase atom rank, and any scalar representation lifts by tensoring with eoe_o. Full details are in Appendix § 10.§ 1. The separations below are stated for scalar targets.

Degree-two bilinear atoms

We first specialize to degree 2, where a scalar coefficient tensor is a symmetric matrix and the atom-rank question reduces to representing quadratic forms

fM(x)=xTMx,M=MT∈Rd×d.f_M(x) = x^{\mathsf{T}} Mx,\qquad M = M^{\mathsf{T}} \in\mathbb{R}^{d \times d}.

A tied atom contributes the signed square α(uTx)2\alpha(u^{\mathsf{T}}x)^2, corresponding to the rank-1 symmetric matrix αuuT\alpha uu^{\mathsf{T}}. An untied atom contributes a product of two linear forms, α(uTx)(vTx)\alpha(u^{\mathsf{T}}x)(v^{\mathsf{T}}x), which after symmetrization becomes

α2(uvT+vuT).\frac{\alpha}{2}\left(uv^{\mathsf{T}} + vu^{\mathsf{T}}\right).

A tied atom therefore contributes one quadratic direction, while an untied atom can couple two. Our first result records this coarse rank comparison.

Theorem 4.5 (Coarse bilinear rank bounds). Let M∈Rd×dM \in\mathbb{R}^{d \times d} be symmetric. If MM is represented by a tied bilinear block of width mm, then rank⁡(M)≤min⁡(m,d)\operatorname{rank}(M) \le\min(m,d). If MM is represented by an untied bilinear block of width mm, then rank⁡(M)≤min⁡(2m,d)\operatorname{rank}(M) \le\min(2m,d). Both bounds are tight.

Proof sketch. Each tied unit contributes the rank-1 symmetric matrix uauaTu_a u_a^{\mathsf{T}}, so a width-mm tied block produces matrices of rank at most min⁡(m,d)\min(m,d). For untied blocks, only the symmetric part of uavaTu_a v_a^{\mathsf{T}} contributes; the resulting matrix 12(uavaT+vauaT)\frac{1}{2}(u_a v_a^{\mathsf{T}} + v_a u_a^{\mathsf{T}}) has image in span⁡{ua,va}\operatorname{span}\{u_a,v_a\} and therefore rank at most two, and summing mm such terms gives the bound. Both bounds are tight by choosing independent coordinate directions.

Theorem 4.5 shows that an untied atom can carry twice the rank of a tied atom, but rank alone does not determine which matrices a given width can represent. The rank-2 gain has a specific sign structure. From the difference-of-squares identity

(u⊤x)(v⊤x)=14((u+v)⊤x)2−14((u−v)⊤x)2,(u^\top x)(v^\top x)=\frac{1}{4}\left((u+v)^\top x\right)^2-\frac{1}{4}\left((u-v)^\top x\right)^2,

a non-collinear untied atom is a difference of two squares. It can pair one positive with one negative quadratic direction—representing interactions like a2−b2=(a+b)(a−b)a^2-b^2=(a+b)(a-b)—but cannot compress two same-sign directions such as a2+b2a^2+b^2 into a single atom. The exact width is therefore controlled not by rank, but by how many positive and negative quadratic directions the target carries.

Let n+(M)n_{+}(M) and n−(M)n_{-}(M) denote the numbers of positive and negative eigenvalues of MM.

Theorem 4.6 (Exact bilinear atom ranks). For every symmetric matrix M∈Rd×dM \in\mathbb{R}^{d\times d},

rtied(M)=rank⁡(M)=n+(M)+n−(M),r_{\mathrm{tied}}(M)=\operatorname{rank}(M)=n_{+}(M)+n_{-}(M),

whereas

runtied(M)=max⁡{n+(M),n−(M)}.r_{\mathrm{untied}}(M)=\max\{n_{+}(M),n_{-}(M)\}.

Proof sketch. For tied atoms, each channel contributes one signed rank-1 square, so it needs one atom per nonzero eigendirection, and the spectral decomposition achieves this number. For untied atoms, the difference-of-squares identity expresses any width-mm representation as M=AA⊤−BB⊤M=AA^\top-BB^\top with A,B∈Rd×mA,B\in\mathbb{R}^{d\times m}. On ker⁡(A⊤)\ker(A^\top), x⊤Mx=−∥B⊤x∥2≤0x^\top Mx=-\lVert B^\top x\rVert^2\leq0, so MM cannot be positive definite on a subspace of dimension greater than mm; the analogous argument on ker⁡(B⊤)\ker(B^\top) gives the same bound for negative directions, so m≥max⁡{n+(M),n−(M)}m\geq\max\{n_{+}(M),n_{-}(M)\}. The matching upper bound is achieved by pairing opposite-sign eigendirections one atom at a time, with unpaired directions handled by the tied special cases u=±vu=\pm v. Full proofs of Theorems 4.5 and 4.6 are in Appendix § 10.§ 2.

Corollary 4.7 (When untied bilinear atoms help). If M⪰0M\succeq0 or M⪯0M\preceq0,

rtied(M)=runtied(M)=rank⁡(M).r_{\mathrm{tied}}(M)=r_{\mathrm{untied}}(M)=\operatorname{rank}(M).

If MM is indefinite,

rtied(M)−runtied(M)=min⁡{n+(M),n−(M)}.r_{\mathrm{tied}}(M)-r_{\mathrm{untied}}(M)=\min\{n_{+}(M),n_{-}(M)\}.

Untied atoms are therefore most width-efficient on indefinite quadratic interactions; they offer no advantage on purely same-sign quadratic structure.

Connection to hard-gated blocks. The same degree-two atom gap appears regionwise in hard-gated CP-MLPs. On a fixed activation pattern μ∈{0,1}m\mu\in\{0,1\}^{m}, ReGLU realizes

Qμ,o=∑a=1mμaWoaua⊤va+va⊤ua2Q_{\mu,o}=\sum_{a=1}^{m}\mu_a W_{oa}\frac{u_a^\top v_a+v_a^\top u_a}{2}

for each output coordinate oo, whereas ReLU2^2 realizes

Qμ,o=∑aμaWoauaua⊤.Q_{\mu,o}=\sum_{a}\mu_a W_{oa}u_a u_a^\top.

Inside each fixed region, ReGLU has access to untied rank-2 atoms with opposite-sign structure, while ReLU2^2 is restricted to tied rank-1 atoms. Region switching adds further piecewise expressivity; Theorem 4.6 isolates the within-region atom difference.

Higher-degree monomial separation

The degree-two analysis showed that untied atoms can be more width-efficient when the target contains opposite-sign quadratic structure. At higher degree, a different kind of separation emerges that is driven not by sign, but by role asymmetry: one direction is repeated many times in the detector slot, while another direction appears only once in the value slot.

For k≥2k\geq2, consider the binary monomial Fk(x1,x2)=x1k−1x2F_k(x_1,x_2)=x_1^{k-1}x_2. This target has the exact shape of a single untied atom: taking u=e1u=e_1 and v=e2v=e_2 gives (u⊤x)k−1(v⊤x)=x1k−1x2(u^\top x)^{k-1}(v^\top x)=x_1^{k-1}x_2, so one untied atom suffices. A tied degree-kk atom, by contrast, has the pure-power form (ℓ⊤x)k(\ell^\top x)^k, so the tied atom rank of FkF_k is exactly its Waring rank.

Definition 4.8 (Waring rank). Let F∈{R,C}\mathbb{F}\in\{\mathbb{R},\mathbb{C}\}, and let f:Fd→Ff:\mathbb{F}^{d}\to\mathbb{F} be a homogeneous polynomial of degree kk. The Waring rank of ff over F\mathbb{F}, denoted Wr⁡F(f)\operatorname{Wr}_{\mathbb{F}}(f), is the smallest integer mm such that

f(x)=∑a=1mcaℓa(x)k,f(x)=\sum_{a=1}^{m}c_a\ell_a(x)^k,

where ca∈Fc_a\in\mathbb{F} and each ℓa\ell_a is a linear form over F\mathbb{F}.

For scalar tied CP-MLPs, the real Waring rank coincides with the tied atom rank: any tied representation f(x)=∑aca(uaTx)kf(x)=\sum_a c_a(u_a^\mathsf{T}x)^k with ua∈Rdu_a\in\mathbb{R}^d is exactly a real Waring decomposition, so rtied(f)=WrR(f)r_{\mathrm{tied}}(f)=\mathrm{Wr}_{\mathbb{R}}(f) for every scalar homogeneous degree-kk polynomial. This is the standard correspondence between symmetric tensor rank and polynomial Waring rank [9]. The following classical formula computes the right-hand side for monomials.

Proposition 4.9 (Waring rank of monomials; Carlini et al., [3, 4, 5]). Let μ(x)=x1α1⋯xnαn\mu(x)=x_1^{\alpha_1}\cdots x_n^{\alpha_n} be a monomial of degree k=∑iαik=\sum_i\alpha_i, with sorted exponents 1≤α1≤⋯≤αn1\leq\alpha_1\leq\cdots\leq\alpha_n. Then

WrC(μ)=1α1+1∏i=1n(αi+1)=∏i=2n(αi+1),\mathrm{Wr}_{\mathbb{C}}(\mu)=\frac{1}{\alpha_1+1}\prod_{i=1}^{n}(\alpha_i+1)=\prod_{i=2}^{n}(\alpha_i+1),

and WrR(μ)=WrC(μ)\mathrm{Wr}_{\mathbb{R}}(\mu)=\mathrm{Wr}_{\mathbb{C}}(\mu) whenever α1=1\alpha_1=1.

The complex formula is Carlini et al., [4]; the real-complex equality for α1=1\alpha_1=1 follows from Carlini et al. [ [5]. For Fk=x1k−1x2F_k=x_1^{k-1}x_2 the sorted exponent vector is (1,k−1)(1,k-1), so

WrR(Fk)=WrC(Fk)=(k−1)+1=k.\mathrm{Wr}_{\mathbb{R}}(F_k)=\mathrm{Wr}_{\mathbb{C}}(F_k)=(k-1)+1=k.

Theorem 4.10 (Degree-kk monomial separation). For every k≥2k\geq2,

runtied(Fk)=1,rtied(Fk)=k.r_{\mathrm{untied}}(F_k)=1,\qquad r_{\mathrm{tied}}(F_k)=k.

A single untied atom therefore represents a degree-kk coefficient tensor that requires kk tied atoms.

Proof. We bound the two ranks separately.

Untied rank. Setting u=e1u=e_1 and v=e2v=e_2, (uTx)k−1(vTx)=x1k−1x2=Fk(x)(u^\mathsf{T}x)^{k-1}(v^\mathsf{T}x)=x_1^{k-1}x_2=F_k(x), so a single untied atom realizes FkF_k and runtied(Fk)≤1r_{\mathrm{untied}}(F_k)\leq1. Since Fk≠0F_k\ne0, it cannot be represented with zero atoms, giving runtied(Fk)=1r_{\mathrm{untied}}(F_k)=1.

Tied rank. Any scalar tied degree-kk representation Fk(x)=∑acaℓa(x)kF_k(x)=\sum_a c_a\ell_a(x)^k is a real Waring decomposition, so rtied(Fk)=WrR(Fk)=kr_{\mathrm{tied}}(F_k)=\mathrm{Wr}_{\mathbb{R}}(F_k)=k by Proposition 4.9.

Exact representation versus approximation. The real border rank r‾f(T)\underline{r}_f(T) is the smallest width permitting arbitrarily accurate approximation of TT by real atoms from family ff. In coefficient norm, a positive approximation gap occurs exactly when m<r‾f(T)m<\underline{r}_f(T). For scalar quadratics, both width classes are closed, so exact and border ranks coincide (Appendix § 10.§ 4). For Fk=x1k−1x2F_k=x_1^{k-1}x_2, however, r‾tied(Fk)=2\underline{r}_{\mathrm{tied}}(F_k)=2, whereas rtied(Fk)=kr_{\mathrm{tied}}(F_k)=k. Thus, at 2≤m<k2\leq m<k, the tied error infimum is zero but unattained. Under any finite bound on ∑a∥wa∥∥ua∥k\sum_a\lVert w_a\rVert\lVert u_a\rVert^k, a positive tied approximation gap remains below the exact rank. Appendix § 10.§ 3 gives the approximation arguments; Appendix § 10.§ 5 extends the separation to sums of independent interaction blocks.

The degree-wise expansion gives necessary width bounds for exact representation of the full analytic function. If a bias-free width-mm block with gate gg represents an analytic target FF near the origin, then

m≥sup⁡k∈S(g)rf(F(k)),f∈{tied,untied},m\geq\sup_{k\in\mathcal{S}(g)}r_f(F^{(k)}),\qquad f\in\{\mathrm{tied},\mathrm{untied}\},

where F(k)F^{(k)} is the degree-kk homogeneous component. Exact representation also requires F(0)=0F(0)=0 and vanishing components at inactive degrees. These conditions are not generally sufficient: all degrees share the same channels, and the gate fixes their coefficient ratios. Appendix § 10.§ 6 gives the proof and an explicit example where the full required width exceeds every individual degree-wise rank.

Experiments

We test whether the atom-rank separations from § 4 appear as observable width and loss effects. Indefinite quadratics realize the degree-two gap (§ 5.1); role-asymmetric monomials realize the higher-degree gap (§ 5.2); pure-power targets show that tying can be the better match (§ 5.3); and gated MLPs show how the separation carries over to actual gates, including under a matched input-projection budget (§ 5.4). Optimization, hyperparameters, and budget checks are in Appendix § 11.

Indefinite quadratic targets

Theorem 4.6 predicts that untied bilinear atoms are more width-efficient than tied atoms only when the target carries both positive and negative directions. To test this, we construct

Q=diag⁡(1n+,−1n−,0d−n+−n−),y(x)=xTQx,Q = \operatorname{diag}(1^{n_+}, -1^{n_-}, 0^{d-n_+-n_-}), \qquad y(x) = x^\mathsf{T} Q x,

with d=8d = 8, and sweep six choices of (n+,n−)(n_+,n_-) covering definite, balanced indefinite, and imbalanced indefinite targets.

Figure 2 confirms the exact threshold: tied blocks require width n++n−n_+ + n_-, untied blocks require width max⁡{n+,n−}\max\{n_+,n_-\}. At one width below the prediction, the loss stays near the target variance; at the predicted width, the loss drops to machine precision. Definite targets show no tied/untied gap, while indefinite targets show the predicted gap min⁡{n+,n−}\min\{n_+,n_-\}.

Quadratic targets with positive and negative directions

Figure 2. Quadratic targets with (n+,n−)(n_+, n_-) positive and negative directions. Left: required widths match the predictions n++n−n_+ + n_- (tied) and max⁡{n+,n−}\max\{n_+, n_-\} (untied). Right: loss stays near the target variance below the predicted width and reaches machine precision at the predicted width.

Role-asymmetric monomial targets

We next test the higher-degree separation of Theorem 4.10. For

Fk(x1,x2)=x1k−1x2,k∈{2,3,4,5},F_k(x_1,x_2) = x_1^{k-1}x_2, \qquad k \in\{2,3,4,5\},

the theory predicts untied atom rank 11 and tied atom rank kk. We fit the target directly in monomial coefficient space and sweep width m=1,…,k+2m = 1,\ldots,k+2; coefficient-space fitting removes sampling noise and isolates the rank effect.

Figure 3 shows the predicted separation at width one: untied atoms reach machine precision at m=1m = 1 for every degree, while tied atoms keep a clearly positive loss. For 2≤m<k2 \le m < k, the tied losses are small but nonzero (about 10−710^{-7} to 10−910^{-9}, with one optimizer outlier at k=4k = 4, m=2m = 2). They drop to machine precision at m=km = k for k≤3k \le3 and at m=k+1m = k + 1 for k=4k = 4; for k=5k = 5 they stay between about 10−710^{-7} and 5×10−105 \times10^{-10} across the tested widths. Below its atom rank the target lies outside the tied model class, but for 2≤m<k2 \le m < k it lies in the class’s closure: the border Waring rank of x1k−1x2x_1^{k-1}x_2 is two [8], via the limit in Equation (1) with ℓ=x1\ell= x_1 and h=x2h = x_2. The plateau therefore reflects the ill-conditioning of border-rank approximations, whose residuals diverge as the residual shrinks, rather than a positive distance to the model class; at m=1m = 1 the floor is genuine, since the set of symmetric rank-at-most-one tensors is closed. The untied advantage is specific to single-value-role targets; products with multiple independent value roles require more atoms (Appendix § 11.7).

Figure 3. Coefficient-space width sweep for Fk=x1k−1x2F_k=x_1^{k-1}x_2, k∈{2,3,4,5}k\in\{2,3,4,5\}. Untied atoms fit at width m=1m=1. Tied atoms keep a positive loss at m=1m=1; for 2≤m<k2\le m<k their losses are small but nonzero, as expected from border rank two, and they reach machine precision at m=km=k for k≤3k\le3. Median over two seeds; bands show min–max.

Pure powers versus role-asymmetric targets

The previous experiment shows a strong untied advantage on role-asymmetric monomials, but tying is not always worse. At width m=1m = 1, we compare tied and untied models on two targets:

x1kandx1k−1x2.x_1^k \qquad\text{and} \qquad x_1^{k-1}x_2.

The first is a pure power and is realized exactly by one tied atom; the second is role-asymmetric and is realized exactly by one untied atom. Figure 4 shows the expected reversal: on pure powers, tied models reach machine precision while untied models must align the redundant value direction with the detector; on role-asymmetric targets, untied models fit at width one while tied models cannot. The low-width winner therefore depends on target structure.

Figure 4. Target structure changes the low-width winner. At m=1m=1, tied atoms fit pure powers x1kx_1^k, while untied atoms fit role-asymmetric monomials x1k−1x2x_1^{k-1}x_2. Bands show min and max over seeds.

Separation in gated MLPs

The coefficient-space experiments above isolate the atom-rank effect; we now test actual gated MLPs on the same witness Fk=x1k−1x2F_k=x_1^{k-1}x_2. We compare tied ReLU2^2, tied SwiGLU (SiLU(zz)zz), untied ReGLU, and untied SwiGLU, using Gaussian inputs with d=4d=4, k∈{2,3}k\in\{2,3\}, and widths m∈{1,2,4}m\in\{1,2,4\}.

Figure 5 shows that the exact polynomial threshold becomes a stable qualitative ordering for gated networks. At k=2k=2, untied ReGLU and SwiGLU reach near-machine precision at m=2m=2, and tied ReLU2^2 at m=4m=4. These widths are twice the atom ranks 1 and 2. Both gates satisfy g(z)−g(−z)=zg(z)-g(-z)=z and g(z)z+g(−z)(−z)=z2g(z)z+g(-z)(-z)=z^2, so each polynomial atom can be built from two gated channels. By the second identity, tied SwiGLU can also represent x1x2x_1x_2 exactly at m=4m=4. Its fits here stay near 10−310^{-3}, whereas the same model reaches 5.5×10−135.5\times10^{-13} at width four in the matched control of Appendix § 11.8, so this gap is an optimization failure. At k=3k=3, no architecture reaches machine precision. ReLU-gated channels are positively homogeneous of degree two and cannot represent a cubic; a SiLU channel that matches the cubic term also contributes quadratic and higher-degree terms, which finite width cannot cancel exactly. Untied SwiGLU is the best-performing model, and for each gate the untied block lies below its tied counterpart at widths m≥2m\ge2.

A control comparing untied width mm against tied width 2m2m, which matches input-projection parameters, gives a degree-dependent picture: at k=2k=2, tied SwiGLU at width four and untied SwiGLU at width two reach comparable losses, while at k=3k=3 untied SwiGLU at width four is about ten times more accurate than tied SwiGLU at width eight. The figure and numbers are in Appendix § 11.8.

Figure 5. Width versus loss for actual gates on x1k−1x2x_1^{k-1}x_2. Left: for k=2k=2, untied ReGLU and SwiGLU reach near-machine precision at smaller width than tied ReLU² and tied SwiGLU. Right: for k=3k=3, the exact rank threshold becomes a qualitative ordering; untied SwiGLU is best, and each untied block stays below its same-gate tied counterpart for m≥2m \ge2. Median over three seeds; bands show min–max.

Conclusion

We introduced CP-MLPs, a tensor framework for comparing tied and untied multiplicative feed-forward blocks. The framework separates the gate from the choice of whether detector and value directions are shared. Our theory shows that this choice changes width efficiency. Untied atoms can represent indefinite quadratic interactions and role-asymmetric monomials with fewer channels, while tied atoms are naturally matched to pure-power targets. For smooth gates such as SwiGLU, these statements apply degree by degree to Taylor coefficient tensors. Experiments confirm the predicted quadratic width thresholds and the width-one untied fits, show tied losses below the exact monomial rank that are consistent with border rank, and demonstrate that the low-width advantage changes with target structure.

Selected gated-MLP comparisons retain an untied advantage under input-projection matching, while others attain comparable errors. These results suggest that the success of GLU-style feed-forward blocks is not only an activation effect: separating detector and value directions changes which interactions are width-efficient. Several open directions remain. Our higher-degree separation is degree-wise rather than a global statement for the full analytic channel, and our experiments use synthetic polynomial targets rather than language-modeling tasks; bridging both gaps, by extending the theory to richer multi-leg atoms and by testing these effects in full Transformer language models, is a natural next step.

Acknowledgments

We acknowledge the support from the Canada CIFAR AI Chair Program and from the Canada Excellence Research Chairs Program. The research was enabled in part by computational resources provided by the Digital Research Alliance of Canada and Mila-Quebec AI Institute.

Preliminaries

This appendix collects the tensor and CP-decomposition notation used throughout the paper.

Tensors and CP decompositions

Let T∈Rd1×⋯×dnT \in\mathbb{R}^{d_1\times\cdots\times d_n} be an order-nn tensor and let xi∈Rdix_i \in\mathbb{R}^{d_i}. We write the multilinear contraction of TT with vectors x1,…,xnx_1,\ldots,x_n as

T[x1,…,xn]:=T×1x1×2⋯×nxn,T[x_1,\ldots,x_n] := T \times_1 x_1 \times_2 \cdots\times_n x_n,

where ×i\times_i denotes contraction along mode ii:

(T×ixi)j1,…,ji−1,ji+1,…,jn=∑k=1diTj1,…,ji−1,k,ji+1,…,jn(xi)k.(T \times_i x_i)_{j_1,\ldots,j_{i-1},j_{i+1},\ldots,j_n} = \sum_{k=1}^{d_i} T_{j_1,\ldots,j_{i-1},k,j_{i+1},\ldots,j_n}(x_i)_k.

Equivalently,

T[x1,…,xn]=∑j1=1d1⋯∑jn=1dnTj1,…,jn(x1)j1⋯(xn)jn.T[x_1,\ldots,x_n] = \sum_{j_1=1}^{d_1}\cdots\sum_{j_n=1}^{d_n} T_{j_1,\ldots,j_n}(x_1)_{j_1}\cdots(x_n)_{j_n}.

When all modes are contracted with vectors, the result is a scalar. If some modes are left uncontracted, the same formula applies to the remaining modes only; in particular, contracting the kk input modes of T∈Rp⊗(Rd)⊗kT \in\mathbb{R}^{p} \otimes(\mathbb{R}^{d})^{\otimes k} while leaving the output mode free gives T[x1,…,xk]∈RpT[x_1,\ldots,x_k] \in\mathbb{R}^{p}, the convention used in § 3.1. We write eie_i for the ii-th standard basis vector.

A canonical polyadic (CP) decomposition represents a tensor as a sum of rank-one outer products:

T=∑a=1rλaua(1)⊗ua(2)⊗⋯⊗ua(n),T = \sum_{a=1}^{r} \lambda_a u_a^{(1)} \otimes u_a^{(2)} \otimes\cdots\otimes u_a^{(n)},

where ua(i)∈Rdiu_a^{(i)} \in\mathbb{R}^{d_i} and λa∈R\lambda_a \in\mathbb{R}. Under contraction, each rank-one term factorizes:

T[x1,…,xn]=∑a=1rλa∏i=1n⟨ua(i),xi⟩.T[x_1,\ldots,x_n] = \sum_{a=1}^{r} \lambda_a \prod_{i=1}^{n} \langle u_a^{(i)}, x_i\rangle.

Thus a CP decomposition expresses a multilinear map as a sum of separable products of linear forms.

This is the tensor structure used throughout the paper. For example, a single CP-MLP channel at degree kk contributes a rank-one raw tensor of the form

wa⊗ua⊗(k−1)⊗va,w_a \otimes u_a^{\otimes(k-1)} \otimes v_a,

whose diagonal evaluation gives

(wa⊗ua⊗(k−1)⊗va)[x,…,x]=wa(uaTx)k−1(vaTx).(w_a \otimes u_a^{\otimes(k-1)} \otimes v_a)[x,\ldots,x] = w_a(u_a^{\mathsf{T}}x)^{k-1}(v_a^{\mathsf{T}}x).

A width-mm CP-MLP sums mm such channel-wise rank-one contributions.

Symmetrization. For any tensor T∈(Rd)⊗kT \in(\mathbb{R}^{d})^{\otimes k}, define

Sym⁡(T)=1k!∑π∈Skπ⋅T.\operatorname{Sym}(T) = \frac{1}{k!}\sum_{\pi\in S_k} \pi\cdot T.

Symmetric CP rank. A symmetric tensor TT has symmetric rank rr if rr is the smallest integer such that

T=∑a=1rλaua⊗k.T = \sum_{a=1}^{r} \lambda_a u_a^{\otimes k}.

Matricization lower bound. For any tensor TT, the rank of any flattening provides a lower bound on its CP rank [14].

Additional framework details

Symmetrization details

Let SkS_k denote the symmetric group on kk elements. For a permutation π∈Sk\pi\in S_k and a tensor T∈Rp⊗(Rd)⊗kT \in\mathbb{R}^p \otimes(\mathbb{R}^d)^{\otimes k}, write π⋅T\pi\cdot T for the tensor obtained by permuting the kk input modes according to π\pi, leaving the output mode fixed.

Definition B.1 (Input-symmetric tensor). A tensor

T∈Rp⊗(Rd)⊗kT \in\mathbb{R}^p \otimes(\mathbb{R}^d)^{\otimes k}

is input-symmetric if π⋅T=T\pi\cdot T = T for every π∈Sk\pi\in S_k. The space of input-symmetric tensors is denoted

Rp⊗Sym⁡k(Rd).\mathbb{R}^p \otimes\operatorname{Sym}^k(\mathbb{R}^d).

Definition B.2 (Input symmetrization). The input symmetrization of T∈Rp⊗(Rd)⊗kT \in\mathbb{R}^p \otimes(\mathbb{R}^d)^{\otimes k} is

Sym⁡(T):=1k!∑π∈Skπ⋅T.\operatorname{Sym}(T) := \frac{1}{k!}\sum_{\pi\in S_k} \pi\cdot T.

Lemma B.3 (Diagonal evaluation sees only the input-symmetric part). For every

T∈Rp⊗(Rd)⊗kT \in\mathbb{R}^p \otimes(\mathbb{R}^d)^{\otimes k}

and every x∈Rdx \in\mathbb{R}^d,

T[x,…,x]=Sym⁡(T)[x,…,x].T[x,\ldots,x] = \operatorname{Sym}(T)[x,\ldots,x].

Proof. For any π∈Sk\pi\in S_k, all input slots of π⋅T\pi\cdot T receive the same vector $x. Hence

(π⋅T)[x,…,x]=T[x,…,x].(\pi\cdot T)[x,\ldots,x] = T[x,\ldots,x].

Averaging over all π∈Sk\pi\in S_k gives the claim. □

Lemma B.4 (Quadratic diagonal evaluation). For every A∈Rd×dA \in\mathbb{R}^{d \times d} and every x∈Rdx \in\mathbb{R}^d,

x⊤Ax=x⊤A+A⊤2x.x^\top A x = x^\top\frac{A + A^\top}{2} x.

Proof. Let

S=12(A−A⊤).S = \frac{1}{2}(A - A^\top).

Then S⊤=−SS^\top= -S. Since x⊤Sxx^\top Sx is a scalar,

x⊤Sx=(x⊤Sx)⊤=x⊤S⊤x=−x⊤Sx.x^\top Sx = (x^\top Sx)^\top= x^\top S^\top x = -x^\top Sx.

Therefore x⊤Sx=0x^\top Sx = 0. The result follows by writing

A=12(A+A⊤)+S.A = \frac{1}{2}(A + A^\top) + S.

□

Derivative tensors. For an analytic scalar function, the degree-kk Taylor coefficient tensor is equivalently the kk-th derivative tensor up to the factor k!k!. In particular, for a homogeneous tensor polynomial

fT(x)=T[x,…,x],f_T(x) = T[x,\ldots,x],

we have

DkfT(0)=k!Sym⁡(T).D^k f_T(0) = k! \operatorname{Sym}(T).

Thus the derivative tensor also records only the input-symmetric part of the raw coefficient tensor. This is the differential version of Lemma B.3.

Expressivity proofs

Scalar-output reduction

Lemma C.1 (Output projection cannot increase channel rank). For every T∈TkT \in\mathcal{T}_k and every η∈(Rp)∗\eta\in(\mathbb{R}^p)^*,

rtied((η⊗I)T)≤rtied(T),runtied((η⊗I)T)≤runtied(T).r_{\mathrm{tied}}\left((\eta\otimes I)T\right) \le r_{\mathrm{tied}}(T), \qquad r_{\mathrm{untied}}\left((\eta\otimes I)T\right) \le r_{\mathrm{untied}}(T).

Proof. Suppose first that TT has a tied representation with mm channels:

T=∑a=1mwa⊗ua⊗k.T = \sum_{a=1}^{m} w_a \otimes u_a^{\otimes k}.

Applying η\eta to the output mode gives

(η⊗I)T=∑a=1mη(wa)ua⊗k.(\eta\otimes I)T = \sum_{a=1}^{m} \eta(w_a)u_a^{\otimes k}.

This is a scalar tied representation with at most mm atoms; atoms with η(wa)=0\eta(w_a)=0 simply drop out. Minimizing over tied representations of TT gives the first inequality.

The detector–value case is identical. If

T=∑a=1mwa⊗Sym⁡(ua⊗(k−1)⊗va),T = \sum_{a=1}^{m} w_a \otimes\operatorname{Sym}\left(u_a^{\otimes(k-1)} \otimes v_a\right),

then

(η⊗I)T=∑a=1mη(wa)Sym⁡(ua⊗(k−1)⊗va),(\eta\otimes I)T = \sum_{a=1}^{m} \eta(w_a)\operatorname{Sym}\left(u_a^{\otimes(k-1)} \otimes v_a\right),

which is a scalar detector–value representation with at most mm atoms.

For the special embedding of a scalar target into one output coordinate, the ranks are exactly preserved.

Corollary C.2 (Scalar separations embed into vector targets). Let S∈Sym⁡k(Rd)S \in\operatorname{Sym}^k(\mathbb{R}^d), and let

T=eo⊗S∈Rp⊗Sym⁡k(Rd).T = e_o \otimes S \in\mathbb{R}^p \otimes\operatorname{Sym}^k(\mathbb{R}^d).

Then

rtied(T)=rtied(S),runtied(T)=runtied(S).r_{\mathrm{tied}}(T) = r_{\mathrm{tied}}(S), \qquad r_{\mathrm{untied}}(T) = r_{\mathrm{untied}}(S).

Consequently, any scalar tied–detector–value separation transfers to the vector-valued CP-MLP setting.

Proof. The lower bounds follow from Lemma C.1 by taking η(y)=yo\eta(y)=y_o, since

(η⊗I)T=S.(\eta\otimes I)T = S.

Thus

rtied(S)≤rtied(T),runtied(S)≤runtied(T).r_{\mathrm{tied}}(S) \le r_{\mathrm{tied}}(T), \qquad r_{\mathrm{untied}}(S) \le r_{\mathrm{untied}}(T).

For the reverse inequalities, take any scalar representation of SS and multiply each atom by the output basis vector eoe_o. This gives a vector-valued representation of T=eo⊗ST=e_o \otimes S with the same number of channels. Hence the ranks are equal.

Bilinear atom-rank proofs

Proof of Theorem 4.5. We prove the tied and untied rank bounds separately.

Tied blocks. A tied bilinear block of width mm represents a matrix of the form

M=∑a=1mαauauaT.M = \sum_{a=1}^{m} \alpha_a u_a u_a^{\mathsf{T}}.

Each term αauauaT\alpha_a u_a u_a^{\mathsf{T}} has rank at most one. Therefore

rank⁡(M)≤∑a=1mrank⁡(αauauaT)≤m.\operatorname{rank}(M) \le\sum_{a=1}^{m} \operatorname{rank}(\alpha_a u_a u_a^{\mathsf{T}}) \le m.

Since M∈Rd×dM \in\mathbb{R}^{d \times d}, also

rank⁡(M)≤d.\operatorname{rank}(M) \le d.

Thus a tied block satisfies

rank⁡(M)≤min⁡(m,d).\operatorname{rank}(M) \le\min(m,d).

This bound is tight. Let r=min⁡(m,d)r = \min(m,d). Choosing

ua=ea,αa=1,a=1,…,r,u_a = e_a,\qquad\alpha_a = 1,\qquad a = 1,\ldots,r,

gives

M=∑a=1reaeaT,M = \sum_{a=1}^{r} e_a e_a^{\mathsf{T}},

which has rank r=min⁡(m,d)r = \min(m,d).

Untied blocks. An untied bilinear block of width mm represents a matrix of the form

M=∑a=1mMa,Ma=αa2(uavaT+vauaT).M = \sum_{a=1}^{m} M_a,\qquad M_a = \frac{\alpha_a}{2}\left(u_a v_a^{\mathsf{T}} + v_a u_a^{\mathsf{T}}\right).

For any z∈Rdz \in\mathbb{R}^{d},

Maz=αa2(ua(vaTz)+va(uaTz))∈span⁡{ua,va}.M_a z = \frac{\alpha_a}{2}\left(u_a(v_a^{\mathsf{T}}z) + v_a(u_a^{\mathsf{T}}z)\right) \in\operatorname{span}\{u_a,v_a\}.

Therefore

rank⁡(Ma)≤2.\operatorname{rank}(M_a) \le2.

Using subadditivity of rank,

rank⁡(M)≤∑a=1mrank⁡(Ma)≤2m.\operatorname{rank}(M) \le\sum_{a=1}^{m} \operatorname{rank}(M_a) \le2m.

Since M∈Rd×dM \in\mathbb{R}^{d \times d},

rank⁡(M)≤d.\operatorname{rank}(M) \le d.

Hence

rank⁡(M)≤min⁡(2m,d).\operatorname{rank}(M) \le\min(2m,d).

This bound is tight. Let r=min⁡(2m,d)r = \min(2m,d).

If r=2sr = 2s is even, choose

ua=e2a−1,va=e2a,αa=1,a=1,…,s.u_a = e_{2a-1},\qquad v_a = e_{2a},\qquad\alpha_a = 1,\qquad a = 1,\ldots,s.

Then

M=12∑a=1s(e2a−1e2aT+e2ae2a−1T).M = \frac{1}{2}\sum_{a=1}^{s}\left(e_{2a-1}e_{2a}^{\mathsf{T}} + e_{2a}e_{2a-1}^{\mathsf{T}}\right).

This is a direct sum of ss independent 2×22 \times2 off-diagonal blocks, each of rank two. Therefore rank⁡(M)=2s=r\operatorname{rank}(M) = 2s = r.

If r=2s+1r = 2s + 1 is odd, use the same ss off-diagonal blocks and add one diagonal atom by taking

us+1=vs+1=e2s+1,αs+1=1.u_{s+1} = v_{s+1} = e_{2s+1},\qquad\alpha_{s+1} = 1.

This adds one independent rank-one block, so the resulting matrix has rank 2s+1=r2s + 1 = r. Thus the untied rank bound is tight. □

Proof of Theorem 4.6. We prove the tied and untied atom ranks separately.

Tied atom rank. By Theorem 4.5, any tied representation of MM with mm atoms satisfies

rank⁡(M)≤m.\operatorname{rank}(M) \le m.

Therefore

rtied(M)≥rank⁡(M).r_{\mathrm{tied}}(M) \ge\operatorname{rank}(M).

For the reverse inequality, use the spectral decomposition of the symmetric matrix MM:

M=∑i=1n+(M)λiqiqiT−∑j=1n−(M)μjpjpjT,λi,μj>0.M = \sum_{i=1}^{n_{+}(M)} \lambda_i q_i q_i^{\mathsf{T}} - \sum_{j=1}^{n_{-}(M)} \mu_j p_j p_j^{\mathsf{T}}, \qquad\lambda_i,\mu_j > 0.

This is a tied representation using one signed rank-one atom for each nonzero eigendirection. Hence

rtied(M)≤n+(M)+n−(M)=rank⁡(M).r_{\mathrm{tied}}(M) \le n_{+}(M) + n_{-}(M) = \operatorname{rank}(M).

Combining the two inequalities gives

rtied(M)=rank⁡(M)=n+(M)+n−(M).r_{\mathrm{tied}}(M) = \operatorname{rank}(M) = n_{+}(M) + n_{-}(M).

Untied atom rank: lower bound. An untied atom satisfies the identity

α2(uvT+vuT)=α4(u+v)(u+v)T−α4(u−v)(u−v)T.\frac{\alpha}{2}(uv^{\mathsf{T}} + vu^{\mathsf{T}}) = \frac{\alpha}{4}(u+v)(u+v)^{\mathsf{T}} - \frac{\alpha}{4}(u-v)(u-v)^{\mathsf{T}}.

Therefore any width-mm untied representation can be written as

M=AAT−BBTM = AA^{\mathsf{T}} - BB^{\mathsf{T}}

for some A,B∈Rd×mA,B \in\mathbb{R}^{d \times m}, after absorbing scalar signs and magnitudes into the columns of AA and BB.

We claim that this implies

n+(M)≤m,n−(M)≤m.n_{+}(M) \le m, \qquad n_{-}(M) \le m.

Indeed, on ker⁡(AT)\ker(A^{\mathsf{T}}),

xTMx=xT(AAT−BBT)x=∥ATx∥2−∥BTx∥2=−∥BTx∥2≤0.x^{\mathsf{T}}Mx = x^{\mathsf{T}}(AA^{\mathsf{T}} - BB^{\mathsf{T}})x = \lVert A^{\mathsf{T}}x\rVert^2 - \lVert B^{\mathsf{T}}x\rVert^2 = -\lVert B^{\mathsf{T}}x\rVert^2 \le0.

If n+(M)>rank⁡(A)n_{+}(M) > \operatorname{rank}(A), a subspace PP of dimension n+(M)n_{+}(M) on which MM is positive definite would satisfy dim⁡(P∩ker⁡AT)≥n+(M)−rank⁡(A)>0\dim(P \cap\ker A^{\mathsf{T}}) \ge n_{+}(M) - \operatorname{rank}(A) > 0, contradicting the inequality above; hence n+(M)≤rank⁡(A)≤mn_{+}(M) \le\operatorname{rank}(A) \le m. Similarly, on ker⁡(BT)\ker(B^{\mathsf{T}}),

xTMx=∥ATx∥2≥0,x^{\mathsf{T}}Mx = \lVert A^{\mathsf{T}}x\rVert^2 \ge0,

so MM cannot be negative definite on a subspace of dimension larger than rank⁡(B)≤m\operatorname{rank}(B) \le m. Hence

n−(M)≤m.n_{-}(M) \le m.

Therefore every untied representation satisfies

m≥max⁡{n+(M),n−(M)}.m \ge\max\{n_{+}(M), n_{-}(M)\}.

So

runtied(M)≥max⁡{n+(M),n−(M)}.r_{\mathrm{untied}}(M) \ge\max\{n_{+}(M), n_{-}(M)\}.

Untied atom rank: upper bound. It remains to construct an untied representation with exactly max⁡{n+(M),n−(M)}\max\{n_{+}(M), n_{-}(M)\} atoms. Use the spectral decomposition

M=∑i=1n+(M)λiqiqiT−∑j=1n−(M)μjpjpjT,λi,μj>0.M = \sum_{i=1}^{n_{+}(M)} \lambda_i q_i q_i^{\mathsf{T}} - \sum_{j=1}^{n_{-}(M)} \mu_j p_j p_j^{\mathsf{T}}, \qquad\lambda_i,\mu_j > 0.

Pair positive and negative eigendirections as much as possible. For a paired block

λqqT−μppT,λ,μ>0,\lambda qq^{\mathsf{T}} - \mu pp^{\mathsf{T}}, \qquad\lambda,\mu> 0,

define

u=λ q+μ p,v=λ q−μ p.u = \sqrt{\lambda}\,q + \sqrt{\mu}\,p, \qquad v = \sqrt{\lambda}\,q - \sqrt{\mu}\,p.

Then

12(uv⊤+vu⊤)=λqq⊤−μpp⊤.\frac{1}{2}\left(uv^{\top}+vu^{\top}\right)=\lambda qq^{\top}-\mu pp^{\top}.

Thus one untied atom represents one positive and one negative eigendirection together.

If any positive eigendirections remain unpaired, represent each λqq⊤\lambda qq^{\top} by the tied special case

u=v=λ q.u=v=\sqrt{\lambda}\,q.

If any negative eigendirections remain unpaired, represent each −μpp⊤-\mu pp^{\top} by

u=μ p,v=−μ p.u=\sqrt{\mu}\,p,\qquad v=-\sqrt{\mu}\,p.

Therefore the total number of atoms used is

max⁡{n+(M),n−(M)}.\max\{n_{+}(M),n_{-}(M)\}.

This proves

runtied(M)≤max⁡{n+(M),n−(M)}.r_{\mathrm{untied}}(M)\leq\max\{n_{+}(M),n_{-}(M)\}.

Combining the lower and upper bounds gives

runtied(M)=max⁡{n+(M),n−(M)}.r_{\mathrm{untied}}(M)=\max\{n_{+}(M),n_{-}(M)\}.

Exact rank, border rank, and bounded approximation

Fix k≥2k\geq2 and f∈{tied,untied}f\in\{\mathrm{tied},\mathrm{untied}\}. All parameters and closures are real. We use the coefficient-tensor Frobenius norm; equivalent norms give the same closure. Define

Cf,m(k)={T:rf(T)≤m},rf(T)=min⁡{m:T∈Cf,m(k)},\mathcal{C}_{f,m}^{(k)}=\{T:r_f(T)\leq m\},\qquad r_f(T)=\min\{m:T\in\mathcal{C}_{f,m}^{(k)}\},

and

ef,m(T)=inf⁡S∈Cf,m(k)∥T−S∥F=dist⁡(T,Cf,m(k)).e_{f,m}(T)=\inf_{S\in\mathcal{C}_{f,m}^{(k)}}\|T-S\|_F=\operatorname{dist}\left(T,\mathcal{C}_{f,m}^{(k)}\right).

The error is positive for m<r‾f(T)m<\underline{r}_f(T), zero but unattained for rf(T)≤m<r‾f(T)r_f(T)\leq m<\underline{r}_f(T), and zero and attained for m≥rf(T)m\geq r_f(T). A positive infimum alone does not guarantee a best approximation [11].

Coefficient sizes. For output wa∈Rpw_a\in\mathbb{R}^p, define

Aat=∥wa∥∥ua∥k,Aau=∥wa∥∥ua∥k−1∥va∥.A_a^t=\|w_a\|\|u_a\|^k,\qquad A_a^u=\|w_a\|\|u_a\|^{k-1}\|v_a\|.

These sizes are invariant under factor rescalings that preserve the raw atom; for scalar output, ∥wa∥=∣wa∣\|w_a\|=|w_a|. Let Cf,m(k)(B)\mathcal{C}_{f,m}^{(k)}(B) contain tensors with a representation using at most mm atoms and total size ∑aAaf≤B\sum_a A_a^f\leq B.

Proposition C.3 (Bounded compactness and forced size divergence). For finite B≥0B\geq0, both classes Cf,m(k)(B)\mathcal{C}_{f,m}^{(k)}(B) are compact. Thus

δf,m(B;T)=min⁡S∈Cf,m(k)(B)∥T−S∥F\delta_{f,m}(B;T)=\min_{S\in\mathcal{C}_{f,m}^{(k)}(B)}\|T-S\|_F

exists, is positive when m<rf(T)m<r_f(T), and converges to ef,m(T)e_{f,m}(T) as B→∞B\to\infty. Every fixed-width sequence converging to a target outside the exact class has Amax⁡→∞A_{\max}\to\infty.

Proof. Normalize the input directions and absorb their norms into the output factors, padding zero atoms as needed. The tied parameter set becomes

{(w~1,…,w~m):∑a∥w~a∥≤B}×(Sd−1)m.\{(\widetilde{w}_1,\ldots,\widetilde{w}_m):\sum_a\|\widetilde{w}_a\|\leq B\}\times(\mathbb{S}^{d-1})^m.

It is compact, as is its image under the continuous tensor-sum map. The untied case uses two unit directions per atom. Compactness gives the positive minimum outside the class. The budgeted classes increase to the unrestricted class, so their distances decrease to ef,m(T)e_{f,m}(T). If Amax⁡A_{\max} did not tend to infinity, a subsequence would have total size at most mCmC for some finite CC; compactness would place its limit in the exact class, a contradiction.

Proposition C.4 (Separation under a common coefficient-size budget). If runtied(T)≤m<rtied(T)r_{\mathrm{untied}}(T) \le m < r_{\mathrm{tied}}(T), there is a finite B0B_0 such that every B≥B0B \ge B_0 permits exact untied width-mm representation, whereas δtied,m(B;T)>0\delta_{\mathrm{tied},m}(B;T)>0. As B→∞B \to\infty, this tied distance has a positive limit if m<rtied(T)m < r_{\mathrm{tied}}(T) and tends to zero otherwise.

Proof. Take B0B_0 to be the total size of an exact untied representation. Apply Proposition C.3 and the definition of border rank to the tied class.

The monomial limit. For independent linear forms ℓ\ell, hh,

Pε=(ℓ+εh)k−(ℓ−εh)k2kε⟶ℓk−1h.(1)P_\varepsilon=\frac{(\ell+\varepsilon h)^k-(\ell-\varepsilon h)^k}{2k\varepsilon}\longrightarrow\ell^{k-1}h. \tag*{(1)}

Hence the tied border rank is at most two. It is at least two because the first-derivative span of ℓk−1h\ell^{k-1}h has dimension two, while a pure power has dimension at most one, and this matrix-rank bound survives closure. Thus rtied(Fk)=2r_{\mathrm{tied}}(F_k)=2 [8]. For k≥3k \ge3, binomial expansion gives error O(ε2)O(\varepsilon^2) and Amax⁡=Θ(∣ε∣−1)A_{\max}=\Theta(|\varepsilon|^{-1}), hence squared error O(Amax⁡−4)O(A_{\max}^{-4}) along this path. This gives an upper bound; optimality is not established. For k=2k=2, the identity is exact for nonzero ε\varepsilon.

Closed quadratic classes

Proposition C.5 (Scalar quadratic closedness). For scalar degree-two targets and every width mm, both tied and untied classes are closed. Their exact and real border ranks coincide.

Proof. By Theorem 4.6, the tied class consists of symmetric matrices of rank at most mm, a closed condition defined by vanishing (m+1)(m+1)-minors. The untied class is

{M=MT:n+(M)≤m, n−(M)≤m},\{M=M^{\mathsf T}: n_+(M)\le m,\ n_-(M)\le m\},

which is closed by continuity of ordered eigenvalues. Distance from a target outside either class is positive and attained: a minimizing sequence can be restricted to a bounded ball, whose intersection with the class is compact.

For x∼N(0,Id)x\sim\mathcal{N}(0,I_d) and symmetric HH,

E[(xTHx)2]=2∥H∥F2+(tr⁡H)2.\mathbb{E}[(x^{\mathsf T}Hx)^2]=2\lVert H\rVert_{\mathrm{F}}^2+(\operatorname{tr}H)^2.

A positive coefficient-norm gap therefore implies a positive optimal population MSE. An observed finite-sample residual can also reflect sampling and optimization. For Q=diag⁡(1,1,−1,−1)Q=\operatorname{diag}(1,1,-1,-1), tied rank-mm spectral truncation gives relative squared Frobenius error (4−m)/4(4-m)/4 for 0≤m≤40\le m\le4; the untied error is 1/21/2 at m=1m=1 and zero at m=2m=2.

Universal comparison and independent interaction blocks

Proposition C.6 (Universal exact- and border-rank comparison). For every degree-kk coefficient tensor TT, with k≥2k\ge2,

runtied(T)≤rtied(T)≤k runtied(T),runtied(T)≤rtied(T)≤2 runtied(T).r_{\mathrm{untied}}(T)\le r_{\mathrm{tied}}(T)\le k\,r_{\mathrm{untied}}(T),\qquad r_{\mathrm{untied}}(T)\le r_{\mathrm{tied}}(T)\le2\,r_{\mathrm{untied}}(T).

Both upper constants are attained by FkF_k.

Proof. Containment gives the lower bounds, including after closure. For independent u,vu,v, a change of coordinates maps (uTx)k−1(vTx)(u^{\mathsf T}x)^{k-1}(v^{\mathsf T}x) to FkF_k, which has a real kk-power decomposition by Theorem 4.10; collinear directions require at most one power. Applying these decompositions to each atom, with its output vector, proves the exact upper bound. Equation (1) puts each untied atom in the closure of the tied width-two class. Summing and taking closure gives the border upper bound. The ranks of FkF_k establish sharpness.

Theorem C.7 (Independent interaction blocks). Let k≥2k \ge2, s≥1s \ge1, d≥2sd \ge2s, and ci∈R∖{0}c_i \in\mathbb{R} \setminus\{0\}. For linearly independent real forms ℓ1,h1,…,ℓs,hs\ell_1,h_1,\ldots,\ell_s,h_s, define

Fk,s(x)=∑i=1sciℓi(x)k−1hi(x).F_{k,s}(x)=\sum_{i=1}^{s}c_i\ell_i(x)^{k-1}h_i(x).

Then

(runtied,rtied,runtied,rtied)(Fk,s)=(s,ks,s,2s).(r_{\mathrm{untied}},r_{\mathrm{tied}},r_{\mathrm{untied}},r_{\mathrm{tied}})(F_{k,s})=(s,ks,s,2s).

Proof. Use the 2s2s independent forms as coordinates. The first derivatives span the 2s2s distinct monomials x2i−1k−2x2ix_{2i-1}^{k-2}x_{2i} and x2i−1k−1x_{2i-1}^{k-1}, including when k=2k=2. Each tied atom contributes at most one derivative direction and each untied atom at most two. Since these matrix-rank bounds survive closure,

rtied≥2s,runtied≥s.r_{\mathrm{tied}} \ge2s,\qquad r_{\mathrm{untied}} \ge s.

One untied atom per summand gives runtied≤sr_{\mathrm{untied}} \le s; Equation (1) gives rtied≤2sr_{\mathrm{tied}} \le2s; and real kk-power decompositions give rtied≤ksr_{\mathrm{tied}} \le ks.

For k≥3k \ge3, absorb cic_i into x2ix_{2i} and apply the additivity theorem for pairwise coprime monomials [4 Theorem 3.2]: the complex Waring rank is ksks, so the real rank is at least ksks. Extra coordinates cannot lower rank, since restricting them to zero preserves a decomposition. For k=2k=2, each block has one positive and one negative eigenvalue, and Theorem 4.6 gives tied rank 2s=ks2s=ks. ▫

Thus the tied approximation gap is positive at s≤m<2ss \le m < 2s, while the untied family is exact. At 2s≤m<ks2s \le m < ks, the tied infimum is zero but unattained.

Necessary width bounds for analytic channels

Proposition C.8 (Degree-wise necessity). Let gg be analytic near zero and let FF be represented near zero by a bias-free width-mm CP-MLP in family ff. Then F(0)=0F(0)=0, inactive Taylor components vanish, and

m≥sup⁡k∈S(g)rf(F(k)).m \ge\sup_{k\in S(g)} r_f(F^{(k)}).

Proof. Equality near zero implies equality of Taylor coefficients. Each active degree is a sum of at most mm atoms, with its nonzero gate coefficient absorbed into the output factors; inactive components vanish. These conditions are necessary, but parameters shared across degrees prevent choosing the decompositions independently. ▫

Proposition C.9 (Degree-wise rank one with SiLU-channel width two). The function

G(x1,x2)=[SiLU⁡(x1)+SiLU⁡(2x1)]x2G(x_1,x_2)=[\operatorname{SiLU}(x_1)+\operatorname{SiLU}(2x_1)]x_2

has untied atom rank one at every active Taylor degree, but its exact untied SiLU-channel width is two.

Proof. The displayed representation uses two channels, and each active component is a nonzero multiple of x1k−1x2x_1^{k-1}x_2. To exclude width one, expand

SiLU⁡(z)=12z+14z2−148z4+O(z6).\operatorname{SiLU}(z)=\frac{1}{2}z+\frac{1}{4}z^2-\frac{1}{48}z^4+O(z^6).

For ASiLU⁡(sx1)x2A\operatorname{SiLU}(sx_1)x_2, the coefficients b2,b3,b5b_2,b_3,b_5 of x1x2,x12x2,x14x2x_1x_2,x_1^2x_2,x_1^4x_2 satisfy

b2=As2,b3=As24,b5=−As448,b22b5b33=−13.b_2=\frac{As}{2},\qquad b_3=\frac{As^2}{4},\qquad b_5=-\frac{As^4}{48},\qquad\frac{b_2^2b_5}{b_3^3}=-\frac{1}{3}.

when b3≠0b_3\ne0. For GG, they are 3/2,5/4,−17/483/2,5/4,-17/48, giving −51/125≠−1/3-51/125\ne-1/3.

Any width-one representation matching the quadratic term must have detector and value forms proportional to x1,x2x_1,x_2 in some order, by unique factorization. The cubic term fixes the detector along x1x_1 and the value along x2x_2, so the invariant excludes all width-one representations. ▫

The obstruction is the gate’s fixed coefficient ratios across degrees. Small held-out function error alone does not bound every Taylor coefficient.

Experimental details

This section documents the experimental methodology in sufficient detail for an independent re-implementation. Code that reproduces every figure and table is available at https://github.com/rarefin/CP-MLPs.

Computational requirements

All experiments run on a standard CPU; no GPU or specialized hardware is required. The experiments of § 5 and Appendices § 11.7–§ 11.8 each complete in about one minute, about five minutes in total on a single CPU core. The additional experiments of Appendices § 11.9–§ 11.13 take about two to three hours in total on one core. Memory usage stays below 1 GB throughout.

The experiments use standard scientific Python: numpy for linear algebra, scipy.optimize.minimize (L-BFGS-B method) for nonlinear least-squares fitting, and matplotlib for figure generation. The experiments of Appendices § 11.10, § 11.11, and § 11.13 also use PyTorch on the CPU, for automatic differentiation and for Adam.

Data generation

The experiments fit two kinds of targets: polynomial coefficient tensors (in coefficient space) and scalar functions of Gaussian inputs (in function space). We describe both.

Coefficient-space targets. A degree-kk polynomial in dd variables corresponds to a coefficient vector T∈RDT \in\mathbb{R}^{D} in the basis of degree-kk monomials, where D=(d+k−1k)D = \binom{d+k-1}{k}. For d=2d = 2 we use the basis indexed by the exponent of x2x_2, so index jj corresponds to the monomial x1k−jx2jx_1^{k-j}x_2^j. The role-asymmetric witness Fk=x1k−1x2F_k = x_1^{k-1}x_2 of Theorem 4.10 therefore has a single nonzero entry at index j=1j = 1; the pure-power target x1kx_1^k has its single nonzero entry at j=0j = 0.

The model’s prediction in coefficient space is the coefficient vector of its parameterized polynomial, computed analytically from the architecture’s atom form (Section D.3). The loss is the squared ℓ2\ell_2 distance between the predicted coefficient vector and the target.

Function-space targets. For experiments using actual Gaussian inputs (§ 5.3 and § 5.4), we sample inputs X∈RN×dX \in\mathbb{R}^{N \times d} with each row drawn i.i.d. from N(0,Id)\mathcal{N}(0, I_d). The sample size NN is given in Table 2. Targets are evaluated pointwise: for y(x)=x1k−1x2y(x) = x_1^{k-1}x_2, we set yi=Xi,1k−1Xi,2y_i = X_{i,1}^{k-1}X_{i,2}; for the indefinite quadratic experiment, yi=Xi⊤QXiy_i = X_i^\top QX_i with Q=diag⁡(1n+,−1n−,0d−n+−n−)Q = \operatorname{diag}(1^{n_+}, -1^{n_-}, 0^{d-n_+-n_-}). The loss is the mean squared error 1N∑i(y^i−yi)2\frac{1}{N}\sum_i(\hat{y}_i-y_i)^2.

ExperimentSectionInput dim ddSample sizeSeedsRestarts
Indefinite quadratic targets§ 5.1825638
Role-asymmetric monomial targets§ 5.22coef. space25
Pure-vs-mixed reversal§ 5.34256105
Gated MLPs§ 5.4420033
Matched-parameter control (Gated MLPs)§ 5.4420033
Scope boundary§D.73coef. space24∗4^*

Table 2. Hyperparameters for each experiment.

Random seeds. Each experiment uses random seeds that are deterministic functions of configuration values. For example, the quadratic experiment uses seed = s * 137 + n_plus * 17 + n_minus for seed index ss and signature (n+,n−)(n_+,n_-); the width-sweep experiment uses seed = s * 37 + k * 11, shared across the widths of a sweep. Fixed seeds make every run reproducible with the same library versions.

Model implementations

Each model is parameterized by a flat parameter vector θ∈RP\theta\in\mathbb{R}^{P} that the optimizer treats as the only variable. The total parameter count PP depends on the architecture and the width mm.

Function-space architectures. Given input X∈RN×dX \in\mathbb{R}^{N \times d} and width mm, all architectures follow the CP-MLP structure of Definition 3.1: project inputs through one or two linear maps, apply an elementwise nonlinearity, multiply per channel, and combine via scalar output coefficients c∈Rmc \in\mathbb{R}^{m} (the scalar specialization of the output direction waw_a).

For tied polynomial atoms with detector matrix U∈Rm×dU \in\mathbb{R}^{m \times d} and coefficients cc:

y^i=∑a=1mca(ua⊤xi)k,θ=(vec⁡(U),c)∈Rm(d+1).\hat{y}_i = \sum_{a=1}^{m} c_a \left(u_a^\top x_i\right)^k,\qquad\theta= \left(\operatorname{vec}(U),c\right) \in\mathbb{R}^{m(d+1)}.

For untied polynomial atoms with detector UU, value V∈Rm×dV \in\mathbb{R}^{m \times d}, and coefficients cc:

y^i=∑a=1mca(uaTxi)k−1(vaTxi),θ=(vec⁡(U),vec⁡(V),c)∈Rm(2d+1).\hat{y}_i = \sum_{a=1}^{m} c_a \left(u_a^{\mathsf{T}} x_i\right)^{k-1}\left(v_a^{\mathsf{T}} x_i\right), \qquad\theta= \left(\operatorname{vec}(U),\operatorname{vec}(V),c\right) \in\mathbb{R}^{m(2d+1)}.

For the four gated architectures in § 5.4, replace the polynomial atoms with their gated analogues from Table 1:

tied ReLU2:y^=∑acaReLU⁡(uaTx)2,untied ReGLU:y^=∑acaReLU⁡(uaTx)(vaTx),\text{tied ReLU}^{2}: \quad\hat{y} = \sum_{a} c_a \operatorname{ReLU}\left(u_a^{\mathsf{T}}x\right)^2, \qquad\text{untied ReGLU:} \quad\hat{y} = \sum_{a} c_a \operatorname{ReLU}\left(u_a^{\mathsf{T}}x\right)\left(v_a^{\mathsf{T}}x\right),
tied SwiGLU:y^=∑acaSiLU⁡(uaTx)(uaTx),untied SwiGLU:y^=∑acaSiLU⁡(uaTx)(vaTx).\text{tied SwiGLU:} \quad\hat{y} = \sum_{a} c_a \operatorname{SiLU}\left(u_a^{\mathsf{T}}x\right)\left(u_a^{\mathsf{T}}x\right), \qquad\text{untied SwiGLU:} \quad\hat{y} = \sum_{a} c_a \operatorname{SiLU}\left(u_a^{\mathsf{T}}x\right)\left(v_a^{\mathsf{T}}x\right).

with SiLU⁡(z)=zσ(z)\operatorname{SiLU}(z) = z\sigma(z) and σ(z)=1/(1+e−z)\sigma(z) = 1/(1+e^{-z}). Tied gated architectures have parameter count m(d+1)m(d+1); untied gated architectures have m(2d+1)m(2d+1). The matched-parameter control of § 5.4 pairs untied width mm against tied width 2m2m, equalizing input-projection parameters at 2md2md.

Coefficient-space implementations. For coefficient-space experiments we compute the coefficient vector of each parameterized polynomial directly, avoiding sampling. For tied atoms, the multinomial theorem gives

(uTx)k=∑∣α∣=k(kα)uαxα,(kα)=k!α1!⋯αd!.\left(u^{\mathsf{T}}x\right)^k = \sum_{\lvert\alpha\rvert=k} \binom{k}{\alpha}u^\alpha x^\alpha, \qquad\binom{k}{\alpha} = \frac{k!}{\alpha_1!\cdots\alpha_d!}.

For untied detector-value atoms,

(uTx)k−1(vTx)=(∑∣β∣=k−1(k−1β)uβxβ)(∑j=1dvjxj),\left(u^{\mathsf{T}}x\right)^{k-1}\left(v^{\mathsf{T}}x\right) = \left(\sum_{\lvert\beta\rvert=k-1}\binom{k-1}{\beta}u^\beta x^\beta\right)\left(\sum_{j=1}^{d}v_jx_j\right),

which expands into a degree-kk multi-index sum. The total predicted coefficient vector is ∑aca\sum_a c_a times the per-atom coefficient vector, and the loss is the squared ℓ2\ell_2 distance from the target.

Optimization protocol

The experiments of § 5 and Appendices § 11.7–§ 11.8 minimize the squared loss using scipy.optimize.minimize with method L-BFGS-B. The iteration limit and tolerances are: indefinite quadratics 3000, ftol=10−22\mathrm{ftol}=10^{-22}, gtol=10−15\mathrm{gtol}=10^{-15}; width sweep 2000, 10−2210^{-22}, 10−1810^{-18}; pure-vs-mixed 5000, 10−2010^{-20}, 10−1510^{-15}; gated MLPs 1500, 10−1810^{-18}, 10−1210^{-12}; matched control 1500, 10−2010^{-20}, 10−1210^{-12}; scope boundary 800, 10−2210^{-22}, 10−1510^{-15}. Appendices § 11.9–§ 11.13 describe their own settings. For each random seed, we run multiple restarts with parameter initialization θ0∼N(0,0.25)\theta_0 \sim\mathcal{N}(0,0.25) (applied independently to each entry of θ\theta). We record the best-of-restarts loss per seed and aggregate (median, min, max) across seeds.

Per-experiment configurations

∗^*Twelve restarts for the untied model at m=1m=1.

For coefficient-space experiments, fits operate directly on the (d+k−1k)\binom{d+k-1}{k} monomial coefficients of the target polynomial, eliminating sampling and function-evaluation noise. For experiments with finite samples, inputs are drawn from N(0,Id)\mathcal{N}(0,I_d) once per seed and held fixed across restarts within that seed.

Convergence criterion

We report best-of-restarts MSE without applying a separate convergence threshold. A configuration is described as fitting the target if the residual reaches machine precision (MSE <10−8< 10^{-8}) and as failing if the residual stays above 10−110^{-1}. The intermediate regime is interpreted as a partial fit due to optimizer-landscape difficulty rather than a representational limitation, and is discussed for the k=5k = 5 case in § 5.2.

To distinguish the plateau of § 5.2 from a slow-optimization artifact, we re-ran the failed configurations with 5×5 \times the L-BFGS iteration budget. The residual stayed within 10% of its plateau value across all configurations tested, confirming that the plateau is robust to optimization budget.

Scope of the role-asymmetric separation

Theorem 4.10 shows that one untied detector-value atom suffices for the role-asymmetric witness Fdv=x1k−1x2F_{\mathrm{dv}} = x_1^{k-1}x_2, where one direction acts as a repeated detector and one direction supplies the single value. This appendix tests the boundary of that statement: targets with multiple independent value roles do not reduce to a single untied atom.

Concretely, at k=3k = 3 we compare tied, untied, and a multi-leg oracle (the smallest atom family rich enough to express any product of kk linear forms) on the role-asymmetric witness FdvF_{\mathrm{dv}} and on the square-free product

Fsf=x1x2⋯xk,F_{\mathrm{sf}} = x_1x_2\cdots x_k,

in which each of the kk variables appears exactly once.

Figure 6 and Table 3 confirm three regimes. On FdvF_{\mathrm{dv}}, untied atoms and the multi-leg oracle both fit at m=1m = 1. Tied atoms keep a positive loss at m=1m = 1, reach small but nonzero losses at m=2m = 2 and m=3m = 3 (about 10−810^{-8} and 10−1010^{-10}, consistent with border rank two), and reach about 10−1310^{-13} at m=4m = 4. On FsfF_{\mathrm{sf}}, the multi-leg oracle fits at m=1m = 1, untied requires m=2m = 2, and tied requires m≥2k−1m \ge2^{k-1}, matching its Waring rank (by Proposition 4.9). The untied advantage is therefore specialized to targets with a single value role and does not extend to arbitrary products.

Figure 6. Scope boundary at k=3k = 3. Untied atoms are rank one on the role-asymmetric witness Fdv=x1k−1x2F_{\mathrm{dv}} = x_1^{k-1}x_2 but not on the square-free product Fsf=x1x2⋯xkF_{\mathrm{sf}} = x_1x_2\cdots x_k; the multi-leg oracle is rank one on both.

TargetArchitecturem=1m = 1m=2m = 2m=3m = 3m=4m = 4
x1k−1x2x_1^{k-1}x_2 (role-asym.)tied4.0×10−14.0 \times10^{-1}2.0×10−82.0 \times10^{-8}2.3×10−102.3 \times10^{-10}1.2×10−131.2 \times10^{-13}
x1k−1x2x_1^{k-1}x_2 (role-asym.)untied (det.-val.)<10−15<10^{-15}6.3×10−156.3 \times10^{-15}2.0×10−152.0 \times10^{-15}<10−15<10^{-15}
x1k−1x2x_1^{k-1}x_2 (role-asym.)multi-leg oracle1.2×10−131.2 \times10^{-13}6.4×10−146.4 \times10^{-14}<10−15<10^{-15}<10−15<10^{-15}
x1x2⋯xkx_1x_2\cdots x_k (square-free)tied6.1×10−16.1 \times10^{-1}3.3×10−13.3 \times10^{-1}1.4×10−11.4 \times10^{-1}1.2×10−151.2 \times10^{-15}
x1x2⋯xkx_1x_2\cdots x_k (square-free)untied (det.-val.)3.3×10−13.3 \times10^{-1}<10−15<10^{-15}<10−15<10^{-15}<10−15<10^{-15}
x1x2⋯xkx_1x_2\cdots x_k (square-free)multi-leg oracle<10−15<10^{-15}<10−15<10^{-15}<10−15<10^{-15}<10−15<10^{-15}

Table 3. Scope boundary at k=3k = 3. Median best-of-restarts MSE over two seeds.

Matched input-projection control in gated MLPs

Untied architectures use two input projections per channel. We compare untied width mm with tied width 2m2m, matching input-projection counts at 2md2md. The tied model has mm additional scalar-output parameters.

Figure 7 shows that the ordering depends on degree and width. At k=2k=2, tied SwiGLU at width four and untied SwiGLU at width two attain comparable median losses, 5.5×10−135.5 \times10^{-13} and 4.3×10−134.3 \times10^{-13}, respectively. At k=3k=3, untied width four reaches 4.0×10−44.0 \times10^{-4}, compared with 4.4×10−34.4 \times10^{-3} for tied width eight, an approximately 10.8-fold difference. These fixed-budget fits show a selected untied advantage, not a uniform separation or minimum-width certificate.

Figure 7. Matched input-projection control on x1k−1x2x_1^{k-1}x_2: untied width mm (dashed) versus tied width 2m2m (solid). Tied SwiGLU uses SiLU(zz)zz. Median over three seeds; bands show min–max. Total parameter counts are m(2d+1)m(2d+1) and 2m(d+1)2m(d+1), respectively.

Border-rank diagnostics

The additional coefficient-space experiments use the apolar norm. If f(x)=∑∣α∣=kfαxαf(x)=\sum_{|\alpha|=k}f_\alpha x^\alpha, then

∥f∥ap2=∑∣α∣=kα!k!fα2\lVert f\rVert_{\mathrm{ap}}^2=\sum_{|\alpha|=k}\frac{\alpha!}{k!}f_\alpha^2

This is the Frobenius norm of the symmetric coefficient tensor and satisfies ∥(u⊤x)k∥ap=∥u∥k\lVert(u^\top x)^k\rVert_{\mathrm{ap}}=\lVert u\rVert^k.

Reported losses are relative squared errors ∥f^−f∥ap2/∥f∥ap2\lVert\hat{f}-f\rVert_{\mathrm{ap}}^2/\lVert f\rVert_{\mathrm{ap}}^2. Unlike the unweighted monomial-coordinate norm used in § 5.2, this convention accounts for multinomial multiplicities. The two norms have the same closure, but numerical losses need not agree.

For the size diagnostic, each configuration has 27 candidate fits: 12 unregularized initializations and five at each of three penalties on ∑aAa2\sum_a A_a^2. We record the smallest loss and the smallest Amax⁡A_{\max} among candidates with loss at most 1.02 times that minimum. At the selected near-best candidate, we also report the intrinsic condition number κ=1/σmin⁡(T)\kappa=1/\sigma_{\min}(T) for the normalized Terracini/Jacobian matrix after removing scaling directions [19]. A small singular value indicates sensitivity of decomposition parameters; it is not a certificate of global optimality. The notation “n.i.” means that local nonuniqueness makes this diagnostic uninformative.

Table 4 follows the three regimes for the original monomial. At tied width one, the positive floor is attained because the class is closed. At width two, the smallest near-best candidate has large size; this is consistent with the forced-divergence statement for any sequence whose error tends to zero. A finite collection of large candidates does not exclude a smaller-size candidate that the optimizer missed. At exact width three, a bounded near-zero fit is recovered.

FamilymmRegimeBest lossBest Amax⁡A_{\max}Near-best Amax⁡A_{\max}κ\kappa
Untied1Exact control1.6×10−291.6 \times10^{-29}1.001.001.7
Tied1Positive infimum0.55560.3850.3851.5
Tied2Zero, unattained5.4×10−125.4 \times10^{-12}83.1783.174.4×1074.4 \times10^7
Tied3Exact width1.5×10−281.5 \times10^{-28}0.4710.471n.i.

Table 4. Coefficient-size diagnostics for F3=x12x2F_3=x_1^2x_2. Exact and border ranks determine the regimes; losses and sizes describe the candidate fits.

For the independent-block target, Table 5 places the positive-to-zero infimum transition at width four, as predicted by Theorem C.7. The large best-fit size at width two does not contradict the positive gap: this candidate approximates one block through a degeneration while leaving the other unfitted, at loss 0.5. Its condition number is 7.9×1067.9 \times10^{6}, whereas a candidate within 2% of the same loss has size 2.27. The theorem identifies the regime; the diagnostics describe how individual fits approach it.

FamilymmRegimeBest lossBest Amax⁡A_{\max}Near-best Amax⁡A_{\max}κ\kappa
Untied2Exact control6.4×10−196.4 \times10^{-19}1.001.001.7
Tied2Positive infimum0.500046.952.278.6×1028.6 \times10^{2}
Tied3Positive infimum0.277866.302.278.6×1028.6 \times10^{2}
Tied4Zero, unattained1.1×10−101.1 \times10^{-10}47.7447.7368.3×1068.3 \times10^{6}
Tied5Zero, unattained9.3×10−119.3 \times10^{-11}34.4034.4051.9×10131.9 \times10^{13}
Tied6Exact width8.3×10−168.3 \times10^{-16}3.723.72n.i.

Table 5. The same diagnostic for F3,2=x12x2+x32x4F_{3,2}=x_1^2x_2+x_3^2x_4. Its tied border width is four and tied exact width is six.

As a closed-class control, tied fits to Q=diag⁡(1,1,−1,−1)Q=\operatorname{diag}(1,1,-1,-1) yield relative coefficient errors 0.7500, 0.5000, and 0.2500 at widths one, two, and three, matching (4−m)/4(4-m)/4. Penalty sweeps along the monomial degeneration give log–log slopes −4.003, −4.002, and −4.002 for squared error versus Amax⁡A_{\max} at k=3,4,5k=3,4,5, consistent with the constructive O(Amax⁡−4)O(A_{\max}^{-4}) path in Equation (1). These slopes do not prove an optimal trade-off.

Recovery of known representations

To separate optimization from expressivity, we retain generating parameters for targets constructed from three atoms. These witnesses establish width-three representability, not minimal rank three. Fitting uses float64 L-BFGS-B with automatic differentiation, d=3d=3, and 60 restarts per configuration. Recovery means relative apolar loss below 10−1010^{-10}. Table 6 reports recovery fractions for the respective target constructions in each family; it should not be read as a comparison on identical targets.

kkTied widthTied widthTied widthUntied widthUntied widthUntied widthBest untied loss at width 3
345345
30.330.320.700.800.921.001.2×10−201.2 \times10^{-20}
40.420.670.750.170.650.882.4×10−182.4 \times10^{-18}
50.770.800.850.000.020.001.7×10−51.7 \times10^{-5}
60.080.200.220.000.000.008.1×10−98.1 \times10^{-9}

Table 6. Recovery fractions for configurations with a width-three generating witness. Witness losses are approximately 10−3210^{-32}. Widths four and five pad the representation with additional channels.

No untied width-three restart reaches the tolerance at k=5k=5 or k=6k=6, despite the generating witnesses. At k=5k=5, widening yields one recovery in 60 attempts at width four and none at width five. Thus the tested extra widths do not reliably resolve the optimization difficulty. In a separate structured example, F4,2F_{4,2} has exact untied rank two, but eight restarts all return loss 0.25. A 40-restart run recovers the representation ten times, with best loss 5.4×10−205.4 \times10^{-20} and median 0.25. These failures cannot be used as rank lower bounds.

Dense random targets and mixtures of degrees

For scalar homogeneous targets with dd variables and k≥3k \ge3, let N=(d+k−1k)N=\binom{d+k-1}{k}. Outside the known defective cases, complex-generic ranks are

rtiedgen=⌈Nd⌉,runtiedgen=⌈N2d−1⌉r_{\mathrm{tied}}^{\mathrm{gen}}=\left\lceil\frac{N}{d}\right\rceil,\qquad r_{\mathrm{untied}}^{\mathrm{gen}}=\left\lceil\frac{N}{2d-1}\right\rceil
ddkkTargetNNTied obs./pred.Untied obs./pred.Obs. ratioAbove prediction
25063/33/21.00Untied
25164/32/22.00Tied
26074/43/31.33—
26175/43/31.67Tied
330104/43/31.33—
331104/43/31.33—
340156/63/32.00—
341157/64/31.75Both
430206/54/41.50Tied
431206/54/41.50Tied
4403511/106/51.83Both
4413510/106/51.67Untied
530358/85/51.60—
531358/85/51.60—

Table 7. Dense random targets: observed recovery widths and complex-generic predictions. “Above” marks recovery only above the prediction and does not certify a larger real rank.

by the Alexander–Hirschowitz theorem and the tangential-Veronese result [2, 1]. A defective case is one in which the dimension count underestimates the required generic width. For the tied family the count increases by one at (d,k)=(3,4),(4,4),(5,3),(5,4)(d,k)=(3,4),(4,4),(5,3),(5,4); for the untied family it increases by one at (d,k)=(3,3),(4,3),(5,3)(d,k)=(3,3),(4,3),(5,3).

We sample coefficients independently from a Gaussian in an apolar-orthonormal basis and fix these predictions before fitting. The observed recovery width is the smallest tested width reaching relative loss below 10−1010^{-10} in 32 restarts. The complex lower threshold applies almost surely to these continuously sampled real targets; it need not be an attainable real width. A real near-zero fit at that width supplies numerical recovery evidence. A fit only at the next width leaves higher real rank, an ill-conditioned limit, and optimizer failure unresolved.

Table 7 recovers several nontrivial corrected counts. The ternary cubic requires an untied prediction of three, rather than the naive two. At (d,k)=(5,3)(d,k)=(5,3), both predictions increase, to tied width eight and untied width five; both are recovered on two targets, with best losses at least 2.5×10−42.5 \times10^{-4} one channel below and below 10−1510^{-15} at the predictions. For tied (3,4)(3,4) target zero, losses below and at width six are 1.47×10−21.47 \times10^{-2} and 9.86×10−199.86 \times10^{-19}. Observed tied/untied width ratios range from one to two; the factor-kk separation of the monomial is a structured example rather than a generic prediction.

Mixtures of degrees. We also fit independent random components at degrees KK, with shared directions but a free coefficient for each channel and degree. With Ntot=∑k∈K(d+k−1k)N_{\mathrm{tot}}=\sum_{k\in K}\binom{d+k-1}{k}, parameter counting gives the reference widths

mtcount=⌈Ntotd−1+∣K∣⌉,mucount=⌈Ntot2d−2+∣K∣⌉.m_t^{\mathrm{count}}=\left\lceil\frac{N_{\mathrm{tot}}}{d-1+|K|}\right\rceil,\qquad m_u^{\mathrm{count}}=\left\lceil\frac{N_{\mathrm{tot}}}{2d-2+|K|}\right\rceil.

These are dimension-count benchmarks, not proved generic-rank formulas for the shared-direction model. The relaxation also differs from a fixed analytic gate because its coefficients are free across degrees.

The untied recovered width is smaller in all seven comparisons in Table 8. The mixture with the largest coefficient dimension recovers both counting benchmarks. Recovery above a benchmark is unresolved and is not attributed to a specific real rank or optimization mechanism.

ddKKNtotN_{\mathrm{tot}}TiedUntied
2{2,3,4,5}\{2,3,4,5\}185 (4), 4 (4)4 (3), 3 (3)
3{2,3}\{2,3\}165 (4), 4 (4)4 (3), 3 (3)
3{2,3,4}\{2,3,4\}317 (7)5 (5)
4{2,3}\{2,3\}307 (6), 7 (6)4 (4), 5 (4)

Table 8. Exploratory degree-mixture fits. Each entry is recovered width (dimension-count benchmark). Paired entries correspond to two independently sampled targets.

Distillation of trained gated blocks

A frozen random tanh network R8→R8\mathbb{R}^{8} \to\mathbb{R}^{8} with hidden width 64 supplies a common target. We train three teachers with 192 parameters each, excluding biases: untied SwiGLU width eight, tied SwiGLU width twelve, and a tanh MLP width twelve. Their training relative MSEs on the common target are 0.295, 0.172, and 0.112, respectively. Tied SwiGLU uses SiLU⁡(z)z\operatorname{SiLU}(z)z, preserving the untied model’s gate while imposing v=uv = u.

For each teacher and m∈{2,3,4,6,8,10,12}m \in\{2, 3, 4, 6, 8, 10, 12\}, we fit untied width mm, tied width mm, and approximately parameter-matched tied width 1.5m1.5m (rounded when necessary). With input and output dimension eight, an untied channel has 24 parameters and a tied channel 16. Counts therefore match exactly for the even base widths in Table 9; odd-width rounding is approximate. Inputs follow N(0,I8)\mathcal{N}(0, I_{8}), with 4,000 training and 4,000 held-out samples. L-BFGS runs for up to 6,000 iterations. The study contains 3×7×3×3=1893 \times7 \times3 \times3 = 189 fits; the table reports median relative held-out MSE over three seeds at the retained representative widths. Recovery uses a descriptive threshold of 10−310^{-3}.

Table 9: Trained-block distillation. Ratios are tied/untied errors; values above one favor untied. Ratios are retained at their reported precision and can differ slightly from ratios of rounded losses.

TeachermmUntied mmTied mmRatioTied 1.5m1.5mRatio
Untied44.02×10−14.02 \times10^{-1}3.96×10−13.96 \times10^{-1}0.992.29×10−12.29 \times10^{-1}0.57
Untied81.03×10−11.03 \times10^{-1}1.16×10−11.16 \times10^{-1}1.138.70×10−48.70 \times10^{-4}0.01
Untied101.37×10−41.37 \times10^{-4}7.51×10−27.51 \times10^{-2}5491.09×10−31.09 \times10^{-3}7.94
Untied128.11×10−68.11 \times10^{-6}8.70×10−48.70 \times10^{-4}1076.11×10−46.11 \times10^{-4}75.3
Tied82.82×10−12.82 \times10^{-1}2.99×10−12.99 \times10^{-1}1.061.64×10−11.64 \times10^{-1}0.58
Tied129.30×10−29.30 \times10^{-2}1.64×10−11.64 \times10^{-1}1.766.47×10−26.47 \times10^{-2}0.70
Tanh83.45×10−13.45 \times10^{-1}3.44×10−13.44 \times10^{-1}1.002.51×10−12.51 \times10^{-1}0.73
Tanh121.81×10−11.81 \times10^{-1}2.51×10−12.51 \times10^{-1}1.381.53×10−11.53 \times10^{-1}0.84

Table 9.

On the untied teacher, the first equal-width median recovery is at width ten for untied students and twelve for tied students. Under parameter matching, tied width twelve already reaches the criterion at base width eight. Untied students achieve lower errors once they enter their recovery regime, but the tied family also recovers the target. Across all widths, geometric-mean tied/untied ratios are 1.12 at equal width and 0.67 under matching for the tied teacher, and 1.06 and 0.78 for the tanh teacher. These controls do not show a consistent untied advantage.

At width ten on the untied teacher, one of three untied seeds fails to recover, with losses 1.4×10−41.4 \times10^{-4}, 1.2×10−51.2 \times10^{-5}, and 7.5×10−27.5 \times10^{-2}; all three recover at width twelve. A separate six-restart diagnostic at untied width eight obtains best loss 2.75×10−32.75 \times10^{-3}, with other losses 0.07–0.15. The teacher parameters themselves give an exact width-eight representation, so these residuals are optimization failures. This experiment tests recovery and precision for imperfect trained teachers; it does not establish independent architectural width lower bounds.

Norm continuation. For tied SwiGLU width twelve fitted to the untied width-eight teacher, the best of three paths with a relaxing normal penalty reaches held-out relative MSE 3.0×10−43.0 \times10^{-4} at squared parameter norm ∥θ∥2≈180\lVert\theta\rVert^{2} \approx180, approximately five times its initial value. One path stalls near 7×10−27 \times10^{-2}. No divergence is observed over the tested range. The result concerns finite-error approximation under a fixed parameterization; it proves neither closedness nor exact representability and does not bound each Taylor coefficient separately. This raw squared parameter norm is distinct from the homogeneous atom sizes used in the coefficient-space diagnostics.

Adam-trained gated scalar regression

We additionally compare tied and untied SiLU-gated scalar blocks at d=8d = 8 on two four-channel teachers. The asymmetric teacher is ∑jajSiLU⁡(pjTx)(qjTx)\sum_j a_j \operatorname{SiLU}(p_j^{\mathsf{T}}x)(q_j^{\mathsf{T}}x) with independent standard Gaussian directions pjp_j, qjq_j and weights aja_j; the tied teacher sets qj=pjq_j = p_j. Here the tied student is the matched-gate specialization SiLU⁡(u⊤x)(u⊤x)\operatorname{SiLU}(u^{\top}x)(u^{\top}x). Training uses 20,000 examples, 5,000 held-out examples, Adam with learning rate 3×10−33 \times10^{-3}, batch size 256, and a cosine schedule for 4,000 steps, and four seeds. Table 10 reports median relative test MSE.

Parameters Untied/tiedAsymmetric teacher UntiedAsymmetric teacher TiedTied teacher UntiedTied teacher Tied
17/180.31620.25800.38080.1899
34/360.08430.10490.01780.0014
51/540.03220.03870.00450.0013
68/720.01880.03700.00100.0006
102/1080.00880.00520.00090.0006
136/1440.00620.00240.00080.0004

Table 10. Adam-trained scalar regression with approximately matched total parameters. Counts are untied/tied; the tied models have slightly larger budgets. Bold marks the smaller error within each teacher and row.

The untied student has lower error on the asymmetric teacher at the intermediate budgets of 34–68 parameters, whereas the tied student is more accurate at the smallest budget and at the two largest budgets. On the tied teacher, the tied student is more accurate at every budget. Under approximately matched total parameters, the untied advantage is therefore limited to a range of budgets on the asymmetric teacher.

References

  1. [1]Hirotachi Abo and Nick Vannieuwenhoven. Most secant varieties of tangential varieties to Veronese varieties are nondefective. Transactions of the American Mathematical Society, 370(1):393–420, 2018. URL https://arxiv.org/abs/1510.02029.
  2. [2]James Alexander and André Hirschowitz. Polynomial interpolation in several variables. Journal of Algebraic Geometry, 4(2):201–222, 1995.
  3. [3]Enrico Carlini, Maria Virginia Catalisano, and Anthony V Geramita. The solution to Waring’s problem for monomials. arXiv preprint arXiv:1110.0745, 2011.arxiv.org/abs/1110.0745
  4. [4]Enrico Carlini, Maria Virginia Catalisano, and Anthony V. Geramita. The solution to the Waring problem for monomials and the sum of coprime monomials. Journal of Algebra, 370:5–14, 2012. doi: 10.1016/j.jalgebra.2012.07.028.DOI
  5. [5]Enrico Carlini, Mario Kummer, Alessandro Oneto, and Emanuele Ventura. On the real rank of monomials. Mathematische Zeitschrift, 286(1):571–577, 2017.
  6. [6]Grigorios G Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Yannis Panagakis, Jiankang Deng, and Stefanos Zafeiriou. Π-nets: Deep polynomial neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7325–7335, 2020.DOI
  7. [7]Grigorios G Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Jiankang Deng, Yannis Panagakis, and Stefanos Zafeiriou. Deep polynomial neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4021–4034, 2022.DOI
  8. [8]Gonzalo Comas and Malena Seiguer. On the rank of a binary form. Foundations of Computational Mathematics, 11(1):65–78, 2011. doi: 10.1007/s10208-010-9077-x.DOI
  9. [9]Pierre Comon, Gene Golub, Lek-Heng Lim, and Bernard Mourrain. Symmetric tensors and symmetric tensor rank. SIAM Journal on Matrix Analysis and Applications, 30(3):1254–1279, 2008.
  10. [10]Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International Conference on Machine Learning, pages 933–941. PMLR, 2017.arxiv.org/abs/1612.08083
  11. [11]Vin de Silva and Lek-Heng Lim. Tensor rank and the ill-posedness of the best low-rank approximation problem. SIAM Journal on Matrix Analysis and Applications, 30(3):1084–1127, 2008. URL https://arxiv.org/abs/math/0607647.
  12. [12]Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, 2021.
  13. [13]Joe Kileel, Matthew Trager, and Joan Bruna. On the expressive power of deep polynomial neural networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
  14. [14]Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM Review, 51(3):455–500, 2009.
  15. [15]Michael T Pearce, Thomas Dooms, Alice Rigg, Jose M Oramas, and Lee Sharkey. Bilinear MLPs enable weight-based mechanistic interpretability. In International Conference on Learning Representations, 2025.
  16. [16]Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.arxiv.org/abs/1710.05941
  17. [17]Noam Shazeer. GLU variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.arxiv.org/abs/2002.05202
  18. [18]David So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le. Searching for efficient transformers for language modeling. In Advances in Neural Information Processing Systems, volume 34, pages 6010–6022, 2021.
  19. [19]Nick Vannieuwenhoven. A condition number for the tensor rank decomposition. Linear Algebra and its Applications, 535:35–86, 2017. URL https://arxiv.org/abs/1604.00052.
  20. [20]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.

Paper details

vlnc:W_a75pjshw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents