In this paper
AbstractIntroductionRelated WorkThe CP-MLP frameworkDegree-wise atomsEffective atoms after symmetrizationExpressivity via CP-MLP atom rankAtom-rank definitionsDegree-two bilinear atomsHigher-degree monomial separationExperimentsIndefinite quadratic targetsRole-asymmetric monomial targetsPure powers versus role-asymmetric targetsSeparation in gated MLPsConclusionAcknowledgmentsPreliminariesTensors and CP decompositionsAdditional framework detailsSymmetrization detailsExpressivity proofsScalar-output reductionBilinear atom-rank proofsExact rank, border rank, and bounded approximationClosed quadratic classesUniversal comparison and independent interaction blocksNecessary width bounds for analytic channelsExperimental detailsComputational requirementsData generationModel implementationsOptimization protocolPer-experiment configurationsConvergence criterionScope of the role-asymmetric separationMatched input-projection control in gated MLPsBorder-rank diagnosticsRecovery of known representationsDense random targets and mixtures of degreesDistillation of trained gated blocksAdam-trained gated scalar regressionReferencesWorksComputer Science
CP-MLPs: A Tensor-Rank Theory of Tied and Untied MLP Blocks
Abstract
Transformer feed-forward blocks combine several design choices: squared activations, gating, and tied versus untied parameter sharing, whose separate contributions to expressivity remain poorly understood. We introduce CP-MLPs, a tensor framework in which each multiplicative unit contributes a rank-1 interaction with three roles: detecting an input feature, reading a value, and writing to the output. Tied gated blocks such as ReLU force the detector and value roles to share the same direction; untied gated blocks such as ReGLU and SwiGLU let them differ. We show that this distinction yields exact, degree-wise tensor-rank separations: untied detector-value units are strictly more width-efficient for indefinite quadratic interactions and role-asymmetric monomials, while tied units are structurally matched to symmetric pure-power targets. The width advantage of untying therefore depends on the target structure. Experiments show loss plateaus below the exact-representation width, but these plateaus alone do not establish a positive approximation gap: arbitrarily accurate approximation is governed by border rank. In trained gated MLPs, untied blocks retain an advantage in selected comparisons at matched input-projection budget. Code is available at https://github.com/rarefin/CP-MLPs.
Introduction
Feed-forward layers account for roughly two-thirds of the parameters in a transformer [20] language model [12], and a wide range of designs have been proposed for them. Common variants include squared activations such as [18], gated blocks such as ReGLU and SwiGLU [17], and tied variants of SwiGLU. Bilinear MLPs are less common as training blocks in modern LLMs, but they provide a clean quadratic base case and have received attention in the interpretability literature [15]. These architectures differ not only in their gating nonlinearity, but also in whether the same direction is used both to detect a feature and to read out its value. In this work we ask when untying these two roles improves expressivity, in the sense of reducing the number of channels required to represent a given interaction.
and tied SwiGLU use the same direction for the detector and value. Untied blocks such as bilinear MLPs, ReGLU, and SwiGLU allow the two to differ. This decouples two design choices that are typically conflated: the choice of gate, and whether the detector and value branches are tied.
The comparison is naturally made after input symmetrization. A polynomial function evaluates its coefficient tensor on repeated copies of the same input, so only the input-symmetric part of the tensor contributes to the function it represents [9]. Tied and untied blocks therefore parameterize the same space of input-symmetric tensors, but through different families of atoms.
Untying helps when the target interaction is better matched to an untied detector-value atom than to a tied pure-power atom.
Our analysis gives exact width separations at fixed polynomial degree. At degree two, untied bilinear atoms can represent certain indefinite quadratic interactions with strictly fewer channels than tied atoms. At higher degree, role-asymmetric monomials are realized by a single untied detector-value atom but require multiple tied pure-power atoms. For smooth gates such as SwiGLU, these statements transfer degree by degree to the Taylor coefficient tensors. These degree-wise results give necessary, but not generally sufficient, width bounds for the full analytic function. The point is not that tying is uniformly worse: tied atoms are well suited to symmetric pure-power targets, and untied atoms to interactions that separate detection from value.
Our coefficient-space experiments distinguish exact rank, border rank, and numerical recovery (Appendices D.9–D.11). A positive approximation gap requires width below border rank, the minimum width permitting arbitrarily accurate approximation. Below exact rank but at or above border rank, the error infimum is zero despite possible numerical loss plateaus. Both families represent pure-power targets at width one, so differences there concern optimization. The untied advantage also appears in selected trained gated-MLP comparisons under parameter matching.
Contributions.
A unified CP-MLP framework. We express bilinear, ReLU, tied SwiGLU, ReGLU, and SwiGLU blocks in a shared atom-based tensor notation with detector, value, and output roles.
Exact atom-rank separations. We prove that untied detector-value atoms achieve strictly lower width on indefinite quadratic interactions and role-asymmetric monomials, while tied atoms remain well matched to pure-power targets.
Empirical link between exact rank, border rank, and loss. We distinguish positive approximation gaps from zero-infimum limits and optimization failures, and evaluate tied and untied gated MLPs under width and parameter-budget controls.
Related Work
Transformer MLP variants and interpretations. Transformer feed-forward designs have diversified considerably. GLU-style activations were introduced for language modeling by Dauphin et al. [ [10], and variants such as ReGLU and SwiGLU were later shown to improve Transformer feed-forward layers [17]. Squared-ReLU activations emerged as effective Transformer primitives through architecture-search studies [18], building on a longer line of activation-function design that includes Swish/SiLU [16]. In parallel, interpretability work has argued that Transformer feed-forward layers behave like structured key-value memories rather than generic nonlinear post-processing [12]. These works motivate studying MLP blocks as structured computational modules, but do not directly isolate the expressivity effect of tying versus untying detector and value branches, which is the focus of this paper.
Tensor, polynomial, and Waring-rank views. Our framework uses canonical polyadic (CP) tensor decomposition, where tensors are written as sums of rank-1 outer products [14]. Recent work on bilinear MLPs also exploits tensor structure to analyze feed-forward blocks, observing that bilinear layers admit a third-order tensor representation [15]. The TT-Net family of Chrysos et al. [ [6, 7] parameterizes polynomial expansions of the input directly through tensor decompositions, demonstrating that such polynomial blocks are competitive with standard activations on a range of tasks. Polynomial networks have also been studied with algebraic and tensor tools, for example through the geometry of polynomial function classes [13]. Our work differs in that we do not propose polynomial networks as an architecture; we use polynomial structure as an analytical lens for comparing existing MLP variants.
For the higher-degree separation we rely on classical Waring rank: a scalar tied degree- channel is a pure power , so tied atom rank coincides with Waring rank. The monomial Waring-rank formula of Carlini et al. [ [3], together with the real-rank result of Carlini et al. [ [5], implies that requires tied atoms but only one untied detector-value atom. This exact-rank separation differs from the approximation threshold: the same monomial has tied border rank two [Comas and and Seiguer, 2011]. Our contribution is to connect these rank facts to practical MLP atom families, showing that tied and untied blocks draw on different atoms inside the same input-symmetric tensor space.
The CP-MLP framework
We use a common tensor notation for the multiplicative feed-forward blocks studied in this paper. Bilinear blocks multiply two linear projections, GLU-style blocks gate one projection by another, and square-style blocks reuse the same projection twice. In every case, each hidden channel produces a product of projected input directions, and the output matrix sums these channel-wise products. At each active degree, the raw coefficient tensor has the structure of a canonical polyadic (CP) decomposition [14], and we call the resulting representation a CP-MLP.
The framework separates two design choices: the gate, which selects the polynomial degrees that appear in a local expansion, and the tying pattern, which selects whether the detector and value directions of a channel are shared. The expressivity analysis in § 4 isolates the effect of the second choice.
Definition 3.1 (CP-MLP block). Fix a scalar gate and let be its coordinatewise extension, . Given weights (rows ) and (columns ), a width- CP-MLP block is
We call the detector direction, the value direction, and the output direction of channel . A tied block constrains ; an untied block parameterizes and separately, allowing equality as a special case. We consider bias-free blocks throughout.
The detector direction selects the scalar feature passed through the gate, while the value direction supplies the linear signal that the gate modulates. Tying forces both roles onto the same direction; untying lets them differ. Figure 1 shows the corresponding three-leg view of a single channel.

Figure 1. Leg diagram of a single CP-MLP channel. The channel computes and writes the result through . At each active polynomial degree, the channel contributes one rank-1 CP atom with output, detector, and value legs.
Degree-wise atoms
Assume the gate is analytic at the origin, (piecewise-analytic gates such as ReLU are treated region-wise; see Table 1). Substituting and multiplying by , one channel contributes ; reindexing by total degree , the degree- term carries the coefficient . Writing for the contraction of the input modes of against (output mode free; conventions in Appendix § 8.§ 1), the degree- component of the block is , with raw coefficient tensor
| Block | Gate | Tying | Signature | Effective atom |
| Bilinear [15] | untied | |||
| ReGLU [17] | untied | {sup}†` | region-wise | |
| SwiGLU [17] | untied | |||
| ReLU2 [18] | tied | {sup}†` | region-wise | |
| Tied SwiGLU | tied |
†Region-wise: ReLU is piecewise linear, so the channel is a degree-2 polynomial inside each fixed activation region.
Table 1. Feed-forward blocks as CP-MLPs. The gate controls which degrees are active through its polynomial signature; the tying pattern controls whether detector and value directions are shared.
Each active degree thus admits a CP decomposition with at most rank-one terms, one per channel. We define for the polynomial signature of the gate, i.e. the set of active degrees.
Definition 3.2 (Polynomial signature). The polynomial signature of is .
The gate selects which degrees are active through ; the tying pattern selects the atom used at each active degree. When , the raw degree- atom collapses to ; when is independent of , it retains the detector-value form .
Effective atoms after symmetrization
The tensor above distinguishes detector and value slots, but the function it represents evaluates the tensor on repeated copies of the same input, so input-slot order is not observable. We write for the symmetrization of , the average of over all permutations of its input modes (with the output mode, when present, left fixed), then
The proof is immediate: every permutation of receives the same vector in every input slot. A full statement, together with the quadratic matrix specialization, is given in Appendix § 9.§ 1.
After symmetrization, each architecture yields a family of input-symmetric atoms; an untied atom can have tensor rank greater than one.
Definition 3.3 (Effective tied and untied CP-MLP atoms). The degree- effective untied atom is
where denotes the space of symmetric order- tensors (Definition B.1). The degree- effective tied atom is the specialization :
Tied and untied blocks therefore parameterize tensors in the same symmetric space , but draw on different atom families: tied blocks use pure-power atoms , while untied blocks use symmetrized detector-value atoms . Setting recovers the tied family as a special case. The benefit of untying is not that antisymmetric components become visible, but that the symmetrized untied family can represent certain symmetric tensors with strictly fewer atoms.
Table 1 places the blocks studied in this paper along the two axes of the framework: the gate and the tying pattern. Bilinear, ReGLU, and SwiGLU differ only in their gates and share the untied detector-value atom structure. ReLU2 and tied SwiGLU use tied atoms, with the same direction serving both the detector and value roles. Tied SwiGLU sets while retaining the SiLU gate, giving .
The next section asks how many atoms of each family are required to represent a given target tensor.
Expressivity via CP-MLP atom rank
The atom families of § 3.2 suggest a natural notion of width: the smallest number of atoms needed to express a given coefficient tensor. Fixing an active degree , each CP-MLP channel contributes one effective atom in , so the question we want to answer is how many such atoms are required to represent a target . We call this minimum the CP-MLP atom rank; since each atom corresponds to one channel at degree , it is the degree-wise width required to represent a coefficient tensor.
For polynomial gates, is the ordinary degree- coefficient tensor; for analytic gates, it is the degree- Taylor coefficient tensor, satisfying . Throughout this section denotes an active degree of the gate, and any nonzero scalar Taylor coefficient is absorbed into the output factor , so the rank comparisons below depend only on the channel atom family.
Atom-rank definitions
Definition 4.1 (Tied CP-MLP atom rank). For , the tied CP-MLP atom rank is the smallest integer such that
Definition 4.2 (Untied CP-MLP atom rank). For , the untied CP-MLP atom rank is the smallest integer such that
Both ranks are minimum-width quantities at a fixed degree, counting the fewest atoms required to express a coefficient tensor in the corresponding family. Since a tied atom is recovered from an untied atom by setting , the untied family contains the tied family, which immediately implies the following inequality.
Proposition 4.3 (Untied atoms contain tied atoms). For every , .
Proof. Any tied representation is also an untied representation with , since . Minimizing over tied representations yields the claim.
The remainder of the section asks when this inequality is strict. We first study degree two, where the coefficient tensor is a symmetric matrix, and then prove a higher-degree separation using a role-asymmetric monomial.
Remark 4.4 (Scalar targets suffice). For a scalar target , embedding gives and : output projection cannot increase atom rank, and any scalar representation lifts by tensoring with . Full details are in Appendix § 10.§ 1. The separations below are stated for scalar targets.
Degree-two bilinear atoms
We first specialize to degree 2, where a scalar coefficient tensor is a symmetric matrix and the atom-rank question reduces to representing quadratic forms
A tied atom contributes the signed square , corresponding to the rank-1 symmetric matrix . An untied atom contributes a product of two linear forms, , which after symmetrization becomes
A tied atom therefore contributes one quadratic direction, while an untied atom can couple two. Our first result records this coarse rank comparison.
Theorem 4.5 (Coarse bilinear rank bounds). Let be symmetric. If is represented by a tied bilinear block of width , then . If is represented by an untied bilinear block of width , then . Both bounds are tight.
Proof sketch. Each tied unit contributes the rank-1 symmetric matrix , so a width- tied block produces matrices of rank at most . For untied blocks, only the symmetric part of contributes; the resulting matrix has image in and therefore rank at most two, and summing such terms gives the bound. Both bounds are tight by choosing independent coordinate directions.
Theorem 4.5 shows that an untied atom can carry twice the rank of a tied atom, but rank alone does not determine which matrices a given width can represent. The rank-2 gain has a specific sign structure. From the difference-of-squares identity
a non-collinear untied atom is a difference of two squares. It can pair one positive with one negative quadratic direction—representing interactions like —but cannot compress two same-sign directions such as into a single atom. The exact width is therefore controlled not by rank, but by how many positive and negative quadratic directions the target carries.
Let and denote the numbers of positive and negative eigenvalues of .
Theorem 4.6 (Exact bilinear atom ranks). For every symmetric matrix ,
whereas
Proof sketch. For tied atoms, each channel contributes one signed rank-1 square, so it needs one atom per nonzero eigendirection, and the spectral decomposition achieves this number. For untied atoms, the difference-of-squares identity expresses any width- representation as with . On , , so cannot be positive definite on a subspace of dimension greater than ; the analogous argument on gives the same bound for negative directions, so . The matching upper bound is achieved by pairing opposite-sign eigendirections one atom at a time, with unpaired directions handled by the tied special cases . Full proofs of Theorems 4.5 and 4.6 are in Appendix § 10.§ 2.
Corollary 4.7 (When untied bilinear atoms help). If or ,
If is indefinite,
Untied atoms are therefore most width-efficient on indefinite quadratic interactions; they offer no advantage on purely same-sign quadratic structure.
Connection to hard-gated blocks. The same degree-two atom gap appears regionwise in hard-gated CP-MLPs. On a fixed activation pattern , ReGLU realizes
for each output coordinate , whereas ReLU realizes
Inside each fixed region, ReGLU has access to untied rank-2 atoms with opposite-sign structure, while ReLU is restricted to tied rank-1 atoms. Region switching adds further piecewise expressivity; Theorem 4.6 isolates the within-region atom difference.
Higher-degree monomial separation
The degree-two analysis showed that untied atoms can be more width-efficient when the target contains opposite-sign quadratic structure. At higher degree, a different kind of separation emerges that is driven not by sign, but by role asymmetry: one direction is repeated many times in the detector slot, while another direction appears only once in the value slot.
For , consider the binary monomial . This target has the exact shape of a single untied atom: taking and gives , so one untied atom suffices. A tied degree- atom, by contrast, has the pure-power form , so the tied atom rank of is exactly its Waring rank.
Definition 4.8 (Waring rank). Let , and let be a homogeneous polynomial of degree . The Waring rank of over , denoted , is the smallest integer such that
where and each is a linear form over .
For scalar tied CP-MLPs, the real Waring rank coincides with the tied atom rank: any tied representation with is exactly a real Waring decomposition, so for every scalar homogeneous degree- polynomial. This is the standard correspondence between symmetric tensor rank and polynomial Waring rank [9]. The following classical formula computes the right-hand side for monomials.
Proposition 4.9 (Waring rank of monomials; Carlini et al., [3, 4, 5]). Let be a monomial of degree , with sorted exponents . Then
and whenever .
The complex formula is Carlini et al., [4]; the real-complex equality for follows from Carlini et al. [ [5]. For the sorted exponent vector is , so
Theorem 4.10 (Degree- monomial separation). For every ,
A single untied atom therefore represents a degree- coefficient tensor that requires tied atoms.
Proof. We bound the two ranks separately.
Untied rank. Setting and , , so a single untied atom realizes and . Since , it cannot be represented with zero atoms, giving .
Tied rank. Any scalar tied degree- representation is a real Waring decomposition, so by Proposition 4.9.
Exact representation versus approximation. The real border rank is the smallest width permitting arbitrarily accurate approximation of by real atoms from family . In coefficient norm, a positive approximation gap occurs exactly when . For scalar quadratics, both width classes are closed, so exact and border ranks coincide (Appendix § 10.§ 4). For , however, , whereas . Thus, at , the tied error infimum is zero but unattained. Under any finite bound on , a positive tied approximation gap remains below the exact rank. Appendix § 10.§ 3 gives the approximation arguments; Appendix § 10.§ 5 extends the separation to sums of independent interaction blocks.
The degree-wise expansion gives necessary width bounds for exact representation of the full analytic function. If a bias-free width- block with gate represents an analytic target near the origin, then
where is the degree- homogeneous component. Exact representation also requires and vanishing components at inactive degrees. These conditions are not generally sufficient: all degrees share the same channels, and the gate fixes their coefficient ratios. Appendix § 10.§ 6 gives the proof and an explicit example where the full required width exceeds every individual degree-wise rank.
Experiments
We test whether the atom-rank separations from § 4 appear as observable width and loss effects. Indefinite quadratics realize the degree-two gap (§ 5.1); role-asymmetric monomials realize the higher-degree gap (§ 5.2); pure-power targets show that tying can be the better match (§ 5.3); and gated MLPs show how the separation carries over to actual gates, including under a matched input-projection budget (§ 5.4). Optimization, hyperparameters, and budget checks are in Appendix § 11.
Indefinite quadratic targets
Theorem 4.6 predicts that untied bilinear atoms are more width-efficient than tied atoms only when the target carries both positive and negative directions. To test this, we construct
with , and sweep six choices of covering definite, balanced indefinite, and imbalanced indefinite targets.
Figure 2 confirms the exact threshold: tied blocks require width , untied blocks require width . At one width below the prediction, the loss stays near the target variance; at the predicted width, the loss drops to machine precision. Definite targets show no tied/untied gap, while indefinite targets show the predicted gap .

Figure 2. Quadratic targets with positive and negative directions. Left: required widths match the predictions (tied) and (untied). Right: loss stays near the target variance below the predicted width and reaches machine precision at the predicted width.
Role-asymmetric monomial targets
We next test the higher-degree separation of Theorem 4.10. For
the theory predicts untied atom rank and tied atom rank . We fit the target directly in monomial coefficient space and sweep width ; coefficient-space fitting removes sampling noise and isolates the rank effect.
Figure 3 shows the predicted separation at width one: untied atoms reach machine precision at for every degree, while tied atoms keep a clearly positive loss. For , the tied losses are small but nonzero (about to , with one optimizer outlier at , ). They drop to machine precision at for and at for ; for they stay between about and across the tested widths. Below its atom rank the target lies outside the tied model class, but for it lies in the class’s closure: the border Waring rank of is two [8], via the limit in Equation (1) with and . The plateau therefore reflects the ill-conditioning of border-rank approximations, whose residuals diverge as the residual shrinks, rather than a positive distance to the model class; at the floor is genuine, since the set of symmetric rank-at-most-one tensors is closed. The untied advantage is specific to single-value-role targets; products with multiple independent value roles require more atoms (Appendix § 11.7).
Figure 3. Coefficient-space width sweep for , . Untied atoms fit at width . Tied atoms keep a positive loss at ; for their losses are small but nonzero, as expected from border rank two, and they reach machine precision at for . Median over two seeds; bands show min–max.
Pure powers versus role-asymmetric targets
The previous experiment shows a strong untied advantage on role-asymmetric monomials, but tying is not always worse. At width , we compare tied and untied models on two targets:
The first is a pure power and is realized exactly by one tied atom; the second is role-asymmetric and is realized exactly by one untied atom. Figure 4 shows the expected reversal: on pure powers, tied models reach machine precision while untied models must align the redundant value direction with the detector; on role-asymmetric targets, untied models fit at width one while tied models cannot. The low-width winner therefore depends on target structure.
Figure 4. Target structure changes the low-width winner. At , tied atoms fit pure powers , while untied atoms fit role-asymmetric monomials . Bands show min and max over seeds.
Separation in gated MLPs
The coefficient-space experiments above isolate the atom-rank effect; we now test actual gated MLPs on the same witness . We compare tied ReLU, tied SwiGLU (SiLU()), untied ReGLU, and untied SwiGLU, using Gaussian inputs with , , and widths .
Figure 5 shows that the exact polynomial threshold becomes a stable qualitative ordering for gated networks. At , untied ReGLU and SwiGLU reach near-machine precision at , and tied ReLU at . These widths are twice the atom ranks 1 and 2. Both gates satisfy and , so each polynomial atom can be built from two gated channels. By the second identity, tied SwiGLU can also represent exactly at . Its fits here stay near , whereas the same model reaches at width four in the matched control of Appendix § 11.8, so this gap is an optimization failure. At , no architecture reaches machine precision. ReLU-gated channels are positively homogeneous of degree two and cannot represent a cubic; a SiLU channel that matches the cubic term also contributes quadratic and higher-degree terms, which finite width cannot cancel exactly. Untied SwiGLU is the best-performing model, and for each gate the untied block lies below its tied counterpart at widths .
A control comparing untied width against tied width , which matches input-projection parameters, gives a degree-dependent picture: at , tied SwiGLU at width four and untied SwiGLU at width two reach comparable losses, while at untied SwiGLU at width four is about ten times more accurate than tied SwiGLU at width eight. The figure and numbers are in Appendix § 11.8.
Figure 5. Width versus loss for actual gates on . Left: for , untied ReGLU and SwiGLU reach near-machine precision at smaller width than tied ReLU² and tied SwiGLU. Right: for , the exact rank threshold becomes a qualitative ordering; untied SwiGLU is best, and each untied block stays below its same-gate tied counterpart for . Median over three seeds; bands show min–max.
Conclusion
We introduced CP-MLPs, a tensor framework for comparing tied and untied multiplicative feed-forward blocks. The framework separates the gate from the choice of whether detector and value directions are shared. Our theory shows that this choice changes width efficiency. Untied atoms can represent indefinite quadratic interactions and role-asymmetric monomials with fewer channels, while tied atoms are naturally matched to pure-power targets. For smooth gates such as SwiGLU, these statements apply degree by degree to Taylor coefficient tensors. Experiments confirm the predicted quadratic width thresholds and the width-one untied fits, show tied losses below the exact monomial rank that are consistent with border rank, and demonstrate that the low-width advantage changes with target structure.
Selected gated-MLP comparisons retain an untied advantage under input-projection matching, while others attain comparable errors. These results suggest that the success of GLU-style feed-forward blocks is not only an activation effect: separating detector and value directions changes which interactions are width-efficient. Several open directions remain. Our higher-degree separation is degree-wise rather than a global statement for the full analytic channel, and our experiments use synthetic polynomial targets rather than language-modeling tasks; bridging both gaps, by extending the theory to richer multi-leg atoms and by testing these effects in full Transformer language models, is a natural next step.
Acknowledgments
We acknowledge the support from the Canada CIFAR AI Chair Program and from the Canada Excellence Research Chairs Program. The research was enabled in part by computational resources provided by the Digital Research Alliance of Canada and Mila-Quebec AI Institute.
Preliminaries
This appendix collects the tensor and CP-decomposition notation used throughout the paper.
Tensors and CP decompositions
Let be an order- tensor and let . We write the multilinear contraction of with vectors as
where denotes contraction along mode :
Equivalently,
When all modes are contracted with vectors, the result is a scalar. If some modes are left uncontracted, the same formula applies to the remaining modes only; in particular, contracting the input modes of while leaving the output mode free gives , the convention used in § 3.1. We write for the -th standard basis vector.
A canonical polyadic (CP) decomposition represents a tensor as a sum of rank-one outer products:
where and . Under contraction, each rank-one term factorizes:
Thus a CP decomposition expresses a multilinear map as a sum of separable products of linear forms.
This is the tensor structure used throughout the paper. For example, a single CP-MLP channel at degree contributes a rank-one raw tensor of the form
whose diagonal evaluation gives
A width- CP-MLP sums such channel-wise rank-one contributions.
Symmetrization. For any tensor , define
Symmetric CP rank. A symmetric tensor has symmetric rank if is the smallest integer such that
Matricization lower bound. For any tensor , the rank of any flattening provides a lower bound on its CP rank [14].
Additional framework details
Symmetrization details
Let denote the symmetric group on elements. For a permutation and a tensor , write for the tensor obtained by permuting the input modes according to , leaving the output mode fixed.
Definition B.1 (Input-symmetric tensor). A tensor
is input-symmetric if for every . The space of input-symmetric tensors is denoted
Definition B.2 (Input symmetrization). The input symmetrization of is
Lemma B.3 (Diagonal evaluation sees only the input-symmetric part). For every
and every ,
Proof. For any , all input slots of receive the same vector $x. Hence
Averaging over all gives the claim. □
Lemma B.4 (Quadratic diagonal evaluation). For every and every ,
Proof. Let
Then . Since is a scalar,
Therefore . The result follows by writing
□
Derivative tensors. For an analytic scalar function, the degree- Taylor coefficient tensor is equivalently the -th derivative tensor up to the factor . In particular, for a homogeneous tensor polynomial
we have
Thus the derivative tensor also records only the input-symmetric part of the raw coefficient tensor. This is the differential version of Lemma B.3.
Expressivity proofs
Scalar-output reduction
Lemma C.1 (Output projection cannot increase channel rank). For every and every ,
Proof. Suppose first that has a tied representation with channels:
Applying to the output mode gives
This is a scalar tied representation with at most atoms; atoms with simply drop out. Minimizing over tied representations of gives the first inequality.
The detector–value case is identical. If
then
which is a scalar detector–value representation with at most atoms.
For the special embedding of a scalar target into one output coordinate, the ranks are exactly preserved.
Corollary C.2 (Scalar separations embed into vector targets). Let , and let
Then
Consequently, any scalar tied–detector–value separation transfers to the vector-valued CP-MLP setting.
Proof. The lower bounds follow from Lemma C.1 by taking , since
Thus
For the reverse inequalities, take any scalar representation of and multiply each atom by the output basis vector . This gives a vector-valued representation of with the same number of channels. Hence the ranks are equal.
Bilinear atom-rank proofs
Proof of Theorem 4.5. We prove the tied and untied rank bounds separately.
Tied blocks. A tied bilinear block of width represents a matrix of the form
Each term has rank at most one. Therefore
Since , also
Thus a tied block satisfies
This bound is tight. Let . Choosing
gives
which has rank .
Untied blocks. An untied bilinear block of width represents a matrix of the form
For any ,
Therefore
Using subadditivity of rank,
Since ,
Hence
This bound is tight. Let .
If is even, choose
Then
This is a direct sum of independent off-diagonal blocks, each of rank two. Therefore .
If is odd, use the same off-diagonal blocks and add one diagonal atom by taking
This adds one independent rank-one block, so the resulting matrix has rank . Thus the untied rank bound is tight. □
Proof of Theorem 4.6. We prove the tied and untied atom ranks separately.
Tied atom rank. By Theorem 4.5, any tied representation of with atoms satisfies
Therefore
For the reverse inequality, use the spectral decomposition of the symmetric matrix :
This is a tied representation using one signed rank-one atom for each nonzero eigendirection. Hence
Combining the two inequalities gives
Untied atom rank: lower bound. An untied atom satisfies the identity
Therefore any width- untied representation can be written as
for some , after absorbing scalar signs and magnitudes into the columns of and .
We claim that this implies
Indeed, on ,
If , a subspace of dimension on which is positive definite would satisfy , contradicting the inequality above; hence . Similarly, on ,
so cannot be negative definite on a subspace of dimension larger than . Hence
Therefore every untied representation satisfies
So
Untied atom rank: upper bound. It remains to construct an untied representation with exactly atoms. Use the spectral decomposition
Pair positive and negative eigendirections as much as possible. For a paired block
define
Then
Thus one untied atom represents one positive and one negative eigendirection together.
If any positive eigendirections remain unpaired, represent each by the tied special case
If any negative eigendirections remain unpaired, represent each by
Therefore the total number of atoms used is
This proves
Combining the lower and upper bounds gives
Exact rank, border rank, and bounded approximation
Fix and . All parameters and closures are real. We use the coefficient-tensor Frobenius norm; equivalent norms give the same closure. Define
and
The error is positive for , zero but unattained for , and zero and attained for . A positive infimum alone does not guarantee a best approximation [11].
Coefficient sizes. For output , define
These sizes are invariant under factor rescalings that preserve the raw atom; for scalar output, . Let contain tensors with a representation using at most atoms and total size .
Proposition C.3 (Bounded compactness and forced size divergence). For finite , both classes are compact. Thus
exists, is positive when , and converges to as . Every fixed-width sequence converging to a target outside the exact class has .
Proof. Normalize the input directions and absorb their norms into the output factors, padding zero atoms as needed. The tied parameter set becomes
It is compact, as is its image under the continuous tensor-sum map. The untied case uses two unit directions per atom. Compactness gives the positive minimum outside the class. The budgeted classes increase to the unrestricted class, so their distances decrease to . If did not tend to infinity, a subsequence would have total size at most for some finite ; compactness would place its limit in the exact class, a contradiction.
Proposition C.4 (Separation under a common coefficient-size budget). If , there is a finite such that every permits exact untied width- representation, whereas . As , this tied distance has a positive limit if and tends to zero otherwise.
Proof. Take to be the total size of an exact untied representation. Apply Proposition C.3 and the definition of border rank to the tied class.
The monomial limit. For independent linear forms , ,
Hence the tied border rank is at most two. It is at least two because the first-derivative span of has dimension two, while a pure power has dimension at most one, and this matrix-rank bound survives closure. Thus [8]. For , binomial expansion gives error and , hence squared error along this path. This gives an upper bound; optimality is not established. For , the identity is exact for nonzero .
Closed quadratic classes
Proposition C.5 (Scalar quadratic closedness). For scalar degree-two targets and every width , both tied and untied classes are closed. Their exact and real border ranks coincide.
Proof. By Theorem 4.6, the tied class consists of symmetric matrices of rank at most , a closed condition defined by vanishing -minors. The untied class is
which is closed by continuity of ordered eigenvalues. Distance from a target outside either class is positive and attained: a minimizing sequence can be restricted to a bounded ball, whose intersection with the class is compact.
For and symmetric ,
A positive coefficient-norm gap therefore implies a positive optimal population MSE. An observed finite-sample residual can also reflect sampling and optimization. For , tied rank- spectral truncation gives relative squared Frobenius error for ; the untied error is at and zero at .
Universal comparison and independent interaction blocks
Proposition C.6 (Universal exact- and border-rank comparison). For every degree- coefficient tensor , with ,
Both upper constants are attained by .
Proof. Containment gives the lower bounds, including after closure. For independent , a change of coordinates maps to , which has a real -power decomposition by Theorem 4.10; collinear directions require at most one power. Applying these decompositions to each atom, with its output vector, proves the exact upper bound. Equation (1) puts each untied atom in the closure of the tied width-two class. Summing and taking closure gives the border upper bound. The ranks of establish sharpness.
Theorem C.7 (Independent interaction blocks). Let , , , and . For linearly independent real forms , define
Then
Proof. Use the independent forms as coordinates. The first derivatives span the distinct monomials and , including when . Each tied atom contributes at most one derivative direction and each untied atom at most two. Since these matrix-rank bounds survive closure,
One untied atom per summand gives ; Equation (1) gives ; and real -power decompositions give .
For , absorb into and apply the additivity theorem for pairwise coprime monomials [4 Theorem 3.2]: the complex Waring rank is , so the real rank is at least . Extra coordinates cannot lower rank, since restricting them to zero preserves a decomposition. For , each block has one positive and one negative eigenvalue, and Theorem 4.6 gives tied rank . ▫
Thus the tied approximation gap is positive at , while the untied family is exact. At , the tied infimum is zero but unattained.
Necessary width bounds for analytic channels
Proposition C.8 (Degree-wise necessity). Let be analytic near zero and let be represented near zero by a bias-free width- CP-MLP in family . Then , inactive Taylor components vanish, and
Proof. Equality near zero implies equality of Taylor coefficients. Each active degree is a sum of at most atoms, with its nonzero gate coefficient absorbed into the output factors; inactive components vanish. These conditions are necessary, but parameters shared across degrees prevent choosing the decompositions independently. ▫
Proposition C.9 (Degree-wise rank one with SiLU-channel width two). The function
has untied atom rank one at every active Taylor degree, but its exact untied SiLU-channel width is two.
Proof. The displayed representation uses two channels, and each active component is a nonzero multiple of . To exclude width one, expand
For , the coefficients of satisfy
when . For , they are , giving .
Any width-one representation matching the quadratic term must have detector and value forms proportional to in some order, by unique factorization. The cubic term fixes the detector along and the value along , so the invariant excludes all width-one representations. ▫
The obstruction is the gate’s fixed coefficient ratios across degrees. Small held-out function error alone does not bound every Taylor coefficient.
Experimental details
This section documents the experimental methodology in sufficient detail for an independent re-implementation. Code that reproduces every figure and table is available at https://github.com/rarefin/CP-MLPs.
Computational requirements
All experiments run on a standard CPU; no GPU or specialized hardware is required. The experiments of § 5 and Appendices § 11.7–§ 11.8 each complete in about one minute, about five minutes in total on a single CPU core. The additional experiments of Appendices § 11.9–§ 11.13 take about two to three hours in total on one core. Memory usage stays below 1 GB throughout.
The experiments use standard scientific Python: numpy for linear algebra, scipy.optimize.minimize (L-BFGS-B method) for nonlinear least-squares fitting, and matplotlib for figure generation. The experiments of Appendices § 11.10, § 11.11, and § 11.13 also use PyTorch on the CPU, for automatic differentiation and for Adam.
Data generation
The experiments fit two kinds of targets: polynomial coefficient tensors (in coefficient space) and scalar functions of Gaussian inputs (in function space). We describe both.
Coefficient-space targets. A degree- polynomial in variables corresponds to a coefficient vector in the basis of degree- monomials, where . For we use the basis indexed by the exponent of , so index corresponds to the monomial . The role-asymmetric witness of Theorem 4.10 therefore has a single nonzero entry at index ; the pure-power target has its single nonzero entry at .
The model’s prediction in coefficient space is the coefficient vector of its parameterized polynomial, computed analytically from the architecture’s atom form (Section D.3). The loss is the squared distance between the predicted coefficient vector and the target.
Function-space targets. For experiments using actual Gaussian inputs (§ 5.3 and § 5.4), we sample inputs with each row drawn i.i.d. from . The sample size is given in Table 2. Targets are evaluated pointwise: for , we set ; for the indefinite quadratic experiment, with . The loss is the mean squared error .
Table 2. Hyperparameters for each experiment.
Random seeds. Each experiment uses random seeds that are deterministic functions of configuration values. For example, the quadratic experiment uses seed = s * 137 + n_plus * 17 + n_minus for seed index and signature ; the width-sweep experiment uses seed = s * 37 + k * 11, shared across the widths of a sweep. Fixed seeds make every run reproducible with the same library versions.
Model implementations
Each model is parameterized by a flat parameter vector that the optimizer treats as the only variable. The total parameter count depends on the architecture and the width .
Function-space architectures. Given input and width , all architectures follow the CP-MLP structure of Definition 3.1: project inputs through one or two linear maps, apply an elementwise nonlinearity, multiply per channel, and combine via scalar output coefficients (the scalar specialization of the output direction ).
For tied polynomial atoms with detector matrix and coefficients :
For untied polynomial atoms with detector , value , and coefficients :
For the four gated architectures in § 5.4, replace the polynomial atoms with their gated analogues from Table 1:
with and . Tied gated architectures have parameter count ; untied gated architectures have . The matched-parameter control of § 5.4 pairs untied width against tied width , equalizing input-projection parameters at .
Coefficient-space implementations. For coefficient-space experiments we compute the coefficient vector of each parameterized polynomial directly, avoiding sampling. For tied atoms, the multinomial theorem gives
For untied detector-value atoms,
which expands into a degree- multi-index sum. The total predicted coefficient vector is times the per-atom coefficient vector, and the loss is the squared distance from the target.
Optimization protocol
The experiments of § 5 and Appendices § 11.7–§ 11.8 minimize the squared loss using scipy.optimize.minimize with method L-BFGS-B. The iteration limit and tolerances are: indefinite quadratics 3000, , ; width sweep 2000, , ; pure-vs-mixed 5000, , ; gated MLPs 1500, , ; matched control 1500, , ; scope boundary 800, , . Appendices § 11.9–§ 11.13 describe their own settings. For each random seed, we run multiple restarts with parameter initialization (applied independently to each entry of ). We record the best-of-restarts loss per seed and aggregate (median, min, max) across seeds.
Per-experiment configurations
Twelve restarts for the untied model at .
For coefficient-space experiments, fits operate directly on the monomial coefficients of the target polynomial, eliminating sampling and function-evaluation noise. For experiments with finite samples, inputs are drawn from once per seed and held fixed across restarts within that seed.
Convergence criterion
We report best-of-restarts MSE without applying a separate convergence threshold. A configuration is described as fitting the target if the residual reaches machine precision (MSE ) and as failing if the residual stays above . The intermediate regime is interpreted as a partial fit due to optimizer-landscape difficulty rather than a representational limitation, and is discussed for the case in § 5.2.
To distinguish the plateau of § 5.2 from a slow-optimization artifact, we re-ran the failed configurations with the L-BFGS iteration budget. The residual stayed within 10% of its plateau value across all configurations tested, confirming that the plateau is robust to optimization budget.
Scope of the role-asymmetric separation
Theorem 4.10 shows that one untied detector-value atom suffices for the role-asymmetric witness , where one direction acts as a repeated detector and one direction supplies the single value. This appendix tests the boundary of that statement: targets with multiple independent value roles do not reduce to a single untied atom.
Concretely, at we compare tied, untied, and a multi-leg oracle (the smallest atom family rich enough to express any product of linear forms) on the role-asymmetric witness and on the square-free product
in which each of the variables appears exactly once.
Figure 6 and Table 3 confirm three regimes. On , untied atoms and the multi-leg oracle both fit at . Tied atoms keep a positive loss at , reach small but nonzero losses at and (about and , consistent with border rank two), and reach about at . On , the multi-leg oracle fits at , untied requires , and tied requires , matching its Waring rank (by Proposition 4.9). The untied advantage is therefore specialized to targets with a single value role and does not extend to arbitrary products.
Figure 6. Scope boundary at . Untied atoms are rank one on the role-asymmetric witness but not on the square-free product ; the multi-leg oracle is rank one on both.
| Target | Architecture | ||||
| (role-asym.) | tied | ||||
| (role-asym.) | untied (det.-val.) | ||||
| (role-asym.) | multi-leg oracle | ||||
| (square-free) | tied | ||||
| (square-free) | untied (det.-val.) | ||||
| (square-free) | multi-leg oracle |
Table 3. Scope boundary at . Median best-of-restarts MSE over two seeds.
Matched input-projection control in gated MLPs
Untied architectures use two input projections per channel. We compare untied width with tied width , matching input-projection counts at . The tied model has additional scalar-output parameters.
Figure 7 shows that the ordering depends on degree and width. At , tied SwiGLU at width four and untied SwiGLU at width two attain comparable median losses, and , respectively. At , untied width four reaches , compared with for tied width eight, an approximately 10.8-fold difference. These fixed-budget fits show a selected untied advantage, not a uniform separation or minimum-width certificate.
Figure 7. Matched input-projection control on : untied width (dashed) versus tied width (solid). Tied SwiGLU uses SiLU(). Median over three seeds; bands show min–max. Total parameter counts are and , respectively.
Border-rank diagnostics
The additional coefficient-space experiments use the apolar norm. If , then
This is the Frobenius norm of the symmetric coefficient tensor and satisfies .
Reported losses are relative squared errors . Unlike the unweighted monomial-coordinate norm used in § 5.2, this convention accounts for multinomial multiplicities. The two norms have the same closure, but numerical losses need not agree.
For the size diagnostic, each configuration has 27 candidate fits: 12 unregularized initializations and five at each of three penalties on . We record the smallest loss and the smallest among candidates with loss at most 1.02 times that minimum. At the selected near-best candidate, we also report the intrinsic condition number for the normalized Terracini/Jacobian matrix after removing scaling directions [19]. A small singular value indicates sensitivity of decomposition parameters; it is not a certificate of global optimality. The notation “n.i.” means that local nonuniqueness makes this diagnostic uninformative.
Table 4 follows the three regimes for the original monomial. At tied width one, the positive floor is attained because the class is closed. At width two, the smallest near-best candidate has large size; this is consistent with the forced-divergence statement for any sequence whose error tends to zero. A finite collection of large candidates does not exclude a smaller-size candidate that the optimizer missed. At exact width three, a bounded near-zero fit is recovered.
| Family | Regime | Best loss | Best | Near-best | ||
| Untied | 1 | Exact control | 1.00 | 1.00 | 1.7 | |
| Tied | 1 | Positive infimum | 0.5556 | 0.385 | 0.385 | 1.5 |
| Tied | 2 | Zero, unattained | 83.17 | 83.17 | ||
| Tied | 3 | Exact width | 0.471 | 0.471 | n.i. |
Table 4. Coefficient-size diagnostics for . Exact and border ranks determine the regimes; losses and sizes describe the candidate fits.
For the independent-block target, Table 5 places the positive-to-zero infimum transition at width four, as predicted by Theorem C.7. The large best-fit size at width two does not contradict the positive gap: this candidate approximates one block through a degeneration while leaving the other unfitted, at loss 0.5. Its condition number is , whereas a candidate within 2% of the same loss has size 2.27. The theorem identifies the regime; the diagnostics describe how individual fits approach it.
| Family | Regime | Best loss | Best | Near-best | ||
| Untied | 2 | Exact control | 1.00 | 1.00 | 1.7 | |
| Tied | 2 | Positive infimum | 0.5000 | 46.95 | 2.27 | |
| Tied | 3 | Positive infimum | 0.2778 | 66.30 | 2.27 | |
| Tied | 4 | Zero, unattained | 47.74 | 47.736 | ||
| Tied | 5 | Zero, unattained | 34.40 | 34.405 | ||
| Tied | 6 | Exact width | 3.72 | 3.72 | n.i. |
Table 5. The same diagnostic for . Its tied border width is four and tied exact width is six.
As a closed-class control, tied fits to yield relative coefficient errors 0.7500, 0.5000, and 0.2500 at widths one, two, and three, matching . Penalty sweeps along the monomial degeneration give log–log slopes −4.003, −4.002, and −4.002 for squared error versus at , consistent with the constructive path in Equation (1). These slopes do not prove an optimal trade-off.
Recovery of known representations
To separate optimization from expressivity, we retain generating parameters for targets constructed from three atoms. These witnesses establish width-three representability, not minimal rank three. Fitting uses float64 L-BFGS-B with automatic differentiation, , and 60 restarts per configuration. Recovery means relative apolar loss below . Table 6 reports recovery fractions for the respective target constructions in each family; it should not be read as a comparison on identical targets.
| Tied width | Tied width | Tied width | Untied width | Untied width | Untied width | Best untied loss at width 3 | |
| 3 | 4 | 5 | 3 | 4 | 5 | ||
| 3 | 0.33 | 0.32 | 0.70 | 0.80 | 0.92 | 1.00 | |
| 4 | 0.42 | 0.67 | 0.75 | 0.17 | 0.65 | 0.88 | |
| 5 | 0.77 | 0.80 | 0.85 | 0.00 | 0.02 | 0.00 | |
| 6 | 0.08 | 0.20 | 0.22 | 0.00 | 0.00 | 0.00 |
Table 6. Recovery fractions for configurations with a width-three generating witness. Witness losses are approximately . Widths four and five pad the representation with additional channels.
No untied width-three restart reaches the tolerance at or , despite the generating witnesses. At , widening yields one recovery in 60 attempts at width four and none at width five. Thus the tested extra widths do not reliably resolve the optimization difficulty. In a separate structured example, has exact untied rank two, but eight restarts all return loss 0.25. A 40-restart run recovers the representation ten times, with best loss and median 0.25. These failures cannot be used as rank lower bounds.
Dense random targets and mixtures of degrees
For scalar homogeneous targets with variables and , let . Outside the known defective cases, complex-generic ranks are
| Target | Tied obs./pred. | Untied obs./pred. | Obs. ratio | Above prediction | |||
| 2 | 5 | 0 | 6 | 3/3 | 3/2 | 1.00 | Untied |
| 2 | 5 | 1 | 6 | 4/3 | 2/2 | 2.00 | Tied |
| 2 | 6 | 0 | 7 | 4/4 | 3/3 | 1.33 | — |
| 2 | 6 | 1 | 7 | 5/4 | 3/3 | 1.67 | Tied |
| 3 | 3 | 0 | 10 | 4/4 | 3/3 | 1.33 | — |
| 3 | 3 | 1 | 10 | 4/4 | 3/3 | 1.33 | — |
| 3 | 4 | 0 | 15 | 6/6 | 3/3 | 2.00 | — |
| 3 | 4 | 1 | 15 | 7/6 | 4/3 | 1.75 | Both |
| 4 | 3 | 0 | 20 | 6/5 | 4/4 | 1.50 | Tied |
| 4 | 3 | 1 | 20 | 6/5 | 4/4 | 1.50 | Tied |
| 4 | 4 | 0 | 35 | 11/10 | 6/5 | 1.83 | Both |
| 4 | 4 | 1 | 35 | 10/10 | 6/5 | 1.67 | Untied |
| 5 | 3 | 0 | 35 | 8/8 | 5/5 | 1.60 | — |
| 5 | 3 | 1 | 35 | 8/8 | 5/5 | 1.60 | — |
Table 7. Dense random targets: observed recovery widths and complex-generic predictions. “Above” marks recovery only above the prediction and does not certify a larger real rank.
by the Alexander–Hirschowitz theorem and the tangential-Veronese result [2, 1]. A defective case is one in which the dimension count underestimates the required generic width. For the tied family the count increases by one at ; for the untied family it increases by one at .
We sample coefficients independently from a Gaussian in an apolar-orthonormal basis and fix these predictions before fitting. The observed recovery width is the smallest tested width reaching relative loss below in 32 restarts. The complex lower threshold applies almost surely to these continuously sampled real targets; it need not be an attainable real width. A real near-zero fit at that width supplies numerical recovery evidence. A fit only at the next width leaves higher real rank, an ill-conditioned limit, and optimizer failure unresolved.
Table 7 recovers several nontrivial corrected counts. The ternary cubic requires an untied prediction of three, rather than the naive two. At , both predictions increase, to tied width eight and untied width five; both are recovered on two targets, with best losses at least one channel below and below at the predictions. For tied target zero, losses below and at width six are and . Observed tied/untied width ratios range from one to two; the factor- separation of the monomial is a structured example rather than a generic prediction.
Mixtures of degrees. We also fit independent random components at degrees , with shared directions but a free coefficient for each channel and degree. With , parameter counting gives the reference widths
These are dimension-count benchmarks, not proved generic-rank formulas for the shared-direction model. The relaxation also differs from a fixed analytic gate because its coefficients are free across degrees.
The untied recovered width is smaller in all seven comparisons in Table 8. The mixture with the largest coefficient dimension recovers both counting benchmarks. Recovery above a benchmark is unresolved and is not attributed to a specific real rank or optimization mechanism.
| Tied | Untied | |||
| 2 | 18 | 5 (4), 4 (4) | 4 (3), 3 (3) | |
| 3 | 16 | 5 (4), 4 (4) | 4 (3), 3 (3) | |
| 3 | 31 | 7 (7) | 5 (5) | |
| 4 | 30 | 7 (6), 7 (6) | 4 (4), 5 (4) |
Table 8. Exploratory degree-mixture fits. Each entry is recovered width (dimension-count benchmark). Paired entries correspond to two independently sampled targets.
Distillation of trained gated blocks
A frozen random tanh network with hidden width 64 supplies a common target. We train three teachers with 192 parameters each, excluding biases: untied SwiGLU width eight, tied SwiGLU width twelve, and a tanh MLP width twelve. Their training relative MSEs on the common target are 0.295, 0.172, and 0.112, respectively. Tied SwiGLU uses , preserving the untied model’s gate while imposing .
For each teacher and , we fit untied width , tied width , and approximately parameter-matched tied width (rounded when necessary). With input and output dimension eight, an untied channel has 24 parameters and a tied channel 16. Counts therefore match exactly for the even base widths in Table 9; odd-width rounding is approximate. Inputs follow , with 4,000 training and 4,000 held-out samples. L-BFGS runs for up to 6,000 iterations. The study contains fits; the table reports median relative held-out MSE over three seeds at the retained representative widths. Recovery uses a descriptive threshold of .
Table 9: Trained-block distillation. Ratios are tied/untied errors; values above one favor untied. Ratios are retained at their reported precision and can differ slightly from ratios of rounded losses.
| Teacher | Untied | Tied | Ratio | Tied | Ratio | |
| Untied | 4 | 0.99 | 0.57 | |||
| Untied | 8 | 1.13 | 0.01 | |||
| Untied | 10 | 549 | 7.94 | |||
| Untied | 12 | 107 | 75.3 | |||
| Tied | 8 | 1.06 | 0.58 | |||
| Tied | 12 | 1.76 | 0.70 | |||
| Tanh | 8 | 1.00 | 0.73 | |||
| Tanh | 12 | 1.38 | 0.84 |
Table 9.
On the untied teacher, the first equal-width median recovery is at width ten for untied students and twelve for tied students. Under parameter matching, tied width twelve already reaches the criterion at base width eight. Untied students achieve lower errors once they enter their recovery regime, but the tied family also recovers the target. Across all widths, geometric-mean tied/untied ratios are 1.12 at equal width and 0.67 under matching for the tied teacher, and 1.06 and 0.78 for the tanh teacher. These controls do not show a consistent untied advantage.
At width ten on the untied teacher, one of three untied seeds fails to recover, with losses , , and ; all three recover at width twelve. A separate six-restart diagnostic at untied width eight obtains best loss , with other losses 0.07–0.15. The teacher parameters themselves give an exact width-eight representation, so these residuals are optimization failures. This experiment tests recovery and precision for imperfect trained teachers; it does not establish independent architectural width lower bounds.
Norm continuation. For tied SwiGLU width twelve fitted to the untied width-eight teacher, the best of three paths with a relaxing normal penalty reaches held-out relative MSE at squared parameter norm , approximately five times its initial value. One path stalls near . No divergence is observed over the tested range. The result concerns finite-error approximation under a fixed parameterization; it proves neither closedness nor exact representability and does not bound each Taylor coefficient separately. This raw squared parameter norm is distinct from the homogeneous atom sizes used in the coefficient-space diagnostics.
Adam-trained gated scalar regression
We additionally compare tied and untied SiLU-gated scalar blocks at on two four-channel teachers. The asymmetric teacher is with independent standard Gaussian directions , and weights ; the tied teacher sets . Here the tied student is the matched-gate specialization . Training uses 20,000 examples, 5,000 held-out examples, Adam with learning rate , batch size 256, and a cosine schedule for 4,000 steps, and four seeds. Table 10 reports median relative test MSE.
| Parameters Untied/tied | Asymmetric teacher Untied | Asymmetric teacher Tied | Tied teacher Untied | Tied teacher Tied |
| 17/18 | 0.3162 | 0.2580 | 0.3808 | 0.1899 |
| 34/36 | 0.0843 | 0.1049 | 0.0178 | 0.0014 |
| 51/54 | 0.0322 | 0.0387 | 0.0045 | 0.0013 |
| 68/72 | 0.0188 | 0.0370 | 0.0010 | 0.0006 |
| 102/108 | 0.0088 | 0.0052 | 0.0009 | 0.0006 |
| 136/144 | 0.0062 | 0.0024 | 0.0008 | 0.0004 |
Table 10. Adam-trained scalar regression with approximately matched total parameters. Counts are untied/tied; the tied models have slightly larger budgets. Bold marks the smaller error within each teacher and row.
The untied student has lower error on the asymmetric teacher at the intermediate budgets of 34–68 parameters, whereas the tied student is more accurate at the smallest budget and at the two largest budgets. On the tied teacher, the tied student is more accurate at every budget. Under approximately matched total parameters, the untied advantage is therefore limited to a range of budgets on the asymmetric teacher.
References
- [1]Hirotachi Abo and Nick Vannieuwenhoven. Most secant varieties of tangential varieties to Veronese varieties are nondefective. Transactions of the American Mathematical Society, 370(1):393–420, 2018. URL https://arxiv.org/abs/1510.02029.
- [2]James Alexander and André Hirschowitz. Polynomial interpolation in several variables. Journal of Algebraic Geometry, 4(2):201–222, 1995.
- [3]Enrico Carlini, Maria Virginia Catalisano, and Anthony V Geramita. The solution to Waring’s problem for monomials. arXiv preprint arXiv:1110.0745, 2011.arxiv.org/abs/1110.0745
- [4]Enrico Carlini, Maria Virginia Catalisano, and Anthony V. Geramita. The solution to the Waring problem for monomials and the sum of coprime monomials. Journal of Algebra, 370:5–14, 2012. doi: 10.1016/j.jalgebra.2012.07.028.DOI
- [5]Enrico Carlini, Mario Kummer, Alessandro Oneto, and Emanuele Ventura. On the real rank of monomials. Mathematische Zeitschrift, 286(1):571–577, 2017.
- [6]Grigorios G Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Yannis Panagakis, Jiankang Deng, and Stefanos Zafeiriou. Π-nets: Deep polynomial neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7325–7335, 2020.DOI
- [7]Grigorios G Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Jiankang Deng, Yannis Panagakis, and Stefanos Zafeiriou. Deep polynomial neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4021–4034, 2022.DOI
- [8]Gonzalo Comas and Malena Seiguer. On the rank of a binary form. Foundations of Computational Mathematics, 11(1):65–78, 2011. doi: 10.1007/s10208-010-9077-x.DOI
- [9]Pierre Comon, Gene Golub, Lek-Heng Lim, and Bernard Mourrain. Symmetric tensors and symmetric tensor rank. SIAM Journal on Matrix Analysis and Applications, 30(3):1254–1279, 2008.
- [10]Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International Conference on Machine Learning, pages 933–941. PMLR, 2017.arxiv.org/abs/1612.08083
- [11]Vin de Silva and Lek-Heng Lim. Tensor rank and the ill-posedness of the best low-rank approximation problem. SIAM Journal on Matrix Analysis and Applications, 30(3):1084–1127, 2008. URL https://arxiv.org/abs/math/0607647.
- [12]Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, 2021.
- [13]Joe Kileel, Matthew Trager, and Joan Bruna. On the expressive power of deep polynomial neural networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
- [14]Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM Review, 51(3):455–500, 2009.
- [15]Michael T Pearce, Thomas Dooms, Alice Rigg, Jose M Oramas, and Lee Sharkey. Bilinear MLPs enable weight-based mechanistic interpretability. In International Conference on Learning Representations, 2025.
- [16]Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.arxiv.org/abs/1710.05941
- [17]Noam Shazeer. GLU variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.arxiv.org/abs/2002.05202
- [18]David So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le. Searching for efficient transformers for language modeling. In Advances in Neural Information Processing Systems, volume 34, pages 6010–6022, 2021.
- [19]Nick Vannieuwenhoven. A condition number for the tensor rank decomposition. Linear Algebra and its Applications, 535:35–86, 2017. URL https://arxiv.org/abs/1604.00052.
- [20]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.




