ReWY: Faster Backpropagation for Delta-Rule Attention - From 7 to 3 Matrix Multiplications

Introduction

Linear attention has become a practical building block of industrial language-model architectures. Kimi Linear uses Kimi Delta Attention (KDA) in an approximately 3:1 mixture with global Multi-Head Latent Attention [11, 13]. Qwen3-Next places Gated DeltaNet (GDN) in 36 of its 48 layers, and the Qwen3.5-0.8B text backbone uses it in 18 of 24 layers [15, 16]. These models repeatedly invoke the same family of delta-rule operators to update a recurrent matrix state. Optimizing that primitive therefore addresses an existing model-computation workload shared across many layers and architectures.

The attraction of these operators is their sequence-length-independent recurrent state and linear sequence complexity. Training, however, must also expose enough parallel work to exploit accelerator matrix units. Chunkwise algorithms express causal interactions within each chunk as dense matrix operations and connect chunks through a state scan [9, 11, 19, 21]. Flash Linear Attention (FLA) provides widely used implementations of this computation; the Kimi authors release their KDA kernel through FLA itself [4, 13]. The systems question is how much work this core operator actually needs to perform, particularly in its backward pass.

A redundant derivative graph. The conventional WY decomposition is useful for parallelizing the forward pass, but differentiating its intermediate operations separately creates avoidable backward work. It forms adjoints of two auxiliary objects—a transformed key matrix and a triangular inverse—only to multiply them again into the gradients that the caller needs. Fusion and tiling can accelerate these products, but executing the same derivative graph still pays for their arithmetic and intermediate lifetimes. At chunk size 64 and key/value width 128, the local WY graph contains seven matrix products and 7.34 million dense FLOPs per chunk/head, before the common state scan and interaction gradients are counted.

Eliminate work, then schedule what remains. Our key observation is that the chunk correction satisfies one triangular residual equation. Differentiating this composite map exposes a shared reverse sensitivity and makes the inverse-adjoint chain unnecessary. We call the resulting algorithm ReWY. As shown in Figure 1, ReWY reduces the local WY computation from seven products to three, removing 42.9% of its dense FLOPs at the above dimensions. The forward recurrence, attention semantics, and exact gradients remain unchanged. This is an operator algorithm change applicable to DeltaNet, GDN, KDA, and GDN2; it does not introduce a new neural-network architecture.

The reduced graph also exposes reuse in the remaining interactions. Matrix intra factors vector-gated pairwise interactions into centered matrix contractions, replacing repeated token-pair processing with matrix-unit work. Gram elimination computes scalar-gate reductions from key-gradient products that are already needed, avoiding a separate key Gram matrix. These transformations complement ReWY: the latter simplifies the coefficient producer, while the former optimize its gradient consumers. Their independent and conditional gains are measured separately.

Turning this algebra into speed requires an execution strategy. Sharing the reverse sensitivity can save products while adding memory traffic; fusing all consumers can instead extend live ranges and restrict occupancy. Our implementations select joint or separate producer–consumer schedules by device and operator, preserve existing FP32 working matrices and accumulations, and use one schedule across the evaluated shapes. Matched saving policies prevent existing FLA activation-saving support from being counted as a new algorithmic contribution. The evaluation measures the complete backward operator, rather than extrapolating a local kernel improvement to the whole computation.

Contributions. This paper makes three contributions:

• A smaller delta-rule derivative graph. ReWY eliminates four local matrix products and the inverse adjoint while recovering all input, state, and gate gradients. We give the resulting formulas in the main text and the full equivalence proof in Appendix § 7.

• Contractions and schedules. Centered matrix intra and Gram elimination remove distinct sources of repeated work. GPU and NPU schedules realize this reuse while controlling shared storage, numerical range, and intermediate lifetimes.

• Complete-operator evaluation. Across six shapes, precision-qualified NVIDIA GPU comparisons yield complete backward speedups of 1.15–1.81×. A separate Ascend 950DT public autograd study yields 1.09–2.89×, with standalone residual and FP32 matrix-intra controls. Ascend 910C comparisons retain their stated precision qualifications. Paired Gram controls establish additional ReWY gains. Tests check every gradient, and separate allocation measurements characterize storage.

Our implementation and reproduction materials are available at https://github.com/flashserve/ReWY.

The layer fractions above motivate the operator’s relevance; they are not runtime fractions. Supplementary A800, Ascend 910C, and scoped H100/H200 full-model backward measurements in § 4.8 quantify the latter using FLA’s existing full models and an identified GDN2 layer port. For example, the core backward accounts for 25.7–31.3% of the unmodified FLA KDA model’s measured backward interval across its input-size sweep. The paper’s main performance claim remains the complete-operator improvement; model integration measures how that improvement carries through the surrounding computation.

Background

Accelerator execution and chunkwise training

GPU and NPU kernels combine matrix operations with elementwise transformations and reductions. A sequence of kernels exchanges intermediate arrays through device memory, whereas a fused kernel can reuse values in registers or on-chip storage. Fusion also keeps more operands and accumulators live. For chunkwise delta-rule operators, the execution schedule must balance this local reuse against the parallelism available across chunks, heads, and feature tiles.

Chunkwise computation separates two kinds of dependency. Within a chunk, triangular coefficient matrices encode the causal token interactions and expose matrix parallelism. Between chunks, a state scan propagates the recurrent memory. The backward pass has the same structure in reverse: a scan accumulates gradients from later chunks, after which local kernels compute the input gradients. Recomputation of forward intermediates or saving them changes the work surrounding these local kernels. We compare the two policies separately.

Delta-rule operators

Consider one chunk of $n$ tokens and one attention head. The incoming memory is $S \in\mathbb{R}^{d_k \times d_v}$, queries and keys are $Q, K \in\mathbb{R}^{n \times d_k}$, and values are $V \in\mathbb{R}^{n \times d_v}$. Token vectors are columns, and the corresponding matrices stack their transposes as rows. A fixed query scale is absorbed into $Q$. An overbar denotes a reverse-mode adjoint, with outer normalization and gate activations handled by the chain rule.

Let $g_i$ be the log-decay vector, $b_i$ the erase gate, and $w_i$ the write gate. The four operators use the recurrence

The decayed state $S_i^{-}$ predicts a value from the gated key. The correction $r_i$ subtracts this prediction from the gated input value, and the rank-one update writes the correction along $k_i$. The output reads the updated state, including the current write. Table 1 specifies the gate parameterizations; $\odot$ denotes elementwise multiplication.

Operator	Log decay $g_i$	Erase $b_i$	Write $w_i$
DeltaNet	$0$	$\beta_1 \mathbf{1}_{d_k}$	$\beta_1 \mathbf{1}_{d_v}$
GDN	$\eta_i \mathbf{1}_{d_k}$	$\beta_i \mathbf{1}_{d_k}$	$\beta_i \mathbf{1}_{d_v}$
KDA	vector	$\beta_i \mathbf{1}_{d_k}$	$\beta_i \mathbf{1}_{d_v}$
GDN2	vector	independent vector	independent vector

The chunk system and conventional WY backward

Let $R \in\mathbb{R}^{n \times d_v}$ stack the corrections, $G_i = \sum_{t \le i} g_t$ be cumulative log decay, and $\gamma= \exp G$. Define

where $B_e, W_e$ stack the erase/write gates and $\oslash$ denotes elementwise division. The factors $E$ and $Y$ account for decay before an erase and after an earlier write. Substituting the unrolled state gives

In matrix form,

The projection $\mathcal{L}_{<}$ retains the strictly lower triangle. Thus $M$ is unit lower triangular, and each coefficient describes how an earlier correction affects a later prediction. Appendix A derives this system for all compatible dimensions.

Writing $A = M^{-1}$, the conventional WY implementation forms $W = AE$ and $U = AZ$, then computes $R = U - WS$ during the state scan. This organization allows the local factors $A, W, U$ to be prepared independently across chunks. The token outputs and outgoing state are

where $\Lambda= \operatorname{diag}(\gamma_n)$, $F = Y\Lambda$, and $C_q = \mathcal{L}_{\le}(Q_gY^{\top})$. The inclusive projection $\mathcal{L}_{\le}$ accounts for the current write in each output.

Given an accumulated correction adjoint $\bar{R}$, the expanded WY backward forms $\bar{W} = -\bar{R}S^{\top}$, then $\bar{A} = \bar{R}Z^{\top} + \bar{W}E^{\top}$, and finally propagates $\bar{A}$ through the inverse as $-A^{\top}\bar{A}A^{\top}$. Together with $\bar{Z} = A^{\top}\bar{R}$ and $\bar{E}_{\mathrm{dir}} = A^{\top}\bar{W}$, this local graph contains seven matrix products. It materializes $\bar{W}$ and $\bar{A}$ even though the downstream computation ultimately requires the coefficient and input gradients. This is the graph simplified by ReWY.

ReWY: Algorithm and Analysis

The algorithm simplifies the local backward graph in two steps. ReWY first removes the inverse-adjoint chain and exposes a shared sensitivity $P$. Matrix intra and Gram elimination then reuse the resulting coefficient adjoint to compute vector- and scalar-gate gradients, respectively. The state scan supplies the inter-chunk adjoints, while device schedules determine where the local intermediates are consumed.

Differentiating the residual equation

The chunk correction obeys $MR = Z - ES$, where $M = I + L$ and $A = M^{-1}$. Rather than materializing derivatives of the forward auxiliaries, ReWY computes a shared sensitivity $P = A^{\top}\bar{R}$. This uses the classical reverse derivative of a linear solve [8]; the contribution is its application to the composite delta-rule graph and the resulting elimination of intermediate adjoints, not a new general differentiation rule. The following result gives the complete local gradient interface. Appendix § 7 proves equivalence to the expanded graph.

Thus the complete erase-factor gradient is $-PS^{\top} + DY$. The three new products are $A^{\top}\bar{R}$, $PR^{\top}$, and $PS^{\top}$; the interaction contractions and state scan are shared with the reference algorithm.

State and gate composition. The local theorem is applied after accumulating both uses of the correction:

Both output and outgoing-state contributions are accumulated before the local pullback. ReWY leaves the reverse-state

scan unchanged; the scan supplies the incoming-state gradient once. Shared scalar erase/write gates combine both paths using an undivided expression at zero or tiny beta. GDN2 retains independent erase/write gradients. Appendix § 7.§ 2 and Corollary B.1 derive these identities; Appendix § 10 gives the full reverse algorithm.

Algorithm 1 ReWY local pullback

Require: $A = M^{-1}$, factors $E$, $Y$, incoming state $S$, correction $R$, accumulated adjoint $\bar{R}$.

Ensure: contributions to $\bar{Z}$, $\bar{E}$, $\bar{Y}$, $\bar{S}$.

1: Compute the shared sensitivity $P \leftarrow A^{\top}\bar{R}$.

2: Compute $D \leftarrow-\mathcal{L}_{<}(PR^{\top})$, retaining the strict causal mask.

3: Set $\bar{Z} \leftarrow P$ and $\bar{E}_{\mathrm{dir}} \leftarrow-PS^{\top}$.

4: Add the interaction terms: $\bar{E} \leftarrow\bar{E}_{\mathrm{dir}} + DY$, $\bar{Y} \leftarrow D^{\top}E$.

5: Supply $\bar{S}_{\mathrm{res}} \leftarrow-E^{\top}P$, or use the equivalent scan term $-W^{\top}\bar{R}$, once.

Arithmetic and intermediate storage

The expanded graph has one product for $\bar{W}$, two for $\bar{A}$, two for the inverse pullback, and one each for $\bar{Z}$ and $\bar{E}_{\mathrm{dir}}$. ReWY computes $P$, $PR^{\top}$, and $PS^{\top}$. Figure 2 compares the dependencies, and Algorithm 1 shows how the reduced graph supplies the common downstream gradients. The seven-product count describes the expanded frozen-reference graph; it is not a claim that every optimized implementation performs seven products. The inspected FlashQLA revision already combines the residual right-hand side but retains the two-sided inverse pullback [17]. This paper does not measure performance against that implementation.

The dense arithmetic removed per chunk/head is

At $n = 64$, $d_k = d_v = 128$, the local WY cost falls from 7.34 to 4.19 million FLOPs (42.9%). This count excludes the unchanged state scan, common interaction products, and activation recomputation; it is not a prediction of complete-backward speedup. Appendix § 7.§ 4 details the cost.

The eliminated adjoints have shapes $\bar{A} \in\mathbb{R}^{n \times n}$ and $\bar{W} \in\mathbb{R}^{n \times d_k}$. The new sensitivity $P \in\mathbb{R}^{n \times d_v}$ feeds value, direct-state, and coefficient gradients, creating a choice between local reuse and shared storage. Fusing its production and consumption can keep $P$ on chip; separate kernels exchange it or the resulting coefficient matrices through device memory. The resulting allocation peak is set by the overlap of these lifetimes. § 4.3 measures that peak for the same complete implementations used in the timing comparison.

Matrix intra for vector-gated interactions

ReWY produces the causal coefficient adjoint $D$; the output path produces $H_q = \mathcal{L}_{\leq}(OR^{\mathsf{T}})$. Both must be propagated to query, key, and gate gradients. With vector decay, each token pair carries a feature-dependent factor $\exp(G_{ik} - G_{jk})$. Evaluating this factor inside each pairwise interaction obscures the matrix structure of the contraction.

GLA and KDA establish separable cumulative-gate factors and stability-aware secondary chunking [11, 20]. We use this structure to reorganize the four backward contractions and their feature-local gradients; separability alone is not a new result. The gate ratio factors into $\exp(G_{ik})$ and $\exp(-G_{jk})$. Scaling the sending features before multiplication and the receiving features afterward yields

and the physical input gradients

Here $\gamma^{-1}$ denotes elementwise reciprocals. For example, the $(i,k)$ entry of $\gamma\odot H_qY$ equals $\sum_j (H_q)_{ij}K_{jk}\exp(G_{ik} - G_{jk})$. The other paths follow the same factorization or its transpose. The matrix-intra stage therefore consists of four matrix products and feature-local chain rules, with the causal masks already included in $H_q$, $D$.

Gram elimination for scalar-gate gradients

For scalar-gated GDN, the remaining gate reduction has a different redundancy. The key gradient already contracts the coefficient adjoint with the keys, but an expanded gate path also forms a key Gram matrix. The gate gradient needs only weighted row and column sums of that matrix, which can be obtained from the existing contractions.

The scalar-gated coefficient is

Let $D=\bar{L}$ include the strict mask, set $\Gamma_{ij}=D_{ij}\exp(G_i-G_j)$, and define

Differentiating the receiving and sending positions of the log gate yields

Centered factors. Separately forming the exponential factors can create large intermediate values even when their ratios are moderate. For a feature-wise center $c$, let $C_c = \operatorname{diag}(\exp c)$ and use

The factors $C_c$ and $C_c^{-1}$ cancel in the coefficient products and in each completed input gradient. Thus the centered factors replace $\gamma$ and $\gamma^{-1}$ by $\alpha=\exp(G-c)$ and $\zeta=\exp(c-G)$, respectively. Appendix § 8 proves the factorization and scaling invariance.

Let $m_k$ and $M_k$ be the minimum and maximum of $G_{ik}$ over tokens, and set $\Delta_k=M_k-m_k$. The midpoint minimizes the largest centered log magnitude:

Consequently, the exponential factors satisfy

The implementation centers feature tiles and checks their gate ranges. Stronger gates use a validated pairwise or sub-chunk path. These safeguards accompany the algebraic factorization.

Matrix intra retains the $O(n^2d_k)$ interaction count but groups it into regular matrix operations with fused feature-local derivatives. The frozen KDA implementation already uses matrix multiplication for part of its intra computation; our consumer reorganizes whole-chunk contractions, centering, and gradient production. Since $D$ can come from either WY backward graph, the matrix-intra contribution can be evaluated separately from residual elimination. Meanwhile, the key-gradient path computes

Interchanging the finite token and feature sums gives

The gate derivative is therefore

The beta contribution uses the same contraction:

This expression is valid at zero beta. The local gate contribution is added to the other cumulative-gate derivatives before the reverse sum. Appendix Appendix § 9 gives the rectangular bilinear identity and its proof.

Given $A_K,B_K$, the replacement costs $O(nd_k)$ elementwise operations and reductions, eliminating a dense $KK^{\mathsf{T}}$ product of approximately $2n^2d_k$ FLOPs and its $n \times n$ intermediate. The transformation applies when the pairwise matrix is consumed through these reductions. Scalar GDN has this structure; vector-gated KDA/GDN2 instead use the feature-dependent matrix-intra factorization. Gram elimination accepts the coefficient adjoint from either WY formulation, providing a second independent component control.

Execution and numerical precision

The transformed graph has a common producer–consumer interface across the four operators. After the reverse-state scan, a producer computes $P$, value/write gradients, and $D$. Consumers propagate the output and residual coefficients to query, key, and gate gradients. The final reverse cumulative sum converts cumulative-gate gradients to token-local gradients. This separation exposes both matrix reuse within a chunk and independent work across chunks and heads.

Joint and separate consumers. A joint kernel consumes the sensitivity and state-direct terms together with the intra contractions, reducing intermediate transfers. A separate schedule shortens live ranges and gives each stage its own task partition. NVIDIA RTX 5070Ti uses a joint KDA consumer with streamed value tiles. The A800 DeltaNet/GDN paths consume $P$ locally without a new global buffer. On Ascend 910C, GDN separates the scalar state-endpoint reduction and gate scan into a Vector kernel. The FP32 vector-gate schedules on Hopper and 950 are specified with their measurements in § 4.7 and § 4.6. Appendix § 11 details the schedules; Appendix § 12.§ 2 identifies precision exclusions.

The state scan remains serial across chunks and parallel across independent value columns. Candidate schedules are fixed per device across the measured shapes. The A800 DeltaNet state kernel retains the original K64 arithmetic with three pipeline stages; GDN retains the original state autotuner. The frozen references keep their native device autotuning, including the multi-stage A800 search space.

Precision and numerical stability. The main experiments use BF16 features, effective erase/write weights, and output adjoints, with FP32 log gates and state interfaces. Our precision contract preserves existing operand precision, stored matrix formats, and FP32 accumulation; exclusions are specified in Appendix § 12.§ 2. Newly shared $P$ storage follows the consumer requirements: RTX 5070Ti uses the input format for DeltaNet/GDN/KDA, while GDN/NPU retains FP32 values for its value/beta consumers. Table 8 specifies these uses.

Quantity	Contract
State scan	FP32 accumulators and initial-state gradient; stored chunk states follow the reference format.
Shared $P$	RTX scalar/KDA paths store the input format after required FP32 uses; GDN/Ascend 910C retains FP32 for value/beta consumers.
Existing matrices	Eligible paths retain FP32 $L/D$ arrays and prescribed GEMM conversions. Output casts cannot repair a lower internal operand precision.
GDN/Ascend 910C	Weighted coefficients and key contractions remain FP32. Gate scalars and both endpoints are reduced before output rounding.

Centered matrix intra uses gate-range dispatch, and small shared beta uses the undivided variant in (9). Independent GDN2 erase/write gates retain separate derivatives. These choices are checked against the complete-gradient thresholds in § 4; partitions, masks, and shared-parameter reductions are derived in Appendix § 10.§ 1.

Experiments

We evaluate complete-backward latency and memory across four delta-rule operators on NVIDIA GPUs and Ascend NPUs. § 4.6 specifies the public-interface timing boundary for Ascend 950DT; § 4.7 reports H100/H200 results. Component experiments separate the effect of ReWY from matrix intra and Gram elimination, connecting the local graph analysis to complete-operator performance.

Experimental setup

The primary latency and memory experiments use an RTX 5070Ti GPU (12 GiB, SM120), an NVIDIA A800-SXM4-80GB GPU (80 GB, SM80), and an Ascend 910C NPU (64 GiB). The GPU environment is PyTorch 2.9.1+cu128 and Triton 3.5.1. The NPU environment is PyTorch 2.7.1, torch_npu 2.7.1.post4, Triton 3.2.0, and CANN 9.0 beta1. Baselines are frozen FLA implementations with equivalent platform adaptations for the NPU; the reference revision and adaptations are listed in Appendix § 12.

Workloads and policies. We vary sequence length, batch size, and head count using six $(B,T,H)$ shapes:

The chunk size is $C=64$, and $d_k=d_v=128$. These shapes vary both the number of serial state updates and the number of independent head/chunk tasks. Each comparison uses either default recomputation (R) or full saving (A) on both sides. KDA uses FLA’s existing saving switch; matched wrappers cache identical intermediates for the other interfaces. One validated implementation is fixed per operator/device across all six shapes and both policies.

Timing and allocation. Complete low-level backward (B) includes all internal stages and required recomputation. Both sides exclude Python autograd overhead and the public wrapper’s final gradient-type conversion; some optimized paths return FP32 where the reference returns BF16. Appendix G reports conversion-inclusive controls. Device-graph events measure warmed latency, with sixty observations per configuration on the GPUs and twenty on the NPU. Speedups are geometric means of the six reference/optimized median-latency ratios. Allocation measurements use warmed incremental peaks, include returned gradients, and report resident workspace separately.

Precision qualifications. Our precision contract preserves the reference input/output types, FP32 accumulations, and existing FP32 working matrices. Entries marked † cast existing matrix operands to lower precision and are reported only as diagnostic comparisons; they do not establish a gain under this contract. Appendix G.2 identifies these exclusions. Newly introduced factor storage is distinct from casting an existing matrix.

under this contract. Appendix G.2 identifies these exclusions. Newly introduced factor storage is distinct from casting an existing matrix.

Complete-backward latency

Figure 3 shows positive aggregate gains for the retained comparisons under both saving policies. On RTX 5070Ti, R/A speedups are 1.15/1.22× for DeltaNet, 1.16/1.23× for GDN, and 1.56/1.81× for KDA. On A800, DeltaNet and GDN achieve 1.15/1.22× and 1.17/1.25×, respectively. These are complete-backward measurements, including the reverse-state scan and recomputation, with the reference autotuning retained.

Table 9 reports one R/A pair per operator and device, with the precision exclusions above. § 4.6 and § 4.7 report the FP32 Ascend 950DT and Hopper comparisons and their timing protocols.

Operator	RTX 5070Ti	A800	Ascend 910C
DeltaNet	1.1469/1.2152	1.1474/1.2237	1.5806/1.7528
GDN	1.1614/1.2264	1.1732/1.2471	1.3739/1.5142
KDA	1.5560/1.8066	1.7943/2.0832	1.5464/1.6291
GDN2	1.7910/2.1458	1.9577/2.2792	2.5518/2.9959

Full saving produces larger aggregate speedups wherever both policies were measured. Removing common recomputation increases the fraction of backward time affected by the transformations, consistent with the complete-operator cost decomposition in Appendix § 10.§ 1. Within each pair, the ratio also varies with the workload. For example, A800 DeltaNet has minimum R/A ratios of 1.05/1.06×, and GDN has minima of 1.08/1.12×. Appendix § 12 reports per-shape variation and the timing protocol.

Backward memory footprint

We measure the same complete implementations at $B = 1$, $T = 8192$, $H = 16$, $d_k = d_v = 128$, with identical forward saved storage within each policy. Figure 4 shows the reduction relative to each reference peak; Table 12 gives absolute MiB values.

Operator	Device	Original	Optimized
DeltaNet	RTX 5070Ti	417.250/257.250	417.250/257.250
DeltaNet	A800	417.250/257.250	385.250/225.250
DeltaNet	NPU	610.508/450.505	518.383/358.380
GDN	RTX 5070Ti	418.250/258.250	417.750/257.750
GDN	A800	418.250/258.250	385.750/225.750
GDN	NPU	612.510/452.508	520.636/360.634
GDN2	RTX 5070Ti	961.000/737.000	833.000/609.000
GDN2	A800	961.000/737.000	801.000/577.000
GDN2	NPU	1015.262/791.259	770.007/546.004
KDA	RTX 5070Ti	772.000/548.000	641.500/417.500
KDA	A800	772.000/548.000	641.531/417.531
KDA	NPU	824.511/600.509	773.009/549.006

Allocation effects depend on the device schedule. On RTX 5070Ti, the DeltaNet peak is unchanged, whereas GDN saves 0.5 MiB. KDA saves 130.50 MiB under both policies. Its remaining R and A peaks are 641.50 and 417.50 MiB, respectively. For A800, the savings are 32 MiB for DeltaNet and 32.5 MiB for GDN, with no new global $P$ storage. Appendix § 14 lists the remaining peaks and their version qualifications. These differences reflect how shared sensitivity and downstream gradients overlap in memory.

Appendix § 13 specifies saved storage, resident workspace, and allocation controls at two shapes. Peaks depend on the tested schedule.

Transformation ablations

The recorded RTX 5070Ti and Ascend 910C controls test each transformation alone and ReWY with a fixed companion. The companion is matrix intra for KDA and GDN2, and Gram elimination for GDN. Standalone results use their respective reference runs. Conditional gains use paired runs with the companion and its schedule fixed. The A800 results above correspond to the complete implementations. All recorded latency ablations are plotted; dagger markers identify historical GPU and Ascend 910C matrix-consumer controls whose precision status is detailed in Appendix § 14.

Standalone and paired gains. The FP32 950 controls in § 4.6 establish the independent effects of matrix intra

Operator	Device	ReWY only	Companion only
KDA	RTX 5070Ti	1.0845/1.1160	1.1996/1.2600
KDA	NPU	1.1402/1.1660	1.3675/1.4179
GDN	RTX 5070Ti	1.0490/1.0852	1.0214/1.0323
GDN	NPU	1.0733/1.0785	1.0538/1.0658
GDN2	RTX 5070Ti	1.1732/1.2026	1.2851/1.3407
GDN2	NPU	1.1103/1.1217	2.0834/2.3268

Operator	Fixed companion	RTX 5070Ti R/A	NPU R/A
KDA	Matrix intra	1.1070/1.1366	1.1308/1.1490
GDN	Gram elimination	1.0485/1.0663	1.0708/1.0889
GDN2	Matrix intra	1.2364/1.2911	1.2248/1.2876

and ReWY for KDA and GDN2. For GDN, Gram elimination alone yields R/A gains of 1.02/1.03× on RTX 5070Ti and 1.05/1.07× on Ascend 910C. Adding ReWY with Gram elimination fixed yields 1.05/1.07× and 1.07/1.09×, respectively. These paired controls establish a producer benefit beyond the companion; standalone measurements from other runs are not multiplied to infer a conditional gain. Component rows with † retain their measured-version qualification.

Component allocation effects. Figure 6 separates measured allocation changes. GDN/NPU’s residual-only control saves 64 MiB, with another 29 MiB saved by adding Gram elimination. The allocation effect depends on the overlap of shared sensitivity and downstream-gradient lifetimes. Table 12 and Table 13 give the complete peaks and component effects; a component ablation is not an additional production version.

Operator	Device	O alone	C alone	O after C	C after O
KDA	RTX 5070Ti	2.50/2.50	130.47/130.47	0.00/0.00	127.97/127.97
KDA	NPU	51.50/51.50	0.00/0.00	51.50/51.50	0.00/0.00
GDN2	NPU	53.25/53.25	0.00/0.00	245.25/245.25	192.00/192.00
GDN	RTX 5070Ti	32.00/32.00	0.00/0.00	32.00/32.00	0.00/0.00
GDN	NPU	64.00/64.00	0.00/0.00	93.00/93.00	29.00/29.00

Numerical validation

Validation for the original evaluation compares outputs, final states, and every input gradient. The separate 950 coverage is specified in § 4.6. An independent CPU FP64 recurrence supplies references through $T = 2048$; the $T = 8192$ workloads compare complete gradients with the frozen baseline. Relative $L_2$ thresholds are 0.8% for ordinary outputs/gradients and 2% for gate, beta, and erase/write gradients. Near-zero references use a maximum absolute threshold of $10^{-6}$, and all values must be finite.

The evaluated implementations pass the main workloads and the tested extended interfaces. The KDA fallback has a bitwise-determinism limitation described in Appendix § 12.§ 3. Numerical passes do not establish the excluded paths’ compliance with the precision contract. These operator checks are separate from the model-level diagnostics in § 4.8. The suite includes tail chunks, unequal widths, nonzero state boundaries, zero/tiny beta, strong decay, and independent GDN2 gates. Graph replay uses intervening inputs to check outputs and temporary-buffer reuse. Appendix § 12 specifies input distributions and additional BF16, FP16, and layout coverage.

Ascend 950DT: public autograd

We evaluate the same four operators on one Ascend 950DT with 96 GiB HBM and 248 KiB usable UB. The software stack is CANN 9.2.0, PyTorch 2.10.0, torch_npu 2.10.0.post7, and Triton-Ascend 3.6.0. The six shapes in Eq. (5), BF16 inputs, $C = 64$, and $d_k = d_v = 128$ are unchanged. Existing FP32 working matrices and accumulations are preserved. KDA and GDN2 use FP32 matrix intra with uniform $32 \times64$ working tiles, distinct from the chunk and head dimensions.

Portability and the reference. The frozen FLA revision is unchanged, with uniform runtime adaptations required by this device: original GDN/GDN2 WY feature tiles are capped at 64 to fit UB, and three original KDA/GDN2 WY kernels use one program per task. The latter change removes repeat-dependent gradient errors observed with their persistent task loops. DeltaNet’s residual producer requires the same scheduling change. Arithmetic-AST checks confirm that all four replacements preserve the original arithmetic after removing only the outer task loop; independent CPU equations additionally check the three original WY replacements. The precise compiler-level cause has not been established. All relevant comparisons use this corrected reference without workload-specific dispatch.

Protocol and acceptance. We time synchronized eager execution through public autograd, including allocations and gradient conversions. B excludes forward execution; FB includes both passes. These ratios are separate from low-level graph measurements and do not measure a complete training step. Thirty alternating ABBA/BAAB rounds yield 60 observations per implementation after at least six warm-up rounds and two seconds of warm-up. Ratios use per-shape medians and their equal-weight geometric mean across six shapes. KDA and GDN2 expose both policies. The public APIs of GDN and DeltaNet support R only.

The combined implementation passes 12 complete checks of forward and backward against a CPU FP64 recurrence at $T = 64$. Each standalone branch passes eight corresponding checks. Every phase also passes ten actual-FLA-input matrix tests, including wide gates and batch two. Across all six shapes, 240 full-gradient/output/state checks pass, including 120 exact same-input repeat comparisons of original and optimized branches. Each timing case rechecks every gradient and optimization dispatch. Thresholds remain 0.8% for ordinary quantities and 2% for gate/beta gradients, with the same near-zero rule. The 950 suite does not establish FP16, variable-length, graph-replay, or model-training convergence coverage.

GDN2 has the largest combined gain, while DeltaNet remains below 1.1× and KDA’s standalone residual gain is modest. The independent FP32 matrix controls demonstrate a substantial contribution for GDN2 under the stated storage contract. Combined, residual-only, and matrix-only ratios are directly measured against their respective references; we do not divide them to infer conditional gains. No new saving capability or 950 memory result is claimed. These operator results do not establish an end-to-end training benefit.

Component	Operator	Policy	B	FB
ReWY	KDA	R	1.0501	1.0158
ReWY	KDA	A	1.0663	1.0190
ReWY	GDN2	R	1.1631	1.0849
ReWY	GDN2	A	1.1874	1.0915
FP32 matrix intra	KDA	R	1.1417	1.0652
FP32 matrix intra	KDA	A	1.1836	1.0724
FP32 matrix intra	GDN2	R	1.8385	1.4385
FP32 matrix intra	GDN2	A	2.0396	1.4820

H100 and H200 backward evaluation

We evaluate all four operators on NVIDIA H100 80GB HBM3 and NVIDIA H200 (both SM90), using PyTorch 2.12.1+cu126 and Triton 3.7.1. We use the six shapes in Eq. (5), $C = 64$, and $d_k = d_v = 128$, with the same frozen FLA reference and matched R/A policies. CUDA Graph device events yield 60 observations per implementation and shape. Table 4 reports geometric means of six ratios of median complete-backward times. This boundary includes required recomputation and excludes the public wrapper’s final gradient-type conversion.

Operator	H100 R	H100 A	H200 R	H200 A
DeltaNet	1.2066	1.3046	1.1776	1.2697
GDN	1.2056	1.2829	1.1819	1.2505
KDA	1.3264	1.4213	1.3065	1.3964
GDN2	1.5680	1.7033	1.5408	1.6527

KDA combines ReWY with FP32 endpoint-scaled matrix intra in a separate schedule. GDN2 uses an FP32 matrix-intra joint consumer and state tiles spanning the full key width and 16 value columns. GDN combines the unified adjoint, Gram elimination, and a liveness-aware joint consumer. DeltaNet uses the unified adjoint and state tiles of key width 64 and value width 16. All retain existing kernel arithmetic and FP32 precision.

All 48 operator/shape/policy combinations per device accelerate. Checks against CPU FP64, complete baseline gradients, and graph replay pass. Gains reflect the complete implementations; Hopper component ablations are not available. Software stacks differ across device groups, so differences in speedup cannot be attributed solely to hardware or scheduling. § 4.8 reports full-model backward; Appendix § 12.§ 4 summarizes operator validation.

Supplementary full-model backward

We integrate the optimized cores into existing FLA language models and measure the complete model backward pass. KDA uses FLA’s KDAForCausalLM; GDN and DeltaNet use its native CausalLM classes with published structures [3, 6, 12]. Each model includes its original projections, convolutions, normalization, MLPs, language-model head, and loss. Only the core implementation changes within each matched comparison.

GDN2 uses the native GatedDeltaNet2 layer in FLA’s common GDN language-model shell, retaining the official 1.3B model’s key dimensions [5, 14]. It is a FLA layer port, not the unchanged official full model. Appendix § 16 specifies configurations and numerical limitations. These are not full Kimi Linear or Qwen3-Next deployments.

Backward-only boundary. All full-model tables report B, the complete loss.backward() interval. Forward/loss evaluation, gradient clearing, optimizer execution, compilation, initialization, and data loading are outside this measured interval. Backward includes every model layer and required recomputation, rather than only the substituted attention core. Both sides use native recomputation R, no layer checkpointing, BF16 autocast, and FP32 parameters, gradients, existing working matrices, and accumulations. Models are randomly initialized and inputs are resident synthetic tokens. The reported acceleration percentage is

The tables show positive backward gains. This percentage measures increased backward execution rate, rather than percentage latency reduction or training throughput. For a group of configurations, we first take the geometric mean of all its original/optimized median-time ratios, then subtract one and multiply by 100; we do not average percentages or filter samples within a group. Original and final optimized full models are compared; component ablations remain at the operator level.

A800. CUDA events bracket backward inside a captured updating graph. Four ABBA blocks yield 60 observations per implementation; forward and optimizer work execute outside the event interval. Table 5 reports the resulting backward times and gains.

Model	$T$	Original B (ms)	Optimized B (ms)	B gain (%)
KDA	2048	112.547	103.068	+9.20
KDA	8192	358.533	325.021	+10.31
GDN	8192	334.440	333.498	+0.28
DeltaNet	2048	87.883	87.056	+0.95
DeltaNet	8192	277.062	274.461	+0.95
GDN2 port	2048	110.450	102.344	+7.92
GDN2 port	8192	371.798	342.511	+8.55

Across KDA’s eight fixed-architecture input-size points, the aggregate B acceleration is 10.04%. The displayed GDN case improves by 0.28%, DeltaNet by about 0.95%, and the GDN2 port by 7.92–8.55%. Backward work outside the substituted core limits the full-model gain.

H100 and H200. Using the stacks in § 4.7, we measure backward event intervals in continuously updating graphs. Four fresh-model ABBA blocks yield 60 observations per implementation and configuration; forward and optimizer work execute outside the measured interval. Table 6 shows fixed-architecture groups with positive aggregate gains on both devices. Each group includes all its input configurations and uses a fixed backend. Each percentage transforms the geometric-mean ratio as in Eq. (6). Comparisons that change architecture or loss workload are excluded.

Model / scope / backend	Points	H100	H200
KDA / default	8	+9.93	+8.83
GDN / published / wide64	2	+4.48	+4.44
DeltaNet / published	2	+1.89	+1.24
GDN2 port / official dims.	2	+6.51	+5.34

Full-model backward includes projections, MLPs, normalization, and other work; its shapes and gate distributions also differ from the standalone sweep. These measurements cannot isolate the causes of the smaller model gains. The tested configurations pass execution and finite-value checks. KDA also has local and trajectory diagnostics. The other families are performance-only; some DeltaNet trajectory diagnostics exceed the acceptance thresholds. Operator passes do not establish full-model numerical acceptance or training convergence (Appendix § 16.§ 2).

Ascend 910C. We use the frozen FLA Triton-Ascend reference with the same necessary platform adaptations on both sides. $B$ uses synchronized eager wall-clock timing after forward has completed and synchronized; it excludes optimizer work. Four ABBA blocks yield 30 $B$ samples per implementation. All models use $B = 1$ and $T = 2048$, with the architecture dimensions in Table 14. KDA and GDN2 combine ReWY with FP32 matrix intra; GDN and DeltaNet use their complete implementations. These measurements have a different timing method from A800, so we do not pool cross-device ratios or compare absolute latencies as hardware throughput.

Model	Layers	Hidden	MLP	$H/K/V$	Params (B)
KDA	24	2048	5632	16/128/128	1.391
GDN	24	2048	5632	8/256/256	1.466
DeltaNet	24	2048	5632	16/128/128	1.366
GDN2 port	18	2304	6208	16/128/128	1.450

Model	Original B (ms)	Optimized B (ms)	B gain (%)
KDA	204.359	187.414	+9.04
GDN	175.469	162.777	+7.80
DeltaNet	169.257	139.724	+21.14
GDN2 port	226.075	158.905	+42.27

Ascend 950DT coverage. The Ascend 950DT experiments in § 4.6 time complete operators, not a complete model. No full-model B measurement is available for that device; no full-model speedup is inferred from its operator ratios.

Numerical scope. KDA on A800 passes layer-VJP tests, graph and eager equivalence, and finite-update checks. Full-network gradient differences remain diagnostics. Other A800 and Ascend 910C models pass interface, finite-value, backend-activation, and execution checks. Appendix § 16.§ 1 reports gradient and parameter differences.

Discussion and Conclusion

ReWY shows that efficient delta-rule training benefits from simplifying the derivative graph before optimizing its kernels. A shared reverse sensitivity removes the inverse adjoint and reduces seven local products to three. Matrix intra and Gram elimination expose further reuse in vector- and scalar-gated interactions. These algebraic identities preserve the existing recurrence and apply to four operator parameterizations already supported by community implementations.

Complete-backward measurements and paired controls establish the operator-level benefit. Allocation results show how intermediate lifetimes affect the mapping of the reduced graph to hardware. Supplementary FLA integrations measure the smaller, workload-dependent gains in full-model backward. Exact arithmetic equivalence does not guarantee identical floating-point trajectories. We therefore distinguish operator validation, historical precision-contract violations, and model-level numerical diagnostics. The contribution reduces the work of an existing linear-attention primitive in a form that accelerator kernels can implement.

A Derivation of the Chunk System

For the recurrence in § 2, unrolling the state entering token $i$’s erase operation gives

Here $\gamma_i=\exp(\sum_{t\le i}g_t)$; a write at $j$ receives decays from $j+1$ through $i$. With the row-stacked factors

substitution yields $R_i=Z_i-E_iS-\sum_{j<i}(E_iY_j^\top)R_j$. Stacking rows gives $MR=Z-ES$, with $M=I+\mathcal{L}_{<}(EY^\top)$. This unit lower triangular system is nonsingular for all compatible dimensions: its strict lower part satisfies $L^n=0$. Forward substitution recovers each correction and therefore the original state sequence. The argument also applies to tail chunks.

Let $\Lambda=\operatorname{diag}(\gamma_n)$, $F=Y\Lambda$, and $C_q=\mathcal{L}_{\le}(Q_gY^\top)$. Unrolling the read and final state gives

The output includes the current write, so $C_q$ has an inclusive mask; the erase system uses a strict mask. The existing forward organization $W=AE$, $U=AZ$, $R=U-WS$, with $A=M^{-1}$, is retained.

Complete Gradients and Equivalence

Residual pullback

Proof of Theorem 3.1. Differentiating $MR=Z-ES$ gives

Since $M^\top P=\bar{R}$, the Frobenius pairing yields

Allowed variations of $L$ are strictly lower triangular, giving $D=-\mathcal{L}_{<}(PR^\top)$. The mask is self-adjoint. Substituting $dL=\mathcal{L}_{<}((dE)Y^\top+E(dY)^\top)$ gives the remaining contributions $DY$ and $D^\top E$. □

Cancellation in the expanded graph. The conventional graph has $\bar{W}=-R\bar{S}^\top$ and $\bar{A}=R\bar{Z}^\top+W\bar{E}^\top=\bar{R}(Z-ES)^\top$. Thus its inverse pullback simplifies to

and $A^\top\bar{W}=-PS^\top$. This removes the two auxiliary adjoints without changing any required gradient.

Output, state, and effective-input gradients

The incoming $\bar{S}^+$ includes later chunks and any terminal-state adjoint. Differentiating Eq. (8) gives the complete map

The scan already supplies $-E^\top P=-W^\top\bar{R}$; it must be accumulated once, not added again by the local consumer. The chain rule for Eq. (7) gives

GDN2 returns independent erase/write gradients. With $\lambda=\gamma_n$, the endpoint contribution is

The cumulative and token-local decay derivatives are

Scalar decay sums over features. The endpoint differentiates $\lambda$; the term in $\bar{Y}$ differentiates the token-local reciprocal factor.

Shared beta and scalar decay

For shared erase/write beta, set $X=\gamma\odot K$. The complete, undivided derivative is

Proof. Multiplying Eq. (9) by $\beta_i$ gives

The quotient is used only for sufficiently nonzero beta. At zero or tiny beta the undivided expression remains defined.

For scalar GDN, the erase-only contribution is $b_i^{\mathrm{erase}}=\langle\gamma_i K_i,\bar{E}_i\rangle$. Its token-local log-gate derivative can be computed from completed FP32 accumulators:

This equals the feature sum of the first three terms in $\bar{G}_i$, including at zero beta. The state endpoint is $\lambda\langle S,\bar{S}\rangle$; the key endpoint is $\lambda\sum_i\langle Y_i,\bar{F}_i\rangle$. Both are added at the last valid token before the reverse sum. The write-beta path is independent of the log gate. The forward base-2 conversion retains the natural-log chain rule; no additional ln 2 factor is introduced.

Arithmetic accounting

Counting a multiply and add as two operations, the seven expanded products and three reduced products cost

The difference is $4n^2d_k+4n^3$, excluding common interactions, the state scan, and recomputation. Triangular-block skipping or repeated production of $P$ changes the executed count, not the identity.

Matrix-Intra Equivalence and Centering

For fixed coefficient $C$ and cumulative gates $G$, define

With $D_G(X)=\exp(G)\odot X$ and $M_C(X)=CX$,

Expanding the finite sum proves the factorization. The scaling maps are self-adjoint and $M_C^*=M_{C^\mathsf{T}}$, hence $T_{C,G}^*=T_{C^\mathsf{T},-G}$. These identities yield the contractions in Eq. (1) and gradients in Eq. (3), preserving both causal masks.

For Eq. (4), diagonal factors cancel in $\widetilde{E}\widetilde{Y}^{\mathsf{T}}=EY^{\mathsf{T}}$ and $\widetilde{Q}_g\widetilde{Y}^{\mathsf{T}}=Q_gY^{\mathsf{T}}$. The contractions transform as

Replacing $\gamma,\gamma^{-1}$ by $\exp(G-c),\exp(c-G)$ cancels these factors in every physical gradient. This holds for any center, including a data-dependent center: it is an invariant execution representation, not a new differentiated operator. Direct-state products retain their original coordinates.

For any feature center $c_k$, the triangle inequality gives $\max(|M_k-c_k|,|m_k-c_k|)\ge(M_k-m_k)/2$. The midpoint attains equality, proving the bound in § 3.3. A tile center uses the extrema over the tile and gives the same conservative bound. Range/layout checks select a validated pairwise or subchunk fallback; the recorded GDN2 threshold is 80 in log-base-2 units. Feature tiling selects columns and does not alter any token contraction.

Gram-Free Reduction Proof

For $X\in\mathbb{R}^{m\times d}$, $Y\in\mathbb{R}^{n\times d}$, and $C\in\mathbb{R}^{m\times n}$, let $J=C\odot(XY^\mathsf{T})$. Finite-sum interchange gives

No symmetry, sign, or mask assumption on $C$ is needed. Taking $X=Y=K$ and $C=\operatorname{diag}(\beta)\Gamma$ proves the scalar-gate identity in § 3.4:

Subtracting the two reductions gives $\bar{G}_i^{(L)}$. The erase-beta contribution is $\langle K_i,(A_K)_i\rangle$, valid at zero beta. Direct contributions are added before shared-beta accumulation and the reverse gate sum.

Elimination requires that consumers need only these reductions and that the feature contractions are already computed. A consumer of the full pairwise matrix prevents it. Vector-gated KDA/GDN2 generally have no single scalar $\Gamma_{ij}$ and instead use matrix intra. Reordered floating-point reductions are subject to the same complete-gradient checks, not an exact floating-point equivalence claim.

Complete Reverse Execution and Boundaries

The complete reverse algorithm traverses chunks in reverse order:

1. Accumulate $\bar{R}$ from both output and outgoing-state paths; form $H_q$ and $\bar{F}$.

2. Apply Algorithm 1, then add the output and final-state contributions to $\bar{Q}_g$, $\bar{Y}$, $\bar{S}$.

3. Apply the factor chain rules in Appendix § 7.§ 2; sum erase/write contributions for shared beta.

4. Add both decay endpoints at the last valid token and reverse-sum $\bar{G}$. Restore prescribed query scaling, normalization, and outer gate derivatives.

The scan supplies the incoming-state term once. Causal masking prevents arbitrary reassociation. In general,

Reassociation is valid within complete blocks below the diagonal; diagonal blocks retain the strict mask.

Partitions and shared parameters

Each valid chunk length has its own unit triangular system. Reverse composition therefore handles nonuniform partitions and tail chunks without a new identity. Feature tiles preserve finite contractions. For a broadcast gate $B_e=B\theta$, the adjoint is $\bar{\theta} = B^{*}\bar{B}$: scalar gates sum features, shared beta sums erase/write paths, and grouped heads sum shared-input contributions.

If only fraction $f$ of baseline time changes with local speedup $s$, the ideal complete-operator ratio is $1/(1-f+f/s)$. The state scan, recomputation, transfers, and launches limit the measured gain; local product counts alone do not predict it.

Schedules and Numerical Contract

Joint consumers reduce transfers but extend live ranges; separate consumers shorten them and add scratch transfers. RTX 5070Ti KDA streams value tiles of width 64; its state scan maintains value groups of width 16 in FP32. A800 DeltaNet uses the original K64 state arithmetic with three stages, while GDN retains the original state autotuner. Their scalar consumers finish FP32 value/weight uses of $P$ before dependent temporaries, avoiding a global $P$ buffer. The references retain native autotuning and candidates keep fixed settings across shapes.

On Ascend 910C, GDN stores weighted residual and output coefficients in FP32. The matrix consumer computes query and key gradients, the erase-beta reduction, and the key endpoint. A separate Vector kernel adds the state endpoint and reverse-sums the gate gradient. This adds one FP32 scalar per token/head and one kernel boundary; complete timing includes both stages and transfers. Hopper and Ascend 950DT schedules are specified with their results in § 4.7 and § 4.6.

Reordered products change floating-point rounding. Centering and range checks constrain exponential magnitudes but do not bound the triangular system’s condition number. No general floating-point stability theorem is claimed.

Zero/tiny beta uses the undivided derivative; GDN2 retains independent erase/write gradients.

Protocol, Precision, and Validation

Frozen controls and measurement scope

The reference is FLA revision a59bc5df8c5841dce2169bb00e4027287197ba77. GPU controls are original. Ascend 910C adaptations retain the equations: an ungated DeltaNet schedule, stable independent key/transpose loads for GDN, and original GDN2 intra equations. The fragmented ungated DeltaNet reference duplicates work under a transpose-transfer compiler limitation and contributes to its larger gain. KDA uses its existing saving switch; other matched wrappers cache identical intermediates. One implementation is fixed per device/operator across six shapes and matched R/A policies.

Original GPU graph runs use six extra warm-up seconds and 30 balanced rounds (60 observations); Ascend 910C uses at least two seconds and ten rounds (20 observations). Each graph contains two calls and times are normalized per call. Per-shape medians are reduced by an equal-weight geometric mean of reference/optimized ratios. Gradients are checked before and after timing and on replay with intervening inputs. The Ascend 950DT public eager boundary, adaptations, and acceptance are separately specified in § 4.6; the Hopper protocol is in § 4.7. These cohorts are not pooled.

Precision eligibility

The dagger-marked complete entries are A800 KDA/GDN2, RTX 5070Ti GDN2, and Ascend 910C KDA/GDN2. Their measured matrix paths cast existing FP32 $L/D$ operands to BF16. The corresponding matrix-only and conditional controls, including the separate RTX KDA controls, have the same limitation. They remain historical timing/allocation records, not evidence for current precision-qualified claims. Residual-only controls and the RTX KDA joint measurement have separate eligibility. Later numerical passes or public-output casts do not restore internal precision.

At $B=1$, $T=2048$, $H=4$, historical conversion-inclusive KDA/GDN Ascend 910C controls give 1.27/1.34 and 1.41/1.61 R/A ratios, respectively. The KDA precision exclusion still applies.

Numerical coverage

The main harness uses feature-normalized Gaussian BF16 queries/keys, Gaussian values/output adjoints, query scale $d_k^{-1/2}$, and FP32 Gaussian initial/terminal states scaled by 0.1. Log gates are $\log(\sigma(x))/16$; erase/write weights lie in $[0.1, 0.9]$. CPU FP64 and complex-step checks cover multiple chunks, unequal widths, nonuniform partitions, nonzero state boundaries, shared gates, and both scalar endpoints. Accelerator extensions include tails, zero/tiny beta, strong decay, and applicable packed/grouped/layout interfaces. RTX KDA/GDN2 and A800 DeltaNet/GDN have recorded BF16 tests at widths 80/112 and 192/160 and FP16 tests at 96/80. Coverage is limited to the tested interfaces; it does not remove precision exclusions or establish convergence.

KDA determinism limitation. An A800 test at $T = 65$, $d_k = d_v = 64$, with every third beta zero, found beta-only repeat differences in 20 of 21 comparisons (maximum $1.49 \times10^{-8}$). The CPU tolerance check passed; the differences arise from unordered FP32 atomics in the fallback. Thus tolerance-based correctness does not imply general bitwise determinism.

Hopper validation

On each of H100 and H200, validation covers all four operators, six shapes, and both saving policies. Checks against CPU FP64 references and complete baseline gradients pass, as do graph-replay checks with intervening inputs. The implementations retain existing FP32 working matrices and accumulations. These operator checks do not establish full-model numerical acceptance or training convergence.

Allocation Definition and Controls

Incremental peaks include warmed backward tensors and returned gradients. Forward auxiliaries exclude inputs and forward outputs and are counted once per underlying storage. Saved bytes match within each policy. Resident workspace is separate: the historical KDA/Ascend 910C matrix path uses 16 MiB per stream. Table 12 and Table 13 retain the complete and component allocation values.

The two audited shapes are $(1, 2048, 4)$ and $(1, 8192, 16)$ in $(B,T,H)$ notation. Each uses both policies and three repetitions. Repeated peaks agree exactly in the recorded GPU controls, with complete-gradient checks. Allocation peaks depend on the tested schedule. Separate component controls cannot be substituted for the complete implementation’s allocation.

Numerical Values for the Result Figures

The tables report measured baselines and component ablations. Appendix § 12.§ 2 specifies precision exclusions. Each † marks the same qualification as in the corresponding figure.

Table 9–Table 13 list the original-cohort values displayed in Figure 3–Figure 6. Supplementary operator values are in Table 2 and Table 4.

Operator	Policy	B	FB
KDA	R	1.2177	1.1057
KDA	A	1.3031	1.1171
GDN	R	1.2192	1.1098
GDN2	R	2.4301	1.6432
GDN2	A	2.8945	1.7260
DeltaNet	R	1.0878	1.0693

Related Work

Delta-rule computation. The WY representation originated as a compact representation of Householder products [1]. DeltaNet uses related structure for parallel sequence

computation. GDN adds adaptive forgetting, KDA channel-wise decay, and GDN2 independent erase and write vectors [9, 11, 19, 21]. Our transformations preserve these operators and their forward computations while simplifying the backward graph.

Matrix derivatives and kernel implementations. The adjoint of a linear solve is classical [8]. ReWY applies it to the composite map $MR = Z - ES$ and eliminates the inverse-adjoint chain while retaining all gate and state derivatives. FLA and its Ascend adaptation provide the chunk operators used as frozen references [4, 7]. The audited FlashQLA revision includes a Hopper path that combines a residual right-hand side before forming the inverse adjoint, as well as scalar-key symmetrization [17]. ReWY further removes the two-sided inverse pullback. cuLA provides hardware-specialized linear-attention kernels [10]; a recent Blackwell study examines shared gated-recurrence backward computation with independent correctness verification [18].

Separable contractions. GLA derives scaled query and key factors and secondary chunking to balance matrix execution with numerical stability [20]. KDA further exploits tied key factors to simplify its chunkwise computation [11]. Our matrix-intra consumer uses these established structures in a whole-chunk backward organization with centering and explicit gate-gradient production; the separable exponential identity itself is not a novelty claim.

Intermediate elimination. FlashAttention reduces intermediate materialization and expresses a backward pairwise reduction as a feature-space inner product [2 Appendix B.2]. ReWY eliminates intermediate adjoints in a triangular system. Matrix intra and Gram elimination reorganize its remaining gated contractions and reductions.

Model Provenance and Numerical Scope

The full models use the frozen FLA revision identified in Appendix § 12. All have vocabulary 32,000; parameter counts include the complete instantiated language model.

KDA’s structure comes from FLA defaults [6]; GDN and DeltaNet use their published dimensions [3, 12]. The GDN2 adapter uses the native FLA layer [5] in a common language-model shell, with dimensions from the official model [14]. Its sigmoid output gate and shell initialization differ from the NVLabs full model. Matching dimensions is not architecture or initialization equivalence. Both compared cores use the same instantiated model; no row reproduces industrial pretrained weights.

Execution and numerical diagnostics

Runs use one device, native $R$, $C = 64$, and no gradient accumulation or layer checkpointing. Both sides use matched random initialization and inputs, BF16 autocast, and FP32 parameters, working matrices, and accumulations. A800 and Hopper record 60 event intervals per implementation across four fresh-model ABBA blocks; Ascend 910C records 30 synchronized eager intervals. Forward and optimizer work are excluded from B. Each ratio uses one physical device and a fixed backend; no case dispatcher selects the faster schedule.

KDA/A800 passes all 24 layer VJPs at the original thresholds: 0.8% for ordinary gradients, 2% for gates, and $10^{-6}$ near zero. With changing inputs, graph and eager executions agree elementwise within each algorithm. Across eight fixed-architecture inputs, initial full-network gradient differences are 1.62–1.64%; after 46 updates, parameter differences are 0.17–0.20% and loss differences at most 0.22%. These diagnostics do not establish every parameter’s accuracy or long-run convergence.

Ascend 910C maximum parameter-gradient differences are 1.94% (KDA), 2.60% (GDN), 2.25% (DeltaNet), and 2.17% (GDN2 port). Some exceed the original thresholds; the model records are performance-only and remain qualified accordingly. Operator correctness checks do not establish full-model numerical acceptance.

Hopper configurations and numerical scope

KDA keeps its architecture fixed and uses eight $(B,T)$ pairs:

Hopper GDN2 uses the official-dimension port with $B = 1$ at $T = 2048$ and $8192$. Hopper GDN and DeltaNet use their published structures at the same two lengths; the separate GDN joint-consumer point uses $B = 1, T = 2048$. Model structure and backend remain fixed within each group.

All eight KDA input configurations have local-VJP, parameter, and loss diagnostics. GDN, GDN2, and DeltaNet measurements are performance-only: finite-value and execution checks do not establish full-model gradient acceptance. DeltaNet trajectory checks can exceed the thresholds.

Changing head counts, feature widths, or loss heads alters model capacity or workload. Such comparisons are excluded from the full-model tables, which keep architecture and loss fixed and report backward only. Operator ratios do not imply a model or complete-training gain.