WorksComputer Science
CertiSum: Bitwise-Identical MXFP4 Training Across Platforms and Parallelism Layouts
Abstract
Training with MXFP4, a 4-bit microscaling format, is non-deterministic: identical inputs can produce different results across parallelism layouts and GPU vendors. These differences complicate debugging and training audits and can cause training and inference engines to disagree. We present CertiSum to make MXFP4 training bitwise invariant across runs, parallelism layouts, and GPU vendors. Its key insight is that native FP32 accumulation of MXFP4 products is often already exact for every summation order. CertiSum collects statistics during quantization to certify native matrix multiplication results, recomputes uncertified outputs exactly, and uses exact kernels directly where certification rarely succeeds. CertiSum keeps quantization blocks unchanged across layouts and merges partial sums without rounding, including in attention and weight-gradient computation. In 200-step Qwen3-8B-Base experiments, CertiSum preserves training states bit for bit across AMD MI350X and NVIDIA GB200 and across changes in data, tensor, and context parallelism on MI350X. Across three 8B–14B models on FineWeb-Edu using MI350X, it adds 11.8–60.5% to training-step latency relative to its non-exact MXFP4 counterpart at the larger evaluated input shapes, with speedups of up to over BF16 and over deterministic BF16.
Introduction & Motivation
The growing scale of large language models (LLMs) has made compute, memory, and energy major costs of training, motivating narrower numerical formats [11, 21] (NVIDIA et al. 2026). Recent developments in low-precision formats have allowed 4-bit floating-point (FP4) and its corresponding microscaling (MX) format, MXFP4, to serve production models such as OpenAI’s gpt-oss [47]. Compared to FP8, FP4 halves memory use and at least doubles matrix throughput on modern accelerators (NVIDIA et al., 2026). With the correct training recipe, MXFP4 has been shown to achieve model quality close to BF16 at a significantly lower estimated cost of the backward pass [60].
Like all floating-point (FP) training, low-precision training is affected by non-determinism. Reductions in machine learning (ML) frameworks can return bitwise-different outputs for the same inputs, for two reasons. First, floating-point addition is non-associative, so the result depends on the summation order, which is set by how each reduction is split across GPUs by data, tensor, and context parallelism (DP, TP, CP) [59, 35] (Li et al. 2020) and across streaming multiprocessors (SMs) within
each kernel. We call this partitioning the parallelism layout (§2.3). Atomic additions, common in the attention backward pass [52, 56], further make the order depend on thread scheduling. Second, the kernel that performs a given operation is itself not fixed. Autotuners and framework heuristics select among implementations based on shape, hardware, and timing measurements, and different implementations reduce in different orders.
Non-determinism undermines several common ML workflows. In reinforcement learning (RL), an inference engine samples tokens and a training engine independently computes their log-probabilities. The two engines often use different kernels and parallelism layouts [58, 41], so nominally on-policy training becomes off-policy [65]. Debugging a training run requires reproducing a step with its exact parallelism layout and kernel selection. Without determinism, discrepancies caused by non-deterministic execution are hard to distinguish from implementation errors [6], such as a numerically unstable algorithm (e.g., a naïve exponential in softmax). Non-determinism also weakens kernel testing. A kernel and its reference implementation sum in different orders, so tests must compare them within a tolerance, and such tolerances could accept many incorrect kernels [54, 57]. Training audits, which re-execute a published training run, possibly on different accelerators, to verify that the released weights follow from the claimed data and code, rely on reproducibility as well. Without it, a backdoor in the model may go undetected [19].
MXFP4 training is more sensitive to the parallelism layout than BF16 training. Figure 1 measures how an LLM’s output tokens and training loss change when training partitions the model across 4 GPUs instead of 2 (using tensor parallelism). Under MXFP4, more output tokens change than under BF16. The workflows above are therefore most affected in the low-precision formats that make training cheaper.
Figure 1. Qwen3-8B-Base [64] on FineWeb-Edu [50], trained on 4 GPUs instead of 2, all else identical. (a) Differing greedy next-token predictions before training. (b) Relative training-loss difference (centered 10-step mean). Repeating the run of each format on two GPUs reproduces the result bit for bit; CertiSum (§3) differs in neither predictions nor loss.
To address this, we propose CertiSum, a method that provides invariance, the property that the values an LLM workload produces depend only on the model and its inputs, not on how or where the computation executes. These values are the model outputs during inference and the losses, gradients, parameters, and optimizer states during training. We target three forms of invariance. Values must be identical across repeated runs with the same configuration (run-to-run invariance), across parallelism layouts and micro-batch counts (parallelization invariance), and across accelerators from different vendors (hardware invariance). Kernels may use any reduction strategy that yields invariant results. We aim to provide these invariances at low latency overhead relative to non-deterministic execution. At production scale, even small overheads are costly [23] (Meta 2024).
Prior work provides only a subset of these invariances, or provides all of them at high cost (Table 1). Order-fixing systems [6, 67, 30] (He & Thinking Machines Lab 2025; Li et al. 2023) provide run-to-run invariance but no hardware invariance across vendors. Most provide parallelization invariance only for specific layout changes (batch size, TP, or DP degree), and some only in the forward pass. Cross-hardware replay [19] provides hardware invariance at 5.2–6.8× the runtime of BF16 training. Order-free summation methods [5, 13, 28] provide all three invariances but with high overhead (up to 16× slower than a non-reproducible general matrix multiplication (GEMM) [28]), since they pay for exactness on every reduction. FoldAttention [1] avoids this overhead but covers only attention. Exact and emulated GEMMs either guarantee exactness only up to a reduction length fixed before runtime [24] or target FP64 accuracy rather than invariance [55]. No existing system provides all three invariances for MXFP4 training across DP, TP, and CP with low overhead.
| Run-to-run | Parallelization | Hardware | Order-free | Backward | Overhead (reported) | Precision (evaluated) | |||||
| Batch | DP | TP | CP | ||||||||
| Order-fixing | Megatron Core (Aithal et al., [6]) | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | 2% | FP8, FP4 | |
| Batch-inv. kernels (He & Thinking Machines Lab, 2025) | ✓ | ✓ | – | – | – | – | ✗ | ✗ | 1.6–2.1× | BF16 | |
| TBIK (Zhang et al., [67]) | ✓ | ✓ | G# | ✓ | – | – | ✗ | – | 22–63% | BF16 | |
| IsoExec (Jiang & SkyRL Team, [30]) | G# | ✓ | G# | ✓ | ✗ | – | ✗ | – | 25% | n/r | |
| EasyScale (Li et al., 2023) | ✓ | ✗ | ✓ | – | – | G#ᵃ | ✗ | ✓ | <1%ᵇ | n/r | |
| Cross-hardware | OPEN-1B (Donaghy et al., [19]) | ✓ | – | G#ᶜ | – | – | ✓ | ✗ | ✓ | 5.2–6.8× | INT8 + BF16 |
| RepDL (Xie et al., [63]) | G# | G# | – | – | – | G# | ✗ | G# | n/r | FP32 | |
| Order-free | ReproBLAS (Ahrens et al., [5]) | ✓ | G# | G# | G# | G# | G#ᵈ | ✓ | – | n/rᵉ | FP32, FP64 |
| ExBLAS (Collange et al., [13]) | ✓ | G# | G# | G# | G# | ✓ | ✓ | – | ≤16×ᶠ | FP64 | |
| FoldAttentionᵍ (Achanta, [1]) | ✓ | ✓ | – | – | – | – | G# | ✓ | ≈0% | BF16 | |
| Exact GEMM | AWE (Hayashi et al., [24]) | G# | – | – | – | – | G# | ✓ | – | n/r | INT8 on FP4 |
| ADP (Schwarz et al., [55]) | – | – | – | – | – | – | G# | – | <10%ʰ | FP64 on INT8 | |
| Ours (CertiSum) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 11.8–60.5% | MXFP4 |
Table 1. Comparison with prior work. A ✓ means guaranteed or supported, ◐ partial or argued only, ✗ not provided or out of scope, – not addressed, and n/r not reported. Overhead is latency relative to non-deterministic execution. ᵃ NVIDIA GPUs only. ᵇ 236% across GPU types (CNNs). ᶜ Claimed for any device count; no replay on a different device count is reported. ᵈ CPU only. ᵉ Latency overhead of 1.2–1.6× argued from previous work. ᶠ GEMM latency overhead (Iakymchuk et al., [28]). ᵍ Attention only. ʰ Targets accuracy, not reproducibility.
Achieving these invariances at low overhead is difficult. Fixing the summation order does not suffice for hardware invariance, because vendors round low-precision GEMM intermediates differently [62], and the same order then yields different bits. An exact summation is invariant to both order and vendor, but it costs more than native FP32 accumulation, and whether that cost is necessary depends on run-time values. MXFP4 GEMMs typically accumulate in FP32 [45, 53] and are exact only if every partial sum fits within the accumulator’s 24-bit significand and exponent range [24]. This depends on the operands’ block scales, the reduction length, and the hardware’s summation order. Exact methods that ignore run-time values must provision for the widest possible value spread [13], paying this cost on every reduction even when native FP32 accumulation would have been exact. Methods that inspect values avoid this cost, but read both input matrices in an extra pass before every GEMM [55].
Neither approach exploits the common case in which FP32’s significand is wide enough for exact accumulation (Figure 2). Our key idea is to certify this cheaply at run time, so that CertiSum keeps the output of the fast native MXFP4 GEMM and pays for exact computation only where certification fails or is not attempted. CertiSum evaluates increasingly expensive certificates, each a sufficient condition for exactness. The cheapest uses row statistics gathered in the same pass as quantization, so checking an output takes constant work independent of the reduction length. This works because MXFP4 elements have few significand bits under a shared per-block scale. On FineWeb-Edu, exact accumulation in any order needs at most 24 bits for 99.2% of GEMM outputs. CertiSum skips the native GEMM for attention and nearly all weight-gradient GEMMs and runs them directly on exact kernels, which account for 46–68% of training-step GEMM FLOPs (Appendix § 16).
Figure 2. Upper bound on the significant bits needed for exact MXFP4 accumulation in any order, for GEMM outputs over 200 steps of training Qwen3-8B-Base on FineWeb-Edu. The bound uses the sum of absolute products; the shaded panel magnifies the tail (log scale).
We make the following key contributions.
We identify hardware invariance as a requirement for reproducible MXFP4 training, which existing methods either do not provide or provide at several times the runtime of native execution.
We present CertiSum, a method that provides all three invariances for MXFP4 training with low overhead by certifying native MXFP4 GEMM results as exact at run time and computing the remaining outputs exactly.
We show that CertiSum produces bitwise-identical Qwen3-8B-Base training across DP, TP, and CP layouts and across AMD MI350X and NVIDIA GB200.
For Qwen3-8B-Base, Qwen3-14B-Base, and Llama-3.1-8B on FineWeb-Edu using AMD MI350X, it adds 11.8–60.5% latency over its non-exact MXFP4 counterpart at the larger evaluated input shapes, where it remains faster than our BF16 and torch titan baselines, and 15.4–77.3% at the smaller ones.
Background
Microscaling Formats
MXFP4, the 4-bit member of the Microscaling (MX) format family, stores each value in 4.25 bits on average, 3.8× fewer bits than BF16. This reduces arithmetic cost as well as the memory and bandwidth costs that limit LLM serving ([21]), while additional techniques mitigate model accuracy degradation ([60, 18, 20]). NVIDIA Blackwell and AMD MI350X GPUs support MXFP4 GEMMs natively ([7, 3]).
The OCP MX specification ([17]) defines block-scaled formats in which low-bit elements share one scale . Each element decodes to . MXFP4 uses , E2M1 elements, which take the values , and an E8M0 scale, which is a power of two. A block of wider-format values is quantized by choosing as a power of two from the block’s largest magnitude and rounding each to the nearest E2M1 value.
Computing , with MXFP4 matrices and , requires dot products of length , each between a row of and a row of . To prevent excessive loss of accuracy [61], the terms of these dot products are accumulated in a wider format, and the specification recommends an FP32 result [17]. The output is therefore not itself in MXFP4. Converting it to MXFP4 is a separate quantization step, needed only when the output is used in another MXFP4 operation [18].
The specification fixes the encodings and the mathematical value of a block dot product, but leaves the accumulation precision, rounding behavior, and order of operations to the implementation [17 b]. Two MX-compliant GEMM implementations can therefore return different bits for identical inputs.
Reduction Order and Exactness
FP addition is non-associative, with in general. This is because floating-point addition rounds each result, with , where is the unit roundoff [26]. The rounding error is set by how far the exact sum lies from the nearest representable value, so different operands and incur different rounding errors. Any change in the order of additions, which we call the reduction order, may therefore change the operation’s output bits.
The accumulation error of a reduction is given by , where is the computed sum and is the exact sum rounded once to the output format. Measuring against rather than excludes the unavoidable final rounding, so this error comes only from intermediate rounding.
The three invariances defined in §1 impose different requirements on a reduction. Run-to-run invariance requires only a repeatable order, which floating-point atomic additions break. Fixing one reduction order regardless of the parallelism layout (§ 2.3) achieves parallelization invariance; prior systems do so for changes in batch size or TP degree [67, 30] (He & Thinking Machines Lab 2025). The accumulation error is then fixed but can be nonzero, so implementations that fix different orders can disagree. Hardware invariance requires every implementation to produce the same bits. Exact accumulation, which sums without intermediate rounding in a sufficiently wide accumulator [24], achieves this for any order, because the correctly rounded sum is unique given identical terms and final rounding mode.
Parallelism Layouts
Distributed ML training partitions the model and inputs across GPUs, and most of these partitions split reductions and change their order. We define a parallelism layout as the set of factors, across GPUs and within each GPU, that determine the reduction order.
Across GPUs, each form of parallelism splits a reduction into per-device partial sums. Tensor parallelism (TP) splits GEMMs along their reduction dimension [59]. Data parallelism (DP) splits the batch-dimension gradient sum across model replicas [12], and micro-batching splits each replica’s share further. Context parallelism (CP) splits attention’s key/value reduction across sequence chunks, in both the forward and backward passes [35 ) ({xref}]§2.4). Changing the degree of any of these moves the split points and therefore changes the reduction order. A collective algorithm then sets the order in which GPUs combine their partial sums. Libraries such as NCCL choose it from message size, GPU count, and network topology (Hu et al., 2025), and rebuilding gradient buckets after a change in GPU count can change the gradient aggregation order (Li et al., 2023).
Within a GPU, kernels split reductions across streaming multiprocessors (SMs) for throughput, as in split-K GEMMs [48] (He & Thinking Machines Lab 2025) and blockwise attention [16 ) ({xref}]§2.4). Kernels choose this split based on the GEMM and micro-batch sizes.
Blockwise Attention
Besides GEMMs, attention is the other dominant reduction in an LLM. Efficient implementations such as FlashAttention compute (omitting scaling by ) blockwise, processing keys in blocks without materializing the full score matrix [15, 35]. For each query row, attention tracks the running maximum of the scores seen so far. It forms , where , and accumulates the unnormalized output and the row sum , dividing by once at the end [14]. Whenever a new key block raises this running maximum, the partial and are rescaled by [14].
Softmax normalizes each query row over all keys, and the output sums the corresponding values, so every accumulated quantity is a reduction over the key/value dimension. Attention’s bitwise output therefore depends on where the key/value dimension is split, whether at kernel block boundaries, at CP-chunk boundaries, or at split-KV boundaries if used [16] (He & Thinking Machines Lab 2025). The order in which the kernel visits blocks also affects the bitwise output.
In the backward pass, sums over key blocks and , sum over query blocks [14]. Its reduction order therefore depends on both the query and key/value partitioning. Under CP, it also depends on the chunk layout and ring order [35]. Kernels that accumulate with atomic additions also make its order run-dependent [14] (He & Thinking Machines Lab 2025).
CertiSum
We introduce CertiSum, a method for exact accumulation of MXFP4 values and their products without prescribing a reduction order. We first explain certification and exact computation, then establish the conditions under which exact merging preserves results across execution layouts, using GEMM and attention as applications. Figure 3 gives an overview of how CertiSum executes a GEMM.
Figure 3. CertiSum GEMM execution; steps ❶–❻ described in §4.
Design Challenge ❶: cheap certification. Exactness depends on the current operand values and block scales. Checking individual product magnitudes would require another reduction for every output. We need reusable input statistics that can certify many outputs without recomputing their dot products.
Design Challenge ❷: soundness and layout invariance. A certificate must cover every reduction order and remain sound when its own evaluation uses rounded arithmetic. Across devices, exact local sums are insufficient if repartitioning changes MX blocks or if partial sums are rounded before merging. The mechanism must preserve both the reduction terms and their exact partial sums.
A Sufficient Condition for Native Exactness
We write GEMM as , where and contain the decoded MXFP4 inputs. For output position , let and be the corresponding input rows, so that and . Their exact dot product is . However an output is computed, CertiSum returns , or for BF16 output, where denotes round-to-nearest-even in format as defined by IEEE 754 (IEEE, 2019). These targets refer to the quantized operands; exact zero is returned as . We call a native MXFP4 kernel with FP32 accumulation natively exact at if its output equals this target for every order and grouping of products and additions it may use.
To bound the precision required by every reduction order, express each row in a common power-of-two unit. For a nonzero row , let be half the smallest MX scale among blocks containing nonzero values, and define analogously. Then and for integers . Zero rows return zero directly. With , the dot product is . We seek a conservative upper bound satisfying
This condition bounds every partial sum, including those formed before cancellation. For any subset , . Each partial sum therefore needs at most 24 significant bits in the common unit. Range checks on the unit , which we call exponent guards, keep nonzero products and partial sums normal and finite in FP32. We assume native kernels that initialize each dot-product accumulator to zero, return exactly representable FP32 products and sums unchanged, and implement the prescribed final conversion without intermediate BF16 rounding. Under these conditions, every product and addition is exact regardless of the reduction tree; this is the error-free condition of Ozaki-type schemes [49, 24], here checked at run time. Appendix B.1 gives the exponent guards and proves native exactness under these conditions.
Evaluating Certificates from Reusable Statistics
To check Equation (1) at a cost independent of the reduction length , instead of computing the products of every output, CertiSum collects row statistics during quantization. For either integer row , these are , , and . These statistics summarize each entire input row and give the following row bound (Figure 4, ❶) for the selected output:
Figure 4. Certificates from row (❶), block (❷), and absolute-product (❸) statistics, in increasing cost and tightness.
The first term bounds every by ; the second exchanges the rows; the third follows from Cauchy–Schwarz. Componentwise maxima of the statistics over groups of rows can certify several outputs together.
When the row bound fails to certify an output, CertiSum tries two tighter but more costly bounds (Figure 4, ❷ and ❸). Block bounds restrict each of the three expressions in Equation (2) to 32-element blocks along . CertiSum sums each expression over blocks, then takes the minimum of the three sums (Appendix § 11). Across all outputs, each sum is a matrix multiplication with reduction length . The tight bound evaluates using a native MXFP4 GEMM on sign-cleared operands with unchanged scales. If all three bounds fail, CertiSum repairs the output (§ 4). For shapes that rarely certify, CertiSum skips the native GEMM and computes all outputs exactly (§ 5).
The row, block, and tight bounds are themselves computed with rounded arithmetic, so CertiSum evaluates each one conservatively, accounting for rounding of the statistics, products, accumulation, and output conversion. In particular, the native MXFP4 GEMM that evaluates the tight bound may itself round, so its verified error allowance must still produce an upper bound . Appendix § 12.§ 2 proves the bound hierarchy and derives sufficient rounding allowances under explicit kernel assumptions.
Repair and Exact Merging
When an output fails certification, where its exactness is unsure, CertiSum recomputes it exactly from the same quantized operands. It has three ways to do so. The first splits the reduction into chunks short enough to pass the native certificate, sums each chunk with the native kernel, and merges the chunk sums exactly, as described below. The second accumulates the products in FP64, as in emulated GEMMs (Ozaki et al., [49]; Schwarz et al., [55]), provided a capacity check shows that no partial sum can round. The third, the integer path, accumulates in multi-word integers, a superaccumulator (Kulisch, [31]; Iakymchuk et al., [28]), and is the fallback when the other two fail their checks (Appendix § 12.§ 3). Direct exact execution uses the same three methods.
Exact merging combines partial sums from different chunks or devices without rounding, so the result does not depend on how the reduction was split. Formally, let disjoint index sets partition a dot product, and let be the exact partial sum of partition . After aligning units by exact power-of-two shifts, CertiSum computes
A capacity check verifies that stays below for a signed int32 accumulator or for an FP64 accumulator. Every partial sum of the then fits in the accumulator, so these additions do not round. Integer addition is associative and commutative, so the result is the same for every partition size, merge order, and reduction tree, the argument behind long-accumulator and reproducible summation (Kulisch, [31]; Collange et al., [13]; Ahrens et al., [5]). Partitions therefore keep their exact partial values until the merge is complete. Rounding a partial sum to the output format first could lose bits that the final result needs. Appendix § 12.§ 3 gives sufficient conditions for exact repair and proves partition invariance.
Preserving Reductions Across Layouts
Exact merging requires every layout to sum the same products . These products would otherwise depend on the parallelism layout: the MX block an element belongs to sets its scale, and blocks formed within a device’s shard change with the shard. CertiSum therefore assigns elements to MX blocks using their positions in the full tensor, not their positions within each device’s shard, including the MX blocks along the token dimension used to quantize weight-gradient operands in MX training recipes (Darvish Rouhani et al., [18]). Tensor-parallel shards and data-parallel or microbatch contributions to a weight gradient are then partial sums over disjoint parts of one reduction. CertiSum keeps each exact and adds them without rounding (§ 4), scaling every contribution by the same global loss normalization.
Attention (§ 2.4) also merges partial sums across partitions, and it ends with a division by a sum, so exact merging must cover both the numerator and the denominator. Query–key scores use the certified or exact dot products above. Unnormalized attention weights and values are quantized in MX blocks fixed by global key and token positions, so splitting keys across devices or kernel blocks yields the same quantized weights (Appendix § 11). To bound the range of retained attention weights, CertiSum prunes key blocks whose scale falls too far below a fixed per-row reference. The reference does not depend on the key partition, so neither does pruning; FoldAttention (Achanta, [1]) fixes a softmax reference for the same reason.
For query and key , let and denote the resulting quantized unnormalized attention weight and value vector. Masked or pruned keys have . CertiSum computes the attention-weight sum and the -weighted sum of value vectors exactly. It obtains the attention output vector by rounding componentwise to FP32:
Key partitions merge their numerator and denominator sums exactly, then round the exact quotient componentwise, the split-key decomposition of blockwise attention [16, 35] with exact instead of rounded merging. Backward contributions use the same exact merge rule.
Exact GEMMs and attention alone do not make a whole training step invariant. Every layout must also see the same batch, start from the same parameters and optimizer state, and use the same random choices, masks, loss normalization, and rounding points. Deterministic training frameworks already fix the batch order, the state, and the random choices [6] (Li et al. 2023). Operators outside these reductions must implement identical numerical maps across supported hardware. CertiSum commits an update only after every required output satisfies its numerical contract and every failed range or capacity check on any GPU has been resolved by repair or recomputation. Under the conditions above, any two accepted executions of the same logical step produce identical parameter and optimizer-state bits. Appendix § 12.§ 5 proves the attention and training-step guarantees.
Implementation Details
For each GEMM, CertiSum chooses between native execution with certification and direct exact computation. Both routes return the same target (§ 3.1); only the cost differs. This section walks through the numbered steps of Figure 3.
Operand preparation ❶. To reuse the operand reads needed for quantization, activation quantization produces the 4-bit MXFP4 elements, block scales, and reusable certificate statistics together. Weights change only at optimizer updates, so CertiSum caches their statistics between updates. The decoded quantized operands are and ; and denote their inputs before quantization.
Route selection ❷. Requiring a full native GEMM before every exact computation would add work even when direct exact execution is preferable. Tensor shapes and previous certificate outcomes guide the choice between native execution with certification ❸ and direct entry into the exact paths ❹ or ❺. This choice determines which work is attempted; acceptance still requires checks on the current operands.
Native execution and certification ❸. Outputs left uncertified by the selected checks of § 3.2 enter the exact paths ❹ or ❺ for repair; failure to certify does not establish that a value is incorrect. To avoid a CPU routing decision before repair, CertiSum supports in-stream repair, whose kernels read flags stored on the GPU and return immediately when no repair is needed.
Exact computation ❹–❺. These paths implement the repair methods of § 4, with chunked native accumulation and FP64 accumulation in ❹ and the integer path in ❺, which routing can also select directly. The certified or exact result supplies the GEMM output ❻.
Evaluation
Experimental Setup
Hardware platforms. We evaluate CertiSum on the two hardware platforms listed in Table 2. Both support MXFP4 GEMMs natively in their matrix units.
| Platform | GPUs | HBM3E/GPU | Used in |
| AMD Instinct MI350X [3] | 1, 2, 4 | 288 GB | all |
| NVIDIA GB200 (NVIDIA, 2025) | 1, 2, 4 | 186 GB | §5.2 |
Table 2. Hardware platforms.
Datasets. We train on FineWeb-Edu [50], a corpus of educational web text, and The Stack v3 [38, 37], a source-code corpus. FineWeb-Edu inputs come from the sample-10BT subset.
Models. We train the base models Qwen3-8B-Base, Qwen3-14B-Base [64], and Llama-3.1-8B [23]. The inference experiments in § 6.4 use post-trained Qwen3-8B [64].
Parallelism layouts. A suffix gives the number of GPUs. For example, TP2 denotes two-way tensor parallelism.
Input shapes. We denote input shapes as , where is the number of sequences and is the tokens per sequence. Thus, 4 × 4096 denotes four sequences of 4,096 tokens each. Shapes are global unless explicitly marked per GPU, as in the DP4 rows of Table 5.
| Qwen3-8B-Base | Qwen3-14B-Base | Llama-3.1-8B | |||||||||||||
| Layout | (ms) | (ms) | (ms) | (ms) | (ms) | (ms) | |||||||||
| 1 GPU | 823.1 | 1.57 | 0.88 | 1112.5 | 0.74 | OOM | 763.7 | 1.49 | 0.83 | 1079.0 | 0.71 | ||||
| 1 GPU | 1042.7 | 1.46 | 0.88 | 1715.7 | 0.61 | OOM | 931.9 | 1.41 | 0.81 | 1587.4 | 0.59 | ||||
| TP2 | 816.4 | 1.20 | 0.90 | 976.9 | 0.84 | 1345.4 | 1.40 | 0.95 | 1512.5 | 0.89 | 749.2 | 1.23 | 0.90 | 895.9 | 0.84 |
| TP4 | 562.3 | 1.12 | 0.88 | 825.1 | 0.68 | 992.3 | 1.38 | 1.00 | 1204.5 | 0.82 | 495.7 | 1.12 | 0.86 | 744.4 | 0.67 |
| CP2 | 911.1 | 1.27 | 0.94 | 1137.5 | 0.80 | 1625.9 | 1.54 | 0.86 | 2058.1 | 0.79 | 823.9 | 1.24 | 0.90 | 1105.4 | 0.75 |
| CP4 | 540.4 | 1.39 | 0.91 | 681.0 | 0.79 | 928.5 | 1.60 | 0.91 | 1123.6 | 0.83 | 475.6 | 1.33 | 0.83 | 648.2 | 0.73 |
| DP4 | 1024.3 | 1.16 | 0.97 | 1298.3 | 0.79 | 1726.8 | 1.31 | 0.89 | 2261.7 | 0.76 | 928.8 | 1.12 | 0.89 | 1250.5 | 0.74 |
| 1 GPU | 312.5 | 1.48 | 1.01 | 359.1 | 0.87 | 549.4 | 1.55 | 0.98 | 638.1 | 0.86 | 281.6 | 1.40 | 0.89 | 346.2 | 0.81 |
| TP2 | 321.2 | 1.43 | 1.13 | 310.8 | 1.03 | 597.9 | 1.77 | 1.35 | 467.3 | 1.28 | 301.7 | 1.45 | 1.13 | 285.1 | 1.06 |
| TP4 | 221.1 | 1.28 | 1.05 | 214.7 | 1.03 | 412.3 | 1.72 | 1.36 | 308.4 | 1.34 | 207.1 | 1.38 | 1.12 | 190.5 | 1.09 |
| CP2 | 516.5 | 1.20 | 0.80 | 690.4 | 0.75 | 961.8 | 1.52 | 0.74 | 1371.1 | 0.70 | 484.9 | 1.20 | 0.80 | 661.9 | 0.73 |
| CP4 | 365.0 | 1.15 | 0.84 | 451.3 | 0.81 | 589.8 | 1.44 | 0.82 | 753.0 | 0.78 | 376.1 | 1.36 | 0.92 | 428.0 | 0.88 |
| DP4 | 450.4 | 1.33 | 1.13 | 470.7 | 0.96 | 699.5 | 1.49 | 1.00 | 807.6 | 0.87 | 384.6 | 1.24 | 0.99 | 447.1 | 0.86 |
Table 5. Training-step latency on AMD MI350X with FineWeb-Edu inputs. Implementations , , , and as in Table 3. Ratios below one favor CertiSum. : sequences tokens, global except DP4 (per GPU). Static-route ; unmarked and all are eager. OOM: all compared implementations ran out of memory before completing the first step.
Implementations. Table 3 lists the compared implementations and the labels that § 6.5 uses for them.
| Label | Implementation |
| E | CertiSum; its native MXFP4 GEMM path uses AITER (Advanced Micro Devices, [2]) on MI350X. At the smaller input shapes, the faster measured result of ordinary (eager) execution and a static-route variant that replays recorded GPU operations. |
| T | MXFP4 twin: CertiSum’s structure with non-exact MXFP4 GEMMs and non-exact versions of our attention kernels. |
| F | BF16: the faster measured result of our BF16 implementation and torchtitan (Liang et al., [34]). |
| D | F in PyTorch deterministic mode (PyTorch Contributors, [51]); deterministic torchtitan uses SDPA attention. |
| Vendor | Native MXFP4 GEMMs through hipBLASLt (Advanced Micro Devices, [4]) with our pinned Tensile library on MI350X and cuBLASLt (NVIDIA, [44]) on GB200, with BF16 FlashAttention on dequantized MXFP4 inputs (AITER on MI350X, FlashAttention4 (Zadouri et al., [66]) on GB200). |
Table 3. Compared implementations.
Appendix § 13 gives further experimental details.
Cross-Platform Invariance
To test whether CertiSum preserves a training trajectory across GPU vendors and GPU counts, we train Qwen3-8B-Base on FineWeb-Edu and The Stack v3 for 200 AdamW steps [36]. Each step uses 1 × 4096.
We run on AMD MI350X and NVIDIA GB200 using one GPU and TP4 on each platform. In this experiment, TP4 divides each GEMM’s output rows and the attention heads among four GPUs and splits no reduction across GPUs.
We compare CertiSum with the vendor baseline of Table 3. All runs start from identical model and optimizer states and consume identical token batches in the same order.
Figure 5 shows training losses above and cross-vendor relative loss differences below. For each method, the latter is , where is the loss at the corresponding training step. All curves are unsmoothed.
Figure 5. Cross-Platform Invariance (MI350X vs. GB200). (a.1), (a.2): FineWeb-Edu training loss on one GPU for the vendor baseline (Table 3) and CertiSum. (a)–(d): relative cross-vendor loss difference on a log scale; exact zeros below the axis break; triangles mark differences below 0.001%.
As shown in Figure 5, CertiSum produces bitwise-identical losses on AMD and NVIDIA at all 200 steps, with one GPU and TP4, on both datasets. Per-step tensor fingerprints and final parameter and optimizer-state hashes match as well, so the whole training state is invariant, not just the loss.
The baseline’s AMD and NVIDIA losses, in contrast, differ from step 0 onward. Median differences are 0.75% and 1.19% on FineWeb-Edu and The Stack v3, respectively, reaching 13.34% and 20.10%.
Under shared numerical definitions, CertiSum therefore trains bitwise identically on both vendors and both GPU counts, satisfying the hardware invariance defined in §1.
Elastic Training
To test whether CertiSum preserves the training trajectory across changes in parallelism layout, we change the GPU count and layout during a run. We reuse the training setup of §5.2 on AMD MI350X GPUs, with a global batch of . Each elastic run resumes from checkpoints between
40-step segments, following 1 GPU DP4 CP2 TP4 DP2. Controls checkpoint and resume at the same boundaries while remaining on one GPU.
Figure 6 compares CertiSum and its non-exact MXFP4 twin with their respective uninterrupted one-GPU references. It plots relative loss difference against training step. Certi-
Figure 6. Elastic Training. Relative loss difference from the uninterrupted one-GPU reference on (a) FineWeb-Edu and (b) The Stack v3. Dots are steps; lines are a centered nine-step geometric mean of positive values; zeros are below the axis break.
Sum matches its reference’s loss bits at all 200 steps on both datasets. Its checkpointed parameters and optimizer states match the one-GPU control bit for bit. The non-exact MXFP4 twin’s losses, in contrast, diverge at step 41, after the first layout change, and remain different thereafter, reaching 4.2% on FineWeb-Edu and 3.0% on The Stack v3. Both methods’ one-GPU controls reproduce the uninterrupted loss bits, so checkpointing alone is not the cause.
Resizing across one GPU, DP, CP, and TP leaves CertiSum’s trajectory untouched, so parallelization invariance holds.
Inference Invariance Across Batching and Layouts
To test whether batching and GPU layout change the tokens generated during inference, we use 32 general-knowledge questions. These questions were generated with Claude Code and are listed verbatim in Appendix § 14. We greedily generate 64 tokens per question using post-trained Qwen3-8B on AMD MI350X GPUs. We run each question alone on one GPU, in a batch of four on one GPU, and in the same batch with TP4.
| Method | Batch 4, 1 GPU | Batch 4, TP4 |
| CertiSum | 32/32 | 32/32 |
| MXFP4 twin | 31/32 | 0/32 |
| Deterministic BF16 | 29/32 | 25/32 |
Table 4. Generated sequences (out of 32) identical to each method’s batch-1, one-GPU reference.
In both batched configurations, CertiSum generates the same 32 sequences as its one-GPU reference, and all 2,048 FP32 log-probability vectors match bit for bit. All three methods reproduce their batch-1 tokens and log-probabilities on repetition, yet deterministic BF16 changes 3 and 7 sequences and the twin 1 and 32, respectively.
Batch composition and TP degree do not change CertiSum’s generated tokens or log-probabilities, so parallelization invariance extends to inference.
Overhead
To measure the latency overhead of CertiSum, we time training steps of the three models using one to four AMD MI350X GPUs on FineWeb-Edu, comparing implementations , , , and of Table 3. We report the median over runs of the mean latency of steps 3–29, taking the slowest GPU at each step; Appendix § 15 gives the run counts and exclusion rule.
Figure 7 plots the results; Table 5 in Appendix § 15 reports and in ms and the ratios , , and for 37 timed model, layout, and input-shape configurations. The main shapes are the larger ones ( and ). CertiSum adds 11.8–60.5% latency over its MXFP4 twin at the main shapes and 15.4–77.3% at the smaller shapes.
Figure 7. Training-step latency on FineWeb-Edu using AMD MI350X, normalized to the MXFP4 twin (dashed line); lower is better. BF16 bars use the faster measured implementation per mode (Table 3). Shapes are sequences tokens, global except DP4 (per GPU). Superscript s: static-route CertiSum; OOM: all implementations ran out of memory.
At the main shapes, CertiSum has lower measured median latency than both ordinary BF16 references. Its latency is 0.81–1.00× that of the faster reference, corresponding to a speedup of up to 1.24×. Qwen3-14B-Base TP4 nearly matches this reference (992.3 versus 994.6 ms). Against the faster deterministic BF16 reference, its latency is 0.59–0.89×, corresponding to a speedup of up to 1.70×.
For smaller inputs, we observe that the advantage over BF16 can reverse. This is likely because certification and routing add kernel launches and host work with a fixed cost per GEMM. As the GEMMs shrink, this cost becomes a larger share of the step. All six smaller TP configurations take 1.05–1.36× the faster ordinary BF16 reference’s latency. In deterministic mode, the range is 1.03–1.34×. All six smaller CP configurations retain lower latency, at 0.74–0.92× and 0.70–0.88× the respective references. Qwen3-8B-Base on one GPU and with DP4 also takes longer than the faster ordinary BF16 reference at the smaller shapes, at 1.01× and 1.13×, but remains faster than deterministic BF16.
On FineWeb-Edu using AMD MI350X, CertiSum is therefore faster than the measured BF16 references at the main input shapes that fit in memory but not at the smaller TP shapes.
Runtime Breakdown
To identify how the training workload affects CertiSum’s runtime, we profile Qwen3-8B-Base, Llama-3.1-8B, and Qwen3-14B-Base on FineWeb-Edu and The Stack v3. For each model, we hold the GPU layout, batch size, and sequence length fixed across datasets. We compare CertiSum with its non-exact MXFP4 twin and our BF16 baseline.
Figure 8 shows traced time per step in seconds by runtime component for each model. These durations include profiling overhead; § 6.5 reports untraced performance.
Figure 8. Traced step time on four AMD MI350X GPUs (DP4, per GPU) on (a) FineWeb-Edu and (b) The Stack v3; bars are the MXFP4 twin, CertiSum, and BF16. Each bar is the slowest GPU, averaged over steps 3–28 of one run. Compute segments are scaled from GPU-averaged kernel sums; crosshatching marks certification, repair, and exact GEMMs; communication excludes overlap; unfilled segments are host/launch gaps and logging.
Moving from FineWeb-Edu to The Stack v3 increases CertiSum’s traced step time by 16–22%, while the twin and BF16 change by less than 0.5%. Within CertiSum, certification, repair, and exact GEMM computation show the largest component increase for every model. In the underlying traces, this group accounts for 83–88% of the increase in summed compute-kernel duration. This percentage describes kernel work before accounting for overlap.
Discussion
Other low-precision formats. We rely on properties of block-scaled formats rather than of MXFP4 specifically: quantized elements are integers in a common power-of-two unit, a bound on their products certifies native accumulation, and exact paths merge partial sums without rounding. MXFP6 (Darvish Rouhani et al., [17]) keeps the power-of-two block scales of MXFP4 and therefore this integer representation. NVFP4 (Alvarez et al., [7]) preserves it as well: an E4M3 block scale is a small integer times a power of two, and the FP32 tensor scale, if computed from the whole tensor, is a common factor that can be applied once to the exact sum. Both would need format-specific bounds and kernels. MXFP8 fits the same design, but its wider elements make native certification succeed less often and shift more work to the exact paths.
Integration into existing frameworks. CertiSum replaces a framework’s GEMM, attention, and gradient-collective operators, their autograd definitions, and the end-of-step update commit (§ 4.1); frameworks with custom operators and autograd functions, such as PyTorch, support these replacements, and their remaining operators must meet the conditions stated there. RL systems that train and sample in different frameworks must install it in both, and audits must record that it was enabled.
Related Work
Table 1 compares CertiSum with prior systems; this section extends the discussion of § 1. Order-fixing systems ([12, 10, 22]) produce identical results only if the order of reductions is fixed, so none makes training invariant to a change of the TP, DP, or CP degree. Verde (Arun et al., [8]) fixes reduction orders across hardware but assumes consistent rounding, which low-precision FP lacks ([62]), and does not cover inter-GPU reductions. Order-free summation avoids fixed ordering by using wide accumulation, in software ([13] (Yakimchuk et al. 2016)) or with hardware support ([39]). Cross-hardware reproduction systems reproduce a given GPU workload deterministically on other hardware, such as CPUs ([9, 19, 63]), but do not provide parallelization invariance on their own. Exact GEMMs have precedent in GEMM emulation ([49, 55, 24]), which splits operands into low-precision components whose products fast matrix routines compute without rounding. AWE (Hayashi et al., [24]) fixes its encoding offline and must therefore provision for worst-case input magnitudes, whereas CertiSum certifies exactness from the run-time operands (§ 3). ADP (Schwarz et al., [55]) also inspects run-time inputs, but chooses its slice count, scaling, and FP64 fallback per call from that call’s operands, so partitioning a reduction differently can change the output bits.
Discussion
Other low-precision formats. We rely on properties of block-scaled formats rather than of MXFP4 specifically:
Conclusion
In this work, we present CertiSum for bitwise-invariant MXFP4 training across runs, parallelism layouts, and GPU vendors. It certifies when native FP32 accumulation of MXFP4 products is exact for every summation order and uses exact kernels where needed. Fixed quantization blocks and exact merging preserve reduction results across layouts. CertiSum preserves training states bit for bit across AMD and NVIDIA GPUs and through DP, TP, and CP changes, while often remaining faster than BF16 training. CertiSum thus lets low-precision training move across GPU counts, parallelism layouts, and vendors without changing its results, enabling exact reproduction for debugging, matching training and inference engines in RL, and training audits on different accelerators.
AI Usage Disclosure
The authors used ChatGPT and Claude through their web interfaces, and Claude Code through both Claude Desktop and the command-line interface. Claude Code assisted with implementation and kernel optimization, and the authors reviewed the generated code. AI tools assisted with the implementation of plotting scripts for figures in the introduction and evaluation. The authors checked these scripts for correctness. The authors conducted the literature search, then used Claude and ChatGPT to check for omissions. The authors wrote the text in the paper and used AI tools to edit it. The authors developed research ideas and experimental design without AI assistance. The authors used Claude Code to generate the 32 general-knowledge questions used for the inference experiments. AI use for data generation was limited to these questions, which are listed verbatim in the appendix.
Mechanism Details
This section details two mechanisms that § 3 summarizes.
Block bounds. Block bounds restrict the row statistics of § 3.2 to each 32-element block along , giving , , and for . Each term of Equation (2), summed over blocks, bounds ; for example, .
Attention quantization. Unnormalized attention weights are quantized in fixed global 32-key blocks, each scaled from its valid scores. To bound how small retained weights can be, CertiSum prunes blocks relative to a fixed row reference, which is the larger scale exponent of the first and query-containing global blocks. This reference fixes rescaling and pruning across key partitions. Value quantization follows global token positions. Backward quantization uses fixed query–key tiles and shares the quantized score gradient between query and key gradients.
Exactness and Invariance Guarantees
We prove the guarantees used in § 3, with its notation , , , , and . Inputs have non-NaN MX scales and finite FP32-representable decoded values; zero rows return . All statements concern the quantized operands. Acceptance requires the stated kernel, range, and capacity checks.
Native Exactness
E2M1 elements are integer multiples of with magnitude at most six in the OCP MX specification ([17]). Since MX scales are powers of two, choosing half the smallest nonzero-block scale as each row’s unit makes , integers. Thus every product and partial sum belongs to . The sufficient condition below is the error-free argument underlying the Ozaki scheme ([49]), which fixes slice widths in advance so that every slice product is exact; CertiSum instead checks a data-dependent bound for each output at run time.
Proposition 1 (Native exactness). Suppose the native kernel starts at zero, returns the exact result of every product and of every addition it performs, whether a binary addition or a fused addition of several aligned terms, whenever that result is representable in FP32, and performs no intermediate BF16 conversion. Sufficient conditions for exact accumulation under every reduction tree are
The kernel must implement the prescribed final conversion and return for exact zero.
Proof. Each tree node sums a subset of the products and has exact value , where and . It therefore has at most 24 significant binary digits. If nonzero, its magnitude is at least and at most , so it is normal and finite in FP32 under IEEE 754 (IEEE, 2019). Every leaf product is representable, and induction over the kernel’s additions, binary or fused, proves that every node is computed exactly. The prescribed conversion then gives c32 or c16.
These are sufficient conditions, not necessary ones. Bounding only the final sum would be unsound, because cancellation can make it small after an earlier partial sum has already rounded. The kernel contract is a separate requirement from MX format compliance. It is stated for fused additions because tensor cores typically align several products and the accumulator to a common exponent and add them in one step inside an adder of limited width. Such an adder satisfies the contract only if it keeps at least 24 significant bits below the largest aligned exponent; otherwise an exactly representable sum can still lose low-order bits. We rely on this property and treat it as a per-kernel assumption to validate on each target platform.
Sound Certificate Evaluation
To justify refinement, let be the th 32-element block, and use the block statistics of Appendix A. The block and tight bounds are
Then . To prove the first inequality, apply , its symmetric counterpart, and Cauchy–Schwarz on each block, then sum. For the second, whole-row maxima dominate block maxima, and Cauchy–Schwarz across blocks gives . Each blockwise sum is bounded by its corresponding row expression; taking minima proves the result. Replacing the statistics of one operand by componentwise maxima over a row group gives an upper bound for every member. The exponent guards must hold for every member too.
Rounding allowance. Mathematical bounds become certificates only when their computed values remain upper bounds. Consider a dot product of nonnegative inputs, supplied exactly or enclosed upward, evaluated as . Assume every rounding satisfies for its exact nonnegative input , where , with no underflow or overflow, as in the standard model of floating-point arithmetic ([26]). If each summand encounters at most roundings, including product formation and internal merges, expanding the reduction gives
Here RU denotes an upper enclosure of the entire expression. For normal results with significant bits, nearest rounding permits ; truncation toward zero permits .
Row statistics, square roots, and subsequent products must likewise be enclosed upward, using directed rounding or a justified error allowance, as in verified floating-point computation ([42]). The row test can avoid the square root by testing an upper bound on against . Block tests apply Equation (7) to their nonnegative reductions. The minimum of independently sound upper bounds remains sound, although rounding allowances need not preserve the bounds’ refinement order.
For the tight test, the exact physical sum is , and every term is nonnegative. This test needs one rounding allowance, a factor for the BF16 conversion of the result. Suppose the kernel satisfies the contract of Proposition 1 and rounds inexact results monotonically, as round-to-nearest and truncation both do, and let be its computed result after BF16 conversion, if the kernel converts. If , Proposition 1 applies to the sign-cleared operands under the guards of Equation (5), so the FP32 sum is exactly. Then , since BF16 keeps eight significant bits. If , some partial sum reaches or exceeds , which is a power of two and therefore representable in FP32 and in BF16; adding further nonnegative terms and rounding monotonically can never bring the computed value below it, so . The test therefore passes only if , and when it passes, . The threshold is finite under the exponent guards and is formed by checked exponent arithmetic. Non-finite evaluations cannot certify an output.
Exact Repair and Merging
Repair must preserve information that native accumulation might lose, as in exact summation with a long accumulator ([31, 13]). Partition a dot product into chunks with exact sums , certified using their local units and guards. Express each chunk sum in the original row-product unit as , . Nonzero chunks rebase by exact integer left shifts; zero chunks contribute zero. Sufficient capacity checks for arbitrary merge orders are
Every subset sum then fits exactly in the selected representation. With exact rebasing and checked scaling, induction over merges preserves . Guarded FP64 accumulation of individual products follows the same argument with one exact product per chunk. A wider format alone is not a proof of exactness.
Integer fallback. Write each E2M1 element as , where and . For a jointly nonzero block, let , and let be the ratio of its block-product unit to the row-product unit . We have , and the E8M0 exponent range of the OCP MX specification (Darvish Rouhani et al., [17]) gives . Hence
The sums range over jointly nonzero blocks. Under the checked condition , 18 radix- limbs provide sufficient capacity, as in the superaccumulator of ExBLAS ([28, 13]). Splitting each shifted term into signed digits of magnitude below keeps every unpropagated limb below in magnitude; checked carry propagation fits int64. Finalization rounds directly to FP32 using its retained bits, then to BF16 if requested. An intermediate conversion that discards low bits is not permitted. Invalid inputs, exceeded capacity, or unsupported output range require escalation or refusal.
Proposition 2 (Partition invariance). If disjoint sets cover a reduction of the same quantized operands, expressed in a common unit, exactly once, exact rebasing and merging of produce the same prescribed output for every partition and merge tree within checked capacity.
Proof. Integer addition is associative and commutative, so every merge yields . Exact rebasing preserves its value, and the same final conversion acts on the same .
This is the reproducibility argument of order-free summation methods ([13, 5]), restated for CertiSum’s merges. Rounding local outputs before merging does not meet this premise. For example, FP32 rounding of and before addition yields zero, whereas their exact sum is one. An ordinary FP32 collective is sufficient when a global certificate covers its intermediate sums.
Attention and Training Invariance
To preserve the reduction terms across layouts, executions must retain the same logical inputs, parameters, optimizer state, masks, and random choices. They must preserve quantizer orientation, global 32-element blocks, backward tiles, pruning decisions, global loss normalization, and rounding boundaries. Partition boundaries along every quantized dimension must fall on multiples of 32, so that no MX block straddles two partitions: the reduction dimension for tensor-parallel shards, the token dimension for micro-batches and data-parallel ranks, and the sequence dimension for context-parallel chunks. A block that straddled two sequentially computed micro-batches would need a scale that depends on data not yet available. Backward tiles impose the same alignment on both of their dimensions. Numerical maps outside exact reductions, including exponentiation and quantization of attention weights, must agree across supported hardware. For attention this means that the exponentiation producing returns identical quantized codes on every supported platform; Appendix § 13.§ 1 describes how MI350X and GB200 meet this requirement. FoldAttention [1] similarly fixes a softmax reference before the scan, which makes partial sums over separate key ranges additive.
The weight is a function of the score , of the scale of its global key block, and of the per-row reference of Appendix § 11, which is fixed by the global block positions of the first and the query-containing key blocks and does not depend on how keys are partitioned. The running maximum of blockwise softmax would not serve, since it depends on the visit order. Pruning is a function of the block scale and the same reference, so it is partition independent as well. Hence and are fixed across layouts. The condition requires at least one unmasked, unpruned key per query row. Under a causal mask, the block that sets the reference contains an unmasked key, is never pruned, and has a nonzero largest quantized weight, so the condition holds; masks that can leave a row with no admissible key are outside the scope of this guarantee. For a row with and fixed and , a key partition contributes the exact pair
These dyadic sums admit exact integer representations. Proposition 2 applies componentwise, so merging the pairs recovers the same and . Finalization must round their exact quotient as in Equation (4). Let be the midpoint of adjacent FP32 candidates bracketing the exact quotient. The exact sign of selects the side containing component ; ties use the even candidate, a standard technique for correctly rounded division [42]. Separately rounding either sum, or relying on an unchecked FP64 division, does not establish this contract.
Corollary (Accepted training steps). Under these conditions, suppose every partition-dependent reduction uses certified or repaired exact partial sums, exact merging, and the prescribed final conversion. This covers reductions over quantized GEMM and attention operands. A training step also contains reductions outside this class, such as the vocabulary-parallel log-sum-exp of cross-entropy, global gradient-norm clipping, and loss normalization. Each of them must either be merged exactly under Proposition 2 or be computed identically on every layout, for example from the same exactly merged inputs in a fixed order. Loss normalization in particular must be applied either as a power-of-two factor to the partial sums or once to the merged sum; applying a general factor to each partial sum before merging rounds before merging and violates the premise of Proposition 2. If other operators implement identical deterministic maps, every accepted step produces identical parameter and optimizer-state bits across layouts and supported hardware.
Proof. Induct through the fixed logical computation graph. Equal predecessors produce identical quantized operands and local-map outputs. Proposition 2 gives equal reduction results; attention follows from the merged pair and common quotient rounding. Equal gradients, old parameters, and optimizer state give equal updates. Repeating this argument proves equality across accepted steps. Acceptance requires all reductions to be complete and all failures resolved across ranks; tentative values affected by a failed reduction must be recomputed before an update is committed.
Experimental Details
This section collects setup details and caveats for the experiments in § 6.
Cross-Platform Invariance
TP4 configuration. The TP4 configuration partitions computation while replicating parameters and optimizer state.
Shared numerical definitions. CertiSum and the vendor baseline use the same numerical definitions outside GEMMs and attention. RoPE and SiLU use shared lookup tables; loss and optimizer operations are explicitly rounded. Cross-entropy uses a shared correctly rounded exponential with a host fallback and host-evaluated logarithms. CertiSum attention fixes the MXFP4 rounding of . GB200 uses threshold comparisons, while MI350X uses native exp2 certified to produce the same quantized codes on reachable inputs.
Layout invariance versus vendor agreement. Each vendor’s one-GPU and TP4 baseline losses match, so invariance across these layouts does not ensure cross-vendor agreement.
Inference Invariance
Decoding setup. Each decoding step recomputes the full prefix, right-padded to 4,096 positions with padded keys and values zeroed.
Repeatability versus invariance. Repeatability alone does not ensure invariance. The twin’s batch-dependent answer arises because its persistent GEMM changes summation order with the output row’s position in the batch.
Overhead
Comparing implementations. These are comparisons between implementations, since TP communication and operating clocks differ between CertiSum and its twin, and for the smaller CP cases the twin was not timed with graph replay. The latency gaps in § 6.5 include these implementation differences.
Out-of-memory configurations. Two further configurations, Qwen3-14B-Base on one GPU at and , ran out of memory in every compared implementation.
NVIDIA GB200. Our GB200 experiments focus on platform invariance (§ 6.2).
General-Knowledge Questions
The inference experiments in § 6.4 use the following 32 general-knowledge questions. They were generated with Claude Code and are reproduced verbatim in their original order.
| General-knowledge questions |
| 1 Why is the sky blue? |
| 2 What causes the seasons on Earth? |
| 3 How does a refrigerator keep food cold? |
| 4 Why do we have leap years? |
| 5 How does a vaccine work? |
| 6 Why does ice float on water? |
| 7 What is a prime number and why do prime numbers matter? |
| 8 How does GPS determine a position? |
| 9 Why do leaves change color in the autumn? |
| 10 How do airplanes stay in the air? |
| 11 What causes the tides in the ocean? |
| 12 How does the human heart pump blood through the body? |
| 13 Why does bread rise when it is baked? |
| 14 How does a rainbow form? |
| 15 What is photosynthesis and why is it important? |
| 16 How do noise-cancelling headphones work? |
| 17 Why do we dream? |
| 18 How does a compass work? |
| 19 What causes earthquakes? |
| 20 How does the internet send data from one computer to another? |
| 21 Why is the ocean salty? |
| 22 How do bees make honey? |
| 23 What is inflation in economics? |
| 24 How does a microwave oven heat food? |
| 25 Why do cats purr? |
| 26 How does the immune system fight infections? |
| 27 What causes thunder and lightning? |
Table 7.
| General-knowledge questions (continued) |
| 28 How do solar panels produce electricity? |
| 29 Why do onions make people cry? |
| 30 How does a credit score work? |
| 31 What is the greenhouse effect? |
| 32 How do volcanoes form? |
Table 8.
Training-Step Latency
AMD MI350X. Table 5 lists the training-step latencies summarized in Figure 7 (§ 6.5). Each run performs 30 training steps; the first three are warm-up. We exclude a run if any timed step exceeds that step’s median across runs of the same configuration by more than both 250 ms and 25%. Main shapes use 7–8 retained CertiSum runs, eight twin runs, and three runs per BF16 reference; smaller shapes use three runs per variant. Our implementations use pretrained weights and FineWeb-Edu; torchtitan uses random initialization, with FineWeb-Edu for Llama and a small C4 test corpus for Qwen. The static-route variant fixes routes during calibration and retains acceptance checks.
Routing and Repair
To evaluate CertiSum’s use of native and exact computation, we measure route allocation and recorded repairs. We instrument the four model/layout configurations in Table 6 on FineWeb-Edu and The Stack v3 using AMD MI350X GPUs. Each configuration and dataset pair uses one 30-step counting run. We sum counters over all GPUs for the 27 steps after warm-up (indices 3–29).
| Dataset | Native | Exact | Int. | Native | Repair | Work (%) |
| Qwen3-8B-Base, 1 GPU, | ||||||
| FineWeb-Edu | 36.92 | 63.08 | 0.0000 | 88.60 | 0.0229 | 0.0069 |
| The Stack v3 | 36.98 | 62.97 | 0.0451 | 88.89 | 0.0186 | 0.0072 |
| Qwen3-8B-Base, TP2, | ||||||
| FineWeb-Edu | 37.19 | 62.81 | 0.0000 | 89.77 | 0.0558 | 0.0123 |
| The Stack v3 | 37.22 | 62.74 | 0.0376 | 89.91 | 0.0521 | 0.0111 |
| Qwen3-14B-Base, TP4, | ||||||
| FineWeb-Edu | 54.23 | 45.76 | 0.0110 | 89.58 | 0.9710 | 0.2318 |
| The Stack v3 | 54.12 | 45.84 | 0.0349 | 89.15 | 0.8497 | 0.1945 |
| Llama-3.1-8B, CP4, | ||||||
| FineWeb-Edu | 32.17 | 67.83 | 0.0005 | 98.81 | 0.1019 | 0.0189 |
| The Stack v3 | 32.16 | 67.84 | 0.0000 | 98.78 | 0.0829 | 0.0154 |
Table 6. Routing and recorded repairs on AMD MI350X. Shapes are global . Assigned work splits nominal step GEMM FLOPs among native, direct exact, and integer-engine (Int.) routes, the last including padding. Forward columns give native candidates as a share of transformer linear-layer outputs and recorded repairs as a share of those. Repair work is a share of nominal GEMM FLOPs. Values are rounded independently.
Table 6 separates work assigned to native GEMMs, direct exact kernels, and the recorded integer engine, as percentages of the step’s nominal GEMM FLOPs. The forward-output columns cover transformer linear layers, excluding attention and the output head.1
The native route handles 88.60–98.81% of transformer linear-layer forward outputs. Instrumented paths record repairs for 0.0186–0.9710% of native forward candidates.
Recorded repair work is 0.0069–0.2318% of the step’s nominal GEMM FLOPs. Qwen3-14B-Base TP4 has the largest recorded repair work, at 0.1945–0.2318%; the two 8B models remain below 0.019%. These counts quantify repair dot products, not all repair arithmetic or runtime overhead.
Exact MXFP4 kernels handle much of the full training step. Native GEMMs account for 32.16–54.23% of assigned work, and direct exact kernels account for 45.76–67.84%. Attention and nearly all weight-gradient work contribute to the exact-kernel share. Recorded integer-engine work is at most 0.0451%, including padding.
We conclude that CertiSum uses the native route for most linear-layer forward outputs. Recorded repairs cover fewer than 1% of native candidates. Direct exact kernels handle much of the full training step.
References
- [1]Achanta, S. FoldAttention: Declared-Reference Softmax for Fast Decode and Deterministic Backward, 2026. URL https://arxiv.org/abs/2609.33410.
- [2]Advanced Micro Devices. AITER: AI Tensor Engine for ROCm. https://github.com/ROCm/aiter, 2026a. Accessed: 2026-10-30.
- [3]Advanced Micro Devices. Introducing AMD CDNA 4 Architecture. White paper, Advanced Micro Devices, Inc., 2026b. URL https://www.amd.com/content/dam/en/documents/instinct-tech-docs/white-papers/amd-cdna-4-architecture-whitepaper.pdf.
- [4]Advanced Micro Devices. hipBLASLt Documentation. https://rocm.docs.amd.com/projects/hipBLASLt/, 2026c. Accessed: 2026-10-29.
- [5]Ahrens, W., Demmel, J., and Nguyen, H. D. Algorithms for Efficient Reproducible Floating Point Summation. ACM Trans. Math. Softw., 46(3), July 2020. ISSN 0098-3500. doi: 10.1145/3389360.
- [6]Aithal, A., Harper, E., Bhavani, S., Xu, Y., and Li, Z. Scale Bitwise-Deterministic Pretraining with NVIDIA Megatron Core. NVIDIA Technical Blog. https://developer.nvidia.com/blog/scale-bitwise-deterministic-pretraining-with-nvidia-megatron-core/, October 2026. Accessed: 2026-10-25.
- [7]Alvarez, E., Almog, O., Chung, E., Layton, S., Stosic, D., Krashinsky, R., and Aubrey, K. Introducing NVFP4 for Efficient and Accurate Low-Precision Inference. NVIDIA Technical Blog, June 2025. URL https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/. Accessed: 2026-10-29.
- [8]Arun, A., St. Arnaud, A., Titov, A., Wilcox, B., Kolobaric, V., Brinkmann, M., Ersoy, O., Fielding, B., and Bonneau, J. Verde: Verification via Refereed Delegation for Machine Learning Programs, 2025. URL https://arxiv.org/abs/2502.19405.
- [9]Badash, E., Boneh, D., Komargodski, I., and Srivastava, M. Hawkeye: Reproducing GPU-Level Non-Determinism. In Chowdhery, A. and Jia, Z. (eds.), Proceedings of Machine Learning and Systems, volume 8, pp. 1830–1842. MLSys, 2026.
- [10]Bai, Y. and Shen, A. Finish RL Training Faster: Bitwise-Consistent, High-Performance, Agent-Driven. vLLM Blog. https://vllm.ai/blog/2026-10-14-train-inference-consistency, October 2026. Accessed: 2026-10-29.
- [11]Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877–1901, 2020.
- [12]Cai, W., Gu, D., Zhong, B., Wang, J., Zheng, Z., Liu, G., Jiang, K., Wang, S., Xiao, W., and Huang, J. FlexTrain: Scalable Hybrid-Parallel Training with Elastic Resource Utilization and Consistent Accuracy. In Chowdhery, A. and Jia, Z. (eds.), Proceedings of Machine Learning and Systems, volume 8, pp. 2082–2095. MLSys, 2026.
- [13]Collange, C., Defour, D., Graillat, S., and Jakymchuk, R. Numerical Reproducibility for the Parallel Reduction on Multi- and Many-Core Architectures. Parallel Computing, 49:83–97, 2015. ISSN 0167-8191. doi: 10.1016/j.parco.2015.09.001. URL https://www.sciencedirect.com/science/article/pii/S0167819115001155.
- [14]Dao, T. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. In Kim, B., Yue, Y., Chaudhuri, S., Fragkiadaki, K., Khan, M., and Sun, Y. (eds.), International Conference on Learning Representations, pp. 35549–35562, 2024.
- [15]Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 16344–16359. Curran Associates, Inc., 2022. doi: 10.52202/068431-1189.
- [16]Dao, T., Haziza, D., Massa, F., and Sizov, G. Flash-Decoding for Long-Context Inference. PyTorch Blog, October 2023. URL https://pytorch.org/blog/flash-decoding/. Accessed: 2026-10-30.
- [17]Darvish Rouhani, B., Garegrat, N., Savell, T., More, A., Han, K.-N., Zhao, R., Hall, M., Klar, J., Chung, E., Yu, Y., Schulte, M., Wittig, R., Bratt, I., Stephens, N., Milanovic, J., Brothers, J., Dubey, P., Cornea, M., Heinecke, A., Rodriguez, A., Langhammer, M., Deng, S., Naumov, M., Mickevicius, P., Siu, M., and Verrilli, C. OCP Microscaling Formats (MX) Specification. Specification Version 1.0, Open Compute Project, September 2023a. URL https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf.
- [18]Darvish Rouhani, B., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., Stosic, D., Elango, V., Golub, M., Heinecke, A., James-Roxby, P., Jani, D., Kolhe, G., Langhammer, M., Li, A., Melnick, L., Mesmakhosroshahi, M., Rodriguez, A., Schulte, M., Shafipour, R., Shao, L., Siu, M., Dubey, P., Mickevicius, P., Naumov, M., Verrilli, C., Wittig, R., Burger, D., and Chung, E. Microscaling Data Formats for Deep Learning, 2023b. URL https://arxiv.org/abs/2310.10537.
- [19]Donaghy, J., Wilcox, B., Ersoy, O., Rastogi, S., St Arnaud, A., Titov, A., Greenberg, J., Fielding, B., and Grieve, H. OPEN-1B: A Fully Auditable Training Run. arXiv preprint arXiv:2609.17380, 2026.
- [20]Gao, Y., Tu, R., Pekhimenko, G., and Vijaykumar, N. DPQuant: Efficient and Differentially-Private Model Training via Dynamic Quantization Scheduling. In Vondrick, C., Hariharan, B., Raffel, C., Pinto, L., Yang, D., and Faust, A. (eds.), International Conference on Learning Representations, 2026.
- [21]Gholami, A., Yao, Z., Kim, S., Hooper, C., Mahoney, M. W., and Keutzer, K. AI and Memory Wall. IEEE Micro, 44 (3):33–39, 2024. doi: 10.1109/MM.2024.3373763.
- [22]Gond, R., Kamath, A. K., Ramjee, R., and Panwar, A. LLM-42: Enabling Determinism in LLM Inference with Verified Speculation. In Proceedings of the ACM SIGOPS 32nd Symposium on Operating Systems Principles, SOSP ’26, pp. 916–931, New York, NY, USA, 2026. Association for Computing Machinery. ISBN 9798400725852. doi: 10.1145/3830418.3843901.
- [23]Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The Llama 3 Herd of Models, 2024. URL https://arxiv.org/abs/2407.21783.
- [24]Hayashi, S., Mukunoki, D., Hoshino, T., and Katagiri, T. AWE: Adaptive Weight Encoding for Exact Integer Matrix Products with Fewer GEMMs on FP4 Tensor Cores, 2026. URL https://arxiv.org/abs/2609.24519.
- [26]Higham, N. J. Accuracy and Stability of Numerical Algorithms. Society for Industrial and Applied Mathematics, 2nd edition, 2002. doi: 10.1137/1.9780898718027. URL https://epubs.siam.org/doi/abs/10.1137/1.9780898718027.
- [28]Iakymchuk, R., Defour, D., Collange, C., and Graillat, S. Reproducible and Accurate Matrix Multiplication. In Scientific Computing, Computer Arithmetic, and Validated Numerics, pp. 126–137. Springer, 2016.
- [30]Jiang, A. and SkyRL Team. IsoExec: Unified Execution to Eliminate Trainer-Inference Mismatch in SkyRL. vLLM Blog. https://vllm.ai/blog/2026-08-21-isoexec. August 2026. Accessed: 2026-10-25.
- [31]Kulisch, U. Computer Arithmetic and Validity: Theory, Implementation, and Applications. De Gruyter, 2nd edition, 2013.
- [34]Liang, W., Liu, T., Wright, L., Constable, W., Gu, A., Huang, C.-C., Zhang, I., Feng, W., Huang, H., Wang, J., Purandare, S. J., Nadathur, G., and Idreos, S. TorchTitan: One-stop PyTorch Native Solution for Production Ready LLM Pretraining. In Yue, Y., Garg, A., Peng, N., Sha, F., and Yu, R. (eds.), International Conference on Learning Representations, pp. 92161–92181, 2025.
- [35]Liu, H., Zaharia, M., and Abbeel, P. RingAttention with Blockwise Transformers for Near-Infinite Context. In Kim, B., Yue, Y., Chaudhuri, S., Fragkiadaki, K., Khan, M., and Sun, Y. (eds.), International Conference on Learning Representations, pp. 3992–4008, 2024.
- [36]Loshchilov, I. and Hutter, F. Decoupled Weight Decay Regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCq7Y.
- [37]Lozhkov, A., Li, R., Ben Allal, L., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., Liu, T., Tian, M., Kocetkov, D., Zucker, A., Belkada, Y., Wang, Z., Liu, Q., Abulkhakov, D., Paul, I., Li, Z., Li, W.-D., Risdal, M., Li, J., Zhu, J., Zhuo, T. Y., Zheltonozhskii, E., Dade, N. O. O., Yu, W., Krauß, L., Jain, N., Su, Y., He, X., Dey, M., Abati, E., Chai, Y., Muenighoff, M., Tang, X., Oblokulov, M., Akiki, C., Marone, M., Mou, C., Mishra, M., Gu, A., Hui, B., Dao, T., Zebaze, A., Dehaene, O., Patry, N., Xu, C., McAuley, J., Hu, H., Scholak, T., Paquet, S., Robinson, J., Anderson, C. J., Chapados, N., Patwary, M., Tajbakhsh, N., Jernite, Y., Muñoz Ferrandis, C., Zhang, L., Hughes, S., Wolf, T., Guha, A., von Werra, L., and de Vries, H. StarCoder 2 and The Stack v2: The Next Generation, 2024. URL https://arxiv.org/abs/2402.19173.
- [38]Lozhkov, A., Larcher, H., Morlon, M., Ben Allal, L., and von Werra, L. The Stack v3: The Largest Open Code Dataset. Hugging Face, 2026. URL https://huggingface.co/datasets/HuggingFaceCode/stack-v3-train.
- [39]Lutz, D. R. and Hinds, C. N. High-Precision Anchored Accumulators for Reproducible Floating-Point Summation. In 2017 IEEE 24th Symposium on Computer Arithmetic (ARITH), pp. 98–105, 2017. doi: 10.1109/ARITH.2017.20.
- [41]MiniMax, Chen, A., Li, A., Gong, B., Jiang, B., Fei, B., Yang, B., Shan, B., Yu, C., Wang, C., Zhu, C., Xiao, C., Du, C., Zhang, C., Qiao, C., Zhang, C., Du, C., Guo, C., Chen, D., Ding, D., Sun, D., Li, D., Jiao, E., Zhou, H., Zhang, H., Ding, H., Sun, H., Feng, H., Cai, H., Zhu, H., Sun, J., Zhuang, J., Cai, J., Song, J., Zhu, J., Li, J., Tian, J., Liu, J., Xu, J., Yan, J., Liu, J., He, J., Feng, K., Yang, K., Xiao, K., Han, L., Wang, L., Yu, L., Feng, L., Li, L., Zheng, L., Du, L., Yang, L., Zeng, L., Yu, M., Tao, M., Chi, M., Zhang, M., Lin, M., Hu, N., Di, N., Gao, P., Li, P., Zhao, P., Ren, Q., Xu, Q., Li, Q., Wang, Q., Tian, R., Leng, R., Chen, S., Chen, S., Shi, S., Weng, S., Guan, S., Yu, S., Li, S., Zhu, S., Li, T., Cai, T., Liang, T., Cheng, W., Kong, W., Li, W., Chen, X., Song, X., Luo, X., Su, X., Li, X., Han, X., Hou, X., Lu, X., Zou, X., Shen, X., Gong, Y., Ma, Y., Wang, Y., Shi, Y., Zhong, Y., Duan, Y., Fu, Y., Hu, Y., Gao, Y., Fan, Y., Yang, Y., Li, Y., Hu, Y., Huang, Y., Li, Y., Xu, Y., Mao, Y., Shi, Y., Wenren, Y., Li, Z., Li, Z., Tian, Z., Zhu, Z., Fan, Z., Wu, Z., Xu, Z., Yu, Z., Lyu, Z., Jiang, Z., Gao, Z., Wu, Z., Song, Z., and Sun, Z. MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention, 2025. URL https://arxiv.org/abs/2506.13585.
- [42]Muller, J.-M., Brunie, N., de Dinechin, F., Jeannerod, C.-P., Joldes, M., Lefèvre, V., Melquiond, G., Revol, N., and Torres, S. Handbook of Floating-Point Arithmetic. Birkhäuser, 2nd edition, 2018.
- [44]NVIDIA. cuBLAS Library: Using the cuBLASLt API. https://docs.nvidia.com/cuda/cublas/, 2026a. Accessed: 2026-10-29.
- [45]NVIDIA. cuda.tile.mma_scaled: Block-Scaled Matrix Multiply-Accumulate. cuTile Python documentation. https://docs.nvidia.com/cuda/cutile-python/generated/cuda.tile.mma_scaled.html, 2026b. Accessed: 2026-10-25.
- [47]OpenAI, Agarwal, S., Ahmad, L., Ai, J., Altman, S., Applebaum, A., Arbus, E., Arora, R. K., Bai, Y., Baker, B., Bao, H., Barak, B., Bennett, A., Bertao, T., Brett, N., Brevdo, E., Brockman, G., Bubeck, S., Chang, C., Chen, K., Chen, M., Cheung, E., Clark, A., Cook, D., Dukhan, M., Dvorak, C., Fives, K., Fomenko, V., Garipov, T., Georgiev, K., Glaese, M., Gogineni, T., Goucher, A., Gross, L., Guzman, K. G., Hallman, J., Hehir, J., Heider, J., Heidecke, J., Helyar, A., Hu, H., Huet, R., Huh, J., Jain, S., Johnson, Z., Koch, C., Kofman, I., Kundel, D., Kwon, J., Kyrylov, V., Le, E. Y., Leclerc, G., Lennon, J. P., Lessans, S., Lezcano-Casado, M., Li, Y., Li, Z., Lin, J., Liss, J., Liu, L., Liu, J., Lu, K., Lu, C., Martinovic, Z., McCallum, L., McGrath, J., McKinney, S., McLaughlin, A., Mei, S., Mostovoy, S., Mu, T., Myles, G., Neitz, A., Nichol, A., Pachocki, J., Paino, A., Palmie, D., Pantuliano, A., Parascandolo, G., Park, J., Pathak, L., Paz, C., Peran, L., Pimenov, D., Pokrass, M., Proehl, E., Qiu, H., Raila, G., Raso, F., Ren, H., Richardson, K., Robinson, D., Rotsted, B., Salman, H., Sanjeev, S., Schwarzer, M., Sculley, D., Sikichi, H., Simon, K., Singhal, K., Song, Y., Stuckey, D., Sun, Z., Tillet, P., Toizer, S., Tsimpourlas, F., Vyas, N., Wallace, E., Wang, X., Wang, M., Watkins, O., Weil, K., Wendling, A., Whinnery, K., Whitney, C., Wong, H., Yang, L., Yang, Y., Yasunaga, M., Ying, K., Zaremba, W., Zhan, W., Zhang, C., Zhang, B., Zhang, E., and Zhao, S. gpt-oss-120b & gpt-oss-20b Model Card, 2025. URL https://arxiv.org/abs/2508.10925.
- [48]Osama, M., Merrill, D., Cecka, C., Garland, M., and Owens, J. D. Stream-K: Work-Centric Parallel Decomposition for Dense Matrix-Matrix Multiplication on the GPU. In Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, PPoPP ’23, pp. 429–431, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400700156. doi: 10.1145/3572848.3577479.
- [49]Ozaki, K., Ogita, T., Oishi, S., and Rump, S. M. Error-free Transformations of Matrix Multiplication and its Applications. Numerical Algorithms, 59(1):95–118, 2012.
- [50]Penedo, G., Kydlíček, H., Ben Allal, L., Lozhkov, A., Mitchell, M., Raffel, C., von Werra, L., and Wolf, T. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 30811–30849. Curran Associates, Inc., 2024. doi: 10.52202/079017-0970. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/370df50ccdf8bde18f8f9c2d9151bda-Paper-Datasets_and_Benchmarks_Track.pdf.
- [51]PyTorch Contributors. torch.use_deterministic_algorithms. https://docs.pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html, 2026. Accessed: 2026-10-29.
- [52]Qiang, X., Chen, H., Sun, S., Leng, J., Liu, X., and Guo, M. DASH: Deterministic Attention Scheduling for High-throughput Reproducible LLM Training. In International Conference on Learning Representations, 2026.
- [53]Salykov, A., Luo, A., Huang, C., and Sun, P. Matrix Core Programming on AMD CDNA3 and CDNA4 Architecture. ROCm Blogs. https://rocm.blogs.amd.com/software-tools-optimization/matrix-cores-cdna/README.html, September 2025. Accessed: 2026-10-25.
- [54]Sarkar, D. The Correctness Illusion in LLM-Generated GPU Kernels, 2026. URL https://arxiv.org/abs/2606.20128.
- [55]Schwarz, A., Anders, A., Brower, C., Bayraktar, H., Gunnels, J., Clark, K., Xu, R. G., Rodriguez, S., Cayrols, S., Tabaszewski, P., and Podlozhnyuk, V. Guaranteed DGEMM Accuracy While Using Reduced Precision Tensor Cores Through Extensions of the Ozaki Scheme. In Proceedings of the Supercomputing Asia and International Conference on High Performance Computing in Asia Pacific Region, SCA/HPCAsia ’26, pp. 91–101, New York, NY, USA, 2026. Association for Computing Machinery. ISBN 9798400720673. doi: 10.1145/3773656.3773670.
- [56]Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 68658–68685. Curran Associates, Inc., 2024. doi: 10.52202/079017-2193.
- [57]Shah, R. and Shrestha, R. A Contract-Grade Verifier for LLM-Generated GPU Kernels, and a Native Blackwell Backward for the Gated-Linear-Recurrence Family, 2026. URL https://arxiv.org/abs/2608.12700.
- [58]Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., and Wu, C. HybridFlow: A Flexible and Efficient RLHF Framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279–1297, 2025.
- [59]Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, 2019. URL https://arxiv.org/abs/1909.08053.
- [60]Tseng, A., Yu, T., and Park, Y. Training LLMs with MXFP4. In Li, Y., Mandt, S., Agrawal, S., and Khan, E. (eds.), Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, volume 258 of Proceedings of Machine Learning Research, pp. 1630–1638. PMLR, 03–05 May 2025.
- [61]Wang, N., Choi, J., Brand, D., Chen, C.-Y., and Gopalakrishnan, K. Training Deep Neural Networks with 8-bit Floating Point Numbers. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
- [62]Xie, P., Xu, S., Wang, Y., Yang, F., and Yang, M. Bit-Accurate Modeling of GPU Matrix Multiply-Accumulate Units: Demystifying Numerical Discrepancy and Accuracy. arXiv preprint arXiv:2511.10909, 2025a.
- [63]Xie, P., Zhang, X., and Chen, S. RepDL: Bit-level Reproducible Deep Learning Training and Inference, 2025b. URL https://arxiv.org/abs/2510.09180.
- [64]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., Yang, K., Yu, L., Deng, L., Li, M., Xue, M., Li, M., Zhang, P., Wang, P., Zhu, Q., Men, R., Gao, R., Liu, S., Luo, S., Li, T., Tang, T., Yin, W., Ren, X., Wang, X., Zhang, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Zhang, Y., Wan, Y., Liu, Y., Wang, Z., Cui, Z., Zhang, Z., Zhou, Z., and Qiu, Z. Qwen3 Technical Report, 2025. URL https://arxiv.org/abs/2505.09388.
- [65]Yao, F., Liu, L., Zhang, D., Dong, C., Shang, J., and Gao, J. Your Efficient RL Framework Secretly Brings You Off-Policy RL Training. Feng Yao’s Notion, August 2025. URL https://fengyao.notion.site/off-policy-rl.
- [66]Zadouri, T., Hoehnerbach, M., Shah, J., Thakkar, V., and Dao, T. FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling. In Chowdhery, A. and Jia, Z. (eds.), Proceedings of Machine Learning and Systems, volume 8, pp. 912–926. MLSys, 2026.
- [67]Zhang, Z., Ding, X., Yuan, J., Liu, R., Mao, H., Xing, J., and Liu, Z. Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch. In Zhang, T., Dudík, M., Jaggi, M., Agarwal, A., Li, S., Schuurmans, D., Zhu, J., Berkenkamp, F., Dong, H., and Bietti, A. (eds.), Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pp. 155026–155042. PMLR, 06–11 Jul 2026.







