betaFeedback

WorksComputer Science

KOSMOS: Self-scheduling Persistent Kernels for Communication-Compute Overlap

Abstract

Tensor and sequence parallelism (TP+SP) and expert parallelism (EP) put collectives on the critical path of transformer layers. State-of-the-art fused kernels mitigate this with fixed roles or schedules, but this rigidity leaves large idle windows. Predicting a communication-computation split does not remove them either; a predetermined split cannot follow data readiness. We present KOSMOS, a library for self-scheduling persistent kernels whose workgroups use queues ordered by consumer demand to switch between communication and computation roles as data become ready, with no scheduler, per-call cost model, or tuned split. KOSMOS fuses every TP+SP collective of a transformer layer with its GEMM, and runs an entire expert layer, from router to fold, in one launch per direction. Evaluated on AMD Instinct MI350X and AMD Instinct MI355X GPUs, KOSMOS speeds up TP+SP end-to-end training in Megatron-LM by 1.10-1.28x in BF16 and 1.10-1.25x in MXFP8. KOSMOS accelerates end-to-end training of DeepSeek-V3 and Qwen3-235B proxies by 1.29-1.83x over DeepEP.

Introduction

Training frontier large language models (LLMs) requires parallelism strategies that trade memory for communication. Memory reduction strategies such as tensor and sequence parallelism (TP+SP) and expert parallelism (EP) in Mixture-of-Experts (MoE) put collectives on the critical path. TP+SP wraps every transformer block in all-gather (AG) and reduce-scatter (RS) collectives, whereas EP in MoE models adds all-to-alls (A2A) around each expert layer. To keep matrix cores and links saturated, overlapping communication and computation is critical to training throughput.

Fusing communication into general matrix multiplication (GEMM) avoids hardware-scheduler contention and moves data at tile granularity, overlapping with the GEMM’s tile-based computation. Prior fused kernels dedicate a fixed number of compute units (CUs) to communication [12] (Zheng et al. 2025a; Sul et al. 2026a) or assign each workgroup (WG) its tasks with a scheduler [1, 14]. However, communication demand is not fixed: In dense collectives it shifts as the kernel progresses, and in MoE it shifts with each batch’s routing. Predicting a split of communication and computation WGs does not solve this issue. A planned split from a cost model before each call is slower than a fixed split in our measurements (Table 2). Even adapting splits at runtime, as mKernel does with a kernel controller, leaves their kernel

1.18×1.18\times slower than the best fixed split [24].

We present KOSMOS (Kernel-Overlapped Self-scheduling Megakernels On Symmetric-memory), a library of persistent kernels in which WGs dynamically change roles as the kernel runs. A WG is either a mover, which mainly communicates, or a builder, which mainly computes GEMM tiles. A WG selects its role from data readiness, with no scheduler or per-call cost model. Scheduling is implicit: Data is produced in the order consumers claim it. Builders start on local data while movers gather; reductions are produced, ordered by groups of tiled rows (band-major), so every rank’s movers start after one step, and dispatch delivers experts in the order each GEMM consumes them.

The switching policy is set by each collective-GEMM pair (Figure 1). In AG+GEMM and MoE dispatch, movers become builders once their transfers are done; in GEMM+RS (Figure 1(A)), builders join the reduction when the GEMM work queue is exhausted; in MoE combine (Figure 1(B)), WGs switch both ways, pushing whenever data is ready and building otherwise. The policy is designed around the shape of the demand: Two-way switching is slower for TP+SP and A2A dispatch collectives whose demand is monotonic; one-way switching leaves combine gaps larger than needed.

Figure 1. CU timelines and work queues on eight GPUs for (A) fused AG and RS and (B) the MoE forward and backward. Each CU runs one persistent WG; queues hold GEMM output tiles, and hatched segments are transfers (color = rank or expert). (A) The AG queue holds local tiles first, then peer tiles interleaved row by row; the RS queue is band-major (pattern = band). AG movers (CUs 0–1 of mm) become builders once their transfers finish; in RS, builders become movers when the GEMM queue empties. (B) The forward queue is expert-major, GEMM1 before GEMM2; the backward queue holds GEMM2 then GEMM1 gradient tiles, interleaving dGdG and dWdW. All CUs first route and plan (purple); dispatchers push, then build; combine movers (CUs dd, d+1d+1) build until a combine unit is ready, then interleave pushes with GEMM2 tiles. Backward: dYdY is dispatched first and the dXdX combine overlaps the GEMM1 gradients.

KOSMOS provides broad coverage, including, to our knowledge, the first readiness-driven TP+SP fused kernels and MoE megakernels for training. Our contributions are:

  1. Self-scheduling persistent kernels, with producer and consumer queues co-designed per collective around data readiness. WGs may then switch roles from local

tile-level checks. Switching lets a shape-only heuristic replace per-shape tuning: within 1% of, and up to 15% faster than, the best single fixed split. A fixed split in turn beats a split predicted per call.

  1. A characterization of where idle time falls in fused communication kernels and a policy rule that reduces it: one-way switching for monotonic demand, two-way switching for interleaved demand.

  1. Complete kernel coverage for training: persistent fused kernels for every TP+SP collective, forward and backward, in BF16 and MXFP8, and the first role-switching MoE layer with a single launch per direction, from router to fold, including gradients, evaluated end-to-end in Megatron-LM on two GPU models.

  1. Communications optimized for bytes/CU/s: collectives for fully connected meshes that saturate the bandwidth while maintaining fine grained access, and measurements showing that on power-capped GPUs the CUs lent to communication cost the GEMM little.

We demonstrate KOSMOS on MI350-series GPUs and expect many of its findings to generalize to other systems: On AMD Instinct™ MI350X and AMD Instinct™ MI355X GPUs, KOSMOS speeds up end-to-end Llama 3 training by up to 1.28×1.28\times over Transformer Engine (TE), and its MoE layers speed up training by up to 1.83×1.83\times over DeepEP.

Background and Motivation

Parallel Strategies

Tensor and Sequence Parallelism

Under TP+SP ([21] (Shoeybi et al. 2020)), each transformer block contains a column-parallel and a row-parallel GEMM pair (Figure 5). Activations outside the pair are sequence-sharded. An AG assembles the sequence before a column-parallel GEMM, and an RS after the row-parallel GEMM reduces sequence shards. The backward pass adds three collectives in BF16: AGs of the output gradients and the saved input, and an RS of the input gradient. MXFP8 gathers output gradients twice, quantized along both dimensions. Each GEMM pair runs five or six collectives per training step, each on the critical path unless overlapped.

Figure 5. Overview of TP+SP kernel fusion in KOSMOS. Top: forward. Bottom: backward. Each dashed box is one KOSMOS kernel in BF16 or MXFP8; the red box is used only for MXFP8.

Fully Sharded Data Parallelism

Fully sharded data parallelism (FSDP) shards only the model states, allowing collectives to gather model states for the next layer to run off of the critical path. Intra-node FSDP ((Zhao et al., 2023; Rajbhandari et al., 2020)) is often been preferred to TP+SP on AMD Instinct nodes ([9]). However, for large-scale distributed training, FSDP is often paired with TP+SP, which then shards activation memory across sequence, head, and channel dimensions, which dominates at scale.

Expert Parallelism

In an MoE layer with EP, the experts are distributed throughout the world size. A router GEMM matches kk experts to each token, then an A2A dispatch sends activations to the selected experts’ GEMMs. An A2A combine then returns the expert outputs to the token’s original rank, with results folded by token. Because routing is decided per pass, transfer volume and destinations change every step, with skew towards GPUs hosting hot experts. The more skewed the data, the more completion times diverge.

MXFP8 Training

MXFP8 ((Rouhani et al., 2023)) stores each element in FP8 and shares one power-of-two (E8M0) scale across each block of 32 consecutive elements, so a tensor moves roughly half the bytes of BF16. A block is quantized in the contracting direction. In training, as each matrix and its transpose are used, we quantize both row-wise and column-wise. Re-quantizing one copy from the other compounds rounding error, so both must be quantized from the higher-precision source. MXFP8 has been shown to be as accurate in training as BF16 ((Mishra et al., 2025)), with twice the theoretical GEMM throughput on MI350-series GPUs.

Motivation and Design Requirements

For TP+SP, the common overlap strategy pipelines chunks of data through transport and compute on two streams. This is generally inefficient, because the two kernels compete for the same CUs without coordinating. In an eight-GPU node in the MI350-series, each GPU has a theoretical bidirectional peak of 1008 GB/s of bandwidth ([2]), split between seven 144 GB/s XGMI links. A single peer is reachable at 1/7th of the aggregate. Full bandwidth usage requires sending to and receiving from all peers simultaneously. Host initiated direct memory access engines cannot provide this fan-out at fine granularity: They lag for small transfers because of their launch and synchronization cost, and they cannot perform reductions needed by TP+SP or EP. The tile-granular overlap therefore requires in-kernel communication. This leads to our first requirement—R1: One kernel, one stream: Communication and computation must share the GPU within one kernel.

In LLM training with power-capped GPUs, GEMM throughput decreases only slightly when CUs are capped. Both MI350X and MI355X ([2 b]) run at their power limit of 1000W and 1400W, respectively, with all 256 CUs active, so when a GEMM runs on fewer CUs, the remaining ones clock higher (Figure 3). On MI350X, our GEMMs keep an average 93% of their full-GPU throughput on 224 CUs (87.5% of the GPU), and 88% on 192 CUs (75% of the GPU) (Figure 2). Reserving CUs for communication, therefore, costs the GEMM less than the CU count suggests. R2: Use the power headroom: Collectives are power cheap, so CUs should be allocated to communication up to the point where the transport saturates.

GEMM-only throughput (TFLOPS) of KOSMOS GEMM kernels and hipBLASLt under a CU cap

Figure 2. GEMM-only throughput (TFLOPS) of KOSMOS GEMM kernels and hipBLASLt under a CU cap, communication off. Top: small shapes (Llama 3 70B, MBS 4); bottom: large shapes (Llama 3 405B, MBS 8); TN, NN, and NT layouts, titled M×N×K. Measured on MI350X.

Power and shader clock (SCLK) values over a Llama 3 70B-FC1-MBS8 GEMM stress test

Figure 3. Power and shader clock (SCLK) values over a Llama 3 70B-FC1-MBS8 GEMM stress test of about 1.1–1.3 s. Both MI350X and MI355X remain at their 1000W and 1400W power caps until dropping to 128 CUs. Both GPUs reach their SCLK peak around 96 CUs.

Overlapping a collective with its GEMM inside one kernel means dividing the GPU’s CUs between communication and computation. Most fused kernels make that division once, leading to windows where WGs sit idle on CUs. For AG, this occurs when the gather has finished before compute is complete. On RS, we have two windows: (1) where communicators wait for the first data to send, and (2) at the end when compute WGs have finished their tiles but the reductions are still occurring. In A2A, dispatches act much like AG with front-loaded communication, but combine traffic is interleaved with each batched GEMM, leaving windows scattered throughout the kernel. Figure 4 shows these windows in traces of fused kernels that keep a fixed split of communication WGs. Retuning the split may reduce idle time, but does not solve this structural problem, as communication demand changes as the fused kernel progresses. R3: Communicators budgeted adaptively: A fixed mover count leaves either matrix cores or interconnects idle, depending on the phase of the kernel. The number of active movers should follow the work that is available.

Advanced Thread Trace (ATT) timelines of three communication and three computation WGs

Figure 4. Advanced Thread Trace (ATT) timeline of three communication and three computation WGs for an AG+GEMM from Llama 70B FC1, with a microbatch size (MBS) of 4; a GEMM+RS from Llama 8B with MBS 8; and MoE forward for DeepSeek-V3 with balanced routing. Dark Blue: working. Light Blue: spinning on a flag or sleeping. White: exited.

MoE layers are a chain of steps whose sizes change from iteration to iteration. Executed as separate kernels, every step requires host overhead, routing decisions must leave the device, and no step can overlap another. Additionally, WGs do not necessarily start in their ID order (Appendix Figure A.1). With cross-WG dependencies (R1, R3), a resident consumer can wait on a producer that has not started; if waiting WGs fill every CU, the producer never spawns and the kernel deadlocks. R4: Persistent and resident: Kernels must be persistent with all WGs resident. When CUs are needed for a second stream of work, kernels must adapt to a WG cap so the whole grid stays resident.

One key corollary follows. R5: Self-scheduling: While kernel design may guide WGs towards certain work, kernels cannot rely on a host round trip (breaks persistence, R4), a separate scheduling kernel (a second kernel, R1), or a scheduler WG (a CU lost to communication or compute, R2). Each WG must decide its own work only from the state it observes (adapting as demand changes, R3).

Overview of KOSMOS

KOSMOS Design

KOSMOS implements each fused operation as a single persistent kernel that launches one WG per CU, with each WG resident for the kernel’s lifetime (R1, R4). Each WG starts in one of two roles: A mover transports data to and from remote peers, and a builder computes GEMM tiles. Depending on the collective, either role can be the producer or the consumer of the other. WGs coordinate only through in-kernel queues and readiness flags, without the host or a scheduler (R5). Rather than launching more WGs as work appears, KOSMOS keeps this fixed resident set and lets WGs change roles as the work changes (R3). Role switching is one-way or two-way. In one-way switching, a WG takes on another role for the rest of the kernel once its own work has finished. In AG and MoE dispatch, the movers become builders after their transfers are complete. In GEMM+RS, builders join the reduce once the GEMM tiles run out (Figure 1(A)). In two-way switching, a WG with no work available in its current role works in the other role temporarily, and returns when its own work appears: In the MoE combine, a set of WGs push any combine unit that is ready and builds GEMM tiles otherwise (Figure 1(B)).

Readiness and Queue Order

Without a scheduler, two things decide how a WG finds work: the unit at which data is declared ready, and the order of the queues it claims from. A WG claims a tile by atomically taking the next entry from a shared queue; it then computes that tile. KOSMOS operates in bands, chunks, tiles, and steps. A tile row is a set of tiles spanning 256 rows. A rank’s chunk is the tokens/TP-size output rows it owns after RS. A band is a group of tile rows within a chunk, and each step produces every chunk’s next band.

Work is tracked at the finest grain the GEMM allows. In gathering collectives, the unit is a 256-row tile row of gathered input. In reducing collectives, it is a band of rows that is sized to the output. Every output tile has exactly one completion point, so a consumer can start work on it as soon as it is finished and enqueued.

Two rules fix the order of work: Upstream before downstream: A WG claims every tile of an earlier step before any tile of a later one. A WG waits only on a tile claimed earlier in queue order, to avoid deadlocks or stalls. For example, in MoE forward, GEMM2 tiles are claimed only once every GEMM1 tile has been claimed (but not necessarily finished), so a waiting tile only ever depends on the work already claimed by a resident WG (Figure 1(B)). Additionally, a two-way WG never blocks on a push (as it builds instead), so waits are always satisfied. Produce in the order consumers need: If consumers are left waiting, we lose

overlap. In AG+GEMM, builders work on the local shard before claiming other work while movers gather the first shards. In GEMM+RS, builders must produce band-major; finishing the first band of every rank’s chunk first allows every rank’s reducers to start after one step. In MoE forward, dispatch pushes experts in the order GEMM1 consumes them, and GEMM2 works on a row block so the combine can push to its destination as soon as GEMM2 finishes it.

Coexistence under a CU Cap

When other work must run concurrently, such as internode FSDP, KOSMOS kernels can be CU capped by calling frameworks to allocate CUs for a second stream of work. Then, AG+GEMM and MoE kernels shrink the builder count and RS kernels shrink both roles proportionally. Every WG of the capped grid then remains resident (R4).

TP+SP Kernels

As shown in Figure 5, TP+SP requires six GEMMs and five or six collectives. KOSMOS names GEMMs by their layout in BLAS col-major convention: “T” for “transpose,” “N” for “no transpose.” For BF16, row-parallel wgrad is able to reuse the gathered dYdY from row-parallel dgrad. In MXFP8 training, dYTdY^{T} is also gathered. To overlap the increased communication, both AGs are fused along with the pair of backward GEMMs for the row-parallel layer into a single kernel. Thus, KOSMOS supports five persistent kernels for both data types (Figure 5).

Fused AG

In the AG kernels (forward AG+TN, backward AG+NN), the movers pull the gathered input directly. Each mover is pinned by its WG index to one slice of one source GPU shard, and copies it one 256-row tile row at a time from the peer into a local buffer. The mover then performs a release-add on a local arrival counter for that peer and tile row. Builders compute the local shard first, then claim peer tiles in gather order. A builder spins on the counters in gather order and performs a single acquire once the count is complete. Once their slice has been transported, the movers perform a one-way switch to the builders.

In MXFP8, the row-parallel backward fuses one more GEMM: Training accuracy requires the weight gradient to use dYdY quantized along the token dimension, so a second column-wise quantized copy of dYdY must also be gathered. KOSMOS gathers both quantized copies of dYdY, and computes the data gradient (NN) and the weight gradient (NT) in one kernel to minimize idle time.

Fused RS

The RS kernel (forward TN+RS) uses a “one-shot” pull, derived from the two-shot all-reduce (Thakur et al., 2005). Builders write their partial tiles into a local staging buffer in band-major order. Movers own a band of the output; they pull it from every rank, sum in FP32 in place in a fixed order, and round once to BF16 before storing directly to the output. Every partial crosses the fabric once and is rounded once, which has less accumulation error than RCCL’s ring, and is bitwise reproducible (Appendix Table A.6). Arrival is detected in the data itself: the staging buffer is filled with a NaN sentinel, so the landing of the data is the signal in and of itself. A BF16 signal NaN is used to avoid hangs that could arise from a matrix fused multiply-add-produced arithmetic NaN. The first builders (up to saturation) that find no GEMM tile left become movers to assist with the reduction (one-way switching).

Bulk Kernels

In the backward pass, two collectives have no data dependency on the GEMM they run beside. The AG of XX is needed only by the weight gradient, so it can run alongside the data gradient GEMM, and the RS of dXdX is not needed by the weight gradient. This becomes a resource optimization problem, so KOSMOS defines a heuristic that uses GB/s/CU and TFLOPS/CU curves to balance WGs. KOSMOS then employs one-way role-switching for movers and builders in bulk fusions to minimize any remaining idle time.

MoE Megakernels

An MoE layer in EP routes each token to kk experts. Then, tokens are dispatched to the GPUs that own those experts. The per-expert token counts change every step, so they are unknown until the layer runs. To avoid host overhead and enable overlap across steps, KOSMOS launches all of the steps below in a single kernel launch (Figure 6).

The KOSMOS single-launch MoE forward kernel

Figure 6. The KOSMOS single-launch MoE forward kernel. Red and blue: local experts; grey hatching: remote experts. Green: movers (light: dispatchers, dark: combiners); purple: builders. The flow runs from Start to End; stages are the forward-pass steps of § 4.4.1.

Forward Pass

  1. Route. All WGs compute the router logits XWTXW^{\mathsf{T}} in FP32 tiles. The last tile to finish a row block of tokens routes that block: score, (group-limited) top-kk, dispatch weights, and aux-loss inputs. The last block to be routed reduces the aux loss. The routing never leaves the device and does not need a separate phase.

  1. Plan. Builders sort the routing into per-expert send lists with a stable counting sort, keeping dWdW bitwise reproducible, since row order sets the wgrad’s accumulation order and MXFP8 scale groups. Only per-expert counts cross ranks. Each rank waits for all peers’ counts, then builds the layout and tile queues on the device. This wait is the forward’s only cross-rank barrier.

  1. Dispatch ∥ GEMM1. A crew of 16-96 movers (Appendix Table B.3) pushes token rows directly into their final, expert-grouped position in each owner’s receive arena, which at EP 8 is slightly faster than pulling (Appendix Table A.7). Push lists are destination-minor, so consecutive movers target different GPUs to keep every link busy. Per-(source, expert) flags at the destination track readiness. In MXFP8, mover quantizes their rows and pad each group to 32 rows before pushing. Each GEMM1 tile, with SwiGLU fused into its epilogue, waits only for the source whose rows it reads, and stores the pre-activations for the backward. In MXFP8, an epilogue quantizes the SwiGLU output row-wise for GEMM2 and column-wise for wgrad2.

  1. GEMM2 ∥ combine. A GEMM2 tile starts as soon as GEMM1 has finished its row block. Up to 128 dual-role movers (Appendix Table B.3) push each (expert, destination) unit back to its token owner once the unit’s row blocks are done, and claim GEMM tiles between pushes (two-way switching). The combine always pushes in BF16, even in MXFP8 training.

  1. Fold. Once every row owed to this rank has arrived, each token’s kk rows are weighted by its FP32 router weights and summed in FP32.

Backward Pass

  1. Dispatch. A crew of 32-64 movers (Appendix Table B.3) pushes the router-weighted dYdY to the expert owners. In MXFP8, dYdY is pushed in BF16 and quantized on the receiver in both the row-wise (dgrad2) and column-wise (wgrad2) layouts. wgrad1 also needs the column-wise XX, which the movers push after dYdY in link time the backward leaves idle; for narrow experts where the backward is communication-bound, it is sent with the forward dispatch instead (Appendix Table B.3). After their pushes, the movers compute the routing-weight gradients dwj=⟨dY,Yj⟩dw_j = \langle dY, Y_j\rangle from the forward’s combine buffer and back-propagate it with the aux-loss gradient through the score function to the logit gradient.

  1. One queue. dgrad2, with the SwiGLU backward fused into its epilogue, is interleaved with wgrad2 of the previous expert. The router’s dXrdX_r and dWrdW_r tiles are queued after dgrad2/wgrad2 to hide the tail associated with the last experts to land.

  1. Combine ∥ wgrad1. Up to 128 dual-role movers (Appendix Table B.3) push dXdX row blocks back to their token owners as dgrad1 completes them, while builders run wgrad1. In BF16, wgrad1 is interleaved into dgrad1; in MXFP8, wgrad1 uses the column-wise XX, which arrives late when sent in the backward. All of dgrad1 runs first to cover its arrival; wgrad1 then overlaps the combine.

  1. Fold. Each token’s kk gradient rows and the router’s dXrdX_r are summed in FP32 into dXdX.

Implementation Details

KOSMOS is written in C++20 and HIP using eight waves and 512 threads per WG; local data share and register usage are kept within single occupancy. GEMM tiles use HipKittens ([18]), a C++ domain-specific language for AMD GPUs, with 256x256 output tiles for all expert and TP+SP GEMMs; the router GEMM uses a 64x128 tile as it only has as many columns as experts. Shape-dependent parameters (mover crew size, transport parameters, GEMM parameters) are fixed with closed-form rules during the first training iteration (Appendix Table B.3).

Evaluation

Experimental Setup

System Specifications

Two single-node systems were used, with eight GPUs (MI350X or MI355X ([2 b]) each, fully connected over XGMI. Both nodes contained two AMD Zen 5 CPUs with 64 cores each. While MI350X runs at a peak clock of 2.2 GHz at up to 1000W, MI355X runs at a peak of 2.4 GHz at up to 1400W. MI350X offers 2.31 PFLOPS of peak dense BF16 matrix multiplication, while MI355X is slightly higher at 2.52 PFLOPS. The peak performance of MXFP8 is 4.62 PFLOPS and 5.03 PFLOPS, respectively.

The benchmarks were run on ROCm 7.14 ([4]) with RCCL 2.30.4 ((AMD, n.d.c)), PyTorch (Paszke et al., 2019) 2.12, ROCm Transformer Engine ((AMD, n.d.f)) 2.18, and ROCm Megatron-LM ((AMD, n.d.d)) commit b0695a0.

Integration

The microbenchmarks were run with a standalone harness in KOSMOS. For TP+SP, a RCCL collective and a GEMM are run serially as a baseline (hipBLASLt ((AMD, n.d.a)) in BF16, a communication-free HipKittens GEMM in MXFP8). Each configuration is run with 24 shapes arising from the Llama3 (Grattafiori et al., [17]) class of models. The 8B, 70B, and 405B model sizes are used with a microbatch size between 1 and 8 and sequence length 8192. MoE runs one layer, both forward and backward, in a standalone harness. Layers of nine MoE models are used: DeepSeek V3 ([15]), DeepSeek V4 Pro ([16]), Qwen3-30B and Qwen3-235B (Yang et al., 2025), Qwen3.5-397B ((Qwen Team, 2026)), Kimi K2.7 Code ((Moonshot AI, 2026)), GLM 5.2 ((Z.ai, 2026)), and Llama 4 Scout and Maverick ((Meta AI, 2025)). The MoE models use sequence length 4096 and MBS 2. KOSMOS is tested against baselines with balanced and skewed routing to simulate different training scenarios.

End-to-end runs are executed using ROCm Megatron-LM. KOSMOS TP+SP kernels are transplanted into TE and run against Megatron-LM using the default ROCm TE path. The 405B model is restricted to 24 layers to fit on a single node. For MoE, the KOSMOS adapter replaces Megatron’s router, token dispatcher, grouped multilayer perceptron (MLP), and shared expert if present. Due to resource constraints, a proxy Qwen-235B model with 24 layers and a DeepSeek-V3 proxy with no dense layers and nine MoE layers are considered. TP+SP models use a global batch size (GBS) of 64, and MoE models use a GBS of 16.

Numerics & Validation

KOSMOS kernels are bitwise run-to-run deterministic: tile claiming changes which WG computes a tile, never the reduction or fold order. Outputs are validated against RCCL+hipBLASLt with ROCm TE’s tolerances ((AMD, n.d.f)), re-poisoned every iteration, and checked twice to catch races. Microbenchmarks run and discard 10 warm-up iterations before 25 timed iterations. Each is run three times with the variant order shuffled, and we report the median of the slowest rank; the median run-to-run spread is 0.4% (max 2.0%). End-to-end results report Megatron-LM’s printed iteration time and throughput. Each configuration is run three times for 20 iterations, with the median result presented.

MoE TFLOPS count actual routed FLOPs.

Ablations

Data Readiness Mechanisms

Table 1 isolates the design decisions behind KOSMOS, changing one mechanism at a time with everything else held fixed. The largest effects come from the order in which data is produced and consumed: local-first AG queues consumed in gather order, band-major RS production, and destination-minor A2A push lists. KOSMOS is optimized for bandwidth per CU: its 8-wave movers saturate XGMI links with half the CUs of a 4-wave kernel (Appendix Table A.4). Fusing the router and shared expert into the MoE launch is a small but consistent gain. Readiness-driven switching only pays off if readiness arrives where WGs look next: Thus, the order of production is as important as the dynamic role switching it enables.

MechanismAlternativeAlternative vs KOSMOS
AG queue orderNo local-first, chunk-major1.17–1.40× slower
RS production orderRow-major1.41–1.45× slower
A2A push orderDestination-major1.12–2.03× slower
8-wave movers4 waves per WG (7–56 WGs)1.23–1.96× slower
Router fusionSeparate router kernels1.01–1.02× slower
Shared-expert fusionSeparate kernels1.02–1.04× slower

Table 1. Design mechanisms ablated in isolation: slowdown of the alternative relative to KOSMOS. Full results in Appendix Appendix § 9.

Role-Switching Policy

To measure the impact of KOSMOS’ derived role switching policies, we compare them to several alternatives (Table 2). For TP+SP, as implied by the one-way relationship of AG/RS and their GEMMs, we found that one-way role switching overall outperformed two-way role switching. One-way role switching is never more than 1% slower than the best single fixed split, and up to 1.15× faster. For AG, communication-bound shapes generally do not benefit from role switching, as movers are active until the very last step: While MLP layers improve by a geometric mean of 1.05×, attention layers average 1.00×. Per-shape results for both GPUs are in Appendix Figures A.2 and A.3.

MI350XMI355X
One-way switchTwo-way switchDedicated combinePlanned splitOne-way switchTwo-way switchDedicated combinePlanned split
TP+SP AG+TN1.032{sup}*`0.965––1.033{sup}*`0.932––
TP+SP TN+RS1.031{sup}*`1.024––1.018{sup}*`1.012––
TP+SP AG+NN1.018{sup}*`0.951––1.017{sup}*`0.918––
MoE fwd, balanced–1.029{sup}*`1.0180.985–1.027{sup}*`1.0080.982
MoE fwd, skewed–1.047{sup}*`1.0290.989–1.051{sup}*`1.0190.914
MoE bwd, balanced–1.052{sup}*`1.0120.973–1.067{sup}*`1.0090.836
MoE bwd, skewed–1.064{sup}*`1.0210.963–1.093{sup}*`1.0090.752

Table 2. Role-policy ablation: geometric-mean speedup over a fixed communication crew. TP+SP: 24 shapes per kernel, fixed crew of 56 WGs. MoE: nine models, fixed crew is the measured peak per shape, also used by the dedicated combine crew (following UniEP (Zheng et al., 2026)). Per-step planned split following Weave (Huang et al., [19]). All policies are implemented in KOSMOS. {sup}*`Shipped KOSMOS policy; bold: fastest per row and GPU.

For MoE, we show the benefits of our two-way role switch in KOSMOS over alternatives. KOSMOS uses an interleaved two-way role switching policy, where WGs decide their role after every tile. We compare this to a policy where dispatch is executed with one-way role switching and combine uses a dedicated WG crew, following UniEP (Zheng et al., 2026), and to a static two-way planned steal, following Weave (Huang et al., [19]), where movers finish dispatch, compute a predetermined number of tiles from a runtime cost model, then return to combine. We find the two-way role switch not only beats a fixed split in all 36 cells on both MI350X and MI355X, but is the best policy on average, with a geometric mean speedup of 1.04×/1.04× on the forward pass and 1.06×/1.08× on the backward pass (MI350X/MI355X). This gain is up to 1.14×/1.16×. Appendix Figures A.4 and A.5 show the speedups per shape on the MI350X and MI355X, respectively.

TP+SP Speedups

TP+SP Microbenchmarks

Figure 7 compares each KOSMOS kernel to the same RCCL collective and hipBLASLt/HipKittens GEMM run without overlap. KOSMOS is faster on all 480 combinations of shape, kernel, precision, and GPU. In BF16, we see a geometric mean speedup of 1.38× on MI350X and 1.41× on MI355X. In MXFP8, the geometric mean is 1.36× on MI350X and 1.39× on MI355X. The MXFP8 GEMMs have twice the peak throughput of BF16, while an MXFP8 all-gather moves 33/64 of the BF16 bytes and the reduce-scatters move BF16 in both precisions, which leaves less compute to hide. Compared against unfused KOSMOS communication and GEMM kernels run without overlap gives similar geometric mean speedups: For BF16, the speedup is 1.39×/1.42×, and for MXFP8 the speedup is 1.37×/1.40×. This shows that most of the benefit over the baselines is due to the overlap mechanisms, rather than the components.

TP+SP microbenchmark speedup charts

Figure 7. TP+SP microbenchmark: KOSMOS speedup over the baseline, geometric mean over 24 Llama 3 shapes per config.

TP+SP End-to-End Training

Table 3 reports Llama 3 training in Megatron with TP 8, where KOSMOS replaces Transformer Engine’s TP+SP collectives and GEMMs. KOSMOS is faster for every model size, precision, and GPU model.

ModelPrec.MI350X SpeedupMI350X TFLOPSMI355X SpeedupMI355X TFLOPS
8BBF161.20–1.27474→5961.24–1.28478→612
MXFP81.21–1.25580→7181.22–1.24597→730
70BBF161.15–1.21664→7691.22–1.28654→834
MXFP81.19–1.22893→10671.21–1.23907→1119
405BBF161.10–1.16570r^{r}→645r^{r}1.17–1.22624r^{r}→757r^{r}
MXFP81.10–1.14862r^{r}→985r^{r}1.16–1.18947r^{r}→1101r^{r}

Table 3. Llama 3 TP+SP end-to-end training, TP 8, sequence 8192, GBS 64: KOSMOS speedup over ROCm Transformer Engine (range over MBS 1–8, same MBS in both arms) and TFLOPS/GPU at each arm’s best MBS (TE → KOSMOS). r^{r}Full recompute in both arms; 70B at MBS 4 and 8 runs all 80 layers, and 405B runs 24 of its 126 layers. Per-MBS results: Appendix Table C.1.

Mixture-of-Experts Speedups

MoE Microbenchmarks

Figure 8 compares one KOSMOS MoE layer against RCCL A2A with TE’s grouped GEMMs. The figure covers nine models, both forward and backward, with balanced and skewed routing. KOSMOS is faster in all 144 cases, 1.28–3.39×. Gains shrink as GEMM compute increases and top-kk decreases: Qwen3-30B (narrow experts, top-8) gains up to 3.39×, while Llama 4 (wide experts, top-1) gains 1.28–1.92×. The forward pass with balanced routing gains the most, and backward pass with skewed routing is sped up the least. We also run KOSMOS components as separate launches; they are 1.25–1.51× slower than the fused layer on geometric mean which we attribute to kernel fusion.

MoE layer speedups over baselines for KOSMOS

Figure 8. MoE layer speedups over baselines for KOSMOS. Bars are per-model speedup for each direction, precision, and routing.

MoE Proxy Training

Table 4 reports end-to-end training of two MoE proxy models. For both, balanced refers to random routing with a uniform distribution, so each expert has an equal expected number of tokens. Skewed results refer to a pure top-kk routing defined by each model. For DeepSeek V3 with Megatron-generated mock data, this was seen to be up to 1.3× the data of balanced for the largest peer, whereas Qwen saw skew up to 4.2× in some iterations. KOSMOS relies on Megatron’s capacity factor flag to size shared buffers accordingly. A greater skew falls back to the default Megatron MoE layer. KOSMOS is faster than Megatron-LM with ROCm’s DeepEP (Zhao et al., 2025a; AMD, 2026) in every configuration, by 1.29–1.83× across GPU models.

MI350XMI355X
RoutingPrec.SpeedupTFLOPSSpeedupTFLOPS
Qwen3-235B
balanced BF161.70359→6101.74387→673
MXFP81.82372→6771.83400→731
skewed BF161.31249→3271.29275→355
MXFP81.57245→3841.54268→411
DeepSeek-V3
balanced BF161.31423→5561.33480→637
MXFP81.39446→6191.39500→696
skewed BF161.33414→5511.36465→631
MXFP81.40437→6131.41488→689

Table 4. MoE proxy training, EP 8: KOSMOS speedup over DeepEP, and TFLOPS/GPU (DeepEP → KOSMOS). Balanced: uniform random routing; skewed: the model’s top-k router. Full results: Appendix Table C.2.

RELATED WORK

Stream-level overlap. Frameworks [22 NVIDIA, n.d.b] run NCCL/RCCL (NVIDIA, n.d.a; AMD, n.d.c) collectives on separate streams, where they hold CUs

for the whole transfer and compete with the GEMM. CoCoNet [20] adds device-side locks, chunking [23] (Wang et al. 2022) shrinks the GEMMs, and copy engines (Zheng et al., 2025b) or new hardware (Pati et al., 2024) cannot reduce or signal per tile.

Fused communication kernels. FLUX [13] fuses RS into a GEMM epilogue and AG into its prologue; ParallelKittens (Sul et al., 2026a), TileLink (Zheng et al., 2025a), and Iris [12] fuse TP+SP collectives per tile with a tuned number of communication streaming multiprocessors (SMs) or WGs. For MoE, COMET (Zhang et al., 2025), Mixture-of-Kittens (Sul et al., 2026b), and the MegaMoE kernels of DeepSeek-V4 [16] (Zhao et al. 2025b) and Primus-Turbo [6] fix communication resources at build time or per shape, before routing is known; Mixture-of-Kittens (a second backward megakernel) and Primus-Turbo (fused operators reused in backward) compute the router outside the kernel. FlashMoE (Aimuyo et al., [1]) and MPK (Cheng et al., [14]) are persistent inference kernels driven by scheduler warps. FLUX, ParallelKittens, COMET, and Mixture-of-Kittens are NVIDIA-only, so § 5.2 instead reproduces the UniEP and Weave policies inside the KOSMOS kernels.

Adaptive communication budgets. UniEP (Zheng et al., 2026) orders one MoE task queue communication-first, with a split set by a cost model and tuner; Weave (Huang et al., [19]) sizes its communication SMs with an in-kernel cost model and lets them steal a precomputed number of GEMM tiles; mKernel (Mao et al., [24]) adapts the split with an in-kernel controller but trails its best fixed split. In KOSMOS, roles follow data readiness alone, with no scheduler, task graph, per-call cost model, or steal budget, and one launch per direction covers every TP+SP collective with its GEMM, and the whole MoE layer, router and gradients included.

Conclusion

Current state-of-the-art overlaps communication in TP+SP and EP training by running it on a separate stream, or by giving it a fixed or predicted share of the GPU inside a fused kernel. We show that on CDNA4 GPUs, both approaches are sub-optimal. Concurrent streams may contend without coordination, and a fixed split strands CUs whenever the demand for communication changes. KOSMOS addresses both by running communication and computation in one self-scheduling persistent kernel per operation or layer. Its WGs switch between moving and building as the work available changes, with in-kernel queues rather than a separate scheduler. Because collectives are limited by CUs up to saturation, giving CUs to communication costs little GEMM throughput, if the kernel adapts. On MI350X and MI355X nodes, KOSMOS speeds up dense model training by up to 1.28× (BF16) and 1.25× (MXFP8), and MoE training by up to 1.74× (BF16) and 1.83× (MXFP8) over DeepEP.

KOSMOS kernels are persistent only for intranode transports: XGMI with hipIPC memory is currently required. Remote direct memory access support and likely a hierarchical set of movers would carry the design to multinode. Thus, end-to-end results are limited to a single-node proxy model. We have evaluated with quasi-FSDP and pipeline parallelism stress tests, but complete multinode verification is future work. Finally, our structure requires one completion point per output tile. This disqualifies GEMM algorithms, such as stream-K (Osama et al., 2023), that compute a single output tile using multiple WGs. Other workloads built on fine-grained collectives, such as all-reduce as used in TP-only training and inference; SUMMA (Van De Geijn & Watts, 1997) and sparse distributed GEMMs; 3D fast Fourier transforms and spectral methods; and stencil solvers could use the same role-switching kernel design.

Overlapping collectives with computation is no longer a question of how many CUs to allocate to communication, but of when. KOSMOS answers this with self-scheduling kernels that produce in the order consumers need, and lets data readiness drive role switches.

Ablations

WGs out of Order

While workgroups of a persistent grid are all resident, they do not start in ID order. The dispatcher has tendencies to fill sorted by XCD rather than ID, but this is not guaranteed either. With oversubscription, the order depends on per-WG work. Variable work per WG can lead to highly unpredictable start order.

Workgroup launch order and time sorted plotted against WG ID number

Figure A.1. Workgroup launch order and time sorted plotted against WG ID number. For persistent kernels, all WGs are resident but order is not guaranteed, and instead leans towards an accelerator complex die (XCD) ID-sorted launch order. With oversubscription, 1024 WGs exhibit different behavior based on the workload assigned to each WG.

Role Switching Policies

These figures give the per-shape results behind Table 2. For TP+SP, one-way switching is never significantly slower than the best fixed split, but up to 15% faster. For MoE, two-way switching beats a fixed split in all 72 cells, and is faster than a dedicated split in 63 of 72 cells and within 2% in the rest. Planned splits appear to lose where routing is skewed within a step, as a fixed crew size is determined before data arrives.

MI350X speedup of one-way and two-way role switching across three fused training formats

Figure A.2. MI350X: speedup of one-way and two-way role switching over a best single fixed split of 56 communication WGs (8 per peer) for the three fused formats in training: AG+TN GEMM, TN GEMM+RS, and AG+NN GEMM. Switching arms use the KOSMOS heuristic crew (Table B.3).

Three-panel bar charts showing speedup of one-way and two-way role switching for AG+TN, TN+RS, and AG+NN fused formats

Figure A.3. MI355X: speedup of one-way and two-way role switching over a best single fixed split of 56 communication WGs (8 per peer) for the three fused formats in training: AG+TN GEMM, TN GEMM+RS, and AG+NN GEMM. Switching arms use the KOSMOS heuristic crew (Table B.3).

KOSMOS speedup over a fixed split of different role-switching policies in MoE on MI350X

Figure A.4. MI350X: speedup over a fixed split of different role-switching policies in MoE. KOSMOS two-way role switching with data readiness role switching, a UniEP-style one-way role switching for dispatch with a dedicated combine crew, and a Weave-style planned split. Crew size for non-KOSMOS policies is the smallest that reaches 98% of the peak dispatch and combine bandwidth (24, 32, or 48 WGs)

KOSMOS speedup over a fixed split of different role-switching policies in MoE on MI355X

Figure A.5. MI355X: speedup over a fixed split of different role-switching policies in MoE. KOSMOS two-way role switching with data readiness role switching, a UniEP-style one-way role switching for dispatch with a dedicated combine crew, and a Weave-style planned split. Crew size for non-KOSMOS policies is the smallest that reaches 98% of the peak dispatch and combine bandwidth (16, 24, 32, or 48 WGs)

Fused vs Separate Router and Shared Expert

Running the router or the shared expert as separate kernels is 1.011.01–1.02×1.02\times and 1.021.02–1.04×1.04\times slower, respectively, than the fused launch (Table A.1).

ComponentModelPassFused (ms)Separate (ms)Without (ms)Speedup
Routerdsv3forward7.5577.6127.1681.007
Routerdsv3backward13.77314.00513.7111.017
Routerqwen235forward3.6223.6793.3981.016
Routerqwen235backward5.8085.9185.7521.019
Shared expertdsv3forward7.6557.9907.2181.044
Shared expertdsv3backward15.53315.79713.8651.017

Table A.1. Fusing the router and the shared expert into the MoE launch. Fused (KOSMOS): one launch. Separate: The router (GEMM, top-k, aux loss) or the shared expert runs as its own kernels (hipBLASLt for the shared-expert GEMMs) on the same stream. Without: the same MoE layer with routing precomputed (router rows) or routed experts only (shared-expert rows).

Local First AG Queue vs Naive Ordering

A naive rank-order queue without local-first is 1.171.17–1.40×1.40\times slower than the shipped queue (Table A.2).

ConfigShapeShipped (ms)Chunk-majorNaive
TN70B-FC1-MBS87.5798.040 (+6.1%)8.873 (+17.1%)
TN405B-QKV-MBS85.5527.314 (+31.7%)7.742 (+39.4%)
NN70B-Down-MBS84.0384.821 (+19.4%)5.109 (+26.5%)
NN405B-OProj-MBS85.1847.022 (+35.5%)7.240 (+39.7%)

Table A.2. AG consumption order, configs 1 (AG+TN) and 3 (AG+NN), TP=8, atomic tile claim (queue order is the schedule). Shipped: local shard first, then peer tiles in the order movers gather them. Chunk-major: local first, peers chunk by chunk (mismatches the gather order). Naive: chunk-major in rank order, no local-first. Time: slowest rank, median of 3 runs × 25 calls; slowdown vs. shipped in parentheses.

Band-Major vs. Row-Major RS Queue

Row-major production is 1.411.41–1.45×1.45\times slower than band-major production (Table A.3).

ShapeBand-major (ms)Row-major (ms)Row / band
8B-OProj-MBS81.3661.9301.41×1.41\times
70B-Down-MBS84.1566.0381.45×1.45\times

Table A.3. RS production order, config 2 (GEMM →\to RS), TP=8. Band-major (shipped) produces band bb of every rank’s chunk before band b+1b+1, so reducers start on the first band; row-major produces each chunk completely before the next.

8 vs. 4 Waves

With four waves per WG, transport is 1.231.23–1.96×1.96\times slower measured by bytes/CU/s at 7–56 WGs (Table A.4).

OpWGs8 waves (shipped)4 waves4 waves, depth 24 waves, pipelined
AG79.77219.13110.52519.460
AG145.0569.7915.4519.920
AG282.7345.0442.8995.063
AG562.2172.7272.2202.729
AG1122.2782.2302.2732.237
RS715.90030.37818.490–
RS147.98815.2059.263–
RS284.0687.6274.683–
RS562.2883.8782.513–
RS1122.3252.2362.259–

Table A.4. Mover waves vs. request depth, 1 GiB transfer (70B-MBS8), TP=8. The shipped 8-wave mover keeps one 16-byte remote load in flight per wave (two waves per single instruction, multiple data unit). 4-wave variants: same loop (depth 1), two loads in flight per wave (depth 2), or load/store interleave (pipelined, AG only). (Bold: fastest)

Dst-Minor vs. Dst-Major Push

Destination-major push lists are 1.12–2.03×\times slower than destination-minor lists (Table A.5).

RoutingPhaseDestination-minor (ms)Destination-major (ms)Slowdown
BalancedForward8.14816.5322.03×\times
BalancedBackward13.78920.5671.49×\times
SkewedForward18.68624.8411.33×\times
SkewedBackward30.07733.6531.12×\times

Table A.5. Push order for EP transport, dsv3, EP=8, routed experts only (router and shared expert excluded). Destination-minor (KOSMOS): Consecutive WGs push to different destinations, one destination per WG. Destination-major: Consecutive WGs push to the same destination. Same kernels and data.

RS Error vs. RCCL Ring, against a FP64 Reference

While KOSMOS accumulates all eight partials in FP32 and rounds to BF16 or MXFP8 once, ring algorithms often keep BF16 partials and round at every hop. The result is 2.21×\times lower reduction error for KOSMOS, and 1.71–1.73×\times lower error against a FP64 GEMM. KOSMOS is also bitwise run-to-run deterministic.

ShapeReduction: KOSMOSReduction: RCCLReduction: Ratioend-to-end: KOSMOSend-to-end: RCCLend-to-end: Ratio
8B-OProj-MBS81.67×10−31.67 \times10^{-3}3.68×10−33.68 \times10^{-3}2.212.34×10−32.34 \times10^{-3}4.04×10−34.04 \times10^{-3}1.73
70B-Down-MBS81.67×10−31.67 \times10^{-3}3.68×10−33.68 \times10^{-3}2.212.35×10−32.35 \times10^{-3}4.02×10−34.02 \times10^{-3}1.71

Table A.6. RS error, TP=8. Every arm reduces the same BF16 partials. Reduction: vs. the FP64 sum of the partials. end-to-end: vs. an FP64 GEMM (32,768-element sample).

Push vs Pull

While pushing rows into the destination’s arena is slightly faster than pulling them this requires more memory than a pull algorithm. As such, for larger EP sizes it may be required to use a pull algorithm instead.

ModelRoutingPhasePush (ms)Pull (ms)Speedup
dsv3balancedforward8.1178.1581.005
dsv3balancedbackward13.78313.7250.996
dsv3balancedstep21.04321.0250.999
dsv3skewedforward18.62819.0041.020
dsv3skewedbackward30.05530.0521.000
dsv3skewedstep45.14445.5061.008
qwen235balancedforward3.7833.8491.017
qwen235balancedbackward5.7105.6990.998
qwen235balancedstep9.2769.3221.005

Table A.7. Push vs. pull EP transport in the full MoE layer (EP=8). Push (KOSMOS): The producer writes rows into the destination’s buffer. Pull: The consumer reads them from the source in the forward dispatch; combine and the backward push in both arms, so the backward rows are a control. Speedup: pull over push time.

Additional KOSMOS Details

Fused Communication Framework Comparison

SystemScopeTrainF/BCrew sized byRolesSwitch triggerSwitches /unitRtrwgradTP+SP
FLUX [13]pair✓––own tile–0––✓
TileLink (Zheng et al., 2025a)pair✗–launch configfixed–0––✓
ParallelKittens (Sul et al., 2026a)pair✗–launch configfixed–0✗✗✓
Iris [12]pair✗–launch configfixed–0––△
COMET (Zhang et al., 2025)pair△∼2 / ?offline, per shapefixed–0✗?✗
Primus MegaMoE (AMD, n.d.b)pair✓3 / 5ᵃautotuned per shapefixed–0✗part.ᵃ✗
UniEP (Zheng et al., 2026)pair✓3 / 6ᵃperf-model tunerone-waycomm tasks exhausted, queue1✗✗ᵃ✗
FlashMoE [1]layer†✗1 / –schedulersched. tasksscheduler warpper task✓–✗
DeepSeek MegaMoE (Zhao et al., 2025b)layer†✗1+aux / –launch configfixed warps–0✗–✗
Weave [19]layer†✗1+aux / –prologue cost modelᵇplanned phasesprecomputed steal countᶜ≤2✗–✗
Mixture-of-Kittens (Sul et al., 2026b)layer✓1+aux / 1+auxlaunch configfixed–0✗ᵈ✓✗
MPK [14]model✗1 / –schedulersched. tasksscheduler SMsper task✓–✗
KOSMOSpair + layer✓1 / 1shape modelone-way + two-wayreadiness at every claimᵉunbounded✓ᶠ✓✓

Table B.1. Fused communication–computation systems. Scope: what one launch fuses (a collective–GEMM pair or a whole MoE layer). Crew sized by: how the number of communication units is chosen. Roles/Switch trigger: whether and when a unit changes between communication and computation. Rtr/wgrad: router and weight gradients inside the launch.

Supported TP+SP Configs

PassCollectiveGEMMKernel
fwdAG XXcol-parallelfused AG+TN
fwdRS YYrow-parallelfused TN+RS
bwdAG dYdYrow-parallel dgradfused AG+NN
bwdAG dYdY, AG dY⊤dY^{\top} †row-parallel dgrad + wgradfused AG+NN(+NT) †
bwdAG XXcol-parallel dgradbulk AG ‖ NN
bwdRS dXdXcol-parallel wgradbulk RS ‖ NT

Table B.2. TP+SP KOSMOS kernels per collective-GEMM pair (†: MXFP8 only).

Heuristics

ParameterKernelInputRule
Movers per peerAG+GEMM, col. fpropgathered width NNBF16: 6 if NN is at least 3584, else 8. MXFP8: 8.
AG+GEMM, row dgradgathered width NNBF16: 6 if NN is at least 4608, else 8. MXFP8: 8.
bulk AGcontraction KKBF16: 2, 4, 6 if KK is at least 7168, 3584, 2304; else 8. MXFP8: 2, 4 if KK is at least 3584, 1792; else 8.
Reducers per peerGEMM+RS, row fproplocal KK, waves ww (tiles per cap)KK at least 4096: 4 (MXFP8 5). KK at least 3072: 6 (MXFP8 8 if ww is at least 16, else 7). KK at most 768: 14, 10, 8 if ww is at least 16, at least 8, or less. Otherwise 8 (MXFP8 10 if ww is at least 16).
bulk RSoutput width nnBF16: 2, 4 if nn is at least 8192, 3584; else 8. MXFP8: 8 if nn is 5120 to 8191, else 4.
Dispatch crewMoE forwardexpert width IIBF16: 32, 24, 16 if II is below 1024, below 3072, or larger. MXFP8: 24576/II rounded to a multiple of 8 between 16 and 48, doubled if the forward also sends the column-wise input. Minimum over ranks.
Combine crewMoE forwardCU capThe smaller of 128 and the cap, rounded down to a multiple of 8.
Dispatch crewMoE backwardtotal bytes, total GEMM work256 WGs times push time over push plus GEMM time, rounded up to a multiple of 8, between 32 and 64.
Combine crewMoE backwardtotal bytes, total GEMM work256 WGs times push time over GEMM time, rounded up to a multiple of 8, between 24 and 128 (MXFP8 32 and 128).
Push burst depthMoE dispatch, combinehidden size HHLargest of 8, 7, 6, 4 that divides H/512H/512.
Split-K factorAG+GEMM row dgrad, bulk AGoutput tilesBF16: 4 or 2 if the FP32 partials fit the workspace (at most 64 MiB), else 1. MXFP8: 1.
bulk RSoutput tilesBF16: 4 if at most 64 tiles and the partials fit the workspace, else 1. MXFP8: 1.
XCD bucketingAG+GEMM (all)tiles, tile rowsOn if at most 1024 tiles and at least 5 tile rows.

Table B.3. Per-shape heuristics, evaluated once per shape on the host. Rules were fitted at TP 8 and EP 8; at other TP sizes mover count is scaled. Mover counts are per remote peer; MoE crews are WGs per GPU. All counts are then fitted to the CU cap so that every mover runs beside at least one builder.

Full Per-Shape and Per-GPU Tables Behind the Summary Figures

ModelMBSLayersBF16 TE baseline ms/iterBF16 KOSMOS ms/iterBF16 KOSMOS TFLOPSBF16 SpeedupMXFP8 TE baseline ms/iterMXFP8 KOSMOS ms/iterMXFP8 KOSMOS TFLOPSMXFP8 Speedup
MI350X
8B13210056.27913.1426.61.2718708.36967.7484.91.250
8B2327727.66435.7524.41.2016729.25454.8619.11.234
8B4327119.45811.7580.71.2255939.64902.3688.51.212
8B8327117.25658.0596.41.2585813.04702.4717.61.236
70B18047371.041279.4713.31.14837514.830841.8954.71.216
70B28044371.238308.3768.61.15832956.227603.61066.61.194
70B480r80^{r}62232.351524.3571.41.20843348.736146.8814.51.199
70B880r80^{r}62377.751415.7572.61.21343643.736258.4812.01.204
405B124r24^{r}56475.751236.1628.11.10237706.934361.3936.51.097
405B224r24^{r}57459.549923.4644.61.15137438.232840.9979.91.140
405B424r24^{r}57912.549899.2644.91.16137347.732671.8985.01.143
405B824r24^{r}58049.149998.7643.61.16137655.0OOM––
MI355X
8B1329996.17915.3427.01.2638647.16971.3485.51.240
8B2327876.56351.8531.31.2406664.85368.7628.91.241
8B4327220.75709.9591.01.2655893.84833.4698.51.219
8B8327059.05515.6611.81.2805649.14619.6730.51.223
70B18048695.239918.9737.61.22037191.530146.4976.71.234
70B28045029.735307.9833.91.27532447.126317.11118.81.233
70B480r80^{r}59942.347738.9616.71.25642303.434680.9848.91.220
70B880r80^{r}58645.846949.1627.11.24941509.834187.4861.21.214
405B124r24^{r}54176.846203.8696.51.17337137.231882.31009.41.165
405B224r24^{r}52786.943727.3735.91.20735251.229995.21072.91.175
405B424r24^{r}52032.342743.1752.91.21734474.229407.81094.31.172
405B824r24^{r}51565.342513.7756.91.21333989.829226.61101.11.163

Table C.1. Table C.1. Llama 3 TP+SP end-to-end training, TP 8, sequence 8192, GBS 64: median over three runs of Megatron-LM’s printed numbers (each the mean over iterations 3–19). Speedup is TE baseline ms/iter over KOSMOS ms/iter. OOM: out of memory. {sup}r`Full recompute in both arms. Layer-reduced proxies: The 405B model runs 24 of its 126 layers with full recompute. 70B at MBS 4 and 8 runs with full recompute. Three runs per point, 20 iterations each.

Table C.2. MoE proxy training, EP 8: median over three runs of Megatron-LM’s printed numbers, 20 iterations each. KOSMOS speedup is the DeepEP ms/iter over KOSMOS’s. DeepEP: Megatron-LM with ROCm’s DeepEP. Balanced: uniform random routing; skewed: the model’s top-k router.

ModelRoutingBackendBF16 ms/iterBF16 TFLOPSBF16 KOSMOS speedupMXFP8 ms/iterMXFP8 TFLOPSMXFP8 KOSMOS speedup
MI350X
Qwen3-235BbalancedKOSMOS545.1610–491.4677–
DeepEP925.63591.698894.73721.821
Qwen3-235BskewedKOSMOS1022.6327–868.1384–
DeepEP1337.72491.3081359.22451.566
DeepSeek-V3balancedKOSMOS612.7556–550.1619–
DeepEP804.84231.314762.84461.387
DeepSeek-V3skewedKOSMOS617.1551–555.4613–
DeepEP822.74141.333778.54371.402
MI355X
Qwen3-235BbalancedKOSMOS494.3673–454.9731–
DeepEP859.73871.739831.14001.827
Qwen3-235BskewedKOSMOS941.6355–810.5411–
DeepEP1211.82751.2871244.22681.535
DeepSeek-V3balancedKOSMOS534.3637–488.7696–
DeepEP709.74801.328680.05001.391
DeepSeek-V3skewedKOSMOS539.7631–493.8689–
DeepEP732.14651.356697.94881.413

Table C.2.

Table C.3. TP+SP microbenchmark, AG+GEMM, column fprop, TP 8, MI350X: per iteration the slowest rank, median. Base: the RCCL collective and the GEMM run back to back on one stream with no overlap (BF16: torch.mm (hipBLASLt); MXFP8: a communication-free HipKittens MXFP8 GEMM). Speedup is base over KOSMOS; vs unf. is KOSMOS’s communication and GEMM WGs run back to back over the fused kernel.

ShapeMNKBF16 Base msBF16 KOSMOS msBF16 SpeedupBF16 vs unf.MXFP8 Base msMXFP8 KOSMOS msMXFP8 SpeedupMXFP8 vs unf.
405B-FC1-MBS1819213312163843.9943.3201.2031.1881.9031.6511.1521.100
405B-QKV-MBS181922304163841.1280.9261.2181.3660.7400.5211.4201.343
70B-FC1-MBS18192716881921.0910.9951.0971.0720.6130.5391.1371.134
70B-QKV-MBS18192128081920.5160.4691.1011.0280.3080.2811.0971.044
8B-FC1-MBS18192358440960.5020.3101.6191.2410.2590.2091.2421.180
8B-QKV-MBS1819276840960.3770.2341.6111.0410.1920.1651.1671.040
405B-FC1-MBS21638413312163848.1086.9081.1741.1373.9433.4761.1341.086
405B-QKV-MBS2163842304163842.1701.5691.3831.4341.3490.8831.5291.413
70B-FC1-MBS216384716881921.9631.7341.1321.1651.0920.9661.1321.181
70B-QKV-MBS216384128081920.9930.7561.3131.2620.5830.4421.3181.240
8B-FC1-MBS216384358440960.7660.5441.4091.3340.4440.3461.2821.265
8B-QKV-MBS21638476840960.4680.3981.1741.0360.2690.2481.0831.048
405B-FC1-MBS432768133121638416.16913.5251.1961.1238.4187.4551.1291.067
405B-QKV-MBS4327682304163844.2682.8321.5071.5152.4931.5891.5681.463
70B-FC1-MBS432768716881924.6643.5301.3211.2372.1741.8941.1481.190
70B-QKV-MBS432768128081921.8321.3531.3541.2881.0660.7861.3571.277
8B-FC1-MBS432768358440961.3480.9381.4361.4600.8220.6191.3281.295
8B-QKV-MBS43276876840960.8730.6931.2601.1940.4890.4111.1891.177
405B-FC1-MBS865536133121638431.27426.6511.1731.13116.92114.9831.1291.085
405B-QKV-MBS8655362304163849.0825.5781.6281.6124.9933.1161.6031.485
70B-FC1-MBS8655367168819210.0357.6781.3071.2575.0064.2381.1811.159
70B-QKV-MBS865536128081923.5282.5321.3941.3442.0741.4791.4021.317
8B-FC1-MBS865536358440962.7511.8291.5051.5361.6211.2251.3241.280
8B-QKV-MBS86553676840961.6681.2891.2941.2190.9190.7661.2001.199
geometric mean1.2521.205

Table C.3.

Table C.4. TP+SP microbenchmark, AG+GEMM, column fprop, TP 8, MI355X: per iteration the slowest rank, median. Base: the RCCL collective and the GEMM run back to back on one stream with no overlap (BF16: torch.mm (hipBLASLt); MXFP8: a communication-free HipKittens MXFP8 GEMM). Speedup is base over KOSMOS; vs unf. is KOSMOS’s communication and GEMM WGs run back to back over the fused kernel.

BF16MXFP8
ShapeMNKBase msKOSMOS msSpeedupvs unf.Base msKOSMOS msSpeedupvs unf.
405B-FC1-MBS1819213312163843.7052.9291.2651.2391.9581.6251.2051.202
405B-QKV-MBS181922304163841.2000.9591.2521.4310.7540.5241.4381.419
70B-FC1-MBS18192716881921.1830.9581.2351.2610.6560.5321.2331.251
70B-QKV-MBS18192128081920.5400.4861.1131.0310.3080.2911.0591.051
8B-FC1-MBS18192358440960.4990.3081.6211.3280.2580.2101.2291.192
8B-QKV-MBS1819276840960.3370.2431.3861.0420.1900.1691.1251.030
405B-FC1-MBS21638413312163847.4275.8481.2701.2083.7353.1761.1761.159
405B-QKV-MBS2163842304163842.3351.6391.4241.5441.3400.9101.4721.491
70B-FC1-MBS216384716881922.2891.6821.3611.3451.2261.0001.2271.247
70B-QKV-MBS216384128081921.0080.8001.2601.3180.5800.4551.2761.270
8B-FC1-MBS216384358440960.7950.5361.4821.4810.4610.3471.3281.319
8B-QKV-MBS21638476840960.4830.4211.1491.0390.2710.2571.0531.045
405B-FC1-MBS432768133121638414.02511.2651.2451.2297.5116.3321.1861.154
405B-QKV-MBS4327682304163844.6482.9241.5901.5812.5251.6641.5181.538
70B-FC1-MBS432768716881924.4903.2061.4011.3712.3931.9231.2451.271
70B-QKV-MBS432768128081921.9261.4331.3441.3471.0500.8161.2881.316
8B-FC1-MBS432768358440961.5340.9431.6261.6240.8470.6201.3661.360
8B-QKV-MBS43276876840960.8910.7341.2151.1860.4970.4301.1551.172
405B-FC1-MBS865536133121638427.34721.9301.2471.23414.56112.4471.1701.157
405B-QKV-MBS8655362304163849.1795.3961.7011.6684.8923.2051.5261.545
70B-FC1-MBS865536716881929.1326.4751.4101.3474.7763.7381.2781.266
70B-QKV-MBS865536128081923.7032.6861.3781.3592.0361.5511.3131.319
8B-FC1-MBS865536358440962.9681.8081.6411.6241.6611.2501.3291.349
8B-QKV-MBS86553676840961.7201.3751.2511.2050.9260.8021.1551.182
geometric mean1.3601.3231.2581.255

Table C.4.

ShapeMNKBF16 Base msBF16 KOSMOS msBF16 SpeedupBF16 vs unf.MXFP8 Base msMXFP8 KOSMOS msMXFP8 SpeedupMXFP8 vs unf.
405B-Down-MBS181921638466562.0151.7031.1831.2521.3821.2531.1031.396
405B-OProj-MBS181921638420481.1400.8191.3921.4600.9670.7661.2611.367
70B-Down-MBS18192819235840.7520.5281.4241.3160.5760.4411.3071.306
70B-OProj-MBS18192819210240.5410.3751.4411.2970.4620.3811.2151.220
8B-Down-MBS18192409617920.3540.2341.5091.3170.2710.2221.2171.238
8B-OProj-MBS1819240965120.2930.2051.4311.2460.2390.2101.1381.203
405B-Down-MBS2163841638466564.4113.6461.2101.2912.7282.4011.1361.464
405B-OProj-MBS2163841638420482.2521.5811.4241.4871.8821.4171.3281.381
70B-Down-MBS216384819235841.3891.0021.3861.4531.1050.7921.3951.560
70B-OProj-MBS216384819210240.9710.7121.3641.3710.8770.7181.2211.333
8B-Down-MBS216384409617920.5930.3811.5551.4770.5020.3841.3091.327
8B-OProj-MBS21638440965120.5030.3561.4131.3090.4400.3791.1611.240
405B-Down-MBS4327681638466569.5817.5711.2651.2875.7134.7631.1991.538
405B-OProj-MBS4327681638420484.3853.0871.4211.5283.6742.7231.3491.398
70B-Down-MBS432768819235842.8371.8801.5091.6012.1671.4361.5091.650
70B-OProj-MBS432768819210241.8861.4461.3041.2981.7001.3501.2591.280
8B-Down-MBS432768409617921.1180.7141.5671.6070.9600.7111.3501.469
8B-OProj-MBS43276840965120.8640.6981.2381.3070.8370.7211.1611.257
405B-Down-MBS86553616384665618.69715.0431.2431.28411.8739.2701.2811.597
405B-OProj-MBS8655361638420489.3506.1961.5091.6027.3225.2971.3821.434
70B-Down-MBS865536819235846.0044.1581.4441.5274.2712.8941.4761.566
70B-OProj-MBS865536819210243.6212.9211.2401.2603.3132.6291.2601.278
8B-Down-MBS865536409617922.1711.4441.5041.5481.8631.3411.3901.434
8B-OProj-MBS86553640965121.7041.3671.2471.3501.6221.3611.1911.306
geometric mean1.3801.3901.2711.379

Table C.5. TP+SP microbenchmark, GEMM+RS, row fprop, TP 8, MI350X: per iteration the slowest rank, median. Base: the RCCL collective and the GEMM run back to back on one stream with no overlap (BF16: torch.mm (hipBLASLt); MXFP8: a communication-free HipKittens MXFP8 GEMM). Speedup is base over KOSMOS; vs unf. is KOSMOS’s communication and GEMM WGs run back to back over the fused kernel.

Table C.6. TP+SP microbenchmark, GEMM+RS, row fprop, TP 8, MI355X: per iteration the slowest rank, median. Base: the RCCL collective and the GEMM run back to back on one stream with no overlap (BF16: torch.mm (hipBLASLt); MXFP8: a communication-free HipKittens MXFP8 GEMM). Speedup is base over KOSMOS; vs unf. is KOSMOS’s communication and GEMM WGs run back to back over the fused kernel.

ShapeMNKBF16MXFP8
Base msKOSMOS msSpeedupvs unf.Base msKOSMOS msSpeedupvs unf.
405B-Down-MBS181921638466562.1641.7951.2061.2711.5141.1841.2791.525
405B-OProj-MBS181921638420481.1940.8401.4211.4331.0160.7701.3201.338
70B-Down-MBS18192819235840.8070.5591.4451.3780.6090.4421.3771.383
70B-OProj-MBS18192819210240.5390.3931.3711.3070.4760.3831.2421.212
8B-Down-MBS18192409617920.3530.2461.4321.3080.2750.2291.2001.218
8B-OProj-MBS1819240965120.2810.2191.2831.2200.2440.2201.1071.173
405B-Down-MBS2163841638466564.3093.3201.2981.3442.9732.1981.3521.603
405B-OProj-MBS2163841638420482.4041.5301.5711.5451.9861.4541.3661.374
70B-Down-MBS216384819235841.5201.0351.4681.5061.1810.7921.4911.572
70B-OProj-MBS216384819210240.9950.7351.3541.3290.9140.7211.2681.286
8B-Down-MBS216384409617920.6200.4011.5451.4930.5170.3881.3341.324
8B-OProj-MBS21638440965120.4830.3831.2611.2670.4520.3841.1761.225
405B-Down-MBS4327681638466568.7646.4671.3551.3515.8084.3651.3311.596
405B-OProj-MBS4327681638420484.6882.9491.5901.5493.8932.7761.4021.400
70B-Down-MBS432768819235843.0471.9111.5941.6302.3151.4581.5881.588
70B-OProj-MBS432768819210241.9721.4561.3541.3081.7771.3931.2761.274
8B-Down-MBS432768409617921.1540.7401.5591.5791.0020.7221.3881.417
8B-OProj-MBS43276840965120.8920.7351.2141.2950.8700.7441.1691.234
405B-Down-MBS86553616384665616.90412.5651.3451.38011.4918.4991.3521.615
405B-OProj-MBS8655361638420489.3295.7961.6101.6127.6975.4081.4231.433
70B-Down-MBS865536819235845.9553.7311.5961.6134.5472.8611.5891.595
70B-OProj-MBS865536819210243.8152.9061.3131.2783.4722.7191.2771.285
8B-Down-MBS865536409617922.2801.4951.5251.5101.9591.3791.4201.425
8B-OProj-MBS86553640965121.7681.4401.2281.3351.6951.4331.1831.294
geometric mean1.4081.4041.3241.384

Table C.6. TP+SP microbenchmark, GEMM+RS, row fprop, TP 8, MI355X: per iteration the slowest rank, median. Base: the RCCL collective and the GEMM run back to back on one stream with no overlap (BF16: torch.mm (hipBLASLt); MXFP8: a communication-free HipKittens MXFP8 GEMM). Speedup is base over KOSMOS; vs unf. is KOSMOS’s communication and GEMM WGs run back to back over the fused kernel.

ShapeMNKBase msKOSMOS msSpeedupvs unf.Base msKOSMOS msSpeedupvs unf.
BF16MXFP8
405B-Down-MBS181926656163842.1721.9761.0991.1272.3141.8401.2581.163
405B-OProj-MBS181922048163841.0690.9401.1371.0701.1710.8711.3451.269
70B-Down-MBS18192358481920.7430.6661.1151.0740.8060.5821.3841.377
70B-OProj-MBS18192102481920.5110.4211.2141.0560.6600.3821.7281.240
8B-Down-MBS18192179240960.4110.2731.5061.0250.4650.2691.7301.386
8B-OProj-MBS1819251240960.3680.2201.6741.1520.3120.2381.3071.101
405B-Down-MBS2163846656163844.2093.6961.1391.2154.9164.1651.1801.145
405B-OProj-MBS2163842048163842.0521.5291.3431.2992.2671.7421.3011.264
70B-Down-MBS216384358481921.4301.1601.2331.1951.5311.0881.4071.397
70B-OProj-MBS216384102481920.8970.7641.1751.0551.0740.6801.5791.442
8B-Down-MBS216384179240960.5750.4211.3661.2560.7370.4551.6191.513
8B-OProj-MBS21638451240960.4630.3801.2171.0960.4840.3971.2201.083
405B-Down-MBS4327686656163849.1287.2531.2581.26010.1038.6791.1641.181
405B-OProj-MBS4327682048163843.9832.7881.4291.4154.5543.6651.2431.205
70B-Down-MBS432768358481922.5952.0701.2531.3512.9932.1601.3861.417
70B-OProj-MBS432768102481921.7311.3511.2811.1801.9521.3291.4691.469
8B-Down-MBS432768179240961.1120.7361.5121.4231.2950.8431.5361.515
8B-OProj-MBS43276851240960.8200.6941.1811.0580.8640.7221.1971.184
405B-Down-MBS86553666561638418.30414.0271.3051.27420.14017.4851.1521.205
405B-OProj-MBS8655362048163848.4865.2051.6301.6299.6077.4151.2961.316
70B-Down-MBS865536358481925.6494.0661.3891.4176.6965.1481.3011.329
70B-OProj-MBS865536102481923.3902.5111.3501.2523.6822.6701.3791.467
8B-Down-MBS865536179240962.0491.3701.4961.5012.4181.6701.4481.551
8B-OProj-MBS86553651240961.5661.2811.2221.1301.6411.3951.1761.201
geometric mean1.3051.2201.3571.302

Table C.7. TP+SP microbenchmark, AG+GEMM, row dgrad, TP 8, MI350X: per iteration the slowest rank, median. Base: the RCCL collective and the GEMM run back to back on one stream with no overlap (BF16: torch.mm (hipBLASLt); MXFP8: a communication-free HipKittens MXFP8 GEMM). Speedup is base over KOSMOS; vs unf. is KOSMOS’s communication and GEMM WGs run back to back over the fused kernel.

Table C.8. TP+SP microbenchmark, AG+GEMM, row dgrad, TP 8, MI355X: per iteration the slowest rank, median. Base: the RCCL collective and the GEMM run back to back on one stream with no overlap (BF16: torch.mm (hipBLASLt); MXFP8: a communication-free HipKittens MXFP8 GEMM). Speedup is base over KOSMOS; vs unf. is KOSMOS’s communication and GEMM WGs run back to back over the fused kernel.

ShapeMNKBase msKOSMOS msSpeedupvs unf.Base msKOSMOS msSpeedupvs unf.
BF16MXFP8
405B-Down-MBS181926656163842.2291.9191.1621.2202.4381.7821.3681.267
405B-OProj-MBS181922048163841.1520.9661.1921.1241.2330.9071.3601.310
70B-Down-MBS18192358481920.7820.6301.2411.2090.8270.5691.4531.462
70B-OProj-MBS18192102481920.5160.4351.1841.0960.6670.3961.6841.260
8B-Down-MBS18192179240960.3960.2791.4161.0860.4570.2621.7461.457
8B-OProj-MBS1819251240960.3280.2261.4501.0730.3110.2481.2551.101
405B-Down-MBS2163846656163844.1903.4091.2291.2684.8063.6041.3331.268
405B-OProj-MBS2163842048163842.2381.6221.3801.3362.4031.8581.2931.305
70B-Down-MBS216384358481921.5051.1191.3451.3521.6151.0461.5441.519
70B-OProj-MBS216384102481920.9310.8041.1571.0921.1010.7201.5281.445
8B-Down-MBS216384179240960.5960.4381.3601.2990.7320.4451.6461.565
8B-OProj-MBS21638451240960.4580.4011.1401.0700.4990.4141.2061.092
405B-Down-MBS4327686656163848.4256.4671.3031.3079.3607.2641.2891.286
405B-OProj-MBS4327682048163844.3662.9241.4931.4654.7243.6741.2861.276
70B-Down-MBS432768358481922.9102.0671.4081.3903.1571.9811.5941.595
70B-OProj-MBS432768102481921.8151.4331.2671.2092.0231.4201.4241.464
8B-Down-MBS432768179240961.1370.7661.4851.4551.3040.8281.5741.584
8B-OProj-MBS43276851240960.8350.7411.1281.0590.8780.7591.1561.171
405B-Down-MBS86553666561638416.54412.4301.3311.33818.27214.5871.2531.298
405B-OProj-MBS8655362048163848.6035.3591.6051.5779.5977.3761.3011.318
70B-Down-MBS865536358481925.6553.9031.4491.4436.4794.3411.4921.500
70B-OProj-MBS865536102481923.5162.6751.3141.2813.8332.8461.3471.470
8B-Down-MBS865536179240962.1621.4531.4871.4642.4541.6461.4911.655
8B-OProj-MBS86553651240961.6131.3701.1771.1151.6871.4721.1461.197
geometric mean1.3141.2551.3981.360

Table C.8.

ShapeMNKBF16MXFP8
Base msKOSMOS msSpeedupvs unf.Base msKOSMOS msSpeedupvs unf.
405B-FC1-MBS1819216384133123.7943.3861.1211.2571.8131.6391.1061.145
405B-QKV-MBS181921638423041.1780.6691.7611.7430.6650.3911.7011.722
70B-FC1-MBS18192819271681.0550.8791.2001.6060.5980.4781.2511.355
70B-QKV-MBS18192819212800.5480.3241.6941.5820.3190.1971.6211.631
8B-FC1-MBS18192409635840.4930.2541.9421.5110.2530.1731.4681.583
8B-QKV-MBS1819240967680.3640.1762.0621.3420.1850.1211.5241.419
405B-FC1-MBS21638416384133128.0146.9591.1521.3544.0153.6701.0941.133
405B-QKV-MBS2163841638423042.2321.3931.6031.6471.2790.7511.7021.770
70B-FC1-MBS216384819271681.9911.8151.0971.5771.1390.9041.2601.380
70B-QKV-MBS216384819212801.0080.6021.6761.5780.5760.3551.6201.547
8B-FC1-MBS216384409635840.7530.4931.5291.5690.4300.2861.5001.698
8B-QKV-MBS21638440967680.4990.3231.5461.3880.2960.2051.4431.498
405B-FC1-MBS432768163841331216.08613.4111.1991.3748.5357.5901.1251.183
405B-QKV-MBS4327681638423044.4252.8241.5671.6602.5521.4781.7271.811
70B-FC1-MBS432768819271684.3353.7191.1661.6392.2891.6661.3741.537
70B-QKV-MBS432768819212801.9201.2181.5761.5561.0940.7101.5421.553
8B-FC1-MBS432768409635841.3820.9331.4801.7200.7970.5331.4951.675
8B-QKV-MBS43276840967680.9080.6171.4721.4030.5310.3851.3781.357
405B-FC1-MBS865536163841331231.46026.5671.1841.39017.22614.8201.1621.221
405B-QKV-MBS8655361638423049.4455.9041.6001.6805.1143.0311.6871.773
70B-FC1-MBS865536819271689.3047.7331.2031.6364.8253.9551.2201.330
70B-QKV-MBS865536819212803.7912.4561.5441.5692.1641.4451.4981.521
8B-FC1-MBS865536409635842.6611.9181.3871.6381.5491.0761.4401.649
8B-QKV-MBS86553640967681.7511.2421.4101.3861.0020.7541.3281.364
geometric mean1.4431.5281.4141.480

Table C.9. TP+SP microbenchmark, bulk AG, column dgrad, TP 8, MI350X: per iteration the slowest rank, median. Base: the RCCL collective and the GEMM run back to back on one stream with no overlap (BF16: torch.mm (hipBLASLt); MXFP8: a communication-free HipKittens MXFP8 GEMM). Speedup is base over KOSMOS; vs unf. is KOSMOS’s communication and GEMM WGs run back to back over the fused kernel.

Table C.10. TP+SP microbenchmark, bulk AG, column dgrad, TP 8, MI355X: per iteration the slowest rank, median. Base: the RCCL collective and the GEMM run back to back on one stream with no overlap (BF16: torch.mm (hipBLASLt); MXFP8: a communication-free HipKittens MXFP8 GEMM). Speedup is base over KOSMOS; vs unf. is KOSMOS’s communication and GEMM WGs run back to back over the fused kernel.

ShapeMNKBF16 Base msBF16 KOSMOS msBF16 SpeedupBF16 vs unf.MXFP8 Base msMXFP8 KOSMOS msMXFP8 SpeedupMXFP8 vs unf.
405B-FC1-MBS1819216384133123.5193.0471.1551.3451.9371.6291.1891.232
405B-QKV-MBS181921638423041.2460.6731.8501.7760.6890.4011.7181.748
70B-FC1-MBS18192819271681.1680.8971.3021.6190.6470.5061.2791.376
70B-QKV-MBS18192819212800.5580.3391.6451.5110.3200.2031.5771.581
8B-FC1-MBS18192409635840.4930.2691.8381.4980.2520.1691.4951.570
8B-QKV-MBS1819240967680.3470.1871.8501.3360.1800.1271.4231.405
405B-FC1-MBS21638416384133127.0956.0061.1811.4053.8003.1201.2181.242
405B-QKV-MBS2163841638423042.4061.3941.7271.7251.3410.7921.6931.745
70B-FC1-MBS216384819271682.2641.7091.3251.6891.2380.9111.3581.467
70B-QKV-MBS216384819212801.0420.6461.6131.5180.5840.3741.5601.517
8B-FC1-MBS216384409635840.7920.4891.6201.6250.4460.2861.5561.792
8B-QKV-MBS21638440967680.5020.3441.4591.3440.2940.2121.3881.476
405B-FC1-MBS432768163841331213.86211.5841.1971.4217.6336.2821.2151.253
405B-QKV-MBS4327681638423044.7012.7771.6931.7322.6511.5611.6991.762
70B-FC1-MBS432768819271684.3853.4561.2691.6512.4111.7581.3711.463
70B-QKV-MBS432768819212802.0001.2921.5481.5211.1260.7121.5811.582
8B-FC1-MBS432768409635841.5040.9281.6211.7250.8400.5211.6131.812
8B-QKV-MBS43276840967680.9350.6621.4111.3440.5320.4031.3201.345
405B-FC1-MBS865536163841331226.95622.6031.1931.46014.84912.1911.2181.282
405B-QKV-MBS8655361638423049.3745.4971.7051.7425.2353.0451.7191.775
70B-FC1-MBS865536819271688.6996.8801.2641.6674.7933.5041.3681.461
70B-QKV-MBS865536819212803.8802.5961.4951.5682.2281.4651.5211.545
8B-FC1-MBS865536409635842.9501.8751.5731.6851.6411.0701.5331.721
8B-QKV-MBS86553640967681.7871.3181.3561.3551.0250.7641.3421.375
geometric mean1.4791.5461.4471.511

Table C.10.

ShapeMNKBF16 Base msBF16 KOSMOS msBF16 SpeedupBF16 vs unf.MXFP8 Base msMXFP8 KOSMOS msMXFP8 SpeedupMXFP8 vs unf.
405B-FC1-MBS1133121638481923.9713.2561.2201.5892.2621.6531.3691.505
405B-QKV-MBS123041638481921.1450.6601.7351.7931.0110.6251.6191.516
70B-FC1-MBS17168819281921.2420.7711.6111.6100.8250.5521.4961.472
70B-QKV-MBS11280819281920.5590.3361.6631.5580.4790.3331.4361.328
8B-FC1-MBS13584409681920.4440.2381.8641.8700.3210.1831.7521.670
8B-QKV-MBS1768409681920.3300.1901.7301.5090.2990.1791.6741.544
405B-FC1-MBS21331216384163848.3297.1551.1641.4434.8273.6961.3061.453
405B-QKV-MBS2230416384163842.1511.2791.6821.8211.9491.1931.6331.525
70B-FC1-MBS271688192163842.3861.5131.5771.6811.5671.0521.4891.431
70B-QKV-MBS212808192163841.0140.6191.6381.5850.8860.6131.4471.329
8B-FC1-MBS235844096163840.7630.4941.5451.7220.5900.3351.7601.644
8B-QKV-MBS27684096163840.5250.3641.4411.2490.5520.3251.6991.568
405B-FC1-MBS413312163843276816.69514.4651.1541.45310.2987.8121.3181.383
405B-QKV-MBS4230416384327684.1932.5731.6301.8853.7802.3391.6161.524
70B-FC1-MBS471688192327684.7313.6731.2881.4883.1512.1291.4801.404
70B-QKV-MBS412808192327681.9221.1951.6081.6021.7131.1741.4581.336
8B-FC1-MBS435844096327681.3960.9671.4441.7591.1150.6221.7921.675
8B-QKV-MBS47684096327680.9320.6511.4331.2761.0370.6081.7061.590
405B-FC1-MBS813312163846553632.76329.8011.0991.41920.77016.2251.2801.346
405B-QKV-MBS8230416384655369.0995.8211.5631.7747.9884.6301.7251.551
70B-FC1-MBS8716881926553610.5768.0941.3071.4176.8134.9021.3901.326
70B-QKV-MBS812808192655363.7242.3841.5621.6133.3152.2941.4451.339
8B-FC1-MBS835844096655362.6951.9561.3781.7982.1881.2071.8131.703
8B-QKV-MBS87684096655361.7361.2261.4161.3022.0171.1621.7351.611
geometric mean1.4761.5821.5511.486

Table C.11. TP+SP microbenchmark, bulk RS, column wgrad, TP 8, MI350X: per iteration the slowest rank, median. Base: the RCCL collective and the GEMM run back to back on one stream with no overlap (BF16: torch.mm (hipBLASLt); MXFP8: a communication-free HipKittens MXFP8 GEMM). Speedup is base over KOSMOS; vs unf. is KOSMOS’s communication and GEMM WGs run back to back over the fused kernel.

Table C.12. TP+SP microbenchmark, bulk RS, column wgrad, TP 8, MI355X: per iteration the slowest rank, median. Base: the RCCL collective and the GEMM run back to back on one stream with no overlap (BF16: torch.mm (hipBLASLt); MXFP8: a communication-free HipKittens MXFP8 GEMM). Speedup is base over KOSMOS; vs unf. is KOSMOS’s communication and GEMM WGs run back to back over the fused kernel.

ShapeMNKBase msKOSMOS msSpeedupvs unf.Base msKOSMOS msSpeedupvs unf.
405B-FC1-MBS1133121638481923.6203.0151.2011.5862.3231.6121.4411.602
405B-QKV-MBS123041638481921.2260.6911.7741.9251.0650.6671.5981.501
70B-FC1-MBS17168819281921.2280.8411.4601.6370.8520.5331.5981.581
70B-QKV-MBS11280819281920.5660.3571.5831.5180.4900.3561.3751.289
8B-FC1-MBS13584409681920.4460.2341.9071.9260.3230.1941.6641.594
8B-QKV-MBS1768409681920.3040.2061.4711.4320.2980.1941.5361.472
405B-FC1-MBS21331216384163847.2316.1881.1691.5134.6383.3481.3851.491
405B-QKV-MBS2230416384163842.3771.3611.7471.9142.0581.2761.6131.512
70B-FC1-MBS271688192163842.3291.6711.3931.6131.6221.0081.6101.560
70B-QKV-MBS212808192163841.0420.6591.5821.5520.9180.6581.3941.291
8B-FC1-MBS235844096163840.8050.5051.5941.8190.5970.3541.6861.617
8B-QKV-MBS27684096163840.5090.3831.3281.2770.5560.3541.5701.499
405B-FC1-MBS413312163843276814.16012.0131.1791.5089.3876.7531.3901.463
405B-QKV-MBS4230416384327684.5962.6251.7511.9024.0262.4841.6211.528
70B-FC1-MBS471688192327684.5373.3681.3471.5863.2151.9831.6211.558
70B-QKV-MBS412808192327682.0101.2781.5731.5631.7781.2621.4091.298
8B-FC1-MBS435844096327681.5460.9931.5561.8481.1370.6601.7231.635
8B-QKV-MBS47684096327680.9260.6901.3411.3021.0560.6561.6101.512
405B-FC1-MBS813312163846553627.92023.7351.1761.51518.36713.4171.3691.435
405B-QKV-MBS8230416384655369.1015.2121.7461.9438.1394.9121.6571.554
70B-FC1-MBS871688192655369.3636.8461.3681.5206.5924.4241.4901.425
70B-QKV-MBS812808192655363.9182.5091.5621.5623.4662.4781.3991.305
8B-FC1-MBS835844096655363.0251.9671.5381.8362.2311.2781.7451.653
8B-QKV-MBS87684096655361.7641.3011.3551.3262.0571.2621.6291.524
geometric mean1.4741.6171.5431.492

Table C.12. TP+SP microbenchmark, bulk RS, column wgrad, TP 8, MI355X: per iteration the slowest rank, median. Base: the RCCL collective and the GEMM run back to back on one stream with no overlap (BF16: torch.mm (hipBLASLt); MXFP8: a communication-free HipKittens MXFP8 GEMM). Speedup is base over KOSMOS; vs unf. is KOSMOS’s communication and GEMM WGs run back to back over the fused kernel.

RoutingModelBaseline msKOSMOS msKOSMOS TFLOPSSpeedup vs KOSMOS serial
MI350Xbalanced
DeepSeek-V317.0417.670752.62.222
Qwen3.5-397B11.0263.791543.82.908
DeepSeek-V4-Pro17.4438.531761.22.045
Kimi-K2.7-Code18.2198.117711.12.244
GLM-5.214.7536.562754.02.248
Qwen3-235B8.7423.405726.62.568
Qwen3-30B4.1581.555397.72.674
Llama 4 Scout3.3582.381865.81.410
Llama 4 Maverick4.3732.710760.81.614
geometric mean2.164
1.400
skewedDeepSeek-V331.35715.792365.51.986
Qwen3.5-397B19.6497.848262.72.504
DeepSeek-V4-Pro30.49217.083380.11.785
Kimi-K2.7-Code32.02216.005360.72.001
GLM-5.227.32413.640362.72.003
Qwen3-235B16.6087.547327.82.201
Qwen3-30B7.0933.126197.92.269
Llama 4 Scout7.5865.737359.41.322
Llama 4 Maverick7.6795.651364.81.359
geometric mean1.898
1.282
MI355Xbalanced
DeepSeek-V315.8566.806848.12.330
Qwen3.5-397B11.2154.008514.42.799
DeepSeek-V4-Pro16.2497.091915.82.292
Kimi-K2.7-Code17.1657.081815.22.424
GLM-5.213.6965.882841.22.329
Qwen3-235B8.4213.463714.42.432
Qwen3-30B4.1031.644376.12.495
Llama 4 Scout3.1832.287901.61.392
Llama 4 Maverick4.0902.423850.81.688
geometric mean2.199
1.469
skewedDeepSeek-V330.07814.522397.52.071
Qwen3.5-397B19.2358.481243.12.268
DeepSeek-V4-Pro29.06814.683442.31.980
Kimi-K2.7-Code30.60814.620394.82.094
GLM-5.226.23012.631391.72.077
Qwen3-235B15.9587.338337.12.175
Qwen3-30B7.0483.368183.62.093
Llama 4 Scout6.5745.123402.41.283
Llama 4 Maverick7.1434.966415.21.438
geometric mean1.911
1.308

Table C.13. MoE microbenchmark, BF16, forward pass, EP 8: per call the slowest rank, median. Baseline: RCCL A2A with TE’s grouped GEMMs. Speedup is baseline over KOSMOS; vs KOSMOS serial is the same KOSMOS kernels as separate launches over the fused layer.

RoutingModelBaseline msKOSMOS msKOSMOS TFLOPSSpeedupvs KOSMOS serial
MI350X
balancedDeepSeek-V325.93914.337805.21.8091.237
Qwen3.5-397B16.2455.918696.72.7451.479
DeepSeek-V4-Pro29.20716.807772.81.7381.187
Kimi-K2.7-Code26.82114.994770.01.7891.249
GLM-5.223.20512.294804.91.8881.252
Qwen3-235B13.4796.040819.12.2321.408
Qwen3-30B6.0681.793690.03.3851.703
Llama 4 Scout6.6754.383940.81.5231.111
Llama 4 Maverick9.3096.198665.21.5021.113
geometric mean1.9961.293
skewedDeepSeek-V348.41530.535378.11.5861.196
Qwen3.5-397B28.23511.970344.42.3591.467
DeepSeek-V4-Pro57.86034.486376.61.6781.127
Kimi-K2.7-Code50.90830.922373.41.6461.220
GLM-5.242.47126.627371.61.5951.186
Qwen3-235B24.93813.154376.11.8961.356
Qwen3-30B10.3824.152297.92.5011.574
Llama 4 Scout14.41810.122407.31.4241.101
Llama 4 Maverick15.24211.350363.31.3431.095
geometric mean1.7441.248
MI355X
balancedDeepSeek-V324.10411.4931004.52.0971.403
Qwen3.5-397B15.7655.162798.83.0541.647
DeepSeek-V4-Pro27.97413.883935.62.0151.270
Kimi-K2.7-Code25.32312.049958.22.1021.398
GLM-5.221.4459.930996.62.1601.410
Qwen3-235B12.7705.176956.02.4671.559
Qwen3-30B5.8341.777696.03.2831.763
Llama 4 Scout5.6513.8881060.41.4531.162
Llama 4 Maverick8.2265.327774.11.5441.140
geometric mean2.1691.403
skewedDeepSeek-V346.07025.534452.11.8041.297
Qwen3.5-397B27.34610.560390.52.5901.578
DeepSeek-V4-Pro55.18228.365457.91.9451.228
Kimi-K2.7-Code49.45025.540452.01.9361.331
GLM-5.240.20621.712455.81.8521.325
Qwen3-235B23.38611.339436.32.0621.460
Qwen3-30B10.0723.772328.02.6701.702
Llama 4 Scout11.7298.816467.71.3301.152
Llama 4 Maverick13.8239.801420.71.4101.135
geometric mean1.9091.345

Table C.14. MoE microbenchmark, BF16, backward pass, EP 8: per call the slowest rank, median. Baseline: RCCL A2A with TE’s grouped GEMMs. Speedup is baseline over KOSMOS; vs KOSMOS serial is the same KOSMOS kernels as separate launches over the fused layer.

RoutingModelTE MXFP8 msKOSMOS msKOSMOS TFLOPSSpeedupvs KOSMOS serial
MI350X
balancedDeepSeek-V315.5545.3261083.92.9211.498
Qwen3.5-397B10.3233.202643.93.2241.464
DeepSeek-V4-Pro14.3995.5161177.42.6111.424
Kimi-K2.7-Code15.9165.5141046.92.8871.505
GLM-5.213.5614.6371067.12.9251.483
Qwen3-235B8.6612.846869.13.0431.418
Qwen3-30B4.2851.591388.62.6931.421
Llama 4 Scout2.7311.7491178.81.5621.111
Llama 4 Maverick3.4501.8991085.91.8171.240
geometric mean2.5661.390
skewedDeepSeek-V329.82511.790489.62.5301.375
Qwen3.5-397B19.3976.396322.43.0331.389
DeepSeek-V4-Pro27.32211.497564.92.3771.364
Kimi-K2.7-Code30.10511.811488.72.5491.394
GLM-5.225.83910.215484.42.5301.386
Qwen3-235B16.4426.002412.22.7391.416
Qwen3-30B7.7702.839217.92.7371.320
Llama 4 Scout5.5863.905527.91.4301.080
Llama 4 Maverick6.0523.794543.31.5951.211
geometric mean2.3281.322
MI355X
balancedDeepSeek-V314.8365.3251084.02.7861.489
Qwen3.5-397B10.0133.364612.82.9761.439
DeepSeek-V4-Pro13.5275.1401263.52.6321.476
Kimi-K2.7-Code15.1545.3631076.42.8261.515
GLM-5.212.9504.6641060.92.7771.477
Qwen3-235B8.3532.870862.02.9111.442
Qwen3-30B4.1741.687366.72.4741.388
Llama 4 Scout2.7131.6981214.21.5981.136
Llama 4 Maverick3.1631.8381121.51.7211.242
geometric mean2.4681.395
skewedDeepSeek-V328.63211.643495.82.4591.380
Qwen3.5-397B18.8226.836301.62.7531.342
DeepSeek-V4-Pro26.34410.897595.92.4181.399
Kimi-K2.7-Code28.93611.726492.32.4681.390
GLM-5.224.85110.066491.52.4691.391
Qwen3-235B15.8196.031410.22.6231.406
Qwen3-30B7.6342.964208.72.5761.287
Llama 4 Scout5.3613.620569.61.4811.093
Llama 4 Maverick5.6823.455596.71.6451.256
geometric mean2.2761.323

Table C.15. MoE microbenchmark, MXFP8, forward pass, EP 8: per call the slowest rank, median. Baseline: RCCL A2A around Transformer Engine MXFP8 grouped GEMMs. Speedup is baseline over KOSMOS; vs KOSMOS serial is the same KOSMOS kernels as separate launches over the fused layer.

RoutingModelTE MXFP8 msKOSMOS msKOSMOS TFLOPSSpeedup vs KOSMOS serial
MI350Xbalanced
DeepSeek-V323.68410.5191097.52.251
Qwen3.5-397B14.6375.149800.82.843
DeepSeek-V4-Pro25.56012.5121038.12.043
Kimi-K2.7-Code25.13311.0211047.62.281
GLM-5.220.7068.9111110.52.324
Qwen3-235B12.3464.6811056.92.637
Qwen3-30B5.5571.699728.03.270
Llama 4 Scout5.1972.7041524.51.921
Llama 4 Maverick8.2894.477920.91.851
geometric mean2.343
skewed
DeepSeek-V344.30721.928526.52.021
Qwen3.5-397B26.2339.945414.62.638
DeepSeek-V4-Pro45.66423.454553.81.947
Kimi-K2.7-Code45.32521.599534.52.098
GLM-5.238.53018.280541.32.108
Qwen3-235B23.16910.053492.22.305
Qwen3-30B10.2173.659338.12.792
Llama 4 Scout10.5476.415642.71.644
Llama 4 Maverick12.9137.873523.71.640
geometric mean2.101
MI355Xbalanced
DeepSeek-V321.9419.0321278.22.429
Qwen3.5-397B13.7485.246786.02.621
DeepSeek-V4-Pro23.44610.5591230.12.220
Kimi-K2.7-Code23.1529.3941228.92.465
GLM-5.219.1717.6631291.32.502
Qwen3-235B11.5534.2681159.32.707
Qwen3-30B5.2811.732714.03.048
Llama 4 Scout4.7662.6161576.41.822
Llama 4 Maverick7.3773.9681039.01.859
geometric mean2.378
skewed
DeepSeek-V341.35019.153602.82.159
Qwen3.5-397B24.8859.154450.42.718
DeepSeek-V4-Pro42.72820.454635.02.089
Kimi-K2.7-Code42.35418.686617.82.267
GLM-5.236.18915.886622.92.278
Qwen3-235B21.7599.105543.42.390
Qwen3-30B9.7083.730331.62.603
Llama 4 Scout9.5395.921696.31.611
Llama 4 Maverick11.6846.847602.21.706
geometric mean2.173

Table C.16. MoE microbenchmark, MXFP8, backward pass, EP 8: per call the slowest rank, median. Baseline: RCCL A2A around Transformer Engine MXFP8 grouped GEMMs. Speedup is baseline over KOSMOS; vs KOSMOS serial is the same KOSMOS kernels as separate launches over the fused layer.

References

  1. [1]Aimuyo, O., Oh, B., and Singh, R. FlashMoE: Fast distributed MoE in a single kernel. In Belgrave, D., Zhang, C., Lin, H., Pascanu, R., Koniusz, P., Ghassemi, M., and Chen, N. (eds.), Advances in Neural Information Processing Systems, volume 38, Main Conference, pp. 100676–100699. Curran Associates, Inc., 2025. doi: 10.52202/085713-3368. URL https://proceedings.neurips.cc/paper_files/paper/2025/file/918938b2d20e95b56072777366f8a211-Paper-Conference.pdf.DOI
  2. [2]AMD. AMD Instinct MI350X GPU datasheet. https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/product-briefs/amd-instinct-mi350x-gpu-brochure.pdf, 2025a. Product page: https://www.amd.com/en/products/accelerators/instinct/mi350/mi350x.html. Accessed 2026-10.
  3. [4]AMD. AMD ROCm Software, including RCCL and hip-BLASLt. https://rocm.docs.amd.com/, 2026. RCCL: https://github.com/ROCm/rocm-systems/tree/develop/projects/rccl; hip-BLASLt: https://github.com/ROCm/rocm-libraries/tree/develop/projects/hipblaslt. Accessed 2026-10.
  4. [6]AMD. Mega MoE (Primus-Turbo documentation). https://github.com/AMD-AGI/Primus-Turbo/blob/main/docs/README_Mega_MoE.md, n.d.b. Accessed 2026-10.
  5. [9]AMD. Training a model with Megatron-LM for ROCm. https://rocm.docs.amd.com/en/docs-6.4.1/how-to/rocm-for-ai/training/benchmark-docker/megatron-lm.html, n.d.e ROCm 6.4.1 documentation. Accessed 2026-10.
  6. [12]Awad, M., Osama, M., and Potter, B. Iris: First-class Multi-GPU programming experience in Triton, 2025. URL https://arxiv.org/abs/2511.12500.
  7. [13]Chang, L.-W. et al. FLUX: Fast software-based communication overlap on GPUs through kernel fusion, 2024. URL https://arxiv.org/abs/2406.06858.
  8. [14]Cheng, X. et al. MPK: A compiler and runtime for Mega-Kernelizing tensor programs. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26), pp. 1909–1926, Seattle, WA, July 2026. USENIX Association. ISBN 978-1-939133-55-7. URL https://www.usenix.org/conference/osdi26/presentation/cheng.arxiv.org/abs/2512.22219
  9. [15]DeepSeek-AI. DeepSeek-V3 technical report, 2025. URL https://arxiv.org/abs/2412.19437.
  10. [16]DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence, 2026. URL https://arxiv.org/abs/2606.19348.
  11. [17]Grattafiori, A. et al. The Llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783.
  12. [18]Hu, W. et al. HipKittens: Fast and furious AMD kernels. In Chowdhery, A. and Jia, Z. (eds.), Proceedings of Machine Learning and Systems, volume 8, pp. 1431–1443. MLSys, 2026. URL https://proceedings.mlsys.org/paper_files/paper/2026/file/bc5fa9843a7905bbed9d83895a88f7f-Paper-Conference.pdf.arxiv.org/abs/2511.08083
  13. [19]Huang, Z. et al. Weave: Fine-grained dynamic SM scheduling in an MoE megakernel for compute-communication overlap, 2026. URL https://arxiv.org/abs/2609.21483.
  14. [20]Jangda, A. et al. Breaking the computation and communication abstraction barrier in distributed machine learning workloads. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’22, pp. 402–416, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392051. doi: 10.1145/3503222.3507778. URL https://doi.org/10.1145/3503222.3507778.
  15. [21]Korthikanti, V. A. et al. Reducing activation recomputation in large Transformer models. In Song, D., Carbin, M., and Chen, T. (eds.), Proceedings of Machine Learning and Systems, volume 5, pp. 341–353. Curan, 2023. URL https://proceedings.mlsys.org/paper_files/paper/2023/file/80083951326cf5b35e5100260d64ed81-Paper-mlsys2023.pdf.arxiv.org/abs/2205.05198
  16. [22]Li, S. et al. PyTorch distributed: Experiences on accelerating data parallel training, 2020. URL https://arxiv.org/abs/2006.15704.
  17. [23]Liang, W. et al. TorchTitan: One-stop PyTorch native solution for production ready LLM pretraining. In Yue, Y., Garg, A., Peng, N., Sha, F., and Yu, R. (eds.), International Conference on Learning Representations, volume 2025, pp. 92161–92181, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/file/e6231c5f46598cfd09ff1970524e0436-Paper-Conference.pdf.arxiv.org/abs/2410.06511
  18. [24]Mao, Z. et al. mKernel: Fast Multi-GPU, multi-node fused kernels, 2026. URL https://arxiv.org/abs/2609.13585.

Paper details

vlnc:W_9q3phv4p
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 15

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents