betaFeedback

WorksComputer Science

SS-HSDP: Efficient and Flexible Sharded Data Parallelism via Runtime-Update Decoupling

Abstract

Sharded data parallelism enables memory-efficient training of large models. While efficient training kernels favor packed, kernel-specific data layouts, optimizers may require variant data structures as optimization units. Existing sharded data-parallel frameworks couple these two choices, limiting runtime efficiency and optimization flexibility. To address this, we present \our, a training framework that separates runtime parameters from update parameters, allowing computation layouts and optimization units to be organized independently. Through shared data views and hierarchical sharding, \our supports this separation while avoiding redundant data movement. This design supports a broad range of optimizers, including matrix-based methods such as Muon, and performance optimizations such as fully zero-copy execution. Experiments on video and language model workloads with MXFP8 training show that \our achieves higher Model FLOPs Utilization (MFU) than FSDP2, veScale-FSDP, and Megatron-FSDP. At 128 GPUs, MFU improves by up to 21.7% over the fastest evaluated baseline, with comparable or lower peak memory usage.

Introduction

ZeRO [32] enables training large models on memory-limited devices by partitioning weights, gradients, and optimizer states across data-parallel ranks. Under ZeRO-3, weights are typically all-gathered before each layer’s computation, and gradients are reduce-scattered after its backward computation. These collectives can overlap with computation in other layers, potentially hiding most of the communication cost in compute-bound workloads. With relatively modest changes to model code, frameworks such as DeepSpeed [33] and PyTorch FSDP2 [31, 47] make this approach practical for large-scale training. This approach has been successfully used to train both language models [14, 15] and video generation models [22, 38].

Recent advances in model architectures, low-precision training and optimization impose new requirements on both runtime computation efficiency and optimizer flexibility. For MoE models [8, 12], redundant movement of their large weights and gradients can introduce substantial overhead. Low-precision weight formats such as MXFP [28] represent weights using quantized values and scale metadata, whose layout must match the requirements of high-performance kernels [9, 23]. Meanwhile, matrix-based optimizers such as Muon [17] operate on matrices that may be submatrices of the weights used for computation. For example, when applying Muon per attention head [18], each head is updated independently, whereas forward and backward computation can use one packed QKV projection, with its values and scales arranged for the target GEMM kernel.

These requirements expose a mismatch between how parameters are computed and how they are updated. Runtime computation requires parameter data to be arranged for optimized kernels, packed across multiple parameters and stored in kernel-specific layouts, whereas optimizers require the same data to be accessed in the units they optimize, such as individual attention heads. Tying update units to the parameter organization used for computation constrains these choices: supporting different update units may require reorganizing data or customizing the optimizer.

Existing frameworks address only part of this mismatch. Megatron-FSDP [24] optimizes the runtime data path with flattened buffers, persistent low-precision weights, and direct gradient writes, but does not support structured optimizer updates. veScale-FSDP [41] aligns ragged shards with the update and quantization structure to support distributed Muon updates and block-scaled quantization, but does not pack or transform parameters into kernel-specific layouts. Our goal is to satisfy both requirements without trading one for the other.

We present SS-HSDP, a framework that decouples runtime parameters for efficient computation from update parameters for flexible optimization. Figure 1 illustrates this separation and its performance benefits. Each update parameter defines a unit of optimization and its associated operations, including optimizer updates and weight casting or quantization. The weights of one or more update parameters are packed or transformed into a runtime parameter’s weight in the representation required by the computation kernel. With careful memory management and hierarchical sharding design, this separation needs no extra memory operations. Rather, it preserves update flexibility while enabling computation- and communication-friendly data layouts that minimize redundant copies and exploit optimized kernels and communication operations.

Figure 1. Parameter separation in SS-HSDP, and MiMo MXFP8 training performance with AdamW and framework capabilities.

Building on this design, SS-HSDP further reduces memory allocations and copies. On supported topologies, it transfers data using copy engines within nodes and host-driven remote memory access (RMA) between nodes, reducing contention for streaming multiprocessors (SMs). As a result, SS-HSDP achieves up to 1.33× speedup and 27.3% memory saving over PyTorch FSDP2 on 128 GPUs. In summary, our contributions are:

  • Identify We identify a mismatch between kernel-specific runtime layouts and optimizer update units, and show how coupling them limits runtime efficiency and optimizer flexibility.

  • Design We design a unified framework that separates runtime parameters from update parameters, with memory management and hierarchical sharding that enable efficient computation and flexible optimizer updates without redundant data movement.

  • Implement A clean, high-performance implementation that supports diverse training scenarios and achieves higher MFU than FSDP2, veScale-FSDP, and Megatron-FSDP.

We will release the code in the future.

Background and Motivation

Notation and ZeRO Training

  • Tensors and shards* A tensor is a multidimensional array of numerical values; a matrix is a two-dimensional tensor. We use lowercase letters for tensors and corresponding uppercase letters for tensor lists, e.g., W=[w0,…,wk]W = [w_0, \ldots, w_k]. A tensor list is a collection of tensors with custom layouts. A logical tensor denotes the complete tensor, regardless of whether all, part, or none of its data are stored locally.

Sharding partitions a tensor or tensor list across a group of ranks, each holding a local shard. The number of ranks in this sharding group is its sharding degree. When shards are replicated, ranks holding replicas of the same shard form a replica group. Superscripts identify the sharding level of a local shard, as defined in the ZeRO sharding paragraph below; the same notation applies to tensors and tensor lists. Table 1 summarizes the notation.

SymbolMeaning
PP, P′P^{\prime}Update and runtime parameters
iiIndex of an update parameter inside a runtime parameter
ww, ssWeight and quantization scales
wmainw^{\mathrm{main}}, ggMain weight and main gradient
mmOptimizer state, e.g., momentum
uu, vvUpdate temporaries
w′w^{\prime}, s′s^{\prime}, g′g^{\prime}Runtime weight, scales, and gradient
WW, SS, GG, MM, UU, VVLists of corresponding tensors
W′W^{\prime}, S′S^{\prime}, G′G^{\prime}Lists of corresponding runtime tensors
F\mathcal{F}Optimizer update
fjf_jThe jjth operation in F\mathcal{F}
G\mathcal{G}Weight casting or quantization
Tw\mathcal{T}_wWeight transform (wi,si)i→(w′,s′)(w_i,s_i)_i \to(w^{\prime},s^{\prime})
Tg\mathcal{T}_gGradient transform g′→(gi)ig^{\prime} \to(g_i)_i
z3,z2,z1z3,z2,z1ZeRO-3/2/1 shard superscripts
sw,sg,sos_w,s_g,s_oSharding degrees at z3,z2,z1z3,z2,z1

Table 1. Notation.

  • Parameters and updates* Following Megatron’s naming convention [25, 34], a parameter PP has several associated tensors for computation and optimization. In mixed-precision training, a parameter has a weight ww stored in a low-precision format such as BF16 or FP8, and an FP32 main weight wmainw^{\mathrm{main}} for optimization [21]. Gradients produced by backward computation are reduced across data-parallel ranks and accumulated over microbatches into a main gradient gg. The optimizer uses gg to update wmainw^{\mathrm{main}} and its optimizer states, such as momentum mm. We consider FP32 main gradients and optimizer states unless otherwise stated. We denote the optimizer update by F\mathcal{F} and weight preparation by G\mathcal{G}:

(wmain,m)→F(wmain,g,m),(w^{\mathrm{main}}, m) \xrightarrow{\mathcal{F}} (w^{\mathrm{main}}, g, m),
(w,s)=G(wmain).(w, s) = \mathcal{G}(w^{\mathrm{main}}).

Depending on the optimizer, additional or different state tensors may be used. G\mathcal{G} casts or quantizes the main weight, producing optional scale metadata ss for quantized formats.

  • ZeRO sharding* ZeRO stage 1 shards main weights and optimizer states; stage 2 additionally shards main gradients; stage 3 further shards computation weights [32]. We

denote local shards at the weight, gradient-accumulation, and optimizer-update levels by superscripts z3, z2, and z1, with sharding degrees sws_w, sgs_g, and sos_o, respectively. For example, wz3w^{z3} and Wz3W^{z3} denote local shards of a weight tensor and a weight tensor list. Conventional ZeRO-3 uses the same sharding group for all three levels. Hybrid sharded data parallelism (HSDP) uses smaller sharding groups and replicates shards across groups to limit repeated weight all-gathers. Outer ZeRO-1 further shards main weights and optimizer states across replica groups [24], reducing persistent memory without enlarging the groups for repeated all-gathers. With gradients accumulated over microbatches, outer communication is needed once per optimizer step.

Training execution We call forward and backward computation and their associated communication runtime. Under ZeRO-3, each layer all-gathers weights before computation and reduce-scatters gradients after backward. These collectives can overlap with computation in other layers. Figure 3 illustrates this layer-wise overlap. After gradients from all microbatches have been accumulated, the update phase applies F\mathcal{F} to parameters across the model and prepares weights through G\mathcal{G} for the next training step.

ZeRO-3 layer-wise overlap of compute, all-gather, and reduce-scatter streams

Figure 3. ZeRO-3 Layer-wise(LL) overlap. FF and BB denote forward and backward; AG and RS denote all-gather and reduce-scatter.

FSDP2 Dataflow

Figure 2 shows PyTorch FSDP2’s BF16 mixed-precision dataflow with FP32 gradient reduction and accumulation [31]. FSDP2 shards each parameter along dimension 0 by default and represents its shards as DTensors [30]. Hybrid sharded data parallelism (HSDP) additionally replicates these shards in replica groups. Before forward and backward computation, G\mathcal{G} casts (wmain)z3(w^{\mathrm{main}})^{z3} to BF16 wz3w^{z3}. Copy-in packs these shards into the all-gather input Wz3W^{z3}. All-gather produces WW in an interleaved layout: data are ordered by rank, leaving each parameter’s shards separated by other parameters’ data. Copy-out assembles each contiguous weight ww for computation. Backward produces per-parameter gradients, which are converted to FP32 and packed by destination rank into a reduce-scatter buffer. HSDP additionally all-reduces the results across replica groups. The reduced shards initialize gz2g^{z2} on the first microbatch and are added to it thereafter. After all microbatches, F\mathcal{F} updates the local main-weight shards (wmain)z3(w^{\mathrm{main}})^{z3}.

Figure 2. FSDP2-style mixed-precision training with two sharding ranks (rank 0 shown). Colored segments track one parameter’s weights, gradients, and optimizer states; white segments belong to other parameters. Solid arrows show the reference dataflow, while numbered dashed paths highlight alternative implementations. AG and RS denote all-gather and reduce-scatter, respectively.

As indicated by the circled numbers in Figure 2, FSDP2’s default communication path incurs repeated weight casts ②, buffer copies ③④, and separate gradient accumulation ⑤. Its parameter-wise sharding also requires additional handling for structured optimizer updates ⑥ and packed or transformed kernel-specific layouts ①, while HSDP replicates main weights and optimizer states rather than further sharding them ⑦. We examine these limitations below.

Challenges and Analysis

Challenges and Analysis

In conventional BF16 training with AdamW, ww, wmainw^{\mathrm{main}}, gg, and the optimizer states have matching shapes, while both F\mathcal{F} and G\mathcal{G} operate elementwise. These tensors can therefore be sharded identically at arbitrary element boundaries, without considering matrix or block structure. Structured updates and block quantization change these requirements. Muon is a matrix-based optimizer [17]; applying it per attention head requires updating each head matrix independently [18], rather than treating a packed projection as one update unit ⑥. Similarly, MXFP quantization in G\mathcal{G} computes a shared scale for a block of weight values [28]. These operations require complete matrices, heads, or quantization blocks; when sharded, metadata must identify the fragments needed for communication.

Runtime computation instead favors packing Q, K, and V projections into one GEMM, with weights and scales arranged in kernel-specific layouts [9, 23] ①. Preparing these representations for every microbatch can be costly. Reusing the prepared weight shards avoids this repeated work ②, but increases persistent memory usage. This additional memory cost can be offset by outer ZeRO-1 ⑦. For communication, compatible contiguous buffer layouts let collectives and computation kernels share storage, avoiding weight and gradient copies ③④. However, evenly sharding these buffers can split matrices, heads, or quantization blocks required by updates. The challenge here is to preserve runtime efficiency without restricting update flexibility or introducing redundant data movement.

Existing frameworks address these requirements in different ways. Megatron-FSDP [24] uses Transformer Engine for kernel-specific low-precision weight transformations ① and retains the prepared weight shards across microbatches ②. Its flattened buffers avoid weight layout copies ③, while compatible kernels write gradients directly into these buffers ④. It also supports outer ZeRO-1 sharding ⑦, but does not support structured optimizer updates ⑥. veScale-FSDP [41] uses structure-aware ragged sharding to accommodate quantization blocks and distributed Muon updates ⑥, and DBuffer to avoid weight layout copies ③. However, neither framework decouples update units from runtime layouts, limiting the combination of update flexibility and runtime efficiency.

SS-HSDP decouples update units from runtime layouts, supporting structured optimizer updates ⑥ while packing and transforming weights for efficient computation ①. Shared buffers and direct gradient writes avoid redundant weight and gradient copies ③④. Prepared weight shards are reused across microbatches to avoid repeated conversion ②, while further sharding main weights and optimizer states across replica groups can offset the additional memory cost ⑦. Custom ZeroSM reduce-scatter accumulates directly into gg without a separate post-reduction add ⑤. Section 3 presents the parameter separation, memory management, and sharding design behind these optimizations.

Design

In this section, we present the core design of SS-HSDP, including the separation of update and runtime parameters, data trees and shard chains for managing flattened buffers and hierarchical sharding, and parameter grouping.

Update and Runtime Parameters

To support runtime efficiency and update flexibility, SS-HSDP divides the parameter into an update parameter and a runtime parameter, denoted by PP and P′P^{\prime}, respectively. An update parameter defines an independent unit of optimization, with the tensors and weight preparation operation G\mathcal{G} defined in Section 2.1. We express its optimizer update as an ordered sequence F=[f0,f1,…,fn]\mathcal{F}=[f_0,f_1,\ldots,f_n], with optional temporary tensors such as uu and vv. Each operation specifies the tensors it reads and writes. Although it contains the low-precision representation (w,s)(w,s), an update parameter is not directly used by forward or backward kernels.

A runtime parameter P′P^{\prime} contains one or more update parameters P0,…,PkP_0,\ldots,P_k whose weight, scale, and main-gradient tensors are denoted by wiw_i, sis_i, and gig_i, respectively. P′P^{\prime} holds the runtime weight w′w^{\prime}, optional scales s′s', and the gradient g′g' produced by backward computation. The transform operations Tw\mathcal{T}_w and Tg\mathcal{T}_g connect the two representations:

(w′,s′)=Tw(w0,s0,…,wk,sk),(g0,…,gk)=Tg(g′).\begin{aligned} (w',s') &= \mathcal{T}_w(w_0,s_0,\ldots,w_k,s_k),\\ (g_0,\ldots,g_k) &= \mathcal{T}_g(g'). \end{aligned}

Tw\mathcal{T}_w prepares the layout required by computation kernels, while Tg\mathcal{T}_g maps the runtime gradient back to the main gradients for optimizer updates.

EG1 We use MXFP8 Q projection with per-head Muon updates as an example. Each Q head has an update parameter PiP_i, while the Q projection is represented by one runtime parameter P′P'. For each head, Fi=[fi0,fi1,fi2]\mathcal{F}_i = [f_{i0}, f_{i1}, f_{i2}], fi0f_{i0} updates mim_i using gig_i and writes a BF16 intermediate to uiu_i. fi1f_{i1} applies Newton–Schulz to uiu_i and writes the FP32 update direction to a separate scratch tensor viv_i. Then fi2f_{i2} updates wimainw_i^{\mathrm{main}} using viv_i. Gi\mathcal{G}_i uses 32×3232 \times32 block quantization in a kk-major layout to produce FP8 wiw_i and UE8M0 sis_i. Then Tw\mathcal{T}_w packs the head weights wiw_i and relayout the scales sis_i into the layout required by DeepGEMM’s MXFP8 kernels for the whole Q projection. This avoids separate per-head GEMM launches. Tg\mathcal{T}_g maps g′g' back to the per-head gig_i.

Packing and unpacking may introduce redundant copies and allocations when the tensors could instead share storage. The update operations must also remain valid when these buffers are sharded. We use data trees and shard chains to address these requirements.

Data Tree and Shard Chain

SS-HSDP stores each tensor list in a contiguous buffer, grouping tensors with the same device and dtype for memory management and communication.

Data Tree A data tree allows update and runtime tensors to share storage despite having different shapes and boundaries. Each tensor or tensor list is represented by a data node. A parent–child link defines a view at a byte offset within the parent; the child’s shape and dtype determine the view’s size. Only the root owns the allocation, and each child accesses this storage through its accumulated offset.

In EG1, the head weights wiw_i are consecutive views of WW, while the runtime weight w′w' views their combined region. The heads can therefore be updated independently and used together as one projection without moving their weight data. The weight-packing part of Tw\mathcal{T}_w requires neither a copy nor an additional allocation, since the packed weight is merely a view of the root allocation, as shown on the left of Figure 4; scale relayout for s′s' is still performed to satisfy the kernel’s layout requirements.

Figure 4. SS-HSDP data structures and execution flow. Left: data-tree views and the Head0 shard chain. Right: packed runtime computation and structured updates. The example uses four ranks, with so=4s_o = 4, sg=sw=2s_g = s_w = 2.

Shard Chain SS-HSDP partitions each flattened tensor list into contiguous chunks and further partitions each local chunk at subsequent levels, recording these partitions as a shard chain:

W⟶Wz3⟶Wz2⟶Wz1.W \longrightarrow W^{z3} \longrightarrow W^{z2} \longrightarrow W^{z1}.

When adjacent levels use the same sharding degree, their local shards cover the same data and can share storage through the data tree. A shard-chain link alone does not create a parent–child link in the data tree.

Each member tensor inherits this chain: its local fragment is the intersection of its range with the local buffer range, or empty if they do not overlap.

wi⟶wiz3⟶wiz2⟶wiz1.w_i \longrightarrow w_i^{z^3} \longrightarrow w_i^{z^2} \longrightarrow w_i^{z^1}.

Each link records the sharding group and partition, mapping local fragments to their positions in the original tensor. The same construction applies to all tensor fields. Together with rank ownership, these mappings locate the fragments required for distributed updates.

The shard chain can also include other partitions. This section focuses on hierarchical data-parallel sharding rather than its composition with tensor or expert parallelism.

Distributed updates For each operation in F\mathcal{F} and for G\mathcal{G}, we declare its input and output tensors and an update unit size for each operand: the number of elements required together, from one element for elementwise operations to a complete matrix for matrix operations. SS-HSDP uses the shard mappings to locate the corresponding operand fragments at the ZeRO-1 level. When all required operands for a unit are local, the operation runs on local views. Otherwise, an execution rank receives the required fragments through point-to-point (P2P) communication, performs the operation, and returns the modified fragments to their owners. Only declared tensors participate in this exchange.

In EG1, fi0f_{i0} and fi2f_{i2} run locally; only fi1f_{i1} requires the complete head intermediate uiu_i. Main weights and momentum remain sharded, and shard boundaries need not coincide with head boundaries. The data tree manages allocations and shared views, while the shard chain preserves each tensor’s logical positions across sharding levels. Together, they preserve packed, evenly sharded runtime buffers without restricting optimizer update units.

Parameter Groups

SS-HSDP groups compatible parameters for shared storage and batched execution. Parameters with different dtypes, quantization formats, or optimizers may require separate tensor lists. The group boundaries for storage, runtime, and updates need not coincide.

An atom group contains compatible runtime parameters and their update parameters. It packs matching tensor fields into separate tensor lists and constructs their data trees and shard chains. A runtime group schedules operations across one or more atom groups, handling coalesced communication, gradient accumulation at ZeRO-2, and the transforms Tw\mathcal{T}_w and Tg\mathcal{T}_g. These transforms can be configured at different sharding levels. An update group schedules optimizer updates F\mathcal{F} and weight preparation G\mathcal{G} at ZeRO-1, batching compatible operations across its atom groups. This organization supports layer-wise runtime execution and model-wide batching of compatible updates.

Execution Flow

We now follow EG1 through the workflow in Figure 4. The example uses four ranks, with ZeRO-1 degree four and ZeRO-2 and ZeRO-3 degree two. We place Tw\mathcal{T}_w at ZeRO-3 and Tg\mathcal{T}_g at ZeRO-2. During setup, SS-HSDP builds the groups, data trees, and shard chains, and initializes or loads the main weights and optimizer states. Before the first training step, G\mathcal{G} initialize weights and scales at ZeRO-1.

Runtime At the first forward pass after initialization or an optimizer update, the runtime group performs the outer all-gather on the FP8 weights and UE8M0 scales prepared by G\mathcal{G} at ZeRO-1. At ZeRO-3, Tw\mathcal{T}_w prepares packed runtime weights and kernel-specific scales ①. These shards are retained across microbatches ②.

For forward and backward computation, all-gather prepares the full runtime weights and scales as needed ③. The GEMMs consume views of these buffers, and backward writes directly into the flattened runtime gradient buffer G′\mathcal{G}' ④. Reduce-scatter accumulates the gradients in FP32 at ZeRO-2 ⑤. This path repeats for each microbatch without repeating weight quantization or scale relayout. Full weight buffers can be released after use, while the runtime weight shards and accumulated gradients remain available.

Update After all microbatches, Tg\mathcal{T}_g maps the accumulated gradients to the update representation at ZeRO-2, and the outer reduce-scatter produces the ZeRO-1 main-gradient shards ⑦. For Head0, the update spans ranks 2 and 1 ⑥. Each rank first executes f0f_0 on its local fragments to update momentum and produce BF16 uu. Rank 2 then gathers the missing fragments of uu from rank 1, executes Newton–Schulz f1f_1, and returns the corresponding FP32 vv fragments to rank 1. Both ranks execute f2f_2 locally to update their main-weight fragments. Main weights and momentum remain sharded throughout this matrix stage. Applying G\mathcal{G} to the updated main weights begins the next cycle.

The design above provides storage and sharding abstractions for decoupled runtime and update parameters. The next section builds on these abstractions to batch updates and transforms, eliminate redundant copies, and reduce contention between communication and computation.

Optimization

The design above enables several optimizations, summarized in Figure 5: batching reduces kernel-launch overhead, shared views and direct gradient writes avoid redundant memory copies, and ZeroSM collectives reduce communication-induced SM contention.

Figure 5. Optimizations enabled by runtime–update separation. (a) Compatible update units are batched per stage and execution rank. (b) Backward kernels write or accumulate directly into runtime-gradient views. (c) SM contention and explicit reduction overhead, shown schematically. (d) ZeroSM all-gather exchanges shards between nodes before broadcasting within each node. (e) Reduce-scatter for one output partition AA; subscripts identify contributing ranks.

Batched Update and Transform

Within each runtime or update group, SS-HSDP batches compatible operations of the same type across parameters, as illustrated in Figure 5a. With independent runtime and update groups, TwT_w and TgT_g can be batched within each layer, while F\mathcal{F} and G\mathcal{G} can be batched across the model. Within each stage, compatible local and P2P-dependent units are batched separately. In our fused Muon implementation, stages f0f_0 and f2f_2 use separate multi-tensor kernels, while f1f_1 groups matrices with matching shapes and layouts for batched GEMMs. We similarly batch MXFP scale-layout transforms across tensors. Batching reduces kernel-launch overhead across parameters, particularly for small per-head updates and scale-layout transforms.

Fully Zero Copy

SS-HSDP arranges ZeRO-1 shards in the flattened buffer to match the group-rank order of hierarchical collectives, avoiding buffer reordering. In Figure 4, the four shards have owners [0, 2, 1, 3] in buffer order. Outer all-gathers within {0, 2} and {1, 3} produce two contiguous halves; runtime all-gathers within {0, 1} and {2, 3} then construct the full buffer directly. Gradient reduce-scatter follows the same layout in reverse, without reordering.

Zero-copy weights When W′W^\prime is a shared-storage view of WW, using the same shard boundaries and rank ownership allows their local shards to share storage as well. In particular, W′z1W^{\prime z1} and W′z3W^{\prime z3} can directly share storage with Wz1W^{z1} and Wz3W^{z3}, respectively. Individual tensors access these tensor lists through sliced views. All-gather therefore reads directly from the existing local shard buffers, and forward and backward kernels consume views of the gathered buffer. No intermediate packing buffer or copy-in/copy-out is needed for weights. Required scale relayouts are still performed.

Direct gradient write The same flattened layout supports direct reduce-scatter, but gradient production and accumulation can still introduce additional memory passes. PyTorch autograd produces per-parameter gradient tensors, which FSDP2 then copies into a packed reduce-scatter input buffer.

As shown in Figure 5b, SS-HSDP supports direct gradient writes from compatible backward kernels, such as those for linear layers, into views of the flattened reduce-scatter input buffer. This avoids separate per-parameter gradient tensors and the subsequent packing copies. The first write must initialize the entire target region, including zero-contribution entries; subsequent contributions accumulate. Otherwise, the buffer must be cleared before accumulating contributions. The remaining addition pass for accumulating reduced gradients across microbatches is fused into the final stage of reduce-scatter, as described next.

ZeroSM Communication

As shown in Figure 3, communication can overlap with computation to hide communication latency. In our profiling, NCCL communication kernels occupy 10–20% of the SMs. This creates SM contention between communication and computation, as illustrated in Figure 5c. When ranks progress at different rates, communication kernels may remain resident while waiting for peer data or synchronization signals, occupying SM resources even without transferring data. Table 2 shows two profiled examples of slower compute kernels under NCCL overlap.

KernelStandaloneNCCL overlap
FlashAttention forward4.7045.774 (+22.7%)
DeepGEMM Grouped GEMM6.0109.087 (+51.2%)

Table 2. Representative kernel latencies (ms). Parentheses show latency increases over standalone execution.

A fixed SM budget for computation kernels can mitigate contention, but also limits their parallelism after communication finishes. We instead use ZeroSM transfers to avoid reserving SM resources for communication.

ZeroSM P2P Within the NVLink domain, GPU copy engines perform peer transfers without using SMs. Our implementation uses registered symmetric allocations for communication buffers. Data-tree views address regions of these registered root buffers, avoiding separate allocations and registrations for individual views. Across nodes, we use NCCL Host remote memory access (RMA) [27]. CPU proxy threads coordinate NIC operations, while RDMA moves the payload without SM communication kernels. Together, these mechanisms support intra-node and inter-node ZeroSM P2P transfers between registered source and destination buffer regions. These bandwidth-bound kernels can use the available SMs and HBM bandwidth for local arithmetic without waiting for remote peers inside the kernel. Figure 5(c) illustrates how reduced SM contention can outweigh this explicit reduction cost.

All-Gather To avoid transferring replicated shards across nodes, we perform inter-node exchange before intra-node broadcast (Figure 5(d)). Compared with exchanging each GPU’s buffer after intra-node broadcast, this reduces inter-node payload by a factor of NN, where NN is the number of participating GPUs per node.

Reduce-Scatter Unlike all-gather, reduce-scatter still requires arithmetic on SMs. We use ZeroSM P2P for communication and launch reduction kernels only after their inputs are ready.

To reduce inter-node traffic, we first aggregate contributions within each node (Figure 5(e)). The input buffer G′G' holds BF16 or FP32 gradients. Intra-node P2P exchange brings slices to local reducers, which sum them in FP32, producing partials such as A{0,1}A_{\{0,1\}} and A{2,3}A_{\{2,3\}}. The partial sums remain FP32 or, for BF16 inputs, are stored in BF16 using stochastic rounding (SR). Inter-node P2P then transfers the partials to their destination ranks. The final FP32 reduction accumulates directly into the persistent buffer G′′2G^{\prime\prime2}, eliminating a separate result buffer and addition pass.

This follows the separation of BF16 communication and FP32 summation in DeepSeek-V4 [11], but applies SR to node-local partial sums. For a finite, in-range sum ss, SR satisfies E[SR⁡(s)∣s]=s\mathbb{E}[\operatorname{SR}(s)\mid s]=s: the rounding error has zero expectation, consistent with unbiased gradient quantization in low-bitwidth training [3]. We write intra-node/inter-node for the dtypes of G′G' and stored node-local partial sums, the latter also serving as the inter-node payload. The modes are FP32 (FP32/FP32), BF16+FP32 (BF16/FP32), and BF16SR (BF16/BF16). § 6.1 evaluates their effect on training loss.

Experiments

End-to-End Performance

Setup. We compare SS-HSDP with PyTorch FSDP2, veScale-FSDP*1, and Megatron-FSDP on representative compute-bound training workloads from video and language models: two internal video models, Qwen3-30B-A3B [44], and a 12-layer MiMo [43] variant. We scale from 32 to 128 GPUs with fixed per-GPU workloads. Detailed settings and measurement procedures are provided in Appendix § 10.§ 1.

Results. Figure 6 reports the MXFP8 results. The upper panels show model FLOPs utilization (MFU) [7], defined as achieved model FLOPs per second divided by peak hardware FLOPs per second. The lower panels show peak memory usage per GPU.

Figure 6. MXFP8 training MFU and peak memory at fixed per-GPU workloads. We use sw=sg=32s_w=s_g=32; for SS-HSDP and Megatron-FSDP, sos_o equals the total GPU count.

SS-HSDP achieves the highest MFU in all tested MXFP8 configurations. At 128 GPUs, the largest relative MFU gain over the fastest baseline is 21.7% on MiMo. MiMo’s large expert-weight volume makes weight quantization and copying costly. Weight reuse avoids repeated quantization across microbatches, while zero-copy execution eliminates redundant weight and gradient copies. These optimizations contribute to its larger performance gain.

Although retaining quantized weight shards increases persistent memory usage, outer ZeRO-1 sharding reduces the memory occupied by main weights and optimizer states. At 128 GPUs, peak memory is comparable to or lower than the lowest-memory baseline on all four models. On MiMo, SS-HSDP uses 125.0 GiB per GPU, 6.2% less than the lowest-memory baseline.

FP8 versus BF16 MXFP8 improves SS-HSDP’s throughput over BF16 by 16.3–18.5% on MiMo and about 7% on Internal MoE, while Qwen3 throughput changes by less than 1% (Appendix § 10.§ 2). Qwen3 profiles show that quantization overhead can offset the savings from

faster expert GEMMs. Long-context attention and normalization times remain the same when expert GEMMs use FP8.

Distributed Muon

Using distributed Muon instead of AdamW increases median step time by only 2.5–4.1% across the tested models and sharding configurations (Table 3). We compare paired runs at 128 GPUs with the same model, token count, sharding layout. Muon updates 88.1% of parameters in Internal MoE and 97.9% in Qwen3; AdamW updates the remainder. The implementation keeps main weights and momentum sharded: elementwise stages run locally, while matrix stages exchange only the required intermediate fragments. These results show that Muon can be applied to most model parameters with modest end-to-end training overhead.

Modelsws_wAdamWMuonOverheadPeak mem. (A/M)
Internal MoE12830.3429.60+2.5%85.8/82.1
6430.2829.40+3.0%88.8/85.1
3230.3729.25+3.9%92.7/89.0
Qwen312827.4126.34+4.1%60.1/59.2
6427.3326.25+4.1%62.8/61.9
3227.1426.23+3.5%66.4/65.5

Table 3. Muon versus AdamW on 128 GPUs with BF16 computation. Peak memory (AdamW/Muon, GiB) includes PyTorch reserved memory and external communication buffers.

Ablation and Analysis

Precision Ablation

To check that BF16SR in § 4.3 does not change training convergence, we ablate communication precision and train a 12-layer, 8.1B-A1.3B Qwen3 MoE variant [44] on FineWeb- Edu [29] for 100B tokens on 128 GPUs with (sw,sg,so)=(8,8,128)(s_w,s_g,s_o)=(8,8,128).

We compare intra/inter-node reduction input precisions: FP32, BF16+FP32, and BF16SR. All experiments use BF16 runtime weights and FP32 optimizer state. The three experiments use identical model, data order, optimizer settings, and seed; only the reduction precision changes. Figure 7 uses FP32 as the loss-difference baseline.

Figure 7. Convergence and loss differences. All runs converge without sustained divergence. Final-10k-step mean loss ranks BF16SR < BF16+FP32 < FP32. Top: training loss. Bottom: ΔL=L−LFP32\Delta L = L - L_{\mathrm{FP32}}. Solid lines show EMA (span 500).

All three experiments converge with training-loss differences on the order of 10−310^{-3}. The final-10k-step mean loss ranks BF16SR < BF16+FP32 < FP32, with differences of −1.10×10−3-1.10\times10^{-3} and −8.39×10−4-8.39\times10^{-4} from FP32, both below 0.05% in magnitude and stable over training.

ZeroSM Communication across Workloads

We vary per-GPU workload size for MiMo at 64 GPUs with sharding degree 64 under BF16 computation and four microbatches per optimizer step, comparing ZeroSM, SM-based communication, and default FSDP2 (Appendix § 10.§ 3).

ZeroSM’s advantage emerges at larger workloads Figure 8 shows a crossover. At the smallest workload, ZeroSM delivers 33.5% lower throughput than SM-based SS-HSDP; at the largest, it gains 8.8%. The trend is consistent with a latency-versus-contention trade-off: communication-bound loads expose collective latency on the critical path, whereas compute-bound loads hide it through overlap and free SM capacity for model kernels.

Figure 8. The benefit of ZeroSM depends on the workload. Throughput relative to default FSDP2 for MiMo (12 layers), using median step times. The dashed line marks default FSDP2; the x-axis (tokens per GPU per microbatch) is logarithmic.

SM-based communication also outperforms FSDP2 SM-based communication retains zero-copy all-gather weight views and direct-gradient writes. Across the five tested workloads, it achieves 1.106–1.566× the throughput of default FSDP2, including 1.561× at the smallest workload. Thus, the throughput advantage over FSDP2 persists without ZeroSM.

Outer ZeRO-1 Sharding and Weight Reuse

Outer ZeRO-1 sharding reduces persistent memory by further partitioning FP32 main weights and optimizer states, without enlarging the groups for repeated weight all-gathers. In the main experiment, SS-HSDP and Megatron-FSDP fix sw=32s_w = 32 and set sos_o to the GPU count, whereas FSDP2 and veScale-FSDP replicate 32-way optimizer shards. From 32 to 64 to 128 GPUs, SS-HSDP’s peak per-GPU memory falls by about 3.9 GiB per doubling on Internal MoE. On MiMo, it falls by about 8.6 GiB per doubling (Figure 6). Outer ZeRO-1 sharding yields greater memory savings for larger models. Weight reuse avoids repeated quantization and scale relayout by retaining low-precision weight shards and scales across microbatches. This requires additional persistent storage, but at 128 GPUs, SS-HSDP’s peak memory remains comparable to or lower than the lowest-memory baseline on all tested MXFP8 workloads in § 5.

Related Work

Large-model pretraining practices Training language models [11, 14, 44] and video models [1, 37, 38, 42] requires flexibility in sharding, runtime numerical representations, and optimizer update granularity. DeepSeek-V4 uses different ZeRO partitions for dense and expert parameters to support Muon, while DeepSeek-V3 [10] and MiMo-V2-Flash [43] combine FP8 computation with higher precision for selected operations and parameters. Kimi K3 [18] and GLM-5 [13] use attention heads as Muon update units.

Runtime optimizations for sharded training Runtime optimizations target communication volume, memory traffic, and GPU resource contention during forward and backward execution. MiCS [46] and ZeRO++ [39] improve communication locality through smaller sharding groups and intra-node backward weight all-gathers, respectively. TorchTitan [19] gathers weights in FP8 for subsequent low-precision computation. Collective bucketing and reordering [45] and topology-aware partitioning [2] improve overlap; DeepCompile [36] prefetches parameters to overlap all-gathers with computation and retains gathered weights to avoid repeated all-gathers. Computation–collective fusion avoids intermediate memory accesses [16]. Host-driven NCCLX and NCCL paths offload data movement to reduce SM usage [26, 35].

Distributed parameter updates Distributed optimizer implementations coordinate matrix inputs, update computation, and persistent states across ranks. Distributed Muon [20] gathers matrix inputs for orthogonalization while retaining sharded optimizer states. Canzona [40] and DMuon [6] assign matrix updates to designated ranks and balance optimizer workloads. Canzona further pipelines input reconstruction, update computation, and result redistribution under tensor parallelism. LinS [5] and InternEvo [4] independently configure parameter, gradient, and optimizer-state sharding to trade memory for communication.

Conclusion

SS-HSDP shows that flexible optimizer updates and efficient runtime execution can be supported within a unified sharded training framework. Shared storage and hierarchical sharding allow structured updates to coexist with packed computation and communication layouts while avoiding redundant data copies. Our evaluation demonstrates that this flexibility can be achieved with improved training efficiency and competitive memory usage. These findings support separating optimizer update requirements from runtime data organization as a practical approach to sharded training.

Implementation Details

Mapping paper terms to the source. A logical parameter is represented by Parameter; the kernel-facing runtime tensor is carried by RuntimeParameter. The source identifiers zero3, zero2, and zero1 denote the retained weight, accumulation, and optimizer shard levels, respectively. Their paper symbols are sws_w, sgs_g, and sos_o. The copy-engine/RMA implementation resides under zero_sm; ZeroSmGrad selects its gradient lifecycle and precision modes. These identifiers are implementation names rather than additional parameter or sharding concepts.

Packing and compatibility. TensorList stores member shapes and offsets and derives the fragments at each shard level. An Atom groups parameters with matching tensor field names, dtypes, devices, optimizer-state layouts, and update buffers. AdamW and Muon parameters with different state schemas therefore use separate Atom objects. Runtime and update groups batch compatible operations across these objects. Each input or output field declares the number of elements per update unit. Fields may use different numbers of elements per unit, but must describe the same number of units.

Allocations and update execution. The data tree uses DataNode objects whose root owns storage. Finalization resolves child offsets and forwards allocation callbacks to that root. These callbacks attach or detach the runtime tensor’s view while retaining its metadata. Update intermediates such as uiu_i and viv_i have update-scoped allocations; executors retaining temporary views are released or rebuilt with those allocations. Stages run in order within one update group, committing intermediates and scalar state for the next stage. Independent groups retain separate schedules.

Execution conditions. Table 4 distinguishes metadata checks from execution conditions whose correctness depends on the producer and scheduler. The direct gradient write mechanism and BF16 communication path are described in § 4; the table records the corresponding storage, hardware, and restart conditions. In particular, explicit SUM normalization and valid gradients for every updated logical parameter connect the runtime’s accumulated values to the intended optimizer step.

ConditionRationaleConsequence if violated
Direct writes cover every element on the first write of the current buffer lifetime; later writes accumulate.Overwrite mode replaces clearing; zero-contribution slices receive explicit zeros.Unwritten slices retain stale data, or repeated overwrites discard contributions.
Compatible field schemas within each Atom; transforms preserve their declared complete units.Packing connects matching fields, dtypes, devices, optimizer states, and update buffers.Planning rejects incompatible metadata; incompatible fields use separate Atom objects.
Ordered stages within each update group; intermediates are produced before use and scalar updates agree across fragments.Each stage consumes committed outputs; independent groups have separate schedules.Reads of temporary buffers before production are rejected; inconsistent scalar updates violate stage semantics.
Nested sharding degrees; equal shards for copy-engine/RMA collectives; balanced node-major ranks and stable registered bases/offsets.Collective layouts and registered addresses describe the same rank-to-data mapping.Invalid geometry and detected binding changes are rejected; lifetime violations risk invalid accesses.
Events or stream order connect producers, consumers, and reuse; shared RS workspace calls are serialized.One arena region serves successive nonoverlapping lifetimes.Concurrent reuse races with unfinished transfers or reductions.
BF16 partials use supported hardware SR; seed/counter state is restored on the same logical rank.The SR instruction and element-to-counter mapping define the numerical path.Unsupported hardware or compiled images are rejected before communication; missing RNG state changes resumed rounding.
BF16 RS uses full-tensor, layout-preserving Cast transforms, matching packed offsets, and separate BF16 input and FP32 accumulation storage.Reduction consumes the input directly while accumulation shards persist.Incompatible transform/layout metadata and detected storage overlap are rejected.
The specialized copy-engine/RMA path runs without CUDA Graph replay.Host protocol counters and Python SR counters have no implemented replay semantics.Graph capture/replay is unsupported.
All contributing shared-parameter forwards are registered before their final-microbatch backwards.Backward-use counts determine eligibility for final synchronization.Sequential forward/backward/forward/backward reuse can synchronize too early.
Every updated trainable logical parameter has a valid final gradient; new accumulation rounds clear prior gradients explicitly.Updates consume gradient shards, without unused-parameter discovery or missing-gradient skipping.Missing or stale contributions enter the update.
Loss, replica, and microbatch normalization are explicit after final synchronization.Gradient collectives use SUM semantics.Omitting the intended scale changes the effective gradient.

Table 4. Execution conditions, their rationale, and the consequence of violating them. Checks are identified where implemented.

Evaluation and Artifact Details

Implementation size. The inspected source snapshot contains 85 Python and native source files, comprising 21,624 physical lines. A source-oriented count excluding blank lines, comments, and Python docstrings gives approximately 17,606 lines: 14,217 in Python and 3,389 in native code. Preprocessor directives and structural delimiters are retained. These totals cover shared infrastructure and specialized backends. The artifact preparation will package the counting script, file manifest, and immutable source identifier alongside this breakdown, so the same rule can be applied to extension examples.

Integration and extension measurements. We will provide runnable examples showing how to convert a model, define update parameters, construct runtime parameters, and configure sharding. The four-head query example packs WQ=[W0;W1;W2;W3]W_Q = [W_0; W_1; W_2; W_3], with Wi∈Rdh×dW_i \in\mathbb{R}^{d_h \times d}, into a BF16 runtime tensor and retains FP32 main weights for per-head Muon. Its execution settings follow Table 4. Each example includes initialization, gradient clearing, first/last microbatch flags, normalization, and update. Engineering measurements count added conversion code, optimizer declarations, backend-specific code, and integration touchpoints for AdamW, expert-wise Muon, and per-head Muon. Shared utilities are reported separately. Correctness checks compare the declared logical updates with a local reference.

Reproduction records. Machine-readable configurations will record hardware, software versions, source revisions, topology, rank mapping, model and optimizer settings, sharding degrees, grouping policy, precision, seeds, and timing boundaries. Raw per-step times and memory observations accompany aggregate statistics. Numerical cases preserve the original inputs, communicated representations, and rounding seeds where feasible, allowing input and partial quantization to be examined independently. Diagnostics document failed configurations and the selection of supported replacements.

Artifact preparation. Release preparation covers licensing, dependency specifications, runnable examples, multi-rank correctness tests, benchmark scripts, and backend configuration guidance. A manifest links each paper experiment to its configuration and output records. Together, these materials make extension size, reproduced behavior, and measured performance traceable to the same implementation snapshot.

Performance Settings

Table 5 gives the workloads used in § 5.1. All runs use 32, 64, or 128 GPUs (eight per node), AdamW, full activation recomputation, and gradient accumulation (GA) of four microbatches per optimizer step. Inputs are synthetic. Per-GPU microbatch size (MBS) and input length remain fixed as GPU count increases.

ModelParams (B)LayersTokensMBSGA
Internal MoE33.8872,18614
Internal Dense19.22472,18614
Qwen330.54824,57644
MiMo (12L)73.51224,57644

Table 5. Performance workloads. Parameter counts include all experts. MBS denotes samples per GPU per microbatch; GA is the number of microbatches per optimizer step.

Weight and gradient sharding degrees are fixed at 32; SS-HSDP and Megatron-FSDP shard optimizer states across all GPUs. MXFP8 uses DeepGEMM for expert GEMMs, or the video FFN of Internal Dense; remaining GEMMs use BF16, with FP32 optimizer states. SS-HSDP and Megatron-FSDP retain quantized weights, whereas FSDP2 and veScale-FSDP quantize on demand.

Software. All end-to-end runs use PyTorch 2.14.0+cu132, CUDA 13.2, NCCL 2.30.7, and DeepGEMM 2.6.1b2. The PyTorch FSDP2 baseline is the implementation in PyTorch 2.14; veScale-FSDP runs from our reproduction; Megatron-FSDP is the implementation in Megatron Core 0.18.2. Baselines use the same model partition, precision settings, and per-GPU workload as SS-HSDP.

Timing uses six-step medians after two warmup steps. Model FLOPs exclude recomputation. We report per-GPU model FLOPs utilization (MFU), i.e., achieved model FLOP/s as a fraction of the GPU’s BF16 peak FLOP/s. Memory is peak PyTorch reserved memory plus external communication buffers, maximized over measured steps and ranks.

BF16 Training Performance

Figure 9 complements the MXFP8 results in § 5.1 with the same workloads, GPU counts, and measurement procedure. Table 6 compares SS-HSDP’s MXFP8 and BF16 throughput, using ratios of median step times. The benefit depends on the workload: it is largest on MiMo, smaller on the internal models, and negligible on Qwen3.

Figure 9. BF16 training performance. Workloads and sharding layouts match Figure 6, with BF16 GEMMs. Top: per-GPU model FLOPs utilization (MFU). Bottom: peak reserved memory plus external communication buffers. Each model uses its own y-axis ranges.

Model32 GPUs64 GPUs128 GPUs
Internal MoE1.072×1.071×1.072×
Internal Dense1.031×1.032×1.027×
Qwen30.995×0.994×1.002×
MiMo (12L)1.185×1.163×1.165×

Table 6. SS-HSDP MXFP8 throughput relative to its own BF16 runs.

Communication-Backend Workload Sweep

§ 6.2 uses a separate BF16 sweep, not the MXFP8 scaling experiment. MiMo (12 layers, 73.5B parameters) runs on eight B200 nodes (eight GPUs per node) for each point, with all three configurations on the same hosts and input shape; otherwise the setup follows Appendix § 10.§ 1, with DeepGEMM BF16 and seed 42. Context and expert parallelism are disabled; profiling and checkpointing are disabled during timing. MiMo uses MBS 1 at 2,048, 8,192, and 24,576, then MBS 4 at 24,576 and 36,864 (98,304 and 147,456 tokens per GPU per microbatch). Token count is a workload descriptor, not an arithmetic-intensity classification.

The sweep has 15 configurations. Each point is the median of zero-based steps 5–14 (ten timed steps after five warmup steps), from one run per configuration; these are within-run samples, not independent repetitions. All accepted runs passed configuration, host-identity, and GPU-clean checks, with no OOM or allocation retry.

ZeroSM uses BF16 runtime gradients with FP32 inter-node reduction and accumulation; SM-based communication uses FP32 gradients and NCCL collectives. Both retain SS-HSDP’s shared weight views and direct gradient writes, but differ in gradient storage, symmetric arenas, hierarchical reduction, and accumulation. ZeroSM offloads payload movement while intra-node and final reductions still use SMs, so the comparison does not isolate transport latency or SM occupancy. Both SS-HSDP modes use (sw,sg,so)=(64,64,64)(s_w, s_g, s_o) = (64, 64, 64), and FSDP2 uses sharding degree 64; the layout was fixed rather than varied, so gains are measured at this sharding degree only. SS-HSDP performs the gradient-to-optimizer reduction only on the final microbatch, and no deferred-replica-all-reduce variant is evaluated.

References

  1. [1]Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: A highly consistent, dynamic and skilled text-to-video generator with diffusion models, 2024.arxiv.org/abs/2405.04233
  2. [2]Chang Chen, Xiuhong Li, Qianchao Zhu, Jiangfei Duan, Peng Sun, Xingcheng Zhang, and Chao Yang. Centauri: Enabling efficient scheduling for communication-computation overlap in large model training via communication partitioning. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, 2024.DOI
  3. [3]Jianfei Chen, Yu Gai, Zhewei Yao, Michael W. Mahoney, and Joseph E. Gonzalez. A statistical framework for low-bitwidth training of deep neural networks. In Advances in Neural Information Processing Systems, volume 33, pages 883–894, 2020.arxiv.org/abs/2010.14298
  4. [4]Qiaoling Chen, Diandian Gu, Guoteng Wang, Xun Chen, Yintong Xiong, Ting Huang, Qinghao Hu, Xin Jin, Yonggang Wen, Tianwei Zhang, and Peng Sun. InternEvo: Efficient long-sequence large language model training via hybrid parallelism and redundant sharding, 2024.arxiv.org/abs/2401.09149
  5. [5]Qiaoling Chen, Qinghao Hu, Guoteng Wang, Yintong Xiong, Ting Huang, Xun Chen, Yang Gao, Hang Yan, Yonggang Wen, Tianwei Zhang, and Peng Sun. Lins: Reducing communication overhead of ZeRO for efficient LLM training. In 2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS), pages 1–10, 2024.DOI
  6. [6]Vincent Chen, Starrick Liu, Regis Cheng, Dance Yang, Shalfun Li, Ryan Yu, Lucy Liang, Hang Su, Roy Gan, Hao Wang, and Qian Wang. Dmuon: Efficient distributed Muon training with near-Adam overhead, 2026.arxiv.org/abs/2606.27153
  7. [7]Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, et al. PaLM: Scaling language modeling with pathways, 2022.arxiv.org/abs/2204.02311
  8. [8]Damai Dai, Chengqi Deng, Chenggang Zhao, R.X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1280–1297, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
  9. [9]DeepSeek-AI. DeepGEMM: Scaling factor format. Official DeepGEMM repository documentation, n.d. Accessed 2026-10-22.
  10. [10]DeepSeek-AI et al. DeepSeek-V3 technical report, 2024.arxiv.org/abs/2412.19437
  11. [11]DeepSeek-AI et al. DeepSeek-V4: Towards highly efficient million-token context intelligence, 2026.arxiv.org/abs/2606.19348
  12. [12]William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.arxiv.org/abs/2101.03961
  13. [13]GLM-5-Team et al. GLM-5: from vibe coding to agentic engineering, 2026.arxiv.org/abs/2602.15763
  14. [14]Aaron Grattafiori et al. The Llama 3 herd of models, 2024.arxiv.org/abs/2407.21783
  15. [15]Dirk Groeneveld et al. OLMo: Accelerating the science of language models, 2024.
  16. [16]Abhinav Jangda, Jun Huang, Guodong Liu, Amir Hossein Nodehi Sabet, Saeed Maleki, Youshan Miao, Madanlal Musuvathi, Todd Mytkowicz, and Olli Saarikivi. Breaking the computation and communication abstraction barrier in distributed machine learning workloads. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2022.
  17. [17]Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024.
  18. [18]Kimi Team et al. Kimi K3: Open frontier intelligence, 2026.arxiv.org/abs/2607.24653
  19. [19]Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, Sanket Jayant Pundare, Gokul Nadathur, and Stratos Idreos. TorchTitan: One-stop PyTorch native solution for production ready LLM pretraining. In International Conference on Learning Representations, 2025.
  20. [20]Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is scalable for LLM training, 2025.arxiv.org/abs/2502.16982
  21. [21]Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In International Conference on Learning Representations, 2018.arxiv.org/abs/1710.03740
  22. [22]NVIDIA. Cosmos world foundation model platform for physical AI, 2025.arxiv.org/abs/2501.03575
  23. [23]NVIDIA. cuBLAS: Block scaling factors layout. cuBLAS 13.4 Documentation, n.d. Accessed 2026-10-22; layouts and supported scaling modes are version-specific.
  24. [24]NVIDIA. Megatron-FSDP. Megatron Core 0.18.2 Documentation, n.d. Accessed 2026-10-22.
  25. [25]NVIDIA. Megatron-LM and Megatron Core. GitHub repository, n.d. Accessed 2026-10-28.
  26. [26]NVIDIA. NCCL release 2.30.7. NCCL Release Notes, n.d. Zero-SM AllGather and All2all; accessed 2026-10-30.
  27. [27]NVIDIA. NCCL user guide: User buffer registration. NCCL 2.31.2 Documentation, n.d. Accessed 2026-10-22.
  28. [28]Open Compute Project. OCP microscaling formats (MX) specification. Technical Report Version 1.0, Open Compute Project, September 2023.
  29. [29]Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale, 2024.arxiv.org/abs/2406.17557
  30. [30]PyTorch. PyTorch DTensor: Distributed tensor. PyTorch Documentation, n.d. Version 2.14; accessed 2026-10-28.
  31. [31]PyTorch. PyTorch FSDP2: torch.distributed.fsdp.fully_shard. PyTorch Documentation, n.d. Version 2.14; accessed 2026-10-22.
  32. [32]Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory optimizations toward training trillion parameter models, 2019.arxiv.org/abs/1910.02054
  33. [33]Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, pages 3505–3506. Association for Computing Machinery, 2020.
  34. [34]Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism, 2019.arxiv.org/abs/1909.08053
  35. [35]Min Si, Pavan Balaji, Yongzhou Chen, et al. Collective communication for 100k+ GPUs, 2025.arxiv.org/abs/2510.20171
  36. [36]Masahiro Tanaka, Du Li, Umesh Chand, Ali Zafar, Haiying Shen, and Olatunji Ruwase. DeepCompile: A compiler-driven approach to optimizing distributed deep learning training, 2025. Version 2, 19 February 2026.
  37. [37]Team Seedance et al. Seedance 2.0: Advancing video generation for world complexity, 2026.arxiv.org/abs/2604.14148
  38. [38]Wan Team. Wan: Open and advanced large-scale video generative models, 2025.arxiv.org/abs/2503.20314
  39. [39]Guanhua Wang, Heyang Qin, Sam Jacobs, Xiaoxia (Shirley) Wu, Connor Holmes, Zhewei Yao, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, and Yuxiong He. ZeRO++: Extremely efficient collective communication for large model training. In International Conference on Learning Representations, pages 50035–50053, 2024.
  40. [40]Liangyu Wang, Siqi Zhang, Junjie Wang, Yiming Dong, Bo Zheng, Zihan Qiu, Shengkun Tang, Di Wang, Rui Men, and Dayiheng Liu. Canzona: A unified, asynchronous, and load-balanced framework for distributed matrix-based optimizers, 2026.arxiv.org/abs/2602.06079
  41. [41]Zezhou Wang, Youjie Li, Zhiqi Lin, Jiacheng Yang, Cong Xie, Guanyu Feng, Zheng Zhong, Ziyue Huang, Hongyu Zhu, Zhi Zhang, Yanghua Peng, and Xin Liu. veScale-FSDP: Flexible and high-performance FSDP at scale. In Proceedings of Machine Learning and Systems, volume 8, pages 763–776. MLSys, 2026. Industry Track.arxiv.org/abs/2602.22437
  42. [42]Bing Wu, Chang Zou, Changlin Li, et al. HunyuanVideo 1.5 technical report, 2025.arxiv.org/abs/2511.18870
  43. [43]Xiaomi LLM-Core Team, Bangjun Xiao, Bingquan Xia, Bo Yang, et al. MiMo-V2-Flash technical report, 2026.
  44. [44]An Yang et al. Qwen3 technical report, 2025.arxiv.org/abs/2505.09388
  45. [45]Ruisi Zhang, Tianyu Liu, Will Feng, Andrew Gu, Sanket Purandare, Wanchao Liang, and Francisco Massa. SimplerFSDP: Simpler fully sharded data parallel with torch.compile, 2024.
  46. [46]Zhen Zhang, Shuai Zheng, Yida Wang, Justin Chiu, George Karypis, Trishul Chilimbi, Mu Li, and Xin Jin. MiCS: Near-linear scaling for training gigantic model on public cloud. Proceedings of the VLDB Endowment, 16(1):37–50, 2022.
  47. [47]Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balıoglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. PyTorch FSDP: Experiences on scaling fully sharded data parallel, 2023.

Paper details

vlnc:W_3dn2bpq8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents