betaFeedback

WorksComputer Science

Zero-SM NCCL: A Systematic Guide to Designing and Deploying SM-Free GPU Communication

Abstract

GPU communication libraries such as NCCL use Streaming Multiprocessors (SMs) for data movement and synchronization, slowing overlapped computation at a cost that grows with interconnect bandwidth. We present Zero-SM NCCL, which communicates within and across scale-up domains without launching any SM kernel. It synchronizes through CUDA stream memory operations, moves data asynchronously with copy engines within a scale-up domain, and drives network RDMA from a stream-ordered CPU proxy. It supports data-movement collectives such as AllGather and AlltoAll, plus one-sided ncclPutSignal and ncclWaitSignal APIs. On GB200 NVL72 and B200 NVL8 systems, Zero-SM collectives reach up to 1.36x higher AllGather bandwidth than NCCL's legacy SM-based kernels and cut the median slowdown of overlapped GEMMs from 33% to 4%, at a higher latency per operation. In Megatron-LM training, they accelerate FSDP by up to 3.5%, and one-sided puts accelerate pipeline parallelism by up to 15%. Zero-SM communication is not a universal win, however. Through an end-to-end study across models and parallelism strategies, we show that it pays off for large transfers in GPU-bound workloads and for deep pipelines with large bubbles, but loses on small messages and host-bound workloads. We distill these findings into guidelines for enabling zero-SM communication in NCCL and building it into other libraries.

Introduction

Large language model (LLM) training and inference run on thousands of GPUs connected by high-bandwidth interconnects such as NVLink and InfiniBand (IB). Every training step exchanges parameters, gradients, and activations among these GPUs, so collective communication sits at the heart of every parallelism strategy. Libraries such as the NVIDIA Collective Communications Library (NCCL) [1] and NVSHMEM [2, 3] have become the de facto runtimes for this GPU-to-GPU data movement, and training frameworks such as Megatron-LM [4] rely on them for every collective. To saturate the interconnect, these libraries run their data transfers and synchronization as kernels on the GPU’s Streaming Multiprocessors (SMs), its general-purpose compute units.

SMs, however, are the scarcest resource in LLM training. Its general matrix multiplications (GEMMs) launch far more thread blocks than a GPU has SMs, so every SM held by communication delays compute. Yet the SMs NCCL needs

to saturate NVLink have grown from 8 on Pascal to 32 on Blackwell (Figure 1A), and SM-based kernels can lower this cost only by giving up bandwidth (Figure 1B). On one B200 node, a 144 MiB AllGather slows the GEMMs it overlaps by about 33% under NCCL’s legacy ring and 6% under its symmetric multicast kernel, against 3.9% when copy engines move the data instead of SMs. In LLM training, zero-SM communication pays off end to end in two ways. Freeing SMs speeds up fully sharded data parallel (FSDP) training by up to 3.5%, and stream-ordered one-sided puts, which decouple pipeline stages, speed up pipeline-parallel (PP) training by up to 14.9%. These speedups save up to $2.6M over a 15T-token pretraining run at on-demand cloud prices [5] (Figure 1C).

Figure 1. (A) SMs that NCCL needs to saturate NVLink per GPU generation, with their share of the GPU’s SMs. (B) Median slowdown of the Qwen3-30B-A3B GEMMs that a 144 MiB AllGather slows by more than 1%, against its CTA count, on one B200 node. Rings mark default CTA counts, and the symmetric kernel is the multicast variant. Legacy beats Zero-SM NCCL only at 4–8 CTAs, where its AllGather is 3.2–5.6× slower. (C) Estimated savings of Zero-SM NCCL, over a 15T-token pretraining run, from freed SMs (FSDP) or one-sided puts (PP), with step-time speedups in parentheses.

Several efforts have sought to reduce the SM cost of communication. NCCL’s symmetric kernels replace multi-stage pipeline protocols with single-shot algorithms [6, 7] that, by default, use a few CTAs instead of tens (§IV-C). However, SMs are still required for data movement and synchronization. NCCLX [8] offloads bulk data movement to dedicated hardware units but retains SM kernels for synchronization, falling short of full elimination. Moreover, most existing approaches target either scale-up or scale-out communication in isolation, leaving no SM-free option for workloads whose communication spans both.

In this work, we present Zero-SM NCCL, a complete solution that enables GPU communication without consuming any SMs. Zero-SM NCCL covers both scale-up and scale-out paths and supports both collective operations (e.g., AllGather and AlltoAll) and one-sided primitives (Put, Wait) that allow users to build custom communication patterns. To achieve this, Zero-SM NCCL combines three techniques. SM-free synchronization based on CUDA stream memory operations replaces all SM-driven barriers and completion tracking. Copy engines (CEs) move data over NVLink. A CPU proxy drives network Remote Direct Memory Access (RDMA) while preserving CUDA stream semantics. Together, these techniques offload both data movement and synchronization from SMs entirely.

The contributions of this work are as follows:

  • To the best of our knowledge, Zero-SM NCCL is the first communication framework that completely eliminates SM usage for both scale-up and scale-out GPU communication, covering collective operations and one-sided primitives within a single, unified design.

  • We introduce new one-sided APIs (ncclPutSignal and ncclWaitSignal) that enable users to build custom zero-SM communication patterns, and implement zero-SM collective algorithms including a hierarchical AllGather that overlaps scale-out transfers with scale-up distribution.

  • We integrate Zero-SM NCCL into the FSDP and PP paths of Megatron-LM and evaluate it on GB200 NVL72 and B200 NVL8 systems, from microbenchmarks and a compute-overlap benchmark on GEMMs to 19 end-to-end training recipes of dense and mixture-of-experts (MoE) models.

  • We show that zero-SM communication is not a universal win. It pays off for large transfers in GPU-bound workloads and for deep pipelines with large bubbles, but loses on small messages and host-bound workloads. We distill these conditions into guidelines that serve as a reference for enabling zero-SM communication in NCCL and for building it into other communication libraries.

All code needed to reproduce our results, including the NCCL patches, benchmark harnesses, and training recipes, is publicly available at https://github.com/spcl/zero-sm-artifact.

Background

GPU Interconnect Landscape

Large clusters use a two-level communication hierarchy. A scale-up domain connects GPUs whose memories are load/store-accessible (LSA) to one another, through NVLink and NVSwitch or PCIe peer access. On Blackwell, NVLink5 provides 1.8 TB/s per GPU (900 GB/s per direction) [9], and NVSwitch adds NVLink SHARP (NVLS), which multicasts writes and reduces data inside the fabric. The scale-out network connects scale-up domains through RDMA-capable NICs. Current DGX systems use up to eight 400 Gb/s InfiniBand or RDMA over Converged Ethernet (RoCE) NICs per node [10], and per-link rates continue to rise [11, 12].

GPU Data Movement Engines

GPU copy engines. Copy engines (CEs) serve operations such as cudaMemcpyAsync and move data over NVLink without occupying SMs once a transfer is launched. In contrast, the Tensor Memory Accelerator (TMA) offloads structured transfers from threads [13, 14] but still needs SM-resident kernels to issue and synchronize them [14].

RNIC DMA engines. RDMA-capable NICs (RNICs), such as NVIDIA ConnectX adapters, contain direct memory access (DMA) engines that read and write GPU memory directly through GPUDirect RDMA [15]. Like CEs, they move data concurrently with SM execution, as long as the control and synchronization around each transfer also avoid SMs.

How NCCL Uses SMs for Communication

NCCL [1, 16] is the primary library for multi-GPU communication on NVIDIA GPUs. Within a scale-up domain, its pipeline protocols (Simple, LL, and LL128) [16] use tens of thread blocks, or cooperative thread arrays (CTAs), whose threads copy data between stages, perform reductions, and poll flags. Across domains, SM kernels pack data and post descriptors to CPU proxy threads, which drive the RDMA operations. On both paths, SMs also perform all synchronization, spinning on flags, executing barriers, and tracking completion. This work is the hardest to offload, because it involves conditional control flow that DMA engines cannot express.

Host- vs. device-initiated communication. NCCL’s collectives are host-initiated. The CPU enqueues operations on CUDA streams, which the GPU then executes. Device-initiated APIs such as NVSHMEM [2, 3, 17] and NCCL’s GPU-Initiated Networking (GIN) [18] instead issue operations from inside CUDA kernels, which enables fusion but requires SMs. Zero-SM NCCL therefore targets the host-initiated API.

Symmetric memory registration. NCCL’s symmetric registration lets all ranks collectively register a buffer, so every peer learns its address at registration time [19]. Zero-SM NCCL relies on it in three ways. CEs copy directly into peer user buffers, NICs access them through GPUDirect RDMA with the remote addresses that one-sided puts need, and signal locations are registered the same way, so peers can write them over both NVLink and the network.

Prior work reduces, but does not completely remove, the SM cost of communication.

NCCL’s symmetric kernels [6, 7] replace multi-stage pipelines with single-shot load/store algorithms that need only a few CTAs, but those CTAs still move data, reduce, and poll flags. MSCCL++ [20] exposes programmable channels whose primitives are all invoked from GPU kernels. Even its port channels, which hand data movement to a CPU proxy, signal and wait inside the kernel.

NCCLX [8] moves data with CEs and RDMA from CPU threads, but launches a small kernel per host-initiated collective to enforce stream order. VCCL [21] offers an SM-free mode that uses CEs and a CPU proxy with stream memory operations, but only for point-to-point transfers. Concurrent HOCCL [22] also moves AllGather, AlltoAll, and Send/Recv data with DMA engines, but stages cross-node data through peer GPUs, adding an NVLink copy at each end, and is evaluated only on 32 H800 GPUs. We do not compare against HOCCL, as it appeared concurrently with this work without public source code. On AMD GPUs, ConCCL [23], DMA-Latte [24], and FiCCO [25] run AllGather and AlltoAll on DMA engines, and PyTorch’s asynchronous tensor parallelism [26] and Flux [27] use CE copies for AllGather. These DMA-based designs are confined to a single node, and the fused operators still wait on SMs.

GPUDirect Async [28] triggers pre-posted IB operations from CUDA streams, HPE’s stream-triggered Message Passing Interface (MPI) [29] does so with Slingshot NIC triggers, and MPIX Stream [30] enqueues MPI operations on streams (see [31] for a survey). These target point-to-point messages, and none provides zero-SM collectives or a copy-engine path.

New hardware such as T3 [32], ACE [33], and the Scalable Hierarchical Aggregation Protocol (SHArP) [34] offloads collectives, and Maia 200 [35] builds signaled DMA transfers into its network protocol, sending remote semaphore updates together with the data. Overlap systems such as TileLink [36] and FlashOverlap [37] instead schedule around the contention between compute and communication, which can slow overlapped compute by up to 40% [38], rather than removing it.

No existing library offers both stream-ordered collectives and one-sided operations that launch no SM kernel at all, within and across scale-up domains, on hardware available today. Zero-SM NCCL fills this gap (§III). In addition, we are the first to systematically analyze when zero-SM communication pays off end to end and to distill guidelines from it.

Zero-SM NCCL Design

Overview and Scope

Zero-SM NCCL eliminates SM consumption in GPU communication by offloading both data movement and synchronization. Figure 2 shows the overall architecture.

Figure 2. Overview of Zero-SM architecture for NCCL’s on-stream host API.

Design requirements. All Zero-SM NCCL operations are designed around the following two requirements:

R1 Avoid SM involvement. A zero-SM operation must not launch GPU kernels or consume SM resources.

R2 Preserve CUDA stream semantics. Zero-SM operations must compose with other CUDA operations. They must be stream-ordered, meaning that an operation starts only after earlier work in the same stream completes, and later work in the stream starts only after the operation completes.

Offload strategy. For scale-up communication among peers, e.g., over NVLink or PCIe, data movement is offloaded to CEs. For scale-out communication over the network, data movement is driven by CPU proxy threads that post RDMA operations to the NIC. In both cases, synchronization is handled by CUDA stream memory operations (§III-B). All buffers and signal locations are symmetrically registered (§II-C), so senders push data directly into peer buffers, like CE copies or RDMA writes, without staging copies or runtime address exchange.

One-sided APIs. Zero-SM NCCL introduces two new one-sided remote memory access (RMA) APIs. ncclPutSignal transfers data to a remote peer and updates a remote signal upon completion, and ncclWaitSignal blocks the CUDA stream until a signal reaches a specified value. Both operations execute without SM kernels. These primitives allow advanced users to construct custom communication patterns (e.g., pipelined exchanges or application-specific all-to-all variants) entirely in zero-SM mode. The collectives described in §III-D share the same underlying mechanisms.

Supported collectives. Zero-SM NCCL targets collectives that do not involve arithmetic, namely AllGather, AlltoAll, Gather, and Scatter. These collectives are pure data movement operations and can be decomposed into pairwise transfers with synchronization, which is exactly the pattern that CEs and proxy threads can execute without SMs. Reduction collectives, such as AllReduce and ReduceScatter, require arithmetic on the data and are not yet supported in zero-SM mode.

SM-Free Synchronization Primitives

A sender must know that the receiver’s buffer is ready, and the receiver that data has arrived. NCCL detects both with SM kernels that spin on flags (§II-C), which violates R1. Zero-SM NCCL instead uses CUDA stream memory operations, which are stream-ordered and do not need SMs, satisfying R1 and R2. The CUDA driver’s stream memory operations, e.g., cuStreamWriteValue64 and cuStreamWaitValue64, write to or wait on a memory location as part of a stream without launching a kernel [39]. Zero-SM NCCL builds its synchronization from both stream memory operations and CE operations.

Sequence-based protocol. To prevent race conditions, each signal location uses a monotonically increasing sequence number rather than a boolean. The sequence number starts at 0 and increments by 1 for every operation. Each wait blocks until the signal reaches the sequence number of the current operation, guaranteeing that stale values from a previous iteration cannot cause premature unblocking.

Flag-based protocol for CUDA graph capture. Zero-SM NCCL also supports CUDA graph capture and replay. This requires a different synchronization protocol from ordinary stream execution. The sequence-number scheme above assumes that the host can assign a fresh expected value whenever it enqueues a new operation. During CUDA graph capture, however, the stream operations are recorded once and then replayed multiple times. The wait condition recorded in the graph cannot be changed on each replay. For instance, if the graph captured a wait for signal value kk, and a previous replay left the signal location at value kk or larger, a later replay could pass the wait immediately without observing the new event.

To avoid this stale-signal problem, Zero-SM NCCL switches to replay-safe flag protocols on both the scale-up and scale-out paths when an operation is captured into a CUDA graph. Each signal location that a captured stream waits on becomes a graph-private flag that the producer sets and the consumer resets within the same replay, so every replay starts from an identical flag state and can proceed only after observing a fresh transition. This costs a few extra stream memory operations per signaled transfer for the resets and the acknowledgment. For simplicity, the remaining sections assume non-graph stream execution.

One-Sided Operations

Scale-Up ncclPutSignal

  1. Scale-Up ncclPutSignal: Within a scale-up domain, ncclPutSignal moves data with the CE and synchronizes with stream memory operations. When rank SS sends to rank RR, the main thread enqueues the following onto SS’s stream (Figure 3, Panel Ⓐ).

Visualization of the Zero-SM one-sided RMA protocols for ncclPutSignal and ncclWaitSignal

Figure 3. Visualization of the Zero-SM one-sided RMA protocols for ncclPutSignal and ncclWaitSignal. Panel Ⓐ shows the scale-up path, while Panel Ⓑ shows the scale-out path. Although Panel Ⓐ illustrates NVLink, the same protocol also applies to PCIe and NVSwitch. The bottom of Panel Ⓑ provides a timeline view of the operations executed on the CUDA streams, main threads, and proxy threads of the sender and receiver.

  1. A device-to-device copy from SS’s local buffer to RR’s symmetrically registered buffer, executed by the CE.

  1. A stream memory operation that writes the current sequence number into a device-resident staging word on SS.

  1. A second device-to-device copy that copies the staged sequence number to RR’s symmetrically registered signal location, notifying RR that the data has arrived.

Publishing the sequence number takes two operations because a stream memory operation cannot target memory on a remote GPU, leaving the CE as the only SM-free path into a peer’s memory. CE copies, as currently exposed by CUDA, only move bytes and offer no atomic operation, so a copy cannot increment the destination, and the value must already sit in device memory to serve as a copy source. The host-side counter cannot be that source either, as it lives in pageable memory and copying from it would force an implicit CPU–GPU synchronization, so a stream memory operation stages the value locally as an immediate operand.

On the receiving rank RR, calling ncclWaitSignal blocks RR’s stream until the signal reaches the expected sequence number. Because the CE transfer and the signal write are both stream-ordered, the signal is guaranteed to be written only after the data transfer completes.

Scale-Out ncclPutSignal

  1. Scale-Out ncclPutSignal: Scale-out communication is more challenging because existing GPU-initiated network paths all rely on SM kernels to trigger, poll, or process network activity, which violates R1. A purely host-driven alternative is also insufficient. Host-based network APIs, such as IB verbs or MPI one-sided operations, execute on the CPU outside CUDA stream scheduling. Used directly, they would break R2 since the CPU may start a network transfer before the GPU has finished producing the send buffer, or the GPU may consume the receive buffer before the network transfer has completed.

Zero-SM NCCL therefore coordinates the main thread with NCCL’s CPU-side proxy infrastructure, using an RMA-specific proxy progress thread to drive one-sided network operations. The main thread handles the API call, creates a descriptor for the network operation, and enqueues stream memory operations that make the descriptor visible to the proxy only after prior CUDA work completes and block later CUDA work until the transfer is done. The proxy thread then polls these descriptors, drives the actual network operation, and reports completion through the stream-visible flag.

When the sender rank invokes ncclPutSignal, the following sequence occurs (Figure 3, Panel Ⓑ).

  1. The main thread creates a descriptor for the network operation, holding the source buffer, the destination window, and a monotonically increasing sequence number. It then appends the descriptor to a CPU-memory queue consumed by the proxy thread. The descriptor is associated with ready and done control flags, which reside in CPU-accessible memory. Depending on platform support, NCCL allocates them either as GDRCopy-mapped GPU memory or as CUDA host memory.

  1. The main thread enqueues cuStreamWriteValue64 (ready, seq) on the stream. This write completes only after all preceding stream operations have finished, so when ready ≥\geq seq, the proxy knows the source buffer contains valid data.

  1. The main thread enqueues cuStreamWaitValue64 (done, seq), which blocks the stream until the proxy thread

signals completion. The main thread then returns immediately.

  1. The proxy thread polls both the descriptor queue and the ready flag. Once a descriptor is available and ready, it initiates the RDMA transfer via the network plugin API (iput/iput_signal).

  1. The proxy thread polls for data transfer completion using the plugin’s test function. Upon completion, it writes the operation’s sequence number into the done flag, unblocking the cuStreamWaitValue64.

Signal update via network plugin backends. The remote signal update is exposed through NCCL’s network plugin interface as an ordered put-with-signal operation (iput_signal). The exact mechanism is backend-specific. An IB backend may implement the signal as an RDMA atomic (e.g., fetch-and-add) after the payload write, while other plugins can provide the same ordering semantics using their native mechanisms, including non-IB backends such as AWS Elastic Fabric Adapter (EFA). Zero-SM NCCL therefore does not depend on a particular NIC primitive. Instead, it relies on the plugin to make the signal visible after the payload transfer is complete. On the receiver, ncclWaitSignal enqueues a cuStreamWaitValue64 on that signal.

Zero-SM Collectives

We describe AllGather in detail, as it is the collective most commonly used in the training workloads we target. The other zero-SM collectives are built from the same primitives and are summarized in §III-D3. Their full implementation is available in the NCCL source code [1].

Scale-Up CE AllGather

  1. Scale-Up CE AllGather: In an NN-GPU AllGather, each GPU contributes one chunk and receives all N−1N-1 chunks from its peers. The protocol has three logical phases.

Phase 1: Barrier for buffer readiness. Before any GPU can write into a peer’s output buffer, all GPUs must have posted their buffers as ready. Zero-SM NCCL implements this barrier on symmetrically registered ready flags. Each rank publishes its ready flag to its peers with CE copies, then waits with stream memory operations until the flags from the other N−1N-1 ranks have arrived. This ensures that every destination buffer is safe to write before the CE data transfers begin.

Phase 2: Data broadcast. Once the readiness barrier is satisfied, each GPU ii enqueues CE transfers to copy its local chunk into the corresponding slot of every peer’s output buffer.

Phase 3: Barrier for global completion. After issuing the CE transfers, Zero-SM NCCL performs a second stream-memory barrier on completion flags. This ensures that every rank has finished its transfers before any CUDA stream proceeds to subsequent work that may consume the AllGather result.

Optimizations. A naive implementation of the CE path would issue many host-side CUDA calls, which can be a significant overhead for small and medium messages. Zero-SM NCCL therefore applies the following techniques.

  • Batching CE and stream-memory operations. Zero-SM NCCL batches both data movement and synchronization to reduce host-side overhead. For data movement, a collective may issue many pairwise copies, such as N−1N-1 remote copies. When supported, Zero-SM NCCL submits these copies through a single cudaMemcpyBatchAsync call. Similarly, the ready and completion barriers are grouped into batched stream-memory operations using cuStreamBatchMemOp. For large batches, Zero-SM NCCL inserts intermediate intra-batch synchronization after a configurable number of CE transfers to limit the amount of unsynchronized in-flight data.

  • Multicast. When NVLS multicast is available within the local scale-up domain, Zero-SM NCCL uses it for both synchronization and data movement, as NCCL’s low-latency symmetric kernels do [7]. A single CE copy to a multicast address is reliably delivered by the NVLink fabric to every GPU in the local team, replacing N−1N-1 unicast copies. When it carries data, the sender’s own chunk also needs no separate self-copy. Multicast delivers to the sender as well, which lowers the effective peak bandwidth, so the tuner keeps data on multicast only below a size threshold and falls back to unicast copies for larger transfers, which hide tail latency better. Synchronization always uses multicast when available, as a barrier publishes the same flag to every peer.

  • Ensuring the zero-SM guarantee for intra-GPU copies. Out-of-place collectives also copy each rank’s own chunk within its GPU. For such same-device copies, cudaMemcpyAsync defaults to SMs rather than the CE, since SMs saturate HBM better. Zero-SM NCCL uses cudaMemcpyFlagPreferOverlapWithCompute to force them onto the CE, trading local-copy speed for the zero-SM guarantee, a loss that is often not observable because each rank’s own chunk is only 1/N1/N of the output and its copy runs concurrently with the peer copies.

Hierarchical AllGather

  1. Hierarchical AllGather: For multi-domain topologies, Zero-SM NCCL uses a hierarchical algorithm that composes the proxy-based scale-out path with the CE-based scale-up path. The implementation treats GPUs with the same local rank across scale-up domains as a rail. Each rail exchanges data across scale-out connections using the RMA proxy, while CE transfers distribute the received data among GPUs within each scale-up domain. The pipeline consists of the following ordered stages, with the receive path operating at chunk granularity so that each remote chunk can be scattered locally as soon as its arrival signal is observed. Figure 4 illustrates the protocol on a two-node example.

Hierarchical Zero-SM AllGather protocol and timeline

Figure 4. Hierarchical Zero-SM AllGather with two nodes and two GPUs per node, where each rank contributes two chunks. Solid blocks denote chunks already present in a GPU’s receive buffer, while dashed blocks denote empty buffer slots. The top half shows data movement across scale-up and scale-out domains, and the bottom half shows the timeline for rank 0’s CUDA stream, CE, and proxy thread. 1 – 5 link the protocol steps to §III-D2.

  1. Cross-domain rail synchronization. Before starting data movement, each rail performs an entry barrier with signal-only RMA operations. The proxy builds a group of zero-byte put-with-signal operations to the matching local-rank peers in the other scale-up domains and waits until the corresponding inbound signals have arrived.

  1. Grouped scale-out puts. Each rank splits its per-rank slice into chunks and builds a single RMA PutGroup descriptor containing all scale-out put-with-signal operations for the remote domains. A stream-memory operation submits the group to the proxy.

  1. Local readiness barrier and self-broadcast. While the proxy progresses the network puts, the CUDA stream performs

a scale-up CE barrier to ensure local receive buffers are ready. Each rank then uses batched CE copies to broadcast its own local chunk into the corresponding receive-buffer slot on the other GPUs in the same scale-up domain.

4 Pipelined receive-and-scatter. For each remote domain and each chunk, the stream waits for the proxy signal indicating that the chunk has arrived from the corresponding rail peer. As soon as a chunk is available, NCCL issues batched CE copies to scatter that chunk from the rail GPU’s receive buffer to the other local GPUs. This overlaps scale-out reception with scale-up distribution instead of waiting for the entire network exchange to finish. Chunks have a fixed maximum size, which trades overlap between network arrival and local scatter against per-chunk synchronization cost.

5 Outgoing completion and final barrier. After all receive-side chunk scatters have been issued, the stream waits on the PutGroup completion operations to ensure all outgoing network puts have completed. A final scale-up CE barrier then prevents later CUDA work from consuming the AllGather result until all local distribution has finished.

Rail schedules. The steps above send each chunk directly to every remote domain on the rail. For large messages and many nodes, pipelined and ring rail schedules perform better, and a tuner selects among the three by node count and per-rank message size (§IV-B2). The pipelined schedule and the tuner are not yet in a released NCCL version and will ship in a future release. We do not detail these schedules, as they follow the same zero-SM principles.

Other Collectives

3) Other Collectives: The other collectives reuse the AllGather structure (a readiness barrier, batched CE copies, and a completion barrier) and differ mainly in which chunks are copied where.

Performance Evaluation

We perform a comprehensive evaluation of Zero-SM NCCL in three steps. Microbenchmarks measure raw communication performance against NCCL’s SM-based kernels and against alternative libraries (§IV-B). An overlap benchmark measures how much each implementation slows the compute it runs

next to (§IV-C). Finally, end-to-end Megatron-LM training on FSDP and PP recipes shows when these effects turn into faster training, and when they do not (§IV-D).

Experimental Setup

Table 1 lists the two systems. They sit at opposite ends of the scale-up design space. On the GB200 NVL72 system, an entire 18-node rack forms one NVLink domain, so every experiment of up to 72 GPUs stays on the CE path. On the B200 NVL8 system, each node is its own NVLink domain, so every multi-node collective takes the hierarchical path of §III-D2. All experiments are based on NCCL 2.32.3, extended with some features that are not yet released, such as the pipelined rail schedule (§III-D2). The patches, run scripts, and recipes for every experiment are in our artifact repository https://github.com/spcl/zero-sm-artifact.

GB200 NVL72B200 NVL8
GPUs per node (SMs)GB200: 4 (148)B200: 8 (148)
NVLink domains18 nodes (72 GPUs)1 node (8 GPUs)
NVLinkNVLink5, 900 GB/s/dirNVLink5, 900 GB/s/dir
Scale-out NICs4× 400 Gb/s per node8× 400 Gb/s per node
CPUGrace (aarch64)2× Xeon Platinum 8570
Driver580.178.04580.167.08

Table 1. Platforms used in the evaluation.

Figure 6. AllGather bus bandwidth and latency. Colored bands show ±1 standard deviation over 10 trials.

Microbenchmarks

Setup. We use NCCL’s performance tests, nccl-tests [40], with 10 trials per data point, and sweep message sizes from 8 B (point-to-point) or 1 KiB (AllGather) to 4 GiB. Within NCCL, we compare three implementations, namely the legacy SM-based ring kernels, the symmetric-memory kernels [6], [7], and Zero-SM NCCL (NCCL_CTA_POLICY=ZERO), labeled CE in the results. NCCL’s default tuner selects the protocol (e.g., LL or Simple) and the symmetric algorithm (the unicast ST or multicast STMC kernel) for each implementation.

We also compare against three alternative libraries described in §II-D, each of which moves part of the communication off the SMs, namely VCCL [21] in both its default and SM-free point-to-point modes, NCCLX’s persistent AllGather (AllGatherP) [8], [41], which moves data with CEs, and MSCCL++ [20], whose port channels hand inter-node transfers to a CPU proxy while SM kernels move all intra-node data.

We tuned NCCLX and MSCCL++ for each GPU count and node layout. For NCCLX we select the best of its three AllGatherP algorithms, and for MSCCL++ we grid-search the thread-block count, threads, and protocol and pick the switch point between its LL and Simple protocols.

Point-to-Point Bandwidth and Latency

1) Point-to-Point Bandwidth and Latency: Figure 5 compares the zero-SM one-sided put with two-sided Send/Recv and with VCCL. Over NVLink, the CE-based put has a higher latency than the Send/Recv kernel (16.7 vs. 9.7 µs at 8 B). It overtakes the kernel from 64 KiB, and ends at 773 vs. 658 GB/s at 4 GiB. VCCL’s SM-free mode reaches a similar bulk bandwidth (774 GB/s) with a latency floor of about 25 µs. Across nodes, the proxy-driven put halves the latency of Send/Recv (15.9 vs. 31.4 µs at 8 B), VCCL’s SM-free mode is slightly faster still (13.0 µs), and all three saturate the 50 GB/s rail, while default VCCL reaches only 27 GB/s.

Point-to-point latency and bandwidth by message size

Figure 5. Point-to-point latency (top) and bandwidth (bottom) by message size on B200 NVL8, within a node over NVLink (left) and across two nodes over one 400 Gb/s rail (right). At bulk sizes, the zero-SM put matches or exceeds the SM-based Send/Recv. Bands show ±1 standard deviation over 10 trials.

AllGather Bandwidth and Latency

2) AllGather Bandwidth and Latency: We focus on AllGather (Fig. 6), the collective most commonly used in the training workloads we target.

Within one NVLink domain, CE is the fastest NCCL implementation for bulk transfers. At 4 GiB it reaches 714–746 GB/s, against 537–678 GB/s for symmetric and 613–698 GB/s for legacy. CE outperforms the SM kernels at bulk sizes because on Blackwell a CE issues wider NVLink transactions than SM-initiated copies. The price is a higher latency floor. Every CE collective is issued from the host and synchronized through stream memory operations, so a 1 KiB AllGather takes 25–29 µs on one node, against 12–18 µs for the SM kernels. For mid-sized messages the symmetric kernel therefore remains faster, and CE overtakes it only from 128–512 MiB. At 72 GPUs the CE floor grows to 52 µs, still less than half of legacy’s 118 µs, because a ring’s latency grows with the number of ranks.

Across NVLink domains (B200, 4 and 8 nodes), the network sets the limit. With the pipelined rail schedule, CE reaches 503 and 436 GB/s at 4 GiB, close to the NIC bound (517 and 450 GB/s). This bound exceeds a node’s 400 GB/s of NIC bandwidth because only the (N−1)/N(N-1)/N of the data from other nodes crosses the network, which bounds bus bandwidth at 400⋅NN−1⋅n−1n400 \cdot\frac{N}{N-1} \cdot\frac{n-1}{n} GB/s for NN nodes and nn GPUs. CE is thus on par with the symmetric GIN kernel (482 and 430 GB/s), while legacy saturates at 372 GB/s.

Alternative libraries. NCCLX’s CE-based AllGatherP matches CE at bulk sizes within one NVLink domain, but takes 60–239 µs per call and reaches only 387 and 339 GB/s across B200 nodes. MSCCL++ instead spends SMs. Its intra-node kernels use up to 35 CTAs of 1,024 threads, which gives it the lowest small-message latency on 4 GB200 GPUs and across B200 nodes and the highest bandwidth from 1 to 64 MiB on 4 GB200 GPUs. Above 32 MiB per rank, its NCCL-API layer switches to a slower full-mesh algorithm (it would normally defer to NCCL, which we disallow), and at 72 GPUs it produced no valid result beyond 16 MiB.

In summary, for raw performance, zero-SM communication matches or exceeds NCCL’s SM-based kernels and the alternative libraries at bulk sizes without using any SMs, while SM-based designs keep the edge for small and medium messages.

Overlap Benchmark

Setup. The overlap benchmark measures how much a concurrent collective operation slows the compute it overlaps, one GEMM at a time. We replay GEMM chains taken from production pretraining traces, with each GEMM’s exact shape, kernel, and grid, on one B200 NVL8 node:

  • Qwen3-30B-A3B: 100 MoE GEMMs in microscaling FP8 (MXFP8), overlapped with a 144 MiB/rank AllGather.

  • Llama-3-8B: 32 FP8 forward, data-gradient, and weight-gradient GEMM triplets (96 GEMMs), overlapped with a 26 MiB/rank AllGather.

Each GEMM launches several hundred thread blocks (640 and 512 for the Qwen and Llama kernels), far more than the 148 SMs of a B200. The GEMMs thus saturate the GPU, which makes the cost of SM-based communication as visible as possible. We replay whole chains rather than single GEMMs for two reasons. The first version of the benchmark paired one GEMM with one AllGather, but these GEMMs run for only tens of microseconds, far shorter than the AllGather, so launch jitter decided whether the two overlapped at all. A chain removes this dependence on alignment, since the AllGather always overlaps some of it. We also observed that a GEMM behaves differently depending on which phase of the collective it overlaps, and a chain exposes the start, steady state, and end of the collective in a single run.

The whole chain is enqueued first. An event recorded before a fixed GEMM releases the AllGather on a second stream. We time every GEMM with the CUDA Profiling Tools Interface (CUPTI) in a clean pass and in an overlap pass, pair the two by process, rank, trial, and chain position, and report the median slowdown over four runs of 8 ranks ×\times 30 trials. We compare four AllGather configurations, namely CE, legacy ring at its default 32 CTAs, and the symmetric kernel at its default 4 CTAs and forced to 32 CTAs.

Results. Figure 7 shows the slowdown of every GEMM in the chain. The SM kernels slow the GEMMs they overlap in proportion to the SMs they occupy. With the Qwen AllGather, legacy NCCL slows the GEMMs by a median of 32.9% and the symmetric kernel by 6.2% at its default 4 CTAs and 24.0% at 32 CTAs. More CTAs thus finish the AllGather sooner but hit each overlapped GEMM harder. CE slows them by only 3.9%, a residual that comes from contention for memory. With the smaller Llama AllGather, CE slows no GEMM by more than 0.2%, while legacy slows them by a median of 9.9%.

Figure 7. Per-GEMM slowdown along a compute chain with one concurrent AllGather. Each bar is one GEMM (median over paired observations).

The timelines also show the inherent limitations of such benchmarks. The slowdown of a GEMM depends not only on how many SMs the collective holds but on which phase of the collective it overlaps.

RecipeTypeGPUsParallelismMBS/GBSGASeq
Llama-405B-G*Dense64DP32 TP21/6428K
Llama-70B-GDense64DP64 TP11/12828K
Llama-70B-H‡Dense64DP32 TP21/6428K
Qwen3-32B-G1Dense64DP32 TP28/25614K
Qwen3-32B-G2Dense32DP16 TP24/6414K
Qwen3-32B-HDense64DP32 TP21/6424K
Nemotron-15B-G1Dense32DP32 TP14/25624K
Nemotron-15B-G2Dense32DP8 TP48/12824K
Nemotron-15B-HDense32DP8 TP42/3224K
Qwen3-235B-GMoE64DP64 TP1 EP21/64116K
Qwen3-Next-HHybrid64DP64 TP1 EP21/6414K
DeepSeek-V3-G†MoE32DP32 TP1 EP41/6424K
Llama-405B-GDense64DP2 TP4 PP8 VPP21/64328K
Llama-70BDense64DP2 TP4 PP4 VPP5 CP21/64328K
Qwen3-32BDense64DP4 TP8 PP2 VPP22/6484K
Nemotron-15BDense32DP4 TP4 PP2 VPP22/3244K
Qwen3-235BMoE64DP4 TP8 PP2 VPP2 EP81/6484K
Qwen3-NextHybrid64DP8 TP1 PP8 VPP2 EP81/6484K
DeepSeek-V3†MoE64DP4 TP1 PP8 VPP4 EP41/64164K

Table 2. END-TO-END MEGATRON-LM PRETRAINING RECIPES. G AND H MARK GPU- AND HOST-BOUND RECIPES. DP, TP, PP, VPP, CP, AND EP ARE THE DATA-, TENSOR-, PIPELINE-, VIRTUAL-PIPELINE-, CONTEXT-, AND EXPERT-PARALLEL SIZES, MBS AND GBS THE MICRO- AND GLOBAL BATCH SIZES, AND GA THE NUMBER OF GRADIENT-ACCUMULATION STEPS, GA=GBS/(MBS⋅DP)GA = GBS/(MBS \cdot DP). FSDP RECIPES HAVE PP=1.

*96-layer shape on B200 (126 layers on GB200). †16-expert proxy with MXFP8 experts and full-iteration CUDA graphs. ‡GPU-bound on B200. Qwen3-Next is a hybrid of linear and full attention with MoE.

collective it overlaps. The GEMM in which the symmetric AllGather starts is hit hardest, the steady-state phase costs far less, and different GEMM types in the same chain react differently. In real training, the alignment between collective phases and compute kernels shifts from one iteration to the next, and several collectives and kernel types overlap at once. A per-GEMM slowdown measured in isolation therefore cannot predict the end-to-end effect, which we measure next.

End-to-End Training

Integration into Megatron-LM. We integrate Zero-SM NCCL into Megatron-LM [4] starting with two communication patterns that exercise both of its collective and one-sided put interfaces, namely the parameter AllGathers of FSDP and the stage-to-stage transfers of PP. Other collectives, such as the AlltoAll of expert parallelism (EP), can likewise be replaced by their zero-SM counterparts. We focus on FSDP and PP as both already have upstream adoption in Megatron-LM.

For FSDP, Megatron-FSDP already allocates its parameter and gradient buffers from an ncclMemAlloc-backed pool when NCCL user buffers are enabled. We register this pool as a symmetric window, so the parameter AllGathers become eligible for the zero-SM path without any change to the FSDP schedule. Because FSDP issues AllGathers back to back, we extend NCCL to pack consecutive zero-SM AllGathers into one plan that shares a single synchronization envelope, amortizing the host issue and completion cost per collective.

For PP, we replace the stage-to-stage ncclSend and ncclRecv with PyTorch symmetric memory’s put_signal and wait_signal, which map to ncclPutSignal and ncclWaitSignal. Each stage writes into a ring of receive slots on its peer, sized by the pipeline warm-up depth, and the receiver consumes the symmetric buffer in place.

Recipes. Table II lists the 12 FSDP and 7 PP recipes, covering dense, MoE, and hybrid models from 15B to 405B parameters on 32 or 64 GPUs. They are a small subset of the possible configurations, chosen to span model types and parallel layouts, which is enough to show when zero-SM NCCL would be beneficial. We run FSDP and PP separately rather than combined, to isolate the effect of the zero-SM AllGather from that of one-sided puts. We classify each recipe as GPU-bound or host-bound, depending on whether the host runs far enough ahead that the GPU never waits for work to be launched.

Arms. In the FSDP recipes, only the AllGather implementation changes. It is the legacy ring, symmetric with unicast (ST) or multicast (STMC), or CE with unicast or multicast. On B200 we report only the multicast arms, because the choice between unicast and multicast had a negligible effect on step time on all recipes. The two TP4 Nemotron-15B recipes have no symmetric arm on B200. In this layout, with two data-parallel ranks per node, every AllGather of the symmetric arm still ran the legacy ring kernel, so we omit it. In the PP recipes, one-sided puts are compared with two-sided Send/Recv.

We did not run the alternative libraries of §IV-B as their low-SM paths are not drop-in replacements for Megatron-LM’s ncclAllGather calls. Integrating them would change more than the AllGather implementation, so a difference in step time could no longer be attributed to the AllGather alone.

Protocol. Differences between node allocations on these systems (1–5%) are non-negligible, so we never compare runs across allocations. Instead, arms are compared in paired blocks. A block runs one session per arm back to back on the same nodes and in rotated order, with warm-up iterations discarded. Each block yields a speedup Tbaseline/Tarm−1T_{\mathrm{baseline}}/T_{\mathrm{arm}}-1 from mean step times TT, and we report the mean and standard deviation over three blocks per recipe.

A host-side artifact. While profiling, we found a platform effect on GB200 that penalizes host-bound training in every arm. Host-side memory unmapping waits on GPU address-translation invalidations, and far longer when buffers are mapped to peer GPUs and NVLink traffic is in flight. We remove it with a small preloaded allocator shim that routes the Python interpreter’s memory arenas through the C library’s malloc and free. Freed arenas are then reused instead of unmapped, which removes these unmaps from the training loop. The shim is active in every arm on both systems, so it does not bias any comparison. It is a temporary workaround, and a more robust, permanent fix is under development.

Cost estimate. Step-time gains of a few percent translate into large savings over a full training run. We convert each speedup into dollars with CoreWeave’s on-demand pricing [5], $10.50 per GB200 GPU-hour and $8.60 per B200 GPU-hour. For a recipe on NN GPUs with price pp per GPU-hour, legacy step time TT, and GG tokens per step (GBS times sequence length), training costs pNT/GpNT/G per token. An arm with speedup ss saves the fraction 1−1/(1+s)1-1/(1+s) of that cost, which we scale to a pretraining run of 15T tokens, the budget of Llama 3 [42].

Note that this is not a full deployment cost model. Production runs use more GPUs and different layouts, and their costs depend on factors we do not model.

FSDP Results

Figure 8 shows the FSDP results. Against the legacy ring, CE speeds up the GPU-bound dense recipes on both systems, by up to 3.5% on GB200 and 1.9% on B200 (the one exception, Qwen3-32B-G2, is within noise), because the ring kernel holds its SMs for the entire duration of every parameter AllGather. Against the symmetric kernel, which uses only a few CTAs, the picture is mixed. CE wins clearly only on Llama-70B-G on GB200 and narrowly on Nemotron-15B-G1 and the MoE DeepSeek-V3-G, ties or wins narrowly on the other GPU-bound recipes, and loses on host-bound recipes and on Qwen3-235B-G. On Qwen3-235B-G (B200), CE loses because plan fusion does not yet cover the faster rail schedules of §III-D2. Combining these results with profiles of each recipe, we identify three conditions under which zero-SM collectives pay off, each of which matters only once the previous ones hold.

Figure 8. FSDP step-time speedup over the legacy ring AllGather (%, mean and ±1 standard deviation over paired blocks). Rows are grouped by their bottleneck. A corner mark and a bold value flag the best arm. The second line of each cell is the estimated saving over a 15T-token pretraining run.

  • 1 The step is GPU-bound, so the freed SMs outweigh the extra host work. Communication kernels slow the compute they overlap by occupying SMs, by 3–4% in our end-to-end profiles. CE removes this cost but adds host-side work per collective. Issuing one CE AllGather takes 26–41 μs on GB200, about 2–3× the 11–17 μs of the symmetric kernel, and 77–86 μs against 5–7 μs on B200. This work stays off the critical path only when the step is GPU-bound.

  • 2 Messages are large enough. CE has a higher fixed cost per collective than SM kernels, a latency floor of 25–29 μs against 12–18 μs on one node, and it overtakes the symmetric kernel in bandwidth only from 128–512 MiB (§IV-B2). Training shows the same threshold.

these per-transfer costs, which leads to two conditions.

  1. There is waiting to remove. The pipeline must be deep, with stages that drift apart, and spend much of the step in its bubble (PP>4PP > 4 and b≳10%b \gtrsim10\%). At b=30%b = 30\% the speedup is 8–15% on GB200. A two-stage pipeline has a single boundary and little stage-to-stage variance, so the one-sided cost is paid in full, and Qwen3-32B loses 3.9% on B200.

  1. Transfers are few and large, so that the fixed cost per put and the staging copy stay small compared with the waiting time removed. With deep interleaving (Llama-70B, b≈2%b \approx2\%, 322 transfers per iteration), this cost cancels the gain.

PP step-time speedup of one-sided puts over two-sided Send/Recv

Figure 9. PP step-time speedup of one-sided puts over two-sided Send/Recv (%, mean and ±1 standard deviation over paired blocks), for the PP recipes of Table II ordered by pipeline-bubble share bb. The second line of each cell is the estimated saving over a 15T-token training run.

On GB200, FSDP gathers of up to about 240 MiB tie or lose against the symmetric kernel, and 820 MiB wins.

1.85.99.99.911.130.430.4
GB200 NVL72−2.7±3.7-2.7 \pm3.7−0.4±5.2-0.4 \pm5.2+2.6±0.1+2.6 \pm0.1+1.7±0.2+1.7 \pm0.2+1.5±3.9+1.5 \pm3.9+8.2±0.6+8.2 \pm0.6+14.9±4.1+14.9 \pm4.1
−812k-812\text{k}−118k-118\text{k}322k322\text{k}1.6M1.6\text{M}92k92\text{k}1.8M1.8\text{M}2.6M2.6\text{M}
B200 NVL80.0±1.20.0 \pm1.2−3.9±0.8-3.9 \pm0.8+5.0±0.3+5.0 \pm0.3+3.0±0.5+3.0 \pm0.5−1.0±0.7-1.0 \pm0.7+2.5±0.3+2.5 \pm0.3+9.4±0.5+9.4 \pm0.5
−8k-8\text{k}−695k-695\text{k}1.3M1.3\text{M}2.6M2.6\text{M}−38k-38\text{k}302k302\text{k}840k840\text{k}
  1. The route matches the concurrent traffic. The choice between multicast and unicast depends on how the competing traffic is spread, for CE and the symmetric kernels alike. A multicast copy is a single stream that injects one copy but advances at the pace of its most congested destination. It therefore wins when links are idle or more evenly loaded, as under the expert all-to-all of MoE recipes. It loses when the load concentrates on a few peers, as under concurrent TP traffic on GB200, where congestion on one link stalls the whole AllGather while independent unicast copies proceed.

These conditions also show that the SMs a collective frees are only part of the story. The kernels interfere with other communication as much as with compute. Under concurrent all-to-all, a unicast CE AllGather on GB200 is delayed by 1.8 ms, six times more than its multicast counterpart, while on B200 all-to-all delays the legacy ring by up to 8.2 ms per AllGather and CE by at most 1.3 ms. Choosing a communication path therefore means accounting for its effect on both the compute and the other collectives it overlaps.

PP Results

Figure 9 shows that one-sided puts speed up every deep pipeline on both systems, by up to 14.9% (Qwen3-Next on GB200). Shallow or heavily interleaved pipelines gain nothing or lose, namely Llama-70B (b≈2%b \approx2\%) and the two-stage Qwen3-32B and Nemotron-15B.

The freed SMs play a minor role, speeding up compute by at most 1–2%. In this case, the gains come primarily from removing the rendezvous of Send/Recv. Pipeline stages rarely finish a micro-batch together, because of uneven layer splits, the embedding and loss on the end stages, and varying expert loads in MoE models. With Send/Recv, a fast stage waits for its slower neighbor at every boundary. A one-sided put writes into a preallocated slot and moves on, so only the receiver waits, and only when it needs the data. The gain therefore grows with the time spent in the pipeline bubble, measured by the bubble share b=(PP−1)/(GA⋅VPP+PP−1)b = (PP - 1)/(GA \cdot VPP + PP - 1).

These savings come at a cost. Every put is issued from the host and synchronized with stream memory operations, adding a fixed cost per transfer that a Send/Recv kernel avoids. The sender must also copy each activation or gradient into a registered staging buffer, since Megatron-LM allocates them in ordinary PyTorch memory. Whether one-sided puts pay off thus depends on how much waiting they remove relative to these per-transfer costs, which leads to two conditions.

Summary

When a workload meets the applicable conditions above, zero-SM NCCL delivers substantial gains. It is not, however, a silver bullet, and enabling it does not by itself make training faster. Real workloads are far more complex than microbenchmarks suggest. Whether zero-SM communication pays off depends not only on its raw performance, but also on how it interacts with everything it overlaps, from compute kernels to other collectives, and on how much host work it adds to the step. The resulting step-time differences range from a few percent on FSDP to nearly 15% on deep pipelines, and over a full pretraining run they amount to hundreds of thousands to millions of dollars. Mapping these effects required careful end-to-end measurement, about 2,800 GPU-hours for the reported FSDP and PP runs alone, across two contrasting systems and 19 recipes. We hope the conditions distilled from these measurements provide a reference for deciding when to enable zero-SM communication in NCCL, and for designing it into other libraries.

Discussion

Other workloads. Weight synchronization in asynchronous reinforcement learning (RL) [43, 44] is a prime fit, a bulk transfer that overlaps SM-bound generation. NCCL-M2N [45], which reshards weights from trainers to inference engines with one-sided transfers, could benefit greatly from Zero-SM NCCL in frameworks such as veRL [43], leaving every SM to generation during an update. Expert-weight prefetching [46], KV-cache transfer in disaggregated serving [47, 48], and ring attention [49] share this pattern, and attention-FFN disaggregation already drives its M-to-N transfers from the CPU to keep SMs free [50, 51].

Applicability beyond NVIDIA GPUs. The design needs only stream-ordered memory waits and writes, SM-free copy engines, and NIC access to registered GPU memory. AMD HIP, for instance, provides stream memory operations, peer access, and batched asynchronous copies [52, 53, 54].

Reductions via NVLink SHARP. Reductions such as AllReduce need arithmetic, but NVLS (§II-A) performs it inside the NVLink fabric and could make them zero-SM.

Limitations. Within a scale-up domain, Zero-SM NCCL has higher latency than SM kernels for small messages, since every operation is issued from the host and synchronized through stream memory operations (§IV-B2). Graph capture costs even more, because its replay-safe protocol needs extra signaling (§III-B). Better CUDA support, such as CE-based atomics for signaling, would reduce both costs. For scale-out synchronization, the proxy path relies on a CPU thread, which consumes host resources. The number of CEs on the GPU may also limit concurrent transfers with high fan-out.

Conclusion

Communication libraries pay for bandwidth with the GPU’s compute, taking more SMs with every faster interconnect. Zero-SM NCCL breaks this link, communicating within and across scale-up domains without a single SM kernel. In Megatron-LM, it speeds up FSDP by up to 3.5% by freeing SMs and PP by up to 14.9% through one-sided puts. These gains are not automatic. We spent about 2,800 GPU-hours of end-to-end training to learn when they hold, and we share the resulting guidelines so that others can adopt zero-SM communication without repeating this effort.

Acknowledgments

The authors thank Maximilian Heer for insightful discussions. The research was conducted as part of the Fast-TrackAI project at the Singapore-ETH Centre, which was established collaboratively between ETH Zurich and the National Research Foundation, Singapore. This research is supported by the National Research Foundation, Singapore (NRF), and the Ministry of Digital Development and Information (MDDI) under the AI Visiting Professorship (Award No. AIVP-2025-005). This work also received funding from the European Research Council (Project PSAP, No. 101002047). The authors used GPT-6 Sol [55] and Claude Opus 5.5 [56] to assist with light editing and proofreading. All content and ideas remain the original work of the authors.

References

  1. [1]NVIDIA Corporation, “NVIDIA Collective Communications Library (NCCL),” https://developer.nvidia.com/nccl, 2026, accessed Oct. 2026.
  2. [2]——, “NVSHMEM,” https://developer.nvidia.com/nvshmem, 2026, accessed Oct. 2026.
  3. [3]Y. Ma, S. Shen, T. Chen, A. Langer, J. Kraus, B. Glick, C. Belusar, J. Hammond, and T. Hoefler, “Demystifying NVSHMEM: A system-level analysis on symmetric memory and device-initiated operations in GPU communication,” arXiv preprint arXiv:2606.05951, 2026.
  4. [4]M. Shoeybi, M. Patwary, P. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-LM: Training multi-million parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053, 2019. [Online]. Available: https://arxiv.org/abs/1909.08053
  5. [5]CoreWeave, “CoreWeave cloud pricing,” https://www.coreweave.com/pricing, 2026, on-demand, North America: GB200 NVL72 $42.00/hr per 4-GPU instance; HGX B200 $68.80/hr per 8-GPU instance. Accessed Sep. 2026.
  6. [6]J. Bachan, K. Ouyang, M. Mubarak, T. Gillis, B. Chang, D. Bureddy, G. Congiu, K. Caton, K. Aubrey, and X. Li, “Enabling fast inference and resilient training with NCCL 2.27,” NVIDIA Technical Blog, Jul. 2025. [Online]. Available: https://developer.nvidia.com/blog/enabling-fast-inference-and-resilient-training-with-nccl-2-27/
  7. [7]S. Shen, A. Korzh, J. Bachan, T. Chen, A. Goel, L. Schneider, P. Kousha, Z. He, S. Jeaugey, K. Iskra, N. Chandawala, J. R. Hammond, and T. Hoefler, “Every microsecond matters: Achieving near speed-of-light latency in GPU collectives,” arXiv preprint arXiv:2607.16100, 2026. [Online]. Available: https://arxiv.org/abs/2607.16100
  8. [8]M. Si, P. Balaji, Y. Chen, C.-H. Chu, A. Gangidi, S. Hasan, S. Iyengar, D. Johnson, B. Liu, R. Ren, D. Shah, A. J. Shetty, G. Steinbrecher, Y. Wang, B. Wu, X. Xie, J. Yang, M. Yang, K. Yu, M. Yu, C. Zhao, W. Bland, D. Boyda, S. Gumadivelli, P. Kannan, C. Lumezanu, R. Miao, Z. Qu, V. Ramesh, M. Samoylov, J. Seidel, S. Sundaresan, F. Tian, Q. Tan, S. Zhang, Y. Zhao, S. Zheng, A. Zhu, and H. Zeng, “Collective communication for 100k+ GPUs,” arXiv preprint arXiv:2510.20171, 2026. [Online]. Available: https://arxiv.org/abs/2510.20171
  9. [9]NVIDIA Corporation, “NVIDIA NVLink and NVLink Switch,” 2026, fifth-generation NVLink: 1.8 TB/s per GPU. Accessed Oct. 2026. [Online]. Available: https://www.nvidia.com/en-us/data-center/nvlink/
  10. [10]——, “NVIDIA DGX B200 User Guide,” 2025, accessed: 2026-06-01. [Online]. Available: https://docs.nvidia.com/dgx/dgx-b200-user-guide/introduction-to-dgxb200.html
  11. [11]E. Tweg, N. Dey, I. Goldwasser, A. Larijani, and I. Ozery, “NVIDIA ConnectX-8 SuperNICs advance AI platform architecture with PCIe gen6 connectivity,” NVIDIA Technical Blog, May 2025. [Online]. Available: https://developer.nvidia.com/blog/nvidia-connectx-8-supernics-advance-ai-platform-architecture-with-pcie-gen6-connectivity/
  12. [12]NVIDIA Corporation, “NVIDIA announces Spectrum-X photonics, co-packaged optics networking switches to scale AI factories to millions of GPUs,” Mar. 2025, press release. [Online]. Available: https://investor.nvidia.com/news/press-release-details/2025/NVIDIA-Announces-Spectrum-X-Photonics-Co-Packaged-Optics-Networking-Switches-to-Scale-AI-Factories-to-Millions-of-GPUs/default.aspx
  13. [13]M. Andersch, G. Palmer, R. Krashinsky, N. Stam, V. Mehta, G. Brito, and S. Ramaswamy, “NVIDIA Hopper architecture in-depth,” NVIDIA Technical Blog, Mar. 2022. [Online]. Available: https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/
  14. [14]NVIDIA Corporation, “CUDA C++ Programming Guide, section 4.12: Asynchronous data copies (using the Tensor Memory Accelerator),” 2026, accessed Oct. 2026. [Online]. Available: https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/async-copies.html
  15. [15]——, “GPUDirect RDMA documentation,” https://docs.nvidia.com/cuda/gpudirect-rdma/, 2026, release 13.4; introduced with Kepler-class GPUs and CUDA 5.0.
  16. [16]Z. Hu, S. Shen, T. Bonato, S. Jeaugey, C. Alexander, E. Spada, J. Dinan, J. Hammond, and T. Hoefler, “Demystifying NCCL: An In-Depth Analysis of GPU Communication Protocols and Algorithms,” in 2025 IEEE Symposium on High-Performance Interconnects (HOTI). Los Alamitos, CA, USA: IEEE Computer Society, Aug. 2025, pp. 48–59. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/HOTI166940.2025.00024arxiv.org/abs/2507.04786
  17. [17]P. Markthub, J. Dinan, S. Potluri, and S. Howell, “Improving network performance of HPC systems using NVIDIA Magnum IO NVSHMEM and GPUDirect Async,” NVIDIA Technical Blog, Nov. 2022. [Online]. Available: https://developer.nvidia.com/blog/improving-network-performance-of-hpc-systems-using-nvidia-magnum-io-nvshmem-and-gpudirect-async/
  18. [18]S. Jeaugey, J. Bachan, P. Markthub, Z. He, S. Das, and F. Ghodsian, “Fusing communication and compute with new device API and copy engine collectives in NVIDIA NCCL 2.28,” NVIDIA Technical Blog, Nov. 2025. [Online]. Available: https://developer.nvidia.com/blog/fusing-communication-and-compute-with-new-device-api-and-copy-engine-collectives-in-nvidia-nccl-2-28/
  19. [19]A. Langer, S. Howell, S. Potluri, J. Dinan, and J. Kraus, “Dynamic symmetric heap allocation in nvshmem,” in OpenSHMEM and Related Technologies. OpenSHMEM in the Era of Exascale and Smart Networks, S. Poole, O. Hernandez, M. Baker, and T. Curtis, Eds. Cham: Springer International Publishing, 2022, pp. 187–198.DOI
  20. [20]C. Hwang, P. Cheng, R. Dathathri, A. Jangda, S. Maleki, M. Musuvathi, O. Saarikivi, A. Shah, Z. Yang, B. Li, C. Rocha, Q. Zhou, M. Ghazirsaeed, S. Anantharaman, and J. Jose, “MSCCL++: Rethinking GPU communication abstractions for AI inference,” in Proc. 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2026, pp. 1201–1215.DOI
  21. [21]M. Zhang, X. Hu, M. Zhang, Z. Chen, Y. Jia, Y. Zhang, D. Liu, Q. Chen, F. Jiao, J. Chen, H. Liu, A. Zeng, S. Duan, R. Gu, Y. Jing, B. Han, W. Chen, W. Xie, J. Hou, Y. Cheng, H. Zhang, B. Xu, M. Xu, and C. Hu, “An efficient, reliable and observable collective communication library in large-scale GPU training clusters,” arXiv preprint arXiv:2510.00991, 2025. [Online]. Available: https://arxiv.org/abs/2510.00991
  22. [22]Y. Fei, G. Zhao, H. Xu, J. Fang, J. Zhu, S. Xu, K. Huang, and Z. Yu, “HOCCL: Offloading collective communication from GPU cores to accelerate distributed training,” arXiv preprint arXiv:2609.34334, 2026.arxiv.org/abs/2609.34334
  23. [23]A. Agrawal, S. Aga, S. Pati, and M. Islam, “ConCCL: Optimizing ML concurrent computation and communication with GPU DMA engines,” in Proc. IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2025, pp. 1–11.
  24. [24]S. Pati, S. Aga, M. Islam, R. Quach, S. Kudchadker, and M. A. Ibrahim, “DMA-Latte: Expanding the reach of DMA offloads to latency-bound ML communication,” in Proc. IEEE/ACM International Symposium on Microarchitecture (MICRO), 2026, To appear; arXiv:2511.06605.arxiv.org/abs/2511.06605
  25. [25]S. Pal, S. Aga, S. Pati, M. Islam, and L. K. John, “Design space exploration of DMA based finer-grain compute communication overlap,” arXiv preprint arXiv:2512.10236, 2025.arxiv.org/abs/2512.10236
  26. [26]Y. Wang, H. He, L. Wright, L. Wehrstedt, T. Liu, and W. Liang, “[distributed w/ TorchTitan] introducing async tensor parallelism in PyTorch,” PyTorch Developer Forum, Sep. 2024. [Online]. Available: https://discuss.pytorch.org/t/distributed-w-torchitan-introducing-async-tensor-parallelism-in-pytorch/209487
  27. [27]L.-W. Chang, W. Bao, Q. Hou, C. Jiang, N. Zheng, Y. Zhong, X. Zhang, Z. Song, C. Yao, Z. Jiang, H. Lin, X. Jin, and X. Liu, “FLUX: Fast software-based communication overlap on GPUs through kernel fusion,” arXiv preprint arXiv:2406.06858, 2024. [Online]. Available: https://arxiv.org/abs/2406.06858
  28. [28]E. Agostini, D. Rossetti, and S. Potluri, “GPUDirect Async: Exploring GPU synchronous communication techniques for InfiniBand clusters,” Journal of Parallel and Distributed Computing, vol. 114, pp. 28–45, 2018.DOI
  29. [29]N. Namashivayam, K. Kandalla, T. White, N. Radcliffe, L. Kaplan, and M. Pagel, “Exploring GPU stream-aware message passing using triggered operations,” arXiv preprint arXiv:2208.04817, 2022.arxiv.org/abs/2208.04817
  30. [30]H. Zhou, K. Raffinetti, Y. Guo, and R. Thakur, “MPIX stream: An explicit solution to hybrid MPI+X programming,” in Proc. 29th European MPI Users’ Group Meeting (EuroMPI/USA), 2022, pp. 1–10.DOI
  31. [31]P. G. Bridges, A. Skjellum, E. D. Suggs, D. Schafer, and P. V. Bangalore, “Understanding GPU triggering APIs for MPI+X communication,” in Recent Advances in the Message Passing Interface (EuroMPI), 2024, pp. 39–55.arxiv.org/abs/2406.05594
  32. [32]S. Pati, S. Aga, M. Islam, N. Jayasena, and M. D. Sinclair, “T3: Transparent tracking & triggering for fine-grained overlap of compute & collectives,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS), 2024, pp. 1146–1164.arxiv.org/abs/2401.16677
  33. [33]S. Rashidi, M. Denton, S. Sridharan, S. Srinivasan, A. Suresh, J. Nie, and T. Krishna, “Enabling compute-communication overlap in distributed deep learning training platforms,” in Proc. ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), 2021, pp. 540–553.DOI
  34. [34]R. L. Graham, D. Bureddy, P. Lui, H. Rosenstock, G. Shainer, G. Bloch, D. Goldenberg, M. Dubman, S. Kotchubievsky, V. Kousnir, L. Levi, A. Margolin, T. Ronen, A. Shpiner, O. Wertheim, and E. Zahavi, “Scalable Hierarchical Aggregation Protocol (SHArP): A Hardware Architecture for Efficient Data Reduction,” in First International Workshop on Communication Optimizations in HPC, COMHPC@SC 2016, Salt Lake City, UT, USA, November 18, 2016, 2016, pp. 1–10. [Online]. Available: https://doi.org/10.1109/COMHPC.2016.006
  35. [35]S. Xu, M. Heddes, J. Peng, T. Savell, M. Tang, P. Ranjan, J. Benson, O. Dekel, S. Dighe, A. Kurpad, A. Levin, M. Mattina, G. Petre, C. Tang, Y. Yu, L. Zhang, and T. Hoefler, “Maia 2000: A software defined dataflow system for large-scale AI acceleration,” 2026. [Online]. Available: https://arxiv.org/abs/2608.24664
  36. [36]S. Zheng, J. Fang, X. Zheng, Q. Hou, W. Bao, N. Zheng, Z. Jiang, D. Wang, J. Ye, H. Lin, L.-W. Chang, and X. Liu, “TileLink: Generating efficient compute-communication overlapping kernels using tile-centric primitives,” in Proc. Machine Learning and Systems (MLSys), 2025.arxiv.org/abs/2503.20313
  37. [37]K. Hong, X. Li, M. Liu, Q. Mao, T. Wu, Z. Huang, L. Chen, Z. Wang, Y. Zhang, Z. Zhu, G. Dai, and Y. Wang, “Efficient and adaptable overlapping for computation and communication via signaling and reordering,” in Proceedings of the 21st European Conference on Computer Systems (EuroSys). ACM, 2026, pp. 1894–1911.DOI
  38. [38]S. Lee, J. Oh, S. Go, and D. Mahajan, “Characterizing compute-communication overlap in GPU-accelerated distributed deep learning: Performance and power implications,” in Proc. IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2025, pp. 353–355.arxiv.org/abs/2507.03114
  39. [39]NVIDIA Corporation, “CUDA Driver API: Stream memory operations,” 2026, CUDA Toolkit Documentation, v13.4. Accessed Oct. 2026. [Online]. Available: https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__MEMOP.html
  40. [40]——, “NCCL tests,” GitHub repository, 2026. [Online]. Available: https://github.com/NVIDIA/nccl-tests
  41. [41]Meta, “TorchComms: Experimental communications API for PyTorch (with the NCCLX backend),” GitHub repository, 2026, version 0.3.0. [Online]. Available: https://github.com/pytorch/torchcomms
  42. [42]A. Graffatoni, A. Dubey, A. Jauhri et al., “The Llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024. [Online]. Available: https://arxiv.org/abs/2407.21783
  43. [43]G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu, “HybridFlow: A flexible and efficient RLHF framework,” in Proceedings of the Twentieth European Conference on Computer Systems (EuroSys), 2025, pp. 1279–1297.DOI
  44. [44]B. Wu, S. Wang, Y. Tang, J. Ding, E. Helenowski, L. Tan, T. Xu, T. Gowda, Z. Chen, C. Zhu, X. Tang, Y. Qian, B. Zhu, and R. Hou, “LlamaRL: A distributed asynchronous reinforcement learning framework for efficient large-scale LLM training,” arXiv preprint arXiv:2505.24034, 2025.arxiv.org/abs/2505.24034
  45. [45]NVIDIA, “NCCL M2N: Cross-group GPU data movement,” https://github.com/NVIDIA/nccl/tree/master/contrib/nccl_m2n_m2n, 2026.
  46. [46]W. Li, J. Peng, Z. Jing, T. Zhang, Z. Long, X. Qiao, X. Chen, D. Yang, K. Duan, and J. Yang, “DWDP: Distributed weight data parallelism for high-performance LLM inference on NVL72,” arXiv preprint arXiv:2604.01621, 2026.arxiv.org/abs/2604.01621
  47. [47]P. Patel, E. Choukse, C. Zhang, A. Shah, Í. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative LLM inference using phase splitting,” in Proceedings of the 51st ACM/IEEE Annual International Symposium on Computer Architecture (ISCA), 2024, pp. 118–132.
  48. [48]Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, “DisServe: Disaggregating prefill and decoding for goodput-optimized large language model serving,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024, pp. 193–210. [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/zhong-yinminarxiv.org/abs/2401.09670
  49. [49]H. Liu, M. Zaharia, and P. Abbeel, “RingAttention with blockwise transformers for near-infinite context,” in The Twelfth International Conference on Learning Representations (ICLR), 2024. [Online]. Available: https://openreview.net/forum?id=WsRHpHH4s0arxiv.org/abs/2310.01889
  50. [50]R. Zhu, Z. Jiang, C. Jin, P. Wu, C. A. Stuardo, D. Wang, X. Zhang, H. Zhou, H. Wei, Y. Cheng, J. Xiao, X. Zhang, L. Liu, H. Lin, L.-W. Chang, J. Ye, X. Yu, X. Liu, X. Jin, and X. Liu, “MegaScale-Infer: Efficient mixture-of-experts model serving with disaggregated expert parallelism,” in Proceedings of the ACM SIGCOMM 2025 Conference, 2025, pp. 592–608.
  51. [51]StepFun, “Step-3 is large yet affordable: Model-system co-design for cost-effective decoding,” arXiv preprint arXiv:2507.19427, 2025.
  52. [52]Advanced Micro Devices, Inc., “HIP Runtime API: Stream Memory Operations,” 2026, ROCm Documentation, accessed June 16, 2026. [Online]. Available: https://rocm.docs.amd.com/projects/HIP/en/latest/reference/hip_runtime_api/modules/stream_memory_operations.html
  53. [53]——, “HIP Runtime API: Peer to Peer Device Memory Access,” 2026, ROCm Documentation, accessed June 16, 2026. [Online]. Available: https://rocm.docs.amd.com/projects/HIP/en/latest/reference/hip_runtime_api/modules/peer_to_peer_device_memory_access.html
  54. [54]——, “HIP Runtime API: Memory Management,” 2026, ROCm Documentation, accessed June 16, 2026. [Online]. Available: https://rocm.docs.amd.com/projects/HIP/en/latest/reference/hip_runtime_api/modules/memory_management.html
  55. [55]OpenAI, “Introducing GPT-6 Sol and Luna,” https://openai.com/index/introducing-gpt-6-sol-and-luna/, Sep. 2026, accessed Oct. 2026.
  56. [56]Anthropic, “System card: Claude Opus 5.5,” https://www-cdn.anthropic.com/fc1b44717c85dc68bcba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf, Sep. 2026, accessed Oct. 2026.

Paper details

vlnc:W_nkxyru29
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

GPT-6 Sol, Claude Opus 5.5

View statement
Looking for related papers…
Contents