betaFeedback

WorksComputer Science

Compiling Specialized GPU-Initiated Networking

Abstract

Machine learning systems specialize computation to the workload and hardware, but communication remains in generic libraries whose device code and transport decisions are not visible to high-level compilers. We present Obaleth, a compiler that makes GPU-initiated communication part of a kernel's intermediate representation, co-optimizes it alongside computation, and lowers it to specialized transport code. Obaleth plans completions around buffer lifetimes and optimizes memory ordering, synchronization, and batching across NVIDIA and AMD GPUs. On H100 GPUs, Obaleth sends small messages up to 3.3×\times faster, and on 256 GPUs runs all-to-all 1.25×\times and 2.3×\times faster than NVSHMEM and NCCL GIN, with much smaller memory and register use. In vLLM, it serves Qwen3-235B on 16 B200 GPUs and answers requests up to 1.29×\times faster than with DeepEP.

Introduction

Machine learning systems use specialized compilers to generate kernels for attention, GEMMs, normalization, and more, just-in-time compiled for tensor shapes, data types, and the GPU’s architecture and resources ([12, 6]). Tile-level languages are particularly prevalent, from the cross-vendor Triton ([48]) and TileLang ([50]) to NVIDIA’s CuTe DSL ([34]) and cuTile ((NVIDIA, 2025b)).

GPU communication, by contrast, lives in libraries of handwritten device code, such as NVSHMEM ([37]) and NCCL ([23]). Libraries offer some malleability through fine-grained GPU-initiated functions that can be inlined into computation kernels, but tile DSLs either link them as LLVM bitcode, as in Triton-distributed ([56]) and PyTorch symmetric memory ([44]), or call them from generated CUDA source, as in MPK ([13]) and DaCe’s multi-GPU backend ([8]). Either way, communication enters at the final lowering stage, without the DSL’s semantic information and outside its optimization passes.

Library device functions are written for the general case: they support any caller, thread geometry, and transport, often through run-time branches, and must reserve queue slots with atomics and order memory and track completions conservatively (Figure 1). While each step individually incurs small costs, together they accumulate: a minimal GPU path issues an 8-byte put in 0.7 μs0.7\,\mu\mathrm{s}, and libraries add up to 4.6 μs4.6\,\mu\mathrm{s} of issue time ([9]).

Figure 1. Posting a chain of commands. While a library pays a reservation, release, doorbell, and completion per call, Obaleth posts the chain with one of each.

Figure 2. Obaleth versus NCCL and NVSHMEM: all-to-all on H100 GPUs.

Higher up the stack, modern workloads like Mixture of Experts (MoE) and KV-cache transfer exhibit distinct communication patterns that cannot be easily expressed with generic point-to-point or collective calls. Expert parallelism (EP) for MoE alone has produced several competing libraries, including DeepEP ([55]), pplx-kernels ([28]), fabric-lib ([29]), UCCL-EP ([31]), NCCL EP ([17]), Hybrid-EP ((Yu et al., 2026)), and MORI ([5]). Similarly, KV-cache transfer has motivated NIXL [36] and Mooncake [45]. Each of these re-implements and tunes parts of the transport by hand, from work-queue construction to signaling, completions, and buffer reuse, for one communication pattern, a set of network interface cards (NICs), and the shapes it targets.

We present OBALETH, a compiler that makes GPU-initiated communication part of the kernel’s intermediate representation (IR), where workload information is visible. OBALETH represents communication as put, signal, and wait operations in the same IR as the computation and lowers them progressively into ordinary device code that builds work-queue entries, rings doorbells, and polls completions. Instead of packaging each call for the general case, OBALETH specializes to the workload, its inputs, and the network topology, and generates only the code the transport and NIC need, optimizing it alongside the computation (Figure 1). We implement OBALETH as MLIR dialects and passes in a plugin for Triton. Our contributions are as follows:

  • RDMA as IR. We define layered IRs, including Queue IR, a backend-agnostic schedule of submissions and completions, that represent GPU-initiated communication and lower it to mlx5, AWS EFA, host-proxy, and peer-memory backends with GPU and CPU submitters, on NVIDIA and AMD GPUs.

  • Communication optimizations. We show that program structure in the kernel IR enables optimizations that libraries cannot perform. OBALETH coalesces independent submissions and reorders their commands so that a batch pays for one NIC ordering point, ties completion and publication to buffer lifetimes, and omits atomics and memory ordering when safe.

  • Tiered adoption model. We offer two tiers of adoption: a generated drop-in device library for existing CUDA and CuTe DSL kernels, and custom Triton kernels that fuse communication with computation, trading integration effort for optimization scope. Adapters describe their communication patterns to the compiler.

OBALETH is faster than communication libraries and uses less memory. On H100 GPUs, it sends 8-byte messages at up to 3.3× NVSHMEM’s rate and 4.2× NCCL GIN’s. On 256 GPUs, its small-message all-to-all round is 1.25× faster than NVSHMEM and 2.3× faster than NCCL GIN (Figure 2). It also allocates only 22 MiB of device memory per rank where the libraries allocate 266–584 MiB. Serving Qwen3-235B [51] in vLLM [25] on 16 B200 GPUs, it lowers time per output token by 10–24% relative to DeepEP and answers requests 1.15–1.29× faster. OBALETH also has lower latency than the NVSHMEM-based communicators of Triton and CuTe DSL (1.3–1.5× in round trip). OBALETH is open source.

Background

GPU-initiated communication. Kernels can communicate and synchronize without returning control to the host [49], sending data as soon as it is produced, which suits latency-critical and high-message-rate workloads. Within NVLink domains, kernels use loads and stores on peer-mapped pointers. Across nodes, remote direct memory access (RDMA) is the default transport, over InfiniBand, RoCE, or AWS EFA hardware. GPUDirect RDMA (Polturi et al., 2013) lets NICs access GPU memory directly, and GPUDirect Async [1, 32] lets kernels trigger network operations themselves.

GPU-initiated RDMA. RDMA NICs expose queue pairs (QPs) of send and receive work queues. To issue an operation such as an RDMA WRITE, the submitter writes a work-queue entry (WQE) into the QP’s send queue (SQ). The shape of this WQE depends on the platform: on NVIDIA ConnectX (mlx5) hardware, a WQE carries either a data segment pointing to registered memory or a payload inlined within the WQE [33]. The submitter then publishes the new producer index in a doorbell record (dbrec) and rings the doorbell at a NIC register in a user access region (UAR) that is memory-mapped over PCIe. Once announced, the NIC fetches the WQEs, reads the payload through DMA (or uses the inlined payload, if available), performs the operations, and, if requested, writes a completion-queue entry (CQE) that the submitter then polls.

Other NICs follow the same flow with some differences in queue placement, ordering, and completion semantics. AWS EFA [46] keeps the SQ in NIC memory that submitters must write WQEs into with memory-mapped I/O (MMIO) stores, unlike in mlx5 where the NIC fetches them. Its Scalable Reliable Datagram (SRD) transport is unordered, unlike mlx5’s reliable-connected (RC) queues, and each of its completions retires only its own request.

Ordering. Correct submission requires WQE and payload stores to be visible to the NIC before the doorbell that announces them and, when one thread rings for WQEs that others wrote, to the ringing thread first. A system-scope fence that enforces this costs 3.7× the issue time of a device-scope one [9].

MLIR and tile languages. MLIR [26] lets compilers lower progressively through coexisting IRs (dialects) while reusing analysis and code generation. Triton, CuTe DSL, and cuTile build on it with dialects that operate on tiles of values. Triton starts from its target-independent IR (TTIR), lowers it to GPU IR with explicit layouts (TTGIR), then to LLVM IR, and finally to PTX (NVIDIA) or AMDGCN (AMD). OBALETH operates at the TTIR stage.

Motivation & Related Work

Specializing Communication

NVSHMEM and rocSHMEM implement the OpenSHMEM model, in which ranks allocate a symmetric heap and kernels issue one-sided operations. NCCL’s device API (GIN) [23, 19], and RCCL [3] instead offer one-sided operations on registered memory windows. Both are written to be compatible with arbitrary callers, which costs them in the code a kernel carries and in what each call does at run time. Libraries with several transports either define separate APIs, as NCCL does, or keep every transport’s code in the kernel, where it consumes registers even when dormant [9]. NVSHMEM offers a build mode without remote transports for this reason [40]. Environment variables and build options tune a library, but only within the implementation it ships.

SystemIn IRSpecializedInter-node
*SHMEM✗✗✓
NCCL/RCCL GIN✗✗✓
PyTorch SymmMem∼∼✓
DeepEP✗H✓
FLUX✗H∼
Comet✗H✗
ParallelKittens✗H✗
Iris∼✓✗
TileLink∼∼✗
Triton-distributed∼∼✓
Mosaic GPU✓✓✗
CoCoNet✓∼✓
Mercury✓✗✓
Obaleth✓✓✓

Table 1. GPU communication methods. In IR: in a compiler IR. Specialized by a compiler (✓) or by hand (H). ∼: partially.

Communication in Existing Tile Languages

Several works enable communication in fused kernels and tile-based DSLs (Table 1). Within peer-accessible memory, Iris [7], TileLink [57], and Pallas Mosaic GPU [21] express transfers as remote loads and stores, Tensor Memory Accelerator (TMA) copies, or copy-engine operations. ParallelKittens [47], FLUX [11], and Comet [54] write them by hand into fused kernels. These systems leave some room for compiler optimizations, but are limited to a single node. Across nodes, Triton-distributed [56] and PyTorch symmetric memory [44] link NVSHMEM or rocSHMEM bitcode into Triton kernels and call their device functions. While Triton-distributed’s dialect models synchronization and symmetric addressing, network operations remain opaque calls whose issue, ordering, batching, and completion the compiler cannot change. Inlining a library’s bitcode does not close this gap, as the inlined body still carries run-time transport dispatch and handling for arbitrary callers. Compilers that represent communication explicitly, such as CoCoNet (Jangda et al., 2022) and Mercury [18], schedule and fuse collectives but leave the transport to NCCL. Only on TPUs does Pallas expose remote DMA beyond peer memory in the IR [22]. To our knowledge, no compiler represents GPU-initiated network transport in its IR.

Workload-Specific Communication Libraries

Generic communication serves communication-heavy workloads poorly, forcing developers to write and tune their own transport code [24]. In MoE EP, each layer dispatches every token to the ranks that host its top-kk experts and combines the expert outputs back, creating a sparse, data-dependent pattern. A plain all-to-all must pad every peer’s buffer to a fixed capacity [27, 16] or first exchange counts with a host synchronization [42], permute tokens into per-peer buffers [17], and send a token once per expert unless it is deduplicated [15, 31]. DeepEP [55] illustrates the effort that specialized libraries invest instead. Its low-latency kernels map one RC queue to each local expert, and its high-throughput kernels deduplicate tokens per node and forward them over NVLink. Other EP libraries make different choices for portability, submission path, and topology. No single library wins everywhere, and each one has an optimal configuration of token counts, hidden sizes, expert counts, network topology, and hardware. Disaggregated inference [58, 41] repeats the story for KV-cache transfer [36, 45], and reinforcement-learning post-training for weight synchronization [29] (Ye et al. 2026).

Hand-written GPU RDMA does not port. Hand-written GPU RDMA code is tied to a NIC family and a deployment. On AMD GPUs, MORI and rocSHMEM each maintain separate device code for NVIDIA ConnectX, Broadcom Thor 2, and AMD Pollara NICs [10, 2, 4 c]. Even on the same hardware, the submission path depends on the deployment environment: the GPU can ring doorbells only if the NIC’s doorbell register is mapped into its address space, which the NVIDIA kernel module restricts to root unless module options permit it. Without it, NVSHMEM falls back to a host thread that forwards doorbells [37], and other libraries delegate submission to CPU proxy threads [19, 31].

Figure 3. Obaleth compilation pipeline. Optimization passes rewrite each IR level using the facts it exposes.

The Opportunity

Consider a fused MoE dispatch kernel that quantizes token blocks, sends them to the ranks hosting their experts, and then sends each peer a token count and a ready flag. With a device library, every write in this kernel is a complete posting sequence (Figure 1): an atomic reservation, a wait for earlier reservations, a fence, a doorbell, and later a completion poll. Libraries pay for each step on every call because they cannot see the following facts in the kernel IR: (i) What follows the call: the chunks, count, and flag sent to a peer form one chain, chains to different experts are independent, and together they can share one reservation, release, doorbell, completion, and NIC ordering point, with the count inside the WQE. (ii) Who else issues: one program sends to each peer, so reserving slots needs no atomic and announcing them requires no wait for other producers. (iii) How long buffers live: a staging slot is overwritten only every PP launches, so the sender needs a completion only that often. These facts are not specific to one workload, and they guide the optimizations of Obaleth.

Obaleth Overview

Obaleth compiles semantic GPU communication into transport code specialized alongside the rest of the kernel and decides how transfers are submitted, published, and completed. Figure 3 shows its compilation pipeline.

Programming Model

Endpoints and regions. Communication operations act on regions and travel over endpoints. An endpoint is a stream of operations from one peer to another and holds the submission and completion state. It is backed by a transport queue — an RC queue pair on mlx5, an SRD queue pair on EFA, a proxy ring, or a peer-memory mapping. Operations on different endpoints are unordered, and a rank may be served by several independent endpoints. Regions are local or remote buffers carrying the address and keys the transport needs. Regions are ordinary allocations and need not be symmetric.

Operations. Table 2 lists Obaleth’s core operations. put and put_inline are used to transfer data, the latter embedding small payloads in the WQE where supported, which saves the NIC a DMA read and needs no sender completion. A signal publishes an epoch that tells the receiver that data has arrived or that a buffer is free. Signals may join a completion group (begin_group/wait_group) that completes many submissions with one wait, and a wait may declare that its sources are reused only every PP launches (wait(t, iteration=i, reuse=P)). Transfers can be disabled with a false predicate without changing the operation sequence, so data-dependent communication keeps a static shape that the compiler can batch.

OperationDescription
endpoint(table, i)Endpoint ii of a table
put(e, dst, src, n)Write nn bytes from src
put_inline(e, dst, x)Write register value xx
signal(e, flag, v)Publish epoch vv to flag
wait(t)Sender completion of tt
wait_signal(flag, v)Wait until `flag \ge v$
drain(e)Complete all work on ee

Table 2. Core operations in obaleth.language (ee: endpoint, tt: token). Transfers also take a predicate, transfers and signals an after token, and wait may declare buffer reuse.

Semantics. Obaleth operations guarantee:

  • Ordering. Transfers and signals return a token, a compiler handle for a communication dependency. Passing it as after orders the new operation after it on the same endpoint. Operations not ordered by tokens are unordered.

  • Sender completion. wait(token) returns once the sources of token and of every operation ordered before it may be overwritten. It makes no guarantees to the receiver.

  • Receiver visibility. wait_signal(flag, v) returns once the local flag holds at least vv, and the transfers ordered before the signal that wrote vv are guaranteed to be visible.

The token also determines how a signal is published: a signal ordered after transfers is strong and carries their visibility, while a standalone signal, such as an acknowledgment, is weak. These guarantees hold on unordered transports such as EFA as well, and are deliberately minimal, as each leaves the compiler room to optimize. Since operations that no token orders are unordered, the compiler may merge independent chains and reorder their commands (§ 5.3), and since wait_signal waits for at least an epoch, it may skip epochs that a later one covers.

Transports. OBALETH takes the transport choice as a compile-time input and specializes the generated code to it. When every peer uses one transport, e.g., in an NVLink domain, the kernels contain only the code for it. With mixed transports, e.g., NVLink and RDMA, OBALETH partitions the communication into transport-homogeneous segments and compiles each one directly, so no Tier-2 operation dispatches on the transport at run time. The transport is similarly specialized to payload shapes and sizes through Triton’s just-in-time compilation.

Adoption Tiers

OBALETH offers two adoption tiers that differ in how much of the program the compiler sees and can therefore optimize.

Tier-1: Library. At this tier, the compiler sees one operation at a time, along with the peer topology. OBALETH generates an NVSHMEM-style device library of one-sided operations specialized for the selected transports and GPU. This tier plugs into existing kernels in CUDA, CuTe DSL, Triton, and other contexts that allow linking external device code (Appendix Listing 2). Isolated calls still contain only the code their transport needs. In mixed deployments, a call selects the transport by peer, in a branch that folds away for statically known peers. The library ships as LLVM bitcode, a relocatable archive, and a generated header.

Tier-2: Custom kernel. At this tier, users compose OBALETH operations in custom Triton kernels. Unlike at Tier-1, the compiler can now see communication together with the computation that produces and uses it. This includes constant sizes, pointer offsets, predicates, and transfer dependencies. Listing 1 shows a fused FP8 MoE dispatch kernel. The kernel contains information beyond simple communication operations, such as a static loop, data movement and transformations, and a quantization step, all of which are visible to OBALETH. From this, the compiler derives that the chunk transfers, the count, and the flag form one submission, that the staged count can be inlined in the work request, and that one thread owns each queue (§ 5.3).

While the portion of the code fed to the compiler uses Triton as the frontend, the generated code is not confined to it. OBALETH exports a compiled communication program in three forms: an inline fragment that can be used in another Triton kernel, fusing the communication into its computation; a device function with bitcode, relocatable objects, and a generated header, which CUDA kernels call

Listing 1. Tier-2: a fused FP8 MoE dispatch kernel in Triton.

@triton.jit
def dispatch(x, counts, endpoints, send, recv, recv_n,
             flag, epoch, RANK, BLOCK, CHUNK, CHUNKS,
             SLOTS):
    peer = tl.program_id(0)    # RANK, SLOTS: tl.constexpr
    ep = obaleth.endpoint(endpoints, peer)  # one issuer
    dst = obaleth.remote(recv, peer)
    n = tl.load(counts + peer)              # tokens for peer
    slot, t = epoch % SLOTS, None
    for c in tl.static_range(CHUNKS):       # unrolled
        q, s = quantize_fp8(load_rows(x, peer, c, n))
        src = send.at(offset(slot, rows(peer, c))
        store_chunk(src, q, s)               # staging
        t = obaleth.put(ep, dst.at(offset(0, RANK, c)), src,
                        CHUNK, predicate=c * BLOCK < n, after=t)
        cnt = send.at(counts.at(offset(slot, peer)), n)
        store_count(cnt, n)                  # staged count
        t = obaleth.put(ep, obaleth.remote(recv_n, peer)
                        .at(RANK), cnt, 4, after=t)  # -> inline
        s = obaleth.signal(ep, obaleth.remote(flag, peer)
                           .at(RANK), epoch, after=t)
        obaleth.wait(s, iteration=epoch, reuse=SLOTS)

like a Tier-1 operation; or a standalone kernel with launch metadata that can be launched in a stream.

Compiler Implementation

OBALETH consists of MLIR dialects and passes, a Python frontend with C++ bindings, and a runtime that creates and manages the endpoints and regions. The dialects and passes are loaded into Triton as a third-party plugin. The runtime bootstraps the transport with an all-gather of rank topology, queue-pair addresses, and region keys, provided by torch.distributed or a TCP store.

RDMA as IR

OBALETH IR represents communication at three levels inside a kernel, plus a dataflow level for framework patterns (Figure 3). Semantic operations (Table 2) act on endpoints, regions, and tokens that the frontend creates from Triton pointers and keys. They are lowered to Queue IR, which defines the schedule of the communication, and then to Triton IR, from which the hardware code is generated.

Queue IR. Queue IR is a backend-agnostic schedule of the commands that each endpoint executes. It defines post_put, post_inline, and post_signal that construct ordered commands, which a submit then announces. A single submission includes one queue reservation, one or more chains of commands (segments), each closed by its own signal, one announcement (i.e., a doorbell ring), and at most one completion. Commands within a segment execute in the order determined by the tokens, and submissions on one endpoint are announced in order. Attributes on submit record the passes’ plans (Appendix § 11, Listing 3).

Having a separate schedule IR has two benefits. First, transport schedules are optimized independently of each target’s descriptor format, ordering rules, and completion model. Passes query the transport’s capabilities — ordered delivery, remote atomics, inline payloads, aggregate completions — and switch their optimizations accordingly. NN optimizations on MM backends therefore do not need N×MN \times M implementations. Second, most optimizations are independent of the frontend and reachable through adapters.

Framework adapters. Integrations whose compute kernels live outside Triton, such as an inference engine’s MoE layer, can describe their computation and communication as a pattern: the logical regions, the transfers among them, and the region effects of the compute phases. At the dataflow level of the IR (Figure 3), passes remove unneeded transfers and staging copies, retarget outputs into final storage, and compute buffer reuse (Appendix § 11.§ 1). The transfers become Tier-2 kernels that pass through the usual pipeline, and the framework launches them.

Triton Integration

Obaleth is an out-of-tree Triton plugin whose module of builtins, obaleth.language, emits Obaleth operations (obaleth.put, . . .) into TTIR. Obaleth’s passes run after Triton’s target-independent optimizations (inlining, common-subexpression elimination (CSE), loop unrolling) and before conversion to TTGIR, where GPU layouts are assigned. Communication operations are scalar in Triton IR: one thread of the program publishes each submission, and Tier-1 functions issue from each calling thread. Triton then lays out the communication code, allocates registers, and compiles it to the GPU’s assembly together with the computation. The transport and planning decisions join Triton’s cache key, so each specialization is compiled as a separate kernel.

Optimizations

Obaleth’s optimizations rewrite the semantic operations and Queue IR. A round of communication costs time on both the GPU that issues it and on the NIC that executes it, and the passes below target both. We measure the contribution of each pass in § 7.1. The complete list of passes is provided in Appendix Table 4.

OptimizationRemovesRequiresTierWhere
Transport specializationRun-time transport dispatch, dormant codePeer transports (R)1, 2—
Doorbell scopeSystem-scope releaseQueue placement (R)1, 2—
Publication planningOrdering point of payload-free signalsTokens (D)1, 2—
Inline forwardingStaging store, source key, NIC DMA readSmall constant payload staged by scalar stores (D)2App. A
Submission batchingPer-command reservation, doorbell, CQEToken chain (D)1, 2§ 5.3
Submission coalescingPer-chain reservation, doorbell, CQEIndependent chains on one endpoint (D)2§ 5.3
Publication orderingPer-signal NIC ordering pointIndependent chains (D), ordered queue (R)2§ 5.3
Issuer analysisReservation atomic, wait for other producersInjective endpoint index (D)2§ 5.3
Producer orderingRedundant block barriersWrites since the last barrier (D)2App. A
Completion planningCQEs and polls between reusesReuse distance (K), queue depth (R)2§ 5.3
Epoch elisionOrdering points between completionsCompletion plan, loop (D)2§ 5.3
Queue reclamationWaits for a poll when the queue is fullKernel-predicated wait (K), single issuer (D)2App. A
Doorbell deferral{sup}∗`Doorbells between completionsCompletion plan, loop (D)2App. A
Shape specializationWork and traffic padded to the boundBound range, uniform live bound (K)FApp. A.1
Valid extentsTransfers of dead paddingValid count (K)2, FApp. A.1
Storage planningUnreached storage, staging copiesRegion effects (D), consumer reads (K)FApp. A.1
Dead communicationTransfers that no consumer readsConsumer graph (D)FApp. A.1
Endpoint fan-outOne queue per large chainByte bound (D), platform profile (R)FApp. A.1
Receive coalescingOne receiver program per laneLane plan (D)FApp. A.1

Table 4. Table 4. Obaleth’s optimization passes. Facts are derived from the IR (D), declared by the kernel or framework (K), or reported by the runtime before compilation (R). F marks Tier-2 kernels compiled from a framework pattern. {sup}∗`Off by default.

Submission batching. Batch formation walks each signal’s after chain back through the transfers it depends on and replaces the chain with one submit at the signal and one poll at the matching wait. Computation and the stores that produce later chunks may sit between the transfers, as in Listing 1, as long as the chain stays in one block and on one endpoint with nothing blocking or synchronizing in between. Disabled transfers stay in the batch as no-op

commands. On mlx5, a chain of NN commands thus costs one reservation, NN contiguous WQEs, one doorbell, and at most one CQE, instead of NN complete posting sequences.

Submission coalescing. Kernels often issue several independent chains on one endpoint, e.g., a dispatch program sends tokens and a flag to each expert hosted on its peer. Coalescing merges such submissions into one: the chains share one reservation, release, doorbell, and completion, while each keeping its own signal (Figure 4). Earlier chains are then announced later, which is legal because the chains are mutually unordered. With 16 chains per round, coalescing cuts the GPU’s issue time from 41.5 to 3.5 μs, but the total time drops only from 43.8 to 39.8 μs, as the bottleneck instead shifts to the NIC.

Optimization passes applied cumulatively to two independent chains on one endpoint

Figure 4. Obaleth’s optimization passes, described in § 5.3, applied cumulatively to two independent chains on one endpoint, removing per-call submissions, NIC holds, and atomics.

Publication ordering. Coalescing exposes the NIC-side cost of strong signals. Each signal is an ordering point: on mlx5, the NIC holds the queue at the signal until the payload ahead of it has landed. We measure this cost at about 2.3 μs for both remote atomics and fenced writes. A merged batch laid out chain-by-chain (P1S1…PKSK)(P_1S_1 \ldots P_KS_K) thus stalls KK times. Publication ordering rewrites such chains to have all payloads precede the signals (Figure 4). With this, every signal still follows its own payload, but the submission only stalls once. The optimization takes the 16-chain round time from 39.8 to 4.3 μs. The pass needs ordered queues; EFA instead retires the payload before each strong signal (§ 5.4). While libraries also batch, they do so only when asked: NCCL GIN aggregates requests when the caller flags every put but the last [38 c], and even then each put-with-signal places its ordering point behind its own payload.

Issuer analysis. When several threads may post to one queue, each must reserve its slots with an atomic and, after writing its WQEs, wait until every earlier reservation has been written before it rings the doorbell. A library cannot rule this case out, so every call pays for it. Obaleth posts each submission from one thread of its program, so it only needs to know whether two programs can select the same endpoint. Issuer analysis checks the index passed to each obaleth.endpoint. If all selections from a table use the same index, and that index is a one-to-one function of the program ID (Appendix § 11.§ 2), each endpoint has a single issuer. The reservation then becomes a plain add, and the wait is removed. NCCL offers the same fast path as an exclusive context that the programmer requests [38], while Obaleth derives it from the IR. Without the analysis, the one-chain stream is 1.6×1.6\times slower and the ping-pong 1.71.7 µs slower per round trip.

Completion planning. Completions ensure that a source buffer may be reused and that a queue slot may be overwritten. Libraries complete conservatively whenever the caller asks: NVSHMEM’s quiet polls every configured queue, used or not, which can double its latency [9]. Obaleth instead defers completions to where they are needed. In Listing 1, reuse=SLOTS declares how many iterations apart its sources are reused, and completion planning turns the wait into a poll every kk-th iteration (Appendix Figure 9), where kk is the largest power of two within the reuse distance, the number of iterations whose commands fit in the queue, and the transport’s preferred interval (16 on mlx5 and EFA). The submission requests a CQE only on the last command, which, on mlx5, retires the commands before it. On EFA, the poll counts completions.

Epoch elision. Each iteration still has one strong signal and thus one NIC ordering point. In a loop that completes every kk iterations, epoch elision publishes signals only on completing iterations and the last, turning others into no-ops of the same size. This is legal because wait_signal waits for at least an epoch, and a strong signal also makes earlier payloads visible. As receivers may see an epoch up to k−1k-1 iterations late, the pass skips kernels that wait on a peer.

Backend Lowering

Backends lower Queue IR to Triton loads, stores, atomics, arithmetic, and structured control flow, with inline PTX/AMDGCN where Triton lacks the required memory ordering. Descriptor formats, opcodes, ordering scopes, and completion formats are fixed at compile time; only queue addresses are read from the endpoint descriptor at run time (Appendix § 11.§ 2, Table 5). The queues the GPU builds for mlx5 are announced either GPU-direct, with the kernel writing the doorbell, or CPU-assisted, through a host-pinned cell that a host thread forwards (§ 3.3). The runtime probes this once per endpoint. The generated code is identical for both submitters, as only the doorbell address changes.

BackendSubmissionStrong signalCompletion
mlx5GPU writes WQEs; GPU or host ringsFenced write or remote atomicOne CQE per planned interval
EFAGPU writes WQEs to the NIC or a ring the host copiesRetire payload, then writeCount CQEs
ProxyGPU writes commands to a host ringHost-posted, per providerHost-published sequence
Peer memoryPeer loads and storesRelease exchangeNone

Table 5. How each backend realizes submission, strong publication, and completion.

mlx5. The emitter generates the RDMA flow described in § 2 directly in Triton IR. As every command’s size is a compile-time constant, each WQE sits at a constant offset in its reservation, even inline writes that span several 64-byte WQE blocks. A strong signal is published as a remote compare-and-swap, which the responder cannot pass writes ahead of [33], even when the payload is registered with relaxed ordering. It is a fenced write if every receiver registration is strictly ordered. A weak signal is a plain write.

EFA. SRD transport is reliable but unordered and has no remote atomics [46], so a signal could overtake its payload. To keep strong signals strong, Obaleth retires the payload before it posts the epoch, which costs an extra round trip and bounds how far sender completion can be deferred. Our AMD backend, which uses no remote atomics either, publishes the same way (§ 7.1).

Peer memory. Over peer memory like NVLink or CUDA IPC, a put becomes a copy of 8-byte loads and stores over all warps (Appendix § 11.§ 2), an inline payload a store from registers, and a signal a system-scope release exchange on the peer’s flag. When a signal follows the copy in the same submission, its release also publishes the copied tile, so the copy ends with a block barrier instead of a system fence.

Evaluation

Setup

Platforms. Appendix Table 6 lists hardware and software versions in detail. The H100 platform has four GPUs per node, connected by NVLink within a node and by 200 Gb/s InfiniBand across nodes. This platform does not let GPUs ring doorbells, so Obaleth and the libraries build queues and WQEs on the GPU and forward only the doorbell through a host thread (CPU-assisted, § 5.4). The B200 platform, used for the vLLM experiments and to confirm the library comparison with GPU doorbells, has eight GPUs per node, 400 Gb/s ConnectX-7 InfiniBand, and the GPU rings the doorbells (GPU-direct, § 5.4).

Baselines. We compare against NCCL’s host and device APIs (NCCL GIN, with GPU-side GDAKI and proxy backends), as well as NVSHMEM’s InfiniBand GPUDirect Async (IBGDA) and IBRC transports. Each library runs its best configuration for each experiment, e.g., fused put-with-signal and warp-cooperative operations, or a packaged collective. Obaleth runs as Tier-1 from CUDA and as Tier-2 Triton kernels. The EP experiments compare against DeepEP (Zhao et al., [55]) 2.5.0 in its default configuration, in the same vLLM build. On H100, we use NVSHMEM 3.4.5, as 3.8.0 is 2.1 µs slower per round trip there.

Methodology. In-kernel timings use %globaltimer, and host-side APIs use CUDA events. Unless noted, each point is the median of five interleaved runs of at least 1000 iterations after 200 warmup iterations, and we verify flags and payloads after timing. For the MoE experiments, both backends serve the same model and expert kernels, with CUDA graphs and dual-batch overlap. Requests arrive at a fixed rate and decode to their natural end of sequence. Before any timing, each model’s outputs are validated against an independent FP32 reference.

Comparison with NCCL and NVSHMEM

Obaleth has lower latency, a higher message rate, and, up to 16 KiB per peer, a higher all-to-all throughput than both libraries, the last also with GPU doorbells, with a fraction of their memory and registers.

Point-to-point. Figure 5a shows the point-to-point latency between two nodes with one GPU each. Tier-2 has the lowest round trip at every size, about 3 µs ahead of both libraries: 9.98 µs at 8 B, against 12.74 for NCCL GIN and 13.12 for NVSHMEM on H100 GPUs. Most of the gap comes from the reservation atomic and system-scope doorbell fence that Obaleth omits (§ 7.1).

Figure 5. Comparison with NCCL and NVSHMEM on H100 GPUs. (a–c) Point-to-point between two nodes: (b) each library at its best issue width and transport, (c) one issuing thread per CTA. (d, e) All-to-all bus bandwidth on 8 and 256 GPUs (4 GPUs per node). (f) Least device memory per rank of any configuration while connecting, without the benchmark’s buffers but with NVSHMEM’s symmetric heap (dashed: without it). (g) Registers per thread of the all-to-all’s RDMA kernel and the ping-pong kernel. (h) Obaleth from CuTe DSL and Triton over each DSL’s NVSHMEM (§ 7.3). Tier-1 uses a GPU-scope doorbell in (b) at 16 and 64 CTAs, (d), and (h), and a system-scope one elsewhere; (a) shows Tier-2 only.

The benefit of batching shows most in the message rate (Figure 5b). With each library at its best issue width, Tier-2 sends 8-byte messages on 16 thread blocks (CTAs), each with its own queue, at 2.8× NVSHMEM’s rate, 3.3× NVSHMEM’s rate and 4.2× NCCL GIN’s, because it rings one doorbell per warp window where the libraries ring one per put. It also reaches the line rate at 4 KiB with 4 CTAs, where NCCL GIN needs 16 (Figure 5c). The libraries lead only where there is little to batch or complete: on a single CTA, and in a pure stream without completions, where NVSHMEM posts from 128–256 threads per queue and Tier-2 from one (Appendix § 15).

All-to-all. Each GPU exchanges with three NVLink peers within its node and over RDMA with the rest. Up to 16 KiB per peer, Tier-2 leads in latency and throughput at every scale. At 256 B per peer, it is 1.32× faster than NVSHMEM’s packaged collective and 1.52× faster than NCCL GIN on 8 GPUs, and 1.25× and 2.3× on 256 (Figure 2), where its bus bandwidth at 16 KiB per peer is 1.6× NCCL GIN’s and 1.2× NVSHMEM’s. From 64 KiB per peer, the exchange is NIC-bound: the time to move the payload over the link outweighs the per-message cost, and the device APIs converge, within 4% of each other from 1 MiB (Figure 5e).

These leads hold with GPU doorbells: on B200, Tier-2 is 1.11–1.16× faster than NCCL GIN and 1.24–1.52× faster than NVSHMEM at 256 B on 16 and 32 GPUs (Appendix Figure 11). Part of the larger lead over NCCL GIN on H100 thus comes from its host-forwarded doorbells.

Resources. OBALETH’s runtime allocates only 4–22 MiB of device memory per rank on 8–256 GPUs, at least 12× less than either library, for its queues and endpoint descriptors (Figure 5f). NCCL allocates 266–426 MiB for its communicator and GIN contexts, and NVSHMEM 584 MiB, of which 512 MiB is the symmetric heap it maps on every rank, however little of it the application uses. OBALETH’s RDMA kernels, which carry one transport’s code for one call pattern, also need fewer registers (Figure 5g). The all-to-all uses 60 per thread, against 128 in library kernels, and the ping-pong uses 62 vs. 210–221. Tier-1’s generic ping-pong kernel uses 255.

Ablation of Compiler Passes

We switch each pass on and off in Tier-2 kernels (Figure 6) and, since one pass’s savings can stay hidden behind another’s bottleneck, report both a waterfall that adds passes in pipeline order and the slowdown when one pass is removed from the defaults (Appendix Figure 8).

Figure 6. Passes in Tier-2 kernels. (a) Streams of 1, 4, and 16 chains of a 64 B put and a signal, passes added in pipeline order. (b) Defaults over all switchable passes off. SnS_n: nn-chain stream; PP: ping-pong of eight 64 B puts, answered by a put or a signal each. A2A: All-to-all on 8 GPUs.

On a stream of chains, the defaults are 8.4× faster than no passes with one chain and 23.9× with 16. With one chain, waiting for a completion every round dominates, and completion planning gives 76% of the gain. With 16 chains, the NIC’s ordering points dominate, and coalescing with publication ordering gives 71% (removing coalescing, which also disables ordering, costs 5.8×). Passes also unmask one another: epoch elision’s waterfall step is only 7% while issuing on the GPU still dominates the round, but once issuer analysis and the GPU-scope doorbell remove that cost, the NIC’s ordering point is the bottleneck and removing elision costs 2.4×. In the ping-pong, issuer analysis and the GPU-scope doorbell give the whole 2.1× gain with a signal per put, and publication planning saves 0.7 μs per round trip.

In the all-to-all on 8 GPUs, the switchable passes add only 1.13–1.22×, as each peer receives one put and one signal per round and there is nothing to coalesce or elide. Most of Tier-2’s lead comes from the lowering that is always on: with every switchable pass off, it still beats NCCL GIN (11.5 against 14.7 μs at 256 B) and Tier-1 (15.9 μs), mainly by posting one batched put-and-signal submission per peer from straight-line code. The remaining Tier-1 gap is retirement: with one issuer per endpoint and a GPU-scope doorbell, Tier-1 waits for its quiet each round, taking 12.1 μs to Tier-2’s 9.2. Retiring every 4th or 32nd round cuts it to 8.1–9.6 μs, and Tier-2 retiring each round takes 13.5. A library sees no reuse distance and must retire when asked.

Other platforms. The passes act on Queue IR, so they run unchanged on other backends. On AMD MI355X GPUs with AMD Pollara RoCE NICs, our GPU-direct backend runs a pipelined stream 3.8–5.7× faster with the passes than without. Even through a host proxy, OBALETH has the lowest round trip up to 32 KiB there, and its all-to-all is 1.24–1.70× and 2.2–2.5× faster than rocSHMEM’s and RCCL’s on 8 and 16 GPUs (Appendix § 14). Our EFA backend is also implemented. We leave full AMD and EFA evaluations to future work.

MoE Expert Parallelism in vLLM

We integrate OBALETH into vLLM as an all-to-all backend for expert parallelism. The integration is a 384-line adapter that guides the compiler with the application configuration: token bound, hidden size, top-k, expert placement, data type, and how the expert kernels read their inputs. It calls the compiled dispatch and combine kernels, whose buffers, shape variants, and queues are derived by pattern passes (Appendix § 11.§ 1). Everything else stays with vLLM. The adapter states only facts vLLM already knows and, unlike with DeepEP, no communication kernel is written or tuned by hand for this workload.

On Qwen3-235B-A22B (BF16, EP16), Obaleth answers requests 1.15–1.29× faster than DeepEP (Table 3). The gain is largest in decode, where each step’s exchange is small and latency-bound, and the time per output token is 10–24% lower, matching the kernels’ 1.7–3.1× lead at one token per rank (§ 7.3). The tail of the time to first token is worse (1.4–2.0× at p95), and accuracy is similar. With the FP8 Qwen3-30B-A3B checkpoint on four GPUs of one node, Obaleth answers 1.21–1.48× faster, again with a higher time to first token. On long GovReport prompts (Huang et al., [20]), where prefill dominates, it answers 1.33× faster.

Requests per second124
Answer (s)5.66 / 7.316.50 / 8.298.10 / 9.29
Speedup1.29×1.28×1.15×
TTFT (ms)125 / 134127 / 140131 / 134
TTFT p95 (ms)376 / 192312 / 218311 / 201
TPOT (ms)21.9 / 28.725.9 / 33.233.4 / 37.3
Correct (of 128)123 / 124124 / 120120 / 122

Table 3. Qwen3-235B-A22B in vLLM on 16 B200 GPUs (EP16), 128 GSM8K (Cobbe et al., [14]) requests per rate. OB ALETH/DeepEP, medians unless noted. Answer: request latency; TTFT: time to first token; TPOT: time per output token.

The gains come from the protocol specialized to the shapes at hand. In decode, the compiled plan has a capacity of one token, one queue per peer, and four commands per peer and phase, where DeepEP keeps 128 progress channels and updates each, empty or not. Over three decode steps of Ling-1T (Ling Team et al., [30]) (EP16), Obaleth’s NICs therefore transmit 5.6× fewer packets, and its send and await kernels take 4.3× less GPU time than DeepEP’s dispatch and combine, which also fuse layout work.

The pattern passes show the computation guiding the communication. Because the pattern states how the expert kernels read their inputs, storage planning sizes each buffer to what they reach, which recovers 16 GiB per GPU, 45% more KV-cache capacity than without the pass and within 5% of DeepEP’s, at no cost in answer time. Shape specialization compiles each call for the live batch instead of the declared bound, and valid extents send only live rows, cutting the bytes sent by 58%. The other pattern passes change the kernels’ service time by less than 3%. These results use one EP configuration at rates that saturate neither engine; it is not the best deployment of these models, as two TP8 replicas of Qwen3-235B serve long reports with 2.4× Obaleth’s throughput. Obaleth’s dispatch packs in a separate kernel here; the fused form of Listing 1 runs at 0.78–0.93× its speed on 8 H100 GPUs in BF16 and 0.44–0.82× with FP8 quantization.

Communication from Other Kernel DSLs

We port our microbenchmarks to CuTe DSL and Triton and compare against the NVSHMEM each DSL ships with: NVSHMEM4Py (CuTe) and Triton-distributed (Zheng et al., [56]) (Figure 5h, Appendix § 16). Called from a DSL, the generated library keeps its lead: with a GPU-scope doorbell, Tier-1’s round trip from CuTe and Triton (10.8 μs) is lower than NCCL GIN’s and NVSHMEM’s even from CUDA in the same runs (12.7 and 13.0 μs). Tier-2 adds batching and leads Triton-distributed, at its faster NVSHMEM transport for each benchmark, by 1.4× in round trip, 2.3–3.3× in all-to-all, and 5.1–10× in message rate. Obaleth allocates 4 MiB per rank in every language, against 2.1 and 1.6 GiB for NVSHMEM4Py and Triton-distributed.

For one MoE layer’s FP8 dispatch and combine, the kernels compiled from the vLLM pattern (§ 7.2) beat DeepEP’s hand-tuned kernels at the small batches of decode: at one token per rank, by 1.7–2.2× on dispatch and 2.2–3.1× on combine. From 64 tokens on Qwen3-235B, DeepEP is faster by up to 1.7× (Figure 7). That cost lies in the compiled kernels, not the transport. Around cuTile pack and reduce kernels, Tier-2 beats NCCL’s all_to_all by up to 1.87× in dispatch and 2.43× in combine, and from 32 tokens this combine beats the compiled one by up to 1.5×.

Figure 7. Compiled MoE dispatch and combine over DeepEP on H100, one FP8 layer, median of five runs (times: Appendix Figure 13).

Conclusion

Obaleth brings GPU-initiated communication into the kernel’s IR, where it is co-optimized with the computation. With semantic operations and a backend-agnostic Queue IR, the compiler batches and reorders submissions, plans completions around buffer lifetimes, and omits atomics and memory ordering, independently of NIC hardware. Obaleth plugs into CUDA, CuTe DSL, and Triton kernels, and into frameworks through pattern adapters. It outperforms vendor libraries with a fraction of their memory, speeds up end-to-end MoE inference, and runs on NVIDIA and AMD GPUs. Code is available on GitHub.

Acknowledgements

Authors from Koç University have received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 949587). We acknowledge the EuroHPC Joint Undertaking for awarding access to the MareNostrum5 supercomputer in Spain where the experiments reported in this paper were partially performed.

AI USAGE DISCLOSURE

The authors used coding assistants with Claude and GPT models to implement the compiler and associated parts described in this work, and to design its experiments and plot the results. The same models were used for light editing of the manuscript, i.e., typo fixes and grammar polish, styling of listings and tables, ensuring compliance with the venue guidelines.

Figure 8. Slowdown of Tier-2’s defaults with one pass removed, for the workloads of Figure 6.

Completion planning and epoch elision on one chain

Figure 9. Completion planning and epoch elision on one chain with wait(reuse=2). Symbols as in Figure 4.

CUDA kernel calling the generated device library

Figure 2. Tier-1: a CUDA kernel calling the generated device library.

Additional Compiler Optimizations

Table 4 is the complete list of optimization passes, each with the cost it removes, requirements, and tiers it applies to. Listing 3 shows the Queue IR that Listing 1 compiles to.

Inline forwarding. Kernels often stage small values, such as counts and offsets, in a buffer only to immediately send them. When a put’s source is written only by scalar stores just before it, and its size is a constant of at most 28 bytes (one 64-byte mlx5 WQE block), Obaleth deletes the stores and sends the values from registers. On mlx5, the payload is put inside the WQE, and over peer memory, it is sent as direct stores. The transfer then no longer needs a staging store, source key, or sender completion, and the NIC is spared a DMA read.

Producer ordering. When every thread of the block writes the payload and one thread announces it, the writers’ stores must be ordered before that thread’s release, a barrier libraries leave to the caller. Obaleth tracks the writes other threads may have made since the last barrier

Queue IR produced by the compiler

Figure 3. Queue IR the compiler produces for Listing 1 with CHUNKS = 2 staged token chunks per peer, after batching and planning. Types are omitted.

and places one barrier before the first submission that reads them. Submissions with no new writes in between share it, and inline payloads need no ordering.

Queue reclamation. A kernel may defer completion with a predicate, wait(t, predicate=p), to complete only after its last window, as NVSHMEM and NCCL callers do when they stream. Nothing then bounds the deferral by the queue’s depth, and a submission that finds the queue full would wait for a poll that never comes. When one program issues on each endpoint, the mlx5 lowering lets that submission retire completed submissions itself: deferred submissions request a CQE whether or not their poll is due, and the capacity check consumes CQEs in order until the batch fits. Planned completion is unaffected, as its interval already fits the queue. With reclamation, a Tier-2 kernel streams 8 B messages at 213 M msg/s on 16 queues and reaches the NIC’s 256 M msg/s on 64 (Appendix § 15).

Doorbell deferral. Doorbells can be deferred, for example, to the end of a loop. Commands in iterations between completions are written to the queue but announced only with the next completing iteration, so the GPU rings once per completion, rather than once per iteration. This pass improves performance when issue time dominates, but hurts it when the bottleneck is NIC-bound. With one chain per round, doorbell deferral reduces the round from 1.12 to 0.81 μ\mus, whereas with 16 chains, it slows it down from 2.41 to 3.06 μ\mus.

Programs from Framework Patterns

Frameworks know facts beyond the kernel IR, such as the maximum number of tokens a step can carry, what other compute kernels read, and the fact that a batch’s live size is the same on every rank. These can be stated by patterns and guide the passes in this section. The pattern produces a plan that Tier-2 kernels are compiled with.

Valid extents. A region may declare that only a prefix of run-time length is valid and that the rest is dead padding with region.valid(count, capacity, size). The compiler then clamps each put by both its requested bytes and the valid bytes that remain in its source so that batches only send live rows. Peer-memory copies are clamped the same way.

Shape specialization. From a declared range of bounds and the fact that the live bound is uniform across ranks, the compiler emits variants at power-of-two capacities and a selector keyed by the live bound. Each variant’s storage, endpoint plan, and completion interval follow from its own bound; in decode, a step that carries one token per rank compiles for a capacity of one. Without the uniformity fact, one variant at the largest bound remains.

Storage planning and dead communication. Storage planning sizes each region, through a dataflow pass, to the extent that its reads and writes reach. A receive buffer that an expert kernel reads through one record per source rank, for example, needs one row per record instead of one per routed copy. On Qwen3-235B, storage planning as a whole adds 45% KV-cache capacity (§ 7.2). An input that every reader masks by the same count may be bound to the framework’s own tensor instead of being copied, and an opaque output may be written directly into the donated buffer that a later copy would fill. Transfers in a generic record interface that no consumer reads, such as returning metadata the combine never uses, are removed with their storage.

Endpoint fan-out. A chain larger than one queue serves well is split across kk endpoints of the same peer, with kk taken from the chain’s byte bound and a per-platform profile measured once. Each lane sends its own row slice and signal, and receive coalescing joins the receiver’s waits for every lane in one program per peer. An unknown platform, or a decode-sized bound, gets one lane. Profiles are measured for B200 GPUs with ConnectX-7 NICs; our H100 platform has none, so its chains use one lane.

Queue IR and Backend Details

Queue IR attributes. submit records the passes’ plans as attributes: whether the completion is deferred, whether one thread owns the queue, whether a block barrier must order the payload’s producers before it, and whether the submission also announces commands built by other threads (Listing 3). Verifiers check that each submission stays on one endpoint and preserves the token order.

Issuer analysis. The analysis accepts an endpoint index that is a one-to-one function of the program ID along one grid axis, possibly shifted, scaled, or rotated as (pid+c) mod N(pid + c) \mathbin{\bmod} N, or linearized over two axes as lane⋅W+peerlane \cdot W + peer. The proof holds only for the grid extents it assumes, e.g., a rotation is a permutation only when the grid’s extent is NN, so the kernel asserts them at run time.

Host proxy. For NICs and platforms without GPU-side queues, the GPU writes compact 32-byte commands into a host-visible ring and publishes each batch with one release store. The command holds source, destination, length, keys, and opcode. A host thread posts the batch with one verbs or libfabric call and publishes a completion sequence that poll reads on the GPU.

Peer memory. Payloads of at least 256 bytes use a cooperative copy: a Triton tensor of 8-byte elements that layout assignment distributes over all warps of the program, copied in chunks of up to 16 KiB that shrink to fit a payload of constant size. Smaller or misaligned payloads use a per-thread loop, and detached Tier-1 functions use a one-warp variant. The copy is complete when its stores are issued, so wait and drain are no-ops.

Doorbell Forwarding

With CPU-assisted submission, the GPU still builds WQEs in its own memory and only delegates the doorbell store to a host thread. By default, every queue a process creates through DEVX shares one doorbell register and a lock for threads that ring it. This reduces the rate at which doorbells are forwarded, as host threads serialize on the register. Giving each endpoint its own doorbell page instead and replacing the lock with atomic writes allows threads to reach the NIC in parallel. With four submitting threads per GPU on 128 GPUs, this reduces the all-to-all round at 256 B per peer 1.8× from 45.0 to 24.5 μs. From 128 KiB per peer, it makes the round 4–7% faster, but slows down 6–14% at 16–64 KiB. This configuration is therefore not selected by default. We note that at 1–4 MiB per peer on 32 GPUs, DEVX-created queues, which NVSHMEM’s IBGDA and NCCL’s GDAKI also use, trail queues the kernel driver creates by 3–4%.

Platforms

Table 6 lists the hardware, submission path, and software versions of the three platforms and the sections that use each.

H100 (§ 7–§ 7.3, App. D, E)B200 (§ 7, § 7.2, App. D)MI355X (§ 7.1, App. C)
GPUs4 H100 per node, 1–64 nodes8 B200 per node, 1–4 nodes8 MI355X per node, 2 nodes
Intra-nodeNVLinkNVLinkxGMI
NetworkConnectX (mlx5), 200 Gb/s InfiniBandConnectX-7 (mlx5), 400 Gb/s InfiniBandAMD Pollara 400, 400 Gb/s RoCEv2
SubmissionCPU-assistedGPU-directHost proxy or GPU-direct (network), peer memory (xGMI)
Driver, runtimeDriver 595.91.07, CUDA 12.6Driver 610.57.04, CUDA 13.0ROCm 7.14.0, HIP 7.14
Triton3.8.03.8.03.8.0
BaselinesNCCL 2.31.2, NVSHMEM 3.4.5; DSLs: NVSHMEM4Py 0.4.1 over NVSHMEM 3.8.0, Triton-distributed with NVSHMEM 3.6.5NCCL 2.32.3, DeepEP 2.5.0, vLLM with the OBALETH adapterrocSHMEM 3.5.0, RCCL 2.30.4

Table 6. Evaluation platforms and software versions.

Preliminary AMD MI355X Results

We ported Obaleth to AMD MI355X GPUs with AMD Pollara RoCE NICs: Tier-2 kernels compile to AMDG­CN through Triton’s AMD backend, and Tier-1 is generated as a HIP device library. Through the host proxy (Table 5), Obaleth has the lowest round trip up to 32 KiB, both across nodes (28.3 against 36.6 μs for rocSHMEM at 8 B) and over xGMI. At 256 B per peer, its all-to-all across two nodes is 1.24–1.70× faster than rocSHMEM’s and 2.2–2.5× faster than RCCL’s on 8 and 16 GPUs (Figure 10a). As the host thread builds the work requests, few passes apply to the proxy, and we implemented no passes specific to AMD. Figure 10b shows the GPU-direct backend, in which the GPU submits to the NIC itself, with and without the passes (§ 7.1).

Figure 10. OBALETH on AMD MI355X GPUs in two nodes, one NIC each. (a) All-to-all with host-proxy submission. (b) A stream of independent 64 B chains whose buffers are reused every 16 rounds. Medians of five interleaved runs.

Additional Results

GPU doorbells. On B200, Tier-2 leads both libraries at every size up to 64 KiB per peer (Figure 11): by 1.06–1.23× over NCCL GIN and 1.05–1.71× over NVSHMEM.

Figure 11. All-to-all on B200 GPUs with GPU doorbells, 8 GPUs per node. One configuration per library (NCCL GIN with warp-issued fused puts; NVSHMEM’s collective). Medians of three runs.

Streaming. Without completions, NVSHMEM posts from 128–256 threads per queue and streams 257 M msg/s on 16 queues, the NIC’s limit, against 213 M msg/s for Tier-2, which writes every WQE of a batch from one thread (about 74 ns per put). On 64 queues, Tier-2 reaches the limit as well (256 M msg/s).

Additional Kernel DSL Results

Figure 12 and Figure 13 give the absolute values behind Figure 5h and 7, with every configuration of the MoE comparison of § 7.3.

Figure 12. OBALETH called from CuTe DSL and Triton kernels, against NVSHMEM as each language ships it (NVSHMEM4Py; Triton-distributed) and, in gray, CUDA libraries. Tier-1 uses a GPU-scope doorbell.

Figure 13. Dispatch and combine for one MoE layer on H100, Qwen3-235B-A22B (top) and Qwen3-30B-A3B (bottom). Median per-call time at the slowest rank, then the median of five interleaved runs. cuTile variants pack and reduce in cuTile kernels around OBALETH’s Tier-2 send kernel or NCCL’s all_to_all. Triton-distributed runs over NVSHMEM’s IBGDA transport (median of three runs).

Setup. The DSL kernels match the CUDA versions call for call. CuTe DSL kernels call Tier-1 as an external device library, or NVSHMEM through its CuTe bindings (NVSHMEM4Py). Triton kernels link Tier-1 as device bitcode or are Tier-2 kernels, and the baseline is Triton-distributed, which links NVSHMEM’s bitcode into Triton kernels. Each library runs the best configuration it ships with, including warp-issued and fused put-with-signal operations and NVSHMEM’s collective where its bindings expose it.

Microbenchmarks. NVSHMEM’s CuTe bindings take 16.0 μs\mu\mathrm{s} per round trip, 3.0 μs\mu\mathrm{s} more than NVSHMEM from CUDA in the same runs. Tier-1’s message rate is below NVSHMEM’s at 16 CTAs, as it rings a doorbell per put, and 2.2–3.6× above it at 64 CTAs. Triton-distributed takes its faster transport per benchmark: IBRC for the round trip and all-to-all, IBGDA for the message rate. At 4 MiB per peer on 8 GPUs, both tiers reach 41 GB/s from both languages, against 11–19 GB/s for NVSHMEM; the drop is NVSHMEM’s own, as its per-peer kernels and its 3.8.0 collective drop the same way from CUDA. OBALETH allocates 4 MiB of device memory per rank in every language, against 2126 and 1614 MiB for NVSHMEM4Py and Triton-distributed.

MoE dispatch and combine. Both models are top-8 of 128 experts (hidden sizes 4096 and 2048) with FP8 in 128-element scale groups. Every configuration receives the same seeded routing, quantizes to FP8 inside its dispatch, and must return the same combined output. Besides OBALETH’s compiled kernels (§ 7.2) and DeepEP’s ElasticBuffer (version 2, over NCCL’s device API), we run Triton-distributed’s low-latency EP layer and cuTile kernels that pack and reduce around either Tier-2’s send kernel or NCCL’s all_to_all. From 64 tokens per rank on Qwen3-235B, DeepEP’s dispatch is nearly flat, while Obaleth’s NVLink send programs spill registers from eight tokens. In cuTile, Tier-2 beats NCCL by 1.051.05–1.87×1.87\times in dispatch from eight tokens per rank and by 1.111.11–2.43×2.43\times in combine at every size. From 32 tokens, the combine also beats the compiled one by up to 1.5×1.5\times, so the cost at large batches lies in the compiled kernels, not the transport. Triton-distributed over IBGDA (median of three runs) is 1.61.6–6.6×6.6\times slower than Obaleth’s compiled kernels in dispatch and 2.02.0–11×11\times in combine.

References

  1. [1]Agostini, E., Rossetti, D., and Potluri, S. GPUDirect Async: Exploring GPU Synchronous Communication Techniques for InfiniBand Clusters. Journal of Parallel and Distributed Computing, 114:28–45, 2018. doi: 10.1016/j.jpdc.2017.12.007.DOI
  2. [2]AMD. AMD Pensando Pollara 400 AI NIC, 2025. URL https://www.amd.com/en/products/network-interface-cards/pensando.html. Accessed 2026-03-17.
  3. [3]AMD. AMD ROCm Communication Collectives Library (RCCL) Documentation, 2026a. URL https://rocm.docs.amd.com/projects/rccl/en/docs-7.14.0/. Accessed 2026-05-17.
  4. [4]AMD. AMD ROCm OpenSHMEM (rocSHMEM) Documentation, 2026b. URL https://rocm.docs.amd.com/projects/rocSHMEM/en/docs-7.14.0/. Accessed 2026-07-20.
  5. [5]AMD. MORI: Modular RDMA Interface, 2026c. URL https://github.com/ROCm/mori. Accessed 2026-07-20.
  6. [6]Ansel, J., Yang, E., He, H., Gimelshein, N., Jain, A., Voznesensky, M., Bao, B., Bell, P., Berard, D., Burovski, E., Chauhan, G., Chourdia, A., Constable, W., Desmaison, A., DeVito, Z., Ellison, E., Feng, W., Gong, J., Gschwind, M., Hirsh, B., Huang, S., Kalambarkar, K., Kirsch, L., Lazos, M., Lezcano, M., Liang, Y., Liang, J., Lu, Y., Luk, C. K., Maher, B., Pan, Y., Puhrsch, C., Reso, M., Saroufim, M., Siraichi, M. Y., Suk, H., Zhang, S., Suo, M., Tillet, P., Zhao, X., Wang, E., Zhou, K., Zou, R., Wang, X., Mathews, A., Wen, W., Chanan, G., Wu, P., and Chintala, S. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS), pp. 929–947, 2024. doi: 10.1145/3620665.3640366.DOI
  7. [7]Awad, M., Osama, M., and Potter, B. Iris: First-Class Multi-GPU Programming Experience in Triton, 2025. URL https://arxiv.org/abs/2511.12500.
  8. [8]Baydamirli, J., Ben-Nun, T., and Unat, D. Autonomous Execution for Multi-GPU Systems: Compiler Support. In SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1129–1140, 2024. doi: 10.1109/SCW63240.2024.00155.DOI
  9. [9]Baydamirli, J., Ismayilov, I., Oktay, K., and Unat, D. GPU-Initiated GPU Communication: Dissecting Down to the Bone, 2026. URL https://arxiv.org/abs/2610.01380.
  10. [10]Broadcom. BCM57608 400G Ethernet NIC Adapters (Thor 2): Product Brief, 2024. URL https://docs.broadcom.com/doc/957608-PB1. Accessed 2026-10-28.
  11. [11]Chang, L.-W., Bao, W., Hou, Q., Jiang, C., Zheng, N., Zhong, Y., Zhang, X., Song, Z., Yao, C., Jiang, Z., Lin, H., Jin, X., and Liu, X. FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion, 2024. URL https://arxiv.org/abs/2406.06858.
  12. [12]Chen, T., Moreau, T., Jiang, Z., Zheng, L., Yan, E., Shen, H., Cowan, M., Wang, L., Hu, Y., Ceze, L., Guestrin, C., and Krishnamurthy, A. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pp. 578–594, 2018.arxiv.org/abs/1802.04799
  13. [13]Cheng, X., Zhang, Z., Zhou, Y., Ji, J., Jiang, J., Zhao, Z., Xiao, Z., Ye, Z., Huang, Y., Lai, R., Jin, H., Hou, B., Wu, M., Dong, Y., Yip, A., Ye, Z., Wang, S., Yang, W., Miao, X., Chen, T., and Jia, Z. MPK: A Compiler and Runtime for Mega-Kernelizing Tensor Programs. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26), pp. 1909–1926, 2026.arxiv.org/abs/2512.22219
  14. [14]Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training Verifiers to Solve Math Word Problems, 2021. URL https://arxiv.org/abs/2110.14168.
  15. [15]DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L., Liang, J., Guo, J., Ni, J., Li, J., Wang, J., Chen, J., Chen, J., Yuan, J., Qiu, J., Li, J., Song, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, L., Zhang, L., Xu, L., Xia, L., Zhao, L., Wang, L., Zhang, L., Li, M., Wang, M., Zhang, M., Zhang, M., Tang, M., Li, M., Tian, N., Huang, P., Wang, P., Zhang, P., Wang, Q., Zhu, Q., Chen, Q., Du, Q., Chen, R. J., Jin, R. L., Ge, R., Zhang, R., Pan, R., Wang, R., Xu, R., Zhang, R., Chen, R., Li, S. S., Lu, S., Zhou, S., Chen, S., Wu, S., Ye, S., Ye, S., Ma, S., Wang, S., Zhou, Jangda, A., Huang, J., Liu, G., Sabet, A. H. N., Maleki, S., Miao, Y., Musuvathi, M., Mytkowicz, T., and Saarikivi, O. Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pp. 402–416, 2022. doi: 10.1145/3503222.3507778.
  16. [16]Fedus, W., Zoph, B., and Shazeer, N. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.arxiv.org/abs/2101.03961
  17. [17]Goldman, A., Boker, N., Sheraizin, M., Admoni, N., Polyakov, A., Bhattacharya, S., Yu, F., Sun, K., Theodorakis, G., Yin, H.-C., Gootzen, P.-J., Shafi, A., Ravid, A., Di Girolamo, S., Dinan, J., Li, X., Venkata, M. G., and Bloch, G. NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL, 2026. URL https://arxiv.org/abs/2603.13606.
  18. [18]Guan, Y., Qiang, X., Pan, Z., Johnson, D., Fang, Y., Zhou, K., Wang, Y., Li, W., Ding, Y., and Aziz, A. Mercury: Unlocking Multi-GPU Operator Optimization for LLMs via Remote Memory Scheduling. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP), pp. 1046–1061, 2025. doi: 10.1145/3731569.3764798.DOI
  19. [19]Hamidouche, K., Bachan, J., Markthub, P., Gootzen, P.-J., Agostini, E., Jeaugey, S., Shafi, A., Theodorakis, G., and Venkata, M. G. GPU-Initiated Networking for NCCL, 2025. URL https://arxiv.org/abs/2511.15076.
  20. [20]Huang, L., Cao, S., Parulian, N., Ji, H., and Wang, L. Efficient Attentions for Long Document Summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pp. 1419–1436, 2021. doi: 10.18653/v1/2021.naacl-main.112.DOI
  21. [21]JAX Authors. Pallas Mosaic GPU, 2026a. URL https://docs.jax.dev/en/latest/pallas/gpu/index.html. Accessed 2026-10-06.
  22. [22]JAX Authors. Distributed Computing in Pallas for TPUs, 2026b. URL https://docs.jax.dev/en/latest/pallas/tpu/distributed.html. Accessed 2026-10-06.
  23. [23]Jeaugey, S., Bachan, J., Markthub, P., He, Z., Das, S., and Ghodsian, F. Fusing Communication and Compute with New Device API and Copy Engine Collectives in NVIDIA NCCL 2.28, 2025. URL https://developer.nvidia.com/blog/fusing-communication-and-compute-with-new-device-api-and-copy-engine-collectives-in-nvidia-nccl-2-28/. Accessed 2026-07-20.
  24. [24]Jin, C., Jiang, Z., Bai, Z., Zhong, Z., Liu, J., Li, X., Zheng, N., Wang, X., Xie, C., Huang, Q., Heng, W., Ma, Y., Bao, W., Zheng, S., Zheng, X., Peng, Y., Lin, H., Liu, X., Jin, X., and Liu, X. MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production. In Proceedings of the 21st European Conference on Computer Systems (EuroSys), pp. 366–382, 2026. doi: 10.1145/3767295.3769325.DOI
  25. [25]Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), pp. 611–626, 2023. doi: 10.1145/3600006.3613165.DOI
  26. [26]Lattner, C., Amini, M., Bondhugula, U., Cohen, A., Davis, A., Pienaar, J., Riddle, R., Shpeisman, T., Vasilache, N., and Zinenko, O. MLIR: Scaling Compiler Infrastructure for Domain Specific Computation. In 2021 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), pp. 2–14, 2021. doi: 10.1109/CGO51591.2021.9370308.DOI
  27. [27]Lepikhin, D., Lee, H., Xu, Y., Chen, D., Fırat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. In International Conference on Learning Representations (ICLR), 2021.
  28. [28]Licker, N., Hu, K., Zaytsev, V., and Chen, L. pplx-kernels: Perplexity MoE Kernels, 2025. URL https://github.com/perplexityai/pplx-kernels. Accessed 2026-10-28.
  29. [29]Licker, N., Hu, K., Zaytsev, V., and Chen, L. fabric-lib: RDMA Point-to-Point Communication for LLM Systems. In Proceedings of the Ninth Conference on Machine Learning and Systems (MLSys), pp. 169–185, 2026. URL https://proceedings.mlsys.org/paper_files/paper/2026/file/89b4b6f55ae611c54065d6fc75075–Paper–Conference.pdf.arxiv.org/abs/2510.27656
  30. [30]Ling Team, Li, A., Liu, B., Hu, B., Li, B., Zeng, B., Ye, B., Tang, C., Tian, C., Huang, C., Zhang, C., Qian, C., Ju, C., Li, C., Tang, C., Fu, C., Ren, C., Wu, C., Zhang, C., Peng, C., Xu, D., Wang, D., Zhang, D., Jin, D., Zhu, D., Hu, D., Zhao, F., Wu, F., Zhu, F., Wang, G., Zhang, H., Zhao, H., Zhang, H., Wang, H., Qian, H., Yu, H., Zhang, H., Zhang, H., Luan, H., Dong, H., Li, H., Li, J., Liu, J., Zhu, J., Sha, J., Wei, J., Yang, J., Ma, J., Wu, J., Huang, J., Tian, J., Zhang, J., Sun, J., Tu, J., Liu, J., Xu, J., Zhou, J., Ou, J., Fang, J., Zhang, K., Hu, K., Shi, K., Tang, K., Chen, K., Mei, L., Liang, L., Xu, L., Zhang, L., Ju, L., Yuan, L., Zhong, L., Ma, L., Liu, L., Yu, L., Cai, L., Zhu, M., Li, M., Chen, M., Xue, M., Cai, M., Yin, M., Jiang, P., Zhao, P., Liu, P., Zhao, Q., Cui, Q., Huang, Q., Yang, Q., Yu, Q., Wei, S., Lian, S., Zheng, S., Song, S., Zhang, S., Zhang, S., Li, S., Liu, S., Guo, T., Zhao, T., Gu, W., Wu, W., Han, W., Fang, W., Wang, W., Shu, X., Shi, X., Lan, X., Zhang, X., Sun, X., Zhao, X., Lu, X., Xu, X., Wang, X., Wang, X., Yang, X., Yang, Y., Xiang, Y., Li, Y., Zhang, Y., Wang, Y., Li, Y., Guo, Y., Fu, Y., Wang, Y., Yang, Y., Yu, Y., Deng, Y., Zhang, Y., Yu, Y., Zhang, Y., He, Y., Gui, Z., Huan, Z., Wang, Z., Zhu, Z., Wang, Z., Zhang, Z., Wang, Z., Zeng, Z., Liu, Z., Xuan, Z., and Tang, Z. Every Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundation, 2025. URL https://arxiv.org/abs/2510.22115.
  31. [31]Mao, Z., Zhang, Y., Cui, C., Huang, Z., You, K., Chen, Z., Xu, Z., Gu, Z., Shenker, S., Raiciu, C., Zhou, Y., and Stoica, I. UCCL-EP: Portable Expert-Parallel Communication, 2025. URL https://arxiv.org/abs/2512.19849.
  32. [32]Markthub, P., Dinan, J., Potluri, S., and Howell, S. Improving Network Performance of HPC Systems Using NVIDIA Magnum IO NVSHMEM and GPUDirect Async, 2022. URL https://developer.nvidia.com/blog/improving-network-performance-of-hpc-systems-using-nvidia-magnum-io-nvshmem-and-gpudirect-async/. Accessed 2026-07-20.
  33. [33]NVIDIA. Mellanox Adapters Programmer’s Reference Manual (PRM), 2020. URL https://network.nvidia.com/files/doc-2020/ethernet-adapters-programming-manual.pdf. Accessed 2026-05-17.
  34. [34]NVIDIA. CUTLASS 4: CuTe DSL, 2025a. URL https://github.com/NVIDIA/cutlass. Accessed 2026-10-03.
  35. [36]NVIDIA. NIXL: NVIDIA Inference Xfer Library, 2025c. URL https://github.com/ai-dynamo/nixl. Accessed 2026-07-20.
  36. [37]NVIDIA. NVIDIA OpenSHMEM Library (NVSHMEM) Documentation, 2026a. URL https://docs.nvidia.com/nvshmem/api/index.html. Accessed 2026-07-20.
  37. [38]NVIDIA. NCCL Release 2.29.7: Release Notes, 2026b. URL https://docs.nvidia.com/deeplearning/nccl/archives/nccl_2297/release-notes/rel_2-29-7.html. Accessed 2026-10-27.
  38. [40]NVIDIA. NVSHMEM Build Configuration: NVSHMEM_BUILD_P2P_ONLY, 2026d. URL https://github.com/NVIDIA/nvshmem/blob/3bcdf3eee1f8287cc0a767e43b00c92117f8b857/cmake_config/NVSHMEMnv.cmake#L39. Accessed 2026-10-03.
  39. [41]Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., and Bianchini, R. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pp. 118–132, 2024. doi: 10.1109/ISCA59077.2024.00019.DOI
  40. [42]Perplexity, Licker, N., Chen, L., and Hoang, P. Efficient and Portable Mixture-of-Experts Communication, 2025. URL https://www.perplexity.ai/hub/blog/efficient-and-portable-mixture-of-experts-communication. Accessed 2026-07-20.
  41. [44]PyTorch. PyTorch Symmetric Memory, 2026. URL https://docs.pytorch.org/docs/2.13/symmetric_memory.html. Accessed 2026-10-03.
  42. [45]Qin, R., Li, Z., He, W., Cui, J., Ren, F., Zhang, M., Wu, Y., Zheng, W., and Xu, X. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), pp. 155–170, 2025.
  43. [46]Shalev, L., Ayoub, H., Bshara, N., and Sabbag, E. A Cloud-Optimized Transport Protocol for Elastic and Scalable HPC. IEEE Micro, 40(6):67–73, 2020. doi: 10.1109/MM.2020.3016891.DOI
  44. [47]Sul, S. H., Arora, S., Spector, B. F., and Ré, C. ParallelKittens: Systematic and Practical Simplification of Multi-GPU AI Kernels. In Proceedings of the Ninth Conference on Machine Learning and Systems (MLSys), pp. 1076–1089, 2026. URL https://proceedings.mlsys.org/paper_files/paper/2026/file/ff997469ac66cf893c4183efeb22212a-Paper-Conference.pdf.arxiv.org/abs/2511.13940
  45. [48]Tillet, P., Kung, H. T., and Cox, D. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages (MAPL), pp. 10–19, 2019. doi: 10.1145/3315508.3329973.DOI
  46. [49]Unat, D., Turimbetov, I., Issa, M. K. T., Saǧbili, D., Vella, F., De Sensi, D., and Ismayilov, I. The Landscape of GPU-Centric Communication. ACM Computing Surveys, 58(12):1–36, 2026. doi: 10.1145/3813799.DOI
  47. [50]Wang, L., Cheng, Y., Shi, Y., Mo, Z., Tang, Z., Xie, W., Wu, T., Ma, L., Xia, Y., Xue, J., Yang, F., and Yang, Z. TileLang: Bridge Programmability and Performance in Modern Neural Kernels. In International Conference on Learning Representations (ICLR), 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/76fb92288bf90360c527efb0d1c2aba6-Abstract-Conference.html.
  48. [51]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., Yang, K., Yu, L., Deng, L., Li, M., Xue, M., Li, M., Zhang, P., Wang, P., Zhu, Q., Men, R., Gao, R., Liu, S., Luo, S., Li, T., Tang, T., Yin, W., Ren, X., Wang, X., Zhang, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Zhang, Y., Wan, Y., Liu, Y., Wang, Z., Cui, Z., Zhang, Z., Zhou, Z., and Qiu, Z. Qwen3 Technical Report, 2025. URL https://arxiv.org/abs/2505.09388.
  49. [54]Zhang, S., Zheng, N., Lin, H., Jiang, Z., Bao, W., Jiang, C., Hou, Q., Cui, W., Zheng, S., Chang, L.-W., Chen, Q., and Liu, X. Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts. In Proceedings of the Eighth Conference on Machine Learning and Systems (MLSys), 2025.arxiv.org/abs/2502.19811
  50. [55]Zhao, C., Zhou, S., Zhang, L., Deng, C., Xu, Z., Liu, Y., Yu, K., Li, J., and Zhao, L. DeepEP: An Efficient Expert-Parallel Communication Library, 2025. URL https://github.com/deepseek-ai/DeepEP. Version 2.5.0, commit b5a33ee. Accessed 2026-10-30.
  51. [56]Zheng, S., Bao, W., Hou, Q., Zheng, X., Fang, J., Huang, C., Li, T., Duanmu, H., Chen, R., Xu, R., Guo, Y., Zheng, N., Jiang, Z., Di, X., Wang, D., Ye, J., Lin, H., Chang, L.-W., Lu, L., Liang, Y., Zhai, J., and Liu, X. Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler, 2025a. URL https://arxiv.org/abs/2504.19442.
  52. [57]Zheng, S., Fang, J., Zheng, X., Hou, Q., Bao, W., Zheng, N., Jiang, Z., Wang, D., Ye, J., Lin, H., Chang, L.-W., and Liu, X. TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitives. In Proceedings of the Eighth Conference on Machine Learning and Systems (MLSys), 2025b.arxiv.org/abs/2503.20313
  53. [58]Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang, H. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 193–210, 2024.arxiv.org/abs/2401.09670

Paper details

vlnc:W_hvb4sqz9
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude, GPT (multiple versions)

View statement
Looking for related papers…
Contents