WorksComputer Science
LiquidCP: Decoupling Attention Tasks from Communication and Kernel Boundaries in Context Parallelism
Abstract
Context parallelism (CP) partitions long sequences across GPUs for LLM training and inference. For variable-length training and full prefill, attention-balanced placement can leave token loads uneven, while ring execution repeats overhead at each computation and communication step. In stage-serialized decode CP (DCP), long-running tasks delay dependent work and leave substantial GPU hardware idle. We propose LiquidCP, which unifies training and inference through a common model of Transfer, Compute, and Reduction tasks with tile-level dependencies, decoupling attention tasks from communication and kernel boundaries. For training and full prefill, MegaRing incorporates BuddyRing placement to balance attention and token loads, and coalesces ready KV regions during forward attention to amortize overhead. For decoding and mixed batches, MegaDCP selects history-KV splits by jointly modeling the completion of attention and its dependent reductions. A shared persistent GPU-resident engine executes both designs. Compared with the fastest baseline per configuration on eight H100 GPUs, MegaRing achieves geometric-mean core-attention speedups of forward and backward, and accelerates a Transformer layer by –. MegaDCP achieves geometric-mean core-attention speedups of for decode-only and for mixed batches. In vLLM serving with KV prefix cache, it achieves an average time-to-first-token speedup of .
Introduction
Long-context training and inference impose substantial computation and memory demands on LLM systems. Context parallelism (CP) addresses these demands by partitioning sequences across GPUs along the token dimension, distributing attention computation and the storage of activations and KV caches [25, 28]. In training and full prefill, query and KV sequences are both long. Grouped-query attention (GQA) [1] uses fewer KV heads than query heads, favoring dataflows that keep queries local and exchange KV, typically around a ring [25, 45]. In decoding and chunked prefill over cached contexts, queries can be much shorter than their KV histories. Decode context parallelism (DCP) instead shards history KV and exchanges queries and partial attention states [2, 37].
Efficient CP requires an end-to-end view of load balance and computation–communication coordination. Placement determines attention work and token loads across GPUs, while task partitioning determines attention parallelism, per-task overhead, and reduction work. Communication and reduction dependencies further determine when tasks become executable and when their outputs become usable. Consequently, balancing aggregate workload alone does not ensure efficient execution: task boundaries and the progression of dependent work also matter.
Training and full prefill. Dynamic CP adapts placement and parallelism degrees to variable-length batches [4, 28]. FCP further combines computation- and memory-aware block placement with communication planning and stage coalescing [42]. Attention and tokenwise operations have different scaling characteristics, making complete-layer performance important when assessing placement: attention work grows approximately quadratically with sequence length, whereas many other operations scale with token count. On the GitHub dataset, Megatron Bridge CP reduces core-attention latency by 46.6% relative to a ring CP baseline using FlashAttention-3 (FA3) [32], while increasing the latency of the remaining Transformer-layer operations by 38.3%. Execution introduces another source of overhead. The conventional ring baselines evaluated here invoke attention separately for each KV communication block, repeating setup and partial-result merging across steps (Section 2.2).
Decoding. Evenly sharded history KV can still yield uneven task completion within each GPU: short queries in decode-only batches limit query parallelism, while heterogeneous histories produce tasks of unequal duration. In mixed batches that combine decoding and chunked prefill requests, long prefill queries can cause FA3’s batch-wide heuristic to leave decoding histories unsplit. The FA3-based DCP paths of state-of-the-art inference frameworks [31, 37] execute communication, attention, and reduction as separate dependent kernel stages, so a few long tasks can delay subsequent work. Additional KV splits expose parallelism but also add task overhead and reduction work. An effective policy must therefore consider attention together with its dependent reductions, while allowing completed portions of the workload to advance (Section 2.2).
In this paper, we present LiquidCP, which coordinates attention-task formation and partitioning with fine-grained fused execution for CP. Building on composable partial attention states [25, 41], we describe execution using Transfer, Compute, and Reduction tasks linked by tile-level dependencies (Section 3). This formulation exposes the trade-off between coalescing tasks to amortize overhead and splitting tasks to expose parallelism. LiquidCP develops policies for both regimes, accounting for input readiness and downstream reduction costs.
For training and full prefill, MegaRing combines structured placement with readiness-driven task formation (Section 4). Its placement algorithm, BuddyRing, jointly selects sequence CP degrees and locations within aligned, nested GPU groups. It searches for placements satisfying attention-work and token-load tolerances while prioritizing reduced fragmentation of short sequences. MegaRing executes groups of different sizes together in one launch. During forward attention, it retains fine-grained KV transfers while dynamically forming Compute tasks from ready, unassigned KV regions across communication blocks and data owners. Computation granularity thus adapts to runtime input availability within prescribed bounds, enabling early computation while amortizing per-task overhead.
For decoding and mixed batches, MegaDCP retains the existing history-KV placement and selects splits using a critical-wave model (§ 7). The model predicts the completion of history attention together with its dependent reductions, accounting for the additional work introduced by splitting and the overlap permitted by the executor.
To realize these policies, we develop a shared GPU-resident engine that fuses communication, attention, and reductions within persistent kernels (§ 8). Building on prior persistent and fused execution techniques [6, 26, 33, 34], the engine coordinates KV arrival, partial-state accumulation, and output completion through device-side protocols. These mechanisms support readiness-driven forward task formation and allow dependent reductions and transfers to advance as their inputs become available, while unrelated attention tasks continue.
We evaluate LiquidCP on eight NVIDIA H100 GPUs against training CP implementations [4, 16, 28, 35, 45] and the FA3-based DCP paths of vLLM and SGLang [31, 37], reporting speedups over the fastest applicable per configuration (Figure 1). Across five variable-length datasets, MegaRing achieves geometric-mean core-attention speedups of 1.23× forward and 1.18× backward, and accelerates a Qwen3-30B-A3B Transformer layer [39] by 1.08–1.16×. On Mooncake trace-derived workloads, MegaDCP achieves geometric-mean core-attention speedups of 1.20× for decode-only and 1.35× for mixed batches. In vLLM serving with preloaded history KV, MegaDCP achieves average speedups of 1.04× in time between tokens and 1.22× in time to first token. This paper makes three contributions:
Figure 1. Core-attention throughput on eight H100 GPUs. Throughput is normalized to the best evaluated baseline in each configuration (dashed line); labels show LiquidCP’s speedups.
Placement and task formation for training. BuddyRing searches structured CP-group placements under token- and attention-load tolerances, while MegaRing dynamically forms forward attention tasks from ready KV regions.
Reduction-aware scheduling for DCP. MegaDCP selects history-KV splits by modeling the completion of attention and its dependent reductions together.
Fine-grained fused execution for CP. A GPU-resident engine supports runtime task formation and coordinates attention, reductions, and transfers through CP-specific readiness and completion protocols.
Background and Motivation
Preliminaries
CP partitions attention along the token dimension, with dataflows determined by the relative query and KV lengths. A partial attention state contains a locally normalized output and the log-sum-exp (LSE) of scores over a KV subset. States over disjoint subsets can be merged into the complete output.
Training and full prefill. For a sequence of length , query and KV lengths satisfy . Grouped-query attention (GQA) uses fewer KV heads than query heads (), favoring KV exchange over exchanging queries and returning partial outputs. Ring CP retains queries on each of ranks and circulates KV shards, computing and merging partial results at each step. For causal attention, zigzag placement divides the sequence into equal blocks and assigns blocks and to rank , balancing attention work (Figure 2(a)). In backward, accumulates locally, while and contributions return to their KV owners. For variable-length batches, dynamic CP [4, 15, 28, 35] adjusts per-sequence parallel degrees and placement to avoid over-partitioning short sequences.
Figure 2. CP dataflows across four ranks. Numbers identify sequence or history-KV partitions; colors indicate owning ranks or query-head groups. (a) Ring CP with zigzag placement. (b) DCP with TP = DCP = 4 and , viewed from Rank 0. denotes rank ’s query-head group; hatched S blocks denote partial attention states. AG, A2A, and Red. denote all-gather, all-to-all, and LSE-weighted reduction.
Decoding. Under tensor parallelism (TP) of degree , each rank owns query heads. When , head-wise partitioning replicates each KV head’s cache across ranks. Decode context parallelism (DCP), supported by vLLM [37] and SGLang [31], shards these replicated caches along the token dimension. We consider groups of ranks sharing one KV head. For appended tokens and history tokens, exchanging queries and partial states moves substantially less data than exchanging history KV when .
In the all-to-all dataflow shown in Figure 2(b), each rank computes causal chunk attention using its local queries and replicated appended KV. Query all-gather enables history attention for all group query heads against each rank’s local history shard. KV splitting divides this shard into independent ranges, whose partial states are combined locally by history reduction. All-to-all returns these contributions to their query-head owners, where final reduction merges them with the local chunk result. Batches may contain only decoding requests (decode-only) or also prefill chunks appended to cached contexts (mixed).
Motivation
Attention and token load balance. Megatron Hybrid CP [28] initializes CP degrees from sequence lengths and places sequences by estimated attention work, without explicitly balancing per-rank token counts. Figure 3(a) separates layer latency into core attention (Attention) and remaining operations (Other). On the GitHub dataset, Megatron Hybrid CP reduces Attention latency by 46.6% relative to Ring FA, while increasing Other latency by 38.3%. Thus, attention gains can coexist with higher costs elsewhere in the layer. Attention work grows approximately quadratically with sequence length, whereas tokenwise work grows linearly with token count; balancing one does not guarantee balancing the other. This motivates a placement objective that accounts for both loads.
Figure 3. CP placement and execution efficiency. (a) Mean forward-plus-backward latency of core attention (top) and remaining layer operations (bottom). (b) Forward (top) and backward (bottom) latency under identical Megatron Hybrid CP assignments, normalized to One-step separately for each dataset and phase. Both panels use eight GPUs with a global 128K-token target.
Communication and ring-step overhead. To separate execution effects from placement, we compare three modes under identical Megatron Hybrid CP assignments. Normal retains overlapped ring communication and per-step computation. Compute-only makes KV available before timing and excludes KV/dKV transfers from the timed region, preserving the same steps and local operations. One-step further combines the local attention work of sequences with the same CP degree into a single call. In forward, removing transfers yields a 1.13–1.19× speedup, and merging steps yields a further 1.21–1.32× speedup (Figure 3(b)). The corresponding backward speedups are 1.15–1.25× and 1.05–1.10×. The first gap measures the aggregate impact of KV and gradients transfers under overlap. The second persists with KV already available and combines changes in invocation count, partial-result merging, and computation shapes.
Insight. KV readiness constrains when computation can begin, but communication steps need not define its boundaries. This motivates exposing inputs through fine-grained transfers while coalescing ready KV regions across communication steps to amortize task and merge overhead.
Execution tails in DCP. The vLLM DCP path executes communication, attention, and reduction as sequential kernels [36]. Consequently, early-finishing SMs cannot advance to later stages until the current stage’s last task finishes. We examine this pattern using a stage-serialized variant of our executor with native FA3 and explicit inter-stage barriers. We define tail idle as an SM’s wait after its last task in a stage until that stage finishes. Its fraction is summed tail idle divided by total SM-time across the plotted stages: 40.1% for the decode-only batch and 48.1% for the mixed batch (Figure 4).
Figure 4. Streaming multiprocessor (SM) activity across serialized DCP stages. Timelines show a selected rank for decode-only (top) and mixed (bottom) batches on eight GPUs with TP = 8 and DCP = 2. Colors indicate stage work; gray indicates tail idle. The horizontal axis concatenates each stage’s work span, from its earliest start to its latest completion.
The native FA3 heuristic [8, 9, 32] uses the batch’s maximum query length to bound KV splitting. In the plotted mixed batch, long prefill queries set this bound too, leaving long-history decoding requests unsplit. Batch-wide query parallelism thus conceals long-running decoding tasks that delay stage completion. In the decode-only batch, splitting exposes more parallel work, but a few last wave tasks still keep other SMs waiting.
Insight. KV splitting reduces work per attention task but adds partial results to merge. This motivates selecting splits by the completion of history attention and its dependent reductions, while allowing ready consumers to proceed once their own inputs are available.
Figure 5. Ring CP and DCP in the task model, viewed on GPU 0. Rows denote tiles (query-head groups in DCP), columns denote KV tiles, and colors identify data owners. Gray cells are masked; “new” denotes appended KV. Arrows, dashed regions, and R nodes denote Transfer, Compute, and Reduction, respectively.
Figure 6. Readiness-driven execution on GPU 1 (G1) with four-GPU zigzag placement. Colors identify KV owners; numbers denote KV tile indices. (a) Valid KV ranges for and ; gray regions are masked. (b) Logical KV order for before task assignment at . (c) Cooperative thread array (CTA) timelines with and . Task labels identify the Q tile and processed KV tiles; arrows indicate dependencies.
A Common Task Model for Context-Parallel Attention
Attention Regions. An attention region consists of query tiles and KV tiles , evaluated under the attention mask. A region may contain noncontiguous tiles owned by different GPUs. For data reuse, our implementation fixes one query tile per forward/decoding region and one KV tile per backward region.
For a query , let partition its valid KV positions. Let and denote the normalized attention output and LSE of scores over . Attention composition gives the complete output [8]:
In backward, each region uses the complete forward outputs and LSEs, together with the output gradient , to produce additive contributions to , , and . If the regions cover every valid query–key interaction exactly once, summing their gradient contributions yields the complete attention gradients.
Primitive Operations. The model uses three logical primitives (Figure 5). Transfer makes operand tiles or partial results available on their consuming GPU. Compute evaluates one region and produces its partial results. Reduction merges output/LSE states or sums gradient contributions for the same target, either incrementally or as a separate task. These primitives may be fused within a single kernel.
Dependencies track the tiles and partial states consumed by each task. A task may execute once its own inputs are available, independently of unrelated tasks. A target becomes complete only after all required contributions have been combined.
Design Implications. Placement determines per-rank attention and token loads. For a fixed placement, region size trades task and merge overhead against input readiness and task parallelism. Coalescing amortizes costs, but waiting to form larger regions can delay dispatch and leave fewer tasks available for parallel execution. Splitting exposes more tasks while creating more partial states and reduction work.
For long queries, MegaRing uses BuddyRing to balance attention and token loads through sequence placement, and coalesces ready KV regions across communication blocks in forward attention (Section 4). For short queries over evenly sharded history KV, MegaDCP chooses splits by jointly modeling history attention and its dependent reductions (§ 7).
MegaRing for Training and Full Prefill
MegaRing transfers KV in fine-grained tiles and forms forward Compute tasks by coalescing ready tiles across communication blocks and KV owners. Its placement algorithm, BuddyRing, balances token and attention loads while limiting sequence fragmentation.
Readiness-Driven Execution
Given a placement, MegaRing indexes each Q tile’s applicable KV tiles in a logical order, with local tiles first and remote tiles appended as their readiness is committed. Let denote the endpoint of the ready prefix and the endpoint of work already assigned to Compute tasks. Thus, contains ready, unassigned KV tiles. Communication CTAs extend the ready prefix, while compute CTAs claim intervals from . After completing their transfers, communication CTAs join computation.
When , a compute CTA claims , where
Assignment advances to $E and fixes the task’s interval; later arrivals are handled by subsequent tasks. Once all required KV tiles are ready, a final nonempty remainder smaller than may also be claimed. Each task merges its partial output and LSE into the Q tile’s running state using Equation 1. The output is complete only after all required KV contributions have been merged. § 8.1 details readiness tracking, task assignment, and result accumulation.
At in Figure 6, has and . Compute CTA 1 claims , combining KV tiles 0 and 3 from G0 and G3 into one task. Both thresholds are measured in KV tiles. trades earlier dispatch against amortization of Q loading and partial-state updates; limits per-task work to preserve parallelism when many tiles are ready. For zigzag-partitioned sequences, we use the empirical defaults
where is the number of KV tiles in a zigzag half-shard. Here, is the sequence length, is CP degree, and
@page 4/13 starts_mid=0 ends_mid=1
Figure 5 (PDF p. 4). Ring CP and DCP in the task model, viewed on GPU 0. Rows denote tiles (query-head groups in DCP), columns denote KV tiles, and colors identify data owners. Gray cells are masked; “new” denotes appended KV. Arrows, dashed regions, and R nodes denote Transfer, Compute, and Reduction, respectively.
Figure 6 (PDF p. 4). Readiness-driven execution on GPU 1 (G1) with four-GPU zigzag placement. Colors identify KV owners; numbers denote KV tile indices. (a) Valid KV ranges for and ; gray regions are masked. (b) Logical KV order for before task assignment at . (c) Cooperative thread array (CTA) timelines with and . Task labels identify the Q tile and processed KV tiles; arrows indicate dependencies.
A Common Task Model for Context-Parallel Attention
Attention Regions. An attention region consists of query tiles and KV tiles , evaluated under the attention mask. A region may contain noncontiguous tiles owned by different GPUs. For data reuse, our implementation fixes one query tile per forward/decoding region and one KV tile per backward region.
For a query , let partition its valid KV positions. Let and denote the normalized attention output and LSE of scores over . Attention composition gives the complete output [8]:
In backward, each region uses the complete forward outputs and LSEs, together with the output gradient , to produce additive contributions to , , and . If the regions cover every valid query–key interaction exactly once, summing their gradient contributions yields the complete attention gradients.
Primitive Operations. The model uses three logical primitives (Figure 5). Transfer makes operand tiles or partial results available on their consuming GPU. Compute evaluates one region and produces its partial results. Reduction merges output/LSE states or sums gradient contributions for the same target, either incrementally or as a separate task. These primitives may be fused within a single kernel.
Dependencies track the tiles and partial states consumed by each task. A task may execute once its own inputs are available, independently of unrelated tasks. A target becomes complete only after all required contributions have been combined.
Design Implications. Placement determines per-rank attention and token loads. For a fixed placement, region size trades task and merge overhead against input readiness and task parallelism. Coalescing amortizes costs, but waiting to form larger regions can delay dispatch and leave fewer tasks available for parallel execution. Splitting exposes more tasks while creating more partial states and reduction work.
For long queries, MegaRing uses BuddyRing to balance attention and token loads through sequence placement, and coalesces ready KV regions across communication blocks in forward attention (Section 4). For short queries over evenly sharded history KV, MegaDCP chooses splits by jointly modeling history attention and its dependent reductions (§ 7).
MegaRing for Training and Full Prefill
MegaRing transfers KV in fine-grained tiles and forms forward Compute tasks by coalescing ready tiles across communication blocks and KV owners. Its placement algorithm, BuddyRing, balances token and attention loads while limiting sequence fragmentation.
Readiness-Driven Execution
Given a placement, MegaRing indexes each Q tile’s applicable KV tiles in a logical order, with local tiles first and remote tiles appended as their readiness is committed. Let denote the endpoint of the ready prefix and the endpoint of work already assigned to Compute tasks. Thus, contains ready, unassigned KV tiles. Communication CTAs extend the ready prefix, while compute CTAs claim intervals from . After completing their transfers, communication CTAs join computation.
When , a compute CTA claims , where
Assignment advances to $E and fixes the task’s interval; later arrivals are handled by subsequent tasks. Once all required KV tiles are ready, a final nonempty remainder smaller than may also be claimed. Each task merges its partial output and LSE into the Q tile’s running state using Equation 1. The output is complete only after all required KV contributions have been merged. § 8.1 details readiness tracking, task assignment, and result accumulation.
At in Figure 6, has and . Compute CTA 1 claims , combining KV tiles 0 and 3 from G0 and G3 into one task. Both thresholds are measured in KV tiles. trades earlier dispatch against amortization of Q loading and partial-state updates; limits per-task work to preserve parallelism when many tiles are ready. For zigzag-partitioned sequences, we use the empirical defaults
where is the number of KV tiles in a zigzag half-shard. Here, is the sequence length, is its CP degree, and is the KV tile size. the number of tokens per KV tile. The thresholds are therefore sequence-specific.
Backward retains one KV tile per task, accumulating its and in registers over the applicable local Q tiles, which are already available. Coalescing multiple KV tiles would require additional gradient accumulators and increase register pressure. We therefore leave backward coalescing to future work.
Workload Placement with BuddyRing
Placement space. Given sequence lengths and a power-of-two GPU count , BuddyRing assigns sequence to an aligned group : consecutive GPUs starting at , with . These disjoint or nested groups form a buddy hierarchy. Admissible group sizes and tile-alignment requirements are specified in Appendix § 12.§ 1. Sequences remain local when and use zigzag placement otherwise. The groups of different sizes are executed in one MegaRing kernel launch.
Placement objective. Each group member receives tokens and estimated attention work , where for causal attention. Let and denote the maximum absolute relative deviations of per-rank token and attention loads from their respective means and . BuddyRing permits deviations up to and (10% and 5% by default), so small imbalances do not force additional splitting. A placement is feasible when its violation
is zero.
BuddyRing ranks placements lexicographically by , a splitting profile, the communication proxy , the number of distinct groups, and finally and . The splitting profile compares, for each length class from shortest to longest, the number of split sequences followed by their summed split depths . Once both load tolerances are satisfied, this ordering prioritizes avoiding fragmentation of short sequences.
Search. BuddyRing uses large sequences to establish a placement and smaller sequences to fill residual capacity. Sequence is structural if ; the remaining sequences are fillers. A Pareto beam search places structural sequences in descending order, exploring their legal groups while retaining partial placements with different load, fragmentation, and communication trade-offs. Fillers are initially placed greedily on single GPUs, and the best completed placements undergo local refinement.
If no feasible placement is found, BuddyRing progressively allows fillers to split, processing longer length classes first and trying smaller groups before larger ones. The search stops at the first feasible relaxation stage; otherwise, it returns the best-ranked placement across all stages. Appendix § 12.§ 1 specifies candidate pruning, filler placement, local refinement, and relaxation. The planner runs once per batch and returns group assignments ordered by decreasing group size to MegaRing’s executor.
In Figure 7, BuddyRing keeps the 2K sequence local. Megatron Hybrid CP splits it across two GPUs when expanding groups initialized with an 8K-token per-GPU target. BuddyRing’s placement meets both load tolerances, reducing from 18.75% to 9.38% and from 14.20% to 0.89%. It uses nine sequence shards, compared with twelve for Megatron Hybrid CP, limiting fragmentation to preserve computational efficiency.
Figure 7. Placement of a 64K-token batch on four GPUs. Colors identify sequences; gray indicates GPUs outside each sequence’s group. “Part.” denotes CP degree × tokens per GPU, with K = 1024. and report token and attention imbalance, respectively.
MegaDCP for Decoding
MegaDCP targets decode-only and mixed batches containing chunked prefill. It keeps history KV evenly sharded and combines dependency-driven execution with KV splitting guided by the joint completion of attention and history reduction.
Execution Model
On each GPU, MegaDCP executes the batch’s DCP stages as a graph of Transfer, Compute, and Reduction tasks within one persistent kernel. The shared attention queue places local chunk tasks before history tasks, allowing chunk attention to overlap query transfers. A history-attention task becomes eligible when all its queries are available; a history-reduction task becomes eligible when all contributing partial attention results are ready. Completed history contributions are published for transfer to their query owners at output-block granularity. Final reduction requires the block’s local chunk and history results and all remote history contributions.
Each compute CTA switches to reduction once all attention tasks are claimed, allowing ready reductions to overlap with attention on other CTAs. § 8 describes task dispatch and completion signaling.
Critical-Wave KV Splitting
Critical-wave model. Splitting changes both when partial results become ready and when attention CTAs release warps for reduction. MegaDCP’s critical-wave model jointly simulates attention and history reduction on a GPU.
One time unit is the per-CTA time to process a history KV tile with all compute CTAs executing attention. With active attention CTAs, the modeled per-tile time is
The term models shared HBM bandwidth, while imposes a calibrated per-tile time floor. An attention task carries tile-equivalents of work, comprising KV tiles and a fixed per-task work overhead . The simulator advances each running attention task at rate , updating whenever an attention task starts or finishes.
History-reduction duration is approximated by the load-independent cost
Here, counts output vectors (query token–head pairs), counts contributing splits per vector, is the per-task overhead, and is the cost per partial vector, in normalized time units.
The simulator dispatches each candidate plan’s attention tasks in order to the first available compute CTA. Here, counts CTAs executing chunk or history attention. A CTA releases its warps to history reduction after completing its last attention task. Each reduction uses one warp and starts in queue order when its warp and inputs are available. The score is the latest simulated completion time across attention and history-reduction tasks; lower is better. The model omits query transfers, result exchange, and final reduction, so its score serves as a proxy for kernel latency.
Task ordering. Decode-only plans with splitting use release-aware longest-processing-time-first (LPT) ordering. Query transfers are prioritized by the attention work they unlock. History tasks are ordered by the transfer epoch of their last required query block, then by decreasing . Unsplit and mixed-batch plans retain the original FIFO order of attention tasks in the shared chunk/history queue.
Plan selection. The planner searches over per-sequence split counts, starting from NoSplit: one Compute task per query tile over its full local history. MegaDCP uses the native FA3 heuristic’s split counts as the FA3-derived reference plan and initial per-sequence search upper bounds. For decode requests with long histories, MegaDCP raises these bounds to at least four, preserving splitting options in mixed batches.
At each iteration, the planner evaluates increasing the split count of each sequence below its limit by one. It also evaluates joint increments for sequences whose tasks tie at the attention tail, since changing one sequence alone may leave the score unchanged. The best-scoring candidate is accepted only if it improves the current score. Search stops when no candidate improves the score or no further splitting is allowed.
MegaDCP selects the best-scoring plan among NoSplit, the search result, and the FA3-derived reference. A split plan is adopted only if its modeled completion time is at least 10% lower than NoSplit. Appendix A.2 specifies parameter calibration, split limits, tie-breaking, and the construction of transfer epochs.
Figure 8 compares plans for local histories of 192, 176, and 2 KV tiles on four compute CTAs. Tuples give their partition counts in this order. NoSplit (1, 1, 1) leaves the long histories on two CTAs and finishes at 152 model time units. The FA3-derived plan (3, 3, 1) exposes more parallelism but creates a second wave of long tasks and more reduction work, finishing at 121.5. The selected plan (2, 2, 1) distributes the long histories across four concurrent tasks while limiting task and reduction overhead, finishing at 104.
Figure 8. Illustrative split plans under the critical-wave model. Tuples list per-sequence KV partition counts; (1, 1, 1) denotes NoSplit. Numbers inside attention blocks label KV-tile counts; hatched blocks are history reductions, drawn on the CTAs whose warps execute them. Block lengths reflect concurrency-dependent processing rates. Gray marks idle time, dashed lines mark model completion, and arrows show reduction dependencies. Illustrative parameters: , , , and . computation and coordinate through readiness flags and completion counters. Appendix A.3 details the publication protocols and CTA configurations.
Execution Engine Implementation
MegaRing and MegaDCP share a GPU-resident engine that implements the task primitives of Section 3 using the persistent execution structure of prior megakernels [6, 33]. The host supplies placement and task metadata; CTAs initially specialize in communication or
MegaRing Specialization
Communication CTAs pull remote KV tiles directly from their owners over NVLink into a local buffer, so each tile is transferred once and reused by all local Q tiles and query heads. MegaRing exposes out-of-order KV arrivals through a per-sequence committed-prefix counter and a logical-physical KV map. Only fully written tiles enter this contiguous, monotonic prefix. For causal zigzag placement, a barrier ensures that all KV needed by front-half queries is committed before publishing KV needed by back-half queries. Compute CTAs combine the applicable remote prefix with local KV to claim eligible intervals (Section 4.1). Forward epilogues merge partial outputs and LSE under a per-Q-tile lock; a separate merged-KV counter tracks completion. Communication CTAs join the compute scheduler after completing their transfers.
Backward accumulates locally and stages and for their KV owners. After local completion, communication CTAs apply Tensor Memory Accelerator (TMA) reduce-add operations to the owners’ gradient accumulators and publish system-scope completion signals once these operations finish. Owners wait for all expected signals before final gradient conversion.
MegaDCP Specialization
MegaDCP dispatches host-prepared descriptors from a unified chunk/history attention queue and separate history- and final-reduction queues. After its last attention task, each compute CTA synchronizes, reuses its shared memory, and repartitions registers for reduction.
Communication CTAs pull query blocks and update local readiness counters. History attention waits for its required query blocks, and attention tasks publish completion flags after writing partial outputs and LSE to local GPU memory. History reductions wait for their inputs and track completion per destination and output block. The last contributor to a remote output block notifies the query owner, which pulls the results and records receive completion after the transfer finishes. Final reduction waits for local chunk-attention flags, the local history-reduction counter, and all remote receive-completion flags to indicate that its enclosing output block’s inputs are complete.
Evaluation
Experimental Setup
Platform and workloads. All experiments run on one node with eight NVIDIA H100 SXM GPUs, each with 80 GB HBM and 450 GB/s NVLink bandwidth per direction. MegaRing uses causal attention with 32 query heads, eight KV heads, and head dimension 128 [16, 39]. We sample 20 batches per dataset and token budget (64K, 128K, and 256K) from ProLong [14], ArXiv, FreeLaw, GitHub, and Pile-CC [13], covering distributions dominated by long or short sequences (Figure 15(a)).
Figure 15. Training and inference workload distributions. (a) Sequence-length CDFs for five training datasets. (b) CDFs of cache length and cache ratio in the Mooncake trace.
For MegaDCP, offline scheduling with prefix caching generates 300 snapshots from the Mooncake trace [27, 30] at arrival-rate scales of 1×, 2×, and 4×. Replaying each snapshot at DCP degrees 2, 4, and 8 yields 630 decode-only and 270 mixed cases. Each decoding request contributes 16 query tokens per iteration, one target and 15 draft tokens as in speculative decoding [11, 22]; chunked prefill processes uncached suffixes in chunks of up to 4K tokens. Appendix § 13 provides additional setup details and workload statistics.
Baselines. MegaRing baselines include zigzag NCCL Ring Attention [25, 29, 45]; overlapped KV all-gather per sequence or across packed batches (Llama-3 style [16]); the placement algorithms of Megatron Hybrid CP [28] and Zeppelin [4], reimplemented with FA3/NCCL; and MagiAttention’s public variable-length API [35]. MegaRing All CP zigzag-partitions every sequence across all GPUs; MegaRing Hybrid uses BuddyRing placement.
MegaDCP baselines reproduce the DCP paths of vLLM and SGLang on a shared FA3 [32] attention kernel with Q all-gather. vLLM-AG-RS [21, 36], vLLM-A2A [2, 37], and SGLang-AG-AR [31, 43] use output reduce-scatter, packed output/LSE all-to-all, and output all-reduce, respectively. SGLang communicates in FP32, whereas vLLM and MegaDCP use BF16. We evaluate eager execution and CUDA Graph replay, reporting Graph results unless noted otherwise.
Measurement. Core-attention timings include CP communication and result reduction and use the slowest rank’s execution time. Throughput is effective attention FLOPs per second per GPU; workload-weighted throughput divides total FLOPs by total time and GPU count. Case latency is the mean over 40 timed iterations for MegaRing and the median over 60 for MegaDCP. Appendix § 13 details the warm-up and measurement procedures. For variable-length workloads, we select the best communication-CTA count from {4, 8, 12, 16} per configuration. Core-attention timings exclude host-side planning, which runs once per batch on the CPU and is reused by every layer. MegaDCP’s planner, including descriptor packing, takes under 3 ms at p95 per decoding step and is included in the end-to-end serving results; BuddyRing depends only on sequence lengths, so it can run in the data loader ahead of each training step.
Core Attention Performance
MegaRing. MegaRing Hybrid outperforms the best baseline in all 15 forward configurations and 14 of 15 backward configurations, with geometric-mean speedups of 1.23× and 1.18×, respectively (Figure 9). On ProLong and ArXiv, All CP already matches or exceeds the best baseline, and BuddyRing adds at most 7%. Placement matters more on FreeLaw, GitHub, and Pile-CC, where splitting short sequences across all eight GPUs leaves too little work per task. All CP falls to 0.17× the best baseline on Pile-CC, whereas Hybrid reaches 1.63× in forward at 64K. Gains diminish at larger token budgets as baselines better amortize per-step costs. Backward benefits less on these short-sequence-heavy datasets because fixed KV tiles prevent coalescing of small ready regions. On Pile-CC backward at 256K, Hybrid’s latency is 8.8% higher than MagiAttention’s.

Figure 9. Workload-weighted forward and backward attention throughput per GPU. Hybrid annotations show speedup over the best baseline per dataset and token budget.
MegaDCP. MegaDCP achieves geometric-mean speedups of 1.20× on decode-only batches and 1.35× on mixed batches over the best CUDA Graph baseline (Figure 10). Decode-only speedups increase from 1.11–1.19× at DCP degree 2 to 1.29–1.31× at degree 8: shorter local histories increase the relative cost of transfers and reductions, making inter-task overlap more valuable. Mixed-batch speedups grow with arrival rate, reaching 1.56× at degree 2 and a 4× rate. § 9.4 examines the contributions of planning and cross-stage execution.
Figure 10. Workload-weighted attention throughput per GPU on decode-only (top) and mixed (bottom) batches. MegaDCP uses eager execution; baselines are evaluated with both eager execution and CUDA Graph replay. Annotations compare MegaDCP with the fastest CUDA Graph baseline.
Uniform-length MegaRing and uniform-history MegaDCP results appear in Appendices § 13.4 and § 13.§ 7.
Sources of MegaRing Gains
Execution mechanisms. Starting from ring CP with external reductions (L1), L2–L5 cumulatively add fused reductions, persistent execution with a fixed queue, CTA role switching, and readiness-driven coalescing, while retaining All-CP placement (Figure 11(a)). Their incremental gains are 1.25–1.31×, 6.1–12.9%, 1.2–2.8%, and 4.0–7.5%, respectively. Fusing reductions yields a substantial improvement; coalescing further amortizes Q loads and partial-result updates across larger tasks. L6 adds BuddyRing placement, improving mean throughput by another 3–8%.

Figure 11. MegaRing ablations. (a) Mean forward throughput on ArXiv at 64K, 128K, and 256K. (b) Workload-weighted forward/backward throughput on ArXiv and FreeLaw at 128K, comparing placements on the same MegaRing kernel with the baselines’ FA3/NCCL execution. MR denotes MegaRing.
Placement. We compare BuddyRing with Megatron-derived and Zeppelin-derived placements using the same MegaRing kernel and sampled batches at 128K tokens. Across all five datasets, BuddyRing achieves geometric-mean forward/backward speedups of 1.11/1.13× over Megatron-derived placement and 1.05/1.07× over Zeppelin-derived placement, with gains on every dataset. Figure 11(b) shows ArXiv and FreeLaw; the larger gains on FreeLaw support balancing token and attention loads while limiting short-sequence partitioning. BuddyRing meets the 10% token-load and 5% attention-load tolerances in all 100 batches. On FreeLaw, mean per-batch (, ) values are (6.4%, 4.3%) for BuddyRing, (44.8%, 33.3%) for Megatron-derived, and (18.7%, 6.8%) for Zeppelin-derived placement. Planner details are in Appendix § 13.
Sources of MegaDCP Gains
Split planning and ordering. We compare MegaDCP with FA3-derived splits and FIFO ordering on the same engine (MegaDCP naive). Critical-wave planning and release-aware ordering provide 1.00–1.03× speedups on decode-only batches. For mixed batches, both variants retain FIFO ordering, and critical-wave split planning provides 1.02–1.29× speedups (Figure 12). For decode-only batches, both variants already split histories and overlap attention with reduction, leaving little additional benefit. For mixed batches, the heuristic leaves long-history decoding requests unsplit alongside prefill chunks. Shortening these tasks gives the largest gain at DCP degree 2 and a 4× arrival rate, where local histories are longest.
Figure 12. Mean per-case median kernel latency on decode-only (left) and mixed (right) batches. Annotations give the ratio of mean latencies, MegaDCP naive over MegaDCP.
Cross-stage execution. At DCP degree 4 and a 4× arrival rate, MegaDCP reduces mean attention-pipeline latency from 128 to 107 s on decode-only batches and from 570 to 394 s on mixed batches relative to vLLM-A2A Graph. Stage profiling shows earlier query-transfer and result-exchange completion, consistent with processing query and output blocks as their dependencies become ready. Appendix § 13.3 provides the timelines and profiling methodology.
End-to-End Performance
We integrate both designs into workloads using Qwen3-30B-A3B [39]; Appendix § 13.§ 8 gives the setup and further analysis. For Transformer-layer training at a 128K token budget, MegaRing Hybrid achieves 1.08–1.16× forward-plus-backward speedups across the five datasets (Figure 13). Faster attention and balanced token loads preserve the gains across the remaining layer operations.
Figure 13. Transformer-layer latency averaged over 20 cases per dataset, decomposed into forward/backward and Core Attention/Other operations. Each MegaRing variant uses its best communication-CTA count per dataset. Annotations above the fastest non-MegaRing baseline show Hybrid’s speedup. Zeppelin denotes our placement-derived baseline without linear-module remapping.
We evaluate vLLM serving with preloaded history KV and single-token decoding on a Mooncake trace segment. Relative to the best vLLM baseline per configuration and metric, MegaDCP achieves 0.99–1.10× speedups in median per-request mean time between tokens (TBT) and 1.02–1.37× in median time to first token (TTFT) (Figure 14). TBT gains are limited by unchanged non-attention computation; TTFT benefits more from the attention gains on mixed batches.
Figure 14. vLLM serving performance across arrival-rate scales and maximum batch sizes. Solid curves show median per-request mean TBT; dashed curves show median TTFT. All backends use CUDA Graphs.
Related Work
Context parallelism and placement. CP uses ring KV exchange, KV all-gather, or head redistribution [16, 18, 25], with hybrid dataflows [12, 17] and improvements in overlap and causal balance [3, 23]. Serving systems adapt parallelism or shard history KV [2, 5, 31, 37, 38]. Variable-length training uses adaptive partitioning, placement, and block mapping [15, 19, 24, 28]; Zeppelin [4] remaps between attention and linear-module layouts. FCP [42] balances computation and memory through fixed-size block placement and flexible peer-to-peer communication. BuddyRing jointly selects sequence CP degrees and aligned power-of-two groups, seeking a single layout that meets token- and attention-load tolerances while limiting short-sequence fragmentation.
Attention task granularity. Single-GPU attention uses tiling, KV splitting, and composable partial states [7, 8, 9, 32, 41]. Across GPUs, MagiAttention [35] combines balanced dispatch with multi-stage overlap, while UltraAttn [40] schedules hierarchically tiled attention. FCP [42] coalesces planned communication and computation sub-stages, separating scheduling granularity from stage-execution granularity. MegaRing retains fine-grained KV transfers and dynamically forms forward Compute tasks from ready, unassigned KV regions across communication blocks and owners. MegaDCP selects history-KV splits by jointly modeling attention and dependent reductions, and orders tasks to advance downstream work.
GPU-resident execution. Megakernels provide persistent execution and dependency management [6, 20, 33]. ParallelKittens [34] and mKernel [26] fuse communication and computation, including Ring Attention; UniEP [44] targets expert-parallel mixture-of-experts (MoE) training. LiquidCP specializes these mechanisms for out-of-order KV arrival, partial-state accumulation, and output-block completion, supporting its task policies and fine-grained progression through attention, reductions, and transfers.
Conclusion
This paper presented LiquidCP, which decouples attention tasks from communication and kernel boundaries in context parallelism. Its common task model of Transfer, Compute, and Reduction primitives places training/full prefill and decoding at opposite ends of one design space: for long queries, MegaRing coalesces ready KV regions and BuddyRing balances attention and token loads, while for short queries, MegaDCP splits history KV by jointly modeling attention and its dependent reductions. On eight H100 GPUs, MegaRing improves core-attention throughput on variable-length training batches by 1.23× (forward) and 1.18× (backward), and MegaDCP by 1.20× on decode-only and 1.35× on mixed batches, in geometric mean over the best baseline per configuration.
Technical Details
BuddyRing Search Details
Candidate groups. BuddyRing tests power-of-two group sizes up to . Causal splits require to divide , keeping all zigzag chunks aligned to 128 tokens. For a structural sequence, the minimum group size is the smallest legal satisfying and , or the largest legal size if none qualifies. The splitting profile in § 6.2 uses length cutoffs of 2K, 4K, 8K, and 16K tokens.
Pareto beam search. For each structural sequence, the beam tries every allowed group for each retained placement and discards dominated candidates. For load , overload is and spread is . For matching per-GPU load bins, BuddyRing keeps the best placement by splitting profile, communication proxy, then distinct group count. The beam keeps the best candidate for each of these three criteria, each spread, and the larger overload. It fills remaining slots by decreasing NSGA-II crowding distance over all criteria [10].
Filler placement. In decreasing order, each filler goes to the GPU that minimizes the larger post-placement overload. Ties favor lower , then lower GPU index, with .
Local repair. Repair considers the highest- sequences on the most and least loaded GPUs for each measure (Table 1). Local sequences can move from a high-load GPU to a low-load GPU or swap between the two. A sequence on a high-load GPU can move to its parent group or, if split, to a same-size or child group containing a low-load GPU. A group containing a marked GPU can move one sequence to each child. Repair takes the best legal improvement until none remains or the iteration limit is reached.
| Parameter | Value |
| Load tolerances | 10% / 5% |
| Structural threshold on | 0.5 |
| Beam width | 64 |
| State-merging bin (of mean load) | 2% |
| Filler smooth-max | 8 |
| Top-ranked placements refined | 8 |
| Maximum repair iterations | 32 |
| GPUs per load extreme / max. sequences per GPU | 2 / 4 |
Table 1. Default BuddyRing parameters.
Relaxation. If placement is infeasible, fillers enter the beam search by length class, longest first, trying group sizes 2, 4, then 8. BuddyRing skips stages with unchanged groups. The final stage permits every legal group for every sequence.
Critical-Wave Model Details
Calibration. We calibrate the model in § 7.2 using per-SM stage timings from Figure 4 (H100, decode-only, DCP degree 2). Of 135 history-attention tasks, the first 132 map to CTAs in index order. The first three CTAs to finish each take one more task. We grid-search the time unit, , and to minimize root-mean-square error (RMSE) in per-CTA busy time. For reduction, we schedule tasks on warps in queue order from observed CTA start times and fit and to sorted CTA busy times by least squares. We use the fitted parameters (Table 2) across all evaluated DCP degrees and workloads.
| Parameter | Value |
| Time per KV tile with all compute CTAs active | |
| Per-tile time floor | 0.43 |
| History-attention overhead | 32 |
| History-reduction overhead | 2.5 |
| Cost per merged partial vector | 0.47 |
Table 2. Critical-wave model parameters measured on H100. Overheads and costs are in model time units.
Fit quality. On the calibration batch, per-CTA RMSE is for attention (measured range 42–67 ) and for reduction (16–35 ). The three CTAs with two attention tasks finish in 64.9–67.3 , compared with the predicted 65.5–66.4 . The fixed cost of 32 tile-equivalents per attention task exceeds the 5–14 KV tiles each task processes in this batch.
Split planning. With compute CTAs, decode requests with at most 16 query tokens and at least local history KV tiles receive a split limit of at least four. Score ties favor earlier attention completion, then fewer attention tasks, then fewer total splits. The planner also uses for chunk-attention tasks. For unsplit sequences, reduction tasks copy runs of output vectors. With local query heads, MegaDCP chooses the largest run in that leaves at least tasks for the 12K warps.
Release-aware ordering. In split decode-only plans, each history task divides its modeled cost equally among the 16-token query blocks it needs. Blocks transfer in decreasing order of their summed costs, in epochs of transfers for communication CTAs and DCP degree . History tasks run in order of their last required block’s epoch, then by decreasing .
Execution Engine Details
MegaRing KV publication. After storing a KV tile for all heads, a communication CTA probes consecutive slots from the committed prefix using acquire–release compare-and-swap (CAS). A failed CAS makes the occupied slot’s KV data visible. After inserting the physical index at , the CTA raises the prefix count to at least with a release atomicMax. Producer warps acquire-load the count before claiming work. Once transfers finish, communication CTAs synchronize locally and reuse shared memory for compute.
MegaDCP CTA configuration. Each 384-thread compute CTA has three warp groups. Per-thread register quotas change from 24/240/240 for attention to 168/168/168 for reduction. The leader claims 12 history-reduction descriptors at once, one per warp. For final reduction, eight warps execute one descriptor at a time per CTA. MegaDCP block transfers. Each 128-row history-attention tile waits for its 16-token query blocks, which communication CTAs pull in host order. After writing the send buffer, history-reduction warps increment the counter for the destination rank and 16-token output block. For a remote block, the last contributing warp sets a ready flag on the query owner’s GPU with system-scope release ordering. The owner’s communication CTA acquire-reads the flag, pulls the output and LSE, then signals local receive completion.
Additional Evaluation Details
Workloads and Baselines
We simulate Mooncake requests in 20 ms steps using trace arrival times and input/output lengths. For each scale in Table 3, we divide interarrival times by that factor and sample 100 snapshots between 300 and 3,600 s. Each decoding step accepts 0–4 of 15 drafts with probabilities (0.20, 0.25, 0.25, 0.20, 0.10), committing 2.75 tokens on average including the target. MegaDCP microbenchmarks use 32 query heads, head dimension 128, and TP degree 8. DCP degrees 2, 4, and 8 use four, two, and one KV heads, respectively, so each group shares one KV head.
| 1× | 2× | 4× | All | |
| Arrival-rate scale | 1× | 2× | 4× | All |
| Snapshots (D/M) | 87/13 | 69/31 | 54/46 | 210/90 |
| Mean requests per batch | ||||
| Decode-only: decode | 8.28 | 17.12 | 34.54 | 17.93 |
| Mixed: decode | 8.38 | 17.26 | 34.26 | 24.67 |
| Mixed: chunk prefill | 1.15 | 1.13 | 1.48 | 1.31 |
| Mean length (tokens) | ||||
| Decode KV history | 12,156 | 12,479 | 12,174 | 12,259 |
| Prefill KV history | 20,992 | 16,574 | 12,040 | 14,523 |
| Prefill chunk | 2,420 | 3,486 | 2,772 | 2,939 |
Table 3. Mooncake snapshots by arrival-rate scale. D/M denotes decode-only/mixed batches. Request counts are means within each batch type. Lengths are means over request occurrences.
The public mKernel [26] (68644c9) and ParallelKittens/ThunderKittens [34] (34b15f7) ring kernels lack the causal mask required for the MegaRing comparison. vLLM uses more Triton-fused operators than SGLang. Against the best eager baseline, MegaDCP’s geometric-mean speedups are 3.72× for decode-only and 1.52× for mixed batches, consistent with greater launch sensitivity for short queries.
Measurement protocol. MegaRing and MegaDCP run 10 and 40 warm-ups, respectively, before timing. CUDA Graph baselines add three eager calls before capture, then use graph replay. Uniform batches use the per-configuration communication-CTA sweep in § 9.1.
MegaRing Ablation Details
The L1 baseline in Figure 11(a) uses eight attention launches, seven NCCL transfers, and eight external reductions.
Our Megatron baseline uses the hybrid-CP scheduler from Megatron-LM [28] (368fa88) to select power-of-two CP degrees from token targets, balance attention work, and expand groups onto idle ranks. Our padded implementation of Zeppelin’s Algorithm 2 [4] allocates GPUs to long sequences in proportion to squared length and packs short sequences locally. Its linear modules retain the attention layout.
With the MegaRing kernel fixed, BuddyRing gains up to 1.16/1.18× in forward/backward throughput over Megatron-derived placement on FreeLaw. With placement fixed, MegaRing outperforms FA3/NCCL by 1.11–1.34× for Megatron and 1.62–2.13× for Zeppelin (Figure 11(b)).
MegaDCP Stage Profiling
Figure 16 orders cases by total effective attention FLOPs across GPUs and smooths per-case medians with a 1 : 2 : 1 moving average. These cross-system timings reflect overlap alongside other implementation differences. MegaDCP is faster in 53 of 54 decode-only cases and all 46 mixed cases. Mean mixed-batch speedups of 1.81× in the lowest FLOP quartile and 1.20× in the highest are consistent with less scope for overlap as attention dominates.
Figure 16. Stage completion at DCP degree 4 and 4× arrivals. Entries give mean times (s) for MegaDCP/vLLM-A2A Graph. Exchange includes history reduction. Both use a common host-side start. MegaDCP (solid) timestamps the last task’s completion in each stage in-kernel and adds launch latency. vLLM-A2A Graph (dashed) accumulates CUDA-event durations scaled per case to latency without profiling events.
Uniform-Length MegaRing Results
BuddyRing and Zeppelin assign each -token sequence to the same GPUs (Table 4). Their comparison isolates execution differences, including fused reductions and readiness-driven coalescing.
| AG | L3 | Ring | Meg. | Magi | Zep. | ||
| 64K | 1 | 1.05/1.22 | 1.03/1.15 | 1.26/1.27 | 1.29/1.28 | 1.06/1.31 | 1.29/1.28 |
| 64K | 2 | 1.20/1.49 | 1.14/1.29 | 1.85/1.84 | 1.13/1.11 | 1.16/1.29 | 1.13/1.11 |
| 64K | 4 | 1.53/2.03 | 1.38/1.51 | 3.06/2.47 | 1.07/1.03 | 1.34/1.31 | 1.09/1.03 |
| 128K | 1 | 1.00/1.10 | 1.02/1.06 | 1.09/1.13 | 1.11/1.14 | 1.00/1.19 | 1.12/1.13 |
| 128K | 2 | 1.04/1.26 | 1.06/1.16 | 1.30/1.38 | 1.31/1.32 | 1.03/1.17 | 1.04/1.05 |
| 128K | 4 | 1.23/1.50 | 1.17/1.24 | 1.80/1.84 | 1.18/1.15 | 1.12/1.14 | 1.10/0.99 |
| 256K | 1 | 1.02/1.02 | 1.00/1.02 | 1.02/0.99 | 1.02/1.00 | 1.01/1.10 | 1.02/1.00 |
| 256K | 2 | 1.02/1.11 | 1.02/1.08 | 1.11/1.10 | 1.13/1.15 | 0.99/1.10 | 0.99/1.00 |
| 256K | 4 | 1.10/1.25 | 1.10/1.13 | 1.36/1.26 | 1.37/1.34 | 1.21/1.10 | 1.00/0.97 |
Table 4. MegaRing Hybrid forward/backward speedups (×) for equal-length sequences totaling tokens. Baselines, in column order, are all-gather, Llama-3 style, Ring FA, Megatron, MagiAttention, and Zeppelin.
Against the fastest baseline per configuration, Hybrid’s geometric-mean speedup is 1.04× in both directions (forward 0.99–1.13×, backward 0.97–1.15×). At 64K (8K tokens per GPU), per-step transfers and merges cost proportionally more. For , Ring FA’s eight-way split of each 16K sequence also makes placement a factor. For a single 128K or 256K sequence, all methods use eight GPUs and the fastest baselines, mostly all-gather variants, hide communication behind attention (Hybrid speedup 0.99–1.06×).
Uniform-History MegaDCP Results
Against vLLM-A2A Graph, the strongest trace baseline, MegaDCP achieves 1.24× on 24 decode-only configurations and 1.20× on 48 mixed configurations (geometric means). The respective ranges are 1.03–1.37× and 1.05–1.53× (Table 5).
| Workload | } | 16K | 32K | 64K | 128K | |
| 0 | 4 | 1.25/1.35/1.35 | 1.22/1.32/1.36 | 1.26/1.37/1.32 | 1.24/1.28/1.30 | |
| 32 | 1.34/1.29/1.37 | 1.24/1.16/1.15 | 1.15/1.14/1.03 | 1.10/1.11/1.09 | ||
| 1 | 4 | 1.05/1.40/1.41 | 1.08/1.14/1.30 | 1.07/1.15/1.22 | 1.17/1.18/1.07 | |
| 32 | 1.42/1.53/1.38 | 1.27/1.38/1.29 | 1.15/1.24/1.23 | 1.12/1.11/1.07 | ||
| 2 | 4 | 1.15/1.33/1.35 | 1.17/1.33/1.24 | 1.07/1.12/1.13 | 1.08/1.06/1.06 | |
| 32 | 1.42/1.36/1.35 | 1.27/1.25/1.23 | 1.16/1.16/1.12 | 1.10/1.08/1.07 |
Table 5. MegaDCP speedups (×) over vLLM-A2A Graph at DCP degrees 2/4/8 for history length per request. Each batch has decode requests with 16 query tokens each and prefill requests with one 2K-token chunk each.
In decode-only batches of 32 requests, speedups fall from 1.29–1.37× at 16K to 1.09–1.11× at 128K as attention dominates.
With four requests, history splitting exposes parallelism across all lengths (1.22–1.37×). Across all 72 configurations, geometric-mean speedups over vLLM-AG-RS Graph and SGLang-AG-AR Graph are 1.30× and 1.70×, respectively.
End-to-End Evaluation Details
Both experiments use Qwen3-30B-A3B [39] (32 query heads, four KV heads, head dimension 128).
Transformer-layer training. Attention uses eight-way CP and the MoE block uses eight-way expert parallelism (EP). All weights, including the expert router, are randomly initialized. In Figure 13, Core Attention includes CP communication. Other covers projections, EP communication, and remaining operations. On ProLong, Other latency for Megatron, Zeppelin, and MagiAttention is close to uniform partitioning. It rises with token imbalance on short-sequence-heavy workloads. Both MegaRing variants stay close to uniform partitioning.
vLLM serving. We replay a contiguous 303 s Mooncake segment [30] (100 warm-up and 1,000 measured requests). Only new prompt tokens need chunked prefill on the decode side of disaggregated serving. With TP and EP degrees of 8 and DCP degree 2, each rank pair shares one KV head. Figure 14 varies arrival-rate scale (1, 2, 4) and maximum batch size (64, 128). Speedups relative to the best vLLM baseline per configuration and metric are averaged arithmetically.
At 4× arrivals and batch size 128, attention takes a larger share of each iteration and TBT speedup reaches 1.10×. At batch size 64, faster iterations shorten the wait for new prompts. With the shared engine, MegaDCP naive reaches up to 1.07× for TBT and 1.29× for TTFT before critical-wave planning and release-aware ordering.
References
- [1]Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). https://arxiv.org/abs/2305.13245
- [2]Nidhi Bhatia, Ankit More, Ritika Borkar, Tiyasa Mitra, Ramon Matas, Ritchie Zhao, Maximilian Golub, Devavatsa Madireddi, Brian Harris, and Bita Darvish Rouhani. 2025. Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding. arXiv:2507.07120 https://arxiv.org/abs/2507.07120 arXiv:2507.07120.
- [3]William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Aknier, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley. 2023. Stripped Attention: Faster Ring Attention for Causal Transformers. arXiv:2311.09431. https://arxiv.org/abs/2311.09431 arXiv:2311.09431.
- [4]Chang Chen, Tiancheng Chen, Jiangfei Duan, Qianchao Zhu, Zerui Wang, Qinghao Hu, Peng Sun, Xiuhong Li, Chao Yang, and Torsten Hoefler. 2026. Zeppelin: Balancing Variable-length Workloads in Data Parallel Large Model Training. In Proceedings of the 21st European Conference on Computer Systems. ACM, 1879–1893. doi:10.1145/3767295.3769369
- [5]Jiefei Chen, Binbin Lin, Jinming Ma, et al. 2026. NanoCP: Request-Level Dynamic Context Parallelism for Data-Expert Parallel Decoding. arXiv:2605.21100 https://arxiv.org/abs/2605.21100 arXiv:2605.21100.
- [6]Xinhao Cheng, Zhihao Zhang, Yu Zhou, et al. 2026. MPK: A Compiler and Runtime for Mega-Kernel Tensor Programs. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26). USENIX Association, 1909–1926. https://www.usenix.org/conference/osdi26/presentation/chengarxiv.org/abs/2512.22219
- [7]Tri Dao, Daniel Y. Fu, Stefano Ermon, Atrii Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv:2205.14135 https://arxiv.org/abs/2205.14135 arXiv:2205.14135.
- [8]Tri Dao, Daniel Haziza, Francisco Massa, and Gregory Sizov. 2023. Flash-Decoding for long-context inference. Stanford CRFM technical blog. https://crfm.stanford.edu/2023/10/12/flashdecoding.html
- [9]Dao Ali Abou El? 2025. FlashAttention Hopper Split Heuristics. Software documentation. https://github.com/Dao-Ailab/flash-attention/blob/v2.8.3/hopper/heuristics.h Version 2.8.3; num_splits_heuristic; accessed 2026-09-25.
- [10]Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and T. Meyarivan. 2002. A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation 6, 2 (2002), 182–197. doi:10.1109/4235.996017
- [11]DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. arXiv:2412.19437 https://arxiv.org/abs/2412.19437 arXiv:2412.19437.
- [12]Jiarui Fang and Shangchun Zhao. 2024. USP: A Unified Sequence Parallelism Approach for Long Context Generative AI. arXiv:2405.07719 https://arxiv.org/abs/2405.07719 arXiv:2405.07719.
- [13]Leo Gao, Stella Biderman, Sid Black, et al. 2020. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027 https://arxiv.org/abs/2101.00027 arXiv:2101.00027.
- [14]Tianyuan Gao, Alexander Wettig, Howard Yen, and Danqi Chen. 2025. How to Train Long-Context Language Models (Effectively). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 7376–7399. doi:10.18653/v1/2025.acl-long.366
- [15]Hao Ge, Junda Feng, Qi Huang, Fangcheng Fu, Xiaonan Nie, Lei Zuo, Haibin Lin, Bin Cui, and Xin Liu. 2025. ByteScale: Communication-Efficient Scaling of LLM Training with a 2048K Context Length on 16384 GPUs. In Proceedings of the ACM SIGCOMM 2025 Conference. ACM, 963–978. doi:10.1145/3718958.3754352DOI
- [16]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 https://arxiv.org/abs/2407.21783 arXiv:2407.21783.
- [17]Diandian Gu, Peng Sun, Qinghao Hu, et al. 2024. LongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism. arXiv:2406.18485 https://arxiv.org/abs/2406.18485 arXiv:2406.18485.
- [18]Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuai-wen Leon Song, Samyam Rajbhandari, and Yuxiong He. 2023. DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. arXiv:2309.14509 https://arxiv.org/abs/2309.14509 arXiv:2309.14509.
- [19]Chenyu Jiang, Zhenkun Cai, Ye Tian, Zhen Jia, Yida Wang, and Chuan Wu. 2025. DCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelism. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. ACM, 221–236. doi:10.1145/3761569.3764849DOI
- [20]Hongyi Jin, Bohan Hou, Guanjie Wang, et al. 2026. Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel. In Proceedings of Machine Learning and Systems, Vol. 8. MLSys, 1917–1933. https://proceedings.mlsys.org/paper_files/paper/2026/hash/53d3f457970d323bd8a0d379c525aa-Abstract-Conference.html
- [21]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles. ACM, 611–626. doi:10.1145/3600006.3613165DOI
- [22]Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202). PMLR, 19274–19286. https://proceedings.mlr.press/v202/leviathan23a.htmlarxiv.org/abs/2211.17192
- [23]Dancheng Li, Rulin Shao, Anze Xie, Eric P. Xing, Xuezhe Ma, Ion Stoica, Joseph E. Gonzalez, and Hao Zhang. 2023. DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Training. arXiv:2310.03294. https://arxiv.org/abs/2310.03294
- [24]Haoyang Li, Fangcheng Fu, Sheng Lin, et al. 2025. Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment. Proceedings of the ACM on Management of Data 3, 6 (2025), 1–30. doi:10.1145/3769802DOI
- [25]Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023. Ring Attention with Blockwise Transformers for Near-Infinite Context. arXiv:2310.01889 https://arxiv.org/abs/2310.01889
- [26]Ziming Ma, Yihan Zhang, Shawn Wei Chew, Shuang Ma, Costin Raiciu, Yang Zhou, Scott Shenker, and Ion Stoica. 2026. mKernel: Fast Multi-GPU, Multi-Node Fused Kernels. arXiv:2609.13585 https://arxiv.org/abs/2609.13585
- [27]Mooncake. 2026. Mooncake FAST25 Trace Release. Software documentation. https://github.com/kvcache-ai/Mooncake/tree/main/FAST25-release Official trace release and schema documentation; accessed 2026-09-25.
- [28]NVIDIA. 2026. Megatron Core: Hybrid Context Parallel Scheduling and BalancedCP Scheduling. Software documentation. https://docs.nvidia.com/megatron-core/developer-guide/latest/apidocs/core.pipeline_parallel.hybrid_cp_schedule.html Official API documentation; accessed 2026-09-25.
- [29]NVIDIA. 2026. NVIDIA Collective Communication Library (NCCL) Documentation. Software documentation. https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/index.html Official documentation; accessed 2026-09-25.
- [30]Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation – A KVCache-centric Architecture for Serving LLM Chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25). USENIX Association, 155–170. https://www.usenix.org/conference/fast25/presentation/qin
- [31]SGLang. 2026. SGLang: Serving Framework Source Code. Software documentation. https://github.com/sgl-project/sglang Source repository; accessed 2026-09-25.
- [32]Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. 2024. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. In Advances in Neural Information Processing Systems, Vol. 37. Neural Information Processing Systems Foundation, Inc. (NeurIPS), 68658–68685. doi:10.52202/079017-2193
- [33]Benjamin Spector, Jordan Juravsky, Stuart Sul, Owen Dugan, Dylan Lim, Dan Fu, Simran Arora, and Chris Ré. 2025. Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B. Hazy Research technical blog. https://hazyresearch.stanford.edu/blog/2025-05-27-no-bubbles
- [34]Stuart H. Sui, Simran Arora, Benjamin F. Spector, and Christopher Ré. 2025. ParallelKittens: Systematic and Practical Simplification of Multi-GPU AI Kernels. arXiv:2511.13940 https://arxiv.org/abs/2511.13940 arXiv:2511.13940.
- [35]Zewei Tao and Yumpeng Huang. 2025. MagAttention: A Distributed Attention Towards Linear Scalability for Ultra-Long Context, Heterogeneous Mask Training. Software and technical documentation. https://github.com/SandAI-org/MagAttention/
- [36]vLLM. 2025. vLLM Attention Operations: Context-Parallel LSE All-Gather and Output Reduce-Scatter. Software documentation. https://docs.vllm.ai/en/v0.11.1/api/vllm/attention/ops/common/ Version 0.11.1; cp_lse_ag_out_rs; accessed 2026-09-25.
- [37]vLLM. 2026. vLLM: Context Parallel Deployment. Software documentation. https://docs.vllm.ai/en/latest/serving/context_parallel_deployment/ Official documentation; accessed 2026-09-25.
- [38]Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles. ACM, 640–654. doi:10.1145/3694715.3695948DOI
- [39]An Yang, Anfeng Li, Baosong Yang, et al. 2025. Qwen3 Technical Report. arXiv:2505.09388 https://arxiv.org/abs/2505.09388 arXiv:2505.09388.
- [40]Haoyu Yang, Zan Zong, Yuyang Jin, Kinman Lei, Jiaao He, Qigang Yang, and Jidong Zhai. 2025. UltraAttn: Efficiently Parallelizing Attention through Hierarchical Context-Tiling. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. ACM, 1769–1784. doi:10.1145/3712285.3759894DOI
- [41]Zhao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, et al. 2025. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving. Proceedings of Machine Learning and Systems 7 (2025).
- [42]Yilong Zhao, Xiaonan Nie, Kan Zhu, Shuang Ma, Zhiqiao Lai, Hongxiang Hao, Yang Zhou, Baris Kasikci, and Ion Stoica. 2026. Unleashing Scalable Context Parallelism for Foundation Models Pre-Training via FCP. Proceedings of Machine Learning and Systems 8 (2026), 2129–2142.
- [43]Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, et al. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In Advances in Neural Information Processing Systems, Vol. 37. Neural Information Processing Systems Foundation, Inc. (NeurIPS), 62557–62583. doi:10.52202/079017-2000DOI
- [44]Size Zheng, Xuegui Zheng, Li-wen Chang, and Jidong Zhai. 2026. UniEP: Unified Expert-Parallel MoE Megakernel for LLM Training. arXiv:2604.19241 https://arxiv.org/abs/2604.19241 arXiv:2604.19241.
- [45]Zilin Zhu. 2026. Ring Flash Attention: Zigzag Ring Attention Implementation. Software documentation. https://github.com/zhuzilin/ring-flash-attention/blob/main/ring_flash_attn/zigzag_ring_flash_attn.py Source repository; accessed 2026-09-25.













