betaFeedback

WorksComputer Science

Understanding GPU Megakernels: Why Residency Can Mislead

Abstract

Megakernels let one persistent GPU program prepare dependent operators early and execute ready tasks together. We study the benefits and costs of these capabilities by reconstructing two LLM megakernels, ThunderKittens (TK) and VDCores, as separate kernels on H100 using their original operator code. Our results show that CTA residency can misrepresent opportunities for useful overlap in both directions. By overlapping startup and operand loading with programmatic dependent launch (PDL), the reconstructions reduce decode-step latency by 5.7% in TK and achieve comparable performance in VDCores when both executions enforce the required memory ordering. Under whole-operator dependencies, TK's persistent execution still gains more from prefetch, but runtime and completion-signal ordering costs outweigh this advantage. Both startup overlap and prefetch depend on resource availability. Persistent CTAs can reuse storage before retiring, whereas separate successors must acquire their full CTA footprints before initializing or loading operands. This admission constraint in both reconstructions motivates finer resource sharing across kernels. Beyond early preparation, joint execution requires ready work and spare capacity in the bottleneck resource. Reserving SMs for communication can add GEMM waves that erase overlap gains, while filling idle SMs offers little benefit once memory bandwidth is saturated. These findings clarify benefits and costs of megakernels and motivate GPU support for earlier resource reuse across kernels.

Introduction

Efficient GPU execution is central to low-latency LLM decoding, where tokens are generated sequentially. Conventional inference systems compose a model from GPU kernels optimized for its individual operators ([16] (Zheng et al. 2024)). In small-batch decoding, short operator executions make startup and data preparation between kernels a significant part of latency.

Recent work explores a different organization: megakernels, which execute multiple operators within a single persistent GPU grid ([5] (Jin et al. 2026; Zheng et al. 2025a)). Keeping a GPU program running across operators can reduce repeated startup, load successor operands early, and schedule ready work together. ThunderKittens’ Llama megakernel (TK) ([31]) uses a GPU-resident interpreter to execute a static task schedule, while VDCores ((He et al., 2026)) decouples memory and compute execution to overlap their work.

What does megakernel execution contribute to inference performance, and when is it preferable to separate kernels? End-to-end comparisons conflate operator implementations, synchronization protocols, and execution organization. We seek to isolate benefits and costs of execution organization and determine which benefits separate kernels can recover.

We conduct an attribution study of execution organization in small-batch LLM decoding on H100. We reconstruct TK as separate kernels using its original operator code, with VDCores as a complementary case study. We vary successor launch and operand-loading times to distinguish startup overlap from prefetch benefits. Traces show when preparation begins and what delays it. We further compare joint and separate execution to identify when overlapping computation with communication or combining ready tasks reduces latency, and when resource-sharing and coordination costs outweigh the gains.

This study reveals a key insight: CTA residency can misrepresent opportunities for useful overlap in both directions. Separate kernels can be blocked from preparing early despite unused capacity, while allowing tasks to execute together need not improve throughput (Figure 1).

Figure 1. How execution organization changes opportunities for useful overlap. Separate kernels retain storage until CTA retirement; persistent CTAs reuse it for prefetch but incur runtime and ordering costs. Red numbers connect mechanisms in (a,b) to findings in (c). Pink marks exposed time or unused capacity; light and dark orange mark runtime costs and completion-signal releases. A/B are operators; W is a weight load.

In our reconstructions, programmatic dependent launch (PDL) recovers opportunities for early preparation by allowing successor kernels to initialize and load operands before their producers finish. After correcting completion-signal ordering in the persistent implementations, the separate-kernel reconstructions reduce decode-step latency by 5.7% in TK and achieve comparable performance in VDCores. Our ablations nevertheless show that TK’s persistent execution gains more from prefetch, although its higher runtime and synchronization costs outweigh that advantage.

TK’s larger prefetch benefit motivates examining what limits early preparation across separate kernels. A persistent CTA can load successor weights into unused storage while the producer is still computing. A separate successor, however, must acquire its full CTA footprint before it can begin preparing. Whole-CTA admission can thus block the loader even when its warps and first weight buffer could fit beside the producer. These constraints motivate GPU interfaces that let preparation acquire resources before a full CTA fits, while preserving efficient operator code.

Beyond preparing dependent operators early, joint scheduling can coordinate work that is ready to run concurrently. We study two such opportunities: overlapping computation with communication and combining ready tiles from independent tasks to fill partial waves. In compute–communication overlap, reserving SMs for communication can force GEMM into additional waves, offsetting the overlap. Worker traces show that idle workers can lack ready tasks. Even with ready work, partial-wave CTAs can saturate bandwidth, so more resident CTAs need not increase useful throughput.

Our main contributions are:

  • Reconstructions of two megakernels, with controlled experiments in TK that distinguish startup and prefetch benefits from execution costs, including the cost of ordering completion signals (Section 4).

  • A diagnosis of how whole-CTA admission blocks useful preparation despite unused producer capacity, motivating interfaces for resource reuse before CTA retirement (Section 5).

  • An analysis of when co-resident execution improves performance, and when dependencies, extra compute waves, or bandwidth saturation leave throughput unchanged or slow computation (Sections 6 and 7).

Background and Scope

GPU Execution and Operator Fusion

An autoregressive decode step executes a dependency graph of operators, including projections, normalization, attention, and collectives. An ordinary kernel launches a grid of cooperative thread arrays (CTAs), which the GPU places on its streaming multiprocessors (SMs). A task or tile is a finer unit of work. Stream-ordered kernels normally release a successor after its producer grid completes. A resident megakernel instead executes tasks from several operators inside one grid.

Fusion can change both scheduling and data movement. MegaMoE, for example, overlaps expert computation with communication and removes intermediate buffers (Zhao et al., 2025). Our mechanism ablations preserve operator computation and compulsory data movement to isolate scheduling effects.

Persistent Execution

Megakernel CTAs execute tasks across operators using static schedules (TK [31]), dynamic schedules (MPK [5]), or both (Event Tensor (Jin et al., 2026)). We examine four capabilities of this organization.

No kernel boundaries. A decode step runs as one launch, avoiding repeated grid startup and initialization (Section 4).

Cross-operator prefetch. TK and MPK reuse shared-memory pages across tasks, loading successor weights while other tasks still compute. Hopper’s Tensor Memory Accelerator (TMA) performs these loads asynchronously. VDCores uses a resident memory core that runs ahead of its compute core (He et al., 2026) (Sections 4 and 5).

Fine-grained dependencies. Tasks signal completion through counters in global memory, so a consumer task can start once the tiles it reads are complete rather than after the whole producer operator (§ 6.2).

Joint scheduling. A runtime that sees tasks from several operators can run independent work together, for example by overlapping computation with communication or by filling SMs that one operator leaves idle (Zheng et al., 2025a; Cheng et al., 2026) (§ 6).

Cross-Kernel Execution Mechanisms

CUDA Graphs replay a recorded dependency graph, amortizing host submission while retaining separate grids [23]. Programmatic dependent launch (PDL) permits a producer to trigger its successor before retirement [21] (Figure 2). However, eligibility does not guarantee immediate admission: successor CTAs still need available SM resources. Once admitted, a successor can load weights and initialize its pipeline before waiting for the producer’s output.

Diagram showing Stream and PDL producer-consumer execution

Figure 2. PDL overlaps consumer initialization and independent loads with producer execution. Before consuming dependent inputs, griddepcontrol.wait waits for producer completion and result visibility.

Graphs and PDL retain separate grids: producer registers and shared memory remain reserved until CTA retirement, and neither creates a shared task pool across operators.

Memory Ordering and Completion Signals

GPU global memory is coherent at L2, but the SMs’ L1 caches are not mutually coherent. Seeing an updated completion counter therefore does not by itself ensure that reads see the producer’s data. Release/acquire synchronization orders the producer’s writes before publishing completion and the consumer’s dependent reads after observing it, at a scope covering both CTAs [22].

In our H100 binaries, a GPU-scope release update adds MEMBAR.ALL.GPU before the counter update. The reader polls with volatile loads that bypass L1, then executes one fence.acquire.gpu after observing completion. This fence compiles to CCTL.IVALL, invalidating local L1 cache lines. The producer barrier orders outstanding accesses before the signal; the consumer invalidation prevents later L1-cached reads from reusing stale data. Volatile polling alone lacks this acquire-side invalidation. Bypassing L1 for the flag alone neither refreshes cached payload data nor orders the producer’s writes.

Across kernels, PDL’s launch trigger permits early execution but does not establish result visibility. Before reading dependent inputs, a successor can use griddepcontrol.wait to wait for producer completion and result visibility [21]. A persistent grid instead needs an explicit in-grid synchronization protocol for data exchanged between CTAs.

Experimental Approach

To separate scheduling effects from operator implementation, we reconstruct megakernels with their original operator code as separate kernels. We vary when successors launch and load operands, when producers trigger them, and how finely results are released.

Setup. Single-GPU experiments use an H100 SXM5 GPU (132 SMs), and GEMM–AllReduce uses eight H100 GPUs. TK decodes Llama-3.2-1B (Raffel et al., 2024) in BF16 at batch size one, with a 32-token prompt and 128 generated tokens. For the main TK results (Figure 3 and Figure 5), we report the latency of a complete decode step averaged over decode positions. Every configuration, including the persistent megakernel, replays one CUDA Graph per position.

Figure 3. TK separate-kernel latency relative to ordered persistent execution with the original dependency granularity. Labels show median paired differences.

Figure 5. TK prefetch benefits and execution costs. Both executions use whole-operator dependencies and required memory ordering. Persistent execution batches completion updates and gains more from prefetch but remains slower. Labels show median paired differences in µs.

Controls. Most TK mechanism ablations keep the operator code and each CTA’s register and shared-memory footprint, switch one mechanism at compile time, and differ from the original compiled code (SASS) only at the targeted sites. Each must reproduce the persistent kernel’s hidden states, KV-cache entries, and logits within per-tensor tolerances. Joint-scheduling studies instead compare complete schedules that are each tuned independently.

Completion-signal ordering. The original TK and VD-Cores persistent implementations use relaxed atomic updates and volatile polling for cross-CTA completion signals, without the ordering described in § 2.4. A standalone test reproducing TK’s signaling sequence exhibits stale reads. We use GPU-scope release updates and an acquire fence after each successful polling loop, labeling these versions Ordered persistent. The main separate-kernel reconstructions use griddepcontrol.wait. Primary comparisons use these ordered versions. The original persistent versions serve only as labeled optimistic references that omit this ordering cost. We assess protocol ordering through source and SASS inspection, separately from numerical correctness checks, because passing output tests alone does not establish ordering guarantees.

Timing and traces. Each TK timing process measures a configuration and the persistent kernel over 14 timed generations after 3 warm-ups. We rotate configuration order across five rounds and report medians. All configurations within a round use the same GPU, with one round on each of five GPUs. VDCores uses seven interleaved timings per round, each averaging 20 graph replays.

The compute-bound MLP experiment (§ 6.2) uses five rounds with seven samples per round. Each sample replays one graph of 20 MLP steps, including counter resets; we report median within-round latency differences per step.

Timestamp traces come from separate instrumented runs to explain execution order and overlap; reported performance comes from uninstrumented timing runs.

Recovering Early Preparation Across Kernels

We first split TK into separate kernels. We then distinguish the startup time hidden by an early launch from the additional benefit of prefetching operands before the preceding operator finishes, and compare prefetch benefits and execution costs in TK.

Launch Overhead and Lost Prefetch

TK ([31]) executes the sixteen layers and LM head of Llama-3.2-1B within a single persistent kernel, whose loader warps fetch the next operator’s weights while its compute warps finish the current one.

We extract a separate kernel for each operator. Each layer contributes five kernels: RMSNorm with QKV projection and RoPE, attention, output projection, RMSNorm with up/gate projection and SwiGLU, and down projection. These kernels execute directly, without the persistent interpreter, in order on a single CUDA stream. With standard CUDA Graph dependencies, each successor starts only after its predecessor grid completes.

This split execution is about 98.1 µs slower per step than the ordered version of the original persistent implementation (Figure 3). Part of this slowdown comes from the startup gaps that remain between kernels even with graph replay. In-kernel timestamps show an all-SM idle interval of about 1 µs between the predecessor’s last CTA exit and the successor’s first CTA entry. Across the step’s 80 boundaries, these gaps add up to 50–100 µs.

Once the successor enters, it must still fetch weights or KV-cache data before computing, exposing memory latency as well as startup. Launching the successor earlier could therefore hide startup alone or also overlap these loads with the producer’s execution. We next separate these two effects within the split execution.

PDL Overlaps Startup and Operand Loading

PDL lets a successor kernel start and prefetch weights or KV-cache data before its predecessor finishes. We enable PDL between successive graph nodes, and each producer triggers its successor after issuing its final weight load. The successor can then initialize and prefetch, but must wait for the producer’s results before consuming dependent inputs.

With PDL, the separate-kernel reconstruction is 60.9 µs faster per step than ordered persistent execution, a 5.7% latency reduction (Figure 3). It remains 3.4% slower than the original persistent implementation, our optimistic reference. The benefit of PDL does not yet distinguish how much comes from earlier startup and how much from prefetch.

To separate these effects, we keep PDL enabled but move independent weight and KV-cache loads after the wait for producer results. CTAs can still enter early and initialize their pipelines. This configuration remains about 97.4 µs faster than standard graph dependencies, showing the startup time hidden by PDL. Startup overlap alone brings separate kernels to parity with ordered persistent execution. Restoring early loads saves a further 61.5 µs.

This overlap raises a placement question. Each extracted CTA retains the original operator’s resource footprint, which nearly exhausts an SM’s registers and shared memory and prevents producer and successor CTAs from residing together. Where can the successor run before the producer grid completes? It turns out that producer CTAs retire at different times across the device. PDL admits a successor CTA to an SM as its producer CTA retires, while producer CTAs continue running elsewhere. We call this a rolling tail (Figure 4).

Figure 4. Staggered CTA retirement lets PDL overlap successor startup and weight loading with producer execution on other SMs. Dependent computation still waits for producer results.

Per-SM traces confirm early successor entry at every boundary and early weight or KV-cache loads at most boundaries. Dependency waits still return after producer-grid completion. The overlap comes from preparing successors on available SMs while producer CTAs run elsewhere, with the result dependency intact.

This makes the producer’s retirement pattern important. TK attention uses only eight SMs, leaving most of the device available to O-projection: 97% of O-projection CTAs issue weights before attention finishes. At the next boundary, the median O-projection CTA retires only about 1 µs before the grid ends, roughly the time a successor CTA needs from entry to its first weight load, so only about 1% of up/gate CTAs issue early. Independent loads benefit only if an SM becomes available early enough to hide them. Rolling tails create this window without requiring producer and successor CTAs to share an SM, but do not provide the same preparation opportunity as reuse inside a live CTA.

As a complementary case, we reconstruct VDCores as separate kernels using its original operator code. We replay one complete decode step of the same Llama-3.2-1B model using VDCores’ original configuration. This reconstruction fuses post-attention RMSNorm into up/gate and the next RMSNorm into down projection. With PDL and griddepcontrol.wait before consuming producer results, it comes within 0.1% of ordered persistent execution.

Why Persistent Execution Is Slower

We examine the execution and synchronization costs that offset persistent execution’s early-preparation benefits. To exclude gains from fine-grained result dependencies, both executions wait for the entire producer before consuming its output. In the persistent variant, each QKV and up/gate runtime instruction publishes one completion update after its final output tile, avoiding redundant per-tile updates. With ordering, this variant is faster than the fine-grained persistent version used above § 6.2.

We first disable prefetch in both executions and retain relaxed signaling in the persistent variant to measure execution-path differences before adding the required ordering. This optimistic reference is still about 79 µs slower per step than separate kernels with PDL. Both execute the same operator code, but separate kernels launch each operator directly, without the persistent interpreter, task dispatch, or storage allocator.

We then measure the additional cost of ordering completion signals within the same persistent implementation. Ordering adds 19.0 µs per step without prefetch but 73.5 µs with prefetch. Loading operands earlier thus makes synchronization more expensive, even with batched completion updates. With prefetch enabled, the relaxed persistent variant is 28.9 µs faster than separate kernels with PDL. Adding the required ordering reverses this comparison: ordered persistent is 43.9 µs slower.

An H100 microbenchmark helps explain this coupling. Release signaling adds MEMBAR.ALL.GPU before the completion-counter update (§ 2.4). We hold the signaling sequence fixed while loader warps on every SM keep background bulk reads in flight. Raising this volume from 64 to 192 KiB per SM increases median release latency from 2.8 to 17.2 µs, although background bandwidth stays near 3.2 TB/s. Relaxed signaling stays near 0.03 µs. When only the signaling SM issues bulk reads, release latency stays near 0.6 µs even at 192 KiB. Release latency therefore depends on device-wide in-flight traffic. Based on these measurements, we hypothesize that once bandwidth saturates, additional reads lengthen memory-system queues, causing release to wait behind this traffic. Prefetch traffic can thus delay the publication of producer completion and offset part of the benefit of loading operands early.

Despite these costs, persistent execution retains a larger net benefit from prefetch. With whole-operator dependencies and required ordering, disabling prefetch increases its latency by 114.8 µs, compared with 61.5 µs for separate kernels with PDL (Figure 5). This motivates examining how persistent execution enables earlier preparation and what prevents separate kernels from accessing the same opportunities (§ 5).

Resource Ownership Constrains Early Work

The split schedule relies on producer CTAs returning resources before a successor can enter. This admission constraint can expose both startup and first operand loads, whereas a persistent CTA can prepare the next operator using resources it already holds. We examine whether separate kernels can access those resources sooner while preserving operator efficiency.

Unused Capacity Can Still Block a Successor

In TK’s persistent megakernel, up/gate weight prefetch can begin before O-projection completes. Each resident CTA reserves thirteen 16 KiB shared-memory pages, but O-projection uses only five. The CTA’s loader can therefore place up/gate’s next weight tile in the remaining pages while O-projection is still running (Figure 6). Traces of the persistent kernel show that the first up/gate weight load issues a median 11.5 µs before O-projection completes.

Figure 6. Capacity reuse at O-projection→up/gate on one SM. O-projection uses pages P0–P4 of a thirteen-page reservation. (a) Separate kernels retain the reservation until CTA retirement. (b) A persistent CTA prefetches up/gate weights into unused pages, then reuses consumed buffers as they become available.

With separate kernels and PDL, however, up/gate must obtain its full CTA resource footprint before it can prefetch weights. Its shared-memory reservation nearly fills an SM, so even releasing O-projection’s unused pages would not leave enough space for the up/gate CTA alongside O-projection. Registers pose a second constraint: O-projection alone occupies 86% of the SM register file, leaving too few registers for the up/gate CTA. The persistent megakernel can prefetch up/gate weights into its unused pages without admitting a separate CTA with both resource requirements. The blocked work is weight loading, but admission tests the resources needed for the entire compute phase.

We test an alternative explanation for the missing prefetch window: a few late O-projection CTAs may delay PDL’s launch trigger for up/gate. PDL permits successor admission only after every producer CTA has issued its trigger [21]. Attention occupies eight of H100’s 132 SMs, so the O-projection CTAs that enter those SMs cannot load weights and issue their triggers until attention finishes. These stragglers could therefore block up/gate admission even when other SMs have capacity. To remove this delay, we pack O-projection’s weight tiles into 124 CTAs that fit on the remaining SMs. The change has little effect on when up/gate can prefetch. Even with earlier triggers, O-projection CTAs still wait for attention results before computing and then finish close together, leaving little capacity for up/gate to enter early. Late triggers from a few stragglers therefore do not explain most of the missing prefetch window.

The same admission constraint limits startup overlap in VDCores. Its attention grid occupies 64 SMs, and traces show that about 47% of O-projection CTAs enter only after attention finishes. These CTAs must initialize queues and barriers and check the input dependency before issuing activation loads, so this startup remains exposed even with weight prefetch disabled. The persistent memory core is already resident and does not repeat this entry path. Resource availability therefore controls both how much successor startup PDL can hide and when prefetch can begin.

Software Recovery Can Add Execution Costs

The admission constraint suggests reducing CTA footprints so that O-projection and up/gate can reside on the same SM. Under whole-CTA admission, however, this requires shrinking the resources reserved for computation even though only the successor’s loader needs to enter early. Both shared-memory reservations and register demand would have to fall. These reductions also constrain the operator’s compute phase and can reduce its efficiency. As a diagnostic of this tradeoff, we evaluate a low-register O-projection that allows two of its own CTAs to share an SM. It is 3.4% slower than the original in isolation. Earlier admission must be weighed against the execution cost of the smaller-footprint operators.

VDCores shows why an improvement at one boundary need not survive across the operator sequence. Reducing shared-memory reservations allows O-projection to prefetch all its weights alongside attention. In instrumented traces, the O-projection level shortens from about 4.7 to 2.9 µs, yet the corresponding full step becomes slower. The early Oprojection CTAs retain their reservations, leaving fewer resources for up/gate to enter next. At other boundaries, smaller reservations remove a weight-buffering stage and slow the pipelines. In these schedules, gaining early loads at one boundary can delay entry or reduce pipeline efficiency at another. Earlier admission changes both the preparation window and the resources available to other work.

Implications for Resource Interfaces

Early weight prefetch needs fewer resources than full computation. At TK’s O-projection→\toup/gate boundary, the loader and its first weight tile could fit beside the producer even though the full successor CTA cannot. The interface should admit this preparation first, then let the successor grow as producer resources are released.

Ordinary inference kernels encounter this constraint. In a GLM-5.2 SGLang configuration, 188 of 822 adjacent same-stream kernel pairs cannot place one CTA from each kernel on the same SM, mostly because of register demand [29] (Zheng et al. 2024; Zhao et al. 2025).

These cases motivate separating shared-memory, register, and warp lifetimes from the complete CTA, preserving efficient computation without requiring a persistent interpreter.

The producer should be able to release unused shared memory and consumed buffers while retaining live buffers, preserving its pipeline as capacity becomes available for successor weights.

Reassigning this storage must preserve forward progress. Before transferring released capacity to the successor, the producer must commit to no further shared-memory allocation. Otherwise, it could wait for capacity held by a successor that awaits its results. Blackwell’s tensor memory (TMEM) provides a precedent: tcgen05.relinquish_alloc_permit commits a CTA to no further TMEM allocation [22]. A shared-memory interface could similarly separate this commitment from the later release of buffers that still contain live data.

Shared-memory release alone is insufficient when registers block the loader. The interface could allow individual producer warps to retire and return their registers and resident-warp slots. For example, a loader warp could retire after issuing its final TMA load and completing any remaining synchronization duties, while the compute warps continue running. Its shared-memory buffers would remain allocated until transfers and consumers finish using them. Warp-level dispatch and resource reclamation, as explored by WarpMan, provide a basis for this behavior [30].

Partial CTA admission must then check the loader’s resource requirements rather than the complete CTA’s. Its warps can

prefetch into available shared memory while compute warps await resources. As the producer releases the remaining capacity, the successor can admit compute warps and grow to its full footprint. Table 1 summarizes these capabilities.

CapabilityEffect on execution
Partial admission and retirementCompleted producer warps release resources so successor loader warps can enter before a full CTA fits.
Phase-scoped releaseProducer returns shared-memory buffers after their last use, before its warps retire.
Ordered growthSuccessor acquires its remaining resources and admits compute warps as capacity becomes available.

Table 1. Proposed resource capabilities for early weight prefetch across kernels. Their performance remains unmeasured.

However, software alone cannot provide these capabilities, because the hardware allocates and reclaims resources per CTA. With identical live CTA footprints, our H100 probes observe admission limits of 145 versus 96 KiB under different allocation histories, consistent with contiguous CTA allocations in a circular shared-memory arena. Under this model, phase-scoped release requires released address ranges to become independently allocatable, while live buffers retain their addresses.

Similarly, for partial admission and ordered growth, admission logic must track changing resource allocations and distinguish resident warps from those awaiting entry, while preserving CTA synchronization state.

When Joint Scheduling Adds Value

Sections 4 and 5 focus on preparing a dependent successor before its producer finishes. The software recovery experiments also show that earlier admission can delay or slow other work. We now ask when sharing resources between independent tasks from different operators improves performance. Communication for completed output tiles can run alongside computation of later tiles, while independent compute tasks can fill the idle SMs of a partial wave. Co-residency does not by itself add execution capacity: ready tasks may use spare bottleneck resources or compete for capacity needed by ongoing work. We examine when the overlap gained outweighs the costs of sharing resources and coordinating execution.

Compute–Communication Overlap

Multi-GPU execution adds inter-GPU communication to the local operator sequence. A tensor-parallel row projection consists of a local GEMM followed by an AllReduce. A fused schedule can begin communicating completed output tiles while later GEMM tiles are still being computed (FigFigure 7a, exploiting the different resources used by computation and communication. Prior systems use this approach for fine-grained compute–communication overlap [27, 4] (Zheng et al. 2025a). MegaMoE applies such overlap to expert-parallel MoE layers, assigning communication to dedicated warps within each CTA rather than to reserved SMs (Zhao et al., 2025). However, communication also needs GPU execution resources, leaving fewer for GEMM. Does hiding communication outweigh the resulting slowdown in computation?

Figure 7. Opportunities for joint scheduling. (a) Releasing GEMM tiles early can hide communication, but reserving resources and signaling add cost. (b) Jointly executing independent tasks may reduce wave quantization when both are ready.

We evaluate Triton-distributed’s fused GEMM–AllReduce implementation (Zheng et al., 2025a), which assigns some SMs to communication and the others to GEMM. Without overlap, GEMM can use all SMs before AllReduce begins, and its best tile configuration may therefore differ. Reusing the fused GEMM’s SM allocation and tile configuration for the sequential baseline could underestimate sequential performance. We therefore tune both implementations independently and compare their full GEMM–AllReduce latencies to determine whether overlap is beneficial.

The experiment covers Qwen3-8B and Qwen3-32B [34] attention-output and MLP-down projections at twelve batch sizes from 1 to 256 with eight-way tensor parallelism on eight H100 GPUs. The fused candidate tunes its tiles and GEMM/communication SM split. The sequential candidate tunes GEMM over all 132 SMs and the following multimem AllReduce independently.

Across the 48 points, fusion is more than 1% faster at 11 points, more than 1% slower at 31, and within 1% at the remaining six (Figure 8). We first examine whether the GEMM provides enough computation to hide communication, then measure how much reserving communication SMs slows that computation.

Figure 8. Fused GEMM–AllReduce latency reduction (%) over independently tuned sequential execution for Qwen3-8B/32B O and down projections on eight H100s (TP8). KK is the local reduction dimension. Outlines mark fused GEMMs requiring multiple waves after SM reservation. Right: median per projection.

Short GEMMs leave limited room for overlap. We estimate the AllReduce time as sequential latency minus GEMM-only latency. This cross-run estimate includes launch and synchronization costs and has an approximate uncertainty of ±1\pm1 μs per point. In these measurements, fusion never wins by more than 1% at the 24 points where GEMM takes less than about 90% of this estimated time. At a given batch size, a model’s attention-output and MLP-down projections produce outputs of the same size and communicate the same amount of data. The down-projection’s larger local reduction dimension supplies more computation to overlap with that communication. O-projection wins at only one of 24 points, whereas ten of the eleven wins are down-projections at batch sizes up to 65.

SM reservation matters at wave boundaries. A wave here is one round of tiles over the SMs assigned to GEMM. We time GEMM alone with and without the communication SM reservation. With the same tile configuration, reserving communication SMs changes isolated GEMM latency by less than 0.3 μs at all 32 single-wave points. The cost rises when the output tiles no longer fit on the remaining SMs and require another wave. For Qwen3-8B O-projection at batch 128, both candidates select the same tile configuration: the 128 output tiles fit on 132 SMs but need a second wave on the fused GEMM’s 116. Timed alone, GEMM slows from 4.9 to 7.2 μs, an increase exceeding the fused schedule’s net loss of 2.2 μs. Fifteen of the sixteen multi-wave configurations are slower than sequential execution. The remaining configuration still wins, showing that overlap can sometimes repay the extra GEMM wave.

Seven single-wave points above the time-ratio threshold still lose, so a long GEMM and a single wave do not guarantee that overlap pays. The gain from hidden communicaгьыл tion must cover the compute slowdown and coordination costs. Wave boundaries make the compute penalty especially abrupt.

Reducing Wave Quantization

Joint scheduling can also reduce wave quantization by filling the idle SMs in an operator’s final, partial wave. If another operator’s tiles are ready, they can run alongside that wave rather than in a later wave of their own (Figure 7b). Two conditions determine whether this opportunity improves latency: independent work must be ready, and the idle SMs must provide capacity in the resource that limits throughput.

Dependencies limit ready work. Fine-grained scheduling can exploit available parallelism, but cannot create it. Finer dependency release can make successor tiles ready before their producer finishes. The opportunity depends on whether consumer readiness coincides with available execution capacity.

Publishing finer-grained readiness also has a cost. In TK, the whole-operator variant used in § 4.3 replaces per-head and per-shard dependencies at QKV→attention and up/gate→down with one completion counter per operator. This permits each producer runtime instruction to publish a single release update after its final tile, while retaining the same tile decomposition and prefetch. With required ordering in both versions, the coarser variant reduces decode-step latency by 1.6% (about 17 µs). In this schedule, the earlier readiness provided by fine-grained dependencies does not repay the cost of more frequent completion updates.

In full-model execution, persistent-worker traces show that ready tasks can remain scarce even with fine-grained scheduling. In MPK’s Qwen3-30B-A3B trace, workers execute tasks for only 40.2% of their resident time, spending the rest between tasks (Figure 9). The task graph helps explain these waits: an attention stage has only four tasks, followed by an O-projection with 32 tasks, for 128 resident workers. This narrow dependency chain limits how many workers can execute, even though the runtime can dispatch individual tasks. Triton-distributed shows similar waiting: its Qwen3-8B TP8 path spends 75.0% of resident CTA-time waiting for dependencies.

Figure 9. Share of resident worker time spent executing tasks versus waiting inside persistent grids. Triton-distributed: Qwen3-8B, TP8, 132 workers. MPK: Qwen3-30B-A3B, one GPU, 128 workers. Fractions refer to resident worker-time, not GPU utilization.

A larger physical grid does not guarantee more useful work. In a GLM-5.2 trace, decode DeepGEMM launches 132 CTAs for 8–128 logical output tiles, while a query-length-four FA3 case uses 36 useful scheduler CTAs behind 132 physical CTAs. Joint scheduling needs additional tiles whose dependencies are satisfied, rather than simply more resident workers.

Bandwidth limits the benefit of additional ready work. Even when independent tasks are ready, filling idle SMs need not improve throughput. A bandwidth-bound task may already saturate memory with only part of the GPU active. Additional memory-bound tasks then compete for the same bandwidth. In contrast, an underfilled compute-bound wave leaves arithmetic units available for other compute tasks. In a steady-state TMA read experiment, 32 of H100’s 132 SMs supply 95.4% of the full-device bandwidth (Figure 10). Filling the remaining SMs offers little additional bandwidth for this workload.

Figure 10. Throughput versus active SMs, normalized to the full GPU, for TMA reads and tensor-core computation. Labels show half-wave throughput loss. The dashed line indicates linear scaling.

The cost of leaving SMs idle grows as the workload becomes more compute-intensive. A kernel that performs arithmetic on its loads needs more SMs than the read stream to approach full throughput: a half-filled wave loses about 17% at 32 FLOP/B and nearly half at 1024 FLOP/B (Figure 10).

We construct a compute-bound MLP, with fused SwiGLU and split-KK down projection, in a persistent Triton kernel with one worker per SM. With 2.12 waves of up/gate tiles and 1.88 waves of down tiles, releasing completed input slices lets ready down tiles fill the sparse producer tail, reducing latency by about 13% versus whole-operator release.

For a given workload, this opportunity should be evaluated after tuning each operator’s tile configuration.

Implications for Runtimes and GPUs

These results suggest three rules for choosing and tuning execution forms.

Tune prefetch together with synchronization. Additional in-flight loads can increase release latency. Tune prefetch with required ordering enabled, and judge its net benefit by complete-step latency.

Account for the cost of earlier admission. If earlier admission requires smaller CTAs, weigh the operator slowdown against the overlap gained. Validate the full sequence: resources held by an early successor can delay later work.

Check wave boundaries before reserving resources. Reserving communication SMs can add a GEMM wave that costs more than the overlap saves. Compare independently tuned joint and separate schedules. For partial-wave packing, require ready work and spare bottleneck capacity: idle SMs alone are insufficient.

Related Work

Persistent execution and graph scheduling. Persistent threads and task-queue frameworks established device-side work distribution before LLM inference [9, 33] (Zheng et al. 2017). Neural-network systems apply residency in several ways. Persistent RNNs and GRNN retain recurrent weights on chip [6, 11], Deep Voice executes WaveNet generation in one launch [3], and VPPS and Cortex support persistent or compiled execution for dynamic and recursive models [15, 7]. Rammer exposes tasks across operator boundaries to a shared scheduler (Ma et al., 2020). Cocktailer, Souffle, and MonoNN also coordinate larger computation graphs [41, 40] (Zhuang et al. 2024), while RECom and RecFlex apply composition to recommendation workloads [25 b]. These designs coordinate work across operators within one kernel. Our reconstruction isolates the schedule effects for a fixed set of operators and data transfers. The operator-efficiency costs we observe also echo earlier work on divergence and register pressure in path-tracing megakernels [17].

LLM megakernels and cross-kernel execution. TK, MPK, Event Tensor, VDCores, and Triton-distributed provide static, dynamic, and resource-decoupled organizations for LLM execution [31, 5] (Jin et al. 2026; He et al. 2026; Zheng et al. 2025a). Event Tensor compares against engines using Graphs and PDL and studies pipelining, wave quantization, and prefetch. Using per-SM traces and prefetch ablations, we explain how

separate kernels recover early preparation. Resource experiments identify opportunities blocked across kernels.

Dependency release and joint scheduling. CUDA Graphs and PDL expose graph replay and early dependent launch [23 a]. Wireframe and BlockMaestro study related launch and dependency mechanisms [1, 2]. Kernelet co-schedules sliced independent kernels (Zhong & He, 2013). Decomposition, T3, FLUX, TileLink, and Triton-distributed enable fine-grained compute–communication overlap [37, 27, 4 a] (Zheng et al. 2025b). DeepGEMM’s Mega MoE and mKernel extend coordinated schedules across distributed operators [20] (Zhao et al. 2025), and Stream-K addresses wave quantization within an operator [24]. Our measurements examine the compute schedule left after resources are assigned to overlapping roles, and the bandwidth and readiness conditions that limit gains from reducing wave quantization across operators.

On-chip resource management. WarpMan, register-file virtualization, RegMutex, scratchpad sharing, Zorua, and Virtual Thread relax ownership or dispatch granularity [30, 13, 14, 36, 12, 35, 38]. SMK and Warped-Slicer partition SM resources among kernels, while FIBER separates register ownership from thread execution [39, 18] (Xu et al. 2016). The resource capabilities discussed here build on this body of work. The reconstructions motivate a specific dependent-kernel use: admitting a loader with a small initial footprint, handing over capacity during execution, and then obtaining the full compute footprint. Evaluating that interface is a future step beyond the measured constraints.

Conclusion

Our reconstructions show that separate kernels with PDL outperform ordered persistent execution in TK and achieve comparable latency in VDCores. Under whole-operator dependencies, TK’s persistent execution obtains a larger prefetch benefit, but runtime and completion-signal ordering costs outweigh this advantage. Joint scheduling can use idle compute capacity, but resource sharing can add compute waves, and idle SMs need not provide spare bandwidth. Together, these results show how residency can misrepresent useful overlap in both directions: whole-CTA admission can block preparation that fits in unused capacity, while co-resident execution can leave throughput unchanged or slow ongoing work. GPU interfaces should let preparation acquire resources before a full CTA fits and runtimes must also account for how sharing those resources changes the execution of other work.

AI USAGE DISCLOSURE

We used OpenAI Codex and Anthropic Claude Code to assist with literature search, experiment and ablation design, code implementation and debugging, analysis of measured results, and manuscript drafting, revision, and review. Code assistance covered GPU kernels, benchmarking and profiling scripts, and data-analysis and plotting scripts. AI assistance was used iteratively in the reconstruction, resource-admission, and joint-scheduling studies and in preparing their presentation.

The reported performance results come from executed GPU experiments. The authors retain responsibility for the experimental methodology, validation of implementations and measurements, interpretation of results, accuracy of references, and final manuscript.

References

  1. [1]Abdolrashidi, A., Tripathy, D., Belviranli, M. E., Bhuyan, L. N., and Wong, D. Wireframe: supporting data-dependent parallelism through dependency graph execution in gpus. In Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO-50 ’17, pp. 600–611, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450349529. doi: 10.1145/3123939.3123976. URL https://doi.org/10.1145/3123939.3123976.
  2. [2]Abdolrashidi, A., Esfeden, H. A., Jahanshahi, A., Singh, K., Abu-Ghazaleh, N., and Wong, D. Blockmaestro: enabling programmer-transparent task-based execution in gpu systems. In Proceedings of the 48th Annual International Symposium on Computer Architecture, ISCA ’21, pp. 333–346. IEEE Press, 2021. ISBN 9781450390866. doi: 10.1109/ISCA52012.2021.00034. URL https://doi.org/10.1109/ISCA52012.2021.00034.
  3. [3]Arik, S. Ö., Chrzanowski, M., Coates, A., Diamos, G., Gibiansky, A., Kang, Y., Li, X., Miller, J., Ng, A., Raimon, J., Sengupta, S., and Shoeybi, M. Deep voice: real-time neural text-to-speech. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, pp. 195–204. JMLR.org, 2017.
  4. [4]Chang, L.-W., Bao, W., Hou, Q., Jiang, C., Zheng, N., Zhong, Y., Zhang, X., Song, Z., Yao, C., Jiang, Z., Lin, H., Jin, X., and Liu, X. Flux: Fast software-based communication overlap on gpus through kernel fusion, 2024. URL https://arxiv.org/abs/2406.06858.
  5. [5]Cheng, X., Zhang, Z., Zhou, Y., Ji, J., Jiang, J., Zhao, Z., Xiao, Z., Ye, Z., Huang, Y., Lai, R., Jin, H., Hou, B., Wu, M., Dong, Y., Yip, A., Ye, Z., Wang, S., Yang, W., Miao, X., Chen, T., and Jia, Z. MPK: A compiler and runtime for Mega-Kernelizing tensor programs. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26), pp. 1909–1926, Seattle, WA, July 2026. USENIX Association. ISBN 978-1-939133-55-7. URL https://www.usenix.org/conference/osdi26/presentation/cheng.arxiv.org/abs/2512.22219
  6. [6]Diamos, G., Sengupta, S., Catanzaro, B., Chrzanowski, M., Coates, A., Elsen, E., Engel, J., Hannun, A., and Satheesh, S. Persistent rnns: Stashing recurrent weights on-chip. In Balcan, M. F. and Weinberger, K. Q. (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 2024–2033, New York, New York, USA, 20–22 Jun 2016. PMLR. URL https://proceedings.mlr.press/v48/diamos16.html.
  7. [7]Fegade, P., Chen, T., Gibbons, P., and Mowry, T. Cortex: A compiler for recursive deep learning models. In Smola, A., Dimakis, A., and Stoica, I. (eds.), Proceedings of Machine Learning and Systems, volume 3, pp. 38–54, 2021. URL https://proceedings.mlsys.org/paper_files/paper/2021/file/eca986d585a03890a412587a2f5ccb43-Paper.pdf.arxiv.org/abs/2011.01383
  8. [9]Gupta, K., Stuart, J. A., and Owens, J. D. A study of persistent threads style gpu programming for gpgpu workloads. In 2012 Innovative Parallel Computing (InPar), pp. 1–14, 2012. doi: 10.1109/InPar.2012.6339596.DOI
  9. [11]Holmes, C., Mawhirter, D., He, Y., Yan, F., and Wu, B. Grnn: Low-latency and scalable rnn inference on gpus. In Proceedings of the Fourteenth EuroSys Conference 2019, EuroSys ’19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450362818. doi: 10.1145/3302424.3303949. URL https://doi.org/10.1145/3302424.3303949.
  10. [12]Jatala, V., Anantpur, J., and Karkare, A. Scratchpad sharing in gpus. ACM Trans. Archit. Code Optim., 14(2), May 2017. ISSN 1544-3566. doi: 10.1145/3075619. URL https://doi.org/10.1145/3075619.
  11. [13]Jeon, H., Ravi, G. S., Kim, N. S., and Annavaram, M. Gpu register file virtualization. In Proceedings of the 48th International Symposium on Microarchitecture, MICRO-48, pp. 420–432, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450340342. doi: 10.1145/2830772.2830784. URL https://doi.org/10.1145/2830772.2830784. V., Mowry, T. C., Jia, Z., and Chen, T. Event tensor: A unified abstraction for compiling dynamic megakernel. In Ninth Conference on Machine Learning and Systems, 2026. URL https://openreview.net/forum?id=PJqfIhAbUha.
  12. [14]Khorasani, F., Asghari Esfeden, H., Farmahini-Farahani, A., Jayasena, N., and Sarkar, V. Regmutex: Inter-warp gpu register time-sharing. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), pp. 816–828, 2018a. doi: 10.1109/ISCA.2018.00073.
  13. [15]Khorasani, F., Esfeden, H. A., Abu-Ghazaleh, N., and Sarkar, V. In-register parameter caching for dynamic neural nets with virtual persistent processor specialization. In Proceedings of the 51st Annual IEEE/ACM International Symposium on Microarchitecture, MICRO-51, pp. 377–389. IEEE Press, 2018b. ISBN 9781538662403. doi: 10.1109/MICRO.2018.00038. URL https://doi.org/10.1109/MICRO.2018.00038.
  14. [16]Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, pp. 611–626, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702297. doi: 10.1145/3600006.3613165. URL https://doi.org/10.1145/3600006.3613165.
  15. [17]Laine, S., Karras, T., and Aila, T. Megakernels considered harmful: wavefront path tracing on gpus. In Proceedings of the 5th High-Performance Graphics Conference, HPG ’13, pp. 137–143, New York, NY, USA, 2013. Association for Computing Machinery. ISBN 9781450321358. doi: 10.1145/2492045.2492060. URL https://doi.org/10.1145/2492045.2492060.
  16. [18]Liu, Z., Leng, J., Zhou, Y., Ding, Y., Zhu, G., Huang, Y., Jin, C., Zhang, C., Sun, S., Feng, Y., Wu, A., Guo, M., Weng, J., Tu, J., and Wang, J. A thread-register decoupled gpu execution model for efficient tensor computation, 2026. URL https://arxiv.org/abs/2608.19628.
  17. [20]Mao, Z., Zhang, Y., Chew, S. W., Ma, S., Raiciu, C., Zhou, Y., Shenker, S., and Stoica, I. mker­nel: Fast multi-gpu, multi-node fused kernels, 2026. URL https://arxiv.org/abs/2609.13585.
  18. [21]NVIDIA Corporation. Programmatic dependent launch and synchronization. CUDA Programming Guide, 2026a. URL https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/programmatic-dependent-launch.html. Accessed: 2026-10-21.
  19. [22]NVIDIA Corporation. Parallel thread execution ISA, version 9.3. CUDA Toolkit 13.3 documentation, 2026b. URL https://docs.nvidia.com/cuda/archive/13.3.0/parallel-thread-execution/index.html. Accessed: 2026-10-21.
  20. [23]NVIDIA Corporation. CUDA Graphs. CUDA Programming Guide, 2026c. URL https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/cuda-graphs.html. Accessed: 2026-10-08.
  21. [24]Osama, M., Merrill, D., Cecka, C., Garland, M., and Owens, J. D. Stream-k: Work-centric parallel decomposition for dense matrix-matrix multiplication on the gpu, 2023. URL https://arxiv.org/abs/2301.03598.
  22. [25]Pan, Z., Zheng, Z., Zhang, F., Wu, R., Liang, H., Wang, D., Qiu, X., Bai, J., Lin, W., and Du, X. Recom: A compiler approach to accelerating recommendation model inference with massive embedding columns. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 4, ASPLOS ’23, pp. 268–286, New York, NY, USA, 2024a. Association for Computing Machinery. ISBN 9798400703942. doi: 10.1145/3623278.3624761. URL https://doi.org/10.1145/3623278.3624761.
  23. [27]Pati, S., Aga, S., Islam, M., Jayasena, N., and Sinclair, M. D. T3: Transparent tracking & triggering for fine-grained overlay of compute & collectives. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS ’24, pp. 1146–1164, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703850. doi: 10. 1145/3620665.3640410. URL https://doi.org/10.1145/3620665.3640410.
  24. [29]Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. Flashattention-3: Fast and accurate attention with asynchrony and low-precision, 2024. URL https://arxiv.org/abs/2407.08608.
  25. [30]Xiang, P., Yang, Y., and Zhou, H. Warp-level divergence in gpus: Characterization, impact, and mitigation. In 2014 IEEE 20th International Symposium on High Performance Computer Architecture (HPCA), pp. 284–295, 2014. doi: 10.1109/HPCA.2014.6835939.DOI
  26. [31]Spector, B., Juravsky, J., Sul, S., Dugan, O., Lim, D., Fu, D., Arora, S., and Ré, C. Look ma, no bubbles! designing a low-latency megakernel for llama-1b. Hazy Research Blog, May 2025. URL https://hazyresearch.stanford.edu/blog/2025-05-27-no-bubbles.
  27. [33]Steinberger, M., Kenzel, M., Boechat, P., Kerbl, B., Dokter, M., and Schmalstieg, D. Whippletree: task-based scheduling of dynamic workloads on the gpu. ACM Trans. Graph., 33(6), November 2014. ISSN 0730-0301. doi: 10.1145/2661229.2661250. URL https://doi.org/10.1145/2661229.2661250.
  28. [34]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., Yang, K., Yu, L., Deng, L., Li, M., Xue, M., Li, M., Zhang, P., Wang, P., Zhu, Q., Men, R., Gao, R., Liu, S., Luo, S., Li, T., Tang, T., Yin, W., Ren, X., Wang, X., Zhang, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Zhang, Y., Wan, Y., Liu, Y., Wang, Z., Cui, Z., Zhang, Z., Zhou, Z., and Qiu, Z. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.
  29. [35]Vijaykumar, N., Hsieh, K., Pekhimenko, G., Khan, S., Shrestha, A., Ghose, S., Jog, A., Gibbons, P. B., and Mutlu, O. Zorua: A holistic approach to resource virtualization in gpus. In 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 1–14, 2016. doi: 10.1109/MICRO.2016.7783718.DOI
  30. [36]Yang, Y., Xiang, P., Mantor, M., Rubin, N., and Zhou, H. Shared memory multiplexing: a novel way to improve gpgpu throughput. In Proceedings of the 21st International Conference on Parallel Architectures and Compilation Techniques, PACT ’12, pp. 283–292, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 9781450311823. doi: 10.1145/2370816.2370858. URL https://doi.org/10.1145/2370816.2370858.
  31. [37]Wang, S., Wei, J., Sabne, A., Davis, A., Ilbeyi, B., Hechtman, B., Chen, D., Murthy, K. S., Maggioni, M., Zhang, Q., Kumar, S., Guo, T., Xu, Y., and Zhou, Z. Overlap communication with dependent computation via decomposition in large deep learning models. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, ASPLOS 2023, pp. 93–106, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450399159. doi: 10.1145/3567955.3567959. URL https://doi.org/10.1145/3567955.3567959.
  32. [38]Yoon, M. K., Kim, K., Lee, S., Ro, W. W., and Annavaram, M. Virtual thread: Maximizing thread-level parallelism beyond gpu scheduling limit. In 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), pp. 609–621, 2016. doi: 10.1109/ISCA.2016.59.DOI
  33. [39]Wang, Z., Yang, J., Melhem, R., Childers, B., Zhang, Y., and Guo, M. Simultaneous multikernel gpu: Multi-tasking throughput processors via fine-grained sharing. In 2016 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 358–369, 2016. doi: 10.1109/HPCA.2016.7446078.DOI
  34. [40]Xia, C., Zhao, J., Sun, Q., Wang, Z., Wen, Y., Yu, T., Feng, X., and Cui, H. Optimizing deep learning inference via global analysis and tensor expressions. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, ASPLOS ’24, pp. 286–301, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703720.DOI
  35. [41]Zhang, C., Ma, L., Xue, J., Shi, Y., Miao, Z., Yang, F., Zhai, J., Yang, Z., and Yang, M. Cocktailer: Analyzing and optimizing dynamic control flow in deep learning. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), pp. 681–699, Boston, MA, July 2023. USENIX Association. ISBN 978-1-939133-34-2. URL https://www.usenix.org/conference/osdi23/presentation/zhang-chen.

Paper details

vlnc:W_ajmqhzrv
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Codex, Claude Code

View statement
Looking for related papers…
Contents