betaFeedback

Reading v1View latest version

WorksComputer Science

Kernel Performance Engineering under Physical Non-Idealities

Abstract

Physical non-idealities in integrated circuit (IC) design and fabrication are concealed by conventional hardware abstractions, yet they can limit kernel performance and obscure optimization opportunities. Using distributed shared memory (DSMEM) on NVIDIA H100 GPUs as a case study, we show how floorsweeping restricts hardware resources available for clustered execution, potentially offsetting the benefits of reduced data movement enabled by DSMEM. Guided by these insights, we develop a wave-aware clustering strategy that accounts for these resource constraints to improve kernel performance. Evaluations using a global-load-and-broadcast microbenchmark, GEMM, and the Hopper version of FlashAttention 4 demonstrate the performance implications of these constraints, with wave-aware clustering improving FlashAttention 4 throughput by up to 5.30% over its baseline. We further show that, under thermal design power (TDP) limits, reduced data movement enables higher Streaming Multiprocessor (SM) frequencies through Dynamic Voltage and Frequency Scaling (DVFS), explaining otherwise unexpected throughput improvements. Extending this analysis beyond GPUs, we show that power constraints also limit throughput scaling on Tenstorrent's Wormhole accelerator, highlighting the broader importance of accounting for physical non-idealities when analyzing and optimizing kernel performance.

Introduction

The fundamental idea of abstraction is pervasive across computer systems. By enabling the philosophy of separation of concerns, abstractions have facilitated the development of software for increasingly complex hardware. Kernel performance engineers have leveraged this principle to simplify their view of the underlying microarchitecture. At the highest level of abstraction, kernels are written in high-level languages, delegating the complexity of the architecture to the compiler. This abstracted view reduces its complexity and improves productivity, but it also restricts the maximum performance achievable by a kernel. As performance demands increase, kernel performance engineers are compelled to break this level of abstraction and work directly with the hardware’s Instruction Set Architecture (ISA). Further, in performance-sensitive domains such as AI, maximizing kernel performance requires engineers to go beyond the ISA and develop a deep understanding of the underlying microarchitecture. Even at this level, physical non-idealities in integrated circuit (IC) design, such as floorsweeping and thermal design power (TDP) limits, can cause the microarchitecture to behave differently under certain conditions. These physical effects are concealed by conventional microarchitectural abstractions, limiting kernel performance engineers’ ability to anticipate their impact and extract peak performance.

Physical non-idealities in hardware are abstracted away by IC vendors when they disclose their microarchitecture. This abstraction aims to provide consistent performance and programmability across different units of a product. Floorsweeping masks manufacturing defects by disabling faulty resources, allowing ICs with different numbers of functional resources to be classified into stock-keeping units (SKUs). Each SKU represents a product configuration with a specified set of functional resources. For example, ICs manufactured with the Hopper GH100 design are categorized into the H100 SXM5 and H100 PCIe SKUs based on their fabrication defects. Further, TDP limits specify a power budget for the IC; however, their impact on operating frequency during kernel execution remains opaque. Similar to higher-level abstractions, this abstraction can also conceal potential kernel optimization opportunities and contribute to uninterpretable performance variations. In this paper, we demonstrate how physical non-idealities affect kernel performance and how understanding these effects can inform kernel performance engineering and explain seemingly unpredictable performance variations, using the distributed shared memory (DSMEM) architectural feature of NVIDIA H100 Graphics Processing Units (GPUs) as a case study.

DSMEM is an architectural feature introduced in the Hopper GPU architecture ([4]). It relies on grouping multiple thread blocks into a unit called a thread block cluster (cluster), which ensures that the thread blocks are mapped to Streaming Multiprocessors (SMs) within a single Graphics Processing Cluster (GPC). This enables each thread block to access the shared memory of other thread blocks within its cluster. The number of thread blocks in a cluster is referred to as the cluster size, and the process of grouping thread blocks into clusters is called clustering. Hopper GPUs support cluster sizes of 2, 4, and 8, with 16 being optional. For a cluster size of 2, thread blocks are always mapped to SMs within the same Texture Processing Cluster (TPC). Hence, we refer to clusters of size 2 as TPC-level clusters and clusters of sizes 4 and 8 as GPC-level clusters.

This architectural feature enables a thread block to load data and broadcast it with other thread blocks within the same cluster when they require the same data, a common scenario in tiled kernels. Interestingly, we observe that this feature is not utilized in modern kernels such as FlashAttention [8, 25, 30]. We also observe that the expected performance gains do not materialize at GPC-level cluster sizes, as shown in Figure 1. A kernel performance engineer familiar with the microarchitecture might attribute this performance degradation entirely to synchronization overheads. However, accounting for physical non-idealities reveals that synchronization overhead is only one of the contributing factors.

Throughput and percentage increase in throughput compared to cluster size 1 across sequence lengths for 128 and 256 head dimensions for FlashAttention 4 Hopper version

Figure 1. Throughput and percentage increase in throughput compared to cluster size 1 across sequence lengths for 128 and 256 head dimensions for FlashAttention 4 Hopper version.

In this paper, we demonstrate the importance of breaking another layer of abstraction by accounting for physical non-idealities in IC design and fabrication when engineering high-performance kernels. First, we show how floorsweeping limits the number of SMs available for clustered execution. Although the NVIDIA H100 GPU exposes 132 SMs, our experiments show that only 120 SMs can be used concurrently at GPC-level cluster sizes. We refer to this number as

the active SM limit. We explain this reduction using the organization of the remaining functional resources into virtual GPCs and the constraints this organization imposes on clustered execution. Consequently, using a GPC-level cluster size can increase the number of execution waves required by a kernel, offsetting the benefits of reduced redundant data loads through DSMEM and ultimately reducing kernel throughput.

With this understanding, we characterize the conditions under which the reduced active SM limit introduces additional execution waves. We define a wave-equalized kernel launch as one whose thread-block count results in the same number of execution waves under both the full and reduced active SM limits. We derive the conditions for such launches and evaluate their performance implications using a global-load-and-broadcast microbenchmark, GEMM, and the Hopper version of FlashAttention 4. With insights from this, we introduce the wave-aware clustering strategy to select between TPC clustering and GPC clustering for a given kernel. We achieve a throughput improvement of up to 5.30% with wave-aware clustering against FlashAttention 4 baseline, which uses cluster size 1.

Furthermore, we explain the discrepancies and seemingly unpredictable variability in kernel performance with DSMEM through a principled analysis of TDP limits and the resulting fluctuations in SM frequency. For kernels operating at the GPU’s power limit, reduced data movement lowers the power consumed by data transfers, allowing the GPU to sustain a higher SM frequency through Dynamic Voltage and Frequency Scaling (DVFS). We demonstrate this effect by incorporating global loading and broadcasting of shared key and value tiles into the Hopper version of FlashAttention 4. This demonstrates the benefit of considering physical non-idealities in kernel performance engineering. Moreover, we evaluate the impact of TDP limits on Tenstorrent’s Wormhole accelerator, demonstrating how power constraints reduce operating frequency and limit throughput scaling as more cores become active. These results demonstrate that the impact of physical non-idealities on kernel performance extends beyond GPUs.

Background and Related Work

Distributed shared memory in NVIDIA Hopper GPU architecture

DSMEM is an architectural feature introduced in the NVIDIA Hopper GPU architecture. It connects the shared memories of multiple SMs into a unified memory space through a Network-on-Chip (NoC) (Prabhakar et al., [21]). Thread blocks can be grouped into a unit called a thread block cluster (cluster), with supported cluster sizes of 2, 4, and 8. However, not all SMs in the GPU can be clustered together, as cluster placement is constrained by the physical organization of the SMs. As shown in Figure 2, the NVIDIA Hopper GPU organizes SMs into pairs within units called TPCs. Nine TPCs form a GPC, and the GPU contains eight such GPCs. Each GPC has a NoC connecting the shared memories of the SMs within it. Consequently, only SMs within the same GPC can participate in a cluster. Furthermore, clusters of size 2 always group SMs within a single TPC. Therefore, we refer to clusters of size 2 as TPC-level clusters and clusters of sizes 4 and 8 as GPC-level clusters. While clustered execution has the potential to reduce data movement within the GPU, it also introduces additional synchronization overhead and scheduling constraints. In this work, we use DSMEM to demonstrate that further trade-offs arise from this architectural feature, which become apparent only when physical non-idealities in IC fabrication are taken into account.

Layout of SMs in the NVIDIA Hopper GPU architecture

Figure 2. Layout of SMs in the NVIDIA Hopper GPU architecture

Impact of physical non-idealities on kernel performance

Prior studies have reported that DSMEM throughput decreases with increasing GPC-level cluster sizes [16 a]. This behavior has been attributed to contention in the crossbar connecting the L1 caches/shared memories of SMs within a GPC [16, 21]. Additionally, the number of SMs available for clustered execution has been observed to decrease at GPC-level cluster sizes [16 NVIDIA, a]. Floorsweeping has been identified as a potential cause of these resource restrictions [5] (He et al. 2026). Concurrent work by Fan et al. (Fan et al., [11] characterizes the physical SM topology and examines its implications for thread-block cluster scheduling, particularly in the context of MIG partitioning (NVIDIA, c) and Green Contexts (NVIDIA, b). Beyond the resource restrictions imposed by floorsweeping, the performance of clustered kernels also depends on the operating frequency sustained within the available power budget. Existing Hopper microbenchmarking studies have characterized DSMEM latency and throughput and separately reported power-induced frequency reductions during tensor-core execution. Studies of GPU power capping and frequency scaling have also characterized performance and energy trade-offs, but have not isolated how DSMEM-based reductions in data movement affect operating frequency [29, 2]. More broadly, the effects of manufacturing-induced process variations and power constraints on CPU and GPU performance have been studied at the microarchitectural level [14, 23, 3, 24]. In this work, we characterize the resource restrictions imposed by floorsweeping on clustered execution and their performance implications. We derive the conditions for wave-equalized kernel launches that avoid additional execution waves despite these restrictions. Furthermore, we investigate how TDP-induced frequency scaling affects clustered kernel performance and explain its combined impact with floorsweeping-related resource restrictions.

Methodology

We evaluate four NVIDIA H100 SXM5 GPUs to assess the impact of physical non-idealities and ensure that the observed effects are not specific to a single GPU. The four GPUs are located in geographically distinct regions to reduce the likelihood that the observations are specific to a particular manufacturing batch. Throughout this paper, NVIDIA H100 refers to the NVIDIA H100 SXM5 SKU unless otherwise specified. We use four CUDA kernels for our experiments: a global-load-and-broadcast microbenchmark, a GPU skyline reverse-engineering kernel, GEMM, and Flash Attention. All kernels are compiled using CUDA 13.0 with the sm_90a compilation target. Each kernel is executed for multiple warm-up and timed iterations, with the exact counts reported in the corresponding experimental sections. All profiling data are collected using NVIDIA Nsight Compute (NCU).

For GEMM, we use the Hopper warp-specialized GEMM kernel from the CUTLASS 4.8.0 library [27]. We configure it with three pipeline stages and use a non-persistent execution mode to ensure that, for a fixed problem size, the number of thread blocks depends only on the tile dimensions. For FlashAttention, we use the FlashAttention 4 Hopper forward kernel (Dao-Abdul, 2026). The kernel takes query (QQ), key (KK), and value (VV) matrices as inputs. With a tiling configuration of BM×BNB_M \times B_N, each thread block processes BMB_M rows of QQ and iteratively loads tiles of BNB_N rows from KK and VV to perform the attention operation. Since multiple thread blocks processing different query tiles require the same KK and VV tiles, there is an opportunity to reduce redundant data loads through a global-load-and-broadcast mechanism using DSMEM. We modify the FlashAttention codebase to incorporate this functionality and evaluate it using non-causal multi-head attention (MHA).

Demystifying Floorsweeping Effects on Clustered Execution

Distributed shared memory enables a thread block to initiate a single global-memory load and broadcast the data into the shared memories of multiple thread blocks within a cluster, eliminating redundant data loads. This is supported by the network connecting the shared memories of SMs within a GPC ([21]). When thread blocks are grouped into a cluster, they are scheduled on SMs within the same GPC. Such data sharing can reduce L2 cache traffic and the cumulative data loading time across thread blocks. However, these expected benefits do not consistently translate into improved kernel performance, as shown in Figure 1. To investigate the underlying causes, we develop a microbenchmark to isolate data loading performance.

Microbenchmarking global load and broadcast

In this microbenchmark, the CPU generates 1 MB of pseudorandom FP32 values, which are then copied to the GPU’s global memory (HBM). We launch the kernel with 256 thread blocks and evaluate cluster sizes of 1, 2, 4, and 8 in separate runs. For cluster sizes greater than 1, the kernel loads and broadcasts the data in successive 128 KB chunks using the two patterns illustrated in Figure 3. In the single-block broadcast pattern, the first thread block in each cluster initiates a Tensor Memory Accelerator (TMA) transfer to load the data into the shared memory of every thread block within the cluster. In contrast, in the all-blocks broadcast pattern, each thread activates a TMA transfer for a distinct 1/(cluster size) portion of the data, broadcasting its portion into the shared memory of every thread block within the cluster.

Global-load and broadcast patterns visualized for cluster size 4

Figure 3. Global-load and broadcast patterns visualized for cluster size 4

The kernel is executed for 50 warm-up iterations, followed by 500 timed iterations to calculate the mean latency. Furthermore, the GPU’s SM and memory clock frequencies are fixed at their maximum values of 1980 MHz and 2619 MHz, respectively. At cluster sizes greater than 1, the all-blocks broadcast pattern issues the same number of TMA transfers as the cluster size 1 configuration, but with smaller transfer sizes than the single-block broadcast pattern. Since smaller TMA transfers can reduce effective memory bandwidth ([15]), the single-block broadcast pattern achieves lower latency than the all-blocks broadcast pattern, as shown in Figure 4.

Mean latency of 256 thread blocks loading 1 MB of data

Figure 4. Mean latency of 256 thread blocks loading 1 MB of data, in 128 KB chunks, from global memory into their shared memories with broadcasting under different cluster sizes

The microbenchmark results also reveal an unexpected trend at GPC-level cluster sizes, similar to that observed in Fig.

  1. Since GPC-level cluster sizes progressively reduce the amount of data transferred from the L2 cache to shared memory, they are expected to achieve lower mean latency than the TPC-level cluster size. However, this expected improvement does not materialize due to the reduced number of active SMs available for GPC-level clustered execution.

Active SM limits at different cluster sizes

To determine the number of active SMs at different cluster sizes, we record the logical SM IDs used by the microbenchmark kernel during each iteration. The logical SM ID is obtained by having the first thread of each thread block read the %smid register, which uniquely identifies the SM executing the thread block. Table 1 reports the number of unique SMs observed within individual iterations and across all iterations for each cluster size.

Unique SMs
Cluster SizePer Iter.All Iters.
1132132
2132132
4120124
8120124

Table 1. Number of unique active SMs at each cluster size

At cluster sizes 1 and 2, the kernel utilizes all 132 SMs in the NVIDIA H100 GPU. However, at GPC-level cluster sizes, only 124 SMs are enabled for clustered execution, of which at most 120 can be used concurrently. Surprisingly, these restrictions arise from floorsweeping.

Effect of floorsweeping on active SM limit

With aggressive transistor scaling, modern AI accelerators are now fabricated with diminishing physical margins, making manufacturing defects a pervasive and critical problem. Conventionally, yield is modeled using the negative binomial (NB) model as the probability of observing zero defects on the die: Y=(αα+λ0)αY = \left(\frac{\alpha}{\alpha+\lambda_0}\right)^\alpha, where λ0=D0×A\lambda_0 = D_0 \times A denotes the expected number of defects per chip, based on the defect density D0D_0 and chip area AA. α\alpha is the clustering parameter controlling the variance of the distribution. For a defect density of D0=0.15±0.075D_0 = 0.15 \pm0.075 per cm2^2 and a GPU die approaching the reticle limit of manufacturing (∼700\sim700–800800 mm2^2), the expected number of defects per chip is approximately one. Under the conventional notion of yield, any chip containing a defect would need to be discarded. However, modern architectures generally incorporate redundant resources combined with binning, allowing partially defective chips to be sold at a lower performance and price target. Defective resources are disabled or bypassed through floorsweeping; together with binning, this creates multiple stock-keeping units (SKUs), which can be thought of as virtually iso-performant buckets with a consistent resource profile. The primary purpose of floorsweeping is to ensure a virtual environment where any chip in the same SKU behaves the same for kernels and software.

Floorsweeping is inherently difficult because the locations of faults can break the homogeneity of the chip. The clustering parameter in the NB model controls the variance of defect counts across chips which stems from the fact that in real-world manufacturing, some chips incur substantially more defects than others. This is also useful for yield, as more defective chips can be binned into lower-performance regimes while less faulty chips remain available for higher SKUs. However, even chips with the same number of defects can exhibit different spatial fault patterns. For instance, in a GPU, some GPCs may lose more TPCs than other GPCs. Fixed physical GPC boundaries can therefore leave chips with the same aggregate resource count with substantially different distributions of active SMs per GPC. Ideally, a uniform SKU would expose a consistent resource distribution and hierarchy despite these physical differences.

NVIDIA’s patent (Duluk Jr et al., [10]) introduces a Skyline as the solution to this problem, which specifies a common distribution of TPCs across virtual GPCs (vGPCs) for a SKU. Skyline selection, illustrated in Figure 5, based on manufacturing yield awareness determines which physical chip configurations can share a common virtual profile. In other words, the goal is to bring as many chips with varied defect patterns as possible under a common virtual abstraction in the SKU while minimizing unnecessary loss of performance or functionality. Accommodating more configurations within the same Skyline, however, can fragment the resources available for clustered execution on less defective chips and may require disabling otherwise healthy SMs simply to match the SKU. The Skyline essentially fixes the number of TPCs exposed within each virtual GPC, i.e., the heights of the towers, while allowing the underlying physical distribution to vary across chips. Each chip maps its surviving TPCs onto the profile, with additional functional TPCs exposed as single-TPC vGPCs called singletons.

Distinct fault patterns mapped to the same skyline

Figure 5. Distinct fault patterns mapped to the same skyline by floorsweeping ([10])

For example, the Skyline 6/8/8/8/9/9/9/9/1×2 represents 68 TPCs across eight multi-TPC vGPCs and two singletons. This organization enables scheduling and context migration across chips with different physical defect patterns within the same SKU. However, singletons can only execute independent thread blocks or clusters that fit within one TPC, i.e., TPC-level clusters; they cannot participate in multi-TPC clusters, i.e., GPC-level clusters. Thus, retaining a functional resource does not necessarily retain its availability for clustered execution. Through this work, we first demonstrate how we can find the skylines for NVIDIA’s Hopper SKUs, and then we show how these constraints affect a performance engineer’s ability to use the chip effectively.

Reverse engineering the skyline

To reverse-engineer the Skyline of a GPU, we use a CUDA kernel that records the SM IDs on which its thread blocks are scheduled. The kernel is launched with 1024 thread blocks, each containing 1024 threads, with the number of thread blocks well exceeding the 132 SMs of the NVIDIA H100. The first thread of each thread block reads the %smid register and stores the resulting ID in an array in global memory. We launch this kernel with increasing cluster sizes and group the SM IDs belonging to the same cluster into sets, leveraging the fact that consecutive thread blocks are grouped together during clustering. We then merge sets that share one or more SM IDs to infer the underlying grouping of SMs. This process is repeated for 1000 iterations to account for variations in thread-block scheduling and improve the coverage of observed mappings. We identify 12 vGPCs in the NVIDIA H100 GPU, with varying numbers of SMs, as shown in Table 2. We assign vGPC IDs in ascending order based on the lowest SM ID within each vGPC. Notably, the GPU contains only eight physical GPCs, despite exposing 12 vGPCs.

TypeCountvGPC IDs
2 SMs vGPC (1 TPC)48,9,10,11
8 SMs vGPC (4 TPCs)10
16 SMs vGPC (8 TPCs)51,2,3,4,5
18 SMs vGPC (9 TPCs)26,7

Table 2. Skyline for NVIDIA H100

Figure 5 illustrates the potential organization of two defective dies with different fault patterns and their corresponding mappings to the H100 skyline of 9/9/8/8/8/8/8/4/1/1/1/1. In the first die, three physical GPCs each contain one faulty TPC, while another contains three faulty TPCs. In the second die, five faulty TPCs are concentrated within the same physical GPC. Despite these differences, floorsweeping allows both dies to expose the same virtual resource profile. Each multi-TPC vGPC is formed from surviving TPCs within a physical GPC, while some remaining functional TPCs are exposed as singletons to match the skyline. This mapping provides a consistent resource hierarchy across the SKU, but may require disabling otherwise functional TPCs and fragmenting physical GPCs into smaller virtual groups. Consequently, even functional TPCs retained as singletons are unavailable for multi-TPC clustered execution. The presence of a vGPC with only four TPCs in the H100 skyline suggests that a substantial number of dies may have faults concentrated within one physical GPC, in line with the earlier discussed clustering, motivating a smaller virtual group to accommodate these defect patterns. While the vGPC abstraction provides uniformity for general GPU usage, these restrictions limit a performance engineer’s ability to use the available compute capacity for topology-dependent features such as DSMEM-based clustering.

Impact of skyline on the active SM limit

The SMs forming a cluster must belong to the same vGPC. Since the Skyline contains four vGPCs with only two SMs each, these vGPCs cannot support clusters of size 4 or 8. Consequently, these eight SMs are unavailable for GPC-level clustered execution, reducing the number of available SMs from 132 to 124. Furthermore, two vGPCs contain 18 SMs each, leaving two SMs unassigned in each vGPC when partitioned into clusters of size 4 or 8. This further reduces the active SM limit to 120 at GPC-level cluster sizes. Since the unassigned SMs can vary between iterations, this explains why the microbenchmark records only 120 unique SM ID per iteration but 124 across all iterations, as shown in Table 1.

Beyond restricting the availability of SMs, the underlying resource organization also influences which SMs are frequently grouped into clusters. As shown in Figure 6, SMs with neighboring logical SM IDs are frequently clustered together. The resulting limit of 120 concurrently active SMs at GPC-level cluster sizes can increase the number of execution waves, thereby reducing kernel performance. The following section characterizes the conditions under which this additional-wave penalty occurs and identifies configurations in which the execution wave count remains unchanged.

Probability of y-SM ID in cluster given x-SM ID in cluster for cluster sizes 4 and 8

Figure 6. Pr(y-SM ID in cluster | x-SM ID in cluster) when a kernel is launched with cluster size 4 and 8

Navigating Active SM Limit Penalties

The active SM limit imposed by floorsweeping can increase the number of execution waves required by a kernel, as the same number of thread blocks must be scheduled across fewer SMs. An execution wave consists of the thread blocks that can execute concurrently on the GPU. The number of execution waves influences kernel execution time, as additional waves require successive scheduling of thread blocks. Consequently, the additional-wave penalty caused by the reduced active SM limit can be avoided when a kernel launch requires the same number of execution waves under both the full 132-SM and reduced 120-SM limits. We refer to such a launch as a wave-equalized kernel launch. The condition for a wave-equalized kernel launch is formalized in Eq. (1). We see popular kernels such as GEMM and FlashAttention often achieve their highest throughput by maximizing data reuse through large tile shapes that fit within the available shared memory and register capacity. The resulting resource requirements typically limit occupancy to one thread block per SM. This restricted occupancy exacerbates the impact of the reduced active SM limit because it minimizes the number of thread blocks in a wave.

⌈# Thread Blocks132⌉=⌈# Thread Blocks120⌉(1)\left\lceil\frac{\#\ \text{Thread Blocks}}{132} \right\rceil= \left\lceil\frac{\#\ \text{Thread Blocks}}{120} \right\rceil \tag*{(1)}

Wave-equalized global load and broadcast within clusters

To examine how wave equalization influences the benefits of global load and broadcast, we compare the original 256-thread-block microbenchmark from § 4 with a 240-thread-block wave-equalized configuration satisfying Eq. (1). Figure 7 shows the change in latency due to clustering, relative to the cluster size 1 baseline, for both configurations. At GPC-level cluster sizes, clustering reduces latency in the wave-equalized 240-block configuration but increases it in the non-wave-equalized 256-block configuration. These results illustrate how additional execution waves can offset the latency benefits of reduced data movement through clustering.

Percentage reduction in mean latency for wave-equalized and non-wave-equalized configurations

Figure 7. Percentage reduction in mean latency of loading 1 MB of data, in 128 KB chunks, from global memory into their shared memories with broadcasting under different cluster sizes.

Wave-equalized GEMM kernel launches

Each GEMM configuration is executed for 100 warm-up iterations, followed by 1000 timed iterations. To evaluate the effect of wave equalization, we first characterize GEMM performance across different cluster sizes. Figure 8 shows the throughput improvement achieved by clustering relative to cluster size 1. We evaluate tile shapes of 128×128128 \times128 and 64×6464 \times64, along with all valid cluster shapes for cluster sizes greater than 1. For each GEMM problem shape and cluster size, we select the configuration that achieves the highest throughput. As shown in Figure 8, the relative throughput benefit of GPC-level clustering increases with matrix size. This is because larger operands lead to more L2 cache misses and consequently more reads from HBM. Broadcasting shared data across thread blocks reduces redundant L2 cache accesses and, as observed in Table 3, HBM reads. The resulting reduction in memory traffic can alleviate memory bottlenecks, potentially improving tensor core utilization and kernel throughput.

Cluster sizeHBM reads (GB)L2 hit rate
1137.6760.18%
269.0471.72%
434.7682.05%
817.5289.61%

Table 3. Size of HBM reads and L2 hit rate for 32768×4096×3276832768 \times4096 \times32768 GEMM with tile size 64×6464 \times64

GEMM throughput improvement across output matrix shapes

Figure 8. M×4096×NM \times4096 \times N GEMM throughput improvement relative to cluster size 1

For the 8192×4096×81928192 \times4096 \times8192 GEMM configuration in Figure 8,

8, GPC-level cluster sizes exhibit significantly lower performance than the TPC-level cluster size. Due to the floorsweeping-induced limit of 120 concurrently active SMs, kernels launched with GPC-level cluster sizes require 12 additional execution waves compared to cluster sizes 1 and 2. To examine the impact of this additional-wave penalty, we evaluate a wave-equalized GEMM configuration.

For a GEMM kernel launch to be wave-equalized, its output dimensions and tile shape must satisfy

⌈MBM⌉⋅⌈NBN⌉1132=⌈MBM⌉⋅⌈NBN⌉1120\left\lceil\frac{M}{B_M}\right\rceil\cdot\left\lceil\frac{N}{B_N}\right\rceil\frac{1}{132} = \left\lceil\frac{M}{B_M}\right\rceil\cdot\left\lceil\frac{N}{B_N}\right\rceil\frac{1}{120}

where M×NM \times N denotes the output matrix dimensions and BM×BNB_M \times B_N denotes the tile shape. We evaluate a 7680 ×\times 4096 ×\times 512 GEMM configuration that satisfies this condition for both evaluated tile shapes and supports all evaluated cluster sizes. Figure 9 shows that GPC-level clustering improves throughput relative to cluster size 1 for this wave-equalized workload. Together, these experiments illustrate how the relative benefits of clustering differ across GEMM workloads with different execution wave counts.

Wave-equalized GEMM throughput improvement

Figure 9. 7680 ×\times 4096 ×\times 512 GEMM throughput improvement relative to cluster size 1

Wave-equalized FlashAttention kernel launches

For a FlashAttention kernel launch to be wave-equalized, it must satisfy Eq. (1). The number of thread blocks in a FlashAttention kernel is given by B×H×⌈SBM⌉B \times H \times\left\lceil\frac{S}{B_M}\right\rceil, where BB is the batch size, HH is the number of attention heads, SS is the sequence length, and BMB_M is the tile size along the query sequence dimension. Accordingly, the wave-equalization condition for FlashAttention can be expressed as

⌈B×H×⌈SBM⌉132⌉=⌈B×H×⌈SBM⌉120⌉\left\lceil\frac{B \times H \times\left\lceil\frac{S}{B_M}\right\rceil}{132}\right\rceil= \left\lceil\frac{B \times H \times\left\lceil\frac{S}{B_M}\right\rceil}{120}\right\rceil

Since each batch and attention head has its own KK and VV matrices, clustering is performed independently for each batch and head. Therefore, to support cluster sizes of up to 8, the additional condition

⌈SBM⌉ mod 8=0\left\lceil\frac{S}{B_M}\right\rceil\bmod8 = 0

must also be satisfied.

To examine how clustering performs across workloads with different execution wave counts, we fix the total number of tokens at 8192, setting the batch size to 8192/S8192/S for a sequence length SS. We fix the head dimension at 256 and evaluate workloads with 16 and 15 attention heads, representing non-wave-equalized and wave-equalized configurations, respectively. Each clustered configuration is compared against the cluster size 1 baseline for the same workload. We evaluate all combinations of BM∈{128,64,32}B_M \in\{128, 64, 32\} and BN∈{80,64,32}B_N \in\{80, 64, 32\}, selecting the tile shape that achieves the highest throughput for each configuration. The maximum value of BNB_N is limited to 80 instead of 128 because a 128 ×\times 128 tile configuration exceeds the available shared memory capacity at a head dimension of 256. Each FlashAttention configuration is executed for 100 warm-up iterations, followed by 1000 timed iterations. Across all evaluated configurations, the 128 ×\times 80 tile shape achieves the highest throughput. With BM=128B_M = 128, the 15-head configurations satisfy the wave-equalization condition for FlashAttention. Figure 10 shows that GPC-level clustering improves FlashAttention throughput relative to cluster size 1 across the evaluated wave-equalized configurations, achieving gains up to 7.32%. These gains exceed those observed in the evaluated non-wave-equalized configurations by a maximum of 12.69 percentage points. With this characterization, we develop a strategy to choose between TPC clustering and GPC clustering for a given kernel by considering its number of execution waves; we call it wave-aware clustering.

FlashAttention throughput improvement across sequence lengths

Figure 10. Percentage increase in throughput over cluster size 1 across sequence lengths for wave-equalized and non-wave-equalized FlashAttention 4 kernel launches

Wave-aware clustering on FlashAttention

FlashAttention 4 kernel made for NVIDIA Hopper GPUs does not use clustering, i.e., uses cluster size 1. We suspect that it is due to the performance variability in clustered execution that is unexplainable from a microarchitectural perspective. As the impact of floorsweeping sheds light on this variation in kernel performance, we propose the wave-aware clustering strategy in Eq. (2), where TB denotes thread blocks, for choosing between TPC-level and GPC-level clustering. To evaluate it, we consider FlashAttention 4 for varying sequence lengths from 2K to 64K, as we see in Figure 11, under two configurations: FA4-H16 and FA4-H15. Both configurations have 256 head dimension and a batch size of 1, while their number of heads are 16 and 15, respectively. As in the previous FlashAttention experiment, the 128 × 80 tile size gave the highest throughput for each configuration, and they are executed for 100 warm-up iterations, followed by 100 timed iterations to measure kernel performance. Moreover, the TPC and GPC clustering conditions represent typical approaches driven by heuristics and profiling of expected workload mix, unaware of the true nature of the non-idealities.

Percentage improvement in throughput over FlashAttention 4 baseline for TPC clustering, GPC clustering, and wave-aware clustering strategies

Figure 11. Percentage improvement in throughput over FlashAttention 4 baseline (uses cluster size 1) for TPC clustering, GPC clustering, and wave-aware clustering strategies.

⌈#TB120⌉−⌈#TB132⌉={0⇒GPC clustering>0⇒TPC clustering(2)\left\lceil\frac{\#\mathrm{TB}}{120} \right\rceil- \left\lceil\frac{\#\mathrm{TB}}{132} \right\rceil= \begin{cases} 0 & \Rightarrow\text{GPC clustering} \\ > 0 & \Rightarrow\text{TPC clustering} \end{cases} \tag*{(2)}

Wave-aware clustering performs better than the FlashAttention 4 baseline that uses cluster size 1 at almost all evaluated kernel configurations and has throughput improvements up to 5.30%. Further, it outperforms strictly using cluster size 2 (TPC clustering) or choosing cluster sizes 4 and 8 (GPC clustering) at all evaluated instances, as we see in Figure 11. The throughput improvements observed with increasing cluster sizes in FlashAttention kernels are not driven by increased tensor core utilization resulting from reduced data movement. Counterintuitively, these improvements arise from the TDP limit imposed during IC design and its impact on operating frequency. We examine this effect in detail in Section 6.

Characterization of wave-equalized kernel launches and wave-aware clustering will help kernel performance engineers interpret performance variations and assess whether clustering is beneficial for a given workload and launch configuration. Moreover, it can also guide the selection of alternative launch geometries where workload constraints permit. These findings further reinforce the importance of looking beyond conventional microarchitectural abstractions and accounting for physical non-idealities in IC fabrication when engineering high-performance kernels.

Demystifying the Impact of Thermal Design Power

Thermal design power (TDP) specifies the amount of heat a cooling system is designed to dissipate during sustained operation. The NVIDIA H100 GPU has a TDP of 700 W, requiring a cooling solution capable of handling this thermal load to avoid performance degradation due to thermal throttling. Since nearly all the electrical power consumed by a GPU is converted into heat, TDP also provides an indication of its sustained power budget. Power consumption consists of two components: static and dynamic power. Static power consumption primarily arises from leakage currents in transistors, whereas dynamic power consumption results from transistor switching activity. Dynamic power typically dominates during compute-intensive workloads and depends on switching activity, operating frequency, and supply voltage. To regulate the power consumption of the GPU, DVFS dynamically adjusts its operating voltage and frequency based on workload demands and power constraints. During periods of low utilization, DVFS can reduce the operating frequency and voltage to lower power consumption. As workload demands increase, the GPU can increase its operating frequency, including the frequency supplied to the SMs, provided sufficient power headroom is available. However, as power consumption approaches the TDP-associated power limit, DVFS may constrain or reduce the operating frequency to maintain power consumption within the permitted budget.

Throughput and percentage increase in throughput over cluster size 1 across sequence lengths for 128 and 256 head dimensions for Flash Attention

Figure 13. Throughput and percentage increase in throughput over cluster size 1 across sequence lengths for 128 and 256 head dimensions for Flash Attention.

Effect of TDP on kernel performance

To evaluate the impact of TDP on kernel performance, we execute FlashAttention kernels with a fixed total token count of 256K, varying the batch size with sequence length. Furthermore, we maintain a constant hidden dimension of 2048 by adjusting the number of attention heads as the head dimension varies. As in the previous FlashAttention experiment, we evaluate all combinations of BM∈{128,64,32}B_M \in\{128, 64, 32\} and BN∈{80,64,32}B_N \in\{80, 64, 32\}, selecting the tile shape that achieves the highest throughput for each configuration. Each configuration is executed for 100 warm-up iterations, followed by 100 timed iterations to measure kernel performance.

Figure 13 shows that the TPC-level cluster size consistently achieves approximately 1.5% and 3% higher throughput than cluster size 1 for head dimensions of 128 and 256, respectively. However, the reduced data movement does not lead to increased tensor core utilization, as shown in Table 4. Table 5 shows that the amount of data read from L2 is approximately halved with each doubling of the cluster size. Nevertheless, tensor core utilization remains high across all cluster sizes. The profiling results indicate lower tensor core utilization at GPC-level cluster sizes because utilization is averaged across all 132 SMs in the GPU. However, the active SM limit restricts concurrent execution to 120

Head dim.Seq. len1248
12816k74.8%74.3%72.6%69.5%
32k76.8%76.2%75.3%72.1%
25616k94.3%94.1%92.5%92.3%
32k95.9%95.8%95.1%95.2%

Table 4. Per SM tensor core utilization for Flash Attention. At cluster sizes 4 and 8, utilization is adjusted to the reduced SM limit ratio.

Metric1248
Mean SM frequency (MHz)1584159316631667
Instruction count (billions)2.973.013.013.01
Tensor core (TC) utilization95.61%95.54%86.05%86.09%
Active SM limit132132120120
Adjusted TC utilization95.61%95.54%94.66%94.70%
L2 read size (GB)54.9335.3518.179.58
Power capping observedYesYesYesYes

Table 5. Metrics collected through NCU profiling for Flash Attention: head dimension 256 and sequence length 32k

SMs, leaving the remaining 12 SMs idle and contributing 0% utilization to the reported average. After accounting for these inactive SMs, the tensor core utilization is comparable across cluster sizes. This indicates that the observed throughput improvements from reduced data movement are not driven by increased tensor core utilization.

We attribute these throughput improvements to the TDP limit and the resulting frequency scaling through DVFS. The example FlashAttention configuration in Table 5 shows that the average SM frequency increases with cluster size. As illustrated in Figure 12, increasing the cluster size reduces redundant data movement within the GPU, consequently lowering the power consumed by data transfers. For kernels operating near the GPU’s power limit, this reduction in power consumption creates additional power headroom, allowing the GPU’s power management system to increase the SM frequency through DVFS while remaining within the TDP limit. Figure 14 shows a strong positive linear relationship between the percentage increases in throughput and SM frequency. The Pearson correlation coefficient (ρ\rho) exceeds 0.9 for every evaluated combination of cluster size and head dimension, indicating a strong positive linear relationship irrespective of the kernel parameters.

Effect of reduced data in a kernel on GPU power allocation under a fixed TDP limit

Figure 12. Effect of reduced data in a kernel on GPU power allocation under a fixed TDP limit.

Percentage of throughput increase against percentage increase in SM frequency for Flash Attention kernel executions with different cluster sizes

Figure 14. Percentage of throughput increase against percentage increase in SM frequency for Flash Attention kernel executions with different cluster sizes

Figure 14 shows a clear separation between the data points corresponding to GPC-level and TPC-level cluster sizes. Within GPC-level clustering, increasing the cluster size from 4 to 8 improves throughput at a head dimension of 256, as expected. However, this trend is reversed at a head dimension of 128. At a head dimension of 256, tensor core utilization approaches its maximum, as shown in Table 4, helping to mask the additional synchronization overhead incurred by increasing the cluster size. In contrast, at a head dimension of 128, tensor core utilization is insufficient to effectively mask these additional synchronization overheads, resulting in lower throughput at cluster size 8 compared to cluster size 4.

Effect of TDP on other AI accelerators

To examine whether our observations generalize beyond GPUs, we also evaluate the impact of TDP on Tenstorrent’s Wormhole n300d (Tenstorrent), a representative spatial dataflow accelerator. Unlike GPUs, which rely on hardware-managed caches, dataflow accelerators explicitly orchestrate data movement between software-managed on-chip scratchpads over a NoC. Each Wormhole chip is organized as a 2D mesh of Tensix cores, where every core contains a matrix engine, a vector engine, and a local SRAM scratchpad, and cores communicate through two independent NoCs. The n300d card integrates two Wormhole chips, each with 64 Tensix cores clocked at 1 GHz, delivering a peak throughput of 64 TFLOP/s per chip.

Figure 15 shows how throughput, clock frequency, and power evolve as we increase the number of active Tensix cores running matrix multiplication on a single Wormhole chip. We evaluate two math fidelity settings: HiFi4, which performs multiple passes through the multiplier to preserve full operand precision, and LoFi, which trades precision for a single pass. With few active cores, both settings scale almost perfectly, and the chip sustains its nominal 1 GHz clock.

Scaling of matrix multiplication on one Wormhole chip under the default power limit

Figure 15. Scaling of matrix multiplication on one Wormhole chip under the default power limit. (a) Achieved throughput versus the number of active Tensix cores for the HiFi4 and LoFi math fidelities. (b) Average core clock (normalized to 1 GHz) and power (normalized to the 85 W limit).

As more cores become active, power rises quickly and eventually hits the TDP limit, at which point the firmware throttles the clock frequency to stay within the power budget. HiFi4 consumes more energy per operation and therefore reaches the limit earlier: its power climbs to 87% of the budget at 16 cores and saturates at 32 cores, after which the clock drops to about 57% of nominal at 64 cores. As a result, HiFi4 throughput plateaus at roughly 31 TFLOPS, only about half of its ideal-scaling value. LoFi stays within the budget up to 32 cores and is throttled only mildly (to about 86% of nominal), reaching around 60 TFLOPS at 64 cores.

These results demonstrate that the impact of TDP limits on kernel performance is non-trivial. Furthermore, understanding these effects provides greater insight into the observed performance characteristics. This reinforces our argument that microarchitectural knowledge alone is insufficient to maximize kernel performance or fully explain the performance achieved.

Discussion

Stringency and portability of wave-equalization conditions

For a kernel launch to be wave-equalized across cluster sizes, it must satisfy Eq. (1). This requires a positive integer nn such that both 132(n−1)<B≤132n132(n - 1) < B \le132n and 120(n−1)<B≤120n120(n - 1) < B \le120n hold, where BB denotes the number of thread blocks and nn is the common number of execution waves. Combining these inequalities yields 132(n−1)<B≤120n132(n - 1) < B \le120n. Since the number of thread blocks must be a positive integer, n≥1n \ge1. Furthermore, satisfying 132(n−1)<120n132(n - 1) < 120n requires n<11n < 11. Therefore, the set of thread-block counts eligible for wave-equalized kernel launches is given by Eq. (3). Although this set is limited, the constraint is unavoidable on the current NVIDIA H100, given the fixed active SM limits of 132 and 120 imposed by floorsweeping. However, incorporating kernel performance considerations into floorsweeping decisions could allow these limits to be selected to expand the set of eligible thread-block counts.

B∈⋃n=110(132(n−1),120n](3)B \in\bigcup_{n=1}^{10} \left(132(n - 1), 120n\right] \tag*{(3)}

Furthermore, the condition for a wave-equalized kernel launch can be readily adapted to another GPU using the microbenchmark described in § 4.2. The numbers of unique SMs observed per iteration under the unclustered (cluster size 1) and GPC-clustered conditions give the complete and reduced active SM limits, respectively. Replacing 132 and 120 in Eq. (1) with these respective limits yields the wave-equalization condition for the target GPU.

Implications for kernel development systems

Performance engineering, spanning compiler-based automatic kernel generation through heuristics, static analysis, and autotuning, etc ([6, 32, 28]), manual performance engineering ([8, 7, 25]), and now agentic kernel development ([20, 12, 22, 31, 1]), relies on appropriate abstractions and knowledge of the microarchitecture. The physical non-idealities discussed in this work challenge several assumptions underlying these approaches. For instance, straightforward, widely accepted heuristics, such as amortizing memory accesses (e.g., amortizing L2 accesses through DSMEM in this work) or increasing compute utilization upto saturation, can adversely affect kernel performance. Further, the implications of such non-idealities extend beyond performance, potentially introducing numerical nondeterminism through changes in floating-point evaluation order, as well as power spikes. Through this work, we provide an initial impetus for the community to consider physical non-idealities more broadly, alongside a closed-form solution for handling manufacturing-yield-related effects on clustered execution. Future work will extend this treatment to other physical non-idealities.

Conclusion

Physical non-idealities in IC design and fabrication introduce an additional layer of abstraction that kernel performance engineers must look beyond to explain and maximize kernel performance. Using DSMEM on NVIDIA H100 GPUs, we show how floorsweeping and the resulting virtual GPC organization restrict the number of SMs eligible for GPC-level clustered execution to 124, with at most 120 active concurrently, despite the GPU exposing 132 SMs. This restriction can increase the number of execution waves and offset the benefits of reduced data movement. We characterize wave-equalized kernel launches and derive the conditions under which the reduced active SM limit introduces no additional execution waves. Evaluations using a global-load-and-broadcast microbenchmark, GEMM, and Hopper FlashAttention 4 illustrate how wave equalization influences the performance trade-offs of clustered execution. With this knowledge, we introduce the wave-aware clustering strategy for choosing between TPC and GPC clustering for a kernel. Across sequence lengths varying from 2K to 64K, wave-aware clustering achieves throughput improvements up to 5.30% over the FlashAttention 4 baseline. Furthermore, we demonstrate that reduced data movement can improve kernel throughput even without increasing tensor core utilization. For kernels operating near the TDP limit, reduced power consumption from data transfers creates additional power headroom, allowing the GPU to sustain higher SM frequencies through DVFS. Our evaluation of the Tenstorrent Wormhole accelerator further demonstrates how power constraints affect operating frequency and achievable throughput beyond GPUs. These findings reinforce our argument that looking beyond conventional microarchitectural abstractions and accounting for physical non-idealities enables kernel performance engineers to explain seemingly unexpected performance behavior and identify optimization opportunities within the physical constraints of the hardware.

Acknowledgments

This work is partially supported by the Ministry of Education, Singapore, under Tier 3 grant MOET32024-0003 and the Advanced Research and Technology Innovation Centre (ARTIC), the National University of Singapore under Grant AFP-RP6.

References

  1. [1]Dai, W., Wu, H., Yu, Q., Gao, H.-a., Li, J., Jiang, C., Lou, W., Song, Y., Yu, H., Chen, J., et al. CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation. arXiv preprint arXiv:2602.24286, 2026.arxiv.org/abs/2602.24286
  2. [2]Afzal, A., Li, M. M., and Panzlaff, M. Modeling and Chasing the Energy-Efficiency Sweet Spots in Modern GPUs. arXiv preprint arXiv:2607.00819, 2026. doi: 10.48550/arXiv.2607.00819.DOI
  3. [3]Aguilera, P., Lee, J., Farmahini-Farahani, A., Morrow, K., Schulte, M., and Kim, N. S. Process Variation-Aware Workload Partitioning Algorithms for GPUs Supporting Spatial-Multitasking. In 2014 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 1–6. IEEE, 2014.
  4. [4]Andersch, M., Palmer, G., Krashinsky, R., Stam, N., Mehta, V., Brito, G., and Ramaswamy, S. Nvidia hopper architecture in-depth, 2022. URL https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/. Accessed: 2026-03-16.
  5. [5]Chen, K. and Patel, D. Dissecting Nvidia Blackwell - Tensor Cores, PTX Instructions, SASS, Floorsweep, Yield. Technical report, SemiAnalysis, 2026. URL https://newsletter.semianalysis.com/p/dissecting-nvidia-blackwell-tensor.
  6. [6]Chen, T., Moreau, T., Jiang, Z., Zheng, L., Yan, E., Shen, H., Cowan, M., Wang, L., Hu, Y., Ceze, L., et al. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pp. 578–594, 2018.arxiv.org/abs/1802.04799
  7. [7]Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=mZn2Xyh9Ec.arxiv.org/abs/2307.08691
  8. [8]Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. Advances in neural information processing systems, 35:16344–16359, 2022.arxiv.org/abs/2205.14135
  9. [10]Duluk Jr, J. F., Hirota, G., Krashinsky, R., Palmer, G., Tuckey, J., Nadadhur, K., Johnson, P. B., and Joginipally, P. Virtualizing Hardware Processing Resources in a Processor, October 21 2025. US Patent 12,449,473.
  10. [11]Fan, X., Wang, J., Cui, W., Zhao, H., Huang, Z., Zhou, Y., Qiu, Y., Sun, S., He, B., Chen, Q., et al. Dissecting How Die Scaling Breaks GPU Fine-grained Scheduling. arXiv preprint arXiv:2609.24270, 2026.arxiv.org/abs/2609.24270
  11. [12]Han, Y., Fan, K., Jiang, H., Xu, W., Xie, W., Zhang, R., Zhu, C., and Zhang, Y. EGG: An Expert-Guided Agent Framework for Kernel Generation. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=8nti23Zqkt.arxiv.org/abs/2606.26758
  12. [14]Lee, J., Sathisha, V., Schulte, M., Compton, K., and Kim, N. S. Improving Throughput of Power-Constrained GPUs using Dynamic Voltage/Frequency and Core Scaling. In 2011 International Conference on Parallel Architectures and Compilation Techniques, pp. 111–120. IEEE, 2011.DOI
  13. [15]Luo, W., Fan, R., Li, Z., Du, D., Liu, H., Wang, Q., and Chu, X. Dissecting the NVIDIA Hopper Architecture through Microbenchmarking and Multiple Level Analysis. arXiv preprint arXiv:2501.12084, 2025a.arxiv.org/abs/2501.12084
  14. [16]Luo, X., Liu, Z., Zhou, Y., Fang, S., Huang, Z., Feng, Y., Zhang, C., Sun, S., Zheng, Z., Leng, J., and Guo, M. Clusterfusion: Expanding operator fusion scope for LLM inference via cluster-level collective primitive. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025b. URL https://openreview.net/forum?id=eX0m4qMYVN.
  15. [20]Ouyang, A., Guo, S., Arora, S., Zhang, A. L., Hu, W., Re, C., and Mirhoseini, A. KernelBench: Can LLMs Write Efficient GPU Kernels? In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=yeoN1iQT1x.arxiv.org/abs/2502.10517
  16. [21]Prabhakar, P. B., Hirota, G., Krashinsky, R., Long, Z., Pharris, B., Dash, R., Tuckey, J., Duluk Jr, J. F., Shah, L., Durant, L., et al. Distributed Shared Memory, March 11 2025. US Patent 12,248,788.
  17. [22]Ran, D., Xie, S., Ji, M., Liu, A., Wu, M., Cao, Y., Guo, Y., Yu, H., Li, L., Hu, Y., Yang, W., and Xie, T. KernelBand: Steering LLM-based Kernel Optimization via Hardware-Aware Multi-Armed Bandits. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=d8e2jBfMpE.arxiv.org/abs/2511.18868
  18. [23]Rountree, B., Ahn, D. H., De Supinski, B. R., Lowenthal, D. K., and Schulz, M. Beyond DVFS: A First Look at Performance under a Hardware-Enforced Power Bound. In 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum, pp. 947–953. IEEE, 2012.DOI
  19. [24]Schoonhoven, R., Veenboer, B., Van Werkhoven, B., and Batenburg, K. J. Going Green: Optimizing GPUs for Energy Efficiency through Model-Steered Auto-Tuning. In 2022 IEEE/ACM International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS), pp. 48–59. IEEE, 2022.arxiv.org/abs/2211.07260
  20. [25]Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision. Advances in Neural Information Processing Systems, 37:68658–68685, 2024.arxiv.org/abs/2407.08608
  21. [27]Thakkar, V., Ramani, P., Cecka, C., Shivam, A., Lu, H., Yan, E., Kosaian, J., Hoemmen, M., Wu, H., Kerr, A., Nicely, M., Merrill, D., Blasig, D., Atluri, A., Qiao, F., Majcher, P., Springer, P., Hohnerbach, M., Wang, J., and Gupta, M. CUTLASS: Hopper Warp-Specialized GEMM Example, 2026. URL https://github.com/NVIDIA/cutlass/blob/0b55a2f691d69981583568fd9eb69687b1f0de8a/examples/48_hopper_warp_specialized_gemm/48_hopper_warp_specialized_gemm.cu. Version 4.8.0, Example 48. Commit 0b55a2f691d69981583568fd9eb69687b1f0de8a.
  22. [28]Tillet, P., Kung, H. T., and Cox, D. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, pp. 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/3315508.3329973. URL https://doi.org/10.1145/3315508.3329973.
  23. [29]Ujeniya, A., Eitzinger, J., Hager, G., and Wellein, G. Architectural Trade-offs in the Energy-Efficient Era: A Comparative Study of Power-Capping NVIDIA H100 and H200. arXiv preprint arXiv:2604.11391, 2026. doi: 10.48550/arXiv.2604.11391.DOI
  24. [30]Zadouri, T., Hoehnerbach, M., Shah, J., Thakkar, V., and Dao, T. FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling. Proceedings of Machine Learning and Systems, 8:912–926, 2026.arxiv.org/abs/2603.05451
  25. [31]Zhang, Z., Wang, R., Li, S., Luo, Y., Hong, M., and Ding, C. CudaForge: An Agent Framework with Hardware Feedback for CUDA Kernel Optimization. arXiv preprint arXiv:2511.01884, 2025.arxiv.org/abs/2511.01884
  26. [32]Zheng, L., Jia, C., Sun, M., Wu, Z., Yu, C. H., Haj-Ali, A., Wang, Y., Yang, J., Zhuo, D., Sen, K., et al. Ansor: Generating High-Performance Tensor Programs for Deep Learning. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pp. 863–879, 2020.arxiv.org/abs/2006.06762

Paper details

vlnc:W_fw4byc9r
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents