WorksComputer Science
Demystifying the Impact of Physical Non-Idealities on Kernel Performance
Abstract
Physical non-idealities in integrated circuit (IC) design, fabrication, and operation are concealed by conventional hardware abstractions, causing unexpected performance variations in kernels. For large-scale workloads, such as AI training and inference, performance unpredictability can lead to substantial losses. We investigate this problem in NVIDIA H100 GPUs with their distributed shared memory (DSMEM) architectural feature that reduces data movement from the L2 cache to Streaming Multiprocessors (SMs) with multicast loads, and uncover that floorsweeping restricts the number of SMs usable for clustered execution, increasing the number of execution waves of kernels. To alleviate this performance penalty, we introduce the wave-aware clustering strategy that accounts for these resource constraints and improves the throughput of the Hopper version of FlashAttention 4 by up to 5.30% over the unclustered baseline. Furthermore, we discover that the power constraint imposed by the thermal design power (TDP) limit causes the DSMEM-based memory bandwidth optimization to improve the throughput even for compute-bound kernels by creating a power headroom for higher SM frequencies through reduced data movement. Extending this analysis beyond GPUs, we find that power constraints also limit throughput scaling on Tenstorrent's Wormhole accelerator, highlighting the broader importance of accounting for physical non-idealities in kernel performance engineering.
Introduction
The fundamental idea of abstraction is pervasive across computer systems. By enabling the philosophy of separation of concerns, abstractions have facilitated the development of software for increasingly complex hardware. At the highest level of abstraction, software is written in high-level languages and lowered to the target hardware with a compiler. This abstraction reduces the complexity of using hardware and improves productivity and portability [6], but it also restricts the maximum performance achievable for the workload [14]. As performance demands increase, performance engineers are compelled to break this level of abstraction and work with the hardware’s Instruction Set Architecture (ISA) to develop kernels that effectively exploit the underlying microarchitecture [14, 32]. Through this work, we show that even at this abstraction level, they are limited because physical non-idealities in integrated circuit (IC) design, fabrication, and operation can cause the microarchitecture to behave differently under certain conditions.
Physical non-idealities are physical constraints that cause ICs to deviate from the expected behaviour that their microarchitecture may imply. Manufacturing defects that occur in IC fabrication are one such constraint. Since it is common for modern ICs to have defects with current manufacturing technologies, IC vendors cannot restrict themselves to selling completely defect-free ICs. As a solution, they disable the defective components and, if required, switch off additional functional components to align ICs to a pre-determined resource profile. This technique is called floorsweeping, and it allows the vendors to hide the distribution of faults and categorize a large percentage of faulty ICs into a single product [12]. For example, ICs manufactured with the NVIDIA GH100 design that have 144 Streaming Multiprocessors (SMs) are grouped
into their enterprise data center-level Graphics Processing Unit (GPU), H100 SXM5, which has only 132 SMs [3]. As users, we do not know how the 12 disabled SMs are distributed in the GPU. Therefore, a nominally identical GPU model can have different internal physical topologies, even when its exposed compute capacity is identical. In addition to floorsweeping, thermal design power (TDP) specifies the maximum power an IC is capable of consuming during a sustained workload. If a kernel attempts to use the microarchitecture in a manner that consumes more power than its TDP, under nominal conditions, the IC would throttle its frequency to reduce its power consumption (NVIDIA, b). Since this is a dynamic process, its impact on kernel performance remains opaque at design time. Physical non-idealities such as these are abstracted away by IC vendors when they disclose their microarchitecture to provide consistent performance and programmability across a product line.
The impact these physical non-idealities have on kernel performance does not vary based on its design, in most cases. However, as kernel performance engineers try to realize the full performance potential of a microarchitecture, they may attempt optimizations that expose performance variations from these effects. In large-scale deployments like AI training and inference, such performance unpredictability can cause significant loss [18]. We explore this issue in NVIDIA H100 GPUs and demonstrate its impact with the distributed shared memory (DSMEM) architectural feature, and show how the impact can be alleviated with kernel mappings that work around these non-idealities.
NVIDIA GPUs before the Hopper architecture rely on sharing data between SMs through the global memory, leading to congestion and consequent performance impacts [3]. The DSMEM introduced in the Hopper architecture reduces the inter-SM communication cost via a Network-on-Chip (NoC) that connects shared memories of SMs in a Graphics Processing Cluster (GPC). As in Figure 1, SMs in the H100 GPU are arranged into pairs called Texture Processing Clusters (TPCs), and nine of those TPCs are grouped into a GPC [3]. To utilize the DSMEM, multiple thread blocks need to be grouped, i.e., clustered, into a unit called a thread block cluster (cluster). It ensures that the thread blocks are mapped to SMs within a single GPC, as shown in Figure 1. This architectural feature enables a thread block to multicast data from the global memory to all SMs’ shared memory in the cluster when they require the same data, a common scenario in tiled kernels (NVIDIA, c). Interestingly, we observe that this feature is not completely utilized in modern kernels such as FlashAttention [10, 32, 39]. As seen in Figure 2, our evaluations on the Hopper version of FlashAttention 4 with multicast load also show that the expected performance gains do not materialize at
Figure 1. Layout of SMs in the NVIDIA H100 GPU. From our reverse engineering, the number of TPCs in each GPC is labelled.
Figure 2. FlashAttention 4 throughput varies across cluster sizes due to physical non-idealities. The plot shows the throughput, percentage increase in throughput, and percentage increase in SM frequency compared to cluster size 1 for FlashAttention 4 with 64K sequence length and 256 head dimension.
cluster sizes 4 and 8. A kernel performance engineer might attribute this performance degradation entirely to synchronization overheads. However, our investigations uncover that it is only one of the contributing factors and that impacts from physical non-idealities dominate here.
In this paper, we first expose how floorsweeping limits the number of SMs available for clustered execution. Although the NVIDIA H100 GPU has 132 SMs [3], our experiments reveal that only 120 SMs can be used concurrently at GPC-level cluster sizes. We refer to this number as the active SM limit. We explain this reduction using the organization of remaining functional resources into virtual GPCs. A reduced active SM limit decreases the number of concurrently scheduled thread blocks in the GPU, i.e., the size of an execution wave, increasing the number of waves required for a kernel execution. We characterize the conditions under which the reduced active SM limit introduces additional execution waves. Further, we define a wave-equalized kernel launch as one whose thread-block count results in the same number of execution waves under both the full and reduced active SM limits. With insights from this, we introduce the wave-aware clustering strategy that is cognizant of the floorsweeping effects. It is able to select the best clustering method for FlashAttention 4 with sequence lengths spanning from 2K to 64K. We achieve a throughput improvement of up to 5.30% against the baseline FlashAttention 4 implementation, which uses cluster size 1.
In contrast to floorsweeping, which causes a performance reduction on DSMEM, the TDP limit has a positive impact. Surprisingly, we find that the throughput improvements from DSMEM do not always occur due to improved tensor core utilization from reduced data movement. In compute-bound kernels, as in Figure 2, the memory bandwidth optimization from DSMEM reduces the power consumed for data transfer, allowing the GPU to sustain a higher SM frequency with Dynamic Voltage and Frequency Scaling (DVFS). We achieve throughput improvements up to 3% through this phenomenon. We extend this analysis beyond GPUs to Tenstorrent’s Wormhole accelerator and uncover that it also has a similar behaviour with the TDP limit, where its expected throughput improvements with a higher number of active cores are restricted by power-limiting frequency variations. These results show that hardware non-idealities cause unexpected performance variations in kernels and that physical-constraint-aware kernel development can unlock meaningful performance improvements.
Background and Related Work
Distributed shared memory in GPUs
DSMEM is an architectural feature introduced in the NVIDIA Hopper GPU architecture. It connects the shared memories of multiple SMs into a unified memory space through a NoC [28]. Thread blocks can be grouped into a unit called a thread block cluster (cluster), with supported cluster sizes of 2, 4, and 8, with 16 being optional. However, not all SMs in the GPU can be clustered together, as cluster placement is constrained by the physical organization of the SMs. As shown in Figure 1, the NVIDIA Hopper GPU organizes SMs into pairs within units called TPCs. Nine TPCs form a GPC, and the GPU contains eight such GPCs. Each GPC has a NoC connecting the shared memories of the SMs within it. Consequently, only SMs within the same GPC can participate in a cluster. Furthermore, clusters of size 2 always group SMs within a single TPC. Therefore, we refer to clusters of size 2 as TPC-level clusters and clusters of sizes 4 and 8 as GPC-level clusters. While clustered execution has the potential to reduce data movement within the GPU, it also introduces additional synchronization overhead and scheduling constraints. In this work, we use DSMEM to demonstrate that unexpected trade-offs arise from this feature, which become apparent only when physical non-idealities are taken into account.
Impact of physical non-idealities
Prior studies have reported that DSMEM throughput decreases with increasing GPC-level cluster sizes [20 a]. This behavior has been attributed to contention in the crossbar connecting the L1 caches/shared memories of SMs within a GPC [20, 28]. Additionally, the number of SMs available for clustered execution has been observed to decrease at GPC-level cluster sizes [20 NVIDIA, a]. Floorsweeping has been identified as a potential cause of these resource restrictions [5] (He et al. 2026). Concurrent work by Fan et al. [13] characterizes the physical SM topology and examines its implications for thread-block cluster scheduling, particularly in the context of MIG partitioning (NVIDIA, e) and Green Contexts (NVIDIA, d). Beyond the resource restrictions imposed by floorsweeping, the performance of clustered kernels also depends on the operating frequency sustained within the available power budget. Existing Hopper microbenchmarking studies have characterized DSMEM latency and throughput and separately reported power-induced frequency reductions during tensor-core execution. Studies of GPU power capping and frequency scaling have also characterized performance and energy trade-offs, but have not isolated how DSMEM-based reductions in data movement affect operating frequency [1] (Ujenjiya et al. 2026). More broadly, the effects of manufacturing variations and power constraints on CPU and GPU performance have been studied at the microarchitectural level [17, 30, 2, 31]. In this work, we first characterize the restrictions imposed by floorsweeping on clustered execution and its performance implications. We propose the conditions for kernel launches that avoid additional execution waves under these restrictions. Furthermore, we investigate how TDP-induced frequency scaling affects clustered kernel performance and explain its combined impact with floorsweeping.
Methodology
We evaluate four NVIDIA H100 SXM5 GPUs to assess the impact of physical non-idealities and ensure that the observed effects are not specific to a single GPU. The four GPUs are located in geographically distinct regions to reduce the likelihood that the observations are specific to a particular manufacturing batch. Throughout this paper, NVIDIA H100 refers to the NVIDIA H100 SXM5 stock-keeping unit (SKU) unless otherwise specified. We use four CUDA kernels for our experiments: a multicast load microbenchmark, a GPU skyline reverse-engineering kernel, GEMM, and FlashAttention. All kernels are compiled using CUDA 13.0 with the sm_90a compilation target. Each kernel is executed for multiple warm-up and timed iterations, with the exact counts reported in the corresponding experimental sections. Furthermore, we record the SM clock frequency using the NVIDIA Management Library (NVML) and the remaining profiling data are collected using NVIDIA Nsight Compute (NCU).
For GEMM, we use the Hopper warp-specialized GEMM kernel from the CUTLASS 4.8.0 library [36]. We configure it with three pipeline stages and use a non-persistent execution mode to ensure that, for a fixed problem size, the number of thread blocks depends only on the tile dimensions. For FlashAttention, we use the FlashAttention 4 Hopper forward kernel (Dao-AILab). The kernel takes query (), key (), and value () matrices as inputs. With a tiling configuration of , each thread block processes rows of and iteratively loads tiles of rows from and to perform the attention operation. Since multiple thread blocks processing different query tiles require the same and tiles, there is an opportunity to reduce redundant data loads through multicast loads using DSMEM. We modify the FlashAttention codebase to incorporate this functionality and evaluate it using non-causal multi-head attention (MHA).
For the Tenstorrent experiments, we use one Wormhole chip of an n300d card and TTNN (Tenstorrent, [35]), Tenstorrent’s official library built on the TT-Metalium software stack (Tenstorrent, [34]). We run matrix multiplication with ttnn.matmul on BF16 inputs, using the official TTNN GEMM benchmark with its default configuration. We vary the number of active Tensix cores from 1 to 64 and keep the work per core fixed. While the benchmark runs, a separate process reads the chip’s core clock and power from the firmware telemetry that the Tenstorrent kernel driver exposes in sysfs. We report the median of the samples taken during each measurement.
Linking Cluster Scaling Limits to Floorsweeping
Distributed shared memory enables a thread block to initiate a multicast load from global memory into the shared memories of the corresponding SMs within a cluster, eliminating redundant data loads. This is supported by the network connecting the shared memories of SMs within a GPC (Prabhakar et al., [28]). When thread blocks are grouped into a cluster, they are scheduled on SMs within the same GPC. Such data sharing can reduce L2 cache traffic and the cumulative data loading time across thread blocks. However, these expected benefits do not consistently translate into improved kernel performance, as shown in Figure 2. To investigate the underlying causes, we develop a microbenchmark to isolate data loading performance.
Microbenchmarking multicast loads
In this microbenchmark, the CPU generates 1 MB of pseudorandom FP32 values, which are then copied to the GPU’s global memory (HBM). We launch the kernel with 256 thread blocks and evaluate cluster sizes of 1, 2, 4, and 8 in separate runs. For cluster sizes greater than 1, the kernel performs multicast loads of the data in successive 128 KB chunks using the two patterns illustrated in Figure 3. In the single-block multicast load pattern, the first thread block in each cluster initiates a Tensor Memory Accelerator (TMA) transfer to load the data into the shared memory of every thread block within the cluster. In contrast, in the all-blocks multicast load pattern, each thread block initiates a TMA transfer for a distinct portion of the data, multicasting its portion into the shared memory of the corresponding SMs of each thread block within the cluster.

Figure 3. Multicast load patterns visualized for cluster size 4.
The kernel is executed for 50 warm-up iterations, followed by 500 timed iterations to calculate the mean latency. Furthermore, the GPU’s SM and memory clock frequencies are fixed at their maximum values of 1980 MHz and 2619 MHz, respectively. At cluster sizes greater than 1, the all-blocks multicast load pattern issues the same number of TMA transfers as the cluster size 1 configuration, but with smaller transfer sizes than the single-block multicast load pattern. Since smaller TMA transfers can reduce effective memory bandwidth [19], the single-block multicast load pattern achieves lower latency than the all-blocks multicast load pattern, as shown in Figure 4.
Figure 4. Mean latency for 256 thread blocks to load 1 MB of data in 128 KB chunks from global memory into shared memory using multicasting, across different cluster sizes.
The microbenchmark results also reveal an unexpected trend at GPC-level cluster sizes, similar to that observed in Figure 2. Since GPC-level cluster sizes progressively reduce the amount of data transferred from the L2 cache to shared memory, they are expected to achieve lower mean latency than the TPC-level cluster size. However, this expected improvement does not materialize due to the reduced number of active SMs available for GPC-level clustered execution.
Active SM limits at different cluster sizes
To determine the number of active SMs at different cluster sizes, we record the logical SM IDs used by the microbenchmark kernel during each iteration. The logical SM ID is obtained by having the first thread of each thread block read the %smid register, which uniquely identifies the SM executing the thread block. Table 1 reports the number of unique SMs observed within individual iterations and across all iterations for each cluster size.
| Cluster Size | Unique SMs | ||
| Per Iter. | All Iters. | ||
| 1 | 132 | 132 | |
| 2 | 132 | 132 | |
| 4 | 120 | 124 | |
| 8 | 120 | 124 |
Table 1. Number of unique active SMs at each cluster size.
At cluster sizes 1 and 2, the kernel utilizes all 132 SMs in the NVIDIA H100 GPU. However, at GPC-level cluster sizes, only 124 SMs are enabled for clustered execution, of which at most 120 can be used concurrently. Surprisingly, these restrictions arise from floorsweeping.
Effect of Floorsweeping on active SM limit
Physical non-idealities can cause ICs to differ from the resource organization implied by their microarchitectural design. For large dies, manufacturing defects are one source of deviation affecting a substantial fraction of chips. The negative binomial (NB) model expresses the zero-defect yield as , where controls the variation in defect counts across chips. The expected number of defects per chip is , where is the defect density and is the die area [7, 4, 21]. At a defect density of 0.15 defects/cm², a large accelerator die near the reticle limit (700–800 mm²) has approximately 1.05 defects on average. Smaller values represent greater clustering, where some chips contain substantially more defects than others.
some chips contain substantially more defects than others. Requiring every chip to be defect-free or have identical defect patterns is impractical. Vendors therefore disable defective components through floorsweeping to expose a usable resource profile. Combined with binning, this allows physically different chips to be sold within the same SKU, while chips with fewer surviving resources may qualify for lower-performance SKUs.
For example, the NVIDIA GH100 design contains 144 SMs, whereas the H100 SXM5 exposes 132. This resource count does not reveal which 12 SMs are disabled or their distribution across the chip. Consequently, GPUs of the same SKU can expose identical compute capacity but different internal physical topologies. Assuming a defect disables a TPC, some physical GPCs may lose more TPCs than others, resulting in different distributions of active SMs per GPC. NVIDIA’s patent [12] introduces a Skyline, specifying a common distribution of TPCs across virtual GPCs (vGPCs) for a SKU. Each tower represents a vGPC, with its height specifying the number of exposed TPCs.
Figure 5 illustrates how dies with different physical fault patterns can expose the same skyline. Each vGPC is formed from surviving TPCs within a physical GPC. A physical GPC with more surviving TPCs than required by its mapped tower can expose the remaining TPCs separately as single-TPC vGPCs, called singletons. Thus, tower heights describe the virtual grouping available to software rather than the physical distribution of surviving resources. Skyline selection aims to accommodate as many physical configurations as possible while minimizing resource loss.
Figure 5. Hypothetical fault patterns mapped to a common skyline through floorsweeping [12].
For example, a skyline of 6/8/8/8/9/9/9/9/1×2 exposes 68 TPCs across eight multi-TPC vGPCs and two singletons. Singletons help accommodate different fault patterns but also constrain clustered execution. They can execute independent thread blocks or TPC-level clusters, but cannot participate in multi-TPC (GPC-level) clusters. Thus, accommodating more configurations may require disabling otherwise functional TPCs or exposing them in smaller virtual groups, limiting the applicability and effectiveness of clustered execution.
Reverse engineering GPU skylines and its impact
The earlier microbenchmark records only 120 unique SM IDs per iteration at cluster sizes 4 and 8, and 124 across all iterations, despite the H100 exposing 132 SMs (Table 1). Since the SMs in a cluster must belong to the same vGPC, both the sizes of these groups and the clusters they can accommodate can fragment available compute capacity. To understand this discrepancy, we reverse-engineer the H100 skyline from the observed SM groupings during clustered execution.
We launch a CUDA kernel with 1024 thread blocks of 1024 threads each. The first thread of each block reads the %smid register and stores its ID in global memory. For increasing cluster sizes, we group the SM IDs belonging to each cluster into sets, leveraging the grouping of consecutive thread blocks during clustering. We merge sets sharing one or more SM IDs to infer the underlying SM groupings, repeating the process for 1000 iterations to account for variations.
We infer 12 vGPCs with the skyline 9/9/8/8/8/8/8/4/1/1/1/1, as shown in Table 2. We assign vGPC IDs in ascending order of the lowest SM ID within each group. This virtual organization contains 12 groups despite the chip having only eight physical GPCs. The four-TPC vGPC potentially exists to accommodate dies with faults largely concentrated within one physical GPC.
| Type | Count | vGPC IDs |
| 2 SMs vGPC (1 TPC) | 4 | 8,9,10,11 |
| 8 SMs vGPC (4 TPCs) | 1 | 0 |
| 16 SMs vGPC (8 TPCs) | 5 | 1,2,3,4,5 |
| 18 SMs vGPC (9 TPCs) | 2 | 6,7 |
Table 2. Inferred skyline for the measured NVIDIA H100.
This skyline explains both observed SM counts. The four singleton vGPCs contain two SMs each and cannot support clusters of size 4 or 8, leaving only 124 of the 132 SMs available for GPC-level clustered execution. Furthermore, each of the two 18-SM vGPCs leaves two SMs unassigned when partitioned into clusters of size 4 or 8. This further reduces the active SM limit to 120 for these cluster sizes in the measured configuration. Since the unassigned SMs can vary between iterations, the microbenchmark observes 120 unique SM IDs per iteration but 124 across all iterations.
Beyond restricting the availability of SMs, the underlying resource organization also influences which SMs are frequently grouped into clusters. As shown in Figure 6, SMs with neighboring logical SM IDs are frequently clustered together at cluster sizes 4 and 8. Thus, a performance engineer
Figure 6. Probability that SM belongs to a cluster given that SM belongs to the same cluster, for cluster sizes 4 and 8.
must account for both the number of available SMs and how they are grouped for clustered execution. The reduction in the active SM limit from 132 to 120 can increase the number of execution waves and reduce kernel performance. The following section characterizes when this additional-wave penalty occurs and identifies configurations in which the execution wave count remains unchanged.
Navigating Active SM Limit Penalties
The active SM limit imposed by floorsweeping can increase the number of execution waves required by a kernel, as the same number of thread blocks must be scheduled across fewer SMs. An execution wave consists of the thread blocks that can execute concurrently on the GPU. The number of execution waves influences kernel execution time, as additional waves require successive scheduling of thread blocks. Consequently, the additional-wave penalty caused by the reduced active SM limit can be avoided when a kernel launch requires the same number of execution waves under both the full 132-SM and reduced 120-SM limits. We refer to such a launch as a wave-equalized kernel launch, formalized in Eq. (1). We see pervasive LLM kernels such as GEMM and FlashAttention often achieve their highest throughput by maximizing data reuse through large tile shapes that fit within the available shared memory and register capacity. The resulting resource requirements typically limit occupancy to one thread block per SM, and this restricted occupancy exacerbates the impact of the reduced active SM limit because it minimizes the number of thread blocks in a wave.
Wave-equalized multicast load within clusters
To examine how wave equalization influences the benefits of multicast load, we compare the original 256-thread-block microbenchmark from § 4 with a 240-thread-block wave-equalized configuration satisfying Eq. (1). Figure 7 shows the change in latency due to clustering, relative to the cluster size 1 baseline, for both configurations. At GPClevel cluster sizes, clustering reduces latency in the wave-equalized 240-block configuration but increases it in the non-wave-equalized 256-block configuration. These results illustrate how additional execution waves can offset the latency benefits of reduced data movement through clustering.

Figure 7. Reduction in latency of loading 1 MB data (128 KB chunks) from global memory to shared memories with multicast across cluster sizes.
Wave-equalized GEMM kernel launches
To evaluate the effect of wave equalization, we first characterize GEMM performance across different cluster sizes. We evaluate tile shapes of and , along with all valid cluster shapes for cluster sizes greater than 1. For each GEMM problem shape and cluster size, we select the configuration that achieves the highest throughput. Furthermore, each configuration is executed for 100 warm-up iterations, followed by 1000 timed iterations.
As shown in Table 3, the L2 hit rate of the unclustered kernel decreases as the operand size increases, because a smaller proportion of the operand data gets reused through the L2 cache. This causes a rise in HBM reads, causing the HBM utilization to increase to near saturation. When the HBM is highly utilized, the bandwidth optimization provided by clustering outweighs its synchronization overheads and reduced active SM limit penalties. Conversely, the clustering bandwidth optimization has no effect when the HBM utilization is low. Interestingly, the impact of reduced active SM limit is pronounced at the middle of these extremes, where the memory bandwidth optimization seems to overpower the synchronization overheads but fails to compensate for the additional execution waves at GPC-level clustering. To examine the impact of working around this additional wave penalty, we evaluate wave-equalized GEMM configurations.
| Metric | 1 | 2 | 4 | 8 | |
| 4K 4K 4K | Throughput (%) | - | -9.6% | -18.8% | -21.4% |
| 4K 4K 4K | HBM Util. (%) | 22.9% | 21.8% | 21.0% | 14.7% |
| 4K 4K 4K | L2 Hit Rate (%) | 89.2% | 85.8% | 82.7% | 86.5% |
| 8K 8K 4K | Throughput (%) | - | +6.1% | +0.4% | -2.2% |
| 8K 8K 4K | HBM Util. (%) | 61.5% | 36.3% | 32.0% | 17.9% |
| 8K 8K 4K | L2 Hit Rate (%) | 74.1% | 79.9% | 79.6% | 89.0% |
| 32K 32K 4K | Throughput (%) | - | +32.0% | +41.2% | +43.2% |
| 32K 32K 4K | HBM Util. (%) | 85.8% | 58.5% | 33.1% | 18.9% |
| 32K 32K 4K | L2 Hit Rate (%) | 60.2% | 71.7% | 82.0% | 89.5% |
Table 3. HBM utilization and the improvement in throughput over cluster 1, for GEMM (tile size ).
For a GEMM kernel launch to be wave-equalized, its output dimensions and tile shape must satisfy
where denotes the output matrix dimensions and denotes the tile shape. We fix and of GEMM to 512 and 4096, respectively, and
vary to satisfy this condition for the tile shape while being eligible for all cluster sizes. Figure 8 shows that GPC-level clustering improves throughput relative to cluster size 1 for these wave-equalized workloads.

Figure 8. GEMM throughput improvement relative to cluster 1. (x-axis) is varied across wave-equalized kernel launches.
Wave-equalized FlashAttention kernel launches
For a FlashAttention kernel launch to be wave-equalized, it must satisfy Eq. (1). The number of thread blocks in a FlashAttention kernel is given by , where is the batch size, is the number of attention heads, is the sequence length, and is the tile size along the query sequence dimension. Accordingly, the wave-equalization condition for FlashAttention can be expressed as
Since each batch and attention head has its own and matrices, clustering is performed independently for each batch and head. Therefore, to support cluster sizes of up to 8, the additional condition must also be satisfied.
To examine how clustering performs across workloads with different execution wave counts, we fix the total number of tokens to 8192, setting the batch size to for sequence length . We fix the head dimension at 256 and evaluate workloads with 16 and 15 attention heads, representing non-wave-equalized and wave-equalized configurations, respectively. Each clustered configuration is compared against the cluster size 1 baseline for the same workload. We evaluate all combinations of and , selecting the tile shape with the highest throughput. The maximum is limited to 80 because a tile exceeds the available shared memory capacity at a head dimension of 256. Each configuration is executed for 100 warm-up iterations, followed by 1000 timed iterations. Across all configurations, the tile achieves the highest throughput. With , the 15-head configurations satisfy the wave-equalization condition for FlashAttention. Figure 9 shows that GPC-level clustering improves throughput over cluster size 1 across wave-equalized configurations, with gains up to 7.32%, exceeding those in non-wave-equalized configurations by up to 12.69 percentage points. Based on this characterization, we propose wave-aware clustering, a strategy that chooses between TPC and GPC clustering for a given kernel based on its number of execution waves.
Figure 9. Percentage increase in throughput over cluster size 1 across sequence lengths for wave-equalized and non-wave-equalized FlashAttention 4 kernel launches.
Wave-aware clustering on FlashAttention
FlashAttention 4 kernel made for NVIDIA Hopper GPUs does not use clustering, i.e., uses cluster size 1. We suspect that it is due to the performance variability in clustered execution that is unexplainable from a microarchitectural perspective. As the impact of floorsweeping sheds light on this variation in kernel performance, we propose the wave-aware clustering strategy in Eq. (2), where TB denotes thread blocks, for choosing between TPC-level and GPC-level clustering. To evaluate it, we consider FlashAttention 4 for varying sequence lengths from 2K to 64K, as we see in Figure 10, under two configurations: FA4-H16 and FA4-H15. Both configurations have 256 head dimension and a batch size of 1, while their number of heads are 16 and 15, respectively. As in the previous FlashAttention experiment, the tile size gave the highest throughput for each configuration, and they are executed for 100 warm-up iterations, followed by 100 timed iterations to measure kernel performance. Moreover, the TPC and GPC clustering conditions represent typical approaches driven by heuristics and profiling of expected workload mix, unaware of the true nature of the non-idealities.
Figure 10. Percentage throughput improvement over FlashAttention-4 (uses cluster size 1) for fixed TPC/GPC clustering and non-ideality-informed wave-aware clustering. Wave-awareness matches the better of the two without requiring ad hoc profiling for each shape.
Wave-aware clustering performs better than the FlashAttention 4 baseline that uses cluster size 1 at almost all evaluated kernel configurations and has throughput improvements up to 5.30%. Further, it outperforms strictly or ad hoc using cluster size 2 (TPC clustering) or choosing cluster sizes 4 and 8 (GPC clustering) at all evaluated instances, as shown in Figure 10. The throughput improvements observed with clustering in FlashAttention here are not driven by increased tensor core utilization resulting from reduced data movement. Counterintuitively, these improvements arise from the TDP limit imposed during IC design and its impact on operating frequency. We explain this further in § 6.
Characterization of wave-equalized kernel launches and wave-aware clustering will help kernel performance engineers interpret performance variations and assess whether clustering is beneficial for a given workload and launch configuration. Moreover, it can also guide the selection of alternative launch geometries where workload constraints permit. These findings further reinforce the importance of looking beyond conventional microarchitectural abstractions and accounting for physical non-idealities in IC fabrication when engineering high-performance kernels.
Demystifying the Impact of Thermal Design Power
Thermal design power (TDP) specifies the heat a cooling system must dissipate during sustained operation. The NVIDIA H100 GPU has a TDP of 700 W, which is its approximate sustained power budget. Power consumption consists of static power, primarily from transistor leakage currents, and dynamic power, resulting from transistor switching activity. Dynamic power typically dominates compute-intensive workloads and depends on switching activity, operating frequency, and supply voltage. To regulate power consumption, DVFS dynamically adjusts the operating voltage and frequency based on workload demands and power constraints. During low utilization, DVFS reduces voltage and frequency to save power, while higher utilization allows the GPU to increase its operating frequency, including that of the SMs, provided sufficient power headroom is available. However, as power consumption approaches the TDP-associated power limit, DVFS may constrain or reduce operating frequency to remain within the permitted budget.
Effect of TDP on kernel performance
To evaluate the impact of TDP on kernel performance, we execute FlashAttention kernels with a fixed total token count of 256K, varying the batch size with sequence length. Furthermore, we maintain a constant hidden dimension of 2048 by adjusting the number of attention heads as the head dimension varies. As in the previous FlashAttention experiment, we evaluate all combinations of and , selecting the tile shape that achieves the highest throughput for each configuration. Each configuration is executed for 100 warm-up iterations, followed by 100 timed iterations to measure kernel performance.
Figure 12 shows that the TPC-level cluster size consistently achieves approximately 1.5% and 3% higher throughput than cluster size 1 for head dimensions of 128 and 256, respectively. However, as we see in Table 4, the reduced data movement does not lead to increased tensor core utilization. It is adjusted to account for the reduced active SM limit ratio, as profiling results indicate a lower utilization at GPC-level cluster sizes because it is averaged across all
Figure 12. Increase in throughput of clustering over cluster size 1 across sequence lengths for Flash Attention.
| Head dim. | Seq. len | 1 | 2 | 4 | 8 |
| 128 | 16K | 74.8% | 74.3% | 72.6% | 69.5% |
| 32K | 76.8% | 76.2% | 75.3% | 72.1% | |
| 256 | 16K | 94.3% | 94.1% | 92.5% | 92.3% |
| 32K | 95.9% | 95.8% | 95.1% | 95.2% |
Table 4. Tensor core utilization for Flash Attention. At cluster sizes 4 and 8, utilization is adjusted to the reduced SM limit ratio.
132 SMs in the GPU. However, the active SM limit restricts concurrent execution to 120 SMs, leaving the remaining 12 SMs idle and contributing 0% utilization to the reported average. After accounting for these inactive SMs, the tensor core utilization is comparable across cluster sizes. This indicates that the observed throughput improvements from reduced data movement are not driven by increased tensor core utilization.
We attribute these throughput improvements to the TDP limit and the resulting frequency scaling through DVFS. Figure 13 shows that the SM frequency increases with cluster size. This increase in frequency results in a proportional improvement in kernel throughput at the TPC-level cluster size, i.e., cluster size 2. As illustrated in Figure 11, increasing the cluster size reduces redundant data movement within the GPU, consequently lowering the power consumed by data transfers. For kernels operating near the GPU’s power limit, this reduction in power consumption creates additional power headroom, allowing the GPU’s power management system to increase the SM frequency through DVFS while remaining within the TDP limit. However, this trend does not hold at GPC-level cluster sizes, i.e., cluster sizes 4 and 8, because of the reduced active SM limit discussed in Section 5.
Figure 11. Effect of reduced data movement in a kernel on GPU power allocation under a fixed TDP limit.
Figure 13. Percentage SM frequency and throughput improvement against cluster size 1 for FlashAttention 4 with 32K sequence length and 256 head dimension. Throughput increases proportionally to frequency variation at TPC-level clustering, and it does not follow in GPC-level clustering due to floorsweeping effects.
Effect of TDP on other AI accelerators
To examine whether our observations generalize beyond GPUs, we further evaluate the impact of TDP on Tensorrent’s Wormhole n300d (Tenstorrent), a representative spatial dataflow accelerator. The n300d card integrates two Wormhole chips, each with 64 Tensix cores at 1 GHz, delivering a peak throughput of 64 TFLOP/s per chip.
Figure 14 shows throughput, clock frequency, and power as the number of active cores grows, for two math fidelities: HiFi4, which makes multiple passes through the multiplier for full precision, and LoFi, which makes a single pass. With few active cores, both scale almost linearly at the nominal 1 GHz clock. As more cores become active, power reaches the TDP limit and the firmware lowers the clock to stay within it. HiFi4 uses more energy per operation and hits the limit first, at 32 cores. By 64 cores, its clock falls to 57% of nominal, and its throughput levels off at about 31 TFLOPS, roughly half of ideal scaling. LoFi stays within the limit up to 32 cores and is throttled only to 86% of nominal, reaching about 60 TFLOPS at 64 cores.
Figure 14. Scaling of matrix multiplication on one Wormhole chip under the default power limit. (a) Achieved throughput versus the number of active Tensix cores for the HiFi4 and LoFi math fidelities. (b) Average core clock (normalized to 1 GHz) and power (normalized to the 85 W limit).
These results demonstrate that the impact of TDP limits on kernel performance is non-trivial. Furthermore, understanding these effects provides greater insight into the observed performance characteristics. This reinforces our argument that microarchitectural knowledge alone is insufficient to maximize kernel performance or fully explain the performance achieved.
Discussion
Stringency and portability of wave-equalization
For a kernel launch to be wave-equalized across cluster sizes, it must satisfy Eq. (1). This requires a positive integer such that both and hold, where denotes the number of thread blocks and is the common number of execution waves. Combining these inequalities yields , which is feasible only for . Therefore, the set of thread-block counts eligible for wave-equalized kernel launches is given by . This
set is limited, but the limitation is unavoidable on the H100 because floorsweeping fixes the active SM limits at 132 and 120. Considering kernel performance when making floorsweeping decisions could allow limits that expand this set. The condition also carries over to other GPUs. The microbenchmark in § 4.2 reports the number of unique SMs observed per iteration. Unclustered runs (cluster size 1) give the complete active SM limit, and GPC-clustered runs give the reduced limit. Substituting these two values for 132 and 120 in Eq. (1) gives the condition for the target GPU.
Implications for kernel development systems
Compiler-based kernel optimization [6, 41, 37], manual performance engineering [10, 9, 32], and now agentic kernel development [27, 15, 29, 40, 8], relies on appropriate abstractions and knowledge of the microarchitecture. The physical non-idealities discussed in this work challenge several assumptions underlying these approaches. For instance, widely accepted heuristics, such as amortizing memory accesses (e.g., amortizing L2 accesses through DSMEM) or increasing compute utilization up to saturation, can hurt performance. These effects also go beyond performance, potentially introducing numerical nondeterminism through changes in floating-point evaluation order, as well as power spikes. Through this work, we provide an initial impetus for the community to consider physical non-idealities more broadly. We derive a closed-form solution for manufacturing-yield effects on clustered execution and leave other non-idealities to future work.
Conclusion
This work shows that physical non-idealities hidden by hardware abstractions can significantly affect kernel performance. On NVIDIA H100 GPUs, floorsweeping restricts the number of SMs available for clustered execution, increasing execution waves. Our wave-aware clustering strategy addresses these constraints, improving the throughput of the Hopper version of FlashAttention 4 by up to 5.30% over the unclustered baseline. We also show that DSMEM-based reductions in data movement create power headroom for higher SM frequencies, benefiting even compute-bound kernels under the TDP limit. The influence of power constraints extends beyond GPUs, as our analysis of Tenstorrent’s Wormhole accelerator reveals similar limitations on throughput scaling. Together, these findings reinforce the need to consider physical non-idealities when analyzing and optimizing kernels across accelerator architectures.
Acknowledgments
This work is partially supported by the Ministry of Education, Singapore, under Tier 3 grant MOET32024-0003 and the Advanced Research and Technology Innovation Centre (ARTIC), the National University of Singapore under Grant AFP-RP6.
References
- [1]Afzal, A., Li, M. M., and Panzlaff, M. Modeling and Chasing the Energy-Efficiency Sweet Spots in Modern GPUs. arXiv preprint arXiv:2607.00819, 2026. doi: 10.48550/arXiv.2607.00819.DOI
- [2]Aguilera, P., Lee, J., Farmahini-Farahani, A., Morrow, K., Schulte, M., and Kim, N. S. Process Variation-Aware Workload Partitioning Algorithms for GPUs Supporting Spatial-Multitasking. In 2014 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 1–6. IEEE, 2014.
- [3]Andersch, M., Palmer, G., Krashinsky, R., Stam, N., Mehta, V., Brito, G., and Ramaswamy, S. Nvidia hopper architecture in-depth, 2022. URL https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/. Accessed: 2026-03-16.
- [4]Bogdanov, Y. I., Bogdanova, N. A., and Dshkhunyan, V. L. Statistical yield modeling for IC manufacture: Hierarchical fault distributions, 2003. URL https://arxiv.org/abs/physics/0303039.
- [5]Chen, K. and Patel, D. Dissecting Nvidia Blackwell - Tensor Cores, PTX Instructions, SASS, Floorsweep, Yield. Technical report, SemiAnalysis, 2026. URL https://newsletter.semianalysis.com/p/dissecting-nvidia-blackwell-tensor.
- [6]Chen, T., Moreau, T., Jiang, Z., Zheng, L., Yan, E., Shen, H., Cowan, M., Wang, L., Hu, Y., Ceze, L., Guestrin, C., and Krishnamurthy, A. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pp. 578–594, 2018.arxiv.org/abs/1802.04799
- [7]Cunningham, J. The use and evaluation of yield models in integrated circuit manufacturing. IEEE Transactions on Semiconductor Manufacturing, 3(2):60–71, 1990. doi: 10.1109/66.53188.DOI
- [8]Dai, W., Wu, H., Yu, Q., Gao, H.-a., Li, J., Jiang, C., Lou, W., Song, Y., Yu, H., Chen, J., Ma, W.-Y., Zhang, Y.-Q., Liu, J., Wang, M., Liu, X., and Zhou, H. CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation. arXiv preprint arXiv:2602.24286, 2026.arxiv.org/abs/2602.24286
- [9]Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=mZn2Xyh19Ec.arxiv.org/abs/2307.08691
- [10]Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. Advances in neural information processing systems, 35:16344–16359, 2022.arxiv.org/abs/2205.14135
- [12]Duluk Jr, J. F., Hirota, G., Krashinsky, R., Palmer, G., Tuckey, J., Nadadhur, K., Johnson, P. B., and Joginipally, P. Virtualizing Hardware Processing Resources in a Processor, October 21 2025. US Patent 12,449,473.
- [13]Fan, X., Wang, J., Cui, W., Zhao, H., Huang, Z., Zhou, Y., Qiu, Y., Sun, S., He, B., Chen, Q., and Guo, M. Dissecting How Die Scaling Breaks GPU Fine-grained Scheduling. arXiv preprint arXiv:2609.24270, 2026.arxiv.org/abs/2609.24270
- [14]Hagedorn, B., Elliott, A. S., Barthels, H., Bodik, R., and Grover, V. Fireiron: A data-movement-aware scheduling language for gpus. In Proceedings of the ACM International Conference on Parallel Architectures and Compilation Techniques, pp. 71–82, 2020.
- [15]Han, Y., Fan, K., Jiang, H., Xu, W., Xie, W., Zhang, R., Zhu, C., and Zhang, Y. EGG: An Expert-Guided Agent Framework for Kernel Generation. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=8nti23Zqkt.arxiv.org/abs/2606.26758
- [17]Lee, J., Sathisha, V., Schulte, M., Compton, K., and Kim, N. S. Improving Throughput of Power-Constrained GPUs using Dynamic Voltage/Frequency and Core Scaling. In 2011 International Conference on Parallel Architectures and Compilation Techniques, pp. 111–120. IEEE, 2011.DOI
- [18]Lin, J., Jiang, Z., Song, Z., Zhao, S., Yu, M., Wang, Z., Wang, C., Shi, Z., Shi, X., Jia, W., Liu, Z., Wang, S., Lin, H., Liu, X., Panda, A., and Li, J. Understanding stragglers in large model training using what-if analysis. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25), pp. 483–498, 2025.
- [19]Luo, W., Fan, R., Li, Z., Du, D., Liu, H., Wang, Q., and Chu, X. Dissecting the NVIDIA Hopper Architecture through Microbenchmarking and Multiple Level Analysis. arXiv preprint arXiv:2501.12084, 2025a.arxiv.org/abs/2501.12084
- [20]Luo, X., Liu, Z., Zhou, Y., Fang, S., Huang, Z., Feng, Y., Zhang, C., Sun, S., Zheng, Z., Leng, J., and Guo, M. Clusterfusion: Expanding operator fusion scope for LLM inference via cluster-level collective primitive. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025b. URL https://openreview.net/forum?id=eX0m4qMYVN.arxiv.org/abs/2508.18850
- [21]Ning, A., Tziantzioulis, G., and Wentzlaff, D. Supply chain aware computer architecture. In Proceedings of the 50th Annual International Symposium on Computer Architecture, ISCA ’23, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400700958. doi: 10.1145/3579371.3589052. URL https://doi.org/10.1145/3579371.3589052.
- [27]Ouyang, A., Guo, S., Arora, S., Zhang, A. L., Hu, W., Re, C., and Mirhoseini, A. KernelBench: Can LLMs Write Efficient GPU Kernels? In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=yeoN1iQT1x.arxiv.org/abs/2502.10517
- [29]Ran, D., Xie, S., Ji, M., Liu, A., Wu, M., Cao, Y., Guo, Y., Yu, H., Li, L., Hu, Y., Yang, W., and Xie, T. KernelBand: Steering LLM-based Kernel Optimization via Hardware-Aware Multi-Armed Bandits. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=d8e2jBfMpE.arxiv.org/abs/2511.18868
- [30]Rountree, B., Ahn, D. H., De Supinski, B. R., Lowenthal, D. K., and Schulz, M. Beyond DVFS: A First Look at Performance under a Hardware-Enforced Power Bound. In 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum, pp. 947–953. IEEE, 2012.DOI
- [31]Schoonhoven, R., Veenboer, B., Van Werkhoven, B., and Batenburg, K. J. Going Green: Optimizing GPUs for Energy Efficiency through Model-Steered Auto-Tuning. In 2022 IEEE/ACM International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS), pp. 48–59. IEEE, 2022.arxiv.org/abs/2211.07260
- [32]Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision. Advances in Neural Information Processing Systems, 37:68658–68685, 2024.arxiv.org/abs/2407.08608
- [34]Tenstorrent. TT-Metal: Low-level programming framework for tensorrent hardware. https://github.com/tenstorrent/tt-metal/tree/main/tt-metal/, 2026a. Accessed: 2026-10-29.
- [35]Tenstorrent. TT-NN: Tenstorrent neural network library. https://github.com/tenstorrent/tt-metal/tree/main/ttnn, 2026b. Accessed: 2026-10-29.
- [36]Thakkar, V., Ramani, P., Cecka, C., Shivam, A., Lu, H., Yan, E., Kosaian, J., Hoemmen, M., Wu, H., Kerr, A., Nicely, M., Merrill, D., Blasig, D., Atluri, A., Qiao, F., Majcher, P., Springer, P., Hohnerbach, M., Wang, J., and Gupta, M. CUTLASS: Hopper Warp-Specialized GEMM Example, 2026. URL https://github.com/NVIDIA/cutlass/blob/0b55a2f691d69981583568fd9eb69687b1f0de8a/examples/48_hopper_warp_specialized_gemm/48_hopper_warp_specialized_gemm.cu. Version 48.0, Example 48. Commit 0b55a2f691d69981583568fd9eb69687b1f0de8a.
- [37]T illet, P., Kung, H. T., and Cox, D. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, pp. 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/3315508.3329973. URL https://doi.org/10.1145/3315508.3329973.
- [39]Zadouri, T., Hoehnerbach, M., Shah, J., Thakkar, V., and Dao, T. FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling. Proceedings of Machine Learning and Systems, 8:912–926, 2026.arxiv.org/abs/2603.05451
- [40]Zhang, Z., Wang, R., Li, S., Luo, Y., Hong, M., and Ding, C. CudaForge: An Agent Framework with Hardware Feedback for CUDA Kernel Optimization. arXiv preprint arXiv:2511.01884, 2025.arxiv.org/abs/2511.01884
- [41]Zheng, L., Jia, C., Sun, M., Wu, Z., Yu, C. H., Hai-Ali, A., Wang, Y., Yang, J., Zhuo, D., Sen, K., Gonzalez, J. E., and Stoica, I. Ansor: Generating High-Performance Tensor Programs for Deep Learning. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pp. 863–879, 2020.arxiv.org/abs/2006.06762










