WorksComputer Science
SparStencil: Retargeting Sparse Tensor Cores to Scientific Stencil Computations via Structured Sparsity Transformation
Abstract
Sparse Tensor Cores offer exceptional performance gains for AI workloads by exploiting structured 2:4 sparsity. However, their potential remains untapped for core scientific workloads such as stencil computations, which exhibit irregular sparsity patterns.This paper presents SparStencil, the first system to retarget sparse TCUs for scientific stencil computations through structured sparsity transformation. SparStencil introduces three key techniques: (1) Adaptive Layout Morphing, which restructures stencil patterns into staircase-aligned sparse matrices via a flatten-and-crush pipeline; (2) Structured Sparsity Conversion, which formulates transformation as a graph matching problem to ensure compatibility with 2:4 sparsity constraints; (3) Automatic Kernel Generation, which compiles transformed stencils into optimized sparse MMA kernels via layout search and table-driven memory mapping. Evaluated on 79 stencil kernels spanning diverse scientific domains, SparStencil achieves up to 7.1x speedup (3.1x on average) over state-of-the-art framework while reducing code complexity and matching or exceeding expert-tuned performance in both compute throughput and memory efficiency.
Introduction
The rapid ascent of artificial intelligence (AI) workloads has fundamentally reshaped the computing landscape. Modern accelerators—especially GPUs—are now engineered not for general-purpose computing, but to saturate the needs of AI-native operations, such as matrix multiplications with structured sparsity. At the heart of this shift are Sparse Tensor Cores, which exploit fine-grained sparsity patterns (e.g., 2:4 sparsity) to skip redundant zero-element computations and significantly boost performance.
Yet, scientific computing has largely remained on the periphery of this acceleration revolution. While sparse TCUs have become a cornerstone of AI performance, their capabilities remain underutilized in scientific applications, where data and compute structures diverge sharply from those in AI. This disconnect reflects a deeper tension: hardware is evolving toward AI-native execution models, but science remains anchored in patterns that predate them.
Nowhere is this divergence more evident than in stencil computations, the backbone of applications such as fluid dynamics [20, 25], weather prediction [1, 5], and earth modeling [21]. These computations iteratively update grid values based on local neighborhoods and account for over 75% of the runtime in long-running simulations [11]. Their ubiquity and cost have earned them recognition as one of the “seven dwarfs” of scientific computing [2, 3].
However, stencil computations yield clustered, irregular sparsity patterns that fundamentally differ from the dense or uniformly sparse structures seen in AI. When mapped into matrix formats for GPU acceleration, these patterns often lead to dense masking, padding, and inefficient memory access—all of which violate the alignment constraints of sparse TCUs and degrade utilization. Prior approaches have largely focused on suppressing this sparsity through dense emulation, missing the opportunity to align it with the execution model of sparse TCUs.
This paper asks a fundamental question: Can we reimagine scientific computing not as a victim of hardware divergence but as a domain whose latent structure can be transformed to align with AI-native accelerators?
We argue yes. And we answer with SparStencil, a stencil computing system that retargets sparse tensor cores originally designed for AI workloads to accelerate scientific stencil computations. Instead of coercing stencil computations into dense formats, SparStencil embraces their clustered sparsity, restructures it into 2:4-compatible formats, and compiles it into optimized sparse TCU kernels—automatically and portably.
SparStencil is guided by four key insights that reshape how stencil sparsity can be efficiently exploited on sparse tensor cores. (1) Sparsity is not overhead—it’s alignment potential. Prior approaches treated stencil-induced sparsity as overhead to be suppressed in order to fit dense compute units. Instead, we view this sparsity as an alignment opportunity for specialized sparse execution. (2) Residual sparsity is a leverage point. Despite masking and padding, stencil workloads retain high residual sparsity (50–80%). These conditions are poorly handled by dense TCUs but can be harnessed by sparse TCUs once the structure is exposed. (3) The irregularity does not mean a lack of structure. Though irregular at first glance, stencil-induced sparsity exhibits consistently clustered spatial patterns. These can be systematically reshaped into alignment-friendly forms with the right abstraction and transformation. (4) Structure-aware compilation closes the loop. Exploiting sparse TCUs demands more than format conversion—it requires compilers that understand and reason over structure, automating the path to performance.
To realize these insights, SparStencil consists of three tightly integrated stages:
First, Adaptive Layout Morphing reformulates diverse stencil patterns into unified matrix-matrix representations through a two-step flatten-and-crush pipeline. It systematically eliminates redundant data accesses while exposing staircase-aligned sparsity, a regularized layout pattern that serves as the structural backbone for downstream hardware-oriented transformations.
Second, Structured Sparsity Conversion transforms staircase-aligned sparsity into 2:4-compatible formats required by sparse TCUs. It is achieved by formulating a column-wise conflict graph, where nodes represent matrix columns and edges encode alignment violations. A hierarchical two-level matching algorithm is then applied to resolve conflicts, ensuring valid 2:4 alignment while minimizing zero-padding and maximizing TCU utilization.
Third, Automatic Kernel Generation compiles the transformed computation into high-performance sparse MMA kernels. It integrates layout exploration, table-driven memory mapping, and code synthesis into a fully automated backend, delivering expert-level performance without manual tuning or architecture-specific reengineering.
We evaluate SparStencil across 79 real-world stencil kernels drawn from a wide spectrum of application domains, including partial differential equation solvers, fluid dynamics, lattice Boltzmann methods, phase field models, and geophysical simulations. Compared to state-of-the-art baselines, SparStencil achieves an average 3.1x speedup, peaking at 7.1x, while significantly reducing code complexity. Our automatic compiler approach matches or exceeds manually optimized expert code in both compute throughput and memory efficiency.
To our knowledge, SparStencil is the first system to map scientific stencil computations onto sparse AI accelerators—not by tuning low-level kernels but by structurally rethinking how stencil workloads are represented and executed. This transformation retargets sparse Tensor Cores to scientific domains, and lays the groundwork for a broader shift toward accelerator-centric design for scientific computing.
Background and Challenges
Sparse Tensor Core
Sparse Tensor Core is a cutting-edge hardware architecture specifically designed to accelerate sparse matrix multiplication, as described in Equation (1).
where is a binary mask matrix in which each group of 4 consecutive elements in a row contains exactly 2 elements equal to 1 and 2 elements equal to 0. This constraint, referred to as 2:4 sparsity, is represented as follows:
While sparse TCUs strictly enforce this 2:4 sparsity pattern, sub-2:4 patterns (0:4 and 1:4) can still be processed. This is achieved by treating one or two elements in the 4-element block as nonzero, even if they are zero. Since multiplying by zero does not affect the result, this approach preserves computational correctness.
At the hardware level, there is an additional constraint; sparse TCUs partition matrices into uniformly sized fragments for computation. Regardless of matrix size variations, these fragments remain fixed (e.g., 16 × 16 × 8, 16 × 32 × 8). [33]
Despite the stringent 2:4 sparsity constraints and invariant fragment size limitations imposed by sparse TCUs, it offers substantial computational potential. Across all supported data types (FP16, BF16, and TF32), sparse TCUs achieve up to 32x speedup over traditional Fused Multiply-Add (FFMA) operations and consistently deliver 2x the performance of dense TCUs, presenting a substantial opportunity to accelerate fundamental operators in traditional scientific computing domains, such as stencil computations. [32]
Stencil Computation
Stencil computations play a critical role in numerous scientific and engineering applications, which are extensively involved in various domains from physical simulations to machine learning. Stencil is also included as one of the seven computational dwarfs presented in the Berkeley View and arises as a principal class of floating-point kernels in high-performance computing.
A stencil contains a pre-defined pattern that updates each point in a -dimensional spatial grid iteratively along the time dimension. These patterns are typically categorized into two main types: star and box stencils. In a star stencil, the computation involves a weighted sum of the central point and its neighboring points along individual dimensions. Conversely, a box stencil computes the weighted sum over a neighborhood that forms a square or cube around the central point in 2D or 3D, respectively.
Divergence and Opportunities
Sparse TCUs have provided dedicated acceleration for the most fundamental matrix multiplication operations in the AI field. However, essential scientific operators such as stencil, face significant challenges in effectively harnessing AI-specific accelerators like TCUs, despite its immense computational potential being a highly valuable yet underutilized resource.
For instance, a direct translation of stencil computations to naive matrix-vector multiplication via a canonical im2row-style approach exhibits severely low accelerator utilization. As shown in Figure 1(a), 87.5% columns are wasted due to the fixed fragment dimensions inherent in TCU architectures.
Figure 1. Architectural mapping challenges of stencil computations on TCUs. (a) Naive matrix-vector mapping achieves only 12.5% compute utilization ( active columns) on an TCU fragment due to rigid dataflow constraints. (b) The clustered sparsity and structured sparsity in TCUs.
Fortunately, some efforts have already been made to bridge this divergence by mapping stencil computations onto dense TCUs, such as TCStencil and ConvStencil. However, these adaptations inevitably introduce more than 50% clustered sparsity, leading to the underutilization of more than half of the TCU’s computational throughput. The clustered sparsity, as illustrated in Figure 1(b), refers to sparsity patterns containing contiguous regions with excessive zero concentrations that violate the structured sparsity requirements (2:4 sparsity) of sparse TCUs.
While clustered sparsity has traditionally been seen as a barrier to dense TCUs, it offers a unique opportunity in the context of sparse TCUs. If we can convert the clustered sparsity of the matrix into structured 2:4 sparsity, we can seamlessly integrate sparse TCUs into stencil computations. Instead of focusing on eliminating zero elements, this approach enables us to leverage the hardware to significantly increase computational density. Consequently, rather than viewing sparsity as a limitation, we propose that it should be seen as an opportunity to harness the capabilities of sparse TCUs, unlocking new pathways for stencil computations.
Challenges
Despite the opportunity, systematically and efficiently mapping stencil computations with various stencil kernels onto sparse TCUs is not a straightforward task. In this subsection, we outline and discuss three key challenges:
Firstly, how can we map stencil kernels onto TCUs with low redundancy and sparsity-awareness? The matrix-based transformation of stencil computations inherently introduces extensive duplication, as sliding-kernel overlap causes each input element to appear repeatedly across the matrix, leading to significant memory overhead. Moreover, existing works (e.g., TCStencil and ConvStencil) lack sparsity awareness, resulting in irregular patterns that cannot be aligned with sparse TCUs. Lacking the ability to restructure such sparsity into a usable form, they often choose to ignore it entirely and fall back to dense computation.
Secondly, how can we convert sparsity into the fine-grained structured patterns required by sparse TCUs? Sparse TCUs demand element-level alignment: every row must contain exactly two non-zeros within each four-element group. Yet stencil sparsity emerges from coarse-grained, spatially entangled structures that defy such regularity. This mismatch in granularity makes element-wise adjustment infeasible—nonzeros cannot simply be shuffled without corrupting stencil semantics. At the same time, coarse layout modifications—such as reordering rows or columns—often introduce cascading disruptions, where changes in one region break alignments elsewhere. As a result, even small transformations risk destabilizing the entire sparsity layout.
Thirdly, how can we automatically generate highly optimized CUDA kernels with optimal layout mapping? Stencil kernels admit a vast space of layout-to-execution mappings, each inducing distinct sparsity patterns, memory traffic, and alignment constraints. This space is too large and irregular for manual tuning, forcing existing TCU-based methods to rely on hand-tuned schedules tailored to specific kernel sizes. Moreover, the intricate interplay between sparsity patterns, TCU constraints, and kernel performance metrics also presents a significant challenge in defining the search space for identifying the optimal layouts that maximize TCU utilization.
SparStencil
In this section, we will provide a detailed introduction to SparStencil. It is a novel approach that leverages sparse TCUs for stencil computations for the first time, extending the boundaries of sparse TCUs for stencil computation.
Adaptive Layout Morphing
Adaptive Layout Morphing is designed with two fundamental objectives: (1) to reformulate stencil computations into matrix representations flexibly, and (2) to implement a sparsity-aware strategy that refines clustered sparsity into a more structured, staircase-like sparsity pattern while consolidating consecutive redundant elements. These objectives are achieved through Stencil Flattening and Duplications Crush, respectively.
Stencil Flattening generalizes the transformation of stencil computations—originally incompatible with TCU—into a unified vector-matrix multiplication format suitable for TCUs. Specifically, the approach unfolds the stencil kernel weights into a single-row vector, the kernel vector, while simultaneously reshaping each region of the input data covered by the sliding kernel into corresponding column vectors. Concatenating these column vectors row-wise forms the input matrix. Consequently, stencil computations, previously executed as repeated weighted summations, become a vector-matrix multiplication. Figure 2 illustrates the stencil flattening process for a kernel applied to a input matrix.
Figure 2. Example of the stencil flattening process.
However, directly applying stencil flattening to TCUs often leads to low hardware utilization, as the kernel vector occupies only one row of the TCU fragment, leaving other rows unused. Moreover, this transformation inherently generates significant redundancy: the input matrix contains numerous duplicated elements, aggravating memory-bound constraints inherent in stencil computations.
To address these issues, we introduce Duplicates Crush.4 By crushing redundant elements within the input matrix, it adaptively transforms the vector-matrix multiplication into matrix-matrix multiplication according to the TCU fragment size while intentionally guiding a staircase-like sparsity pattern, laying the groundwork for the subsequent transformation. The Duplicates refers to the element duplicates caused by the horizontal and vertical sliding of the kernel in stencil computations, which can be categorized along the two directions into two types: horizontal duplicates and vertical duplicates.
The horizontal duplicates are illustrated in Figure 3(a). During horizontal sliding, the stencil kernel repeatedly covers certain elements due to overlapping regions at adjacent positions. These repeated elements are reflected in matrix after Stencil Flattening, such as and , and , as well as and . In Figure 3(a), is partitioned into three submatrices, where each consists solely of elements from the -th input row. Due to horizontal kernel sliding, adjacent kernels in Figure 3(a) share two elements per row. This overlap induces horizontal duplicates in , with adjacent columns in each sharing two identical elements. Generally, these horizontal duplicates can be mathematically expressed as:
Figure 3. The distribution of duplicates after stencil flattening. (a) The horizontal duplicate distribution at the element level within a sub-matrix. (b) The vertical duplicates distribution between sub-matrices at the sub-matrix level.
where , and is a submatrix of .
The vertical duplicates are caused by the vertical sliding of the stencil kernel. As illustrated in Figure 3(b), elements from the first row and the second row are covered simultaneously by these adjacent kernels. After flattening, these repeated elements appear as submatrices in , where the elements from the first row become submatrix and the elements from the second row become submatrix . Viewing the matrix as consisting of submatrix , each composed of the elements from the -th row, the vertical redundancy in manifests as duplication between these submatrices. Mathematically, this can be expressed as:
where denotes the element located at the position within the submatrix of .
After analyzing the distribution of duplicates, we now introduce the Duplicates Crush. It eliminates the duplicates in matrix after Stencil Flattening, while expanding the vector into a matrix and intentionally introducing a staircase-like sparsity pattern within to prepare for the transformation. Similar to how duplicates are distributed, the crushing procedure is categorized along the stencil kernel’s two sliding directions: horizontal and vertical.
We begin with horizontal duplicate crushing, as characterized in Equation (3). Since these duplicates are confined within each sub-matrix of , it suffices to illustrate the process at the sub-matrix level for uniform application across all . The procedure, illustrated in Figure 4(a), proceeds as follows: (1) Consider the flattened matrices and in sub-matrix form, and select and as representative examples. (2) In , duplicate values appear diagonally across columns. These are eliminated by retaining one representative per group and reorganizing the unique entries into a single-column format. (3) Concurrently, is vertically expanded into multiple rows with deliberate zero-padding to align with the new structure, forming a staircase-like sparsity pattern. Although not directly compatible with sparse TCUs, the staircase pattern imposes a regular layout that facilitates the subsequent transformation.
Figure 4. The process of Duplicates Crush. Figure (a) illustrates the horizontal duplicates crush at the element level within a sub-matrix. Figure (b) shows the vertical duplicates crush between sub-matrices at the sub-matrix level.
We next describe vertical duplicate crushing, which targets duplicates across sub-matrices in , as defined in Equation (4). The crushing process is again applied at the sub-matrix level, as illustrated in Figure 4(b), and proceeds as follows: (1) After the horizontal duplicate crushing, vertical duplicates in exhibit diagonal alignment at the sub-matrix level, similar to the horizontal case. (2) Redundant sub-matrices across the three columns are removed, retaining one representative per group. The remaining sub-matrices are then reorganized into a single column structure. (3) Meanwhile, is similarly expanded into multiple rows at the sub-matrix level.
The above procedures illustrate both horizontal and vertical duplicate crushing. In general, horizontal crushing can be applied to every columns within each , while vertical crushing merges every columns of . The values of and are determined during the subsequent layout exploration phase.
Structured Sparsity Conversion
Although Adaptive Layout Morphing guides sparsity into a staircase-like structure, the resulting pattern remains irregular and violates the 2:4 constraint required by sparse TCUs. To bridge this gap, we propose Structured Sparsity Conversion, a general transformation that reorganizes intermediate sparsity into a hardware-aligned 2:4 format.
A core component of this process is the Permutation Invariant Transformation (PIT), which preserves computation semantics during layout reordering. PIT simultaneously permutes the columns of and the corresponding rows of along the -dimension, ensuring the result of remains unchanged. Formally, let and . Then,
guaranteeing that the multiplication remains invariant under any shared permutation —a crucial property for enabling valid sparse pattern adaptation.
While PIT guarantees correctness, not all permutations yield valid 2:4 structured sparsity. To make the search space tractable, we observe that the 2:4 constraint can be equivalently expressed as a composition of 1:2 and 0:2 subpatterns (§ 2.1). This motivates the core problem of Structured Sparsity Conversion: finding a permutation that rearranges nonzeros into conflict-free 1:2 or 0:2 groupings while minimizing padding. We formalize this as follows: Given a matrix derived from layout morphing with parameters , we construct an augmented matrix by appending zero columns. The goal is to find a permutation such that the transformed matrix satisfies a 1:2 or 0:2 sparsity pattern while minimizing . We model this as a graph-theoretic matching problem:
Definition 1 (Conflict Graph). Given , its conflict graph is , where represents the columns of , and if and only if such that and .
Definition 2 (Augmented Matching Graph). Given zero columns , the augmented graph is , where is inherited from the original conflict graph and no additional edges are introduced.
Definition 3 (Valid Matching). A matching is valid if it satisfies: (i) coverage: each appears in exactly one edge in ; and (ii) conflict-freedom: .
Problem 1 (Minimum Zero-Column Matching). Find the minimal such that a valid matching exists in , with .
To solve the problem, we introduce the following definition:
Definition 4 (-Staircase Matrix). A matrix has the -staircase property if:
where denotes indices.
As described in § 3.1, the matrix can be partitioned into blocks , exhibiting a self-similar -staircase structure: (i) Global Staircase: treating each block as a scalar, the block-level matrix satisfies the -staircase property; (ii) Local Staircase: each nonzero block individually satisfies the same property. Figure 5 (a) illustrates this self-similarity.
Figure 5. An example of Sparsity Conversion. (a) The self-similar staircase property in the matrix undergoing layout morphing. (b) The global conflict graph and the local conflict graph. (c) The process of hierarchical two-level matching algorithm.
The self-similar structure of induces two levels of conflict graphs, as illustrated in Figure 5 (b): (i) Global Conflict Graph : each node corresponds to a block column , with iff such that and ; (ii) Local Conflict Graph : each node corresponds to a column in , with iff such that and .
After constructing two-level conflict graphs, the following theorem can be derived based on the -staircase property:
Theorem 1 (Non-Conflict Property). Let be a matrix with a staircase pattern and its conflict graph . Then any two nodes separated by at least hops are conflict-free:
Proof. Conflict free: By the staircase offset, columns and satisfy:
Thus, no shared nonzero rows exist.
With Theorem 1, we propose Hierarchical Two-Level Matching to solve Problem 1, which is shown in Algorithm 1. The process of Algorithm 1 is shown in Figure 5(c). Furthermore, Theorem 2 establishes the correctness and optimality of this algorithm.
Theorem 2 (Correctness and Optimality). Let be the matching returned by Algorithm 1, and let be the set of inserted zero-columns. Then: (i) Validity: for all , we have ; (ii) Minimality: achieves the minimum value in Problem 1.
Proof. (i) Validity. Let . If , then ; if , then . Hence, all matched pairs satisfy . (ii) Minimality. Since all subgraphs are structurally identical, it suffices to analyze a single subgraph of size . If , then: when is even, a perfect matching exists; when odd, exactly one node remains unmatched—this is trivially optimal. If , the algorithm pairs with , matching nodes. Assume a better matching exists. Validity requires for any pair , so no pair with can exist, contradicting the assumption. Thus, the matching is optimal.
Algorithm 1: Hierarchical Two-Level Matching
Input: Global graph with nodes, subgraph size , stencil size
Output: Perfect matching
Initialize , , ;
for to do
if unmatched and then
;
Initialize , ;
for each unmatched do
for to do
;
if unmatched then
if then
;
else
Add zero node , ;
Initialize ;
for , to do
;
;
Theorem 2 ensures that Algorithm 1 achieves both correctness and optimality with linear complexity , since each vertex is traversed at most once. While this is guaranteed under the -staircase assumption, our approach remains broadly applicable even when the input deviates from this structure. In such cases, the same conflict graph formulation allows us to fall back to the classical Blossom algorithm [12] to compute maximum matchings over arbitrary sparsity patterns. Despite its worst-case complexity of , the Blossom algorithm remains practical in our setting, as real-world stencil kernels typically yield small and sparse conflict graphs.
Automatic Kernel Generation
Despite methods for leveraging sparse TCUs, manual tuning for sparse TCUs is both labor-intensive and inherently suboptimal due to the high variability of stencil kernels and TCU fragments. To address this issue, we propose Automatic Kernel Generation, a compiler framework that enables end-to-end scheduling through two phases: (1) layout exploration dynamically determines the optimal data layout by automatically searching over a tunable space, minimizing memory transfer overhead while maximizing computational density. (2) Code Generation emits expert-level CUDA kernels via metadata-driven sparse MMA, lookup-based memory mapping, and double-buffered pipelines, consistently outperforming hand-tuned implementations.
Layout Exploration. The configuration parameters in § 3.1 modulate data layout patterns and induce clustered sparsity in TCU, where and denote the number of consecutive columns to be merged in and . The configuration creates distinct memory-computation trade-offs that necessitate automated exploration to identify optimal configurations. To systematically evaluate these configurations, we adopt an analytical model tailored for stencil computations on TCU [9], shown in Equation (6), 7 and 8.
Compute Evaluation. As defined in Equation (7), depends on , with , , and fixed by hardware. Given an input, a stencil kernel, and a TCU fragment of size , applying layout morphing yields , with , , and . The corresponding MMA count is:
Substituting into Equation (7) yields .
Memory Evaluation. SparStencil stores the original input I and the pre-converted kernel matrix A′ in global memory. As stencil kernels are typically small, the footprint of A′ is negligible compared to I, which remains layout-invariant and thus independent of . Consequently, and in Equation (8) are constant, while and depend on A′ and B′ after layout morphing:
We compute via Equation (8), and total latency using Equation (6). The optimal configuration is selected by exhaustive search:
For each layout candidate , SparStencil conducts static graph analysis to construct a conflict graph from the symbolic sparsity of A′. A provably optimal permutation is then derived via Algorithm 1 to satisfy the 2:4 sparsity constraint under structural constraints. The resulting permutation is applied via PIT, yielding a layout conforming to sparse TCU execution.
Code Generation. With optimal parameters and the PIT transformation applied, the resulting irregular stencil patterns and hardware-specific layouts create highly scattered memory access patterns that are impractical to analyze manually. To address this, SparStencil automates CUDA kernel generation by integrating metadata encoding, lookup table acceleration, and a highly optimized CUDA template.
| Symbol | Description |
| , , | Total, compute, and memory time |
| / | Shared memory read/write volume |
| , | MMA operations number, TCUs number |
| / | Global memory read/write volume |
| , | Cycles per TCU op, GPU core frequency |
| , | Bandwidth of global/shared memory |
Table 1. Summary of symbols used in performance modeling.
Metadata. Computing sparse matrix multiplication on sparse TCUs necessitates the generation of metadata to indicate the positions of non-zero elements in matrix . The metadata generation process is straightforward: it involves traversing matrix , recording the indices of nonzero elements, and encoding them into a compact format.
Lookup Table. During layout transformation, computing address pointer offsets for transferring data from global to shared memory introduces significant computational overhead due to frequent integer division and modulus operations, which are inefficient on GPUs. Moreover, these computations often exhibit redundancy across different processing blocks. To mitigate this, we precompute pointer offsets on the host and store them in lookup tables, which are then passed to the CUDA kernel. This approach eliminates redundant computations, significantly improving memory access efficiency.
Based on prior analysis, SparStencil instantiates optimized CUDA kernels from a predefined template tailored for memory efficiency, compute-transfer overlap, and sparse TCU utilization. The kernel follows a three-stage pipeline: (1) asynchronous loading via lookup tables to prefetch global memory into shared memory; (2) sparse MMA execution using metadata, interleaved with the next load to maximize throughput; and (3) writing results back to global memory, completing a double-buffered loop.
Evaluation
Setup
Implementation. We implement SparStencil with CUDA C++ and PTX. SparStencil is compiled with NVCC 12.4.
Machine. Our platform is composed of an AMD EPYC 7543 processor and an Nvidia A100 GPU with sparse Tensor Core. The A100 GPU is connected to the motherboard via PCIe Gen4. The A100 GPU is equipped with 108 SMs, each containing 4 sparse Tensor Cores [31].
State-of-the-arts. We compare SparStencil with a wide range of state-of-the-arts, including cuDNN, AMOS, Brick, DRStencil, TCStencil, and ConvStencil in FP16. Moreover, considering that ConvStencil employs 3x temporal fusion for small kernels, SparStencil adopts the same approach in our comparative evaluation to ensure a fair performance comparison.
Benchmarks. We use various stencil kernels with different shapes as benchmarks. The specifics are detailed in Table 2 [18, 46].
| Kernel | Points | Problem Size | Block |
| Heat-1D | 3 | 10240000 × 10000 | 1024 |
| 1D5P | 5 | 10240000 × 10000 | 1024 |
| Heat-2D | 5 | 10240 × 10240 × 10240 | 32 × 64 |
| Box-2D9P | 9 | 10240 × 10240 × 10240 | 32 × 64 |
| Star-2D13P | 13 | 10240 × 10240 × 10240 | 32 × 64 |
| Box-2D49P | 49 | 10240 × 10240 × 10240 | 32 × 64 |
| Heat-3D | 7 | 1024 × 1024 × 1024 × 1024 | 8 × 64 |
| Box-3D27P | 27 | 1024 × 1024 × 1024 × 1024 | 8 × 64 |
Table 2. Configuration for Stencil Benchmarks
Metrics. Most works on stencil evaluate performance using GStencil/s [8] [28, 45, 46, 47, 48] (Gigastencils per second, denoting the number of stencil points updated per second) as a metric. We also adopt this metric, as defined in Equation 12.
where denotes the number of iterations, denotes the dimensionality of the stencil, denotes the size of the -th dimension, and denotes the total execution time in seconds.
Preprocessing Overhead Analysis
Figure 8 illustrates the overhead associated with metadata, lookup table, and transformation across different stencils.
Figure 8. The percentages of overhead across various stencils. TS means Transformation, MD means Metadata and LUT means Lookup Table.
Overall overhead is minimal and quickly amortized. For 1D stencils, a brief spike is observed (1D-5P: LUT peaks near 30%) but rapidly diminishes. Higher-dimensional stencils, such as Box-3D27P, show consistently low and stable overhead due to favorable reuse patterns.
2D stencils (Box-2D9P, Box-2D49P) exhibit a slightly higher sustained overhead, occasionally exceeding 10%. Compared to 1D stencils with simpler memory access and 3D stencils with higher data reuse in tensor cores, 2D stencils have a less optimal balance between sparsity transformation and computation-to-memory ratio, leading to relatively higher preprocessing costs.
State-of-the-art Comparison
Figure 6 compares SparStencil against state-of-the-art baselines. SparStencil consistently outperforms all reference methods, achieving 2.89x–60.35x speedup over cuDNN, which lacks Tensor Core support for stencil patterns and underperforms on one-channel convolutions. In contrast, AMOS falls short due to inefficient stencil-to-TCU mapping.
Figure 6. Performance comparison of SparStencil with state-of-the-arts.
Against stencil-specialized baselines like DRStencil and ConvStencil, SparStencil retains a clear advantage. On Star-2D13P and Box-2D49P, it achieves up to 38.89% higher performance, attributed to reduced memory traffic via layout search and superior compute throughput from sparse TCUs. This synergy of memory and compute underpins SparStencil’s overall performance lead.
Performance Breakdown
Figure 7 illustrates the incremental performance gains from each optimization stage using Box-2D49P as a representative case. Applying Layout Morphing on dense TCUs yields a 1.58x speedup over the CUDA baseline, attributed to the superior matrix multiplication capabilities of dense TCUs. Subsequently, employing PIT to harness sparse TCUs results in an additional 1.22x speedup, as sparse TCUs deliver double the computational power of dense TCUs. Notably, for problem sizes 256 and 768, PIT resulted in 0.79x and 0.90x performance, respectively, as the memory overhead of PIT exceeds the compute efficiency gains from sparse TCUs at limited scales. Finally, the application of further performance-boosting optimizations achieves an additional 1.24x improvement. With this, we have completed all optimizations in SparStencil.
Figure 7. Performance breakdown of SparStencil
Adaptivity and Sparsity Across Varying Configurations
To better evaluate adaptivity, we omit the 3x temporal fusion used in Section 4.3, as it masks performance variation for small kernels. Here, we assess adaptivity and runtime sparsity across two dimensions: varying stencil sizes and diverse stencil patterns.
For varying stencil sizes, as illustrated in Figure 9, SparStencil consistently outperforms dense baselines across all stencil sizes and TCU fragments, achieving up to 4.1× speedup and maintaining sparsity below 60%. Heatmaps show that optimal compute density emerges in specific regions, which are effectively discovered by our automated search, demonstrating the system’s ability to adaptively align diverse stencil patterns with TCU-efficient layouts.
Figure 9. Performance, sparsity, and compute density across stencil sizes and layout configurations. Top: Throughput (left) and sparsity ratio (right) across stencil sizes on different TCU fragments. Bottom: Performance and compute density over for two representative 2D stencil kernels.
For diverse stencil patterns, as illustrated in Figure 10, across 79 stencil kernels spanning 9 application domains, SparStencil consistently achieves higher throughput, reaching up to 156.7 GStencil/s and outperforming cuDNN and ConvStencil by 6.3x and 3.1x on average, respectively. SparStencil maintains a stable performance, demonstrating robustness to stencil structural diversity. In terms of compute density, SparStencil achieves 17.92x and 4.46x over cuDNN and ConvStencil, respectively. Thanks to its adaptive layout search, SparStencil effectively reduces memory footprint and maintains stable performance even under extreme sparsity.
Figure 10. Performance and compute density breakdown across 79 real-world stencil kernels spanning 9 application domains. Top: end-to-end throughput (GStencil/s) of SparStencil, cuDNN, and ConvStencil; Bottom: corresponding compute intensity.
GPU Resource Utilization Comparison
Figure 11 shows hardware utilization across six metrics. SparStencil achieves higher SM utilization (74.5%) and occupancy (96.9%) than ConvStencil (18.3%, 61.3%) and cuDNN (59.4%, 88.5%), enabled by sparsity-aware layout. L1/TEX cache and memory throughput reach 64.5% and 64.1%, indicating effective on-chip reuse. In contrast, DRAM throughput (17.5%) and L2 cache activity (52.6%) are lower than cuDNN (43.5%, 61.6%). This reduction reflects SparStencil’s layout-aware access patterns, which promote reuse in L1/shared memory, reducing dependence on L2 and minimizing global memory pressure.
Figure 11. Hardware utilization comparison.
Evaluating SparStencil Beyond FP16: Performance at FP64
Given that current sparse TCUs lack direct support for FP64 computation, our primary evaluation mainly focuses on FP16 precision. To further evaluate its broader applicability, we extend our analysis to FP64-enabled dense TCUs, benchmarking against state-of-the-art approaches as shown in Table 3.
| Method | Heat-2D | Box-2D9P | Star-2D13P | Box-2D49P |
| AMOS | 10.16 | 10.23 | 10.51 | 10.59 |
| cuDNN | 64.33 | 64.57 | 17.05 | 17.15 |
| DRStencil | 55.46 | 57.63 | 50.16 | 20.28 |
| ConvStencil | 65.83 | 62.76 | 64.37 | 63.93 |
| SparStencil | 72.49 | 73.25 | 71.34 | 67.28 |
Table 3. Performance Comparison of SparStencil and SOTA methods under FP64 precision across different stencils (GFlops/s)
Despite the absence of hardware-level sparsity acceleration, SparStencil consistently outperforms existing methods across all stencil configurations, achieving speedups ranging from 1.11x to 7.13x. This advantage stems from adaptive layout morphing and search strategy, which automatically restructures stencil matrices to optimize computational and memory throughput, ensuring high efficiency even without specialized sparse TCU acceleration.
These results validate two critical insights: (1) SparStencil remains effective and achieves performance leadership even on dense TCUs, demonstrating its robustness beyond FP16-restricted architectures. (2) Future sparse TCUs with FP64 support will further amplify SparStencil’s benefits, as our sparse-aware optimization framework is inherently aligned with next-generation hardware trends.
Related Work
The optimization of stencil computations has been extensively studied on both CPUs [23, 46] and GPUs [27, 34, 37].
On CPUs, vectorization [17, 18, 23, 42], data reuse strategies [36, 40, 50], and tiling techniques, such as diamond [4, 7], time skewing [22, 43], rectangular [39], and tessellating tiling [46], are key methods for improving performance.
On GPUs, spatial tiling [13] [26, 50], temporal tiling [6, 14] [19, 29, 35, 41], unrolling [15], prefetching [38], and streaming [37] techniques effectively leverage GPU parallelism and memory hierarchies. Frameworks like Brick [52] [50, 51] exploit fine-grained data reuse, while DRStencil [44] applies fusion-partition optimization for stencil acceleration. Recent research also explores Tensor Cores, with TCStencil [24] applying them directly, ConvStencil [9] bridging convolution and stencil computations, FlashFFT [16] restructures FFT-based stencil computations on TCU and LoraStencil [49] reduces redundancy through low-rank decomposition of stencil. Furthermore, libraries like cuDNN [10] [30] optimize stencil-related operations, and AMOS [53] maps depth-wise convolutions, equivalent to stencil operations, to hardware units, including Tensor Cores.
Conclusion
This paper presents SparStencil, a novel system leveraging sparse Tensor Cores for stencil computations via Permutation Invariant Transformation. It integrates Adaptive Layout Morphing, Structured Sparsity Conversion, and Automatic Kernel Generation to optimize stencil computations on sparse TCUs. Evaluations demonstrate its effectiveness, outperforming state-of-the-art approaches, with potential benefits for scientific and engineering applications.
References
- [1]Yulong Ao, Chao Yang, Xinliang Wang, Wei Xue, Haohuan Fu, Fangfang Liu, Lin Gan, Ping Xu, and Wenjing Ma. 2017. 26 PFLOPS Stencil Computations for Atmospheric Modeling on Sunway TaihuLight. In 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS). 535–544. doi:10.1109/IPDPS.2017.9
- [2]Krste Asanovic, Ras Bodik, Bryan Christopher Catanzaro, Joseph James Gebis, Parry Husbands, Kurt Keutzer, David A Patterson, William Lester Plishker, John Shalf, Samuel Webb Williams, et al. 2006. The landscape of parallel computing research: A view from berkeley. (2006).
- [3]Krste Asanovic, Rastislav Bodik, James Demmel, Tony Keaveny, Kurt Keutzer, John Kubiatowicz, Nelson Morgan, David Patterson, Koushik Sen, John Wawrzynek, David Wessel, and Katherine Yelick. 2009. A View of the Parallel Computing Landscape. Commun. ACM 52, 10 (oct 2009), 56–67. doi:10.1145/1562764.1562783
- [4]Vinayaka Bandishti, Irshad Pananilath, and Uday Bondhugula. 2012. Tiling Stencil Computations to Maximize Parallelism. In Proceedings of the 2012 International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’12). IEEE Computer Society, USA, 1–11. doi:10.1109/SC.2012.107DOI
- [5]Tal Ben-Nun, Linus Groner, Florian Deconinck, Tobias Wicky, Eddie Davis, Johann Dahm, Oliver D. Elbert, Rhea George, Jeremy McGibbon, Lukas Trümper, Elynn Wu, Oliver Fuhrer, Thomas Schultess, and Jersten Hoefler. 2022. Productive Performance Engineering for Weather and Climate Modeling with Python. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis. 1–14. doi:10.1109/SC41404.2022.00078DOI
- [6]Uday Bondhugula, Vinayaka Bandishti, and Irshad Pananilath. 2017. Diamond Tiling: Tiling Techniques to Maximize Parallelism for Stencil Computations. IEEE Transactions on Parallel and Distributed Systems 28, 5 (May 2017), 1285–1298. doi:10.1109/TPDS.2016.2615094DOI
- [7]Uday Bondhugula, Albert Hartono, J. Ramanujam, and P. Sadayappan. 2008. A Practical Automatic Polyhedral Parallelizer and Locality Optimizer. In Proceedings of the 29th ACM SIGPLAN Conference on Programming Language Design and Implementation (Tucson, AZ, USA) (PLDI ’08). Association for Computing Machinery, New York, NY, USA, 101–113. doi:10.1145/137558.1375595
- [8]Peng Chen, Mohamed Wahib, Shinichiro Takizawa, Ryousei Takano, and Satoshi Matsuoka. 2019. A Versatile Software Systolic Execution Model for GPU Memory-Bound Kernels. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (Denver, Colorado) (SC ’19). Association for Computing Machinery, New York, NY, USA, Article 53, 81 pages. doi:10.1145/3295500.3356162DOI
- [9]Yuetao Chen, Kun Li, Yuhao Wang, Donglin Bai, Lei Wang, Lingxiao Ma, Liang Yuan, Yunquan Zhang, Ting Cao, and Mao Yang. 2024. ConvStencil: Transform Stencil Computation to Matrix Multiplication on Tensor Cores. In Proceedings of the 29th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming (Edinburgh, United Kingdom) (PPoPP ’24). Association for Computing Machinery, New York, NY, USA, 333–347. doi:10.1145/3627535.3638476
- [10]Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. 2014. cudnn: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759 (2014).
- [11]Alain Denzel, Rahul Bera, Nastaran Hajinazar, Gagandeep Singh, Geraldo F. Oliveira, Juan Gómez-Luna, and Onur Mutlu. 2023. Casper: Accelerating Stencil Computation using Near-cache Processing. arXiv:2112.14216 [cs.AR]. https://arxiv.org/abs/2112.14216
- [12]Jack Edmonds. 1965. Paths, Trees, and Flowers. Canadian Journal of Mathematics 17 (1965), 449–467. doi:10.4153/CJM-1965-045-4DOI
- [13]Thomas L. Falch and Anne C. Elster. 2014. Register Caching for Stencil Computations on GPUs. In 2014 16th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing. 479–486. doi:10.1109/SYNASC.2014.70DOI
- [14]Tobias Grosser, Albert Cohen, Paul H. J. Kelly, J. Ramanujam, P. Sadayappan, and Sven Verdoolaege. 2013. Split Tiling for GPUs: Automatic Parallelization Using Trapezoidal Tiles. In Proceedings of the 6th Workshop on General Purpose Processor Using Graphics Processing Units (Houston, Texas, USA) (GPGPU-6). Association for Computing Machinery, New York, NY, USA, 24–31. doi:10.1145/2458523.2458526DOI
- [15]Tobias Gysi, Christoph Müller, Oleksandr Zinenko, Stephan Herhut, Eddie Davis, Tobias Wicky, Oliver Fuhrer, Torsten Hoefler, and Tobias Grosser. 2021. Domain-Specific Multi-Level IR Rewriting for GPU: The Open Earth Compiler for GPU-Accelerated Climate Simulation. ACM Trans. Archit. Code Optim. 18, 4, Article 51 (sep 2021), 23 pages. doi:10.1145/3469030DOI
- [16]Haozhi Han, Kun Li, Wei Cui, Donglin Bai, Yiwei Zhang, Liang Yuan, Yifeng Chen, Yunquan Zhang, Ting Cao, and Mao Yang. 2025. FlashFFTStencil: Bridging Fast Fourier Transforms to Memory-Efficient Stencil Computations on Tensor Cores. In Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming (Las Vegas, NV, USA) (PPoPP ’25). Association for Computing Machinery, New York, NY, USA, 355–368.
- [52]Tuowen Zhao, Samuel Williams, Mary Hall, and Hans Johansen. 2018. Delivering Performance-Portable Stencil Computations on CPUs and GPUs Using Bricks. In 2018 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC). 59–70. doi:10.1109/P3HPC.2018.00009DOI
- [53]Size Zheng, Renze Chen, Anjiang Wei, Yicheng Jin, Qin Han, Liqiang Lu, Bingyang Wu, Xiuhong Li, Shengwen Yan, and Yun Liang. 2022. AMOS: enabling automatic mapping for tensor computations on spatial accelerators with hardware abstraction. In Proceedings of the 49th Annual International Symposium on Computer Architecture. 874–887.






