betaFeedback

WorksComputer Science

Efficient Reconfigurable Architectures for Deep Learning

Abstract

The growing computational demands of deep learning (DL) have driven AI-optimized reconfigurable devices, such as the AMD Versal AI Engines (AIEs), the Intel Stratix 10 NX Tensor Blocks (TBs), and the AMD Ryzen AI NPUs. Their heterogeneous resources and novel dataflow models make efficient exploitation nontrivial, and FPGAs lack native support for structured sparsity. This dissertation optimizes dense general matrix multiplication (GEMM), the dominant DL kernel, on these devices, proposes FPGA architectural enhancements for structured sparsity, and extends the techniques to AMD Ryzen AI NPUs. First, MaxEVA optimizes GEMM on Versal AIEs, achieving up to 2.19×\times higher throughput and 20.4% higher energy efficiency than the state of the art. It is then extended to arbitrary matrix sizes, and a systematic framework is proposed for the Stratix 10 NX TBs. Both frameworks reach state-of-the-art int8 throughput of up to 77 and 68 TOPS, and energy efficiency of up to 0.94 and 1.35 TOPS/W, on Versal and Stratix, respectively. Next, Systolic Sparse Tensor slices (SSTs) are proposed as in-fabric hard blocks supporting dense, 2:4, 1:4, and a new 1:3 sparsity level. SST-based GEMM accelerators achieve up to 5×\times higher frequency and 10.9×\times lower area than conventional FPGA logic, and on sparse ViT and CNN models SSTs deliver up to 3.52×\times speedup over dense in-fabric acceleration with up to 13.3% area increase. Finally, a common systematic methodology is proposed to optimize GEMM across the XDNA and XDNA2 AMD NPUs, reaching state-of-the-art performance of up to 6.76 and 38.05 TOPS (int8) and 3.14 and 14.71 TOPS (bf16), respectively.

Copyright

by

Endri Taka

2026

The Dissertation Committee for Endri Taka certifies that this is the approved version of the following dissertation:

Efficient Reconfigurable Architectures for Deep Learning

Committee:

Diana Marculescu, Supervisor

Lizy Kurian John

Andreas Gerstlauer

Michael Schulte

Aman Arora

Dissertation

Presented to the Faculty of the Graduate School of The University of Texas at Austin in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy

The University of Texas at Austin for structured sparsity and extends these optimization techniques to next-generation AMD Ryzen AI NPU architectures.

First, this dissertation presents MaxEVA, a framework that optimizes GEMM performance on the Versal AIE devices by efficiently exploiting the AIE’s architectural features and resolving performance bottlenecks at multiple levels. MaxEVA achieves up to 2.19×2.19 \times higher throughput and up to 20.4% higher energy efficiency compared to the state-of-the-art approach, providing insights that help close the knowledge gap in optimizing GEMM workloads on these devices.

The AMD Versal and Intel Stratix 10 NX take contrasting approaches to AI acceleration. The Versal pairs out-of-fabric programmable vector processors (the AIEs) with traditional reconfigurable logic, whereas the Stratix 10 NX incorporates in-fabric hard blocks (the TBs), each comprising multiple dot-product engines. To this end, the MaxEVA framework is extended to support arbitrary matrix sizes by incorporating an additional level of memory hierarchy utilizing the Versal’s FPGA memory resources, and proposes a systematic framework for optimizing GEMM workloads by efficiently exploiting the Stratix 10 NX’s TBs. Both frameworks maximize performance and energy efficiency by leveraging each device’s architectural characteristics, achieving state-of-the-art throughput for 8-bit integer (int8) precision of up to 77 and 68 TOPS and energy efficiency of up to 0.94 and 1.35 TOPS/W on the Versal and Stratix, respectively. This study provides guidelines for optimizing GEMM-based applications on both platforms while highlighting their respective programmability trade-offs and challenges.

To address the sparsity gap, Systolic Sparse Tensor slices (SSTs) are proposed as in-fabric hard blocks that bring structured sparsity support directly into FPGA architectures. SSTs efficiently accelerate a wide variety of DL workloads across multiple degrees of sparsity, supporting dense operation, 2:4 (50%) and 1:4 (75%) sparsity, and a new 1:3 (66.7%) level for additional flexibility. On GEMM accelerators, SST-based designs achieve up to 5×5 \times higher frequency and up to 10.9×10.9 \times lower area than conventional FPGA logic. On state-of-the-art sparse vision transformer (ViT) and convolutional neural network (CNN) models, SSTs deliver up to 3.52×3.52 \times speedup over dense in-fabric acceleration with a moderate area increase of up to 13.3%.

Finally, AMD Ryzen AI processors integrate a reconfigurable NPU that represents an evolution of the Versal AIE architecture. A common methodology is proposed to optimize GEMM workloads across both NPU generations, XDNA and XDNA2, exploiting their architectural features and addressing performance bottlenecks at the system level. End-to-end evaluation across various GEMM sizes demonstrates state-of-the-art throughput of up to 6.76 TOPS (XDNA) and 38.05 TOPS (XDNA2) for int8 precision, and up to 3.14 TOPS (XDNA) and 14.71 TOPS (XDNA2) for brain floating point (bf16) precision. Collectively, this dissertation advances GEMM acceleration across reconfigurable and AI-specialized hardware, spanning dense and sparse acceleration on FPGAs and dense acceleration on NPUs.

Chapter 1: Introduction

The rapid growth of deep learning (DL) workloads Jouppi et al. [1]; Sheng et al. [2] has driven the development of AI-optimized hardware, including GPUs NVIDIA [3, 4]; AMD [5, 6] and ASICs Jouppi et al. [1, 7]; Firoozshahian et al. [8]; Int (2019); Ama (2022). Reconfigurable platforms have followed the same trend, with AI-optimized architectures proposed by both industry Langhammer et al. [10]; Gaide et al. [11]; Nurvitadhi et al. [12]; Nurvitadhi et al. [13]; Cairncross et al. (2023) and academia Arora et al. (2021a, 2022a, 2023); Boutros et al. (2019); Eldafrawy et al. (2020); Boutros et al. (2018).

Within this landscape, reconfigurable architectures are particularly attractive for accelerating DL workloads. Unlike GPUs and ASICs, whose datapaths, supported numerical precisions, and dataflows are fixed at design time, and whose long design cycles hinder adaptation to rapidly evolving DL models, FPGAs allow these aspects to be tailored to the workload, while offering low-latency streaming and high energy efficiency Boutros et al. (2024); Nurvitadhi et al. (2017); Fowers et al. (2018). The recent integration of AI-specific hardened blocks, such as the AMD Versal AI Engines and Altera/Intel Stratix 10 NX tensor blocks, combines this flexibility with substantially higher compute density. Within these platforms, general matrix multiplication (GEMM) is the natural optimization target, as it is the computational core of DL workloads: fully connected layers, attention projections in transformers, and convolutions lowered to matrix form all reduce to GEMM, which accounts for up to 90% of execution time Adolf et al. (2016a); Wang et al. (2021a).

Improving GEMM efficiency therefore translates directly into end-to-end gains in performance and energy. However, realizing the theoretical peak throughput of these platforms is challenging, as it requires careful management of data movement, on-chip memory, as well as placement and routing. Moreover, they lack native support for features such as structured sparsity, which GPUs already exploit. This dissertation addresses these challenges by analyzing the Versal and Stratix architectures in depth, developing frameworks that bridge the gap between peak and achieved GEMM performance, and proposing hardware and mapping extensions that overcome their current limitations.

This chapter introduces the three main directions explored in this dissertation. First, we present the leading AI-optimized FPGA architectures targeted in this work — namely the AMD Versal AI Engine and the Altera/Intel Stratix 10 NX — and the frameworks developed to optimize dense GEMM performance on each. Second, we discuss the lack of native sparsity support in these FPGA architectures, introducing the systolic sparse tensor slices (SSTs), in-fabric hardware blocks designed to accelerate structured sparsity in DL workloads. Finally, we extend our GEMM framework to include reconfigurable NPUs, presenting a methodology for optimizing GEMM workloads across AMD’s Ryzen AI NPU generations.

AI-Optimized FPGA Architectures

The two major FPGA vendors have taken different approaches to optimizing their devices for DL. AMD introduced the Versal Adaptive Compute Acceleration Platform (ACAP) Ahmad et al. (2019); Alok (2020), which combines the AI Engine (AIE) with programmable logic and scalar processors. In contrast, Altera/Intel released the Stratix 10 NX Langhammer et al. [10], which retains the conventional FPGA architecture but replaces DSP blocks with new AI Tensor Blocks (TBs). Consequently, the Versal AIE is an out-of-fabric solution consisting of programmable vector processors operating at high frequency, whereas TBs are in-fabric blocks comprising multiple dot-product engines that run at lower FPGA fabric frequency.

Given these differences, each platform requires its own design approach. We therefore present systematic methodologies and optimization techniques for mapping GEMM onto each device, and propose frameworks that maximize GEMM throughput. and energy efficiency on Versal and Stratix while achieving high resource utilization. As discussed earlier, GEMM occupies up to 90% of DL execution time, which makes it a representative workload for evaluating these architectures.

Sparsity Support in Modern DL Architectures

Contemporary DL workloads exhibit varying degrees of sparsity, i.e., the presence of zero values in weight or activation tensors Hoefler et al. (2021). Sparsity arises naturally in activations (as a by-product of non-linear activation functions such as ReLU) and is induced in weights through pruning, which removes redundant parameters from over-parameterized models. Since zero-valued operands do not need to be stored or computed, exploiting sparsity reduces computational and memory requirements, and has become a common strategy in efficient DL accelerators Liu et al. (2022a).

Typically, zeros in modern deep neural networks (DNNs) are distributed randomly within the data tensors Hoefler et al. (2021); Liu et al. (2022a). This random sparsity is commonly referred to as unstructured sparsity. Leveraging unstructured sparsity has been the main focus of numerous sparse hardware accelerators over the past years, targeting both ASICs Han et al. (2016); Zhang et al. (2016); Pal et al. (2018); Shomron et al. (2019); Hegde et al. (2019); Gondimalla et al. (2019); Chen et al. (2019); Zhang et al. (2020) as well as FPGAs Lu and Liang (2018); Jain et al. (2021); Xie et al. (2021); Liang et al. (2022). However, unstructured sparse accelerators require complex hardware structures, which result in high area overheads Wu et al. (2023). The random location of zeros in unstructured sparsity causes unpredictable and low hardware utilization, rendering inference speedup inefficient Liu et al. (2020a,b); Han et al. (2016).

To address these challenges in sparsity acceleration, structured sparsity has recently been proposed Mishra et al. (2021); Liu et al. (2022a); Jeong et al. (2023). Structured sparsity imposes constraints on the sparsity patterns in data tensors, enabling low area overheads and highly energy-efficient hardware enhancements for sparsity exploitation. To this end, Nvidia A100 GPUs introduced support for the 2:4 structured sparsity, where two out of every four consecutive values must be non-zero. Furthermore, structured sparse ASIC accelerators have also been proposed, demonstrating up to 3.1×3.1 \times less energy consumption per inference compared to state-of-the-art unstructured sparse accelerators Liu et al. (2022a).

Although industrial FPGA architectures Langhammer et al. [10]; Gribok and Pasca (2024) and academic proposals for in-fabric tensor blocks Arora et al. (2021a, 2022a) have improved support for the computational demands of DL, they primarily target dense computation. This results in a significant limitation of their efficiency and applicability to most real-world DL models, which inherently exhibit sparsity. To address this challenge, this dissertation proposes incorporating flexible in-fabric slices into the FPGA architecture to efficiently support both sparse and dense DL workloads. We employ 2D systolic slices similar to Arora et al. (2021b, 2022b), which are augmented with structured sparsity features and FPGA-specific architecture optimizations.

AMD Ryzen AI Reconfigurable NPU Architecture

AMD released the Ryzen AI processors Rico et al. (2024), featuring multi-core CPUs, an integrated GPU, and a new neural processing unit (NPU). The NPU features the XDNA architecture and serves as a dedicated DL accelerator integrated with x86 processors. The NPU architecture leverages the characteristics of modern DL workloads, where compilers can determine most of the control flow statically. In particular, AMD NPUs provide an explicit data movement architecture, which both reduces hardware complexity and enables high performance, while offering substantially higher energy-efficiency compared to dynamically scheduled architectures, such as CPUs and GPUs Rico et al. (2024). This lower power and energy footprint has driven the integration of NPUs into edge platforms, enabling DL applications to run efficiently under tight power and thermal budgets. The Ryzen AI NPU is an evolution of the Versal AIE architecture Rico et al. (2024), retaining its spatial array of programmable vector processors with local memory while introducing architectural changes that significantly affect GEMM mapping, such as data reuse, tiling, and data movement. In contrast to the NPU, Versal devices integrate an FPGA fabric, which acts as an additional level of memory hierarchy and enables efficient exploitation of data reuse in GEMM. Furthermore, the flexibility of the fabric allows customized tiling schemes Taka et al. (2024); Zhuang et al. (2023a) and tailored data layout transformations Wang et al. (2025a), which the NPU must instead handle within its fixed memory hierarchy and DMA capabilities.

In this dissertation, we conduct a comprehensive study to optimize GEMM workloads on AMD Ryzen AI NPUs. The architectural differences discussed above introduce new design challenges, highlighting the necessity for a separate methodology rather than a direct port of Versal approaches. Having identified this important research gap, we propose a unified systematic framework to maximize GEMM performance across the two current generations, namely XDNA and XDNA2. The framework targets end-to-end GEMM performance and introduces a novel procedure for system-level optimization.

Research Objectives

First, we focus on the Versal AIE, a new class of reconfigurable architecture. To address the design challenges this architecture introduces, we propose the MaxEVA framework Taka et al. (2023), a systematic methodology that maximizes the performance and energy efficiency of GEMM workloads on the AIE array. MaxEVA exploits key architectural features of the AIE, namely local memory sharing, static circuit switching, and broadcasting, while overcoming challenges that lead to sub-optimal performance: limited I/O and switch bandwidth, reduced number of AIE-FPGA interface tiles, and routing congestion.

Second, we build on MaxEVA Taka et al. (2023) and extend it with an additional tional memory hierarchy level that uses the Versal FPGA’s on-chip resources. Since Versal and Stratix 10 NX take fundamentally different approaches to AI acceleration (out-of-fabric vector processors versus in-fabric tensor blocks), comparing them fairly requires a GEMM design that is equally well optimized for each. We therefore develop a framework to design, map, and optimize a configurable GEMM accelerator on Stratix 10 NX that exploits its in-fabric TBs, enabling a direct comparison of the two architectures Taka et al. (2024). Both frameworks Taka et al. (2024) involve extensive design space exploration (DSE) and analytical modeling to maximize GEMM performance, data reuse, and memory utilization, and thus the overall energy efficiency.

Third, we address the lack of sparsity support in existing in-fabric blocks by proposing the SST slices Taka et al. (2025a), which accelerate a wide range of DL workloads. To the best of our knowledge, SSTs are the first in-fabric FPGA blocks to natively exploit structured sparsity. Specifically, they support 2:4 (50% sparsity), 1:3 (66.7%), and 1:4 (75%) patterns, in addition to dense operation. These levels align with the most common sparsity levels in DNNs, since sparsity higher than 75% typically leads to significant accuracy degradation Zhu et al. (2019); Liu et al. (2020a); Fang et al. (2022); Zhou et al. (2021); Zhang et al. (2022a).

The 1:3 sparsity bridges the gap between 50% and 75% sparsity, increasing flexibility and providing both acceleration and storage benefits, while can be supported in SSTs without incurring additional area overheads (by reusing the area allocated for 2:4 and 1:4). This sparsity flexibility enables the development of tailored solutions for DL models, since each model exhibits distinct levels of sparsity. Moreover, we introduce dedicated interconnects between the SST slices to facilitate their efficient integration within the FPGA architecture. These dedicated interconnects demonstrate substantial routing savings in FPGAs compared to prior work on 2D systolic blocks Arora et al. (2021b, 2022b), where such interconnects are not employed.

Finally, we propose a new systematic methodology to optimize GEMM performance on Ryzen AI NPUs Taka et al. (2026, 2025b). A key requirement of our methodology is that matrices remain in standard layout format (row- and column-major) in main memory (DRAM), rather than requiring custom or pre-transformed data layouts. This enables seamless integration with DL tensor libraries, such as GGML GGML (2025), and supports the development of high-performance GEMM libraries, similar to those available for GPUs Nvidia (2025a,b); AMD [6]. Furthermore, our GEMM designs efficiently exploit several architectural features of the NPU, e.g., on-the-fly tensor transformations, to address critical performance bottlenecks, particularly through maximization of DRAM bandwidth and sophisticated data movement design between the NPU and DRAM.

Dissertation Statement

Currently, the dominant hardware platforms for DL workloads are GPUs [4]; [5] and specialized ASIC accelerators [7]; [8]; Ama (2022). However, the rapid evolution of DL workloads creates a strong demand for hardware platforms that combine high efficiency with architectural flexibility. Reconfigurable architectures offer a promising alternative due to their ability to adapt to emerging workloads while maintaining competitive performance and efficiency. Although AI-optimized reconfigurable platforms have been proposed both in industry [10]; [11]; [13] and academia Arora et al. (2021a, 2023); Boutros et al. (2018), their large-scale adoption for DL acceleration has yet to be fully realized.

This work presents a comprehensive evaluation of prominent AI-optimized reconfigurable architectures, i.e., AMD Versal ACAP, Altera/Intel Stratix 10 NX and AMD Ryzen AI NPUs, using GEMM as the primary workload, and provides insights into their performance and efficiency characteristics. Building on these insights, we propose architectural enhancements to reconfigurable platforms that enable efficient acceleration of a wide range of dense and sparse DL workloads, including in-fabric support for structured sparsity, a capability absent from existing FPGA architectures. Ultimately, this dissertation aims to facilitate the broader adoption of reconfigurable architectures for DL acceleration through a detailed analysis of their efficiency, programmability trade-offs, and associated architectural challenges.

Contributions

This dissertation makes the following key contributions:

  • MaxEVA Framework Taka et al. (2023): We propose an optimization methodology based on analytical modeling to maximize the performance of GEMM on the Versal AIE. Our methodology is generalizable to any Versal AIE device and addresses various performance bottlenecks, leading to maximized utilization of the AIE resources. Furthermore, we propose a sophisticated AIE kernel placement strategy to effectively leverage the most efficient data movement mechanisms of the Versal AIE architecture. Demonstration on the AMD VC1902 device shows up to 5.44 and 77.01 TOPS throughput for single-precision floating point (fp32) and 8-bit integer (int8) precisions, respectively, outperforming the state-of-the-art approach by showcasing up to 2.19×2.19 \times higher performance and 20.4% higher energy efficiency.

  • Efficient GEMM Frameworks for Leading AI-Optimized FPGAs Taka et al. (2024): We propose novel systematic frameworks to optimize GEMM performance on leading commercial AI-optimized FPGAs. For Versal, we leverage the MaxEVA framework and extend it to incorporate an additional memory hierarchy level utilizing the Versal FPGA’s on-chip resources. We maximize performance via DSE and analytical modeling, and we propose a novel RAM optimization scheme to overcome severe limitations of Vitis high-level synthesis (HLS). For Stratix, we develop a novel framework to design, map and optimize a configurable GEMM accelerator by exploiting the device’s in-fabric TBs. Our framework involves extensive DSE and analytical modeling to maximize GEMM

performance. Demonstration of our frameworks on various GEMM workloads for int8 precision, exhibits up to 77 and 68 TOPS throughput with 100% AIE and 91% TB utilization for Versal and Stratix, respectively. Moreover, our designs attain up to 0.94 and 1.35 TOPS/W energy efficiency, with 88% and 94% on-chip memory for Versal and Stratix, respectively.

  • Systolic Sparse Tensor Slices Taka et al. (2025a): We propose incorporating structured sparsity support into FPGA architectures to address the challenge of efficiently accelerating sparse DL workloads. We architect 2D systolic in-fabric blocks, named SST slices, that support multiple degrees of sparsity to efficiently accelerate a wide variety of DL workloads. Our sparse SST-based designs attain up to 5×5\times higher FPGA frequency and 10.9×10.9\times lower area, compared to traditional FPGAs. Moreover, evaluation of the proposed SSTs on state-of-the-art sparse vision transformers (ViTs) and convolutional neural networks (CNNs) exhibits up to 3.52×3.52\times speedup with minimal area increase of up to 13.3%, compared to dense in-fabric acceleration.

  • GEMM Optimization on AMD Ryzen AI NPUs Taka et al. (2026, 2025b): We propose a systematic methodology to optimize GEMM workloads on Ryzen AI NPUs, combining analytical modeling with hardware profiling. Our GEMM implementation generates sufficiently long contiguous DRAM accesses to maximize main-memory bandwidth utilization, which in turn maximizes GEMM performance across the NPU array. We leverage novel tensor transformations on the NPU’s DMA engines to optimize DRAM bandwidth while keeping matrices in standard row- or column-major layout in DRAM, without explicit pre-tiling. Evaluated on two mini PCs with NPUs from different generations, our approach achieves state-of-the-art performance of up to 6.76 TOPS (XDNA) and 38.05 TOPS (XDNA2) for int8, and 3.14 TOPS (XDNA) and 14.71 TOPS (XDNA2) for brain floating point (bf16) precision.

Dissertation Organization

The remainder of this dissertation is organized as follows. Chapter 2 reviews related work in AI-optimized FPGA architectures, structured sparsity support for DL accelerators, and prior applications of the AMD Ryzen AI NPUs. Chapter 3 provides background on the three reconfigurable architectures central to this dissertation: the AMD Versal ACAP, the Intel Stratix 10 NX, and the AMD Ryzen AI NPUs. Chapter 4 presents the MaxEVA framework, which addresses the design, mapping, and optimization of GEMM on the Versal AI Engine. Chapter 5 extends MaxEVA to support arbitrary matrix sizes and introduces a parallel framework for the Altera/Intel Stratix 10 NX. Chapter 6 presents the architectural design of the SST slices and their application to sparse GEMM and DL acceleration. Chapter 7 presents a methodology for optimizing GEMM workloads across the XDNA and XDNA2 generations of AMD Ryzen AI NPUs. Finally, Chapter 8 concludes the dissertation and discusses directions for future work.

This chapter reviews prior work in three areas that correspond to the contributions of this dissertation. First, we discuss work targeting the two leading AI-optimized FPGA architectures from industry: AMD Versal and Intel Stratix 10 NX. Second, we survey prior work on structured sparsity support in modern DL accelerators. Finally, we review prior work on the AMD Ryzen AI NPU across across a range of workloads.

Leading AI-Optimized FPGAs

AMD Versal ACAP

Several prior works leverage the Versal ACAP architecture across multiple application domains. In particular, CHARM Zhuang et al. (2023b,c) proposes a framework to automate the process of GEMM acceleration on Versal ACAP. Their experimental results on the VC1902 exhibit higher energy efficiency of up to 7.2×7.2 \times and 1.7×1.7 \times compared to traditional FPGAs (AMD/Xilinx U250) and GPUs (NVIDIA A100), respectively. Furthermore, additional research on Versal focuses on accelerating specific DL workloads, such as CNNs Jia et al. (2022), Zhang et al. (2022b), and graph neural networks (GNNs) Zhang et al. (2022c); Chen et al. (2023).

Mhatre et al. (2025a) presents a thorough analysis of GEMM workloads of varying shapes and sizes on AMD Versal, aiming to provide insights for optimal performance. Moreover, GAMA Mhatre et al. (2025b) proposes a design that maximizes memory utilization on the second generation of Versal AIE (AIE-ML), attaining state-of-the-art performance of up to 165 TOPS. Other works include AIE compilers Chatarasi et al. (2020), arbitrary precision integer multiplication Yang et al. (2023), as well as acceleration of atmospheric simulations Singh et al. (2023); Brown (2023).

Considering all prior work, our MaxEVA framework Taka et al. (2023) detailed in this dissertation, represents the state-of-the-art GEMM implementation on Versal AIE, achieving notable performance gains of 2.19×2.19 \times and 20.8% for int8 and fp32, respectively, as well as up to 20.4% higher energy efficiency, over prior works. Nevertheless, MaxEVA only targets small matrix sizes that fit within the Versal AIE. We address this limitation in our subsequent work Taka et al. (2024), extending MaxEVA to support arbitrary GEMM sizes by implementing an additional level of memory hierarchy exploiting Versal’s FPGA resources.

Intel Stratix 10 NX

The Intel Stratix 10 NX architecture was introduced in Langhammer et al. [10], including a discussion of the TB operating modes and design trade-offs. This work also presents a TB design used for feneral matrix-vector (GEMV) as well as GEMM operations. Multiple other works have targeted the Stratix 10 NX for DL workloads. In Boutros et al. (2020), a GEMV accelerator is mapped on AI TBs and incorporated into an enhanced Brainwave NPU overlay Fowers et al. (2018). They show a speedup of up to 3.5×3.5 \times compared to all prior works on FPGA-based acceleration of recurrent neural networks (RNN) Guan et al. (2017); Rybalkin et al. (2018); Que et al. (2020), as well as the baseline NPU with legacy DSPs Nurvitadhi et al. [13]. Additionally, in Stan et al. (2022), the Stratix 10 NX is utilized to enhance the CNN HPIPE accelerator Hall and Betz (2020). They demonstrate a 4×4 \times speedup over prior FPGA accelerators for CNN. In Langhammer et al. [10], a method to assemble higher than int8 precision multipliers is presented. Finally, in Shipton et al. (2021), a speech-generation model is implemented on Stratix 10 NX, substantially outperforming a V100 GPU implementation.

In this dissertation, we develop a framework, which includes a detailed, systematic approach for automatically generating a configurable GEMM accelerator on Stratix 10 NX Taka et al. (2024). Furthermore, we perform an extensive DSE and we explore various trade-offs in GEMM design, which are not thoroughly examined in prior work targeting the Intel Stratix 10 NX device.

Architectural Support for Sparsity in AI Accelerators

Structured sparsity is currently supported by various commercial architectures. In particular, the 2:4 sparsity is supported by state-of-the-art GPUs, i.e., Nvidia A100 [3], Nvidia H100 [4] and AMD MI300 [5], as well as the new AMD Versal AIE-ML processors AMD (2024a,b). In academic research, hardware support for structured sparsity has also been investigated in ASIC DNN accelerators Liu et al. (2022a, 2020b); Wu et al. (2023). Moreover, prior academic research Zhu et al. (2019) introduces structured sparsity features in the tensor cores of modern GPUs. Finally, structured sparsity support has also been proposed inside the matrix engines Jeong et al. (2023) and vector units Titopoulos et al. (2024) of CPUs. Considering all prior work, we are the first to incorporate structured sparsity support into the FPGA architecture. Moreover, our proposed in-fabric blocks support multiple levels of structured sparsity (beyond merely 2:4), to enable efficient acceleration for most contemporary DNN models.

In-fabric hard blocks have been commercially incorporated into several AI-optimized FPGAs. More specifically, the Intel Stratix 10 NX [10] replaces the traditional DSP slices with tensor blocks. These tensor blocks comprise multiple dot-product engines to more efficiently support AI applications. More recently, Intel announced the Agilex-5 FPGAs Gribok and Pasca (2024); Intel (2024a), which introduce new AI-enhanced DSP blocks. These new DSPs include multiple dot-product operations, while also maintaining various features of the traditional Agilex DSP blocks. Finally, the Achronix Speedster7t FPGAs incorporate machine learning processor (MLP) blocks Cairncross et al. (2023); Achronix (2024a). These MLP hard blocks feature multiplier arrays, adder trees, accumulators and tightly coupled memories to the computational blocks.

In-fabric enhancements have also been proposed in academia. In Arora et al. (2021b, 2022b), the authors enrich the FPGA architecture with 2D systolic tensor slices, showing substantial efficiency benefits in DNN acceleration over traditional FPGAs. Moreover, in Rasoulinezhad et al. (2019), a special pattern of dedicated wires between DSP blocks is proposed, which enables more efficient mapping of systolic arrays on FPGAs. However, all aforementioned commercial and academic in-fabric blocks target primarily dense computation. In this dissertation, we propose incorporating the SST slices Taka et al. (2025a), which are architected to enable efficient computation for both sparse and dense AI workloads. The SST slices employ a 2D systolic dataflow similar to Arora et al. (2021b, 2022b), but are augmented with sparsity features. Moreover, we introduce vertical dedicated wires between the SSTs to more efficiently map GEMM accelerators into FPGA architectures.

Finally, prior research Cao et al. (2019); Liu and Brown (2021); Fang et al. (2022); Cai et al. (2023); Yang et al. (2024) implement DNN accelerators that support multi-level structured sparsity on traditional (non AI-optimized) FPGAs. In contrast, we also introduce the novel 1:3 sparsity pattern Taka et al. (2025a), which is absent from any prior work and offers increased sparsity flexibility. In this dissertation, we demonstrate the importance of this flexibility for various state-of-the-art DNNs. Furthermore, our SST-based GEMM designs show remarkable performance and area efficiency gains compared to traditional FPGAs.

Application Mapping on Ryzen AI NPUs

Several workload mapping on Ryzen AI NPUs are covered in prior publications. The authors in Zhuang et al. (2025) map a ResNet layer on XDNA, utilizing only 20% of the XDNA cores. In Deshmukh et al. (2025), a compiler framework for dynamic attention folding on AMD NPUs is presented. Moreover, in Karami et al. (2025) the authors perform characterization of generative AI workloads on Ryzen AI processors. Other works leverage the NPUs to perform stencil applications Xu et al. (2025), accelerate Fortran intrinsics Brown and Rodriguez-Canal (2025), and implement a variant of the fast Fourier transform (FFT) Nozaki et al. (2024).

Moreover, some prior works have also explored mapping GEMM on AMD NPUs. In Rösti and Franz (2025), the authors explore fine-tuning GPT-2 on the Ryzen AI processors. Specifically, they offload the time-intensive GEMM computation on the first-generation XDNA and attain 2.8×2.8 \times speedup compared to a CPU-only implementation. Furthermore, they utilize a GEMM implementation similar to the non-optimized programming example in AMD [6]. In Fang et al. (2025), a task-based programming model for dataflow accelerators is presented, demonstrating a GEMM implementation on XDNA. In particular, they attain up to 5.04 TOPS and 1.95 TOPS for int8 and bf16 data types, respectively, which closely matches the performance in AMD [6].

Furthermore, in Wang et al. (2026) the authors introduce MLIR-AIR, which is a novel open-source compiler stack, operating at a higher level compared to the lower-level MLIR-AIE framework AMD [6]. They demonstrate up to 79% compute efficiency compared to MLIR-AIE. More recently, Wang et al. (2025b) introduced asymmetric tile buffering, which decouples the buffered tile dimensions of the input and output operands. They demonstrate up to 28.5 TOPS GEMM performance on XDNA2 using block floating-point (bfp16), a datatype in which a block of values shares one common exponent. Finally, Pang et al. (2026) presents the TileFuse library, targeting mixed-precision quantization formats such as W4A16, and attaining up to 12 TOPS GEMM performance on XDNA2.

In this dissertation, we present a systematic framework Taka et al. (2026, 2025b) to optimize GEMM performance across both NPU generations (XDNA and XDNA2), while retaining matrices in standard layout in DRAM, i.e., row- and column-major order, without explicit pre-tiling. Our designs leverage the multi-dimensional tensor addressing feature across the entire NPU hierarchy to transform matrices into the tiled layout expected by NPU cores. We show that our optimized GEMM implementation on the first-generation XDNA achieves superior performance over all prior work, reaching up to 6.76 TOPS (34% higher) for int8 and 3.14 TOPS (62% higher) for bf16. For the second-generation XDNA2, we attain state-of-the-art performance of up to 38 TOPS for int8 and 14.7 TOPS for bf16.

We note that Wang et al. (2025b) assumes explicit matrix pre-tiling, which provides optimal DRAM bandwidth, and focuses on bfp16 precision. However, retaining matrices in standard layout in DRAM and exploiting the multi-dimensional DMA addressing features of the NPUs for bfp16 remains an open problem, due to significant challenges in data storage and limitations of NPU DMA transformations. A direct comparison is therefore not directly applicable, since our work in this dissertation focuses on int8 and bf16 formats, for which we implement sophisticated DMA transformations. Similarly, Pang et al. (2026) targets mixed-precision formats and achieves only up to 12 TOPS on XDNA2 (we attain up to 38 TOPS); however, since we focus on single-precision formats in this work, a direct comparison is not applicable here either.

Chapter 3: Background & Architecture Overview

This chapter provides background on the three reconfigurable architectures central to this dissertation: the AMD Versal ACAP, the Intel Stratix 10 NX, and the AMD Ryzen AI NPUs. For each architecture, we describe the main architectural features, the primary data movement and communication mechanisms, as well as their associated programming models and tools.

AMD Versal ACAP

Architecture Overview

The architecture of the Versal ACAP is depicted in Figure 3.1. The Versal ACAP comprises the processor system (PS), the programmable logic (PL), as well as the novel AIE array Ver (2022). The PS consists of scalar ARM processors, while the PL includes the traditional FPGA resources, e.g., LUTs, FFs, DSP slices, and on-chip memory resources (BRAMs/URAMs).

Versal ACAP architecture diagram

Figure 3.1. Versal ACAP architecture.

The Versal AIE is a 2D array consisting of identical AIE tiles. Each tile includes an AIE core, a memory unit and an interconnect (switch) AMD/Xilinx (2021). The AIE cores are architected as very-long instruction word (VLIW) programmable processors featuring single-instruction multiple-data (SIMD) vector units. The AIE array provides three levels of parallelism. First, instruction-level parallelism is realized by executing up to 7 instructions every clock cycle (7-way VLIW). Second, data-level parallelism is achieved through vector operations, where multiple data can be processed each clock cycle (SIMD). Third, spatial-level parallelism is attained via the concurrent execution of multiple AIE cores (up to 400). Communication between different AIE cores is achieved by local memory sharing access for neighboring cores, or by programmable switches for distant cores (Figure 3.1).

The AIE array communicates efficiently with the PL via the dedicated AIE-PL tiles, located on the last row of the AIE, as shown in Figure 3.1. These tiles provide AXI4-Streaming interface with the PL, while also supporting clock domain crossing between the AIE and the PL. The Versal ACAP additionally includes a network-on-chip (NoC) to enable flexible communication with the off-chip memory (DDR).

AI Engine Array Data Movement

Each AIE tile has 16KB of program memory to store VLIW instructions, as well as 32KB of data memory divided into 8 banks of 4KB. For higher memory requirements, AIEs can access data memory directly from their neighbors, for a total of 128KB. Figure 3.2 shows this direct access (AIEs highlighted in green), which constitutes the main data movement mechanism of the AIE array AMD/Xilinx (2021). However, notice that while each AIE is able to directly access memory from its north and south directions, the east and west access depend on the row location of the AIE. In particular, the AIE array is arranged on alternate even and odd rows, where the cores in even rows can access memory on the west direction, while in odd rows, the east module is accessed. Finally, we note that AIEs on the edges of the array have fewer memory accesses on both north/south and east/west directions following the pattern described above.

Figure 3.2. Data movement mechanisms in AI Engine array.

For non-neighboring AIEs, the communication is realized through the direct memory access (DMA) mechanism using the programmable switches, as shown in Figure 3.2 (AIEs highlighted in orange). Compared to direct access, non-neighboring communication through DMA has increased communication latency and requires more memory resources. The AXI4-Stream switches can be configured for either circuit-switching or packet-switching. Circuit-switching provides dedicated connections which are statically configured at compilation time. In contrast, packet-switching allows routing to different destinations by dynamically setting a destination header at the beginning of each packet. Due to static configuration, circuit-switching exhibits deterministic latency between connections, while also supporting broadcast to multiple output channels. Conversely, packet-switching can cause resource contention, leading to non-deterministic latency AMD/Xilinx (2021). In this work, we only exploit the most efficient circuit-switching mechanism, without the need of explicit packet-switching proposed in Zhuang et al. (2023b,d).

Programming Model & Tools

An AIE kernel running on a single AIE core can be programmed in high-level C/C++ AMD/Xilinx (2022a), or low-level SIMD intrinsics AMD/Xilinx (2022b). The mapping of multiple kernels on the AIE array is realized through the adaptive data flow (ADF) graph modeling. The nodes in the ADF correspond to AIE kernels while the edges represent connections between them AMD/Xilinx (2022c). The PL can be programmed in C/C++ using Vitis HLS Vit (2022a) or low-level RTL. Finally, AMD/Xilinx provides the Vitis V++ tool Vit (2022b) to integrate the AIE graph system and the PL kernels.

Intel Stratix 10 NX

Architecture Overview

The Intel Stratix 10 NX device’s PL comprises the traditional FPGA resources, i.e., ALMs (adaptive logic modules), M20K blocks (Intel’s BRAMs, 20Kbit in size Int (2023a)), as well as the new AI-optimized TBs. TBs replace the traditional variable-precision DSP blocks (Figure 3.3), by dropping many of their legacy, high-precision operating modes and replacing them with scalar, vector, and tensor operating modes for several DL-optimized data types. The TB maintains the interface of the legacy DSP. In this work, we focus on the Tensor int8 mode, which supports three 10-element 8×88 \times8 signed fixed-point dot-product operations and three 3232-bit fixed-point additions for accumulation on partial products. TBs in an FPGA column are physically grouped in chains, each consisting of 36 TBs. The TBs within a chain are cascaded using dedicated wires to propagate operands and accumulated products. Neighboring chains are not cascaded. Within a chain, TBs can be logically grouped in independent arrays of configurable length.

Figure 3.3. Architecture of Stratix 10 NX Tensor Blocks.

Figure 3.3 illustrates an example of an array of 18 TBs that lies within a TB chain, as well as a simplified block diagram of the TB operating in the int8 mode. Each TB contains two banks of three ping-pong registers for storing operands (bank 0, 1), where each register holds ten 8-bit values. Moreover, it includes three 10-element dot-product engines (Dot 10). The first input to each dot-product engine comes from its corresponding register, while the second input is broadcast from the 80-bit wide data_in port to all three engines. Finally, three independent 32-bit fixed-point adders are responsible for adding the generated dot-products to the cascade input from the previous TB. As shown in Figure 3.3, accumulation results are propagated along the TB array through cascade connections between a block’s casc_accum_out port and the following block’s casc_accum_incasc\_accum\_in port. The latency of the dot-product calculation plus the cascade accumulation is equal to two cycles. The array’s final TB outputs are exposed through three 24-bit wide data_outdata\_out ports.

There are three methods for loading operands in an array’s TB registers: (i) parallel load mode, (ii) side load mode and (iii) cascade mode. In this work, we focus on the cascade mode, since it leads to less routing congestion, as mentioned in Langhammer et al. [10]. We refer the reader to Langhammer et al. [10] for a detailed description of the other modes and their trade-off analysis. In cascade mode, TB0TB_0 (Figure 3.3) performs no computation and acts only as a loading port, where operands enter the array through its 80-bit data_indata\_in port. These operands are then propagated to subsequent TBs (TB1TB_1–TB17TB_{17} in Figure 3.3) from the casc_data_outcasc\_data\_out port of one TB to the next’s casc_data_incasc\_data\_in port and eventually stored in the TB registers. This requires three cycles per TB, leading to longer loading latencies as the array grows in size. However, computation can occur concurrently due to the ping-pong registers, allowing to hide the loading latency.

Programming & Tools

While the TBs support many modes of operation, there is currently no support for programming them using Intel HLS Int (2023b) or other high-level synthesis tools. Consequently, the developer is responsible for implementing a TB-based design exclusively at the register transfer level (RTL), using languages such as Verilog or VHDL, within Intel’s Quartus Prime software Qua (2026).

AMD Ryzen AI NPUs

Architecture Overview

The architecture of both XDNA and XDNA2 is illustrated in Figure 3.4. The NPU is a modular and scalable architecture, comprising a 2D array of identical compute tiles (CompTiles) Rico et al. (2024); AMD (2024a). CompTiles include the processing cores which operate out of a local memory (L1\mathrm{L1}). The NPU core is a VLIW processor with SIMD datapath, supporting both fixed-point and floating-point operations. NPUs incorporate a second level of on-chip memory (L2\mathrm{L2}) via the memory tiles (MemTiles). These are arranged in a single row, located below the array of CompTiles (Figure 3.4). Finally, NPUs include a last row of interface tiles (ShimTiles) to provide communication with the main off-chip memory (DRAM).

Architecture diagram of Ryzen AI NPUs

Figure 3.4. Architecture of Ryzen AI NPUs.

Data movement between the levels of memory hierarchy is facilitated by DMA engines, which are integrated across all NPU tiles. DMAs move data between the NPU tiles by utilizing the configurable interconnects (switches), shown as black squares in Figure 3.4. The DMA engines of ShimTiles read/write data to DRAM via the NPU NoC and SoC-level fabric of the Ryzen AI chips Rico et al. (2024); Subramon et al. (2023). Task scheduling and data movement are orchestrated by an on-chip command processor. This processor controls the data movement between ShimTiles and DRAM at runtime, allowing the NPU to focus exclusively on the main computation. The command processor is also responsible for (re-)configuring the NPU compute kernels, switches, and DMA transfers.

The XDNA has 20 compute cores, which are organized as a 4×54 \times5 array (rows ×\times columns) of CompTiles Rico et al. (2024), while the XDNA2 contains 32 cores as a 4×84 \times8 array AMD (2024c). Both NPU generations have 64 KB of memory per L1 tile and 512 KB of memory per L2 tile Rico et al. (2024); AMD (2024a); Deshmukh et al. (2025). XDNA and XDNA2 natively support int8, int16 and bf16 precisions. Additionally, XDNA2 has hardware support for bfp16 datatype AMD [6], where a block of eight numbers shares one common exponent AMD [6]; Darvish Rouhani et al. (2020). XDNA2 offers increased theoretical peak compute capabilities, delivering up to 50 TOPS AMD [6] (2024c), compared to the 10 TOPS of XDNA Rico et al. (2024).

Data Movement Architecture

Ryzen AI NPUs incorporate a dedicated data movement architecture that allows programmers to explicitly configure data transfers between all levels of the memory hierarchy. Data moves from a source DMA channel to a destination DMA channel through a circuit- or packet-switched stream, using one or more configurable switches between them. The source channel is a memory-map to stream (MM2S) channel which reads data from memory and pushes it to the stream switches. Conversely, the destination channel receives data from the stream switches and writes it to the memory-mapped space (S2MM). Each CompTile and ShimTile includes two MM2S and two S2MM DMA channels, while MemTiles incorporate six MM2S and six S2MM channels AMD (2024a).

Figure 3.5a shows an example of moving an input buffer AA from DRAM to an L2 MemTile, and eventually to L1, where it can be utilized by the corresponding core. In a similar fashion, Figure 3.5b depicts the data movement of an output buffer from L1 to L2, and eventually to DRAM. The synchronization of data buffers between the DMAs and the corresponding module, e.g., the NPU core or DRAM, is managed by hardware lock units AMD (2024a); Rico et al. (2024).

Figure 3.5. Data movement across the memory hierarchy in Ryzen AI NPUs: input buffer AA (a) and output buffer CC (b).

DMAs run independently and in parallel with computation on cores. They are programmed by configuring a sequence of buffer descriptors (BDs). A BD contains all the required information associated with a specific DMA transfer, such as the amount of data to read/write, the memory addresses involved, and the locks to acquire/release before and after each transfer AMD (2024a,b). BDs support both linear memory addressing and multi-dimensional address generation, enabling on-the-fly data layout transformations required for DL tensors. CompTiles and ShimTiles support each 3D tensor addressing, while MemTiles incorporate 4D addressing. This important DMA addressing feature is extensively exploited by our GEMM implementation across all tiles in the NPU architecture, allowing tensors to be stored in regular order in DRAM.

Programming & Tools

In this work, we use IRON, an open-source close-to-metal toolchain for developing programs on Ryzen AI NPUs AMD [6]. As a low-level toolkit, IRON enables fine-grained control of the NPU architectural attributes, such as explicit data movement and complex access patterns supported in DMAs, while also providing convenient programming abstractions Hunhoff et al. (2025). These characteristics allow explicit control of the NPU architectural features, which is critical for achieving high performance.

IRON is based on a multi-level intermediate representation (MLIR) Lattner et al. (2021) dialect, named “AIE”, and offers a user-friendly interface to this dialect through Python bindings MLIR (2025). Hence, Python scripts can be used to generate code that describes both the data movement across the NPU hierarchy and the compute kernels that run on the cores. The single-core compute kernels can be written in high-level C++ AMD [6], or low-level SIMD intrinsics AMD [6]. Finally, two options are available for single-core kernel compilation: the proprietary xchesscc compiler AMD [6] (2024b) and the open-source Peano tool AMD [6].

Chapter 4: MaxEVA: Maximizing the Efficiency of Matrix Multiplication on Versal AI Engine

In this chapter, we discuss the details of the MaxEVA framework Taka et al. (2023); max (2023). The MaxEVA framework addresses the design, mapping, and optimization of GEMM on the AIE array. MaxEVA assumes that input/output data buffers are placed in PL BRAMs, as repeatedly used in practice to efficiently exploit data reuse in large matrices Zhuang et al. (2023b); Wang et al. (2021b); Zhuang et al. (2023d); Zhang et al. (2019). This assumption restricts the matrix sizes that can be processed entirely on-chip; we revisit and resolve this limitation in Chapter 5, where we extend MaxEVA with an additional level of memory hierarchy to support arbitrary matrix sizes.

In this work, we focus on VC1902 device of the VCK190 board AMD/Xilinx (2022d), which has a total of 400 AIEs organized as an 88 rows ×\times 5050 columns array. The AIE/PL communication is established through the AIE-PL interface tiles. However, not all existing columns in the AIE array can interface with PL. For instance, in VC1902 there are only 39 AIE-PL tiles AMD/Xilinx (2023). The small AIE-PL bandwidth is one of the main challenges when designing GEMM workloads on AIEs, which MaxEVA successfully overcomes. Finally, we note that although we show our method on the VC1902, our work can be generalized in straightforward fashion to any Versal device.

MaxEVA Framework

Matrix Multiplication Tiling Scheme

Figure 4.1 depicts a simplified example of our proposed tiling scheme. In this design, the single-core tiling size (M×K×NM \times K \times N) is determined by the single GEMM kernel, which is mapped to exactly one AIE core. Since the Versal AIE comprises multiple cores, we map multiple GEMM kernels on the AIE array (described by the parameters XX, YY, ZZ as explained below). With this scheme, the final GEMM size running on the entire AIE array is (X⋅M)×(Y⋅K)×(Z⋅N)(X \cdot M) \times(Y \cdot K) \times(Z \cdot N). To this end, X,Y,Z,M,K,NX, Y, Z, M, K, N constitute the configurable integer parameters which are optimized by the MaxEVA framework.

Figure 4.1. Simplified view of tiling scheme for X=3X=3, Y=4Y=4, Z=2Z=2.

Matrix Multiplication Mapping on AI Engine Array

To overcome the reduced number of AIE-PL interface tiles, and thus avoid the PL Input/Output (PLIO) bottleneck, we leverage two principal properties of the GEMM algorithm. First, we exploit the inherent data reuse in GEMM to reduce the number of incoming PLIOs (inputs to AIE array), by broadcasting inputs AA and BB (Figure 4.1) to multiple AIEs. Second, we utilize the native reduction of the Y⋅KY \cdot K dimension in GEMM to decrease the number of outgoing PLIOs (outputs of AIE array) by performing reduction on the AIE itself, instead of the PL. With this method, we are able to efficiently map X⋅Y⋅ZX \cdot Y \cdot Z GEMM kernels (each performing a GEMM computation of M×K×NM \times K \times N size), and X⋅(Y−1)⋅ZX \cdot(Y - 1) \cdot Z Add kernels (each reducing partial results of M×NM \times N size).

Figure 4.2 shows a high-level mapping diagram of GEMM and Add kernels on the AIE array. In our design, there exist groups of YY GEMM kernels along with their corresponding adder trees (Y−1Y - 1 adders on each group). More specifically, there are X⋅ZX \cdot Z of such groups in total, all executing in a parallel fashion. Each Ax,yA_{x,y} and By,zB_{y,z} PLIO input data are broadcasted to their corresponding GEMM kernels, ZZ and XX times, respectively, as governed by the GEMM algorithm. In this design, there are in total X⋅Y+Y⋅ZX \cdot Y + Y \cdot Z PLIO inputs and X⋅ZX \cdot Z PLIO outputs. We note here that broadcasts are realized through the programmable switches of the AIE array and are statically configured during compilation (via circuit-switching).

GEMM mapping on the AIE array using input broadcasting and output adder tree reduction

Figure 4.2. GEMM mapping on AIE array by leveraging input broadcasting and output adder tree reduction.

Figure 4.3 illustrates a group comprising 4 GEMM kernels along with its adder tree (3 Add kernels). As mentioned above, each GEMM kernel executes on a separate AIE core. However, we map the whole adder tree to a single AIE core, where all Add kernels are executed in a sequential fashion. We make such design choice for various reasons. First, we note that only GEMM kernels contribute to throughput, while Add kernels are only used to reduce the output PLIOs. Thus, we maximize the number of GEMM kernels, by minimizing the AIE cores used to run the Add kernels. Second, Add kernel’s execution time (latency) is much lower than GEMM kernel’s latency. Therefore, we are able to map multiple Add kernels to a single AIE core, without any performance degradation (as we show in § 4.3.1). Third, when mapping multiple kernels to a single AIE core, memory resources are reduced compared to mapping to several AIE cores. As depicted in Figure 4.3, double buffers are inserted between separate AIE cores to effectively overlap computation with communication (and thus increasing the compute utilization of AIEs). However, between the Add kernels only single buffers are used, since all Add kernels are executed sequentially. This results in twofold memory buffer savings compared to if they were executed on separate AIEs. Overall, the throughput of the entire design is determined by the computationally heavy GEMM kernels, since the whole adder tree latency is lower than GEMM latency (§ 4.3.1).

Group of GEMM kernels and its adder tree, showing four GEMM kernels mapped to separate AIE cores and an adder tree mapped to one AIE core, with double and single buffers

Figure 4.3. Group of GEMM kernels (each mapped to one AIE core) and its adder tree (whole tree mapped to a single AIE core).

AI Engine Kernels Modeling & Optimization

Because the design space of the configurable integer parameters XX, YY, ZZ, MM, KK, NN is large, we propose an analytical model to maximize the throughput of GEMM on the AIE array. Our model takes as input device-specific parameters and constraints, e.g., I/O bandwidth, AIE peak throughput, number of PLIOs, and finds the optimal solution based on our mapping methodology. In this section, first, we describe the optimization of the single GEMM kernel, and then we discuss mapping the entire GEMM to the AIE array.

Single AI Engine Kernel Optimization: MM, KK, NN parameters.

Our model takes as input a lower bound of the efficiency of the single GEMM kernel, eff_lbeff\_lb, to ensure that the achieved throughput is very close to the theoretical AIE throughput. Here, we define efficiency (effeff) as the fraction of the achieved throughput to the peak throughput of the vector processor inside the AIE core. In particular, each AIE vector processor is able to achieve up to 128 MACs/cyc (multiply-accumulate operations per clock cycle) for int8 precision, while for fp32 the peak throughput is 8 MACs/cyc AMD/Xilinx (2021). Defining peak_MACspeak\_MACs as the AIE core peak throughput and kernel_cyckernel\_cyc as the latency (in clock cycles) of the GEMM kernel of M×K×NM \times K \times N size, we obtain the following constraint:

eff≥eff_lb⇒(M⋅K⋅Nkernel_cyc)/peak_MACs≥eff_lb⇒kernel_cyc≤M⋅K⋅N/(eff_lb⋅peak_MACs)(1)eff \ge eff\_lb \Rightarrow \left(\frac{M \cdot K \cdot N}{kernel\_cyc}\right) / peak\_MACs \ge eff\_lb \Rightarrow kernel\_cyc \le M \cdot K \cdot N / (eff\_lb \cdot peak\_MACs) \tag*{(1)}

Next, we optimize our design by ensuring that I/O bandwidth does not become a performance bottleneck. There are two I/O bandwidth considerations: the PLIO bandwidth for both inputs and outputs, as well as the bandwidth of the AXI4-Stream switches. According to AMD/Xilinx (2021), the bandwidth for both I/Os is BW_IO=4Bytes/cycBW\_IO = 4Bytes/cyc. To assure that our design is not I/O limited, we require the latency of input (a_cyca\_cyc, b_cycb\_cyc) and output (c_cycc\_cyc) data transmission to be lower than the GEMM kernel latency. Therefore, we get:

{a_cyc,b_cyc,c_cyc}≤kernel_cyc⇒{M⋅K⋅sizeof(a)/BW_IO≤kernel_cyc,K⋅N⋅sizeof(b)/BW_IO≤kernel_cyc,M⋅N⋅sizeof(c)/BW_IO≤kernel_cyc}(2)\begin{aligned} \{a\_\mathrm{cyc}, b\_\mathrm{cyc}, c\_\mathrm{cyc}\} &\le kernel\_\mathrm{cyc} \Rightarrow\\ \{M \cdot K \cdot sizeof(a)/BW\_IO &\le kernel\_\mathrm{cyc}, \\ K \cdot N \cdot sizeof(b)/BW\_IO &\le kernel\_\mathrm{cyc}, \\ M \cdot N \cdot sizeof(c)/BW\_IO &\le kernel\_\mathrm{cyc}\} \tag*{(2)} \end{aligned}

By combining equations (1) and (2), we have the following:

N≥eff_lb⋅peak_MACs⋅sizeof(a)/BW_IO(3)N \ge eff\_lb \cdot peak\_MACs \cdot sizeof(a)/BW\_IO \tag*{(3)}
M≥eff_lb⋅peak_MACs⋅sizeof(b)/BW_IO(4)M \ge eff\_lb \cdot peak\_MACs \cdot sizeof(b)/BW\_IO \tag*{(4)}
K≥eff_lb⋅peak_MACs⋅sizeof(c)/BW_IO(5)K \ge eff\_lb \cdot peak\_MACs \cdot sizeof(c)/BW\_IO \tag*{(5)}

Notice that aa, bb are the inputs of the GEMM kernel, while cc is the output of either the GEMM or the Add kernel (both have same output size of M×NM \times N). The sizeof()sizeof() function calculates the size (in Bytes) of input/output data types. This is particularly important for int8 computation, since we retain all accumulations in 32-bits (int32). In this case, the size of inputs aa, bb is 1 Byte, while the size of output cc is 4 Bytes.

Finally, we define a constraint that all input/output buffers of the single GEMM kernel should fit within the local AIE memory. By not exceeding the local AIE memory, we are able to maximize the number of GEMM kernels that execute in parallel on the AIE array. Each AIE core needs some system memory for its operation, e.g., stack. The AIE data memory is partitioned in 4KB banks; we leave one bank for system memory. This implies an available space of 28KB for our/user buffers. Because both input and output buffers of GEMM kernels are double buffered (see Figure 4.3), we get:

{M⋅K⋅sizeof(a)+K⋅N⋅sizeof(b)+M⋅N⋅sizeof(c)}≤14KB(6)\{M \cdot K \cdot sizeof(a) + K \cdot N \cdot sizeof(b) + M \cdot N \cdot sizeof(c)\} \le14KB \tag*{(6)}

The solution of MM, KK, NN can be formulated as an integer programming (IP) optimization problem, where we maximize the number of MACs of the single GEMM kernel by having eq. 4.3–4.6 as constraints. Increasing the number of MACs will lead to more data reuse in the vector registers of the AIE core, resulting in higher kernel efficiency. The lower bound of the efficiency (eff_lbeff\_lb) can be assigned based on the performance constraints of the application and the achievable throughput. In this work, we are able to achieve 95% of GEMM kernel efficiency (§ 4.3.1), which we set it as our lower bound. The configurable parameters are evaluated through exhaustive search and top-ranked design points are reported. We note that solving the IP exhaustively is a suitable approach, since the search space is significantly reduced by considering only powers of two for MM, KK, NN (as discussed in § 4.3.1).

2) Multiple AI Engine Kernels Optimization: XX, YY, ZZ parameters.

To obtain an optimal mapping onto the AIE array, we require our entire design to fit in the total number of AIE cores (AIE_coresAIE\_cores). We also require the number of utilized input/output PLIOs to not exceed the available PLIOs of the device. In particular, for VC1902, PLIO_in=78PLIO\_in = 78 and PLIO_out=117PLIO\_out = 117 AMD/Xilinx (2021, 2023). Based on the discussion at § 4.1.2 and the above, the following can be expressed:

X⋅Y⋅Z+X⋅Z≤AIE_cores(7)X \cdot Y \cdot Z + X \cdot Z \le AIE\_cores \tag*{(7)}
X⋅Y+Y⋅Z≤PLIO_in(8)X \cdot Y + Y \cdot Z \le PLIO\_in \tag*{(8)}
X⋅Z≤PLIO_out(9)X \cdot Z \le PLIO\_out \tag*{(9)}

The optimization of XX, YY, ZZ is evaluated through exhaustive search by maximizing the total number of GEMM kernels (X⋅Y⋅ZX \cdot Y \cdot Z), which leads to maximized throughput in GEMM. Again, exhaustive search is sufficient because all constants in eq. (7)–(9) are in the order of hundreds (reasonably small) AMD/Xilinx (2021). Multiple top-ranked design points are reported, from which we explore various options (refer to § 4.3.2 for details).

AI Engine Kernel Placement

To leverage the most efficient neighboring data sharing mechanism of the AIE, we propose a sophisticated kernel placement strategy. Figure 4.4 illustrates an example. of the multiplication procedure, where each multiplication symbol denotes a GEMM kernel, while the addition symbol indicates the adder tree mapped to a single AIE core. We place each group of YY GEMM kernels along with its adder tree, in a manner to avoid any DMA usage in the buffer connections between GEMM and Add kernels (Figure 4.3). For instance, when considering the group starting at (0,0)(0, 0) location in Figure 4.4, the adder tree is able to access the memory of 3 (out of 4) GEMM kernels directly (along the north, south and east direction). Notice that in this case, the GEMM kernel located at (1,0)(1, 0) does not directly communicate with the adder tree. However, the output buffer of this GEMM kernel can be placed to its north location (1,1)(1, 1); this is possible because of direct memory sharing between neighboring AIEs as shown in Figure 3.2. From here the adder tree AIE can access it directly, thus ensuring no DMA usage.

Figure 4.4. Examples of proposed placement strategy for Y=3,4Y = 3, 4.

Another placement example is the group starting at (0,5)(0, 5). Although, this placement is similar to the (0,0)(0, 0) case described above, notice that the adder tree is located on the opposite side. This is because the local data sharing is realized only on the west direction in even rows. We observe that in this case, the adder tree can only access 2 out of 4 GEMM kernels directly (located at (0,6)(0, 6) and (1,5)(1, 5)). However, the output buffers of all GEMM kernels can be placed such that no DMA is used. For instance, the output buffers of (0,5)(0, 5) and (0,7)(0, 7) kernels can be placed at (1,5)(1, 5) and (1,7)(1, 7) locations (east access), respectively, which can be both directly accessed by the adder tree.

An example for a group of Y=3Y = 3 kernels is also shown in Figure 4.4, where the explanation of its placement is similar to the examples discussed above. Finally, we note that the input data buffers of the GEMM kernels, as well as the output data buffer of the adder tree (refer to Figure 4.3) can be placed on any free memory space, again only accessing memory directly. We let the AMD/Xilinx AIE PnR (place and route) tool to make such decisions and optimize the whole AIE array mapping.

We exploit the aforementioned placement strategy to fill the entire AIE ar-

ray, thus mapping multiple GEMM/Add kernels. Figure 4.5 shows our two proposed placement patterns, P1P1 and P2P2. Notice that in pattern P1P1, we use a new “TT”–like shape outlined in orange color, while all other shapes of both patterns are similar to the examples in Figure 4.4. This is required to fill the whole AIE array and ensure that no AIEs remain unutilized. However, each “TT”–like shape will lead to a small DMA usage (one GEMM output buffer), as we show in the next section. Finally, we only propose patterns for Y=3,4Y = 3, 4, because these constitute the top-ranked solutions based on our optimization approach, as we also present in the next section.

Figure 4.5. Proposed placement patterns.

Evaluation

In this section, we first report single AIE kernel experimental results. Second, we present results and evaluation of full GEMM mapping on the AIE array. Finally, we perform a comprehensive comparison with the state-of-the-art approach, which exhibits the superiority of the MaxEVA framework.

We compile and simulate our designs by using the AMD/Xilinx Vitis 2022.1 version. Across all experiments, the AIE frequency is set to the maximum frequency of the VC1902 on VCK190, i.e., 1.25 GHz, while the PL operates at 312.5 MHz, as recommended in (AMD/Xilinx 2022b,c). To ensure rate matching without performance reduction between AIE and PL, we set the PLIO width to 128-bits AMD/Xilinx (2022b). Moreover, we use the AIE simulator AMD/Xilinx (2022c) to calculate the throughput of our designs, which we report as an average of 10 runs. Finally, power consumption is estimated through the AIE XPE tool AMD/Xilinx (2022e).

Since our focus is to achieve maximum GEMM throughput from the AIE array, isolating the AIE implementation from any source of performance degradation is crucial to obtain accurate evaluation. Such sources of performance reduction may arise from any design mapped to PL causing stalls, as well as the limited DRAM bandwidth. Therefore, we simulate our AIE designs without utilizing the PL and DRAM. We simulate the state-of-the-art CHARM framework as well by leveraging their open-source code Zhuang et al. (2023b,d), in the exactly same manner, with the same assumptions. This ensures a thorough and fair comparison between MaxEVA and CHARM frameworks, thus enabling us to report accurate throughput, power, and AIE resource utilization. However, we note that the CHARM code includes only fp32 implementation. In Zhuang et al. (2023d) the authors report results for int8 implementation as well, but their code is not publicly available. Therefore, for int8, we show a qualitative comparison based on their reported results.

Performance of Single AI Engine Kernels

For maximum efficiency, GEMM and Add kernels have been designed to leverage the vector processors of the AIEs. Our kernels are written in C/C++ utilizing the AIE APIs and several optimization compiler directives (pragmas) that perform software pipelining, loop unrolling/flattening and explicit independence between data AMD/Xilinx (2022b). Moreover, during our kernel design experimentation, we found that GEMM kernels with powers of two dimensions produce higher efficiency. Hence, in this work, we use powers of two during the optimization of the MM, KK, NN parameters, as also proposed in Zhuang et al. (2023b); Zhang et al. (2022c).

Table 4.1 presents the single AIE kernel results. For int8, the 32×128×3232 \times128 \times32 GEMM kernel was the only solution that satisfied all constraints in our optimization procedure (§ 4.1.3). All other values for MM, KK, NN are either I/O bandwidth limited or exceed the memory constraint of 14KB (eq. (6)). In contrast, for fp32, there are many top-ranked solutions that maximize the number of MACs, e.g., 16×64×3216 \times64 \times32, 64×16×3264 \times16 \times32, 32×32×3232 \times32 \times32, etc. However, we notice that all best solutions exhibit the same number of MACs (equal to 32768). Since 32×32×3232 \times32 \times32 GEMM kernel is one of our optimized solutions and also used in state-of-the-art CHARM Zhuang et al. (2023b,d), we obtain its code from their open-source code base. This ensures a fair comparison between their approach and ours.

Kernel typeKernel size M×K×NM \times K \times NLatency (cyc)Throughput (MACs/cyc)Efficiency
GEMM int832×128×3232 \times128 \times321075 (1×1 \times)121.9395.26%
Add int3232×3232 \times32164 (0.15×0.15 \times)6.2478.05%
GEMM fp32 Zhuang et al. (2023b,d)32×32×3232 \times32 \times324329 (1×1 \times)7.5794.70%
Add fp3232×3232 \times32167 (0.04×0.04 \times)6.1376.65%

Table 4.1. Single AI Engine kernel results.

From Table 4.1, we observe that the int8 GEMM kernel utilizes the vector processor of the AIE efficiently - a very high efficiency of 95.26% is achieved. The fp32 GEMM kernel obtained from Zhuang et al. (2023b,d) also presents very high efficiency (94.70%), and is designed by using AIE intrinsics. Moreover, we observe that int8 and fp32 Add kernels have very similar latencies, which are both significantly lower than GEMM latencies (164 vs. 1075 cycles for int8, and 167 vs. 4329 cycles for fp32). This validates that multiple Add kernels are able to run sequentially into a single AIE, without causing any performance degradation. We also observe that the relative latency ratio of Add kernel to GEMM kernel is notably lower for fp32 (0.04×\times) compared to int8 (0.15×\times). These relative ratios indicate that the AIE core running even multiple Add kernels remains idle for substantially longer for the fp32 case, affecting its power consumption accordingly (§ 4.3.2). Finally, we note that Add kernels also exhibit high efficiency (78.05% and 76.65% for int8 and fp32, respectively), though not as high as the efficiency of GEMM kernels. This performance difference is due to less data reuse on the AIE vector registers by Add kernels compared to GEMM kernels.

Performance of Matrix Multiplication on AI Engine Array

We utilize the MaxEVA framework to optimize the performance of the entire GEMM application. To map multiple kernels to the AIE array we wrote a parameterized C++ code for any values of XX, YY, ZZ by exploiting the ADF graph model.

MaxEVA vs. state-of-the-art CHARM for fp32 precision

From our multiple AIEs optimization of X×Y×ZX \times Y \times Z parameters, we found that the 10×4×810 \times4 \times8 solution maximizes the number of GEMM kernels. In this case, there are 320 GEMM kernels and 80 cores which run Add kernels, hence, all 400 AIE cores are utilized. However, this solution was not feasible because the AIE PnR tool failed due to routing congestion. This is due to the extra routing needed because of DMA usage (pattern P1), as well as the 100% utilization of the AIE cores, leaving no free space for successful routing. Our second top-ranked solution, i.e., 13×4×613 \times4 \times6, does not present any routing issues and is successfully mapped to the AIE array. In Table 4.2 we show this solution (row 1), which achieves a very high throughput of 5442.11 GOPS, outperforming the state-of-the-art approach by 20.8% (CHARM presents 4504.46 GOPS).

MaxEVA Cfg. X×Y×ZX \times Y \times Z (pat.)GEMM kernelsTotal AIE coresMemory banksDMA banksPLIOsThroughput (GOPS)Power (W)Energy Eff. (GOPS/W)
1. 13×4×613 \times4 \times6 (P1)312390 (97.5%)3138 (98.1%)18154 (79.0%)5442.11 (+20.8%)43.83124.16 (+20.4%)
2. 10×3×1010 \times3 \times10 (P2)300400 (100%)3190 (99.7%)0160 (82.1%)5405.33 (+20.0%)44.66121.03 (+17.4%)
3. 11×4×711 \times4 \times7 (P1)308385 (96.3%)3106 (97.1%)18149 (76.4%)5414.39 (+20.2%)44.01123.03 (+19.3%)
4. 11×3×911 \times3 \times9 (P2)297396 (99.0%)3176 (99.3%)0159 (81.5%)5382.27 (+19.5%)44.13121.96 (+18.3%)
5. 12×4×612 \times4 \times6 (P1)288360 (90.0%)2934 (91.7%)16144 (73.8%)5031.19 (+11.7%)40.68123.68 (+20.0%)
6. 12×3×812 \times3 \times8 (P2)288384 (96.0%)3092 (96.6%)0156 (80.0%)5225.05 (+16.0%)42.28123.58 (+19.9%)
CHARM384384 (96.0%)3086 (96.4%)080 (41.0%)4504.46 (+0%)43.69103.10 (+0%)

Table 4.2. Evaluation of several MaxEVA configurations for fp32 and comparison with state-of-the-art approach.

When further comparing the 13×4×613 \times4 \times6 MaxEVA solution to CHARM, we can observe from Table 4.2 (row 1) that our method utilizes the AIE array slightly more (390 vs. 384 AIE cores). However, we use considerably fewer cores for GEMM kernels (312 vs. 384), while the remaining (390−312=78390-312=78) AIE cores are used to run Add kernels. Also notice that CHARM has only GEMM kernels. Therefore, our solution is also able to achieve less AIE core power consumption (25.62 W vs. 26.95 W), because the cores that run the fp32 Add kernels remain idle most of the time. However, our implementation uses more memory banks than CHARM (3138 vs. 3086 out of 3200 available), which leads to higher data memory power consumption (18.21 W vs. 16.74 W). When computing the total AIE power as the summation of AIE core power and data memory power (AMD/Xilinx 2022e), we observe that our 13×4×613 \times4 \times6 design exhibits slightly higher power than CHARM (43.83 W vs. 43.69 W). Hence, our highest throughput solution presents also 20.4% higher energy efficiency compared to CHARM. We note here that our method of input broadcasting and output adder tree reduction, utilizes efficiently the available PLIOs (79% for 13×4×613 \times4 \times6). On the contrary, CHARM severely under-utilizes the device’s PLIOs (only 41%), which acts as a performance bottleneck for their design. Finally, we observe a very small DMA usage of 18 banks due to the “TT”–like shapes of pattern P1P1 (see Figure 4.5), as expected.

MaxEVA vs. state-of-the-art CHARM for int8 precision

Since int8 CHARM implementation is not open-sourced, we perform a qualitative comparison of performance. In Zhuang et al. (2023d) the authors report GEMM throughput of 28.15 TOPS for int8 CHARM design, when operating at 1 GHz frequency. To fairly compare with our results, we scale the aforementioned value to 1.25 GHz (our frequency), thus becoming 35.19 TOPS. In contrast, MaxEVA presents int8 maximum throughput of 77.01 TOPS, which is 2.19×2.19 \times higher than CHARM (Table 4.3). To get more confidence, we do a similar qualitative comparison for fp32 results. When scaling for fp32, we get a CHARM performance of 4342.33 GOPS at 1.25GHz. But our experimental results in Table 4.2 show a performance of 4504.46 GOPS for fp32 CHARM implementation. This small performance difference of 3.73% is expected because the authors in Zhuang et al. (2023d) measure the end-to-end performance on the VCK190. Thus, they experience sources of performance degradation, including the required zero padding Zhuang et al. (2023d). However, this small difference indicates that our experiments are accurate, and also validates our ∼2.19×\sim2.19\times performance gain for int8 over CHARM. This substantial performance gain is because CHARM utilizes only 192 AIE cores (48%) for int8, due to routing congestion issues Zhuang et al. (2023d). On the contrary, MaxEVA utilizes efficiently the entire AIE array, by mapping 390 cores (97.5%) for the highest throughput design (row 1 in Table 4.3). Finally, we note that due to the absence of open-source code, power for int8 CHARM cannot be calculated through the XPE tool, thus we are not able to present energy efficiency comparison.

MaxEVA Cfg. X×Y×ZX \times Y \times Z (pat.)GEMM kernelsTotal AIE coresMemory banksDMA banksPLIOsThroughput (TOPS)Power (W)Energy Eff. (TOPS/W)
1. 13×4×613 \times4 \times6 (P1)312390 (97.5%)3112 (97.3%)18154 (79.0%)77.01 (2.19×)66.831.152
2. 10×3×1010 \times3 \times10 (P2)300400 (100%)3194 (99.8%)0160 (82.1%)76.08 (2.16×)65.521.161
3. 11×4×711 \times4 \times7 (P1)308385 (96.3%)3096 (96.8%)18149 (76.4%)75.67 (2.15×)66.791.133
4. 11×3×911 \times3 \times9 (P2)297396 (99.0%)3178 (99.3%)0159 (81.5%)74.66 (2.12×)65.831.134
5. 12×4×612 \times4 \times6 (P1)288360 (90.0%)2918 (91.2%)16144 (73.8%)71.25 (2.02×)62.131.147
6. 12×3×812 \times3 \times8 (P2)288384 (96.0%)3080 (96.3%)0156 (80.0%)72.93 (2.07×)63.241.153
CHARM192192 (48.0%)–––35.19 (1×)––

Table 4.3. Evaluation of several MaxEVA configurations for int8 (results for CHARM obtained from Zhuang et al. (2023d)).

Placement Patterns Comparison

To provide a comprehensive evaluation of the proposed placement patterns, we show the two top-ranked solutions for each pattern in Table 4.2, Table 4.3 (rows 1–4). Based on the results, in general, we observe that pattern P2P2 has higher total AIE core and memory usage compared to P1P1, because it uses more Add kernels. However, higher AIE core usage does not necessarily lead to higher core power consumption. For instance, pattern P2P2 10×3×1010\times3\times10 design utilizes the entire AIE array (400 cores, feasible routing for pattern P2P2 since no DMA is used), but exhibits lower AIE core power than P1P1 13×4×613\times4\times6 design (25.54 W vs. 25.62 W and 47.44 W vs. 48.65 W for fp32 and int8, respectively). This is attributed to the fact that P2P2 has fewer GEMM kernels than P1P1 (300 vs. 312), and more cores that run Add kernels which remain mostly idle (100 vs. 78). However, when also including the memory power, the total power consumption depends on the number of memory banks used as well. In particular, when comparing 10×3×1010\times3\times10 with 13×4×613\times4\times6, the former shows slightly higher total power for fp32 (44.66 W vs. 43.83 W), while for int8 its power is lower (65.52 W vs. 66.83 W) than the latter.

In general, we observe from Table 4.2, Table 4.3 that the higher the number of GEMM kernels, the higher the throughput. However, this does not always hold true. For instance, for int8 precision, the 10×3×1010 \times3 \times10 design presents slightly higher throughput than 11×4×711 \times4 \times7 (76.08 vs. 75.67 TOPS), despite the fact that it has fewer GEMM kernels (300 vs. 308). This very small performance difference (<1%) is due to memory conflicts (leading to a few stalls), caused by dissimilarities in buffer optimizations from the AIE PnR tool AMD/Xilinx (2022c).

To quantify the effect of DMA usage on GEMM performance, we also implement the highest common solution (same number of GEMM kernels) between our two placement patterns (Table 4.2, Table 4.3, rows 5–6). In particular, when comparing 12×4×612 \times4 \times6 (P1P1) with 12×3×812 \times3 \times8 (P2P2), which both have 288 GEMM kernels, we notice that throughput is higher in P2P2 for both precisions. For instance, for int8, P1P1 attains 71.25 TOPS, while P2P2 achieves 72.93 TOPS. This is attributed to the DMA resources used in P1P1, which increase latency compared to P2P2, where no DMA is used. However, from Table 4.2, Table 4.3 we observe that P2P2 has higher energy efficiency for int8 (1.153 vs. 1.147 TOPS/W), while for fp32 the opposite occurs (123.58 vs. 123.68 GOPS/W). This arises from the fact that cores running Add kernels remain idle for significantly fewer cycles for int8 compared to fp32 (Table 4.1). To this end, we observe a higher percentage difference of total power for fp32 when comparing the aforementioned P1P1 and P2P2 solutions (40.68 W vs. 42.28 W for fp32, and 62.13 W vs. 63.24 W for int8). We notice that although in most cases P2P2 and P1P1 present higher energy efficiency for int8 and fp32, respectively, this relationship is complicated and depends on the number of GEMM kernels, the total cores used, as well as the memory banks and switch routing (as optimized by the AIE PnR tool).

Throughout all design points, 13×4×613 \times4 \times6 (P1P1) exhibits both highest throughput (5442.11 GOPS, 20.8% over CHARM) and energy efficiency (124.16 GOPS/W, 20.4% higher than CHARM), for fp32 precision. However, for int8, 13×4×613 \times4 \times6 (P1P1) has the highest throughput (77.01 TOPS, 2.19× higher than CHARM), while 10×3×1010 \times3 \times10 (P2P2) exhibits the greatest energy efficiency (1.161 TOPS/W). Finally, all of our optimized designs present very high resource utilization, using up to 100% AIE cores, 99.8% AIE memory and 82.1% PLIOs.

Variation of Performance under Different Matrix Sizes

We also explore the performance variation when altering the input matrix sizes (as powers of two) of the highest throughput design (Figure 4.6). The throughput is estimated by supposing that tiling is performed in PL for large matrix sizes, and also the PL does not cause any stalls (commonly attained in practice Zhuang et al. (2023d)). As expected, we observe that as the matrix size increases, the throughput also increases, and for large enough matrices it converges to its peak value. This is ascribed to zero padding in matrices such that they fit the native GEMM size of the 13×4×613 \times4 \times6 design. In particular, the 13×4×613 \times4 \times6 design is able to perform a GEMM of 416×128×192416 \times128 \times192 and 416×512×192416 \times512 \times192 size for fp32 and int8, respectively. To this end, we notice that for square matrices larger than ∼2K×2K×2K\sim2\mathrm{K} \times2\mathrm{K} \times2\mathrm{K}, less padding is needed throughout tiling, leading to almost peak performance.

Figure 4.6. Variation of throughput for different square matrix sizes for the 13×4×613 \times4 \times6 design.

Going a step further, we estimate the performance of full DNN inference, under the same assumptions as above. More specifically, when considering the MLP used in Zhuang et al. (2023b), MaxEVA achieves a throughput of 4735.94 GOPS. In contrast, when scaling the reported results from Zhuang et al. (2023b) to 1.25 GHz, we get 3670.88 GOPS, showcasing a higher MaxEVA performance of 29% over CHARM.

Chapter Summary & Impact

The Versal AIE architecture introduces a new paradigm in reconfigurable computing, posing several unique design challenges. To address these challenges, we propose MaxEVA, a framework that maximizes GEMM efficiency on Versal AIE by effectively leveraging the AIE’s architectural characteristics and resolving performance bottlenecks from multiple perspectives. Our experimental results demonstrate performance gains over the state-of-the-art design of up to 2.19×2.19 \times higher throughput and 20.4% higher energy efficiency. MaxEVA is broadly applicable across Versal AIE platforms and GEMM-based DL workloads, providing a foundation for the extensions presented in the following chapters

Chapter 5: Efficient Approaches for GEMM Acceleration on Leading AI-Optimized FPGAs

In this chapter, we present the frameworks developed for GEMM optimization on two leading AI-optimized FPGAs: the AMD Versal ACAP and the Intel Stratix 10 NX Taka et al. (2024)1. We extend the MaxEVA framework, detailed in Chapter 4, to support arbitrary matrix sizes, and we introduce a new framework for the Stratix 10 NX device that efficiently exploits its in-fabric AI TBs. This work provides insights and guidelines for optimizing GEMM-based applications on both platforms, while also delving into their programmability trade-offs and associated challenges.

Versal ACAP & Stratix 10 NX Characteristics

Table 5.1 demonstrates the characteristics of the two exemplar devices, where we showcase our proposed methods; the Versal VC1902 and Stratix 10 NX 2100. Both are large chips with roughly equal number of logic elements/cells, and similar on-chip memory capacity. Additionally, both devices have nearly equal theoretical peak throughput (int8) capabilities, under similar power envelopes (135 vs. 143 TOPS, and 165 vs. 125 W, for Versal VCK and Stratix Langhammer et al. [10], respectively).

DeviceVersal VC1902Stratix 10 NX 2100
FPGA
Logic Elements/Cells1968K2073K
On-chip Memory20.5 MB16.75 MB
DSP Slices1968–
Tensor Blocks–3960
AIE Cores400–
AIE Memory12.5 MB–
Processing SystemARM A72 + R5F–
DRAM TechnologyDDR4HBM2
Peak DRAM BW102.4 GB/s512 GB/s
Theor. Peak TOPS (int8)*135 TOPS143 TOPS
Peak Power165 W125 W
Process7nm TSMC14nm Intel

Table 5.1. Hardware platform characteristics.

Besides their distinct architectures, the two devices also present differences in their DRAM technology and the manufacturing nodes (Table 5.1). Versal has 5×5\times lower bandwidth (BW) than Stratix. Moreover, Versal is manufactured in a 7nm7\mathrm{nm} TSMC process, while Stratix uses 14nm14\mathrm{nm} Intel. The main focus of this paper is to provide a comprehensive evaluation of various aspects in GEMM optimization, emphasizing architecture-specific methodologies, which are largely agnostic to DRAM

and manufacturing technology. Thus, we perform experiments on designs that operate within on-chip memory. However, to enable a complete and thorough analysis we extensively examine off-chip memory considerations and requirements for both devices.

GEMM Design & Optimization

In this section, we describe the design, mapping, and optimization of GEMM-based workloads on both Versal ACAP and Stratix 10 NX FPGAs.

GEMM Implementation on Versal ACAP

We leverage the MaxEVA open-source code max (2023), and extend it to include on-chip buffers in the PL, tiling logic, as well as Load/Store units to communicate with DDR. The PL is designed using Vitis HLS, as extensively used in prior works on Versal to enhance productivity Zhuang et al. (2023b); Zhang et al. (2022c); Brown (2023); Zhang et al. (2022b). Optimization of the PL design is attained through analytical modeling for maximization of on-chip data reuse (to reduce DDR BW requirements, as VC1902 has limited BW, Table 5.1). Moreover, this method effectively resolves severe memory over-utilization issues caused by Vitis HLS. In the following sections, we provide a brief overview of the MaxEVA AIE design, and we elaborate on the PL design and optimization.

GEMM Multi-Level Tiling Scheme

Figure 5.1 illustrates the tiling scheme used in MaxEVA for the AIE, as well as our tiling method on PL. Each AIE core executes a matrix multiplication (MatMul) kernel of M×K×NM \times K \times N size (first tiling level). The parameters X,Y,ZX, Y, Z determine the multiple MatMul kernels running on the entire AIE array, as discussed in the next section (second tiling level). We incorporate an additional level of buffering in the PL, introducing three new parameters: U,V,WU, V, W (third tiling level). The AIE-specific parameters (X,Y,Z,M,K,N)(X, Y, Z, M, K, N) are optimized by utilizing the MaxEVA framework, while the PL-specific parameters (U,V,W)(U, V, W) are optimized using our proposed PL optimization procedure (introduced in § 5.2.4). All the aforementioned parameters determine the AA, BB and CC matrix sizes supported out of on-chip memory in the PL, as depicted in Figure 5.1. Therefore, we define two GEMM sizes. First, the compute GEMM size, i.e., (X⋅M)×(Y⋅K)×(Z⋅N)(X \cdot M) \times(Y \cdot K) \times(Z \cdot N), running on the AIE array. Second, the native buffer size, i.e., (U⋅X⋅M)×(V⋅Y⋅K)×(W⋅Z⋅N)(U \cdot X \cdot M) \times(V \cdot Y \cdot K) \times(W \cdot Z \cdot N), of the data stored in the on-chip buffers inside the PL.

Figure 5.1. Multi-level tiling scheme for GEMM on Versal ACAP.

GEMM Mapping on AIE Array

The limited number of AIE-PL tiles on Versal devices (notably VC1902 has only 39 AIE-PL tiles out of 50 columns in the AIE AMD/Xilinx (2023)), is one of the main challenges in GEMM design. To overcome the limited PLIO bottleneck, MaxEVA utilizes the following two techniques Taka et al. (2023). First, the number of input PLIO ports is reduced by broadcasting input data to multiple AIEs. Second, the output PLIO ports are decreased by performing adder tree reduction (via Add kernels) on the AIE. This approach exploits only the most efficient circuit-switching AIE mechanism, as opposed to packet-switching used in Zhuang et al. (2023b,c). Readers unfamiliar with MaxEVA are encouraged to read Chapter 4 first, which describes the framework in detail. In the following sections, we describe our PL implementation and optimization procedure.

PL Implementation

The lower part of Figure 5.2 (PL) shows a high-level block diagram of the PL design. The AA, BB and CC matrices are stored in on-chip buffers, exploiting the PL BRAM and URAM resources. To provide sufficient bandwidth to/from the AIE, each buffer is partitioned (utilizing the HLS pragma array_partition), to exactly match the corresponding AIE PLIO ports. We note that although not shown in Figure 5.2 for simplicity, we employ double-buffering to effectively overlap GEMM computation with external off-chip DRAM communication.

GEMM accelerator design on Versal AIE and PL, showing an AIE array, PL buffers, MatMul and Add kernels, PLIOs, adders, AIE cores, and broadcasting

Figure 5.2. GEMM accelerator design on Versal AIE and PL.

We set the PLIO width to 128-bits to ensure rate matching between AIE and PL without performance loss Taka et al. (2023); AMD/Xilinx (2022b). In addition, while the data type of the AA and BB buffers is int8, all accumulations are performed in 32-bits. Thus, when sending data from the AA and BB buffers to the AIE, we concatenate 16 8-bit values for each input PLIO to form a 128-bit vector. In contrast, when receiving data from the output PLIOs of the AIE, we pack 4 32-bit values and store them in the CC buffer. The logical size of the PL buffers is shown in Figure 5.1. However, in the physical implementation, the buffers have a width of 128-bits to match the PLIO width, and are partitioned into smaller buffers (Figure 5.2). The partition factors ({A,B,C}part\{A, B, C\}_{part}) and depths ({A,B,C}depth\{A, B, C\}_{depth}) of the buffers are expressed as:

Apart=2⋅X⋅Y,Adepth=U⋅V⋅M⋅K/16(10)A_{part} = 2 \cdot X \cdot Y,\qquad A_{depth} = U \cdot V \cdot M \cdot K/16 \tag*{(10)}
Bpart=2⋅Y⋅Z,Bdepth=V⋅W⋅K⋅N/16(11)B_{part} = 2 \cdot Y \cdot Z,\qquad B_{depth} = V \cdot W \cdot K \cdot N/16 \tag*{(11)}
Cpart=2⋅X⋅Z,Cdepth=U⋅W⋅M⋅N/4(12)C_{part} = 2 \cdot X \cdot Z,\qquad C_{depth} = U \cdot W \cdot M \cdot N/4 \tag*{(12)}

We multiply the partition factor by 2 for double-buffering. We also divide the depth by 16 and 4 to match the 128-bit packing.

Partial results from the AIE are accumulated in the PL to handle the reduction across tiles in the V⋅Y⋅KV \cdot Y \cdot K dimension in Figure 5.1, or for larger matrices if needed. As depicted in Figure 5.2, we implement 4 32-bit adders in soft logic for each partitioned CC buffer (4 ⋅X⋅Z\cdot X \cdot Z adders in total, all executing in parallel). Each new partial result from every PLIO port (AXI4-Stream interface) is accumulated to its corresponding CC buffer address every clock cycle. This PL logic is pipelined (using the pragma pipeline) with an initiation interval (II) of 1, such that in every clock cycle a new partial result can be accumulated. To this end, one load (for the current PLIO values) and one store operation (for the previous PLIO values) is required for every partitioned CC buffer. Thus, we configure the CC buffers in simple dual-port mode Ver (2020); Vit (2022a) (pragma bind_storage). This ensures a stall-free PL implementation that does not introduce any throughput degradation in the entire design.

In contrast, the input buffers AA, BB are configured in single-port mode, since either only a load (send data to AIE) or store operation (receive data from DRAM) is required in each cycle for double-buffering. Moreover, Load/Store units are implemented in the PL (not shown in Figure 5.2) to communicate with DRAM. Finally, it is important to note that our implementation is symmetric in terms of the first and last dimensions in both the compute GEMM size and the native buffer size (refer to § 5.2.2 for definitions and § 5.4.1.2 for evaluation).

Memory Optimization Strategy

To maximize the data reuse of the on-chip buffers, we propose an optimization methodology based on analytical modeling. Our model utilizes the multiple configurations of BRAMs and URAMs to identify the optimal values of the UU, VV, WW parameters, as well as the buffer mapping to BRAMs and URAMs. Although Vitis HLS includes the capability to automatically map buffers to memory resources (impl=AUTO in pragma bind_storage, we found that it fails to find an operational mapping in several cases (§ 5.4.1.1). This automatic mapping generated by Vitis HLS leads to severe over-utilization of memory resources, and thus, failure to place and route (PnR). To the best of our knowledge, this HLS limitation has not been identified in any prior work. Several works have focused on optimizing the logical-to-physical memory mapping during the synthesis/PnR phases Voss et al. (2019); Karchmer and Rose (1994); Tessier et al. (2007), while others Vasiljevic and Chow (2014) propose tools to assist designers at the user-level with automatic memory mapping targeting BRAMs, but not URAMs. Our approach focuses on overcoming this HLS limitation by identifying an optimal mapping to both BRAMs and URAMs, and subsequently guiding the HLS tool according to this mapping (through impl={BRAM/URAM} in pragma bind_storage).

The inputs of our model include a MaxEVA solution (X,Y,Z,M,K,NX, Y, Z, M, K, N parameters), and PL-specific parameters, i.e., BRAM, URAM configurations supported in Versal devices Ver (2020) and the available on-chip memory resources. The model produces as outputs the optimal U,V,WU, V, W parameters, as well as the AA, BB and CC buffer mapping to BRAMs/URAMs.

Besides the optimal parameter finding and mapping to memory resources, our model also estimates the BRAM/URAM utilization with 100% accuracy in all cases (§ 5.4.1.1). First, we model the BRAM/URAM utilization based on the depth of each partitioned buffer (all buffers have 128-bits width and are highly partitioned). Figure 5.3a shows an example of our modeling when 512<depth≤1K512 < depth \le1\mathrm{K}. In this case, BRAMs are configured as 1K×361\mathrm{K}\times36 and 4 36K BRAMs are needed to construct the 128-bitwidth buffer. Observe that a portion of the rightmost BRAM in Figure 5.3a is not utilized in this situation. To this end, we define BRAM/URAM efficiency as the fraction of the logical buffer size to the total size of memory blocks used. For instance, when assuming that all 1K entries are used in Figure 5.3a, the BRAM efficiency is determined by the utilized width, i.e., 128/(36⋅4)=88.89%128/(36\cdot4)=88.89\%.

BRAM configurations example and proposed modeling

Figure 5.3. BRAM configurations example and proposed modeling.

In Figure 5.3b, we show another example when 1K<depth≤2K1K < depth \le2K. In this case, BRAMs are configured as 2 independent 1K×18 blocks. With 7 2K×18 blocks, 126-bits can be mapped. The remaining 2-bits can be efficiently mapped by packing 2K×2-bits on a single 1K×18 block. This is possible since 2K×2-bits can be logically viewed as 1K×4-bits, where additional multiplexing logic is needed to determine the corresponding 2-bits, based on the address. Thus, 7.5 36K BRAMs (or 15 18K blocks) are required in total. The remaining 18K BRAM (outlined in orange in Figure 5.3b), can be used for other purposes, since it is physically independent.

Similarly, we also model the cases when depth≤512depth \le512 (2 36K BRAMs), and 2K<depth≤4K2K < depth \le4K (15 36K BRAMs) Ver (2020). In addition, since URAMs support one configuration (4K×72), 2 URAMs are required for depth≤4Kdepth \le4K. In eq. (13) and 5.5, we summarize the number of BRAMs (fBf_B) and URAMs (fUf_U), respectively, as functions of the buffer depth.

fB(depth)={2,if depth≤5124,if 512<depth≤1K7.5,if 1K<depth≤2K15,if 2K<depth≤4K(13)f_B(depth) = \begin{cases} 2, & \text{if } depth \le512 \\ 4, & \text{if } 512 < depth \le1K \\ 7.5, & \text{if } 1K < depth \le2K \\ 15, & \text{if } 2K < depth \le4K \end{cases} \tag*{(13)}
fU(depth)=2,if depth≤4K(14)f_U(depth) = 2,\quad\text{if } depth \le4K \tag*{(14)}

Afterwards, we establish a constraint that limits each buffer’s depth to 4K (eq. (15)), since all buffers are highly partitioned with a relatively small depth. This constraint allows a very high BRAM, URAM efficiency, as shown in § 5.4.1.2.

{Adepth,Bdepth,Cdepth}≤4K(15)\{A_{depth}, B_{depth}, C_{depth}\} \le4K \tag*{(15)}

Finally, we impose constraints to ensure that the total number of utilized BRAMs and URAMs does not exceed the device’s available resources (B36KB_{36K} and U288KU_{288K}). Eq. (16) and 5.8 show an example of the constraints for BRAMs and URAMs, respectively. In this example, buffers AA, BB have been mapped to BRAMs, while buffer CC is mapped to URAMs.

Apart⋅fB(Adepth)+Bpart⋅fB(Bdepth)≤B36K(16)A_{part} \cdot f_{B}(A_{depth}) + B_{part} \cdot f_{B}(B_{depth}) \le B_{36K} \tag*{(16)}
Cpart⋅fU(Cdepth)≤U288K(17)C_{part} \cdot f_{U}(C_{depth}) \le U_{288K} \tag*{(17)}

Constraints similar to 5.7, 5.8 are applied for all permutations of mapping buffers AA, BB and CC to BRAMs and URAMs.

The solution of UU, VV, WW and buffer mapping to BRAMs, URAMs can be formulated as an integer programming (IP) optimization problem with the aforementioned constraints. We solve the IP exhaustively by setting the maximization of on-chip data reuse as the objective. The data reuse of all buffers is encapsulated in the product of U⋅V⋅WU \cdot V \cdot W. Notice from Figure 5.1 that buffers AA and BB are reused WW and UU times in GEMM, respectively, while CC is reused and updated (during accumulation) VV times. In addition, note that maximizing data reuse also leads to maximization of the PL buffer sizes, under the resource constraints. We report, implement and explore the trade-offs of multiple top-ranked solutions in § 5.4.1.

GEMM Implementation on Stratix 10 NX

We implement a configurable MatMul accelerator consisting of a control logic and a spatial 2D layout on the Stratix 10 NX architecture. The accelerator’s TBs operate out of local, on-chip memory consisting of AA, BB and CC buffers (Figure 3.3) of size M′⋅K′M' \cdot K', K′⋅N′K' \cdot N' and M′⋅N′M' \cdot N', respectively. The M′×K′×N′M' \times K' \times N' MatMul size is defined as the native buffer size, similar to Versal. We conduct a DSE to optimize for accelerator throughput and employ analytical modeling to optimize for on-chip data reuse. Below we delineate the TB layout, the dataflow, the memory architecture, optimization strategies and the automatic code generation tool we developed.

TB Layout

We utilize groups of TB arrays of configurable length that run in parallel and operate in the cascade loading mode. An array’s first TB (colored white in Figure 5.4) serves as its point of entry for AA data (Figure 5.4), performing no computation. Subsequent TBs in the array receive their AA data (and store them in their registers) via their casc_data_in port and their BB data through their data_in port (Figure 3.3). The latency of loading AA blocks is three cycles per TB, but can be hidden when overlapped with dot-product operations. While bank 0 (Figure 3.3) is being multiplied with a set of BB blocks, bank 1 can be loaded and vice-versa. An array’s accumulated outputs are exposed through the data_out port of its final TB.

Figure 5.4. 2D TB layout and GEMM dataflow on Stratix 10 NX.

We define four architecture parameters for the TB layout:

  • Parameter TBlenTB_{len}: Length of a TB array, equal to four in Figure 5.4. This length can be less than or equal to 36.

  • Parameter KpK_p: Set of arrays that work in parallel across the K′K' dimension, which we refer to as a reduction group (KpK_p equal to two in Figure 5.4). The data_out outputs of all the arrays in a reduction group are fed into its corresponding adder tree, the output of which is accumulated at the CC buffer (Figure 5.4). These adders are implemented in soft logic.

  • Parameter NpN_p: Set of reduction groups that contain the same AA blocks, but get multiplied with different BB blocks (equal to two in Figure 5.4). NpN_p allows

exploiting parallelism across N′N'. We refer to this set of reduction groups as an NpN_p block.

  • Parameter MpM_p: Number of NpN_p blocks that allow parallelism across M′M', equal to three in Figure 5.4. Each NpN_p block uses different AA blocks, but the same BB blocks.

Dataflow

Each TB holds a 3×103 \times10 block of AA in its registers, represented by a shape in Figure 5.4, and it multiplies it with a 10×110 \times1 block of BB in each clock cycle. This BB block belongs to a set of N′/NpN'/N_p 10×110 \times1 blocks, represented by a color in the BB Buffer (Figure 5.4). The TB will sequentially multiply its AA block with all BB blocks in the colored set. Given enough BB blocks in this set (i.e., a large enough N′N^{\prime}), the TB’s register loading latency can be fully hidden. Different TBs of the same array hold separate AA blocks (shapes) and process different sets of BB blocks (colors). Partial dot-products are propagated along an array through cascade connections and added to the next dot-products on-the-fly. This process takes two clock cycles per TB (§ 3.2.1). Therefore, a TB starts processing data two cycles after its previous TB. Finally, an array’s accumulated dot-products are exposed at its last TB.

Different arrays in the same reduction group hold separate blocks of AA (shapes) and process separate sets of BB blocks (colors) in parallel, each generating a new output every clock cycle. Reduction groups of the same NpN_p block hold the same AA blocks (shapes), but get multiplied with different sets of BB blocks (colors) in parallel, allowing parallelism across N′N^{\prime}. Finally, different NpN_p blocks contain different sets of AA blocks (shapes), but they all get multiplied with the same BB blocks (colors) in parallel, allowing parallelism across M′M^{\prime}.

The total number of utilized TBs is equal to TBlen⋅Kp⋅Np⋅MpTB_{len} \cdot K_p \cdot N_p \cdot M_p. We define the compute GEMM size as (Mp⋅3)×([TBlen−1]⋅Kp⋅10)×(Np)(M_p \cdot3) \times([TB_{len} - 1] \cdot K_p \cdot10) \times(N_p). This is the size of the blocks the accelerator processes at the lowest memory hierarchy level. We refer to this GEMM size as DM′×DK′×DN′D_{M^{\prime}} \times D_{K^{\prime}} \times D_{N^{\prime}}. The multiplication by 3 and 10 accounts for the size of the AA, BB blocks. We subtract by 1 for the wasted TB of each array and we multiply by KpK_p for all arrays in a reduction group. Due to tiling over the compute GEMM size, M′M^{\prime}, K′K^{\prime} and N′N^{\prime} need to be multiples of DM′D_{M^{\prime}}, DK′D_{K^{\prime}} and DN′D_{N^{\prime}}, respectively. Moreover, N′N^{\prime} must be sufficiently large to allow hiding the TB register loading latency.

Architectural Considerations

Below we elaborate on architectural considerations related to the four architecture parameters, along with the constraints they introduce.

Parameter TBlenTB_{len}: As mentioned, the TB chain granularity is equal to 36. Hence, as multiple arrays may fit in a chain, the array length TBlenTB_{\mathrm{len}} must be a factor of 36 to avoid fragmentation during placement of a design with a large number of TBs. Sound values for TBlenTB_{\mathrm{len}} are 36, 18, 12 and 9. Low TBlenTB_{\mathrm{len}} leads to more wasted TBs (first TB in every array). Meanwhile, this increases flexibility in executing different matrix sizes, as they do not require a high reduction dimension (K′K^{\prime}). Large TBlenTB_{\mathrm{len}} increases latency for loading AA, but also results in higher peak throughput. A TBlenTB_{\mathrm{len}} of 36 has a loading latency of 36⋅3=10836 \cdot3 = 108 cycles. However, because each AA block is multiplied with a set of BB blocks, if this set contains at least 108 blocks, depending on N′N^{\prime}, this latency can be hidden.

Parameter KpK_p: A TB array processes (TBlen−1)⋅10(TB_{\mathrm{len}}-1) \cdot10 elements across the reduction dimension (K′K^{\prime}) each cycle. KpK_p increases the parallelism across K′K^{\prime} (i.e., DK′D_{K^{\prime}}), while keeping the signal fan-out low. However, if DK′D_{K^{\prime}} is large, but K′K^{\prime} is small, some TBs in a reduction group will be under-utilized.

Parameter NpN_p: In order to utilize more TBs, parallelism across N′N^{\prime} and M′M^{\prime} is also essential. NpN_p requires broadcasting AA blocks across all reduction groups in an NpN_p block, while MpM_p requires broadcasting BB blocks across all NpN_p blocks (Figure 5.4). However, each array has only one input for AA data (at its initial TB), but a considerably larger amount of inputs for BB data (equal to TBlen−1TB_{\mathrm{len}}-1). As a result, MpM_p increases the fanout of a lot more signals than NpN_p does, which increases FPGA routing congestion. Finally, a higher NpN_p leads to a larger minimum required value of N′N^{\prime} to hide the TB loading latency.

Parameter MpM_p: MpM_p enables parallelism across M′M^{\prime}, but also increases the fan-out of BB blocks, which, as mentioned, can lead to significant routing congestion in the FPGA, adversely affecting the attainable clock frequency and performance.

Memory Architecture

Evidently, when a large number of TBs are used, high BRAM bandwidth is needed to feed them blocks of AA and BB. We employ a memory architecture that is designed to (i) enable high compute throughput and (ii) to maximize data reuse, which will amortize off-chip BW requirements. In a similar manner to Versal, we maintain separate buffers for AA, BB and CC and we utilize double-buffering. The Load/Store units perform read-only operations on CC buffers and write-only operations on AA and BB buffers. The compute logic performs read and write operations on CC and read-only operations on AA and BB. M20K blocks configured in simple dual-port mode are used to implement all buffers. For AA and BB, the compute and Load/Store units have their own port for reading and writing, respectively. For double-buffering, we double the depth of AA and BB, and split CC to two equal-sized buffers. A more detailed description of the architecture of buffers AA, BB and CC is provided below.

Buffers AA, BB: As illustrated in Figure 5.4, a separate AA block must be fed to each array of a reduction group and across different NpN_p blocks. Additionally, each TB in an array (except for TB0TB_0), receives separate BB blocks every cycle. Similarly, all arrays in all reduction groups of an NpN_p block require different blocks of BB each clock cycle. Also, the bitwidth of the AA and BB ports is 80-bits. Therefore, the partition factors ApartA_{part}, BpartB_{part} of AA, BB and the depth AdepthA_{depth}, BdepthB_{depth} of each smaller partitioned buffer of AA, BB are:

Bpart=(TBlen−1)⋅Kp⋅Np(18)B_{part} = (TB_{len} - 1) \cdot K_p \cdot N_p \tag*{(18)}
Bdepth=2⋅K′⋅N′/(Bpart⋅10)(19)B_{depth} = 2 \cdot K' \cdot N' / (B_{part} \cdot10) \tag*{(19)}
Apart=Mp⋅Kp,Adepth=2⋅M′⋅K′/(Apart⋅10)(20)A_{part} = M_p \cdot K_p,\quad A_{depth} = 2 \cdot M' \cdot K' / (A_{part} \cdot10) \tag*{(20)}

We multiply by 2 for double-buffering and we also multiply by 10, because 80-bits is 10 bytes. The 80-bit width of these buffers is implemented using M20Ks operating in the 512×40512 \times40, 1024×201024 \times20 or 2048×102048 \times10 configurations Int (2023a). We model the number of M20Ks required to implement each buffer, fM80f_{M80}, as a function of the depth. For the AA buffer, since AdepthA_{depth} is sufficiently large, all M20K configurations lead to equivalent, accurate solutions in our model. However, BdepthB_{depth} was always less than 1024. If depth≤512depth \le512, two M20Ks in 512×40512 \times40 mode can be used. If 512<depth≤1024512 < depth \le1024, both 512×40512 \times40 and 1024×201024 \times20 configurations lead to a usage of four M20Ks. As a result, in all our designs fM80f_{M80} is estimated as:

fM80(depth)=2⋅⌈depth/512⌉(21)f_{M80}(depth) = 2 \cdot\left\lceil depth/512 \right\rceil \tag*{(21)}

The symbol ⌈ ⌉\lceil\ \rceil denotes rounding up to the next integer.

Buffer CC: A TB has three data_out ports, so reduction groups generate three distinct CC values each. The partition factor CpartC_{part} and the depth CdepthC_{depth} of each smaller buffer is:

Cpart=Mp⋅Np⋅3⋅2,Cdepth=M′⋅N′⋅2/Cpart(22)C_{part} = M_p \cdot N_p \cdot3 \cdot2,\quad C_{depth} = M' \cdot N' \cdot2/C_{part} \tag*{(22)}

We multiply by 2 for double-buffering. The valid M20K configurations for the 32-bitwidth of the CC buffers are 2048×82048 \times8, 1024×161024 \times16 or 512×32512 \times32. The number of M20Ks, fM32f_{M32}, required for CC buffers is also modeled as a function of the depth. Since the depth of the CC buffers is sufficiently large, all above M20K configurations led to equivalent, accurate solutions in our model, estimated by:

fM32(depth)=⌈depth/512⌉(23)f_{M32}(depth) = \left\lceil depth/512 \right\rceil \tag*{(23)}

Optimization Strategies

Similar to Versal, we formulate the selection of M′M', K′K', N′N' for a given TB configuration as an IP problem and solve it exhaustively to optimize for data reuse, by maximizing the product M′⋅K′⋅N′M' \cdot K' \cdot N'. We impose two constraints on the IP. First, the utilized M20Ks must not exceed the device’s available M20K blocks (BM20KB_{M20K}). Second, N′N' is set to be sufficiently large to hide TB register loading latency:

Apart⋅fM80(Adepth)+Bpart⋅fM80(Bdepth)+Cpart⋅fM32(Cdepth)≤BM20K(24)A_{part} \cdot f_{M80}(A_{depth}) + B_{part} \cdot f_{M80}(B_{depth}) + C_{part} \cdot f_{M32}(C_{depth}) \leq B_{M20K} \tag*{(24)}
N′≥TBlen⋅3⋅Np(25)N' \geq TB_{len} \cdot3 \cdot N_p \tag*{(25)}

Since throughput is directly related to operating frequency, we implement several optimizations to shorten critical paths. These include replication of control logic and insertion of a configurable number of pipeline stages along the data and address datapaths. Additionally, we conduct an extensive exploration on various TB architecture parameters, to find configurations that maximize frequency, and thus, throughput (see § 5.4.2).

In an effort to reduce signal fan-out, we also tried inserting registers between NpN_p blocks in a systolic fashion in order to propagate BB blocks, thus minimizing broadcasting. However, our experiments showed no frequency improvements, while greatly increasing ALM usage (up to ∼40%\sim40\%, as opposed to a maximum of ∼19%\sim19\% with the proposed design, see Table 5.4). Such a large number of soft-logic pipeline registers is undesirable, as it can reduce energy efficiency and also render logic resources unavailable, as mentioned in Langhammer et al. [10]. Thus, the systolic distribution approach was not considered in our final designs.

TB config.ALMsBRAMsTBsFreq. (MHz)Thrpt. (TOPS)Power (W)En. Eff. (TOPS/W)
18×16×4×3124K (18%)6304 (92%)3456 (87%)34968.0051.11.331
18×8×8×3123K (17%)6064 (89%)3456 (87%)34567.2150.21.340
9×16×5×5127K (18%)5840 (85%)3600 (91%)35066.9452.51.275
12×8×6×6100K (14%)6144 (90%)3456 (87%)33864.0048.61.317
18×16×3×4108K (15%)6272 (92%)3456 (87%)32763.7147.31.347
9×16×6×4131K (19%)6464 (94%)3456 (87%)34262.8850.71.241
18×8×3×881K (12%)6064 (89%)3456 (87%)32162.4046.51.342
9×8×10×5124K (18%)5840 (85%)3600 (91%)32061.2148.71.257
18×8×5×5101K (14%)6150 (90%)3600 (91%)30161.0845.41.346
18×4×8×691K (13%)6080 (89%)3456 (87%)31260.6946.21.315

Table 5.4. Evaluation of 10 top-ranked GEMM designs on Stratix 10 NX.

Automatic RTL Code Generation

We develop a Python-based tool that automatically generates the aforementioned architecture’s RTL code, including the control logic, TB layout and memory. The input configurations of the tool are the four architecture parameters, the M′M^{\prime}, K′K^{\prime}, N′N^{\prime} dimensions, and the number of pipeline stages for both address and data.

Evaluation

In this section, we discuss and compare various aspects of GEMM design and optimization on both Versal and Stratix devices.

Versal DSE Evaluation

For the Versal AIE, we obtain the two most efficient solutions from MaxEVA. As found in Taka et al. (2023), solution P1P1 13×4×613 \times4 \times6 (X×Y×ZX \times Y \times Z) shows the highest throughput, while the P2P2 10×3×1010 \times3 \times10 presents the highest energy efficiency. Both use a 32×128×3232 \times128 \times32 (M×K×NM \times K \times N) single AIE MatMul kernel with 95% throughput efficiency.

For the PL, we utilize our model (§ 5.2.4), which takes as input a MaxEVA solution (X,Y,Z,M,K,NX, Y, Z, M, K, N parameters) and produces the optimal values of U,V,WU, V, W. We perform DSE on the 5 top-ranked U×V×WU \times V \times W solutions, for each of the two MaxEVA solutions (10 designs in total).

We use the AMD/Xilinx Vitis 2022.1 version to implement and compile our designs. The PL part is designed using Vitis HLS, while AIE-PL linking is achieved via the V++ compiler. Throughout all experiments, the AIE frequency is set to its maximum value of 1.25 GHz, while the PL frequency ranges from 275–300 MHz, depending on the PL configuration. To calculate the throughput of our designs, we use hardware emulation in Vitis, while power is estimated through the post-implementation Vivado Power Analysis Tool Viv (2022).

Model Estimation

In Table 5.2, we present 4 top-ranked solutions of our PL optimization procedure (parameters U×V×WU \times V \times W and buffer mapping to BRAM/URAM resources). First, we observe that our model estimates the BRAM/URAM utilization with 100% accuracy in all cases. For example, for solution 4×2×44 \times2 \times4 (P1P1), the model suggests (“Model Est.” column) that buffers {A,B,C}\{A, B, C\} should be mapped to {B,U,U}\{B, U, U\}, where BB, UU denote BRAM, URAM, respectively. When we guide the HLS tool to use this mapping, both our model and HLS synthesis report identical BRAM/URAM utilization, i.e., 780 (81%) / 408 (88%). Letting the HLS tool instead to automatically map buffers (“HLS AUTO” column), results in a severe over-utilization of URAMs (616 or 133%), without any BRAM usage. Vivado PnR attempts to implement this HLS AUTO solution by mapping the surplus URAMs to BRAMs. However, it generates an error reporting a 119.4% BRAM and 99.8% URAM utilization. A similar result is observed for another solution, i.e., 4×2×44 \times2 \times4 (P2P2). However, for the other two solutions shown in Table 5.2, HLS AUTO is able to successfully find an efficient mapping. In these cases, both our model and HLS AUTO produce exactly the same BRAM/URAM utilization, e.g., 416 (43%) / 408 (88%), for 2×2×82 \times2 \times8 (P1P1). We note that HLS AUTO fails to implement 5 of the 10 top solutions (Table 5.3), justifying the necessity of our approach.

U×V×WU \times V \times W (MaxEVA P.){A,B,C}\{A, B, C\}Model Estimation BRAMsModel Estimation URAMsHLS AUTO BRAMsHLS AUTO URAMs
4×2×44 \times2 \times4 (P1){B,U,U}\{\mathcal{B}, \mathcal{U}, \mathcal{U}\}780 (81%)408 (88%)0 (0%)616 (133%)
4×2×44 \times2 \times4 (P2){B,B,U}\{\mathcal{B}, \mathcal{B}, \mathcal{U}\}900 (93%)400 (86%)0 (0%)640 (138%)
2×2×82 \times2 \times8 (P1){B,U,U}\{\mathcal{B}, \mathcal{U}, \mathcal{U}\}416 (43%)408 (88%)416 (43%)408 (88%)
2×8×22 \times8 \times2 (P2){U,U,B}\{\mathcal{U}, \mathcal{U}, \mathcal{B}\}800 (83%)240 (52%)800 (83%)240 (52%)

Table 2. Optimization model estimation for various solutions and comparison with HLS AUTO mapping.

GEMM Performance

In Table 5.3, we show various metrics for our 10 top solutions. In all solutions, throughput is calculated on their native buffer sizes (Sec. 5.2.1.1) Overall, we observe that all designs exhibit high throughput ranging from 75.4–77.01 TOPS. However, to maintain such high throughput, several designs require higher DRAM BW compared to the VC1902’s BW (102.4 GB/s). We note that we calculate DRAM BW as the worst-case of concurrent loads for buffers AA, BB and stores for CC (all as 8-bits due to quantization in DL). With more sophisticated data reuse techniques as in Jia et al. (2022), this requirement can be amortized. However, in this work, we consider the worst-case BW scenario for the sake of generality. Therefore, we only examine designs that stay within the BW of the VC1902 device (highlighted in bold in Table 5.3).

From these designs, 2×2×82 \times2 \times8 (P1) shows both the highest throughput, i.e., 76.93 TOPS, and the best energy efficiency, i.e., 0.938 TOPS/W. For all valid solutions, throughput ranges from 75.40–76.93 TOPS, which is 58.9–60.1% of the theoretical peak throughput of VC1902, and the same as the state-of-the-art MaxEVA Taka et al. (2023). Additionally, energy efficiency ranges from 0.911–0.938 TOPS/W. In all cases, we notice a very high resource utilization, up to 94% BRAMs, 88% URAMs, and 100% AIE cores, with a small LUT usage of up to 11%. Moreover, we observe high RAM efficiency of 75.7–90.2%, as a direct result of our modeling and optimization methodology. Finally, although not shown in Table 5.3 for brevity, the swapping of the first and last GEMM dimensions results in equivalent solutions (symmetrical design, see § 5.2.3.1). For instance, solution 2×2×82 \times2 \times8 (P1) can be swapped to 8×2×28 \times2 \times2 (P1), resulting in a native buffer size of 1536×1024×8321536 \times1024 \times832 (compute GEMM size becomes 192×512×416192 \times512 \times416 in this case).

U×V×WU \times V \times W (MaxEVA P.)LUTsBRAMsURAMsAIE coresPL Fq. (MHz)Thrpt. (TOPS)Power (W)En. Eff. (TOPS/W)
2×8×22 \times8 \times2 (P1)85K (9%)630 (65%)304 (66%)390 (98%)30077.0178.60.980
2×2×82 \times2 \times8 (P1)91K (10%)422 (44%)408 (88%)390 (98%)29076.9382.00.938
3×2×53 \times2 \times5 (P1)94K (10%)792 (82%)408 (88%)390 (98%)27876.7282.70.932
4×2×44 \times2 \times4 (P1)90K (10%)792 (82%)408 (88%)390 (98%)27876.7282.30.928
2×4×42 \times4 \times4 (P1)97K (11%)792 (82%)408 (88%)390 (98%)27876.7282.80.927
2×8×22 \times8 \times2 (P2)92K (10%)806 (83%)240 (52%)400 (100%)30076.0878.30.971
2×7×22 \times7 \times2 (P2)92K (10%)806 (83%)240 (52%)400 (100%)30076.0877.80.977
2×6×22 \times6 \times2 (P2)91K (10%)806 (83%)240 (52%)400 (100%)30076.0877.50.982
4×2×44 \times2 \times4 (P2)100K (11%)912 (94%)400 (86%)400 (100%)27575.4082.80.911
4×2×34 \times2 \times3 (P2)100K (11%)912 (94%)400 (86%)400 (100%)27575.4082.00.919

Table 3. Evaluation of 10 top-ranked GEMM designs on Versal VC1902.

Stratix DSE Evaluation

We conducted a DSE on 100 designs with different TB architecture parameters to optimize for performance and energy efficiency. The TB utilization of all explored designs is 85%–91%. The native buffer sizes (§ 5.2.5) were set based on our IP solver. The RTL (Verilog) code was automatically generated using our Python tool, and afterwards implemented on Intel Quartus 2021.1 (with a patch for Stratix 10 NX support from Intel). We use ModelSim Mod (2023) to calculate throughput and the Quartus Power Analyzer to estimate power Qua (2018).

Table 5.4 shows the evaluation of the top 10 designs. Configurations are a combination of the 4 TB architecture parameters (TBlen×Kp×Np×MpTB_{len} \times K_p \times N_p \times M_p). Designs are ranked based on their throughput when running GEMM on their native buffer sizes. Our solutions achieve high throughput, up to 68 TOPS, which is 47.6% of the theoretical peak of the NX 2100 device. We note that our throughput is directly related to our achieved frequencies, which are similar to prior work Stan et al. (2022);

Boutros et al. (2020); ); Langhammer et al. [10]. The maximum achieved energy efficiency is 1.347 TOPS/W. All designs present a small ALM (12–19%), but very high BRAM (85–94%) and TB (87–91%) utilization. Moreover, they present very high RAM efficiency (81.2–90%) and low BW requirements (79.3–92.6 GB/s).

Overall, we notice that the top designs have a TBlenTB_{len} of 18, 12 or 9. Designs with a TBlenTB_{len} of 36 achieve lower frequency and require a longer time for PnR, as they have less flexibility during placement. Note that design 9×16×5×5 has a lower throughput than 18×16×4×3, despite having higher TB utilization and operating at a higher frequency. This is attributed to a smaller TBlenTB_{len}, and therefore more wasted TBs operating in parallel load mode. Notice also how designs 18×16×4×3 and 18×8×8×3 perform better than designs 18×16×3×4 and 18×8×3×8, respectively. They achieve a higher frequency, due to a lower MpM_p, which leads to lower overall signal fan-out and FPGA routing congestion.

Insights & Discussion

Next, we discuss and compare various aspects of GEMM design and optimization on both Versal and Stratix devices.

1) FPGA Frequency: We examine the reliance of our solutions on the attainable FPGA frequency. While Versal’s AIE array operates at a fixed frequency (1.25 GHz), the PL (FPGA) frequency varies from 275-300 MHz across all top solutions (Table 5.3). Despite the PL frequency variation, we observe a stable performance (∼2%\sim2\%) for Versal. We further depict the impact of PL frequency on throughput in Figure 5.5a for design 2×2×82 \times2 \times8 (P1P1). We notice a negligible throughput decrease (<1.5%<1.5\%) for a wide frequency range (290 to 250 MHz). This is attributed to higher AIE computation time compared to communication with PL, illustrating the weak dependence of performance on a wide PL frequency range. However, for frequencies lower than 250 MHz performance degrades more severely (∼16%\sim16\% from 250 to 200 MHz).

Figure 5.5. Performance vs. PL frequency (a), and square matrix dimensions (b) of 2×2×82 \times2 \times8 (P1P1) on Versal VC1902.

Conversely, for Stratix, we observed a higher performance range (∼12%\sim12\%) across all solutions in Table 5.4. This is attributed to the direct relationship of performance to frequency, illustrating the strong dependence of the NX on PL frequency.

2) GEMM Scalability: Furthermore, we explore the scalability of our solutions when altering the matrix dimensions. For Versal, in Figure 5.5b, we present the throughput of the best overall design 2×2×82 \times2 \times8 (P1P1), for square matrices (powers-of-two) ranging from 512 to 32K. Zero-padding is applied to align to the compute GEMM size (Table 5.3). We notice that our design scales effectively, leading to almost its native peak throughput for dimensions ∼2\sim2K and higher. We note here that native peak refers to the achieved throughput of our designs when running GEMM on their native buffer sizes (Tables 5.3, Table 5.4).

Similarly, for Stratix, we show the scalability of the two bolded designs in Table 5.4. The first (Figure 5.6a) has a high DK′D_{K'}, and presents one of the highest throughput among all top solutions. The second (Figure 5.6b) has the lowest DK′D_{K'}. As DK′D_{K'} is much higher than DM′D_{M'} and DN′D_{N'} (Table 5.4), designs with flexible (small) DK′D_{K'} require less zero-padding along the reduction dimension (K′K') and scale better. We observe that, while the design in Figure 5.6a has higher native peak throughput, the design in Figure 5.6b scales better, due to lower DK′D_{K'}. Overall, GEMM scalability on both devices directly depends on the compute GEMM size. By properly adjusting this size, our methods can be exploited in straightforward fashion to target specific matrix dimensions e.g., long and narrow matrices. However, for the sake of generality, we demonstrate GEMM scalability on square matrices, without targeting specific sizes.

Figure 5.6. Performance vs. square matrix dimensions of designs 9×16×5×59 \times16 \times5 \times5 (a) and 9×8×10×59 \times8 \times10 \times5 (b) on Stratix 10 NX.

3) Achieved Performance: As mentioned, Versal achieves ∼60%\sim60\% of its theoretical peak throughput. This can be ascribed to multiple reasons. First, the nonideal (95%) efficiency Taka et al. (2023); Zhuang et al. (2023b) of the AIE MatMul kernels. Second, the necessity to introduce Add kernels on the AIE, which do not contribute to throughput, but occupy some of the cores (§ 5.2.3). Third, the inevitable stalls caused by memory conflicts on the AIE array AMD/Xilinx (2022c). In contrast, Stratix achieves ∼47%\sim47\% of its theoretical peak throughput. This is because the theoretical peak assumes 100% TB utilization with no wasted TBs in cascade mode, and a very high operating frequency of 600MHz Langhammer et al. [10], which are infeasible to attain in practice.

Our results in Tables 5.3, Table 5.4 indicate that, on average, Versal attains 19.8% higher throughput, while Stratix presents 41.6% higher energy efficiency on GEMM workloads. However, we note that the focus of this study strays from a competitive comparison among the two AI-optimized FPGA architectures. Instead, we emphasize the distinct design methods required for each device, due to their considerably different architecture styles (out-of-fabric vs. in-fabric). A broader set of full AI-workloads would be needed to explore complicated trade-offs between the two architectures, which we leave as future work.

4) Programmability Trade-Offs: The two technologies require completely different programming methods. In particular, Versal utilizes high-level C/C++ software for programming the AIE array. Efficient, high-level programming constructs can enable high performance of the AIE array’s SIMD processors on vectorizable workloads (with 95% AIE kernel efficiency). For productivity purposes, the PL part is typically programmed in HLS. However, HLS inefficiencies can lead to performance degradation, which can be mitigated by exploiting sophisticated methods (§ 5.2.4). Finally, AMD/Xilinx provides automation tools, such as Vitis V++, to reduce the complexity of integrating the entire system.

On the contrary, no high-level tools are available for programming Stratix 10 NX. In particular, Intel HLS Int (2023b) has no support for TBs, thus requiring exclusive coding in RTL for design, verification, and full-system integration. This resulted in ∼40×\sim40 \times higher number of lines of code in Stratix compared to Versal. Nonetheless, our Python-based generator significantly reduced this complexity. Moreover, because the performance of Stratix is directly analogous to frequency, unlike Versal, more programming effort is required to meet timing goals, and an extensive DSE is needed to identify high-throughput designs. In our case, this led to ∼5×\sim5 \times higher total tool compilation time for Stratix compared to Versal. While each individual design required 3–6 hours to compile on both devices, the extensive exploration on Stratix greatly increased the design space. Finally, based on our estimations, it might be 1.5–2×\times more productive to program Versal due to the increased programming effort required by Stratix. However, we note that design productivity is highly dependent on designer skills.

_5) GEMM Optimization Insights:_ Both devices exhibit design challenges due to their high complexity. To address complexity, systematic methodologies, e.g., analytical modeling, are essential for optimization of GEMM-based applications.

Versal AIE is a novel architecture that introduces several new design challenges. A comprehensive exploitation of AIE architectural attributes, e.g., local memory sharing and circuit switching, is crucial to optimize performance in GEMM Taka et al. (2023). Moreover, challenges, such as the reduced number of AIE-PL tiles and AIE routing congestion, require refined techniques to prevent performance degradation (§ 5.2.3). Finally, the limited DRAM BW of Versal devices introduces additional challenges in maintaining the high throughput of the AIEs.

In contrast, Stratix 10 NX preserves a traditional FPGA architecture, while introducing in-fabric TBs embedded in the PL. The inherent complexity of TBs necessitates the consideration of multiple operating modes to effectively map GEMM workloads (§ 5.2.5). Moreover, as throughput is directly related to frequency, performance depends on both low-level RTL optimizations and the intricacies of the FPGA architecture. RTL techniques, e.g., replication of control logic and insertion of pipeline stages, are essential in shortening critical paths.

Chapter Summary & Impact

In this chapter, we propose frameworks to optimize GEMM-based applications targeting the two leading AI-optimized FPGA architectures. We present a thorough evaluation of several aspects in GEMM design by efficiently leveraging the unique and substantially different architectural attributes of the AMD Versal ACAP and Altera/Intel Stratix 10 NX devices. Our experimental results show that our frameworks achieve up to 77 and 68 TOPS (int8) throughput, with up to 0.94 and 1.35 TOPS/W energy efficiency, for Versal and Stratix, respectively. This study provides fundamental insights regarding the architectural characteristics, programmability trade-offs, challenges, and limitations inherent in both Versal and Stratix AI-optimized FPGAs, motivating the hardware-level sparsity support explored in the next chapter.

Chapter 6: Systolic Sparse Tensor Slices: FPGA Building Blocks for Sparse and Dense AI Acceleration

In this chapter, we present the architectural details of our proposed SSTs Taka et al. (2025a), their integration into SST-based GEMM implementations, and the resulting inference speedups on sparse DL workloads compared to dense in-fabric FPGA acceleration. Moreover, we show that sparse accelerators exploiting SSTs offer substantial performance, scalability, and area advantages over traditional FPGAs.

Architecture & Design Overview

Fine-Grained Structured Sparsity

In this work, we leverage the regular patterns of fine-grained structured sparsity to architect SST slices with low area overhead. Figure 6.1a depicts a 50% unstructured sparse matrix, where the non-zero data are distributed randomly, i.e., there is no specific pattern of their locations. In contrast, in Figure 6.1b, the 2:42:4 structured sparsity pattern is illustrated, which has the same sparsity level (50%), but in every group of four consecutive elements there are two non-zero values. Notice that the location of the two non-zero values can vary significantly within the four-element group, offering fine-grained sparsity flexibility. These types of constraints enable low area hardware enhancements, allowing for efficient exploitation of sparsity.

50% unstructured sparse matrix and 2:4 structured sparse matrix with compressed format

Figure 6.1. 50% unstructured sparse matrix (a) and 2:4 (50%) structured sparse matrix along with compressed format (b).

A 2:42:4 sparse matrix can be efficiently stored in compressed format by saving only the non-zero values. The location of each non-zero data is encoded using 2-bit indices, as shown in Figure 6.1b. Similar to 2:42:4, for 1:41:4 (75%) sparsity one every four consecutive elements is non-zero, while for 1:31:3 (66.7%) sparsity there is one non-zero every three consecutive elements. For all aforementioned patterns, 2-bit indices are required to encode the location of each non-zero element. This is a very efficient compressed format, as we show in § 6.2.8, where comparison among other formats is performed.

Systolic Sparse Tensor Slices Architecture

In this section, we present an overview of the proposed SST slices. The core compute unit of the SSTs is a 4×44 \times4 systolic array (SA) Kung (1982), as depicted in Figure 6.2. A 2D SA consists of homogeneous processing elements (PEs), where each PE performs a multiply–accumulate (MAC) operation and forwards the input operands to the neighboring PEs. This architecture allows maximization of data reuse in GEMM, while also delivering high performance due to its regular and highly scalable design. To this end, SAs have become a prime architecture in many DNN accelerators Norrie et al. (2021); Jouppi et al. [7]; Liu et al. (2022a); Jeong et al. (2023); Wei et al. (2017); Ye et al. (2023); Samajdar et al. (2020). In this work, we show that incorporating SST slices in FPGAs leads to high performance and scalable dense and sparse GEMM accelerators.

Systolic Sparse Tensor slice architecture

Figure 6.2. Systolic Sparse Tensor slice architecture.

SST Operation

We enhance the systolic PEs with sparse features by introducing sparse processing elements (SPEs), while maintaining the properties of the SAs discussed above.

Our SSTs utilize an output stationary SA, consisting of 16 SPEs. The accumulations remain stationary in the SPEs, while input operands are propagated to their neighbors every clock cycle. The adataa_{\mathrm{data}} of an input matrix AA are propagated and reused across SPEs horizontally, while the bdatab_{\mathrm{data}} of an input matrix BB are propagated and reused vertically (Figure 6.2). The matrix AA can be either sparse or dense (typically to map weights), while BB is dense (typically to map input activations). The architecture of the SPEs is delineated in § 6.1.3.

Besides the 4×44 \times4 SPE grid, we also implement pipeline registers to delay the input operands for systolic data setup Jouppi et al. [1] (arranged in triangular manner in Figure 6.2). Systolic setup is needed at the interface for loading input matrices (typically from on-chip memory), when chaining multiple SSTs to construct larger SA grids (§ 6.1.4). Multiplexers are used to select either the systolic setup or directly the input data, and are configured statically (during bitstream loading).

The SSTs support both int8 and bf16 precisions, while accumulations are realized int32 and fp32, respectively, similar to Nvidia GPUs Mishra et al. (2021) and Google TPUs Norrie et al. (2021). When the SA operation is completed, the output matrix CC is extracted via the c_datac\_data output ports. We note that SPEs finish their operation in a diagonal fashion cycle after cycle. In the first cycle, SPE00SPE00 finishes, in the second cycle both SPE01SPE01 and SPE10SPE10 finish, etc. (Figure 6.2). However, to maintain regularity, thus simplifying the downstream logic (typically in CLBs), we extract the output values in a column-wise manner (four values per cycle). To achieve that, we introduce a six-element buffer (implemented as a register array), to store the data before getting extracted column-wise. This is particularly important in sustaining 100% SPE utilization (16 MACs per cycle) at the steady-state (matrices processed one after the other).

Since SPEs complete their operation diagonally, the six upper-triangular SPE outputs, shown in Figure 6.2, need to be stored in the buffer. Suppose a cycle TT where the outputs of the biggest diagonal, i.e., SPESPEs {03,12,21,30}\{03,12,21,30\} are generated. In cycle TT, the values of the first column, i.e., SPESPEs {00,10,20,30}\{00,10,20,30\} can be extracted, where SPESPEs {00,10,20}\{00,10,20\} are loaded from the buffer, while only SPE30SPE30 is directly extracted. In cycle T+1T+1, the values of SPESPEs {03,12,21}\{03,12,21\} replace the position of SPESPEs {00,10,20}\{00,10,20\} in the buffer (since they have been extracted). Therefore, the locations of the six-element buffer are being effectively reused, and in cycle T+1T+1, the second column can be extracted, i.e., SPESPEs {01,11,21,31}\{01,11,21,31\}. In a similar fashion, the rest two columns are extracted in cycles T+2T+2 and T+3T+3, respectively, while the buffer locations are efficiently reused due to replacement.

Global Routing Interface & Dedicated Wires

As illustrated in Figure 6.2, the a_dataa\_data and b_datab\_data input ports as well as c_datac\_data output ports are connected to the global FPGA routing resources. When chaining multiple SSTs, a_dataa\_data and b_datab\_data are forwarded to their next in the chain SSTs, horizontally and vertically, respectively. Horizontally, the data are propagated via the a_data_outa\_data\_out using the FPGA routing resources. However, vertically, we utilize dedicated wires to propagate the data (via the b_ded_outb\_ded\_out ports) and connect them to the next SST in the same FPGA column (via the b_ded_inb\_ded\_in ports). These vertical dedicated wires provide efficient connections without the usage of the global routing resources, matching the columnar nature of the modern FPGA fabric Boutros and Betz (2021); Boutros et al. (2024). This approach significantly reduces routing resources, as opposed to Arora et al. (2021b, 2022b), where all inputs/outputs (I/Os) of the in-fabric tensor slices are connected to global routing (see detailed comparison in § 6.2.3).

Besides the I/Os shown in Figure 6.2, the SSTs include also several dynamic control signals. In particular, an input enable signal is used to control the operation of the SSTs. This signal should be deasserted when the operation of the SST needs to be stalled. Moreover, an accumulate signal is utilized to control the accumulation in the SPEs. This signal remains asserted when accumulation is desirable in the SPEs, e.g., during tiling to process larger matrices. However, for every new matrix operation the accumulate signal should be deasserted for one cycle. Another input signal is the d_typed\_type, which dynamically selects the precision, i.e., int8 or bf16. Finally, a 2-bit sparsity_level signal is employed to dynamically select the sparsity level of the matrix AA, i.e., dense, 2:4, 1:3, 1:4. This signal is utilized internally in the SSTs to control the operation of the supported sparsity modes, as discussed in the next section.

Regarding the outputs of the SST, an accumulate_out signal is used for systolic distribution of the accumulate signal when chaining multiple SSTs. This systolic distribution makes the control logic significantly simpler, since the accumulate is set only for the first SST in the 2D array layout, while being distributed to the remaining SSTs, as explained in § 6.1.4. Another output signal is the valid_out, which is asserted when the output c_datac\_data are extracted column-wise (Figure 6.2). This signal simplifies the downstream logic, since only valid_out\mathit{valid\_out} needs to be checked for valid output data.

Sparse Processing Element

The SPE architecture in all supported sparsity modes is illustrated in Figure 6.3.

Figure 6.3. Sparsity modes in systolic sparse elements of the SST slices (multiplexing logic omitted for clarity).

When the SPE is configured in dense mode (Figure 6.3a), it operates as a regular dense PE. In particular, one MAC operation is performed every cycle, while the elements of matrices AA and BB are forwarded to their neighboring SPEs (via pipeline registers) horizontally and vertically, respectively. The output value of matrix CC in each SPE is calculated after KK cycles, when considering the M×K×NM \times K \times N matrix dimensions depicted in Figure 6.3a.

In 2:4 mode (50% sparsity), matrix AA is stored in a compressed format of M×K/2M \times K/2 size for both values and the 2-bit indices, as shown in Figure 6.3b. In order to achieve speedup over the dense case, we increase the number of ports of each SPE in the vertical dimension, to load four elements of the matrix BB in parallel. Furthermore, the 2-bit index is also loaded in the SPE to select the corresponding BB values (via a 4:1 multiplexer), which need to get multiplied with each AA value. However, in 2:4 sparsity, for every four values of the matrix BB, two MAC operations need to be performed. Hence, the four BB values remain in the SPE registers for two clock cycles (the four green registers in Figure 6.3b are loaded every two cycles). To correctly synchronize the systolic operation, two pipeline stages are required for both the AA values and their indices (blue and yellow registers in Figure 6.3b), which are loaded every cycle. In this manner, one MAC operation is performed every cycle, ensuring 100% utilization. Moreover, the output value of matrix CC in each SPE is calculated in K/2K/2 cycles, achieving 2×2 \times acceleration over dense operation.

Similar to 2:4, for 1:4 (75%) sparsity, four BB values are loaded in the SPE. However, in this case the BB values are loaded every cycle, while only one pipeline stage is required for the AA value and its index, as shown in Figure 6.3c. The output

value needs K/4K/4 cycles to be calculated, offering 4× speedup over the dense case. To fill the sparsity gap between 50% and 75%, we leverage the same hardware enhancements for 2:4 and 1:4 sparsity to also support the 1:3 (66.7%) pattern. In particular, as illustrated in Figure 6.3d, the 1:3 operation is similar to 1:4, with the main difference being that three BB values are loaded every cycle instead of four. The indices ensure that the fourth BB value is never selected, thus no additional hardware is required. In this case, K/3K/3 cycles are needed for every SPE output value, providing 3× acceleration. Finally, we note that similar to 1:3, 2:3 sparsity can also be supported by directly utilizing the 2:4 mode, leading to 33.3% sparsity. However, a matrix is typically considered sparse when it has sparsity of 50% or higher AMD (2024b). Hence, in this work, we do not consider the 2:3 pattern.

In Table 6.1, we present a summary of the supported sparsity levels. First, we show the compression ratio, i.e., the memory reduction over dense storage due to compressed format, for both int8 and bf16 precisions. For all sparsity levels, bf16 offers a higher compression ratio over int8, e.g., 1.78× vs. 1.6× for 2:4 sparsity. This is because 2-bit indices are required for both 8-bit and 16-bit data types, resulting in relatively lower overhead for the compressed representation in bf16 compared to int8. This compressed format substantially reduces both on-chip and off-chip memory requirements, achieving up to 3.56× reduction (Table 6.1).

Sparsity levelCompres. ratio int8Compres. ratio bf16Speedup over denseSPE util.
Dense (0%)1×1×1×100%
2:4 (50%)1.6×1.78×2×100%
1:3 (66.7%)2.4×2.67×3×100%
1:4 (75%)3.2×3.56×4×100%

Table 6.1. Summary of supported sparsity levels in SST slices.

Second, we notice that every sparsity level is translated to its corresponding speedup, e.g., 4× for 1:4 (75%) sparsity, achieving 100% SPE utilization in all cases (Table 6.1). For all sparsity levels, data reuse is maximized since both indices and values are propagated and reused horizontally and vertically, similar to dense operation.

GEMM Design Utilizing Multiple SST Slices

In this section, we describe parametric GEMM implementations for both dense and sparse configurations, utilizing multiple SST slices. Both implementations are highly regular and scale effectively on the FPGA fabric, attaining high frequencies as shown in § 6.2.4.

Dense Implementation

Figure 6.4a depicts a parametric GEMM accelerator comprising a 2D array of SST slices, which are configured in dense mode (§ 6.1.3). The 2D array consists of Y⋅XY \cdot X SST slices, implementing a total SA size of (Y⋅4)×(X⋅4)(Y \cdot4) \times(X \cdot4). This size is denoted as the native size of the GEMM accelerator. On-chip memory buffers are implemented to store the input matrices AA, BB and the output matrix CC. The input buffers AA, BB are located in the left and top edges of the 2D array, respectively, and are partitioned into banks, providing sufficient bandwidth to feed the SSTs. In particular, for buffer AA, YY banks are required, while for buffer BB, XX banks are needed. Since each SST slice includes a 4×44 \times4 SA, each bank needs to provide a bandwidth of 32-bits per cycle when SSTs are configured for int8 precision, while for bf16, 64-bits per cycle are required. Regarding the output buffer CC, X⋅YX \cdot Y banks are needed due to the output stationary architecture of the SSTs, each receiving an output of 128-bits per cycle (four 32-bit values as explained in § 6.1.2).

Figure 6.4. 2D systolic GEMM design: dense implementation (a) and dynamic configuration of all supported sparsity modes (b).

The data from the buffers AA, BB are propagated in a systolic fashion between the SST slices, as illustrated in Figure 6.4a. Horizontally, the AA data are propagated via the global routing resources of the FPGA fabric. Vertically, the BB data are inserted via global routing wires in the first SST at each vertical chain (YY SSTs in Figure 6.4a), while being forwarded to the next SSTs via dedicated wires. It is important to note here

that the SSTs comprising each vertical chain are physically contiguous in the FPGA. Nevertheless, in the horizontal dimension, the YY chains might not follow the logical arrangement shown in Figure 6.4a inside the FPGA fabric, due to the routing flexibility of FPGAs. This depends on decisions made by the FPGA place and route (PnR) algorithm. Finally, notice the (static) systolic data setup configuration specifically for the SST slices that interface with the buffers AA, BB (left and top edge of the 2D array).

Control logic, mapped to the CLB resources of the FPGA, is utilized to orchestrate the entire operation of the GEMM design. We also implement tiling logic (in CLBs) to exploit data reuse in GEMM, as well as to support arbitrary GEMM sizes based on the available on-chip memory resources. When mapping an arbitrary GEMM of M′×K′×N′M' \times K' \times N' dimensions, M′M' must be a multiple of (Y⋅4)(Y \cdot4), while N′N' must be a multiple of (X⋅4)(X \cdot4), since the native size of the accelerator is (Y⋅4)×(X⋅4)(Y \cdot4) \times(X \cdot4). Note that there is no constraint on the reduction K′K' dimension. Finally, although not shown in Figure 6.4a, accumulate signals utilized during tiling (§ 6.1.2) are propagated in a systolic fashion among the 2D array of SSTs, similar to the AA, BB data. The control logic sets the accumulate signal only for the first SST in both vertical and horizontal dimensions (i.e., the SST fed by buffers A1A_1 and B1B_1), which significantly simplifies the overall logic.

Dynamic Sparse Configuration

Figure 6.4b illustrates a parametric GEMM design that is dynamically configured to support all the sparsity modes in the SSTs, i.e., dense, 2:4, 1:3 and 1:4. This dynamic configuration is particularly important for layer-wise sparsity exploitation in DNNs, since each layer might require different sparsity level for optimal trade-off between DNN accuracy and speedup (§ 6.2.7). We note that the implementation is similar to the dense design (Figure 6.4a), with main differences lying in the design of buffers AA and BB, as well as in the vertical and horizontal propagation of the data. Horizontally, the AA data are kept in the buffer banks in compressed format for the sparse modes (2:4, 1:3 and 1:4). In this case, both non-zero data and indices are loaded in the SSTs and are propagated horizontally (see § 6.1.3). More specifically, for int8, each buffer AA bank needs to provide a bandwidth of 40-bits per cycle, due to the additional 8-bits indices for the four vertical SPEs at the interface of each SST. Similarly, for bf16, 72-bits per cycle are required.

Vertically, four BB banks are needed to feed the SSTs due to the 4×4\times increase in ports for 2:4 and 1:4 sparsity acceleration (§ 6.1.3). Each BB bank provides the same bandwidth as the dense design (Figure 6.4a), i.e., 32-bits and 64-bits per cycle for int8 and bf16, respectively. Similar to the dense design, the BB data are inserted in the first SST at each vertical chain and dedicated wires are used to propagate them vertically (Figure 6.4b). These dedicated wires are particularly important for the sparse design, since otherwise 4×4\times more vertical wires would be required to use global routing compared to the dense design (in the case where only non-dedicated wires are employed).

Besides sparse operation, the design in Figure 6.4b supports also dense computation. This is because dense computation might still be needed, even if all weights in DNNs are sparse. For instance, in Transformer-based DNNs Vaswani et al. (2017); Devlin et al. (2019); Dosovitskiy et al. (2020), the QKV (Query, Key, Value) self-attention GEMM computations do not involve weights, and are typically computed as dense. For dense operation, only one BB bank is sufficient at each vertical chain, e.g., Bx,0B_{x,0} (Figure 6.4b). However, multiplexing logic can be employed to utilize the remaining Bx,1B_{x,1}, Bx,2B_{x,2}, Bx,3B_{x,3} banks. This is particularly important to ensure efficient utilization of on-chip memory resources, which leads to maximized data reuse and thus optimized energy efficiency Taka et al. (2024, 2023). Finally, for 1:3 sparsity, only banks Bx,0B_{x,0}, Bx,1B_{x,1}, Bx,2B_{x,2} are required, leaving bank Bx,3B_{x,3} unused. However, similar to the dense operation, additional multiplexing logic can be employed to reuse this bank when it comprises multiple BRAMs, which we do not explore in this work (see § 6.2.4 for implementation details).

The dynamic configuration among all sparsity levels is implemented in the control logic (using CLBs). However, FPGA accelerators can be designed in a custom fashion depending on the sparsity of each DNN. For instance, a specific DNN might require only 1:3 sparsity and dense computation across all of its layers. The control and tiling logic for sparsity is similar to the dense design (Figure 6.4a), showcasing a marginal increase in CLB resources (see § 6.2.4).

Evaluation

FPGA Architecture

The traditional FPGA architecture comprises CLBs, BRAMs, DSP slices and routing resources, i.e., connection and switch boxes (CBs and SBs). We enrich the FPGA architecture with our proposed in-fabric SST slices. The SST columns repeat every 15 FPGAs columns, occupying only 14% of the total FPGA area, which is sufficient for DNN applications as also found in Arora et al. (2022b) (refer to Figure 6.8 for the layout of the proposed FPGA architecture). We exploit the automated transistor sizing tool, COFFE Boutros et al. (2023); Yazdanshenas and Betz (2019), to model the delays and areas of the FPGA components. These delays and areas are subsequently used to describe the FPGA architecture in the Verilog-to-Routing (VTR) tool flow Murray et al. (2020), facilitating FPGA architecture exploration. We utilize the 7nm FinFET ASAP7 predictive process design kit (PDK) Clark et al. (2016), for the SPICE simulations conducted by COFFE. Specifically, we exploit the typical corner (TTTT) ASAP7 model Clark et al. (2016); Vashishtha et al. (2017). Finally, COFFE was run with cost function of area×delayarea \times delay for four transistor sizing iterations (2–4 iterations are typically sufficient Chiasson and Betz (2013)).

SST-based GEMM implemented in VTR

Figure 6.8. SST-based GEMM implemented in VTR (SA size: 12x12). Blue: CLB, Orange: BRAM, Yellow: DSP, Purple: SST.

We model a modern Agilex-like FPGA similar to Arora et al. (2021b, 2022b). Table 6.2 shows the FPGA logic (CLB) and routing (CBs and SBs) architectural parameters used in COFFE. Each CLB contains 10 fracturable logic elements (FLEs), which consist of 6-input lookup tables (LUTs), registers and two adder bits. Since

ParameterValueParameterValue
FLEs per CLB (NN)10Frac. LUT size (KK)6
FLE independent inputs2Adders per FLE2
Channel width (WW)300Wire length (LL)4/16
CLB inputs (II)60CLB outputs (OO)40
FLE outputs to routing (OrO_r)4Feedback FLE outputs (OfbO_{fb})2
SB flexibility (FsF_s)3Input connection flex. (FcinF_{cin})0.15
Output connection flex. (FcoutF_{cout})0.1Input crossbar flex. (FclocalF_{clocal})0.5

Table 6.2. FPGA logic tile and routing parameters in COFFE.

COFFE does not model 7nm optimized memory circuits, we obtain the model of the 20Kbit BRAMs used in the recent work Boutros et al. (2023) for 7nm FPGAs. The delay and area values of these BRAMs are conservatively estimated using the 14nm Stratix 10 architecture values, as described in Boutros et al. (2023). Finally, we leverage the DSP used in Arora et al. (2022b), which supports multiple precisions and modes, closely matching the commercial Agilex DSP Intel (2020). However, since this DSP is modeled in 22nm, we scale down its area and delay values to 7nm, exploiting the scaling factors from Stillmaker and Baas (2017).

SST Slices Implementation

We implement the SST slices utilizing the ASAP7 7.5-track standard cell library Vashishtha et al. (2017); ASA (2023). Specifically, we use the regular threshold voltage (RVT) cells of the typical (TTTT) corner of the library. COFFE’s hybrid flow was exploited, where the core of the SSTs is implemented using the standard cell flow, while the interface to the programmable routing (local crossbar, drivers for dedicated wires, etc.) is implemented in the full custom flow using SPICE simulations. The standard cell flow uses Synopsys Design Compiler for synthesis, Cadence Innovus for PnR and Synopsys Primetime for timing analysis. The full custom flow uses the TTTT ASAP7 SPICE model and COFFE was run with the same configurations as in the previous section.

To accurately quantify the area overhead of the sparsity enhancements described in § 6.1.3, we remove the sparsity features of the SST slices, to implement a 2D systolic dense tensor slice. We retain all the design features delineated in § 6.1.2, except the dedicated wires. For this dense tensor slice, we utilize the approach proposed in Arora et al. (2021b, 2022b), where all I/O ports are connected to global routing. We refer to this tensor slice as SDT_GIO, which is similar to Arora et al. (2021b, 2022b). The dense SDT_GIO slices are used as the baseline comparison with our proposed sparse SSTs, while also allowing for quantification of the routing savings due to dedicated wires in SSTs (§ 6.2.3).

Table 6.3 shows the area and frequency results for both the SST and SDT_GIO slices obtained from COFFE. At the standard-cell core level, we observe a low area overhead of 22.9% for the sparsity enhancements of the SST slices. For both SST and SDT_GIO slices, we implement a 50% populated input crossbar to enhance their routability inside the FPGA fabric Yazdanshenas and Betz (2019). Each SST slice has 333 global routing inputs and 202 outputs, along with 256 dedicated inputs/outputs. In contrast, the SDT_GIO slice has 259 global routing inputs and 258 outputs, without dedicated I/Os. Given the aforementioned global routing I/Os, the SST slices need to access 6 SBs, thus spanning 6 CLB (logic) tiles. However, since more outputs are needed for the SDT_GIOs (primarily due to absence of dedicated wires), they need to span 7 CLB tiles. Therefore, when calculating the total tile area for both slices (standard-cell core and routing interface), we obtain a low area increase of 19.6% for SSTs vs. SDT_GIOs (Table 6.3). Finally, we observe that both slices can operate at high frequencies for the supported precisions (>838> 838 MHz in all cases), similar to commercial 7nm FPGA hard blocks AMD (2024d); Intel (2024b).

In-fabric sliceSSTSDT_GIO
Standard-cell core4530.8 (+22.9%)3687.8
Input crossbar1511.4985.6
Dedicated output routing40.20
Switch Box (SB)1374.31603.4
Connection Box (CB)734.9571.6
Total tile8191.6 (+19.6%)6848.4
Freq. (MHz)int8
928.6935.3
bf16
838.1850.5

Table 6.3. Area and frequency of SST vs. SDT_GIO slices (post PnR).

Wirelength Gains Utilizing Dedicated Wires

To quantify the benefits of the proposed dedicated wires in SSTs, we implement dense GEMMs utilizing our SSTs slices (Figure 6.4a) and compare them with dense GEMMs using SDT_GIOs slices, which support only global routing. To this end, similar to the enhanced FPGA architecture with SST slices, we create an architecture that has SDT_GIOs instead of SSTs. To facilitate the exploration, we implement a parametric Python script that generates the Verilog code for arbitrary GEMM sizes. The Verilog designs are synthesized and implemented on the two aforementioned FPGA architectures (one with SST and one with SDT_GIO slices), exploiting the VTR flow. In all cases, we perform a 10 seed-sweep in VTR, and report the design that attains the maximum FPGA frequency.

Figure 6.5 illustrates the total routing wirelength of the two dense GEMM accelerators, for various SA sizes at both int8 and bf16 precisions. These designs use 512-element deep banks for buffers AA, BB and CC (§ 6.1.4), exploiting the 512×40512 \times40-bit BRAM configuration mode (also found in recent Intel FPGAs Intel (2024c,d, 2023)). For the SA sizes shown in Figure 6.5, we observe a significant wirelength reduction due to dedicated wires in the SSTs, ranging from 15.5–29.9% for int8 precision, compared to the SDT_GIO-based GEMM designs. Similarly, for bf16, the wirelength reduction ranges from 23.9–31.2%, showcasing the importance of vertical dedicated wires for 2D GEMM implementations on FPGA architectures.

Figure 6.5. Routing wirelength comparison of dense GEMM mapped to SST and SDT_GIO slices, for various SA sizes.

Regarding the maximum attainable FPGA frequency, both dense GEMM implementations achieve high frequencies ranging from 668–731 MHz. In particular, the smallest designs, i.e., 20×2020 \times20 SA size, attain the highest frequency of ∼731 MHz for both SSTs and SDT_GIOs at int8 and bf16 precisions. Similarly, the largest designs, i.e., 40×4040 \times40 SA size, both attain ∼668 MHz. We notice that dense GEMMs mapped to SST and SDT_GIO slices attain almost similar FPGA frequencies (<1% difference for all tested SA sizes depicted in Figure 6.5). This can be explained by the fact that the critical path is typically in the control logic implemented in CLBs, which is the same for both dense GEMMs mapped to SSTs and SDT_GIOs.

Sparse SST-based GEMM Implementation

In this section, we present the sparse GEMM implementation shown in Figure 6.4b, which allows dynamic configuration for all supported sparsity modes in SSTs, i.e., dense, 2:4, 1:3 and 1:4. We use two baselines for our comparison. First, we compare with a GEMM design mapped to SDT_GIOs using a FPGA architecture that replaces SSTs with SDT_GIOs. Since SDT_GIOs have a 2D dense systolic dataflow that does not support sparse computation (§ 6.2.2), we retain the zeros of the sparse matrices, therefore operating similarly to dense. Hence, this implementation is similar to that shown in the previous section (§ 6.2.3), but it has the same amount of on-chip memory as the sparse SST-based design. In particular, we implement four memory banks in the vertical dimension to feed each SDT_GIO (similar to Figure 6.4b). This allows for the same data reuse opportunities as in the SST-based design, thus enabling a fair comparison. Second, we implement the same sparse SST-based design, mapped to CLBs and DSPs of a traditional FPGA architecture, i.e., without SSTs and SDT_GIOs. Finally, we note that all tested designs utilize 512-element deep banks and are based on a 10 seed-sweep maximization of FPGA frequency, similar to § 6.2.3.

FPGA Frequency

Figure 6.6 depicts the FPGA frequency of the sparse GEMMs mapped to SST slices, SDT_GIO slices as well as CLBs and DSPs. First, we observe the high FPGA frequency of GEMMs mapped to SST and SDT_GIO slices, ranging from 596–625 MHz for int8 and from 578–610 MHz for bf16. We note that the slightly lower frequencies of the SDT_GIO-based GEMMs compared to the previous section are attributed to the higher BRAM usage. For all tested SA sizes, we notice a negligible difference in FPGA frequency (<1%<1\%) between the SST-based and the SDT_GIO-based GEMMs. However, when comparing the SST-based vs. the CLB+DSP GEMMs, we observe considerably higher frequencies, ranging from 2.8–4.4×\times and from 2.7–5×\times for int8 and bf16, respectively.

Figure 6.6. FPGA frequency comparison of sparse GEMM using SSTs vs. SDT_GIOs vs. CLBs+DSPs, for various SA sizes.

Second, note that the sparse GEMMs utilizing SSTs scale effectively when increasing the SA size. In particular, we observe a minor decrease in FPGA frequency (4.6% and 5.2% for int8 and bf16, respectively), when considering the 40×4040 \times40 vs. the 20×2020 \times20 SA size. In contrast, the sparse GEMM mapped to CLBs and DSPs, does not scale as effectively, where we notice a substantial frequency drop of 36.6% and 49.6% for int8 and bf16, respectively. These results emphasize the significant performance advantages and scalability of in-fabric sparse hard blocks compared to traditional FPGAs.

GEMM Area

We calculate the total area of our GEMM designs as the summation of the utilized logic area and the FPGA routing area. While the utilized logic area is reported in VTR, the tool does not report the used routing area. Thus, we estimate the routing area by summing the area of all utilized multiplexers in both the SB and CB resources of our designs. Figure 6.7 shows the total area of our sparse GEMM designs in minimum width transistor area (MWTA) units Murray et al. (2020). As shown in Table 6.3, we observed a 19.6% area overhead of the SSTs vs. the SDT_GIOs at the tile level. However, here we calculate this area overhead at the GEMM implementation level, as this provides more accurate estimations for DNN accelerators. When comparing the sparse GEMM mapped to SSTs vs. the GEMM using SDT_GIOs, we observe a small area increase ranging from 10.2–15.9% for int8, and from 13.3–19.4% for bf16, across all SA sizes. We note that the lowest area increase, i.e., 10.2% and 13.3% for int8 and bf16, respectively, occurs for the highest, 40×4040 \times40 SA size. This is mainly attributed to the dedicated wires in SSTs, where the increase of the SA size (thus the total slices), leads to more routing area in the GEMM design using SDT_GIOs.

Figure 6.7. Total area comparison of various sparse GEMM implementations utilizing SSTs vs. SDT_GIOs vs. CLBs+DSPs.

As shown in Figure 6.7, we notice a substantial area increase of the sparse GEMM mapped to CLBs and DSPs vs. using SSTs, of up to 7×7 \times for int8, and 10.9×10.9 \times for bf16. Moreover, we observe that this area difference increases with the SA size. In particular, for 20×2020 \times20 SA size, we obtain a difference of 5.5×5.5 \times and 8.3×8.3 \times for int8 and bf16, respectively. However, for 40×4040 \times40 size, the difference increases to 7×7\times and 10.9×10.9\times for int8 and bf16, respectively. Hence, our results exhibit the significant area efficiency and scalability of the proposed sparse GEMM designs using in-fabric SSTs, over traditional FPGAs.

Resource Usage & Efficiency of Sparse SST-based GEMM

Table 6.4 presents the resource usage and several evaluation metrics of the sparse GEMM designs, for SA size of 40×4040 \times40. First, we notice a small CLB increase for the SST vs. the SDT_GIO GEMM, of 26.2% and 29.1% for int8 and bf16, respectively. This is ascribed to the additional control logic of including all supported sparsity levels, compared to only control logic for dense (since for SDT_GIOs the zeros of the sparse matrices are retained, effectively operating similar to dense). Second, note that the BRAM usage is the same in all GEMM designs, despite the additional bandwidth requirement due to the indices of the compressed sparse format (§ 6.1.4). This is due to the use of the 512×40512 \times40-bit BRAM mode (§ 6.2.3). For example, for int8, buffer AA banks in the dense design of Figure 6.4a, need a bitwidth of 4⋅8=324 \cdot8 = 32-bits, which is smaller than the 40-bits width of the BRAMs. However, for the sparse operation of Figure 6.4b, they need additionally 8-bits for the indices (40-bits total), which is exactly the same as the BRAMs bitwidth. Similar conclusions can be drawn for bf16. Hence, no additional BRAMs are needed for the higher bandwidth requirements of sparse matrices in compressed format.

Pr.GEMM Cfg.CLBsBRAMsDSP slicesSST slicesSDT_GIO slicesFreq. (MHz)Area (GMWTA)Wirelength (CLB seg. units)Eff. Thrpt. (TOPS)Area Eff. (TOPS/GMWTA)
int8SST6654500100–6010.145493K7.6953.03
int8SDT_GIO5274500–1005960.132578K1.9114.47
int8CLB+DSP22650450400––1381.0227373K1.771.732
bf16SST7195000100–5780.162684K7.4045.68
bf16SDT_GIO5575000–1005800.143725K1.8613.01
bf16CLB+DSP270655002400––1161.76415212K1.480.839

Table 6.4. Evaluation and comparison of sparse GEMM designs using SSTs vs. SDT_GIOs vs. CLBs+DSPs (SA size: 40×4040 \times40).

Table 6.4 shows the effective throughput and area efficiency of our GEMM designs. The effective throughput (TOPS) is calculated on the GEMM native size (40×4040 \times40 in this case, see § 6.1.4), and similar to Mishra et al. (2021); NVIDIA [3]; AMD [5] when acceleration can be attained from sparsity. In particular, since 1:4 sparsity is included in our SST-based and CLB+DSP designs, their effective throughput is increased by a factor of four. When comparing the SST-based GEMM vs. the SDT_GIO-based design, we observe a throughput gain of 4.03×4.03 \times and 3.98×3.98 \times for int8 and bf16, respectively. Moreover, we observe 3.66×3.66 \times (int8) and 3.51×3.51 \times (bf16) higher area efficiency (TOPS/GMWTA, area in Giga MWTAs). In addition, when comparing with the GEMM using CLBs and DSPs, we obtain 4.34×4.34 \times (int8) and 5×5 \times (bf16) higher throughput for the SST-based GEMM. Regarding the area efficiency, an immense 30.62×30.62 \times (int8) and 54.45×54.45 \times (bf16) difference is observed, primarily due to the increased usage of the traditional FPGA resources. To this end, notice the very high CLB and wirelength usage for the CLB+DSP vs. the SST-based GEMM. For instance, for int8, there is a 34.1×34.1 \times higher CLB and a 15×15 \times higher wirelength usage.

Finally, in Figure 6.8 we show a screenshot of a sparse SST-based design implemented in VTR. Notice the physically contiguous vertical SST chains due to dedicated wires. Also, observe the 2D physical layout after VTR PnR, compared to the logical

2D array of Figure 6.4b.

DNN Speedup Estimation Using SST Slices

We provide a speedup estimation for actual DNNs when leveraging our proposed SST slices. In particular, we utilize the DeiT Touvron et al. (2021) and ConvNeXt Liu et al. (2022b) models, which attain state-of-the-art accuracy for vision tasks. We apply two types of sparsity to determine the optimal sparsity configuration for each aforementioned models. First, we utilize uniform sparsity, i.e., all layers have the same sparsity, by pruning weights based on their magnitudes Mishra et al. (2021). Second, we exploit layer-wise sparsity by adopting the neural architecture search (NAS) methodology proposed in Huang et al. (2024) to identify the optimal sparsity level of each layer, aiming to minimize accuracy degradation. Accuracy is evaluated on the ImageNet-1K dataset Deng et al. (2009), with int8 quantization applied using the post-training method proposed in Yuan et al. (2022).

For speedup estimation, we develop an analytical model based on our results in § 6.2.4, comparing the sparse SST-based GEMM to the SDT GIO-based GEMM, with the latter operating on dense matrices. Our model exploits the matrix sizes for each layer of the DeiT and ConvNeXt models, and applies zero padding to align with the native size of the GEMM accelerator (§ 6.1.4). We utilize our largest implemented GEMM design, i.e., 40x40 SA size, and assume 100 GB/s bandwidth for DRAM modeling, which is typical in modern FPGAs Achronix (2024b); AMD (2024e). We note here that utilizing GEMM computations provides a reliable model for estimating speedup, since GEMM is the core computation in contemporary DNNs, accounting for more than 90% of the total execution time Adolf et al. (2016b); Wang et al. (2021a). Other operations e.g., softmax, layernorm, can be effectively overlapped by DNN accelerators. Moreover, GEMM-based estimation (convolutions implemented as GEMM for ConvNeXt) is also performed in multiple works targeting speedup calculations due to sparsity exploitation Wu et al. (2023); Jeong et al. (2023); Zhu et al. (2019); Liu et al. (2020a); Tu et al. (2023).

Table 6.5 presents the speedup estimation and accuracy for our two ViT models, i.e., DeiT-S and DeiT-B, and the ConvNeXt-S model. When applying uniform 2:4 sparsity, we observe a negligible accuracy decrease in DeiT-B for bf16 (0.07%), or even slightly higher accuracy for int8 (by +0.05%). However, we notice a larger decrease for DeiT-S, e.g., 0.48% for int8. This is due to the fact that smaller models (DeiT-S with 4.7 GOPS) exhibit less redundancy compared to larger models (DeiT-B with 17.6 GOPS), making them more resilient to sparsity. Note that even with uniform 2:4 sparsity across all layers, dense computation is still needed (Table 6.5), due to QKV computation in ViTs (§ 6.1.4). In addition, for ConvNeXt-S, we retain its depth-wise convolution layers as dense, since we observed a severe accuracy degradation when attempting to sparsify them. For instance, when applying 2:4 sparsity to all other layers except the (36) depth-wise layers in ConvNeXt-S, we obtain an accuracy degradation of only 0.47%. However, uniform 2:4 sparsity can result in significant speedup, e.g., 1.93×1.93 \times in ConvNeXt-S, compared to dense operation. We note that both int8 and bf16 cases deliver nearly indistinguishable speedups. over dense computation, since the frequency difference between the SST-based and SDT-GIO-based GEMM accelerators is less than 1% (§ 6.2.4).

DNN modelSparsity levelsNumber of layersIn-fabric sliceSpeedupTop-1 accuracy (%)Weight red.
Dense2:41:31:4int8bf16int8bf16
DeiT-S (4.7 GOPS)dense48–––SDT-GIO1×1\times79.56 (-0.00%)79.79 (-0.00%)1×1\times1×1\times
[dense, 2:4]–48––SST1.88×1.88\times79.08 (-0.48%)79.20 (-0.59%)1.58×1.58\times1.76×1.76\times
[dense, 2:4, 1:3, 1:4]326163SST2.14×2.14\times77.95 (-1.61%)78.33 (-1.46%)1.81×1.81\times2.02×2.02\times
[dense, 1:3]––48–SST2.61×2.61\times73.36 (-6.20%)73.95 (-5.84%)2.32×2.32\times2.61×2.61\times
DeiT-B (17.6 GOPS)dense48–––SDT-GIO1×1\times81.40 (-0.00%)81.80 (-0.00%)1×1\times1×1\times
[dense, 2:4]–48––SST1.91×1.91\times81.45 (+0.05%)81.73 (-0.07%)1.59×1.59\times1.77×1.77\times
[dense, 2:4, 1:3, 1:4]6151314SST2.35×2.35\times81.28 (-0.12%)81.60 (-0.20%)2.05×2.05\times2.26×2.26\times
[dense, 1:3]––48–SST2.75×2.75\times81.10 (-0.30%)81.25 (-0.55%)2.36×2.36\times2.64×2.64\times
[dense, 1:4]–––48SST3.52×3.52\times80.37 (-1.03%)80.69 (-1.11%)3.12×3.12\times3.50×3.50\times
ConvNeXt-S (8.7 GOPS)dense108–––SDT-GIO1×1\times82.99 (-0.00%)83.09 (-0.00%)1×1\times1×1\times
[dense, 2:4]3672––SST1.93×1.93\times82.52 (-0.47%)82.62 (-0.47%)1.46×1.46\times1.66×1.66\times
[dense, 2:4, 1:3, 1:4]365976SST2.07×2.07\times82.30 (-0.69%)82.39 (-0.70%)1.52×1.52\times1.74×1.74\times
[dense, 1:3]36–72–SST2.81×2.81\times81.44 (-1.55%)81.53 (-1.56%)1.95×1.95\times2.32×2.32\times
[dense, 1:4]36––72SST3.63×3.63\times81.05 (-1.94%)81.16 (-1.93%)2.34×2.34\times2.89×2.89\times

Table 6.5. Speedup estimation, accuracy and weight memory reduction of various DNNs when mapped to SSTs vs. SDT-GIOs.

When applying layer-wise sparsity (all supported levels), we observe higher speedup in all cases, e.g., 2.35×2.35\times in DeiT-B. In this case, for DeiT-B, accuracy degradation is still negligible, e.g., 0.12%. However, we observe higher degradation for DeiT-S (1.61% for int8) and ConvNeXt-S (0.69% for int8), since they are both small models. When applying higher sparsity (uniform 1:3 and 1:4), accuracy degrades even further, but higher speedup is attained, showcasing the accuracy-speedup trade-off. If we constraint the accuracy degradation to be ∼\sim1% (acceptable in the vast majority of applications Sun et al. (2020)), the optimal speedup is attained for different sparsity configurations in DNNs (bolded in Table 6.5). Specifically, for DeiT-S, uniform 2:4 is optimal (1.88×1.88\times speedup), while for DeiT-B, uniform 1:4 shows best results (3.52×3.52\times speedup). For ConvNeXt-S, layer-wise sparsity (all supported levels) exhibits optimal results (2.07×2.07\times speedup). However, when allowing higher accuracy degradation (for instance, within 2%) higher speedup can be attained, e.g., 3.63×3.63\times for uniform 1:4 in ConvNeXt-S. Finally, notice the higher weight memory reduction as sparsity increases (Table 6.5). Our results demonstrate the importance of supporting multiple sparsity levels in the SST slices.

Versal AIE-ML Sparsity Comparison

The new Versal AIE-ML AMD (2024a,b) includes out-of-fabric processors that support 2:4 sparsity. Therefore, we aim to compare the attainable acceleration due to sparsity, as well as the efficiency of the compressed sparse format between our SSTs and the AIE-ML. We exploit AMD’s optimized codes for dense and sparse GEMM kernels for the AIE-ML AMD (2023b,c), using 16-bits for the outputs as this leads to substantially higher compute utilization AMD (2023b). We compile and simulate the AIE-ML designs using Vitis 2023.2 on the VEK280 platform AMD (2024f).

Figure 6.9a shows the compute utilization of a 64×64×6464 \times64 \times64 GEMM, mapped to both SST slices and the AIE-ML. We notice that the AIE-ML achieves higher than 90% utilization in dense GEMM for both precisions. However, for 2:4 sparsity, we observe a substantially lower utilization, i.e., 51.6% for int8 and 50.4% for bf16. We note that the compute utilization is calculated as the effective utilization of only non-zero computation in the sparse case. Thus, we obtain a marginal speedup of 13% (int8) and 4.5% (bf16), for the 2:4 sparse GEMM over the dense case for the AIE-ML. Moreover, for higher sparsity, i.e., 1:3 and 1:4, the compute utilization in AIE-ML drops more significantly (Figure 6.9a). While higher than 2:4 sparsity can be stored in a compressed format in the AIE-ML, only 2:4 sparsity is inherently supported. Consequently, no further acceleration can be achieved, leading to significantly decreased effective utilization.

Figure 6.9. Compute utilization (a) and compression ratio (b) comparison of a 64×64×64 GEMM using SSTs vs. AIE-ML.

This marginal speedup due to sparsity in the AIE-ML is primarily attributed to limited vector load/store bandwidth, causing the sparse GEMM to become I/O bound AMD (2023b, 2024a). In contrast, as depicted in Figure 6.9a, our SST-based GEMM achieves 100% utilization for all supported sparsity levels. This is because we architect our SST slices in a fashion that guarantees the corresponding speedup of each sparsity level, e.g., 4×4 \times for 1:4 sparsity (refer to § 6.1.3 for design details).

Figure 6.9b shows the compression ratios achieved by the SSTs vs. the AIE-ML. The AIE-ML uses a bitmap-based compressed format AMD (2024b), which is also used in various works accelerating structured sparsity Liu et al. (2022a, 2020a); Fang et al. (2022); Cai et al. (2023); Yang et al. (2024). Instead, we utilize an index-based format (§ 6.1.1). For 2:4 sparsity, both approaches present the same compression ratio (Figure 6.9b). However, for 1:3 and 1:4, our approach leads to higher compression ratio, up to 20% for int8 and 11% for bf16, showcasing the storage efficiency of the index-based format. Moreover, as shown in Fang et al. (2022), bitmap achieves higher compression ratio over traditional sparse formats, e.g., CSR, CSC, COO Hoefler et al. (2021), for sparsity levels of 50–87.5%. This highlights that the supported index-based sparse format provides superior compression over other formats.

Impact on Non-AI Benchmarks

We demonstrate the flexibility of our enhanced FPGA architecture with SST slices, by exploring the maximum attainable frequency of non-AI benchmarks. We select non-AI benchmarks from the VTR benchmark suite Murray et al. (2020), which target various application domains, i.e., sha, mmcl, arm_core, stereovision2, LU32PEEng, bgm, raygentop, and blob_merge (see VTR (2023) for domains). When comparing our enhanced FPGA with a traditional FPGA on the aforementioned benchmarks, we observe, on average, a negligible decrease (<1%<1\%) in maximum frequency. Hence, our proposed FPGAs maintain the performance and flexibility of traditional FPGAs, while offering highly efficient sparse and dense DNN acceleration (§ 6.2.3–§ 6.2.7).

Insights & Discussion

Very High Sparsity in DNN Layers

In the previous sections, we exhibit the importance and efficiency of our proposed in-fabric SST slices for accelerating sparse DNNs. However, our SSTs support up to 75% (1:4) structured sparsity. Prior work Han et al. (2016, 2015) has shown that a few DNN layers can have very high random sparsity, e.g., >90%>90\%. In this case, unstructured sparse FPGA accelerators mapped to traditional CLB and DSP resources, e.g., Fowers et al. (2014); Jain et al. (2020); Lu et al. (2019), can be exploited to efficiently compute these few high-sparsity layers. All other DNN layers with less sparsity can be mapped to our SSTs (§ 6.2.4 & § 6.2.7), enabling both structured and unstructured sparse FPGA accelerators to operate synergistically, for maximized efficiency. Moreover, for very high sparsity (>90%>90\%), the CSR, CSC, COO formats utilized in unstructured sparse accelerators, offer the most efficient compression Hoefler et al. (2021). Instead, for lower sparsity, the index-based format utilized in SSTs is the most efficient (§ 6.2.8). This emphasizes the custom flexibility of the enhanced FPGAs with in-fabric SST slices in supporting highly efficient sparse DNN accelerators.

Efficient In-Fabric Slices

In-fabric slices have attained commercial success in AI-optimized FPGAs, e.g., the Intel tensor blocks Langhammer et al. [10] and the new Intel AI-enhanced DSPs Gribok and Pasca (2024); Intel (2024a). Moreover, as we showed in Chapter 5, the comparison of the leading in-fabric (Intel Stratix 10 NX) to out-of-fabric (AMD Versal ACAP) AI-optimized FPGAs, resulted in ∼44%\sim44\% higher energy efficiency in GEMM, favoring the in-fabric solutions. However, these efficient in-fabric slices primarily target dense AI acceleration, rendering them insufficient for most DNNs, which are sparse. Instead, our proposed flexible SST slices support both dense and sparse computation (multiple sparsity levels), showcasing substantial advantages over dense in-fabric blocks and traditional FPGAs (§ 6.2.4), for actual sparse DNN workloads (§ 6.2.7). Note that although we are inspired by prior academic research that utilize dense 2D systolic in-fabric blocks Arora et al. (2021b, 2022b), the methodology described in § 6.1 for introducing sparsity support can be generalized in straightforward fashion to other in-fabric FPGA blocks.

Chapter Summary & Impact

Structured sparsity support has been incorporated in state-of-the-art GPUs, e.g., Nvidia H100 and AMD MI300, as well as the new Versal AIE-ML processors. Although FPGA architectures have been enriched with in-fabric blocks for DNN acceleration, these blocks are primarily designed for dense operation. This leads to insufficient computation for most contemporary DNN models, which display varying levels of sparsity. To effectively address this challenge, we propose flexible in-fabric blocks, named SST slices, that support multiple levels of structured sparsity. We show that our sparse GEMM accelerators exploiting the SST slices offer substantial performance, scalability, and area advantages over traditional FPGAs. Demonstration on various state-of-the-art sparse DNN models utilizing our SST slices, exhibits up to 3.52×3.52 \times speedup with marginal accuracy degradation compared to dense in-fabric acceleration.

Chapter 7: Striking the Balance: GEMM Performance Optimization Across Generations of Ryzen AI NPUs

In this chapter, we present the methodology we propose to optimize GEMM workloads across the two current NPU generations, XDNA and XDNA2 Taka et al. (2026, 2025b). As an evolution of the Versal AI Engine, these NPUs introduce new architectural constraints that require rethinking several of the optimization strategies developed in earlier chapters. Our methodology addresses key performance bottlenecks at the system level and provides significant insights for various aspects in GEMM optimization.

GEMM Design & Optimization

Our approach consists of a parametrized GEMM mapping that exploits the unique features of the NPU architecture, combined with an exhaustive parameter search based on an integer programming formulation of the performance characteristics of our chosen mapping. In this section, we describe the details of both.

Multi-Level Tiling Method

Figure 7.1 illustrates the multi-level tiling scheme we use to partition the input and output matrices. The inner-most (first) tiling level is defined by the supported shapes of the AIE API library for the single-core GEMM kernel AMD [6], and is expressed via the parameters r×s×tr \times s \times t. The AIE API provides multiple optimized modes for each supported precision and enables portability across NPU generations. The second tiling level is determined by the GEMM kernel that is supported out of local L1 memory, denoted as mct×kct×nctm_{\mathrm{ct}} \times k_{\mathrm{ct}} \times n_{\mathrm{ct}}. Since the NPU is a multi-core architecture, we partition GEMM across multiple cores to exploit spatial-level parallelism and maximize performance. The third tiling level achieves this partitioning and corresponds to the GEMM size operating on the entire NPU array (parameters and mapping delineated in the next section). Finally, the outer-most (fourth) tiling level is dictated by the final GEMM sizes of input matrices AA (typically activations) and BB (typically weights), and output matrix CC, expressed as M×K×NM \times K \times N.

Figure 7.1. Proposed GEMM multi-level tiling scheme.

GEMM Mapping Strategy on NPUs

NPU Array Mapping & Core Design

Our mapping strategy is to parallelize GEMM in space across the MM and NN dimensions, while reduction across the KK dimension is performed in time. In this manner, all NPU cores perform the same GEMM computation on different data and operate independently. This leads to maximized performance, since no data communication occurs between the cores, as opposed to Versal devices, where the reduction dimension is partitioned across multiple cores, e.g., Taka et al. (2023); Mhatre et al. (2025b). By using the broadcast feature of the NPU architecture, we can attain spatial parallelization and exploit the inherent data reuse of the GEMM algorithm. This parallelization is expressed via the number of single-core tiles in the MM and NN dimensions, and is defined by the parameters mrowsm_{\mathrm{rows}} and ncolsn_{\mathrm{cols}}, respectively (Figure 7.1). In this work, we map the parameters mrowsm_{\mathrm{rows}} and ncolsn_{\mathrm{cols}} in a straightforward fashion to the number of rows and columns of the two NPU architectures, respectively. In particular, we broadcast each input AA tile across one row in the NPU array, e.g., A0A0 in Figure 7.2a and Figure 7.2b for XDNA and XDNA2, respectively. Similarly, each BB tile is broadcast across one column of NPU cores. Note that due to the absence of a ShimTile in the last column of XDNA, we choose to map GEMM across 4 rows and 4 columns for this architecture, similar to AMD [6]; Rösti and Franz (2025); Fang et al. (2025). This enables a symmetric 4×44 \times4 (mrows×ncolsm_{\mathrm{rows}} \times n_{\mathrm{cols}}) GEMM implementation, as depicted in Figure 7.2a. For XDNA2, we utilize the entire 4×84 \times8 array, which results in an asymmetric mapping (Figure 7.2b).

GEMM mapping strategy on XDNA (a) and XDNA2 (b)

Figure 7.2. GEMM mapping strategy on XDNA (a) and XDNA2 (b).

Each core performs a GEMM of mct×kct×nctm_{\mathrm{ct}} \times k_{\mathrm{ct}} \times n_{\mathrm{ct}} size. To handle the reduction across KK, the single-core kernel also loads previous partial results of mct×nctm_{\mathrm{ct}} \times n_{\mathrm{ct}} size, performs accumulation and stores the updated results back to the output tiles. For example, the upper-left core in Figure 7.2b, sequentially loads tiles A0A0 and B0B0, A0′A0' and B0′B0', etc., in order to perform reduction. In this fashion, the output CC tiles remain stationary in the L1 memory of each core (output stationary mapping). When all tiles complete their accumulation (K/kctK/k_{\mathrm{ct}} tiles in total), the output tile is transferred to an L2 MemTile, and eventually to DRAM. The MemTile design is detailed in § 7.1.3. Subsequently, a fast vectorized kernel initializes the output CC tile to zero, preparing it for the next accumulation in GEMM tiling.

To overlap GEMM computation with DMA transfers of matrix tiles, we employ double-buffering on both L1 and L2 for the input tiles AA and BB. However, we retain the output CC tiles as a single buffer on each core. Since the output tiles are transferred only once for each complete reduction, when K/kctK/k_{\mathrm{ct}} is sufficiently large, the infrequent added latency of a single-buffered output transfer becomes negligible. At the same time, this design choice frees up valuable L1 memory, thereby enabling higher flexibility in tiling parameter optimization. The larger tile sizes enabled by this additional memory ultimately lead to higher end-to-end GEMM performance in the general case (refer to § 7.4.2).

Memory Tile Design

The input tiles are temporarily stored in L2 MemTiles before being broadcast to each NPU core. Note that we load multiple input tiles across the reduction dimension KK into MemTiles, e.g., A0A0 and A0′A0' in Figure 7.2a and Figure 7.2b. This is an important aspect of the mapping; loading multiple tiles allows us to access a larger amount of contiguous data from DRAM, where matrices are stored in regular order (row- and column-major). Such long contiguous reads result in increased DRAM BW utilization, which is essential for maximizing GEMM performance at the system level. In this work, we retain matrices AA and CC in row-major order, while matrix BB is either in row- or column-major order. To this end, we introduce another parameter, kmtk_{\mathrm{mt}}, which specifies the size of the tiles loaded into L2. In particular, for matrix AA, each L2 MemTile loads a tile of mct×kmtm_{\mathrm{ct}} \times k_{\mathrm{mt}} size. Similarly, when BB is in column-major order, each L2 MemTile loads a tile of kmt×nctk_{\mathrm{mt}} \times n_{\mathrm{ct}} size. Storing matrix AA in row-major and BB in column-major provides sufficient contiguous DRAM access for both matrices, which leads to higher GEMM performance (§ 7.3.1). However, when BB is in row-major, MemTiles load the same tile as CompTiles, i.e., kct×nctk_{\mathrm{ct}} \times n_{\mathrm{ct}}, since contiguous data are accessible across the nctn_{\mathrm{ct}} dimension.

Due to the 4×44 \times4 symmetric design of XDNA, each MemTile holds the same amount of input tiles, as illustrated in Figure 7.2a. Specifically, each MemTile broadcasts BB tiles within its own column. The AA tiles are broadcast across the four rows; we map them in a regular fashion to the four MemTiles: the MemTile in column 0 holds A0 (which will be broadcast across row 0), the MemTile in column 1 holds A1 (which will be broadcast across row 1), etc. In contrast, for XDNA2, we map the four AA tiles to eight MemTiles in an alternating pattern across even columns due to its 4×84 \times8 asymmetric design. The mapping for BB remains the same. In this design, even MemTiles hold more tiles than their odd counterparts, as depicted in the logical view of Figure 7.2b. This particular mapping aids the IRON tool to leverage the NPU architectural feature of directly accessing the memory of the neighboring MemTile (AMD 2024a). Therefore, when buffer sizes exceed the capacity of a specific MemTile, IRON physically allocates buffers to a neighboring MemTile.

Observe that four output CC tiles are aggregated across each column into a MemTile (Figure 7.2a and Figure 7.2b). This is because four CC tiles need to be transferred concurrently, while ShimTiles provide only two S2MM channels. Hence, we exploit the six S2MM channels available in each MemTile (AMD 2024a) to temporarily store the four output tiles, before they are transferred to DRAM via ShimTiles.

We define the native GEMM size, as (mct⋅mrows)×kmt×(nct⋅ncols)(m_{\mathrm{ct}} \cdot m_{\mathrm{rows}}) \times k_{\mathrm{mt}} \times(n_{\mathrm{ct}} \cdot n_{\mathrm{cols}}). This corresponds to the GEMM size that operates natively on the entire NPU array, while also ensuring high performance. Moreover, note that although we arbitrarily map matrix AA across rows and matrix BB across columns, the reverse mapping is equally feasible, yielding symmetrical solutions across the MM and NN dimensions. Finally, although an input stationary mapping can also be employed, it would not be adequate to efficiently support arbitrary GEMM dimensions. Specifically, partial results would need to be temporarily stored in MemTiles, and subsequently reloaded to CompTiles for exploitation of data reuse in GEMM. This would require three input channels for efficient GEMM computation, while CompTiles provide two inputs channels.

On-The-Fly Tensor Transformations

We extensively exploit the multi-dimensional addressing feature of DMAs to reorganize data into tiled layouts, as needed by the NPU cores. This enables matrices to be stored in standard order in DRAM (row- and column-major). The single-core GEMM kernels assume that matrices are pre-tiled AMD (2025f). In particular, the kernels running on each core expect r×s×tr \times s \times t-sized tiles and both data within tiles as well as the tiles themselves to be in row-major order, as illustrated in the upper part of Figure 7.3, for the case of matrix AA. The aforementioned distribution of tiles across multiple cores, along with the requirement for contiguous DRAM accesses necessitates multiple data layout transformations, as explained below.

Figure 7.3. On-the-fly DMA transformations for matrix A.

As matrices are transferred from DRAM to NPU, DMAs apply a series of transformations depending on the addressing capabilities of each NPU tile. Figure 7.3 shows the transformations of each DMA channel associated with the transfer of matrix AA tiles. Initially, a ShimTile MM2S channel reads a tile of mct×Km_{\mathrm{ct}} \times K size from DRAM. Since ShimTiles support 3D addressing, the row-major mct×Km_{\mathrm{ct}} \times K tile is transformed into multiple smaller mct×kmtm_{\mathrm{ct}} \times k_{\mathrm{mt}} tiles via the MM2S channel (parameters: mctm_{\mathrm{ct}}, kmtk_{\mathrm{mt}}, KK). Before each mct×kmtm_{\mathrm{ct}} \times k_{\mathrm{mt}} tile is stored in the MemTile, another 3D transformation occurs at the S2MM MemTile channel, partitioning it into several mct×kctm_{\mathrm{ct}} \times k_{\mathrm{ct}} tiles (parameters: mctm_{\mathrm{ct}}, kctk_{\mathrm{ct}}, kmtk_{\mathrm{mt}}).

Each MemTile holds a mct×kmtm_{\mathrm{ct}} \times k_{\mathrm{mt}} tile, which it sequentially transmits to the corresponding CompTiles as a series of smaller mct×kctm_{ct} \times k_{ct} tiles. Since the CompTiles expect pre-tiled data, the data layout of each of the smaller tiles requires transformation. Describing this transfer requires five parameters, namely rr, ss, mctm_{ct}, kctk_{ct}, kmtk_{mt}. However, MemTiles only support 4D addressing. To circumvent this, we decompose the transformation into two separate transformations, by utilizing the hardware’s data layout transformation features in both the MM2S MemTile (output) channel and the S2MM CompTile (input) channel. First, the MM2S MemTile channel partitions data into several mct×sm_{ct} \times s tiles, as depicted in Figure 7.3 (parameters: ss, mctm_{ct}, kctk_{ct}, kmtk_{mt}). This enables address linearization within the r×sr \times s tile, thereby allowing a subsequent 3D transformation in the S2MM CompTile channel to reorganize data into the required layout (effective parameters: r⋅sr \cdot s, mctm_{ct}, kctk_{ct}).

Address generation in the NPU DMAs occurs at 32-bit granularity AMD (2024a,b). DMAs alone cannot perform layout transformations at smaller-precision data types, e.g., individual elements of int8 or bf16 data types. However, shuffle instructions running on the AIE cores can be utilized to support this operation and enable the fine-grained data swizzling. In our application, this becomes relevant in use cases where an int8 or bf16 matrix BB is stored in column-major order in DRAM. To this end, we modify the GEMM kernel to utilize shuffling instructions, by using the AIE API transpose function AMD [6], such that both data within tiles and the tiles themselves are in column-major order. Subsequently, we apply similar transformations to matrix BB as demonstrated for matrix AA, across the NPU hierarchy (see Figure 7.3).

Furthermore, when matrix BB is in row-major, only one 4D transformation is required in the MemTiles (parameters: ss, tt, kctk_{ct}, nctn_{ct}), since each MemTile holds a kct×nctk_{ct} \times n_{ct} tile (§ 7.1.2). In a similar fashion, matrix CC tiles (row-major) entail a single 4D transformation in the MemTiles (parameters: rr, tt, mctm_{ct}, nctn_{ct}).

Data Movement Between NPU & DRAM

The outer-most (fourth) level of GEMM tiling encompasses data movement between the NPU and DRAM. The NPU interfaces with DRAM via ShimTiles and the control of data movement is orchestrated by the on-chip command processor. ShimTiles are equipped with an input task queue to facilitate DMA transfers, where tasks are submitted sequentially. When a task terminates, it issues a task-completion token, which the command processor uses to synchronize between multiple DRAM transfers. Each ShimTile has access to 16 BDs AMD (2024a), which are used to specify DMA transfers. When a complex data movement pattern requires more than 16 BDs on a ShimTile, we can reuse (reconfigure) BDs. Before reconfiguring a BD, it is necessary to properly synchronize and ensure that the previous transfer associated with the BD has completed AMD [6].

Figure 7.4 illustrates a simplified view of the data movement between the NPU and DRAM. Each BD is utilized to describe data movements in a fine-grained manner. In particular, for matrix AA, each BD defines one ShimTile DMA transfer of mct×Km_{\mathrm{ct}} \times K size. Similarly, for matrix BB, each BD defines a transfer of K×nctK \times n_{\mathrm{ct}}. For matrix CC, each BD describes a transfer of (mct⋅mrows)×nct(m_{\mathrm{ct}} \cdot m_{\mathrm{rows}}) \times n_{\mathrm{ct}}, since mrowsm_{\mathrm{rows}} output tiles are aggregated per column. The command processor program in our implementation maps and inserts BDs into ShimTile task queues as determined by the GEMM mapping strategy, e.g., BDA0BD_{A0}, BDB0BD_{B0}, and BDC0BD_{C0} to ShimTile 0, BDA1BD_{A1}, BDB1BD_{B1}, and BDC2BD_{C2} to ShimTile 1, etc. Moreover, BDA4BD_{A4}, BDB0BD_{B0}, and BDC1BD_{C1} are also pushed into the queue of ShimTile 0, as dictated by GEMM tiling. Similarly, our program enqueues BDs describing the GEMM transfers into the input task queues of each ShimTile in a sequential fashion. We note that the simplified example in Figure 7.4 is directly applicable to XDNA (ncols=4n_{\mathrm{cols}} = 4, see § 7.1.2), while the BD mapping for XDNA2 can be determined in a straightforward fashion.

Simplified view of outer-most fourth GEMM tiling level showing Matrix A, Matrix B, Matrix C, ShimTile task queues, and NPU–DRAM transfers

Figure 7.4. Simplified view of outer-most (fourth) GEMM tiling level, determined by NPU–DRAM transfers (mrowsm_{\mathrm{rows}}, ncols=4n_{\mathrm{cols}} = 4).

Depending on the target GEMM dimensions, i.e., MM, KK, NN, more than the maximum of 16 BDs in each ShimTile might be required. BD reconfiguration is needed in this situation, which might result in performance degradation (§ 7.4.3).

To address this challenge, we propose the following procedure, which enables DMA data transfers to overlap with BD reconfiguration. Initially, we submit to the task queue five BDs for each of the three AA, BB, and CC DMA transfers. This efficiently utilizes 15 out of the 16 BDs available in each ShimTile. Each DMA transfer begins immediately once its associated BD reaches the front of the queue. For instance, BDA0BD_{A0} and BDB0BD_{B0} transfers start first in Figure 7.4. Subsequently, the command processor waits for a task-completion token for each output transfer in a sequential manner, e.g., first for BDC0BD_{C0}, then for BDC1BD_{C1}, etc. Notice that the command processor only needs to wait for completion of each BD associated with the output matrix, e.g., BDC0BD_{C0}, since once it completes, the corresponding input BDs, e.g., BDA0BD_{A0} and BDB0BD_{B0}, have also finished. Therefore, once each output BD completes, the three retired BDs can be safely reconfigured, and the next three BDs are inserted into the queue (if available). This ensures that 15 BDs are in the queue in the steady-state operation, allowing DMA data movement to overlap efficiently with BD reconfiguration. This process is repeated iteratively until GEMM tiling is completed.

The fine-grained BD description of NPU–DRAM data movement discussed above allows supporting very large GEMM dimensions. GEMM dimensionality is limited by the NPU registers bitwidth utilized in multi-dimensional tensor addressing AMD (2024b). For instance, when storing BB in column-major, the GEMM programming example in [6] allows only a reduction KK dimension of up to ∼4\sim4K for bf16 on XDNA2, while our approach allows sizes >64>64K in all dimensions.

Analytical Modeling Optimization

To maximize GEMM performance, we propose an optimization methodology based on analytical modeling. First, we focus on single-core performance to gain insights for the on-chip compute part, and afterwards we extend our methodology to optimize performance at the system level by incorporating off-chip DRAM BW constraints.

Single-Core GEMM Optimization

Our model utilizes architectural parameters, such as the peak compute throughput of the cores (peak_MACspeak\_MACs in MACs/cycle) and the DMA BW (DMA_BWDMA\_BW in Bytes/cycle), in order to identify the optimal mctm_{\mathrm{ct}}, kctk_{\mathrm{ct}}, nctn_{\mathrm{ct}} parameters that maximize performance. We define the efficiency (effeff) as the fraction of the attained compute throughput to the peak throughput of the core. Eq. (26) expresses the compute cycles of the GEMM kernel (CcompC_{\mathrm{comp}}), while Eq. (27) and Eq. (28) formulate the number of cycles needed for the DMA transfers of AA (CAcommCA_{\mathrm{comm}}) and BB (CBcommCB_{\mathrm{comm}}) tiles, respectively. Here, ty(⋅)ty(\cdot) indicates the data type size (in Bytes) of the matrices.

Ccomp=mct⋅kct⋅nct/(eff⋅peak_MACs)(26)C_{\mathrm{comp}} = m_{\mathrm{ct}} \cdot k_{\mathrm{ct}} \cdot n_{\mathrm{ct}}/(eff \cdot peak\_MACs) \tag*{(26)}
CAcomm=mct⋅kct⋅ty(A)/DMA_BW(27)CA_{\mathrm{comm}} = m_{\mathrm{ct}} \cdot k_{\mathrm{ct}} \cdot ty(A)/DMA\_BW \tag*{(27)}
CBcomm=kct⋅nct⋅ty(B)/DMA_BW(28)CB_{\mathrm{comm}} = k_{\mathrm{ct}} \cdot n_{\mathrm{ct}} \cdot ty(B)/DMA\_BW \tag*{(28)}

Next, we define the constraint in Eq. (29) to ensure that the single-core GEMM remains compute bound (not bounded by the DMA BW for AA and BB tiles). Furthermore, Eq. (30) restricts the GEMM buffers to fit within the 64KB L1 memory capacity (with 1KB reserved for stack). Finally, we also impose the straightforward constraint that mctm_{\mathrm{ct}}, kctk_{\mathrm{ct}}, nctn_{\mathrm{ct}} need to be multiples of rr, ss, tt, respectively (not shown).

Ccomp≥{CAcomm,CBcomm}(29)C_{\mathrm{comp}} \ge\{CA_{\mathrm{comm}}, CB_{\mathrm{comm}}\} \tag*{(29)}
{2⋅mct⋅kct⋅ty(A)+2⋅kct⋅nct⋅ty(B)+mct⋅nct⋅ty(C)}≤63 KB(30)\{2 \cdot m_{\mathrm{ct}} \cdot k_{\mathrm{ct}} \cdot ty(A) + 2 \cdot k_{\mathrm{ct}} \cdot n_{\mathrm{ct}} \cdot ty(B) + m_{\mathrm{ct}} \cdot n_{\mathrm{ct}} \cdot ty(C)\} \le63\ \mathrm{KB} \tag*{(30)}

The solution of mctm_{\mathrm{ct}}, kctk_{\mathrm{ct}}, nctn_{\mathrm{ct}} can be formulated as an integer programming (IP) optimization problem utilizing the aforementioned constraints. The IP is solved exhaustively by setting the maximization of the number of MACs (mct⋅kct⋅nctm_{\mathrm{ct}} \cdot k_{\mathrm{ct}} \cdot n_{\mathrm{ct}}) as the main objective. This increases data reuse in GEMM, thereby maximizing the overall efficiency. Furthermore, due to the output stationary GEMM mapping, we impose a second objective to minimize the output CC tile, i.e., the product mct⋅nctm_{\mathrm{ct}} \cdot n_{\mathrm{ct}}. This is essential in reducing the number of loads/stores for accumulations and decreasing memory stalls caused by bank conflicts. Evidently, the two optimization objectives lead to increased kctk_{\mathrm{ct}} and reduced mctm_{\mathrm{ct}}, nctn_{\mathrm{ct}} (sufficiently large to not become DMA BW bound). Moreover, notice that we do not impose DMA constraints on the output CC buffer, as opposed to AA and BB (Eq. (27), 7.3), while also retaining CC as a single buffer (Eq. (30)). This significantly increases the search space, thus increasing performance in the general GEMM case (§ 7.4.2), which is an essential aspect of the system-level optimization discussed below.

System-Level NPU Array Optimization

First, we analytically express the DRAM accesses for each matrix in GEMM. Eq. (31) captures the DRAM reads needed for matrix AA (AmemA_{\mathrm{mem}}). The first term, mct⋅mrows⋅K⋅ty(A)m_{\mathrm{ct}} \cdot m_{\mathrm{rows}} \cdot K \cdot ty(A), represents the DRAM reads (in Bytes) during GEMM tiling along the KK dimension. This is due to the output stationary mapping and because AA is broadcast across rows (§ 7.1.2). The second term, N/(nct⋅ncols)N/(n_{\mathrm{ct}} \cdot n_{\mathrm{cols}}), describes the repeat factor of the aforementioned read accesses due to tiling along the NN dimension. In a similar manner, the third term, M/(mct⋅mrows)M/(m_{\mathrm{ct}} \cdot m_{\mathrm{rows}}), captures the repeat across the MM dimension.

Amem=(mct⋅mrows⋅K⋅ty(A))(Nnct⋅ncols)(Mmct⋅mrows)⇒Amem=M⋅K⋅N⋅ty(A)/(nct⋅ncols)(31)A_{\mathrm{mem}} = \left(m_{\mathrm{ct}} \cdot m_{\mathrm{rows}} \cdot K \cdot ty(A)\right)\left(\frac{N}{n_{\mathrm{ct}} \cdot n_{\mathrm{cols}}}\right)\left(\frac{M}{m_{\mathrm{ct}} \cdot m_{\mathrm{rows}}}\right) \Rightarrow A_{\mathrm{mem}} = M \cdot K \cdot N \cdot ty(A)/(n_{\mathrm{ct}} \cdot n_{\mathrm{cols}}) \tag*{(31)}

Similarly, Eq. (32) represents the DRAM reads for matrix BB (BmemB_{\mathrm{mem}}), while Eq. (33) shows the DRAM writes for the output matrix CC (CmemC_{\mathrm{mem}}). We note here that these equations provide an elegant and compact representation of data reuse and tiling scheme in GEMM.

Bmem=M⋅K⋅N⋅ty(B)/(mct⋅mrows)(32)B_{mem} = M \cdot K \cdot N \cdot ty(B)/(m_{ct} \cdot m_{rows}) \tag*{(32)}
Cmem=M⋅N⋅ty(C)(33)C_{mem} = M \cdot N \cdot ty(C) \tag*{(33)}

Furthermore, Eq. (34) models the GEMM compute time on the NPU (TcompT_{comp}), while Eq. (35) expresses the total DRAM access time (TmemT_{mem}). Here, peak_TOPSpeak\_TOPS denotes the theoretical peak throughput of the NPU array, calculated at the maximum operating frequency. Furthermore, DRAM_BWDRAM\_BW is the effective DRAM BW achieved during GEMM execution on the NPU. Note that, since all NPU cores execute the same GEMM kernel independently, the single-core efficiency effeff, as defined in § 7.1.6.1, directly corresponds to the entire NPU array efficiency in Eq. (34).

Tcomp=2⋅M⋅K⋅N/(eff⋅peak_TOPS)(34)T_{comp} = 2 \cdot M \cdot K \cdot N/(eff \cdot peak\_TOPS) \tag*{(34)}
Tmem=(Amem+Bmem+Cmem)/DRAM_BW(35)T_{mem} = (A_{mem} + B_{mem} + C_{mem})/DRAM\_BW \tag*{(35)}

As discussed previously (§ 7.1.6.1), minimizing mctm_{ct} and nctn_{ct}, as well as maximizing kctk_{ct} is essential to attain high efficiency, and thus maximized GEMM performance (equivalent to minimized TcompT_{comp}). However, from Eq. (31) and Eq. (32), we notice that nctn_{ct} and mctm_{ct} parameters appear on the denominator of DRAM accesses for AA and BB, respectively. Therefore, the DRAM access time, TmemT_{mem}, increases as mctm_{ct} and nctn_{ct} decrease. This highlights the inverse relationship between compute and memory time: as one increases, the other decreases. Thus, the optimal GEMM performance is attained at the balanced point where they intersect (i.e., Tcomp≈TmemT_{comp} \approx T_{mem}).

In order to find the aforementioned optimal balanced point we empirically set starting values for mctm_{ct}, kctk_{ct}, nctn_{ct}, and effeff parameters, and perform the iterative procedure explained below (starting values reduce iterations to typically <5<5). These starting values are set based on the results of the single-core kernel optimization (§ 7.1.6.1), and the effective DRAM_BWDRAM\_BW during GEMM execution (measured via micro-benchmarking, refer to § 7.2.2.1). First, we measure the actual GEMM performance on the NPU device as well as the single kernel efficiency for the starting values, verifying that GEMM is memory bound (Tcomp<TmemT_{comp} < T_{mem}, due to low mctm_{ct}, nctn_{ct}, and high kctk_{ct}). In each iteration, we decrease the parameter kctk_{ct} (as a multiple of ss), and solve exhaustively an IP similar to the previous § 7.1.6.1. However, for that specific iteration, we fix the kctk_{ct} parameter and the objective is to maximize the product of mct⋅nctm_{ct} \cdot n_{ct}. This leads to maximized mctm_{ct}, nctn_{ct} values, given the DMA BW and L1 memory constraints, ensuring a highly optimized GEMM kernel (maximized number of MACs). This is essential in identifying the optimal balanced point, because it results in the smallest possible increment of TcompT_{comp} in each iteration, while also ensuring maximized possible GEMM performance given the specific parameters of that iteration. We note here that the efficiency (effeff) for a specific iteration is estimated based on each previous point measurements of each iteration. We iteratively measure GEMM performance on the NPU device of the top-ranked solution for each IP, verifying that in each step performance is higher compared to the previous. While GEMM performance is increasing in each step, at some point we observe lower performance, and the iteration stops. Evidently, at this specific point GEMM has become compute bound (Tcomp>TmemT_{comp} > T_{mem}), while also performance is lower. Therefore, the balanced point where GEMM performance is maximized has been identified (mctm_{ct}, kctk_{ct}, nctn_{ct} parameters of the previous iteration).

Evaluation

For the experimental evaluation, we use two representative mini PCs, corresponding to two NPU generations. The Minisforum UM790 Pro Minisforum (2025), equipped with Ryzen 9 7940HS processor (Phoenix Point) AMD (2025m), is used for XDNA. For XDNA2, we use the ASRock 4×4 Box Industrial (2025), featuring the AMD Ryzen AI 7 350 processor (Krackan Point) AMD (2025n). Both mini PCs have dual-channel DDR5-5600 MT/s DRAM. The CPUs run Ubuntu 24.04 LTS and serve as the host. In particular, the CPUs allocate buffers in DRAM using Xilinx Runtime (XRT) [6], configure the NPU, invoke the NPU for GEMM execution, and process the completion notification. Finally, throughout all experiments, NPUs are configured at their maximum performance level, i.e., turbo mode [6], utilizing XDNA driver commands [6].

Single-Core GEMM Performance

We exploit the AIE API [6] to design highly optimized single-core GEMM kernels. The kernels are compiled using the xchesscc tool, employing various compiler directives, e.g., software pipelining and loop unrolling/flattening, to attain high efficiency. Performance is assessed via hardware profiling utilizing the NPU trace unit [6] (AMD 2024a), thereby enabling cycle accurate measurements. For int8, besides full output precision (int32), we also perform precision reduction to 16- and 8-bits, which is a common technique to increase GEMM performance in AIE architectures [6] (); Zhuang et al. 2024; ); Deng et al. 2024; ); Mhatre et al. 2025b). Moreover, for bf16, we retain the output precision to bf16. Finally, for XDNA2 we emulate the bf16 precision utilizing the bfp16 hardware datapath, which leads to increased GEMM performance (emulation attained via a specific xchesscc flag at compile time, as mentioned in [6]).

In Table 7.1, we present the top-ranked solutions of the single-core optimization procedure described in § 7.1.6.1. For int8 precision, we attain very high throughput, ranging from 192.0–233.0 MACs/cycle and from 384.0–450.6 MACs/cycle for XDNA and XDNA2, respectively. Similarly for bf16, we achieve 112.6 MACs/cycle for XDNA, and 158.1 MACs/cycle for XDNA2. Observe that in both cases, XDNA2 attains higher throughput compared to XDNA, since it has higher peak compute throughput capabilities per core. We note here that, since performance is measured via hardware tracing, the results in Table 7.1 include the inevitable memory stalls due to bank conflicts, thus reflecting the actual NPU performance (when not bounded by DRAM BW). Finally, very high L1 memory usage is achieved across all solutions, ranging from 94–97%.

Dev.Precision In-OutKernel Size mct×kct×nctm_{ct} \times k_{ct} \times n_{ct}Throughput MACs/cycleL1 Core Mem. (KB)
XDNAint8-int864×232×6464 \times232 \times64233.062.0 (97%)
int8-int1664×216×6464 \times216 \times64217.662.0 (97%)
int8-int3248×280×4848 \times280 \times48192.061.5 (96%)
bf16-bf1664×104×6464 \times104 \times64112.660.0 (94%)
XDNA2int8-int864×232×6464 \times232 \times64450.662.0 (97%)
int8-int1664×216×6464 \times216 \times64419.862.0 (97%)
int8-int3248×280×4848 \times280 \times48384.061.5 (96%)
bf16-bf1648×152×4848 \times152 \times48158.161.5 (96%)

Table 7.1. Single-core GEMM results for XDNA & XDNA2.

GEMM Performance on NPU Array

In this section, we present GEMM performance on the entire NPU array. Performance is evaluated using wall-clock time, thereby capturing the actual performance observed by the users (includes OS overheads, NPU dispatch time, etc.) [6]. All reported results represent the average of 100 runs.

Optimal Balanced GEMM Kernel

As shown in the previous section, GEMM kernels can attain very high throughput. However, when using the optimum compute kernel sizes of Table 7.1, we observe that GEMM performance on the NPU array remains low. For instance, for int8-int16 precision, we obtain only 17.86 TOPS on XDNA2 at ∼4K square GEMM size. However, the peak compute capability of this kernel on the XDNA2 array is 48.36 TOPS, when calculated at maximum operating frequency (identified through XDNA driver commands [6], i.e., 1 GHz and 1.8 GHz for XDNA and XDNA2, respectively). This implies that GEMM is memory bound at this specific kernel size, due to low values of mctm_{ct} and nctn_{ct} (inverse relationship of compute and memory). Therefore, we leverage the optimization methodology described in § 7.1.6.2 in order to identify the optimal balanced kernel, i.e., where compute and memory become balanced. First, we use micro-benchmarking to estimate the effective DRAM BW that is available to the NPU when running GEMM workloads. For micro-benchmarking, we imitate GEMM transfers from DRAM to NPU array and vice-versa, observing ∼15\sim15 GB/s and ∼50\sim50 GB/s for XDNA and XDNA2, respectively. Afterwards, we exploit these values to select starting points for the iterative optimization procedure of § 7.1.6.2.

Tables 7.2 and 7.3 present the two top-ranked solutions, for XDNA and XDNA2, respectively. First, notice that kernel sizes have lower kctk_{\mathrm{ct}} and higher mctm_{\mathrm{ct}}, nctn_{\mathrm{ct}} compared to kernels in Table 7.1. This decreases their compute throughput; for example, 96×112×9696 \times112 \times96 for int8-int16 (Table 7.2) achieves 192.0 MACs/cycle, compared to 217.6 MACs/cycle of 64×216×6464 \times216 \times64 (Table 7.1). However, GEMM performance on the NPU array increases, since compute and memory become balanced. For example, when using the 96×112×9696 \times112 \times96 kernel, GEMM performance on XDNA for ∼4\sim4K GEMM size is 5.85 TOPS (Table 7.2) compared to 4.03 TOPS of 64×216×6464 \times216 \times64 (not shown in Table 7.1).

Precision In-OutKernel Size mct×kct×nctm_{ct} \times k_{ct} \times n_{ct}Product mct⋅nctm_{ct} \cdot n_{ct}Thrghpt. MACs/cycL1 Core Mem. (KB)L2 Total Mem. (KB)Peak TOPSGEMM Size M×K×NM \times K \times NActual TOPS
int8-int8112×112×112112 \times112 \times11212.3K212.561.3 (96%)980 (48%)6.804032 ×\times 4032 ×\times 40326.52
112×104×128112 \times104 \times12814.0K207.462.8 (98%)1004 (49%)6.634032 ×\times 4160 ×\times 40966.48
int8-int1696×112×9696 \times112 \times969.0K192.060.0 (94%)960 (47%)6.144224 ×\times 4032 ×\times 42245.85
80×104×12880 \times104 \times12810.0K186.962.3 (97%)996 (49%)5.984160 ×\times 4160 ×\times 40965.75
int8-int3280×88×9680 \times88 \times967.5K146.060.3 (94%)964 (47%)4.674160 ×\times 4224 ×\times 42244.42
64×80×12864 \times80 \times1288.0K133.162.0 (97%)992 (48%)4.264096 ×\times 4160 ×\times 40964.09
bf16-bf1696×56×9696 \times56 \times969.0K99.860.0 (94%)960 (47%)3.194224 ×\times 4032 ×\times 42243.12
96×48×11296 \times48 \times11210.5K97.360.0 (94%)960 (47%)3.114224 ×\times 4032 ×\times 40323.02

Table 7.2. Evaluation of two top-ranked solutions for XDNA across various data types (matrix BB in column-major).

Second, observe that when further increasing the product of mct⋅nctm_{\mathrm{ct}} \cdot n_{\mathrm{ct}} (thus decreasing kctk_{\mathrm{ct}} to fit in L1), GEMM performance on NPU array drops. For instance, for int8-int8, kernel 160×64×144160 \times64 \times144 has higher mct⋅nctm_{\mathrm{ct}} \cdot n_{\mathrm{ct}} product compared to 144×72×144144 \times72 \times144 (22.5K vs. 20.3K), but performance is lower: 36.13 vs. 37.35 TOPS (Table 7.3). This is because at this point GEMM has become compute bound, but compute has also become lower (throughput drops to 322.6 from 343.0 MACs/cycle). The peak compute throughput on the entire XDNA2 array when attaining single-core throughput of 322.6 MACs/cycle is 37.16 TOPS. However, the actual GEMM performance is 36.13 TOPS for ∼4\sim4K GEMM size, as shown in Table 7.3. This ∼3\sim3% difference is mainly attributed to DMA transfers of CC tiles (single output buffer), occurring at the end of each complete reduction across KK (every K/kctK/k_{\mathrm{ct}} tiles). Another smaller contribution comes from the vectorized zeroing kernel executing every K/kctK/k_{\mathrm{ct}} tiles (§ 7.1.2), which is typically <10% of GEMM kernel time (thus <0.15% for the ∼4\sim4K GEMM of this particular example). Both output DMA transfers and zeroing kernel cycles have been verified using NPU tracing. Note that as KK becomes higher, these effects are further amortized. Other contributions include the DRAM transfer time for initial AA and BB tiles, along with final output CC tiles, as well as overheads due to wall-clock time measurements, e.g., NPU dispatch time AMD [6]. In summary, the bolded solutions across all precisions in Table 7.2 and Table 7.3 represent the optimal balanced point between compute and memory, where end-to-end GEMM performance is maximized.

Precision In-OutKernel Size mct×kct×nctm_{ct} \times k_{ct} \times n_{ct}Product mct⋅nctm_{ct} \cdot n_{ct}Thrghpt. MACs/cycL1 Core Mem. (KB)L2 Total Mem. (KB)Peak TOPSGEMM Size M×K×NM \times K \times NActual TOPS
int8-int8144×72×144144 \times72 \times14420.3K343.060.8 (95%)2106 (51%)39.524032 ×\times 4320 ×\times 460837.35
160×64×144160 \times64 \times14422.5K322.660.5 (95%)2064 (50%)37.164480 ×\times 4224 ×\times 460836.13
int8-int16128×72×112128 \times72 \times11214.0K307.261.8 (97%)2084 (51%)35.394096 ×\times 4320 ×\times 448030.77
160×64×96160 \times64 \times9615.0K271.462.0 (97%)2016 (49%)31.264480 ×\times 4224 ×\times 460829.59
int8-int3296×64×9696 \times64 \times969.0K256.060.0 (94%)2016 (49%)29.494224 ×\times 4224 ×\times 460824.74
128×56×80128 \times56 \times8010.0K209.962.3 (97%)2036 (50%)24.184096 ×\times 4032 ×\times 448021.67
bf16-bf16112×48×96112 \times48 \times9610.5K137.260.0 (94%)2496 (61%)15.814032 ×\times 4224 ×\times 460814.52
160×40×80160 \times40 \times8012.5K124.162.5 (98%)2400 (59%)14.304480 ×\times 4160 ×\times 448013.67

Table 7.3. Evaluation of two top-ranked solutions for XDNA2 across various data types (matrix BB in column-major).

We note that results in Table 7.2, Table 7.3 correspond to matrix BB in column-major (results for BB in row-major are presented in § 7.3.1). Across all results, the contiguous parameter kmtk_{mt}, which is utilized to increase the effective DRAM BW (and hence GEMM performance), is configured as shown below (§ 7.3). Finally, the optimization procedure requires less than 30 minutes to identify the optimum solution for each data type (evaluated on the corresponding mini PCs). The overall execution time is dominated by the compilation time of xchesscc and IRON (up to 5 minutes per iteration), while the exhaustive search takes less than 1 s in all cases.

Contiguous kmtk_{\mathrm{mt}} Parameter

In this section, we present the impact of the contiguous kmtk_{\mathrm{mt}} parameter on GEMM performance. Figure 7.5 depicts the GEMM performance when varying the parameter kmtk_{\mathrm{mt}} and utilizing the optimal balanced kernels for two arbitrary data types (∼\sim4K GEMM size and BB in column-major). For instance, for XDNA, when kmtk_{\mathrm{mt}} is equal to kctk_{\mathrm{ct}} (= 56 in Figure 7.5a), GEMM performance is very low, i.e., 1.27 TOPS. As kmtk_{\mathrm{mt}} increases (in multiples of kctk_{\mathrm{ct}}) performance improves. This is because it enables higher effective DRAM BW, by traversing more contiguous elements for both AA (row-major) and BB (column-major) matrices. However, at some point, this enhancement becomes saturated, i.e., higher kmtk_{\mathrm{mt}} provides marginal improvement. Therefore, we empirically select the smaller value where GEMM performance becomes saturated for each data type, i.e., kmtk_{\mathrm{mt}}=224 in Figure 7.5a. Note that selecting the smallest value while also maintaining high GEMM performance is essential, since it reduces any potential zero padding needed to align the final GEMM dimensions with the native GEMM size (§ 7.1.2). For example, for the bf16-bf16 case, the native GEMM size operating natively on the entire 4×\times4 XDNA array is 384×\times224×\times384. We note here that this results in reduced L2 memory utilization, ranging from 47–61% across all data types (Table 7.2 & Table 7.3). While maximized L2 memory usage can be attained for higher kmtk_{\mathrm{mt}} values (e.g., 96% for kmtk_{\mathrm{mt}}=560 in Figure 7.5a), this only leads to marginal performance improvement in GEMM (<1%).

Figure 7.5. GEMM performance while varying parameter kmtk_{\mathrm{mt}} for bf16-bf16 96×56×9696 \times56 \times96 (a) and int8-int16 128×72×112128 \times72 \times112 (b).

Similarly, for XDNA2, for int8-int16, we set kmtk_{\mathrm{mt}}=432 (Figure 7.5b). In this case, the native GEMM size on the XDNA2 array becomes 512×\times432×\times896. We note here that the three latest points of Figure 7.5b are enabled via the neighboring memory sharing in MemTiles, as a direct result of the GEMM mapping strategy on XDNA2

(§ 7.1.3). For all other data types, we set the parameter kmtk_{\mathrm{mt}} in a similar fashion. Specifically, for XDNA, we set kmtk_{\mathrm{mt}} equal to 448 for int8-int8 and int8-int16, while for int8-int32 we set it to 352. On XDNA2, we use 432 for int8-int8, and 384 for both int8-int32 and bf16-bf16.

It is important to mention here that the non-optimized GEMM example in [6], cannot support sufficient contiguous elements. For a fair comparison, we utilize our optimized balanced kernels and modify their implementation to support single output CC buffers (allowing it to fit in L1). For instance, for the data types shown in Figure 7.5, we attain 2.4×2.4 \times and 3.6×3.6 \times higher performance for XDNA and XDNA2, respectively. These results highlight the importance of accessing sufficient contiguous elements in GEMM performance.

Roofline GEMM Performance Sweeps

In Figure 7.6 we show roofline GEMM performance sweeps (in linear scale) for various data types. Each point represents a matrix size that is a multiple of the native GEMM size (using the optimal balanced kernel). We select more than 400 points for each case (separately for BB in column- and row-major), up to 8K-

Figure 7.6. Roofline GEMM performance sweeps for various matrix sizes on XDNA & XDNA2.

sized matrices, without favoring any particular MM, KK, NN dimension. First, notice that when arithmetic intensity (ARI) is low, i.e., small matrix sizes, performance is limited. In this case, GEMM is severely memory bound. However, as ARI increases, GEMM performance improves, and becomes more stabilized after a specific value. In all cases, storing matrix BB in column-major provides, on average, higher performance compared to row-major. This is because of accessing sufficient contiguous data for both AA and BB matrices (determined by kmtk_{\mathrm{mt}} parameter). However, for BB in row-major, the contiguous access is limited to the nctn_{\mathrm{ct}} parameter, while only for AA (row-major), kmtk_{\mathrm{mt}} contiguous data are traversed. To this end, for XDNA, we observe, on average, 4.8%, 4.4%, and 0.57% higher performance, for int8-int8, int8-int16, and bf16-bf16, respectively.

Moreover, we notice that for XDNA2 the difference between column- and row-major is higher. In particular, we observe, on average, 19.1%, 25.2%, and 8.7% higher performance, for int8-int8, int8-int16, and bf16-bf16, respectively. This difference between XDNA and XDNA2 is presumably attributed to complex interaction between the NPU NoC, the SoC-level fabric and DRAM Rico et al. (2024); Subramon et al. (2023), which affects the effective DRAM BW that NPU perceives (although both mini PC devices are equipped with the same DRAM technology). Also, we note that for both XDNA and XDNA2, the difference between column- and row-major for bf16 is lower compared to int8. This is because of accessing a larger number of bytes for bf16 across the nctn_{\mathrm{ct}} dimension (when BB is in row-major), which increases GEMM performance in this case, thereby lowering their difference.

For XDNA, we note that performance gets stabilized after a specific ARI value for BB in both row- and column-major formats (almost resembling a line in Figure 7.6a, 7.6b, 7.6c). Moreover, for XDNA2, we observe that for BB in column-major, GEMM performance also resembles a stable line. However, for BB in row-major it displays a more scattered distribution (Figure 7.6d, 7.6e, 7.6f). This is because XDNA2 has higher reliance on the effective DRAM BW (due to attaining significantly higher absolute TOPS values), which is substantially increased and stabilized when accessing sufficient contiguous data across both AA and BB matrices. For example, for int8-int16 on XDNA2, we measure a variability of only 5% for BB in column-major, while for row-major the variability is 19% (ARI>1600\mathrm{ARI} > 1600). Finally, across all points in GEMM sweeps, XDNA attains up to 6.76 (int8-int8), 6.05 (int8-int16), 4.57 (int8-int32), and 3.14 (bf16-bf16) TOPS. Similarly, XDNA2 achieves up to 38.05 (int8-int8), 31.52 (int8-int16), 25.31 (int8-int32), and 14.71 (bf16-bf16) TOPS (int8-int32 sweep omitted for brevity).

Insights & Discussion

Performance Across Multiple GEMM Sizes in DL Workloads

Modern DL workloads perform GEMMs with a wide range of sizes across their layers. Our employed output stationary mapping allows arbitrary GEMM dimensions to be supported, by applying zero-padding to align with the native GEMM size (§ 7.1.2). Zero-padding can be applied efficiently by utilizing the NPU’s architectural support for on-the-fly zero-padding in MemTile channels AMD (2024a); leveraging this feature is left for future work.

Switching between different GEMM sizes can incur critical performance inefficiencies. To this end, one approach could be to reconfigure the NPU array with a dedicated GEMM design for each size. When quantifying the reconfiguration latency of the entire GEMM design, we measure a delay of 3.4 ms and 4.9 ms on XDNA and XDNA2, respectively. However, this reconfiguration latency is comparable to the GEMM execution time (e.g., a ∼4K\sim4\mathrm{K} square GEMM for int8-int16 on XDNA2 takes 5.2 ms). This underscores that reconfiguring the entire design can impose substantial overheads.

When retaining the same GEMM design on the NPU, only two parameters require reconfiguration across different problem sizes (MM, KK, NN): (i) the total number of output tiles M⋅N/(mct⋅nct)M \cdot N/(m_{\mathrm{ct}} \cdot n_{\mathrm{ct}}), and (ii) the number of tiles across the reduction dimension K/kctK/k_{\mathrm{ct}} Rösti and Franz (2025). According to our measurements, this negligible reconfiguration does not incur any noticeable performance overhead at the system level. Hence, identifying the optimal parameters, i.e., mctm_{\mathrm{ct}}, kctk_{\mathrm{ct}}, nctn_{\mathrm{ct}}, kmtk_{\mathrm{mt}}, § 7.2.2, and reusing them across different GEMM sizes is essential for high-performance DL deployment. Finally, we note that the system-level GEMM results presented are specific to the mini PCs used in this evaluation. However, our optimization methodology is generalizable to any NPU device in the current two generations.

Single Output Buffer

Due to output stationary mapping, we retain the output CC tiles as single buffers and apply double-buffering only for inputs AA and BB (§ 7.1.2). This is crucial in maximizing performance in the general GEMM case, because it enables significantly higher flexibility in single-core parameter optimization. In particular, it allows increased values for mct×kct×nctm_{\mathrm{ct}} \times k_{\mathrm{ct}} \times n_{\mathrm{ct}} parameters, compared to utilizing double-buffering (constrained by L1 memory size). This results in identifying a balanced kernel that has higher performance, thereby enabling higher GEMM performance at the system level. For example, when using the optimization methodology described in § 7.1.6.2 and apply double-buffering for CC, we identify the 112×48×96112 \times48 \times96 size as the optimal balanced kernel for int8-int16 data type on XDNA2. For ∼4K\sim4\mathrm{K} square GEMM, this kernel provides 26.1 TOPS. However, the 128×72×112128 \times72 \times112 kernel of Table 7.3 (single CC buffer), provides 30.77 TOPS, representing an 18% performance improvement. Similarly, on XDNA, double-buffering on CC provides 2.76 TOPS (80×40×9680 \times40 \times96 kernel), while single buffer offers 3.12 TOPS (96×56×9696 \times56 \times96 kernel on Table 7.2, 13% higher). The transfer of the output CC tiles in the single buffer case gets amortized as the reduction KK dimension becomes sufficiently high (typically <5%<5\% degradation in GEMM performance when K/kct>20K/k_{\mathrm{ct}}>20).

NPU–DRAM Data Movement & BD Reconfiguration

The procedure delineated in § 7.1.5 enables efficient overlapping of NPU–DRAM data movement with BD reconfiguration. To quantify its impact on performance, we modify our design to synchronize and reconfigure BDs sequentially (without overlap). In this case, for int8-int16 on XDNA2, we notice only 22.21 TOPS at ∼4K\sim4\mathrm{K} square GEMM, while the overlapped design of Table 7.3 exhibits 30.77 TOPS (28% decrease for non-overlapped design). Similarly, for XDNA, for int8-int16 (Table 7.2), we observe a 27% degradation in performance. This highlights the critical role of overlapping DMA transfers with BD reconfiguration in attaining high GEMM performance.

Chapter Summary & Impact

In this chapter, we present a methodology to maximize GEMM performance on Ryzen AI NPUs. We observe an inverse relationship between compute and off-chip memory utilization and identify the optimal balance point at which performance is maximized, using a combination of analytical modeling and hardware profiling. Our methodology attains state-of-the-art GEMM performance and generalizes across both current AMD Ryzen AI NPU generations, achieving throughput of up to 6.76 TOPS (XDNA) and 38.05 TOPS (XDNA2) for int8 precision. Similarly, for bf16 precision, our GEMM implementations attain up to 3.14 TOPS (XDNA) and 14.71 TOPS (XDNA2). This study provides key insights into the performance aspects that govern GEMM optimization on reconfigurable NPU architectures.

Chapter 8: Conclusion and Future Work

Modern DL workloads are predominantly deployed on GPUs and specialized ASIC accelerators. However, their rapid evolution demands platforms that combine high efficiency with architectural flexibility. Reconfigurable architectures are well positioned to offer this combination, yet their large-scale adoption for DL acceleration has yet to be fully realized. This dissertation addresses this gap through a comprehensive evaluation of three prominent AI-optimized reconfigurable platforms: the Versal ACAP, Intel Stratix 10 NX, and AMD Ryzen AI NPUs. Using GEMM as the primary workload, this evaluation yields insights into their performance and efficiency characteristics. Building on these insights, this dissertation proposes architectural enhancements that enable efficient acceleration of both dense and sparse DL workloads. The proposed in-fabric sparsity support addresses a key gap in the literature on structured sparse acceleration. Together, these contributions advance the case for reconfigurable architectures as an efficient and adaptable platform for DL acceleration.

Summary of Contributions

First, we target the AMD Versal AIE, whose array of programmable vector processors introduces a new paradigm for GEMM acceleration on reconfigurable hardware. We propose MaxEVA, a framework to map and optimize GEMM workloads onto the AIE by exploiting its architectural features and resolving performance bottlenecks from multiple angles. On the VC1902 device, MaxEVA reaches up to 5.44 TOPS (fp32) and 77.01 TOPS (int8) throughput, outperforming the prior state-of-the-art by up to 2.19×2.19 \times in throughput and 20.4% in energy efficiency, and establishes the foundation the remaining work builds on.

Second, we extend this foundation in two ways: we add a memory hierarchy level to MaxEVA to support arbitrary matrix sizes beyond what fits in on-chip memory resources of the Versal FPGA, and we introduce a parallel framework for the Intel Stratix 10 NX, whose in-fabric tensor blocks take a fundamentally different architectural approach from the AIE. The two frameworks achieve up to 77 and 68 TOPS (int8) throughput, with up to 0.94 and 1.35 TOPS/W energy efficiency, for Versal and Stratix respectively. Notably, Stratix 10 NX delivers ∼44%\sim44\% higher energy efficiency, suggesting that in-fabric hardened compute blocks are a promising path to efficient FPGA acceleration for DL. Direct comparison also yields practical insights and guidelines on GEMM optimization, programmability, architectural attributes, and limitations across both platforms.

Third, building on the efficiency advantage of in-fabric hardened AI blocks, we address a key limitation of existing architectures: these blocks are optimized primarily for dense computation and cannot efficiently exploit the structured sparsity present in most modern DL workloads. To this end, we architect the SSTs, in-fabric hard blocks that support 2:4 and 1:4 structured sparsity, as well as a new 1:3 sparsity mode, in addition to dense operation. Applied to GEMM accelerators, SST-based designs achieve up to 5×5\times higher FPGA frequency and 10.9×10.9\times lower area than traditional FPGA logic, with up to 3.52×3.52\times speedup on sparse ViT and CNN models at minimal area cost of up to 13.3%. Where the first two contributions optimize dense GEMM on existing devices, this work proposes a new hardware architecture to close this important sparsity gap.

Finally, we extend this work beyond FPGAs to AMD’s Ryzen AI NPUs, a direct architectural evolution of the Versal AIE. We present a common methodology for optimizing GEMM across both NPU generations, XDNA and XDNA2, achieving up to 6.76 and 38.05 TOPS (int8), and 3.14 and 14.71 TOPS (bf16), respectively. While many optimization principles from earlier work carry over to this architecture, the NPU’s distinct execution model and system-level constraints require a new methodology tailored specifically to XDNA and XDNA2, underscoring that generalizing across reconfigurable AI hardware demands adaptation rather than direct reuse.

Future Work

First, for MaxEVA, the proposed mapping methodology extends directly to other special cases of GEMM, such as matrix-vector multiplication (GEMV), which we leave as future work. Moreover, for the Versal vs. Stratix comparison, future work could broaden the evaluation to include actual end-to-end DL workloads rather than GEMM alone, as well as additional devices, to provide a more holistic view of the advantages and disadvantages of each platform.

Second, for the SST slices, our current design supports static sparsity targeting DNN weights; future work could extend this support to dynamic sparsity in activations. Additional modes could further increase flexibility, such as element-wise computation within SSTs, similar to those explored in prior tensor-slice designs Arora et al. (2021b, 2022b). GEMV is already supported directly by the SSTs at full utilization with a batch size of four. Improving utilization at a batch size of one is possible by adding multiplexing logic, as shown in Arora et al. (2022b), and is left for future work. Future work could explore multiple input crossbars to further reduce the SSTs’ area overhead, as well as a broader architectural study of area allocation and the trade-offs between area and FPGA routability introduced by the SST routing circuitry. Finally, future work could pursue an in-depth exploration of the architectural placement and organization of FPGA building blocks, e.g., logic and memory blocks, SST slices, etc., to optimize the architecture for DL workloads without significantly degrading the performance of non-DL workloads.

Third, for the GEMM optimization methodology on Ryzen AI NPUs, XDNA2 adds hardware support for bfp16 precision, in which multiple values share a common exponent. This introduces additional challenges for data layout transformations that exploit the NPU’s multi-dimensional DMA addressing, which we leave for future work. Additionally, similar to the other works in this dissertation, the proposed optimization methodology extends directly to GEMV, which we also leave as future work.

Throughout this dissertation, we deliberately focus on GEMM as the primary workload. As the dominant computational kernel in most modern DL models, it provides a natural and representative basis for studying and optimizing reconfigurable architectures for DL. A key direction for future work is to build on the architectural insights and optimization techniques developed here to support full end-to-end DL workload deployment on these devices, extending beyond GEMM optimization to complete model execution.

Works Cited

References

  1. [1]Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. In-Datacenter Performance Analysis of a Tensor Processing Unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, ISCA ’17, page 1–12, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450348928. doi: 10.1145/3079856.3080246. URL https://doi.org/10.1145/3079856.3080246.
  2. [2]Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU, 2023.
  3. [3]NVIDIA. NVIDIA A100 Tensor Core GPU Architecture. https: //images.nvidia.com/aem-dam/en-zz/Solutions/data-center/ nvidia-ampere-architecture-whitepaper.pdf, 2020.
  4. [4]NVIDIA. NVIDIA H100 Tensor Core GPU. https://resources.nvidia. com/en-us-tensor-core/nvidia-tensor-core-gpu-datasheet, 2023.
  5. [5]AMD CDNA 3 Architecture. https://www.amd.com/content/ dam/amd/en/documents/instinct-tech-docs/white-papers/ amd-cdna-3-white-paper.pdf, 2023a.
  6. [6]AMD. AMD CDNA 4 Architecture. https://www.amd.com/ content/dam/amd/en/documents/instinct-tech-docs/white-papers/ amd-cdna-4-architecture-whitepaper.pdf, 2025a.
  7. [7]Norman P. Jouppi, Doe Hyun Yoon, Matthew Ashcraft, Mark Gottscho, Thomas B. Jablin, George Kurian, James Laudon, Sheng Li, Peter Ma, Xiaoyu Ma, Thomas Norrie, Nishant Patil, Sushma Prasad, Cliff Young, Zongwei Zhou, and David Patterson. Ten Lessons From Three Generations Shaped Google’s TPUv4i : Industrial Product. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), pages 1–14, 2021. doi: 10.1109/ISCA52012.2021.00010.
  8. [8]Amin Firoozshahian, Joel Coburn, Roman Levenstein, Rakesh Nattoji, Ashwin Kamath, Olivia Wu, Gurdeepak Grewal, Harish Aepala, Bhasker Jakka, Bob Dreyer, Adam Hutchin, Utku Diril, Krishnakumar Nair, Ehsan K. Aredestani, Martin Schatz, Yuchen Hao, Rakesh Komuravelli, Kunming Ho, Sameer Abu Asal, and et al. MTIA: First Generation Silicon Targeting Meta’s Recommendation Systems. In Proceedings of the 50th Annual International Symposium on Computer Architecture, ISCA ’23, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400700958. doi: 10.1145/3579371.3589348. URL https://doi.org/10.1145/3579371.3589348. Goya Inference Platform White Paper. https://habana.ai/wp-content/uploads/pdf/habana_labs_goya_whitepaper.pdf, 2019.
  9. [10]Martin Langhammer, Eriko Nurvitadhi, Bogdan Pasca, and Sergey Gribok. Stratix 10 NX architecture and applications. In The 2021 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pages 57–67, 2021a.DOI
  10. [11]Brian Gaide, Dinesh Gaitonde, Chirag Ravishankar, and Trevor Bauer. Xilinx Adaptive Compute Acceleration Platform: VersalTM Architecture. FPGA ’19, page 84–93, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450361378. doi: 10.1145/3289602.3293906. URL https://doi.org/10.1145/3289602.3293906.
  11. [12]Eriko Nurvitadhi, Jeffrey Cook, Asit Mishra, Debbie Marr, Kevin Nealis, Philip Colangelo, Andrew Ling, Davor Capalija, Utku Aydonat, Aravind Dasu, and Sergey Shumarayev. In-Package Domain-Specific ASICs for Intel Stratix 10 FPGAs: A Case Study of Accelerating Deep Learning Using TensorTile ASIC. In 2018 28th International Conference on Field Programmable Logic and Applications (FPL), pages 106–1064, 2018. doi: 10.1109/FPL.2018.00027.DOI
  12. [13]E. Nurvitadhi, D. Kwon, A. Jafari, A. Boutros, J. Sim, P. Tomson, H. Sumbul, G. Chen, P. Knag, R. Kumar, R. Krishnamurthy, S. Gribok, B. Pasca, M. Langhammer, D. Marr, and A. Dasu. Why Compete When You Can Work Together: FPGA-ASIC Integration for Persistent RNNs. In 2019 IEEE 27th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), pages 199–207, 2019.DOI

Paper details

vlnc:W_7raw384y
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents