betaFeedback

WorksPhysics and Astronomy

Hybrid programming-model strategies for GPU offloading of electronic structure calculation kernels

Abstract

To address the challenge of performance portability, and facilitate the implementation of electronic structure solvers, we developed the Basic Matrix Library (BML) and Parallel, Rapid O(N) and Graph-based Recursive Electronic Structure Solver (PROGRESS) libraries. BML implements linear algebra operations necessary for electronic structure kernels using a unified user interface for various matrix formats (dense, sparse) and architectures (CPUs, GPUs). Focusing on Density Functional Theory (DFT) and Tight-Binding (TB) models, PROGRESS implements several solvers for computing the single-particle density matrix and relies on BML. In this paper, we describe the general strategies used for these implementations on various computer architectures, using OpenMP target functionalities on GPUs, in conjunction with third-party libraries to handle performance critical numerical kernels. We demonstrate the portability of this approach and its performance on benchmark problems.

Introduction

Performance portability is a significant challenge for application programs that are run on modern HPC resources. For example, software solutions targeting portability such as OpenMP sometimes can have a hard time delivering performance, either due to the lack of maturity of compilers or the fine granularity of control needed for some specific kernels. On the other hand, writing kernels in a vendor specific language targeting one specific GPU may not be portable and can lead to software maintenance difficulties. Recent trends in HPC only make this problem more acute, as many leadership computing facilities have adopted hardware composed of heterogeneous compute nodes containing CPUs and GPUs, where a majority of the acceleration is provided by the GPUs [1, 2, 3].

In practice, it is usually best to use existing libraries when possible if those libraries implement the numerical kernels one needs. For electronic structure applications, linear algebra libraries are the most common dependency. For dense linear algebra on CPUs, standard interfaces developed for BLAS [4, 5, 6] and Lapack [7] have facilitated the use and development of these libraries, and several well optimized solutions exist. On GPUs, the situation is more complicated. Vendors offer optimized implementations of BLAS and Lapack in platform specific libraries such as cuBLAS, cuSolver (Nvidia), rocBLAS, rocSolver (AMD) and MKL (Intel). There is however no common interface to these libraries. As a consequence, application codes need to have platform specific wrappers around the functions they intend to use. In addition, application developers need to understand the details of all these interfaces. There are several reasons why the GPU situation is not as user friendly as it is on CPUs. First, there is the choice of having two possible locations for data arrays passed as arguments, either allocated on the host or the device, as well as for the return value, when there is one. Then there is also the option of enabling several kernels to execute asynchronously on the device, using for instance GPU streams. When writing platform specific GPU kernels, there also can be extra kernel arguments depending on the architecture to ensure optimality such as, for instance, different hardware/run time parameters, different thread-block grid sizes or user controlled cache (shared memory) sizes. As a result of these various issues and GPU technology changes, user interfaces are not as stable as one would like. For sparse linear algebra, the situation is also complicated. Unlike the dense format for which there are not too many ways of laying out the data, there are several sparse formats including compressed sparse row (CSR), compressed sparse column (CSC), coordinate list (COO), and ELLPACK just to mention a few. In addition, even for a given format such as CSR, there are variants, as some libraries may or may not expect data in a row to be ordered by column indexes.

Some open source alternatives have emerged in recent years, some of which are targeting multiple architectures and thus facilitating portability of application codes using those. At the node level, the MAGMA library [8, 9] offers the functionalities of BLAS and Lapack on GPUs, and is already fully functional on Nvidia and AMD GPUs. The SLATE project is implementing a distributed and scalable dense linear algebra library for distributed memory accelerator-based computer systems, aiming to provide performance and portability to various hardware (CPUs, GPUs, accelerators) [10]. On the sparse matrix side, Ginkgo[11] is being actively developed and already offers a lot of functionalities on GPU architectures.

With the development of the Parallel, Rapid O(N)O(N) and Graph-based Recursive Electronic Structure Solver (PROGRESS) and the Basic Matrix Library (BML), our goal is to facilitate the development of performant and portable electronic structure solvers by providing the necessary linear algebra tools in a hardware agnostic way. By providing several matrix formats, specifically a dense format and several sparse formats, BML facilitates the development of reduced complexity algorithms that can exploit any possible sparsity of the Hamiltonian and density matrix. We reported on this concept and the BML library a few years ago in Ref. [12], with a focus on CPU implementations. In this paper we extend this concept to implementations on GPUs, demonstrating some of these ideas on various hardware such as Nvidia V100, AMD MI250X and Intel GPUs. We describe in particular the implementation model used to offload calculations to GPUs, using OpenMP in combination with third-party libraries. Electronic structure calculations are an important class of applications that require heavy use of linear algebra kernels. Here, electronic structure calculations refers broadly to the many ways of numerically evaluating the state of electrons in a physical system (molecule, periodic solid, etc.) as necessary to derive other physical quantities of interest. In this paper we will restrict our discussion to mean-field models such as Density Functional Theory (DFT) or Tight-Binding methods. Some algorithms and implementations discussed are targeting large scale simulations and make use of matrix sparsity to reduce computational complexity to O(N)O(N). Moreover, fast time-to-solution is also of high interest in the community, specifically to speedup wall-clock times in quantum molecular dynamics (QMD) and enable better modeling with longer time-scales for medium-size systems, on the order of 1,000 electrons.

The idea of isolating all the linear algebra operations of an electronic structure code into a separate library is a natural design choice, and at the same time allows for multiple application codes to share this implementation. Several other research groups have made efforts towards identifying and isolating software libraries and have made them available to the community. The DBCSR library[13, 14], which the CP2K simulation package[15] relies on, is designed to efficiently perform sparse matrix-matrix multiplication, among other operations. It provides a distributed implementation using MPI and runs on Nvidia and AMD GPUs via CUDA and HIP, respectively. The Electronic Structure Infrastructure (ELSI) project [16] provides an open-source software interface to facilitate the implementation and optimal use in electronic structure codes of high-performance solver libraries including traditional eigensolvers, O(N)O(N) complexity algorithms, and other reduced complexity methods. The Electronic Structure Library (ESL) [17] is a community-maintained library of software specific to electronic structure simulations, which includes among others the ELSI library just mentioned. On the application side, an example of an electronic structure code that recently embraced this separation of operations and the use of more libraries is SIESTA [18].

After introducing the PROGRESS and BML libraries in Section II, we discuss in § 3 the specific problem of electronic structure these libraries are targeting: computing the single-particle density matrix (DM). We then discuss in § 4 some GPU-friendly algorithms implemented in PROGRESS as possible alternatives to a direct dense diagonalization. In § 5, we describe our general strategy to offload computational kernels to the GPU using OpenMP. We then describe some more specific strategies used for the dense matrix format in § 6 (using the MAGMA and MKL libraries) and sparse matrix format in Section VII (using AMD roc-Sparse library and the Hypre library). Finally in Section VIII, we discuss distributed memory solvers that leverage the shared memory solvers discussed in previous sections.

PROGRESS and BML libraries

The Basic Matric Library (BML) is designed to implement the linear algebra operations necessary to implement matrix-based electronic structure solvers. Its purpose is to hide all the implementation details of numerically intensive kernels, including architecture specific code and interfaces with third-party libraries, from the user.

BML is written in C. This facilitates interoperability with other languages such as Fortran and C++. BML supports four datatypes for all operations: single precision, double precision, single complex, double complex. To avoid writing essentially the same code for all the data types, C macros are used extensively for data types and in function names, and the C preprocessor is used to generate specifics code associated with each data type. To avoid naming conflicts with other packages, all BML function names use a prefix bml_.

BML supports four different matrix formats: dense, ELLPACK-R, CSR and ELLBLOCK. ELLPACK-R is a sparse format with a fixed memory allocation allowing a pre-determined maximum number MM of non-zero elements per row [19]. It is less adaptable than CSR since a growing number of non-zeroes may reach the limit $M and lead to a failure. But it has the performance advantage of avoiding a lot of memory allocation/deallocation during routine usage. ELLBLOCK is a generalization of ELLPACK-R where each element is a block[20]. Note that BML CSR format implementation is not exactly what is usually referred to as CSR. In BML, rows of data and column indexes are stored independently of each other instead of being packed together into a single array. This facilitates memory reallocation when the number of non-zeroes in a row changes. In addition, BML supports a memory distributed format, “distributed2d” which builds on top of the four non-distributed formats.

BML matrices are stored as a C struct, which includes, besides a pointer to the data storage, other parameters necessary to fully describe that matrix, such as its type (e.g. dense or ELLPACK-R), its data type (float, double, complex), and the number of rows in the matrix. An example for an ELLPACK-R matrix is shown in Listing 1.

struct bml_matrix_ellpack_t
{
    /* matrix type identifier. */
    bml_matrix_type_t matrix_type;
    /* data type */
    bml_matrix_precision_t matrix_precision;
    /* number of rows. */
    int N;
    /* number of columns in storage */
    int M;
    /* matrix elements */
    void *value;
    /* column indexes */
    int *index;
    /* number of non zeros per row */
    int *nnz;
    ...
}

Listing 1: C struct used for ELLPACK-R matrix storage

The PROGRESS (Parallel, Rapid O(N) and Graph-based Recursive Electronic Structure Solver) library is a collection of algorithms used in electronic structure calculations, with a focus on iterative solvers based on matrix polynomials. It is written mostly in FORTRAN, but also offers a C-interface for routines expected to be called directly by an application code. More specifically, it implements several versions of the “second-order spectral projection” (SP2) DM solver[21], a Chebyshev polynomial expansion of the DM[22], as well as some iterative methods to compute the inverse square root of an overlap matrix as is often necessary in a non-orthogonal basis set or a non-orthogonal tight-binding model. All these implementations rely on BML matrices and their functionalities. They are mostly matrix format agnostic, and available for all data types available in BML. Several of these algorithms show an O(N)O(N) computational complexity with the matrix size NN when sparse matrix formats are used, and an appropriate threshold is used to discard small matrix elements. Figure 1 shows the software stack for a typical application using PROGRESS and BML, including third-party dependencies. The PROGRESS and BML library are both open source, licensed under the BSD 3-clause license, and available on GitHub[23, 24].

Software stack showing the PROGRESS and BML libraries and their integration within an electronic structure application

Figure 1. Software stack showing the PROGRESS and BML libraries and their integration within an electronic structure application.

To evaluate a solver’s performance, we developed some benchmark drivers within PROGRESS. Our main benchmark test is based on a physical system, a small peptide chain solvated in water with periodic boundary conditions (Figure 2). It consists of 523 atoms. From this atomic configuration, we build a tight-binding Hamiltonian represented by a matrix of size 1081×10811081 \times1081. To construct this Hamiltonian we call a Density Functional Based Tight Binding (DFTB)[25] Hamiltonian builder implemented in PROGRESS that uses DFTB parameters from the LATTE (Los Alamos Transferable Tight-binding for Energetics) code[26]. We build larger Hamiltonians by replicating this system by a factor of two or three in each direction. This gives us a series of Hamiltonians of increasing size to study computational cost, computational complexity, and parallel scaling. The sparsity of each DM for these benchmark problems are shown in Tab. I. This resulting “soft matter” system is what one would typically encounter in a biophysical molecular dynamics (MD) simulation.

Small peptide chain solvated in water

Figure 2. Small peptide chain solvated in water, used for PROGRESS benchmarks. The figure was created using PyMOL[27].

PROGRESS and BML rely on the CMake build system[28]. To facilitate the development of these libraries, and avoid breaking the code when changes are made by developers not familiar with all the functionalities and implementation details, an extensive suite of unit tests has been developed over time and continue to be enhanced. These tests cover all the matrix formats and matrix datatypes, and are run through Ctest, the testing driver provided by CMake.

ReplicasNNDM sparsity
1×1×11,08157.5%
2×1×12,16278.7%
3×1×13,24385.8%
2×2×14,32489.3%
3×2×16,48692.9%
2×2×28,64894.6%
3×3×19,72995.3%
3×2×212,97296.4%
3×3×219,45897.6%
3×3×329,18798.4%

Table 1. Percentage of zero-valued elements in DM in PROGRESS benchmark problem, using a cutoff threshold of 10−610^{-6}.

Single-particle density matrix and the associated eigenvalue problem

Given a symmetric (or Hermitian) N×NN \times N matrix HH representing a Hamiltonian operator in a finite basis set, the task of computing the single-particle density matrix DD in that same basis set can be accomplished by following a straightforward procedure

  1. Compute all the eigenvalues ϵi\epsilon_i and eigenvectors vi\mathbf{v}_i of HH.

  1. Given a chemical potential μ\mu, the Fermi-Dirac distribution function is given by

fμ(ϵ)=11+exp⁡(β(ϵ−μ)).(1)f_\mu(\epsilon) = \frac{1}{1+\exp(\beta(\epsilon-\mu))}. \tag*{(1)}
  1. The single particle density matrix is given by

D=V(fμ(ϵ1)fμ(ϵ2)⋱fμ(ϵN))VT,(2)D = V \begin{pmatrix} f_\mu(\epsilon_1) & & \\ & f_\mu(\epsilon_2) & \\ & & \ddots\\ & & & f_\mu(\epsilon_N) \end{pmatrix} V^{T}, \tag*{(2)}

where VV is the N×NN \times N matrix made of the ordered and orthonormal eigenvectors

V=(v1v2…vN).V = (\mathbf{v}_1 \mathbf{v}_2 \ldots\mathbf{v}_N).

Note that if there is a gap in the eigenvalue spectrum, and no eigenvalue close to μ\mu, fμ(ϵ)f_\mu(\epsilon) takes either the value 1 (ϵ<μ\epsilon< \mu) or 0 (ϵ>μ\epsilon> \mu), provided β\beta is not too small. In this case we have an insulator. The value μ\mu in practice can be determined by setting an occupation (number of electrons) and finding the μ\mu which results in this occupation through an iterative process.

In practice, the Hamiltonian matrix can come, for example, directly from a discretization of the problem in a small basis set (such as a set of Gaussian-shaped orbitals centered on the atoms), through a parameterized tight-binding (TB) approximation, or from the projection of the Hamiltonian operator onto an auxilliary set of wave functions built iteratively during the search for the numerical solution in a larger numerical basis set (e.g. plane waves, finite elements or finite difference cases). Depending on the discretization of the problem and the solver adopted, the size and degree of sparsity of this matrix HH can vary significantly.

The matrix size also depends on the algorithm used to solve the electronic structure problem. When working with wave-functions, NN can be substantially smaller than in a TB or Gaussian-based approach. It can be as small as the number of occupied orbitals, as in a Conjugate Gradient solver[29], in which case no diagonalization is needed for the projected Hamiltonian (all states are fully occupied). But wavefunction-based solvers often use NN larger than the number of occupied orbitals in order to speedup the solver, or to solve for partial occupancy in metallic systems[30]. In addition, solvers such as Block-Davidson will involve solving a Rayleigh-Ritz problem, often referred to in the field of electronic structure as subspace diagonalization, for 2N×2N2N \times2N matrices[31, 32, 33]. A Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) solver[34] will involve an even larger space, with a Rayleigh-Ritz procedure in 3N×3N3N \times3N matrix space.

Note that for wavefunction-based approaches, distribution of the wavefunction over nodes and cores can substantially reduce time-to-solution. However, the Rayleigh-Ritz process used to compute the orbitals occupation does involve information from all the distributed parts, so that the resulting synchronization communication step is often the bottleneck in the strong scaling limit [35, 36], and thus requires efficient algorithms to solve that problem.

GPU friendly algorithms

Matrix multiplications have two advantageous properties when it comes to their implementations on GPUs: (i) very simple arithmetic operations, (ii) high arithmetic intensity (floating point operations per memory load). When compared to the operations involved in solving a dense eigenvalue problem on a GPU, the use of standard dense diagonalization algorithms are typically not very efficient. Thus solvers based on matrix-matrix multiplications are able to better utilize the massively parallel threads on a GPU and may offer better performance [20, 37, 38].

Significant development in matrix-matrix multiplication based iterative solvers for the density matrix happened in the 1990’s. Their primary purpose was to reduce algorithmic complexity from O(N3)O(N^3) to O(N)O(N) by utilizing the sparsity in the Hamiltonian matrix and in functions of the Hamiltonian matrix (see Ref. 39 for a review). The key idea is to replace the diagonalization of the Hamiltonian matrix to evaluate the Fermi-Dirac function (see previous Section) with a much cheaper polynomial approximation that one can easily apply to a Hamiltonian matrix. Often in these approaches, we define a shifted and rescaled Hamiltonian matrix H~\widetilde{H} such that the eigenvalues of this matrix are all inside the interval [a,b][a,b]. A good polynomial approximation would then map the interval [a,b][a,b] to a Fermi-Dirac function with the appropriate chemical potential μ\mu so that

D≈pμ(H~),(3)D \approx p_{\mu}(\widetilde{H}) , \tag*{(3)}

where the subscript of the polynomial pp denotes the dependence on the chemical potential μ\mu. Another type of approximate expansion is a recursive polynomial expansion

D≈pn(pn−1(⋯p2(p1(H~))),(4)D \approx p_n(p_{n-1}(\cdots p_2(p_1(\widetilde{H}))) , \tag*{(4)}

so that the polynomial pμp_{\mu} is replaced by a composition of generally simpler or lower-order polynomials: pμ=pn∘pn−1∘⋯∘p2∘p1p_{\mu}=p_n \circ p_{n-1}\circ\cdots\circ p_2\circ p_1.

Although electronic structure problems can be quite large, domain scientists are often limited to solving problems of more modest sizes for which a very fast time-to-solution can be achieved. This is usually the case of quantum molecular dynamics where an electronic structure problem needs to be solved at each timestep to accurately compute atomic forces and propagate the atoms along the MD trajectories. In these more moderately-sized problem cases, even matrix multiplications cannot always fully utilize the available resources on GPU devices and substantial portions of the GPU remain idle. Therefore finding further parallelism in the evaluation of these polynomials is beneficial (see § 4 B).

SP2 solver

An example of a recursive expansion is the “second-order spectral projection” (SP2) algorithm[21] as implemented in PROGRESS. In SP2, one starts with

D0≡H~=εNI−HεN−ε1,(5)D_0 \equiv\widetilde{H} = \frac{\varepsilon_N I-H}{\varepsilon_N-\varepsilon_1}, \tag*{(5)}

the shifted and rescaled Hamiltonian, after which, the density matrix is computed iteratively using the recursion

Dm+1=Dm2(6)D_{m+1}=D_m^2 \tag*{(6)}

if the trace of DmD_m is larger than the number of electrons, and

Dm+1=2Dm−Dm2(7)D_{m+1}=2D_m-D_m^2 \tag*{(7)}

if the trace of DmD_m is smaller than the number of electrons. The density matrix DD is then approximated by DnD_n for a sufficient number of iterations, nn. Listing 2 shows a sketch of a Fortran code implementing an SP2 solver based on BML matrices and functionalities. We should emphasize that such an implementation is independent of the matrix format, matrix data type, and underlying computer architecture. Other variants of the SP2 algorithm have also been implemented in PROGRESS.

We use SP2 in our PROGRESS benchmark, specifically to evaluate performance of matrix-multiplication-based algorithms compared to dense diagonalization. For the bio-system benchmark described in Section II, SP2 converges to the specific tolerance used in that benchmark in 22 iterations, and its cost is typically dominated by the 22 matrix-matrix multiplications used in these iterations.

Chebyshev polynomial expansion

Recursive Fermi operator expansion techniques, like SP2, have known challenges in the case of a vanishing electronic bandgap. A better technique in this case is to use a serial Chebyshev expansion of the Fermi operator[22]

fμ(H)≈∑n=0LcnTn(H),(8)f_{\mu}(H)\approx\sum_{n=0}^{L}c_nT_n(H) , \tag*{(8)}

where TnT_n is the nnth Chebyshev polynomial and cnc_n is the nnth Chebyshev expansion coefficient. Currently in PROGRESS, and in some other codes [40], this expansion is computed via the Chebyshev polynomial recursion property

Tn+1(H)=2HTn(H)−Tn−1(H),(9)T_{n+1}(H)=2HT_n(H)-T_{n-1}(H) , \tag*{(9)}
!SP2 solver to compute DM d_bml for Hamiltonian h_bml
!using a cutoff threshold tau
subroutine prg_sp2(h_bml,d_bml,tau,...)
    use bml
    !Declare BML matrices ...
    type(bml_matrix_t),intent(in) :: h_bml
    type(bml_matrix_t),intent(inout) :: d_bml
    type(bml_matrix_t) :: x2_bml
    ...
    !initialize d_bml as shifted, rescaled Hamiltonian
    call bml_copy(h_bml, d_bml)
    call bml_add_identity(d_bml,-1.*emax)
    call bml_scale(-1.0/(emax-emin), d_bml)
    !create BML matrix x2 as a copy of d_bml
    call bml_copy_new(d_bml, x2_bml)
    ...
    !SP2 loop
    do i=0,maxiter
        tr = bml_trace(d_bml)
        ! X2 <- X * X
        call bml_multiply_x2(d_bml,x2_bml,tau)
        if(tr-nel <= 0.) then
            call bml_add(2.,d_bml,-1.,x2_bml,tau)
        else
            call bml_copy(x2_bml, d_bml)
        end if
        !check for convergence
        ...
    end do
    call bml_deallocate(x2_bml)
end subroutine

caption: Listing 2: Illustration of use case for BML matrices in SP2 solver as implemented in PROGRESS library.

for n>0n > 0, which results in a serialized summation – each term in the summation for fμf_\mu needs to be computed in sequence. Moreover, each recursion requires a single matrix multiplication, so that for a Chebyshev expansion of polynomial order LL, approximately LL matrix multiplications are required. In other words, the number of matrix multiplications to approximate the Fermi operator and compute the density matrix scales linearly with the number of expansion terms.

In 1973, Paterson and Stockmeyer [41] showed that a generic polynomial PP in powers of xx of order LL could be rewritten in such a way that only ∼2L\sim2\sqrt{L} multiplications in xx are needed to evaluate P(x)P(x). Several decades later, Liang et al. [42, 43] showed that this same idea could be applied directly to Chebyshev polynomial expansions of the Fermi-Dirac operator. The Chebyshev expansion in Eq. (8) can then be written as

∑n=0LcnTn=∑i=0k−1di,0Ti+Tk(∑i=0k−1di,1Ti+Tk(∑i=0k−1di,2Ti+⋯+Tk(∑i=0k−1di,m−1Ti)⋯ )),(10)\sum_{n=0}^{L} c_n T_n = \sum_{i=0}^{k-1} d_{i,0}T_i + T_k\left(\sum_{i=0}^{k-1}d_{i,1}T_i + T_k\left(\sum_{i=0}^{k-1}d_{i,2}T_i+\cdots+T_k\left(\sum_{i=0}^{k-1}d_{i,m-1}T_i\right)\cdots\right)\right), \tag*{(10)}

for coefficients di,jd_{i,j}, that can be determined from the Chebyshev coefficients cnc_n, and positive integers k,mk,m such that L+1=kmL+1=km. Not only did this algorithm replace a diagonalization with matrix-matrix multiplications, it also substantially reduced the number of multiplications needed to approximate the Fermi-Dirac operator for a given Chebyshev expansion size. Instead of the nearly LL multiplications required for the serial recursion from Eq. (9), the number of multiplications now scaled with the square root of the polynomial order, L\sqrt{L}. The large timing discrepancies between matrix-matrix multiplications and diagonalization previously mentioned then have an even more pronounced effect, making this approach very well-suited to modern GPU devices. The determination of the coefficients di,jd_{i,j} relies on the multiplicative recursion property of Chebyshev polynomials [37]: for n,mn,m non-negative integers,

TnTm(H)=12(Tn+m+T∣n−m∣)(H).(11)T_nT_m(H)=\frac{1}{2}(T_{n+m}+T_{\lvert n-m\rvert})(H). \tag*{(11)}

Note that each sum, SjS_j, on the right hand side of Eq. (10), where

Sj=∑i=0k−1di,jTi,0≤j≤m−1.(12)S_j=\sum_{i=0}^{k-1}d_{i,j}T_i,\qquad0\leq j\leq m-1. \tag*{(12)}

is completely independent of every other sum SjS_j, for j≠j′j\ne j' once the first kk Chebyshev polynomials TiT_i, i=0,…,ki=0,\ldots,k are known. This introduces parallelism into Eq. (10) since each sum can be calculated concurrently with every other sum. In addition, the first kk Chebyshev polynomials, T1T_1, T2T_2 up to TkT_k, can be calculated in a semi-parallel way too. The idea is to use the Chebyshev polynomial multiplication identity from Eq. (11) to calculate each TiT_i, i=2,…,ki=2,\ldots,k. Starting from T0T_0 and T1T_1, the second Chebyshev polynomial can be computed as

T2=2T1T1−T0,(13)T_2=2T_1T_1-T_0, \tag*{(13)}

and similarly, once T0T_0, T1T_1 and T2T_2 are known, the next two Chebyshev polynomials can be computed via:

T3=2T2T1−T1T4=2T2T2−T0,(14)\begin{aligned} T_3&=2T_2T_1-T_1 \\ T_4&=2T_2T_2-T_0, \tag*{(14)} \end{aligned}

using only previously computed TiT_i. Each Chebyshev polynomial on the left in Eq. (14) can therefore be computed in parallel. This parallelization on a single GPU is implemented using CUDA and HIP streams, on Nvidia and AMD GPUs, respectively, through MAGMA queues. The combination of this parallelization along with the square root scaling Chebyshev expansion approach leads to orders-of-magnitude speed ups over diagonalization when constructing the single-particle density matrix through a Chebyshev expansion. In Figure 3, we show the speed up obtained on an AMD MI250X GPU. Further details of this approach, and on how to compute the set of coefficients {di,j}\{d_{i,j}\}, can be found in Ref. [37].

Speed-up factor over diagonalization versus number of Chebyshev expansion terms for $N=500$, $N=1000$, $N=2000$, and $N=4000$

Figure 3. Speed up over diagonalization of the density matrix construction (using a known chemical potential) on an AMD MI250X GPU with multiple GPU streams for N=500N = 500 and N=1000N = 1000. Only a single stream is used for N=2000N = 2000, 4000. The MAGMA library is used for diagonalization which was found to be faster than the diagonalization routine in rocSolver.

Offloading with OpenMP

There are two components of GPU offloading, data movement and compute. Since most accelerated architectures have separate memory spaces for host and device, with limited bandwidth between them, we attempt to minimize data movement whenever possible. OpenMP [44, 45] offers a portable, pragma-based, framework for data movement between host and device, as well as compute functionality on both. The OpenMP standard is supported by multiple compiler vendors [46], with varying degrees of compatibility, especially with regard to offload support. GPU-offloading capabilities of OpenMP have been successfully used by various applications[47, 48, 49], including some in the electronic structure community[50, 51, 52] and for large-scale sparse eigensolvers [53]. In this section we outline the general strategy adopted in BML for using OpenMP to offload compute to accelerated devices.

Data Allocation

The BML matrix format includes both the base matrix data (one or more data arrays, depending on the format) as well as certain metadata (e.g. matrix format, distribution mode, local array bounds) in a C struct (see Listing 1). We only offload the data arrays to the device, at allocation time, using persistent data allocation on the device. For example, in the ELLPACK-R format, we offload the values, index and number of non-zeros arrays using a combination of allocation and updates from host to device (see Listing 3). Subsequently, we assume that the data on the device is correct, and synchronization

between device and host is only performed when needed. We also modify the corresponding deallocation functions to ensure that device-side memory is freed when host-side arrays are destroyed.

// allocate arrays on GPU
double* A_value=A->value;
#pragma omp target enter data map(alloc:A_value[:N*M])
int* A_index=A->index;
#pragma omp target enter data map(alloc:A_index[:N*M])
int* A_nnz=A->nnz;
#pragma omp target enter data map(alloc:A_nnz[:N])
// copy data from CPU to GPU
#pragma omp target update to(A_value[:N*M])
#pragma omp target update to(A_index[:N*M])
#pragma omp target update to(A_nnz[:N])

Listing 3: GPU memory allocation and update for BML matrix data

OpenMP Compute

For simple computations, or those computations which are not performance-critical, we can use OpenMP to perform device computations. For example, scaling an ELLPACK-R matrix;

size_t MbyN = N * M;
#pragma omp target teams distribute parallel for
  for (size_t i = 0; i < MbyN; i++)
  {
    A_value[i] = scale * A_value[i];
  }

Listing 4: Example of GPU-offloaded kernel using OpenMP directives.

In practice, OpenMP may not offer the ability to generate kernels with optimal performance on some devices, due to the inability to access certain fine-grained parallelism which is only available via vendor-specific methods, such as CUDA [54]. For this reason, we generally utilize a mixture of OpenMP, for data movement and execution of simple or non-performance-critical compute, with vendor-specific libraries (rocSparse, oneAPI, etc) for performance-critical kernels (see Sections VII A, VII B and VII B). Particularly for sparse matrices, we find that native OpenMP is unable to match the performance of vendor-optimized functions (see Figure 6). Fortunately, it is relatively easy to pass data to vendor libraries through raw pointers in C as allocated by OpenMP on the GPU. In these cases, we utilize the OpenMP use_device_ptr functionality to perform compute on the device-side data arrays previously allocated. For example, Listing 5 shows the interface to call an eigensolver using Intel oneAPI MKL libraries;

Comparison of time-to-solution of SP2 solver between sparse matrices using rocSPARSE and dense matrices using MAGMA on AMD MI250X GPU

Figure 6. Comparison of time-to-solution of SP2 solver between sparse matrices using rocSPARSE and dense matrices using MAGMA on AMD MI250X GPU for the PROGRESS 3D bio-benchmark. OpenMP Offload timings for small matrix sizes are also shown, for comparison.

In some cases, we have utilized entirely separate workflows, such as MAGMA, which encompass both data movement and compute, bypassing OpenMP (see § 6 A).

For sparse matrix functionality, most vendor-supplied libraries only use the CSR format, both on CPU and accelerator. In contrast, BML uses CSR (slightly modified), ELLPACK-R and ELLBLOCK formats. In order to leverage the performance of the vendor-supplied libraries, we therefore developed a set of functions to perform the appropriate data transformations. Here we show a simple example of transforming data from a canonical CSR format to ELLPACK-R on the GPU;

#pragma omp target teams distribute parallel for
for (int i = 0; i < A_N; i++)
{
    A_nnz[i] = csrRowPtr[i + 1] - csrRowPtr[i];
    for (int j = 0; j < A_nnz[i]; j++)
    {
        int idx = csrRowPtr[i] + j;
        A_value[ROWMAJOR(i, j, A_N, A_M)]
            = csrVal[idx];
        A_index[ROWMAJOR(i, j, A_N, A_M)]
            = csrColInd[idx];
    }
}

Listing 6: CSR to ELLPACK-R conversion using OpenMP target directives. Here “ROWMAJOR” is a C macro that returns the 1-d index of an element (i,j)(i,j) for a matrix of size A_M × A_N.

Although these data transformations inevitably introduce some overhead, the cost is outweighed by the potential penalty of using sub-optimal compute for performance-critical functions such as sparse matrix multiplication in the native ELLPACK-R format using OpenMP.

Offloading dense linear algebra solvers

MAGMA on Nvidia and AMD GPUs

The MAGMA library [8, 55] offers most of the functionalities we need for the dense matrix format. It essentially implements the whole set of functions typically available in BLAS and Lapack for the GPU, and more. It currently supports Nvidia and AMD GPUs, and is expected to support Intel GPUs in the near future. For offloading the dense matrix format to GPU in BML, the choice was made to rely heavily, and mostly, on MAGMA, for memory allocation, data movement and linear algebra operations on Nvidia and AMD GPUs.

As an exception, we observed that the diagonalization function available in the cuSolver library “cusolverDnDsyevd” from Nvidia significantly outperforms the MAGMA functions and have thus added an interface to it. Calling a cuSOLVER function from within a MAGMA code turns out to be easy since both codes work with simple C pointers, and a data array allocated by MAGMA can be directly used by any cuSolver function. The rocSolver library is the AMD equivalent of Nvidia’s cuSolver and is easy to integrate with MAGMA as well. However we did not find that the diagonalization function in that library to be more performant than the MAGMA one.

In Figure 4, we plot the time-to-solution for the SP2 solver (based on “magma_dgemm” function) compared to dense diagonalization (using “magma_dsyevd_gpu” function) for the PROGRESS 3D bio-benchmark, as measured on AMD MI250X GPU. SP2 significantly out-performs the dense diagonalization for all matrix sizes shown here (up to N=29,187N=29,187), but more significantly for the smaller sizes.

Comparison of time-to-solution between SP2 solver and dense diagonalization using MAGMA functionalities on AMD MI250X GPU for the PROGRESS 3D bio-benchmark

Figure 4. Comparison of time-to-solution between SP2 solver and dense diagonalization using MAGMA functionalities on AMD MI250X GPU for the PROGRESS 3D bio-benchmark.

Note that MAGMA provides a 2-stage eigensolver that could potentially be faster than the divide and conquer version we are currently using. However its current interface only supports data on the CPU, and thus would require extra copies between the GPU and CPU in our implementation.

MKL on Intel GPUs

When offloading computation to Intel GPUs, we use the oneAPI MKL libraries [56]. At present we have only offloaded the computations for BML dense format, but we anticipate that the methodology will be directly transferable to the appropriate sparse formats. The offload of data to the Intel GPU is accomplished using OpenMP functionality, as described in § 5. That is, we allocate the appropriate array, in this case a single dense block, on the GPU when the BML matrix is initialized. Subsequently, the correct state of the matrix is assumed to be that on the GPU, and all computation is performed on the GPU when possible in order to minimize data movement and to maximally leverage the improved GPU compute relative to CPU.

The Intel model for GPU offload is enabled via offloaded versions of their oneAPI MKL libraries. The call signature is largely the same as the corresponding CPU function, with the addition of an appropriate OpenMP pragma. For example, the call for matrix addition on CPU (computing αA+βB\alpha A+\beta B) in double precision is given by, using the BLAS C-interface:

cblas_dscal(n, alpha, A->matrix, inc);
cblas_daxpy(n, beta, B->matrix, inc,
            A->matrix, inc);

Listing 7: Computing αA+βB\alpha A+\beta B using the BLAS C-interface

and the corresponding GPU offload is:

double* A_matrix=A->matrix;
double* B_matrix=B->matrix;
#pragma omp target variant dispatch \
    use_device_ptr(A_matrix)
cblas_dscal (n, alpha, A_matrix, inc);
#pragma omp target variant dispatch \
    use_device_ptr(A_matrix, B_matrix)
cblas_daxpy (n, beta, B_matrix, inc, A_matrix, inc);

Listing 8: Computing αA+βB\alpha A+\beta B using oneAPI MKL

Note that for complex data types, the MKL CBLAS calls on the GPU require an “&” in front of alpha and beta to get the address of a complex number. Note also the target variant dispatch and use_device_ptr pragmas. Otherwise the call signature of the offloaded function appears identical to the corresponding host-side call.

With the appropriate functions offloaded, we can then compare the performance of the SP2 algorithm on GPU with diagonalization, on both CPU and GPU. We used the PROGRESS soft matter synthetic Hamiltonians described in Ref. 37 for this comparison, which is based on a simpler code implementation and allowed us to work around some of the compiler issues we faced on Intel GPUs available to us. The resulting timings on the Sunspot testbed are shown in Figure 5. Sunspot is a precursor to Aurora [57], where each node consists of 2 Intel Xeon CPU Max Series (codename Sapphire Rapids or SPR) and 6 Intel Data Center GPU Max Series (codename Ponte Vecchio or PVC). Each Xeon has 52 physical cores supporting 2 hardware threads per core. We utilize 16 threads per process on CPU, to simulate using 1/6 of the CPU available per GPU. We also repeated the test with 52 CPU threads, equivalent to a full socket, which shows some improvement at larger sizes. As seen on other systems, the SP2 solver is performing well compared to the dense diagonalization, and GPU outperforms CPU.

PROGRESS benchmark timings for dense matrices

Figure 5. PROGRESS benchmark timings measured on Intel Sunspot system for dense matrices of N=1024 up to N=16384 using the oneAPI MKL library. Times are in milliseconds. CPU runs used either 16 or 52 threads, as indicated in legend. This work was done on a pre-production supercomputer with early versions of the Aurora software development kit.

Although the choice to implement the offloaded functions with (almost) identical signatures to the CPU version makes the implementation straightforward from a coding standpoint, the lack of maturity of the software stack is an issue. Functions are being ported to the GPU gradually, and compiler updates may break working builds so frequent regression testing is needed. Overall, this is to be expected with a software stack in this stage of development.

Leveraging third-party libraries for O(NN) sparse linear algebra solvers

When it comes to iterative sparse solvers, such as SP2, the key kernel needed is a sparse-sparse matrix multiply. Compared to others such as sparse-dense matrix multiply, this function is not implemented in many libraries. Previously, threaded sparse matrix methods in BML were implemented using the ELLPACK-R sparse matrix format. Our initial approach to GPU acceleration used OpenMP offload. Unfortunately this led to bottlenecks in matrix-matrix multiplications and additions – the performance of the offloaded versions of these functions was lacking. We thus switched to the use of third-party libraries for these numerical kernels.

AMD rocSparse library

As faster methods were needed for AMD GPUs to prepare for the availability of Frontier at the Oak Ridge Leadership Facility[3], we sought to replace these functions with AMD rocSPARSE methods. The CSR data used by rocSPARSE is obtained using an ELLPACK-R to CSR translation code within BML (see § 5). Calls to rocSPARSE methods are then made within an omp target data region, to enable host or device pointers to be passed to rocSPARSE methods as appropriate. The code block in Listing 9 illustrates the general approach of calling a rocSPARSE function using the GPU pointer to a BML data array (the approach is similar to the way the MKL methods are used on Intel GPUs as described in Section VIB). In this code block, the function f() performs a computation on A_matrix on the GPU. The use_device_ptr(A_matrix) clause instructs the compiler to pass the GPU pointer for A_matrix to f(). The variable A_N is not included in the use_device_ptr() clause, indicating that the value on the host will be used.

double* A_matrix=A->matrix;
#pragma omp target data use_device_ptr(A_matrix)
{
    f(..., A_matrix,A_N,...)
}

Listing 9: Using OpenMP offload for a rocSPARSE function “f” call and a BML matrix “A”.

The most computationally intensive kernel in SP2 is the sparse matrix multiplication; therefore, we first developed an interface to use the rocsparse_spgemm() function inside BML for this kernel. Several other bottlenecks subsequently were identified. In particular, the addition of two matrices in BML was accelerated using the function rocsparse_csrgeam().

As of at least ROCm 5.3, the rocSPARSE methods require sorted column indices. This is important for our applications as the ELLPACK-R format is unsorted, and matrices can become unsorted during the course of calculations. For example, computing the matrix transpose leads to unsorted column indices. Moreover, even though the rocSPARSE methods require sorted matrices on input, they can produce unsorted matrices on output. We therefore changed the BML rocSPARSE code to sort matrices as needed. This was accomplished using the rocSPARSE rocsparse_csrsort function. In addition, to ensure that the resulting sparse matrices satisfy the thresholding criterion for including matrix elements, our implementation uses the rocSPARSE function rocsparse_xprune_csr2csr).

Figure 6 compares the performance of the SP2 solver for the PROGRESS 3D bio-benchmark using the rocSPARSE (red triangles) vs. MAGMA (blue diamonds) solvers. The builds used the Cray CCE compilers version 15, and AMD ROCm version 5.1. The rocSPARSE method shows approximate linear scaling, as expected for the sparse SP2 density matrix algorithms. Due to the approximate linear scaling, the rocSPARSE timings for large matrix sizes are smaller than the MAGMA timings (up to more than 10x). Two OpenMP Offload timings for small matrices are shown for comparison (Figure 6, orange squares). These data points show that the time required for the density matrix build using the rocSPARSE method is orders of magnitude smaller than that achieved for the OpenMP Offload code. Overall, the figure shows the substantial performance increase on GPU achieved when taking advantage of sparsity using rocSPARSE compared to the dense format using MAGMA for large matrix sizes. It also shows that rocSPARSE overcomes the performance limitations of the OpenMP Offload methods.

hypre library

As mentioned above, functionalities for multiplying two sparse matrices by each other are not found in many linear algebra libraries. The focus is often on solving sparse linear systems using iterative solvers, where multiplying a sparse matrix by a vector is the key ingredient. That being said, electronic structure solvers are not the only ones using sparse-sparse matrix multiplications. Another area where those are heavily used is in the algebraic multigrid community. There, coarse grid operators are often computed as the product AL=RAlPA_L = R A_l P, where AlA_l is the discretized operator at the fine level, ALA_L the discretized operator at the coarse level, and RR and PP are restriction and prolongation operators, which are used to coarsen or refine the data between levels. All these operators are represented as sparse matrices, and thus the computation of ALA_L is typically the result of two consecutive sparse-sparse matrix multiplications.

The hypre linear solver library [58, 59] is a well-known open-source library with scalable Algebraic Multigrid solver capabilities, among other solvers. The sparse matrix-matrix multiplication routines in hypre provide access to both vendor library routines, as well as internal algorithms independent of the vendor. These internal algorithms are ported to HPC hardware with Nvidia, AMD and Intel GPUs based on their respective native programming paradigms. Thus, through the same interface to hypre, BML could access and utilize this performance critical sparse matrix functionality on different HPC platforms, leading to a performant and portable alternative to vendor-specific libraries. Additionally, leveraging open-source libraries such as hypre provides security through access to software capabilities and personnel with technical expertise and a shared interest in addressing software stack issues and challenges on emerging HPC hardware.

Accessing sparse matrix-matrix multiplication routines through hypre follows a similar approach to integrating BML with vendor libraries. First, matrix data is converted from ELLPACK-R format to standard CSR format on the device. hypre’s internal CSR matrix data structure uses raw C array pointers to store the matrix data. This interface makes it convenient to directly pass device pointers to standard CSR data, allocated by OpenMP, into hypre’s internal CSR matrix data structure without incurring additional overhead. Next, with the data on the device, the sparse matrix-matrix multiplication is performed using functionality provided by hypre. Finally, the result is converted back from CSR to ELLPACK-R using the same approach as for vendor libraries. Listing 10 provides a code snippet of how the integration with hypre is realized. Note that unlike rocSparse, hypre’s interface does not put any requirement on the order of the elements in a CSR row.

Initial challenges to this effort came from having a consistent compiler stack to enable an interoperable build of hypre with BML, with OpenMP Offload and vendor-specific compiler constraints. These challenges were eventually resolved. Performance results for the PROGRESS SP2 benchmark using the sparse-sparse matrix multiplication from the hypre library are shown in Figure 7 for an Nvidia V100 GPU. Two data points using Nvidia cuSPARSE library are also shown. These correspond to the two smallest matrix sizes in the PROGRESS benchmark – the only two we were able to complete using Nvidia CUDA 11 toolkit, due to the large memory requirements for the cuSPARSE sparse-sparse matrix multiplication. They show a significantly better performance using hypre (factor 6-7X). The results from Section VII A using rocSPARSE on AMD MI250X GPU, are also shown for reference. Taking into account the differences in hardware performance – about 3X more flops and 1.8X better memory bandwidth for the AMD - GPU – they indicate a comparable performance for hypre and rocSPARSE sparse-sparse matrix multiplications.

Time-to-solution of sparse O(N) SP2 solver using hypre, rocSPARSE, and cuSPARSE

Figure 7. Time-to-solution of sparse O(N)O(N) SP2 solver using the hypre library on NVIDIA V100 GPU for the PROGRESS 3D bio-benchmark. For comparison, we also plot two data points when using the Nvidia cuSparse library on same V100 GPU, as well as the results using the rocSparse SP2 library on AMD MI250X GPU.

/* create hypre csr matrix */
matA = hypre_CSRMatrixCreate( A_N,A_N,nnzA );
matB = hypre_CSRMatrixCreate( B_N,B_N,nnzB );
#pragma omp target data use_device_ptr( \
    csrRowPtrA,csrColIndA,csrValA, \
    csrRowPtrB,csrColIndB,csrValB)
{
    hypre_CSRMatrixI(matA) = csrRowPtrA;
    hypre_CSRMatrixJ(matA) = csrColIndA;
    hypre_CSRMatrixData(matA) = csrValA;

    hypre_CSRMatrixI(matB) = csrRowPtrB;
    hypre_CSRMatrixJ(matB) = csrColIndB;
    hypre_CSRMatrixData(matB) = csrValB;
}
/* perform matrix multiplication */
matC  = hypre_CSRMatrixMultiplyDevice(matA, matB);

Distributed memory solvers

Distributed memory approaches are very attractive for large problems that do not fit in the memory of a single node. But they can also be used to speedup time-to-solution. There is obviously always a cost to distributing a problem across multiple nodes due to communication and possibly extra computation for overlapping work. But in many cases, these costs can be managed to a reasonable fraction of the compute time, and calculations can benefit from distributed resources.

Here we distinguish between two general ways of distributing an atomistic simulation: distributing the linear algebra problem of computing a DM, or dividing the physical system into a set of (potentially overlapping) subsystems. The ScaLAPACK library[60] is probably the most well-known library implementing distributed linear algebra operations. In BML, we implemented a distributed matrix format which leverages the operations implemented for the various shared memory formats in BML. This approach is described in Section VIII A. Partitioning the DM computation at the physical level was proposed by Yang and Lee in 1995 using a Divide and Conquer approach [61]. In Section VIII B, we present a related approach implemented in PROGRESS. It relies purely on matrix elements to determine the partitioning of the system, and is thus very appropriate for a library implementation totally independent of any specific electronic structure code. It also leverages shared memory solvers implemented in BML, which are used for the linear algebra operations done for each physical sub-system.

Distributed linear algebra

In order to leverage the implementation done for all the shared memory matrix formats in BML, we introduced a distributed matrix format where each block owned by an MPI task is a BML matrix in a non-distributed (shared memory) format. This allows us to have a distributed format for all the shared memory matrix formats already implemented in BML. The restrictions we introduce with this format are that (i) the sub-matrices owned by each MPI task have to be square matrices, (ii) the number of MPI tasks used has to be an integer squared. We named that new format “distributed2d”. This format is built with the BML library when BML is configured to be built with MPI.

Our implementation is non-intrusive, leaving the shared memory matrix formats untouched. It consists mainly of “wrapper” functions calling sub-matrix operations when possible, implementing the “distributed2d” matrix operations as combinations of “shared memory” matrix operations. Some operations, such as the Frobenius norm for instance, need an MPI reduction at the end to get the global values. Some operations (multiplication, transpose, …) require more substantial communications. Our current implementation of distributed matrix-matrix multiplication is based on Cannon’s algorithm. Some operations are intrinsically more intrusive; for instance computing the bounds on the eigenvalues of a matrix using Gershgorin circles. The strategy in this case is to add functionalities to the basic formats to be used by “distributed2d” implementation. Some operations are beyond the scope of this project. For instance, implementing a distributed eigensolver would require a lot of work beyond the resources of this project. Fortunately, other libraries offer good distributed eigensolvers. In BML, our eigensolver is thus simply interfacing with an existing solver. For CPU, we have implemented an interface to ScaLAPACK[60]. For GPU, we have implemented an interface to the ELPA library [62].

Graph-based Divide electronic structure

Niklasson and collaborators recent findings[63] have demonstrated, both theoretically and practically, that there exists a bijective correspondence between matrix functions of sparse matrices and the same functions applied to only certain graph-restricted domains (parts) of the matrices. This theory enables the decomposition of the problem of computing a sparse matrix function into sub-problems involving much smaller dense matrices. The technique achieves near-perfect parallelism, where computations can be executed with distributed memory and minimal communication [64, 65]. The significance is twofold: it not only offers a systematic approach to addressing matrix function calculations by breaking them down into manageable components, but also capitalizes on the power of parallel computing to handle these computations concurrently.

The original article used electronic structure calculations (the application of the Fermi-Dirac function) to validate this technique and was coined “graph-based linear-scaling electronic structure theory.” The utilization of electronic structure concepts to verify this theoretical result underscores its practical relevance in real-world applications, particularly in the domain of large-scale scientific computing. Other researchers embraced this concept and translated it into libraries designed for the parallel application of matrix functions on massively large scales [66].

Within the context of the PROGRESS library, we apply the Fermi-Dirac function to compute the system’s DM. This technique however can be generalized to other functions. Given a Hamiltonian matrix HH, and a threshold τ\tau, one can create a graph GG by constructing its adjacency matrix AA defined as

Aij={1if ∣Hij∣≥τ and i≠j0otherwise.(15)A_{ij} = \begin{cases} 1 & \text{if } \lvert H_{ij} \rvert\ge\tau\text{ and } i \ne j \\ 0 & \text{otherwise.} \end{cases} \tag*{(15)}

This graph can be partitioned into several components (or parts) using various methods referred to as graph partitioning techniques [67]. We define a partition Π\Pi of graph GG into mm parts, as a collection of mm sets of nodes of GG, Π={π1,…πi,…πm}\Pi= \{\pi_1,\ldots\pi_i,\ldots\pi_m\}.

The PROGRESS library provides several options for graph partitioning. Perhaps the most straightforward one is to simply divide the matrix index list into relatively regular segments. PROGRESS also offers partitioning through the use of the METIS library, which implements various algorithms based on multiple constraint partitionings [68]. In addition, PROGRESS provides a partitioning algorithm with the objective of minimizing the total number of arithmetic operations if a O(N3)O(N^3) complexity would dominate mathematical operations performed over each set of nodes. PROGRESS also implements several variations of graph partitioning inspired by the Kernighan-Lin method [69]. Finally, there is a method in which a METIS partitioning is refined using a simulated annealing technique to minimize the number of operations in a O(N3)O(N^3) complexity algorithm.

Within the context of electronic structure, two concepts immediately follow the idea of a partition: core partition and core-halo partition. A partition is called a core partition, Πc={π1c,…πkc,…πmc}\Pi_c = \{\pi_1^c,\ldots\pi_k^c,\ldots\pi_m^c\}, if every node in GG belongs to one and only one of the πkc∈Πc\pi_k^c \in\Pi_c. If we extend each core component πkc∈Πc\pi_k^c \in\Pi_c to also include the neighboring nodes of every node in πkc\pi_k^c, this defines a core-halo partition, Πch={π1ch,…πkch,…πmch}\Pi_{ch} = \{\pi_1^{ch},\ldots\pi_k^{ch},\ldots\pi_m^{ch}\}. Thus if a node ll belongs to πkc\pi_k^c, then every directly connected neighboring node oo (where Al,o=1A_{l,o}=1) belongs to πkch\pi_k^{ch}.

Given a partition Π\Pi of GG, for every element πk∈Π\pi_k \in\Pi, we can extract a submatrix hαβk:=Hi∈πk,j∈πkh_{\alpha\beta}^k := H_{i\in\pi_k,j\in\pi_k}. There is then a one-to-one mapping between indices of hh and the indices of HH, that is if sks_k is the size (number of nodes) of πk\pi_k, α∈[1,sk]↔i∈πk\alpha\in[1,s_k] \leftrightarrow i \in\pi_k. BML implements the functionalities to extract a submatrix hh from a global matrix HH, and to map data from a submatrix hh into a global matrix HH.

To prevent discontinuities, our approach involves extracting submatrices from core-halo partitions, applying the necessary functions to these submatrices, and then subsequently constructing the final matrices. This is accomplished by first constructing a core-halo partition from a core partition. The extension is based on an “auxiliary” matrix that defines the connectivity of the orbitals. This auxiliary matrix can be the Hamiltonian itself, the overlap matrix, or a previously computed DM. A previously computed DM can be obtained from previous time steps (or a previous SCF step) during a geometry optimization and/or a QMD simulation. From this extension we can extract the submatrix hk−chh^{k-ch} for each part kk and apply a matrix function to it. Once the function has been evaluated for every part kk we can then extract the submatrices corresponding to the “core” partition and map them back to a full system density matrix PP, where P≈f(H)P \approx f(H).

In Listing 11 we present a general algorithm that applies this graph-based approach to computing the density matrix. The inputs are the Hamiltonian (h_bml)

General density matrix graph-based distributed solver routine

Figure 11. General density matrix graph-based distributed solver routine as implemented in PROGRESS.

matrix, and the auxiliary matrix (g_bml) that serves as a guess for the connectivity of the orbitals. In our PROGRESS implementation, all the matrices in the interface are sparse matrices. The construction of the graph is handled using the BML ELLPACK-R format given the sizes of the adjacency matrices that are typically involved in large systems that need memory distributed techniques. The auxiliary matrix is converted into a weighted adjacency matrix by taking the absolute values of every entry and a thresholding operation is applied to control the extent of the resulting graph. The resulting graph is then used in a graph partitioning algorithm to get the parts. The graph parts are then used to define the Hamiltonian sub-matrices (dense BML matrices) that are then solved for independently and concurrently using, for example, a regular diagonalization method (prg_build_DM()). From the several DM sub-matrices obtained, each MPI task will reconstruct locally a “partially filled” full DM, before the full DM is assembled by summing up all contributions using an MPI reduction operation.

Fig. Figure 8 shows the average DM element error and the average CH size as a function of the threshold value picked to build the adjacency matrix. We see that we get a well controlled error (Frobenius norm of the difference between the graph-based DM and the DM obtained using the dense diagonalization) by modifying the thresholding parameter used to construct the adjacency matrix. The average element error follows a linear function of the threshold on a log-log plot, indicating that the error is a polynomial of the threshold (error ∼τ2.2\sim\tau^{2.2} in this case). When the submatrices are extracted, they contain a halo region (extra layer of surrounding orbitals) which is an extension from the cores (extracted set of atoms) arising from the overlapping graph-partitioning process. The smaller the threshold, the larger the overlap between the different parts and the smaller the error committed. On the other hand, the smaller the threshold, the larger the individual Hamiltonians that need to be solved independently and the higher the computational cost.

Error vs. thresholding parameter and average core+halo sizes

Figure 8. Error vs. thresholding parameter used to construct the adjacency matrix (blue) with average sizes of core+halo parts (red). These results were obtained from the PROGRESS benchmark Hamiltonian for N=2162N=2162 and a partition into 8 parts.

We tested this graph-based approach performance by evaluating the wall-clock time for the construction of DM as a function of the problem size. In Figure 9, we show the timings obtained and compared those to a standard dense diagonalization. The computational cost as a function of problem size is remarkably promising. On a single Intel CPU node the graph-based approach has a very low prefactor scaling as compared to the regular DM construction method (see Figure 9) and becomes competitive for matrix size N=2000N=2000 and beyond. We also see a larger speedup for larger system sizes.

Wall-clock time versus problem size

Figure 9. Wall-clock time vs. problem size for a fixed threshold of 0.01 for the graph-based solver using 8 parts (red). This is compared with the wall-clock time of a dense diagonalization of the entire problem (blue). Runs on a single Intel Core i7 CPU node with 12 OpenMP threads.

Speedup with distributed graph solver

Figure 10. Speedup with distributed graph solver compared to serial global solver as a function of the threshold used for graph partitioning. Results obtained for the PROGRESS benchmark with N=2162N=2162.

With modern hybrid architectures, with GPU-accelerated nodes, speedups can be obtained only for larger problems. The overhead associated with this distributed graph-based approach includes (i) partitioning the global matrix into “core+halo” parts, (ii) extracting the dense submatrices associated with each part from large sparse matrices, (iii) communications between MPI tasks to gather the calculated submatrices into the resulting global DM. In Table II, we show the matrix and submatrix sizes used by the graph-solver, as well as performance numbers obtained on Nvidia GPUs using a molecular system from the PROGRESS benchmark described in Section II. In this case, the difference between the time spent in the local solver and the total time for the distributed solver shows the time spent in the overhead operations (3.6 s). While it cannot be eliminated totally, we expect future code optimization to reduce it significantly.

Number of parts4
Number of MPI tasks4
Number of GPUs4
Matrix size12,972
Number of core nodes/part3149–3314
Number of core+halo nodes/part5657–5810
Wall-clock time for local solver (s)2.0
Total wall-clock time distributed solver (s)5.6
Wall-clock time single GPU solver (s)8.7
Speedup1.55 X

Table II. Performance of graph-based density matrix construction for a Hamiltonian matrix of size 12,972. Run on one node of Summit at OLCF (Nvidia V100 GPU). Local, single GPU solver is Nvidia cuSolver “cusolverDnDsyevd”. The graph was constructed from the (precomputed) DM itself with a threshold of 10−210^{-2}.

Concluding remarks

Even though the software development environment using high-performance computing resources with GPU accelerators has improved substantially in recent years, it is still a challenge to produce software that is performant, portable and maintainable.

With modern hybrid architectures where more than 90% of the flops are those on the GPU, more and more scientific code needs to be developed and optimized for GPU execution. From a numerical point of view, maximal utilization of a GPU is both a lot of work and technically challenging, and may even require algorithm redesign. Dense eigenvalue solvers routinely used in the electronic structure community do not get the speedup one might expect on a GPU based on flops specifications, while other solvers based on matrix multiplications perform much better.

In this paper, we demonstrated some ideas on how to address some of these issues, and described libraries (PROGRESS and BML) where these techniques are implemented. From a performance point of view, when comparing the dense matrix-multiplication-based iterative solver SP2 with traditional dense diagonalization, we showed some significant speedups using AMD and Intel GPUs. For O(N)O(N) solvers based on sparse SP2, focusing on Nvidia and AMD GPUs, we demonstrated how to leverage third-party libraries for core numerical kernels within an OpenMP offload implementation to achieve better performance than dense solvers for matrix sizes beyond N=3,000N = 3{,}000. We also showed how some distributed memory solvers can be implemented, leveraging the performance of the shared memory operations implemented in BML. For these, the challenge remains to keep data transfer overhead low in comparison to on-GPU operations which are extremely fast on high-end devices.

From a practical point of view, the software stack can still be quite unstable and building a set of working libraries and code together remains a challenge. In addition, OpenMP offload support differs from compiler to compiler which, with fewer debugging options than on CPUs, can require significant code development efforts. While we expect software stack stability and reliability to improve with the maturity of the technology used in today’s largest HPC resources, we also expect some challenges to persist for quite some time.

Acknowledgements

This manuscript has been authored in part by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. This research was supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the DOE Office of Science and the National Nuclear Security Administration (NNSA).

Portions of this work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344

The authors would like to thank Rui Peng Li, LLNL, for his insightful comments and assistance in the integration of hypre into the BML library.

Author declarations

The authors have no conflict of interest to disclose.

Data availability

The data and code supporting the findings of this study are available from the corresponding authors upon reasonable request.

References

  1. [1]Tom Deakin, James Cownie, Wei-Chen Lin, and Simon McIntosh-Smith. Heterogeneous programming for the homogeneous majority. In 2022 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC), pages 1–13, 2022.DOI
  2. [2]L. Luo, T. P. Straatsma, L. E. Aguilar Suarez, R. Broer, D. Bykov, E. F. D’Azvedo, S. S. Faraji, K. C. Gottiparthi, C. De Graaf, J. A. Harris, R. W. A. Havenith, H. J. Aa. Jensen, W. Joubert, R. K. Kathir, J. Larkin, Y. W. Li, D. I. Lyakh, O. E. B. Messer, M. R. Norman, J. C. Ofelein, R. Sankaran, A. F. Tillack, A. L. Barnes, L. Visscher, J. C. Wells, and M. Wibowo. Pre-exascale accelerated application development: The ORNL Summit experience. IBM Journal of Research and Development, 64(3/4):11:1–11:21, 2020.DOI
  3. [3]Scott Atchley, Christopher Zimmer, John Lange, David Bernholdt, Veronica Melesse Vergara, Thomas Beck, Michael Brim, Reuben Buddardija, Sunita Chandrasekaran, Markus Eisenbach, Thomas Evans, Matthew Ezell, Nicholas Frontiere, Antigoni Georgiadou, Joe Glenski, Philipp, Grete, Steven Hamilton, John Holmen, Axel Huebl, Daniel Jacobson, Wayne Joubert, Kim Mcmahon, Elia Merzari, Stan Moore, Andrew Myers, Stephen Nichols, Sarp Oral, Thomas Papatheodore, Danny Perez, David M. Rogers, Evan Schneider, Jean-Luc Vay, and P. K. Yeung. Frontier: Exploring exascale. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’23, New York, NY, USA, 2023. Association for Computing Machinery.DOI
  4. [4]C. L. Lawson, R. J. Hanson, D. R. Kincaid, and F. T. Krogh. Basic linear algebra subprograms for fortran usage. ACM Trans. Math. Softw., 5(3):308–323, sep 1979.DOI
  5. [5]Jack J. Dongarra, Jeremy Du Croz, Sven Hammarling, and Richard J. Hanson. An extended set of fortran basic linear algebra subprograms. ACM Trans. Math. Softw., 14(1):1–17, 1988.
  6. [6]J. J. Dongarra, Jeremy Du Croz, Sven Hammarling, and I. S. Duff. A set of level 3 basic linear algebra subprograms. ACM Trans. Math. Softw., 16(1):1–17, mar 1990.
  7. [7]E. Anderson, Z. Bai, C. Bischof, S. Blackford, J. Demmel, J. Dongarra, J. Du Croz, A. Greenbaum, S. Hammarling, A. McKenney, and D. Sorensen. LAPACK Users’ Guide. Society for Industrial and Applied Mathematics, Philadelphia, PA, third edition, 1999.
  8. [8]Jack Dongarra, Mark Gates, Azzam Haidar, Jakub Kurzak, Piotr Luszczek, Stanimire Tomov, and Ichitaro Yamazaki. Accelerating numerical dense linear algebra calculations with GPUs. Numerical Computations with GPUs, pages 1–26, 2014.DOI
  9. [9]Cade Brown, Ahmad Abdelfattah, Stanimire Tomov, and Jack Dongarra. Design, optimization, and benchmarking of dense linear algebra algorithms on AMD GPUs. In 2020 IEEE High Performance Extreme Computing Conference (HPEC), pages 1–7, 2020.DOI
  10. [10]Mark Gates, Asim Yarkhan, Dalal Sukkari, Kadir Akbudak, Sebastien Cayrols, Daniel Bielich, Ahmad Abdelfattah, Mohammed Al Farhan, and Jack Dongarra. Portable and efficient dense linear algebra in the beginning of the exascale era. In 2022 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC), pages 36–46, 2022.DOI
  11. [11]Hartwig Antz, Terry Cojean, Goran Flegar, Fritz Göbel, Thomas Grützmacher, Pratik Nayak, Tobias Ribizel, Yuhsiang Mike Tsai, and Enrique S. Quintana-Ortí. Ginkgo: A Modern Linear Operator Algebra Framework for High Performance Computing. ACM Transactions on Mathematical Software, 48(1):2:1–2:33, February 2022.
  12. [12]Nicolas Bock, Christian F A Negre, Susan M Nizsiewski, Jamaludin Mohd-Yusof, Bálint Aradi, Jean-Luc Fattebert, Daniel Osei-Kuffuor, Timothy C Germann, and Anders M N Niklasson. The basic matrix library (BML) for quantum chemistry. J. Supercomput., 74(11):6201–6219, 2018.DOI
  13. [13]Urban Bornstik, Joost VandeVondele, Valery Weber, and Juerg Hutter. Sparse Matrix Multiplication: The Distributed Block-Compressed Sparse Row Library. Parallel Computing, 40(5-6), 2014.DOI
  14. [14]Ole Schütt, Peter Messmer, Jürg Hutter, and Joost VandeVondele. GPU-Accelerated Sparse Matrix–Matrix Multiplication for Linear Scaling Density Functional Theory, chapter 8, pages 173–190. John Wiley & Sons, Ltd, 2016.
  15. [15][15*] Thomas D. Kühne, Marcella Iannuzzi, Mauro Del Ben, Vladimir V. Rybkin, Patrick Seewald, Frederick Stein, Glenski, Philipp, Grete, Steven Hamilton, John Holmen, Axel Huebl, Daniel Jacobson, Wayne Joubert, Kim Mcmahon, Elia Merzari, Stan Moore, Andrew Myers, Stephen Nichols, Sarp Oral, Thomas Papadotheodore, Danny Perez, David M. Rogers, Evan Schneider, Jean-Luc Vay, and P. K. Yeung. Frontier: Exploring exascale. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’23, New York, NY, USA, 2023. Association for Computing Machinery.
  16. [16]Victor Wen zhe Yu, Carmen Campos, William Dawson, Alberto García, Ville Havu, Ben Hourahine, William P. Huhn, Mathias Jacquelin, Weile Jia, Murat Keçeli, Raul Laasner, Yingzhou Li, Lin Lin, Jianfeng Lu, Jonathan Moussa, Jose E. Roman, Álvaro Vázquez-Mayagoitia, Chao Yang, and Volker Blum. ELSI — an open infrastructure for electronic structure solvers. Computer Physics Communications, 256:107459, 2020.
  17. [17]Michael J. T. Oliveira, Nick Papior, Yann Pouillon, Volker Blum, Emilio Artacho, Demian Caliste, Fabiano Corsetti, Stefano de Gironcoli, Alin M. Elena, Alberto García, Víctor M. García-Suárez, Luigi Genovese, William P. Huhn, Georg Huhs, Sebastian Kokott, Emine Küçükbenli, Ask H. Larsen, Alfio Lazzaro, Irina V. Lebedeva, Yingzhou Li, David López-Durán, Pablo López-Tarifa, Martin Lüders, Miguel A. L. Marques, Jan Minar, Stephan Mohr, Arash A. Mostofi, Alan O’Cais, Mike C. Payne, Thomas Ruh, Daniel G. A. Smith, José M. Soler, David A. Strubbe, Nicolas Tancongco-Dejean, Dominic Tildesley, Marc Torrent, and Victor Wen-zhe Yu. The CECAM electronic structure library and the modular software development paradigm. The Journal of Chemical Physics, 153(2):024117, 07 2020.DOI
  18. [18]Irina V. Lebedeva, Alberto García, Emilio Artacho, and Pablo Ordejón. Modular implementation of the linear- and cubic-scaling orbital minimization methods in electronic structure codes using atomic orbitals. Royal Society Open Science, 10(4):230063, 2023.DOI
  19. [19]F. Vázquez, G. Ortega, J.J. Fernández, and E.M. Garzón. Improving the performance of the sparse matrix vector product with GPUs. In 2010 10th IEEE International Conference on Computer and Information Technology, pages 1146–1151, 2010.DOI
  20. [20]Susan M Nniszewski, James Belak, Jean-Luc Fattebert, Christian FA Negre, Stuart R Slattery, Adetokunbo A Adedoyin, Robert F Bird, Choongseok Chang, Guanye Chen, Stéphane Ethier, Shane Fogerty, Salman Habib, Christoph Junghans, Damien Lebrun-Grandie, Jamaludin Mohd-Yusof, Stan G Moore, Daniel Osei-Kuffuor, Steven J Plimpton, Adrian Pope, Samuel Temple Reeve, Lee Ricketson, Aaron Scheinberg, Amil Y Sharma, and Michael E Wall. Enabling particle applications for exascale computing platforms. Int. J. High Perform. Comput. Appl., 35(6):572–597, 2021.
  21. [21]Anders M. N. Niklasson, C. J. Tymczak, and Matt Challaсombe. Trace resetting density matrix purification in O(N) self-consistent-field theory. The Journal of Chemical Physics, 118(19):8611–8620, 05 2003.DOI
  22. [22][22*] S. Goedecker and M. Teter. Tight-binding electronic structure calculations and tight-binding molecular dynamics with localized orbitals. Phys. Rev. B, 51:9455–9464, Apr 1995.
  23. [23]Christian F. A. Negre, Nicolas Bock, and Susan M Mniszewski. BML, version 2.4, 2023. https://github.com/lanl/bml.
  24. [24]Anders M. Niklasson, Susan M Mniszewski, Christian F. A. Negre, Michael E. Wall, Marc J. Cawkwell, and Nicolas Bock. PROGRESS, version 1.3, 2023. https://github.com/lanl/qmd-progress.
  25. [25]Th. Frauenheim, G. Seifert, M. Elstner, Z. Hajnal, G. Jungnickel, D. Porezag, S. Suhai, and R. Scholz. A self-consistent charge density-functional based tight-binding method for predictive materials simulations in physics, chemistry and biology. physica status solidi (b), 217(1):41–62, 2000.
  26. [26]N. Bock, M. J. Cawkwell, J. D. Coe, A. Krishnapriyan, M. P. Kroonblawd, A. Lang, C. Liu, E. Martinez Saez, S. M. Mniszewski, C. F. A. Negre, A. M. N. Niklasson, E. Sanville, M. A. Wood, and P. Yang. LATTE: Developer repository for the LATTE code, 2023. https://github.com/lanl/LATTE.
  27. [27]Schrödinger, LLC. The PyMOL molecular graphics system, version 1.8. November 2015.
  28. [28]CMake. https://cmake.org, (accessed: 2023-12-19).
  29. [29]I. Stich, R. Car, M. Parrinello, and S. Baroni. Conjugate gradient minimization of the energy functional: A new method for electronic structure calculation. Phys. Rev. B, 39:4997–5004, Mar 1989.pubmed.ncbi.nlm.nih.gov/9948888
  30. [30]G. Kresse and J. Furthmüller. Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set. Computational Materials Science, 6(1):15–50, 1996.DOI
  31. [31]A.R. Tackett, N.A.W. Holzwarth, and G.E. Matthews. A projector augmented wave (PAW) code for electronic structure calculations, part II: pwpaw for periodic solids in a plane wave basis. Computer Physics Communications, 135(3):348–376, 2001.
  32. [32]Chao Yang, Juan C. Meza, and Lin-Wang Wang. A constrained optimization algorithm for total energy minimization in electronic structure calculations. Journal of Computational Physics, 217(2):709–721, 2006.DOI
  33. [33]J-L Fattebert. A robust solver for wavefunction-based density functional theory calculations. Electronic Structure, 4(1):015002, 2022.arxiv.org/abs/2111.11947
  34. [34]François Bottin, Stéphane Leroux, Andrew Knyazev, and Gilles Zérah. Large-scale ab initio calculations based on three levels of parallelization. Computational Materials Science, 42(2):329–336, 2008.DOI
  35. [35]Jürgen C. Becker and Marc Torrent. Parallel eigensolvers in plane-wave density functional theory. Computer Physics Communications, 187:98–105, 2015.DOI
  36. [36]Massimiliano Lupo Pasini, Bruno Turcksin, Wenjun Ge, and Jean-Luc Fattebert. A parallel strategy for density functional theory computations on accelerated nodes. Parallel Computing, 100:102703, 2020.DOI
  37. [37]Joshua Finkelstein, Christian F. A. Negre, and Jean-Luc Fattebert. A fast, dense Chebyshev solver for electronic structure on GPUs. J. Chem. Phys., 159(10):101101, 2023.arxiv.org/abs/2306.12616
  38. [38]Sameer Khadatkar and Phani Motamarri. Subspace recursive Fermi-operator expansion strategies for large-scale DFT eigenvalue problems on HPC architectures. The Journal of Chemical Physics, 159(3):031102, 07 2023.DOI
  39. [40]Stephan Mohr, William Dawson, Michael Wagner, Damien Caliste, Takahito Nakajima, and Luigi Genovese. Efficient computation of sparse matrix functions for large-scale electronic structure calculations: The CheSS library. Journal of Chemical Theory and Computation, 13(10):4684–4698, 2017.DOI
  40. [41]Michael S. Paterson and Larry J. Stockmeyer. On the number of nonscalar multiplications necessary to evaluate polynomials. SIAM Journal on Computing, 2(1):60–66, 1973.DOI
  41. [42]WanZhen Liang, Chandra Saravanan, Yihan Shao, Roi Baer, Alexis T Bell, and Martin Head-Gordon. Improved Fermi operator expansion methods for fast electronic structure calculations. J. Chem. Phys., 119(8):4117–4125, 2003.DOI
  42. [43]Wan Zhen Liang, Roi Baer, Chandra Saravanan, Yihan Shao, Alexis T Bell, and Martin Head-Gordon. Fast methods for resumming matrix polynomials and Chebyshev matrix polynomials. J. Comp. Phys., 194(2):575–587, 2004.DOI
  43. [44]OpenMP Application Programming Interface. https://www.openmp.org/spec-html/5.2/openmp.html.
  44. [45]Bronis R. de Supinski, Thomas R. W. Scogland, Alejandro Duran, Michael Klemm, Sergi Mateo Bellido, Stephen L. Olivier, Christian Terboven, and Timothy G. Mattson. The ongoing evolution of OpenMP. Proceedings of the IEEE, 106(11):2004–2019, 2018.DOI
  45. [46]OpenMP Compilers and Tools. https://www.openmp.org/resources/openmp-compilers-tools.
  46. [47]Seonmyeong Bak, Colleen Bertoni, Swen Boehm, Reuben Budidjarja, Barbara M. Chapman, Johannes Doerfert, Markus Eisenbach, Hal Finkel, Oscar Hernandez, Joseph Huber, Shintaro Iwasaki, Vivek Kale, Paul R.C. Kent, JaeHyuk Kwack, Meifeng Lin, Piotr Luszczek, Ye Luo, Buu Pham, Swaroop Pophale, Kiran Ravikumar, Vivek Sarkar, Thomas Scogland, Shilei Tian, and P.K. Yeung. OpenMP application experiences: Porting to accelerated nodes. Parallel Computing, 109:102856, 2022.DOI
  47. [48]Jean-Luc Fattebert, Stephen DeWitt, Aurelien Perron, and John Turner. Thermo4PFM: Facilitating phase-field simulations of alloys with thermodynamic driving forces. Computer Physics Communications, 288:108739, 2023.
  48. [49]Adrian S. Sabau, Lang Yuan, Jean-Luc Fattebert, and John A. Turner. An OpenMP GPU-offload implementation of a non-equilibrium solidification cellular automata model for additive manufacturing. Computer Physics Communications, 284:108605, 2023.DOI
  49. [50]Ye Luo, Peter Doak, and Paul Kent. A high-performance design for hierarchical parallelism in the QMCPACK Monte Carlo code. In 2022 IEEE/ACM International Workshop on Hierarchical Parallelism for Exascale Computing (HiPar), pages 22–27, 2022.DOI
  50. [51]Dipayan Datta and Mark S. Gordon. Accelerating coupled-cluster calculations with GPUs: An implementation of the density-fitted CCSD(T) approach for heterogeneous computing architectures using OpenMP directives. Journal of Chemical Theory and Computation, 19(21):7640–7657, 2023.DOI
  51. [52]Buu Q. Pham, Melisa Alkan, and Mark S. Gordon. Porting fragmentation methods to graphical processing units using an OpenMP application programming interface: Offloading the Fock build for low angular momentum functions. Journal of Chemical Theory and Computation, 19(8):2213–2221, 2023.DOI
  52. [53]Fazlay Rabbi, Christopher S. Daley, Hasan Metin Aktulga, and Nicholas J. Wright. Evaluation of directive-based gpu programming models on a block eigensolver with consideration of large sparse matrices. In Sandra Wienke and Sridutt Bhalachandra, editors, Accelerator Programming Using Directives, pages 66–88, Cham, 2020. Springer International Publishing.DOI
  53. [54]Jamaludin Mohd-Yusof and Nikolay Sakharnykh. Fast sparse matrix multiplication for QMD using parallel merge. In GPU Technology Conference (GTC), 2015.
  54. [55]MAGMA: Matrix Algebra on GPU and Multicore Architectures. https://icl.utk.edu/magma/index.html.
  55. [56]Intel oneAPI Base Toolkit. https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit.html.
  56. [57]Aurora Media Kit. https://www.alcf.anl.gov/aurora-media-kit.
  57. [58]hypre: High performance preconditioners. https://llnl.gov/casc/hypre, https://github.com/hypre-space/hypre.DOI
  58. [59]Robert D. Falgout, Ruipeng Li, Björn Sjögreen, Lu Wang, and Ulrike Meier Yang. Porting hypre to heterogeneous computer architectures: Strategies and experiences. Parallel Computing, 108:102840, 2021.DOI
  59. [60]L. Blackford, J. Choi, A. Cleary, J. Demmel, Inderjit S. Dhillon, J. Dongarra, S. Hammarling, G. Henry, A. Petitet, K. Stanley, D. Walker, and R. C. Whaley. ScaLAPACK Users’ Guide. SIAM, 1997.DOI
  60. [61]Weitao Yang and Tai-Sung Lee. A density-matrix divide-and-conquer approach for electronic structure calculations of large molecules. The Journal of Chemical Physics, 103(13):5674–5678, 10 1995.DOI
  61. [62]Victor Wen zhe Yu, Jonathan Moussa, Pavel Kůs, Andreas Marek, Peter Messmer, Mina Yoon, Hermann Lederer, and Volker Blum. GPU-acceleration of the ELPA2 distributed eigensolver for dense symmetric and hermitian eigenproblems. Computer Physics Communications, 262:107808, 2021.DOI
  62. [63]Anders M N Niklasson, Susan M Mniszewski, Christian F A Negre, Marc J Cawkwell, Pieter J Swart, Jamal Mohd-Yusof, Timothy C Germann, Michael E Wall, Nicolas Bock, Emanuel H Rubensson, and Hristo Djidjev. Graph-based linear scaling electronic structure theory. J. Chem. Phys., 144(23):234101, June 2016.DOI
  63. [64]Hristo N. Djidjev, Georg Hahn, Susan M. Mniszewski, Christian F.A. Negre, Anders M.N. Niklasson, and Vivek B. Sardeshmukh. Graph Partitioning Methods for Fast Parallel Quantum Molecular Dynamics, pages 42–51. Society for Industrial and Applied Mathematics, 2016.arxiv.org/abs/1605.01118
  64. [65]Purnima Ghale, Matthew P Kroonblawd, Sue Mniszewski, Christian F A Negre, Robert Pavel, Sergio Pino, Vivek Sardeshmukh, Guangjie Shi, and Georg Hahn. Task-based parallel computation of the density matrix in quantum-based molecular dynamics using graph partitioning. SIAM J. Sci. Comput., 39(6):C466–C480, January 2017.DOI
  65. [66]William Dawson and Takahito Nakajima. Massively parallel sparse matrix function calculations with NTPOLy. Comput. Phys. Commun., 225:154–165, April 2018.
  66. [67]Hamilton Wilfried Yves Adoni, Tarik Nahhal, Moez Krichen, Brahim Aghezzaf, and Abdeltif Elbyed. A survey of current challenges in partitioning and processing of graph-structured data in parallel and distributed systems. Distrib. Parallel Databases, 38(2):495–530, 2020.DOI
  67. [68]METIS - serial graph partitioning and fill-reducing matrix ordering. http://glaros.dtc.umn.edu/gkhome/metis/metis/overview. Accessed: 2022-5-3.
  68. [69]B W Kernighan and S Lin. An efficient heuristic procedure for partitioning graphs. The Bell System Technical Journal, 49(2):291–307, February 1970.DOI

Paper details

vlnc:W_b6jsgbjs
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents