betaFeedback

WorksPhysics and Astronomy

Efficient implementation of the quasiparticle self-consistent GWGW method on GPU

Abstract

We have developed a multi-GPU version of the quasiparticle self-consistent GWGW (QSGW), a cutting-edge method for describing electronic excitations in a first-principles approach. While the QSGW calculation algorithm is inherently well-suited for GPU computation due to its reliance on large-scale tensor operations, achieving a maintainable and extensible implementation is not straightforward. Addressing this, we have developed a GPU version within the ecalj package, utilizing module-based programming style in modern Fortran. This design facilitates future development and code sustainability. Following the summary of the QSGW formalism, we present our GPU implementation approach and the results of benchmark calculations for two types of systems to demonstrate the capability of our GPU-supported QSGW calculations.

Graphical Abstract

Figure Graphical Abstract. Σ~C\widetilde{\Sigma}^{C}, WCW^{C}, Σx\Sigma^{x}, ΣCx\Sigma^{Cx}, VQSGWxcV_{\mathrm{QSGW}}^{xc}, εqn\varepsilon_{qn}, Ψqn\Psi_{qn}; CPUs; multi-GPU; QSGW80+SO; Ref. [Otsuka]; Exp. [PL]; Exp. [Abs]; Over 10 times acceleration with 4 GPUs vs. 4 CPU nodes — enabling wider application of QSGW.

Highlights

Efficient implementation of the quasiparticle self-consistent GWGW method on GPU

Masao Obata, Takao Kotani, Tatsuki Oda

  • Efficient QSGW calculations leveraging multi-GPU acceleration. Achieved significant speedup in QSGW calculations utilizing cutting-edge GPU technology; A GPU node achieved 10-fold speedup over four CPU nodes. Our implementation ensures maintainability and facilitates future development.

  • Enhanced applicability to large-scale systems. Introduced a mixed precision approach, achieving a speed increase of more than 3 times while preserving numerical accuracy, thereby enabling further applications to large-scale systems.

  • Accurate electronic structures and new physical insights. The electronic structures obtained from QSGW calculations agree well with the experimental band gaps of InAs/GaSb superlattices, and suggest a possible structural stabilization mechanism in Ni2MnGa\mathrm{Ni}_{2}\mathrm{MnGa}, attributed to a valley structure in the density of states near the Fermi level.

Introduction

The first-principles GWGW approximation (GWA) [1, 2] based on the many-body perturbation theory (MBPT) [3] is a powerful method to describe electronic excitations of materials. GWA contains not only the exchange effect in the Hartree-Fock theory but also the correlational effect, where we use the dynamical screened Coulomb interaction WW instead of the bare Coulomb interaction vv. WW is given in the random phase approximation (RPA), which Hedin and Lundqvist called the linearized time-dependent Hartree approximation [3]. The quasiparticle (QP) energies in MBPT imply the energies to add or remove an electron from the system, directly corresponding to the experimental spectra obtained from photoemission spectroscopy. GWA has been employed to calculate QP energies as perturbative corrections to the eigenvalues obtained from the local density approximation (LDA) or the generalized gradient approximation (GGA) within density functional theory (DFT). Hybertsen and Louie pioneered the first-principles GWA in 1986, demonstrating that it accurately predicts the band gaps of simple semiconductors [4]. This approach is commonly referred to as the one-shot GWGW method. However, the one-shot GWGW method can be problematic in terms of self-consistency. In particular, it tends to underestimate band gaps significantly in certain materials, such as ZnO and InN [2, 5, 6]. In cases where there is no band gap at the LDA level, such as Ge, the one-shot GWGW cannot modify connectivity of eigenvalue dispersion in the Brillouin zone because no off-diagonal matrix elements are included [7].

To overcome the problem of self-consistency in the one-shot GWGW, we developed the quasiparticle self-consistent GWGW (QSGW) method, in which a self-consistent one-body Hamiltonian H0H^0 is obtained within GWA, offering a good independent-particle picture for describing electronic excitations in materials [7, 8]. QSGW provides an improved description of the band gaps and the effective masses compared to the experimental values [2, 7, 9]. It also accurately describes the positions and bandwidths of localized 3d3d orbitals [10, 11, 12]. This is in contrast to the cases of LDA/GGA, which often give too extended 3d3d orbitals. Note that such localization changes cannot be described in the one-shot GW, which does not treat changes of eigenfunctions from LDA/GGA. The effect of UU in the LDA+UU method is rather naturally incorporated in QSGW through WW, which is determined self-consistently within the QSGW cycle.

We had implemented QSGW in the ecalj package [2, 13]. We can handle even metals because we use the coarser kk mesh for the self-energy calculation to reduce too much computational effort, rather than the mesh used for the self-consistency of the charge. Furthermore, we can handle magnetic materials [14, 15, 16]. Since a self-energy interpolation in the Brillouin zone is built-in, we can generate eigenfunctions at any kk points without requiring additional procedures, such as Wannier interpolations [17]. Based on the results of QSGW, linear response functions such as dielectric functions, magnetic susceptibilities, impact ionization energies, etc, can be calculated, enabling a deeper understanding of material properties and facilitating the design of advanced materials for various applications, including electronics, optoelectronics, and spintronics. This procedure may resolve the intrinsic problems related to the electronic excitations, which are hardly described in LDA. For example, we can perform very high-resolution calculations on spin waves [18, 19], virtually equivalent to the 100 × 100 × 100 division of the Brillouin zone.

Therefore, QSGW is promising for the application of a wide range of materials, not only bulk materials but also interfaces, surfaces, and so on. For such applications, we have two problems. The first is to achieve automatic and robust computation just from atomic positions without manual setting of various computational settings. The second is how to increase the computational speed of QSGW, which requires 100∼1000 times computational effort rather than LDA/GGA. This article focuses on the latter problem, while the first point is described elsewhere [20, 21].

In this paper, we have implemented the multiple graphics processing unit (GPU) version of QSGW in ecalj to increase the computational speed. Although recent studies have explored QSGW calculations accelerated by GPUs, detailed descriptions of implementation strategies and performance evaluations are still lacking [22]. Furthermore, our implementation offers advantages due to the choice of basis set used to represent the eigenfunctions. Specifically, we have implemented QSGW on the PMT method [20], which employs a mixed basis set consisting of both Muffin-tin orbitals (MTOs) and augmented plane waves (APWs) [23]. The use of an appropriate basis set reduced the need for manual tuning of basis-set parameters [20]. The PMT method enables us to treat systems involving vacuum regions, such as slab supercell models [24], which are difficult to handle using MTOs alone. Our implementation is written in an object-oriented style using Fortran modules, making the code structure transparent and maintainable. The source code is publicly available on GitHub [13].

After showing the computational algorithm in QSGW with key formulas in § 4, we will explain how to implement QSGW on multiple GPUs in § 5. In § 7, we demonstrate the performance of the implementation for two examples, Type-II superlattice InAs/GaSb and the ferromagnetic shape-memory alloy Ni2_2MnGa. Both are essential materials from the perspective of practical applications.

Theory and Algorithm to perform QSGW

Formulation of QSGW

QSGW is a method to determine the one-body Hamiltonian H0H^0 of the independent-particle picture of electronic excitation [8]. H0H^0 gives eigenvalues and eigenfunctions of QPs. Theoretically, the existence of H0H^0 of QPs is supported by Landau’s Fermi liquid theory [25], where QPs are considered to be the fundamental excitations of electronic systems. Note that QPs can be well illustrated for insulators with band gaps: we have well-defined quasiparticles with infinite lifetime below the impact ionization threshold [23, 26], as long as we neglect phonons. This is because there are no decay channels for such QPs [24]. QPs represent the dressed electrons propagating in materials. Strictly speaking, QPs are defined only well below the threshold. In this sense, H0H^0 is just a theoretical construction to calculate physical quantities on top of H0H^0.

Let us start from the basics of GWA. Although any H0H^0 can be chosen, we start from the Kohn-Sham Hamiltonian that incorporates the exchange-correlation potential VxcV^{\mathrm{xc}} within LDA. From such H0H^0, we have a set of eigenvalues εkn\varepsilon_{kn} and eigenfunctions Ψkn(r)\Psi_{kn}(\mathbf{r}) that satisfy H0Ψkn(r)=εknΨkn(r)H^0\Psi_{kn}(\mathbf{r}) = \varepsilon_{kn}\Psi_{kn}(\mathbf{r}). GWA gives the self-energy as Σ(r,r′,t−t′)=iG(r,r′,t−t′)W(r,r′,t−t′+δ)\Sigma(\mathbf{r},\mathbf{r}',t-t') = iG(\mathbf{r},\mathbf{r}',t-t')W(\mathbf{r},\mathbf{r}',t-t' + \delta), where the infinitesimal positive δ\delta is required to ensure that the exchange term is correct. In ω\omega space, we have Σ(r,r′,ω)\Sigma(\mathbf{r},\mathbf{r}',\omega) as

Σ(r,r′,ω)=i2π∫G(r,r′,ω−ω′)W(r,r′,ω′)e−iδω′ dω′,(1)\Sigma(\mathbf{r},\mathbf{r}',\omega) = \frac{i}{2\pi}\int G(\mathbf{r},\mathbf{r}',\omega-\omega')W(\mathbf{r},\mathbf{r}',\omega')e^{-i\delta\omega'}\,d\omega', \tag*{(1)}

where the Green’s function GG and the dynamical screened Coulomb interaction WW are made of εkn\varepsilon_{kn} and Ψkn(r)\Psi_{kn}(\mathbf{r}). This Σ(r,r′,ω)\Sigma(\mathbf{r},\mathbf{r}',\omega) plays the role of VLDAxcV_{\mathrm{LDA}}^{\mathrm{xc}} in LDA in determining QPs. In other words, we obtain the one-body Green’s function

G=1ω−(H0+Σ−Vxc)(2)G = \frac{1}{\omega-(H^0+\Sigma-V^{\mathrm{xc}})} \tag*{(2)}

in GWA. We can apply GWA to H0H^0 that contains any VxcV^{\mathrm{xc}}. As discussed in § 3, the one-shot GWGW evaluates only the diagonal part of Σ−Vxc\Sigma-V^{\mathrm{xc}} as the correction of the eigenvalues. The one-shot GWGW is often referred to as G0W0G_0W_0. It is important to note the difference in purpose between one-shot GWGW and QSGW. The one-shot GWGW is to evaluate the difference between quasiparticle energies and eigenvalues in DFT, while QSGW is to determine the one-body Hamiltonian H0H^0.

As discussed in § 3, a problem with the one-shot GWGW is its lack of self-consistency. For example, in systems such as the InAs/GaSb superlattice, which we will use as a sample in the benchmark section 4, we expect some corrected charge transfer between the layers if the electron density is evaluated using the corrected GG in GWA. To take into account such a corrected charge transfer, a new theoretical framework will be required beyond the one-shot GWGW. This can be achieved by self-consistency; we should employ the scheme of self-consistent perturbation theory. Specifically, we need to choose the starting H0H^0 so that the perturbation correction Σ−Vxc\Sigma-V^{\mathrm{xc}} is minimized. This is the idea of QSGW originally introduced in 2004 [10]. In QSGW, we will repeat the self-consistency cycle until Σ−Vxc\Sigma-V^{\mathrm{xc}} is minimized.

To achieve a closed-form self-consistency cycle in QSGW, we need to remove the energy dependence of Σ\Sigma in GWA in Eq. (2). That is, we replace Σ\Sigma with the static self-energy VQSGWxcV_{\mathrm{QSGW}}^{\mathrm{xc}}. Thus, in practice, the QSGW iteration cycle is given as in Fig. 1. At the top of Fig. 1, for a given ΔVxc\Delta V^{\mathrm{xc}} keeping, we determine H0H^0 such that the electron density n(r)n(\mathbf{r}) is self-consistently obtained. H0H^0 contains an extra term, a non-local static potential ΔVxc=VQSGWxc−VLDAxc\Delta V^{\mathrm{xc}}=V_{\mathrm{QSGW}}^{\mathrm{xc}}-V_{\mathrm{LDA}}^{\mathrm{xc}}, which is the difference in the exchange-correlation term between QSGW and LDA. At the beginning of the iteration cycle, we set ΔVxc\Delta V^{\mathrm{xc}} to zero. In the next step, we generate the eigenvalues and eigenfunctions of H0H^0. Subsequently, after some steps (Green, Red, and Blue boxes in Fig. 1), we finally have ΔVxc\Delta V^{\mathrm{xc}}. The four computationally-expensive steps for calculating VQSGWxcV_{\mathrm{QSGW}}^{\mathrm{xc}}, depicted in red and blue boxes, are as follows.

  1. Core-exchange self-energy ΣCx\Sigma^{\mathrm{Cx}}

  2. Exchange self-energy Σx\Sigma^{\mathrm{x}}

  3. The dynamical screened Coulomb interaction WW (red box in Fig. 1)

  4. Static version of correlation self-energy Σ~c\widetilde{\Sigma}^{\mathrm{c}}

We need WW to calculate the static version of the correlation self energy Σ~c\widetilde{\Sigma}^{\mathrm{c}}. The matrix elements of the correlation self-energy Σ~c\widetilde{\Sigma}^{\mathrm{c}} is written as

Σ~nmc(k)=⟨Ψkn|Re⁡[Σc(εkn)+Σc(εkm)2]|Ψkm⟩,(3)\widetilde{\Sigma}^{\mathrm{c}}_{nm}(\mathbf{k}) = \left\langle\Psi_{kn}\middle|\operatorname{Re}\left[\frac{\Sigma^{\mathrm{c}}(\varepsilon_{kn})+\Sigma^{\mathrm{c}}(\varepsilon_{km})}{2}\right]\middle|\Psi_{km}\right\rangle, \tag*{(3)}

where Re means taking the Hermitian part. To eliminate the ω\omega dependence of Σc\Sigma^{\mathrm{c}}, we use Eq. (3), which is one of the reasonable choices [8]. Together with other terms, we obtain VQSGWxc=ΣCx+Σx+Σ~cV_{\mathrm{QSGW}}^{\mathrm{xc}}=\Sigma^{\mathrm{Cx}}+\Sigma^{\mathrm{x}}+\widetilde{\Sigma}^{\mathrm{c}} in the expansion of Ψkn\Psi_{kn} of H0H^0. After self-consistency is achieved, {Ψkn}\{\Psi_{kn}\} used for expansion in Eq. (3) is in agreement with {Ψkn}\{\Psi_{kn}\} generated in the next step of the iteration cycle.

We divide all electrons into core and valence electrons by the default setting in ecalj. Except for cores, we treat all electrons as valence electrons. We use the frozen-core approximation; core electron densities are fixed to be those calculated for spherical atoms in the top box of Figure 1. However, we generate core eigenfunctions restricted within Muffin-tin (MT) regions for the calculation of ΣCx\Sigma^{\mathrm{Cx}}. We neglect contributions of cores to WW, based on the examination in Ref. [8]. Thus, the cores affect the valence electrons only through ΣCx\Sigma^{\mathrm{Cx}}, in addition to their frozen electron density. This treatment of core electrons is justified by comparing cases in which the cores are treated as frozen cores or as valence electrons in the local orbital method.

Figure 1. Self-consistent cycle of QSGW. The name of the program that performs each task is noted at the top right corner of the task box. The top lmf, which solves the one-particle equation with given ΔVxc\Delta V^{\mathrm{xc}}, performs self-consistent calculations. The ΔVxc\Delta V^{\mathrm{xc}} is set to zero in the initial loop. The QSGW cycle continues iteratively, checking if the difference between the new and old diagonal terms of self-energy is below a certain threshold ΔQP\Delta_{\mathrm{QP}}. If not, the process repeats.

An empirical correction can be introduced in QSGW to improve agreement with experimental values. This is because QSGW gives a systematic overestimation of exchange effects [2]. This overestimation is reasonable considering the nature of QSGW because the vertex corrections [27, 28] are not considered. Since the vertex correction is expected to reduce the energies of the intermediate states due to the correlational motions between electrons and holes, we anticipate that the actual effective interaction between electrons will be smaller than that evaluated in QSGW. The electrons should be polarized more easily than they would be without the vertex correction. To account for this effect of vertex correction in a simple manner, we can use QSGW80, a hybrid with a mixing ratio of 80% QSGW and 20% LDA, rather than 100% QSGW (QSGW100). We have investigated the capability of QSGW80, demonstrating a universal improvement in agreement with experimental data on band gaps and effective masses in semiconductors and insulators [2]. For QSGW80, we use 0.8×ΔVxc0.8 \times\Delta V^{\mathrm{xc}} instead of ΔVxc\Delta V^{\mathrm{xc}}.

Hereafter, we will show detailed equations for the four steps, followed by the representation of eigenfunctions in the MT division of space.

Representation of eigenfunctions in the MT division of space

Our method is based on an all-electron electronic structure calculation in the PMT method [23, 29], implemented in the ecalj package [13]. The PMT is a linearized mixed-basis method that utilizes not only MTOs but also APWs to represent the eigenfunctions, employing the MT division of the space. In this context, the PMT is a combination of linearized MTO (LMTO) and linearized APW (LAPW) methods. The APW effectively represents the smooth behavior of eigenfunctions in the interstitial region, whereas the MTO effectively represents the eigenfunctions just outside of MTs. In ecalj, the parameters for defining MTOs are determined in a simple manner. This is because APWs, with a cutoff energy of approximately 3 Ry, can effectively complement the expansion of eigenfunctions in the interstitial region, thereby avoiding the need to tune the parameters of MTOs [23]. Since the MTO part can expand the eigenfunctions near the MT boundaries very well, we can roughly take the cutoff energy of the APWs as that of the computed eigenfunctions relative to some averaged level of interstitial potential. Exactly speaking, we also use local orbitals [30]. The local orbitals are important in cases; for example, we represent 3d3d orbitals with local orbitals for Ga, 2p2p for Na as well.

For given H0H^0, we obtain eigenfunctions Ψkn(r)\Psi_{\mathbf{k}n}(\mathbf{r}) expanded in the PMT bases. We re-expand Ψkn(r)\Psi_{\mathbf{k}n}(\mathbf{r}) in the MT space of division as the sum of the augmentation parts within the MTs and the plane wave (PW) parts in the interstitial region as follows:

Ψkn(r)=∑RuαRuknφRuk(r)+∑GβGknPGk(r),(4)\Psi_{\mathbf{k}n}(\mathbf{r}) = \sum_{Ru} \alpha^{\mathbf{k}n}_{Ru}\varphi^{\mathbf{k}}_{Ru}(\mathbf{r}) + \sum_{\mathbf{G}} \beta^{\mathbf{k}n}_{\mathbf{G}}P^{\mathbf{k}}_{\mathbf{G}}(\mathbf{r}), \tag*{(4)}

where the interstitial plane wave (IPW) is defined as

PGk(r)={0if r∈any MTexp⁡(i(k+G)⋅r)otherwise(5)P^{\mathbf{k}}_{\mathbf{G}}(\mathbf{r}) = \begin{cases} 0 & \text{if } \mathbf{r} \in\text{any MT} \\ \exp\left(i(\mathbf{k}+\mathbf{G}) \cdot\mathbf{r}\right) & \text{otherwise} \end{cases} \tag*{(5)}

and φRuk(r)\varphi^{\mathbf{k}}_{Ru}(\mathbf{r}) are Bloch sums of the atomic functions φRu(r)\varphi_{Ru}(\mathbf{r}) defined within the MT at R\mathbf{R},

φRuk(r)≡∑TφRu(r−R−T)exp⁡(ik⋅T).(6)\varphi^{\mathbf{k}}_{Ru}(\mathbf{r}) \equiv\sum_{\mathbf{T}} \varphi_{Ru}(\mathbf{r}-\mathbf{R}-\mathbf{T})\exp(i\mathbf{k}\cdot\mathbf{T}). \tag*{(6)}

T\mathbf{T} and G\mathbf{G} are lattice translation vectors in real and reciprocal spaces, respectively.

Overview of QSGW calculations

In our method to perform GWA, we need not only the basis set to expand eigenfunctions but also the basis set to expand the product of eigenfunctions. The basis set is referred to as the mixed product basis set (MPB) {MIk(r)}\{M^{\mathbf{k}}_I(\mathbf{r})\}, introduced by Kotani in Ref. [29]. The MPB consists of the product basis (PB) within MTs, by Aryasetiawan and Gunnarsson [31], and the IPW in the interstitial region. Since {MIk(r)}\{M^{\mathbf{k}}_I(\mathbf{r})\} contains IPWs that are not orthogonal, we have an overlap matrix OIJk=⟨MIk∣MJk⟩O^{\mathbf{k}}_{IJ} = \langle M^{\mathbf{k}}_I \mid M^{\mathbf{k}}_J \rangle. We introduce an orthonormal MPB {Eμk}\{E^{\mathbf{k}}_\mu\} that diagonalizes the Coulomb matrix for convenience. An orthonormal MPB was originally introduced by Friedrich, Blügel, and Schindlmayr in Ref. [32]. Let the transformation coefficients from {MIk}\{M^{\mathbf{k}}_I\} to {Eμk}\{E^{\mathbf{k}}_\mu\} be zμIkz^{\mathbf{k}}_{\mu I}, and let νμ(k)\nu_\mu(\mathbf{k}) be the eigenvalue of ν(k)\nu(\mathbf{k}), i.e., ∣Eμk⟩=∑IzμIk∣MIk⟩|E^{\mathbf{k}}_\mu\rangle= \sum_I z^{\mathbf{k}}_{\mu I}|M^{\mathbf{k}}_I\rangle, ν(k)∣Eμk⟩=νμ(k)∣Eμk⟩\nu(\mathbf{k})|E^{\mathbf{k}}_\mu\rangle= \nu_\mu(\mathbf{k})|E^{\mathbf{k}}_\mu\rangle. The values of zμIkz^{\mathbf{k}}_{\mu I} and νμ(k)\nu_\mu(\mathbf{k}) are obtained by solving the generalized eigenvalue equation, ∑J(vIJk−νμ(k)OIJk)zμJk=0\sum_J(v^{\mathbf{k}}_{IJ}-\nu_\mu(\mathbf{k})O^{\mathbf{k}}_{IJ})z^{\mathbf{k}}_{\mu J}=0, where vIJk=⟨MIk∣ν∣MJk⟩v^{\mathbf{k}}_{IJ}=\langle M^{\mathbf{k}}_I|\nu|M^{\mathbf{k}}_J\rangle. By using these, the Coulomb interaction can be represented by matrix elements as follows:

ν(k)=∑μ∣Eμk⟩νμ(k)⟨Eμk∣.(7)\nu(\mathbf{k}) = \sum_\mu|E^{\mathbf{k}}_\mu\rangle\nu_\mu(\mathbf{k}) \langle E^{\mathbf{k}}_\mu|. \tag*{(7)}

Equation (7), as introduced in Ref. [32], is used in the all-electron full-potential GWA. This corresponds to the representation in the plane wave expansion of the Coulomb interaction, given by ν(k+G,k+G′)=4πδGG′∣k+G∣2\nu(\mathbf{k}+\mathbf{G},\mathbf{k}+\mathbf{G}') = \frac{4\pi\delta_{\mathbf{G}\mathbf{G}'}}{|\mathbf{k}+\mathbf{G}|^2}. The case of μ=1\mu=1 corresponds to the largest eigenvalue of νμ\nu_\mu, which is νμ=1(k)∼4π∣k∣2\nu_{\mu=1}(\mathbf{k}) \sim\frac{4\pi}{|\mathbf{k}|^2}. We handle this divergence numerically by replacing 1/r1/r with exp⁡(−κrr)/r\exp(-\kappa_{\mathrm{r}}r)/r, where κr\kappa_{\mathrm{r}} is small enough (1×10−5 bohr−11 \times10^{-5}\ \mathrm{bohr}^{-1}, usually). By introducing {Eμk}\{E_{\mu}^{k}\} as the new MPB, our GW calculations are formulated using the matrix elements ZZ in terms of EμkE_{\mu}^{k} and the wave functions Ψ\Psi which are written as follows:

Zkμq−kn′,qn=⟨EμkΨq−kn′∣Ψqn⟩.(8)Z_{k\mu}^{\mathbf{q}-\mathbf{k}n',\mathbf{q}n}=\langle E_{\mu}^{k}\Psi_{\mathbf{q}-\mathbf{k}n'}\mid\Psi_{\mathbf{q}n}\rangle. \tag*{(8)}

We expect that the set of mixed product bases {MJk}\{M_{J}^{k}\} (or {Eμq}\{E_{\mu}^{q}\}, equivalently) must have the ability to expand the product of eigenfunctions very well. However, we need to keep the number of basis functions as small as possible to minimize computational efforts. Since the size of the Zμqq−kn,kn′Z_{\mu q}^{\mathbf{q}-\mathbf{k}n,\mathbf{k}n'} array can be significantly large, we prepare it on the fly for computations. The expansion (4) is designed for the convenience of computing. In the following GW calculations, we use Zμqkn,q+kn′Z_{\mu q}^{\mathbf{k}n,\mathbf{q}+\mathbf{k}n'} with vμ(k)v_{\mu}(\mathbf{k}), in addition to eigenvalues.

The exchange self-energy is written as

Σnmx(q)=⟨Ψqn∣Σx∣Ψqm⟩=−∑kBZ∑n′occ∑μ[Zkμq−kn′,qn]∗vμ(k)Zkμq−kn′,qm.(9)\begin{aligned} \Sigma_{nm}^{\mathrm{x}}(\mathbf{q})=\langle\Psi_{\mathbf{q}n}\mid\Sigma^{\mathrm{x}}\mid\Psi_{\mathbf{q}m}\rangle\\ &=-\sum_{\mathbf{k}}^{\mathrm{BZ}}\sum_{n'}^{\mathrm{occ}}\sum_{\mu}\left[Z_{k\mu}^{\mathbf{q}-\mathbf{k}n',\mathbf{q}n}\right]^{*}v_{\mu}(\mathbf{k})Z_{k\mu}^{\mathbf{q}-\mathbf{k}n',\mathbf{q}m}. \tag*{(9)} \end{aligned}

Replacing occupied valence states in (9) with core states, we can calculate core exchange ΣCx\Sigma^{\mathrm{Cx}}. Note that the MPB for ΣCx\Sigma^{\mathrm{Cx}} differs from that for valence electrons because the former must represent products of the core and valence eigenfunctions. Specifically, we need to evaluate Zkμq−kn′,qnZ_{k\mu}^{\mathbf{q}-\mathbf{k}n',\mathbf{q}n}, where the intermediate states Ψq−kn′\Psi_{\mathbf{q}-\mathbf{k}n'} belong to the cores just within MTs. The screened Coulomb interaction Wμν(q,ω)W_{\mu\nu}(\mathbf{q},\omega) is calculated from the dielectric function ϵμν(q,ω)\epsilon_{\mu\nu}(\mathbf{q},\omega) and the Lindhard polarization function Πμν(q,ω)\Pi_{\mu\nu}(\mathbf{q},\omega) i.e,

ϵμν(q,ω)=δμν−vμ(q)Πμν(q,ω)vν(q),(10)\epsilon_{\mu\nu}(\mathbf{q},\omega)=\delta_{\mu\nu}-\sqrt{v_{\mu}(\mathbf{q})}\Pi_{\mu\nu}(\mathbf{q},\omega)\sqrt{v_{\nu}(\mathbf{q})}, \tag*{(10)}
Wμν(q,ω)=vμ(q)ϵμν−1(q,ω)vν(q).(11)W_{\mu\nu}(\mathbf{q},\omega)=\sqrt{v_{\mu}(\mathbf{q})}\epsilon_{\mu\nu}^{-1}(\mathbf{q},\omega)\sqrt{v_{\nu}(\mathbf{q})}. \tag*{(11)}

where the Πμν(q,ω)\Pi_{\mu\nu}(\mathbf{q},\omega) is written as

Πμν(q,ω)=∑kBZ∑nocc∑n′unoccZqμkn,q+kn′[Zqνkn,q+kn′]∗ω−(ϵq+kn′−ϵkn)+iδ+∑kBZ∑nunocc∑n′occZqμkn,q+kn′[Zqνkn,q+kn′]∗−ω−(ϵkn−ϵq+kn′)+iδ(12)\Pi_{\mu\nu}(\mathbf{q},\omega)=\sum_{\mathbf{k}}^{\mathrm{BZ}}\sum_{n}^{\mathrm{occ}}\sum_{n'}^{\mathrm{unocc}}\frac{Z_{q\mu}^{\mathbf{k}n,\mathbf{q}+\mathbf{k}n'}[Z_{q\nu}^{\mathbf{k}n,\mathbf{q}+\mathbf{k}n'}]^{*}}{\omega-(\epsilon_{\mathbf{q}+\mathbf{k}n'}-\epsilon_{\mathbf{k}n})+i\delta}+\sum_{\mathbf{k}}^{\mathrm{BZ}}\sum_{n}^{\mathrm{unocc}}\sum_{n'}^{\mathrm{occ}}\frac{Z_{q\mu}^{\mathbf{k}n,\mathbf{q}+\mathbf{k}n'}[Z_{q\nu}^{\mathbf{k}n,\mathbf{q}+\mathbf{k}n'}]^{*}}{-\omega-(\epsilon_{\mathbf{k}n}-\epsilon_{\mathbf{q}+\mathbf{k}n'})+i\delta} \tag*{(12)}
=∑kBZ∑nocc∑n′unoccZqμkn,q+kn′[Zqνkn,q+kn′]∗ω−(ϵq+kn′−ϵkn)+iδ+∑kBZ∑nocc∑n′unoccZqμ−kn∗,−q−kn′[Zqν−kn∗,−q−kn′]∗−ω−(ϵ−q−kn′−ϵ−kn)+iδ.(13)=\sum_{\mathbf{k}}^{\mathrm{BZ}}\sum_{n}^{\mathrm{occ}}\sum_{n'}^{\mathrm{unocc}}\frac{Z_{q\mu}^{\mathbf{k}n,\mathbf{q}+\mathbf{k}n'}[Z_{q\nu}^{\mathbf{k}n,\mathbf{q}+\mathbf{k}n'}]^{*}}{\omega-(\epsilon_{\mathbf{q}+\mathbf{k}n'}-\epsilon_{\mathbf{k}n})+i\delta}+\sum_{\mathbf{k}}^{\mathrm{BZ}}\sum_{n}^{\mathrm{occ}}\sum_{n'}^{\mathrm{unocc}}\frac{Z_{q\mu}^{-\mathbf{k}n^{*},-\mathbf{q}-\mathbf{k}n'}[Z_{q\nu}^{-\mathbf{k}n^{*},-\mathbf{q}-\mathbf{k}n'}]^{*}}{-\omega-(\epsilon_{-\mathbf{q}-\mathbf{k}n'}-\epsilon_{-\mathbf{k}n})+i\delta}. \tag*{(13)}

Note that Zqμ−kn∗,−q−kn′=⟨EμqΨ−kn∗∣Ψ−q−kn′⟩Z_{q\mu}^{-\mathbf{k}n^{*},-\mathbf{q}-\mathbf{k}n'}=\langle E_{\mu}^{q}\Psi_{-\mathbf{k}n}^{*}\mid\Psi_{-\mathbf{q}-\mathbf{k}n'}\rangle. In the second term of Eq. (13), we interexchange nn and n′n' of Eq. (12) with keeping momentum conservation. Πμν(q,ω)\Pi_{\mu\nu}(\mathbf{q},\omega) can be divided into time-revesal and anti-time-reversal parts as discussed in Ref. [8]. If we assume time-reversal symmetry of H0H^{0}, Ψkn∗\Psi_{\mathbf{k}n}^{*} is the degenerated eigenfunction of Ψ−kn\Psi_{-\mathbf{k}n}. Then the numerator of Eq. (13) at the second term is identical to that of the first term; thus the antitime-reversal parts disappear. Then we have

Πμν(q,ω)=∑kBZ∑nocc∑n′unoccZqμkn,q+kn′[Zqνkn,q+kn′]∗(1ω−ϵq+kn′+ϵkn+iδ−1ω+ϵq+kn′−ϵkn−iδ).(14)\Pi_{\mu\nu}(\mathbf{q},\omega)=\sum_{\mathbf{k}}^{\mathrm{BZ}}\sum_{n}^{\mathrm{occ}}\sum_{n'}^{\mathrm{unocc}}Z_{q\mu}^{\mathbf{k}n,\mathbf{q}+\mathbf{k}n'}[Z_{q\nu}^{\mathbf{k}n,\mathbf{q}+\mathbf{k}n'}]^{*}\left(\frac{1}{\omega-\epsilon_{\mathbf{q}+\mathbf{k}n'}+\epsilon_{\mathbf{k}n}+i\delta}-\frac{1}{\omega+\epsilon_{\mathbf{q}+\mathbf{k}n'}-\epsilon_{\mathbf{k}n}-i\delta}\right). \tag*{(14)}

Hereafter, we assume the time-reversal symmetry. This definition is the same as that in the Fetter-Walecka textbook [33]. At ω=0\omega=0, this is a negative definite matrix. The matrix part Zqμkn,q+kn′[Zqνkn,q+kn′]∗Z_{q\mu}^{\mathbf{k}n,\mathbf{q}+\mathbf{k}n'}[Z_{q\nu}^{\mathbf{k}n,\mathbf{q}+\mathbf{k}n'}]^{*} in Eq. (14) is Hermitian, and Πμν(q,ω)=Πμν(q,−ω)\Pi_{\mu\nu}(\mathbf{q},\omega)=\Pi_{\mu\nu}(\mathbf{q},-\omega) is satisfied for general complex ω\omega where we can neglect ±iδ\pm i\delta. In practice, we use the Hilbert transformation to calculate the Hermitian part of Π\Pi after computing the anti-Hermitian part. The anti-Hermitian part of Π\Pi, written as Im⁡Π\operatorname{Im}\Pi, is written as

Im⁡[Πμν(q,ωj)]=πi∑kBZ∑nocc∑n′unoccZqμkn,q+kn′[Zqνkn,q+kn′]∗wn′nq,k(ωj),(15)\operatorname{Im}[\Pi_{\mu\nu}(\mathbf{q},\omega_j)] = \frac{\pi}{i} \sum_{\mathbf{k}}^{\mathrm{BZ}} \sum_n^{\mathrm{occ}} \sum_{n'}^{\mathrm{unocc}} Z_{q\mu}^{kn,q+kn'}[Z_{q\nu}^{kn,q+kn'}]^*w_{n'n}^{q,\mathbf{k}}(\omega_j), \tag*{(15)}

where wn′nq,k(ωj)w_{n'n}^{q,\mathbf{k}}(\omega_j) is the tetrahedron weight to accumulate Im⁡Π\operatorname{Im}\Pi. This weight is calculated as follows:

wn′nq,k(ωj)=∑k∈BZtet(itet)14w(n′,n,q,k,itet,ωj),(16)w_{n'n}^{q,\mathbf{k}}(\omega_j) = \sum_{\mathbf{k}\in\mathrm{BZtet}(i_{\mathrm{tet}})} \frac{1}{4}w(n',n,\mathbf{q},\mathbf{k},i_{\mathrm{tet}},\omega_j), \tag*{(16)}
w(n′,n,q,k,itet,ωj)=∫BZtet(itet)d3k∫ωLjωRjδ(ω−εq+kn′+εkn)dω,(17)w(n',n,\mathbf{q},\mathbf{k},i_{\mathrm{tet}},\omega_j) = \int_{\mathrm{BZtet}(i_{\mathrm{tet}})} d^3k \int_{\omega_{Lj}}^{\omega_{Rj}} \delta(\omega-\varepsilon_{\mathbf{q}+\mathbf{k}n'}+\varepsilon_{\mathbf{k}n})d\omega, \tag*{(17)}

where ωj\omega_j denotes a histogram bin [ωLj,ωRj][\omega_{Lj},\omega_{Rj}] for ω\omega along the real axis to accumulate the imaginary part. We employ a logarithmic mesh for ωj\omega_j such as ωj=10−5×1.03j\omega_j=10^{-5}\times1.03^j Ry, resulting in ∼100\sim100 mesh points to cover all excitation energies. We evaluate the sum for k\mathbf{k} of the right-hand side of Eq. (15) using a tetrahedron method for the polarization function [20, 34]. Note that the products Zqμkn,q+kn′[Zqνkn,q+kn′]∗Z_{q\mu}^{kn,q+kn'}[Z_{q\nu}^{kn,q+kn'}]^* are the positive definite matrix. The tetrahedron method is crucial for performing accurate GWGW calculations efficiently. We expect the correct sum rule to be satisfied within the space spanned by the PMT bases, which are used to expand eigenfunctions. The real (Hermitian) part is obtained through the Hilbert transformation as

Re⁡[Πμν(q,ωi)]=∑jKijRe⁡[Πμν(q,ωj)],(18)\operatorname{Re}[\Pi_{\mu\nu}(\mathbf{q},\omega_i)] = \sum_j K_{ij}\operatorname{Re}[\Pi_{\mu\nu}(\mathbf{q},\omega_j)], \tag*{(18)}

where KijK_{ij} is the Hilbert transformation matrix, a discretized version of 1/(ωi−ωj)1/(\omega_i-\omega_j), generated in advance with taking into account the size of histogram bins. By construction, our manner to obtain Πμν(q,ω)\Pi_{\mu\nu}(\mathbf{q},\omega) satisfies the Kramers-Kronig relation very well. Then we obtain Πμν(q,ωi)\Pi_{\mu\nu}(\mathbf{q},\omega_i) as the sum of their imaginary part and real part. When we have no time-reversal symmetry, we accumulate not only the positive energy part of Eq. (15), but also the negative energy part for occupied n′n' and unoccupied nn in Eq. (12). As an approximation, we may use Eq. (14) even for systems without time-reversal, however, we may need some careful examination.

The correlation part of the screened Coulomb interaction Wμνc(k,ω)=Wμν(k,ω)−vμ(k)δμνW_{\mu\nu}^{c}(\mathbf{k},\omega)=W_{\mu\nu}(\mathbf{k},\omega)-v_{\mu}(\mathbf{k})\delta_{\mu\nu} is given by the matrix inversion of Eq. (10) as

Wμνc(k,ω)=vμ(k)(εμν−1(k,ω)−δμν)vν(k).(19)W_{\mu\nu}^{c}(\mathbf{k},\omega)=\sqrt{v_{\mu}(\mathbf{k})}\left(\varepsilon_{\mu\nu}^{-1}(\mathbf{k},\omega)-\delta_{\mu\nu}\right)\sqrt{v_{\nu}(\mathbf{k})}. \tag*{(19)}

With Wμνc(k,ω)W_{\mu\nu}^{c}(\mathbf{k},\omega), we have a formula for the correlation self energy as

Σnmc(q,ω)=∑k,n′μ,ν∫−∞∞i dω′2π[Zqμq−kn′,qn]∗Wμνc(k,ω′)Zqνq−kn′,qme−iδω′ω−ω′−εq−kn′±iδ.(20)\Sigma_{nm}^{c}(\mathbf{q},\omega)=\sum_{\mathbf{k},n'\mu,\nu}\int_{-\infty}^{\infty}\frac{i\,d\omega'}{2\pi}\frac{[Z_{q\mu}^{q-kn',qn}]^*W_{\mu\nu}^{c}(\mathbf{k},\omega')Z_{q\nu}^{q-kn',qm}e^{-i\delta\omega'}}{\omega-\omega'-\varepsilon_{\mathbf{q}-\mathbf{k}n'}\pm i\delta}. \tag*{(20)}

Here, we use −iδ-i\delta for the occupied states of q−kn′\mathbf{q}-\mathbf{k}n' and +iδ+i\delta for the unoccupied states. The factor e−iδω′e^{-i\delta\omega'} is neglected due to the behavior of Wc(k,ω′)W^{c}(\mathbf{k},\omega') at ω′→±∞\omega'\to\pm\infty.

In QSGW, we have to calculate Σnmc(q,εqn)=⟨Ψqn∣Σc(q,εqn)∣Ψqm⟩\Sigma_{nm}^{c}(\mathbf{q},\varepsilon_{qn})=\langle\Psi_{qn}|\Sigma^{c}(\mathbf{q},\varepsilon_{qn})|\Psi_{qm}\rangle as discussed in Eq. (3). The integration contour for ω′\omega' in Eq. (20) is deformed to the one shown in Fig. 15 of [8]. We evaluate Σnmc(q,εqn)\Sigma_{nm}^{c}(\mathbf{q},\varepsilon_{qn}) as the sum of the imaginary-axis-contour contribution Ic(q,εqn)I^{c}(\mathbf{q},\varepsilon_{qn}) and the real-axis-contour (or pole) contribution Rc(q,εqn)R^{c}(\mathbf{q},\varepsilon_{qn}), that is,

Σnmc(q,εqn)=Inmc(q,εqn)+Rnmc(q,εqn).(21)\Sigma_{nm}^{c}(\mathbf{q},\varepsilon_{qn})=I_{nm}^{c}(\mathbf{q},\varepsilon_{qn})+R_{nm}^{c}(\mathbf{q},\varepsilon_{qn}). \tag*{(21)}

Here Inmc(q,εqn)I^{\mathrm{c}}_{nm}(\mathbf{q},\varepsilon_{q n}) is written as

Inmc(q,εqn)=−∑k,n′∑μ,ν∫−∞∞dω2π[Zkμq−kn′,qn]∗Wμνc(k,iω)Zkνq−kn′,qmεqn−iω−εq−kn′I^{\mathrm{c}}_{nm}(\mathbf{q},\varepsilon_{q n})=-\sum_{\mathbf{k},n'}\sum_{\mu,\nu}\int_{-\infty}^{\infty}\frac{d\omega}{2\pi}\frac{[Z^{\mathbf{q}-\mathbf{k}n',q n}_{k\mu}]^{*}W^{\mathrm{c}}_{\mu\nu}(\mathbf{k},i\omega)Z^{\mathbf{q}-\mathbf{k}n',q m}_{k\nu}}{\varepsilon_{q n}-i\omega-\varepsilon_{\mathbf{q}-\mathbf{k}n'}}
≃discretized−∑k,n′∑i∑μ,νwiI(εqn−εq−kn′)[Zkμq−kn′,qn]∗Wμνc(k,iωiI)Zkνq−kn′,qm.(22)\mathrel{\mathop{\simeq}^{\mathrm{discretized}}}-\sum_{\mathbf{k},n'}\sum_i\sum_{\mu,\nu}w_i^{\mathrm{I}}(\varepsilon_{q n}-\varepsilon_{\mathbf{q}-\mathbf{k}n'})[Z^{\mathbf{q}-\mathbf{k}n',q n}_{k\mu}]^{*}W^{\mathrm{c}}_{\mu\nu}(\mathbf{k},i\omega_i^{\mathrm{I}})Z^{\mathbf{q}-\mathbf{k}n',q m}_{k\nu}. \tag*{(22)}

The mesh points {iωiI}\{i\omega_i^{\mathrm{I}}\} and the corresponding discretized integration weight {wI(ΔE,iωiI)∣for iωiI}\{w^{\mathrm{I}}(\Delta E,i\omega_i^{\mathrm{I}})|\text{for }i\omega_i^{\mathrm{I}}\} satisfy

∫−∞∞dω2πX(iω)ΔE−iω≃discretized∑iωiIwI(ΔE,iωiI)X(iωiI),(23)\int_{-\infty}^{\infty}\frac{d\omega}{2\pi}\frac{X(i\omega)}{\Delta E-i\omega}\mathrel{\mathop{\simeq}^{\mathrm{discretized}}}\sum_{i\omega_i^{\mathrm{I}}}w^{\mathrm{I}}(\Delta E,i\omega_i^{\mathrm{I}})X(i\omega_i^{\mathrm{I}}), \tag*{(23)}

where X(iω)X(i\omega) behaves monotonically as Wμνc(k,ω)W^{\mathrm{c}}_{\mu\nu}(\mathbf{k},\omega) along the imaginary axis: Wμνc(k,ω)W^{\mathrm{c}}_{\mu\nu}(\mathbf{k},\omega) is a positive-definite hermitian with monotonic damping for ω→±i∞\omega\to\pm i\infty. To determine wI(ΔE,iωiI)w^{\mathrm{I}}(\Delta E,i\omega_i^{\mathrm{I}}), we use the method given by Aryasetiawan and Gunnarsson [31]. Since Wc(ω)=Wc(−ω)W^{\mathrm{c}}(\omega)=W^{\mathrm{c}}(-\omega), we can reduce the sum for iωiIi\omega_i^{\mathrm{I}} only for ωiI≥0\omega_i^{\mathrm{I}}\geq0. We typically use only 10 points for {ωiI}\{\omega_i^{\mathrm{I}}\}. The contribution Rc(q,εqn)R^{\mathrm{c}}(\mathbf{q},\varepsilon_{q n}) is written as

Rnmc(q,εqn)=∑k,n′∑μ,ν[Zkμq−kn′,qn]∗Wμνc(k,εqn−εq−kn′)Zkνq−kn′,qm×[θ(εqn−EF)θ(εqn−εq−kn′)θ(εq−kn′−EF)−θ(EF−εqn)θ(εq−kn′−εqn)θ(EF−εq−kn′)]≃discretized∑k,n′∑ωiR∑μ,νwiR(εqn−EF,εq−kn′−EF)[Zkμq−kn′,qn]∗Wμνc(k,ωiR)Zkνq−kn′,qm.(24)\begin{aligned} R^{\mathrm{c}}_{nm}(\mathbf{q},\varepsilon_{q n})={}&\sum_{\mathbf{k},n'}\sum_{\mu,\nu}[Z^{\mathbf{q}-\mathbf{k}n',q n}_{k\mu}]^{*}W^{\mathrm{c}}_{\mu\nu}(\mathbf{k},\varepsilon_{q n}-\varepsilon_{\mathbf{q}-\mathbf{k}n'})Z^{\mathbf{q}-\mathbf{k}n',q m}_{k\nu}\\ &\times\left[\theta(\varepsilon_{q n}-E_{\mathrm{F}})\theta(\varepsilon_{q n}-\varepsilon_{\mathbf{q}-\mathbf{k}n'})\theta(\varepsilon_{\mathbf{q}-\mathbf{k}n'}-E_{\mathrm{F}})\right.\\ &\left.\qquad-\theta(E_{\mathrm{F}}-\varepsilon_{q n})\theta(\varepsilon_{\mathbf{q}-\mathbf{k}n'}-\varepsilon_{q n})\theta(E_{\mathrm{F}}-\varepsilon_{\mathbf{q}-\mathbf{k}n'})\right]\\ &\mathrel{\mathop{\simeq}^{\mathrm{discretized}}}\sum_{\mathbf{k},n'}\sum_{\omega_i^{\mathrm{R}}}\sum_{\mu,\nu}w_i^{\mathrm{R}}(\varepsilon_{q n}-E_{\mathrm{F}},\varepsilon_{\mathbf{q}-\mathbf{k}n'}-E_{\mathrm{F}})[Z^{\mathbf{q}-\mathbf{k}n',q n}_{k\mu}]^{*}W^{\mathrm{c}}_{\mu\nu}(\mathbf{k},\omega_i^{\mathrm{R}})Z^{\mathbf{q}-\mathbf{k}n',q m}_{k\nu}. \tag*{(24)} \end{aligned}

Here, the mesh points ωiR\omega_i^{\mathrm{R}} are the boundaries of the histogram bins ωj\omega_j used in Eq. (17). The corresponding discretized integration weight wiR(εqn−EF,εq−kn′−EF)w_i^{\mathrm{R}}(\varepsilon_{q n}-E_{\mathrm{F}},\varepsilon_{\mathbf{q}-\mathbf{k}n'}-E_{\mathrm{F}}) is the interpolation coefficients to obtain WcμνW^{\mu\nu}_{\mathrm{c}} at εqn−εq−kn′\varepsilon_{q n}-\varepsilon_{\mathbf{q}-\mathbf{k}n'} from Wc(ωiR)W^{\mathrm{c}}(\omega_i^{\mathrm{R}}). In addition, the factor from the step function should be included. Since we employ the three-point interpolation, wiR(εqn−EF,εq−kn′−EF)w_i^{\mathrm{R}}(\varepsilon_{q n}-E_{\mathrm{F}},\varepsilon_{\mathbf{q}-\mathbf{k}n'}-E_{\mathrm{F}}) are non-zero only at three of the mesh points ωiR\omega_i^{\mathrm{R}}. Note that the lifetime of QP (= the impact ionization rate) due to electron-electron scattering is only included in Rc(q,εqn)R^{\mathrm{c}}(\mathbf{q},\varepsilon_{q n}) [23]. In the calculation of the static version of the correlation self-energy, WμνcW^{\mathrm{c}}_{\mu\nu} in RnmcR^{\mathrm{c}}_{nm} is hermitized.

Regarding the division Σnmc(q,ω)=Rnmc(q,ω)+Inmc(q,ω)\Sigma^{\mathrm{c}}_{nm}(\mathbf{q},\omega)=R^{\mathrm{c}}_{nm}(\mathbf{q},\omega)+I^{\mathrm{c}}_{nm}(\mathbf{q},\omega), only Rnmc(q,εqn)R^{\mathrm{c}}_{nm}(\mathbf{q},\varepsilon_{q n}) contributes to the lifetime of QPs. For the electron at qnq n (εqn>EF\varepsilon_{q n}>E_{\mathrm{F}}), there is a factor θ(εqn−εq−kn′)θ(εq−kn′−EF)\theta(\varepsilon_{q n}-\varepsilon_{\mathbf{q}-\mathbf{k}n'})\theta(\varepsilon_{\mathbf{q}-\mathbf{k}n'}-E_{\mathrm{F}}). When Wμνc(k,εqn−εq−kn′)W^{\mathrm{c}}_{\mu\nu}(\mathbf{k},\varepsilon_{q n}-\varepsilon_{\mathbf{q}-\mathbf{k}n'}) has the non-zero imaginary part, Rnmc(q,εqn)R^{\mathrm{c}}_{nm}(\mathbf{q},\varepsilon_{q n}) includes the decay rate to the lower energy electron εq−kn′\varepsilon_{\mathbf{q}-\mathbf{k}n'} with an electron-hole pair. The details of discussions about the lifetime of QPs and the impact ionization rate in semiconductors were presented in [23]. Since we use Gaussian broadening at energies εq−kn′\varepsilon_{\mathbf{q}-\mathbf{k}n'} for the discretized sum in Eq. (24), we should truncate the Gaussian weights by the step functions in Eq. (24).

We have two key points not yet explained. One is the improved offset-Γ\Gamma method [8], which treats the divergence of Wc(k→0,ω)W^{\mathrm{c}}(\mathbf{k}\to0,\omega) in Eqs. (9) and (20). Instead of Wc(k→0,ω)W^{\mathrm{c}}(\mathbf{k}\to0,\omega), we use a non-divergent effective interaction W‾c(k=0,ω)\overline{W}^{\mathrm{c}}(\mathbf{k}=0,\omega) in Eq. (20). This W‾c(k=0,ω)\overline{W}^{\mathrm{c}}(\mathbf{k}=0,\omega) is an average of Wc(k,ω)W^{\mathrm{c}}(\mathbf{k},\omega) in k\mathbf{k} within the Γ\Gamma cell, which is defined as the microcell of the BZ containing k=0\mathbf{k}=0 [20] at the center. To obtain W‾c(k=0,ω)\overline{W}^{\mathrm{c}}(\mathbf{k}=0,\omega), we have to calculate Wc(k,ω)W^{\mathrm{c}}(\mathbf{k},\omega) at some k\mathbf{k} points near k=0\mathbf{k}=0 (offset Gamma points). The number of k\mathbf{k} points depends on the crystal symmetry. v(k→0)v(\mathbf{k}\to0) is replaced by v‾(k→0)\overline{v}(\mathbf{k}\to0), as well. Then, we can take a simple discrete sum for both expressions of Eqs. (9) and (20). The other point is how to perform an interpolation to give VQSGWxcV_{\mathrm{QSGW}}^{xc} at any q\mathbf{q} in the BZ. For the interpolation, we expand the difference ΔVxc=VQSGWxc−VLDAxc\Delta V^{xc}=V_{\mathrm{QSGW}}^{xc}-V_{\mathrm{LDA}}^{xc} in the MTQSGW, omitting APWs for expansion. We have performed empty-sphere tests to complement the missing APWs [20]; however, including APWs, probably with the contracted plane wave [35], remains a subject for future work.

Multi-GPU implementation of QSGW

Recently, multi-GPU computations have gained significant attention in scientific computing. For example, GPUs are used for first-principles calculations in DFT, as described in Refs. [36, 37]. In the GW calculations, multi-GPU implementations have recently emerged [22, 38, 39, 40]. One of the computationally expensive parts is the evaluation of the polarization functions. This part was successfully accelerated by GPUs in plane-wave basis codes. In QSGW, not only the polarization function but also the self-energy calculation are computationally expensive, since we have to calculate off-diagonal elements of the self-energy as in Eqs. (9) and (20). Here, we demonstrate how to implement a multi-GPU version of QSGW in the ecalj package.

Computational cost of QSGW

QSGW calculations comprise four computationally intensive steps, as outlined in § 4. Table 1 shows the order of computational costs for the computational kernels in each step, together with those in LDA. To illustrate the size of actual calculations, we display the size for a benchmark, (InAs)10(GaSb)10(\mathrm{InAs})_{10}(\mathrm{GaSb})_{10}. For supplied crystal structures, in addition to the number of divisions in the BZ, these numbers are determined by our default setting. The key points regarding the sizes in Table 1 are as follows: (1) As shown in the numbers of (InAs)10(GaSb)10(\mathrm{InAs})_{10}(\mathrm{GaSb})_{10}, the number of product basis functions (npbn_{\mathrm{pb}}) is only several times larger than the number of basis for the wave function (nmbn_{\mathrm{mb}}). In principle, we may expect npb/nmb∼8=23n_{\mathrm{pb}}/n_{\mathrm{mb}} \sim8 = 2^3 (to expand products of plane waves). However, we had verified that such a large npbn_{\mathrm{pb}} is not necessary in our construction of MPB [8]. (2) We use the intermediate states nall=nocc+nunoccn_{\mathrm{all}} = n_{\mathrm{occ}} + n_{\mathrm{unocc}} which is almost the same as nmbn_{\mathrm{mb}} (the number of basis functions ≃\simeq the eigenfunctions). Thus, we handle the space spanned by the PMT basis functions very well, that the artificial truncation is avoided. Some eigenfunctions are missing at the diagonalization step of H0H^0 because of the poor linear dependency of the overlap matrix. (3) In our default setting, nqpn_{\mathrm{qp}} (where nqp<nalln_{\mathrm{qp}} < n_{\mathrm{all}}) is determined to include quasiparticle states less than Fermi energy +2∼+3+2 \sim+3 Ry. We had checked that this is a suitable truncation to reduce computational costs [20].

Table 1: Size parameters in QSGW and computational cost of each part of the computation. To have an idea of the size of actual calculations, we show the size for a case of our benchmark for (InAs)10(GaSb)10(\mathrm{InAs})_{10}(\mathrm{GaSb})_{10} (40 atoms in a cell). See details in Table 2.

SymbolDescription(InAs)10(GaSb)10(\mathrm{InAs})_{10}(\mathrm{GaSb})_{10}
nIBZn_{\mathrm{IBZ}}number of k points in the irreducible Brillouin zone9
nBZn_{\mathrm{BZ}}number of k points in the Brillouin zone16
ncoren_{\mathrm{core}}number of core states1180
noccn_{\mathrm{occ}}number of occupied states460
nunoccn_{\mathrm{unocc}}number of unoccupied states1348
nalln_{\mathrm{all}}number of all states =nocc+nunocc=n_{\mathrm{occ}}+n_{\mathrm{unocc}}1808
nCpbn_{\mathrm{Cpb}}number of product basis for core-exchange self-energy calculation8121
npbn_{\mathrm{pb}}number of product basis5231
nmbn_{\mathrm{mb}}number of basis functions for eigenfunctions1834
nqpn_{\mathrm{qp}}number of states to calculate self-energy713
nspinn_{\mathrm{spin}}number of spin components1
nωn_{\omega}number of frequency mesh along real ω\omega68
nωIn^{\mathrm{I}}_{\omega}number of imaginary frequency mesh (including ω=0\omega= 0)11
kernelEq.
Computational cost
Outer loop size
Array size
Core-exchange self-energyZZEq. (8)
ΣCx\mathcal{\Sigma}^{\mathrm{Cx}}Eq. (9)
O(nqpncorenCpbnmb)\mathcal{O}(n_{\mathrm{qp}}n_{\mathrm{core}}n_{\mathrm{Cpb}}n_{\mathrm{mb}})
O(nqp2ncorenCpb)\mathcal{O}(n_{\mathrm{qp}}^{2}n_{\mathrm{core}}n_{\mathrm{Cpb}})
nIBZnBZnspinn_{\mathrm{IBZ}}n_{\mathrm{BZ}}n_{\mathrm{spin}}
nIBZnBZnspinn_{\mathrm{IBZ}}n_{\mathrm{BZ}}n_{\mathrm{spin}}
(nqpncorenCpb)(n_{\mathrm{qp}}n_{\mathrm{core}}n_{\mathrm{Cpb}})
Exchange self-energyZZEq. (8)
Σx\Sigma^{\mathrm{x}}Eq. (9)
O(nqpnoccnpbnmb)\mathcal{O}(n_{\mathrm{qp}}n_{\mathrm{occ}}n_{\mathrm{pb}}n_{\mathrm{mb}})
O(nqp2noccnpb)\mathcal{O}(n_{\mathrm{qp}}^{2}n_{\mathrm{occ}}n_{\mathrm{pb}})
nIBZnBZnspinn_{\mathrm{IBZ}}n_{\mathrm{BZ}}n_{\mathrm{spin}}
nIBZnBZnspinn_{\mathrm{IBZ}}n_{\mathrm{BZ}}n_{\mathrm{spin}}
(nqpnoccnpb)(n_{\mathrm{qp}}n_{\mathrm{occ}}n_{\mathrm{pb}})
Screened Coulomb interactionZZEq. (8)
Im⁡Π\operatorname{Im}\PiEq. (15)
Re⁡Π\operatorname{Re}\PiEq. (18)
ε−1v\varepsilon^{-1}vEq. (19)
O(noccnoccnpbnmb)\mathcal{O}(n_{\mathrm{occ}}n_{\mathrm{occ}}n_{\mathrm{pb}}n_{\mathrm{mb}})
O(noccnunoccnpb2)\mathcal{O}(n_{\mathrm{occ}}n_{\mathrm{unocc}}n_{\mathrm{pb}}^{2})
O(npb2nω)\mathcal{O}(n_{\mathrm{pb}}^{2}n_{\omega})
O(nωnpb3)\mathcal{O}(n_{\omega}n_{\mathrm{pb}}^{3})
nIBZnBZnspinn_{\mathrm{IBZ}}n_{\mathrm{BZ}}n_{\mathrm{spin}}
nIBZnBZnspinn_{\mathrm{IBZ}}n_{\mathrm{BZ}}n_{\mathrm{spin}}
nIBZn_{\mathrm{IBZ}}
nIBZn_{\mathrm{IBZ}}
(noccnunoccnpb)(n_{\mathrm{occ}}n_{\mathrm{unocc}}n_{\mathrm{pb}})
(npb2nω)(n_{\mathrm{pb}}^{2}n_{\omega})
Correlation self-energyZZEq. (8)
RcR_{\mathrm{c}}Eq. (24)
IcI_{\mathrm{c}}Eq. (22)
O(nqpnallnpbnmb)\mathcal{O}(n_{\mathrm{qp}}n_{\mathrm{all}}n_{\mathrm{pb}}n_{\mathrm{mb}})
O(nqpnallnpb2)\mathcal{O}(n_{\mathrm{qp}}n_{\mathrm{all}}n_{\mathrm{pb}}^{2})
O(nqpnallnpb2nωI)\mathcal{O}(n_{\mathrm{qp}}n_{\mathrm{all}}n_{\mathrm{pb}}^{2}n_{\omega}^{\mathrm{I}})
nIBZnBZnspinn_{\mathrm{IBZ}}n_{\mathrm{BZ}}n_{\mathrm{spin}}
nIBZnBZnspinn_{\mathrm{IBZ}}n_{\mathrm{BZ}}n_{\mathrm{spin}}
nIBZnBZnspinn_{\mathrm{IBZ}}n_{\mathrm{BZ}}n_{\mathrm{spin}}
(nqpnallnpb)(n_{\mathrm{qp}}n_{\mathrm{all}}n_{\mathrm{pb}})
LDASetup H0H^{0}O(nmb2nall)\mathcal{O}(n_{\mathrm{mb}}^{2}n_{\mathrm{all}})
Solve H0∣Ψkn⟩=εkn∣Ψkn⟩H^{0}|\Psi_{kn}\rangle= \varepsilon_{kn}|\Psi_{kn}\rangleO(nmb3)\mathcal{O}(n_{\mathrm{mb}}^{3})
nIBZnspinn_{\mathrm{IBZ}}n_{\mathrm{spin}}
nIBZnspinn_{\mathrm{IBZ}}n_{\mathrm{spin}}
(nmbnall)(n_{\mathrm{mb}}n_{\mathrm{all}})

Table 1.

As shown in Table 1, the computational cost of WW and the correlation part of self energy are particularly high, making them the most dominant components of the QSGW calculation. Based on our experience, the current computation time for QSGW is approximately 500 times longer than that for LDA/GGA calculations in the 10-atom system because of the significantly higher computational cost compared to LDA/GGA calculations. The fact that the size of the array scales with the cube of the size of the system indicates that memory partitioning is necessary for large systems. The following section discusses the method for using GPUs in QSGW in ecalj.

Parallelization scheme with GPU

We have two versions of code to perform QSGW, which are simply switched by a C preprocessor option (cpp). One is just the MPI (CPU) version. The other is the GPU version. In this case, we use one MPI process per GPU, as shown in Figure 2. In principle, our implementation enables the sharing of a GPU among multiple MPI processes by using Multiprocess-Service (MPS) or Multi-Instance-GPU (MIG) techniques [41]. However, such an approach is only effective if the computational cost per MPI process is small and a single GPU resource cannot be fully utilized. Additionally, it is unsuitable for large-scale systems because the

Parallelization scheme for QSGW calculations

Figure 2. Parallelization scheme for QSGW calculations.

GPU memory of each MPI process decreases. Thus, we now arrive at the conclusion that the MPI process per GPU is a best practice, as in Figure 2.

Maintaining the computational code is vital for the continuation of future research and development. It would be impractical if the GPU version were merely a one-off optimization that lacks maintainability. Thus, we have carefully designed the above implementations.

At the MPI level, we optimize the Fortran codes. Subsequently, we introduce the GPU version by replacing the expensive parts with GPU coding. Our code is based on module-based programming using the latest modern Fortran. We utilize vector operations and minimize the use of do loops whenever possible. Intrinsic functionalities such as block, findloc, counts, associate, and array constructors are fully used. In object-oriented programming terminology, we utilize modules to mimic the singleton pattern. That is, modules contain various data generated simply by calling the subroutines in modules (initialization and/or some instructions). The set of data can be read by other subroutines, but cannot be modified by them. The Fortran singleton pattern is helpful for robust and efficient implementation. One of the advantages is that one can safely read any data in any subroutine by directly accessing the modules where the data is stored. It is essential to make no copies of the data. For example, the number of atoms is defined as a module variable in a specific module. We access the number of atoms exclusively through this variable, which helps to avoid data inconsistency within the code.

In the GPU version, we utilize OpenACC programming to eliminate code duplication between the MPI and GPU versions as much as possible. OpenACC enables the effective utilization of GPUs by inserting directives into existing code, allowing the source code to be shared between CPU and GPU versions. Moreover, previous research has reported that GPU implementations of one-shot GWGW calculation using OpenACC directives achieve performance comparable to or slightly lower than CUDA implementations [40]. It indicates that OpenACC is one of the suitable options for straightforwardly achieving high performance while ensuring high code maintainability. However, it is important to note that implementing OpenACC programming alone has limitations. Due to its simplicity and limited functionality, OpenACC is more challenging to optimize and tune code than CUDA programming. Therefore, we employ a GPU-accelerated math library as much as possible for the kernel calculations. As mentioned, the GWGW calculations involve tensor contractions, as shown in Eqs. (9), (15), (22), and (24), which can be executed through GPU-accelerated matrix multiplication. Matrix multiplications are particularly well suited for GPUs; for example, recent NVIDIA GPUs have been able to accelerate matrix multiplication using Tensor Core.

Effective data transfer between the CPU and GPU is another crucial point in achieving high computational efficiency in GPU usage. Since data transfer between CPU and GPU is quite time-consuming, we design data control for GPU so that the expensive part of computations is performed within the GPU alone. Consequently, the CPU only performs small but complicated computations that are difficult to parallelize efficiently, as well as the data management and computational control.

The GPU usage implementation procedure on ecalj

Generally, the development of GPU-accelerated computational code follows these steps:

  1. Identify the bottleneck parts of the computation

  2. Move the bottleneck computations to the GPU

  3. Optimize data transfer between CPU and GPU

In Step I, performance analysis tools such as the Intel VTune Profiler are valuable tools. Regarding Step II, we have developed low-level Fortran modules for core operations, such as matrix multiplication and matrix inversion calculations. These modules have CPU and GPU versions of core operations routines with a unified interface. In the higher-level modules, simply switching the called routines allows the use of either CPU or GPU. The called routines are switched using a combination of Fortran module aliases and preprocessor directives. The data transfer and small-scale GPU operations in the high-level modules are controlled by OpenACC programming, ensuring common code for both CPU and GPU. Furthermore, the variables used exclusively by the GPU are clearly defined using the ‘device’ attribute in CUDA Fortran. On the other hand, for public attribute variables used in both the CPU and GPU, the variable scope can become unclear, negatively affecting the code’s maintainability. Therefore, we restrict such variables to a limited number of large arrays. Step III minimizes data transfer by offloading all large-scale data operations to the GPU. During the development stage, we utilized NVIDIA Nsight Systems to identify the low-efficiency parts of GPU usage and the bottlenecks caused by data transfer.

To effectively implement GPU usage on ecalj, we optimized our code before introducing GPU, focusing on the following two concepts. First, it is crucial for reducing memory usage and enhancing the robustness of calculations, especially in large-scale systems. We divided the internal npbn_{\mathrm{pb}} in the calculation of WW using MPI, thereby reducing the memory usage per MPI process. Moreover, segmented batch processing for the intermediate state of ZZ is also implemented for these four steps because its size at kk and qq points is proportional to the cube of the system, which far exceeds the available memory in large-scale systems. Second, as mentioned earlier, the kernel calculations in the ecalj code were replaced with matrix multiplication.

Algorithm 1 Pseudo code to calculate WcW^c with our MPI-GPU architecture shown in Figure 2. MPI communications are at lines 26 and 30. At lines 2 and 3, we assign a QKM block to the MPI process pp. Exactly speaking, we need additional q\mathbf{q} and k\mathbf{k} for the offset-Γ\Gamma method evaluating Wc(q=0,ωj)W^c(\mathbf{q}=0,\omega_j), but not shown here explicitly. To calculate ZZ at Line 12, we employ further MPI parallelization, which is not shown here.

  1. Bootstrap initialization to store data sets in modules. The data sets are for crystal structure, frequency mesh, eigenvalues, eigenfunctions, and computational settings.

  2. Set up divisions for MPI processes as {q}Q∣Q=0,1,2,⋯ ,NQ−1\{\mathbf{q}\}_{Q} \mid Q=0,1,2,\cdots,N_Q-1, {k}K∣K=0,1,2,⋯ ,NK−1\{\mathbf{k}\}_{K} \mid K=0,1,2,\cdots,N_K-1, {μ}M∣M=0,1,2,⋯ ,NM−1\{\boldsymbol{\mu}\}_{M} \mid M=0,1,2,\cdots,N_M-1.

  3. Generate functions Q(p)Q(p), K(p)K(p), M(p)M(p), so that a QKM block {q}Q(p)×{k}K(p)×{μ}M(p)\{\mathbf{q}\}_{Q(p)} \times\{\mathbf{k}\}_{K(p)} \times\{\boldsymbol{\mu}\}_{M(p)} is assigned to the MPI process pp, where p=0,1,2,⋯ ,NMPI−1p=0,1,2,\cdots,N_{\mathrm{MPI}}-1.

  4. — Following steps are at each MPI process pp —

  5. for q←{q}Q(p)\mathbf{q} \leftarrow\{\mathbf{q}\}_{Q(p)} do

  6. Im⁡Πq(1:npb,{μ}M(p),1:nω)←0\operatorname{Im}\Pi_{\mathbf{q}}(1:n_{\mathrm{pb}},\{\boldsymbol{\mu}\}_{M(p)},1:n_\omega) \leftarrow0 ▷ Π\Pi for q\mathbf{q} will be stored on Πq\Pi_{\mathbf{q}}

  7. Read vq(1:npb)v_{\mathbf{q}}(1:n_{\mathrm{pb}}) ▷ vμ(q)v_\mu(\mathbf{q}) in Eq. (7)

  8. Calculate tetrahedron weights w(1:nunocc,1:nocc,1:nω,{k})w(1:n_{\mathrm{unocc}},1:n_{\mathrm{occ}},1:n_\omega,\{\mathbf{k}\}) ▷ wk,qn′nw_{\mathbf{k},\mathbf{q}}^{n' n} in Eq. (16)

  9. for k←{k}K(p)\mathbf{k} \leftarrow\{\mathbf{k}\}_{K(p)} do

  10. Batch division 1:nunocc1:n_{\mathrm{unocc}} into {n′}b\{n'\}_b where b=1,2,⋯ ,Bbatchb=1,2,\cdots,B^{\mathrm{batch}} ▷ Avoid huge memory for ZZ

  11. for b←1:Bbatchb \leftarrow1:B^{\mathrm{batch}} do

  12. Calculate Z(1:npb,{n′}b,1:nocc)Z(1:n_{\mathrm{pb}},\{n'\}_b,1:n_{\mathrm{occ}}) ▷ For k\mathbf{k}, q\mathbf{q} in Eq. (8)

  13. for iω←1:nωi_\omega\leftarrow1:n_\omega do ▷ iωi_\omega specifies the histogram bins to accumulate the imaginary part

  14. Set up pair index {n(ipair)′,n(ipair)∣ipair=1,2,⋯ ,npair}\{n'_{(i_{\mathrm{pair}})},n_{(i_{\mathrm{pair}})}\mid i_{\mathrm{pair}}=1,2,\cdots,n_{\mathrm{pair}}\} for w(n(ipair)′,n(ipair),iω,k)≠0w(n'_{(i_{\mathrm{pair}})},n_{(i_{\mathrm{pair}})},i_\omega,\mathbf{k})\ne0 and n(ipair)′∈{n′}bn'_{(i_{\mathrm{pair}})}\in\{n'\}_b

  15. !$acc kernel

  16. for ipair←1:npairi_{\mathrm{pair}} \leftarrow1:n_{\mathrm{pair}} do

  17. Zpair(ipair,1:npb)←Z(1:npb,n(ipair)′,n(ipair))Z_{\mathrm{pair}}(i_{\mathrm{pair}},1:n_{\mathrm{pb}})\leftarrow Z(1:n_{\mathrm{pb}},n'_{(i_{\mathrm{pair}})},n_{(i_{\mathrm{pair}})})

  18. WZpair(ipair,{μ}M(p))←Zpair(ipair,{μ}M(p))w(n(ipair)′,n(ipair),iω,k)WZ_{\mathrm{pair}}(i_{\mathrm{pair}},\{\boldsymbol{\mu}\}_{M(p)})\leftarrow Z_{\mathrm{pair}}(i_{\mathrm{pair}},\{\boldsymbol{\mu}\}_{M(p)})w(n'_{(i_{\mathrm{pair}})},n_{(i_{\mathrm{pair}})},i_\omega,\mathbf{k})

  19. end for

  20. !$acc end kernel

  21. Im⁡Πq(1:npb,{μ}M(p),iω)←Im⁡Πq(1:npb,{μ}M(p),iω)+Zpair†×WZpair\operatorname{Im}\Pi_{\mathbf{q}}(1:n_{\mathrm{pb}},\{\boldsymbol{\mu}\}_{M(p)},i_\omega)\leftarrow\operatorname{Im}\Pi_{\mathbf{q}}(1:n_{\mathrm{pb}},\{\boldsymbol{\mu}\}_{M(p)},i_\omega)+Z_{\mathrm{pair}}^\dagger\times WZ_{\mathrm{pair}} ▷ GEMM

  22. end for

  23. end for

  24. end for

  25. if K(p)==0K(p)==0 then ▷ Reduction for k\mathbf{k}

  26. Πq(1:npb,{μ}M(p),iω)←∑p′∈Q(p′),M(p′)==M(p)Πqp′(1:npb,{μ}M(p),iω)\Pi_{\mathbf{q}}(1:n_{\mathrm{pb}},\{\boldsymbol{\mu}\}_{M(p)},i_\omega)\leftarrow\sum_{p'\in Q(p'),M(p')==M(p)}\Pi_{\mathbf{q}}^{p'}(1:n_{\mathrm{pb}},\{\boldsymbol{\mu}\}_{M(p)},i_\omega) ▷ MPI reduce sum. Superscript p′p' of Πqp′\Pi_{\mathbf{q}}^{p'} indicates data belonging to the process p′p'.

  27. Perform Hilbert transformation to Im⁡Πq\operatorname{Im}\Pi_{\mathbf{q}}, resulting in Πq\Pi_{\mathbf{q}} at {ωj∣ωj\{\omega_j\mid\omega_j along Real and Imag axes}\}. ▷ GEMM

  28. for jωj_\omega for {ωj∣ωj\{\omega_j\mid\omega_j along Real and Imag axes}\} do

  29. if M(p)==0M(p)==0 then ▷ Reduction for MPB

  30. Πqjω(1:npb,1:npb)←⨁p′Πqp′(1:npb,{μ}M(p′),jω)\Pi_{\mathbf{q}j_\omega}(1:n_{\mathrm{pb}},1:n_{\mathrm{pb}})\leftarrow\bigoplus_{p'}\Pi_{\mathbf{q}}^{p'}(1:n_{\mathrm{pb}},\{\boldsymbol{\mu}\}_{M(p')},j_\omega) ▷ MPI gather. Superscript p′p' of Πqp′\Pi_{\mathbf{q}}^{p'} indicates data belonging to the process p′p'.

  31. Calculate ϵ(1:npb,1:npb)\epsilon(1:n_{\mathrm{pb}},1:n_{\mathrm{pb}}) from vqv_{\mathbf{q}} and Πqjω\Pi_{\mathbf{q}j_\omega}. ▷ For q,ω=ωj\mathbf{q},\omega=\omega_j in Eq. (10), OpenACC

  32. Calculate ϵ−1(1:npb,1:npb)\epsilon^{-1}(1:n_{\mathrm{pb}},1:n_{\mathrm{pb}}) ▷ For q,ω=ωj\mathbf{q},\omega=\omega_j in Eq. (19), MATINV

  33. Calculate Wc(1:npb,1:npb)W^c(1:n_{\mathrm{pb}},1:n_{\mathrm{pb}}) ▷ For q,ω=ωj\mathbf{q},\omega=\omega_j in Eq. (19), OpenACC

  34. Write Wc(1:npb,1:npb)W^c(1:n_{\mathrm{pb}},1:n_{\mathrm{pb}}) into a file. ▷ At q=0\mathbf{q}=0, Wc(1:npb,1:npb)W^c(1:n_{\mathrm{pb}},1:n_{\mathrm{pb}}) instead.

  35. end if

  36. end for

  37. end if

  38. end for

To accomplish this, we restructured the arrays and changed the order of operations to utilize matrix operations. With respect to this point, the following details the specific implementation of WW and the correlation self-energy Σ\Sigma.

Screened Coulomb interaction calculation

Due to the above development, our QSGW calculations code works with massive MPI parallelization, regardless of GPU usage. In the calculation of WW, computations for specific q\mathbf{q} points are executed within each divided MPI communicator. Within the intracommunicator, MPI parallelization of MPB and k\mathbf{k} points is implemented. In our implementation, Π\Pi and WW are the largest arrays, whose specific size will be described later; therefore, this memory partitioning is crucial for practical applications. MPB parallelization enables the memory partitioning for Π\Pi, thereby reducing the array size of Π\Pi stored in each MPI process. For this reason, MPB parallelization is prioritized over k-point parallelization; therefore, the latter is set to one in our QSGW calculations. Meanwhile, k-point parallelization can be utilized for the calculation of the macroscopic dielectric function in the further analysis stage because it usually requires dense k-point sampling to reach convergence [42]. When calculating the dielectric function (ϵ\epsilon) from Π\Pi, as shown in Eq. (10), the data within the intra-communicator about MPB is gathered. The inverse matrix in Eq. (11) is obtained using cuSOLVER in the GPU version.

The kernel part of the Π\Pi calculation involves constructing ZZ in Eq. (8) and sum for the occupied orbitals nn and for the unoccupied orbitals n′n' at k points, described in Eq. (15). The overall implementation is illustrated in Alg. 1, where {q}Q\{q\}_Q, {k}K\{k\}_K and {μ}M\{\mu\}_M indicate q-point, k-point, and MPB parallelization, respectively. GEMM (general matrix multiplication) and OpenACC directives ($acc) indicate the regions to be computed on the GPU; the detailed specifications are omitted for simplicity. As mentioned earlier, segmented batch processing of intermediate states, due to the array size of ZZ exceeding the available memory, is introduced as an internal loop, as described in Alg. 1, line 10. As shown in Eq. (17), ww only has contributions from specific nn and n′n' for a given ω\omega. Reconstructing ww and ZZ only from nn and n′n' that contribute to the summation, we can perform the summation using the GEMMs routines. The calculations of ZZ and the summation part of Π\Pi, which are the predominant components of Π\Pi computation, as well as all computations involving ZZ and Π\Pi, are performed on the GPU. Therefore, the primary data transfers between the CPU and GPU involve reading the wave functions, which are the results of lmf, from files and transferring them to the GPU, as well as copying WW created on the GPU to the CPU for file output.

Static version of correlation self-energy calculation

In the implementation of the static version of correlation self energy, the World is divided by MPI parallelization for kk, qq, and ωi\omega_i. The kernel operations involve the construction of ZZ and the ω′\omega' integration shown in Eqs. (22) and (24), as well as the summation over n′n' and MPB. Algorithm 2 outlines the pseudo code for the calculation of Σ~c\widetilde{\Sigma}^{c}. The GEMM operations are repeated for ωi\omega_i to perform the imaginary axis integration, which is the most time-consuming calculation in the correlation self energy. The real axis integration only has values between specific orbitals (nn, n′n'); therefore, a similar implementation can be employed for the Im⁡Π\operatorname{Im}\Pi. Note that since this is the calculation of the static version of self energy, the real-axis integration involves the hermitization of WW, leading to the vanishing of the imaginary component of self energy. Although computation can be accelerated on the GPU, file reading, which involves data transfer via the CPU, becomes a significant bottleneck in the GPU version. Therefore, the WW data is stored in memory in advance on the GPU version. In contrast, it is read from the file each time on the CPU version because this treatment is not applicable to the CPU version due to the insufficient memory of each MPI process.

The MPI parallelization for ω′\omega' is less efficient than that of kk and qq due to the calculation of ZZ outside of the ω′\omega' loop. Therefore, the division of kk and qq is basically prioritized over ω′\omega' parallelization. However, the division of ω′\omega' has the advantage of reducing the array size of WW to be stored in memory, which helps avoid GPU memory exhaustion in the GPU version where WW is stored in GPU memory. Consequently, the ω′\omega' parallelization is used in the largest system of this benchmark (InAs)10_{10}(GaSb)10_{10} in the GPU version, but it is inactive in other cases.

Mixed precision approach

In GPU utilization, using single-precision or Tensor-Float (TF) operations, such as TF32, can improve computational speed at the expense of computational accuracy. For example, on NVIDIA’s A100 GPU, matrix multiplication using TF32 operations has eight times the computational performance compared to double precision (DP) with Tensor Core operations. We have confirmed that using TF32 operations in complex matrix multiplication benchmarks improves computational speed by more than 5 times. Since our computation-heavy tasks of the screened Coulomb interaction and self-energy calculations are mainly performed by complex matrix multiplication, the mixed precision (MP) approach is expected to increase computational speed drastically. Moreover, the array size will be half that of the DP case, which is beneficial for memory usage. Based on such motivations, we have implemented MP operations in GPU calculations.

Algorithm 2 Pseudo code to calculate Σc\Sigma^{\mathrm{c}} (hsfp0_sc) with our MPI-GPU architecture. MPI communications are at line 50. This is for the correlation self-energy. Ensure that frequency ωi\omega_i and weight wiw_i are not confused.

  1. Bootstrap initialization to store data sets in modules. Data sets are for crystal structure, frequency mesh, eigenvalues, eigenfunctions, and computational settings.

  2. Set up divisions for MPI processes as {{q}Q∣Q=0,1,2,⋯ ,NQ−1}\{\{q\}_{Q}\mid Q = 0,1,2,\cdots,N_{Q}-1\}, {{k}K∣K=0,1,2,⋯ ,NK−1}\{\{k\}_{K}\mid K = 0,1,2,\cdots,N_{K}-1\}, {{ωiI}Ω∣Ω=0,1,2,⋯ ,NΩ−1}\{\{\omega_i^{\mathrm{I}}\}_{\Omega}\mid\Omega= 0,1,2,\cdots,N^{\Omega}-1\}, as well as {{ωiR}Ω∣Ω=0,1,2,⋯ ,NΩ−1}\{\{\omega_i^{\mathrm{R}}\}_{\Omega}\mid\Omega= 0,1,2,\cdots,N^{\Omega}-1\}.

  3. Generate functions Q(p)Q(p), K(p)K(p), Ω(p)\Omega(p) for assigning a QKΩI\Omega^{\mathrm{I}} block {q}Q(p)×{k}K(p)×{ωiI}Ω(p)\{q\}_{Q(p)}\times\{k\}_{K(p)}\times\{\omega_i^{\mathrm{I}}\}_{\Omega(p)} to the MPI process pp, where p=0,1,2,⋯ ,NMPI−1p=0,1,2,\cdots,N_{\mathrm{MPI}}-1. In the same manner, Q(p)Q(p), K(p)K(p), ΩR(p)\Omega^{\mathrm{R}}(p) for a QKΩR\Omega^{\mathrm{R}} as well.

  4. ——— Following steps are at each MPI process pp ———

  5. Σc(1:nqp,1:nqp,{q})←0\Sigma^{\mathrm{c}}(1:n_{qp},1:n_{qp},\{q\})\leftarrow0 ▷ For Σc(q,εqn)\Sigma^{\mathrm{c}}(q,\varepsilon_{qn})

  6. for q←{q}Q(p)q\leftarrow\{q\}_{Q(p)} do

  7. Read vq(1:npb)v_q(1:n_{pb}) ▷ vμ(q)v_{\mu}(q) in Eq. (7)

  8. if GPU version: Read WIc(1:npb,1:npb,{ωiI}Ω(p))W^{\mathrm{c}}_{\mathrm{I}}(1:n_{pb},1:n_{pb},\{\omega_i^{\mathrm{I}}\}_{\Omega(p)}), WRc(1:npb,1:npb,{ωiR}Ω(p))W^{\mathrm{c}}_{\mathrm{R}}(1:n_{pb},1:n_{pb},\{\omega_i^{\mathrm{R}}\}_{\Omega(p)}) ▷ Wc(q,ω)W^{\mathrm{c}}(q,\omega) in Eqs. (22) and (24)

  9. for k←{k}K(p)k\leftarrow\{k\}_{K(p)} do

  10. Batch division 1:nall1:n_{\mathrm{all}} into {n′}b\{n'\}_b where b=1,2,⋯ ,Bbatchb=1,2,\cdots,B^{\mathrm{batch}}. ▷ Avoid huge memory for ZZ

  11. for b←1:Bbatchb\leftarrow1:B^{\mathrm{batch}} do

  12. ZWc({n′}b,1:nqp,1:npb)←0ZW^{\mathrm{c}}(\{n'\}_b,1:n_{qp},1:n_{pb})\leftarrow0

  13. Calculate Z(1:npb,{n′}b,1:nqp)Z(1:n_{pb},\{n'\}_b,1:n_{qp}) ▷ For k,qk,q in Eq. (8)

  14. ——— QKΩI\Omega^{\mathrm{I}} block ———

  15. Calculate the weight {wiI(εqn−εq−kn′)∣n′∈{n′}b,n∈nqp,ωiI∈{ωiI}Ω(p)}\{w_i^{\mathrm{I}}(\varepsilon_{qn}-\varepsilon_{q-kn'})\mid n'\in\{n'\}_b,n\in n_{qp},\omega_i^{\mathrm{I}}\in\{\omega_i^{\mathrm{I}}\}_{\Omega(p)}\} ▷ Eq. (22)

  16. for ωiI←{ωiI}Ω(p)\omega_i^{\mathrm{I}}\leftarrow\{\omega_i^{\mathrm{I}}\}_{\Omega(p)} do ▷ QKΩI\Omega^{\mathrm{I}} block

  17. !$acc kernel

  18. for n←1:nqpn\leftarrow1:n_{qp}, n′∈{n′}bn'\in\{n'\}_b do

  19. Zw(1:npb,n′,n)←Z(1:npb,n′,n)wiI(εqn−εq−kn′)Z_w(1:n_{pb},n',n)\leftarrow Z(1:n_{pb},n',n)w_i^{\mathrm{I}}(\varepsilon_{qn}-\varepsilon_{q-kn'})

  20. end for

  21. !$acc end kernel

  22. if GPU version: Wc(1:npb,1:npb)←Wc(1:npb,1:npb,ωiI)W^{\mathrm{c}}(1:n_{pb},1:n_{pb})\leftarrow W^{\mathrm{c}}(1:n_{pb},1:n_{pb},\omega_i^{\mathrm{I}})

  23. if CPU version: Read Wc(1:npb,1:npb)W^{\mathrm{c}}(1:n_{pb},1:n_{pb}) from a file. ▷ For q,ωIq,\omega^{\mathrm{I}} of Wc(q,iωI)W^{\mathrm{c}}(q,i\omega^{\mathrm{I}}) in Eq. (22)

  24. ZWc←Zw†×WcZW^{\mathrm{c}}\leftarrow Z_w^{\dagger}\times W^{\mathrm{c}} ▷ GEMM, accumulate the first dimension of ZwZ_w and WcW^{\mathrm{c}}.

  25. end for

  26. Reshape ZWc({n′}b,1:nqp,1:npb)ZW^{\mathrm{c}}(\{n'\}_b,1:n_{qp},1:n_{pb}) to ZWc(1:npb,{n′}b,1:nqp)ZW^{\mathrm{c}}(1:n_{pb},\{n'\}_b,1:n_{qp}) ▷ OpenACC

  27. ——— QKΩR\Omega^{\mathrm{R}} block ———

  28. Calculate the weight {wiR(εqn−EF,εq−kn′−EF)∣n′∈{n′}b,n∈nqp,ωiR∈{ωiR}Ω(p)}\{w_i^{\mathrm{R}}(\varepsilon_{qn}-E_{\mathrm{F}},\varepsilon_{q-kn'}-E_{\mathrm{F}})\mid n'\in\{n'\}_b,n\in n_{qp},\omega_i^{\mathrm{R}}\in\{\omega_i^{\mathrm{R}}\}_{\Omega(p)}\} ▷ Eq. (24)

  29. Set up pair index {ni(jpair)′,ni(jpair)∣jpair=1,2,⋯ ,npair,i}\{n'_{i(j_{\mathrm{pair}})},n_{i(j_{\mathrm{pair}})}\mid j_{\mathrm{pair}}=1,2,\cdots,n_{\mathrm{pair},i}\} for wiR(εqni(jpair)−EF,εq−kni(jpair)′−EF)≠0w_i^{\mathrm{R}}(\varepsilon_{qn_{i(j_{\mathrm{pair}})}}-E_{\mathrm{F}},\varepsilon_{q-kn'_{i(j_{\mathrm{pair}})}}-E_{\mathrm{F}})\ne0, ωiR∈{ωiR}Ω(p)\omega_i^{\mathrm{R}}\in\{\omega_i^{\mathrm{R}}\}_{\Omega(p)}, and ni(jpair)′∈{n′}bn'_{i(j_{\mathrm{pair}})}\in\{n'\}_b

  30. for ωiR←{ωiR}Ω(p)\omega_i^{\mathrm{R}}\leftarrow\{\omega_i^{\mathrm{R}}\}_{\Omega(p)} do ▷ QKΩR\Omega^{\mathrm{R}} block

  31. !$acc kernel

  32. for jpair←1:npair,ij_{\mathrm{pair}}\leftarrow1:n_{\mathrm{pair},i} do

  33. Zw(1:npb,jpair)←Z(1:npb,ni(jpair)′,ni(jpair))wiR(εqni(jpair)−EF,εq−kni(jpair)′−EF)Z_w(1:n_{pb},j_{\mathrm{pair}})\leftarrow Z(1:n_{pb},n'_{i(j_{\mathrm{pair}})},n_{i(j_{\mathrm{pair}})})w_i^{\mathrm{R}}(\varepsilon_{qn_{i(j_{\mathrm{pair}})}}-E_{\mathrm{F}},\varepsilon_{q-kn'_{i(j_{\mathrm{pair}})}}-E_{\mathrm{F}})

  34. end for

  35. !$acc end kernel

  36. if GPU version: Wc(1:npb,1:npb)←Wc(1:npb,1:npb,ωiR)W^{\mathrm{c}}(1:n_{pb},1:n_{pb})\leftarrow W^{\mathrm{c}}(1:n_{pb},1:n_{pb},\omega_i^{\mathrm{R}})

  37. if CPU version: Read Wc(1:npb,1:npb)W^{\mathrm{c}}(1:n_{pb},1:n_{pb}) from a file. ▷ For q,ωRq,\omega^{\mathrm{R}} of Wc(q,ω)W^{\mathrm{c}}(q,\omega) in Eq. (24)

  38. ZwWc←Zw†×[Wc+Wc†]/2Z_wW^{\mathrm{c}}\leftarrow Z_w^{\dagger}\times[W^{\mathrm{c}}+W^{\mathrm{c}\dagger}]/2 ▷ GEMM

  39. !$acc kernel

  40. for jpair←1:npair,ij_{\mathrm{pair}}\leftarrow1:n_{\mathrm{pair},i} do

  41. ZWc(1:npb,ni(jpair)′,ni(jpair))←ZWc(1:npb,ni(jpair)′,ni(jpair))+ZwWc(jpair,1:npb)ZW^{\mathrm{c}}(1:n_{pb},n'_{i(j_{\mathrm{pair}})},n_{i(j_{\mathrm{pair}})})\leftarrow ZW^{\mathrm{c}}(1:n_{pb},n'_{i(j_{\mathrm{pair}})},n_{i(j_{\mathrm{pair}})})+Z_wW^{\mathrm{c}}(j_{\mathrm{pair}},1:n_{pb})

  42. end for

  43. !$acc end kernel

  44. end for

  45. Σc(1:nqp,1:nqp,q)←Σc(1:nqp,1:nqp,q)+∑iPB∈nPB,n′∈{n′}bZWc(iPB,n′,1:nqp)×Z(iPB,n′,1:nqp)\Sigma^{\mathrm{c}}(1:n_{qp},1:n_{qp},q)\leftarrow\Sigma^{\mathrm{c}}(1:n_{qp},1:n_{qp},q)+\sum_{i_{\mathrm{PB}}\in n_{\mathrm{PB}},n'\in\{n'\}_b}ZW^{\mathrm{c}}(i_{\mathrm{PB}},n',1:n_{qp})\times Z(i_{\mathrm{PB}},n',1:n_{qp}) ▷ GEMM

  46. end for

  47. end for

  48. end for

  49. if K(p)==0K(p)==0 and Ω(p)==0\Omega(p)==0 then ▷ Reduction for q,k,ωq,k,\omega

  50. Σc(1:nqp,1:nqp,{q})←∑p′Σcp′(1:nqp,1:nqp,{q})\Sigma^{\mathrm{c}}(1:n_{qp},1:n_{qp},\{q\})\leftarrow\sum_{p'}\Sigma^{\mathrm{c}p'}(1:n_{qp},1:n_{qp},\{q\}) ▷ Σcp′\Sigma^{\mathrm{c}p'} denotes Σc\Sigma^{\mathrm{c}} at the MPI process p′p'. MPI reduce sum.

  51. Write Σc(1:nqp,1:nqp,{q})\Sigma^{\mathrm{c}}(1:n_{qp},1:n_{qp},\{q\})

  52. end if

However, since TF32 operations reduce computational precision, they can affect the accuracy of the results. Thus, a verification of the computational accuracy of MP is a necessity for the application. Note that our MP implementation does not replace all GPU operations with single-precision or TF32. For example, operations such as matrix inversion and the calculation of the IPW overlap matrix, which may require high numerical precision, are performed using DP arithmetic.

The numerical accuracy of the MP approach may depend on the materials and computational settings. While further validation across a broader range of materials and computational settings is necessary, this study primarily focuses on two benchmark systems to evaluate the computational efficiency and numerical accuracy of the MP approach.

Benchmark and Discussion

Benchmark system

In this study, we benchmark two physical systems: a Type-II superlattice of the semiconductor compound (InAs)n_n(GaSb)n_n (n=2,4,6,8,10n = 2, 4, 6, 8, 10) and the modulated martensite (10M, 14M) and non-modulated (NM) phases of Ni2_2MnGa. The former has a narrow direct band gap that can be adjusted by the number of stacking layers (nn), making it suitable for infrared sensor applications [43]. The latter is known as a ferromagnetic shape memory alloy and is expected to be applied to functional materials, such as high-speed actuators [44, 45, 46]. In both cases, issues with the physical accuracy of the evaluation of electronic correlations have been pointed out in calculations at the LDA/GGA level, and therefore analysis using more precise methods such as QSGW is desired [9, 15, 47].

Figure 3 illustrates their crystal structures. In (InAs)n_n(GaSb)n_n, the unit cells are tetragonal, and the construction of the structures follows the data described in Ref. [9]. We assumed a constant interface distance between InAs and GaSb. In Ni2_2MnGa, the unit cells of the modulated phases are triclinic, and the structure is reproduced by building tetragonal blocks, where 10M (14M) has a 32ˉ32ˉ3\bar{2}3\bar{2} (52ˉ52ˉ5\bar{2}5\bar{2}) shearing sequence. The aspect ratio of the Ni tetragonal sublattice (c/ac/a) and the density are determined from experimental values, specifically c/a=1.21c/a = 1.21 and 49.411549.4115 Å3^3/f.u. [48]. The scale of the calculations is shown in Table 2. As mentioned above, since ZsizeZ_{\mathrm{size}} exceeded the memory capacity available to a single MPI, they are handled in segments. The segment size is determined on the basis of the available memory size. Note that, in general, the amount of memory available per MPI process differs between the CPU and GPU versions, resulting in different segmented batch sizes for intermediate states.

Structures of (InAs)₈(GaSb)₈ and Ni₂MnGa 14M

Figure 3. Structures of (InAs)8_8(GaSb)8_8(a) and Ni2_2MnGa 14M (b), where In, As, Ga, Sb, Mn, and Ni are represented by pink, green, red, yellow, blue, and gray spheres, respectively. The solid black lines represent the unit cell, and the dashed orange lines in (b) represent the boundaries of the nano twins.

(InAs)n(GaSb)n(\mathrm{InAs})_n(\mathrm{GaSb})_nNi2MnGa\mathrm{Ni}_2\mathrm{MnGa}
n=2n=2n=4n=4n=6n=6n=8n=8n=10n=10NM10M14M
natomn_{\mathrm{atom}}81624324082028
nspinn_{\mathrm{spin}}11111222
ncoren_{\mathrm{core}}2364727089441180132330462
nvaln_{\mathrm{val}}9218427636846092230322
nqpn_{\mathrm{qp}}206287428571713123301421
nalln_{\mathrm{all}}361722109014441808269670940
nmbn_{\mathrm{mb}}366732110614641834281690971
npbn_{\mathrm{pb}}1356209731394189523198224793471
nCpbn_{\mathrm{Cpb}}19343253487365018121128431994479
nBZn_{\mathrm{BZ}}16(32)16(32)161616482424
nIBZn_{\mathrm{IBZ}}9(18)9(18)999141414
ZsizeZ_{\mathrm{size}} (GiB)1.506.4721.8251.47100.480.487.4520.47
WsizeW_{\mathrm{size}} (GiB)2.225.3111.8921.1833.031.167.4214.54

Table 2. Setting parameters, where these values represent the maximum in all kk points. nvaln_{\mathrm{val}} is the number of valence electrons, which matches noccn_{\mathrm{occ}} in insulator systems. natomn_{\mathrm{atom}} is the number of atoms in the unit cell. The array size of ZZ (ZsizeZ_{\mathrm{size}}) is per kk and qq point and size of WW (WsizeW_{\mathrm{size}}) is per qq point. These sizes become half in the case of the MP approach. In (InAs)n(GaSb)n(\mathrm{InAs})_n(\mathrm{GaSb})_n, nBZn_{\mathrm{BZ}} of 16 (32) came from 4×4×14 \times4 \times1 (4×4×24 \times4 \times2), while in Ni2MnGa\mathrm{Ni}_2\mathrm{MnGa}, nBZn_{\mathrm{BZ}} of 24 (48) corresponds to 4×3×24 \times3 \times2 (4×3×44 \times3 \times4). The numbers in parentheses at nBZn_{\mathrm{BZ}} and nIBZn_{\mathrm{IBZ}} indicate the values used in calculating the band gap.

All benchmarks are performed at the Supercomputer Center, ISSP, Univ. of Tokyo (system C: Kugui). For the GPU version, the ACC one node in Kugui installed a CPU (AMD EPYC 7763), and four GPUs (NVIDIA A100 40GB for HGX) are used with four MPI processors. For the CPU version, four CPU nodes in Kugui, with a total of 8 CPUs (AMD EPYC 7763), are used with 128 MPI with four-thread parallelization. As shown in Table 1 and Table 2, the QSGW calculations use significantly more memory than the LDA calculations. Although memory segmentation is implemented, this parallel setting ensures that each MPI process has the necessary memory to perform the calculations. This thread parallelization is achieved only through the MKL BLAS and LAPACK libraries. The current efficiency of thread parallelization (4 threads) is measured to be 1.6 times faster than the flat MPI version in the case of (InAs)4(GaSb)4(\mathrm{InAs})_4(\mathrm{GaSb})_4. The observed reduction in thread parallelization efficiency can be attributed to the decreased efficiency of matrix operations. This phenomenon has been observed in our benchmarks of complex matrix multiplication: suppression of thread-parallel efficiency occurs in highly MPI-parallelized computations. The NVIDIA HPC SDK Version 24.7 compiler and its GPU-accelerated math libraries (cuBLAS, cuSOLVER) are used for our implementation.

Benchmark results

Figure 4 shows the total computational time (elapsed time) for the four main tasks in one cycle of QSGW calculations. This comparison examines the calculation time performed on four CPU nodes versus one GPU node. The computational time on the GPU is successfully reduced compared to that of the CPU. In the paper, we define the acceleration ratio as the ratio of computational time between different approaches. For example, the acceleration ratio of GPU/CPU is estimated from the computational time of the CPU divided by that of the GPU. In (InAs)n(GaSb)n(\mathrm{InAs})_n(\mathrm{GaSb})_n with (n≥6n \ge6) and 14M, an acceleration ratio of more than 10 times was achieved. Furthermore, the MP approach showed an acceleration ratio of up to 38 times compared to the CPU, which is approximately 3 times faster than the GPU version. The acceleration ratio is almost saturated on larger systems, implying that the computational resources of GPU are fully utilized. In contrast, the acceleration ratio of the smaller system is relatively small. Such low efficiency is due to the inefficient use of GPU computational resources in smaller systems and bottlenecks in parts not yet optimized for GPUs.

Figure 4. Computational time of Σ\Sigma. The meaning of the key is as follows. CPU (4 nodes): This is the reference to measure the speed of GPU computations. The total number of cores is 128×4128 \times4. GPU (1 node): Use one node with four GPUs. GPUMP^{\mathrm{MP}} (1 node): Use one node with four GPUs with mixed precision GPU code. In the figure, ×12 means that the computational speed of the GPU (1 node) is worth (128×4)×12(128 \times4) \times12 cores (48 nodes). Note that the CPU (4 nodes) results were already below the linear scaling proportional to the number of cores, despite our efforts to use the CPU cores as efficiently as possible. See the text for details.

It is important to note that the ideal computational performance of four CPU nodes is 20.2 TFLOPS, while that of one GPU node is 38.8 TFLOPS (77.6 TFLOPS with Tensor Core). Compared to this, our GPU computation speed appears excessive because a speedup of at most 3.8 (77.6/20.2) times is expected, based on a simple ratio from the ideal values. This is attributed to the fact that we reduce the number of

MPI processes to secure sufficient CPU memory. The number of MPI parallelizations on the CPU (128) is 32 times larger than on the GPU (4), leading to bottlenecks due to file input/output (I/O) and MPI communications. Ignoring the efficiency of thread parallelization, the performance of 128 CPU cores is 5.05 TFLOPS, whereas the performance of the 1 GPU node with Tensor Core is 15.3 times larger than that.

Figure 5 shows the computation times for the four main tasks. When comparing the four tasks, the computation time increases in the order of exchange self-energy, core-exchange self-energy, screened Coulomb interaction, and correlation self-energy. Moreover, the proportion of correlation self-energy calculation increases as the system size increases. For all of these main tasks, the GPU accelerates calculations, reducing computation time compared to 4 CPU node calculations. The accelerations by GPU in smaller systems, such as (InAs)2_2(GaSb)2_2 and Ni2_2MnGa NM structure, are relatively lower than those of larger systems. The reason is that the GPU’s computational resources are not fully utilized in smaller systems, and CPU computation and data transfer costs are relatively high. However, in larger systems, the acceleration ratio of these tasks increases to around 10 and 30 in the GPU and GPUMP^{\mathrm{MP}}, respectively. This indicates that GPU resources are utilized more efficiently, and the costs of CPU computation and data transfer are relatively low.

Figure 5. Computational time of the predominant four tasks. For the acceleration ratio, see the left (right) vertical axis for between GPU (GPUMP^{\mathrm{MP}}) and CPU.

Figure 6(a-Figure 6d) illustrates the computation time for the kernel parts of each task listed in Table 1. IcI^{c} and RcR^{c} correspond to the computational time of the QKΩIQK\Omega I and QKΩRQK\Omega R blocks, respectively, in Algo. 2. As noted in Algo. 2, line 45, the multiplication of the matrix with ZZ is shared for IcI^{c} and RcR^{c} parts, thus they are not entirely consistent with Eqs. (22) and (24). Note that the data of the calculation of ϵ−1\epsilon^{-1} by GPUMP^{\mathrm{MP}} are not listed because we did not employ the MP approach in this part. As shown in Figure 6(a-Figure 6d), the computation time of these tasks basically increases exponentially, which is described in Table 1. Except for Σc\Sigma^{c}, the calculation of ZZ takes the most time. In contrast, for Σc\Sigma^{c}, the main computation is the imaginary-axis integration (IcI^{c}), followed by ZZ and real-axis integration (RcR^{c}). The GPU and GPUMP^{\mathrm{MP}} versions drastically improve the computational time for all kernel parts. Figure 6 (e-l) shows the acceleration ratio between the GPU and the CPU (e-h) and between the GPUMP^{\mathrm{MP}} and the GPU (i-l). Even considering that the current CPU thread parallel efficiency is lower than the ideal value of 4, at approximately 1.5, the GPU acceleration ratio from CPU on ZZ and RcR^{c} appears excessive. This is because, as previously mentioned, in the CPU version, the ZZ computation for WW involves MPI communication due to the parallel computation for MPB, and the RR computation is limited by the file I/O for WW. We assign one MPI process to one GPU, which allows the GPU version to utilize more memory per MPI process than the CPU version, contributing to the excessive acceleration. In GPUMP\mathrm{GPU}^{\mathrm{MP}}, computations, which mainly consist of matrix multiplication, such as the calculation of IcI^{c}, achieve approximately a four-fold speedup compared to GPU.

Figure 6. Computational time of kernel calculation in the predominant four tasks and those of acceleration ratio between GPU (GPUMP^{\mathrm{MP}}) and CPU. In the topmost figures, the vertical and horizontal axes are set to a logarithmic scale.

Accuracy of mixed precision calculation

As shown in Figure 4, the computation speed in the MP approach (GPUMP\mathrm{GPU}^{\mathrm{MP}}) is more than three times faster compared to that of the DP approach (GPU). Although this approach enables faster computations, there are concerns regarding the reduction of the accuracy of the calculations. To verify the accuracy of the computation results, we compared the quasiparticle self-energy obtained from these approaches. Specifically, in the first cycle of QSGW calculations, we evaluated the differences in the diagonal components of the self-energies on (InAs)6_6(GaSb)6_6 and Ni2_2MnGa 10M as a representative of each system.

Figure 7 shows the difference in self energies between MP and DP as a function of the initial eigenvalue. It shows that the difference between DP and MP varies with energy, tending to be larger at lower energy levels. Additionally, the difference is smaller in the case of core-exchange self energy. This may be because the core exchange self-energy calculation uses only the wave functions within the MT spheres, whose basis functions are analytically orthogonal. The mean differences and their root mean square were −7-7 meV and 9 meV in the (InAs)6_6(GaSb)6_6 and 5 meV and 6 meV in 10M systems, respectively.

Figure 7. Self-energy and its components differences between MP and DP at Γ\Gamma point, where ΔΣCx\Delta\Sigma^{\mathrm{Cx}}, ΔΣx\Delta\Sigma^{\mathrm{x}}, ΔΣc\Delta\Sigma^{\mathrm{c}} and ΔΣ\Delta\Sigma are the differences on core-exchange, exchange, correlation, and total self-energies, respectively. The horizontal axis represents the quasiparticle energy in LDA/GGA calculations.

Electronic structure from large-scale QSGW calculations

This subsection addresses the electronic structures obtained from the benchmark calculations. By considering interfaces and modulation structures, large-scale QSGW calculations enable direct and non-empirical treatment of their contributions. The DP approach is used for the following results. The convergence criterion for the QSGW calculations was set to ΔQP=0.005\Delta_{\mathrm{QP}} = 0.005 eV for quasiparticle energies below 20 eV.

Type-II superlattice of (InAs)n_n(GaSb)n_n

Figure 8(a) shows the band gap obtained by QSGW80 with spin-orbit coupling (SOC) correction. The results of n=3n = 3 and n=5n = 5 are obtained using the same setting of kk points with n=4n = 4 and n=6n = 6, respectively. We find that the band gap is maximized at n=3n = 3 and reduced by increasing or decreasing nn. This trend agrees with the previous QSGW80 + SO results of Otsuka et al. [9] and the experimental results [49, 50]. For n=10n = 10, the band gap value is 0.18 eV, a decrease of about 30% from that of n=3n = 3 (0.27 eV). Note that the calculated band gaps for small values of nn are slightly lower than those reported by Otsuka [9], despite both being obtained using the ecalj package. This discrepancy arises from differences in the numerical treatment of the atomic base set and k→0k \to0 on the WcW_c, as described in § 4.3.

Figure 8. Band gap (a) with respect to nn. The dashed lines are a guide for the eye. Band dispersion of n=3n = 3 (b) and n=10n = 10 (c). Ref. [Otsuka] is [9], Exp. [PL] and Exp. [Abs] are experimental values from photoluminescence and optical absorption. [49, 50] These experimental data are extrapolated values to zero kelvin. The upper and lower edges of the shadowed area in (a) are the band gap obtained from the non-self-consistent calculation on 70% and 90% of hybrid QSGW approaches from the result of QSGW80. The shadowed area in (b) and (c) is the projected band dispersion of bulk InAs (blue) and GaSb (yellow) to the two-dimensional BZ (2D-BZ) with band offset correction. The upper horizontal axis indicates the symbols in 2D-BZ.

In previous studies using GW0GW_0 calculations [51], the obtained band gap of 0.234 eV for n=10n = 10 is in good agreement with the experimental results of 0.234 eV and 0.228 eV [50]. However, it is larger than for n=8n = 8, contrary to the experimental facts and the present results. Such discrepancies may be attributed to the fact that the GW0GW_0 approach does not update the wave functions from the LDA ones. The gray area is based on the results of QSGW80 and represents the region enclosed by the band gaps calculated using QSGW70 and QSGW90 with SOC. The experimental results appear between QSGW80 and QSGW90, implying that the effect of the vertex may not be significant in this material.

The dispersion curves with projected areas of the bulk band are shown in Figure 8(b) and (Figure 8c). We evaluated the band offset to the bulk band of InAs and GaSb based on the energy levels of the core states without SOI, specifically, the Sb 3p and As 2p states. Note that bulk bands were calculated with the same in-plane lattice constant as the (InAs)n_n(GaSb)n_n case. The calculated offset values for the Fermi level of n=10n = 10 are 0.25 eV and −0.10-0.10 eV for InAs and GaSb, respectively. Considering these offsets, the conduction band minimum (CBM) of InAs is higher than the valence band maximum (VBM) of GaSb, indicating the charge transfer from InAs to GaSb by forming the interface. The obtained value of the difference between the CBM of InAs and the VBM of GaSb is 0.10 eV, which is in good agreement with the previous investigation based on experimental measurements of 0.15 eV [52]. The quantum well states (e1e_1, e2e_2) are clearly observed in the band dispersion of n=10n = 10. The dispersion of e1e_1 and e2e_2 in the Z-Γ\Gamma region is due to the interaction between the quantum well states across the interface. Since the e1e_1 and e2e_2 states have different parities, their dispersions in Z-Γ\Gamma are reversed. The bandwidths of e1e_1 and e2e_2 are 0.18 eV and 0.20 eV, respectively, while the heavy hole state (hh1hh_1) exhibits a dispersionless band in the Z-Γ\Gamma region. This behavior is also reported with the previous model calculations [37, 53]. It suggests that, along the transport direction (i.e., zz-direction), e1e_1 and e2e_2 possess high group velocities, whereas efficient transport of hh1hh_1 would require the application of a strong electric field.

Figure 9. Total density of states (DOS) (a) and its enlarged plots near the Fermi level (b) of the NM, 10M, and 14M structures on Ni2_2MnGa. The origins of the vertical axis of (a) are set as ±3\pm3 and ±6\pm6 for the data of 10M and 14M, respectively. The top (bottom) graph indicates the results of QSGW (GGA). In all cases, the Fermi energy is set to 0 eV.

Ni2_2MnGa martensite phases of NM, 10M, and 14M

Figure 9 shows the density of states (DOS) of Ni2_2MnGa in the NM, 10M, and 14M phases, assuming a ferromagnetic state. The results of the PBE version of the generalized gradient approximation (GGA) are also shown for the comparison [54]. By comparison of the QSGW and GGA results, the peak of unoccupied states in minority spins appears around 1.3 eV on GGA, whereas it is approximately 0.4 eV higher energy in QSGW. Additionally, the DOS intensity in QSGW is higher, resulting from the narrowing of the bandwidth. These differences between the QSGW and GGA results arise from the improvement of the excessive delocalization of the dd electrons typically observed in GGA.

Focusing on the vicinity of the Fermi level, in GGA, the DOS of minority spin states for all NM, 10M, and 14M structures are located near the peak. In contrast, in QSGW, the Fermi level is near the valley of the DOS. It is at the bottom in the case of 10M and 14M structures, in contrast to the case in NM. This implies that the modulated structures observed in 10M and 14M are realized by stabilizing the electronic structure from NM. The peak structure of the electronic states obtained from the GGA may be suppressed by inducing deformation in the crystal structure. In fact, it has been reported that GGA overestimates the optimized c/ac/a ratio on the NM structure and energetically stabilizes the artificial structure (4O) compared to 10M and 14M [55].

One of the reasons for the difference in the electronic structure of minority spins between GGA and QSGW is due to the difference in the magnitude of the exchange splitting. The calculated magnetic moments per formula unit for the NM, 10M, and 14M structures are 4.60, 4.51, and 4.50 μB\mu_{\mathrm{B}}, respectively, in QSGW. These are in contrast to GGA values, 4.17 (NM), 4.13 (10M), and 4.16 μB\mu_{\mathrm{B}} (14M). The magnetic moments in QSGW are overestimated compared to the experimental values for the martensite phase: 4.23 [56], 4.16 [57], and 4.17 μB\mu_{\mathrm{B}} [48]. This increase can be attributed to the enhanced localization of the dd electrons, which leads to a larger intra-atomic exchange splitting. Indeed, even within the framework of DFT, the use of advanced exchange-correlation potentials beyond GGA, such as meta-GGA (SCAN) or hybrid functionals (HSE03), has also been reported to yield increased magnetic moments in Ni2_2MnGa alloy. Specifically, for the NM structure, magnetic moment of 5.15 μB\mu_{\mathrm{B}} (c/ac/a=1.08) with HSE03 and 4.73 μB\mu_{\mathrm{B}} (c/ac/a=1.17) with SCAN were reported in Ref. [58], while a value of 4.67 μB\mu_{\mathrm{B}} (c/ac/a=1.21) was obtained with SCAN in Ref. [47]. The overestimation of magnetic moments compared to experimental values is likely due to the absence of spin fluctuations in the calculations [59]. Incorporating such effects would be expected to reduce the calculated values.

summary

We have developed a GPU-accelerated QSGW calculation code by extending the code ecalj. This implementation was carried out using OpenACC programming and GPU-accelerated math libraries to maintain the maintainability of computational code. Two benchmarks have shown that GPU computations successfully accelerated the QSGW calculation, with one GPU node performing calculations equivalent to approximately 40 CPU nodes. Moreover, in our benchmark, the number of MPI parallelizations in GPU computations was significantly smaller than CPU computations due to the computational performance of the GPU. This also helped reduce bottlenecks in MPI communication and file I/O, which in turn contributed to the acceleration. The mixed precision approach was also developed, achieving a speedup of over three times with the double precision version while maintaining acceptable numerical accuracy in self-energy. The developed GPU-accelerated QSGW code facilitates applications to large-scale systems, such as interfaces, surfaces, and superlattice structures, paving the way to accurate electronic structure calculations in extended systems.

Acknowledgement

The first-principles quasiparticle self-consistent GWGW calculations were performed using the Supercomputer Center, Institute for Solid State Physics, University of Tokyo, Japan. This work was partially supported by JSPS KAKENHI, Grant-in-Aid for Scientific Research (Nos. 21KK0083, 24K08229, 24K17608, and 25K01665). The crystal structure illustrations were produced using VESTA [60].

References

  1. [1]F. Aryasetiawan, O. Gunnarsson, The GW method, Rep. Prog. Phys. 61 (3) (1998) 237. doi:10.1088/0034-4885/61/3/002.DOI
  2. [2]D. Deguchi, K. Sato, H. Kino, T. Kotani, Accurate energy bands calculated by the hybrid quasiparticle self-consistent GW method implemented in the ecalj package, Jpn. J. Appl. Phys. 55 (5) (2016) 051201. doi:10.7567/JJAP.55.051201.DOI
  3. [3]L. Hedin, S. Lundqvist, Effects of Electron-Electron and Electron-Phonon Interactions in the One-Electron States of Solids, in: Solid State Phys., Vol. 23, Elsevier, 1970, pp. 1–181. doi:10.1016/S0081-1947(08)60615-3.DOI
  4. [4]M. S. Hybertsen, S. G. Louie, Electron correlation in semiconductors and insulators: Band gaps and quasiparticle energies, Phys. Rev. B 34 (8) (1986) 5390. doi:10.1103/PhysRevB.34.5390.DOI
  5. [5]M. Usuda, N. Hamada, T. Kotani, M. van Schilfgaarde, All-electron GW calculation based on the LAPW method: Application to wurtzite ZnO, Phys. Rev. B 66 (12) (2002) 125101. doi:10.1103/PhysRevB.66.125101.DOI
  6. [6]M. Usuda, H. Hamada, K. Shiraishi, A. Oshiyama, Band structures of wurtzite InN and Ga1−xInxN by all-electron GW calculation, Jpn. J. Appl. Phys. Lett. (part 2) 43 (2004) L407.
  7. [7]M. van Schilfgaarde, T. Kotani, S. Faleev, Quasiparticle self-consistent GW theory, Phys. Rev. Lett. 96 (2006) 226402. doi:10.1103/PhysRevLett.96.226402.DOI
  8. [8]T. Kotani, M. van Schilfgaarde, S. V. Faleev, Quasiparticle self-consistent GW method: A basis for the independent-particle approximation, Phys. Rev. B 76 (2007) 165106. doi:10.1103/PhysRevB.76.165106.DOI
  9. [9]J. Otsuka, T. Kato, H. Sakakibara, T. Kotani, Band structures for short-period (InAs)n(GaSb)n superlattices calculated by the quasiparticle self-consistent GW method, Jpn. J. Appl. Phys. 56 (2) (2017) 021201. doi:10.7567/JJAP.56.021201.
  10. [10]S. V. Faleev, M. van Schilfgaarde, T. Kotani, All-electron self-consistent GW approximation: Application to Si, MnO, and NiO, Phys. Rev. Lett. 93 (12) (2004) 126406. doi:10.1103/PhysRevLett.93.126406.DOI
  11. [11]F. Bruneval, N. Vast, L. Reining, M. Izquierdo, F. Sottile, N. Barrett, Exchange and correlation effects in electronic excitations of Cu2O, Phys. Rev. Lett. 97 (2006) 267601. doi:10.1103/PhysRevLett.97.267601.DOI
  12. [12]S. W. Jang, T. Kotani, H. Kino, K. Kuroki, M. J. Han, Quasiparticle self-consistent GW study of cuprates: electronic structure, model parameters, and the two-band theory for Tc, Sci. Rep. 5 (2015) 12050. doi:10.1038/srep12050.DOI
  13. [13]https://github.com/tkotani/ecalj.
  14. [14]H. Yoon, S. W. Jang, J. H. Sim, T. Kotani, M. J. Han, Magnetic force theory combined with quasi-particle self-consistent GW method, J. Phys. Condens. Matter 31 (2019) 405503. doi:10.1088/1361-648X/ab2b7e.DOI
  15. [15]M. Obata, T. Kotani, T. Oda, Intrinsic instability to martensite phases in ferromagnetic shape memory alloy Ni2MnGa: Quasiparticle self-consistent GW investigation, Phys. Rev. Mater. 7 (2023) 024413. doi:10.1103/physrevmaterials.7.024413.DOI
  16. [16]J. Luštinec, M. Obata, R. Majumder, K. Hyodo, T. Kotani, L. Kalvoda, T. Oda, Comparative study of electronic structure in ferromagnetic heusler alloys Ni₂MnX (X = Al, Ga, In) using the quasi-particle self-consistent GW method, J. Magn. Soc. Jpn. 48 (6) (2024) 94–107. doi:10.3379/smjmag.2411R001.DOI
  17. [17]N. Marzari, A. A. Mostofi, J. R. Yates, I. Souza, D. Vanderbilt, Maximally localized Wannier functions: Theory and applications, Reviews of Modern Physics 84 (4) (2012) 1419–1475. doi:10.1103/RevModPhys.84.1419.DOI
  18. [18]H. Okumura, K. Sato, T. Kotani, Spin-wave dispersion of 3d ferromagnets based on quasiparticle self-consistent GW calculations, Phys. Rev. B 100 (2019) 054419. doi:10.1103/PhysRevB.100.054419.DOI
  19. [19]H. Okumura, K. Sato, T. Kotani, Nonlinear extension of the dynamical linear response of spin: Extended heisenberg model, J. Phys. Soc. Jpn. 90 (9) (2021) 094710. doi:10.7566/JPSJ.90.094710.DOI
  20. [20]T. Kotani, Quasiparticle self-consistent GW method based on the augmented plane-wave and muffin-tin orbital method, J. Phys. Soc. Jpn. 83 (9) (2014) 094711. doi:10.7566/JPSJ.83.094711.DOI
  21. [21]S. Takano, T. Kotani, M. Obata, H. Fujii, K. Sato, H. Saito, T. Oda, in preparation.
  22. [22]H. Ness, F. Corsetti, D. Pashov, B. VerstiicheI, G. W. Winkler, M. van Schilfgaarde, R. M. Lutchyn, Self-consistent quasiparticle GW and hybrid functional calculations for Al/InAs/Al heterojunctions: Band offset and spin-orbit coupling effects, Phys. Rev. B 110 (2024) 195301. doi:10.1103/PhysRevB.110.195301.DOI
  23. [23]T. Kotani, M. van Schilfgaarde, Fusion of the LAPW and LMTO methods: The augmented plane wave plus muffin-tin orbital method, Phys. Rev. B 81 (2010) 125117. doi:10.1103/PhysRevB.81.125117.DOI
  24. [24]H. Sakakibara, T. Kotani, M. Obata, T. Oda, Finite electric-field approach to evaluate the vertex correction for the screened Coulomb interaction in the quasiparticle self-consistent GW method, Phys. Rev. B 101 (2020) 205120. doi: 10.1103/PhysRevB.101.205120.DOI
  25. [25]L. Landau, On the conservation laws for weak interactions, Nucl. Phys. 3 (1) (1957) 127. doi:10.1016/0029-5582(57)90061-5.DOI
  26. [26]R. Fujita, K. Konaga, Y. Ueoka, Y. Kamakura, N. Mori, T. Kotani, Analysis of anisotropic ionization coefficient in bulk 4H-SiC with full-band Monte Carlo simulation, in: 2017 International Conference on Simulation of Semiconductor Processes and Devices (SISPAD), IEEE, Kamakura, Japan, 2017, pp. 289–292. doi:10.23919/SISPAD.2017.8085321.DOI
  27. [27]M. Shishkin, M. Marsman, G. Kresse, Accurate quasiparticle spectra from self-consistent GW calculations with vertex corrections, Phys. Rev. Lett. 99 (2007) 246403. doi:10.1103/PhysRevLett.99.246403.DOI
  28. [28]W. Chen, A. Pasquarello, Accurate band gaps of extended systems via efficient vertex corrections in GW, Phys. Rev. B 92 (2015) 041115. doi:10.1103/PhysRevB.92.041115.DOI
  29. [29]T. Kotani, M. van Schilfgaarde, An-electron GW approximation with the mixed basis expansion based on the full-potential LMTO method, Solid State Commun. 121 (9) (2002) 461. doi:https://doi.org/10.1016/S0038-1098(02)00028-5.
  30. [30]Introduction to the LAPW Method, in: D. J. Singh, L. Nordström (Eds.), Planewaves, Pseudopotentials and the LAPW Method, Springer US, Boston, MA, 2006, pp. 43–52. doi:10.1007/978-0-387-29684-5_4.DOI
  31. [31]F. Aryasetiawan, O. Gunnarsson, Product-basis method for calculating dielectric matrices, Phys. Rev. B 49 (1994) 16214. doi:10.1103/PhysRevB.49.16214.DOI
  32. [33]A. L. Fetter, J. D. Walecka, Quantum Theory of Many-Particle Systems, Dover Publications, 1971.
  33. [34]J. Rath, A. J. Freeman, Generalized magnetic susceptibilities in metals: Application of the analytic tetrahedron linear energy method to sc, Phys. Rev. B 11 (1975) 2109. doi:10.1103/PhysRevB.11.2109.DOI
  34. [35]T. Kotani, Contracted plane wave satisfying periodic gauge, arXiv preprint arXiv:2010.03959 (2020).arxiv.org/abs/2010.03959
  35. [36]W. Jia, J. Fu, Z. Cao, L. Wang, X. Chi, W. Gao, L.-W. Wang, Fast plane wave density functional theory molecular dynamics calculations on multi-GPU machines, J. Comput. Phys. 251 (2013) 102. doi:https://doi.org/10.1016/j.jcp.2013.05.005.
  36. [37]S. Das, P. Motamarri, V. Subramanian, D. M. Rogers, V. Gavini, DFT-FE 1.0: A massively parallel hybrid CPU-GPU density functional theory code using finite-element discretization, Comput. Phys. Commun. 280 (2022) 108473. doi:https://doi.org/10.1016/j.cpc.2022.108473.
  37. [38]D. Pashov, S. Acharya, W. R. Lambrecht, J. Jackson, K. D. Belashchenko, A. Chantis, F. Jamet, M. van Schilfgaarde, QuestaaI: A package of electronic structure methods based on the linear muffin-tin orbital technique, Comput. Phys. Commun. 249 (2020) 107065. doi:https://doi.org/10.1016/j.cpc.2019.107065.
  38. [39]J. Barker, D. Pashov, J. Jackson, Electronic structure and finite temperature magnetism of yttrium iron garnet, Electron. Struct. 2 (4) (2021) 044002. doi:10.1088/2516-1075/abd097.DOI
  39. [40]V. W.-z. Yu, M. Govoni, GPU acceleration of large-scale full-frequency GW calculations, J. Chem. Theory Comput. 18 (8) (2022) 4690–4707. doi:10.1021/acs.jctc.2c00241.
  40. [41]B. Li, T. Patel, S. Samsi, V. Gadepally, D. Tiwari, MISO: exploiting multi-instance GPU capability on multi-tenant GPU clusters, in: Proceedings of the 13th Symposium on Cloud Computing, SoCC ’22, Association for Computing Machinery, New York, NY, USA, 2022, p. 173. doi:10.1145/3542929.3563510.DOI
  41. [42]M. Gajdoš, K. Hummer, G. Kresse, J. Furthmüller, F. Bechstedt, Linear optical properties in the projector-augmented wave methodology, Phys. Rev. B 73 (2006) 045112. doi:10.1103/PhysRevB.73.045112.DOI
  42. [43]Y.-M. Niquet, C. Delerue, Band offsets, wells, and barriers at nanoscale semiconductor heterojunctions, Phys. Rev. B 84 (2011) 075478. doi:10.1103/PhysRevB.84.075478.DOI
  43. [44]K. Ullakko, J. K. Huang, C. Kantner, R. C. O’Handley, V. V. Kokorin, Large magnetic-field-induced strains in Ni₂MnGa single crystals, Appl. Phys. Lett. 69 (1996) 1966. doi:10.1063/1.117637.DOI
  44. [45]O. Heczko, V. Kopecký, A. Sozinov, L. Straka, Magnetic shape memory effect at 1.7 K, Appl. Phys. Lett. 103 (2013) 072405. doi:10.1063/1.4817941.DOI
  45. [46]A. Sozinov, N. Lanska, A. Soroka, W. Zou, 12% magnetic field-induced strain in Ni-Mn-Ga-based non-modulated martensite, Appl. Phys. Lett. 102 (2013) 021902. doi:10.1063/1.4775677.DOI
  46. [47]D. R. Baigutlin, V. V. Sokolovskiy, O. N. Miroshkina, M. A. Zagrebin, J. o. Nekolaine, Electronic structure beyond the generalized gradient approximation for Ni₂MnGa, Phys. Rev. B 102 (2020) 045127. doi:10.1103/PhysRevB.102.045127.
  47. [48]P. J. Webster, K. R. Ziebeck, S. L. Town, M. S. Peak, Magnetic order and phase transformation in Ni₂MnGa, Philos. Mag. B 49 (1984) 295. doi:10.1080/13642817408246515.DOI
  48. [49]A. P. Ongstad, R. Kaspi, C. E. Moeller, M. L. Tilton, D. M. Gianardi, J. R. Chavez, G. C. Dente, Spectral blueshift and improved luminescent properties with increasing GaSb layer thickness in InAs–GaSb type-II superlattices, J. Appl. Phys. 89 (4) (2001) 2185. doi:10.1063/1.3379178.DOI
  49. [50]B. Klein, E. Plis, M. N. Kutty, N. Gautam, A. Albrecht, S. Myers, S. Krishna, Varshni parameters for InAs/GaSb strained layer superlattice infrared photodetectors, J. Phys. D: Appl. Phys. 44 (7) (2011) 075102. doi:10.1088/0022-3727/44/7/075102.DOI
  50. [51]Z. Taghipour, E. Shojaee, S. Krishna, Many-body perturbation theory study of type-II InAs/GaSb superlattices within the GW approximation, J. Phys.: Condens. Matter 30 (32) (2018) 325701. doi:10.1088/1361-648X/aacdce.DOI
  51. [52]I. Vurgaftman, J. R. Meyer, L. R. Ram-Mohan, Band parameters for III–V compound semiconductors and their alloys, J. Appl. Phys. 89 (11) (2001) 5815. doi:10.1063/1.1368156.
  52. [53]S. Mukherjee, A. Singh, A. B. Ramdhakar, B. Muralidharan, Carrier localization and miniband modeling of inas/gasb based type-ii superlattice infrared detectors, J. Phys. D: Appl. Phys. 54 (34) (2021) 345104. doi:10.1088/1361-6463/ac0702.DOI
  53. [54]J. P. Perdew, K. Burke, M. Ernzerhof, Generalized gradient approximation made simple, Phys. Rev. Lett. 77 (1996) 3865. doi:10.1103/PhysRevLett.77.3865.DOI
  54. [55]M. Zelený, L. Straka, A. Sozinov, O. Hečzko, Ab initio prediction of stable nanotwin double layers and 4O structure in Ni₂MnGa, Phys. Rev. B 94 (2016) 224108. doi:10.1103/PhysRevB.94.224108.DOI
  55. [56]K. Ooiwa, K. Endo, A. Shinogi, A structural phase transition and magnetic properties in a Heusler alloy Ni₂MnGa, J. Magn. Magn. Mater. 104 (PART 3) (1992) 2011. doi:10.1016/0304-8853(92)91645-A.
  56. [57]M. Khan, J. Brock, I. Sugerman, Anomalous transport properties of Ni₂Mn₁−xCrₓGa Heusler alloys at the martensite-austenite phase transition, Phys. Rev. B 93 (5) (2016) 054419. doi:10.1103/PhysRevB.93.054419.DOI
  57. [58]J. Janovec, M. Zelený, O. Hečko, A. Ayuela, Localization versus delocalization of d-states within the Ni₂MnGa Heusler alloy, Sci. Rep. 12 (1) (2022) 20577. doi:10.1038/s41598-022-23575-1.
  58. [59]L. Sponza, P. Pisanti, A. Vishina, D. Pashov, C. Weber, M. van Schilfgaarde, S. Acharya, J. Vidal, G. Kotliar, Self-energies in itinerant magnets: A focus on Fe and Ni, Phys. Rev. B 95 (2017) 041112(R). doi:10.1103/PhysRevB.95.041112.DOI
  59. [60]K. Momma, F. Izumi, VESTA 3 for three-dimensional visualization of crystal, volumetric and morphology data, J. Appl. Crystallogr. 44 (6) (2011) 1272. doi:10.1107/S0021889811038970.DOI

Paper details

vlnc:W_nxhdntnj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents