betaFeedback

WorksPhysics and Astronomy

Adaptive Finite Differencing in High Accuracy Electronic Structure Calculations

Abstract

A new method for implementing the kinetic energy operator for real-space, grid-based electronic structure codes is developed. It is based on multi-order Adaptive Finite Differencing (AFD) and uses atomic pseudo orbitals produced by the corresponding pseudopotential codes to optimize the standard finite difference (SFD) operators for improved accuracy. Results are presented for a variety of test systems and Bravais lattice types, including the well-known Delta test for 71 elements in the periodic table, the Mott insulator NiO, and borax decahydrate, which contains covalent, ionic, and hydrogen bonds. The tests show that an 8th-order AFD operator leads to the same average Delta value as that achieved by plane-wave codes and is typically far more accurate and a much lower computational cost than a 12th-order SFD operator. The scalability of real-space electronic calculations is demonstrated for a 2,016-atom NiO cell, for which the computational time decreases nearly linearly when scaled from 18 to 144 CPU-GPU nodes.

INTRODUCTION

Density functional theory (DFT) 1–3 has enabled accurate, ab initio predictions of materials properties and explanations of a wide range of experimental data. Tens of thousands of DFT calculations are performed each year. The reliability of DFT predictions has led to materials-genome-type projects, 4–7 in which a large set of possible material compositions and structures are screened by DFT calculations in order to identify those with promising properties. Those with the most potential are then suggested or selected for experimental synthesis and evaluation.

Reliability of predictions is critical to any evaluation of potential predictions, and the accuracy of DFT calculations is thus a factor when choosing a suitable computational method. However, full-precision DFT calculations can be computationally expensive, especially if complex structures or multi-component systems are involved, and the choice of approach and code often involves a tradeoff between accuracy and practicality. Fortunately, the methodology has advanced to the point where the results of several widely-used DFT codes agree well with each other and a set of benchmark high-accuracy calculations for a large set of elements across the periodic table.8 However, the tradeoffs remain for more complex systems or large survey studies, requiring practical compromises between the computational expense, accuracy, and even the choice of the computational method. This paper describes a significant advance in the class of methods that use real-space grids to solve DFT equations. By using adaptive finite differencing to discretize the kinetic energy operator, the real-space results agree with those of plane-wave-based codes using much lower grid densities than those previously required and reproducing the benchmark DFT results8 at the same accuracy level as those of plane-wave-based and all-electron codes. The improved discretization enables high-accuracy real-space calculations at a substantially reduced cost while leveraging the well-known advantages of realspace methods of easy parallelization across many nodes, including multi-CPU and multi-GPU configurations, and avoiding the use of Fast Fourier Transform algorithms, which require global communication across nodes.

Briefly, in density functional theory, the ground state electronic structure of many-electron systems is solved numerically, typically after introducing the Kohn-Sham equation,2 which is derived variationally from the total energy functional after approximating the kinetic energy by that of non-interacting electrons. The electrons are described as single particles in the presence of an external potential stemming from nuclei and externally applied fields and an effective potential, introduced by interactions with other electrons and the effects of the Pauli exclusion principle and spin:

[−∇2+Vext(r)+Veff(r)]ψi(r)=εiψi(r)i=1,…,N(1)\left[ -\nabla^2 + V_{ext}(r) + V_{eff}(r) \right]\psi_i(r) = \varepsilon_i\psi_i(r) \quad i=1,\ldots,N \tag*{(1)}

for the NN electrons in the system. The effective potential is further split into the classical electron-electron repulsion part and the exchange-correlation part that accounts for all effects left out by the classical non-interacting electrons picture. While it has been proven that the exact exchange-correlation potential is only a functional of the total electron density,1 its functional form is unknown. Nevertheless, excellent approximations exist, starting with the local density approximation, which pointwise approximates this potential with that of the uniform-density electron gas through a “Jacobs ladder” of functionals with increasing complexity and accuracy9 Non-local approximations to exchange-correlation functionals have also been devised, including various forms of hybrid functionals.[12] [10, 11]

Depending on the treatment of the external potential from the nuclei, the methods to solve the Kohn-Sham equation can be categorized into two groups, the first being the all-electron method that includes both core and valence electrons in the calculations.[13, 14, 15] The other group replaces the nuclei and inert core electrons with atom-derived pseudopotentials[16, 17, 18, 19] or effective core potentials[20, 21, 22] that are constructed to accurately describe the interactions of valence electrons with the ionic cores. This dramatically reduces the number of electrons that must be included in calculations. Furthermore, because the valence electrons experience a softer pseudopotential instead of the real potential, the wave functions are smoother, and the basis set required to describe them is much smaller. Another advantage is the implicit inclusion of relativistic effects in the pseudopotential so that relativistic effects do not need to be explicitly addressed in the numerical solution of the Kohn-Sham equations.[23] For these reasons, most large DFT calculations use pseudopotentials. When pseudopotentials are carefully constructed from atomic all-electron calculations, they are transferable to different atomic environments and provide results that agree well with those of all-electron calculations.[24] [8]

The norm-conserving pseudopotentials[16, 19] use angular momentum projectors to force pseudo wavefunctions to agree exactly with atomic all-electron wavefunctions beyond a matching radius, typically chosen slightly beyond the core radius. A Kleinman-Bylander form of pseudopotentials[25] converts the angular projectors to limited-range 3D projectors, significantly simplifying their applications in calculations that use plane-wave basis sets. Ultrasoft pseudopotentials[17] relax the norm-conservation criteria and further reduce the basis set size but introduce sharp augmented charge densities near the atomic cores that require careful numerical treatment. Recently, Hamann[19] incorporated the main aspects of ultrasoft pseudopotentials into a norm-conserving form and generated pseudopotentials with multiple projectors for each angular momentum. As a result, these pseudopotentials are nearly as soft as the ultrasoft ones while avoiding the complexity of the augmented charge density.

In the last few decades, considerable work has been devoted to developing methods for solving the Kohn-Sham equations using pseudopotentials. The wave functions are usually expanded in a basis set, and the resulting secular equations are solved using matrix methods, which provide wave function information used to analyze physical phenomena. Many physics-, chemistry- or mathematics-motivated bases exist, including plane waves[26, 27, 28, 29, 30, 31, 32], atomic or atomic-like orbitals[33, 34, 35, 36], Gaussians,[37, 38, 39, 40, 41, 42, 43, 44] real-space grids,[45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60] wavelets,[61, 62, 63, 64] and finite elements.[65, 66, 67]

The most widely used approaches use plane waves as a basis, taking advantage of the fact that the kinetic energy operator is diagonal in Fourier space, while the potential is diagonal in real space (except for the non-local projectors and non-local exchange). The transformations between the two spaces use Fast Fourier Transforms (FFTs), which have the well-known Mlog⁡MM\log M scaling with the number of plane waves or grid points. Another advantage of plane-wave calculations is using a single, easy-to-understand parameter determining the basis set size and thus the accuracy: the highest kinetic energy of plane waves included in the calculations – the plane-wave cutoff.

Since real space is the dual to Fourier space, methods using uniformly-spaced grids[45, 46, 47, 48, 49, 52, 53, 54, 55, 56, 57, 58, 59, 60] should offer similar advantages to plane waves and avoid extensive use of FFTs, the price being an inexact treatment of the kinetic energy operator. As shown in the sections below, adaptive discretization of this operator leads to dramatically improved accuracy, facilitating grid-based calculations with modest grid sizes and enabling highly accurate large-scale calculations on modern parallel computers.

The rest of the paper is organized as follows. We first present an analysis of the finite difference operator in the context of electronic structure calculations, followed by a procedure to generate standard finite difference operators for non-orthogonal lattices. Next, we introduce an adaptive finite difference operator optimized for electronic structure calculations and present the results of the Δ\Delta test, a well-known test8 evaluating the accuracy of electronic structure codes, comparing grid-based results to highly accurate all-electron benchmarks for 71 elements. The following subsections examine energy convergence with grid spacing, and show an example of a highly accurate, large-scale DFT calculation obtained using the adaptive operator, parallelized over the hundreds of CPU-GPU nodes of an exascale supercomputer. All the grid-based calculations use the Real-space MultiGrid (RMG) code.[47, 48, 68]

RESULTS

Analysis of the finite difference operator

Real-space, grid-based methods for solving the Kohn-Sham equations typically use finite differences (FD) to implement the kinetic energy operator −12∇2-\frac{1}{2}\nabla^2 appearing in the Kohn-Sham equation. The space locality inherent to FD operators is computationally efficient and suitable for parallelization via domain decomposition, which partitions the computational volume into subvolumes assigned to different nodes. In plane-wave methods, the Laplacian operator is trivial, while the charge density calculations and applications of the potential need transferring of all the wave functions from the plane wave representation to real space using Fast Fourier Transforms (FFTs), which require communications across the entire computational domain. However, the plane-wave treatment of the Laplacian is exact within the basis set defined by the G-space cutoff, while the FD approximation introduces errors. Minimizing the FD error is thus necessary to exploit the computational advantages possible in real space. The various approaches have included non-uniform grids,[45] higher-order central finite difference operators,[46, 69] the Mehrstellen discretization in our previous version of the RMG package,[47, 48] and adaptive coordinates and grids.[50, 70]

We begin the analysis by considering the standard central finite difference approximation for the second derivative of a function f(x)f(x) on a uniform grid of spacing hh, using 2n+12n + 1 grid points, as given by Eq. (2).

The coefficients aia_i are chosen to make the expression exact for polynomials of degree less than or equal to 2n2n

fn′′(x0)=a0f(x0)+∑i=1nai(f(x0+ih)+f(x0−ih))+Rn(h2n),(2)f_n^{\prime\prime}(x_0) = a_0 f(x_0) + \sum_{i=1}^{n} a_i \left(f(x_0 + ih) + f(x_0 - ih)\right) + R_n(h^{2n}), \tag*{(2)}

where the truncation error is

Rn=bnh2nf2n+2(x0)+O(h2n+2)(3)R_n = b_n h^{2n} f^{2n+2}(x_0) + O(h^{2n+2}) \tag*{(3)}

and the bnb_n arise from the Taylor series expansions used to derive Eq. (2), with b1=112b_1 = \frac{1}{12}, b2=124b_2 = \frac{1}{24} etc. There are alternate representations of the Laplacian where the coefficients are optimized for a different set of properties. For example, controlling dispersion is required for stable finite difference solutions to the wave equation, and there is a large body of work on this specific topic[71, 72, 73]. The goal is to find coefficients for use in Eq. (2) that are optimized for electronic structure calculations. Let f(x)f(x) be represented by a plane wave expansion of the form below

f(x)=∑j=1,NCjexp⁡(iGjx).(4)f(x) = \sum_{j=1,N} C_j \exp(iG_jx). \tag*{(4)}

The first term of the truncation error in Fourier space can be written as

Rn=bnh2n∑j=1,N(iGj)2n+2Cjexp⁡(iGjx0).(5)R_n = b_n h^{2n} \sum_{j=1,N} (iG_j)^{2n+2} C_j \exp(iG_jx_0). \tag*{(5)}

For each frequency, the truncation errors of different orders are proportional to the CjC_j's, which are determined by the Fourier transform of the original function. Therefore, a linear combination of finite difference operators of different orders can be constructed to reduce the truncation error.

The new adaptive operator can be written as

f{new}′′(x0)=(1+M)fn′′(x0)−Mfn−1′′(x0).(6)f''_{\{new\}}(x_0) = (1+M)f''_n(x_0)-Mf''_{n-1}(x_0). \tag*{(6)}

The new composite operator is guaranteed to be exact for polynomials up to degree 2(n−1)2(n-1), while a further variation of MM may be used to minimize the truncation error for a target class of functions. This method of optimizing finite difference operators by mixing stencils of a different order is not necessarily appropriate for arbitrary functions but has been used successfully for specific purposes.[73] In our RMG implementation and tests, we found that MM is always positive. This is supported by the error analysis below.

Analyzing the error in more detail is useful when the adaptive operator is applied to plane waves. For a given reciprocal vector G\mathrm{G}, the truncation errors for a standard nn-th order operator can be expressed as a power series

Rns(G)=1h2∑m=1,2…bn,m(ihG)2n+2m(7)R^s_n(G) = \frac{1}{h^2}\sum_{m=1,2\ldots} b_{n,m}(ihG)^{2n+2m} \tag*{(7)}

Here we ignore the phase term eiGx0e^{iGx_0}. For a periodic system, the values of hGhG are always in the range of [−π,π][-\pi,\pi]. The bn,mb_{n,m} decrease with increasing mm, and the signs of the errors alternate for m = even or odd, but the errors for hG≫1hG \gg1 are still large compared to the small hGhG range.

The truncation error for the adaptive nn-th order operator is

Rna(G,M)=(1+M)Rns(G)−MRn−1s(G)(8)R^a_n(G,M) = (1+M)R^s_n(G)-MR^s_{n-1}(G) \tag*{(8)}

For a given nn, the constants bn,mb_{n,m} may be evaluated using a symbolic math package (e.g., SymPy[74]). For the 8th order adaptive operator (n=4n=4) we obtain the truncation error

R4a(G,M)=1h2[−M560(hG)8+[13156+M1680](hG)10−[113860+19M201600](hG)12+⋯ ](9)R^a_4(G,M) = \frac{1}{h^2}\left[-\frac{M}{560}(hG)^8+\left[\frac{1}{3156}+\frac{M}{1680}\right](hG)^{10}-\left[\frac{1}{13860}+\frac{19M}{201600}\right](hG)^{12}+\cdots\right] \tag*{(9)}

Figure 1 is a graph of the truncation error vs. GG from Eq. (9) for M=0.33M=0.33 as well as graphs of the errors for the 6th6^{\mathrm{th}} and 8th8^{\mathrm{th}} order standard operators used to construct the adaptive. It’s clear from the graph and Eq. (7) that the error increases rapidly as hG→1hG \to1. Therefore, a necessary condition for using finite differences in electronic structure calculations is for the wavefunction components to be small when ∣hG∣>1\lvert hG\rvert> 1 and approach zero as ∣hG∣\lvert hG\rvert is close to its maximum π\pi. Figure 1(a) shows that for G>X≅0.29G > X \cong0.29 the adaptive operator is more accurate than the standard 8th8^{\mathrm{th}} while the zoomed view in Figure 1(b) shows that for G<XG < X the standard 8th8^{\mathrm{th}} order is slightly more accurate. The absolute magnitude of the errors is very different in the two regions, with the adaptive operator having a significantly flatter error curve over a broader range than the standard operators. The total truncation error depends on both Rna(G,M)R_n^a(G,M) and the profile of ψ(G)\psi(G), with the adaptive process optimizing MM to minimize the total error. However, the calculations presented later show that even a fixed value of M=0.33M=0.33 leads to more accurate results for a wide range of atomic species and grid spacings. While the reason for this is not immediately obvious, it can be understood by focusing on the differential absolute errors in the regions to the left and right of G=XG=X in Figure 1. The absolute errors for plane waves with G<XG<X are small for both adaptive and standard operators. In a high-cutoff calculation, where the wavefunctions have little weight for G>XG>X, the overall error and the differences between the operators are small. For a less converged calculation, where the wavefunctions have significant components for G>XG>X, the adaptive operator has a significantly lower total error because the errors for the standard operators increase rapidly in this region. As noted earlier, the 8th8^{\mathrm{th}}-order adaptive operator with M=0.33M=0.33 performs well over a wide range of atomic species and grid spacings, but as the grid density becomes very high, there is a point where the standard operators are more accurate because the wavefunctions have vanishing components for G>XG>X. The fully adaptive operator overcomes this issue by optimizing MM for specific calculations, decreasing MM as the convergence level increases.

Absolute value of the finite difference truncation error as a function of G

Figure 1. Absolute value of the finite difference truncation error as a function of GG, up to the Nyquist frequency 12Gmax⁡\frac{1}{2}G_{\max}. The 8th8^{\mathrm{th}}-order AFD operator uses a mixing value of M=0.33M=0.33.

Finite difference operator by Taylor Series expansion

For cubic or orthorhombic lattice systems, the lattice vectors are orthogonal to each other, and the FD operators can be obtained separately for each direction. For a general lattice type, where the lattice vectors are not orthogonal, such as monoclinic or triclinic systems, the following process is used to generate the initial coefficients for a specific order. Then, the procedure discussed in the next subsection is used to optimize the FD operator further to reach high accuracy without increasing the computational cost.

First, a few special axes in the real-space grid are chosen appropriately for the lattice type; axes that coincide with the lattice vectors are always included. 2n2n grid points along each axis are used for the following minimization. For each point, the function f(x,y,z)f(x,y,z) can be expressed as the Taylor series expansion

f(xi,yj,zk)=f(x0,y0,z0)+∑lmpf(lmp)l!m!p!hxlhymhzp+δ(ijk)(10)f(x_i,y_j,z_k)=f(x_0,y_0,z_0)+\sum_{lmp}\frac{f^{(lmp)}}{l!m!p!}h_x^l h_y^m h_z^p+\delta(ijk) \tag*{(10)}

Where flmnf^{lmn} are the derivatives to be determined and l+m+p<2n+1l+m+p<2n+1 for the 2n2n-order FD operator. We define an error norm RR by

R=∑ijk∣δ(ijk)∣2hijk3(11)R=\sum_{ijk}\frac{|\delta(ijk)|^2}{h_{ijk}^3} \tag*{(11)}

where δ(ijk)\delta(ijk) is defined in equation (10) and hijkh_{ijk} is the distance from the point (xi,yj,zk)(x_i,y_j,z_k) to the point (x0,y0,z0)(x_0,y_0,z_0). The summation is over the set {ijk}\{ijk\} of points included in the FD operator. For monoclinic or triclinic systems, the sampling grid points are on axes along the lattice vectors and also additional axes. The FD coefficients can be obtained by minimizing the norm in Eq. 11, and we have

∂R∂f(lmp)=0(12)\frac{\partial R}{\partial f^{(lmp)}}=0 \tag*{(12)}

This will generate a set of linear equations and the derivatives f(lmp)f^{(lmp)} can be obtained by solving these linear equations. [75] One can always avoid the singularity of the linear equations by including more grid points and axes.

For orthogonal lattices, the resulting FD kinetic energy operator is separable. For non-orthogonal lattices, cross terms occur, but both the kinetic energy operator and the first derivatives are obtained from the same sampling points with predetermined coefficients. Recently, a Kronecker product formulation of the kinetic energy operator for non-orthogonal lattices has been developed. [76] All of the derivatives are applied separately on the lattice vectors, but the cross terms ∂2/∂ζ∂η\partial^2/\partial\zeta\partial\eta need to be applied sequentially on each lattice vector. We have not explored this formulation.

Adaptive finite difference operator

A generalization from the one-dimensional mixed operator case discussed above to three dimensions is trivial, i.e., the truncation error in one lattice frequency is always proportional to the component of the target function at that frequency. This is true for all lattice types in solid-state physics, and the operators along the different axes are determined by the method described above.

For electronic structure applications in which the Kohn-Sham equation is solved numerically, the target functions are the Kohn-Sham wave functions. To a first-order approximation, these may be written as a linear combination of the pseudo atomic orbitals ψatomic\psi_{atomic}. In a non-atomic environment, the ψatomic\psi_{atomic} should still provide a good representation of the Kohn-Sham wave functions in the regions near the ionic cores, as is implicitly assumed in the linear combination of atomic orbitals (LCAO) methods for calculating the electronic structure of molecules and solids.[77] In the interstitial regions, their deviation from the pseudo atomic orbitals may be larger, but as they are much less rapidly varying than in the core regions, an operator of the above form, which is exact for polynomials of degree 2(n−1)2(n-1), is highly accurate for a sufficiently large nn. For these reasons, the pseudo atomic orbitals were chosen as the optimization functions for the finite difference operators. In analogy with pseudopotentials, operators constructed using this procedure are highly accurate for isolated atoms and should also be transferrable to solids or molecules.

The optimization procedure requires a reference standard. For a function discretized on a uniform real space grid, the most accurate Laplacian operator is the FFT method. Unfortunately, this would require transforming all the real-space wave functions to the reciprocal space via FFTs. For very large calculations in which the wave functions are distributed across many computing nodes, the FFTs are too slow because they require global communications.

Nevertheless, the FFT Laplacian is well-suited as a reference standard because it only needs to be computed once for each atomic orbital. The overall computational cost is negligible. We choose the objective function to be minimized as the norm of the difference between the kinetic energies of the real-space atomic orbitals computed via FFT versus their values computed by finite differences. Weighting factors are applied based on the atomic orbital occupations and the number of atoms of each kind in the anticipated large calculation. The final formula is

F(M)=∑i=1,Nwi∣⟨ψi(r)∣LFD(M)−LFFT∣ψi⟩∣2,(13)F(M) = \sum_{i=1,N} w_i \left| \langle\psi_i(r) | L_{FD}(M) - L_{FFT} | \psi_i \rangle\right|^2, \tag*{(13)}

where ψi\psi_i is one of the distinct atomic orbitals, and wiw_i is the weight factor equal to the total occupancy for this orbital. LFDL_{FD} is the finite differential Laplacian operator and LFFTL_{FFT} is the FFT Laplacian operator. The minimization process for F(M)F(M) can be performed by using a least-squares fit of the resulting data to determine the minima. If the deviation is too large for any orbital than a given tolerance, usually 10−410^{-4}, it indicates that the grid spacing needs to be decreased. Alternatively, one can automatically determine the needed grid density by requiring that all deviations are below the tolerance.

Figure 2(a) and Figure 2(b) show the Ni 3dxz3d_{xz} orbital in real and GG spaces, respectively. Although the truncation errors for the high-frequency plane waves are large, the orbitals have very small components in this range, which can be clearly seen in Figure 2(b). Figure 2(c) and Figure 2(d) show the truncation errors in real and GG spaces for this orbital with different finite-difference operators. As was noted before, the truncation error at a specific GG vector is proportional to the orbital’s component at this frequency. This can be clearly seen in Figure 2(d), where the errors have the same trends for both 6{sup}th- and 8{sup}th-order standard FD operators, independent of the grid spacings or the effective cutoff energies. In real space, the errors also have the same trends for both 6{sup}th- and 8{sup}th-order SFDs. Therefore, the errors are dramatically reduced with the adaptive 8{sup}th-order FD operator. In fact, it has better accuracy than the standard 12{sup}th-order FD operator (see below). The optimization uses the atomic orbitals as the target functions, and the resulting AFD is transferable to the wavefunctions in the real applications. Figure 2(e) and Figure 2(f) show the finite-difference errors (compared to the FFT results) for the highest occupied molecular orbital (HOMO) and the lowest unoccupied molecular orbital (LUMO) of NiO in a 64-atom cell. In the interstitial regions, the wavefunctions are smooth, and the errors from both SFD and AFD are small. In the regions near atoms, the errors from AFD are significantly smaller than those from SFD. This indicates that AFD is more accurate than SFD not only for the atomic orbitals but also for the extended-state wave functions. The more accurate total energy in the next section also proves that. Since a high-order FD operator requires more computational resources than the lower-order one, not only because of the size of the FD tensor but also the communication cost of reaching adjacent domains when domain decomposition is used in massively parallel calculations, the adaptive finite difference (AFD) operator leads to more accurate results at a significantly lower computational cost. We discuss the computational advantages of AFD below.

Orbital and discretization errors in real space and reciprocal G space

Figure 2. Orbital and discretization errors in real space and reciprocal GG space. Ni 3dxz3d_{xz} orbital in (a) real space and (b) GG space. Discretization errors of Ni 3dxz3d_{xz} orbital in (c) real space and (d) GG space. Finite difference errors for (e) HOMO and (f) LUMO wave functions of NiO along the [111][111] direction. Ni-O distance is 3.61 Å.

Accuracy across the periodic table

The DFT Δ\Delta test[78] [8] was selected to test the new operators in the RMG code[79] across a wide range of atomic species and lattice types. The Δ\Delta test evaluates the average variance between the equations of state calculated by different codes and has become a standard way of comparing the accuracy and reproducibility of DFT codes using different all-electron or pseudopotential methods, basis sets, and algorithms for 71 different atomic species. Published results listing the average Δ\Delta are available[80] for a large selection of DFT codes, using highly-converged all-electron results obtained by the WIEN2k code[81] as the reference standard. For our tests, we used a mixture of ONCV, GBRV, and pslibrary pseudopotentials which tracked closely with the SSSP accuracy library.1 The correspondence was not exact, as RMG does not support PAW-based pseudopotentials used for some atomic species in the SSSP library. Figure 3 provides an element-by-element comparison of the individual Δ\Delta values for the 8th8^{\mathrm{th}}-order adaptive and the 12th12^{\mathrm{th}}-order standard finite difference operators, while Table 1 lists the average Δ\Delta values for the 8th8^{\mathrm{th}}, 10th10^{\mathrm{th}} and 12th12^{\mathrm{th}}-order standard operators as well as the 8th8^{\mathrm{th}}-order adaptive and 8th8^{\mathrm{th}}-, 10th10^{\mathrm{th}}- and 12th12^{\mathrm{th}}-order operators with fixed mixing. The latter are generated using the same mixing value for all species, making them easy to implement. They produce lower delta values than the standard operators of the same order, although not as low as the fully adaptive 8th8^{\mathrm{th}}.

Comparison of $\Delta$ values for individual atomic species

Figure 3. Δ\Delta value comparison [meV] for individual atomic species calculated by RMG version 5.3 for the 8{sup}th order adaptive and 12{sup}th order standard operators.

Finite difference methodΔ\Delta value (meV)
8th8^{\mathrm{th}} order standard3.753
10th10^{\mathrm{th}} order standard1.540
12th12^{\mathrm{th}} order standard0.802
8th8^{\mathrm{th}} order fixed (M=0.33)1.241
10th10^{\mathrm{th}} order fixed (M=0.416)0.553
12th12^{\mathrm{th}} order fixed (M=0.5)0.486
8th8^{\mathrm{th}} order adaptive0.298

Table 1. Average delta values for various finite difference methods.

Energy convergence with the grid spacing

In this subsection, we examine the convergence of the total energy versus grid resolution, represented by the corresponding plane-wave kinetic energy cutoff, of various forms of finite-differencing of the kinetic energy, compared to a reference value obtained by a plane-wave code that treats the kinetic energy exactly at that cutoff. For a given grid, we compute the kinetic energy cutoff as the radius squared of the inscribed sphere in the corresponding reciprocal space, which is consistent with the procedures used in plane wave codes. While the Δ\Delta test consists of calculations for elemental solids, we use supercells with more than one type of atom here, employing norm-conserving pseudopotentials (NCPP).[16, 19] While the AFD method is equally applicable to ultrasoft pseudopotentials (USPP),[17] the wavefunctions from NCPP are typically more rapidly varying than those from USPP and therefore provide a more rigorous test of the quality of the operator. The reported results were generated using the SG15 potentials developed by Schlipf and Gygi 83 using Hamann’s ONCV code.[19] [84] The first system is the Mott insulator NiO, represented by a 64-atom supercell of NiO in an antiferromagnetic configuration, with Ni 3s23p63s^{2}3p^{6} semi-core states included in valence. Figure 4(a) shows the difference in total energy ΔE\Delta E between a reference value and RMG v.5.3 using various forms of the kinetic energy operator. The reference values were generated using the open-source plane-wave code Quantum Espresso (QE)[30] [85] that has an exact representation of the operator within its basis set. While all of the finite-difference-based operators achieve good agreement with QE at high energy cutoffs, the adaptive operators are considerably more accurate at lower cutoffs. Indeed, the 8th8^{\mathrm{th}}-order AFD operator is more accurate than the 12th12^{\mathrm{th}}-order standard operator up to a 205 Rydberg cutoff, with the difference at 124 Rydbergs being a factor of 11.

Differences in total energies and time per SCF step for NiO and borax

Figure 4. Differences in total energies between the reference energy obtained using Quantum Espresso and RMG for (a) a 64-atom NiO cell in an antiferromagnetic configuration, and (b) an 86-atom borax decahydrate Na2[B4O5(OH)4]⋅8H2O\mathrm{Na}_{2}[\mathrm{B}_{4}\mathrm{O}_{5}(\mathrm{OH})_{4}]\cdot8\mathrm{H}_{2}\mathrm{O} cell, as functions of plane-wave kinetic energy cutoff; (c) time per SCF step for grid densities required to reach prescribed accuracies in comparison to the reference energy.

The second test case is borax decahydrate, Na2[B4O5(OH)4]⋅8H2O\mathrm{Na}_{2}[\mathrm{B}_{4}\mathrm{O}_{5}(\mathrm{OH})_{4}]\cdot8\mathrm{H}_{2}\mathrm{O}, which has a complex monoclinic structure with 86 atoms in the primitive cell and a mixture of covalent, ionic, and hydrogen bonds. 86 As shown in Figure 4(b), the adaptive finite differencing is more accurate than the standard 10th10^{\mathrm{th}}- and 12th12^{\mathrm{th}}-order finite-difference formulas at all tested cutoffs, with the difference exceeding a factor of 200 at 117 Rydbergs, and converges more quickly to the reference value.

Figure 4(c) shows the time per SCF step at a prescribed accuracy for borax decahydrate, compared to the reference Quantum Espresso plane wave calculation. For the same accuracy, the standard finite difference operators require significantly denser grids than the adaptive operator, and thus their times per SCF step are substantially greater.

Large-scale DFT calculations using the adaptive operator

The adaptive finite-difference operator enables high-accuracy calculations for large systems by taking advantage of the enhanced parallelization enabled by domain decomposition when using

Scaling of the 2,016-atom NiO calculation with the number of Crusher (Frontier) nodes

Figure 5. Scaling of the 2,016-atom NiO calculation with the number of Crusher (Frontier) nodes.

real-space grids. As an example, we consider a 2,016-atom NiO cell in an antiferromagnetic configuration, also with Ni 3s23p63s^{2}3p^{6} semi-core states included in valence. Based on the convergence versus grid spacing results in the left panel of Figure 2, a grid spacing corresponding to an energy cutoff of 167 Ry was selected. The calculation was spin-polarized and used 14,128 Kohn-Sham wavefunctions for each spin channel. The results were obtained using the Frontier development platform Crusher at ORNL. Crusher nodes have the same hardware as the exascale Frontier, but there are only 192 nodes versus Frontier’s 9,472. Each node contains a 64-core AMD Epyc 7453 CPU and four AMD MI250X GPUs. The RMG calculations used all CPU cores and all GPUs in each node. The scaling results for 18 to 144 nodes are presented in Figure 4. A minimum of 18 nodes is required due to the memory footprint of the calculations.

The decrease in the computational time is linear from 18 to 72 nodes, but as the number of nodes increases further, the speedup rate decreases because the computational work per node becomes too small to saturate the nodes fully. It should be noted that even on 18 nodes, the time per SCF step, 230 seconds, makes high-accuracy calculations of this size practical even when the problem requires extensive geometry optimizations, transition path studies, or ab initio molecular dynamics.

DISCUSSION

The promise of real-space electronic structure calculations has always centered on their easy parallelizability on massively parallel computer architectures through domain decomposition, because real-space operations are inherently local and thus amenable to parallel execution without extensive communication. However, for grid-based calculations, it is well-established that high-order discretization of the kinetic energy operator is required, which makes the calculations more costly and less local by increasing the communications beyond the local domain. In contrast, plane-wave calculations represent the kinetic energy operator exactly for a given cutoff but require global fast Fourier transform operations. The adaptive finite-differencing scheme developed in this paper relies on optimizing the kinetic energy expression near atomic cores for each atomic specie separately, resulting in finite-difference coefficients that are transferable to different atomic environments in analogy with pseudopotentials, which both grid- and plane-wave-based methods use. The resulting adaptive kinetic energy operator is far more accurate than standard high-order discretizations and lower-order, decreasing the costs of both calculations and communications. We have tested the accuracy of this operator in electronic structure calculations using the well-known Δ\Delta test for 71 elemental solids, and the average error of the calculations was the same as those of the well-established plane-wave codes VASP and Quantum Espresso. The accuracy of the adaptive operator was further established with multi-specie tests on NiO and borax decahydrate, which exhibit a range of complex bonding arrangements. The scalability of real-space grid methodology was then confirmed in highly-accurate calculations for a 2,016-atom NiO supercell, where the computational time decreased nearly linearly when scaling from 18 to 144 CPU-GPU nodes.

METHODS

The adaptive finite difference operator for the kinetic energy was implemented in version 5 of the open-source RMG code for all crystallographic space groups. The calculations for the Mott insulator NiO used the DFT+U formalism87 with PBE exchange-correlation functional88 and U = 6.5 eV. Due to supercell sizes, only Γ\Gamma point sampling was used. The calculations for borax decahydrate used the PBE functional and were also at Γ\Gamma. The Δ\Delta test calculations used structures provided at https://molmod.ugent.be/sites/default/files/Delta_v3-1_0.zip with varying k-point meshes depending on the specific element. All input and output data are available from the depository listed below.

DATA AVAILABILITY

All input and output files used to report the results and create the graphics are available at https://github.com/RMGDFT/AFD_5.3.

CODE AVAILABILITY

The open-source RMG code is distributed through github.com/RMGDFT.

Appendix 1. Grid spacings for Delta tests

Elementh (bohr)Elementh (bohr)Elementh (bohr)Elementh (bohr)
Ag0.248Fe0.212Na0.209Sc0.252
Al0.240F0.236Nb0.208Se0.289
Ar0.221Ga0.246Ne0.229Si0.271
As0.281Ge0.245Ni0.210Sn0.248
Au0.207He0.211N0.244Sr0.358
Ba0.209Hf0.229Os0.203S0.305
Be0.259Hg0.241O0.251Ta0.208
Bi0.248H0.255Pb0.250Tc0.249
Br0.243In0.307Pd0.235Te0.259
B0.262Ir0.192Po0.264Ti0.264
Ca0.274I0.347Pt0.197Tl0.264
Cd0.260Kr0.213P0.260V0.225
Cl0.280K0.220Rb0.213W0.299
Co0.274Li0.228Re0.213Xe0.233
Cr0.226Lu0.169Rh0.229Y0.268
Cs0.210Mg0.289Rn0.214Zn0.235
Cu0.180Mn0.243Ru0.208Zr0.248
C0.280Mo0.238Sb0.236

Table A1. Average grid spacings hh for different elements in the Delta tests. For a general lattice type, hh is defined as h=V/(nx∗ny∗nz)3h = \sqrt[3]{V/(n_x * n_y * n_z)}, where V is the volume of the unit cell at equilibrium, nxn_x, nyn_y, and nzn_z are the numbers of grid points along the three lattice vectors.

References

Acknowledgments

This research was supported by the U.S. Department of Energy’s Exascale Computing Project (ECP), Project Number: 17-SC-20-SC. Computing resources at the Oak Ridge Leadership Computing Facility were provided through the ECP User Program and the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program.

Author Contributions

E.B. and W.L. devised the adaptive operator and performed the calculations. All authors analyzed the results and wrote the manuscript.

Competing Interests

The authors declare no competing interests.

References

  1. [12]Chai, J.-D. & Head-Gordon, M. Systematic optimization of long-range corrected hybrid density functionals. J. Chem. Phys. 128, 084106 (2008).DOI
  2. [13]Wimmer, E., Krakauer, H., Weinert, M. & Freeman, A. J. Full-potential self-consistent linearized-augmented-plane-wave method for calculating the electronic structure of molecules and surfaces: O2 molecule. Phys. Rev. B 24, 864–875 (1981).DOI
  3. [14]Blaha, P., Schwarz, K., Sorantin, P. & Trickey, S. B. Full-potential, linearized augmented plane wave programs for crystalline systems. Comput. Phys. Commun. 59, 399–415 (1990).
  4. [15]Methfessel, M., Rodriguez, C. O. & Andersen, O. K. Fast full-potential calculations with a converged basis of atom-centered linear muffin-tin orbitals: Structural and dynamic properties of silicon. Phys. Rev. B 40, 2009–2012 (1989).pubmed.ncbi.nlm.nih.gov/9992074
  5. [16]Hamann, D. R., Schlueter, M. & Chiang, C. Norm-Conserving Pseudopotentials. Phys. Rev. Lett. 43, 1494 (1979).DOI
  6. [17]Vanderbilt, D. Soft self-consistent pseudopotentials in a generalized eigenvalue formalism. Phys. Rev. B 41, 7892 (1990).pubmed.ncbi.nlm.nih.gov/9993096
  7. [18]Blöchl, P. E. Projector augmented-wave method. Phys. Rev. B 50, 17953–17979 (1994).pubmed.ncbi.nlm.nih.gov/9976227
  8. [19]Hamann, D. R. Optimized norm-conserving Vanderbilt pseudopotentials. Phys. Rev. B 88, 085117 (2013).DOI
  9. [20]Kahn, L. R., Baybutt, P. & Truhlar, D. G. Ab initio effective core potentials: Reduction of all-electron molecular structure calculations to calculations involving only valence electrons. J. Chem. Phys. 65, 3826–3853 (1976).DOI
  10. [21]Hay, P. J., Wadt, W. R. & Kahn, L. R. Ab initio effective core potentials for molecular calculations. II. All-electron comparisons and modifications of the procedure. J. Chem. Phys. 68, 3059–3066 (1978).DOI
  11. [22]Hay, P. J. & Wadt, W. R. Ab initio effective core potentials for molecular calculations. Potentials for the transition metal atoms Sc to Hg. J. Chem. Phys. 82, 270–283 (1985).DOI
  12. [23]Kleinman, L. Relativistic norm-conserving pseudopotential. Phys. Rev. B 21, 2630 (1980).DOI
  13. [24]Borlido, P., Doumont, J., Tran, F., Marques, M. A. L. & Botti, S. Validation of Pseudopotential Calculations for the Electronic Band Gap of Solids. J. Chem. Theory Comput. 16, 3620–3627 (2020).arxiv.org/abs/2003.10177
  14. [25]Kleinman, L. & Bylander, D. M. Efficacious Form for Model Pseudopotentials. Phys. Rev. Lett. 48, 1425 (1982).DOI
  15. [26]Kresse, G. & Furthmüller, J. Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set. Comput. Mater. Sci. 6, 15–50 (1996).DOI
  16. [27]Kresse, G. & Joubert, D. From ultrasoft pseudopotentials to the projector augmented-wave method. Phys. Rev. B 59, 1758–1775 (1999).DOI
  17. [28]Gonze, X. et al. First-principles computation of material properties: the ABINIT software project. Comput. Mater. Sci. 25, 478–492 (2002).DOI
  18. [29]Clark, S. J. et al. First principles methods using CASTEP. Z. Für Krist. - Cryst. Mater. 220, 567–570 (2005).DOI
  19. [30]Giannozzi, P. et al. QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials. J. Phys. Condens. Matter 21, 395502 (2009).DOI
  20. [31]Gygi, F. Architecture of Qbox: A scalable first-principles molecular dynamics code. IBM J. Res. Dev. 52, 137–144 (2008).DOI
  21. [32]Hutter, J., Iannuzzi, M., Schiffmann, F. & VandeVondele, J. cp2k: atomistic simulations of condensed matter systems. WIREs Comput. Mol. Sci. 4, 15–25 (2014).DOI
  22. [33]Soler, J. M. et al. The SIESTA method for ab initio order-N materials simulation. J. Phys. Condens. Matter 14, 2745–2779 (2002).DOI
  23. [34]Lu, W. C. et al. Molecule intrinsic minimal basis sets. I. Exact resolution of ab initio optimized molecular orbitals in terms of deformed atomic minimal-basis orbitals. J. Chem. Phys. 120, 2629–2637 (2004).pubmed.ncbi.nlm.nih.gov/15268406
  24. [35]Blum, V. et al. Ab initio molecular simulations with numeric atom-centered orbitals. Comput. Phys. Commun. 180, 2175–2196 (2009).DOI
  25. [36]Jensen, F. Atomic orbital basis sets. WIREs Comput. Mol. Sci. 3, 273–295 (2013).DOI
  26. [37]Boys, S. F. Electronic Wave Functions. I. A General Method of Calculation for the Stationary States of Any Molecular System. Proc. R. Soc. Lond. Ser. A 200, 542–554 (1950).DOI
  27. [38]Gill, P. M. W. Molecular integrals Over Gaussian Basis Functions. Adv. Quantum Chem. 25, 141–205 (1994).DOI
  28. [39]Pople, J. A. & Hehre, W. J. Computation of Electron Repulsion Integrals Involving Contracted Gaussian Basis Functions. J. Comput. Phys. 27, 161–168 (1978).DOI
  29. [40]Curtiss, L. A., Raghavachari, K., Redfern, P. C. & Pople, J. A. Assessment of Gaussian-2 and density functional theories for the computation of enthalpies of formation. J. Chem. Phys. 106, 1063 (1997).DOI
  30. [41]Gill, P. M. W. & Pople, J. A. The prism algorithm for two-electron integrals. Int. J. Quantum Chem. 40, 753–772 (1991).DOI
  31. [42]Frisch, M. J. et al. Gaussian 03, Revision C.02.
  32. [43]Schmidt, M. W. et al. General atomic and molecular electronic structure system. J. Comput. Chem. 14, 1347–1363 (1993).
  33. [44]Aprà, E. et al. NWChem: Past, present, and future. J. Chem. Phys. 152, 184102 (2020).DOI
  34. [45]Bernholc, J., Yi, J. Y. & Sullivan, D. J. Structural Transitions in Metal-Clusters. Faraday Discuss. 217–228 (1991).DOI
  35. [46]Che likowsky, J. R., Troullier, N. & Saad, Y. Finite-difference-pseudopotential method: Electronic structure calculations without a basis. Phys. Rev. Lett. 72, 1240–1243 (1994).pubmed.ncbi.nlm.nih.gov/10056658
  36. [47]Briggs, E. L., Sullivan, D. J. & Bernholc, J. Large-scale electronic-structure calculations with multigrid acceleration. Phys. Rev. B 52, R5471 (1995).DOI
  37. [48]Briggs, E. L., Sullivan, D. J. & Bernholc, J. Real-space multigrid-based approach to large-scale electronic structure calculations. Phys. Rev. B 54, 14362–14375 (1996).pubmed.ncbi.nlm.nih.gov/9985442
  38. [49]Seitsonen, A. P., Puska, M. J. & Nieminen, R. M. Real-space electronic-structure calculations: Combination of the finite-difference and conjugate-gradient methods. Phys. Rev. B 51, 14057–14061 (1995).pubmed.ncbi.nlm.nih.gov/9978330
  39. [50]Gygi, F. & Galli, G. Real-space adaptive-coordinate electronic-structure calculations. Phys. Rev. B 52, 2229–2232 (1995).pubmed.ncbi.nlm.nih.gov/9981378
  40. [51]Ono, T. & Hirose, K. Timesaving Double-Grid Method for Real-Space Electronic-Structure Calculations. Phys. Rev. Lett. 82, 5016–5019 (1999).DOI
  41. [52]Mortensen, J. J., Hansen, L. B. & Jacobsen, K. W. Real-space grid implementation of the projector augmented wave method. Phys. Rev. B 71, 035109 (2005).DOI
  42. [53]Hodak, M., Wang, S., Lu, W. & Bernholc, J. Implementation of ultrasoft pseudopotentials in large-scale grid-based electronic structure calculations. Phys. Rev. B Condens. Matter Mater. Phys. 76, 085108–8 (2007).
  43. [54]Andrade, X. et al. Real-space grids and the Octopus code as tools for the development of new simulation approaches for electronic systems. Phys. Chem. Chem. Phys. 17, 31371–31396 (2015).DOI
  44. [55]Michaud-Rioux, V., Zhang, L. & Guo, H. RESCU: A real space electronic structure method. J. Comput. Phys. 307, 593–613 (2016).DOI
  45. [56]Ghosh, S. & Suryanarayana, P. SPARC: Accurate and efficient finite-difference formulation and parallel implementation of Density Functional Theory: Isolated clusters. Comput. Phys. Commun. 212, 189–204 (2017).DOI
  46. [57]Ghosh, S. & Suryanarayana, P. SPARC: Accurate and efficient finite-difference formulation and parallel implementation of Density Functional Theory: Extended systems. Comput. Phys. Commun. 216, 109–125 (2017).DOI
  47. [58]Xu, Q. et al. Ab initio electronic structure calculations using a real-space Chebyshev-filtered subspace iteration method. J. Phys. Condens. Matter 31, 455901 (2019).DOI
  48. [59]Noda, M. et al. SALMON: Scalable Ab-initio Light–Matter simulator for Optics and Nanoscience. Comput. Phys. Commun. 235, 356–365 (2019).DOI
  49. [60]Liou, K.-H., Biller, A., Kronik, L. & Chelikowsky, J. R. Space-Filling Curves for Real-Space Electronic Structure Calculations. J. Chem. Theory Comput. 17, 4039–4048 (2021).DOI
  50. [61]Harrison, R. J., Fann, G. I., Yanai, T. & Beylkin, G. Multiresolution quantum chemistry in multiwavelet bases. Lect. Notes Comput. Sci. 103–110 (2003).
  51. [62]Genovese, L. et al. Daubechies wavelets as a basis set for density functional pseudopotential calculations. J. Chem. Phys. 129, 014109–014109 (2008).DOI
  52. [63]Yanai, T., Harrison, R. J. & Handy, N. C. Multiresolution quantum chemistry in multiwavelet bases: time-dependent density functional theory with asymptotically corrected potentials in local density and generalized gradient approximations. Mol. Phys. Int. J. Interface Chem. Phys. 103, 413 (2005).
  53. [64]Beylkin, G., Fann, G., Harrison, R. J., Kurcz, C. & Monzón, L. Multiresolution representation of operators with boundary conditions on simple domains. Appl. Comput. Harmon. Anal. 33, 109–139 (2012).DOI
  54. [65]Pask, J. E., Klein, B. M., Fong, C. Y. & Sterne, P. A. Real-space local polynomial basis for solid-state electronic-structure calculations: A finite-element approach. Phys. Rev. B 59, 12352–12358 (1999).DOI
  55. [66]Motamarri, P., Nowak, M. R., Leiter, K., Knap, J. & Gavini, V. Higher-order adaptive finite-element methods for Kohn–Sham density functional theory. J. Comput. Phys. 253, 308–343 (2013).DOI
  56. [67]Das, S., Motamarri, P., Subramanian, V., Rogers, D. M. & Gavini, V. DFT-FE 1.0: A massively parallel hybrid CPU-GPU density functional theory code using finite-element discretization. Comput. Phys. Commun. 280, 108473 (2022).DOI
  57. [68]Bernholc, J. et al. Large-scale applications of real-space multigrid methods to surfaces, nanotubes, and quantum transport. Phys. Status Solidi B 217, 685–701 (2000).
  58. [69]Wang, J. & Beck, T. L. Efficient real-space solution of the Kohn–Sham equations with multiscale techniques. J. Chem. Phys. 112, 9223–9228 (2000).DOI
  59. [70]Waghmare, U. V. et al. HARES: an efficient method for first-principles electronic structure calculations of complex systems. Comput. Phys. Commun. 137, 341–360 (2001).DOI
  60. [71]Diaye*, M. N., Hewett, R. J., Atle, A. & Calandra, H. Optimized finite difference coefficients for the Helmholtz equation. in SEG Technical Program Expanded Abstracts 2015 3744–3748 (Society of Exploration Geophysicists, 2015). doi:10.1190/segam2015-5827540.1.
  61. [72]Zhang, J.-H. & Yao, Z.-X. Optimized finite-difference operator for broadband seismic wave modeling. GEOPHYSICS 78, A13–A18 (2013).DOI
  62. [73]Kindelan, M., Moscoso, M. & Gonzalez-Rodriguez, P. Optimized Finite Difference Formulas for Accurate High Frequency Components. Math. Probl. Eng. 2016, e7860618 (2016).DOI
  63. [74]SymPy. https://www.sympy.org/en/index.html.
  64. [75]Liszka, T. & Orkisz, J. The finite difference method at arbitrary irregular grids and its application in applied mechanics. Comput. Struct. 11, 83–95 (1980).DOI
  65. [76]Sharma, A. & Suryanarayana, P. On real-space Density Functional Theory for non-orthogonal crystal systems: Kronecker product formulation of the kinetic energy operator. Chem. Phys. Lett. 700, 156–162 (2018).DOI
  66. [77]Slater, J. C. & Koster, G. F. Simplified LCAO Method for the Periodic Potential Problem. Phys. Rev. 94, 1498–1524 (1954).DOI
  67. [78]Lejaeghere, K., Van Speybroeck, V., Van Oost, G. & Cottenier, S. Error Estimates for Solid-State Density-Functional Theory Predictions: An Overview by Means of the Ground-State Elemental Crystals. Crit. Rev. Solid State Mater. Sci. 39, 1–24 (2014).DOI
  68. [79]The RMG code http://www.rmgdft.org/.
  69. [80]Comparing Solid State DFT Codes, Basis Sets and Potentials | Center for Molecular Modeling. https://molmod.ugent.be/deltacodesdft.
  70. [81]Blaha, P. et al. WIEN2k: An APW+lo program for calculating the properties of solids. J. Chem. Phys. 152, 074101 (2020).DOI

Paper details

vlnc:W_qm5ah9fd
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents