betaFeedback

WorksPhysics and Astronomy

sdos: spectral densities of large sparse Hermitian and generalized eigenproblems

Abstract

sdos is an open Python package that computes the density of states (DOS) and related spectral quantities of very large sparse Hermitian matrices A, and of generalized problems A psi = lambda B psi, from Chebyshev moments estimated with random vectors. The density is reconstructed by the kernel polynomial method (KPM, the default) or by maximum entropy (Maxent); the same interface gives band energies, Fermi levels, local and projected densities, harmonic thermodynamics, exact HOMO-LUMO gaps and eigenpairs in a spectral window. Interfaces cover electronic (empirical tight-binding, DFTB, SIESTA) and vibrational (matrix-free LAMMPS Hessians) problems. I checked most ingredients against dense diagonalization or an independent code; the exceptions are listed in Sec. V. On a workstation (4 threads, 19 GB) the DOS of an 884,736-atom amorphous silicon model (a 6^3 tiling of a 4096-atom cell; 3.5 million orbitals; the orthogonal empirical tight-binding model of Kwon et al.) takes 71 s (M=100, R=2, no exact moments; 383 s with the default exact moments), and a matrix-free 2,097,152-atom crystal 10 min in 1.8 GB. A 17^3 replica of a 216-atom amorphous cell (1,061,208 atoms) agrees with exact Bloch spectra: electronic DOS in 229 s, matrix-free ACE vibrational DOS in 2.2 h on one GPU. Shift-invert, incomplete-Cholesky setup, exact moments of high order and Maxent at high moment counts do not scale. The underlying methods are due to Silver, Roder, Kress, Voter, Drabold, Sankey and collaborators; the Introduction states what is added.

Introduction

The method of this report applies to any large sparse Hermitian matrix HH (and to any matrix-free linear operator, and to the generalized problem Aψ=λBψA\psi= \lambda B\psi with BB positive definite). It computes the spectral density ρ(E)=N−1Tr⁡δ(E−H)\rho(E) = N^{-1}\operatorname{Tr}\delta(E-H), where EE is the energy (the spectral variable) and NN the dimension, and quantities derived from it. Nothing in the algorithm depends on where the matrix comes from. I illustrate the method with electronic and vibrational problems, because I know them, because exact references exist for them, and because the density of states determines band energies, Fermi levels and thermodynamics and, through its local variants, much of the interpretation of electronic and vibrational structure in disordered and nanostructured materials. These applications are examples of a more general tool. For matrices of dimension N∼105N \sim10^5–10710^7, as arise for amorphous networks, nanostructures and large supercells, diagonalization (O(N3)O(N^3) time, O(N2)O(N^2) memory) is out of reach. The moments μn=N−1Tr⁡mn(H~)\mu_n = N^{-1}\operatorname{Tr}m_n(\widetilde{H}) of ρ\rho with respect to Chebyshev polynomials of the scaled operator, on the other hand, cost O(N)O(N) operations each. They come from matrix-vector products with a few random vectors, because the trace can be replaced by a stochastic estimator [1, 2]. Two routes lead from moments to a density. The kernel polynomial method (KPM) [3, 4, 5] is a damped Chebyshev series; it is linear, easy to use, and positive with the Jackson kernel [6]. The principle of maximum entropy [7, 8, 9] finds the positive density of maximal entropy that reproduces the moments. It has an information-theoretic foundation [10] and, for the same moments, resolves sharp features better. Both belong to the family of linear-scaling electronic-structure methods reviewed by Goedecker [11] and Ordejón [12]; related moment methods are the recursion method [13], the plane-wave moments method [14] and Fermi-operator expansions [15], and modern surveys of stochastic spectral-density estimation are in Ref. [16].

Electronic problems in a non-orthogonal basis and vibrational problems with unequal masses are generalized eigenproblems, Ax=λBxA x = \lambda B x. Röder, Silver and Kress [17] showed that KPM extends to this case by applying B−1AB^{-1}A with a preconditioned conjugate-gradient (PCG) solve of BB for every Chebyshev step, and demonstrated it for a titanate/platinum interface and amorphous diamond; Röder, Silver, Drabold and Dong [18] treated amorphous diamond in the same way. SDOS implements this idea and adds alternatives for applying B−1B^{-1} (block-Jacobi, incomplete Cholesky, algebraic multigrid, and a fixed Chebyshev polynomial pp with p(B)≈B−1p(B) \approx B^{-1}, which replaces the inverse by a matrix polynomial so that the recursion is applied to one fixed, explicit operator p(B)Ap(B)A and needs no inner solves), a Maxent reconstruction, exact low-order moments, spectral windows, projections and a set of interfaces to tight-binding and force-field codes. Open tight-binding codes for large systems, such as KITE [19], Kwant (with its kp module) [20], pybinding [21] and TB-PLAS [22], use Chebyshev-based methods (KPM, or time propagation in TBPLAS) for orthogonal tight-binding models; I did not find an overlap matrix in their documentation or papers. What is offered here is an open, tested implementation and a quantitative validation, and I want to be clear about what is old. Chebyshev moments, KPM and Maxent for densities of states, impartial random vectors, locally truncated (exact) low-order moments and non-orthogonal KPM with PCG all come from the work of Silver, Röder, Voter, Kress, Drabold, Sankey and collaborators [3, 4, 9, 17, 18, 23, 24, 25]. Silver and Röder [25], in particular, fitted up to 1024 exact Chebyshev moments by maximum entropy for a tight-binding Hamiltonian of amorphous diamond with 16 384 states; their prescription is described in § 2 C. Voter, Kress and Silver [24] obtain low-order moments from locally truncated Hamiltonians, which, as I understand it, is the same local computation as my exact low-order moments. What I believe is new here is more limited: (i) using these exact moments up to order KK together with sampled moments above KK in one estimator, with an argument for when it is unbiased and a measurement of when it is not useful (§ 2 B and III C); (ii) a Maxent solver with a moment-uncertainty floor and a staged (bootstrapped) climb in the number of moments, which I tested against exact spectra but not against the Silver–Röder prescription; (iii) alternatives for the overlap inverse; and (iv) the open implementation, with chunked builders and matrix-free operators for 10610^6 atoms and more. I may have missed earlier descriptions of (i) and (ii). Each section says what is established elsewhere and what was measured here.

§ 2 describes the methods; Sec. III compares with exact results; Sec. IV gives four applications (a 2 097 152-atom crystal, the fullerene-to-graphene series, an amorphous Si model with DFTB and ACE, and a million-atom replica of a 216-atom cell); § 2.5 treats cost, memory and limits; Sec. VI gives recommendations and Sec. VII the conclusions and availability.

Methods

Moments from random vectors

Let HH be Hermitian with spectrum in [Emin⁡,Emax⁡][E_{\min}, E_{\max}], and H~=(H−b)/a\tilde{H}=(H-b)/a with the interval widened by a margin ϵw\epsilon w (w=Emax⁡−Emin⁡w=E_{\max}-E_{\min}) on each side, a=12w(1+2ϵ)a=\frac{1}{2}w(1+2\epsilon), b=12(Emax⁡+Emin⁡)b=\frac{1}{2}(E_{\max}+E_{\min}), so that the spectrum of H~\tilde{H} lies in (−1,1)(-1,1) even though the extreme eigenvalues are only estimated (ϵ=0.01\epsilon=0.01 by default). The density per orbital is ρ(E)=1aρ~(x)\rho(E)=\frac{1}{a}\tilde{\rho}(x), x=(E−b)/ax=(E-b)/a, where EE is the energy (the eigenvalue, for a general matrix), with Chebyshev moments

μn=1NTr⁡Tn(H~)=∫−11ρ~(x)Tn(x) dx.(1)\mu_n=\frac{1}{N}\operatorname{Tr} T_n(\tilde{H})=\int_{-1}^{1}\tilde{\rho}(x)T_n(x)\,\mathrm{d}x. \tag*{(1)}

A random vector ξ\xi with E[ξξ†]=1\mathbb{E}[\xi\xi^\dagger]=1 gives μn≈1Nξ†Tn(H~)ξ\mu_n\approx\frac{1}{N}\xi^\dagger T_n(\tilde{H})\xi (for unit-variance entries), and averaging over RR vectors reduces the sampling error as R−1/2R^{-1/2}; the error of each moment also falls as N−1/2N^{-1/2} (a bound is given below), so large systems need fewer vectors [1, 2, 16]. sdos draws Rademacher (±1\pm1) vectors for real and random phases for complex problems (Gaussian vectors are available; they have a larger variance, Table I). For a Hermitian matrix AA with E[ξξ†]=1\mathbb{E}[\xi\xi^\dagger]=1 and independent entries, the variance of the estimate ξ†Aξ\xi^\dagger A\xi of Tr⁡A\operatorname{Tr}A is 2∑i≠jAij22\sum_{i\ne j}A_{ij}^{2} for Rademacher vectors (real symmetric AA), ∑i≠j∣Aij∣2\sum_{i\ne j}|A_{ij}|^{2} for random phases and 2∥A∥F22\lVert A\rVert_{F}^{2} for real Gaussian vectors; Rademacher vectors have the smallest variance among real vectors with independent unit-variance entries [2]; random phases halve the off-diagonal variance of a real symmetric AA at the price of complex arithmetic. With A=Tn(H~)A=T_n(\tilde{H}) and the factor N−1N^{-1} of the normalized moment, the standard deviation of μn\mu_n for one Rademacher vector is bounded by 2/N\sqrt{2/N}, whatever the locality of HH: ∥Tn(H~)∥2≤1\lVert T_n(\tilde{H})\rVert_2\le1 gives ∥Tn(H~)∥F2≤N\lVert T_n(\tilde{H})\rVert_F^2\le N and sd⁡(μn)=N−1(2∑i≠j[Tn2]ij)1/2≤2/N\operatorname{sd}(\mu_n)=N^{-1}(2\sum_{i\ne j}[T_n^2]_{ij})^{1/2}\le\sqrt{2/N}. The covariance of the moments of different order and the cases where it matters are in Sec. III C. Variance-reduction schemes for stochastic trace estimation that I did not implement or test include probing [26] and low-rank deflation as in Hutch++ [27]. With vn=Tn(H~)v_n=T_n(\tilde{H}), the inner product ⟨u∣v⟩=N−1u†v\langle u|v\rangle=N^{-1}u^\dagger v, the recursion vn+1=2H~vn−vn−1v_{n+1}=2\tilde{H}v_n-v_{n-1} and the doubling identities

vectorsstd (meV/at.)time (s)efficiency
Rad.192.80.231.0×
Gauss.315.40.230.4×
impartial (3)25.60.3536.7×
Rad. + exact K=411.21.7538.0×
Rad. + exact K=84.48.9947.3×
Rad. + exact K=161.479.8452.2×

Table I. Random vectors and exact low moments, band energy of DTW4096 (Kwon, M=100M = 100, 12 seeds with R=2R = 2 each; std is the standard deviation of an estimate with R=2R = 2 vectors, 2\sqrt{2} times larger per vector). Efficiency = (variance × time)−1^{-1} relative to plain Rademacher vectors; the uncertainty of each entry is about 30%.

μ2n=2⟨vn∣vn⟩−μ0,(2)\mu_{2n}=2\langle v_n|v_n\rangle-\mu_0, \tag*{(2)}
μ2n+1=2⟨vn+1∣vn⟩−μ1,(3)\mu_{2n+1}=2\langle v_{n+1}|v_n\rangle-\mu_1, \tag*{(3)}

give MM moments from M/2M/2 matrix–vector products per vector. For complex Hermitian HH all inner products are conjugated and everything else is unchanged; real arithmetic is used whenever the operator is real. The operator may be an explicit sparse matrix (CSR) or any matrix-free linear operator; Hermiticity is checked (explicitly or by randomized probes for matrix-free operators), which matters for finite-difference Hessians that are Hermitian only to O(δ2)O(\delta^2).

Spectral bounds. The bounds are estimated with ARPACK (Lanczos)1, starting from a fixed vector (a Gaussian vector from a fixed internal seed, independent of the random vectors used for the moments). Two runs therefore give identical bounds, and the same bounds serve all moment vectors. The vectors of the moments themselves come from the user’s seed and differ from run to run if none is given. Lanczos Ritz values lie inside the spectrum, hence the margin ϵ\epsilon. For matrix-free operators built from finite differences of forces the estimate can fail badly (§ 2.5, item iii), and the package does not detect an over-wide window. For explicit matrices Gershgorin bounds are available as a safe fallback.

Exact low-order moments and impartial vectors

Drabold and Sankey [9] introduced impartial vectors, chosen to weigh all eigenvectors equally, and computed low-order moments exactly; Voter, Kress and Silver [24] and Röder [25] evaluate low-order moments from locally truncated Hamiltonians. The substitution below uses that local computation. The construction of Drabold and Sankey minimizes the penalty P(ξ)=⟨ξ∣ξ⟩−1)2+(⟨ξ∣H∣ξ⟩−μ1)2+(⟨ξ∣H2∣ξ⟩−μ2)2P(\xi)=\langle\xi|\xi\rangle-1)^2+(\langle\xi|H|\xi\rangle-\mu_1)^2+(\langle\xi|H^2|\xi\rangle-\mu_2)^2 by conjugate gradients, with the exactly known μ1=N−1Tr⁡H\mu_1=N^{-1}\operatorname{Tr}H and μ2=N−1Tr⁡H2\mu_2 = N^{-1}\operatorname{Tr}H^2 (both O(N)O(N) from the matrix), so that the first three moments of ξ\xi are exact. This narrows the distribution of the band energy over the random vectors. Only three moments are constrained in the construction of Drabold and Sankey; constraining more makes the ensemble measurably biased (Sec. III C), so sdos keeps three. The recommended way to use exact low moments is substitution: μn\mu_n for n≤Kn \le K can be computed exactly in O(N)O(N) because [Tn(H~)]ii[T_n(\widetilde{H})]_{ii} depends only on a neighbourhood of radius ≈n/2\approx n/2 hops of site ii, and in every sample the sampled μn\mu_n with n≤Kn \le K is replaced by the exact one (exact_moments=KK). The moments with n>Kn > K remain the sampled estimates and the Chebyshev recursion is unchanged. The reconstruction uses one list, μ0,…,μK\mu_0,\ldots,\mu_K exact and μK+1,…,μM−1\mu_{K+1},\ldots,\mu_{M-1} sampled, of one matrix. Each μn\mu_n is unbiased on its own, so every linear functional of the list (the KPM density, the integrated density, the band energy at a fixed Fermi level) is unbiased, and the variance falls sharply (Table I). The list is not exactly the moment list of a positive density; this does not matter for KPM, and with Maxent it is absorbed by the sampled uncertainties of R≥2R \ge2 vectors (Sec. III C), so with a single vector, which has no variance estimate, sdos uses K=0K = 0. The default is K=4K = 4 when AA is an explicit sparse matrix and BB is absent or diagonal, and K=0K = 0 otherwise, because the exact moments need the matrix elements. The cost per site grows polynomially with KK, not with NN. Table I measures all options on one problem: at equal cost every variance-reduction option beats plain Rademacher vectors by a factor of about 35–50, with K=8K = 8–16 comparable (the quoted uncertainty is 30%).

Reconstruction: KPM and maximum entropy

KPM. The default reconstruction is the damped series

ρ~(x)=1π1−x2[g0μ0+2∑n≥1gnμnTn(x)]\widetilde{\rho}(x) = \frac{1}{\pi\sqrt{1-x^2}}\left[g_0\mu_0 + 2\sum_{n\ge1} g_n\mu_nT_n(x)\right]

with the Jackson kernel gng_n [5, 6] (positive, energy resolution ≈πa/M\approx\pi a/M throughout this report, taken as the Gaussian σ\sigma of the kernel, so that the full width at half maximum is about 2.355πa/M2.355\pi a/M; the Fejér sums [28, 29] are the simplest damping with a positive kernel); the Lorentz kernel is available for Green-function-like uses. The density is evaluated on the Chebyshev–Gauss nodes xk=cos⁡θkx_k = \cos\theta_k, θk=π(k−12)/npts\theta_k = \pi(k - \frac{1}{2})/n_{\mathrm{pts}} with npts=max⁡(2M,1000)n_{\mathrm{pts}} = \max(2M,1000) by default; the integrated density is the analytic integral of the damped series, and the band energy and Fermi level are obtained from it on a uniform grid of 20000 points in θ\theta (trapezoid rule in θ\theta). KPM is linear in the moments and never fails to converge. Beyond the resolution at which the sampling noise dominates, more moments only sharpen speckle. For a finite system with NN levels, a resolution finer than the mean level spacing resolves individual levels.

Maximum entropy. Maxent [7, 8, 9, 25] writes ρ~(x)=m(x)exp⁡[∑nλnTn(x)]\widetilde{\rho}(x) = m(x)\exp[\sum_n\lambda_nT_n(x)] with a default model mm and finds λ\lambda by Newton’s method on the convex dual

D(λ)=∫meλ⋅T dx−λ⋅μ+12∑nσn2λn2(4)\mathcal{D}(\lambda) = \int m e^{\lambda\cdot T}\,dx - \lambda\cdot\mu+ \frac{1}{2}\sum_n \sigma_n^2\lambda_n^2 \tag*{(4)}

whose integrals are evaluated by the midpoint rule on a uniform grid in θ=arccos⁡x\theta= \arccos x of max⁡(8192,64M)\max(8192,64M) points (so that dx=sin⁡θ dθdx = \sin\theta\,d\theta and a flat model is 12sin⁡θ\frac{1}{2}\sin\theta); the integrated density is the cumulative midpoint sum, and the band energy follows from a trapezoid rule in energy over its increments. The dual has gradient equal to the moment residual plus σj2λj\sigma_j^2\lambda_j and Hessian ∫meλ⋅TTjTk dx+diag⁡(σj2)\int m e^{\lambda\cdot T}T_jT_k\,dx + \operatorname{diag}(\sigma_j^2). Since TjTk=12(Tj+k+T∣j−k∣)T_jT_k = \frac{1}{2}(T_{j+k}+T_{|j-k|}), the integral is a function of the Chebyshev moments of the current iterate meλ⋅Tm e^{\lambda\cdot T}. Here σn\sigma_n are the standard errors of the sampled moments (zero for exact moments), and the solver uses an eigenvalue-floored Newton step with backtracking. The default model is uniform (flat); the Chebyshev weight, the Jackson-KPM density and a KPM model at fewer moments gave no measurable difference in accuracy or conditioning. Two practical points follow from my tests. First, with a spectral window (the interval [Emin⁡,Emax⁡][E_{\min},E_{\max}] used for the scaling) much wider than the support (the interval that contains the eigenvalues; a poor bound), the multipliers become very large (up to 10410^4–10510^5 for exact moments) although the density remains accurate; stochastic moments with their errors tolerate even a factor-two window. Second, Maxent with exact moments of a density with singularities (log van Hove peaks, Dirac cusps) is a badly conditioned problem. This is an empirical observation; I have no proof. Silver and Röder [25] fitted 1024 exact moments for a 16384-state amorphous-diamond Hamiltonian; they fitted Gibbs-damped moments (the positive-kernel factors gmM×Kg_m^{M\times K}) with the kernel-polynomial density as the default model, evaluated the Fourier integrals on a fine grid of M×K×IM \times K \times I pixels (they use K=8K = 8, I=4I = 4), used moment uncertainties at the numerical precision for exact data and continued in the regularization parameter α\alpha (an outer loop that decreases α\alpha from a large value) rather than in the number of moments, none of which I tested, so the non-convergence reported here refers to the solver described in this section (undamped moments, flat model), not to theirs. The exact moments of a density with van Hove peaks and band edges lie close to the boundary of the set that an exponential exp⁡(∑nλnTn)\exp(\sum_n \lambda_n T_n) can reproduce, the multipliers grow as MM increases, and the Newton iteration stops converging: for DTW4096 it did not converge in 300 steps for M≥100M \ge100 (Table IV). The behaviour depends on the density: for C3840_{3840} with M=150M = 150 exact moments the plain solver does converge (403 steps without the floor, 27 with it; Sec. IV B). With sampled moments the sampling error regularizes the problem. The same regularization can be applied to exact moments: sdos assigns every moment except μ0\mu_0 a minimum uncertainty σn≥s0\sigma_n \ge s_0 with s0=10−4s_0 = 10^{-4} by default (sigma_floor); the moments are normalized, ∣μn∣≤1|\mu_n| \le1, so this is an absolute scale. The dual then has a finite solution and Newton’s method converges in 9–24 steps at all MM tested (Table IV). For stochastic moments of a 16 384-orbital system with R=4R = 4 and K=0K = 0 the sampled errors (σn\sigma_n between 7×10−47 \times10^{-4} and 4×10−34 \times10^{-3} for n>Kn > K) exceed the floor, which then changes nothing (identical results for M=100M = 100, 170 and 300). The floor does act on the exact moments n≤Kn \le K (σn=0\sigma_n = 0 there; K=4K = 4 by default), and it is essential for R=1R = 1, where there is no variance estimate (Table III). The floor is a bias of order s0s_0 in the moments, which was found to be ≤0.02\le0.02 meV/atom in the band energy at s0=10−4s_0 = 10^{-4} and 0.2 meV/atom at 10−310^{-3}; sigma_floor=0 restores the exact-moment problem. Whether 10−410^{-4} is small enough depends on the quantity. A shift δμn\delta\mu_n of the moments changes an integrated quantity ∫fρ~ dx\int f \tilde{\rho}\,dx by ∑nfnδμn\sum_n f_n\delta\mu_n, with fnf_n the Chebyshev coefficients of ff; for the band energy f=(ax+b)Θ(xF−x)f = (ax + b)\Theta(x_F - x) and ∣fn∣∼a/n|f_n| \sim a/n, so the worst case, with every moment displaced coherently by s0s_0, is ∼s0aln⁡M\sim s_0 a \ln M (a few meV per orbital for a=10a = 10 eV), and a random-sign shift is ∼1.3 s0a\sim1.3\,s_0a. These are bounds on the bias of the constraints, not on the fitted density, whose error was measured to be far smaller (above). A quantity that depends on fine structure at high order (a feature narrower than πa/M\pi a/M, a sharp edge, a spectral weight below 10−410^{-4}) or that needs a precision better than ∼s0a\sim s_0a may be affected: reduce sigma_floor, accepting slower convergence and, in the limit s0=0s_0 = 0, the non-convergence documented above. In practice: with exact moments the default floor gives the exact-moment Maxent density, to a band-energy error of 0.02 meV/atom or less, in a few tens of Newton steps where the plain solver fails; with sampled moments, K=0K = 0 and R≥4R \ge4, the floor is inactive. More random vectors do not help the exact-moment problem, because the exact moments do not depend on RR. For sampled moments they lower the sampling error and with it the pointwise error (Table III), but convergence is not the issue there: Newton converges in 12–15 steps for R≥4R \ge4.

MMΔEBS\Delta E_{\mathrm{BS}} (meV/atom) KPMΔEBS\Delta E_{\mathrm{BS}} (meV/atom) MaxentΔEBS\Delta E_{\mathrm{BS}} (meV/atom) Maxent s0s_0pointwise L1L_1 KPMpointwise L1L_1 Maxentpointwise L1L_1 Maxent s0s_0Newton steps MaxentNewton steps Maxent s0s_0
20−198.23-198.23+22.52+22.52+22.52+22.523.34930.11220.112299
40−18.24-18.24+1.56+1.56+1.56+1.560.28620.03830.03831111
70−0.99-0.99+0.12+0.12+0.10+0.100.07150.01160.01143417
100+0.263+0.263–+0.017+0.0170.0332–0.0082–22
170+0.297+0.297–−0.005-0.0050.0119–0.0068–23
300+0.138+0.138–−0.012-0.0120.0053–0.0082–24
400+0.085+0.085–−0.017-0.0170.0037–0.0123–22

Table IV. Truncation error on DTW4096 (Kwon, Γ\Gamma) with exact moments (no sampling noise): band-energy error and L1L_1 distance to the exact spectrum broadened by 0.05 eV. Maxent: exact-moment constraints; dashes: the solver did not reach tolerance 10−910^{-9} in 300 Newton steps, so no result is reported (no steps are listed either). Maxent s0s_0: every moment but μ0\mu_0 given the uncertainty s0=10−4s_0 = 10^{-4} (§ 2 C); all converged.

Bootstrapped Maxent. maxent_bootstrap climbs in the number of moments: it solves for M=16,32,64,…M = 16, 32, 64, \ldots in turn, starts each stage from the previous multipliers, and after each stage restricts the integration support of the dual to the interval where the density found at that stage exceeds 10−410^{-4} of its maximum, padded by two resolution lengths π/M\pi/M (in the scaled variable xx). No states are allowed outside the edges, so the multipliers need not build huge exponentials to suppress the density in the empty margin of the window. The support is a modelling statement, checked by the residual of the moments; a stage that fails to converge is widened and retried, and if it still fails the last converged stage is returned (with converged=False and the number of moments used). For graphene (Xu Hamiltonian, exact kk-sum on a 240×240240 \times240 mesh) with exact moments, M=400M = 400, the bootstrap converges in 29 s without the uncertainty floor, where the plain solver does not converge in 300 steps (46 s; L1L_1 distance to the exact spectrum broadened by 0.05 eV: 0.0446 for the bootstrap, and 0.0449 for the last iterate of the plain solver, which is not a converged result). With the default floor both converge, the bootstrap in 2.0 s and the plain solver in 3.8 s (31 steps), at L1=0.0444L_1 = 0.0444 (timings on 2 threads, approximate). The support barely shrinks for graphene, because the band fills its window, so here the gain comes from the warm starts.

Integrated quantities. The integrated density N(E)N(E), the Fermi level for a given electron count, and the band energy EBS=2∑Ei<EFEiE_{\mathrm{BS}} = 2\sum_{E_i<E_F} E_i follow from the reconstruction (fermi_and_band_energy), with jackknife error bars for Maxent (the spread of the result when each random vector is left out in turn). In a gap the integrated density is flat, so EFE_F is not determined by the moments to better than the gap: sdos reports the interval over which N(E)N(E) equals the electron count within noise (fermi-interval) and warns; the band energy is nearly independent of where EFE_F is placed in a true gap. Exact HOMO and LUMO levels are obtained separately (Sec. III).

Generalized problems

For Aψ=λBψA\psi= \lambda B\psi, Y=B−1AY = B^{-1}A is self-adjoint in the BB-inner product and similar to B−1/2AB−1/2B^{-1/2}AB^{-1/2}, so Tr⁡Tm(Y~)=Tr⁡Tm(B−1/2AB−1/2)\operatorname{Tr} T_m(\tilde{Y}) = \operatorname{Tr} T_m(B^{-1/2}AB^{-1/2}) and ⟨ξ∣Tm(Y~)∣ξ⟩\langle\xi|T_m(\tilde{Y})|\xi\rangle is an unbiased estimator for any ξ\xi with E[ξξ†]=1\mathbb{E}[\xi\xi^\dagger] = 1 [[17]]. YY is not Hermitian in the Euclidean product, so the doubling identities do not apply: one application of B−1B^{-1} per moment is needed, and the cost per moment is that of a solve. Spectral bounds of YY use a Lanczos process in the BB-inner product. SDOS offers the following ways of applying B−1B^{-1}, all verified to give the same moments to tolerance (Table V): (i) PCG to a relative residual 10−1210^{-12} with block-Jacobi (default), incomplete Cholesky IC(0) with a Manteuffel shift ([30, 31]) (the shift needed is reported, since the overlap of a minimal basis is not an M-matrix), or algebraic multigrid; (ii) a fixed polynomial p(B~)≈B~−1p(\tilde{B}) \approx\tilde{B}^{-1} on the diagonally scaled B~=D−1/2BD−1/2\tilde{B} = D^{-1/2}BD^{-1/2} (the scaled problem Y~=D1/2YD−1/2\tilde{Y} = D^{1/2}YD^{-1/2} is similar to YY), of degree ≃12κln⁡(2/ϵ)\simeq\frac{1}{2}\sqrt{\kappa}\ln(2/\epsilon) from the condition number κ\kappa.

S−1S^{-1} byΓ\Gamma (real)k (complex Hermitian)
time (s)iter./solveΔEBS\Delta E_{\mathrm{BS}} (meV/atom)L1L_1 to exacttime (s)iter./solveL1L_1 to Γ\Gamma exact
PCG, block-Jacobi5423.6−70.0±35-70.0 \pm350.02412523.60.021
PCG, IC(0)7911.0−70.0±35-70.0 \pm350.02415511.00.021
Chebyshev polynomial6426.0−70.0±35-70.0 \pm350.02413526.00.021

Table V. DFTB electronic problem of DTW4096 (16 384 orbitals, 179 nonzeros per row): KPM, M=170M = 170, R=6R = 6. At Γ\Gamma the three ways of applying S−1S^{-1} give identical band energies (the same moments); the error is against the dense generalized spectrum and is a sampling error (this run, seed 11: −70-70 meV/atom, with ±35\pm35 the standard error estimated from its six vectors; other seeds give −122-122, +69+69 and +153+153 meV/atom with estimated errors of 84–94, so that estimate is itself uncertain and the spread over seeds, about 110 meV/atom at R=6R = 6, is the reliable figure; see text), and L1L_1 is the distance to the exact spectrum broadened by 0.1 eV. The dense complex generalized problem does not fit in memory here, so the k point is compared with the Γ\Gamma spectrum of the same cell.

In detail: the extreme eigenvalues of B~\widetilde{B} are found by ARPACK (tolerance 10−810^{-8}) and widened by 2% on each side to guard against bounds that are too tight; pp is the Chebyshev interpolant of 1/t1/t on that interval; its degree starts from ln⁡(2/ε)/ln⁡(1/q)\ln(2/\varepsilon)/\ln(1/q) with q=(κ−1)/(κ+1)q=(\sqrt{\kappa}-1)/(\sqrt{\kappa}+1) and is raised until max⁡∣1−tp(t)∣≤ε\max|1-tp(t)|\leq\varepsilon on a grid of 4001 points (ε=10−12\varepsilon=10^{-12} by default, cheb_tol); p(B)p(B) is applied to blocks of vectors by the Clenshaw recurrence, one product with B~\widetilde{B} per degree, and a check on a random vector confirms the tolerance. The inverse is then replaced by a matrix polynomial, so the recursion is applied to one fixed operator and needs no inner solves; the error of the polynomial is a known, controlled approximation. For a diagonal BB (masses) BB is absorbed into AA once and no solves are needed. For the DFTB overlap of silicon (κ(B~)=3.85\kappa(\widetilde{B})=3.85) the polynomial has degree 26 and PCG with block-Jacobi needs about 24 iterations per solve (Table V); the iteration count is independent of the system size (§ 2.5).

Eigenvectors in spectral windows, exact gaps and projections

Windows. window_eigs returns the eigenpairs with λ\lambda in a window together with their inverse participation ratio IPR=∑sws2\mathrm{IPR}=\sum_s w_s^2 (site weights wsw_s from the Mulliken populations, see Projections below), by shift-invert (sparse LDL†LDL^\dagger of A−τBA-\tau B, ARPACK) or by Chebyshev-filtered subspace iteration [32]. Shift-invert is not O(N)O(N): the fill of a sparse factorization grows faster than linearly in three dimensions; the filtered iteration (a Chebyshev polynomial filter) is the scalable fallback. A filter of a given degree separates the states in the window from the rest of the spectrum only at distances from the window edges larger than about 2πa/degree2\pi a/\mathrm{degree}, where aa is the half-width of the spectrum.

Exact HOMO–LUMO gaps. By Sylvester’s law of inertia, the number of eigenvalues below τ\tau equals the number of negative pivots of A−τBA-\tau B; count_states_below evaluates it from a symmetric factorization without pivoting, and homo_lumo finds HOMO and LUMO by two bisections, with no eigensolver and no assumption on the gap size or on degeneracy. Because of the fill, this is more expensive than the density of states above ∼2×104\sim2\times10^4 orbitals (§ 2.5), so the exact gap is a tool for moderate sizes.

Projections. For Aψ=λBψA\psi=\lambda B\psi with eigenvectors normalized ψ†Bψ=1\psi^\dagger B\psi=1, the weight of orbital ii in state nn is the Mulliken weight win=Re⁡ψin∗(Bψn)iw_{in}=\operatorname{Re}\psi_{in}^{*}(B\psi_n)_i [33] (equal to ∣ψin∣2|\psi_{in}|^2 for B=1B=1). For overlapping orbitals winw_{in} is not a probability: it can be negative or exceed one (the sum over ii is one, ∑iwin=ψn†Bψn\sum_i w_{in}=\psi_n^\dagger B\psi_n), so a local or projected density built from it is a signed quantity, not a positive local spectral density, and it can be negative at some energies for orbitals that overlap strongly with their neighbours. This is a property of the Mulliken partition, as for Mulliken charges; the total density is unaffected.

KPM handles signed densities; Maxent assumes positivity and is not used for projections. A positive alternative is the Löwdin partition, which needs B1/2B^{1/2} and is not provided. The order of the operators matters. With Y=B−1AY=B^{-1}A and ∑nψnψn†=I\sum_n\psi_n\psi_n^\dagger=I the spectral projector is ∑nψnψn†δ(E−λn)=δ(E−Y)B−1\sum_n\psi_n\psi_n^\dagger\delta(E-\lambda_n)=\delta(E-Y)B^{-1}. Its diagonal element [δ(E−Y)B−1]ii=∑n∣ψin∣2δ(E−λn)[\delta(E-Y)B^{-1}]_{ii}=\sum_n|\psi_{in}|^2\delta(E-\lambda_n) integrates to (B−1)ii(B^{-1})_{ii}, not to one, so it is not the Mulliken density. With the Mulliken weight winw_{in} defined above,

LDOSi(E)=Re⁡∑nψin∗(Bψn)iδ(E−λn)=Re⁡[Bδ(E−Y)B−1]ii=Re⁡[δ(E−Y†)]ii,(5)\begin{aligned} \mathrm{LDOS}_i(E)=\operatorname{Re}\sum_n\psi_{in}^{*}(B\psi_n)_i\delta(E-\lambda_n) \\ &=\operatorname{Re}\left[B\delta(E-Y)B^{-1}\right]_{ii}=\operatorname{Re}\left[\delta(E-Y^\dagger)\right]_{ii}, \tag*{(5)} \end{aligned}

because BYB−1=AB−1=Y†BYB^{-1}=AB^{-1}=Y^\dagger (Y†Y^\dagger the Euclidean adjoint of YY) and f(Y)B−1=B−1f(AB−1)f(Y)B^{-1}=B^{-1}f(AB^{-1}) (the matrix BB does not commute through the function; it conjugates the argument). Its Chebyshev moments are the diagonal elements [Tn(Y~†)]ii=eiTTn(Y~†)ei[T_n(\widetilde{Y}^\dagger)]_{ii}=\mathbf{e}_i^\mathsf{T}T_n(\widetilde{Y}^\dagger)\mathbf{e}_i of the adjoint operator, which is applied to a vector as Y†v=A(B−1v)Y^\dagger v=A(B^{-1}v): first the inverse of BB, then AA (the reverse of Yv=B−1(Av)Yv=B^{-1}(Av)) used for the total density. The code applies exactly this operator (GeneralizedOperator.apply_adjoint), and a unit test with dense BB, real and complex, compares the resulting moments of single orbitals with the Mulliken weights Re⁡ψn∗(Bψn)\operatorname{Re}\psi_n^*(B\psi_n) of the generalized eigenproblem to 10−710^{-7}. Then PDOSg=∑i∈gLDOSi\mathrm{PDOS}_g=\sum_{i\in g}\mathrm{LDOS}_i. Sum rules hold exactly: ∑iLDOSi=Nρ\sum_i\mathrm{LDOS}_i=N\rho, each LDOSi_i integrates to one, and PDOSg_g integrates to ngn_g, the number of orbitals in the group. Two estimators are provided (projected random vectors were used for local densities by Sankey, Drabold and Gibson [23]): “shared” (one random vector is propagated once and every group reads its own partial inner product, all groups for the price of one DOS, at the cost of extra noise for small groups) and “separate” (a vector supported on the group), and ldos propagates unit vectors for a deterministic result.

Vibrations

For a force-constant matrix Φ\Phi and masses mim_i, the dynamical matrix Dij=Φij/mimjD_{ij}=\Phi_{ij}/\sqrt{m_i m_j} has eigenvalues ω2\omega^2. vdos works with λ=ω2\lambda=\omega^2 and reports g(ω)=2ωρ(ω2)g(\omega)=2\omega\rho(\omega^2); negative eigenvalues (instabilities) are reported, not clipped, with the weight below the resolution. The three rigid translations are known eigenvalues at λ=0\lambda=0 and can be removed exactly from the moments (zero_modes=3), which matters for the entropy and the low-temperature heat capacity because the kernel would smear their weight into small positive ω\omega. harmonic_thermo gives the zero-point energy, UU, FF, SS and CvC_v per atom and warns below the temperature where the lowest frequencies are unresolved.

Matrix-free Hessians. For large models HessianOperator applies Φu=−[F(x0+su)−F(x0−su)]/2s\Phi_u=-[F(x_0+su)-F(x_0-su)]/2s with any force provider (LAMMPS through its Python module). In this case a product costs two force calls and translations are projected out. Units and conventions: positions in Å, forces in eV/Å, masses in atomic mass units (the LAMMPS metal units), so Φ\Phi is in eV/Å2^2 and DD in eV Å−2^{-2} u−1^{-1}, with ω2=λ\omega^2 = \lambda and ℏω=64.654\hbar\omega= 64.654 meV ×λ/[eV A˚−2 u−1]\times\sqrt{\lambda/[\mathrm{eV}\,\mathrm{\AA}^{-2}\,\mathrm{u}^{-1}]} in the thermodynamics (the factor is the argument energy_per_sqrt_lambda of harmonic_thermo, for other unit systems). The cell is periodic as defined by the force provider (LAMMPS applies the minimum image and its own neighbour lists), and the Hessian is that of the structure at Γ\Gamma. For a vector vv the displacement is δx=s m−1/2v\delta x = s\,m^{-1/2}v (components v/miv/\sqrt{m_i}), where the scalar ss is chosen such that the rms of the Cartesian components of δx\delta x, (13N∑∣δxiα∣2)1/2\left(\frac{1}{3N}\sum|\delta x_{i\alpha}|^2\right)^{1/2}, equals 10−310^{-3} Å regardless of NN and of the norm of vv (so the rms displacement per atom, as a vector, is 3×10−3\sqrt{3}\times10^{-3} Å); the central difference then gives Φ δx\Phi\,\delta x with an O(δ2)O(\delta^2) error and the result is multiplied by m−1/2m^{-1/2} again to give DvDv. No Hessian is stored.

Interfaces and implementation

SDOS is a pure-Python package (NumPy [34], SciPy [35]; Python 3.14 in the present tests); everything else is optional. Tight-binding models: the Hamiltonian of Kwon et al. [36], an empirical, orthogonal sp3sp^3 tight-binding total-energy model for silicon (a band-structure term plus a repulsive pair potential; sdos uses only its Hamiltonian matrix, with four orbitals per atom), with Bloch twists (complex Hermitian matrices), a chunked assembler for 10610^6 atoms and a matrix-free periodic stencil for crystals; and the Xu–Wang–Chan–Ho carbon model [37]. DFTB Slater–Koster (.skf) files, read from a user-supplied directory (the pbc-0-3 set [38] is distributed under CC BY-SA 4.0 and requires citation of Refs. [39, 40]; it is not bundled), non-self-consistent [41, 42] (zeroth-order Hamiltonian H0H_0 and overlap SS). SIESTA HH and SS through sisl [43, 44, 45], and LAMMPS force fields [46] through the matrix-free Hessian. A command-line tool (sdos) covers the common cases. The test suite has 186 tests (run with OpenMP/BLAS capped at 4 threads), including the comparisons with exact diagonalization listed in Sec. III, a regression test against the original single-file code and tests that the documented examples run.

Input, result files and analysis. The simplest matrix file is plain text with one entry per line, i j a_ij (a fourth column is the imaginary part); one or both triangles, 0- or 1-based indices and the storage are detected, and the other triangle is generated by conjugation (Matrix Market and SciPy .npz are also read). The density-of-states stage needs only the matrix and produces a density ρ(E)\rho(E) per orbital, normalized to one. A result written with sdos H.dat -o result.npz (here H.dat is a plain-text list of the nonzero matrix elements; or res.save is a self-describing file: ρ\rho, its error bars and the integrated density, the averaged and per-sample Chebyshev moments, the spectral window, the method and its settings, and for vibrational ω\omega and g(ω)g(\omega)). sdos analyze result.npz (or sdos.load) then works from the file alone, without the matrix. Whatever needs information from outside the matrix enters only here: for electrons the number of electrons (given directly or as the sum of the valence of each atom, from a file) and the spin degeneracy give EFE_F and the band energy; for vibrations a temperature grid gives the zero-point energy and UU, FF, SS, CVC_V (the masses and the three zero modes change the moments and are therefore applied at the density-of-states stage). The analysis can also rebuild the density from the first M′M' saved moments with KPM or Max-ent, which costs no matrix products. Saving and loading reproduces the arrays exactly, and EFE_F and EBSE_{\mathrm{BS}} from a loaded file agree with the in-memory values to 10−1210^{-12} for both methods (tests); reconstruction from a loaded vibrational result is not supported, and the generalized problem was not tested through the file.

The DFTB reader. Reading the DFTB format correctly required three conventions that I did not find stated in the documentation available to me (the DFTB+ manual; I could not retrieve the separate Slater–Koster format description, and some of them may be documented there), which I recovered by comparing whole matrices with DFTB+ [47] (Sec. IIIE): a table announced as ngrid rows contains ngrid - 1, row jj sits at r=j Δrr = j\,\Delta r (r=0r = 0 is the omitted row) and the first 19 rows of the silicon file are placeholders for r<0.4r < 0.4 Bohr; the pp orbitals are ordered (py,pz,px)(p_y,p_z,p_x); and each integral is continued past the end of its table by a quintic that reaches zero, with vanishing first and second derivatives, one Bohr later and matches value, slope and curvature of the table’s last row. The non-orthogonal pair convention follows from the file name: in A–B.skf the first orbital of each integral sits on A, so a (pp on A, ss on B) block uses the B–A file.

Validation

The electronic examples use the Kwon Hamiltonian [36], an empirical, orthogonal sp3sp^3 tight-binding total-energy model of silicon; its spectra are those of that model, so agreement with an exact spectrum below tests the method, not the model. Each test below is compared with exact diagonalization or with an independent code; the benchmarks use DTW4096, a 4096-atom model of amorphous silicon from the Djordjevic–Thorpe–Wooten family [48, 49] (relaxed at zero pressure with ACE, cell 43.6×43.7×43.943.6 \times43.7 \times43.9 Å3^3), at Γ\Gamma unless stated, since the cells are large and the kk-point dependence of an amorphous Si cell this size is weak.

Conventions. Unless stated otherwise, L1=∫∣ρest−ρref∣ dEL_1 = \int|\rho_{\mathrm{est}} - \rho_{\mathrm{ref}}|\,dE over the whole spectrum with both densities normalized to one, where ρref\rho_{\mathrm{ref}} is the histogram of the exact eigenvalues broadened by a Gaussian of the width given in each case (0.05 eV for the tests with exact moments, 0.1 eV for the stochastic ones; a narrower reference penalizes the sharper Maxent curves, a wider one the broader KPM curves). Band energies are for the electron count of the neutral system, in meV per atom, and “error” is estimate minus exact. A sampling error quoted as ± is the standard error over the RR vectors unless it is marked 2σ2\sigma. Table II lists every feature that I checked, with the system, the reference, the quantity and the observed error; the last rows list what was not checked or not reproduced.

Unit tests

The package is tested against the following, all of which pass: (1) a cubic Anderson model; (2) the same with Peierls phases (complex Hermitian); (3) a 216-atom diamond-structure silicon cell with the Kwon Hamiltonian; (4) a random sparse AA with a sparse positive definite BB whose condition number is tuned from ≈10\approx10 to ≈104\approx10^{4}, real and complex; (5) a Keating [51] dynamical matrix with unequal masses in both forms (mass-weighted, and generalized with the mass matrix); (6) PCG (block-Jacobi) and the Chebyshev polynomial give the same moments (the other preconditioners, IC(0) and AMG, are exercised in a further test and compared on DTW4096 in Table V); (7) regression against the original code. Beyond these: Maxent robustness (large windows, default models), the Mulliken sum rules, local exact moments against the dense matrix, the inertia counts, and the interfaces (Secs. III D–III F).

Accuracy against the number of moments and vectors

To separate truncation from sampling I compute the moments exactly from the dense spectrum of DTW4096 (Kwon TB, 16 384 orbitals) and reconstruct (Table IV, Figure 1a,Figure 1b). KPM with the Jackson kernel converges steadily (L1L_1 error 3.7×10−33.7 \times10^{-3} at M=400M = 400); Maxent with exact-moment constraints is more accurate for the same moments up to M=70M = 70, the last size at which it converged: at M=70M = 70 its band-energy error is 0.12 meV/atom against 1.0 and its L1L_1 distance is six times smaller, at M=40M = 40 1.6 against −18-18 meV/atom. From M=100M = 100 the plain Maxent solver no longer converges with exact moments of this singular density; how many exact moments the plain solver manages depends on the problem (for some it works to several hundred); those results are not reported (dashes in Table IV, no points for “Maxent, exact moments” in Figure 1). With the uncertainty floor s0=10−4s_0 = 10^{-4}(§ 2 C) every case converges: the band-energy errors are +0.02, −0.005-0.005, −0.012-0.012 and −0.017-0.017 meV/atom at M=100M = 100, 170, 300 and 400, 5–60 times smaller than KPM’s (+0.09 to +0.30 meV/atom), and the L1L_1 distance to the 0.05 eV-broadened exact spectrum is 0.007–0.012. The L1L_1 distance of Maxent s0s_0 is smallest at M=170M = 170 (0.0068) and then rises (0.0082 and 0.0123 at M=300M = 300 and 400; Figure 1b), while KPM’s keeps falling and is smaller at M=300M = 300 and 400 (0.0053 and 0.0037); the rise is not caused by the floor: at M=400M = 400 the L1L_1 distance is 0.0117, 0.0123 and 0.0128 for s0=10−3s_0 = 10^{-3}, 10−410^{-4} and 10−510^{-5} (s0=10−6s_0 = 10^{-6} did not converge in 300 steps, so I give no number), and at M=300M = 300 it is 0.0082–0.0083 for s0=10−5s_0 = 10^{-5} to 10−310^{-3} (0.0095 for 10−610^{-6}, 285 steps; script bench_accuracy_floor.py, the setup of Table IV). I attribute it to the ill-conditioning of the exact-moment problem for a singular density (§ 2 C) and in part to the test, because a sharp Maxent curve is compared with a spectrum broadened by 0.05 eV, so for resolving fine structure KPM remains the more accurate and for integrated quantities Maxent is better here. With sampled moments the sampling error dominates (Figure 1c). Table III shows what more random vectors do to Maxent with sampled moments: the L1L_1 error falls with RR (roughly as R−1/2R^{-1/2} from R=4R = 4 to 64), and the solver converges in 12–15 Newton steps for R≥4R \ge4; a single vector without the floor does not converge.

Figure 1. Accuracy on DTW4096 (Kwon TB, Γ\Gamma). (a) Band-energy error from exact moments: KPM (Jackson), Maxent with exact-moment constraints (converged cases only) and Maxent with the uncertainty floor s0=10−4s_0 = 10^{-4}. (b) Pointwise L1L_1 distance to the exact spectrum broadened by 0.05 eV. Maxent with exact-moment constraints is shown only where the solver converged. (c) Standard deviation of the band energy over 16 seeds at M=100M = 100 against the number of random vectors RR (plain Rademacher vectors, no exact-moment substitution), with the R−1/2R^{-1/2} law.

The sampling error of the band energy falls as R−1/2R^{-1/2} (Figure 1c) with ≈270\approx270 meV/atom per random vector for plain Rademacher vectors (193 at R=2R = 2, Table I); the default exact-moment substitution reduces this to ≈16\approx16 meV/atom per vector (11 at R=2R = 2). Note that comparing a sharp Maxent curve with an exact spectrum broadened by a Gaussian penalizes Maxent for sharpness; the truncation test above avoids this by comparing integrated quantities and a narrow (0.05 eV) reference.

Small systems: validity of exact-moment substitution

The test of Table I is one system with N=16 384N = 16\,384. I repeated the comparison where the argument for substitution could fail, with exact references from dense diagonalization: a periodic chain and a dimerized (gapped) chain with N=32N = 32–4096, the Kwon tight-binding Hamiltonian of cubic diamond silicon cells 232^3–535^3 (N=256N = 256, 864, 2048, 4000 orbitals, Γ\Gamma) and the three-dimensional Anderson model (W=3W = 3, N=512N = 512, 1000, 4096). The moments of random vectors were computed in the eigenbasis, which gives the same numbers as the recursion. Errors of the band energy EBSE_{\rm BS} (half filling) and of the cumulative distribution are measured against the same reconstruction fed with the exact moment list, so that truncation is common to all estimators.

Consistency. A list μ0,…,μm\mu_0,\ldots,\mu_m is the moment list of a positive measure on [−1,1][-1,1] if and only if the Toeplitz matrix Tjk=μ∣j−k∣T_{jk} = \mu_{|j-k|} is positive semidefinite. The exact lists satisfy λmin⁡≥−6×10−14\lambda_{\min} \ge-6 \times10^{-14}. Lists from one random vector or an average of 16 vectors, and impartial vectors, never violate it (all seeds, all systems). The spliced lists (K=2K = 2–16, one or 16 vectors) violate it in 80–100% of the seeds, with λmin⁡\lambda_{\min} down to −0.02-0.02 to −0.7-0.7, at an order m∗≈12m^* \approx12–40 that does not grow with NN (Figure 2); the

Figure 2. Smallest eigenvalue of the Toeplitz matrix TmT_m (median over seeds) against m/Nm/N (Kwon TB: the tight-binding Hamiltonian of Kwon et al.): exact moments and plain or impartial vectors (all curves on top of each other, never negative) and spliced lists K=4,16K = 4, 16 (negative beyond m≈12m \approx12–40).

TABLE II. Summary of the validation. “Dense” is a dense diagonalization of the same matrix. DTW4096: 4096-atom model of amorphous silicon with the Kwon tight-binding Hamiltonian (16 384 orbitals) or DFTB; “Kwon TB” is the empirical, orthogonal tight-binding total-energy Hamiltonian of Kwon et al. [36]; its spectra are those of this model and not of a first-principles calculation.

FeatureSystemReferenceQuantityObserved error
Exact local moments (K≤40K \le40)chain, dimerized chain, Kwon TB, 64–864 orbitalsdense momentsmax⁡n∣Δμn∣\max_n \lvert\Delta\mu_n\rvert≤5×10−15\le5 \times10^{-15}
KPM truncation (Jackson)DTW4096, Kwon TBdense, exact momentsEBSE_{\mathrm{BS}}; L1L_1 (0.05 eV)+0.09 to +0.30 meV/atom for M=100M = 100–400; L1=3.7×10−3L_1 = 3.7 \times10^{-3} at M=400M = 400
Maxent, exact moments, floor 10−410^{-4}DTW4096, Kwon TBdense, exact momentsEBSE_{\mathrm{BS}}; L1L_1 (0.05 eV)−0.017 to +0.02 meV/atom for M=100M = 100–400; L1=0.007L_1 = 0.007–0.012
Sampling errorDTW4096, Kwon TB, M=100M = 100denses.d. of EBSE_{\mathrm{BS}} over seeds≈270\approx270 meV/atom per Rademacher vector (193 at R=2R = 2), ∝R−1/2\propto R^{-1/2}; ≈16\approx16 per vector (11 at R=2R = 2) with K=4K = 4 (Table I)
Substitution and impartial vectors at small NNchain, dimerized chain, Kwon TB, Anderson, N=32N = 32–4000denseRMSE of EBSE_{\mathrm{BS}} (M=100M = 100, R=4R = 4)0.04–0.10 (K=4K = 4) and 0.004–0.014 (K=16K = 16) of plain vectors (Sec. III C)
Complex Hermitian (kk twist)Kwon 64 atoms, DFTB 216 atomsunion of primitive-cell bandseigenvalues; Hermiticity10−910^{-9}, 10−810^{-8} eV; 10−1210^{-12}
Matrix-free stencilKwon TB, 16316^3 cells (131 072 orbitals)exact k-sumL1L_1 (0.1 eV), M=170M = 170, R=6R = 6KPM 0.036, Maxent 0.047; Hermiticity 1.8×10−171.8 \times10^{-17}
DFTB matrices64-atom Si cell, Si–H clusterDFTB+ WriteRealHSmatrix elements; eigenvaluesHH: 7.9×10−77.9 \times10^{-7}, SS: 4.3×10−84.3 \times10^{-8} eV; eigenvalues 3×10−63 \times10^{-6} eV
Generalized, three B−1B^{-1} modesDTW4096, DFTB, Γ\Gammadense generalizedEBSE_{\mathrm{BS}}; L1L_1 (0.1 eV)identical moments in the three modes; sampling error rms 110 meV/atom at R=6R = 6, 80 at R=24R = 24 (two seeds), −0.3±7.7-0.3 \pm7.7 at R=96R = 96 (eight seeds; moment zz-scores rms 1.00, mean 0.006±0.0360.006 \pm0.036); L1=0.024L_1 = 0.024 (Table V)
SIESTA matrices; inertia countC240_{240}, C540_{540} (960, 2160 orbitals)dense eigh(HH, SS)HOMO–LUMO gap2×10−102 \times10^{-10}, 4×10−114 \times10^{-11} eV
Matrix-free HessianSi 512 atoms, ACELAMMPS dynamical_matrixmatrix, relative2.4×10−52.4 \times10^{-5}
Harmonic thermodynamicsDTW4096, ACE and SWdense dynamical matrixZPE; CvC_v (300 K); CvC_v (100 K)ACE: −0.66%, +0.27%, +1.4%; SW: −0.54%, +0.29%, +0.92% (Table VI)
Shift-invert windowDTW4096densethree levels in the gap7×10−147 \times10^{-14} eV
17317^3 replica of R10, Kwon TB4 244 832 orbitalsexact Bloch spectramax error of N(E)N(E)Maxent 0.04%, KPM 0.08% of NN (Sec. IV D)
17317^3 replica of R10, ACE3 183 624 degrees of freedomexact Bloch spectraZPE; low-frequency mode countZPE 0.03–0.07%; 0.5% of the modes below 9 meV at M=600M = 600
2 097 152-atom crystalKwon TB, 64364^3 cells, matrix-freeexact k-sum of the 8-atom cellband edges; gapedges within resolution; Maxent follows the exact edges (0.46, 1.24 eV); no L1L_1 computed
Not reproducedC3840_{3840} gapRef. [50] Fig. 3gap0.284 against 0.10 eV (open)
Not checkedcomplex generalized at k; Mulliken weights and IPR; 10610^6-atom runs——only sum rules or an internal comparison (§ 2.5, item vii)
Table 2.

Table 1.

MMRRs0=10−4s_0 = 10^{-4}s0=0s_0 = 0L1L_1
100129–0.069
100412120.046
1001614140.022
1006415150.014
300126–0.126
300412120.087
3001614140.038
3006415150.018

Table 3. TABLE III. Maxent with sampled moments on DTW4096 (Kwon TB, Γ\Gamma), no exact-moment substitution. Newton steps with the default floor s0=10−4s_0 = 10^{-4} and with s0=0s_0 = 0 (“–”: not converged in 499 steps), and L1L_1 distance to the exact spectrum broadened by 0.05 eV (floor 10−410^{-4}; the same for s0=0s_0 = 0 when it converges).

empty arcs of the spectral measure (band-edge margin, gaps) make λmin⁡\lambda_{\min} of the exact list small at low order, and the N−1/2N^{-1/2} noise of the sampled tail exceeds it.

KPM. At equal cost the root-mean-square error of EBSE_{\rm BS} (M=100M = 100, R=4R = 4) is 0.04–0.10 (K=4K = 4) and 0.004–0.014 (K=16K = 16) times that of plain Rademacher vectors, for every system and NN from 32 to 4000; impartial vectors give 0.08–0.15 (Figure 3). The estimates of EBSE_{\rm BS} have a bias that is statistically significant with 500 seeds, for plain vectors as well as spliced ones, but much smaller than the plain-vector error. Its origin is the Fermi level: with EFE_{\rm F} held at its exact value, EBSE_{\rm BS} is a linear functional of the moments and is unbiased to within two standard errors for plain vectors and for K=4K = 4, 16 (2000 seeds). The impartial vectors are unbiased at the level of one standard error for N≳500N \gtrsim500 and show an rms zz-score of 1.3–1.6 for N≲100N \lesssim100. Constraining more moments (script zscore_constrained_impartial.py, 1000–1500 vectors, rms zz of the unconstrained moments nn above the constrained ones and below 64) gives 1.0–1.5 for 4, 8 and 16 constrained moments (1.5 for the Kwon TB with 864 orbitals, 1.3–1.4 for the Anderson model with 1000 sites, about 1.0 for the Kwon TB with 2048 orbitals and for a chain with 512 sites), against 0.8–1.0 for the three constrained moments of Drabold and Sankey: a bias of at most about one standard error of the mean of the vectors at these sizes.

Figure 3. Root-mean-square error of the KPM band energy per orbital, divided by the bandwidth, against NN (M=100M = 100, R=4R = 4; 500 seeds for N≤1024N \le1024, 200 above), for plain Rademacher vectors, exact-moment substitution (K=4K = 4, 16), impartial vectors and impartial vectors with K=4K = 4.

Maxent. With sampled uncertainties from R=16R = 16 vectors, substitution with K=4K = 4 lowers the error of EBSE_{\rm BS} by 6–40 times relative to plain vectors and Maxent converges in 90–100% of the runs (K=16K = 16 needs 20–66 Newton steps, against 11–17 for plain vectors). With a single vector the uncertainty is only the floor, the inconsistent list has no solution, and Maxent with K=4K = 4 or 16 does not converge (0–20% of the runs reach the tolerance in 500 steps) and returns a band energy and a cumulative distribution that are far worse than with plain vectors, at every size tested (chain, dimerized chain, Kwon,

Anderson). Plain and impartial vectors converge with R=1R = 1. These R=1R = 1 and R=16R = 16 runs used M=100M = 100, the chain and the dimerized chain with N=64N = 64 and 512, Kwon with N=256N = 256 and 864 and Anderson with N=512N = 512, with 20 seeds (5 for the non-converging R=1R = 1 splices); the runs without the uncertainty floor and with M>NM > N were not repeated. For R=2R = 2, 3, 4 and 8 (script e2_maxent_R.py; M=100M = 100; chain and dimerized chain with N=512N = 512, Kwon with N=256N = 256 (60 seeds) and 864, Anderson with N=512N = 512, 40 seeds each unless stated) Maxent with K=4K = 4 converged in 90–100% of the runs (typically 95–98%, with no trend in RR), in 9–19 Newton steps (median), and lowered the rms error of EBSE_{\rm BS} by factors of 8–40 against plain vectors with the same RR; plain vectors converged in 95–100% of the runs. The failure is therefore specific to R=1R = 1. These runs were limited to N≤864N \leq864.

Resolution at the level spacing. For N=256N = 256 and 1024, KPM with MM up to 4N4N stays stable and its error is set by the stochastic noise, not by MM. Maxent from the exact moment list, with the default floor, needs 42–129 Newton steps for M≤NM \leq N and did not converge at M=NM = N for the Kwon system with N=256N = 256. The package warns when M≥NM \geq N and suggests dense diagonalization, which is cheaper and exact for such small problems.

Complex Hermitian problems

A Bloch twist k\mathbf{k} makes the Hamiltonian of a periodic supercell complex Hermitian, Hij=h(dij)eik⋅dijH_{ij} = h(d_{ij})e^{i\mathbf{k}\cdot d_{ij}}, and its spectrum is the union of the primitive-cell bands at the k\mathbf{k} points folded onto it. For a 64-atom Kwon supercell and a 216-atom DFTB supercell (the generalized problem), the spectrum of the twisted supercell equals this union to 10−910^{-9} and 10−810^{-8} eV, respectively, and HH is Hermitian to 10−1210^{-12}; the same builder is used in chunked form for 10610^6 atoms and agrees with the unchunked one exactly. The matrix-free operator applies HH as a stencil: the product with a vector is a sum of 128 shifted 4×44 \times4 blocks (one for each neighbour pair of the cell), so that HH is never stored. For a 16316^3-cell supercell (131 072 orbitals) at a generic twist the stencil has a Hermiticity defect of 1.8×10−171.8 \times10^{-17} and its KPM and Maxent densities (M=170M = 170, R=6R = 6) agree with the exact kk-sum with L1L_1 distances 0.036 and 0.047 against the 0.1 eV-broadened reference. For DTW4096 with the DFTB model the density at k=(b1+b2+b3)/4k = (\mathbf{b}_1+\mathbf{b}_2+\mathbf{b}_3)/4 (bi\mathbf{b}_i the reciprocal-lattice vectors of the cell; a quarter of each, so kk is not a reciprocal-lattice vector and the matrices are not gauge-equivalent to those at Γ\Gamma), a genuinely complex generalized problem, looks like the Γ\Gamma one (L1L_1 to the Γ\Gamma spectrum 0.021 against 0.024 for Γ\Gamma itself, Table V); a dense complex generalized diagonalization of 16 384 orbitals does not fit in this machine, so there is no exact reference at that kk point (at a reciprocal-lattice vector GG the twisted matrices are unitarily equivalent to those at Γ\Gamma, which would give an exact reference; I did not use such a point).

Non-orthogonal problems: DFTB and SIESTA

Comparison with DFTB+. DFTB+ [47] can write the real-space Hamiltonian HH (the zeroth-order, non-self-consistent one) and SS (WriteRealHS) for a randomly displaced 64-atom silicon cell of 10.9 Å (where pairs interact through two periodic images) the matrices built by sdos from the same .skf files agree with DFTB+ to 7.9×10−77.9 \times10^{-7} eV in HH (largest element 10.8 eV) and 4.3×10−84.3 \times10^{-8} in SS, and the generalized eigenvalues to 3×10−63 \times10^{-6} eV. A distorted Si–H cluster (which tests the heteronuclear sign convention and the use of the reversed-pair file) agrees to 7.3×10−77.3 \times10^{-7} eV and 5.5×10−85.5 \times10^{-8}. The residual is the difference between my cubic spline and DFTB+’s polynomial interpolation; in the interior of the table the two agree to 10−1010^{-10}.

DTW4096. Table V gives the DFTB problem of DTW4096: 16 384 orbitals with 179 nonzeros per row, dense generalized spectrum computed exactly (Cholesky reduction and a dense eigensolver, 210 s, about three dense matrices of 2.1 GB each), gap 0.0095 eV. The three ways of applying S−1S^{-1} give the same moments (identical band energy). The error against the dense generalized spectrum is a sampling error, not a bias. For R=6R = 6 and four seeds it is −70-70, −122-122, +69+69 and +153+153 meV/atom (rms 110), for R=24R = 24 and two seeds −3-3 and +113+113 (rms 80), and for R=96R = 96 (one run) +2+2 meV/atom (script dtw_dftb_Rscan.py). This is consistent with the R−1/2R^{-1/2} law (expected rms 55 and 27 meV/atom from the R=6R = 6 value), within the small number of runs. To test unbiasedness directly I ran eight seeds at R=96R = 96 with the Chebyshev B−1B^{-1} mode and M=100M = 100 (about 9 min each; script dtw_dftb_unbiased.py), and compared every stochastic moment with the dense generalized moment: the zz-scores (stochastic minus exact, over the sample standard error) have rms 0.82–1.22 per seed (pooled 1.00 over 792 moments) and mean 0.006 (the standard error would be 1/792=0.0361/\sqrt{792} = 0.036 for independent moments; the eight per-seed means scatter by only 0.011, so the moments are not independent and I quote no further error), and ΔEBS\Delta E_{\mathrm{BS}} (the KPM band energy at the same MM, so truncation cancels) is −0.3±7.7-0.3 \pm7.7 meV/atom (seed spread 22, range −42-42 to +25+25), consistent with zero bias and with the R−1/2R^{-1/2} law. The standard error estimated from the vectors of a single run is itself uncertain at small RR (35 to 94 meV/atom at R=6R = 6), so Table V should be read with the seed spread, not its ±\pm. For the intended use this means that R=6R = 6 vectors give the shape of the density of states but a band energy to only ≈±0.1\approx\pm0.1 eV/atom, and the energy differences between structures that matter in a workflow need either many more vectors (the cost is linear in RR) or an independent estimate of the error; the variance reductions of Table I are not available here, because exact substitution requires a diagonal BB and impartial vectors are refused for a generalized problem. Using the same random vectors for two similar structures should cancel part of the noise; I have not tested it. IC(0) halves the number of CG iterations (11 against 24 per solve) but is slower in wall-clock time because of its setup.

SIESTA. The SIESTA Harris/single-zeta matrices of the fullerenes C240_{240} and C540_{540} (960 and 2160 orbitals), generated with SIESTA 5.4.2 [45] and read through sisl, give HOMO–LUMO gaps from the inertia bisection that agree with the dense eigh⁡(H,S)\operatorname{eigh}(H,S) gaps to 2×10−102 \times10^{-10} and 4×10−114 \times10^{-11} eV (1.2995 and 0.9220 eV). For the series C60n2_{60n^2} up to C2160_{2160} the PCG needs 59 iterations per solve at every size and the cost is linear in NN; SIESTA itself ran out of memory at C3840_{3840} (12 GB) because of its N2N^2 setup, which is a reason to build HH, SS by the sparse route above.

Vibrations

The operator. For a 512-atom silicon structure with the ACE potential [52, 53] for the silicon–hydrogen system of Rosset and Deringer [54] the matrix-free products agree with the exact dynamical matrix written by LAMMPS’ dynamical_matrix command to a relative 2.4×10−52.4 \times10^{-5} (the expected O(δ2)O(\delta^2) error of the finite difference) with the same units, using two force calls per product.

DTW4096DTW4096. For the 4096-atom cell (3 rigid translations removed exactly) the matrix-free results agree with the exact dense spectra of both potentials (Table VI): for Stillinger–Weber [55] (dense matrix built column by column through the same force interface) the zero-point energy is 0.5% and Cv(300 K)C_v(300\ \mathrm{K}) 0.3% off; for ACE, whose exact matrix I obtained with LAMMPS (Kokkos on the GPU, pair_style pace product) in about ten minutes, the ZPE is −0.66%-0.66\% and Cv(300 K)C_v(300\ \mathrm{K}) +0.27%+0.27\% off, and the exact matrix has negative eigenvalues and exactly three zero modes. The largest error is the heat capacity at 100 K (+1.4%), where the result depends on the lowest frequencies, which have the poorest resolution in λ=ω2\lambda= \omega^2. Figure 4b shows the density: the acoustic band peaks near 5 THz (≈21\approx21 meV) and the optic band near 14 THz (≈58\approx58 meV). KPM and Maxent agree to four digits for ACE and to three for Stillinger–Weber.

potentialmethodZPE (meV/atom)Cv(100 K)C_v(100\ \mathrm{K})Cv(300 K)C_v(300\ \mathrm{K})Cv(600 K)C_v(600\ \mathrm{K})
SWexact (dense)66.9350.85582.29352.7863
SWsdos KPM66.5730.86372.30022.7890
SWsdos Maxent66.5460.86472.30072.7892
ACEexact (GPU dynamic_matrix)60.1040.93692.41272.8294
ACEsdos KPM59.7090.94992.41922.8319
ACEsdos Maxent59.7090.94992.41932.8319

Table VI. Harmonic thermodynamics of DTW4096 (4096 atoms, relaxed with each potential): matrix-free sdos (M=170M = 170, R=4R = 4, two force calls per product, three rigid translations removed exactly) against the exact dynamical matrix. CvC_v in kBk_{\mathrm{B}}/atom.

Figure 4. DTW4096 (a 4096-atom model of a-Si) at Γ\Gamma. (a) Electronic density of states (Kwon TB), KPM and Maxent (M=170M = 170, R=12R = 12) against the exact spectrum. (b) Vibrational density of states with the ACE potential: matrix-free KPM (M=170M = 170, R=4R = 4, two force calls per product) against the exact dynamical matrix of LAMMPS (grey).

Windows and gaps

On DTW4096 the shift-invert window around the gap returns the same three levels as the dense spectrum to 7×10−147 \times10^{-14} eV. The exact fullerene gaps from the inertia bisection are in Table VII.

CNatC_{\mathrm{Nat}}gap (eV)ReqR_{\mathrm{eq}} (Å)gap Req/ℏvFR_{\mathrm{eq}}/\hbar v_{\mathrm{F}}
601.5373.541.295
2401.0157.071.711
5400.71310.611.803
9600.54914.151.851
15000.44617.681.879
21600.37421.221.891
29400.32324.761.906
38400.28428.291.915
48600.25331.831.919
60000.22835.361.922

Table VII. Exact HOMO–LUMO gaps (inertia bisection, Xu Hamiltonian, LCBOP-relaxed icosahedral fullerenes) and the confined-Dirac-cone scaling; ℏvF=4.196\hbar v_{\mathrm{F}} = 4.196 eV Å from the graphene bands of the same Hamiltonian, Req=(NatA0/4π)1/2R_{\mathrm{eq}} = (N_{\mathrm{at}}A_{0}/4\pi)^{1/2} with A0A_{0} the graphene area per atom.

Applications

A 2097152-atom crystal of silicon

Extended Data Figure 9 of Deringer et al. [56] shows the tight-binding DOS of a cubic diamond supercell of “2097172” atoms (34.7584 nm on a side) computed with the Kwon tight-binding Hamiltonian [36], 170 moments and Maxent. A cell of 34.7584 nm is exactly 64364^3 cubic cells of 0.5431 nm, i.e. 2097152 atoms (Ref. [56], of which I am a coauthor, prints 2097172), and the matrix has 8 388 608 orbitals. Because the cell is periodic and translation invariant, the Hamiltonian is applied as a stencil (128 shifted 4×44 \times4 blocks per vector) and never stored. With M=170M = 170 and R=4R = 4 vectors the KPM density takes 603 s on 1 thread (358 s for the bounds, 246 s for the moments), with a peak of 1.8 GB; the Maxent density from the same moments takes the same time (the fit adds less than 1 s); Figure 5. The exact reference is the 8-atom cell bands on the 64364^3 mesh (exactly the Γ\Gamma spectrum of the supercell), a histogram with exactly 8 388 608 orbitals. Both methods reproduce the exact spectrum. The valence peaks are at −6-6 and −4-4 eV, the conduction features at +2+2 to +3+3 eV and the sharp upper edge at 6.7 eV, as in the published figure. The band runs from −13.5-13.5 to 6.65 eV. The methods differ most in the gap (Figure 5b). The exact band edges on this mesh are 0.46 and 1.24 eV, and Maxent follows them: its density is below 10−310^{-3} between 0.44 and 1.24 eV and below 10−410^{-4} between 0.49 and 1.20 eV. KPM at 170 moments (resolution πa/M=0.19\pi a/M = 0.19 eV, a=10.28a = 10.28 eV; FWHM 0.45 eV) fills the gap down to 10−410^{-4} (minimum 1.1×10−41.1 \times10^{-4} at 0.82 eV), a Jackson-broadened floor. For anything that depends on the gap this is the advantage of Maxent for integrated quantities, noted in Sec. III B. The same calculation takes 60 s at 32332^3 cells (262 144 atoms) and 264 s at 48348^3 cells (884 736 atoms). The time grows with the exponent 1.12 against 1.01 for the explicit matrices of § 2.5: 1.09 for the moments and 1.13 for the bounds, whose ARPACK iteration needs more products as the system grows (162, 222, 372, 392 and 522 at 838^3, 16316^3, 24324^3, 32332^3 and 48348^3 cells, against 340 for the moments). For a crystal the bounds can be given analytically.

Figure 5. Kwon tight-binding DOS of diamond silicon with 64364^3 cubic cells (2 097 152 atoms, 8 388 608 orbitals), M=170M = 170 moments, R=4R = 4 vectors, matrix-free; grey: exact kk-sum. (a) Whole band. (b) Gap region on a logarithmic scale.

Figure 9. DOS of the one-million-atom a-Si model of Morrow et al. [61, 62] with the Kwon tight-binding Hamiltonian (Γ point, 4 million orbitals). (a) KPM and Maxent, M=300M = 300, R=2R = 2, against the exact 4096-atom DTW spectrum (0.08 eV broadening). (b) Gap region on a log scale; Maxent from M=60M = 60 to 600 (R=2R = 2 and 4) and DTW4096.

Fullerenes and graphene

Drabold, Ordejón, Dong and Martin [50] followed the icosahedral fullerenes C60n2C_{60n^2} from C60C_{60} to C3840C_{3840} with moment-Maxent densities of states (“of order 150 moments”, no artificial broadening), a shifted-Lanczos HOMO–LUMO gap and local densities of states, and observed that the DOS of C2160C_{2160} and C3840C_{3840} approaches graphene; the geometries were relaxed with an order-NN tight-binding scheme [57]. I repeat the calculation with the Xu–Wang–Chan–Ho Hamiltonian [37] on geometries relaxed with the LCBOP potential [58] (which reproduces the strain energies and radii of Ref. [57] to within a few percent, while REBO and AIREBO overestimate the strain by 35–70%), generated with a Caspar–Klug (Goldberg) construction for any (m,k)(m,k). Figure 6a shows C3840C_{3840} from M=150M = 150 moments. Maxent follows the exact spectrum (a 0.1 eV Gaussian-broadened histogram of the 15 360 exact eigenvalues, computed in 124 s), including the van Hove peaks near +9+9 and +12+12 eV above EFE_F. KPM at 150 moments (resolution πa/M≈0.42\pi a/M \approx0.42 eV, a=19.9a = 19.9 eV) is visibly broader. This is the regime of the 1995 calculation, and the sharp graphene-like features it showed are reproduced. The Maxent curve in Figure 6a,Figure 6b uses the default uncertainty floor s0=10−4s_0 = 10^{-4}, with 12 random vectors the sampled errors are larger and the floor is inactive. To see what s0s_0 does here I also fitted exact moments, computed from the exact spectrum, at M=150M = 150 (green dashed line). The L1L_1 distance to the exact spectrum, broadened by 0.1 eV, is 0.064 with the floor and 0.065 without it (the plain solver converges here, in 403 steps instead of 27), against 0.070 for Maxent with the 12 random vectors and 0.099 for KPM. At this MM the floor is therefore free, and the stochastic error costs Maxent little. The exact HOMO–LUMO gap of C3840C_{3840} in the Xu model is 0.284 eV against 0.10 eV in their Figure 3 (and 1.537 against 1.62 eV for C60C_{60}). The difference may come from the different relaxations of the geometry (the cages of Ref. [50] were relaxed with the order-NN scheme of Ref. [57], mine with LCBOP), but the comparison with a sphere below argues against this, since a much larger change of geometry changes the gap by less than 1% at C3840C_{3840}; I did not resolve it, and the shifted-Lanczos gap of 1995 may not have been converged.

Figure 6. Fullerene C3840_{3840} with the Xu Hamiltonian on an LCBOP-relaxed geometry. (a,b) DOS from M=150M = 150 moments (blue: KPM; red: Maxent) against the exact eigenvalues (grey, 0.1 eV Gaussian), full band and near EFE_F; green dashed: Maxent with the floor s0=10−4s_0 = 10^{-4} from the exact moments, M=150M = 150. (c) gap Req/ℏvFR_{\mathrm{eq}}/\hbar v_F against the number of atoms; dashed: the smooth-sphere value 2.

To see what the relaxation does, the same Hamiltonian was employed on the unrelaxed spherical geometry: the Caspar–Klug construction projected onto a sphere of the graphene-equivalent radius (28.29 Å). All atoms are then at the same distance from the centre, and the bond lengths scatter between 1.13 and 1.56 Å (rms 0.10 Å). The LCBOP geometry has bonds between 1.41 and 1.45 Å (rms 0.003 Å), but its distances from the centre range from 26.9 to 31.6 Å (rms 1.1 Å): the relaxed cage is faceted. The two exact spectra of C3840 (Figure 7) have almost the same gap (0.286 and 0.284 eV), with 93 and 72 levels within 0.6 eV of EFE_F, but otherwise differ strongly. The same comparison for smaller cages (Figure 7) shows the gap of the sphere larger than the relaxed one by 10% at C240 (1.119 and 1.015 eV), 8% at C960 (0.591 and 0.549 eV) and 4% at C2160 (0.391 and 0.374 eV), and the van Hove peaks of the relaxed cages sharper at every size (those of the sphere are smeared); the band of the sphere is also wider (−42 to +25 eV against −22 to +17 eV) because of its short bonds. The sharp graphene-like features of the 1995 spectrum thus require the flat regions of a faceted cage and do not occur on a uniformly curved one. Part of the smearing on the sphere is bond-length disorder, which is the strain that the relaxation removes, so this comparison does not separate curvature from strain.

Figure 7. Exact spectra of C240_{240}, C960_{960}, C2160_{2160} and C3840_{3840} with the Xu Hamiltonian (0.1 eV Gaussian broadening): LCBOP-relaxed geometry (grey) and the unrelaxed spherical Caspar–Klug geometry (orange). The HOMO–LUMO gaps (sphere / relaxed) are given in the panel titles.

The gap, a property of the lowest levels near the Dirac point, is insensitive to both. Maxent from 150 moments (stochastic, R=12R = 12) follows the exact sphere spectrum with an L1L_1 distance of 0.118 against 0.155 for KPM.

The exact gaps (Table VII) follow gap ∝Nat−0.47\propto N_{\mathrm{at}}^{-0.47} (fit over C240–C6000; fits starting at C60, C960 and C2160 give −0.43, −0.48 and −0.48), which is the scaling of a confined massless Dirac fermion, hvF/Rh v_F/R with R∝Nat1/2R \propto N_{\mathrm{at}}^{1/2}.

The graphene bands of the same Hamiltonian have a true Dirac point (gap at K <10−14< 10^{-14} eV) with ℏvF=4.20\hbar v_{\mathrm{F}} = 4.20 eV Å (6.4×1056.4 \times10^{5} m/s). The product gap · Req/ℏvFR_{\mathrm{eq}}/\hbar v_{\mathrm{F}} rises with NatN_{\mathrm{at}} to 1.92 at C6000_{6000} (Figure 6c), close to the value 2 of a smooth sphere without pentagons, whose lowest levels are ±ℏv/R\pm\hbar v/R. The twelve pentagons carry a fictitious gauge flux that modifies the continuum spectrum [59, 60]. The 4% is therefore read as agreement of the scaling, not as a test. The weight of the states within 0.6 eV of EFE_{\mathrm{F}} on the 12 pentagon neighbourhoods (3% of the atoms), the 30 edge strips and the 20 facet interiors (shift-invert on C960_{960}, C2160_{2160}, C3840_{3840}; weights summed over each Ih_{\mathrm{h}} multiplet) shows that the pentagons matter much more than the edges: for C3840_{3840} the pentagon enhancement ranges between 0.26 and 3.4 across multiplets, the edge and interior enhancements between 0.75 and 1.3; the states stay extended (IPR ∼1/N\sim1/N). The class widths (2.5 and 3 Å) are my choice and the edge distance uses the mean radius of a faceted cage, so the ranking of the multiplets is more reliable than the numbers.

Figure 8 shows how the density of states evolves with the size of the fullerene, from C60_{60} to C150000_{150000}: the molecular levels of C60_{60} merge into bands, the van Hove peaks sharpen, and near EFE_{\mathrm{F}} the DOS of C150000_{150000} cannot be distinguished from that of graphene on the scale of the figure. For the cost, the 600 000-orbital problem C150000_{150000} takes 372 s and 1.35 GB for 1500 moments and two vectors (timing on an unrelaxed ideal-sphere geometry, valid as a cost but not as a spectrum; the DOS of Figure 8 is for the relaxed geometry and took 471 s), with 37 nonzeros per row and the exact-gap bisection capped at 25 000 orbitals because of fill.

Figure 8. Density of states of the fullerenes C₆₀ to C₁₅₀₀₀₀ (LCBOP-relaxed geometries, Xu Hamiltonian, KPM with M=1500M = 1500 moments, resolution πa/M≈0.042\pi a/M \approx0.042 eV) against the DOS of graphene from the same Hamiltonian (dashed). (a) Full band, curves offset vertically. (b) Near EFE_F.

Amorphous silicon with DFTB and ACE

Figure 4 and Table V, Table VI summarize DTW4096 as a stress test of both branches: the electronic DOS from the Kwon Hamiltonian (exact gap 0.0149 eV at Γ\Gamma) and the DFTB problem (generalized, real and complex, gap 0.0095 eV), and the vibrational DOS with the ACE potential [54] that relaxed the structure. The small gaps are properties of this T-point cell, not of the method; the zero-point energy is 60.1 meV/atom and Cv(300 K)=2.41 kBC_{\mathrm{v}}(300\ \mathrm{K}) = 2.41\ k_{\mathrm{B}}/atom. The tiling of DTW4096 into 232^{3} cells with the SW potential reproduces the single-cell ZPE per atom to 2×10−52 \times10^{-5} (67.0505 and 67.0516 meV/atom for 4096 and 32 768 atoms; matrix-free KPM with M=100M = 100, R=2R = 2 on the same SW-relaxed structure as Table VI). The same structure at M=170M = 170, R=4R = 4 gives 66.573 meV/atom (Table VI) against 66.935 exact, so the dependence on MM and RR (the M=100M = 100 run used R=2R = 2, the M=170M = 170 run R=4R = 4) is larger than the difference between the two cells.

Large amorphous models have been treated with this kind of method before: the Supplementary Information of Morrow et al. [61] shows the electronic density of states of a one-million-atom amorphous-silicon model (their Supplementary fig:3a in the arXiv version, arXiv:2308.16868), computed by the maximum-entropy method of Ref. [9] and compared for increasing numbers of moments, and a gap similar to that of a 512-atom model treated with a hybrid functional. I repeated the calculation for the same structure, using the one-million-atom main-text model deposited by those authors [62] (cubic cell of 273.6 Å, periodic, Γ\Gamma point; file aSi_1Matom_maintext.xyz) with the orthogonal Kwon tight-binding Hamiltonian: 4 million orbitals, 65 nonzeros per row. At M=300M = 300, R=2R = 2 the DOS takes 58 s (plus 58 s for the spectral bounds and 18 s to build HH) with a peak of 10.6 GB; at M=600M = 600, R=4R = 4 it takes 217 s. The band energy is −19.882±0.002-19.882 \pm0.002 and −19.887±0.003-19.887 \pm0.003 eV/atom for the two runs (Maxent from the saved moments agrees to ≤0.5\leq0.5 meV at every MM from 60 to 600, and converged in 15–16 Newton steps with the default floor, Sec. II C); the Fermi level is 0.52–0.54 eV. The DTW4096 value (−20.038-20.038 eV/atom, exact) belongs to a different structure and is not a reference for this model. Figure 9 shows the result. The spectrum has a pseudogap, not a clean gap: between 0.1 and 0.9 eV the density stays at (4(4–7)×10−37) \times10^{-3} per orbital (0.02–0.03 states/atom/eV), against ≈10−3\approx10^{-3} for DTW4096. The two runs differ by RR (2 vs 4) and MM (300 vs 600) and agree in this region to ∼15%\sim15\%, so the level is neither stochastic noise nor Chebyshev resolution (πa/M≈0.11\pi a/M \approx0.11 eV at M=300M = 300); it is a property of this structure in this Hamiltonian (with a bond cutoff of 2.85 Å the deposited structure has 6558 threefold atoms, 0.66%, in line with the 0.7% dangling bonds quoted by Morrow et al., and 16 809 fivefold atoms, 1.68%, so about 2.3% coordination defects in all). I did not test the Kwon Hamiltonian against a first-principles calculation of this structure.

A million-atom replica of a 216-atom amorphous cell

V. L. Deringer suggested a test on a periodic replica of a small defect-free amorphous model, the way Ref. [56] treats crystalline Si. I took the 216-atom model R10 of Rosset et al. (the defect-free model of Ref. [54], relaxed here with the same ACE potential) and tiled it 17317^3 times: 1 061 208 atoms, 4 244 832 orbitals, 3 183 624 vibrational degrees of freedom. This is a stringent test for a different reason than DTW4096: the exact spectrum is known for the whole replica, since for a periodic tiling it is the union of the spectra of the cell’s Bloch matrices H(k)H(k) and D(k)D(k) on the 17317^3 mesh of kk points (time-reversal symmetry halves the work). The relation is exact only while the replica stays periodic; the relaxed cell is a stable minimum (no negative eigenvalue), and a short anneal followed by minimization of a 636^3 replica returned to the same structure, so I did not anneal the large one. It also means that the large system contains no states other than those of the 216-atom cell at the 17317^3 kk points: a larger sampling of the same network, not a larger network. The electronic and vibrational tests below are therefore tests of the method at scale, not new physics of amorphous silicon.

Electronic structure. With the Kwon Hamiltonian, M=600M = 600, R=4R = 4 and no exact moments the DOS takes 229 s (17 s to build H, peak 12.2 GB). The exact gap of the replica is 1.351 eV (−0.179 to 1.172 eV). Figure 10(a) shows the gap and the tails. The exact edges are sharp, because every state is one of the 216-atom cell’s levels; the Maxent density with M=300M = 300 and 600 follows them, and drops below 10−610^{-6} per orbital within the gap except for a few narrow spikes (up to ∼ 10−410^{-4} at M=600M = 600).

DOS per orbital and vibrational-mode weights

Figure 10. 17317^3 replica of the 216-atom model R10 (1 061 208 atoms). (a) Kwon tight-binding DOS (4 244 832 orbitals, R=4R = 4) in the gap and tails, against the exact spectrum from the Bloch matrices (0.02 eV bins). (b) Exact vibrational modes of the same replica: median and 10th percentile of the weight on a sound wave of the same wave vector, and median in-cell IPR (times 216; uniform is 1), per 1 meV bin. The dotted line (7 meV) marks the step, which lies near the zone-face frequency ℏvπ/L≈6.4\hbar v\pi/L \approx6.4 meV of the 216-atom cell (see text); it is not a property of the network.

KPM, at the same moment count, has a floor of 10−510^{-5}–10−410^{-4} per orbital that rises toward the band edges, so a KPM calculation at this MM cannot distinguish the gap from a faint pseudogap, whereas Maxent can. In cumulative terms the largest error in N(E)N(E) against the exact count is 1 800 states out of 4 244 832 (0.04%) for Maxent at M=600M = 600 and 3 200 (0.08%) for KPM; these errors sit in the band tails, not inside the bands.

Vibrations. The matrix-free ACE spectrum (lammps with Kokkos/CUDA on one 12 GB GPU, two force evaluations per matrix–vector product, M=600M = 600, R=2R = 2) takes 2.2 h and 1.1 GB of host memory. The zero-point energy agrees with the exact Bloch value to 0.03% (KPM and Maxent, M=170M = 170 and 300) and 0.07% (Maxent M=600M = 600); at temperatures above 50 K CvC_v and SS agree to better than 0.1%. At 5 K, where only the lowest frequencies contribute, KPM at M=600M = 600 is off by −18%-18\% (CvC_v) and −30%-30\% (SS), Maxent by −3%-3\% and −2%-2\%. The largest error in the integrated mode count below 9 meV, where the exact replica has 25 297 modes, is 118 (KPM) and 130 (Maxent) at M=600M = 600, about 0.5%, but at M=170M = 170 it is 760 (KPM). Figure 11(a) shows the whole VDOS. KPM and Maxent follow the exact histogram over the acoustic and optic bands, including the dip at 28 meV; above about 60 meV, where the spectrum falls to zero at 68.2 meV, both oscillate about the exact curve, and Maxent more strongly (at M=600M = 600 the oscillation is off the scale of the figure, and it has a spike at the exact maximum). The fraction of modes above 65 meV is nevertheless right (1.38% for Maxent, 1.41% for KPM, 1.39% exact). Figure 11(b) divides by ω2\omega^2: below 7 meV the exact VDOS is Debye-like, with a flat g/ω2≈3×10−5 meV−3g/\omega^2 \approx3 \times10^{-5}\ \mathrm{meV}^{-3} that both methods reproduce to the resolution of the fit, and above it g/ω2g/\omega^2 rises. Below about 3 meV the reconstructions oscillate around the exact histogram, whose bins there hold only a few hundred modes. Figure 10(b) shows the character of the exact modes, measured by the weight on the plane wave eik⋅rϵe^{i\mathbf{k}\cdot\mathbf{r}} \boldsymbol{\epsilon} (summed over polarizations) with the wave vector k\mathbf{k} of the Bloch state. Below ℏω≈6\hbar\omega\approx6 meV (1.5 THz) the modes are sound waves of the cell: this weight is ≥0.99\ge0.99 (median) and the in-cell participation ratio equals 1/2161/216, as for a uniform cell. The weight falls from 0.95 to 0.03 (median) between 6–7 and 7–8 meV, and above it the modes remain extended in the cell (IPR below 1.5/216) but have no weight on the plane wave of the same k\mathbf{k}. This step is not a property of the network: it marks the Brillouin-zone boundary of the 216-atom cell. The Debye plateau of Figure 11(b) (from the exact mode counts below 3–6 meV, g/ω2=2.7g/\omega^2 = 2.7–2.8×10−5 meV−32.8 \times10^{-5}\ \mathrm{meV}^{-3}) gives ωD=(3/(g/ω2))1/3≈48\omega_D = (3/(g/\omega^2))^{1/3} \approx48 meV and, with 0.0477 atoms/Å3^3 (216 atoms in L3L^3), a mean sound velocity v≈5.1v \approx5.1 km/s (ℏv≈34\hbar v \approx34 meV Å), so the zone face at π/L=0.19\pi/L = 0.19 Å−1^{-1} (cubic cell, L=16.55L = 16.55 Å) lies at ℏvπ/L≈6.4\hbar v\pi/L \approx6.4 meV; above it the plane-wave content of a Bloch state sits at k+G\mathbf{k} + \mathbf{G}, not at k\mathbf{k}. To test this I computed the same weight for DTW4096 (cell 43.7 Å, zone face at 2.3 meV; ten k\mathbf{k} points including Γ\Gamma, 5 049 modes below 14 meV, Bloch matrices from the Gammapoint ACE dynamical matrix, each pair assigned to its minimum image, which is valid because the force cutoff is much smaller than L/2L/2; script dtw_bloch.py). The weight at the same k\mathbf{k} drops from ≈1\approx1 to a median 0.21 at 2–3 meV and to zero above 3 meV, i.e. at that cell’s zone face, whereas the weight on the best plane wave k+G\mathbf{k} + \mathbf{G} (all G\mathbf{G} within two reciprocal-lattice steps, adequate below about 9 meV) decreases gradually: the medians are 0.99, 0.97, 0.75, 0.69, 0.44, 0.25 and 0.14 in the 1 meV bins from 2–3 to 8–9 meV. The cell size, not the network, therefore sets the step in Figure 10(b), and I do not claim a propagating–diffuson crossover at 7 meV. The gradual decay of the best-plane-wave weight over roughly 4–9 meV in DTW4096 is the physical statement; I did not compute it for the replica. Eigenvectors. For a 333^3 replica (17 496 degrees of freedom) shift-invert with the Hessian assembled from finite differences found all 58 modes below 7 meV in 393 s, matching the exact eigenvalues to 10−810^{-8} (residual 3×10−143 \times10^{-14}). Shift-invert does not scale here (the ACE Hessian has about 900 nonzeros per row), so for the 17317^3 replica the modes shown in Figure 10(b) are Bloch eigenvectors of the cell, not shift-invert results.

Vibrational density of states

Figure 11. Vibrational DOS of the 17317^3 replica of R10 from the matrix-free ACE operator (M=600M = 600, R=2R = 2) against the exact Bloch levels. (a) Whole spectrum per mode; the Maxent M=600M = 600 curve, which oscillates strongly above 60 meV, is omitted, and the Maxent M=300M = 300 curve also goes off scale above about 62 meV. (b) The low-frequency region divided by ω2\omega^2 for the exact levels (gray, 1 meV bins), KPM M=600M = 600 (blue), Maxent M=300M = 300 (green, dashed) and Maxent M=600M = 600 (orange), as in (a); the dotted line (7 meV) is the same as in Figure 10.

Cost, memory and limits

Machine. All timings are from one Intel Core i5-14400F workstation (WSL2 Linux 6.18, 19 GB of memory visible, 16 logical CPUs of which at most 8 threads were used, usually 4), Python 3.14.4, NumPy 2.5.4, SciPy 1.18.1; wall-clock times of single runs. The GPU (an RTX 5070) is used for the LAMMPS ACE dynamical matrix of DTW4096 and for the matrix-free ACE vibrational spectrum of the million-atom replica (Sec. IV D). The scaling benchmarks use periodic tilings of DTW4096, i.e. not independent structures: they measure computational cost, not physics.

Orthogonal case. Table VIII and Figure 12 for the Kwon Hamiltonian on tilings up to 636^3 copies (684 736 atoms, 3 538 944 orbitals, 64 nonzeros per row) the whole DOS call (M=100M = 100, R=2R = 2) scales linearly with the number of orbitals (fitted exponent 1.013; moments 1.05, bounds 1.01, matrix build 0.97) and takes 7 s at the largest size: 17 s build, 29 s bounds, 24 s moments (100 matrix–vector products, 0.24 s each). With the default exact moments (K=4K = 4) the same call takes 383 s at the largest size (342 s for the moments, 14× the stochastic moments of the plain run), and the moments again scale linearly with the number of orbitals (exponent 1.09). Peak memory is 9.7 GB at the largest size: the CSR matrix itself is 2.7 GB, the rest is transient copies (the scaled operator and the chunk stack), and the memory exponent 0.79 I attribute to the fixed interpreter and library baseline at small sizes (I did not profile this); a leaner assembly would lower the peak. The matrix-free stencil avoids storing the matrix at all (8.4 million orbitals in 1.8 GB, Sec. IV A) but needs a translation-invariant structure.

atomsorbitalsbuild (s)bounds (s)moments (s)total (s)peak mem. (GB)moments, K=4K = 4 (s)total, K=4K = 4 (s)
4 09616 3840.10.10.10.30.151.01.2
32 768131 0720.61.00.92.50.458.610.0
110 592442 3682.13.62.88.51.231.836.6
262 1441 048 5764.98.97.121.03.192.3104.2
512 0002 048 0009.717.414.441.55.5177.8201.0
884 7363 538 94417.329.424.170.89.7341.8382.5

Table VIII. Orthogonal case: Kwon tight binding on periodic tilings of DTW4096, KPM with M=100M = 100 moments and R=2R = 2 vectors, 4 threads; columns 3–7: no exact-moment substitution (K=0K = 0); last two columns: the default K=4K = 4 (moments, and the whole call including build and bounds), from a separate run. Timings are one run each.

Figure 12. Cost scaling on tilings of DTW4096 (M=100M = 100, R=2R = 2). (a) Wall time against the number of orbitals: orthogonal Kwon (total and moments only), non-orthogonal DFTB with PCG/block-Jacobi and with the Chebyshev B−1B^{-1}; dotted: linear scaling. (b) Peak memory.

Dependence on sparseness. The moments cost about nnz×M×R/2\mathrm{nnz} \times M \times R/2 multiply–adds (the doubling identities give two moments per product), so the time is linear in the number of nonzeros (Table IX). From 15 to 255 nonzeros per row I measure 0.8–0.9 ns per nonzero, moment and vector (the entry “ns per (nnz MR)(\mathrm{nnz}\,MR)” of the table is per moment, half the time per multiply–add, i.e. 1.6–1.8 ns per multiply–add), and up to 1.8× more at 3 per row, where the per-row overhead matters. This is a bare recursion on a random matrix without locality; the Kwon calculation above (0.24 s per product at 64×3.5×10664 \times3.5 \times10^6 nonzeros and R=2R = 2) corresponds to ≈0.5\approx0.5 ns, with the whole call also paying for the bounds and the build. Sparseness does not change the stochastic error, which falls as (NR)−1/2(NR)^{-1/2}; the number of moments is set by the bandwidth divided by the resolution. Maxent after the moments costs the same at any sparsity. Dense exact diagonalization scales as N3N^3 and should win only for small or nearly full matrices with MRMR comparable to NN; I did not time that case.

nonzeros per row37153163127255
time (s)0.060.090.130.260.531.112.28
ns per (nnz MRMR)1.471.070.800.800.820.870.89

Table IX. Time of the Chebyshev recursion against the number of nonzeros per row (random symmetric sparse matrix, N=50000N = 50000, M=100M = 100, R=2R = 2 Rademacher vectors, 4 threads; bare recursion: no bounds, no reconstruction).

Non-orthogonal case. Table X: DFTB has 179 nonzeros per row (cutoff 6.0 Å, 4 orbitals per atom) and a Hamiltonian and an overlap matrix, so the peak memory at the same number of orbitals is 2.3 to 3.5 times that of the orthogonal case with PCG and 2.3 to 4.7 times with the Chebyshev polynomial over the four sizes (at the largest, 3.2 and 4.5; ratios from the unrounded peaks in the benchmark files). The time of the whole DOS call grows with the exponent 0.97 (PCG, block-Jacobi, four sizes up to 1 048 576 orbitals) or 0.94 (Chebyshev polynomial, four sizes) and the CG iteration count per solve is independent of size (23.7 at 16 384 orbitals, 23.9 at 442 368 and at 1 048 576), as it should be for a fixed local preconditioner of the well-conditioned overlap. IC(0) reduces the iterations to 11 per solve, but my incomplete Cholesky is a Python loop per column and takes longer overall, so I ran it only for the small sizes. A compiled factorization would change that conclusion. A million-atom non-orthogonal DFTB problem does not fit in 19 GB as explicit HH and SS: at 262 144 atoms the PCG run peaked at 9.8 GB and the Chebyshev run at 14.0 GB (912 s, under a 16 GB cap; the first attempt, under a 10 GB cap, was killed), so four times that size would need 40–56 GB.

atomsorbitalsnnz/rowbuild (s)PCG (BJ) (s)Chebyshev (s)PCG (IC(0)) (s)peak mem., PCG (GB)peak mem., Cheb. (GB)
4 09616 38417911518350.350.35
32 768131 07217941171422881.41.8
110 592442 36817915380398–4.35.8
262 1441 048 57617926806912–9.814.0

Table X. Non-orthogonal case: DFTB (pbc-0-3, non-self-consistent) HH and SS on tilings of DTW4096, KPM with M=100M = 100, R=2R = 2; time of the whole density-of-states call for the three ways of applying S−1S^{-1} (PCG with block-Jacobi, Chebyshev polynomial, PCG with IC(0)). IC(0) was run only for the small sizes (its setup is a Python loop). At 262 144 atoms the Chebyshev row is a rerun under a 16 GB cap (peak 14.0 GB; the first attempt, under a 10 GB cap, was killed); the PCG row at that size was recovered from the log of the first run (peak 9.8 GB), and the build time is that of the rerun.

Vibrations. A matrix-free product costs two force evaluations. With Stillinger–Weber on one CPU thread a 12 288-degree-of-freedom model takes 5 ms per force call (11 s for M=100M = 100, R=2R = 2 and the bounds) and 98 304 degrees of freedom 37 ms per call (191 s, 5124 calls including the bounds), i.e. linear in the force cost. Only about 200 of these calls belong to the moments (M=100M = 100, R=2R = 2); about 4900 (some 2450 Lanczos products) go to the spectral bounds and the Hermiticity probes. In the 4096-atom run the bounds took 9.0 s of 9.9 s and the 102 products of the moments 0.8 s, so the bounds dominate the cost of a matrix-free vibrational density of states at these sizes. I did not investigate why ARPACK needs so many products here; it is not the finite-difference step (changing it from 10−310^{-3} to 10−510^{-5} Å changes the count of force calls for the bounds of the 4096-atom model only from 2124 to 2584), and the window found is correct (Stillinger–Weber, [0, 1.844]). With ACE the force call dominates: on CPU it takes 0.73 s for 4096 atoms (4 threads) and the matrix-free DOS of DTW4096 took 1306 s for 1768 force calls; with a Kokkos/CUDA build of LAMMPS on one 12 GB GPU through its Python module the million-atom replica of Sec. IV D (1220 force calls for M=600M = 600, R=2R = 2, with the spectral window taken from the exact force-constant spectrum) took 2.2 h, i.e. about 6.6 s per call including the transfer of positions and forces through Python, and 1.1 GB of host memory.

What does not scale. (i) Shift–invert and the exact HOMO–LUMO bisection are not O(N)O(N) because of fill; for the Xu fullerenes the bisection takes 27 s at 15 360 orbitals and 62–80 s at 19 440–24 000, and the exact-gap bisection costs more than the DOS above ∼2×104\sim2 \times10^{4} orbitals. (ii) Exact moments of order KK cost a time that grows quickly with KK (1.8, 9 and 80 s per run at K=4K = 4, 8, 16 for 16384 orbitals), although it is linear in NN for fixed KK. (iii) The spectral bounds are not negligible and grow with NN (162 to 522 matrix–vector products from 16 384 to 3.5 × 10610^{6} orbitals of the stencil, against 340 for the moments with M=170M = 170, R=4R = 4); for a crystal they can be given analytically. For a matrix-free operator they can also be wrong: on a relaxed one-million-atom ACE Hessian (the Morrow et al. model of Sec. IV C, whose vibrational density of states is not part of this report; finite-difference step 10−310^{-3} Å) the Lanczos window came out as [-29, 69] in units of eV Å−2^{-2} amu−1^{-1}, whereas the spectrum lies in about [0, 1.34] (a plain 30-step Lanczos iteration with full reorthogonalization and a step of 10−410^{-4} Å converges to extreme Ritz values 0.0035 and 1.336; the exact ranges of the 216-atom cell replica and of DTW4096 are [0, 1.11] and [0, 1.27]). The cause is the finite-difference step: the displacement is normalized to an rms of 10−310^{-3} Å over all 3N3N components, so a Lanczos vector that localizes on a few atoms is displaced by a large amount per atom, the product is then not that of a fixed linear operator, and the Ritz values leave the spectrum (the same iteration with the step of 10−310^{-3} Å reaches [-5.3, 20] after 20 steps). I inferred the localization from this step dependence and did not measure it. With a window of [-29, 69] and M=600M = 600 moments the resolution is 0.26, a quarter of the band, and the result is useless; the symptom is a density with weight at λ<0\lambda< 0 (the result is flagged negative_indicated) and a window much wider than the physical range; a window that is too narrow is as bad (the moments of a run with [-0.05, 1.3] grew to 1017910^{179}). For matrix-free vibrational problems I therefore recommend estimating the window with a smaller step (here 10−410^{-4} Å) or passing bounds from a known range (a crystal, or a small cell of the same potential), and checking the window before a long run; the runs of this report that used it (Sec. IV D) took the window from the exact spectrum of the force-constant matrix. (iv) Max-ent with many exact moments stops converging unless the moments are given a small uncertainty (Sec. III B). Even then its pointwise L1L_1 error against the 0.05 eV-broadened exact spectrum is smallest at M=170M = 170 and rises at M=300M = 300 and 400, while KPM’s keeps falling (Table IV). The floor is not the cause (Sec. III B); part of it is the test (a sharp Maxent curve is compared with a broadened spectrum), and for the band energy, an integrated quantity, Maxent is the more accurate at every MM in Table IV. I do not claim that a converged Maxent is worse than KPM in general: for integrated quantities it is better, and only for pointwise resolution at large MM do I recommend KPM. (v) The DFTB matrices are large: 179 nonzeros per row. (vi) The statistical error of a moment decreases with NR\sqrt{NR}, so large systems need few vectors, but the band energy per atom of one 4096-atom cell with R=6R = 6 vectors has an uncertainty of order 0.1 eV/atom (Table V; rms 110 meV/atom at R=6R = 6) unless exact moments or impartial vectors are used (Table I), and for the generalized problem the variance reductions of Table I are not available; integrated quantities should be reported with their error. (vii) Not checked against an independent reference: the complex generalized problem

lem at a kk point (Sec. III D), the generalized problem through the result file (§ 2 G), the Mulliken weights and IPR (tested through sum rules and one shift-invert case), and the one-million-atom amorphous run. (viii) In the replica of Sec. IV D Maxent oscillates strongly where the vibrational spectrum ends abruptly (above 60 meV at M=600M = 600), although the integrated weight is right; shift-invert is practical only for small replicas, and the exact references there exist because the replica is periodic. (ix) Finally, all numbers are Γ\Gamma-point or single-twist values for one structure each, except the crystal and replica comparisons, whose references are exact kk-sums; the dependence on the structure and the cell size is a separate question.

Recommendations

  1. KPM is the default: linear, positive (Jackson), never fails, resolution πa/M\pi a/M; for the maximum resolution of a large system increase MM (a few thousand is cheap; cost is linear in MM). Beyond the

mean level spacing or the noise floor, more moments only sharpen sampling speckle.

  1. Maxent (keep the default uncertainty floor, which makes it converge with exact or nearly exact moments) when high-order moments are expensive or noisy (M≲300M \lesssim300), when positivity and sharp edges matter, and for integrated quantities (band energy, gap region); from identical exact moments at M=20M = 20–7070 (the sizes where it converged) it gave 8–12× lower band-energy errors and 6–30× lower L1L_1 distances (Table IV). Use the bootstrapped solver; with the default uncertainty floor it converges also on exact moments, but its pointwise error, measured against a broadened exact spectrum, rises beyond a few hundred moments of singular densities (Table IV).

  1. Random vectors: the default exact-moment substitution (K=4K = 4) is almost always worth it for KPM, and for Maxent with R≥2R \ge2 vectors; with Maxent and a single vector use K=0K = 0 or impartial vectors (Sec. III C); K=8K = 8–1616 for band energies; impartial vectors constrain only three moments and are nearly as efficient at lower cost per vector (Table I).

  1. Generalized problems: PCG with block-Jacobi is the default and my plain recommendation; the Chebyshev polynomial needs no inner solves, which makes it the option for matrix-free or GPU settings where no iterative-solver stack is wanted, but in Table X it was slower and needed more memory than PCG; IC(0) lowers the iteration count only if its setup is cheap (mine is not).

  1. Report the sampling error of every integrated quantity, and overlay KPM and Maxent: where they agree, the feature is in the moments.

Conclusions and availability

SDOS computes densities of states and related spectral quantities of very large sparse Hermitian and generalized problems. The ingredients are established: stochastic traces, KPM, Maxent, and non-orthogonal KPM by PCG. What is added is their combination, the choices for the overlap inverse, bootstrapped Maxent, windows, projections, the interfaces, and a validation against exact diagonalization and independent codes from 10310^3 to 8×1068 \times10^6 orbitals, including the exact Bloch spectra of a million-atom replica of a 216-atom amorphous cell, for which Maxent resolves the gap edges that KPM smears and the matrix-free ACE vibrational spectrum reproduces the zero-point energy to 0.07%. The moment-based computations scale linearly in the number of orbitals for explicit matrices (exponent 1.01) and nearly so for the matrix-free crystal (1.12). Three conventions of the DFTB file format that I did not find in the documentation available to me are recorded here. The limits are stated in § 2.5.

Availability. The package is licensed under BSD-3-Clause (copyright D. A. Drabold, 2026) and is available at https://github.com/Drabold/sdos and archived on Zenodo (version 0.1.0, concept DOI 10.5281/zenodo.24023874); it installs with pip install sdos (PyPI) or from a clone with pip install -e .. The repository contains the examples and benchmark scripts of this report; the large run data (million-atom structures and exact spectra) and the DFTB parameter files are not included. All scripts, saved results and the generation of the figures and tables of this report are in the examples/ and paper/ directories of the repository; the tests run with pytest.

Acknowledgments

AI assistance: the package code, the tests and the benchmark and analysis scripts were written, and the text of this report was drafted, with Claude Sonnet 5.5 and Claude Opus 5.5 (Anthropic), used through Claude Code. I reviewed and validated all AI-assisted output and made the design decisions; I am responsible for all scientific statements. I acknowledge the Frauenheim group’s pbc-o-3 Slater–Koster set (CC BY-SA 4.0), whose required references I cite. I thank R. N. Silver, H. Röder, J. J. Dong, P. Ordéjon and R. M. Martin for their insights and collaboration, C. Ugwumadu for a careful reading of the whole manuscript and many helpful suggestions, E. T. Drabold for advice on hardware selection, V. L. Deringer for suggesting the million-atom replica test, and L. A. M. Rosset and V. L. Deringer for the silicon–hydrogen ACE potential used in the benchmarks.

References

References

  1. [1]J. Skilling, The eigenvalues of mega-dimensional matrices, in Maximum Entropy and Bayesian Methods (Springer, Dordrecht, 1989) pp. 455–466.DOI
  2. [2]M. Hutchinson, A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines, Communications in Statistics - Simulation and Computation 19, 433 (1990).
  3. [3]R. N. Silver and H. Röder, Densities of states of mega-dimensional Hamiltonian matrices, International Journal of Modern Physics C 5, 735 (1994).DOI
  4. [4]R. Silver, H. Roeder, A. Voter, and J. Kress, Kernel polynomial approximation for densities of states and spectral functions, Journal of Computational Physics 124, 115 (1996).DOI
  5. [5]A. Weisse, G. Wellein, A. Alvermann, and H. Fehske, The kernel polynomial method, Reviews of Modern Physics 78, 275 (2006).
  6. [6]D. Jackson, On approximation by trigonometric sums and polynomials, Transactions of the American Mathematical Society 13, 491 (1912).DOI
  7. [7]L. R. Mead and N. Papanicolaou, Maximum entropy in the problem of moments, Journal of Mathematical Physics 25, 2404 (1984).DOI
  8. [8]I. Turek, A maximum-entropy approach to the density of states within the recursion method, Journal of Physics C: Solid State Physics 21, 3251 (1988).DOI
  9. [9]D. A. Drabold and O. F. Sankey, Maximum entropy approach for linear scaling in the electronic structure problem, Physical Review Letters 70, 3631 (1993).
  10. [10]E. T. Jaynes, Where do we stand on maximum entropy?, in The Maximum Entropy Formalism, edited by R. D. Levine and M. Tribus (MIT Press, Cambridge, MA, 1978) pp. 15–118.
  11. [11]S. Goedecker, Linear scaling electronic structure methods, Reviews of Modern Physics 71, 1085 (1999).
  12. [12]P. Ordejón, Order-n tight-binding methods for electronic-structure and molecular dynamics, Computational Materials Science 12, 157 (1998).
  13. [13]R. Haydock, V. Heine, and M. J. Kelly, Electronic structure based on the local atomic environment for tight-binding bands, Journal of Physics C: Solid State Physics 5, 2845 (1972).
  14. [14]L.-W. Wang, Calculating the density of states and optical-absorption spectra of large quantum systems by the plane-wave moments method, Physical Review B 49, 10154 (1994).
  15. [15]S. Goedecker and L. Colombo, Efficient linear scaling algorithm for tight-binding molecular dynamics, Physical Review Letters 73, 122 (1994).
  16. [16]L. Lin, Y. Saad, and C. Yang, Approximating spectral densities of large matrices, SIAM Review 58, 34 (2016).
  17. [17]H. Röder, R. N. Silver, and J. D. Kress, The kernel polynomial method for non-orthogonal electronic structure calculations, in High Performance Computing 1996: Grand Challenges in Computer Simulation, edited by A. Tentner (Society for Computer Simulation, San Diego, 1996) report CONF-960482, OSTI ID 390779.
  18. [18]H. Röder, R. N. Silver, D. A. Drabold, and J. J. Dong, Kernel polynomial method for a nonorthogonal electronic-structure calculation of amorphous diamond, Physical Review B 55, 15382 (1997).
  19. [19]S. M. João, M. Andelković, L. Covaci, T. G. Rapopport, J. M. V. P. Lopes, and A. Ferreira, KITE: high-performance accurate modelling of electronic structure and response functions of large molecules, disordered crystals and heterostructures, Royal Society Open Science 7, 191809 (2020).
  20. [20]C. W. Groth, M. Wimmer, A. R. Akhmerov, and X. Waintal, Kwant: a software package for quantum transport, New Journal of Physics 16, 063065 (2014).
  21. [21]D. Moldovan, M. Andelković, and F. Peeters, Pybinding v0.9.5: a Python package for tight-binding calculations (2020).
  22. [22]Y. Li, Z. Zhan, X. Kuang, Y. Li, and S. Yuan, TBPLaS: A tight-binding package for large-scale simulation, Computer Physics Communications 285, 108632 (2023).
  23. [23]O. F. Sankey, D. A. Drabold, and A. Gibson, Projected random vectors and the recursion method in the electronic-structure problem, Physical Review B 50, 1376 (1994).
  24. [24]A. F. Voter, J. D. Kress, and R. N. Silver, Linear-scaling tight binding from a truncated-moment approach, Physical Review B 53, 12733 (1996).
  25. [25]R. N. Silver and H. Röder, Calculation of densities of states and spectral functions by chebyshev recursion and maximum entropy, Physical Review E 56, 4822 (1997).
  26. [26]J. M. Tang and Y. Saad, A probing method for computing the diagonal of a matrix inverse, Numerical Linear Algebra with Applications 19, 485 (2012).
  27. [27]R. A. Meyer, C. Musco, C. Musco, and D. P. Woodruff, Hutch++: Optimal stochastic trace estimation, in Symposium on Simplicity in Algorithms (SOSA) (2021) pp. 142–155.DOI
  28. [28]L. Fejér, Untersuchungen über Fouriersche Reihen, Mathematische Annalen 58, 51 (1904).DOI
  29. [29]R. Courant and D. Hilbert, Methods of Mathematical Physics, Vol. 1 (Interscience, New York, 1953).
  30. [30]J. A. Meijerink and H. A. van der Vorst, An iterative solution method for linear systems of which the coefficient matrix is a symmetric m-matrix, Mathematics of Computation 31, 148 (1977).
  31. [31]T. A. Manteuffel, An incomplete factorization technique for positive definite linear systems, Mathematics of Computation 34, 473 (1980).
  32. [32]Y. Zhou, Y. Saad, M. L. Tiago, and J. R. Chelikowsky, Self-consistent-field calculations using chebyshev-filtered subspace iteration, Journal of Computational Physics 219, 172 (2006).
  33. [33]R. S. Mulliken, Electronic population analysis on lcaomo molecular wave functions. i, The Journal of Chemical Physics 23, 1833 (1955).
  34. [34]C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, et al., Array programming with NumPy, Nature 585, 357 (2020).
  35. [35]P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, et al., SciPy 1.0: fundamental algorithms for scientific computing in python, Nature Methods 17, 261 (2020).
  36. [36]I. Kwon, R. Biswas, C. Z. Wang, K. M. Ho, and C. M. Soukoulis, Transferable tight-binding models for silicon, Physical Review B 49, 7242 (1994).pubmed.ncbi.nlm.nih.gov/10009461
  37. [37]C. H. Xu, C. Z. Wang, C. T. Chan, and K. M. Ho, A transferable tight-binding potential for carbon, Journal of Physics: Condensed Matter 4, 6047 (1992).DOI
  38. [38]T. Frauenheim, The pbc–o–3 slater–koster parameter set, Universität Bremen, CC BY-SA 4.0 (dftb.org) (2017).
  39. [39]A. Sieck, Doctoral thesis, Ph.D. thesis, University of Paderborn (2000), cited as required by the README of the pbc-o-3 Slater–Koster set.
  40. [40]A. Sieck, T. Frauenheim, and K. A. Jackson, Shape transition of medium-sized neutral silicon clusters, physica status solidi (b) 240, 537 (2003).
  41. [41]D. Porezag, T. Frauenheim, T. Köhler, G. Seifert, and R. Kaschner, Construction of tight-binding-like potentials on the basis of density-functional theory: Application to carbon, Physical Review B 51, 12947 (1995).
  42. [42]M. Elstner, D. Porezag, G. Jungnickel, J. Elsner, M. Haugk, T. Frauenheim, S. Suhai, and G. Seifert, Self-consistent-charge density-functional tight-binding method for simulations of complex materials properties, Physical Review B 58, 7260 (1998).
  43. [43]N. Papior, sisl: a python package for tight-binding, dft and siesta post-processing, https://github.com/zerothi/sisl (2026), software; accessed 2026.
  44. [44]J. M. Soler, E. Artacho, J. D. Gale, A. García, J. Junquera, P. Ordejón, and D. Sánchez-Portal, The SIESTA method for ab initio order- n materials simulation, Journal of Physics: Condensed Matter 14, 2745 (2002).DOI
  45. [45]A. García, N. Papior, A. Akhmerov, E. Artacho, V. Blum, E. Bosoni, et al., Siesta : Recent developments and applications, The Journal of Chemical Physics 152, 124101 (2020).
  46. [46]A. P. Thompson, H. M. Aktulga, R. Berger, D. S. Bolintineanu, W. M. Brown, P. S. Crozier, et al., LAMMPS - a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales, Computer Physics Communications 271, 108171 (2022).DOI
  47. [47]B. Hourahine, B. Aradi, V. Blum, F. Bonafé, A. Buccheri, C. Camacho, et al., DFTB+, a software package for efficient approximate density functional theory based atomistic simulations, The Journal of Chemical Physics 152, 124101 (2020).
  48. [48]F. Wooten, K. Winer, and D. Weaire, Computer generation of structural models of amorphous Si and Ge, Physical Review Letters 54, 1392 (1985).pubmed.ncbi.nlm.nih.gov/10031020
  49. [49]B. R. Djordjević, M. F. Thorpe, and F. Wooten, Computer model of tetrahedral amorphous diamond, Physical Review B 52, 5685 (1995).pubmed.ncbi.nlm.nih.gov/9981754
  50. [50]D. A. Drabold, P. Ordejón, J. Dong, and R. M. Martin, Spectral properties of large fullerenes: From cluster to crystal, Solid State Communications 96, 833 (1995).DOI
  51. [51]P. N. Keating, Effect of invariance requirements on the elastic strain energy of crystals with application to the diamond structure, Physical Review 145, 637 (1966).DOI
  52. [52]R. Drautz, Atomic cluster expansion for accurate and transferable interatomic potentials, Physical Review B 99, 014104 (2019).DOI
  53. [53]Y. Lysogorsky, C. v. d. Oord, A. Bochkarev, S. Menon, M. Rinaldi, T. Hammerschmidt, et al., Performance implementation of the atomic cluster expansion (PACE) and application to copper and silicon, npj Computational Materials 7, 97 (2021).
  54. [54]L. A. M. Rosset and V. L. Deringer, Atomic cluster expansion potential for the Si-H system, Physical Review Materials 10, 044604 (2026).arxiv.org/abs/2510.15633
  55. [55]F. H. Stillinger and T. A. Weber, Computer simulation of local order in condensed phases of silicon, Physical Review B 31, 5262 (1985).pubmed.ncbi.nlm.nih.gov/9936488
  56. [56]V. L. Deringer, N. Bernstein, G. Csányi, C. Ben Mahmoud, M. Ceriotti, M. Wilson, D. A. Drabold, and S. R. Elliott, Origins of structural and electronic transitions in disordered silicon, Nature 589, 59 (2021).DOI
  57. [57]S. Itoh, P. Ordejón, D. A. Drabold, and R. M. Martin, Structure and energetics of giant fullerenes: An order-molecular-dynamics study, Physical Review B 53, 2132 (1996).pubmed.ncbi.nlm.nih.gov/9983678
  58. [58]J. H. Los and A. Fasolino, Intrinsic long-range bond-order potential for carbon: Performance in monte carlo simulations of graphitization, Physical Review B 68, 024107 (2003).
  59. [59]J. González, F. Guinea, and M. A. H. Vozmediano, Continuum approximation to fullerene molecules, Physical Review Letters 69, 172 (1992).pubmed.ncbi.nlm.nih.gov/10046217
  60. [60]J. González, F. Guinea, and M. A. H. Vozmediano, The electronic spectrum of fullerenes from the Dirac equation, Nuclear Physics B 406, 771 (1993).DOI
  61. [61]J. D. Morrow, C. Ugwumadu, D. A. Drabold, S. R. Elliott, A. L. Goodwin, and V. L. Deringer, Understanding defects in amorphous silicon with million-atom simulations and machine learning, Angewandte Chemie International Edition 63, e202403842 (2024).
  62. [62]J. D. Morrow, C. Ugwumadu, D. A. Drabold, S. R. Elliott, A. L. Goodwin, and V. L. Deringer, Research data for “understanding defects in amorphous silicon with million-atom simulations and machine learning”, Zenodo (2024), doi:10.5281/zenodo.10794053.

Paper details

vlnc:W_p74f8rnm
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Claude Sonnet 5.5, Claude Opus 5.5

View statement
Looking for related papers…
Contents