In this paper
AbstractIntroductionMethodsMoments from random vectorsExact low-order moments and impartial vectorsReconstruction: KPM and maximum entropyGeneralized problemsEigenvectors in spectral windows, exact gaps and projectionsVibrationsInterfaces and implementationValidationUnit testsAccuracy against the number of moments and vectorsSmall systems: validity of exact-moment substitutionComplex Hermitian problemsNon-orthogonal problems: DFTB and SIESTAVibrationsWindows and gapsApplicationsA 2097152-atom crystal of siliconFullerenes and grapheneAmorphous silicon with DFTB and ACEA million-atom replica of a 216-atom amorphous cellCost, memory and limitsRecommendationsConclusions and availabilityAcknowledgmentsReferencesWorksPhysics and Astronomy
sdos: spectral densities of large sparse Hermitian and generalized eigenproblems
Abstract
sdos is an open Python package that computes the density of states (DOS) and related spectral quantities of very large sparse Hermitian matrices A, and of generalized problems A psi = lambda B psi, from Chebyshev moments estimated with random vectors. The density is reconstructed by the kernel polynomial method (KPM, the default) or by maximum entropy (Maxent); the same interface gives band energies, Fermi levels, local and projected densities, harmonic thermodynamics, exact HOMO-LUMO gaps and eigenpairs in a spectral window. Interfaces cover electronic (empirical tight-binding, DFTB, SIESTA) and vibrational (matrix-free LAMMPS Hessians) problems. I checked most ingredients against dense diagonalization or an independent code; the exceptions are listed in Sec. V. On a workstation (4 threads, 19 GB) the DOS of an 884,736-atom amorphous silicon model (a 6^3 tiling of a 4096-atom cell; 3.5 million orbitals; the orthogonal empirical tight-binding model of Kwon et al.) takes 71 s (M=100, R=2, no exact moments; 383 s with the default exact moments), and a matrix-free 2,097,152-atom crystal 10 min in 1.8 GB. A 17^3 replica of a 216-atom amorphous cell (1,061,208 atoms) agrees with exact Bloch spectra: electronic DOS in 229 s, matrix-free ACE vibrational DOS in 2.2 h on one GPU. Shift-invert, incomplete-Cholesky setup, exact moments of high order and Maxent at high moment counts do not scale. The underlying methods are due to Silver, Roder, Kress, Voter, Drabold, Sankey and collaborators; the Introduction states what is added.
Introduction
The method of this report applies to any large sparse Hermitian matrix (and to any matrix-free linear operator, and to the generalized problem with positive definite). It computes the spectral density , where is the energy (the spectral variable) and the dimension, and quantities derived from it. Nothing in the algorithm depends on where the matrix comes from. I illustrate the method with electronic and vibrational problems, because I know them, because exact references exist for them, and because the density of states determines band energies, Fermi levels and thermodynamics and, through its local variants, much of the interpretation of electronic and vibrational structure in disordered and nanostructured materials. These applications are examples of a more general tool. For matrices of dimension –, as arise for amorphous networks, nanostructures and large supercells, diagonalization ( time, memory) is out of reach. The moments of with respect to Chebyshev polynomials of the scaled operator, on the other hand, cost operations each. They come from matrix-vector products with a few random vectors, because the trace can be replaced by a stochastic estimator [1, 2]. Two routes lead from moments to a density. The kernel polynomial method (KPM) [3, 4, 5] is a damped Chebyshev series; it is linear, easy to use, and positive with the Jackson kernel [6]. The principle of maximum entropy [7, 8, 9] finds the positive density of maximal entropy that reproduces the moments. It has an information-theoretic foundation [10] and, for the same moments, resolves sharp features better. Both belong to the family of linear-scaling electronic-structure methods reviewed by Goedecker [11] and Ordejón [12]; related moment methods are the recursion method [13], the plane-wave moments method [14] and Fermi-operator expansions [15], and modern surveys of stochastic spectral-density estimation are in Ref. [16].
Electronic problems in a non-orthogonal basis and vibrational problems with unequal masses are generalized eigenproblems, . Röder, Silver and Kress [17] showed that KPM extends to this case by applying with a preconditioned conjugate-gradient (PCG) solve of for every Chebyshev step, and demonstrated it for a titanate/platinum interface and amorphous diamond; Röder, Silver, Drabold and Dong [18] treated amorphous diamond in the same way. SDOS implements this idea and adds alternatives for applying (block-Jacobi, incomplete Cholesky, algebraic multigrid, and a fixed Chebyshev polynomial with , which replaces the inverse by a matrix polynomial so that the recursion is applied to one fixed, explicit operator and needs no inner solves), a Maxent reconstruction, exact low-order moments, spectral windows, projections and a set of interfaces to tight-binding and force-field codes. Open tight-binding codes for large systems, such as KITE [19], Kwant (with its kp module) [20], pybinding [21] and TB-PLAS [22], use Chebyshev-based methods (KPM, or time propagation in TBPLAS) for orthogonal tight-binding models; I did not find an overlap matrix in their documentation or papers. What is offered here is an open, tested implementation and a quantitative validation, and I want to be clear about what is old. Chebyshev moments, KPM and Maxent for densities of states, impartial random vectors, locally truncated (exact) low-order moments and non-orthogonal KPM with PCG all come from the work of Silver, Röder, Voter, Kress, Drabold, Sankey and collaborators [3, 4, 9, 17, 18, 23, 24, 25]. Silver and Röder [25], in particular, fitted up to 1024 exact Chebyshev moments by maximum entropy for a tight-binding Hamiltonian of amorphous diamond with 16 384 states; their prescription is described in § 2 C. Voter, Kress and Silver [24] obtain low-order moments from locally truncated Hamiltonians, which, as I understand it, is the same local computation as my exact low-order moments. What I believe is new here is more limited: (i) using these exact moments up to order together with sampled moments above in one estimator, with an argument for when it is unbiased and a measurement of when it is not useful (§ 2 B and III C); (ii) a Maxent solver with a moment-uncertainty floor and a staged (bootstrapped) climb in the number of moments, which I tested against exact spectra but not against the Silver–Röder prescription; (iii) alternatives for the overlap inverse; and (iv) the open implementation, with chunked builders and matrix-free operators for atoms and more. I may have missed earlier descriptions of (i) and (ii). Each section says what is established elsewhere and what was measured here.
§ 2 describes the methods; Sec. III compares with exact results; Sec. IV gives four applications (a 2 097 152-atom crystal, the fullerene-to-graphene series, an amorphous Si model with DFTB and ACE, and a million-atom replica of a 216-atom cell); § 2.5 treats cost, memory and limits; Sec. VI gives recommendations and Sec. VII the conclusions and availability.
Methods
Moments from random vectors
Let be Hermitian with spectrum in , and with the interval widened by a margin () on each side, , , so that the spectrum of lies in even though the extreme eigenvalues are only estimated ( by default). The density per orbital is , , where is the energy (the eigenvalue, for a general matrix), with Chebyshev moments
A random vector with gives (for unit-variance entries), and averaging over vectors reduces the sampling error as ; the error of each moment also falls as (a bound is given below), so large systems need fewer vectors [1, 2, 16]. sdos draws Rademacher () vectors for real and random phases for complex problems (Gaussian vectors are available; they have a larger variance, Table I). For a Hermitian matrix with and independent entries, the variance of the estimate of is for Rademacher vectors (real symmetric ), for random phases and for real Gaussian vectors; Rademacher vectors have the smallest variance among real vectors with independent unit-variance entries [2]; random phases halve the off-diagonal variance of a real symmetric at the price of complex arithmetic. With and the factor of the normalized moment, the standard deviation of for one Rademacher vector is bounded by , whatever the locality of : gives and . The covariance of the moments of different order and the cases where it matters are in Sec. III C. Variance-reduction schemes for stochastic trace estimation that I did not implement or test include probing [26] and low-rank deflation as in Hutch++ [27]. With , the inner product , the recursion and the doubling identities
| vectors | std (meV/at.) | time (s) | efficiency |
| Rad. | 192.8 | 0.23 | 1.0× |
| Gauss. | 315.4 | 0.23 | 0.4× |
| impartial (3) | 25.6 | 0.35 | 36.7× |
| Rad. + exact K=4 | 11.2 | 1.75 | 38.0× |
| Rad. + exact K=8 | 4.4 | 8.99 | 47.3× |
| Rad. + exact K=16 | 1.4 | 79.84 | 52.2× |
Table I. Random vectors and exact low moments, band energy of DTW4096 (Kwon, , 12 seeds with each; std is the standard deviation of an estimate with vectors, times larger per vector). Efficiency = (variance × time) relative to plain Rademacher vectors; the uncertainty of each entry is about 30%.
give moments from matrix–vector products per vector. For complex Hermitian all inner products are conjugated and everything else is unchanged; real arithmetic is used whenever the operator is real. The operator may be an explicit sparse matrix (CSR) or any matrix-free linear operator; Hermiticity is checked (explicitly or by randomized probes for matrix-free operators), which matters for finite-difference Hessians that are Hermitian only to .
Spectral bounds. The bounds are estimated with ARPACK (Lanczos)1, starting from a fixed vector (a Gaussian vector from a fixed internal seed, independent of the random vectors used for the moments). Two runs therefore give identical bounds, and the same bounds serve all moment vectors. The vectors of the moments themselves come from the user’s seed and differ from run to run if none is given. Lanczos Ritz values lie inside the spectrum, hence the margin . For matrix-free operators built from finite differences of forces the estimate can fail badly (§ 2.5, item iii), and the package does not detect an over-wide window. For explicit matrices Gershgorin bounds are available as a safe fallback.
Exact low-order moments and impartial vectors
Drabold and Sankey [9] introduced impartial vectors, chosen to weigh all eigenvectors equally, and computed low-order moments exactly; Voter, Kress and Silver [24] and Röder [25] evaluate low-order moments from locally truncated Hamiltonians. The substitution below uses that local computation. The construction of Drabold and Sankey minimizes the penalty by conjugate gradients, with the exactly known and (both from the matrix), so that the first three moments of are exact. This narrows the distribution of the band energy over the random vectors. Only three moments are constrained in the construction of Drabold and Sankey; constraining more makes the ensemble measurably biased (Sec. III C), so sdos keeps three. The recommended way to use exact low moments is substitution: for can be computed exactly in because depends only on a neighbourhood of radius hops of site , and in every sample the sampled with is replaced by the exact one (exact_moments=). The moments with remain the sampled estimates and the Chebyshev recursion is unchanged. The reconstruction uses one list, exact and sampled, of one matrix. Each is unbiased on its own, so every linear functional of the list (the KPM density, the integrated density, the band energy at a fixed Fermi level) is unbiased, and the variance falls sharply (Table I). The list is not exactly the moment list of a positive density; this does not matter for KPM, and with Maxent it is absorbed by the sampled uncertainties of vectors (Sec. III C), so with a single vector, which has no variance estimate, sdos uses . The default is when is an explicit sparse matrix and is absent or diagonal, and otherwise, because the exact moments need the matrix elements. The cost per site grows polynomially with , not with . Table I measures all options on one problem: at equal cost every variance-reduction option beats plain Rademacher vectors by a factor of about 35–50, with –16 comparable (the quoted uncertainty is 30%).
Reconstruction: KPM and maximum entropy
KPM. The default reconstruction is the damped series
with the Jackson kernel [5, 6] (positive, energy resolution throughout this report, taken as the Gaussian of the kernel, so that the full width at half maximum is about ; the Fejér sums [28, 29] are the simplest damping with a positive kernel); the Lorentz kernel is available for Green-function-like uses. The density is evaluated on the Chebyshev–Gauss nodes , with by default; the integrated density is the analytic integral of the damped series, and the band energy and Fermi level are obtained from it on a uniform grid of 20000 points in (trapezoid rule in ). KPM is linear in the moments and never fails to converge. Beyond the resolution at which the sampling noise dominates, more moments only sharpen speckle. For a finite system with levels, a resolution finer than the mean level spacing resolves individual levels.
Maximum entropy. Maxent [7, 8, 9, 25] writes with a default model and finds by Newton’s method on the convex dual
whose integrals are evaluated by the midpoint rule on a uniform grid in of points (so that and a flat model is ); the integrated density is the cumulative midpoint sum, and the band energy follows from a trapezoid rule in energy over its increments. The dual has gradient equal to the moment residual plus and Hessian . Since , the integral is a function of the Chebyshev moments of the current iterate . Here are the standard errors of the sampled moments (zero for exact moments), and the solver uses an eigenvalue-floored Newton step with backtracking. The default model is uniform (flat); the Chebyshev weight, the Jackson-KPM density and a KPM model at fewer moments gave no measurable difference in accuracy or conditioning. Two practical points follow from my tests. First, with a spectral window (the interval used for the scaling) much wider than the support (the interval that contains the eigenvalues; a poor bound), the multipliers become very large (up to – for exact moments) although the density remains accurate; stochastic moments with their errors tolerate even a factor-two window. Second, Maxent with exact moments of a density with singularities (log van Hove peaks, Dirac cusps) is a badly conditioned problem. This is an empirical observation; I have no proof. Silver and Röder [25] fitted 1024 exact moments for a 16384-state amorphous-diamond Hamiltonian; they fitted Gibbs-damped moments (the positive-kernel factors ) with the kernel-polynomial density as the default model, evaluated the Fourier integrals on a fine grid of pixels (they use , ), used moment uncertainties at the numerical precision for exact data and continued in the regularization parameter (an outer loop that decreases from a large value) rather than in the number of moments, none of which I tested, so the non-convergence reported here refers to the solver described in this section (undamped moments, flat model), not to theirs. The exact moments of a density with van Hove peaks and band edges lie close to the boundary of the set that an exponential can reproduce, the multipliers grow as increases, and the Newton iteration stops converging: for DTW4096 it did not converge in 300 steps for (Table IV). The behaviour depends on the density: for C with exact moments the plain solver does converge (403 steps without the floor, 27 with it; Sec. IV B). With sampled moments the sampling error regularizes the problem. The same regularization can be applied to exact moments: sdos assigns every moment except a minimum uncertainty with by default (sigma_floor); the moments are normalized, , so this is an absolute scale. The dual then has a finite solution and Newton’s method converges in 9–24 steps at all tested (Table IV). For stochastic moments of a 16 384-orbital system with and the sampled errors ( between and for ) exceed the floor, which then changes nothing (identical results for , 170 and 300). The floor does act on the exact moments ( there; by default), and it is essential for , where there is no variance estimate (Table III). The floor is a bias of order in the moments, which was found to be meV/atom in the band energy at and 0.2 meV/atom at ; sigma_floor=0 restores the exact-moment problem. Whether is small enough depends on the quantity. A shift of the moments changes an integrated quantity by , with the Chebyshev coefficients of ; for the band energy and , so the worst case, with every moment displaced coherently by , is (a few meV per orbital for eV), and a random-sign shift is . These are bounds on the bias of the constraints, not on the fitted density, whose error was measured to be far smaller (above). A quantity that depends on fine structure at high order (a feature narrower than , a sharp edge, a spectral weight below ) or that needs a precision better than may be affected: reduce sigma_floor, accepting slower convergence and, in the limit , the non-convergence documented above. In practice: with exact moments the default floor gives the exact-moment Maxent density, to a band-energy error of 0.02 meV/atom or less, in a few tens of Newton steps where the plain solver fails; with sampled moments, and , the floor is inactive. More random vectors do not help the exact-moment problem, because the exact moments do not depend on . For sampled moments they lower the sampling error and with it the pointwise error (Table III), but convergence is not the issue there: Newton converges in 12–15 steps for .
| (meV/atom) KPM | (meV/atom) Maxent | (meV/atom) Maxent | pointwise KPM | pointwise Maxent | pointwise Maxent | Newton steps Maxent | Newton steps Maxent | |
| 20 | 3.3493 | 0.1122 | 0.1122 | 9 | 9 | |||
| 40 | 0.2862 | 0.0383 | 0.0383 | 11 | 11 | |||
| 70 | 0.0715 | 0.0116 | 0.0114 | 34 | 17 | |||
| 100 | – | 0.0332 | – | 0.0082 | – | 22 | ||
| 170 | – | 0.0119 | – | 0.0068 | – | 23 | ||
| 300 | – | 0.0053 | – | 0.0082 | – | 24 | ||
| 400 | – | 0.0037 | – | 0.0123 | – | 22 |
Table IV. Truncation error on DTW4096 (Kwon, ) with exact moments (no sampling noise): band-energy error and distance to the exact spectrum broadened by 0.05 eV. Maxent: exact-moment constraints; dashes: the solver did not reach tolerance in 300 Newton steps, so no result is reported (no steps are listed either). Maxent : every moment but given the uncertainty (§ 2 C); all converged.
Bootstrapped Maxent. maxent_bootstrap climbs in the number of moments: it solves for in turn, starts each stage from the previous multipliers, and after each stage restricts the integration support of the dual to the interval where the density found at that stage exceeds of its maximum, padded by two resolution lengths (in the scaled variable ). No states are allowed outside the edges, so the multipliers need not build huge exponentials to suppress the density in the empty margin of the window. The support is a modelling statement, checked by the residual of the moments; a stage that fails to converge is widened and retried, and if it still fails the last converged stage is returned (with converged=False and the number of moments used). For graphene (Xu Hamiltonian, exact -sum on a mesh) with exact moments, , the bootstrap converges in 29 s without the uncertainty floor, where the plain solver does not converge in 300 steps (46 s; distance to the exact spectrum broadened by 0.05 eV: 0.0446 for the bootstrap, and 0.0449 for the last iterate of the plain solver, which is not a converged result). With the default floor both converge, the bootstrap in 2.0 s and the plain solver in 3.8 s (31 steps), at (timings on 2 threads, approximate). The support barely shrinks for graphene, because the band fills its window, so here the gain comes from the warm starts.
Integrated quantities. The integrated density , the Fermi level for a given electron count, and the band energy follow from the reconstruction (fermi_and_band_energy), with jackknife error bars for Maxent (the spread of the result when each random vector is left out in turn). In a gap the integrated density is flat, so is not determined by the moments to better than the gap: sdos reports the interval over which equals the electron count within noise (fermi-interval) and warns; the band energy is nearly independent of where is placed in a true gap. Exact HOMO and LUMO levels are obtained separately (Sec. III).
Generalized problems
For , is self-adjoint in the -inner product and similar to , so and is an unbiased estimator for any with [[17]]. is not Hermitian in the Euclidean product, so the doubling identities do not apply: one application of per moment is needed, and the cost per moment is that of a solve. Spectral bounds of use a Lanczos process in the -inner product. SDOS offers the following ways of applying , all verified to give the same moments to tolerance (Table V): (i) PCG to a relative residual with block-Jacobi (default), incomplete Cholesky IC(0) with a Manteuffel shift ([30, 31]) (the shift needed is reported, since the overlap of a minimal basis is not an M-matrix), or algebraic multigrid; (ii) a fixed polynomial on the diagonally scaled (the scaled problem is similar to ), of degree from the condition number .
| by | (real) | k (complex Hermitian) | ||||||
| time (s) | iter./solve | (meV/atom) | to exact | time (s) | iter./solve | to exact | ||
| PCG, block-Jacobi | 54 | 23.6 | 0.024 | 125 | 23.6 | 0.021 | ||
| PCG, IC(0) | 79 | 11.0 | 0.024 | 155 | 11.0 | 0.021 | ||
| Chebyshev polynomial | 64 | 26.0 | 0.024 | 135 | 26.0 | 0.021 |
Table V. DFTB electronic problem of DTW4096 (16 384 orbitals, 179 nonzeros per row): KPM, , . At the three ways of applying give identical band energies (the same moments); the error is against the dense generalized spectrum and is a sampling error (this run, seed 11: meV/atom, with the standard error estimated from its six vectors; other seeds give , and meV/atom with estimated errors of 84–94, so that estimate is itself uncertain and the spread over seeds, about 110 meV/atom at , is the reliable figure; see text), and is the distance to the exact spectrum broadened by 0.1 eV. The dense complex generalized problem does not fit in memory here, so the k point is compared with the spectrum of the same cell.
In detail: the extreme eigenvalues of are found by ARPACK (tolerance ) and widened by 2% on each side to guard against bounds that are too tight; is the Chebyshev interpolant of on that interval; its degree starts from with and is raised until on a grid of 4001 points ( by default, cheb_tol); is applied to blocks of vectors by the Clenshaw recurrence, one product with per degree, and a check on a random vector confirms the tolerance. The inverse is then replaced by a matrix polynomial, so the recursion is applied to one fixed operator and needs no inner solves; the error of the polynomial is a known, controlled approximation. For a diagonal (masses) is absorbed into once and no solves are needed. For the DFTB overlap of silicon () the polynomial has degree 26 and PCG with block-Jacobi needs about 24 iterations per solve (Table V); the iteration count is independent of the system size (§ 2.5).
Eigenvectors in spectral windows, exact gaps and projections
Windows. window_eigs returns the eigenpairs with in a window together with their inverse participation ratio (site weights from the Mulliken populations, see Projections below), by shift-invert (sparse of , ARPACK) or by Chebyshev-filtered subspace iteration [32]. Shift-invert is not : the fill of a sparse factorization grows faster than linearly in three dimensions; the filtered iteration (a Chebyshev polynomial filter) is the scalable fallback. A filter of a given degree separates the states in the window from the rest of the spectrum only at distances from the window edges larger than about , where is the half-width of the spectrum.
Exact HOMO–LUMO gaps. By Sylvester’s law of inertia, the number of eigenvalues below equals the number of negative pivots of ; count_states_below evaluates it from a symmetric factorization without pivoting, and homo_lumo finds HOMO and LUMO by two bisections, with no eigensolver and no assumption on the gap size or on degeneracy. Because of the fill, this is more expensive than the density of states above orbitals (§ 2.5), so the exact gap is a tool for moderate sizes.
Projections. For with eigenvectors normalized , the weight of orbital in state is the Mulliken weight [33] (equal to for ). For overlapping orbitals is not a probability: it can be negative or exceed one (the sum over is one, ), so a local or projected density built from it is a signed quantity, not a positive local spectral density, and it can be negative at some energies for orbitals that overlap strongly with their neighbours. This is a property of the Mulliken partition, as for Mulliken charges; the total density is unaffected.
KPM handles signed densities; Maxent assumes positivity and is not used for projections. A positive alternative is the Löwdin partition, which needs and is not provided. The order of the operators matters. With and the spectral projector is . Its diagonal element integrates to , not to one, so it is not the Mulliken density. With the Mulliken weight defined above,
because ( the Euclidean adjoint of ) and (the matrix does not commute through the function; it conjugates the argument). Its Chebyshev moments are the diagonal elements of the adjoint operator, which is applied to a vector as : first the inverse of , then (the reverse of ) used for the total density. The code applies exactly this operator (GeneralizedOperator.apply_adjoint), and a unit test with dense , real and complex, compares the resulting moments of single orbitals with the Mulliken weights of the generalized eigenproblem to . Then . Sum rules hold exactly: , each LDOS integrates to one, and PDOS integrates to , the number of orbitals in the group. Two estimators are provided (projected random vectors were used for local densities by Sankey, Drabold and Gibson [23]): “shared” (one random vector is propagated once and every group reads its own partial inner product, all groups for the price of one DOS, at the cost of extra noise for small groups) and “separate” (a vector supported on the group), and ldos propagates unit vectors for a deterministic result.
Vibrations
For a force-constant matrix and masses , the dynamical matrix has eigenvalues . vdos works with and reports ; negative eigenvalues (instabilities) are reported, not clipped, with the weight below the resolution. The three rigid translations are known eigenvalues at and can be removed exactly from the moments (zero_modes=3), which matters for the entropy and the low-temperature heat capacity because the kernel would smear their weight into small positive . harmonic_thermo gives the zero-point energy, , , and per atom and warns below the temperature where the lowest frequencies are unresolved.
Matrix-free Hessians. For large models HessianOperator applies with any force provider (LAMMPS through its Python module). In this case a product costs two force calls and translations are projected out. Units and conventions: positions in Å, forces in eV/Å, masses in atomic mass units (the LAMMPS metal units), so is in eV/Å and in eV Å u, with and meV in the thermodynamics (the factor is the argument energy_per_sqrt_lambda of harmonic_thermo, for other unit systems). The cell is periodic as defined by the force provider (LAMMPS applies the minimum image and its own neighbour lists), and the Hessian is that of the structure at . For a vector the displacement is (components ), where the scalar is chosen such that the rms of the Cartesian components of , , equals Å regardless of and of the norm of (so the rms displacement per atom, as a vector, is Å); the central difference then gives with an error and the result is multiplied by again to give . No Hessian is stored.
Interfaces and implementation
SDOS is a pure-Python package (NumPy [34], SciPy [35]; Python 3.14 in the present tests); everything else is optional. Tight-binding models: the Hamiltonian of Kwon et al. [36], an empirical, orthogonal tight-binding total-energy model for silicon (a band-structure term plus a repulsive pair potential; sdos uses only its Hamiltonian matrix, with four orbitals per atom), with Bloch twists (complex Hermitian matrices), a chunked assembler for atoms and a matrix-free periodic stencil for crystals; and the Xu–Wang–Chan–Ho carbon model [37]. DFTB Slater–Koster (.skf) files, read from a user-supplied directory (the pbc-0-3 set [38] is distributed under CC BY-SA 4.0 and requires citation of Refs. [39, 40]; it is not bundled), non-self-consistent [41, 42] (zeroth-order Hamiltonian and overlap ). SIESTA and through sisl [43, 44, 45], and LAMMPS force fields [46] through the matrix-free Hessian. A command-line tool (sdos) covers the common cases. The test suite has 186 tests (run with OpenMP/BLAS capped at 4 threads), including the comparisons with exact diagonalization listed in Sec. III, a regression test against the original single-file code and tests that the documented examples run.
Input, result files and analysis. The simplest matrix file is plain text with one entry per line, i j a_ij (a fourth column is the imaginary part); one or both triangles, 0- or 1-based indices and the storage are detected, and the other triangle is generated by conjugation (Matrix Market and SciPy .npz are also read). The density-of-states stage needs only the matrix and produces a density per orbital, normalized to one. A result written with sdos H.dat -o result.npz (here H.dat is a plain-text list of the nonzero matrix elements; or res.save is a self-describing file: , its error bars and the integrated density, the averaged and per-sample Chebyshev moments, the spectral window, the method and its settings, and for vibrational and ). sdos analyze result.npz (or sdos.load) then works from the file alone, without the matrix. Whatever needs information from outside the matrix enters only here: for electrons the number of electrons (given directly or as the sum of the valence of each atom, from a file) and the spin degeneracy give and the band energy; for vibrations a temperature grid gives the zero-point energy and , , , (the masses and the three zero modes change the moments and are therefore applied at the density-of-states stage). The analysis can also rebuild the density from the first saved moments with KPM or Max-ent, which costs no matrix products. Saving and loading reproduces the arrays exactly, and and from a loaded file agree with the in-memory values to for both methods (tests); reconstruction from a loaded vibrational result is not supported, and the generalized problem was not tested through the file.
The DFTB reader. Reading the DFTB format correctly required three conventions that I did not find stated in the documentation available to me (the DFTB+ manual; I could not retrieve the separate Slater–Koster format description, and some of them may be documented there), which I recovered by comparing whole matrices with DFTB+ [47] (Sec. IIIE): a table announced as ngrid rows contains ngrid - 1, row sits at ( is the omitted row) and the first 19 rows of the silicon file are placeholders for Bohr; the orbitals are ordered ; and each integral is continued past the end of its table by a quintic that reaches zero, with vanishing first and second derivatives, one Bohr later and matches value, slope and curvature of the table’s last row. The non-orthogonal pair convention follows from the file name: in A–B.skf the first orbital of each integral sits on A, so a ( on A, on B) block uses the B–A file.
Validation
The electronic examples use the Kwon Hamiltonian [36], an empirical, orthogonal tight-binding total-energy model of silicon; its spectra are those of that model, so agreement with an exact spectrum below tests the method, not the model. Each test below is compared with exact diagonalization or with an independent code; the benchmarks use DTW4096, a 4096-atom model of amorphous silicon from the Djordjevic–Thorpe–Wooten family [48, 49] (relaxed at zero pressure with ACE, cell Å), at unless stated, since the cells are large and the -point dependence of an amorphous Si cell this size is weak.
Conventions. Unless stated otherwise, over the whole spectrum with both densities normalized to one, where is the histogram of the exact eigenvalues broadened by a Gaussian of the width given in each case (0.05 eV for the tests with exact moments, 0.1 eV for the stochastic ones; a narrower reference penalizes the sharper Maxent curves, a wider one the broader KPM curves). Band energies are for the electron count of the neutral system, in meV per atom, and “error” is estimate minus exact. A sampling error quoted as ± is the standard error over the vectors unless it is marked . Table II lists every feature that I checked, with the system, the reference, the quantity and the observed error; the last rows list what was not checked or not reproduced.
Unit tests
The package is tested against the following, all of which pass: (1) a cubic Anderson model; (2) the same with Peierls phases (complex Hermitian); (3) a 216-atom diamond-structure silicon cell with the Kwon Hamiltonian; (4) a random sparse with a sparse positive definite whose condition number is tuned from to , real and complex; (5) a Keating [51] dynamical matrix with unequal masses in both forms (mass-weighted, and generalized with the mass matrix); (6) PCG (block-Jacobi) and the Chebyshev polynomial give the same moments (the other preconditioners, IC(0) and AMG, are exercised in a further test and compared on DTW4096 in Table V); (7) regression against the original code. Beyond these: Maxent robustness (large windows, default models), the Mulliken sum rules, local exact moments against the dense matrix, the inertia counts, and the interfaces (Secs. III D–III F).
Accuracy against the number of moments and vectors
To separate truncation from sampling I compute the moments exactly from the dense spectrum of DTW4096 (Kwon TB, 16 384 orbitals) and reconstruct (Table IV, Figure 1a,Figure 1b). KPM with the Jackson kernel converges steadily ( error at ); Maxent with exact-moment constraints is more accurate for the same moments up to , the last size at which it converged: at its band-energy error is 0.12 meV/atom against 1.0 and its distance is six times smaller, at 1.6 against meV/atom. From the plain Maxent solver no longer converges with exact moments of this singular density; how many exact moments the plain solver manages depends on the problem (for some it works to several hundred); those results are not reported (dashes in Table IV, no points for “Maxent, exact moments” in Figure 1). With the uncertainty floor (§ 2 C) every case converges: the band-energy errors are +0.02, , and meV/atom at , 170, 300 and 400, 5–60 times smaller than KPM’s (+0.09 to +0.30 meV/atom), and the distance to the 0.05 eV-broadened exact spectrum is 0.007–0.012. The distance of Maxent is smallest at (0.0068) and then rises (0.0082 and 0.0123 at and 400; Figure 1b), while KPM’s keeps falling and is smaller at and 400 (0.0053 and 0.0037); the rise is not caused by the floor: at the distance is 0.0117, 0.0123 and 0.0128 for , and ( did not converge in 300 steps, so I give no number), and at it is 0.0082–0.0083 for to (0.0095 for , 285 steps; script bench_accuracy_floor.py, the setup of Table IV). I attribute it to the ill-conditioning of the exact-moment problem for a singular density (§ 2 C) and in part to the test, because a sharp Maxent curve is compared with a spectrum broadened by 0.05 eV, so for resolving fine structure KPM remains the more accurate and for integrated quantities Maxent is better here. With sampled moments the sampling error dominates (Figure 1c). Table III shows what more random vectors do to Maxent with sampled moments: the error falls with (roughly as from to 64), and the solver converges in 12–15 Newton steps for ; a single vector without the floor does not converge.
Figure 1. Accuracy on DTW4096 (Kwon TB, ). (a) Band-energy error from exact moments: KPM (Jackson), Maxent with exact-moment constraints (converged cases only) and Maxent with the uncertainty floor . (b) Pointwise distance to the exact spectrum broadened by 0.05 eV. Maxent with exact-moment constraints is shown only where the solver converged. (c) Standard deviation of the band energy over 16 seeds at against the number of random vectors (plain Rademacher vectors, no exact-moment substitution), with the law.
The sampling error of the band energy falls as (Figure 1c) with meV/atom per random vector for plain Rademacher vectors (193 at , Table I); the default exact-moment substitution reduces this to meV/atom per vector (11 at ). Note that comparing a sharp Maxent curve with an exact spectrum broadened by a Gaussian penalizes Maxent for sharpness; the truncation test above avoids this by comparing integrated quantities and a narrow (0.05 eV) reference.
Small systems: validity of exact-moment substitution
The test of Table I is one system with . I repeated the comparison where the argument for substitution could fail, with exact references from dense diagonalization: a periodic chain and a dimerized (gapped) chain with –4096, the Kwon tight-binding Hamiltonian of cubic diamond silicon cells – (, 864, 2048, 4000 orbitals, ) and the three-dimensional Anderson model (, , 1000, 4096). The moments of random vectors were computed in the eigenbasis, which gives the same numbers as the recursion. Errors of the band energy (half filling) and of the cumulative distribution are measured against the same reconstruction fed with the exact moment list, so that truncation is common to all estimators.
Consistency. A list is the moment list of a positive measure on if and only if the Toeplitz matrix is positive semidefinite. The exact lists satisfy . Lists from one random vector or an average of 16 vectors, and impartial vectors, never violate it (all seeds, all systems). The spliced lists (–16, one or 16 vectors) violate it in 80–100% of the seeds, with down to to , at an order –40 that does not grow with (Figure 2); the
Figure 2. Smallest eigenvalue of the Toeplitz matrix (median over seeds) against (Kwon TB: the tight-binding Hamiltonian of Kwon et al.): exact moments and plain or impartial vectors (all curves on top of each other, never negative) and spliced lists (negative beyond –40).
TABLE II. Summary of the validation. “Dense” is a dense diagonalization of the same matrix. DTW4096: 4096-atom model of amorphous silicon with the Kwon tight-binding Hamiltonian (16 384 orbitals) or DFTB; “Kwon TB” is the empirical, orthogonal tight-binding total-energy Hamiltonian of Kwon et al. [36]; its spectra are those of this model and not of a first-principles calculation.
| Feature | System | Reference | Quantity | Observed error |
| Exact local moments () | chain, dimerized chain, Kwon TB, 64–864 orbitals | dense moments | ||
| KPM truncation (Jackson) | DTW4096, Kwon TB | dense, exact moments | ; (0.05 eV) | +0.09 to +0.30 meV/atom for –400; at |
| Maxent, exact moments, floor | DTW4096, Kwon TB | dense, exact moments | ; (0.05 eV) | −0.017 to +0.02 meV/atom for –400; –0.012 |
| Sampling error | DTW4096, Kwon TB, | dense | s.d. of over seeds | meV/atom per Rademacher vector (193 at ), ; per vector (11 at ) with (Table I) |
| Substitution and impartial vectors at small | chain, dimerized chain, Kwon TB, Anderson, –4000 | dense | RMSE of (, ) | 0.04–0.10 () and 0.004–0.014 () of plain vectors (Sec. III C) |
| Complex Hermitian ( twist) | Kwon 64 atoms, DFTB 216 atoms | union of primitive-cell bands | eigenvalues; Hermiticity | , eV; |
| Matrix-free stencil | Kwon TB, cells (131 072 orbitals) | exact k-sum | (0.1 eV), , | KPM 0.036, Maxent 0.047; Hermiticity |
| DFTB matrices | 64-atom Si cell, Si–H cluster | DFTB+ WriteRealHS | matrix elements; eigenvalues | : , : eV; eigenvalues eV |
| Generalized, three modes | DTW4096, DFTB, | dense generalized | ; (0.1 eV) | identical moments in the three modes; sampling error rms 110 meV/atom at , 80 at (two seeds), at (eight seeds; moment -scores rms 1.00, mean ); (Table V) |
| SIESTA matrices; inertia count | C, C (960, 2160 orbitals) | dense eigh(, ) | HOMO–LUMO gap | , eV |
| Matrix-free Hessian | Si 512 atoms, ACE | LAMMPS dynamical_matrix | matrix, relative | |
| Harmonic thermodynamics | DTW4096, ACE and SW | dense dynamical matrix | ZPE; (300 K); (100 K) | ACE: −0.66%, +0.27%, +1.4%; SW: −0.54%, +0.29%, +0.92% (Table VI) |
| Shift-invert window | DTW4096 | dense | three levels in the gap | eV |
| replica of R10, Kwon TB | 4 244 832 orbitals | exact Bloch spectra | max error of | Maxent 0.04%, KPM 0.08% of (Sec. IV D) |
| replica of R10, ACE | 3 183 624 degrees of freedom | exact Bloch spectra | ZPE; low-frequency mode count | ZPE 0.03–0.07%; 0.5% of the modes below 9 meV at |
| 2 097 152-atom crystal | Kwon TB, cells, matrix-free | exact k-sum of the 8-atom cell | band edges; gap | edges within resolution; Maxent follows the exact edges (0.46, 1.24 eV); no computed |
| Not reproduced | C gap | Ref. [50] Fig. 3 | gap | 0.284 against 0.10 eV (open) |
| Not checked | complex generalized at k; Mulliken weights and IPR; -atom runs | — | — | only sum rules or an internal comparison (§ 2.5, item vii) |
Table 1.
| 100 | 1 | 29 | – | 0.069 |
| 100 | 4 | 12 | 12 | 0.046 |
| 100 | 16 | 14 | 14 | 0.022 |
| 100 | 64 | 15 | 15 | 0.014 |
| 300 | 1 | 26 | – | 0.126 |
| 300 | 4 | 12 | 12 | 0.087 |
| 300 | 16 | 14 | 14 | 0.038 |
| 300 | 64 | 15 | 15 | 0.018 |
Table 3. TABLE III. Maxent with sampled moments on DTW4096 (Kwon TB, ), no exact-moment substitution. Newton steps with the default floor and with (“–”: not converged in 499 steps), and distance to the exact spectrum broadened by 0.05 eV (floor ; the same for when it converges).
empty arcs of the spectral measure (band-edge margin, gaps) make of the exact list small at low order, and the noise of the sampled tail exceeds it.
KPM. At equal cost the root-mean-square error of (, ) is 0.04–0.10 () and 0.004–0.014 () times that of plain Rademacher vectors, for every system and from 32 to 4000; impartial vectors give 0.08–0.15 (Figure 3). The estimates of have a bias that is statistically significant with 500 seeds, for plain vectors as well as spliced ones, but much smaller than the plain-vector error. Its origin is the Fermi level: with held at its exact value, is a linear functional of the moments and is unbiased to within two standard errors for plain vectors and for , 16 (2000 seeds). The impartial vectors are unbiased at the level of one standard error for and show an rms -score of 1.3–1.6 for . Constraining more moments (script zscore_constrained_impartial.py, 1000–1500 vectors, rms of the unconstrained moments above the constrained ones and below 64) gives 1.0–1.5 for 4, 8 and 16 constrained moments (1.5 for the Kwon TB with 864 orbitals, 1.3–1.4 for the Anderson model with 1000 sites, about 1.0 for the Kwon TB with 2048 orbitals and for a chain with 512 sites), against 0.8–1.0 for the three constrained moments of Drabold and Sankey: a bias of at most about one standard error of the mean of the vectors at these sizes.
Figure 3. Root-mean-square error of the KPM band energy per orbital, divided by the bandwidth, against (, ; 500 seeds for , 200 above), for plain Rademacher vectors, exact-moment substitution (, 16), impartial vectors and impartial vectors with .
Maxent. With sampled uncertainties from vectors, substitution with lowers the error of by 6–40 times relative to plain vectors and Maxent converges in 90–100% of the runs ( needs 20–66 Newton steps, against 11–17 for plain vectors). With a single vector the uncertainty is only the floor, the inconsistent list has no solution, and Maxent with or 16 does not converge (0–20% of the runs reach the tolerance in 500 steps) and returns a band energy and a cumulative distribution that are far worse than with plain vectors, at every size tested (chain, dimerized chain, Kwon,
Anderson). Plain and impartial vectors converge with . These and runs used , the chain and the dimerized chain with and 512, Kwon with and 864 and Anderson with , with 20 seeds (5 for the non-converging splices); the runs without the uncertainty floor and with were not repeated. For , 3, 4 and 8 (script e2_maxent_R.py; ; chain and dimerized chain with , Kwon with (60 seeds) and 864, Anderson with , 40 seeds each unless stated) Maxent with converged in 90–100% of the runs (typically 95–98%, with no trend in ), in 9–19 Newton steps (median), and lowered the rms error of by factors of 8–40 against plain vectors with the same ; plain vectors converged in 95–100% of the runs. The failure is therefore specific to . These runs were limited to .
Resolution at the level spacing. For and 1024, KPM with up to stays stable and its error is set by the stochastic noise, not by . Maxent from the exact moment list, with the default floor, needs 42–129 Newton steps for and did not converge at for the Kwon system with . The package warns when and suggests dense diagonalization, which is cheaper and exact for such small problems.
Complex Hermitian problems
A Bloch twist makes the Hamiltonian of a periodic supercell complex Hermitian, , and its spectrum is the union of the primitive-cell bands at the points folded onto it. For a 64-atom Kwon supercell and a 216-atom DFTB supercell (the generalized problem), the spectrum of the twisted supercell equals this union to and eV, respectively, and is Hermitian to ; the same builder is used in chunked form for atoms and agrees with the unchunked one exactly. The matrix-free operator applies as a stencil: the product with a vector is a sum of 128 shifted blocks (one for each neighbour pair of the cell), so that is never stored. For a -cell supercell (131 072 orbitals) at a generic twist the stencil has a Hermiticity defect of and its KPM and Maxent densities (, ) agree with the exact -sum with distances 0.036 and 0.047 against the 0.1 eV-broadened reference. For DTW4096 with the DFTB model the density at ( the reciprocal-lattice vectors of the cell; a quarter of each, so is not a reciprocal-lattice vector and the matrices are not gauge-equivalent to those at ), a genuinely complex generalized problem, looks like the one ( to the spectrum 0.021 against 0.024 for itself, Table V); a dense complex generalized diagonalization of 16 384 orbitals does not fit in this machine, so there is no exact reference at that point (at a reciprocal-lattice vector the twisted matrices are unitarily equivalent to those at , which would give an exact reference; I did not use such a point).
Non-orthogonal problems: DFTB and SIESTA
Comparison with DFTB+. DFTB+ [47] can write the real-space Hamiltonian (the zeroth-order, non-self-consistent one) and (WriteRealHS) for a randomly displaced 64-atom silicon cell of 10.9 Å (where pairs interact through two periodic images) the matrices built by sdos from the same .skf files agree with DFTB+ to eV in (largest element 10.8 eV) and in , and the generalized eigenvalues to eV. A distorted Si–H cluster (which tests the heteronuclear sign convention and the use of the reversed-pair file) agrees to eV and . The residual is the difference between my cubic spline and DFTB+’s polynomial interpolation; in the interior of the table the two agree to .
DTW4096. Table V gives the DFTB problem of DTW4096: 16 384 orbitals with 179 nonzeros per row, dense generalized spectrum computed exactly (Cholesky reduction and a dense eigensolver, 210 s, about three dense matrices of 2.1 GB each), gap 0.0095 eV. The three ways of applying give the same moments (identical band energy). The error against the dense generalized spectrum is a sampling error, not a bias. For and four seeds it is , , and meV/atom (rms 110), for and two seeds and (rms 80), and for (one run) meV/atom (script dtw_dftb_Rscan.py). This is consistent with the law (expected rms 55 and 27 meV/atom from the value), within the small number of runs. To test unbiasedness directly I ran eight seeds at with the Chebyshev mode and (about 9 min each; script dtw_dftb_unbiased.py), and compared every stochastic moment with the dense generalized moment: the -scores (stochastic minus exact, over the sample standard error) have rms 0.82–1.22 per seed (pooled 1.00 over 792 moments) and mean 0.006 (the standard error would be for independent moments; the eight per-seed means scatter by only 0.011, so the moments are not independent and I quote no further error), and (the KPM band energy at the same , so truncation cancels) is meV/atom (seed spread 22, range to ), consistent with zero bias and with the law. The standard error estimated from the vectors of a single run is itself uncertain at small (35 to 94 meV/atom at ), so Table V should be read with the seed spread, not its . For the intended use this means that vectors give the shape of the density of states but a band energy to only eV/atom, and the energy differences between structures that matter in a workflow need either many more vectors (the cost is linear in ) or an independent estimate of the error; the variance reductions of Table I are not available here, because exact substitution requires a diagonal and impartial vectors are refused for a generalized problem. Using the same random vectors for two similar structures should cancel part of the noise; I have not tested it. IC(0) halves the number of CG iterations (11 against 24 per solve) but is slower in wall-clock time because of its setup.
SIESTA. The SIESTA Harris/single-zeta matrices of the fullerenes C and C (960 and 2160 orbitals), generated with SIESTA 5.4.2 [45] and read through sisl, give HOMO–LUMO gaps from the inertia bisection that agree with the dense gaps to and eV (1.2995 and 0.9220 eV). For the series C up to C the PCG needs 59 iterations per solve at every size and the cost is linear in ; SIESTA itself ran out of memory at C (12 GB) because of its setup, which is a reason to build , by the sparse route above.
Vibrations
The operator. For a 512-atom silicon structure with the ACE potential [52, 53] for the silicon–hydrogen system of Rosset and Deringer [54] the matrix-free products agree with the exact dynamical matrix written by LAMMPS’ dynamical_matrix command to a relative (the expected error of the finite difference) with the same units, using two force calls per product.
. For the 4096-atom cell (3 rigid translations removed exactly) the matrix-free results agree with the exact dense spectra of both potentials (Table VI): for Stillinger–Weber [55] (dense matrix built column by column through the same force interface) the zero-point energy is 0.5% and 0.3% off; for ACE, whose exact matrix I obtained with LAMMPS (Kokkos on the GPU, pair_style pace product) in about ten minutes, the ZPE is and off, and the exact matrix has negative eigenvalues and exactly three zero modes. The largest error is the heat capacity at 100 K (+1.4%), where the result depends on the lowest frequencies, which have the poorest resolution in . Figure 4b shows the density: the acoustic band peaks near 5 THz ( meV) and the optic band near 14 THz ( meV). KPM and Maxent agree to four digits for ACE and to three for Stillinger–Weber.
| potential | method | ZPE (meV/atom) | |||
| SW | exact (dense) | 66.935 | 0.8558 | 2.2935 | 2.7863 |
| SW | sdos KPM | 66.573 | 0.8637 | 2.3002 | 2.7890 |
| SW | sdos Maxent | 66.546 | 0.8647 | 2.3007 | 2.7892 |
| ACE | exact (GPU dynamic_matrix) | 60.104 | 0.9369 | 2.4127 | 2.8294 |
| ACE | sdos KPM | 59.709 | 0.9499 | 2.4192 | 2.8319 |
| ACE | sdos Maxent | 59.709 | 0.9499 | 2.4193 | 2.8319 |
Table VI. Harmonic thermodynamics of DTW4096 (4096 atoms, relaxed with each potential): matrix-free sdos (, , two force calls per product, three rigid translations removed exactly) against the exact dynamical matrix. in /atom.
Figure 4. DTW4096 (a 4096-atom model of a-Si) at . (a) Electronic density of states (Kwon TB), KPM and Maxent (, ) against the exact spectrum. (b) Vibrational density of states with the ACE potential: matrix-free KPM (, , two force calls per product) against the exact dynamical matrix of LAMMPS (grey).
Windows and gaps
On DTW4096 the shift-invert window around the gap returns the same three levels as the dense spectrum to eV. The exact fullerene gaps from the inertia bisection are in Table VII.
| gap (eV) | (Å) | gap | |
| 60 | 1.537 | 3.54 | 1.295 |
| 240 | 1.015 | 7.07 | 1.711 |
| 540 | 0.713 | 10.61 | 1.803 |
| 960 | 0.549 | 14.15 | 1.851 |
| 1500 | 0.446 | 17.68 | 1.879 |
| 2160 | 0.374 | 21.22 | 1.891 |
| 2940 | 0.323 | 24.76 | 1.906 |
| 3840 | 0.284 | 28.29 | 1.915 |
| 4860 | 0.253 | 31.83 | 1.919 |
| 6000 | 0.228 | 35.36 | 1.922 |
Table VII. Exact HOMO–LUMO gaps (inertia bisection, Xu Hamiltonian, LCBOP-relaxed icosahedral fullerenes) and the confined-Dirac-cone scaling; eV Å from the graphene bands of the same Hamiltonian, with the graphene area per atom.
Applications
A 2097152-atom crystal of silicon
Extended Data Figure 9 of Deringer et al. [56] shows the tight-binding DOS of a cubic diamond supercell of “2097172” atoms (34.7584 nm on a side) computed with the Kwon tight-binding Hamiltonian [36], 170 moments and Maxent. A cell of 34.7584 nm is exactly cubic cells of 0.5431 nm, i.e. 2097152 atoms (Ref. [56], of which I am a coauthor, prints 2097172), and the matrix has 8 388 608 orbitals. Because the cell is periodic and translation invariant, the Hamiltonian is applied as a stencil (128 shifted blocks per vector) and never stored. With and vectors the KPM density takes 603 s on 1 thread (358 s for the bounds, 246 s for the moments), with a peak of 1.8 GB; the Maxent density from the same moments takes the same time (the fit adds less than 1 s); Figure 5. The exact reference is the 8-atom cell bands on the mesh (exactly the spectrum of the supercell), a histogram with exactly 8 388 608 orbitals. Both methods reproduce the exact spectrum. The valence peaks are at and eV, the conduction features at to eV and the sharp upper edge at 6.7 eV, as in the published figure. The band runs from to 6.65 eV. The methods differ most in the gap (Figure 5b). The exact band edges on this mesh are 0.46 and 1.24 eV, and Maxent follows them: its density is below between 0.44 and 1.24 eV and below between 0.49 and 1.20 eV. KPM at 170 moments (resolution eV, eV; FWHM 0.45 eV) fills the gap down to (minimum at 0.82 eV), a Jackson-broadened floor. For anything that depends on the gap this is the advantage of Maxent for integrated quantities, noted in Sec. III B. The same calculation takes 60 s at cells (262 144 atoms) and 264 s at cells (884 736 atoms). The time grows with the exponent 1.12 against 1.01 for the explicit matrices of § 2.5: 1.09 for the moments and 1.13 for the bounds, whose ARPACK iteration needs more products as the system grows (162, 222, 372, 392 and 522 at , , , and cells, against 340 for the moments). For a crystal the bounds can be given analytically.
Figure 5. Kwon tight-binding DOS of diamond silicon with cubic cells (2 097 152 atoms, 8 388 608 orbitals), moments, vectors, matrix-free; grey: exact -sum. (a) Whole band. (b) Gap region on a logarithmic scale.
Figure 9. DOS of the one-million-atom a-Si model of Morrow et al. [61, 62] with the Kwon tight-binding Hamiltonian (Γ point, 4 million orbitals). (a) KPM and Maxent, , , against the exact 4096-atom DTW spectrum (0.08 eV broadening). (b) Gap region on a log scale; Maxent from to 600 ( and 4) and DTW4096.
Fullerenes and graphene
Drabold, Ordejón, Dong and Martin [50] followed the icosahedral fullerenes from to with moment-Maxent densities of states (“of order 150 moments”, no artificial broadening), a shifted-Lanczos HOMO–LUMO gap and local densities of states, and observed that the DOS of and approaches graphene; the geometries were relaxed with an order- tight-binding scheme [57]. I repeat the calculation with the Xu–Wang–Chan–Ho Hamiltonian [37] on geometries relaxed with the LCBOP potential [58] (which reproduces the strain energies and radii of Ref. [57] to within a few percent, while REBO and AIREBO overestimate the strain by 35–70%), generated with a Caspar–Klug (Goldberg) construction for any . Figure 6a shows from moments. Maxent follows the exact spectrum (a 0.1 eV Gaussian-broadened histogram of the 15 360 exact eigenvalues, computed in 124 s), including the van Hove peaks near and eV above . KPM at 150 moments (resolution eV, eV) is visibly broader. This is the regime of the 1995 calculation, and the sharp graphene-like features it showed are reproduced. The Maxent curve in Figure 6a,Figure 6b uses the default uncertainty floor , with 12 random vectors the sampled errors are larger and the floor is inactive. To see what does here I also fitted exact moments, computed from the exact spectrum, at (green dashed line). The distance to the exact spectrum, broadened by 0.1 eV, is 0.064 with the floor and 0.065 without it (the plain solver converges here, in 403 steps instead of 27), against 0.070 for Maxent with the 12 random vectors and 0.099 for KPM. At this the floor is therefore free, and the stochastic error costs Maxent little. The exact HOMO–LUMO gap of in the Xu model is 0.284 eV against 0.10 eV in their Figure 3 (and 1.537 against 1.62 eV for ). The difference may come from the different relaxations of the geometry (the cages of Ref. [50] were relaxed with the order- scheme of Ref. [57], mine with LCBOP), but the comparison with a sphere below argues against this, since a much larger change of geometry changes the gap by less than 1% at ; I did not resolve it, and the shifted-Lanczos gap of 1995 may not have been converged.
Figure 6. Fullerene C with the Xu Hamiltonian on an LCBOP-relaxed geometry. (a,b) DOS from moments (blue: KPM; red: Maxent) against the exact eigenvalues (grey, 0.1 eV Gaussian), full band and near ; green dashed: Maxent with the floor from the exact moments, . (c) gap against the number of atoms; dashed: the smooth-sphere value 2.
To see what the relaxation does, the same Hamiltonian was employed on the unrelaxed spherical geometry: the Caspar–Klug construction projected onto a sphere of the graphene-equivalent radius (28.29 Å). All atoms are then at the same distance from the centre, and the bond lengths scatter between 1.13 and 1.56 Å (rms 0.10 Å). The LCBOP geometry has bonds between 1.41 and 1.45 Å (rms 0.003 Å), but its distances from the centre range from 26.9 to 31.6 Å (rms 1.1 Å): the relaxed cage is faceted. The two exact spectra of C3840 (Figure 7) have almost the same gap (0.286 and 0.284 eV), with 93 and 72 levels within 0.6 eV of , but otherwise differ strongly. The same comparison for smaller cages (Figure 7) shows the gap of the sphere larger than the relaxed one by 10% at C240 (1.119 and 1.015 eV), 8% at C960 (0.591 and 0.549 eV) and 4% at C2160 (0.391 and 0.374 eV), and the van Hove peaks of the relaxed cages sharper at every size (those of the sphere are smeared); the band of the sphere is also wider (−42 to +25 eV against −22 to +17 eV) because of its short bonds. The sharp graphene-like features of the 1995 spectrum thus require the flat regions of a faceted cage and do not occur on a uniformly curved one. Part of the smearing on the sphere is bond-length disorder, which is the strain that the relaxation removes, so this comparison does not separate curvature from strain.
Figure 7. Exact spectra of C, C, C and C with the Xu Hamiltonian (0.1 eV Gaussian broadening): LCBOP-relaxed geometry (grey) and the unrelaxed spherical Caspar–Klug geometry (orange). The HOMO–LUMO gaps (sphere / relaxed) are given in the panel titles.
The gap, a property of the lowest levels near the Dirac point, is insensitive to both. Maxent from 150 moments (stochastic, ) follows the exact sphere spectrum with an distance of 0.118 against 0.155 for KPM.
The exact gaps (Table VII) follow gap (fit over C240–C6000; fits starting at C60, C960 and C2160 give −0.43, −0.48 and −0.48), which is the scaling of a confined massless Dirac fermion, with .
The graphene bands of the same Hamiltonian have a true Dirac point (gap at K eV) with eV Å ( m/s). The product gap · rises with to 1.92 at C (Figure 6c), close to the value 2 of a smooth sphere without pentagons, whose lowest levels are . The twelve pentagons carry a fictitious gauge flux that modifies the continuum spectrum [59, 60]. The 4% is therefore read as agreement of the scaling, not as a test. The weight of the states within 0.6 eV of on the 12 pentagon neighbourhoods (3% of the atoms), the 30 edge strips and the 20 facet interiors (shift-invert on C, C, C; weights summed over each I multiplet) shows that the pentagons matter much more than the edges: for C the pentagon enhancement ranges between 0.26 and 3.4 across multiplets, the edge and interior enhancements between 0.75 and 1.3; the states stay extended (IPR ). The class widths (2.5 and 3 Å) are my choice and the edge distance uses the mean radius of a faceted cage, so the ranking of the multiplets is more reliable than the numbers.
Figure 8 shows how the density of states evolves with the size of the fullerene, from C to C: the molecular levels of C merge into bands, the van Hove peaks sharpen, and near the DOS of C cannot be distinguished from that of graphene on the scale of the figure. For the cost, the 600 000-orbital problem C takes 372 s and 1.35 GB for 1500 moments and two vectors (timing on an unrelaxed ideal-sphere geometry, valid as a cost but not as a spectrum; the DOS of Figure 8 is for the relaxed geometry and took 471 s), with 37 nonzeros per row and the exact-gap bisection capped at 25 000 orbitals because of fill.
Figure 8. Density of states of the fullerenes C₆₀ to C₁₅₀₀₀₀ (LCBOP-relaxed geometries, Xu Hamiltonian, KPM with moments, resolution eV) against the DOS of graphene from the same Hamiltonian (dashed). (a) Full band, curves offset vertically. (b) Near .
Amorphous silicon with DFTB and ACE
Figure 4 and Table V, Table VI summarize DTW4096 as a stress test of both branches: the electronic DOS from the Kwon Hamiltonian (exact gap 0.0149 eV at ) and the DFTB problem (generalized, real and complex, gap 0.0095 eV), and the vibrational DOS with the ACE potential [54] that relaxed the structure. The small gaps are properties of this T-point cell, not of the method; the zero-point energy is 60.1 meV/atom and /atom. The tiling of DTW4096 into cells with the SW potential reproduces the single-cell ZPE per atom to (67.0505 and 67.0516 meV/atom for 4096 and 32 768 atoms; matrix-free KPM with , on the same SW-relaxed structure as Table VI). The same structure at , gives 66.573 meV/atom (Table VI) against 66.935 exact, so the dependence on and (the run used , the run ) is larger than the difference between the two cells.
Large amorphous models have been treated with this kind of method before: the Supplementary Information of Morrow et al. [61] shows the electronic density of states of a one-million-atom amorphous-silicon model (their Supplementary fig:3a in the arXiv version, arXiv:2308.16868), computed by the maximum-entropy method of Ref. [9] and compared for increasing numbers of moments, and a gap similar to that of a 512-atom model treated with a hybrid functional. I repeated the calculation for the same structure, using the one-million-atom main-text model deposited by those authors [62] (cubic cell of 273.6 Å, periodic, point; file aSi_1Matom_maintext.xyz) with the orthogonal Kwon tight-binding Hamiltonian: 4 million orbitals, 65 nonzeros per row. At , the DOS takes 58 s (plus 58 s for the spectral bounds and 18 s to build ) with a peak of 10.6 GB; at , it takes 217 s. The band energy is and eV/atom for the two runs (Maxent from the saved moments agrees to meV at every from 60 to 600, and converged in 15–16 Newton steps with the default floor, Sec. II C); the Fermi level is 0.52–0.54 eV. The DTW4096 value ( eV/atom, exact) belongs to a different structure and is not a reference for this model. Figure 9 shows the result. The spectrum has a pseudogap, not a clean gap: between 0.1 and 0.9 eV the density stays at – per orbital (0.02–0.03 states/atom/eV), against for DTW4096. The two runs differ by (2 vs 4) and (300 vs 600) and agree in this region to , so the level is neither stochastic noise nor Chebyshev resolution ( eV at ); it is a property of this structure in this Hamiltonian (with a bond cutoff of 2.85 Å the deposited structure has 6558 threefold atoms, 0.66%, in line with the 0.7% dangling bonds quoted by Morrow et al., and 16 809 fivefold atoms, 1.68%, so about 2.3% coordination defects in all). I did not test the Kwon Hamiltonian against a first-principles calculation of this structure.
A million-atom replica of a 216-atom amorphous cell
V. L. Deringer suggested a test on a periodic replica of a small defect-free amorphous model, the way Ref. [56] treats crystalline Si. I took the 216-atom model R10 of Rosset et al. (the defect-free model of Ref. [54], relaxed here with the same ACE potential) and tiled it times: 1 061 208 atoms, 4 244 832 orbitals, 3 183 624 vibrational degrees of freedom. This is a stringent test for a different reason than DTW4096: the exact spectrum is known for the whole replica, since for a periodic tiling it is the union of the spectra of the cell’s Bloch matrices and on the mesh of points (time-reversal symmetry halves the work). The relation is exact only while the replica stays periodic; the relaxed cell is a stable minimum (no negative eigenvalue), and a short anneal followed by minimization of a replica returned to the same structure, so I did not anneal the large one. It also means that the large system contains no states other than those of the 216-atom cell at the points: a larger sampling of the same network, not a larger network. The electronic and vibrational tests below are therefore tests of the method at scale, not new physics of amorphous silicon.
Electronic structure. With the Kwon Hamiltonian, , and no exact moments the DOS takes 229 s (17 s to build H, peak 12.2 GB). The exact gap of the replica is 1.351 eV (−0.179 to 1.172 eV). Figure 10(a) shows the gap and the tails. The exact edges are sharp, because every state is one of the 216-atom cell’s levels; the Maxent density with and 600 follows them, and drops below per orbital within the gap except for a few narrow spikes (up to ∼ at ).

Figure 10. replica of the 216-atom model R10 (1 061 208 atoms). (a) Kwon tight-binding DOS (4 244 832 orbitals, ) in the gap and tails, against the exact spectrum from the Bloch matrices (0.02 eV bins). (b) Exact vibrational modes of the same replica: median and 10th percentile of the weight on a sound wave of the same wave vector, and median in-cell IPR (times 216; uniform is 1), per 1 meV bin. The dotted line (7 meV) marks the step, which lies near the zone-face frequency meV of the 216-atom cell (see text); it is not a property of the network.
KPM, at the same moment count, has a floor of – per orbital that rises toward the band edges, so a KPM calculation at this cannot distinguish the gap from a faint pseudogap, whereas Maxent can. In cumulative terms the largest error in against the exact count is 1 800 states out of 4 244 832 (0.04%) for Maxent at and 3 200 (0.08%) for KPM; these errors sit in the band tails, not inside the bands.
Vibrations. The matrix-free ACE spectrum (lammps with Kokkos/CUDA on one 12 GB GPU, two force evaluations per matrix–vector product, , ) takes 2.2 h and 1.1 GB of host memory. The zero-point energy agrees with the exact Bloch value to 0.03% (KPM and Maxent, and 300) and 0.07% (Maxent ); at temperatures above 50 K and agree to better than 0.1%. At 5 K, where only the lowest frequencies contribute, KPM at is off by () and (), Maxent by and . The largest error in the integrated mode count below 9 meV, where the exact replica has 25 297 modes, is 118 (KPM) and 130 (Maxent) at , about 0.5%, but at it is 760 (KPM). Figure 11(a) shows the whole VDOS. KPM and Maxent follow the exact histogram over the acoustic and optic bands, including the dip at 28 meV; above about 60 meV, where the spectrum falls to zero at 68.2 meV, both oscillate about the exact curve, and Maxent more strongly (at the oscillation is off the scale of the figure, and it has a spike at the exact maximum). The fraction of modes above 65 meV is nevertheless right (1.38% for Maxent, 1.41% for KPM, 1.39% exact). Figure 11(b) divides by : below 7 meV the exact VDOS is Debye-like, with a flat that both methods reproduce to the resolution of the fit, and above it rises. Below about 3 meV the reconstructions oscillate around the exact histogram, whose bins there hold only a few hundred modes. Figure 10(b) shows the character of the exact modes, measured by the weight on the plane wave (summed over polarizations) with the wave vector of the Bloch state. Below meV (1.5 THz) the modes are sound waves of the cell: this weight is (median) and the in-cell participation ratio equals , as for a uniform cell. The weight falls from 0.95 to 0.03 (median) between 6–7 and 7–8 meV, and above it the modes remain extended in the cell (IPR below 1.5/216) but have no weight on the plane wave of the same . This step is not a property of the network: it marks the Brillouin-zone boundary of the 216-atom cell. The Debye plateau of Figure 11(b) (from the exact mode counts below 3–6 meV, –) gives meV and, with 0.0477 atoms/Å (216 atoms in ), a mean sound velocity km/s ( meV Å), so the zone face at Å (cubic cell, Å) lies at meV; above it the plane-wave content of a Bloch state sits at , not at . To test this I computed the same weight for DTW4096 (cell 43.7 Å, zone face at 2.3 meV; ten points including , 5 049 modes below 14 meV, Bloch matrices from the Gammapoint ACE dynamical matrix, each pair assigned to its minimum image, which is valid because the force cutoff is much smaller than ; script dtw_bloch.py). The weight at the same drops from to a median 0.21 at 2–3 meV and to zero above 3 meV, i.e. at that cell’s zone face, whereas the weight on the best plane wave (all within two reciprocal-lattice steps, adequate below about 9 meV) decreases gradually: the medians are 0.99, 0.97, 0.75, 0.69, 0.44, 0.25 and 0.14 in the 1 meV bins from 2–3 to 8–9 meV. The cell size, not the network, therefore sets the step in Figure 10(b), and I do not claim a propagating–diffuson crossover at 7 meV. The gradual decay of the best-plane-wave weight over roughly 4–9 meV in DTW4096 is the physical statement; I did not compute it for the replica. Eigenvectors. For a replica (17 496 degrees of freedom) shift-invert with the Hessian assembled from finite differences found all 58 modes below 7 meV in 393 s, matching the exact eigenvalues to (residual ). Shift-invert does not scale here (the ACE Hessian has about 900 nonzeros per row), so for the replica the modes shown in Figure 10(b) are Bloch eigenvectors of the cell, not shift-invert results.

Figure 11. Vibrational DOS of the replica of R10 from the matrix-free ACE operator (, ) against the exact Bloch levels. (a) Whole spectrum per mode; the Maxent curve, which oscillates strongly above 60 meV, is omitted, and the Maxent curve also goes off scale above about 62 meV. (b) The low-frequency region divided by for the exact levels (gray, 1 meV bins), KPM (blue), Maxent (green, dashed) and Maxent (orange), as in (a); the dotted line (7 meV) is the same as in Figure 10.
Cost, memory and limits
Machine. All timings are from one Intel Core i5-14400F workstation (WSL2 Linux 6.18, 19 GB of memory visible, 16 logical CPUs of which at most 8 threads were used, usually 4), Python 3.14.4, NumPy 2.5.4, SciPy 1.18.1; wall-clock times of single runs. The GPU (an RTX 5070) is used for the LAMMPS ACE dynamical matrix of DTW4096 and for the matrix-free ACE vibrational spectrum of the million-atom replica (Sec. IV D). The scaling benchmarks use periodic tilings of DTW4096, i.e. not independent structures: they measure computational cost, not physics.
Orthogonal case. Table VIII and Figure 12 for the Kwon Hamiltonian on tilings up to copies (684 736 atoms, 3 538 944 orbitals, 64 nonzeros per row) the whole DOS call (, ) scales linearly with the number of orbitals (fitted exponent 1.013; moments 1.05, bounds 1.01, matrix build 0.97) and takes 7 s at the largest size: 17 s build, 29 s bounds, 24 s moments (100 matrix–vector products, 0.24 s each). With the default exact moments () the same call takes 383 s at the largest size (342 s for the moments, 14× the stochastic moments of the plain run), and the moments again scale linearly with the number of orbitals (exponent 1.09). Peak memory is 9.7 GB at the largest size: the CSR matrix itself is 2.7 GB, the rest is transient copies (the scaled operator and the chunk stack), and the memory exponent 0.79 I attribute to the fixed interpreter and library baseline at small sizes (I did not profile this); a leaner assembly would lower the peak. The matrix-free stencil avoids storing the matrix at all (8.4 million orbitals in 1.8 GB, Sec. IV A) but needs a translation-invariant structure.
| atoms | orbitals | build (s) | bounds (s) | moments (s) | total (s) | peak mem. (GB) | moments, (s) | total, (s) |
| 4 096 | 16 384 | 0.1 | 0.1 | 0.1 | 0.3 | 0.15 | 1.0 | 1.2 |
| 32 768 | 131 072 | 0.6 | 1.0 | 0.9 | 2.5 | 0.45 | 8.6 | 10.0 |
| 110 592 | 442 368 | 2.1 | 3.6 | 2.8 | 8.5 | 1.2 | 31.8 | 36.6 |
| 262 144 | 1 048 576 | 4.9 | 8.9 | 7.1 | 21.0 | 3.1 | 92.3 | 104.2 |
| 512 000 | 2 048 000 | 9.7 | 17.4 | 14.4 | 41.5 | 5.5 | 177.8 | 201.0 |
| 884 736 | 3 538 944 | 17.3 | 29.4 | 24.1 | 70.8 | 9.7 | 341.8 | 382.5 |
Table VIII. Orthogonal case: Kwon tight binding on periodic tilings of DTW4096, KPM with moments and vectors, 4 threads; columns 3–7: no exact-moment substitution (); last two columns: the default (moments, and the whole call including build and bounds), from a separate run. Timings are one run each.
Figure 12. Cost scaling on tilings of DTW4096 (, ). (a) Wall time against the number of orbitals: orthogonal Kwon (total and moments only), non-orthogonal DFTB with PCG/block-Jacobi and with the Chebyshev ; dotted: linear scaling. (b) Peak memory.
Dependence on sparseness. The moments cost about multiply–adds (the doubling identities give two moments per product), so the time is linear in the number of nonzeros (Table IX). From 15 to 255 nonzeros per row I measure 0.8–0.9 ns per nonzero, moment and vector (the entry “ns per ” of the table is per moment, half the time per multiply–add, i.e. 1.6–1.8 ns per multiply–add), and up to 1.8× more at 3 per row, where the per-row overhead matters. This is a bare recursion on a random matrix without locality; the Kwon calculation above (0.24 s per product at nonzeros and ) corresponds to ns, with the whole call also paying for the bounds and the build. Sparseness does not change the stochastic error, which falls as ; the number of moments is set by the bandwidth divided by the resolution. Maxent after the moments costs the same at any sparsity. Dense exact diagonalization scales as and should win only for small or nearly full matrices with comparable to ; I did not time that case.
| nonzeros per row | 3 | 7 | 15 | 31 | 63 | 127 | 255 |
| time (s) | 0.06 | 0.09 | 0.13 | 0.26 | 0.53 | 1.11 | 2.28 |
| ns per (nnz ) | 1.47 | 1.07 | 0.80 | 0.80 | 0.82 | 0.87 | 0.89 |
Table IX. Time of the Chebyshev recursion against the number of nonzeros per row (random symmetric sparse matrix, , , Rademacher vectors, 4 threads; bare recursion: no bounds, no reconstruction).
Non-orthogonal case. Table X: DFTB has 179 nonzeros per row (cutoff 6.0 Å, 4 orbitals per atom) and a Hamiltonian and an overlap matrix, so the peak memory at the same number of orbitals is 2.3 to 3.5 times that of the orthogonal case with PCG and 2.3 to 4.7 times with the Chebyshev polynomial over the four sizes (at the largest, 3.2 and 4.5; ratios from the unrounded peaks in the benchmark files). The time of the whole DOS call grows with the exponent 0.97 (PCG, block-Jacobi, four sizes up to 1 048 576 orbitals) or 0.94 (Chebyshev polynomial, four sizes) and the CG iteration count per solve is independent of size (23.7 at 16 384 orbitals, 23.9 at 442 368 and at 1 048 576), as it should be for a fixed local preconditioner of the well-conditioned overlap. IC(0) reduces the iterations to 11 per solve, but my incomplete Cholesky is a Python loop per column and takes longer overall, so I ran it only for the small sizes. A compiled factorization would change that conclusion. A million-atom non-orthogonal DFTB problem does not fit in 19 GB as explicit and : at 262 144 atoms the PCG run peaked at 9.8 GB and the Chebyshev run at 14.0 GB (912 s, under a 16 GB cap; the first attempt, under a 10 GB cap, was killed), so four times that size would need 40–56 GB.
| atoms | orbitals | nnz/row | build (s) | PCG (BJ) (s) | Chebyshev (s) | PCG (IC(0)) (s) | peak mem., PCG (GB) | peak mem., Cheb. (GB) |
| 4 096 | 16 384 | 179 | 1 | 15 | 18 | 35 | 0.35 | 0.35 |
| 32 768 | 131 072 | 179 | 4 | 117 | 142 | 288 | 1.4 | 1.8 |
| 110 592 | 442 368 | 179 | 15 | 380 | 398 | – | 4.3 | 5.8 |
| 262 144 | 1 048 576 | 179 | 26 | 806 | 912 | – | 9.8 | 14.0 |
Table X. Non-orthogonal case: DFTB (pbc-0-3, non-self-consistent) and on tilings of DTW4096, KPM with , ; time of the whole density-of-states call for the three ways of applying (PCG with block-Jacobi, Chebyshev polynomial, PCG with IC(0)). IC(0) was run only for the small sizes (its setup is a Python loop). At 262 144 atoms the Chebyshev row is a rerun under a 16 GB cap (peak 14.0 GB; the first attempt, under a 10 GB cap, was killed); the PCG row at that size was recovered from the log of the first run (peak 9.8 GB), and the build time is that of the rerun.
Vibrations. A matrix-free product costs two force evaluations. With Stillinger–Weber on one CPU thread a 12 288-degree-of-freedom model takes 5 ms per force call (11 s for , and the bounds) and 98 304 degrees of freedom 37 ms per call (191 s, 5124 calls including the bounds), i.e. linear in the force cost. Only about 200 of these calls belong to the moments (, ); about 4900 (some 2450 Lanczos products) go to the spectral bounds and the Hermiticity probes. In the 4096-atom run the bounds took 9.0 s of 9.9 s and the 102 products of the moments 0.8 s, so the bounds dominate the cost of a matrix-free vibrational density of states at these sizes. I did not investigate why ARPACK needs so many products here; it is not the finite-difference step (changing it from to Å changes the count of force calls for the bounds of the 4096-atom model only from 2124 to 2584), and the window found is correct (Stillinger–Weber, [0, 1.844]). With ACE the force call dominates: on CPU it takes 0.73 s for 4096 atoms (4 threads) and the matrix-free DOS of DTW4096 took 1306 s for 1768 force calls; with a Kokkos/CUDA build of LAMMPS on one 12 GB GPU through its Python module the million-atom replica of Sec. IV D (1220 force calls for , , with the spectral window taken from the exact force-constant spectrum) took 2.2 h, i.e. about 6.6 s per call including the transfer of positions and forces through Python, and 1.1 GB of host memory.
What does not scale. (i) Shift–invert and the exact HOMO–LUMO bisection are not because of fill; for the Xu fullerenes the bisection takes 27 s at 15 360 orbitals and 62–80 s at 19 440–24 000, and the exact-gap bisection costs more than the DOS above orbitals. (ii) Exact moments of order cost a time that grows quickly with (1.8, 9 and 80 s per run at , 8, 16 for 16384 orbitals), although it is linear in for fixed . (iii) The spectral bounds are not negligible and grow with (162 to 522 matrix–vector products from 16 384 to 3.5 × orbitals of the stencil, against 340 for the moments with , ); for a crystal they can be given analytically. For a matrix-free operator they can also be wrong: on a relaxed one-million-atom ACE Hessian (the Morrow et al. model of Sec. IV C, whose vibrational density of states is not part of this report; finite-difference step Å) the Lanczos window came out as [-29, 69] in units of eV Å amu, whereas the spectrum lies in about [0, 1.34] (a plain 30-step Lanczos iteration with full reorthogonalization and a step of Å converges to extreme Ritz values 0.0035 and 1.336; the exact ranges of the 216-atom cell replica and of DTW4096 are [0, 1.11] and [0, 1.27]). The cause is the finite-difference step: the displacement is normalized to an rms of Å over all components, so a Lanczos vector that localizes on a few atoms is displaced by a large amount per atom, the product is then not that of a fixed linear operator, and the Ritz values leave the spectrum (the same iteration with the step of Å reaches [-5.3, 20] after 20 steps). I inferred the localization from this step dependence and did not measure it. With a window of [-29, 69] and moments the resolution is 0.26, a quarter of the band, and the result is useless; the symptom is a density with weight at (the result is flagged negative_indicated) and a window much wider than the physical range; a window that is too narrow is as bad (the moments of a run with [-0.05, 1.3] grew to ). For matrix-free vibrational problems I therefore recommend estimating the window with a smaller step (here Å) or passing bounds from a known range (a crystal, or a small cell of the same potential), and checking the window before a long run; the runs of this report that used it (Sec. IV D) took the window from the exact spectrum of the force-constant matrix. (iv) Max-ent with many exact moments stops converging unless the moments are given a small uncertainty (Sec. III B). Even then its pointwise error against the 0.05 eV-broadened exact spectrum is smallest at and rises at and 400, while KPM’s keeps falling (Table IV). The floor is not the cause (Sec. III B); part of it is the test (a sharp Maxent curve is compared with a broadened spectrum), and for the band energy, an integrated quantity, Maxent is the more accurate at every in Table IV. I do not claim that a converged Maxent is worse than KPM in general: for integrated quantities it is better, and only for pointwise resolution at large do I recommend KPM. (v) The DFTB matrices are large: 179 nonzeros per row. (vi) The statistical error of a moment decreases with , so large systems need few vectors, but the band energy per atom of one 4096-atom cell with vectors has an uncertainty of order 0.1 eV/atom (Table V; rms 110 meV/atom at ) unless exact moments or impartial vectors are used (Table I), and for the generalized problem the variance reductions of Table I are not available; integrated quantities should be reported with their error. (vii) Not checked against an independent reference: the complex generalized problem
lem at a point (Sec. III D), the generalized problem through the result file (§ 2 G), the Mulliken weights and IPR (tested through sum rules and one shift-invert case), and the one-million-atom amorphous run. (viii) In the replica of Sec. IV D Maxent oscillates strongly where the vibrational spectrum ends abruptly (above 60 meV at ), although the integrated weight is right; shift-invert is practical only for small replicas, and the exact references there exist because the replica is periodic. (ix) Finally, all numbers are -point or single-twist values for one structure each, except the crystal and replica comparisons, whose references are exact -sums; the dependence on the structure and the cell size is a separate question.
Recommendations
KPM is the default: linear, positive (Jackson), never fails, resolution ; for the maximum resolution of a large system increase (a few thousand is cheap; cost is linear in ). Beyond the
mean level spacing or the noise floor, more moments only sharpen sampling speckle.
Maxent (keep the default uncertainty floor, which makes it converge with exact or nearly exact moments) when high-order moments are expensive or noisy (), when positivity and sharp edges matter, and for integrated quantities (band energy, gap region); from identical exact moments at – (the sizes where it converged) it gave 8–12× lower band-energy errors and 6–30× lower distances (Table IV). Use the bootstrapped solver; with the default uncertainty floor it converges also on exact moments, but its pointwise error, measured against a broadened exact spectrum, rises beyond a few hundred moments of singular densities (Table IV).
Random vectors: the default exact-moment substitution () is almost always worth it for KPM, and for Maxent with vectors; with Maxent and a single vector use or impartial vectors (Sec. III C); – for band energies; impartial vectors constrain only three moments and are nearly as efficient at lower cost per vector (Table I).
Generalized problems: PCG with block-Jacobi is the default and my plain recommendation; the Chebyshev polynomial needs no inner solves, which makes it the option for matrix-free or GPU settings where no iterative-solver stack is wanted, but in Table X it was slower and needed more memory than PCG; IC(0) lowers the iteration count only if its setup is cheap (mine is not).
Report the sampling error of every integrated quantity, and overlay KPM and Maxent: where they agree, the feature is in the moments.
Conclusions and availability
SDOS computes densities of states and related spectral quantities of very large sparse Hermitian and generalized problems. The ingredients are established: stochastic traces, KPM, Maxent, and non-orthogonal KPM by PCG. What is added is their combination, the choices for the overlap inverse, bootstrapped Maxent, windows, projections, the interfaces, and a validation against exact diagonalization and independent codes from to orbitals, including the exact Bloch spectra of a million-atom replica of a 216-atom amorphous cell, for which Maxent resolves the gap edges that KPM smears and the matrix-free ACE vibrational spectrum reproduces the zero-point energy to 0.07%. The moment-based computations scale linearly in the number of orbitals for explicit matrices (exponent 1.01) and nearly so for the matrix-free crystal (1.12). Three conventions of the DFTB file format that I did not find in the documentation available to me are recorded here. The limits are stated in § 2.5.
Availability. The package is licensed under BSD-3-Clause (copyright D. A. Drabold, 2026) and is available at https://github.com/Drabold/sdos and archived on Zenodo (version 0.1.0, concept DOI 10.5281/zenodo.24023874); it installs with pip install sdos (PyPI) or from a clone with pip install -e .. The repository contains the examples and benchmark scripts of this report; the large run data (million-atom structures and exact spectra) and the DFTB parameter files are not included. All scripts, saved results and the generation of the figures and tables of this report are in the examples/ and paper/ directories of the repository; the tests run with pytest.
Acknowledgments
AI assistance: the package code, the tests and the benchmark and analysis scripts were written, and the text of this report was drafted, with Claude Sonnet 5.5 and Claude Opus 5.5 (Anthropic), used through Claude Code. I reviewed and validated all AI-assisted output and made the design decisions; I am responsible for all scientific statements. I acknowledge the Frauenheim group’s pbc-o-3 Slater–Koster set (CC BY-SA 4.0), whose required references I cite. I thank R. N. Silver, H. Röder, J. J. Dong, P. Ordéjon and R. M. Martin for their insights and collaboration, C. Ugwumadu for a careful reading of the whole manuscript and many helpful suggestions, E. T. Drabold for advice on hardware selection, V. L. Deringer for suggesting the million-atom replica test, and L. A. M. Rosset and V. L. Deringer for the silicon–hydrogen ACE potential used in the benchmarks.
References
References
- [1]J. Skilling, The eigenvalues of mega-dimensional matrices, in Maximum Entropy and Bayesian Methods (Springer, Dordrecht, 1989) pp. 455–466.DOI
- [2]M. Hutchinson, A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines, Communications in Statistics - Simulation and Computation 19, 433 (1990).
- [3]R. N. Silver and H. Röder, Densities of states of mega-dimensional Hamiltonian matrices, International Journal of Modern Physics C 5, 735 (1994).DOI
- [4]R. Silver, H. Roeder, A. Voter, and J. Kress, Kernel polynomial approximation for densities of states and spectral functions, Journal of Computational Physics 124, 115 (1996).DOI
- [5]A. Weisse, G. Wellein, A. Alvermann, and H. Fehske, The kernel polynomial method, Reviews of Modern Physics 78, 275 (2006).
- [6]D. Jackson, On approximation by trigonometric sums and polynomials, Transactions of the American Mathematical Society 13, 491 (1912).DOI
- [7]L. R. Mead and N. Papanicolaou, Maximum entropy in the problem of moments, Journal of Mathematical Physics 25, 2404 (1984).DOI
- [8]I. Turek, A maximum-entropy approach to the density of states within the recursion method, Journal of Physics C: Solid State Physics 21, 3251 (1988).DOI
- [9]D. A. Drabold and O. F. Sankey, Maximum entropy approach for linear scaling in the electronic structure problem, Physical Review Letters 70, 3631 (1993).
- [10]E. T. Jaynes, Where do we stand on maximum entropy?, in The Maximum Entropy Formalism, edited by R. D. Levine and M. Tribus (MIT Press, Cambridge, MA, 1978) pp. 15–118.
- [11]S. Goedecker, Linear scaling electronic structure methods, Reviews of Modern Physics 71, 1085 (1999).
- [12]P. Ordejón, Order-n tight-binding methods for electronic-structure and molecular dynamics, Computational Materials Science 12, 157 (1998).
- [13]R. Haydock, V. Heine, and M. J. Kelly, Electronic structure based on the local atomic environment for tight-binding bands, Journal of Physics C: Solid State Physics 5, 2845 (1972).
- [14]L.-W. Wang, Calculating the density of states and optical-absorption spectra of large quantum systems by the plane-wave moments method, Physical Review B 49, 10154 (1994).
- [15]S. Goedecker and L. Colombo, Efficient linear scaling algorithm for tight-binding molecular dynamics, Physical Review Letters 73, 122 (1994).
- [16]L. Lin, Y. Saad, and C. Yang, Approximating spectral densities of large matrices, SIAM Review 58, 34 (2016).
- [17]H. Röder, R. N. Silver, and J. D. Kress, The kernel polynomial method for non-orthogonal electronic structure calculations, in High Performance Computing 1996: Grand Challenges in Computer Simulation, edited by A. Tentner (Society for Computer Simulation, San Diego, 1996) report CONF-960482, OSTI ID 390779.
- [18]H. Röder, R. N. Silver, D. A. Drabold, and J. J. Dong, Kernel polynomial method for a nonorthogonal electronic-structure calculation of amorphous diamond, Physical Review B 55, 15382 (1997).
- [19]S. M. João, M. Andelković, L. Covaci, T. G. Rapopport, J. M. V. P. Lopes, and A. Ferreira, KITE: high-performance accurate modelling of electronic structure and response functions of large molecules, disordered crystals and heterostructures, Royal Society Open Science 7, 191809 (2020).
- [20]C. W. Groth, M. Wimmer, A. R. Akhmerov, and X. Waintal, Kwant: a software package for quantum transport, New Journal of Physics 16, 063065 (2014).
- [21]D. Moldovan, M. Andelković, and F. Peeters, Pybinding v0.9.5: a Python package for tight-binding calculations (2020).
- [22]Y. Li, Z. Zhan, X. Kuang, Y. Li, and S. Yuan, TBPLaS: A tight-binding package for large-scale simulation, Computer Physics Communications 285, 108632 (2023).
- [23]O. F. Sankey, D. A. Drabold, and A. Gibson, Projected random vectors and the recursion method in the electronic-structure problem, Physical Review B 50, 1376 (1994).
- [24]A. F. Voter, J. D. Kress, and R. N. Silver, Linear-scaling tight binding from a truncated-moment approach, Physical Review B 53, 12733 (1996).
- [25]R. N. Silver and H. Röder, Calculation of densities of states and spectral functions by chebyshev recursion and maximum entropy, Physical Review E 56, 4822 (1997).
- [26]J. M. Tang and Y. Saad, A probing method for computing the diagonal of a matrix inverse, Numerical Linear Algebra with Applications 19, 485 (2012).
- [27]R. A. Meyer, C. Musco, C. Musco, and D. P. Woodruff, Hutch++: Optimal stochastic trace estimation, in Symposium on Simplicity in Algorithms (SOSA) (2021) pp. 142–155.DOI
- [28]L. Fejér, Untersuchungen über Fouriersche Reihen, Mathematische Annalen 58, 51 (1904).DOI
- [29]R. Courant and D. Hilbert, Methods of Mathematical Physics, Vol. 1 (Interscience, New York, 1953).
- [30]J. A. Meijerink and H. A. van der Vorst, An iterative solution method for linear systems of which the coefficient matrix is a symmetric m-matrix, Mathematics of Computation 31, 148 (1977).
- [31]T. A. Manteuffel, An incomplete factorization technique for positive definite linear systems, Mathematics of Computation 34, 473 (1980).
- [32]Y. Zhou, Y. Saad, M. L. Tiago, and J. R. Chelikowsky, Self-consistent-field calculations using chebyshev-filtered subspace iteration, Journal of Computational Physics 219, 172 (2006).
- [33]R. S. Mulliken, Electronic population analysis on lcaomo molecular wave functions. i, The Journal of Chemical Physics 23, 1833 (1955).
- [34]C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, et al., Array programming with NumPy, Nature 585, 357 (2020).
- [35]P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, et al., SciPy 1.0: fundamental algorithms for scientific computing in python, Nature Methods 17, 261 (2020).
- [36]I. Kwon, R. Biswas, C. Z. Wang, K. M. Ho, and C. M. Soukoulis, Transferable tight-binding models for silicon, Physical Review B 49, 7242 (1994).pubmed.ncbi.nlm.nih.gov/10009461
- [37]C. H. Xu, C. Z. Wang, C. T. Chan, and K. M. Ho, A transferable tight-binding potential for carbon, Journal of Physics: Condensed Matter 4, 6047 (1992).DOI
- [38]T. Frauenheim, The pbc–o–3 slater–koster parameter set, Universität Bremen, CC BY-SA 4.0 (dftb.org) (2017).
- [39]A. Sieck, Doctoral thesis, Ph.D. thesis, University of Paderborn (2000), cited as required by the README of the pbc-o-3 Slater–Koster set.
- [40]A. Sieck, T. Frauenheim, and K. A. Jackson, Shape transition of medium-sized neutral silicon clusters, physica status solidi (b) 240, 537 (2003).
- [41]D. Porezag, T. Frauenheim, T. Köhler, G. Seifert, and R. Kaschner, Construction of tight-binding-like potentials on the basis of density-functional theory: Application to carbon, Physical Review B 51, 12947 (1995).
- [42]M. Elstner, D. Porezag, G. Jungnickel, J. Elsner, M. Haugk, T. Frauenheim, S. Suhai, and G. Seifert, Self-consistent-charge density-functional tight-binding method for simulations of complex materials properties, Physical Review B 58, 7260 (1998).
- [43]N. Papior, sisl: a python package for tight-binding, dft and siesta post-processing, https://github.com/zerothi/sisl (2026), software; accessed 2026.
- [44]J. M. Soler, E. Artacho, J. D. Gale, A. García, J. Junquera, P. Ordejón, and D. Sánchez-Portal, The SIESTA method for ab initio order- n materials simulation, Journal of Physics: Condensed Matter 14, 2745 (2002).DOI
- [45]A. García, N. Papior, A. Akhmerov, E. Artacho, V. Blum, E. Bosoni, et al., Siesta : Recent developments and applications, The Journal of Chemical Physics 152, 124101 (2020).
- [46]A. P. Thompson, H. M. Aktulga, R. Berger, D. S. Bolintineanu, W. M. Brown, P. S. Crozier, et al., LAMMPS - a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales, Computer Physics Communications 271, 108171 (2022).DOI
- [47]B. Hourahine, B. Aradi, V. Blum, F. Bonafé, A. Buccheri, C. Camacho, et al., DFTB+, a software package for efficient approximate density functional theory based atomistic simulations, The Journal of Chemical Physics 152, 124101 (2020).
- [48]F. Wooten, K. Winer, and D. Weaire, Computer generation of structural models of amorphous Si and Ge, Physical Review Letters 54, 1392 (1985).pubmed.ncbi.nlm.nih.gov/10031020
- [49]B. R. Djordjević, M. F. Thorpe, and F. Wooten, Computer model of tetrahedral amorphous diamond, Physical Review B 52, 5685 (1995).pubmed.ncbi.nlm.nih.gov/9981754
- [50]D. A. Drabold, P. Ordejón, J. Dong, and R. M. Martin, Spectral properties of large fullerenes: From cluster to crystal, Solid State Communications 96, 833 (1995).DOI
- [51]P. N. Keating, Effect of invariance requirements on the elastic strain energy of crystals with application to the diamond structure, Physical Review 145, 637 (1966).DOI
- [52]R. Drautz, Atomic cluster expansion for accurate and transferable interatomic potentials, Physical Review B 99, 014104 (2019).DOI
- [53]Y. Lysogorsky, C. v. d. Oord, A. Bochkarev, S. Menon, M. Rinaldi, T. Hammerschmidt, et al., Performance implementation of the atomic cluster expansion (PACE) and application to copper and silicon, npj Computational Materials 7, 97 (2021).
- [54]L. A. M. Rosset and V. L. Deringer, Atomic cluster expansion potential for the Si-H system, Physical Review Materials 10, 044604 (2026).arxiv.org/abs/2510.15633
- [55]F. H. Stillinger and T. A. Weber, Computer simulation of local order in condensed phases of silicon, Physical Review B 31, 5262 (1985).pubmed.ncbi.nlm.nih.gov/9936488
- [56]V. L. Deringer, N. Bernstein, G. Csányi, C. Ben Mahmoud, M. Ceriotti, M. Wilson, D. A. Drabold, and S. R. Elliott, Origins of structural and electronic transitions in disordered silicon, Nature 589, 59 (2021).DOI
- [57]S. Itoh, P. Ordejón, D. A. Drabold, and R. M. Martin, Structure and energetics of giant fullerenes: An order-molecular-dynamics study, Physical Review B 53, 2132 (1996).pubmed.ncbi.nlm.nih.gov/9983678
- [58]J. H. Los and A. Fasolino, Intrinsic long-range bond-order potential for carbon: Performance in monte carlo simulations of graphitization, Physical Review B 68, 024107 (2003).
- [59]J. González, F. Guinea, and M. A. H. Vozmediano, Continuum approximation to fullerene molecules, Physical Review Letters 69, 172 (1992).pubmed.ncbi.nlm.nih.gov/10046217
- [60]J. González, F. Guinea, and M. A. H. Vozmediano, The electronic spectrum of fullerenes from the Dirac equation, Nuclear Physics B 406, 771 (1993).DOI
- [61]J. D. Morrow, C. Ugwumadu, D. A. Drabold, S. R. Elliott, A. L. Goodwin, and V. L. Deringer, Understanding defects in amorphous silicon with million-atom simulations and machine learning, Angewandte Chemie International Edition 63, e202403842 (2024).
- [62]J. D. Morrow, C. Ugwumadu, D. A. Drabold, S. R. Elliott, A. L. Goodwin, and V. L. Deringer, Research data for “understanding defects in amorphous silicon with million-atom simulations and machine learning”, Zenodo (2024), doi:10.5281/zenodo.10794053.









