betaFeedback

WorksPhysics and Astronomy

DL_POLY 5: Calculation of system properties on the fly for very large systems via massive parallelism

Abstract

Modelling has become a third distinct line of scientific enquiry, alongside experiments and theory. Molecular dynamics (MD) simulations serve to interpret, predict and guide experiments and to test and develop theories. A major limiting factor of MD simulations is system size and in particular the difficulty in handling, storing and processing trajectories of very large systems. This limitation has become significant as the need to simulate large system sizes of the order of billions of atoms and beyond has been steadily growing. Examples include interface phenomena, composite materials, biomaterials, melting, nucleation, atomic transport, adhesion, radiation damage and fracture. More generally, accessing new length and energy scales often brings qualitatively new science, but this has currently reached a bottleneck in MD simulations due to the traditional methods of storing and post-processing trajectory files. To address this challenge, we propose a new paradigm of running MD simulations: instead of storing and post-processing trajectory files, we calculate key system properties on-the-fly. Here, we discuss the implementation of this idea and on-the-fly calculation of key system properties in the general-purpose MD code, DL_POLY. We discuss code development, new capabilities and the calculation of these properties, including correlation functions, viscosity, thermal conductivity and elastic constants. We give examples of these on-the-fly calculations in very large systems. Our developments offer a new way to run MD simulations of large systems efficiently in the future.

PROGRAM SUMMARY/NEW VERSION PROGRAM SUMMARY

Program Title: DL_POLY_5

CPC Library link to program files: (to be added by Technical Editor)

Developer’s repository link: https://gitlab.com/ccp5/dl-poly

Licensing provisions: L-GPL v3.0

Programming language: Fortran 2008

Supplementary material:

Nature of problem: Molecular dynamics is utilised for modelling in many domains including physics, chemistry, biology, materials science, and their interfaces. These applications often call for large scale simulations targeting high-fidelity timescales and ever larger length scales. In all these cases efficient use of computation, storage, and input/output (I/O) load handling on both the software and hardware sides is required to facilitate analysis.

Solution method: DL_POLY provides an efficient set of algorithms for molecular simulation alongside a domain-decomposition strategy to distributed computation efficiently across massively parallel CPU systems. These include parallel I/O handling, link-cells, smooth particle mesh Ewald electrostatics, and now a general purpose on-the-fly correlation framework. This latter development addresses the problem of storing and analysing large trajectory files by iteratively computing correlations at runtime.

Introduction

For a few decades now, molecular simulations have grown into a third distinct line of scientific enquiry in condensed matter, alongside experiment and theory [1, 2]. Molecular dynamics (MD) simulations give coordinates and momenta of simulated particles as a function of time and thus provide the system’s trajectory in phase space. This enables us to calculate most important system properties including those related to structure, statistics, dynamics, transport and so on.

MD results are used in a number of ways to provide insight into experiments, from interpretation to prediction and guiding new ones, as well as to test theories and provide insight at the atomic level resolution for materials, scenarios or processes not yet synthesised by researchers or constructed by experimentalists. MD is particularly useful in fields where measurements are inaccessible in experiment, where empirical theory of phenomena at the larger engineering scales can be examined at short time and length scales, such as damage in materials [3]. Another popular use is extracting statistics and analytics that are impossible to capture by experiment such as cheminformatics analysis (formation of hydrogen bond networks, hydrophobicity, π\pi-π\pi stacking, etc.), the dynamics of defects in materials (defects and faults in crystalline matrices, amorphisation processes) as well as coarse-grained modelling using augmented force-fields of complex dynamics of, for example, proteins. Generally, MD results are also used to generate synthetic benchmark data for fortifying machine-learnt (ML) models.

This interaction between experiments, theory and simulation has proven to be very productive and has naturally found wide applications in physics, chemistry, materials and earth sciences, engineering, biological and medical sciences etc, from purely academic research to real-world industrial applications. These applications include those in the strategic priority areas of energy, environment, advanced materials and health. It is hard to imagine the current state of these areas without MD and other types of computer modelling. Many practising scientists have taken MD simulations on board to the extent that they consider themselves as combined experimentalist-modellers or theorist-modellers.

As for any method, MD simulations have their limitations and associated domain of applicability. Several important ones have been relaxed or lifted through the developments of the last decades; others remain unsolved, including the impact of small system size on on simulated properties, such as diffusion coefficients [4].

One of the strengths of MD is its ability to access small length scales of nanometers and short time scales of picoseconds or beyond where few experimental methods can measure. Early MD simulations were able to simulate about 100 particles for typically tens of picoseconds. With increasing com puter power, simulations of millions of atoms on a personal computer is now routine, with simulation times extending to nanoseconds and above.

The appetite of the modelling community to simulate larger systems is constantly growing. In many areas, large systems are necessary to reach the required length or energy scales and match these to experiments and relevant physical processes. These areas include interface phenomena, composite materials, biomaterials, melting, nucleation, atomic transport, adhesion, radiation damage and fracture.

About a decade ago, we simulated system sizes approaching 1 billion atoms with realistic many-body potentials for iron, using over 60,000 parallel processors [5]. This large system size was required in order to contain radiation damage due to high-energy (in MeV) recoil energy. A representative picture of the damage is shown in Figure 1. We note that the system size, in the order of micrometers, is in interesting proximity to the length scales visible through an optical microscope; simulations and experiments met at this length-scale for the first time.

A representative 0.5 MeV collision cascade

Figure 1. A representative 0.5 MeV collision cascade from our earlier work [5] showing displaced atoms at 0.2, 1.5 ps and 100 ps. The system size is a cube of 2000 Å size with 0.1 billion atoms. Animations showing the propagation of the collision cascade can be watched from [6].

We found that the nature of the damage and defects in these structures were new and could not be extrapolated from lower-energy events. This had a clear value from scientific perspective and showed that, as is often the case in physics, exploring new length- and energy-scales brings about new effects [5].

Running these very large simulations, not only brought scientific insights, but also computational insights highlighting the scale of the data. In single precision, the uncompressed positions of 100 million atoms for one timestep costs approximately 1.2GB of disc storage. Computed velocity and force data brings this to 3.6GB. In an MD simulation, certain structural and dynamical properties (for example, viscosity or velocity correlation functions, momentum currents, and radial distribution functions) often need over 10,000–100,000 configurations to be computed, requiring usually two orders of magnitude more timesteps in order to get good-quality data. For such analysis the data requirement reaches 100’s of Terabytes. On the other hand, the work file system of a high-performance facility such as UK’s ARCHER2 service [7] has a total of 10.2 PB [8] to be shared between all users (a maximum of 60 TB allocated to a single project or team). Going beyond to 1 billion atoms will require Petabytes of data and yet simulations of this magnitude are increasingly being performed [9, 10, 11, 12]. The magnitude of these storage requirements presents a significant storage challenge which must be addressed before any post-processing to obtain results.

Second, a high-performance computing facility such as ARCHER2 spends a significant amount of its time writing the data rather than actually carrying out the numerical simulation, which is an inefficient way of utilising the HPC resource. Even with efficient parallelisation of writing data to storage, a best outcome means that writing a single sample configuration costs as much in CPU time as a timestep. In the case of 100 million atoms run on 65,000 processors, this fraction was in excess of 5% for a full production run, including reading and writing. This implies that each simulation run involving writing the trajectory file, even if feasible in terms of file size, wastes considerable compute power. Furthermore, the larger the simulation (in model size and time evolution) and the larger the count of processors used to run it, the larger the waste.

In addition to these problems many MD use-cases rely on the analysis of rare events such as protein folding or phase transitions. Both accessing these events and the analysis of them in MD exacerbates the issue of trajectory storage. In this domain the exploitation of collective variables (CV) [13] both to inform sampling or otherwise bias dynamics and to detect or drive these events is well used. As a result software such as PLUMED [14, 15, 16] and A4MD [17] act as in situ monitors of MD simulations to ingest data trajectory data at runtime to monitor CVs and act upon them.

The problems however extend more generally and will either prevent most users from running and analysing MD simulations of large systems or make these simulations inefficient. In general this challenge can be partially addressed with multiple methodologies related to data compression. Plain ASCII is often a default and the simplest schemes merely reduce written significant figures (for example this is optional in DL_POLY, and also with GROMACS’s XTC format [18]). Nevertheless, this comes at potentially significant accuracy losses. Alternatively common lossless formats such as GZIP are also offered, for example in LAMMPS [19] which can achieve significant file size reductions. However more advanced algorithms specifically for floating-point data exists such as ZFP [20]. This compression can be performed at a relatively low cost including into efficient domain specific data-layouts such as H5MD [21] that utilises HDF5 [22] for MD data. For example lossless compression ratios of around 1.44 with ZFP compared with 1.09 for GZIP on the same test data suite can be achieved. With lossy compression this rises up to 10.98 or 226.3 with 32-bit or 16-bit precision. ZFPs throughput rates of around 20 MB/s to 60 MB/s for 64 and 32 bit precision on single CPU cores makes this a negligible cost [20]. In the lossless case this can be expected to result in a 30% reduction in I/O overheads. If 32-bit (or less) lossy precision is acceptable using ZFP compressed trajectory output and on-the-fly methods will likely perform similarly in terms of workflow core hours. Since the data foot-print will be reduce by around 90% (or more). Note however that the on-the-fly method involves no precision loss by bit-per-float reduction or lossy compression. It may also be used in conjunction with a more manageable lossy trajectory write-out. For extremely large systems lossless storage and post-processing analysis are still problematic.

Alternatively these problems can be addressed by bypassing trajectory storage entirely. In the simplest cases observables can trivially be computed each step, on-the-fly, without (or whilst) saving the trajectory such as the stress tensor. A more complex example is our radiation damage case study - we overcame these problems by implementing a special on-the-fly algorithm in the general-purpose MD package, DL_POLY [23, 24]. This algorithm specifically finds and analyses displaced and defect atoms in the very large billion-atom system. This was a non-trivial task as the algorithm had to be made consistent with the domain-decomposition infrastructure of DL_POLY [25, 26]. These simulations have provided a useful case study and a motivation for the development of a more general code, with a view to benefit a wider modelling community and to approach the problems in a general way.

Recall that some properties such as energy can be sampled and averaged at each time-step, and these are calculated by DL_POLY and other popular MD codes as a standard output. However, there are other important properties relying on time correlation functions which need to be calculated over the entire trajectory and which are currently calculated by writing out the trajectory file and analysing it a posteriori. One common example is the velocity autocorrelation function related to the phonon density of states and power spectrum of the system. Other examples include k-space densities and currents[27]. For very large system sizes, the problems discussed above make performing these analyses from trajectory files impractical.

Instead of attempting to save a trajectory and subsequently analysing it, we have enabled the calculation of important physical properties on-the-fly. This sort of on-the-fly calculation will be crucial to the execution and analysis of ever-larger MD simulations, particularly as there is a growing need to simulate larger systems, length and energy scales, coupled with the availability of scalable MD codes and massive parallel computing facilities.

Supported by the EPSRC funding, we have extended the UK flagship MD code, DL_POLY, to calculate system properties on-the-fly culminating in the DL_POLY 5 code available at [28]. DL_POLY is a general-purpose massively parallel MD simulation package that uses a highly efficient set of methods and algorithms, including domain decomposition, linked cells, Daresbury Advanced Fourier Transform, Trotter derived Velocity Verlet integration and RATTLE [24]. Written to support academic and industrial research, DL_POLY has a wide range of applications and can run on a wide range of computers: from single processor workstations to massively parallel high-performance facilities. During development a strong emphasis was placed on efficient utilization of multi-processor power by optimising memory workload and distribution. This makes it possible to efficiently simulate large systems of billions of atoms. Simulation of systems of this size has been tested, and we are not aware of reasons preventing from running larger systems. DL_POLY 5 is hosted on GitLab [28] has been a free and open source project (GPLv3.0) since 2020.

In this paper, we document and detail the developments of DL_POLY related to on-the-fly calculation of some key system properties. This included general and substantial refactoring across DL_POLY’s modules to enable the new developments. We begin by discussing the theory and implementation of on-the-fly calculations of correlation functions. We then show results from simulations of small and large scales - up to 100 million atoms - for calculating different properties including viscosity and thermal conductivity, elastic constants, currents, and rigid body properties. These results are compared with experimental results and previous MD simulations. After showing the case studies we discuss benchmarks of our work alongside the costs and benefits or using on-the-fly methods in DL_POLY, and more generally. We finish with an overview of DL_POLY’s recent history since our last published work [24], the 4th and now 5th major releases. Including in particular the refactoring which laid the groundwork for the on-the-fly developments.

System properties on-the-fly

On-the-fly methods

Transforming data as it is acquired is a paradigm often termed “online algorithms” [29]. The basic idea is that, instead of processing an entire data set at once, an on-the-fly calculation processes the data incrementally, often as the data are generated, received, or simply by processing a data set in chunks. This can incur performance advantages by eliminating the requirement to store the full data set on disc or in memory (RAM). Alternatively input/output (I/O) penalties can be significant when reading from disk, and RAM may be limited. In some cases, the full data set may be either too large to store in memory, or perhaps even unbounded. One example of unbounded data is the internet, where a continuous stream of “big”-data is encountered on a daily basis [30]. In these applications a sliding-window technique is often used where statistics and other data transformations are defined in reference to a fixed-size window [31] which could be temporal or spatial.

In MD, the phase space configuration of atoms is mapped through simulation time-steps into a trajectory. In order to calculate any properties defined by these time-linked configurations the trajectory may be stored on disc, and an algorithm applied. For example storing atom velocities and calculating the velocity auto-correlation function (VAF). For small systems, it may be possible to store these data at every step, obtaining a VAF accurate to the time-step dtdt. As the system size scales this becomes infeasible both in terms of storage and I/O penalties if storing to disc. By conceptualising the simulation trajectory as a big-data stream, we can apply the same ideas of online algorithms to calculate system properties on-the-fly. For modest systems, this presents a convenience. However for large systems, this methodology can give access to properties which otherwise require infeasible storage requirements or some accuracy trade-off.

Multiple system properties depend upon the calculation of correlation functions. As an incomplete list: (1) The VAF is one example which can in turn be used to calculate the vibrational density of states. (2) From Green-Kubo theory [32, 33] the thermal conductivity and viscosity can be calculated by using correlation functions of the heat flux and stress tensor respectively [32, 33]. (3) Elastic constants may also be calculated using either strain correlations [34] or stress correlations [35, 36]. (4) kk-space density correlations may be used to determine thermal-conductivity [37], and momentum current correlations may be used to calculate structure factors and liquid phonon spectra [27, 38].

Motivated by these examples, and others, in DL_POLY we implemented a general on-the-fly correlation algorithm, so that we are able to calculate these values at runtime; that is as the system is simulated, without storing trajectory data to disc.

Correlation functions at runtime

Given two observable quantities X(sΔt)X(s\Delta t) and Y(sΔt)Y(s\Delta t) in MD, where s∈1,2,…Ss \in1, 2, \ldots S are simulation time-steps for a time-step of Δt\Delta t, we can evaluate the correlation function of these two values at time-lags l=1,2,…,L≪Sl = 1, 2, \ldots, L \ll S as [33],

CXY(lΔt)=1S−l∑s′=1S−lX(s′Δt)Y((s′+l)Δt)(1)C_{XY}(l\Delta t) = \frac{1}{S-l}\sum_{s'=1}^{S-l} X(s'\Delta t)Y((s'+l)\Delta t) \tag*{(1)}

for one trajectory. Typically statistics are accumulated over multiple trajectories. In any case, directly evaluating equation 1 is cumbersome as SS and LL increase. Even more so if the observable is averaged over atoms of molecules (for example, in the case of the VAF). Therefore, often the discrete Fourier transforms of XX and YY are taken to access the correlation function using the inverse transform of their product [33]. Whether the sum is taken directly or a Fourier transform method is used XX and YY must be stored in memory or on disc for the computation.

We use the multiple-tau correlation algorithm [39] within DL_POLY to calculate correlations on-the-fly. The method works by maintaining an hierarchical set of blocks which are updated during the simulation. The blocks store observed data (velocities, stresses, etc.) in a decreasing resolution. The first block stores the observed data intact whilst lower blocks store block averages, computed over the previous level. This is controlled by three parameters, the number of blocks bb, the number of points within each block pp, and the size of the block average m<pm < p. See Appendix § 11 for algorithmic details of our implementation based upon [39].

This approach is similar to the velocity correlator introduced by Frenkel and Smit [2] with the additional separation of pp (data points stored at each level) and mm the averaging parameter. By increasing bb and pp (with mm fixed), later correlation times may be collected at a fixed accuracy. Increasing mm has the effect of calculating much longer correlation times for a trade off in accuracy and increased performance. The maximum lag-time of a correlation given these parameters is defined by (p−1)mluΔt(p-1)m^{l}u\Delta t where Δt\Delta t is the simulation time-step and the frequency the correlation is updated is uu. The trade off in performance terms scales as O(pm+1m)O(p^{\frac{m+1}{m}}). The actual performance will also depend on the values correlated (e.g. velocity across all atoms, or system stress) and other system options (such as electrostatics).

In DL_POLY we support correlating key quantities such as components of heat-flux, velocity, the stress tensor, rigid body positions, velocities, and angular velocities, as well as k-space resolved densities, heat-fluxes, currents, stress tensors, and energy currents. Additionally common statistical values reported by DL_POLY’s statistics module including, but not limited to volume, kinetic/potential energy, and temperature. We do this by defining each as an observable. Quantities that are observable may be juxtaposed in the CONTROL file to define arbitrary correlations. For example a velocity-velocity correlation may be defined by v_x-v_x where _x indicates the component correlated, and stress_xy-stress_xy defines a stress correlation of the xyxy-component of the stress tensor. Scalar quantities such as volume can be requested using e.g. volume-volume. The power of the juxtaposition is that any pair of observables can be correlated.

For each correlation, independently, we also support specifying the parameters bb, pp, and mm separately, and provide the ability to set the frequency each correlation is updated separately from the frequency of other statistics and output in DL_POLY. This enables high-resolution correlations to be calculated without additional I/O overhead, and without forcing low-resolution correlations to be calculated at unnecessarily high time-resolutions. Differences in correlation frequency are handled in the correlation output by automatically accounting for the different effective time-units in the outputted lag times.

During the simulation, correlators are stored indexed by the unique names of the observables they correlate, i.e. at any point in the simulation, we may trivially obtain the current correlation value for say the heat-flux in xx, heat_flux_x-heat_flux_x. Not only does this serve to maintain uniqueness, but also allows us to easily report these values, or process them further, at runtime. The result of each correlator is reported in a new file COR which reports the value of each correlation function as well as potentially a set of derived quantities, at user specified intervals. These derived quantities include viscosity, thermal conductivity, and elastic constants as detailed in the next sections.

Viscosity and thermal conductivity

The viscosity and thermal conductivity are important properties characterising liquids, for example in the evaluation of the performance of molten salts as nuclear reactor coolant fluids, since this is determined by their transport and thermal diffusion characteristics [40, 41]; these can be calculated using Green-Kubo theory from correlations of shear-stress and heat-flux respectively [32, 33, 42].

The shear-viscosity may be calculated using the integral

η=VkBT∫0∞dt⟨σxy(t) σxy(0)⟩,(2)\eta= \frac{V}{k_{\mathrm{B}}T}\int_{0}^{\infty} \mathrm{d}t\langle\sigma_{xy}(t)\,\sigma_{xy}(0)\rangle, \tag*{(2)}

where ⟨⋅⟩\langle\cdot\rangle is the ensemble average, TT is the system temperature and kBk_{\mathrm{B}} is Boltzmann’s constant. Finally the stress tensor, for pairwise additive potential UU, is defined as

σαβ=1V∑imivαivβi−1V∑i,j≠i∂U(rij)∂rrαijrβijrij,(3)\sigma_{\alpha\beta} = \frac{1}{V}\sum_i m^i v_\alpha^i v_\beta^i - \frac{1}{V}\sum_{i,j\ne i}\frac{\partial U(r^{ij})}{\partial r}\frac{r_\alpha^{ij}r_\beta^{ij}}{r^{ij}}, \tag*{(3)}

where mim^i, viv^i, rir^i are particle ii’s mass, velocity, and position, and rij=ri−rjr^{ij}=r^i-r^j, and VV is the system volume. Finally, α\alpha and β\beta are orthogonal Cartesian coordinates.

Similarly the thermal conductivity κ\kappa is related to the heat flux by

κ=V3kBT2∫0∞dt⟨q(t)⋅q(0)⟩.(4)\kappa= \frac{V}{3k_{\mathrm{B}}T^2}\int_{0}^{\infty} \mathrm{d}t\langle\mathbf{q}(t)\cdot\mathbf{q}(0)\rangle. \tag*{(4)}

Where the heat flux is

q=1V∑i=1N(Eivi+12∑j≠ifij⋅virij),(5)\mathbf{q} = \frac{1}{V}\sum_{i=1}^{N}\left(E^i\mathbf{v}^i+\frac{1}{2}\sum_{j\ne i}\mathbf{f}^{ij}\cdot\mathbf{v}^i r^{ij}\right), \tag*{(5)}

where EiE^{i} is the total energy (kinetic plus potential) of particle ii, and fij\mathbf{f}^{ij} is the force exerted on ii due to jj. In Eq. (4) the dot product has the function of averaging over the dimensions xx, yy, and zz.

For multi-component systems with significant asymmetry (e.g. mass difference) it is necessary to correct for mass flux effects when calculating κ\kappa in order to compare to experimental data [43]. For example the effect is significant in LiF but not in KCl [44] which have mass ratios 6.94118.99=0.37\frac{6.941}{18.99}=0.37 and 35.45339.098=0.91\frac{35.453}{39.098}=0.91 respectively. This can be done by forming multiple Green–Kubo formulae using the heat flux q\mathbf{q} and partial momentum densities for particular species,

jIs=1V∑i∈Ismivi(t),(6)\mathbf{j}_{I_s}=\frac{1}{V}\sum_{i\in I_s}m_i\mathbf{v}^{i}(t), \tag*{(6)}

where IsI_s is the set of atom indices for atoms of species ss. In DL_POLY the partial momentum densities may be calculated, and correlated, for a selection of species by the user. The values themselves may also be written to the HEATFLUX file alongside q\mathbf{q}. In general it is possible to take advantage of momentum conservation in order to not calculate all partial momentum densities. A user is then able to form the necessary correlations for the corrected Green–Kubo formula required.

Comparison to experiment

The correlation functions integrated in equations 2 and 4 can be calculated during an MD run using DL_POLY’s on-the-fly correlator. In each case we can approximate the integral by computing the required correlation to some maximum time-lag TT and performing a numerical integration of the correlation functions’ values. In each case, when multiple congruent correlation are present (e.g. xyxy and zxzx shear stresses), DL_POLY automatically calculates the derived properties (viscosity) listing them for each correlation and also as an averaged quantity. Care must be taken to collect adequate statistics for accurate calculations of these derived quantities. For example Zhang et al. [42] detail a method for choosing an optimal cutoff lag-time.

For argon, there is experimental data for κ\kappa and η\eta at a variety of pressure and temperatures compiled by the National Institute of Standards and Technology (NIST) [45]. DL_POLY has been used successfully before to calculated viscosity for argon by directly correlating the outputted stress tensor [46]. Here we report new data including thermal conductivity using

Viscosity for supercritical Argon as a function of temperature, showing NIST data, DL POLY data for $N=500$, and DL POLY data for $N=10^{8}$

Figure 2. Viscosity for supercritical Argon, as calculated using DL_POLY’s on-the-fly correlations. In each case experimental data from NIST is compared to data from N=500N = 500 atom simulations averaged over 20 simulations, and large scale simulations with N=108N = 10^{8} atoms averaged over 5 simulations, also compared to NIST [45]

DL_POLY’s on-the-fly correlator.

For all simulation work, we began by equilibrating for 10510^{5} steps in an NPT ensemble, followed by 10410^{4} steps with an NVE ensemble to account for changing thermostats, and then production runs, where statistics are calculated, ran for 10610^{6} time-steps. In all cases the time-step was 0.001 ps. For correlations we collect data for lag times up to 5 ps. For the small systems (N=500N = 500) we collect data on independently seeded simulations for 20 replicates to ensure adequate averaging of the Green-Kubo results. For the large-scale systems, we were able to simulate 5 replicates for each data point.

Figure 2 shows the small- and large-scale system results for viscosity (see Eq. (2)). We find good agreement with the NIST data as expected [46]. Additionally we find similarly consistent results for the large-scale simulations. Likewise for thermal conductivity, Eq. 4, we also find the calculated results are consistent with the experimental data (see Figure 3). The thermal conductivity data notably deviates more, this is due to increased noise between different replicates. For viscosity, we find standard deviations, across the different simulation results, of the order 10−610^{-6} Pa for both system sizes, whereas for thermal conductivity the smaller systems’ standard deviations are of the order 10−310^{-3} but for the large-scale systems this rose to 10−210^{-2} at 250 K and 10−310^{-3} at 500 K and 750 K.

Thermal conductivity for supercritical Argon

Figure 3. Thermal conductivity for supercritical Argon, as calculated using DL_POLY’s on-the-fly correlations. In each case experimental data from NIST is compared to data from N=500N = 500 atom simulations averaged over 20 simulations, and large scale simulations with N=108N = 10^{8} atoms averaged over 5 simulations, also compared to NIST [45]. See also figure 2.

Both viscosity and thermal conductivity show an interesting feature: the minima which bound these properties from below. The minima are due to the dynamical crossover of particle dynamics [46]. Interestingly, the values at the minima are fixed by fundamental physical constants such as the Planck constant and electron mass [47, 48, 49].

Elastic constants

Another use case for the on-the-fly correlator is the calculation of elastic constants. In general, the elastic constants of a system are defined by the elements of the elasticity tensor CαβμνC_{\alpha\beta\mu\nu}. The values of which can be calculated, in full, using either the stress-fluctuation of strain-fluctuation methods, or by examining system stress after applying a set of strains. The stress-fluctuation method has been found to be more reliable and faster converging [50, 51, 52, 53].

For the calculation of elastic constants, we use this method where the

elastic constants are given by,

Cαβμν=⟨CαβμνB⟩−VkBT[⟨σαβσμν⟩−⟨σαβ⟩⟨σμν⟩]+2NkBTV(δαμδβν+δανδβμ).(7)C_{\alpha\beta\mu\nu} = \langle C^{\mathrm{B}}_{\alpha\beta\mu\nu}\rangle- \frac{V}{k_{\mathrm{B}}T}\left[\langle\sigma_{\alpha\beta}\sigma_{\mu\nu}\rangle- \langle\sigma_{\alpha\beta}\rangle\langle\sigma_{\mu\nu}\rangle\right] + \frac{2Nk_{\mathrm{B}}T}{V}\left(\delta_{\alpha\mu}\delta_{\beta\nu} + \delta_{\alpha\nu}\delta_{\beta\mu}\right). \tag*{(7)}

where the parameters NN is the atom count, δij\delta_{ij} is the Kronecker delta, and σαβ\sigma_{\alpha\beta} is the microscopic stress tensor as in equation 3. Finally the Born term is defined as,

CαβμνB=1V∑i,j≠i(∂2U(rij)∂rij2−1rij∂U(rij)∂rij)rαijrβijrμijrνijrij2.(8)C^{\mathrm{B}}_{\alpha\beta\mu\nu} = \frac{1}{V}\sum_{i,j\ne i}\left(\frac{\partial^2 U(r^{ij})}{\partial r^{ij2}}-\frac{1}{r^{ij}}\frac{\partial U(r^{ij})}{\partial r^{ij}}\right)\frac{r^{ij}_{\alpha}r^{ij}_{\beta}r^{ij}_{\mu}r^{ij}_{\nu}}{r^{ij2}}. \tag*{(8)}

The 81 components of the elasticity tensor CαβμνC_{\alpha\beta\mu\nu} can be reduced to 21 independent values for physical media, and further to 3 for physical media with cubic symmetry. Using Voigt notation we identify C11=13(C1111+C2222+C3333)C_{11}=\frac{1}{3}(C_{1111}+C_{2222}+C_{3333}), C12=13(C1122+C1133+C2233)C_{12}=\frac{1}{3}(C_{1122}+C_{1133}+C_{2233}), and C44=13(C2323+C3131+C1212)C_{44}=\frac{1}{3}(C_{2323}+C_{3131}+C_{1212}). DL_POLY automatically reports the values of the elastic constants the 21 possible elastic constants when the required stress correlation terms are present. To compare with experimental data, from the elastic constants we can derive the bulk and shear modulus. The bulk modulus is

B=13(C11+2C12).(9)B = \frac{1}{3}(C_{11}+2C_{12}). \tag*{(9)}

For the shear modulus there is a choice of averaging, KVK^{V} (Voigt) and KRK^{R} (Reuss) which give upper and lower bounds of the experimental results [54]. These take the form

KV=C11−C12+3C445,(10)K^{V} = \frac{C_{11}-C_{12}+3C_{44}}{5}, \tag*{(10)}
KR=54/(C11−C12)+3/C44.(11)K^{R} = \frac{5}{4/(C_{11}-C_{12})+3/C_{44}}. \tag*{(11)}

Both KK and BB are reported for experimentally observed argon crystals. Note that the isothermal compressibility is often reported where χ=B−1\chi= B^{-1}, and the bulk modulus itself may also be more simply calculated from fluctuations in volume in an NPT ensemble as [1]

B=⟨V⟩kBT⟨V2⟩−⟨V⟩2.(12)B = \frac{\langle V \rangle k_{\mathrm{B}}T}{\langle V^2 \rangle- \langle V \rangle^2}. \tag*{(12)}

Comparison to simulations and experiments

The following methodology was used in all our simulation results. An FCC Argon crystal was initialised and equilibrated in NPT, for 10510^5 steps (timestep 1 fs) with a Nosé-hoover thermostat with thermo- and baro-stat couplings of 0.1 ps and 0.1 ps respectively, and a cutoff of 12 Å. The potential was the Lennard-Jones potential with the same parameters as [50].

After equilibration production runs (NVT) were simulated for 10610^6 steps unless otherwise specified. For these runs, the atoms were re-scaled to the average simulation cell from the equilibration phase. Statistical averages were calculated on a rolling window of 10 ps updated every step, and correlations computed for a maximum lag time of 10 ps, also updated every step using DL_POLY’s on-the-fly correlators. Everything else was kept the same. For each replicate n=1,2,…,20n = 1, 2, \ldots, 20, this procedure was carried out with a different random seed for equilibration and production. The methodology is similar to [50], except they generate initial inputs through Monte-Carlo (MC) methods.

Figure 4 compares DL_POLY’s calculated elastic constants with existing simulation results on argon FCC crystals, including MD and MC results, which are notably quite sparsely reported along the temperature axis. Previous simulation results are obtained from MC [35] and MD [51, 52, 50, 53]. We find that the values reported by DL_POLY are consistent with previous simulation results. The results from Pereverzev [52] and Squire et al. [35] follow broadly the same trend in temperature, and the remaining results reported at 60 K are also consistent with DL_POLY’s.

Figure 4. Elastic constants for FCC Argon, as calculated by DL_POLY compared to previous simulation results. In general there is good agreement with data that exists, notably existing data is quite sparse. Our data is averaged over n=20n = 20 initial conditions, and statistics collected over T=106T = 10^{6} steps, with a 10 ps maximum correlation lag time.

Given that we find consistent results for the individual elastic constants against simulated data we now compare with experimental values. For the bulk modulus (BB) these follow the same averaging scheme. DL_POLY’s results, Figure 5, bound the experimental values of Dobbs and Jones [55] as expected. The bulk modulus is also in good agreement with Peterson et al. [56]. Although there is a notable plateau trend at ≲20\lesssim20 K, which is not captured in simulations. Not that Pereverzev [52] also finds a linear relationship at low temperature consistent with our results (see Figure 4).

Plot of bulk and shear modulus versus temperature

Figure 5. The bulk and shear modulus, derived from DL POLY elastic constants (see Figure 4 for the raw data), compared with experimental data. Voigt and Reuss averaging give upper and lower bounds on the shear modulus respectively as expected. The Bulk modulus is also in good agreement Like [52] we find a linear relationship in C11C_{11} and C12C_{12} with temperature and hence the same is true for BB.

Currents

In DL_POLY 5’s currents module is capable of calculating various collective properties resolved to user supplied k-space vectors. To begin, the most basic functionality is the computation of k-space density, as well as transverse and longitudinal currents. These are defined as [27]

n(k,t)=∑ieik⋅ri(t),(13)n(\mathbf{k},t)=\sum_i e^{i\mathbf{k}\cdot\mathbf{r}^i(t)}, \tag*{(13)}
jL(k,t)=∑i(vi(t)⋅k^)k^eik⋅ri(t),(14)\mathbf{j}_{L}(\mathbf{k},t)=\sum_i\left(\mathbf{v}^i(t)\cdot\hat{\mathbf{k}}\right)\hat{\mathbf{k}}e^{i\mathbf{k}\cdot\mathbf{r}^i(t)}, \tag*{(14)}
jT(k,t)=∑i[vi(t)−(vi(t)⋅k^)k^]eik⋅ri(t).(15)\mathbf{j}_{T}(\mathbf{k},t)=\sum_i\left[\mathbf{v}^i(t)-\left(\mathbf{v}^i(t)\cdot\hat{\mathbf{k}}\right)\hat{\mathbf{k}}\right]e^{i\mathbf{k}\cdot\mathbf{r}^i(t)}. \tag*{(15)}

Equation 13 may be used to determine the intermediate scattering function using the correlations

F(k,t)=1N⟨n(k,t)n(−k,0)⟩,(16)F(\mathbf{k},t)=\frac{1}{N}\langle n(\mathbf{k},t)n(-\mathbf{k},0)\rangle, \tag*{(16)}

which can be used to determine thermal-conductivity in fluids [37], as well as the static structure factor as the t=0t=0 correlation value. The dynamic structure factor may also be obtained using the Fourier transform of F(k,t)F(\mathbf{k},t). Analogous correlations of jL(k,t)\mathbf{j}_{L}(\mathbf{k},t) and, jT(k,t)\mathbf{j}_{T}(\mathbf{k},t) may be used to determine transverse and longitudinal propagating modes. These collective properties relate to the overall motion of the particles of the system. Collective excitations form an integral and well known part of the theory of solid and gaseous states, but for liquids, which combine strong interactions with dynamical disorder, this has historically not been the case.

Alongside these currents, DL_POLY can also calculate the k-space energy density, energy currents, and k-dependent stress tensor. The energy density is given by

e(k,t)=12∑iEieik⋅ri(t).(17)e(\mathbf{k},t)=\frac{1}{2}\sum_i E^i e^{i\mathbf{k}\cdot\mathbf{r}^i(t)}. \tag*{(17)}

and the k\mathbf{k}-dependent stress and energy currents from equations (18) and (19).

σα,β(k)=∑i(mvαivβi−12∑j≠irαijrβij∣rij∣2P(k,rij))eik⋅ri(t),(18)\sigma_{\alpha,\beta}(\mathbf{k}) = \sum_i \left(mv_{\alpha}^{i}v_{\beta}^{i} - \frac{1}{2}\sum_{j\ne i}\frac{r_{\alpha}^{ij}r_{\beta}^{ij}}{\lvert r^{ij}\rvert^2}P(\mathbf{k},r^{ij})\right)e^{i\mathbf{k}\cdot\mathbf{r}^{i}(t)}, \tag*{(18)}
qα(k)=12∑i[Ei′vαi−12∑j≠i∑b(vbi+vbj)(rαijrbij∣rij∣2)P(k,rij)]eik⋅ri(t),(19)\mathbf{q}_{\alpha}(\mathbf{k}) = \frac{1}{2}\sum_i\left[E_i'v_{\alpha}^{i}-\frac{1}{2}\sum_{j\ne i}\sum_b\left(v_b^i+v_b^j\right)\left(\frac{r_{\alpha}^{ij}r_b^{ij}}{\lvert r^{ij}\rvert^2}\right)P(\mathbf{k},r^{ij})\right]e^{i\mathbf{k}\cdot\mathbf{r}^{i}(t)}, \tag*{(19)}

where

Pk(k,r)=∣r∣∂U(∣r∣)∂r1−e−ik⋅rik⋅r(20)P_k(\mathbf{k},\mathbf{r})=\lvert\mathbf{r}\rvert\frac{\partial U(\lvert\mathbf{r}\rvert)}{\partial r}\frac{1-e^{-i\mathbf{k}\cdot\mathbf{r}}}{i\mathbf{k}\cdot\mathbf{r}} \tag*{(20)}

A user may request correlations of combinations of these various currents (component wise). Each k\mathbf{k}-point specified by the user will result in one correlation matching these requests in the CONTROL file. One use case for current correlations is calculating phonon dispersion curves. These dispersions can be calculated in DL_POLY via the calculation of current correlations

CL(k,t)=⟨jLz(k,t)jLz(k,0)⟩,(21)C_L(\mathbf{k},t)=\langle j_L^z(\mathbf{k},t)j_L^z(\mathbf{k},0)\rangle, \tag*{(21)}
2CT(k,t)=⟨jTx(k,t)jTx(k,0)⟩+⟨jTy(k,t)jTy(k,0)⟩.(22)2C_T(\mathbf{k},t)=\langle j_T^x(\mathbf{k},t)j_T^x(\mathbf{k},0)\rangle +\langle j_T^y(\mathbf{k},t)j_T^y(\mathbf{k},0)\rangle. \tag*{(22)}

From these correlation functions, the spectra are found from the Fourier transforms CL(k,ω)C_L(\mathbf{k},\omega) and CT(k,ω)C_T(\mathbf{k},\omega). The frequencies corresponding to maxima of these Fourier transforms of each wave-vector are interpreted as the spectrum of collective modes [27, 38].

In liquids in particular, it is possible to observe an interesting feature of spectra not seen in solids: kk-gap in the solid-like transverse spectrum meaning a threshold value below which no such propagating modes exist [57, 38]. The kk-gap can be observed by plotting the relationship between these maxima and k=∣k∣k=\lvert\mathbf{k}\rvert.

To calculate these data in DL_POLY we first equilibrate an argon system of 4,000 atoms in NPT to the desired temperature. We then expand to other desired system sizes using DL_POLY’s nfold capability. We then calculate the correlations functions on-the-fly over 10610^6 simulation steps (1 fs per step), for all physically reasonable k\mathbf{k}-points, kzkz. Our correlations are resolved for 10240 points per block, at a correlation frequency of every 10 steps (lags up to 102400 fs). The resulting correlation functions are first averaged across 20 independent trajectories. The frequency maxima are then found by taking

Fourier transforms of these averaged correlations. The result of this is shown in Figure 6 where we see the increasing kk-gap for the transverse mode for temperatures of 85 K and 134.5 K.

Dispersion curves for Argon at 85 K and 134.5 K for system sizes 4,000, 32,000, 108,000, and 10,976,000

Figure 6. Dispersion curves for Argon at 85 K and 134.5 K for system sizes 4,000, 32,000, 108,000, and 10,976,000.

Rigid body correlations

In DL_POLY rigid bodies can be defined as part of user input in the FIELD file. Molecules may have single or multiple rigid body components. During a simulation each rigid body has position, velocity and angular velocity data which may be correlated in the same manner as per-atom correlations. That is DL_POLY will determine distinct rigid body types based on distinct rigid body components within distinct molecular types, calculate requested correlations at runtime, and average these results for each.

Using this on-the-fly correlation functionality we are able to reproduce the results of Brodka and Zerda [58] for SF6_6. We do this by simulating rigid bodies of SF6_6 molecules with the same Lennard-Jones potential parameters. For equilibration we set the temperature to 296 K, and pressure to the experimental values listed for the pressure by Brodka and Zerda. We simulate in NPT for 100 ps to ensure equilibration. The resulting densities were similar to the 1.5 and 1.9 g/cm3^3 listed in their results. We then rescaled our resulting configurations to match those densities exactly, and then proceed in NVE collecting statistics over 1 ns simulation steps. Brodka and Zerda use 1000 equilibration steps and 4200 production steps at a higher time-step (5 fs vs. 1 fs). Our results for the VAF are shown in Figure 7 which match well.

Comparison to Brodka and Zerda’s results for SF$_6$ rigid body VAFs

Figure 7. Comparison to Brodka and Zerda’s results (digitised) for SF6\mathrm{SF}_{6} rigid body VAFs [58]. Temperature 296 K with pressure and density indicated in the legend. NN indicates the rigid body count.

The VAF can also be used as a definition of the Frenkel line in supercritical fluids [59, 60], which marks the crossover between liquid-like and gas-like particle dynamics and thermodynamics. Using the rigid body VAF functionality we perform this analysis for rigid body CH4_4 molecules. We follow the process of Yang et al. [61], using the same potential [62]. We first equilibrate our system in NPT at the desired PP (900 bar) and TT conditions for 50 ps and then collect statistics over a further 50 ps. As before we find the equilibrated simulation densities to be approximately the same as the NIST experimental data for supercritical CH4_4 [45], and for increasing TT there is a loss of the VAF minimum as shown in Figure 8.

VAF for CH₄ rigid molecules at 900 bar for increasing $T$, showing the crossover at the Frenkel line

Figure 8. VAF for CH4_4 rigid molecules at 900 bar for increasing TT, showing the crossover at the Frenkel line.

Benchmarks and Trade-offs

We have identified a variety of applications of the on-the-fly correlator in DL_POLY. In each case the calculated correlations forgo the need for stor

ing the underlying trajectory. However for different use-cases this trajectory is more or less significant. Consider for example the viscosity and thermal conductivity. DL_POLY, analogously to many other packages, already calculates the heat flux and stress tensor at each time-step. This data (the 3 components of heat flux or 6 components of the stress tensor per step) is small and does not scale with the system size. Therefore, the printing of the full trajectory (velocities and forces) can be avoided in these cases. For the VAF the velocities are required, while for currents the reduced data are still much more significant. In these cases the written data scales with the system size explicitly (atom velocities) or implicitly (possible kk-points). For the thermal-conductivity and viscosity, the use of the correlator is a significant convenience (and a well defined reproducible workflow) but not a large performance boost. We can realise the full benefit of avoiding trajectory storage and benchmark our work where heavy I/O can be avoided, for example computing the VAF.

Strong and weak scaling: the velocity autocorrelation function

In Figure 9 and Figure 10 we show strong and weak scaling benchmarks for calculating the VAF of molten LiF with 10610^6 atoms, using the Buckingham potential in the NVE ensemble. That data for all scatter points can be found in tabulated form in Appendix Appendix E. Additionally we present that same data in Figure 11 in the format of CPU core hours for the full workflows (on-the-fly versus saving the trajectory and analysing the trajectory).

Strong scaling benchmarks with four panels: On-the-fly, Saving, Replay, and Python FFT, showing wall time relative to baseline versus core count for frequencies 1, 2, 4, 8, 16, and 32

Figure 9. Strong scaling benchmarks (calculating the VAF) with 1,000,000 LiF atom simulations. Frequency indicates the sampling interval i.e. either correlation frequency or trajectory writing frequency. In all cases the y axis is the wall time relative to the baseline simulation (without trajectory writing or correlation) on a log scale. The baseline wall times were 123.19 s, 66.14 s, and 75.4 s for the 128, 640, and 1280 core simulations. The absolute wall times for the simulations can be found in Table E.1 and the absolute wall times for the post-processing methods in Table E.2.

Weak scaling benchmarks for on-the-fly, saving, replay, and Python FFT methods

Figure 10. Weak scaling benchmarks (calculating the VAF) with 1,000 LiF atoms per core. Frequency indicates the sampling interval i.e. either correlation frequency or trajectory writing frequency. In all cases the y axis is the wall time relative to the baseline simulation (without trajectory writing or correlation) on a log scale. The baseline wall times were 16.0 s, 16.43 s, 20.34 s, 26.86 s, and 49.89 s for the N=8000N = 8000, 27000, 64000, 125000, and 512000 atom simulations. The absolute wall time data can be found in Table E.3 for the on-the-fly simulations, in Table E.4 for trajectory saving, in Table E.5 for the trajectory replay post-processing method, and in Table E.6 for the Python FFT based post-processing.

Core hour analysis for the strong and weak benchmark data

Figure 11. Core hour analysis for the strong and weak benchmark data presented in Figure 9 and Figure 10. The circular scatter points indicate the full on-the-fly workflow core hour cost. The downward triangles indicate the combination of a simulation with trajectory saving followed by calculating correlations by trajectory replay (which is generally the fastest post-processing method presented here).

In all cases the simulations were identical except for the mutually exclusive modes of trajectory writing or correlating. The simulations were run for 1000 time-steps with a cutoff of 8 Å and an SPME precision of 10−610^{-6}. STATIS and OUTPUT were suppressed by setting their output frequency greater than the number of steps. In the figure captions “Frequency ff” refers to a sampling interval of ff simulation time steps i.e. the frequency of trajectory saving, or correlation. Post-processing of the trajectories was performed by enabling DL_POLY’s replay feature (also using the on-the-fly correlator) and in Python using FFTW [63] (also using simple multiprocessing across atoms) to cover basic user post-processing analysis strategies. All simulations were performed on the Sulis Tier 2 HPC system utilising the standard “compute nodes”. These were Dell PowerEdge R6525 compute nodes each with 2×2 \times AMD EPYC 7742 (Rome) 2.25 GHz 64-core processors (128 cores per node) with 512 GB DDR4-3200 RAM per node. The interconnect was Mellanox ConnectX-6 HDR100 (100 Gbit/s) InfiniBand and the storage system utilised was General Parallel File System (GPFS) . The default OpenMPI IO subsystem was used for these benchmarks. Sulis also offers ROMIO [64] which had little impact. In the case of the trajectory saving and replay benchmarks care was taken to select the best performing split of MPI processes as readers and writers. These are subsets of the same MPI count used for domain decomposition. They can work independently on the same I/O file. All MPI tasks depend on the I/O dedicated MPI tasks and no compute can take place until all I/O has been completed as commitment to the memory of the I/O subsystem. This can be controlled in DL_POLY’s CONTROL file directly. Typically this meant using equal or less than the node size (128).

The strong scaling benchmarks show that DL_POLY’s correlator module is able to make use of the increasing core counts well. Especially at higher frequencies when compared to writing the trajectory, as expected given DL_POLY’s domain decomposition. In comparison purely saving the trajectory incurs a much larger performance impact (when saving every step the data measures ∼285\sim285 GB). Except for the cases of low frequency (64-128 steps), and more widely for the single node case. As remarked above, but not shown in these data, the benchmarks for trajectory saving and replay depend heavily on the user’s selection of reader and writer processes. A clear benefit of the on-the-fly method is this is simply not a concern. For example in the worst case of the 1280 core simulations and saving every step, choosing 64/1280 cores as writers resulted in the reported wall time, 3019 s. But choosing 1280/1280 cores as writer results in a wall time of 5513.36 s. Finally note for a complete picture of the workflow the wall times for saving the trajectory must be added to one of the post-processing modes. Consider the best case for saving the trajectory (128 cores and 128 saving frequency). The simulation saving the trajectory took 154 s and the post-processing wall times were 64 s or 32 s for replay and the FFT method respectively. The on-the-fly wall time was 202 s. A ∼5%\sim5\% saving or ∼8%\sim8\% loss in each case.

Both post-processing methodologies are dominated by reading the trajectory data rather than the computation of the correlation. This can be seen by the poor scaling with increased core count which we include for complete picture. Further improvements could likely be obtained by using dedicated data-processing partitions where available. The weak scaling benchmarks show a similar picture. A tight spread between the frequencies using the on-the-fly method and a widening gap between the heavy I/O and light I/O when saving or post-processing the data. Again we also see the impact of multi-node I/O where the wall times are impacted depending on the user’s choice of read and writer processes.

Finally, the core hour comparison in Figure 11 again indicates the benefits that can be obtained using the on-the-fly method. In this case we see the impact of the combined cost of trajectory saving and analysis compared with using the on-the-fly method. The increased core hours come from I/O operations. Depending on accuracy requirements use of compression schemes such as ZFP could reduce this by 30% (64-bit, lossless) or more than 90% (32-bit, lossy). In the worst cases (strong scaling, 1280 cores) the over head is approximately between 102−101.5≈6810^2 - 10^{1.5} \approx68 core hours and 103.5−101.5≈313010^{3.5} - 10^{1.5} \approx3130 core hours. At 30% and 90% data reductions this could lead to overheads of 48 core hours and 312 core hours respectively.

When to use on-the-fly correlations

DL_POLY’s on-the-fly correlations module provides the ability to calculate correlations without saving trajectories. We have demonstrated that this can be a significant saving, particular in calculating the VAF. In general on-the-fly correlations make efficient use of DL_POLY’s parallelisation, with low overhead. The primary use case is for large systems where analysis would normally require the trajectory to be written. Especially where analysis would dictate frequent writing. Concrete examples do depend on the hardware used. In general it is notable that when saving a significant trajectory a user must think carefully about the I/O setup of the job (read and writer splits).

However by bypassing storage of the trajectory the user is left unable to conduct further analysis without re-running the simulation. This could manifest from the need for unforeseen analysis or incorrect parametrisation. On this latter point an example would be computing a correlation to a far too short time scale, cutting off important features for analysis, or too infrequent correlation leading to poor resolution. One solution is to conduct preliminary analysis on small systems where I/O is negligible in order to determine the analysis requirements relating to correlation parameters. In this case the on-the-fly correlator can then be used for much larger systems. Or paying the price of trajectory storage for a smaller sample of large simulations, and using the on-the-fly correlator to about additional replicates for good statistics. A middle ground can be found in many cases. That is by saving the trajectory coarse enough that the runtime impact is moderate in order to preserve data for later use, whilst calculating certain properties on-the-fly at a much finer resolution. It is then down to the user to determine a reasonable saving frequency for unanticipated analysis, which is always a problem.

An advantageous alternative for the DL_POLY user results from DL_POLY’s replay feature. That is a HISTORY file can be fed back into DL_POLY to re-calculate statistics, including new ones at finer or coarser scales, leveraging the domain decomposition and parallel I/O in DL_POLY. As shown in the benchmarks this has a competitive performance compared to using fast Fourier transform methods accessible with simple scientific programming as core counts and/or problem sizes increase. Of course there is a trade-off as well. Simple highly specific targeted analysis can obtain results with little overhead. Replaying a HISTORY file however, does incur overhead as DL_POLYis doing more than required.

In general across MD software on-the-fly methods will be subject to these concerns. But this will also change depending on the parallelisation strategy. DL_POLY is a CPU based software. When heterogeneous compute systems can be leveraged (such as CPU-GPU systems) trajectory saving could become less cumbersome if the CPU can largely write out data whilst the GPU does the bulk of the MD simulation workload. However the efficiency of this setup will depend on the relative speed of the CPU write-out (plus data transfer penalties) to the GPU compute speed. In cases where this is a bottleneck (e.g. high time-step resolution analysis), on-the-fly methods could be utilised on the GPU side to reduce the communication of data. Another way to avoiding I/O/transfer overheads.

DL_POLY, massive parallelism and scalability

The DL_POLY project was conceived in 1993 by W Smith and originally released in 1994 as DL_POLY_2 [65, 66] (now known as DL_POLY_CLASSIC [67]) using a replicated data (RD) approach as its underlying parallelisation strategy [68, 69], written in Fortran77 [70] and using MPI [71]. Later in 2003, led by I. Todorov, with core work carried out by W. Smith and I. Bush, DL_POLY 3 was released which developed the foundation of the domain decomposition (DD) parallelisation strategy [25, 26], including a Smooth Particle Mesh Ewald (SPME) method suitable for the DL_POLY DD [72].

This strategy provided for an extremely efficient memory distribution across a variety of test systems [26].

The release of DL_POLY_4 included a major rewrite providing an extension of particle dynamics to rigid body dynamics [73], inclusion of two-temperature thermostat functionality [74] for high-energy events and the inclusion of parallel I/O [75]. The rigid body dynamics provided a natural extension of the parallel performance capabilities for biological and biochemical systems in which constrained bond dynamics was a bottleneck. The enhancement in performance of rigid body (RB) dynamics and constrained bond (CB) dynamics is demonstrated in Figure 12 where both weak and strong scaling tests were performed on the same SPC water system and model with scaled sizes in the case of weak scaling. Further developments in 2012 led to the inclusion of Dissipative Particle Dynamics integration [76] and a three seeded random number generation [77] for using repeatable random series for stochastically dependent thermostating, kinetic fields and pressure tensors in a fully deterministic manner across domains and time series.

Weak and strong scaling benchmarks of SPC water systems

Figure 12. Weak and strong scaling benchmarks of SPC water systems. This data was first produced with DL_POLY 4.04 in May 2013 on UK’s National HPC service HECToR (Cray XE6).

A few years ago an extensive refactoring effort took place that is the base of the current version of DL_POLY. These changes were introduced mainly due to the need for being able to add new complex tasks with maximum code reuse and allow various easy optimisations. The main changes intro duced were: i) using object oriented Fortran 2008 as the minimal standard required, ii) restructuring the code to be fully modular, with all functions and subroutines exchanging data only via parameters rather than via modules - the removal of implicit common blocks. Only one singleton remains for dealing with I/O for errors. iii) introducing an automated test system with both regression and unit tests. Testing was extended from 24 tests to over 200 tests. iv) atomic configurations are stored in an array of structures that is cache friendly with respect to positions and forces (although this has a non-trivial trade-off with vectorisability of the code), v) electrostatics was rewritten to allow per-particle decomposition and allowing the computation of long range contributions from any power potential, see next section for details. vi) The CONTROL file format was redesigned (see appendix Appendix C). In general all entries now are consistently of the form token-value-unit (where applicable), STATIS, RDF and other output files can be written in the yaml format [78] for easier postprocessing, vii) a modern software infrastructure is in place with full CI/CD via gitlab.com, vi) license was changed to an open-source license GPL-3.0, viii) dlpoly-py was developed [79] - a companion python package that can read inputs and outputs of DL_POLY and initiate simulations. Without this extensive refactoring, implementation of the Empirical Valence Bond [80] and the new Chemshell interface [81] would not have been possible.

SPME refactoring

The Ewald method [82] is a method for the calculation of long-range (conditionally convergent until infinity) forces. This can be achieved in a number of ways, the usual physical description being the introduction of a screening charge distribution and a correction to account for the screening so as to accelerate the summation of the long-range component in Fourier space. This can also be seen as performing a sum from 0→α0 \to\alpha in real-space and from 0→α0 \to\alpha in reciprocal space to construct a full sum of 0→∞0 \to\infty in finite time (lim⁡x→01x=∞\lim_{x \to0} \frac{1}{x} = \infty).

The smooth particle mesh Ewald (SPME) approach by Essmann et al. [83] uses cardinal B-Splines to interpolate the complex exponentials that result from the above approach in a regular fashion. This allows the use of a standard Fast Fourier Transform (FFT), so reducing the scaling of the method with system size to O(Nlog⁡(N))O(N \log(N)). However this is at the cost of some accuracy due to the interpolation, though this can be negated by using higher order splines to provide a better description of the exponentials.

The SPME approach in DL_POLY has been refactored to enable computation of per-particle contributions to energies, forces and stresses. The calculation of per-particle contributions has been used to implement the computation of Green-Kubo thermal conductivity with Ewald methods (See § 2.3).

This new method coincidentally enables the expansion to not only alternative orders of 1r\frac{1}{r} potentials, but also derivatives of arbitrary order, thereby paving the way to correct multipolar expansions of arbitrary order. Direct computations of higher-order potentials allow “exact” computation of, e.g. Lennard-Jones (6-12) [84] potentials without cut-offs, enabling calculation of the effect of these cutoffs on actual dynamics. Currently implemented are functional forms for: r−1r^{-1}, r−2r^{-2}, r−6r^{-6}, r−12r^{-12} and a general form for r−nr^{-n} and their derivatives for both the real- and reciprocal-space parts (see appendix Appendix § 8).

Benchmarks of the Refactored Code

As part of the DL_POLY_5 developments, we have recently benchmarked 2-body and SPME-enabled systems composed of ∼\sim2 million atoms and 2-body simulations composed of ∼\sim1 billion atoms across increasing MPI process counts on Archer2 [7]. We examine processor counts ranging from 128 in the small systems to 262,144 in the larger system. Representing a 35% concurrent utilisation of Archer2’s 750,080 CPU cores [85]. The results can be found in Figure 13. Primarily the results continue to demonstrate the scalability of DL_POLY. But secondly, it also demonstrated the scalability of the newly refactored MD algorithms, with the reworked DD memory distribution that favoured purely local communications and minimised all-to-all communications as much as possible. A comprehensive survey of DL_POLY performance on modern hardware can be found in [86].

Strong scaling benchmarks for DL_POLY for Argon and NaCl systems

Figure 13. Strong scaling benchmarks for DL_POLY for Argon (2-body) and NaCl (2-body and SPME electrostatics) systems. Benchmarks were conducted on Archer2 [7] all are timed over 1ps simulation time. The system sizes were of the order of millions and billions for Argon, and millions for NaCl. Processor counts vary from 128 in the smaller systems, up to 262,144 for the 1 billion atom system. Our largest benchmark represents a 35% concurrent utilisation of Archer2 [85]

Conclusions

We have used the paradigm of on-the-fly calculations to develop a general purpose correlation module in DL_POLY 5. This functionality exposes key properties derived from simulation data which may be composed into correlation functions by the user. This method provides a convenient way to perform correlation analysis as well as eliminating the cumbersome I/O operations inherent in saving trajectory data for post-processing. It also presents the continuous availability of correlation function during simulation time. This saving becomes significant for high temporal resolution correlations defined at the atomic or molecular level such as the VAF.

DL_POLY continues to show the same scalability for large CPU counts as past versions. Our correlation implementation naturally takes advantage of the DD algorithm and multi-processing architecture used in DL_POLY to distribute the calculation of atomistic correlations. We aim to continue the development of on-the-fly methods in and beyond DL_POLY 5. For example, to date we have exposed rigid body molecular positions, velocities, and angular velocities. A natural extension is to correlate properties of flexible molecular species and other atomic groupings more generally. In particular the more complex sub-components of flexible molecules such as atomic bonds or angle based interactions etc.

Author contributions

H. L. Devereux: Methodology; Software; Validation; Formal analysis; Investigation; Data Curation; Writing - Original Draft; Writing - Review & Editing; Visualization. C. Cockrell: Methodology; Software; Validation; Formal Analysis; Investigation; Writing - Original Draft; Writing - Review & Editing. A. M. Elena: Conceptualization; Methodology; Software; Validation; Formal Analysis; Investigation; Resources; Data Curation; Writing - Original Draft; Writing - Review & Editing; Supervision; Project administration; Funding Acquisition. Ian Bush: Software; Validation; Data Curation; Writing - Review & Editing; Aidan B. G. Chalk: Software; Validation; Data Curation. Jim Madge: Software; Validation; Data Curation. Ivan Scivetti: Software; Validation; Data Curation. J. S. Wilkins: Software; Validation; Data Curation; Writing - Original Draft; Writing - Review & Editing. I. T. Todorov: Conceptualization; Methodology; Software; Validation; Formal Analysis; Investigation; Resources; Data Curation; Writing - Original Draft; Writing - Review & Editing; Supervision; Project administration; Funding Acquisition. W. Smith: Software; Conceptualization; Methodology; Validation; Data Curation. K. Trachenko: Conceptualization; Methodology; Validation; Formal analysis; Investigation; Resources; Writing; Writing-Review & Editing; Supervision; Project administration; Funding acquisition.

Data statement

The DL_POLY 5 source code is hosted on GitLab [28] (GPLv3.0). The input files and submission scripts to reproduce the statistical data for our on-the-fly methods can be found on Gitlab https://gitlab.com/apw951/dl_poly_workflows/-/tree/dlpoly_5_preprint?ref_type=tags.

Acknowledgements

We are grateful to EPSRC (for grant No. EP/W029006/1). This research utilised Queen Mary’s Apocrita HPC facility, supported by QMUL Research-IT http://doi.org/10.5281/zenodo.438045, STFC Scientific Computing Department’s SCARF cluster and the Sulis Tier 2 HPC platform hosted by the Scientific Computing Research Technology Platform at the University of Warwick and funded by EPSRC GrantS EP/T022108/1 and the HPC Midlands+ consortium. Via our membership of the UK’s HEC Materials Chemistry Consortium, which is funded by EPSRC (EP/R029431), this work used the ARCHER UK National Supercomputing Service (http://www.archer.ac.uk)

Kernels and derivation for SPME

Below follows a derivation for the generic kernels for order 1/rp1/r^p potentials.

gpg_p.

As described in Essmann et al. [83] the general form of the gg function for 1/rp1/r^p is given as:

gp(x)=2Γ(p2)∫x∞sp−1exp⁡(−s2) ds(23)g_p(x) = \frac{2}{\Gamma\left(\frac{p}{2}\right)} \int_x^\infty s^{p-1}\exp\left(-s^2\right)\,\mathrm{d}s \tag*{(23)}

From Gradshteyn and Ryzhik [87] (pp. 108–109) and [83] we find for r−1r^{-1} (Coulomb):

g1(x)=2Γ(12)∫x∞exp⁡(−s2) ds(24)g_1(x) = \frac{2}{\Gamma\left(\frac{1}{2}\right)} \int_x^\infty\exp\left(-s^2\right)\,\mathrm{d}s \tag*{(24)}

where:

∫exp⁡(−βxn) dx=12πβ erf⁡(βx)\int\exp\left(-\beta x^n\right)\,\mathrm{d}x = \frac{1}{2}\sqrt{\frac{\pi}{\beta}}\,\operatorname{erf}\left(\sqrt{\beta}x\right)
g1(x)=2ππ2(erf⁡(∞)−erf⁡(x))(25)g_1(x) = \frac{2}{\sqrt{\pi}}\frac{\sqrt{\pi}}{2}\left(\operatorname{erf}(\infty)-\operatorname{erf}(x)\right) \tag*{(25)}
g1(x)=erfc⁡(x)(26)g_1(x) = \operatorname{erfc}(x) \tag*{(26)}

For the general case (1/rp1/r^p) we find:

gp=2Γ(p2)∫x∞sp−1exp⁡(−s2) ds(27)g_p = \frac{2}{\Gamma\left(\frac{p}{2}\right)} \int_x^\infty s^{p-1}\exp\left(-s^2\right)\,\mathrm{d}s \tag*{(27)}

where:

∫xmexp⁡(±axn) dx=±xm+1−nnaexp⁡(±axn)∓m+1−nna∫xm−nexp⁡(±axn) dx\begin{aligned} \int x^m\exp\left(\pm ax^n\right)\,\mathrm{d}x &= \pm\frac{x^{m+1-n}}{na}\exp\left(\pm ax^n\right) \\ &\mp\frac{m+1-n}{na}\int x^{m-n}\exp\left(\pm ax^n\right)\,\mathrm{d}x \end{aligned}

So for our simpler case:

∫xp−1exp⁡(−x2) dx=−xp−22+p−22∫xp−3exp⁡(−x2) dx(28)\int x^{p-1}\exp\left(-x^2\right)\,\mathrm{d}x = -\frac{x^{p-2}}{2} +\frac{p-2}{2}\int x^{p-3}\exp\left(-x^2\right)\,\mathrm{d}x \tag*{(28)}

Which, using a series expansion gives for even powers:

∑n=0p/2[x2nn!]exp⁡(−x2)(29)\sum_{n=0}^{p/2}\left[\frac{x^{2n}}{n!}\right]\exp\left(-x^2\right) \tag*{(29)}

and for odd powers:

∑n=1p−1[2nn!!x2n−1]exp⁡(−x2)π+erfc⁡(x)(30)\sum_{n=1}^{p-1}\left[\frac{2^n}{n!!}x^{2n-1}\right]\frac{\exp\left(-x^2\right)}{\sqrt{\pi}}+\operatorname{erfc}\left(x\right) \tag*{(30)}

where:

n!!=∏j=1n+12(2j−1)n!!=\prod_{j=1}^{\frac{n+1}{2}}(2j-1)

which gives us the final form of the equation:

gp(x)={2Γ(p2)[∑n=0p/2x2nn!]exp⁡(−x2),p∈even2Γ(p2)[∑n=1p−12nn!!x2n−1]exp⁡(−x2)π+erfc⁡(x),p∈odd(31)g_p(x)= \begin{cases} \frac{2}{\Gamma\left(\frac{p}{2}\right)} \left[\sum_{n=0}^{p/2}\frac{x^{2n}}{n!}\right]\exp\left(-x^2\right), & p\in\text{even}\\[1.2em] \frac{2}{\Gamma\left(\frac{p}{2}\right)} \left[\sum_{n=1}^{p-1}\frac{2^n}{n!!}x^{2n-1}\right]\frac{\exp\left(-x^2\right)}{\sqrt{\pi}}+\operatorname{erfc}\left(x\right), & p\in\text{odd} \end{cases} \tag*{(31)}

fpf_p.

fp(x)=2xp−3Γ(p2)∫x∞s2−pexp⁡(−s2) ds(32)f_p(x)=\frac{2x^{p-3}}{\Gamma\left(\frac{p}{2}\right)}\int_x^\infty s^{2-p}\exp\left(-s^2\right)\,\mathrm{d}s \tag*{(32)}

There are two simpler to solve cases, these are: where p=1p=1:

f1=1πexp⁡(−x2)x2,(33)f_1=\frac{1}{\sqrt{\pi}}\frac{\exp\left(-x^2\right)}{x^2}, \tag*{(33)}

and where pp is odd and p>3p > 3

fp=1p! Γ(p2)[∑n=0p2((p−2n)!(−1)n+1x2n)exp⁡(−x2)+(−x2)p−32−1Ei(−x2)](34)f_p = \frac{1}{p!\,\Gamma\left(\frac{p}{2}\right)}\left[\sum_{n=0}^{\frac{p}{2}}\left((p-2n)!(-1)^{n+1}x^{2n}\right)\exp\left(-x^2\right)+\left(-x^2\right)^{\frac{p-3}{2}-1}E_i\left(-x^2\right)\right] \tag*{(34)}

The general case of fpf_p is harder to evaluate. Consider

I(t,x)=∫x∞stexp⁡(−s2) ds(35)I(t,x)=\int_x^\infty s^t\exp\left(-s^2\right)\,\mathrm{d}s \tag*{(35)}

This can be written

I(t,x)=∫x∞[st−1][sexp⁡(−s2)] ds(36)I(t,x)=\int_x^\infty\left[s^{t-1}\right]\left[s\exp\left(-s^2\right)\right]\,\mathrm{d}s \tag*{(36)}

Which can easily be integrated by parts to give

I(t,x)=t−12I(t−2,x)+12xt−1exp⁡(−x2)(37)I(t,x)=\frac{t-1}{2}I(t-2,x)+\frac{1}{2}x^{t-1}\exp\left(-x^2\right) \tag*{(37)}

This can be rearranged to give us an upwards recurrence

I(t+2,x)=t+12I(t,x)+12xt+1exp⁡(−x2)(38)I(t+2,x)=\frac{t+1}{2}I(t,x)+\frac{1}{2}x^{t+1}\exp\left(-x^2\right) \tag*{(38)}

From which given I(0,x)I(0,x) and I(1,x)I(1,x) we can calculate all values of the integral for positive tt. These starting integrals are straightforward:

I(0,x)=π2erfc⁡(x)(39)I(0,x)=\frac{\sqrt{\pi}}{2}\operatorname{erfc}(x) \tag*{(39)}
I(1,x)=12exp⁡(−x2)(40)I(1,x)=\frac{1}{2}\exp\left(-x^2\right) \tag*{(40)}

For negative tt we need a downwards recurrence. This can also be obtained by rearranging the above

I(t−2,x)=2t−1[I(t,x)−12xt−1exp⁡(−x2)](41)I(t-2,x)=\frac{2}{t-1}\left[I(t,x)-\frac{1}{2}x^{t-1}\exp\left(-x^2\right)\right] \tag*{(41)}

While we can use the above for even negative tt it is not suitable for odd as stepping from t=1t=1 to t=−1t=-1 introduces a division by zero. We can solve this by noting

I(−1,x)=12E1(x)(42)I(-1,x)=\frac{1}{2}E_1(x) \tag*{(42)}

where E1(x)E_1(x) is the first order exponential integral and using this as the starting point for odd negative tt.

From I(t,x)I(t,x) fpf_p is trivially calculated

g(x)=2Γ(p2)∫x∞sp−1exp⁡(−s2) ds(43)g(x)=\frac{2}{\Gamma\left(\frac{p}{2}\right)}\int_x^\infty s^{p-1}\exp\left(-s^2\right)\,\mathrm{d}s \tag*{(43)}
g(x)=2Γ(p2)[F(∞)−F(x)](44)g(x)=\frac{2}{\Gamma\left(\frac{p}{2}\right)}[F(\infty)-F(x)] \tag*{(44)}

where : F[s]=∫ssp−1exp⁡(−s2) dsF[s]=\int^s s^{p-1}\exp\left(-s^2\right)\,\mathrm{d}s

dg(x)dx=2Γ(p2)[−F′(x)](45)\frac{\mathrm{d}g(x)}{\mathrm{d}x}=\frac{2}{\Gamma\left(\frac{p}{2}\right)}[-F'(x)] \tag*{(45)}
dg(x)dx=2Γ(p2)[−xp−1exp⁡(−x2)](46)\frac{\mathrm{d}g(x)}{\mathrm{d}x}=\frac{2}{\Gamma\left(\frac{p}{2}\right)}\left[-x^{p-1}\exp\left(-x^2\right)\right] \tag*{(46)}
dfpdx\frac{\mathrm{d}f_p}{\mathrm{d}x}
f(x)=2xp−3Γ(p2)∫x∞s2−pexp⁡(−s2) ds(47)f(x)=\frac{2x^{p-3}}{\Gamma\left(\frac{p}{2}\right)}\int_{x}^{\infty}s^{2-p}\exp\left(-s^{2}\right)\,\mathrm{d}s \tag*{(47)}
f(x)=2xp−3Γ(p2)[F(∞)−F(x)](48)f(x)=\frac{2x^{p-3}}{\Gamma\left(\frac{p}{2}\right)}\left[F(\infty)-F(x)\right] \tag*{(48)}

where: F[s]=∫ss2−pexp⁡(−s2) dsF[s]=\int^{s}s^{2-p}\exp\left(-s^{2}\right)\,\mathrm{d}s

df(x)dx=ddx[2xp−3Γ(p2)][F(∞)−F(x)]+2xp−3Γ(p2)ddx[F(∞)−F(x)](49)\frac{\mathrm{d}f(x)}{\mathrm{d}x}=\frac{\mathrm{d}}{\mathrm{d}x}\left[\frac{2x^{p-3}}{\Gamma\left(\frac{p}{2}\right)}\right]\left[F(\infty)-F(x)\right] +\frac{2x^{p-3}}{\Gamma\left(\frac{p}{2}\right)}\frac{\mathrm{d}}{\mathrm{d}x}\left[F(\infty)-F(x)\right] \tag*{(49)}
=p−3x2Γ(p2)xp−3[F(∞)−F(x)]+2xp−3Γ(p2)[−F′(x)](50)=\frac{p-3}{x}\frac{2}{\Gamma\left(\frac{p}{2}\right)}x^{p-3}\left[F(\infty)-F(x)\right] +\frac{2x^{p-3}}{\Gamma\left(\frac{p}{2}\right)}\left[-F'(x)\right] \tag*{(50)}
=p−3xf(x)−2xp−3Γ(p2)[x2−pexp⁡(−x2)](51)=\frac{p-3}{x}f(x)-\frac{2x^{p-3}}{\Gamma\left(\frac{p}{2}\right)}\left[x^{2-p}\exp\left(-x^{2}\right)\right] \tag*{(51)}
=p−3xf(x)−2xΓ(p2)exp⁡(−x2)(52)=\frac{p-3}{x}f(x)-\frac{2}{x\Gamma\left(\frac{p}{2}\right)}\exp\left(-x^{2}\right) \tag*{(52)}

N.B. our xx is actually:

x=πmβ(53)x=\frac{\pi m}{\beta} \tag*{(53)}

so we find that:

df(x)dm→dxdmdf(x)dx(54)\frac{\mathrm{d}f(x)}{\mathrm{d}m}\rightarrow\frac{\mathrm{d}x}{\mathrm{d}m}\frac{\mathrm{d}f(x)}{\mathrm{d}x} \tag*{(54)}

which gives:

πβdf(x)dx(55)\frac{\pi}{\beta}\frac{\mathrm{d}f(x)}{\mathrm{d}x} \tag*{(55)}

as our actual stress kernel.

Per-particle contributions for SPME

This section follows the notation of the original SPME paper [83]. α\alpha, β\beta, γ\gamma label the lattice vectors. Note all is in the basis of the lattice vectors, should cartesian values be required a basis transformation is needed at the end (see the end of the force calculation for an example in the code)

Consider

ΩABC=12πV∑m≠0S(m)mαAmβBmγCf(m)\Omega^{ABC} = \frac{1}{2\pi V} \sum_{\mathbf{m}\ne0} S(\mathbf{m})m_{\alpha}^{A}m_{\beta}^{B}m_{\gamma}^{C}f(\mathbf{m})

where AA, BB, CC are the derivatives in uu, vv, ww.

It will be shown that all of the energy, force and stresses can be expressed in this form

Given the definition of the structure factor we can write:

ΩABC=12πV∑m≠0∑jqje2πim⋅rjmαAmβBmγCf(m)\Omega^{ABC} = \frac{1}{2\pi V} \sum_{\mathbf{m}\ne0}\sum_j q_j e^{2\pi i\mathbf{m}\cdot\mathbf{r}_j}m_{\alpha}^{A}m_{\beta}^{B}m_{\gamma}^{C}f(\mathbf{m})

Now applying the standard SPME methods.

First start working in scaled coordinates, where the scaling is by the size of the FFT grid we intend to use. So for a grid of size K1∗K2∗K3K_1*K_2*K_3

uμj=Kμrμju_{\mu j}=K_{\mu}r_{\mu j}
ΩABC=12πV∑jqj∑m≠0mαAmβBmγCf(m)∏μe2πimμuμjKμ\Omega^{ABC} = \frac{1}{2\pi V}\sum_j q_j\sum_{\mathbf{m}\ne0}m_{\alpha}^{A}m_{\beta}^{B}m_{\gamma}^{C}f(\mathbf{m})\prod_{\mu}e^{2\pi i\frac{m_{\mu}u_{\mu j}}{K_{\mu}}}

Now applying the SPME approximation:

e2πimμKμuμj≈bμ(mμ)∑kμ=−∞∞Mn(uμj−kμ)e2πimμKμkμe^{2\pi i\frac{m_{\mu}}{K_{\mu}}u_{\mu j}}\approx b_{\mu}(m_{\mu})\sum_{k_{\mu}=-\infty}^{\infty}M_n(u_{\mu j}-k_{\mu})e^{2\pi i\frac{m_{\mu}}{K_{\mu}}k_{\mu}}

Apply periodic boundary conditions:

e2πimμKμuμj≈bμ(m)∑nμ=−∞∞∑kμ=0Kμ−1Mn(uμj−kμ−nμKμ)e2πimμKμkμe^{2\pi i\frac{m_{\mu}}{K_{\mu}}u_{\mu j}}\approx b_{\mu}(m)\sum_{n_{\mu}=-\infty}^{\infty}\sum_{k_{\mu}=0}^{K_{\mu}-1}M_n(u_{\mu j}-k_{\mu}-n_{\mu}K_{\mu})e^{2\pi i\frac{m_{\mu}}{K_{\mu}}k_{\mu}}

However we can go one step further. Differentiating AA times on both sides w.r.t. uμju_{\mu j} and rearranging gives

mμAe2πimμKμuμj≈[Kμ2πi]Abμ(m)×∑nμ=−∞∞∑kμ=0Kμ−1∂AMn(uμj−kμ−nμKμ)∂uμjAe2πimμKμkμm_\mu^A e^{2\pi i \frac{m_\mu}{K_\mu}u_{\mu j}} \approx\left[\frac{K_\mu}{2\pi i}\right]^A b_\mu(m) \times\sum_{n_\mu=-\infty}^{\infty}\sum_{k_\mu=0}^{K_\mu-1}\frac{\partial^A M_n(u_{\mu j}-k_\mu-n_\mu K_\mu)}{\partial u_{\mu j}^A}e^{2\pi i \frac{m_\mu}{K_\mu}k_\mu}

The properties of cardinal B splines means this is easy to evaluate as

dduMn(u)=Mn−1(u)−Mn−1(u−1)\frac{d}{du}M_n(u)=M_{n-1}(u)-M_{n-1}(u-1)

and since the splines are evaluated by recursion we already have all the terms we need already.

Inserting this into the above gives

ΩABC=12πV∑j∑n1,n2,n3=−∞∞∑k1=0K1−1∑k2=0K2−1∑k3=0K3−1[QjABC(k,n)×∑m≠0[∏μbμ(m)e2πimμkμKμ]f(m)]\Omega^{ABC}=\frac{1}{2\pi V}\sum_j\sum_{n_1,n_2,n_3=-\infty}^{\infty}\sum_{k_1=0}^{K_1-1}\sum_{k_2=0}^{K_2-1}\sum_{k_3=0}^{K_3-1}\left[Q_j^{ABC}(\mathbf{k},\mathbf{n})\times\sum_{\mathbf{m}\ne\mathbf{0}}\left[\prod_\mu b_\mu(\mathbf{m})e^{2\pi i\frac{m_\mu k_\mu}{K_\mu}}\right]f(\mathbf{m})\right]
QjABC(k,n)=qj[Kα2πi]A∂A∂uαjAMn(uαj−kα−nαKα)×[Kβ2πi]B∂B∂uβjBMn(uβj−kβ−nβKβ)×[Kγ2πi]C∂C∂uγjCMn(uγj−kγ−nγKγ)Q_j^{ABC}(\mathbf{k},\mathbf{n})=q_j\left[\frac{K_\alpha}{2\pi i}\right]^A\frac{\partial^A}{\partial u_{\alpha j}^A}M_n(u_{\alpha j}-k_\alpha-n_\alpha K_\alpha)\times\left[\frac{K_\beta}{2\pi i}\right]^B\frac{\partial^B}{\partial u_{\beta j}^B}M_n(u_{\beta j}-k_\beta-n_\beta K_\beta)\times\left[\frac{K_\gamma}{2\pi i}\right]^C\frac{\partial^C}{\partial u_{\gamma j}^C}M_n(u_{\gamma j}-k_\gamma-n_\gamma K_\gamma)

Now the sum over mm is almost a discrete Fourier transform. The problems are missing out zero and the infinite range.

We can make it one if

  1. We define f(0)=0f(\mathbf{0})=0 – which it will be for a neutral cell

  1. We only sum out to Kμ−1K_{\mu}-1. This is OK if f(m)f(\mathbf{m}) has decayed to negligible values by the edge of the grid. This should be the case if we have converged the calculation carefully – ultimately the decaying potential term will kill off the function provided we have chosen the grid size and Ewald parameter correctly.

Thus we can define a per particle contribution as

ΩABC=∑jωjABC(56)\Omega^{ABC} = \sum_j \omega_j^{ABC} \tag*{(56)}
ωjABC=12πV∑n1n2n3=−∞∞∑k1K1−1∑k2K2−1∑k3K3−1QjABC(k,n)∑m≠0[∏μbμ(m)e2πimμkμ/Kμ]f(m)(57)\omega_j^{ABC} = \frac{1}{2\pi V} \sum_{n_1n_2n_3=-\infty}^{\infty} \sum_{k_1}^{K_1-1} \sum_{k_2}^{K_2-1} \sum_{k_3}^{K_3-1} Q_j^{ABC}(\mathbf{k},\mathbf{n}) \sum_{\mathbf{m}\ne0} \left[\prod_{\mu} b_\mu(\mathbf{m})e^{2\pi i m_\mu k_\mu/K_\mu}\right] f(\mathbf{m}) \tag*{(57)}
QjABC(k,n)=qj[Kα2πi]A∂A∂uαjAMn(uαj−kα−nαKα)×[Kβ2πi]B∂B∂uβjBMn(uβj−kβ−nβKβ)×[Kγ2πi]C∂C∂uγjCMn(uγj−kγ−nγKγ)(58)\begin{aligned} Q_j^{ABC}(\mathbf{k},\mathbf{n}) = q_j &\left[\frac{K_\alpha}{2\pi i}\right]^A \frac{\partial^A}{\partial u_{\alpha j}^A}M_n(u_{\alpha j}-k_\alpha-n_\alpha K_\alpha)\times\\ &\left[\frac{K_\beta}{2\pi i}\right]^B \frac{\partial^B}{\partial u_{\beta j}^B}M_n(u_{\beta j}-k_\beta-n_\beta K_\beta)\times\\ &\left[\frac{K_\gamma}{2\pi i}\right]^C \frac{\partial^C}{\partial u_{\gamma j}^C}M_n(u_{\gamma j}-k_\gamma-n_\gamma K_\gamma) \tag*{(58)} \end{aligned}

Redefinition of control files

Old Style.

P-cresol 340

temperature   340
ensemble nve       1.0
pressure           0.000
steps              10200000
multiple           1
print              2000
stack              100
stats              2000
trajectory         200000 2000 0
timestep           0.0005
cutoff             10.0
delr               0.5
rvdw cutoff        10.000E+00
ewald precision           1.0000E-06
cap          1.0000E+04
shake tolerance           1.0000E-05
quaternion tolerance      1.0000E-05
job time               2.5000E+05
close time             1.0000E+02
finish

New Style.

title P-cresol 340

io_file_config CONFIG
io_file_field FIELD
io_file_statis STATIS
io_file_revive REVIVE
io_file_revcon REVCON
io_file_cor COR
io_file_currents CURRENTS

print_frequency 2000 steps
stats_frequency 2000 steps
stack_size 100 steps

vdw_cutoff 10.0 ang
padding 0.125 ang
cutoff 10.0 ang
coul_method spme
spme_precision 1e-06

ensemble nve
temperature 340.0 K
pressure_hydrostatic 0.0 katm
timestep 0.0005 ps

time_run 10200000 steps
time_equilibration 0 steps
restart clean

traj_calculate ON
traj_start 200000 steps
traj_interval 2000 steps
traj_key pos

initial_minimum_separation 0.0 ang
equilibration_force_cap 10000.0 k_B.temp/ang

shake_tolerance 1e-05 ang

Implementation of general correlations

In order to support arbitrary correlations in a modular and extensible manner, we separate the code into two components. First implementing the multi-tau algorithm [39], and second supplying an interface to submit data for correlation at runtime.

For the first, Algorithm 1 details updating a correlator with newly observed data xx and yy. In this listing the correlator structure includes rolling shift arrays for the data to be stored in, accumulators, and correlation values, all index by bib_i the block index in the hierarchical data structure. Every mm steps the algorithm proceeds to the next block by averaging the accumulated values of xx and yy that have previously been submitted. This step results in the increasing granularity when bi>1b_i > 1 if m>1m > 1. Every pp steps the shift arrays will again update from the start. The values of xx and yy may be real or complex (in which case a conjugate must be taken). In order to read out the correlation values, the implicit time lags due to any coarse graining need to be unwrapped as shown in Algorithm 2.

During a simulation multiple correlators are spawned, identified with correlating particular observables. As the simulation progresses, whenever a correlation must be updated the get method for the observables it tracks are called and the data submitted with bi=1b_i = 1 (the first block).

For the second part, the interface calls for a pair of abstract Observable types to be supplied which each implement a get method. This method obtains the data to be correlated and acts as a kernel. By designing the module in this way a single loop over arbitrary pairs of Observable types may be completed to compute all correlations, without the need for additional selection logic. The specific functionality is implemented in types deriving from Observable as opposed to including this logic in the statistics collection subroutine itself. For some types this is a relatively trivial task. For example obtaining the velocity of a particular atom. Some a little more complex such as selecting one of the six current types which are also separated by atomic species and k-point. In all cases however, computation is separated from obtaining the data.

Benchmark data

**input:** $x$: first value correlated, $y$: second value correlated, $b_i$: the
    correlator block to update, cor: correlator data to update.
**if** $b_i > \mathrm{cor.blocks}$ **then**
    **return**
**end**
$\mathrm{cor.blocksUsed} \leftarrow \max(b_i,\mathrm{cor.blocksUsed})$
$\mathrm{cor.shift}[b_i,\mathrm{cor.shift}[b_i,:]] \leftarrow x,y$
$\mathrm{cor.validShift}[b_i] \leftarrow \max(\mathrm{cor.shift}[b_i],1)$
$\mathrm{cor.accumulator}[b_i,:] \leftarrow \mathrm{cor.accumulate}[b_i,:]+x,y$
$\mathrm{cor.accumulated}[b_i] \leftarrow \mathrm{cor.accumulated}[b_i]+1$
**if** $\mathrm{cor.accumulated}[b_i] > \mathrm{cor.m}$ **then**
    $u \leftarrow \mathrm{cor.accumulator}[b_i,1]/\mathrm{cor.accumulated}[b_i]$
    $v \leftarrow \mathrm{cor.accumulator}[b_i,2]/\mathrm{cor.accumulated}[b_i]$
    $\mathrm{Update}(u,v,b_i+1,\mathrm{cor})$
    $\mathrm{cor.accumulator}[b_i,:] \leftarrow 0;\ \mathrm{cor.accumulated}[b_i,:] \leftarrow 0$
**end**
$i \leftarrow \mathrm{cor.shift}[b_i]$
**if** $i == 1$ **then**
    $j \leftarrow i$
    **for** $n \leftarrow 1,\mathrm{cor.validShifts}[b_i]$ **do**
        $\mathrm{cor.values}[b_i,n] \leftarrow$
        $\mathrm{cor.values}[b_i,n]+\mathrm{cor.shift}[b_i,n,1]*\mathrm{Conj}(\mathrm{cor.shift}[b_i,j,2])$
        $\mathrm{cor.count}[b_i,n] \leftarrow \mathrm{cor.count}[b_i,n]+1$
        $j \leftarrow j-1$
        **if** $j \leq 0$ **then**
            $j \leftarrow j+p$
        **end**
    **end**
**else**
    $j \leftarrow i-\lfloor\mathrm{cor.p}/\mathrm{cor.m}\rfloor+1$
    **for** $n \leftarrow \lfloor\mathrm{cor.p}/\mathrm{cor.m}\rfloor+1,\mathrm{cor.p}$ **do**
        **if** $j \leq 0$ **then**
            $j \leftarrow j+\mathrm{cor.p}$
        **end**
        $\mathrm{cor.values}[b_i,n] \leftarrow$
        $\mathrm{cor.values}[b_i,n]+\mathrm{cor.shift}[b_i,n,1]*\mathrm{Conj}(\mathrm{cor.shift}[b_i,j,2])$
        $\mathrm{cor.count}[b_i,n] \leftarrow \mathrm{cor.count}[b_i,n]+1$
        $j \leftarrow j-1$
    **end**
**end**
$\mathrm{cor.shift}[b_i] \leftarrow \mathrm{cor.shift}[b_i]+1$
**if** $\mathrm{cor.shift}[b_i] > p$ **then**
    $\mathrm{cor.shift}[b_i] \leftarrow 1$
**end**

Algorithm 1: Updating a correlator

input : cor: correlator to readout.
output: values: the correlation values, times: the time lags between correlation values.

$\tau \leftarrow 1$
for $i \leftarrow 1$, cor.p do
    if cor.count$[1, i] > 0$ then
        values$[\tau] \leftarrow$ values$[\tau]$ + cor.values$[1, i]$/cor.count$[1, i]$
        times$[\tau] \leftarrow i - 1$
        $\tau \leftarrow \tau + 1$
    end
end
for $k \leftarrow 1$, cor.blocksUsed do
    for $i \leftarrow \lfloor\mathrm{cor.p}/\mathrm{cor.m}\rfloor + 1$, cor.p do
        if cor.count$[k, i] > 0$ then
            values$[\tau] \leftarrow$ values$[\tau]$ + cor.values$[k, i]$/cor.count$[k, i]$
            times$[\tau] \leftarrow (i - 1) * m^{k-1}$
            $\tau \leftarrow \tau + 1$
        end
    end
end

Algorithm 2: Readout of a correlation

MethodOn-the-flyTrajectory Saving
MPI Cores12864012801286401280
Baseline (s)123.1966.1475.40123.1966.1475.40
Sampling intervalWall time (s)
1617.24190.31145.131072.343896.314182.64
2401.74129.71103.83597.862464.962383.92
4312.1599.4590.40353.981623.541280.41
8268.3387.9482.55238.281305.37548.13
16245.2581.0884.52179.88689.58650.48
32235.4980.9681.70147.60374.37457.52

Table E.1. Absolute wall times for the strong scaling benchmarks in figure 9 for the on-the-fly and trajectory saving methods. The baseline walls times indicate simulations performed with neither trajectory saving or on-the-fly correlations.

MethodReplayPython FFT
MPI Cores12864012801286401280
Baseline (s)123.1966.1475.40123.1966.1475.40
Sampling intervalWall time (s)
1807.661027.331197.842367.952513.562815.13
2351.52505.57604.711182.091241.041395.91
4123.73264.06405.30599.24656.39720.67
848.62154.71213.92295.79320.55365.25
1632.8779.75124.14148.95178.81193.01
3223.6355.9493.8878.7491.49110.22

Table E.2. Absolute wall times for the strong scaling benchmarks in figure 9 for post-processing methods. The baseline walls times indicate simulations performed with neither trajectory saving or on-the-fly correlations.

Atoms80002700064000125000512000
Baseline (s)16.0016.4320.3426.8649.89
Sampling intervalWall time (s)
134.1454.0361.6470.61114.15
222.3632.6531.6245.1680.36
417.7924.8124.6534.3068.73
817.5221.2221.3130.9762.61
1617.1218.1020.2628.6259.04
3215.9817.8720.0229.6557.70

Table E.3. Absolute wall times for the on-the-fly method weak scaling benchmarks in figure 10. The baseline walls times indicate simulations performed with neither trajectory saving or on-the-fly correlations.

Atoms80002700064000125000512000
Baseline (s)16.0016.4320.3426.8649.89
Sampling intervalWall time (s)
1126.32116.81530.03437.782701.92
263.2868.59147.26197.841802.02
439.1939.5769.90130.181356.89
826.8935.8648.4372.15997.88
1621.3220.0633.3252.30542.28
3216.2618.1725.2236.30322.02

Table E.4. Absolute wall times for the trajectory saving method weak scaling benchmarks in Figure 10. The baseline walls times indicate simulations performed with neither trajectory saving or on-the-fly correlations.

Atoms80002700064000125000512000
Baseline (s)16.0016.4320.3426.8649.89
Sampling intervalWall time (s)
129.6970.5873.13258.03804.66
29.4024.3120.6463.96401.23
44.9813.348.9033.96215.81
82.655.576.6026.34116.58
161.755.015.0416.5748.43
321.442.874.0615.3045.05

Table E.5. Absolute wall times for the replay post-processing method weak scaling benchmarks in Figure 10. The baseline walls times indicate simulations performed with neither trajectory saving or on-the-fly correlations.

Atoms80002700064000125000512000
Baseline (s)16.0016.4320.3426.8649.89
Sampling intervalWall time (s)
120.0865.65145.54291.431242.89
210.2732.9773.62144.26632.86
45.2316.8037.0775.26314.30
82.778.2618.9537.75160.83
161.304.309.7119.2090.36
320.752.275.0610.0245.99

Table E.6. Absolute wall times for the Python FFT post-processing method weak scaling benchmarks in Figure 10. The baseline walls times indicate simulations performed with neither trajectory saving or on-the-fly correlations.

References

  1. [1]M. P. Allen and D. J. Tildesley. Computer Simulation of Liquids. Oxford University Press, 2007.
  2. [2]D. Frenkel and B. Smit. Understanding molecular simulation: from algorithms to applications, volume 2. Academic Press San Diego, 2002.
  3. [3]K Nordlund. Computational materials science of ion irradiation. Nuclear Instruments and Methods in Physics Research Section B: Beam Interactions with Materials and Atoms, 188(1-4):41–48, 2002.DOI
  4. [4]Alper T Celebi, Seyed Hossein Jamali, André Bardow, Thijs JH Vlugt, and Othonas A M Moultós. Finite-size effects of diffusion coefficients computed from molecular dynamics: a review of what we have learned so far. Molecular Simulation, 47(10-11):831–845, 2021.DOI
  5. [5]E. Zarkadoula, S. L. Daraszewicz, D. M. Duffy, M. A. Seaton, I. T. Todorov, K. Nordlund, M. T. Dove, and K. Trachenko. The nature of high-energy radiation damage in iron. J. Phys.: Condens. Matt., 25(12):125402, feb 2013. doi: 10.1088/0953-8984/25/12/125402. URL https://dx.doi.org/10.1088/0953-8984/25/12/125402.
  6. [6]0.5 MeV collision cascade animation. https://ccmpp.ph.gmul.ac.uk/~kostya/rad.html. Accessed: 2025-01-14.
  7. [7]George Beckett, Josephine Beech-Brandt, Kieran Leach, Zoë Payne, Alan Simpson, Lorna Smith, Andy Turner, and Anne Whiting. Archer2 service description, December 2024. URL https://doi.org/10.5281/zenodo.14507040.
  8. [8]Archer2 user documentation: Data management and transfer. (https://docs.archer2.ac.uk/user-guide/data/), Accessed: 2025-01-21.
  9. [9]Kien Nguyen-Cong, Jonathan T Willman, Stan G Moore, Anatoly B Belonoshko, Rahulkumar Gayatri, Evan Weinberg, Mitchell A Wood, Aidan P Thompson, and Ivan I Oleynik. Billion atom molecular dynamics simulations of carbon at extreme conditions and experimental time and length scales. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–12, 2021.DOI
  10. [10]Abhishek Singharoy, Christopher Maffeo, Karelia H Delgado-Magnero, David JK Swainsbury, Melih Sener, Ulrich Kleinekathöfer, John W Vant, Jonathan Nguyen, Andrew Hitchcock, Barry Isra lewitz, et al. Atoms to phenotypes: molecular design principles of cellular energy metabolism. Cell, 179(5):1098–1111, 2019.DOI
  11. [11]Abigail Dommer, Lorenzo Casalino, Fiona Kearns, Mia Rosenfeld, Nicholas Wauer, Surl-Hee Ahn, John Russo, Sofia Oliveira, Clare Morris, Anthony Bogetti, et al. # covidisairborne: Ai-enabled multiscale computational microscopy of delta sars-cov-2 in a respiratory aerosol. The international journal of high performance computing applications, 37(1):28–44, 2023.DOI
  12. [12]Jan A Stevens, Fabian Grünewald, PA van Tilburg, Melanie König, Benjamin R Gilbert, Troy A Brier, Zane R Thornburg, Zaida Luthey-Schulten, and Siewert J Marrink. Molecular dynamics simulation of an entire cell. Frontiers in Chemistry, 11:1106495, 2023.
  13. [13]Giacomo Fiorin, Michael L Klein, and Jérôme Hénin. Using collective variables to drive molecular dynamics simulations. Molecular Physics, 111(22-23):3345–3362, 2013.DOI
  14. [14]Massimiliano Bonomi, Davide Branduardi, Giovanni Bussi, Carlo Camilloni, Davide Provasi, Paolo Raiteri, Davide Donadio, Fabrizio Marinelli, Fabio Pietrucci, Ricardo A Broglia, et al. Plumed: A portable plugin for free-energy calculations with molecular dynamics. Computer Physics Communications, 180(10):1961–1972, 2009.DOI
  15. [15]Gareth A Tribello, Massimiliano Bonomi, Davide Branduardi, Carlo Camilloni, and Giovanni Bussi. Plumed 2: New feathers for an old bird. Computer physics communications, 185(2):604–613, 2014.DOI
  16. [16]Promoting transparency and reproducibility in enhanced molecular simulations. Nature methods, 16(8):670–673, 2019.DOI
  17. [17]Silvina Caino-Lores, Michel Cuendet, Jack Marquez, Ekaterina Kots, Trilce Estrada, Ewa Deelman, Harel Weinstein, and Michela Taufer. Runtime steering of molecular dynamics simulations through in situ analysis and annotation of collective variables. In Proceedings of the Platform for Advanced Scientific Computing Conference, pages 1–11, 2023.
  18. [18]Mark James Abraham, Teemu Murtola, Roland Schulz, Szilárd Páll, Jeremy C Smith, Berk Hess, and Erik Lindahl. Gromacs: High performance molecular simulations through multi-level parallelism from laptops to supercomputers. SoftwareX, 1:19–25, 2015.DOI
  19. [19]Aidan P Thompson, H Metin Aktulga, Richard Berger, Dan S Bolintineanu, W Michael Brown, Paul S Crozier, Pieter J Int’t Veld, Axel Kohlmeyer, Stan G Moore, Trung Dac Nguyen, et al. Lammps-a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales. Computer physics communications, 271:108171, 2022.
  20. [20]Peter Lindstrom. Fixed-rate compressed floating-point arrays. IEEE transactions on visualization and computer graphics, 20(12):2674–2683, 2014.DOI
  21. [21]Pierre de Buyl, Peter H Colberg, and Felix Höfling. H5md: A structured, efficient, and portable file format for molecular data. Computer Physics Communications, (6):1546–1553, 2014.DOI
  22. [22]The hdf group, hierarchical data format version 5. URL http://www.hdfgroup.org/HDF5.
  23. [23]A Diver, O Dicks, AM Elena, IT Todorov, and K Trachenko. Evolution of amorphous structure under irradiation: zircon case study. Journal of Physics: Condensed Matter, 32(41):415703, 2020.DOI
  24. [24]I. T. Todorov, W. Smith, K. Trachenko, and M.T. Dove. DL_POLY_3: new dimensions in molecular dynamics simulations via massive parallelism. J. Mater. Chem., 16:1911, 2006.DOI
  25. [25]I. T. Todorov and W Smith. DL_POLY_3: the CCP5 national UK code for molecular–dynamics simulations. Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, 362(1822):1835–1852, 2004.
  26. [26]Ilian T. Todorov, William Smith, Kostya Trachenko, and Martin T. Dove. DL_POLY_3: new dimensions in molecular dynamics simulations via massive parallelism. Journal of Materials Chemistry, 16(20):1911–1918, 2006.DOI
  27. [27]U. Balucani and M. Zoppi. Dynamics of the Liquid State. Oxford University Press, 2003.
  28. [28]DL_POLY, 2024. URL https://gitlab.com/ccp5/dl-poly.
  29. [29]Richard M. Karp. On-line algorithms versus off-line algorithms: How much. In Algorithms, Software, Architecture: Information Processing 92: Proceedings of the IFIP 12th World Computer Congress, volume 1, page 416, 1992.
  30. [30]Georg Krempl, Indre Žliobaitė, Dariusz Brzeziński, Eyke Hüllermeier, Mark Last, Vincent Lemaire, Tino Noack, Ammar Shaker, Sonja Sievi, Myra Spiliopoulou, et al. Open challenges for data stream mining research. ACM SIGKDD explorations newsletter, 16(1):1–10, 2014.DOI
  31. [31]Mayur Datar, Aristides Gionis, Piotr Indyk, and Rajeev Motwani. Maintaining stream statistics over sliding windows. SIAM journal on computing, 31(6):1794–1813, 2002.
  32. [32]Robert Zwanzig and Raymond D Mountain. High-frequency elastic moduli of simple fluids. The Journal of Chemical Physics, 43(12):4464–4471, 1965.DOI
  33. [33]Mark E Tuckerman. Statistical mechanics: theory and molecular simulation. Oxford university press, 2023.
  34. [34]John R. Ray and Aneesur Rahman. Statistical ensembles and molecular dynamics studies of anisotropic solids. II. The Journal of Chemical Physics, 82(9):4243–4247, 05 1985. ISSN 0021-9606. doi: 10.1063/1.448813. URL https://doi.org/10.1063/1.448813.
  35. [35]D. R. Squire, A. C. Holt, and W. G. Hoover. Isothermal elastic constants for argon, theory and monte carlo calculations. Physica, 42(3):388–397, 1969.DOI
  36. [36]J. F. Lutsko. Generalized expressions for the calculation of elastic constants by computer simulation. Journal of applied physics, 65(8):2991–2997, 1989.DOI
  37. [37]Bingqing Cheng and Daan Frenkel. Computing the heat conductivity of fluids from density fluctuations. Physical Review Letters, 125(13):130602, 2020.DOI
  38. [38]C. Yang, M. T. Dove, V. V. Brazhkin, and K. Trachenko. Emergence and evolution of the k-gap in spectra of liquid and supercritical states. Phys. Rev. Lett., 118:215502, 2017.DOI
  39. [39]Jorge Ramírez, Sathish K. Sukumaran, Bart Vorselaars, and Alexei E. Lichtman. Efficient on the fly calculation of time correlation functions in computer simulations. The Journal of Chemical Physics, 133(15):154103, 10 2010. ISSN 0021-9606. doi: 10.1063/1.3491098. URL https://doi.org/10.1063/1.3491098.
  40. [40]P. R. Kasten M. W. Rosenthal and R. B. Briggs. Molten-salt reactors—history, status, and potential. Nuclear Applications and Technology, 8(2):107–117, 1970. doi: 10.13182/NT70-A28619. URL https://doi.org/10.13182/NT70-A28619.
  41. [41]Benjamin A. Frandsen, Stella D. Nickerson, Austin D. Clark, Andrew Solano, Raju Baral, Johnny Williams, Jörg Neuefeind, and Matthew Memmott. The structure of molten FLiNaK. Journal of Nuclear Materials, 537:152219, 2020. ISSN 0022-3115. doi: https://doi.org/10.1016/j.jnucmat.2020.152219. URL https://www.sciencedirect.com/science/article/pii/S0022311520304736.
  42. [42]Yong Zhang, Akihito Otani, and Edward J. Maginn. Reliable viscosity calculation from equilibrium molecular dynamics simulations: A time decomposition method. Journal of chemical theory and computation, 11(8):3537–3546, 2015.DOI
  43. [43]Jeff Armstrong and Fernando Bresme. Thermal conductivity of highly asymmetric binary mixtures: how important are heat/mass coupling effects? Physical Chemistry Chemical Physics, 16(24):12307–12316, 2014.DOI
  44. [44]C. Cockrell, M. Withington, H. L. Devereux, A. M. Elena, I. T. Todorov, Z. K. Liu, S. L. Shang, J. S. McCloy, P. A. Bingham, and K. Trachenko. Thermal conductivity and thermal diffusivity of molten salts: insights from molecular dynamics simulations and fundamental bounds. arXiv preprint arXiv:2409.03775, 2024.arxiv.org/abs/2409.03775
  45. [45]National Institute of Standards and Technology database. URL https://webbook.nist.gov/chemistry/fluid.
  46. [46]C. Cockrell, V. V. Brazhkin, and K. Trachenko. Universal interrelation between dynamics and thermodynamics and a dynamically driven “c” transition in fluids. Phys. Rev. E, 104:034108, Sep 2021. doi: 10.1103/PhysRevE.104.034108. URL https://link.aps.org/doi/10.1103/PhysRevE.104.034108.DOI
  47. [47]K. Trachenko and V. V. Brazhkin. Minimal quantum viscosity from fundamental physical constants. Sci. Adv., 6:eaba3747, 2020.DOI
  48. [48]K. Trachenko and V. V. Brazhkin. The quantum mechanics of viscosity. Physics Today, 74(12):66, 2021.
  49. [49]K. Trachenko, M. Baggioli, K. Behnia, and V. V. Brazhkin. Universal lower bounds on energy and momentum diffusion in liquids. Phys. Rev. B, 103:014311, 2021.DOI
  50. [50]Germain Clavier, Nicolas Desbiens, Emeric Bourasseau, Véronique Lachet, Nadège Brusselle-Dupend, and Bernard Rousseau. Computation of elastic constants of solids using molecular simulation: comparison of constant volume and constant pressure ensemble methods. Molecular Simulation, 43(17):1413–1422, 2017.
  51. [51]D. J. Quesnel, D. S. Rimai, and L. P. DeMejo. Elastic compliances and stiffnesses of the fcc Lennard-Jones solid. Physical Review B, 48(10):6795, 1993.
  52. [52]Andrey Pereverzev. Isothermal and adiabatic elastic constants from virial fluctuations. Physical Review E, 106(4):044110, 2022.DOI
  53. [53]Aidan Thompson and Germain Clavier. A general method for calculating local stress and elastic constants for arbitrary many-body interaction potentials in LAMMPS. Technical report, Sandia National Lab.(SNL-NM), Albuquerque, NM (United States), 2022.DOI
  54. [54]Richard Hill. The elastic behaviour of a crystalline aggregate. Proceedings of the Physical Society. Section A, 65(5):349, 1952.
  55. [55]ER Dobbs and Gwyn Owain Jones. Theory and properties of solid argon. Reports on Progress in Physics, 20(1):516, 1957.
  56. [56]O. G. Peterson, D. N. Batchelder, and R. O. Simmons. Measurements of X-ray lattice constant, thermal expansivity, and isothermal compressibility of argon crystals. Phys. Rev., 150:703–711, Oct 1966. doi: 10.1103/PhysRev.150.703. URL https://link.aps.org/doi/10.1103/PhysRev.150.703.DOI
  57. [57]M. Baggioli, M. Vasin, V. Brazhkin, and K. Trachenko. Gapped momentum states. Physics Reports, 865:1, 2020.
  58. [58]A Brodka and TW Zerda. A molecular dynamics simulation of sulphur hexafluoride. Molecular Physics, 76(1):103–112, 1992.
  59. [59]C. Cockrell, V. V. Brazhkin, and K. Trachenko. Transition in the supercritical state of matter: Review of experimental evidence. Physics Reports, 941:1–27, 12 2021. ISSN 0370-1573. doi: 10.1016/J.PHYSREP.2021.10.002.DOI
  60. [60]V. V. Brazhkin, Yu. D. Fomin, A. G. Lyapin, V. N. Ryzhov, E. N. Tsiok, and Kostya Trachenko. “liquid-gas” transition in the supercritical region: Fundamental changes in the particle dynamics. Phys. Rev. Lett., 111:145901, 2013.
  61. [61]C. Yang, V. V. Brazhkin, M. T. Dove, and K. Trachenko. Frenkel line and solubility maximum in supercritical fluids. Phys. Rev. E, 91:012112, 2015.
  62. [62]Ioannis Skarmoutsos, Leonidas I. Kampanakis, and Jannis Samios. Investigation of the vapor–liquid equilibrium and supercritical phase of pure methane via computer simulations. Journal of molecular liquids, 117(1-3):33–41, 2005.DOI
  63. [63]M. Frigo and S.G. Johnson. The design and implementation of fftw3. Proceedings of the IEEE, 93(2):216–231, 2005. doi: 10.1109/JPROC. 2004.840301.DOI
  64. [64]Rajeev Thakur, William Gropp, and Ewing Lusk. On implementing MPI-IO portably and with high performance. In Proceedings of the Sixth Workshop on Input/Output in Parallel and Distributed Systems, page 23–32, May 1999.
  65. [65]W Smith and TR Forester. DL_POLY_2.0: A general-purpose parallel molecular dynamics simulation package. Journal of molecular graphics, 14(3):136–141, 1996.
  66. [66]W Smith, CW Yong, and PM Rodger. DL_POLY: Application to molecular simulation. Molecular Simulation, 28(5):385–471, 2002.
  67. [67]DL_POLY_CLASSIC. https://gitlab.com/DL_POLY_Classic. Accessed: 2025-01-21.
  68. [68]W Smith and TR Forester. Parallel macromolecular simulations and the replicated data strategy: I. the computation of atomic forces. Computer physics communications, 79(1):52–62, 1994.
  69. [69]W Smith and TR Forester. Parallel macromolecular simulations and the replicated data strategy: II. the RD-SHAKE algorithm. Computer physics communications, 79(1):63–77, 1994.
  70. [70]Walt Brainerd. Fortran 77. Communications of the ACM, 21(10):806–820, 1978.
  71. [71]David W. Walker and Jack J. Dongarra. MPI: a standard message passing interface. Supercomputer, 12:56–68, 1996.
  72. [72]I. J. Bush, Ilian T. Todorov, and W. Smith. A DAFT DL_POLY distributed memory adaptation of the smoothed particle mesh Ewald method. Computer physics communications, 175(5):323–329, 2006.DOI
  73. [73]I. Todorov, L. Ellison, and W. Smith. Rigid body molecular dynamics within the domain decomposition framework of DL_POLY_4. Parallel Computing Technologies: Lecture Notes in Computer Science (V. Malyshkin (Ed.)), 7979:429–435, 2013. doi: https://doi.org/10.1007/978-3-642-39958-9_40.
  74. [74]M. A. Seaton, I. T. Todorov, S. L. Daraszewicz, G. S. Khara, and D. M. Duffy. Domain decomposition of the two-temperature model in DL_POLY_4. Molecular Simulation, 47:180–187, 2021. doi: https://doi.org/10.1080/08927022.2018.1511902.
  75. [75]I. J. Bush, Ilian T. Todorov, and W. Smith. Optimisation of the I/O for distributed data molecular dynamics applications. Cray User Group Proceedings 2010, 2010. URL https://cug.org/5-publications/proceedings_attendee_lists/CUG10CD/.
  76. [76]T. Shardlow. Splitting for dissipative particle dynamics. SIAM Journal on Scientific Computing, 24:1267–1282, 2003. doi: https://doi.org/10.1137/S1064827501392879.
  77. [77]M. Seaton, I. Todorov, and A. Yaser. Efficient domain decomposition of dissipative particle dynamics via choice of pseudorandom number generator. Parallel Computing Technologies: Lecture Notes in Computer Science (V. Malyshkin (Ed.)), 7979:250–257, 2013. doi: https://doi.org/10.1007/978-3-642-39958-9_23.
  78. [78]Oren Ben-Kiki, Clark Evans, and Brian Ingerson. Yaml ain’t markup language (yaml™) version 1.1. Working Draft 2008, 5(11), 2009.
  79. [79]Alin Elena, Jacob Wilkins, and Harvey Devereux. dlpoly-py, March 2025. URL https://doi.org/10.5281/zenodo.14983396.
  80. [80]Ivan Scivetti, Kakali Sen, Alin M. Elena, and Ilian Todorov. Reactive molecular dynamics at constant pressure via nonreactive force fields: Extending the empirical valence bond method to the isothermal-isobaric ensemble. The Journal of Physical Chemistry A, 124(37):7585–7597, 2020. doi: 10.1021/acs.jpca.0c05461. URL https://doi.org/10.1021/acs.jpca.0c05461. PMID: 32820921.
  81. [81]You Lu, Matthew R. Farrow, Pierre Fayon, Andrew J. Logsdail, Alexey A. Sokol, C. Richard A. Catlow, Paul Sherwood, and Thomas W. Keal. Open-source, Python-based redevelopment of the ChemShell multiscale QM/MM environment. Journal of Chemical Theory and Computation, 15(2):1317–1328, 2019. doi: 10.1021/acs.jctc.8b01036. URL https://doi.org/10.1021/acs.jctc.8b01036. PMID: 30511845.
  82. [82]P. P. Ewald. Die Berechnung optischer und elektrostatischer Gitterpotentiale [On the calculation of optical and electrostatic grid potentials]. Annalen der Physik, 369(3):253–287, 1921.
  83. [83]Ulrich Essmann, Lalith Perera, Max L. Berkowitz, Tom Darden, Hsing Lee, and Lee G. Pedersen. A smooth particle mesh Ewald method. The Journal of Chemical Physics, 103(19):8577–8593, 11 1995. ISSN 0021-9606. doi: 10.1063/1.470117. URL https://doi.org/10.1063/1.470117.
  84. [84]J. E. Lennard-Jones. On the determination of molecular fields. Proc. R. Soc. Lond. A, 106(738):463–477, 1924.
  85. [85]Archer2 user documentation: Archer2 hardware. (https://docs.archer2.ac.uk/user-guide/hardware/#system-overview), . Accessed: 2025-02-21.
  86. [86]Martyn F. Guest, Alin M. Elena, and Aidan B. G. Chalk. DL_POLY - a performance overview analysing, understanding and exploiting available hpc technology. Molecular Simulation, 47(2-3):194–227, 2021.
  87. [87]I. S. Gradshteyn and I. M. Ryzhik. Table of Integrals, Series, and Products. Elsevier, 2007. ISBN 9780123736376.

Paper details

vlnc:W_rv8duhay
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents