betaFeedback

WorksPhysics and Astronomy

Accelerating phase-field simulations on exascale computing systems for faster-than-real-time precipitate aging predictions

Abstract

Three-dimensional phase-field simulations are a gold-standard for microstructure prediction for materials, but their computational cost often limits application-relevant calculations to modest domain sizes and timescales. Here, we present a holistic approach for accelerating Fourier pseudospectral phase-field simulations by combining performance-portable GPU computing, large-scale distributed-memory parallelism, and high-order implicit-explicit time integration within the MEUMAPPS C++ framework. We demonstrate this approach with a 1.5-billion-grid-point simulation of the growth and coarsening of 1,920 gamma'' precipitates in a Ni-Nb-Fe alloy. The simulated seven-hour heat treatment is completed in 5.5 hours, making the calculation faster than real time and an estimated 217-501x faster than a CPU-only, first-order baseline. This performance makes three-dimensional simulations of thousands of interacting precipitates tractable, enabling quantitative studies ocollective microstructural phenomena, large simulation ensembles, and real-time integration into controlling and interpreting experiments. Benchmarking shows single-node GPU speedups of up to 17.5x relative to comparable CPU resources, near-ideal strong scaling to 4096 GPUs for a 12-billion-grid-point problem, and a further 2.6-2.9x acceleration from fourth-order time integration at scientifically relevant error tolerances. These results establish a performance-portable strategy for exploiting leadership-scale GPU systems that is applicable to a broad class of Fourier pseudospectral simulations beyond phase-field models such as fluid dynamics and crystal plasticity simulations.

Introduction

Phase-field models have been demonstrated to accurately describe the dynamics of phase separation and evolution across a variety of scientific domains [1, 2, 3], but 3D3\mathrm{D} phase-field simulations usually require much more time to run than the underlying physics – taking days or weeks to calculate phenomena that occur in seconds or hours [4, 5, 6]. In this article we present a multiprong approach to accelerate phase-field simulations of solid-state precipitation in a nickel-base alloy. Combining GPU acceleration, large-scale distributed computing, and high-order implicit-explicit (ImEx) time integration, we demonstrate a faster-than-real-time simulation of aging of nearly 2000 precipitates under standard heat treatment conditions and estimate a two order-of-magnitude decrease in time-to-solution over a baseline approach. This simulation, with 1.5 billion grid points, is one of the largest phase-field simulations conducted to date.

Phase-field models provide a description of the evolution of a multiphase system governed by thermodynamic driving forces and using a thermodynamically consistent diffuse transition between phases. In metal alloys, phase-field models can describe the evolution of second-phase particles (“precipitates”) that form through solid-solid phase transformations. The evolution of precipitates can have a large influence on the properties of materials, impacting strength as well as the susceptibility to fracture and corrosion. For reviews on phase-field models both generally and for precipitation, refer to Refs. [1, 2, 3, 7, 8].

Phase-field simulations of precipitation are computationally intensive, often requiring large physical domains to accommodate many precipitates and long time horizons. One particular area of interest is in coarsening dynamics, where the larger precipitates grow at the expense of smaller ones, and particle-resolving methods (like phase field) can be used to test the validity of mean-field models [4, 9, 10]. 2D2\mathrm{D} simulations of coarsening dynamics have shown that thousands of particles are necessary to probe how the dynamics are affected by factors such as elasticity, with Ref. [9] as an example where the coarsening of 4000 particles was simulated until only 100 remained. Performing these studies in 3D3\mathrm{D} has proven difficult, owing to the massive computational expense. As an example, a simulation of the coarsening of 510 precipitates by Poulsen et. al [4] required weeks to finish on 200 CPU cores. The Fourier pseudospectral method has been widely used for phase-field simulations, especially for solid-state transformations where the ease of implementing ImEx time integration and nonlinear elasticity solvers is attractive [3, 7, 8, 11, 12, 13, 14]. The Fourier pseudospectral method is a spatial discretization method where field variables are represented by a series expansion in sine and cosine functions and is part of a broader class of spectral methods [15]. Compared to other common spatial discretization methods for partial differential equations, such as the finite difference method and finite element methods that use a local representation of the field for derivatives, Fourier pseudospectral methods use a global basis. This global basis yields excellent (exponential) convergence of the truncation error, but means that different parallelization strategies are needed than for local methods [16]. Outside of phase-field applications, Fourier pseudospectral methods are used in a wide variety of applications including: crystal plasticity, fracture, and turbulent fluid flow [15, 17, 18]. Historically, pseudospectral phase-field simulations have been carried out using low-order ImEx time integration schemes. In their pioneering work, Chen and Shen [11] developed what is now the standard ImEx splitting approach for phase-field models. They present first, second, and third order schemes where the stiff term is handled implicitly using a backward difference formula (BDF) and the nonlinear term is handled explicitly using an Adams-Bashforth (AB) scheme. In practice often only the first-order scheme is used [12, 14, 19, 20, 21, 22].

Most large-scale phase-field simulations of precipitation have been performed on modest CPU-only resources that are small compared to modern leadership-scale supercomputers, albeit with some initial forays into larger calculations on many CPU cores or GPUs. For example, Refs. [4, 23, 24, 25] contain examples of simulations using the finite difference or finite element methods and between 16 and 1024 CPU cores. For pseudospectral methods, simulations with up to 1767 CPU cores have been reported [19]. GPU calculations using the pseudospectral method have been reported, usually with one GPU [22, 26, 27, 28], but up to four GPUs [29] or six GPUs [30]. Comparing one CPU core to one GPU, speedups of 40–58×\times have been observed [26, 31], highlighting the potential of GPUs for these calculations. Looking outside of precipitation simulations, 100s to 10,000s of GPUs have been utilized for finite element and finite difference phase-field simulations of dendritic solidification [32, 33, 34] and 65,536 CPU cores have been utilized for pseudospectral phase-field crystal simulations [35]. These examples highlight the untapped opportunity for pseudospectral phase-field simulations to leverage the 10,000s of GPUs available on modern supercomputers.

In this article we present a new holistic approach for accelerating phase-field simulations of precipitation using the Fourier pseudospectral method. Although this work focuses on phase-field simulations, the approach is broadly applicable to other applications of the Fourier pseudospectral method. The contributions of this work are five-fold:

  1. A demonstration of the decrease in calculation time using GPUs compared to only

CPUs, with up to a 17.5× speedup on 8 GPUs versus 64 CPU cores.

  1. Strong and weak scaling tests from 8 to 8192 GPUs, demonstrating near-ideal strong scaling up to 4,096 GPUs for a simulation with 12 billion grid points.

  1. An investigation of the use of first- through fourth-order implicit-explicit additive Runge–Kutta (ImEx-ARK) time integration schemes, demonstrating a 2.9× speed-up for the fourth-order method over the standard first-order method for science-relevant levels of error.

  1. A demonstration of the above methods to accelerate a phase-field simulation of precipitation by 217-501×, making the simulation faster than the corresponding real-time experiment.

  1. A discussion of the implementation of the above approaches in the MEUMAPPS C++ software framework.

Results and Discussion

To demonstrate the computational performance of our holistic approach to accelerating Fourier pseudospectral phase-field simulations, we present a simulation of the growth and coarsening of nearly 2000 precipitates on 1024 GPUs. We choose a technologically relevant setting for this demonstration – the evolution of γ′′\gamma^{\prime\prime} precipitates in a Ni-Fe-Nb alloy. This alloy has been used as a ternary surrogate for IN718 [19, 36], a common precipitation-hardened nickel-base superalloy used for applications where high strength is needed at high temperatures, and γ′′\gamma^{\prime\prime} precipitates are the most important strengthening phase for IN718 [10]. Due to its industrial importance, the evolution of γ′′\gamma^{\prime\prime} in binary and ternary surrogates of IN718 has been the subject of a number of studies using phase-field simulations [10, 19, 36, 37, 38, 39, 40]. The model parameters we use for this material system are taken from [36]. All simulations are carried out at 700 C, a typical heat treatment temperature for IN718 for precipitation hardening. We simulate 7h of evolution of 1920 γ′′\gamma^{\prime\prime} precipitates distributed over the phase’s three orientation variants in a 576 nm ×\times 576 nm ×\times 576 nm domain (a 115231152^3 simulation grid). Snapshots of the simulated evolution are given in Figure 1. Starting with small nuclei in a supersaturated matrix at t=0t = 0, the system is in a growth mode until approximately t=3t = 3 h, when the volume fraction of precipitates stabilizes. From t=3t = 3 h to the end of the simulation (t=7t = 7 h), the system undergoes coarsening, with some precipitates growing while others dissolve.

Snapshots showing the evolution of $\gamma^{\prime\prime}$ precipitates at 0 h, 1 h, 3 h, and 7 h

Figure 1. Snapshots of the evolution of γ′′\gamma^{\prime\prime} precipitates in a Ni-Fe-Nb alloys over the course of seven simulated hours of aging at 700 C. The color indicates each of the three orientation variants of γ′′\gamma^{\prime\prime} found in this system. At the start of the simulation 1920 nuclei are present split between the orientation variants. These nuclei grow for approximately the first 3 h of aging and then the system transitions into coarsening.

With over 1.5 billion grid points, this simulation is among the largest phase-field simulations to date [5, 33, 41] and, with the approaches described in this article, was completed in less than 5.5 hours of wall time on 1024 GPUs (128 nodes on the Frontier supercomputer at the Oak Ridge Leadership Computing Facility). This simulation runs faster than the associated experiment, which opens up the potential for performing first-of-a-kind experiments (e.g., at x-ray or neutron sources) with simulations running alongside the experiment to assist in interpreting results and/or to steer the experiment.

Using the analysis in the following sections, we estimate a transformational decrease in the time-to-solution from months to hours in the large-scale growth and coarsening simulation in Figure 1 from combining GPU acceleration (§ 2.1) with high-order time integration (§ 2.3) and scaling to a large number of nodes (§ 2.2) compared to a baseline scenario using a first-order time integration scheme and limited, CPU-only computing hardware. The methods that underpin these three acceleration approaches are discussed in § 4.

We estimate the overall speedup using multiplicative factors for each of the three acceleration approaches. For GPU acceleration, we estimate a speedup of 7.6×7.6 \times to 17.5×17.5 \times based on the single node CPU-GPU comparisons on Frontier in § 2.1. For high-order time integration, we estimate a speedup of 2.6×2.6 \times relative to the first-order scheme with Δt=0.125\Delta t = 0.125s (scaling the 2.9×2.9 \times value from § 2.3 given a decrease in Δt\Delta t from 2.252.25s to 2.02.0s with the high-order method). Combining these two factors, we expect a CPU configuration using 128 Frontier nodes (8192 CPU cores) with the first-order scheme to be 19.8×19.8 \times to 45.5×45.5 \times slower (4.5 to 11 days to complete). As discussed in the introduction, 8192 CPU cores is far more than have typically been used for phase-field simulations of precipitation. Therefore, we choose 512 CPU cores (eight Frontier nodes) as our baseline level, which is within the common range for phase-field simulations of precipitation in the literature. The baseline choice is also motivated by the memory requirements of simulations on a 1152×1152×11521152 \times1152 \times1152 grid, running on substantially fewer CPU cores may not be possible. In this more limited setting with eight Frontier nodes, we estimate that multi-node scaling provides an additional 11×11 \times speedup (equal to the speedup observed in § 2.2 going from 8 to 128 nodes). Therefore, multiplying the three parts of the estimated speedup yields a 217×217 \times to 501×501 \times speedup for the demonstration simulation configuration over the baseline scenario. As such, a simulation like the one presented in Figure 1 would take between 50 and 115 days rather than 5.5 hours using the methods presented in this article.

In the following sections we discuss the three components of our acceleration approach in detail and provide performance benchmarks for each component that yield the per-approach acceleration factors used in the preceding analysis.

Improved performance via GPU acceleration

The first of the three components of our acceleration strategy is improved performance through GPU acceleration. Our GPU acceleration strategy includes acceleration of three aspects of the problem: the use of performance portable data structures and parallel execution patterns (described in § 4.3), GPU-accelerated FFTs (described in § 4.5), and GPU-accelerated time integration (described in § 4.4). Using these

approaches the data remains resident on the GPU throughout the simulation, eliminating the costly transfers between the GPU and CPU that bottleneck performance. Aside from some one-off processes during initialization, the only time field data is transferred from the GPU to the CPU is when writing outputs to file for visualization.

To demonstrate the performance increase for this fully GPU-resident approach, we perform single-node benchmark simulations using CPU-only and GPU-accelerated simulations on the OLCF Summit and Frontier supercomputers (refer to § 4.7 for details of the computing systems). The key results of this investigation are presented in Table 1 and Table 2 and Figure 2. Here all performance tests use the full compute capabilities of the node, comparing all of the CPU cores to all of the GPUs. This type of full-node comparison gives performance information relevant to the practical speedup of using a GPU-accelerated node versus a CPU-only node.

Node typeDomainTotal (s)Per point (s)MPI (s, %)GPU Speedup
CPU1683168^{3}2615.50×10−55.50 \times10^{-5}19.8 (7.6%)-
GPU1683168^{3}55.41.17×10−51.17 \times10^{-5}11.3 (20%)4.7×
CPU3363336^{3}21955.79×10−55.79 \times10^{-5}133 (6.0%)-
GPU3363336^{3}2366.22×10−66.22 \times10^{-6}66.4 (28%)9.3×
CPU5043504^{3}51083.99×10−53.99 \times10^{-5}284 (5.6%)-
GPU5043504^{3}----

Table 1. Single-node CPU and GPU timings on Summit (44-core IBM Power9 CPU, NVIDIA V100 GPUs). No GPU performance data is available for the 5043504^{3} domain due to the memory limitations of the V100 GPUs.

Node typeDomainTotal (s)Per point (s)MPI (s, %)GPU Speedup
CPU1683168^{3}1473.09×10−53.09 \times10^{-5}20.7 (14%)-
GPU1683168^{3}19.24.04×10−64.04 \times10^{-6}2.65 (14%)7.6×
CPU3363336^{3}15484.08×10−54.08 \times10^{-5}247 (16%)-
GPU3363336^{3}88.32.33×10−62.33 \times10^{-6}23.4 (27%)17.5×
CPU5043504^{3}51174.00×10−54.00 \times10^{-5}641 (13%)-
GPU5043504^{3}3022.36×10−62.36 \times10^{-6}79.0 (26%)16.9×

Table 2. Single-node CPU and GPU performance on Frontier (64-core AMD EPYC CPU, AMD MI250X GPUs).

Single-node profiling results for the time spent in FFT library calls, Kokkos loops, and other operations for CPU-only and GPU calculations on the OLCF Summit and Frontier supercomputers

Figure 2. Single-node profiling results for the time spent in FFT library calls, Kokkos loops, and other operations for CPU-only and GPU calculations on the OLCF Summit and Frontier supercomputers.

The benchmark problem for this section is a version of the γ′′\gamma^{\prime\prime} precipitation simulation presented above, but for three smaller grid sizes: 1683168^{3} (a 84 nm cube), 3363336^{3} (a 168 nm cube), and 5043504^{3} (a 252 nm cube). Multiples of 168 grid points per direction were chosen to permit efficient MPI domain decompositions over 6, 8, 42, and 64 MPI tasks (the number of full-node MPI tasks for Summit GPUs, Frontier GPUs, Summit CPUs, and Frontier CPUs, respectively). The 3363336^{3} domain is the largest domain that will fit in Summit GPU memory for a single node, and the 5043504^{3} domain is the largest domain that will fit in Frontier GPU memory for a single node. On Summit, the CPU calculations range from 112,896 to 3,048,192 points per MPI rank and the GPU calculations range from 790,272 to 21,337,344 points per MPI rank. On Frontier, the CPU calculations range from 74,088 to 2,000,376 points per MPI rank and the GPU calculations range from 592,704 to 16,003,008 points per MPI rank. Benchmark simulations in all three domain sizes had a consistent number density of precipitates (6 in the 1683168^{3} domain, 12 in the 3363336^{3} domain, and 18 in the 5043504^{3} domain), with pseudorandom positions and pseudorandom orientation variant assignment. Each benchmark test is conducted for 120 simulated seconds, using third-order ImEx time integration (see § 2.3) and a fixed time step size of 1.25 s. Given the fixed number of grid points, fixed time step size, and consistent number of iterations for the nonlinear mechanics solver, preliminary tests indicated that 120 second simulations had per-time-step timings representative of longer simulations such as the simulation in Figure 1.

Table 1 shows the single-node performance on the Summit supercomputer, where each node has 42 usable IBM Power9 CPU cores (with 2 cores reserved for system processes) and 6 NVIDIA V100 GPUs. On the CPUs, the total wall time increases as the domain size increases, as expected. The average time-per-grid-point is approximately flat at 55.0–57.9 ms for the two smaller domain sizes and then decreases slightly to 39.9 ms for the largest domain size. This decrease may be due to the increased work per rank in the compute-heavy portions of the calculation hiding overheads elsewhere. On the GPUs, the total wall time (and accordingly the average per-point times) are much faster than on CPUs, yielding a 4.7×\times speedup for the 1683168^{3} domain and a 9.3×\times speedup for the 3363336^{3} domain. The time per-grid-point decreases substantially from the 1683168^{3} domain to the 3363336^{3} domain. This observation is consistent with previous findings that GPU performance degrades with low thread occupancy. The fraction of total wall time spent on MPI communication decreases slightly with increasing system size on CPU, likely due to a more favorable ratio of volume (work) to surface area (communication) scaling. On CPUs MPI communication costs are low, always under 8% of total wall time. The total time spent on MPI communication for the GPU calculations is approximately half of the time spent for the CPU calculations. Possible reasons for communication being faster on GPUs could be the faster memory speeds on the GPUs and/or the reduced number of messages (6 MPI ranks on GPUs versus 42 ranks on CPUs). The decrease in MPI time is less than the overall GPU speedup, so the fraction of time spent on communication increases relative to the CPU-only tests. Different from the CPU-only tests, the fraction of time spent on MPI communication increases from the 1683168^{3} domain to the 3363336^{3} domain. Overall, these tests demonstrate a significant advantage to using the GPUs on Summit for an appropriately sized problem, with more than a 9× speedup using all 6 of the V100 GPUs on the node compared to solely the 42 CPU cores.

Table 2 shows the single-node performance on the Frontier supercomputer, where each node has 64 NVIDIA EPYC CPU cores and 8 AMD MI250X GPUs. The larger memory of the MI250X compared to the V100 allows all three domain sizes to be tested on the Frontier GPUs. On the CPUs, the calculations on Frontier were slightly faster than those on Summit, requiring 30.9-40.8 ms per grid point. On the GPUs, the per-grid-point time is 4.044.04 ms for the 1683168^{3} domain, and it decreases to about 2.32.3 ms for the larger domains. This same trend is observed on Summit where the per-grid-point time improves with higher thread occupancy. The per-grid-point performance then saturates at the 3363336^{3} domain, suggesting that the GPU is at full occupancy. The GPU speedup on Frontier is even larger than observed for Summit – 7.6×7.6\times for the smallest domain and about 17×17\times for the larger domains. In absolute terms, the time spent on MPI communication is similar or higher for CPU calculations on Frontier compared to Summit and lower for GPU calculations on Frontier compared to Summit. Frontier’s improved MPI communication on GPUs compared to Summit may be due to a difference in the node architectures; Frontier has direct connections between its GPUs, while Summit has direct connections between groups of three GPUs which are in turn connected via the CPU. The reduction in MPI communication time is key to the improved GPU speedup on Frontier compared to Summit. For example, if the Frontier GPU calculation for the 3363336^{3} domain had the same 66.466.4 s MPI time as the Summit calculation, the GPU speedup would only be 11.8×11.8\times rather than 17.5×17.5\times. Overall, these tests demonstrate an even larger benefit to using the GPUs on Frontier than on Summit for an appropriately sized problem, with more than a 17×17\times speedup using all 8 of the MI250X GPUs on the node compared to solely the 64 CPU cores.

Comparing the total wall time for GPU calculations on Summit and Frontier, the Frontier calculations were 2.9×2.9\times faster and 2.7×2.7\times faster than the Summit calculations for the 1683168^{3} and 3363336^{3} domains, respectively. The improved performance on the Frontier nodes is expected due to the higher nominal peak performance of the Frontier nodes (191.2 double-precision TFLOPS) versus the Summit nodes (46.8 double-precision TFLOPS) and the higher nominal memory bandwidth on Frontier (1.6 TB/s) versus Summit (900 GB/s). While the observed speedup of 2.72.7-2.9×2.9\times is less than the 4.1×4.1\times increase in nominal peak FLOPS, it is larger than the 1.8×1.8\times increase in memory bandwidth. MEUMAPPS C++ performance is in line with previous comparisons of Summit and Frontier showing a speedup of 1.71.7-6×6\times for applications co-developed with supercomputer facility staff [42].

Figure 2 gives a breakdown of the timings for groups of operations in the main loop for the 3363336^{3} domain calculations. Profiling data identifies the time spent on fast Fourier transform (FFT) calls to the heFFTe library and on loops using the Kokkos library. Together these two groups of operations reflect 65-93% of the total wall time of the main loop, with roughly an equal split between them. GPU speedups are observed on both Summit and Frontier nodes for FFT calls, Kokkos loops, and for the remaining (“other”) operations. On the Summit node, FFT calls have a 9.6×9.6\times speedup, Kokkos loops have a 17.6×17.6\times speedup, and the other operations have a 1.8×1.8\times speedup. On the Frontier node, FFT calls have a 21.5×21.5\times speedup, Kokkos loops have a 15.2×15.2\times speedup, and the other operations have a 14.3×14.3\times speedup. These results demonstrate that the overall GPU speedup observed in Tables 1 and 2 is achieved by GPU speedups in both the FFTs and the Kokkos loops.

The only case where substantial GPU speedups were not observed was for the “other” category on the Summit node. This poor speedup of 1.8×1.8 \times was observed consistently during testing and was not a spurious result from a single test. Of the 77 s spent in this category, the majority of the time has no specific entry associated with it (e.g., output, Kokkos deep copies) and region profiles suggest that this time is distributed across multiple functions. With the much larger GPU speedups for the FFT and Kokkos loops, the time spent on other operations increases from 6% to 35% of the time in the main loop, causing a meaningful impact on overall performance. Identification of the specific operations driving poor GPU speedup of the “other” category is left for future work. The remaining test cases in this article focus on the newer, higher performing Frontier supercomputer, where the “other” category exhibits similar GPU speedups to FFTs and Kokkos loops.

In summary, the single node tests demonstrate that the performance portability strategies in MEUMAPPS C++ are effective in generating large GPU speedups (4.74.7-17.5×17.5 \times) on both NVIDIA and AMD GPUs. As expected, the GPU speedup improves as the domain size, and therefore GPU occupancy, increases. The observed GPU speedup is due to GPU speedups of both FFT calls to the heFFTe library and for loops using the Kokkos library.

Improved performance via HPC scaling

Now we turn from single-node performance to scaling across multiple nodes, the second component of our acceleration strategy. Scaling to multiple nodes requires distributed memory parallelism, with explicit communication between each computational node. Minimizing the time the simulation spends waiting for data from another node is the driver for efficient multi-node scaling. For the Fourier pseudospectral method that we employ, nearly all of the distributed memory communication occurs during FFT operations, and therefore our interface to a scalable FFT library (heFFTe, as described in Sec. 4.5) is the key enabler for reducing simulation times with multi-node simulations.

The benchmark problem setting is the same as for the single-node tests, the growth of pseudorandomly placed γ′′\gamma^{\prime\prime} particles with fixed initial number density for multiple domain sizes. We focus our benchmarking on the newer Frontier supercomputer, with scaling behavior presented in Figure 3. Three domain sizes are considered, testing up to 12.2 billion grid points on 8192 GPUs. The smallest domain, 5763576^{3}, is close to the limit of what will fit into GPU memory on one Frontier node. The other two sizes, 115231152^{3} and 230432304^{3}, are factors of two and four larger in each dimension (8 and 64 times more total grid points). A combined strong and weak scaling test [32] was performed with these three domain sizes, and the results are shown in Figure 3. In this combined test, the lowest number of GPUs for each domain size is chosen to follow a weak scaling test (where the number of GPUs increases proportionally with the total number of grid points), and then a strong scaling test (increasing GPUs counts for a fixed total number of grid points) is performed for each domain size, with successively doubling of the GPU count until the time to solution increases. With this test matrix, corresponding places along the strong scaling tests for each domain size constitute weak scaling tests (i.e., moving left to right in Figure 3, the first data point from each domain size form a weak-scaling test with the highest occupancy, the second data point from each domain size form a weak-scaling test with the second highest occupancy, etc.). Tests on fewer GPUs for the 115231152^3 and 230432304^3 domain sizes are not possible due to the GPU memory limitations.

Combined strong and weak scaling plot for MEUMAPPS C++ on the OLCF Frontier supercomputer

Figure 3. Combined strong and weak scaling plot for MEUMAPPS C++ on the OLCF Frontier supercomputer.

Starting with the strong scaling results, Figure 3 demonstrates efficient strong-scaling to thousands of GPUs. For the 5763576^3 domain, the test starts on 8 GPUs (one Frontier node) and maintains a parallel efficiency above 70% up through 32 GPUs. From 64 through 256 GPUs the scaling is sub-ideal (35-53% parallel efficiency), but the time-to-solution continues to steadily decrease. At 512 GPUs the time-to-solution is barely faster than at 256 GPUs, and at 1024 GPUs overhead from distributing the computational finally increases enough that the time-to-solution increases. For the 115231152^3 domain, the test starts on 64 GPUs and parallel efficiency above 70% is maintained up through 1024 GPUs. At 2048 GPUs, the scaling abruptly worsens and the time-to-solution increases. For the 230432304^3 domain, the test starts on 512 GPUs and parallel efficiency above 70% is maintained up through 4096 GPUs. At 8192 GPUs, the scaling abruptly worsens and the time-to-solution increases.

For the weak scaling results, Figure 3 shows consistent deviation from ideal scaling. The relevant weak scaling tests are the first four with the highest occupancy, where strong scaling is near-ideal for all three domain sizes. For the weak scaling test at the highest occupancy (23,887,872 grid points per GPU), the time to solution increases from 568s to 1077s to 1379s for the 5763576^3, 115231152^3, and 230432304^3 domain sizes, respectively, corresponding to parallel efficiencies of 53% to 41%. At the second highest occupancy (11,943,936 grid points per GPU), the time to solution increases from 326 s to 643 s to 793 s (parallel efficiencies of 51% and 41%). At the third highest occupancy (5,971,968 grid points per GPU), the time to solution increases from 206 s to 322 s to 381 s (parallel efficiencies of 64% and 54%). At the fourth highest occupancy (2,985,984 grid points per GPU), the time to solution increases from 133 s to 168 s to 228 s (parallel efficiencies of 79% and 59%).

The scaling tests indicate strengths and limits of the MEUMAPPS C++ framework for use on GPU supercomputers. For a large enough problem size (230432304^3 in our tests), MEUMAPPS C++ can efficiently use at least 4096 GPUs. The per-GPU problem size where strong-scaling worsens is in the range of 1,492,992 to 2,985,984 grid points per GPU. Generally, a 6-11× reduction in the time to solution was attained by adding GPUs beyond the minimum required for the problem to fit in GPU memory. The parallel performance for weak scaling is in the 40-80% range. The 3D FFT algorithm is known to have non-ideal weak scaling, going as O(N3log⁡(N))\mathcal{O}(N^3 \log(N)) [43]. However, for the problem sizes examined here, the theoretical non-ideality from the log⁡(N)\log(N) would only reduce the parallel efficiency to ~80%. Benchmarks from the heFFTe library on the Summit supercomputer are consistent with those theoretical expectations with 80% parallel efficiency for 11.2 million grid points per GPU [16]. The decreased weak scaling performance in MEUMAPPS C++ may be due to increased GPU register pressure resulting from the many fields being stored in GPU memory simultaneously and is an area for future optimization within the code.

Improved performance via time integration approach

Finally, we turn to the third component of our acceleration strategy, high-order ImEx time integration. The computational cost of a Fourier pseudospectral method is approximately proportional to the number of right-hand side (RHS) evaluations, with each time step during the simulation involving one or more RHS evaluations depending on the time integration method and order. The total number of RHS evaluations can be decreased either by reducing the number of time steps (i.e. taking larger time steps) or reducing the number RHS evaluations per time step (i.e. lowering the order of the method). Using additive Runge-Kutta methods (described in § 4.4 and implemented in the SUNDIALS library), we find that for physically relevant error levels that the simulations are faster with large, high-order time steps than the common [12, 14, 19, 20, 21, 22] approach of small, first-order time steps.

To demonstrate the performance differences across time-step size and method order, we perform single-node benchmark simulations using GPU-accelerated nodes on the Frontier supercomputer. The benchmark problem for this section involves two growing precipitates for a single γ′′\gamma^{\prime\prime} orientation variant, the smallest, simplest scenario of precipitate evolution that is more complex than monotonic growth of a single precipitate.

This test is performed in a cubic computational domain with 32 nm in each dimension and a 0.5 nm grid spacing (i.e., 64364^3 domain). One precipitate seed with a 2.5 nm radius is placed at (10 nm, 15.5 nm, 15.5 nm) and a second seed with a 3 nm radius is placed at (20.5 nm, 20.5 nm, 20.5 nm). 1500 s of heat treatment was simulated, enough to see the particles complete the growth phase of evolution (where the precipitates near the equilibrium volume fraction) and begin coarsening (where the larger particle grows at the expense of the smaller particle). The small size and short length of this test case makes reference simulations with a very small time step computationally tractable.

Using the two-particle test case we examine the error convergence and computational performance of our time integration strategy using methods ranging from first order to fourth order. The key results of this investigation are presented in Figure 4.

Results evaluating different time integration methods/configurations with the two-particle test case: (a) Error convergence; (b) Method efficiency

Figure 4. Results evaluating different time integration methods/configurations with the two-particle test case. a) A plot of the L2L_2 error for various fixed time step sizes using tight elasticity tolerances. The markers denote the results of the numerical experiments and the dashed lines denote the expected reduction of error given the order of accuracy of the method. b) A plot of the time-to-solution versus the L2L_2 error for different time integrator orders with fixed or adaptive step sizes and a loose or tight elasticity solver tolerance. The blue region denotes simulation results with L2L_2 error below the physically relevant level.

fig:4a shows the convergence of the L2L_2 error of the simulations with decreasing time step size (Δt\Delta t). The error is calculated as the L2L_2 norm of the difference between the solution and a reference solution at the end of the simulation (t=1500t = 1500 s). The reference solution is from a highly resolved simulation using the third order accurate method with Δt=0.25\Delta t = 0.25 s. All simulations in fig:4a were carried out with a tight elasticity solver tolerance (ϵel=10−16\epsilon_{\mathrm{el}} = 10^{-16}) such that the differences between the test results and the reference solution is dominated by temporal discretization error. For the first, second, and third order methods the expected order of accuracy is clearly observed. With the fourth order method, we observe fourth order convergence between the three largest time steps considered (Δt=1.875\Delta t = 1.875 s, 2.0 s, 2.25 s). However, this trend does not hold for the remaining three time step sizes (Δt=1.0\Delta t = 1.0 s, 1.25 s, 1.5 s) where the observed order of accuracy falls between second and third order. A degradation in the observed order of accuracy could in principle be caused by an inadequate reference solution, where the difference between test solutions and the reference is dominated by temporal discretization error in the reference rather than the test solution, but is unlikely to be the case here. Assuming the observed third order convergence continues to hold for the nominally third order method, extrapolating from the error at Δt=1.875\Delta t = 1.875 s the expected L2L_2 error for the reference solution is ∼10−5\sim10^{-5}, 40×\times smaller than the measured error for the smallest time step size for the nominally fourth-order method and 120×\times larger than the measured error for the time step size where the nominally fourth-order method deviates from the fourth-order trend line. Instead, we believe the degradation in the observed convergence rate is due to order reduction when integrating stiff systems [44, 45, 46].

In addition to the convergence tests, as a secondary check, we manually implemented the first-order ImEx Euler method, utilized by other phase-field models [11, 12, 14, 19, 20, 21, 22], into MUMPS C++ and compared the results to those obtained using the SUNDIALS library implementation of the same method (i.e., those shown in Figure 4a). The pointwise absolute error between the two solutions is close to floating-point precision, less than 10−1610^{-16}. Together, the convergence tests and the pointwise error verification provide strong evidence that the high-order time integration methods proposed here

behave as expected and are correctly implemented in MEUMAPPS C++.

The approximate maximum time step sizes reported in Figure 4a vary significantly with the order of accuracy of the integration method and were found to be 0.5 s for first and second order methods, 1.5 s for the third order method, and 2.25 s for the fourth order method. These values were found empirically by increasing the step size in increments of 0.25 s until the simulation was unable to successfully reach the final time of 1500 s. As discussed in § 4, the ImEx schemes are limited by the stability of the explicit portion of the method. In general, the higher order ImEx methods have a larger explicit linear stability region and can allow for larger time step sizes (at additional computational expense).

Figure 4b gives the time-to-solution for simulations to reach the end of the two particle test case (t=1500t = 1500 s) versus the L2L_2 error (as defined for Figure 4a) on one node of the OLCF Frontier supercomputer using all eight of the GPUs available. Time-to-solution comparisons between the manually implemented first-order method and the SUNDIALS implementation of the same method yielded run times that differed by less than the run-to-run variation observed (∼7%). Thus, we incur no appreciable overhead utilizing the SUNDIALS library and only report results from the more flexible SUNDIALS implementations for the remainder of this article. The work-precision diagram (Figure 4b) enables us to determine the most efficient approach for a particular level of accuracy. The level of error is a key criterion for scientific investigations using phase-field simulations; a faster time-to-solution may not be preferred if it causes an increase in error. However, this preference for lower error is not absolute; below a certain level of error, the changes in the solution cease to be useful for scientific insight or decision making. In Figure 4b the region below the physically useful error is shown in blue. Correspondingly, the data points in the white region in the figure may have time integration error that is too large for the simulation results to be useful. The physically useful error is inherently application-dependent and subjective. The criterion we choose for our analysis is whether the interface contour (i.e., the 0.5 level set of the order parameter field) in a slice down the center of the larger of the two particles is clearly distinguishable from the contour for the highly resolved reference solution when viewing the full extent of the particle. This criterion corresponds to convergence of a key quantity of interest: the shape of the simulated particles. Simulations using the first order method reach this criterion between the Δt=0.25\Delta t = 0.25 s and Δt=0.125\Delta t = 0.125 s. Interpolating between these points, we place the upper bound on the physically useful error at 0.40 for a first-order simulation with Δt=0.35\Delta t = 0.35 s.

For simulations using the tight elastic solver tolerance, as used for the temporal convergence tests, the time-to-solution increases approximately linearly in the log-log plot with decreasing error. For a similar time-to-solution the first order simulations have much higher error than the higher-order simulations. The higher order simulations are bunched in a band, with only minor variation in the time-to-solution for similar levels of error. Note that even the data points with the highest error for the second through fourth order methods are well below the physically meaningful level. While one may expect that a shorter time-to-solution could be achieved with these methods by using larger time step sizes to obtain solutions with larger errors still below the useful threshold, the maximum time step possible is limited by the stability of the explicit terms in the ImEx splitting. Since all second- through fourth-order simulations have adequate accuracy, the best performing simulation is simply the one with the shortest run time. Therefore, for the tight elastic solver tolerance, the best performing simulation is the third-order simulation with Δt=1.5 s\Delta t = 1.5\ \mathrm{s} with a time-to-solution of 800 s. This time-to-solution is approximately 2.7×2.7 \times faster than the only first-order result with error below the physically useful level (Δt=0.125 s\Delta t = 0.125\ \mathrm{s} and a time-to-solution of 2192 s) and is 1.5×1.5 \times faster than the next first-order result (Δt=0.25 s\Delta t = 0.25\ \mathrm{s} and a time-to-solution of 1197 s) which has an unacceptable level of error.

While these simulations with a tight elasticity solver tolerance are appropriate for calculating the temporal discretization error, a looser tolerance reduces the time-to-solution significantly without adding physically relevant error. Figure 4b also includes simulation results with a loose elasticity solver tolerance (ϵel=10−6\epsilon_{\mathrm{el}} = 10^{-6}). With this tolerance, the elasticity solver exits after one iteration for nearly all time steps; further loosening of the tolerance does not lead to appreciable reduction in time-to-solution. With the change to the loose tolerance, the time-to-solution drops by about an order of magnitude with the overall range of times-to-solution going from between 646 s and 1415 s down to between 63 s and 167 s. The additional error introduced by the elasticity solve has negligible effect on the first order results due to their already large error. For the higher order simulation results, the looser tolerance increases the L2L_2 error and compresses the range of errors, but the results are still well below the acceptable error. With the loose tolerance, the best performing simulation is the fourth order simulation with Δt=2.25 s\Delta t = 2.25\ \mathrm{s} with a time-to-solution of 87 s. This time-to-solution is approximately 2.9×2.9 \times faster than the only first-order result with error below the physically useful level (Δt=0.125 s\Delta t = 0.125\ \mathrm{s} and a time-to-solution of 250 s) and is 1.4×1.4 \times faster than the next first-order result (Δt=0.25 s\Delta t = 0.25\ \mathrm{s} and a time-to-solution of 125 s) which has an unacceptable level of error.

In Figure 4b, we also present simulation results using the fourth-order integrator with adaptive time step sizes. The tolerances for adaptivity are applied pointwise to the change in the (unitless) order parameter field. Three values of the absolute tolerance were chosen: 2×10−92 \times10^{-9}, 5×10−95 \times10^{-9}, and 1×10−81 \times10^{-8}. A relative tolerance of 1×10−91 \times10^{-9} was used for all tests, with the result being insensitive to its value. Of these simulations, the 5×10−95 \times10^{-9} absolute tolerance value resulted in the fastest time-to-solution, 87 s, the same time-to-solution as the fastest fixed time step simulation, albeit with slightly reduced L2L_2 error. The simulations with adaptive time steps have the same stability limitations as the fixed time step simulations. Here, the error increases rapidly above Δt=2.25 s\Delta t = 2.25\ \mathrm{s} due to instability, and the time step size controller keeps the time step sizes at or below this level. Although adaptive time stepping does not improve time-to-solution, these findings point to a robustness benefit – the adaptive scheme automatically finds the appropriate time step without manual trial-and-error with different fixed values of Δt\Delta t.

The time-to-solution for these simulations is dominated by the number of right-hand side (RHS) evaluations. The total number of RHS evaluations is the product of the number of RHS evaluations per time step and the number of time steps. Because of the choice of ImEx splitting (see § 4) the explicit and implicit RHS functions are evaluated the same number of times each step as RHS evaluations are not necessary as part of the implicit stage solves. The number of RHS evaluations per time step is as follows: one for the first-order method, three for the second-order method, four for the third-order method, and six for the fourth-order method. Examining the results with the loose elasticity tolerance, the fourth-order method with the highest stable time step size (six RHS evaluations per time step, 667 time steps) has three times fewer total RHS evaluations than the first-order simulation at the upper bound of the physically relevant error (one RHS evaluation per time step, 12,000 time steps), almost exactly matching the observed 2.9× speedup in time-to-solution. This relationship is somewhat weaker for the tight elastic tolerance, where the variable number of iterations in the nonlinear elasticity solve cause variation in the time per RHS evaluation (e.g., the speedup between the fourth-order, Δt=2.25\Delta t = 2.25 s simulation compared to the first-order, Δt=0.125\Delta t = 0.125 s simulation is only 2.7×).

In summary, using an ImEx-ARK time integration approach, we demonstrated expected convergence behavior for first- through fourth-order schemes and a 2.9× speedup under scientifically relevant simulation conditions using a fourth-order scheme compared to the widely used first order scheme. The second- and third-order order schemes also provided a speedup relative to the first-order scheme. All stable simulations using the second- through fourth- order schemes exhibited L2L_2 error well below the physically relevant level. Therefore, in addition to reduced time-to-solution compared to the first-order simulations (on the edge of the physically relevant error), these results suggest that any simulation with the second- through fourth- order schemes that are stable, also have acceptable error. Given these findings, the fourth-order scheme was used for the other studies in this article.

Limitations and future work

While the speedup demonstrated in the previous section is already paradigm shifting, further improvements are likely possible. The time integrator study in Sec. 2.3 and the large-scale growth and coarsening simulation highlight that the higher order ImEx methods enable larger time step sizes and produce errors well below the physically relevant level. However, the maximum step size is still limited by stability not the solution error. This suggests alternative partitionings of the right-hand side (e.g., treating some of the nonlinear terms implicitly) may improve stability and allow the use of larger time step sizes while still having error below the physically relevant level. The challenge with such an approach will be ensuring an efficient nonlinear solve for the implicit stages to ensure gains in step size are not offset by increased solver costs. Second, the fourth-order time integration scheme exhibited order-reduction as the time step size decreased. When tighter error control is required, developing ImEx methods with higher weak stage order (building on the work in [47, 48, 49] to alleviate order reduction with explicit and implicit methods) may yield improved convergence.

The ability to take larger time steps near the physically relevant error level would increase the likelihood of performance gains from adaptive time integration. A preliminary test using adaptive time integration for the same simulation conditions in Figure 1 revealed similar results to what was seen in § 2.3 – a slightly larger time step size (Δt=2.28s\Delta t = 2.28\mathrm{s} vs. 2s2\mathrm{s}) but a slightly longer time-to-solution (5.6h5.6\mathrm{h} vs. 5.5h5.5\mathrm{h}) due to overheads associated with adaptive time integration.

Two improvements to node-level performance may be worth investigation. First, the performance of the Kokkos MDRangePolicy construct is known to be sensitive to tiling [50]. Autotuning the tiling parameter may result in increased performance for time spent in Kokkos loops. We did not investigate this type of optimization in this article because it necessarily depends on the grid size and computational resources – instead our focus is on general acceleration strategies. Second, we have not implemented Kokkos host-parallel (e.g. OpenMP or C++ Threads) backends and have focused on the more performant GPU backends. Using shared-memory parallelism for CPU-only calculations may provide a speedup compared to our use of intranode MPI communication. However, we expect any speedup to be limited, on the order of the 6-16% of the total time spent in MPI communication in Table 1 and Table 2.

More detailed profiling of scaling tests may identify the portions of the code causing the current scaling limits. Identification of bottlenecks for weak scaling are particularly of interest due to the substantially sub-O(N3log⁡(N))O(N^3\log(N)) scaling. However, at the application level it is unclear what optimizations could change the scaling behavior, since nearly all MPI communication is performed during FFT library calls. If the sub-ideal weak scaling is due to GPU register pressure, as hypothesized in § 2.2, reducing the number of variables stored on the GPU may improve scaling performance (albeit potentially at the expense of recomputing previously stored fields).

With the large simulation domains now accessible with MEUMAPPS C++, visualization becomes a challenge. One of the reasons for performing the demonstration simulation in a 115231152^3 domain rather than a 230432304^3 was due to our estimate that rendering 70 frames for visualizing would take days of computing time with the standard HPC visualization tool VisIt [51]. In-situ visualization (where visualization calculations are performed during a simulation with co-allocated resources), may provide a tractable path forward for large simulations with MEUMAPPS C++ [52].

Although this article focuses on the implementation of the phase field model for solid-state phase transformation developed by Radhakrishnan et al. [14], MEUMAPPS C++ is designed in a modular fashion to allow rapid development of different physics applications. One straightforward extension of the current model is the incorporation of the more complex CALPHAD representation of the homogeneous free energy functions (e.g., using the Thermo4PFM library [53]) rather than the parabolic functions used here. Phase-field models for phenomena other than precipitation can also be implemented with MEUMAPPS C++. For example, the MEUMAPPS C++ repository [54] contains an application to solve the Cahn-Hillard equation for the PFHub benchmark problem [55]. Pseudospectral methods are extensively used for phase-field crystal models [56], making them an attractive set of models to implement with MEUMAPPS C++. Beyond phase-field models, other models often solved using pseudospectral methods such as computational fluid dynamics and crystal plasticity are also candidates for implementation with MEUMAPPS C++.

Conclusion

In this article we demonstrated a holistic approach to accelerate pseudospectral phase-field simulations using a combination of GPU acceleration, distributed memory scaling, and high-order time integration implemented in the MEUMAPPS C++ framework. Substantial GPU acceleration was shown on both NVIDIA and AMD GPUs with a 4.74.7-9.3×9.3 \times speedup observed on NVIDIA GPUs (comparing 6 NVIDIA V100 GPUs to 42 CPU cores) and a 7.67.6-17.5×17.5 \times speedup observed on AMD GPUs (comparing 8 AMD MI250X GPUs to 64 CPU cores). Using performance portability layers in MEUMAPPS C++, these calculations on CPUs and both types of GPUs required no hardware-specific code. Scaling tests showed near-ideal strong scaling up to 4096 GPUs for a 230432304^3 domain and effective use of hundreds of GPUs for smaller domains. Switching from the first-order implicit-explicit scheme typically used to a fourth-order scheme leads to a 2.62.6-2.9×2.9 \times speedup, with error reduced to well below the physically meaningful level. We demonstrated that this holistic approach drastically accelerates an application-relevant simulation that investigates the growth and coarsening of 1920 γ′′\gamma^{\prime\prime} precipitates in a Ni-Nb-Fe alloy. A simulation of a 7h heat treatment of this system was completed in just 5.5 hours. We presented an estimate that this is a 217217-501×501 \times speedup over a baseline approach (a CPU-only calculation using first order time integration running on 512 cores). Therefore, a calculation that would normally take months can be completed in an afternoon.

While high-order time integration, GPU acceleration, and distributed memory scaling individually are not new, we demonstrate how recent advances in these areas lead to transformative performance improvements when combined. Furthermore this demonstration was not for a toy problem; by implementing these approaches in MEUMAPPS C++ we tested these approaches for a full-physics application-relevant simulation of alloy heat treatment. The approach presented in this article makes 3D simulations of thousands of precipitates tractable. Such large-scale simulations are necessary to quantitatively evaluate coarsening behavior, examine long-range interactions (e.g., rafting), to examine the collective behavior of systems with many phases and/or orientation variants, and many other phenomena. Even for smaller-scale problems, the performance improvements presented here enable more advanced uses of phase-field simulations, such as large ensembles of simulations to quantify uncertainty and integration of faster-than-real-time simulations into experiments. The holistic approach to accelerating pseudospectral calculations and the MEUMAPPS C++ implementation are relevant to a wide range of applications, not only the phase-field simulations of precipitation that are the focus of this article. Other uses of phase-field models (e.g., grain evolution and phase-field crystal), fluid dynamics, and crystal plasticity are particularly relevant uses. Across all of these applications, the ability to leverage hundreds or thousands of GPUs will become increasingly important as AI applications motivate the continued proliferation of GPU-accelerated computing computing clusters. The combination of approaches for GPU acceleration, parallel scaling, and time integration presented here give researchers using pseudospectral methods the capability to take full advantage of these computational resources and thereby unlock new insights with their simulations.

Methods

Phase-field model formulation

In this article we use a Kim-Kim-Suzuki (KKS) phase-field model of solid-state phase transformations for our test cases. Specifically, we use the multi-component, multiphase MEUMAPPS-SS model described in Refs. [14, 19]. We refer readers to those references (and the references therein) for a detailed description of the model; here we briefly lay out the model equations. The total free energy is given by

F=∫(fb+fel+fch) dV,(1)\mathcal{F} = \int\left(f_b + f_{el} + f_{ch}\right)\,\mathrm{d}V, \tag*{(1)}

where each component of the free energy density is given by

fb=12∑v[∇ϕv]Tκv[∇ϕv]+∑v∑q>vωvq∣ϕvϕq∣+ωˉ∑qϕq(1−∑vϕv),(2)f_b = \frac{1}{2}\sum_v [\nabla\phi_v]^{T}\kappa_v[\nabla\phi_v] + \sum_v\sum_{q>v}\omega_{vq}|\phi_v\phi_q| + \bar{\omega}\sum_q\phi_q\left(1-\sum_v\phi_v\right), \tag*{(2)}
fel=∑vh(ϕv)felv+[1−∑vh(ϕv)]felmatrix,(3)f_{el} = \sum_v h(\phi_v)f_{el}^{v} + \left[1-\sum_v h(\phi_v)\right]f_{el}^{matrix}, \tag*{(3)}
fch=∑vh(ϕv)fchv+[1−∑vh(ϕv)]fchmatrix,(4)f_{ch} = \sum_v h(\phi_v)f_{ch}^{v} + \left[1-\sum_v h(\phi_v)\right]f_{ch}^{matrix}, \tag*{(4)}

where

h(ϕ)=ϕ3(6ϕ2−15ϕ+10),(5)h(\phi)=\phi^3(6\phi^2-15\phi+10), \tag*{(5)}
felp=12ϵijel,p:Cijklpϵklel,p,(6)f_{\mathrm{el}}^{p} = \frac{1}{2}\epsilon_{ij}^{\mathrm{el},p} : C_{ijkl}^{p} \epsilon_{kl}^{\mathrm{el},p}, \tag*{(6)}
ϵijel,p=ϵˉij+12(∂ui∂xj+∂uj∂xi)−ϵij∗,p,(7)\epsilon_{ij}^{\mathrm{el},p} = \bar{\epsilon}_{ij} + \frac{1}{2}\left(\frac{\partial u_i}{\partial x_j} + \frac{\partial u_j}{\partial x_i}\right) - \epsilon_{ij}^{*,p}, \tag*{(7)}
fchp=Ap+∑c[Ap,c(Xp,c−Bp,c)].(8)f_{\mathrm{ch}}^{p} = A^{p} + \sum_c \left[A^{p,c}\left(X^{p,c} - B^{p,c}\right)\right]. \tag*{(8)}

The index vv refers to precipitate phases and the index pp refers to all phases (i.e., the precipitate phase and the matrix phase). The index cc refers to solute elements. Note, we made slight changes in notation (especially in equation (4)) from [14, 19] for a general number of phases and components. For equations (2) to (7), the following model constants were introduced: κv\kappa_v (gradient energy tensor per phase), ωvq\omega_{vq} (pairwise barrier), ωˉ\bar{\omega} (double-well barrier), ApA^p (homogeneous free energy offset), Ap,cA^{p,c} (homogeneous free energy curvature), Bp,cB^{p,c} (homogeneous free energy reference composition), CijklpC_{ijkl}^{p} (single-phase stiffness tensor), ϵˉ\bar{\epsilon} (mean strain), and ϵij∗,p\epsilon_{ij}^{*,p} (single-phase eigenstrain).

The time evolution equations are derived using the standard variational approach for a KKS model, assuming conservation of the composition fields and no conservation of the order parameter fields. For the composition fields, XcX^c, this yields

∂Xc∂t=∇⋅(Mc∇∂fch∂Xc),(9)\frac{\partial X^c}{\partial t} = \nabla\cdot\left(M^c \nabla\frac{\partial f_{\mathrm{ch}}}{\partial X^c}\right), \tag*{(9)}

where

Xc=∑vh(ϕv)Xv,c+[1−∑vh(ϕv)]Xmatrix,c,(10)X^c = \sum_v h(\phi_v)X^{v,c} + \left[1 - \sum_v h(\phi_v)\right]X^{\mathrm{matrix},c}, \tag*{(10)}

and McM^c is the mobility for element cc. As is standard in KKS models, the Xp,cX^{p,c} are constrained such that the chemical potentials of an element are equal in all phases at the interface,

∂fp∂Xp,c=∂fp′∂Xp′,c.(11)\frac{\partial f^p}{\partial X^{p,c}} = \frac{\partial f^{p'}}{\partial X^{p',c}}. \tag*{(11)}

For the time evolution of the order parameter fields, ϕv\phi_v, we make a simplifying assumption from Refs. [14, 19]. In those references the MEUMAPPS-SS model is derived in a multi-phase-field context, where ∑pϕp=1\sum_p \phi_p = 1, which results in an expression that depends on the pointwise number of non-zero fields [57]. Such a formulation is prone to numerical artifacts as ϕv\phi_v values cross the threshold for being “non-zero”. Therefore, in this work, we use the simpler multi-order-parameter expression, which has no assumption on ∑pϕp\sum_p \phi_p [6, 58], given by

∂ϕv∂t=−LδFδϕv,(12)\frac{\partial\phi_v}{\partial t} = -L \frac{\delta\mathcal{F}}{\delta\phi_v}, \tag*{(12)}

where LL is the interface mobility. The previous use of the multi-phase-field expression was motivated by the desire to simulate overlapping nuclei for variants in Ti-6Al-4V to examine competition between orientation variants [14] and the ∑pϕp=1\sum_p \phi_p = 1 constraint was important to maintain sensical interpolated values in these overlap regions. For the test cases in this article, with the more typical situation of well-separated precipitates that never overlap, the multi-order-parameter expression is sufficient (and should lead to near-identical results as the multi-phase-field expression).

The final governing equation for the displacement field, which is assumed to be a quasi-equilibrium at all times [13], is given by

∇⋅{Cijkl[ϵˉij+12(∂ui∂xj+∂uj∂xi)−(∑vh(ϕv)ϵij∗,p)]}=0,(13)\nabla\cdot\left\{ C_{ijkl} \left[ \bar{\epsilon}_{ij} + \frac{1}{2}\left(\frac{\partial u_i}{\partial x_j} + \frac{\partial u_j}{\partial x_i}\right) - \left(\sum_v h(\phi_v)\epsilon_{ij}^{*,p}\right) \right] \right\} = 0, \tag*{(13)}

where

Cijkl={∑vh(ϕv)[Cijklv]−1+[1−∑vh(ϕv)][Cijklmatrix]−1}−1.(14)C_{ijkl} = \left\{ \sum_v h(\phi_v)\left[C_{ijkl}^{v}\right]^{-1} + \left[1-\sum_v h(\phi_v)\right]\left[C_{ijkl}^{\mathrm{matrix}}\right]^{-1} \right\}^{-1}. \tag*{(14)}

In summary, the governing equations for the MEUMAPPS-SS model used in this article are given by (9), (12), and (13).

MEUMAPPS C++ and the MEUMAPPS-SS application

In this article, we implement the model described in the preceding section in the MEUMAPPS-SS application using the MEUMAPPS C++ framework for solving systems of PDEs with the pseudospectral method and the numerical and computational approaches laid out in this article. A preliminary version of MEUMAPPS C++ was described in Ref. [30]. MEUMAPPS C++ and its MEUMAPPS-SS application generalize and extend the original MEUMAPPS-SS code [14, 59], which was written in Fortran and was used in Refs. [14, 19, 36]. MEUMAPPS C++ was restructured to be similiar to the core library/application structure of the PRISMS-PF framework [25], where the core library contains shared functionality (e.g., library interfaces, generic input/output, initial condition utilities, and a mechanics solver) and the applications contain the implementation of functionality tied to a particular use case (e.g., use-case-dependent input parameters, governing equations, and postprocessing analysis). In addition to the core library and the MEUMAPPS-SS application, the MEUMAPPS C++ repository [54] contains applications for the Allen-Cahn equation, the Cahn-Hilliard equation (PFHub Benchmark 1a [55]), and a simple coupled Allen-Cahn-Cahn-Hilliard system. Although application development for MEUMAPPS C++ has been focused on phase-field models, the framework is designed to be general for other models amenable to pseudospectral methods.

Next, we discuss briefly the general numerical approach used to solve the governing equations, with a more detailed description of some aspects in later sections. We apply a pseudospectral discretization in space, using FFTs to move from real space (where the physical inputs and outputs are defined) to reciprocal space (where the time integration is performed). In reciprocal space the PDE solve over the entire domain is replaced with ODEs at every grid point. Following a common approach in pseudospectral phase-field simulations since the seminal work by Chen and Shen [11], we use an ImEx scheme to advance the ODE system in time. This choice of time integration approach allows us to exploit the structure of (9) and (12) for greater efficiency. The reciprocal space counterparts of (9) and (12) can be rewritten as

∂X~c∂t=Mc∣k∣2X~c+FnonlinearX,(15)\frac{\partial\widetilde{X}^{c}}{\partial t}=M^{c}|\mathbf{k}|^{2}\widetilde{X}^{c}+F^{X}_{nonlinear}, \tag*{(15)}
∂ϕ~v∂t=L(∑i,jκi,jkikj)ϕ~v+Fnonlinearϕ,(16)\frac{\partial\widetilde{\phi}^{v}}{\partial t}=L\left(\sum_{i,j}\kappa_{i,j}k_{i}k_{j}\right)\widetilde{\phi}^{v}+F^{\phi}_{nonlinear}, \tag*{(16)}

where the right-hand side is partitioned into a stiff diffusive term and a less stiff nonlinear term. The diffusive term is linear and treated implicitly, while the nonlinear terms are handled explicitly. With the pseudospectral discretization, the linear solve necessary for the implicit portion of the time integration method is trivial to compute so the time advance can proceed without any (nontrivial) linear or nonlinear solves.

For a general form of the single-phase homogeneous free energy density, fchpf^{p}_{ch}, satisfying Eq. (10) and (11) requires the solution of a pointwise nonlinear equation [60]. However, following a common choice in the literature [6, 14, 19, 58, 60] we use parabolic single-phase homogeneous free energy densities (see Eq. (8)), such that the single phase compositions are given by an analytical expression. To solve Eq. (13), nonlinear mechanical equilibrium, we use the iterative approach of Hu and Chen [12, 13].

Performance portability layer

One of the contributions of this work is to demonstrate the benefit of performance portability approaches to effectively utilize diverse modern computing hardware. Performance portability is the concept of developing software that runs on a variety of computing hardware without duplicate work and doing so without significant sacrifices in performance versus software specifically targeting the hardware [61]. Performance portability focuses on on-node parallelism (rather than between-node distributed memory parallelism which is typically accomplished with a Message Passing Interface (MPI) library). The current increase in diversity of architectures for supercomputers highlights the need for performance portability; as of writing the top ten largest supercomputers include systems with NVIDIA/CUDA, AMD/HIP, and Intel/oneAPI hardware [62].

In this work we use the Kokkos library [61, 63] as our performance portability layer. Kokkos provides MEUMAPPS C++ with data structures and parallel execution operations that have architecture-specific backends. The specific data layouts and execution patterns used depend on the backend but come from the same Kokkos calls in MEUMAPPS C++. Thus, very little of MEUMAPPS C++ has architecture-specific code, and the architecture-specific code it has is largely limited to precompiler macros that provide the proper header files to include. MEUMAPPS C++ has support for the serial, CUDA, and HIP backends in Kokkos (targeting CPU, NVIDIA GPU, and AMD GPU architectures, respectively). Support for the experimental SYCL backend (for Intel GPUs) is planned and is not expected to require significant development. Support for the experimental OpenMP target backend (using OpenMP for GPU offload) is not planned, with architecture-specific backends being preferred, but adding support is not expected to require significant development. MEUMAPPS C++ does not currently support the host parallel backends (OpenMP, C++ Threads, and HPX), as the focus is on GPU utilization. MEUMAPPS C++ relies heavily on Kokkos data structures, with the Kokkos::View, a type of multidimensional array, being the data structure used for storing all field variables. As shown in Table 1 and Table 2, kernels written as Kokkos lambda functions or functors occupy the vast majority of the run-time of the code in MEUMAPPS C++ itself (i.e., not counting the library calls to heFFTe for the distributed 3D FFTs).

One of the contributions in this work is an approach for translating between Views and the data structures for the other libraries we use, SUNDIALS and heFFTe. The translation approach is discussed in the following two subsections.

Time integration

To advance the model equations in time, we utilize ImEx-ARK time integration methods from the ARKODE [64] package in the SUNDIALS [65, 66] library of time integrators and nonlinear solvers. Unlike the multi-step method presented in Ref. [11], ImEx-ARK methods are one-step, multi-stage methods. Our primary motivation for choosing ImEx-ARK methods is their greater stability at high order [67, 68], the potentially lower cost of error estimates for step size adaptivity with lower order methods compared to step-doubling approaches with ImEx multi-step methods [69], and the availability of an established, general-purpose, GPU-accelerated library (ARKODE) for C++ applications.

ImEx-ARK methods target mixed stiff/non-stiff systems of the form

y′=fI(t,y)+fE(t,y),y(t0)=y0,(17)y^{\prime} = f^{I}(t,y) + f^{E}(t,y), \qquad y(t_{0}) = y_{0}, \tag*{(17)}

where yy is the state vector of composition fields and order parameters with the initial condition y0y_{0} at time t0t_{0}. The right-hand side function fIf^{I} contains the stiff linear terms in Eqs. (15) and (16) which are treated implicitly while fEf^{E} includes the remaining nonlinear terms which are treated explicitly. A step from tn−1t_{n-1} to tnt_{n} with an ImEx-ARK method is given by

zi=yn−1+Δtn∑j=1iAi,jIfI(tn,jI,zj)+Δtn∑j=1i−1Ai,jEfE(tn,jE,zj),i=1,…,s,(18)z_i = y_{n-1} + \Delta t_n \sum_{j=1}^{i} A^I_{i,j} f^I(t^I_{n,j}, z_j) + \Delta t_n \sum_{j=1}^{i-1} A^E_{i,j} f^E(t^E_{n,j}, z_j), \quad i=1,\ldots,s, \tag*{(18)}
yn=yn−1+Δtn∑i=1s(biIfI(tn,iI,zi)+biEfE(tn,iE,zi)),(19)y_n = y_{n-1} + \Delta t_n \sum_{i=1}^{s} \left(b^I_i f^I(t^I_{n,i},z_i) + b^E_i f^E(t^E_{n,i},z_i)\right), \tag*{(19)}

where the method for computing the internal stages, ziz_i, and the new solution, yny_n, is defined by a pair of ss-stage Butcher tables containing the implicit (AI∈Rs×s,bI∈Rs,cI∈Rs)(A^I \in\mathbb{R}^{s \times s}, b^I \in\mathbb{R}^s, c^I \in\mathbb{R}^s) and explicit (AE∈Rs×s,bE∈Rs,cE∈Rs)(A^E \in\mathbb{R}^{s \times s}, b^E \in\mathbb{R}^s, c^E \in\mathbb{R}^s) coefficients. The implicit and explicit internal stage times are denoted by tn,jI=tn−1+cjIΔtnt^I_{n,j} = t_{n-1} + c^I_j \Delta t_n and tn,jE=tn−1+cjEΔtnt^E_{n,j} = t_{n-1} + c^E_j \Delta t_n, respectively, with step size Δtn=tn−tn−1\Delta t_n = t_n - t_{n-1}. In general, computing ziz_i requires solving a nonlinear algebraic system of equations. However, by treating just the stiff, linear terms implicitly we only need to solve a linear system for each stage. Moreover, the pseudospectral discretization makes the resulting linear solves trivial to compute.

For some methods, the Butcher table may include an additional set of coefficients, b~I∈Rs\tilde{b}^I \in\mathbb{R}^s and b~E∈Rs\tilde{b}^E \in\mathbb{R}^s, that define a (typically) lower-order embedded solution,

y~n=yn−1+hn∑i=1s(b~iIfI(tn,iI,zi)+b~iEfE(tn,iE,zi)).(20)\tilde{y}_n = y_{n-1} + h_n \sum_{i=1}^{s} \left(\tilde{b}^I_i f^I(t^I_{n,i},z_i) + \tilde{b}^E_i f^E(t^E_{n,i},z_i)\right). \tag*{(20)}

The difference between yny_n and y~n\tilde{y}_n provides an estimate of the local truncation error (LTE) in a step which can be utilized to adapt the step size. An attempted step is accepted if ∥LTE∥WRMS≤1\lVert\mathrm{LTE}\rVert_{\mathrm{WRMS}} \le1 in the weighted root-mean-square (WRMS) norm,

∥v∥WRMS=(1N∑i=1N(viwi)2)1/2,(21)\lVert v \rVert_{\mathrm{WRMS}} = \left(\frac{1}{N}\sum_{i=1}^{N}(v_i w_i)^2\right)^{1/2}, \tag*{(21)}

where NN is the length of the vector vv and the weights wiw_i are given by

wi=(rtol∣yn−1,i∣+atol)−1,(22)w_i = \left(\mathrm{rtol}\left|y_{n-1,i}\right|+\mathrm{atol}\right)^{-1}, \tag*{(22)}

with the relative and absolute tolerances, rtol and atol, respectively. If a step attempt is rejected, a new step size is computed based on the LTE, and the step is repeated. After successfully completing a step, the LTE is similarly used to determine the step size for the next step attempt. The results presented above utilize the first order ImEx Euler method, the second order method from [70], as well as the third and fourth order methods from [71]. The second through fourth order methods include embeddings for adaptive step size selection.

Applying an ImEx splitting to Eqs. (15) and (16) enables larger time steps than possible with a fully explicit method as treating the stiff linear terms implicitly alleviates the primary stability restriction on the step size. This choice of splitting also avoids the complexity and expense of a nonlinear solve in each stage when an implicit method is applied to all of the terms. As such, the trade-off with this choice of splitting is that implicit solves can be computed quickly and easily, but the time step sizes are still subject to the stability limitations of the nonlinear terms.

Interfacing with SUNDIALS requires, at a minimum, providing the time integrator with a vector object to operate on state data, yy, and functions to evaluate fIf^I and fEf^E. As noted above, the problem state is stored in separate multidimensional Kokkos Views for XcX^c and ϕv\phi^v. The local data underlying each View is a one-dimensional array, passed with a pointer, that is trivially wrapped by the native SUNDIALS vector objects (NVectors) which support NVIDIA, AMD, and Intel GPUs [72]. The SUNDIALS update is indifferent to the ordering of elements in the one-dimensional array because the time integration is a pure point-wise operation. The distributed state vector (i.e., yy) is formed by constructing an MPI-aware ManyVector [73] (itself an NVector) comprised of the local subvectors wrapping the View data. When evaluating fIf^I or fEf^E, the state data wrapped by the input NVector is copied into the MEUMAPPS-SS objects for computing the FFTs and evaluating the relevant physics. Relative to the other computations in MEUMAPPS-SS, this data copy incurs no meaningful overhead. Finally, to take advantage of the special structure of the linear solves with a pseudospectral discretization, we wrap the MEUMAPPS-SS linear solve computation as a SUNDIALS linear solver object (SUNLinearSolver) enabling ARKODE to directly call the MEUMAPPS-SS native linear solver for the Poisson systems in each implicit stage.

Scalable, performance-portable fast Fourier transforms

To compute multidimensional distributed FFTs, MEUMAPPS C++ relies on the highly efficient fast Fourier transforms for exascale (heFFTe) library [16] which unifies multiple parallel FFT approaches found in different DOE applications [74, 75]. The two primary kernels underlying parallel FFTs are (1) computing one-dimensional transformations and (2) data reshaping operations to align the data such that all the entries along a single dimension belong to the same MPI rank. To compute the one-dimensional FFTs, heFFTe interfaces with a number of different on-node backends such as FFTW [76] and vendor optimized libraries for AMD, Intel, and NVIDIA GPUs. As such, the main focus of heFFTe is on the intricacies of the communication algorithms for data reshaping operations which usually exceed the cost of the one-dimensional transformations, especially with GPU acceleration [77, 78].

The heFFTe interface operates on flat arrays of local data with a corresponding box3d object that describes the global indices of the data and the data ordering. As the data underlying the Kokkos Views are one-dimensional arrays, it is straightforward to pack/unpack the state data into a contiguous array for FFT computations. Unlike with the SUNDIALS integration, the heFFTe integration requires specific ordering of the contiguous array to maintain the spatial relationships in each field. For consistency, arrays in MEUMAPPS C++ are always passed to heFFTe using the Kokkos LayoutLeft ordering. The fields with FFTs applied to them in MEUMAPPS C++ are usually four- or five-dimensional Kokkos Views, with related fields (e.g., the vector of composition fields) being stacked into a single View. Each three-dimensional field is individually deep copied from the higher dimensional View into a temporary LayoutLeft View to pass into heFFTe. For Views residing on the GPU, the default layout is LayoutLeft whereas it is LayoutRight when residing on the CPU. With Views declared with the proper layout, the Kokkos::deep copy operation automatically changes the layout during the deep copy. After the FFT operation is completed, the field is deep copied back into the appropriate higher dimensional View. In principle, the deep copy is not necessary and a pointer to the appropriate memory location inside the higher dimensional View could be passed to heFFTe along with whether it is ordered LayoutLeft or LayoutRight. However, the structure of the higher-dimensional views complicates the striding of the array data in memory across the different fields, and so further examination of removing these deep copies is left to future work. Profiling indicates that these deep copies are responsible for less than 5% of the total run time across different domain sizes and hardware, and therefore do not substantially change the overall performance of the simulations.

heFFTe has several configuration options involving communication patterns and indexing during reshaping, some of which have a significant effect on performance. heFFTe supports “slab”, “pencil”, and “brick” domain decompositions. In this study, a “pencil” MPI decomposition was used for all variables (each MPI subdomain extended the full length of the domain along exactly one dimension). The default MEUMAPPS C++ algorithm was used to create the pencils based on the domain size and the number of processes, selecting the number of subdomains in the real-space y and z directions as the closest factor pair for the number of MPI ranks that results in equally sized subdomains (using the lower of the factors for y and the higher of the factors for z). To eliminate a reshape operation in the 3D FFT, the subdomain decomposition is rotated in reciprocal space, using the lower of the factors for x and the higher of the factors for y. This default decomposition strategy was used for all calculations in this article. heFFTe includes several communication patterns, of which pipelined point-to-point (“p2p-pl”) and all-to-all with variably message sizes (“a2av”) were used in this article. Following standard practice, p2p-pl was used for the single-node tests and a2av was used for the scaling tests and the large growth and coarsening simulation, although testing showed minimal differences in performance for the two strategies. heFFTe also includes an option to reorder the data for the 1D FFTs for contiguous memory layout (versus leaving it as-is and striding across the memory). Initial testing showed a slight performance benefit for using the reorder option, and thus that option was used for all simulations in this article.

The MEUMAPPS C++ implementation has a modular interface to FFT libraries, allowing straightforward extensibility to alternative libraries. In addition to heFFTe, MEUMAPPS C++ has experimental support for the AccFFT library [79]. However, initial testing revealed substantially improved performance for heFFTe over AccFFT and therefore only heFFTe results are cited in this article.

Profiling

To collect profiling data MEUMAPPS C++ utilizes Caliper [80], a performance analysis library that provides a general abstraction layer for code instrumentation and performance measurement in HPC applications. At the user level, Caliper defines a simple, low-overhead API for source-code annotation to mark regions of interest. Behind this interface, Caliper can combine performance measurements from different sources with the annotation context to perform a wide range of performance analysis tasks and generate reports in human or machine-readable formats. The report configuration can be set at runtime with built-in options for common workflows e.g., the “runtime-report” configuration collects timings for annotated regions. Some libraries commonly used in scientific computing, including Kokkos, have built-in support for Caliper. The results presented in this article use the configuration options runtime-report(profile.mpi,mem.highwatermark,profile.kokkos), calc.inclusive, mpi-report to collect timings. In addition to the timings created by default, key regions of MEUMAPPS C++ were instrumented with custom marked regions. The timing information in Table 1 and Table 2 and Figure 2 was generated from scripts analyzing Caliper text file outputs (runtime-report(profile.mpi) for MPI timing, runtime-report(profile.kokkos), and custom marked regions for FFT timing).

Computing systems

The calculations in this article are performed on two supercomputers at the Oak Ridge Leadership Computing Facility (OLCF), Summit and Frontier. The Summit supercomputer contains 4608 IBM Power System AC922 nodes [42, 81]. Each node contains two IBM Power9 CPUs (totaling 42 usable cores, with 2 cores reserved for the operating system) and six NVIDIA V100 GPUs. Each Power9 CPU has 256 GB of memory (512 GB per node) and each V100 GPU has 16 GB of memory (96 GB per node). Each node has approximately 42 TFLOPS of theoretical throughput for double precision datatypes. The nodes are split into subdomains containing one CPU and three GPUs. Resources within a subdomain are connected with NVLink, and the subdomains are connected between the CPUs with an X-Bus. Summit nodes are connected using a dual-rail EDR InfiniBand network with a non-blocking fat-tree topology. The networking interface card for each node is connected to the CPUs.

The Frontier supercomputer contains 9856 HPE/Cray EX nodes [42, 82]. Each node contains one 64-core AMD EPYC CPU and four AMD MI250X accelerators. Each MI250X accelerator contains two graphics compute dies; for practical purposes each compute die functions as a separate GPU. Following standard convention [82], we refer to each MI250X graphics compute die as a GPU. The CPU on a Frontier node has 512 GB of memory and each GPU has 64 GB of memory (512 GB per node). Each node has approximately 191 TFLOPS of theoretical throughput for double precision datatypes. All GPUs on the node are directly connected to each other and to the CPU. Frontier nodes are connected using a HPC Slingshot network with a dragonfly topology. Each Frontier node has four networking interface cards, connected to pairs of GPUs.

Author Contributions

SD: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Writing - Original Draft, Writing - Review & Editing, Visualization; DG: Conceptualization, Methodology, Software, Validation, Formal analysis, Writing - Original Draft, Writing - Review & Editing; PF: Conceptualization, Methodology, Software, Writing - Review & Editing; YS: Methodology, Software, Writing - Review & Editing; MS: Methodology, Software, Writing - Original Draft, Writing - Review & Editing, Funding acquisition; CW: Conceptualization, Methodology, Writing - Review & Editing, Funding acquisition; BR: Conceptualization, Methodology, Software, Validation, Writing - Original Draft, Writing - Review & Editing, Funding acquisition.

All authors read and approved the final manuscript.

Acknowledgements

This research was supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration. This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. Support for this work was also provided in part by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Scientific Discovery through Advanced Computing (SciDAC) Program through the Frameworks, Algorithms, and Scalable Technologies for Mathematics (FASTMath) Institute. This work was performed in part under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. LLNL-JRNL-2020868. The authors also acknowledge guidance and advice from the Kokkos team, particularly Bruno Turcksin, Damien Lebrun-Grandie, and Christian Trott.

Competing Interests

All authors declare no financial or non-financial competing interests.

Data Availability

The datasets generated and analyzed during the current study are available in the OLCF Constellation dataset repository, 10.13139/OLCF/3989587.

Code Availability

The underlying code for this study is available in the Oak Ridge National Laboratory code repository and can be accessed via this link https://code.ornl.gov/meumapps/meumapps.

References

  1. [1]S. DeWitt and K. Thornton, “Phase Field Modeling of Microstructural Evolution,” pp. 67–87. Cham: Springer International Publishing, 2018.DOI
  2. [2]M. R. Tonks and L. K. Aagesen, “The phase field method: Mesoscale simulation aiding material discovery,” Annual Review of Materials Research, vol. 49, no. Volume 49, 2019, pp. 79–102, 2019.DOI
  3. [3]N. Moelans, B. Blanpain, and P. Wollants, “An introduction to phase-field modeling of microstructure evolution,” Calphad, vol. 32, no. 2, pp. 268–294, 2008.
  4. [4]S. Poulsen and P. Voorhees, “Early stage phase separation in ternary alloys: A test of continuum simulations,” Acta Materialia, vol. 113, pp. 98–108, 2016.DOI
  5. [5]S. Ghosh, C. K. Newman, S. DeWitt, J.-L. Fattebert, B. Radhakrishnan, J. Coleman, S. Nichols, and J. Balak, “Massively parallel phase-field simulations targeting exascale,” Computational Materials Science, vol. 261, p. 114323, 2026.DOI
  6. [6]S. DeWitt, E. L. Solomon, A. R. Natarajan, V. Araullo-Peters, S. Rudraraju, L. K. Aagesen, B. Puchala, E. A. Marquis, A. van der Ven, K. Thornton, and J. E. Allison, “Misfit-driven β''' precipitate composition and morphology in Mg-Nd alloys,” Acta Materialia, vol. 136, pp. 378–389, 2017.DOI
  7. [7]L.-Q. Chen, “Phase-field models for microstructure evolution,” Annual Review of Materials Research, vol. 32, no. Volume 32, 2002, pp. 113–140, 2002.DOI
  8. [8]C. Shen and Y. Wang, Coherent Precipitation — Phase Field Method, pp. 2117–2142. Dordrecht: Springer Netherlands, 2005.DOI
  9. [9]K. Thornton, N. Akaiwa, and P. W. Voorhees, “Dynamics of late-stage phase separation in crystalline solids,” Phys. Rev. Lett., vol. 86, pp. 1259–1262, Feb 2001.pubmed.ncbi.nlm.nih.gov/11178058
  10. [10]N. Zhou, D. Lv, H. Zhang, D. McAllister, F. Zhang, M. Mills, and Y. Wang, “Computer simulation of phase transformation and plastic deformation in IN718 superalloy: Microstructural evolution during precipitation,” Acta Materialia, vol. 65, pp. 270–286, 2014.DOI
  11. [11]L. Chen and J. Shen, “Applications of semi-implicit Fourier-spectral method to phase field equations,” Computer Physics Communications, vol. 108, no. 2, pp. 147–158, 1998.DOI
  12. [12]S. Hu and L. Chen, “A phase-field model for evolving microstructures with strong elastic inhomogeneity,” Acta Materialia, vol. 49, no. 11, pp. 1879–1890, 2001.DOI
  13. [13]A. Durga, P. Wollants, and N. Moelans, “Evaluation of interfacial excess contributions in different phase-field models for elastically inhomogeneous systems,” Modelling and Simulation in Materials Science and Engineering, vol. 21, p. 055018, jun 2013.DOI
  14. [14]B. Radhakrishnan, S. Gorti, and S. S. Babu, “Phase field simulations of autocatalytic formation of alpha lamellar colonies in Ti-6Al-4V,” Metallurgical and Materials Transactions A, vol. 47, no. 12, pp. 6577–6592, 2016.DOI
  15. [15]J. P. Boyd, Chebyshev and Fourier Spectral Methods. New York: Dover Publications, 2nd ed., 2000.
  16. [16]A. Ayala, S. Tomov, A. Haidar, and J. Dongarra, “heFFTe: Highly efficient FFT for exascale,” in Computational Science – ICCS 2020 (V. V. Krzhizhanovskaya, G. Závodszky, M. H. Lees, J. J. Dongarra, P. M. A. Sloot, S. Brissos, and J. Teixeira, eds.), (Cham), pp. 262–275, Springer International Publishing, 2020.DOI
  17. [17]P. Shanthraj, Spectral Solvers for Crystal Plasticity and Multi-physics Simulations, pp. 1347–1372. United States: Springer Nature, Feb. 2019.DOI
  18. [18]S. Yoon, D. Jeong, C. Lee, H. Kim, S. Kim, H. G. Lee, and J. Kim, “Fourier-spectral method for the phase-field equations,” Mathematics, vol. 8, no. 8, 2020.DOI
  19. [19]Y. Song, B. Radhakrishnan, S. Gorti, and R. Acharya, “Precipitate growth kinetics under inhomogeneous concentration fields using a phase-field model,” Phys. Rev. Mater., vol. 5, p. 053401, May 2021.DOI
  20. [20]V. Attari, D. Khatamsaz, D. Allaire, and R. Arroyave, “Towards inverse microstructure-centered materials design using generative phase-field modeling and deep variational autoencoders,” Acta Materialia, vol. 259, p. 119204, 2023.DOI
  21. [21]N. Provatas and K. Elder, Phase-Field Methods in Materials Science and Engineering. Weinheim, Germany: Wiley-VCH Verlag GmbH & Co. KGaA, 1st ed., 2010.DOI
  22. [22]A. Boccardo, M. Tong, S. Leen, D. Tourret, and J. Segurado, “Efficiency and accuracy of GPU-parallelized Fourier spectral methods for solving phase-field models,” Computational Materials Science, vol. 228, p. 112313, 2023.
  23. [23]R. Darabi, A. Reis, and J. Cesar de Sa, “Exploring the dynamic transition from aging to additive manufacturing: Investigating microstructural evolution of metastable γ″ precipitates in Inconel 625 through coupled phase-field simulation,” Advanced Engineering Materials, vol. 27, no. 7, p. 2402374, 2025.
  24. [24]M. Tegeler, O. Shchyglo, R. D. Kamachali, A. Monas, I. Steinbach, and G. Sutmann, “Parallel multiphase field simulations with OpenPhase,” Computer Physics Communications, vol. 215, pp. 173–187, 2017.
  25. [25]S. DeWitt, S. Rudraraju, D. Montiel, W. B. Andrews, and K. Thornton, “PRISMS-PF: A general framework for phase-field modeling with a matrix-free finite element method,” npj Computational Materials, vol. 6, no. 1, p. 29, 2020.DOI
  26. [26]X. Shi, H. Huang, G. Cao, and X. Ma, “Accelerating large-scale phase-field simulations with GPU,” AIP Advances, vol. 7, p. 105216, 10 2017.DOI
  27. [27]J. Lee and K. Chang, “Effect of magnetic ordering on the spinodal decomposition of the Fe-Cr system: A GPU-accelerated phase-field study,” Computational Materials Science, vol. 169, p. 109088, 2019.
  28. [28]Z. Liu, Y. Zhao, G. Shang, X.-G. Lu, and Y. Liu, “An accelerated scheme for solving inhomogeneous elasticity in phase-field method and application to Ni-based multicomponent alloys,” Materials Today Communications, vol. 38, p. 107829, 2024.
  29. [29]S. Ghosh, H. Kumar, C. Brandl, A. N. Choudhury, S. Bhattacharyya, and R. Mukherjee, “Effect of molybdenum addition on precipitate coarsening kinetics in Inconel 740h: A phase-field study,” Intermetallics, vol. 175, p. 108513, 2024.
  30. [30]J. A. Turner, J. Belak, N. Barton, M. Bement, N. Carlson, R. Carson, S. DeWitt, J.-L. Fattebert, N. Hodge, Z. Jibben, W. King, L. Levine, C. Newman, A. Plotkowski, B. Radhakrishnan, S. T. Reeve, M. Rolchigo, A. Sabau, S. Slattery, and B. Stump, “ExaAM: Metal additive manufacturing simulation at the fidelity of the microstructure,” The International Journal of High Performance Computing Applications, vol. 36, no. 1, pp. 13–39, 2022.
  31. [31]C. O. Yenusah, Phase-Field Simulations of the Precipitation Kinetics and Microstructure Development in Nickel-Based Superalloys. PhD thesis, 2022.
  32. [32]S. Ghosh, C. K. Newman, and M. M. Francois, “Tusas: A fully implicit parallel approach for coupled phase-field equations,” Journal of Computational Physics, vol. 448, p. 110734, 2022.DOI
  33. [33]T. Takaki, “Large-scale phase-field simulations for dendrite growth: A review on current status and future perspective,” IOP Conference Series: Materials Science and Engineering, vol. 1274, p. 012009, jan 2023.DOI
  34. [34]T. Shimokawabe, T. Aoki, T. Takaki, T. Endo, A. Yamanaka, N. Maruyama, A. Nukada, and S. Matsuoka, “Peta-scale phase-field simulation for dendritic solidification on the TSUBAME 2.0 supercomputer,” in Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’11, (New York, NY, USA), Association for Computing Machinery, 2011.
  35. [35]T. Pinomaa, J. Aho, J. Suviranta, P. Jreidini, N. Provatas, and A. Laukkanen, “OpenPFC: An open-source framework for high performance 3D phase field crystal simulations,” Modelling and Simulation in Materials Science and Engineering, vol. 32, p. 045002, mar 2024.DOI
  36. [36]S. Radhakrishnan, Balasubramaniam anrunged Gorti, Y. Song, and R. Acharya, “Predictive tools for customizing heat treatment of additively manufactured aerospace components,” tech. rep., Oak Ridge National Laboratory, 2022.DOI
  37. [37]F. Schleifer, M. Holzinger, Y.-Y. Lin, U. Glatzel, and M. Fleck, “Phase-field modeling of γ′/γ″ microstructure formation in Ni-based superalloys with high γ″ volume fraction,” Intermetallics, vol. 120, p. 106745, 2020.DOI
  38. [38]R. Laskowski, K. Wang, R. Ahluwalia, K. Bai, G. Vastola, and Y.-W. Zhang, “Phase field model for multiphase alloys under arbitrary thermal history: An application to IN718 super-alloy,” Journal of Alloys and Compounds, vol. 861, p. 158630, 2021.DOI
  39. [39]R. Shi, D. P. McAllister, N. Zhou, A. J. Detor, R. DiDomizio, M. J. Mills, and Y. Wang, “Growth behavior of γ′/γ″ coprecipitates in Ni-base superalloys,” Acta Materialia, vol. 164, pp. 220–236, 2019.DOI
  40. [40]Y. Ji, Y. Lou, M. Qu, J. D. Rowatt, F. Zhang, T. W. Simpson, and L.-Q. Chen, “Predicting coherency loss of γ″ precipitates in IN718 superalloy,” Metallurgical and Materials Transactions A: Physical Metallurgy and Materials Science, vol. 47, no. 6, pp. 3235 – 3247, 2016. Cited by: 46.DOI
  41. [41]A. Plotkowski, M. Rolchigo, G. Wagner, S. T. Reeve, J. Coleman, G. Knapp, L. Levine, A. To, S. DeWitt, F. Dugast, S. Mahadevan, C. Newman, B. Stump, M. Bement, and J. Turner, “Metal additive manufacturing simulation across length, time, and computing scales,” International Materials Reviews, vol. 71, no. 3, pp. 254–293, 2026.DOI
  42. [42]R. D. Budiardja, M. Berrill, M. Eisenbach, G. R. Jansen, W. Joubert, S. Nichols, D. M. Rogers, A. Tharrington, and O. E. Bronson Messer, “Ready for the frontier: Preparing applications for the world’s first exascale system,” in High Performance Computing (A. Bhatele, J. Hammond, M. Baboulin, and C. Kruse, eds.), (Cham), pp. 182–201, Springer Nature Switzerland, 2023.DOI
  43. [43]D. Pekurovsky, “P3DFFT: A framework for parallel computations of Fourier transforms in three dimensions,” SIAM Journal on Scientific Computing, vol. 34, no. 4, pp. C192–C209, 2012.DOI
  44. [44]A. Prothero and A. Robinson, “On the stability and accuracy of one-step methods for solving stiff systems of ordinary differential equations,” Mathematics of Computation, vol. 28, no. 125, pp. 145–162, 1974.DOI
  45. [45]J. M. Sanz-Serna, J. G. Verwer, and W. Hundsdorfer, “Convergence and order reduction of Runge-Kutta schemes applied to evolutionary problems in partial differential equations,” Numerische Mathematik, vol. 50, pp. 405–418, 1986.
  46. [46]J. Verwer, “Convergence and order reduction of diagonally implicit Runge-Kutta schemes in the method of lines,” in Numerical Analysis: Proceedings of the Dundee Conference on Numerical Analysis, 1975, pp. 220–237, Jan. 1986.
  47. [47]A. Biswas, D. I. Ketcheson, S. Roberts, B. Seibold, and D. Shirokoff, “Explicit Runge-Kutta methods that alleviate order reduction,” SIAM Journal on Numerical Analysis, vol. 63, no. 4, pp. 1398–1426, 2025.arxiv.org/abs/2310.02817
  48. [48]A. Biswas, D. I. Ketcheson, B. Seibold, and D. Shirokoff, “Design of DIRK schemes with high weak stage order,” Communications in Applied Mathematics and Computational Science, vol. 18, no. 1, pp. 1–28, 2023.DOI
  49. [49]A. Biswas, D. Ketcheson, B. Seibold, and D. Shirokoff, “Algebraic structure of the weak stage order conditions for Runge–Kutta methods,” SIAM Journal on Numerical Analysis, vol. 62, no. 1, pp. 48–72, 2024.DOI
  50. [50]Kokkos Team, “MDRangePolicy - Kokkos Core Wiki,” 2025. Accessed: 2025-03-31.
  51. [51]H. Childs, E. Brugger, B. Whitlock, J. Meredith, S. Ahern, D. Pugmire, K. Biagas, M. Miller, C. Harrison, G. H. Weber, H. Krishnan, T. Fogal, A. Sanderson, C. Garth, E. W. Bethel, D. Camp, O. Rübel, M. Durant, J. M. Favre, and P. Navrátil, “Visit: An end-user tool for visualizing and analyzing very large data,” in High Performance Visualization–Enabling Extreme-Scale Scientific Insight, pp. 357–372, October 2012.
  52. [52]V. A. Mateevitsi, A. Sewell, M. Bode, P. Fischer, J. H. Göbbert, J. A. Insley, I. Kavroulakis, D. Konioris, Y.-H. Lan, M. Min, D. Papageorgiou, M. E. Papka, S. Petruzza, S. Rizzi, and A. Tomboulides, “Visuals on the house: Optimizing HPC workflows with no-cost CPU visualization,” in 2024 IEEE 14th Symposium on Large Data Analysis and Visualization (LDAV), pp. 69–70, 2024.DOI
  53. [53]J.-L. Fattebert, S. DeWitt, A. Perron, and J. Turner, “Thermo4PFM: Facilitating phase-field simulations of alloys with thermodynamic driving forces,” Computer Physics Communications, vol. 288, p. 108739, 2023.
  54. [54]S. DeWitt, P. Fackler, Y. Song, B. Radhakrishnan, and S. Gorti, “MEUMAPPS (C++ version).” [Computer Software] https://doi.org/10.11578/dc.20220114.1, jan 2022.
  55. [55]A. Jokisaari, P. Voorhees, J. Guyer, J. Warren, and O. Heinonen, “Benchmark problems for numerical implementations of phase field models,” Computational Materials Science, vol. 126, pp. 139–151, 2017.DOI
  56. [56]H. Emmerich, H. Löwen, R. Wittkowski, T. Gruhn, G. I. Tóth, G. Tegze, and L. G. and, “Phase-field-crystal models for condensed matter dynamics on atomic length and diffusive time scales: an overview,” Advances in Physics, vol. 61, no. 6, pp. 665–743, 2012.DOI
  57. [57]R. Shi and Y. Wang, “Variant selection during α precipitation in Ti–6Al–4V under the influence of local stress – a simulation study,” Acta Materialia, vol. 61, no. 16, pp. 6006–6024, 2013.DOI
  58. [58]Y. Ji, A. Issa, T. Heo, J. Saal, C. Wolverton, and L.-Q. Chen, “Predicting β′ precipitate morphology and evolution in Mg–RE alloys using a combination of first-principles calculations and phase-field modeling,” Acta Materialia, vol. 76, pp. 259–271, 2014.DOI
  59. [59]B. Radhakrishnan, S. B. Gorti, and Y. Song, “MEUMAPPS (Microstructure Evolution Using Massively Parallel Phase-field Simulations).” [Computer Software] https://doi.org/10.11578/dc.20201001.96, may 2020.
  60. [60]S. G. Kim, W. T. Kim, and T. Suzuki, “Phase-field model for binary alloys,” Phys. Rev. E, vol. 60, pp. 7186–7197, Dec 1999.pubmed.ncbi.nlm.nih.gov/11970661
  61. [61]C. Trott, L. Berger-Vergiat, D. Poliakoff, S. Rajamanickam, D. Lebrun-Grandie, J. Madsen, N. Al Awar, M. Gligoric, G. Shipman, and G. Womeldorff, “The Kokkos ecosystem: Comprehensive performance portability for high performance computing,” Computing in Science & Engineering, vol. 23, no. 5, pp. 10–18, 2021.DOI
  62. [62]TOP500, “November 2025 top500 list,” 2025. 66th edition of the TOP500 supercomputer ranking.
  63. [63]C. R. Trott, D. Lebrun-Grandié, D. Arndt, J. Ciesko, V. Dang, N. Ellingwood, R. Gayatri, E. Harvey, D. S. Hollman, D. Ibanez, N. Liber, J. Madsen, J. Miles, D. Poliakoff, A. Powell, S. Rajamanickam, M. Simberg, D. Sunderland, B. Turcksin, and J. Wilke, “Kokkos 3: Programming model extensions for the exascale era,” IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 4, pp. 805–817, 2022.DOI
  64. [64]D. R. Reynolds, D. J. Gardner, C. S. Woodward, and R. Chinomona, “ARKODE: A flexible IVP solver infrastructure for one-step methods,” ACM Transactions on Mathematical Software, vol. 49, no. 2, pp. 1–26, 2023.DOI
  65. [65]D. J. Gardner, D. R. Reynolds, C. S. Woodward, and C. J. Balos, “Enabling new flexibility in the SUNDIALS suite of nonlinear and differential/algebraic equation solvers,” ACM Transactions on Mathematical Software (TOMS), vol. 48, no. 3, pp. 1–24, 2022.DOI
  66. [66]A. C. Hindmarsh, P. N. Brown, K. E. Grant, S. L. Lee, R. Serban, D. E. Shumaker, and C. S. Woodward, “SUNDIALS: Suite of nonlinear and differential/algebraic equation solvers,” ACM Transactions on Mathematical Software (TOMS), vol. 31, no. 3, pp. 363–396, 2005.
  67. [67]R. R. Rosales, B. Seibold, D. Shirokoff, and D. Zhou, “Unconditional stability for multistep ImEx schemes: Theory,” SIAM Journal on Numerical Analysis, vol. 55, no. 5, pp. 2336–2360, 2017.DOI
  68. [68]J. Frank, W. Hundsdorfer, and J. G. Verwer, “On the stability of implicit-explicit linear multistep methods,” Applied Numerical Mathematics, vol. 25, no. 2-3, pp. 193–205, 1997.DOI
  69. [69]R. A. Mara’Beh, J. M. Mantas, P. González, and R. J. Spiteri, “Performance comparison of variable-stepsize IMEX SBDF methods on advection-diffusion-reaction models,” Computers & Mathematics with Applications, vol. 190, pp. 41–56, 2025.
  70. [70]F. X. Giraldo, J. F. Kelly, and E. M. Constantinescu, “Implicit-explicit formulations of a three-dimensional nonhydrostatic unified model of the atmosphere (NUMA),” SIAM Journal on Scientific Computing, vol. 35, no. 5, pp. B1162–B1194, 2013.DOI
  71. [71]C. Kennedy and M. Carpenter, “Additive Runge-Kutta schemes for convection-diffusion-reaction equations,” Applied Numerical Mathematics, vol. 44, no. 1-2, pp. 139–181, 2003.DOI
  72. [72]C. J. Balos, D. J. Gardner, C. S. Woodward, and D. R. Reynolds, “Enabling GPU accelerated computing in the SUNDIALS time integration library,” Parallel Computing, vol. 108, p. 102836, 2021.DOI
  73. [73]D. R. Reynolds, D. J. Gardner, C. J. Balos, and C. S. Woodward, “SUNDIALS multiphysics + MPIManyVector performance testing,” Tech. Rep. LLNL-TR-791538, Lawrence Livermore National Laboratory, 2019.arxiv.org/abs/1909.12966
  74. [74]A. Ayala, S. Tomov, M. Stoyanov, and J. Dongarra, “Scalability issues in FFT computation,” in Parallel Computing Technologies: 16th International Conference, PaCT 2021, Kaliningrad, Russia, September 13–18, 2021, Proceedings 16, pp. 279–287, Springer, 2021.DOI
  75. [75]A. Ayala, S. Tomov, P. Luszczek, S. Cayrols, G. Raghhianthi, and J. Dongarra, “FFT benchmark performance experiments on systems targeting exascale,” Univ. Tennessee at Knoxville, Knoxville, TN, USA, Tech. Rep. ICL-UT-22-02, 2022.
  76. [76]M. Frigo and S. G. Johnson, “The design and implementation of FFTW3,” Proceedings of the IEEE, vol. 93, no. 2, pp. 216–231, 2005. Special issue on “Program Generation, Optimization, and Platform Adaptation”.DOI
  77. [77]A. Ayala, S. Tomov, M. Stoyanov, A. Haidar, and J. Dongarra, “Performance analysis of parallel FFT on large multi-GPU systems,” in 2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 372–381, IEEE, 2022.DOI
  78. [78]A. Ayala, S. Tomov, P. Luszczek, S. Cayrols, G. Raghhianthi, and J. Dongarra, “Analysis of the communication and computation cost of FFT libraries towards exascale,” tech. rep., Technical Report ICL-UT-22-07. https://icl. utk. edu/files/publications/2022 . . ., 2022.
  79. [79]A. Gholami, J. Hill, D. Malhotra, and G. Biros, “AccFFT: A library for distributed-memory FFT on CPU and GPU architectures,” arXiv preprint arXiv:1506.07933, 2015.arxiv.org/abs/1506.07933
  80. [80]D. Boehme, T. Gamblin, D. Beckingsale, P.-T. Bremer, A. Gimenez, M. LeGendre, O. Pearce, and M. Schulz, “Caliper: Performance Introspection for HPC Software Stacks,” International Conference for High Performance Computing, Networking, Storage and Analysis (SC’16), (Salt Lake City, Utah, USA), Nov. 2016. LLNL-CONF-699263.
  81. [81]Oak Ridge Leadership Computing Facility, “Summit user guide,” 2024. Accessed: 2026-06-03.
  82. [82]Oak Ridge Leadership Computing Facility, “Frontier user guide,” 2026. Accessed: 2026-06-03.

Paper details

vlnc:W_bwkrzr43
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents