betaFeedback

WorksPhysics and Astronomy

Scalability of the asynchronous discontinuous Galerkin method for compressible flow simulations

Abstract

The scalability of time-dependent partial differential equation (PDE) solvers based on the discontinuous Galerkin (DG) method is increasingly limited by data communication and synchronization requirements across processing elements (PEs) at extreme scales. To address these challenges, asynchronous computing approaches that relax communication and synchronization at a mathematical level have been proposed. In particular, the asynchronous discontinuous Galerkin (ADG) method with asynchrony-tolerant (AT) fluxes has recently been shown to recover high-order accuracy under relaxed communication, supported by detailed analyses of its accuracy and stability. However, the scalability of this approach in modern large-scale parallel DG solvers has not yet been systematically investigated. In this paper, we address this gap by implementing the ADG method coupled with AT fluxes in the open-source finite element library deal.II. We employ a communication-avoiding algorithm (CAA) that reduces the frequency of inter-process communication while accommodating controlled delays in ghost value exchanges. We first demonstrate that applying standard numerical fluxes in this asynchronous setting degrades the solution to first-order accuracy, irrespective of the polynomial degree. By incorporating AT fluxes that utilize data from multiple previous time levels, we successfully recover the formal high-order accuracy of the DG discretization. The accuracy of the proposed method is rigorously verified using benchmark problems for the compressible Euler equations. Furthermore, we evaluate the performance of the method through extensive strong-scaling studies for both two- and three-dimensional test cases. Our results indicate that CAA substantially suppresses synchronization overheads, yielding speedups of up to 1.9x in two dimensions and 1.6x in three dimensions compared to a baseline synchronous DG solver.

Introduction

The discontinuous Galerkin (DG) method has emerged as a powerful framework for the numerical solution of partial differential equations (PDEs), particularly for hyperbolic systems characterized by shocks, sharp gradients, or discontinuities. By combining key features of finite volume and finite element methods, DG achieves high-order accuracy while preserving strict locality of computations through the use of discontinuous basis functions [1]. This locality enables element-wise operations and eliminates the need for global linear solves when explicit time integration is employed, resulting in high arithmetic intensity and favorable performance characteristics on modern high-performance computing (HPC) architectures. Despite these advantages, the scalability of DG solvers on distributed-memory systems is increasingly constrained by data movement and synchronization costs. In parallel DG implementations, numerical flux evaluations across processing-element (PE) boundaries require frequent communication of solution data. As simulations scale to large process counts, the surface-to-volume ratio of each subdomain grows, causing communication and synchronization overheads to dominate the runtime. This challenge is further amplified for high-order discretizations, high-dimensional problems, and fine-grained domain decompositions, posing a major obstacle to the efficient use of emerging exascale systems [2, 3, 4, 5].

Substantial research effort has therefore been directed toward improving the parallel efficiency of DG-based solvers. Algorithmic strategies such as hybridized discontinuous Galerkin (HDG) methods reduce the number of globally coupled degrees of freedom by introducing additional interface unknowns [6, 7]. Parallel-in-time approaches, including Parareal methods, further increase concurrency by decomposing the temporal dimension alongside space [8, 9, 10]. In parallel, hardware-driven advances, most notably GPU acceleration, have enabled DG solvers to exploit asynchronous data transfers and performance-portable programming models [11, 12]. Matrix-free DG formulations have emerged as a particularly effective approach, delivering high arithmetic intensity and reduced memory traffic, and have been shown to outperform hybridized variants in many large-scale scenarios [13, 14, 15, 16]. These developments are embodied in modern DG software frameworks such as ExaDG, built on top of the deal.II finite element library, which has demonstrated strong scaling to hundreds of thousands of cores for challenging fluid dynamics benchmarks [17, 2]. Comparable scalability advances have been reported in other DG frameworks, including DUNE and FLEXI, which emphasize computation–communication overlap and explicit time integration for large-scale compressible and incompressible flow simulations [18, 3, 19, 4]. Nevertheless, even these state-of-the-art solvers exhibit noticeable deviations from ideal scaling at extreme process counts, where communication and synchronization costs become dominant.

An alternative strategy for mitigating communication bottlenecks is to relax data communication and synchronization requirements between PEs at a mathematical level, allowing communication-dependent computations to proceed using delayed data [20, 21, 22, 23]. In particular, we are interested in the asynchronous computing approach based on finite-difference schemes introduced in [24, 22], and the subsequent development of asynchrony-tolerant (AT) finite-difference schemes [25]. These AT schemes employ wider spatio-temporal stencils to recover high-order accuracy despite relaxed communication. An accurate strategy for coupling such AT schemes with multi-stage Runge–Kutta time integration was later developed, demonstrating improved scalability at extreme scales [26]. These methods have been validated for a range of linear and nonlinear problems, including large-scale simulations of compressible turbulence and reacting flows [27, 28, 29, 30].

Building on these ideas, asynchronous computing was recently extended to the discontinuous Galerkin framework through the introduction of the asynchronous discontinuous Galerkin (ADG) method [31, 32]. In ADG, inter-process communication is relaxed for multiple time steps, and delayed data are used in numerical flux evaluations at PE boundaries. While this approach substantially reduces synchronization overheads, it was shown that the use of standard numerical fluxes restricts the accuracy of ADG schemes to first-order, irrespective of the polynomial degree. To overcome this limitation, asynchrony-tolerant (AT) numerical fluxes were proposed, enabling the recovery of high-order accuracy by leveraging locally stored flux histories. These developments establish the theoretical foundation for accurate asynchronous DG methods, which have been validated for combustion simulations [33]. However, their practical accuracy and performance for realistic multidimensional problems in modern DG solvers on massively parallel systems remain open questions.

The focus of the present study is the implementation, validation, and performance characterization of the ADG method with asynchrony-tolerant fluxes in a modern, large-scale DG solver. The main contributions of this paper are summarized as follows:

  • Implement the asynchronous discontinuous Galerkin (ADG) method with asynchrony-tolerant (AT) fluxes in the open-source finite element library deal.II, building upon the matrix-free DG solver provided in step-76 [34].

  • Employ a communication-avoiding algorithm (CAA) that performs inter-process communication only at prescribed intervals and combine it with AT fluxes to enable asynchronous execution with high-order accuracy.

  • Verify the accuracy of the ADG method with AT fluxes for higher-dimensional compressible Euler equations.

  • Perform a comprehensive strong-scaling and profiling analysis to quantify the performance benefits of the CAA-based DG solver relative to a baseline synchronous DG solver.

The remainder of this paper is organized as follows. § 2 reviews the DG formulation and the synchronous solver framework. § 3 introduces the asynchronous DG method and the communication-avoiding algorithms for standard and AT fluxes. Implementation details within deal.II are discussed in § 4. Numerical accuracy and performance results are presented in § 5, followed by concluding remarks and future directions in § 6.

Standard discontinuous Galerkin (DG) method

We begin with a brief overview of the discontinuous Galerkin (DG) formulation for the compressible Euler equations, which are a system of hyperbolic partial differential equations (PDEs) describing inviscid, compressible fluid flow. Solutions of these equations typically exhibit wave-like behavior and may develop discontinuities, such as shocks, even from smooth initial conditions, making them a canonical benchmark for high-order numerical methods.

Let us express the dd-dimensional compressible Euler equations in the following conservative form

∂w∂t+∇⋅F(w)=G(w),(1)\frac{\partial\mathbf{w}}{\partial t} + \nabla\cdot\mathbf{F}(\mathbf{w}) = \mathbf{G}(\mathbf{w}), \tag*{(1)}

defined on a spatial domain Ω⊂Rd\Omega\subset\mathbb{R}^{d} with appropriate initial and boundary conditions, and t∈[0,tfinal]t \in[0,t_{\mathrm{final}}]. Here, w∈Rd+2\mathbf{w} \in\mathbb{R}^{d+2} denotes the vector of conserved variables, F(w)∈R(d+2)×d\mathbf{F}(\mathbf{w}) \in\mathbb{R}^{(d+2)\times d} the convective flux tensor, and G(w)∈Rd+2\mathbf{G}(\mathbf{w}) \in\mathbb{R}^{d+2} a source term, given by

w=[ρρuE],F(w)=[ρuρu⊗u+Ip(E+p)u],G(w)=[0ρgρu⋅g].(2)\mathbf{w} = \begin{bmatrix} \rho\\ \rho\mathbf{u} \\ E \end{bmatrix}, \qquad \mathbf{F}(\mathbf{w}) = \begin{bmatrix} \rho\mathbf{u} \\ \rho\mathbf{u} \otimes\mathbf{u} + I p \\ (E+p)\mathbf{u} \end{bmatrix}, \qquad \mathbf{G}(\mathbf{w}) = \begin{bmatrix} 0 \\ \rho\mathbf{g} \\ \rho\mathbf{u}\cdot\mathbf{g} \end{bmatrix}. \tag*{(2)}

In these expressions, ρ\rho is the density, u∈Rd\mathbf{u} \in\mathbb{R}^{d} the velocity vector, and EE the total energy, defined as

E=pγ−1+ρ2∥u∥2,(3)E = \frac{p}{\gamma-1} + \frac{\rho}{2}\lVert\mathbf{u}\rVert^{2}, \tag*{(3)}

where pp denotes the pressure and γ\gamma is the ratio of specific heats (taken as γ=1.4\gamma= 1.4 for air). The identity matrix I∈Rd×dI \in\mathbb{R}^{d \times d} and the outer product ⊗\otimes follow standard notation, while g\mathbf{g} represents acceleration due to gravity or, more generally, an external force per unit mass acting on the fluid.

DG spatial discretization

To construct a numerical approximation, the spatial domain is partitioned into NEN_E non-overlapping elements, Ω≈Ωh=⋃e=1NEΩe\Omega\approx\Omega_h = \bigcup_{e=1}^{N_E}\Omega_e. Within the DG framework, the solution is approximated locally on each element by polynomials, allowing for discontinuities across element interfaces. Let {φje}j=1Ndof\{\varphi_j^e\}_{j=1}^{N_{\mathrm{dof}}} denote a set of polynomial basis functions of degree NpN_p on element Ωe\Omega_e, where NdofN_{\mathrm{dof}} is the number of local degrees of freedom (DoFs) per element. The element-wise DG approximation is then written as

whe(x,t)=∑j=1Ndofw^je(t)φje(x),(4)\mathbf{w}_h^e(\mathbf{x},t) = \sum_{j=1}^{N_{\mathrm{dof}}}\widehat{\mathbf{w}}_j^e(t)\varphi_j^e(\mathbf{x}), \tag*{(4)}

with w~je(t)\widetilde{\mathbf{w}}_j^e(t) denoting the time-dependent local DoFs. The global DG solution is obtained as the direct sum of the element-wise approximations, wh(x,t)≈⨁e=1Newhe(x,t)\mathbf{w}_h(\mathbf{x},t)\approx\bigoplus_{e=1}^{N_e}\mathbf{w}_h^e(\mathbf{x},t).

Substituting wh\mathbf{w}_h into Eq. (1) yields the residual

Rh=∂wh∂t+∇⋅F(wh)−G(wh),(5)\mathcal{R}_h=\frac{\partial\mathbf{w}_h}{\partial t}+\nabla\cdot\mathbf{F}(\mathbf{w}_h)-\mathbf{G}(\mathbf{w}_h), \tag*{(5)}

which must be minimized to obtain a numerical solution with a desired accuracy. Following the Galerkin approach, we choose test functions from the same space as the basis functions and impose orthogonality of the residual with respect to the L2L^2 inner product. This leads to the element-wise formulation

(φie,∂whe∂t)Ωe+(φie,∇⋅F(whe))Ωe−(φie,G(whe))Ωe=0.(6)\left(\varphi_i^e,\frac{\partial\mathbf{w}_h^e}{\partial t}\right)_{\Omega_e}+\left(\varphi_i^e,\nabla\cdot\mathbf{F}(\mathbf{w}_h^e)\right)_{\Omega_e}-\left(\varphi_i^e,\mathbf{G}(\mathbf{w}_h^e)\right)_{\Omega_e}=0. \tag*{(6)}

Applying integration by parts to the flux divergence term yields the standard DG weak form

(φie,∂whe∂t)Ωe−(∇φie,F(whe))Ωe+(φie,n⋅F^(whe))∂Ωe−(φie,G(whe))Ωe=0,(7)\left(\varphi_i^e,\frac{\partial\mathbf{w}_h^e}{\partial t}\right)_{\Omega_e}-\left(\nabla\varphi_i^e,\mathbf{F}(\mathbf{w}_h^e)\right)_{\Omega_e}+\left(\varphi_i^e,\mathbf{n}\cdot\widehat{\mathbf{F}}(\mathbf{w}_h^e)\right)_{\partial\Omega_e}-\left(\varphi_i^e,\mathbf{G}(\mathbf{w}_h^e)\right)_{\Omega_e}=0, \tag*{(7)}

where n\mathbf{n} denotes the outward unit normal and F^\widehat{\mathbf{F}} is a numerical flux enforcing inter-element coupling. This formulation can be written compactly as

Medwhedt=Lh(t,whe),(8)\mathcal{M}_e\frac{d\mathbf{w}_h^e}{dt}=\mathcal{L}_h(t,\mathbf{w}_h^e), \tag*{(8)}

with Me\mathcal{M}_e denoting the element-local mass matrix and Lh\mathcal{L}_h the spatial DG operator incorporating both volume and surface contributions.

Numerical flux

Due to the discontinuous nature of the DG basis functions, the solution obtained over the discretized domain is, in general, discontinuous across element interfaces. As a result, at the common boundary between two neighboring elements, multiple solution values are defined, one from each element sharing the interface. To illustrate this, Figure 1 shows two adjacent elements, Ωe−1\Omega_{e-1} and Ωe\Omega_e, in a two-dimensional setting, represented here as quadrilateral elements with quadratic basis functions. At the shared face, the solution interface value from the left element is denoted by w−\mathbf{w}^{-}, while the interface value from the right element is denoted by w+\mathbf{w}^{+}. In general, these two values are not equal, leading to ambiguous flux evaluations at the interface.

Image not extracted

Figure 1. Illustration of numerical flux computation across a common edge of two quadrilateral elements in a two-dimensional domain. Circles denote the local nodal degrees of freedom of the elements.

To ensure a unique, conservative coupling between neighboring elements, a single-valued numerical flux function F^(w)=F^(w−,w+)\widehat{\mathbf{F}}(\mathbf{w})=\widehat{\mathbf{F}}(\mathbf{w}^{-},\mathbf{w}^{+}) is introduced. This flux weakly enforces inter-element continuity, guarantees local conservation, and provides the necessary information exchange between elements in the DG formulation.

In this work, we employ the local Lax-Friedrichs (Rusanov) flux [17], defined as

F^(w−,w+)=12[F(w−)+F(w+)]+λ2(w−−w+)⊗n−,(9)\widehat{F}(w^{-}, w^{+}) = \frac{1}{2}\left[F(w^{-}) + F(w^{+})\right] + \frac{\lambda}{2}\left(w^{-} - w^{+}\right) \otimes\boldsymbol{n}^{-}, \tag*{(9)}

where n−\boldsymbol{n}^{-} denotes the outward unit normal associated with the interior state w−w^{-}. The dissipation parameter λ\lambda is chosen as

λ=12max⁡(∥u−∥2+(c−)2,∥u+∥2+(c+)2),(10)\lambda= \frac{1}{2}\max\left(\sqrt{\lVert\boldsymbol{u}^{-}\rVert^{2} + (c^{-})^{2}}, \sqrt{\lVert\boldsymbol{u}^{+}\rVert^{2} + (c^{+})^{2}}\right), \tag*{(10)}

with cc denoting the local speed of sound. This choice ensures stability and robustness of the DG discretization in the presence of strong gradients and discontinuities, which are common in compressible flow simulations.

Time integration

The semi-discrete system in Eq. (8) constitutes a set of ordinary differential equations in time. To preserve the locality and scalability of the DG formulation, explicit time-integration schemes are typically employed. In this work, we use low-storage explicit Runge-Kutta (LSERK) methods, which are widely adopted in large-scale DG solvers due to their favorable stability and memory characteristics.

A generic ss-stage two-storage LSERK scheme can be written as

kme=Me−1Lh(tn+∂mΔt,rme),rm+1e=whe,n+Δt amkme,whe,n+1=whe,n+Δt bmkme,m=1,…,s,(11)\begin{aligned} k_m^e &= \mathcal{M}_e^{-1}\mathcal{L}_h\left(t^n + \partial_m\Delta t, r_m^e\right),\\ r_{m+1}^e &= w_h^{e,n} + \Delta t\,a_m k_m^e,\\ w_h^{e,n+1} &= w_h^{e,n} + \Delta t\,b_m k_m^e,\qquad m=1,\ldots,s, \tag*{(11)} \end{aligned}

where the coefficients ama_m, bmb_m, and ∂m\partial_m are determined from the corresponding Butcher tableau [35].

In a serial implementation, this fully discrete scheme advances the solution by looping over all elements and evaluating the spatial operator locally. Because the scheme is explicit, no global linear solve is required. Instead, element-level computations can be performed independently, apart from the data dependencies introduced by the numerical fluxes. This locality makes DG methods particularly well-suited for parallelization, enabling efficient large-scale simulations through concurrent execution across elements.

Parallel implementation

The parallel implementation of the discontinuous Galerkin (DG) method in this work targets distributed-memory systems using the Message Passing Interface (MPI). The computational domain is decomposed into multiple subdomains and distributed across processing elements (PEs), each of which owns a subset of the mesh elements. Based on data dependencies, local elements are classified as interior elements, whose DG operator evaluations rely solely on locally available data, and PE-boundary elements, for which numerical flux evaluations require solution values from neighboring PEs. The implementation builds upon the distributed-memory parallel infrastructure provided by the open-source finite element library deal.II. This framework offers explicit control over ghost-value exchange and enables a clear separation between communication and computation, which is essential for exposing communication-computation overlap in large-scale DG solvers. Detailed solver-specific aspects of the deal.II implementation are discussed in § 4.

Communication between PEs is required only for evaluating numerical fluxes on PE-boundary elements. Solution values associated with neighboring elements are exchanged via MPI and stored in ghost (buffer) elements. The parallel communication model follows a standard ghost-exchange pattern, where ghost-value updates are explicitly initiated and completed outside the element-local computational kernels. This design decouples communication from computation and enables partial overlap of data exchange with local work. In practice, each PE initiates nonblocking point-to-point communication to exchange ghost values with its neighboring PEs and then proceeds with the evaluation of DG contributions for interior elements that do not depend on remote data. Once these computations are completed, a synchronization step ensures that all required ghost values have been received. The PE then evaluates the contributions for its boundary elements using the updated ghost data. This element-centric execution model preserves locality, minimizes idle time, and forms the basis for overlapping communication with computation.

Image not extracted

Figure 2. (a) Domain decomposition of 144 elements among four PEs: PE-0 (green), PE-1 (orange), PE-2 (blue), and PE-3 (white); (b) classification of interior and PE-boundary elements in the subdomain of PE-0.

Figure 2 illustrates this domain decomposition for a two-dimensional example. Part (a) of the figure exhibits the domain decomposition of 144 elements among 4 PEs, namely, ‘PE-0’, ‘PE-1’, ‘PE-2’, and ‘PE-3’, where the elements of these PEs are represented by green, orange, blue and white colors, respectively. Each of the PEs has 36 local elements. Part (b) of the figure depicts the subdomain of PE-0, demonstrating the interior elements in light green and the boundary elements of PE-0 that rely on values from neighboring PEs for flux computation in dark green. The required values from different PEs are communicated and stored in buffer elements. The buffer elements that store the values from PE-1 are shown in orange, and for storing the communicated values from elements of PE-2, the buffer elements are shown in blue.

This parallel execution model serves as the foundation for both the synchronous and asynchronous algorithms considered in this work. In the following subsection, we describe the standard synchronous algorithm, in which communication and synchronization are enforced at every Runge-Kutta stage.

Synchronous algorithm

The synchronous algorithm (SA) for the discontinuous Galerkin (DG) method combined with a low-storage explicit Runge–Kutta (RK) time integration scheme is summarized in Algorithm 1. This execution model represents the baseline parallel strategy employed in a DG-based parallel PDE solver. Following domain decomposition across processing elements (PEs), the simulation proceeds through a time-stepping loop that dominates the overall computational cost. Each time step consists of multiple RK stages. At the beginning of every RK stage, non-blocking MPI communication is initiated to exchange ghost values associated with PE-boundary elements. While this communication is in progress, the algorithm evaluates all DG contributions, both volume and face integrals, for interior elements in ΩI\Omega_I, whose numerical fluxes depend exclusively on locally owned degrees of freedom. After completing the interior-element computations, a synchronization step is enforced to ensure that all communicated ghost values have been received. Once synchronization is complete, the algorithm proceeds to evaluate the DG operator for PE-boundary elements in ΩB\Omega_B, where numerical fluxes at inter-process interfaces require the most recent data from neighboring PEs. Using these updated ghost values, the stage solution is assembled and the solution is advanced to the next RK stage.

Although non-blocking communication allows partial overlap between communication and computation, the requirement to synchronize ghost values at every RK stage introduces frequent global synchronization points. As a result, the synchronous algorithm can become increasingly communication-bound at large process counts, particularly for high-order discretizations and large-scale simulations. This limitation motivates the development of asynchronous discontinuous Galerkin method discussed in the next section.

Algorithm 1 Synchronous algorithm (SA). Baseline DG-RK implementation with communication and synchronization at every Runge-Kutta stage. (ΩI\Omega_I: interior elements not requiring MPI ghost data; ΩB\Omega_B: PE-boundary elements)

  1. Initialization

  2. for each time step nn do

  3. for each RK stage mm do

  4. Initiate MPI communication of ghost values for PE-boundary elements /* Compute contributions for interior elements */

  5. for each element Ωe\Omega_e in ΩI\Omega_I do

  6. Compute volume contribution using latest local values

  7. for each face ff in ∂Ωe\partial\Omega_e do

  8. Compute numerical flux using latest local values: F^n+θm=F^((whn+θm)−,(whn+θm)+)\widehat{F}^{n+\theta_m}=\widehat{F}((w_h^{n+\theta_m})^-,(w_h^{n+\theta_m})^+)

  9. end for

  10. Update local stage solution for Ωe\Omega_e

  11. end for

  12. Finish MPI communication (synchronize ghost values) /* Compute contributions for PE-boundary elements */

  13. for each element Ωe\Omega_e in ΩB\Omega_B do

  14. Compute volume contribution using latest local values

  15. for each face ff in ∂Ωe\partial\Omega_e do

  16. Compute numerical flux using latest values: F^n+θm=F^((whn+θm)−,(whn+θm)+)\widehat{F}^{n+\theta_m}=\widehat{F}((w_h^{n+\theta_m})^-,(w_h^{n+\theta_m})^+)

  17. end for

  18. Update local stage solution for Ωe\Omega_e

  19. end for

  20. Advance solution to next RK stage

  21. end for

  22. end for

Asynchronous DG method

Despite extensive optimization efforts and the use of computation-communication overlap, the communication and synchronization requirements enforced at the end of each Runge-Kutta (RK) stage introduce substantial overhead in large-scale DG solvers. At extreme process counts, this overhead increasingly dominates the overall runtime and limits scalability. To address this issue, a new asynchronous approach for the discontinuous Galerkin (DG) method has been proposed that relaxes communication and synchronization constraints at the mathematical level [32]. This approach allows the solver to continue advancing in time even when data from neighboring PEs are not yet available, by temporarily relying on previously communicated values.

To illustrate the idea, we consider a fully discrete update equation corresponding to the semi-discrete system in Eq. (8), using an explicit Euler scheme for time integration. The update equation for advancing the solution from time level nn to n+1n+1 with step size Δt\Delta t can be represented as

whe,n+1=whe,n+Δt Me−1Lh(tn,whe,n),(12)w_h^{e,n+1}=w_h^{e,n}+\Delta t\,M_e^{-1}\mathcal{L}_h\left(t^n,w_h^{e,n}\right), \tag*{(12)}

where the right-hand-side operator uses the numerical flux F^en=F^((we−)n,(we+)n)\widehat{F}_e^n=\widehat{F}((w_e^-)^n,(w_e^+)^n) evaluated with solution values from the current time level.

In contrast, the asynchronous discontinuous Galerkin (ADG) method permits the use of delayed data at PE boundaries when communication is avoided. The update equation can then be written as

whe,n+1=whe,n+Δt Me−1Lh(tn,whe,n,whe,n−k~),(13)w_h^{e,n+1}=w_h^{e,n}+\Delta t\,M_e^{-1}\mathcal{L}_h\left(t^n,w_h^{e,n},w_h^{e,n-\widetilde{k}}\right), \tag*{(13)}

where the numerical flux at PE-boundary elements is evaluated using values from an earlier time level n−k~n-\widetilde{k},

F^estd,n=F^((whe,n−k~)−,(whe,n−k~)+)=F^en−k~.(14)\widehat{F}_e^{\mathrm{std},n}=\widehat{F}\left((w_h^{e,n-\widetilde{k}})^-,(w_h^{e,n-\widetilde{k}})^+\right)=\widehat{F}_e^{n-\widetilde{k}}. \tag*{(14)}

Here, k~\tilde{k} denotes the delay with respect to the current time level nn, and is bounded by a parameter LL, called the maximum allowable delay, such that, k~∈{0,1,…,L−1}\tilde{k} \in\{0,1,\ldots,L-1\}. The standard synchronous DG method is recovered as the special case k~=0\tilde{k}=0.

The effect of delayed boundary updates is illustrated in Figure 3 for a one-dimensional domain decomposed between two processing elements, denoted PE-0 and PE-1. The elements Ωe−1\Omega_{e-1} and Ωe\Omega_e represent the right and left boundary elements of PE-0 and PE-1, respectively. Figure 3(a) shows the standard synchronous implementation, in which boundary values are exchanged at every time step and stored at buffer nodes (shown in blue). Fig. 3(b) depicts an asynchronous execution based on a communication-avoiding algorithm (CAA) [28, 26]. In this illustrative example, PEs communicate and synchronize boundary values for two consecutive time steps, followed by three time steps without communication, corresponding to the maximum allowable delay of L=4L=4. This specific pattern is chosen solely for visualization purposes; in practice, different communication intervals may be used.

Image not extracted

Figure 3. Illustration of a discretized one-dimensional domain decomposed into two processing elements (PEs): (a) synchronous execution and (b) asynchronous execution based on the communication-avoiding algorithm with maximum allowable delay L=4L=4.

At time t0t^0, boundary values u0u^0 are communicated and stored in the buffers, and are subsequently used to compute u1u^1 at the PE-boundary elements. Communication is repeated at t1t^1 to update the buffers with u1u^1, which is then used to compute u2u^2 at t2t^2. These updates follow the standard DG formulation. Starting from t2t^2, communication is suspended and the buffer values are no longer refreshed; these delayed values are indicated in red in the figure. The update at t3t^3 therefore uses boundary values u1u^1, corresponding to a delay k~=1\tilde{k}=1. Subsequent updates at t4t^4 and t5t^5 proceed with delays k~=2\tilde{k}=2 and k~=3\tilde{k}=3, respectively. After this communication-free phase, boundary values are refreshed at t5t^5, and the updates at t6t^6 and t7t^7 again follow the standard DG method. This pattern repeats throughout the simulation. Interior elements always use the standard DG update, as all required neighboring values are locally available. Only PE-boundary elements alternate between synchronous and asynchronous updates, depending on the availability of the latest boundary data in the buffers.

Communication-avoiding algorithm with the standard flux

Contrary to the synchronous DG method, the asynchronous DG (ADG) method allows communication to be relaxed for a limited number of time steps. To realize this idea in practice, we employ the communication-avoiding algorithm (CAA), previously introduced in [28, 26]. The CAA performs inter-process communication only periodically, once every LL time steps, while allowing the solver to continue advancing in time using previously communicated data during the intervening steps. Algorithm 2 summarizes the resulting execution pattern. At the beginning of each time step nn, the algorithm determines whether communication should be performed by evaluating the condition n%L=0n\%L = 0. Based on this condition, a boolean flag communication is set. When communication is enabled, ghost-value exchange is explicitly initiated and completed via MPI calls, ensuring that all ghost buffers contain data corresponding to the most recent time level. When communication is disabled, these calls are skipped and the solver proceeds without synchronization.

As in the synchronous algorithm, computations are organized in a nested loop over Runge-Kutta (RK) stages and elements. Interior elements always use the standard DG update, since all required neighboring values are locally available. For PE-boundary elements, however, numerical flux evaluations may be affected by ghost values that are no longer up-to-date when communication is avoided. To account for this, the CAA introduces a delay parameter kk, which represents the number of time steps elapsed since the most recent communication. The delay parameter kk is initialized to zero whenever communication is performed and is incremented by one at each subsequent time step for which communication is skipped. For example, if L=5L = 5 and communication occurs at time step n−1n - 1, then the numerical flux computed at time level n−1n - 1 is reused at time steps nn through n+4n + 4. Consequently, the numerical flux used at any time level n′∈{n,…,n+4}n' \in\{n, \ldots, n + 4\} is given by F^en′−k−1\widehat{F}_e^{n'-k-1}, with kk taking the respective values 0,1,…,40, 1, \ldots, 4. Within each RK stage, interior elements are updated first and independently of communication. For PE-boundary elements, volume contributions are always evaluated using locally available data. For face nodes that do not lie on PE boundaries, or when communication is enabled, the numerical flux is computed using the latest solution values. When communication is disabled and a face node lies on a PE boundary, the numerical flux is not recomputed; instead, a previously stored flux value is reused. These flux values are stored in a local buffer during the first RK stage of a time step when communication is enabled and are subsequently reused during the following time steps, until the communication flag is reset to true.

In this variant of the CAA, referred to as CAA with the standard flux, delayed flux values are used directly in the numerical flux computation at PE boundaries. As a result, only a single set of boundary flux values from the most recent communication step needs to be stored, leading to minimal additional memory overhead proportional to the number of PE-boundary degrees of freedom. While this approach enables communication avoidance and can significantly improve scalability, it is known to reduce the formal accuracy of the scheme due to the use of delayed boundary information [32]. In the next subsection, we describe how this limitation is addressed through the use of asynchrony-tolerant numerical fluxes.

Asynchrony-tolerant (AT) fluxes

In general, a discontinuous Galerkin method using basis polynomials of degree NpN_p coupled with an optimal choice of numerical flux function exhibits a convergence rate of O(hNp+1)O(h^{N_p+1}) in the L2L_2 norm. However, although the asynchronous discontinuous Galerkin (ADG) method maintains stability and consistency with the same numerical flux, it displays poor accuracy. A thorough error analysis based on a statistical framework has been conducted in the previous work [32] to investigate this discrepancy in accuracy. The analysis reveals that the use of delayed values in computing numerical fluxes at PE boundaries restricts the accuracy of the ADG scheme to first-order, regardless of the degree of the basis polynomials employed.

Subsequently, a new flux termed the asynchrony-tolerant (AT) flux has been introduced to recover the accuracy of the ADG schemes. For a time level nn, the AT flux, denoted as F^eAT,n\widehat{F}_e^{\mathrm{AT},n}, utilizes previously computed numerical fluxes F^en−k\widehat{F}_e^{n-k}, F^en−k−1\widehat{F}_e^{n-k-1}, …\ldots, F^en−k−Np\widehat{F}_e^{n-k-N_p}, achieving an expected accuracy of O(hNp+1)O(h^{N_p+1}), which can be expressed as

F^eAT,n=∑l=L1L2c~lF^en−l=F^en+O(hNp+1).(15)\widehat{F}_e^{\mathrm{AT},n} = \sum_{l=L_1}^{L_2} \widetilde{c}^{l}\widehat{F}_e^{n-l} = \widehat{F}_e^n + O\left(h^{N_p+1}\right). \tag*{(15)}

Here, the two limits L1L_1 and L2L_2 represent the extent of consecutive time levels required for approximating the numerical flux. The lower limit is L1=k~L_1 = \widetilde{k} based on the latest value whe,n−k~w_h^{e,n-\widetilde{k}} in the buffer, whereas L2L_2 depends on the desired accuracy of the AT flux. The unknown coefficients c~l\widetilde{c}^{l} can be determined with the help of the

Algorithm 2 Communication-avoiding algorithm (CAA) with the standard flux. ($\Omega_I$: interior elements not requiring MPI ghost data; $\Omega_B$: PE-boundary elements)
1: Initialization
2: Set delay counter $k \leftarrow 0$
3: for each time step $n$ do
    /* Set communication flag and update delay parameter */
4:     if $(n\%L) == 0$ then
5:         communication $\leftarrow$ true
6:         $k \leftarrow 0$
7:     else
8:         communication $\leftarrow$ false
9:         $k \leftarrow k + 1$
10:    end if
11:    for each RK stage $m$ do
12:        if communication is true then
13:            Initiate MPI communication of ghost values
14:        end if
    /* Compute contributions for interior elements */
15:        for each element $\Omega_e \in \Omega_I$ do
16:            Compute volume and face contributions using latest local values
17:            Update local RK-stage solution for $\Omega_e$
18:        end for
19:        if communication is true then
20:            Finish MPI communication (synchronize ghost values)
21:        end if
    /* Compute contributions for PE-boundary elements */
22:        for each element $\Omega_e \in \Omega_B$ do
23:            Compute volume contribution using latest local values
24:            for each face node $f \in \partial\Omega_e$ do
25:                if $f$ is not a PE-boundary node or communication is true then
26:                    Compute standard numerical flux using latest available values
27:                    if $m = 0$ and $f$ lies on a PE boundary then
28:                        Store numerical flux in flux_buffer
29:                    end if
30:                else
31:                    Use delayed numerical flux from flux_buffer at time level $n-k$
32:                end if
33:            end for
34:            Update local RK-stage solution for $\Omega_e$
35:        end for
36:        Advance solution to the next RK stage
37:    end for
38: end for

following constraints

∑l=L1L2clζ(−lΔt)ζζ!={1,ζ=00,0<ζ<Np+1r(16)\sum_{l=L_1}^{L_2} c_l^\zeta\frac{(-l\Delta t)^\zeta}{\zeta!} = \begin{cases} 1, & \zeta= 0 \\ 0, & 0 < \zeta< \dfrac{N_p+1}{r} \end{cases} \tag*{(16)}

Parameter rr, derived from the stability analysis of the numerical scheme, satisfies the CFL relation Δt∼Δxr\Delta t \sim\Delta x^r. For r=1r = 1, the upper limit is L2=k~+NpL_2 = \tilde{k} + N_p, and the respective coefficients for the AT flux for the second and third-order accuracy, are reported in Table 1.

The construction of the AT flux in Eq. (15) is inherently multi-step in nature, as it relies on numerical fluxes from multiple previous time levels. When combined with multi-stage explicit Runge-Kutta (RK) schemes, special care is required to consistently account for the intermediate stage times. The coupling of multi-level asynchrony-tolerant schemes with low-storage multi-stage RK methods has been systematically analyzed in [26], where two different approaches were proposed. The first is a naive approach, in which delayed fluxes are directly incorporated at each RK stage by accounting for the fractional advancement within the time step. While straightforward to implement and robust in practice, this approach limits the overall temporal accuracy to at most third order, regardless of the formal order of the underlying AT scheme. The second approach introduces a more elaborate high-order accurate approach that preserves the full accuracy of both the multi-level AT scheme and the RK integrator, at the expense of increased algorithmic complexity.

In the present work, we restrict our implementation to the naive approach and focus on demonstrating its effectiveness for second- and third-order accurate ADG schemes. Specifically, for an ss-stage explicit RK method, the delay parameter used in the AT flux coefficients is updated at every stage according to

k~m=k~+δm,m=0,1,…,s−1,\tilde{k}_m = \tilde{k} + \delta_m,\qquad m = 0,1,\ldots,s-1,

where kk denotes the integer-valued delay associated with the time step, and δm\delta_m represents the fractional time-step offset corresponding to the mm-th RK stage. The stage-specific delays kmk_m are then used to evaluate the AT flux coefficients c~l\tilde{c}^{l} in Eq. (15).

This formulation allows fractional time-step advancements to be consistently incorporated into the AT flux evaluation without modifying the structure of the RK scheme. Although it restricts the attainable order of accuracy, it is sufficient for the second- and third-order schemes considered in this study and enables a clean integration of AT fluxes into existing low-storage RK-based DG solvers. A detailed discussion of both approaches and their theoretical properties can be found in [26].

OrderCoefficients c~l\tilde{c}^{l} with k~=k\tilde{k}=k
2c~k=(k+1)\tilde{c}^{k}=(k+1), c~k+1=−k\tilde{c}^{k+1}=-k
3c~k=k2+3k+22\tilde{c}^{k}=\frac{k^{2}+3k+2}{2}, c~k+1=−(k2+2k)\tilde{c}^{k+1}=-(k^{2}+2k), c~k+2=k2+k2\tilde{c}^{k+2}=\frac{k^{2}+k}{2}

Table 1. Coefficients of second- and third-order asynchrony-tolerant (AT) fluxes.

communication-avoiding algorithm with AT fluxes

The communication-avoiding algorithm (CAA) combined with the asynchrony-tolerant (AT) flux realizes an accurate asynchronous discontinuous Galerkin (ADG) scheme and is summarized in Algorithm 3. In contrast to the CAA with the standard flux, this variant employs AT fluxes at PE-boundary elements during communication-free phases in order to recover the formal order of accuracy of the DG discretization in the presence of delayed boundary data. As introduced in § 3.2, the AT flux at a given time level requires access to numerical fluxes from Np+1N_p+1 consecutive past time levels at process boundaries. Consequently, the communication pattern in CAA-AT differs from that of the standard-flux variant. Specifically, communication is enabled for Np+1N_p+1 consecutive time steps to populate the boundary flux history, followed by LL time steps during which communication is avoided. This pattern is implemented at the beginning of each time step using the condition nmod(L+Np+1)<(Np+1)n \mathop{\mathrm{mod}} (L+N_p+1)<(N_p+1), which sets a boolean communication flag. When this condition is satisfied, ghost-value exchange is performed and standard numerical fluxes at PE-boundary faces are computed and stored. When the condition is not satisfied, communication is skipped and AT fluxes are constructed from the stored flux history.

As in the synchronous algorithm and the CAA with the standard flux, computations are organized in nested loops over Runge-Kutta (RK) stages and elements. Interior elements always employ the standard DG update, since all required neighboring values are locally available. For PE-boundary elements, the treatment of numerical fluxes depends on the communication state. If communication is enabled, standard numerical fluxes are computed using the latest ghost values. During communication-free time steps, numerical fluxes at PE-boundary faces are evaluated using the AT flux formulation, which combines previously stored flux values. The effective delay entering the AT flux at RK stage mm is given by km=k+∂mk_m = k + \partial_m, where kk denotes the number of consecutive time steps elapsed since the most recent communication and ∂m\partial_m is the fractional RK-stage offset associated with the time-integration scheme. The delay counter kk is reset to zero whenever communication is performed and incremented by one at each subsequent time step for which communication is avoided. Numerical fluxes computed during communication phases are stored at the first RK stage and shifted in a rolling buffer to maintain the required history for AT flux evaluation.

In a dd-dimensional domain, the number of nodes on a PE boundary scales as (Np+1)d−1(N_p+1)^{d-1}. Accordingly, the additional memory required by the CAA-AT approach scales with the number of PE-boundary nodes, the number of stored time levels (Np+1)(N_p+1), and the number of solution variables. This overhead remains modest relative to the total solution storage and is independent of the number of interior degrees of freedom. By combining periodic communication with asynchrony-tolerant fluxes, the CAA-AT approach preserves local conservation and recovers the high-order accuracy of the underlying DG discretization while significantly reducing the frequency of communication and synchronization. This makes the method particularly well-suited for large-scale simulations in communication-dominated regimes, as demonstrated by the numerical results presented in § 5.

DG solver based on the deal.II library

Our implementation uses deal.II, an open-source finite-element library written in C++ that provides extensive support for high-order discretizations, matrix-free operator evaluation, and scalable parallel execution on distributed-memory systems [17]. In addition to a wide range of built-in solvers, deal.II offers a flexible infrastructure for developing custom application codes, making it well-suited for research on advanced numerical methods and parallel algorithms. To concretely realize the parallel DG execution model described in § 2.4, we build upon the step-76 tutorial solver provided with deal.II, which is designed for the solution of the compressible Euler equations using a discontinuous Galerkin (DG) discretization in space and low-storage explicit Runge–Kutta (LSERK) schemes for time integration. The DG formulation for the compressible Euler equations has been reviewed in § 2, and the LSERK coefficients used in the solver are listed in Table 2. The solver includes test cases for both two- and three-dimensional flows and supports multiple numerical fluxes, including the local Lax-Friedrichs and Harten-Lax-van Leer-Contact (HLLC) fluxes.

Schemeama_mbmb_mcmc_m
LSERK(2,2)
Stage 11.0000000000.5000000000.000000000
Stage 2–0.5000000001.000000000
LSERK(3,3)
Stage 10.7557263520.2451702870.000000000
Stage 20.3869544770.1848960520.755726352
Stage 3–0.5699336610.632125000

Table 2. Coefficients of ss-stage, oo-order low-storage explicit Runge-Kutta schemes, LSERK(s,o)(s,o), used for time integration.

The evaluation of the DG operator in step-76 is performed in a fully matrix-free fashion. Both volume and face integrals, as well as the application of the inverse mass matrix, are computed without assembling global matrices, relying instead on sum-factorization techniques to reduce computational complexity and memory traffic. This matrix-free, element-centric execution model enables efficient use of modern CPU architectures and exposes a high degree of parallelism across elements.

Parallelization is realized through domain decomposition across processing elements (PEs) using MPI. To facilitate overlap of communication and computation, deal.II internally partitions the locally owned elements into three groups, commonly referred to as part-0, part-1, and part-2. Elements in part-1 correspond to PE-boundary elements whose numerical fluxes depend on degrees of freedom from neighboring PEs and therefore require MPI communication. Whereas, elements in part-0 and part-2 are interior elements that can be processed independently of inter-process data. At each Runge–Kutta stage, the solver initiates the exchange of ghost values required for PE-boundary elements by calling vector_update_ghosts_start(), which triggers non-blocking MPI communication using the MPI calls MPI_Isend/Irecv. While this communication is in progress, the solver proceeds with the evaluation of DG contributions for elements in part-0, which consist of interior elements whose computations are completely independent of ghost data. Once the computations for part-0 are complete, communication is finalized using the call vector_update_ghosts_finish(), which performs MPI_Wait/all operation, ensuring that all required ghost values are available. The solver then evaluates the DG operator for elements in part-1, corresponding to PE-boundary elements whose numerical fluxes depend on data from neighboring processes. After completing the element-local DG operator

Algorithm 3 Communication-avoiding algorithm (CAA) with asynchrony-tolerant (AT) fluxes. (ΩI\Omega_I: interior elements not requiring MPI ghost data; ΩB\Omega_B: PE-boundary elements)

  1. Initialization

  2. Set delay counter k←0k \leftarrow0

  3. for each time step nn do /* Set communication flag and update delay parameter */

  4. if n%(L+Np+1)<(Np+1)n\%(L+N_p+1)<(N_p+1) then

  5. communication ←\leftarrow true

  6. k←0k \leftarrow0

  7. else

  8. communication ←\leftarrow false

  9. k←k+1k \leftarrow k+1

  10. end if

  11. for each RK stage mm do

  12. if communication is true then

  13. Initiate MPI communication of ghost values

  14. end if /* Compute contributions for interior elements */

  15. for each element Ωe∈ΩI\Omega_e\in\Omega_I do

  16. Compute volume and face contributions using latest local values

  17. Update local RK-stage solution for Ωe\Omega_e

  18. end for

  19. if communication is true then

  20. Finish MPI communication (synchronize ghost values)

  21. end if /* Compute contributions for PE-boundary elements */

  22. for each element Ωe∈ΩB\Omega_e\in\Omega_B do

  23. Compute volume contribution using latest local values

  24. for each face node f∈∂Ωef\in\partial\Omega_e do

  25. if ff is not a PE-boundary node or communication is true then

  26. Compute standard numerical flux using latest available values

  27. if m=0m=0 and ff lies on a PE boundary then

  28. Shift previously stored fluxes: flux_buffer(l+1l+1) ←\leftarrow flux_buffer(ll), for l=0,…,Np−1l=0,\ldots,N_p-1

  29. Store numerical flux in flux_buffer(0)

  30. end if

  31. else

  32. Set stage-dependent delay km←k+∂mk_m\leftarrow k+\partial_m

  33. Compute AT flux using coefficients corresponding to kmk_m: F^kmn+∂m=∑l=0Npc~km+l\widehat{F}^{n+\partial_m}_{k_m}=\sum_{l=0}^{N_p}\widetilde{c}_{k_m+l} flux_buffer(ll)

  34. end if

  35. end for

  36. Update local RK-stage solution for Ωe\Omega_e

  37. end for

  38. Advance solution to next RK stage

  39. end for

  40. end for

evaluation, the solver initiates the accumulation of contributions to shared degrees of freedom by calling vector_compress_start(). This operation is required primarily for continuous finite-element discretizations, where degrees of freedom are shared across element boundaries. While this communication is ongoing, the solver proceeds with computations for elements in part-2, which again do not depend on inter-process data. Finally, the communication is completed via vector_compress_finish(), finalizing the update of the distributed solution vectors.

This structured execution pattern enables partial overlap of communication and computation at multiple points within each Runge-Kutta stage. While synchronization is still required at well-defined points, the partitioning into part-0, part-1, and part-2 reduces idle time and mitigates—but does not eliminate—the scalability limitations imposed by communication and synchronization in the synchronous DG algorithm. In this work, the synchronous DG solver provided by step-76 serves as the baseline against which we compare the proposed asynchronous DG solvers based on the communication-avoiding algorithms described in § 3.1 and § 3.3. The same matrix-free discretization, numerical fluxes, and time integration schemes are used in both cases, ensuring that any observed performance differences can be attributed solely to changes in the communication and synchronization strategy.

Numerical results

In this section, we assess the accuracy and scalability of the asynchronous discontinuous Galerkin (ADG) method with both standard and asynchrony-tolerant (AT) numerical fluxes, and compare its performance against the baseline synchronous DG method. The ADG solver in deal.II is implemented using the communication-avoiding algorithms described in § 3.1 and § 3.3.

Experimental setup

All numerical experiments are conducted using the DG solver step-76 for the compressible Euler equations provided by the deal.II library. The solver includes two benchmark configurations: the two-dimensional isentropic vortex test case and the three-dimensional flow around a cylinder test case, which are described below.

Two-dimensional isentropic vortex test case. The first benchmark is a two-dimensional problem defined on a rectangular domain [0,10]×[−5,5][0,10] \times[-5,5] with Dirichlet boundary conditions. It consists of an isentropic vortex transported through a uniform background flow field (see Figure 4), with the exact solution given by

Image not extracted

Figure 4. Initial density profile for the two-dimensional domain for the isentropic vortex test case.

u=1−βe(1−r2)y−y02π,v=βe(1−r2)x−x02π,ρ=(1−γ−116γπ2β2e2(1−r2))1γ−1,p=ργ,(17)\begin{aligned} u &= 1-\beta e^{(1-r^2)}\frac{y-y_0}{2\pi},\\ v &= \beta e^{(1-r^2)}\frac{x-x_0}{2\pi},\\ \rho&= \left(1-\frac{\gamma-1}{16\gamma\pi^2}\beta^2 e^{2(1-r^2)}\right)^{\frac{1}{\gamma-1}},\\ p &= \rho^\gamma, \tag*{(17)} \end{aligned}

where r=(x−t−x0)2+(y−y0)2r = \sqrt{(x-t-x_0)^2+(y-y_0)^2}, x0=5x_0 = 5, y0=0y_0 = 0, β=5\beta= 5, and γ=1.4\gamma= 1.4 [1]. This analytical solution is used to prescribe both the initial and boundary conditions.

Three-dimensional flow around a cylinder test case. The second benchmark is a three-dimensional channel flow with an embedded circular cylinder. The computational domain is defined as [0,2.2]×[0,0.41]×[0,0.41][0, 2.2]\times[0, 0.41]\times[0, 0.41], with a cylinder of diameter 0.1 placed parallel to the zz-axis and centered at (0.2,0.2,0)(0.2, 0.2, 0). A constant inflow condition is prescribed at the inlet using Dirichlet boundary conditions, while a subsonic outflow condition is imposed at the outlet by prescribing the energy and extrapolating mass and momentum from the interior. The top and bottom channel walls, as well as the cylinder surface, satisfy a no-penetration boundary condition.

The initial condition corresponds to a uniform subsonic flow with density ρ=1\rho= 1, velocity u=(0.4,0,0)\mathbf{u} = (0.4, 0, 0), and total energy E=c2γ(γ−1)+12ρ∥u∥2E = \frac{c^2}{\gamma(\gamma- 1)} + \frac{1}{2}\rho\lVert\mathbf{u}\rVert^2, resulting in a Mach number of Ma=0.307Ma = 0.307. This benchmark follows the classical setup described in [36].

Computational platform. All simulations are performed on the PARAM Pravega supercomputer at the Indian Institute of Science (IISc), Bengaluru, India. Each compute node is equipped with dual-socket Intel Xeon Cascade Lake 8268 processors (48 cores per node), 4 GB of RAM per core, and is connected via a fat-tree interconnect. In total, the system consists of 428 compute nodes. The experimental configuration varies key numerical parameters, including the polynomial degree NpN_p, the number of elements per spatial direction, and the coefficients of the low-storage explicit Runge–Kutta (LSERK) scheme. Mesh resolution is controlled by a global refinement parameter GG, where each increment in GG doubles the number of elements along each coordinate direction. Figure 5 illustrates representative domain decompositions for both test cases. For the two-dimensional isentropic vortex test case, part (a) shows a configuration with G=4G = 4, resulting in 4096 elements distributed evenly across 256 MPI processes. For the three-dimensional flow around a cylinder test case, part (b) depicts a setup with G=2G = 2, comprising 25,600 elements distributed over 320 MPI processes.

Image not extracted

Figure 5. Domain decompositions for (a) the two-dimensional isentropic vortex test case with 4096 elements distributed across 256 PEs and (b) the three-dimensional flow around a cylinder test case with 25,600 elements distributed across 320 PEs.

Profiling

This subsection presents a detailed profiling analysis of the step-76 solver in deal.II, with the objective of quantifying the relative costs of computation and communication and identifying the dominant scalability bottlenecks. In particular, we examine the time spent in ghost-value exchange, vector compression, and element-local DG operator evaluations, which together account for the majority of the runtime at large process counts. The profiling results are obtained using user-defined timers embedded in the solver. Based on the global refinement parameter GG and the polynomial degree NpN_p, we consider representative configurations that are sufficiently large to expose communication overheads. For the two-dimensional isentropic vortex test case, we analyze the configurations G=6,Np=2G = 6, N_p = 2 (65,536 elements, 2.4 million DoFs) and G=7,Np=2G = 7, N_p = 2 (262,144 elements, 9.4 million DoFs). For the three-dimensional flow around a cylinder test case, we use the configurations G=3,Np=2G = 3, N_p = 2 (204,800 elements, 27.6 million DoFs) and G=4,Np=2G = 4, N_p = 2 (1.6 million elements, 221.2 million DoFs). The two-dimensional cases are run for 5000 time steps, while the three-dimensional cases are run for 1000 time steps. All profiling results are based on multiple independent executions to account for run-to-run variability. Each configuration is executed five times, and for every run the solver reports the minimum, maximum, and average time spent in each measured operation. To obtain robust timing estimates, we report the mean of the average times computed over these five runs for each configuration and process count.

2D: isentropic vortex test case
GG34567
NEN_E1,0244,09616,38465,536262,144
DoFs36,864147,456589,8242.4M9.4M
3D: flow around a cylinder test case
GG1234–
NEN_E3,20025,600204,8001.6M–
DoFs432,0003.5M27.6M221.2M–

Table 3. Problem configurations for the two-dimensional isentropic vortex and three-dimensional flow around a cylinder test cases with polynomial degree Np=2N_p = 2.

Figure 6 presents a breakdown of computation and communication times as a function of the number of MPI processes. We report the time spent in ghost-value exchange performed by vector_update_ghosts_start and vector_update_ghosts_finish, vector compression (vector_compress_start and vector_compress_finish), and element-local DG operator evaluations grouped according to the internal partitions part-0, part-1, and part-2. In addition, we show the total computation time, defined as the sum of the three compute partitions, and the combined computation and communication time. We first consider the two-dimensional results shown in Figure 6(a) and (b), corresponding to the configurations G=6,Np=2G = 6, N_p = 2 and G=7,Np=2G = 7, N_p = 2, respectively. The number of MPI processes ranges from 128 to 16,416, corresponding to 4 to 342 compute nodes.

Image not extracted

Figure 6. Breakdown of communication and computation times for (a) 2.4M DoFs and (b) 9.4M DoFs for the two-dimensional isentropic vortex test case, and (c) 27.6M DoFs and (d) 221.2M DoFs for the three-dimensional flow around a cylinder test case.

For the largest configuration, 48 processes per node are used, whereas all smaller configurations employ 32 processes per node. For all process counts, the total computation time exhibits near-ideal strong scaling for both cases, reflecting the efficiency of the matrix-free DG operator evaluation. For the lower-resolution case (G=6,Np=2G = 6, N_p = 2), the total computation time saturates beyond 8192 processes, as the number of elements per process becomes very small and the number of PE-boundary elements is reduced to as few as four at the highest process count. In this regime, further reductions in computation time are no longer possible, and fixed overheads dominate. More importantly, communication costs begin to overwhelm computation significantly earlier. For G=6,Np=2G = 6, N_p = 2, the time spent in ghost-value synchronization (vector_update_ghosts_finish) becomes comparable to or larger than the total computation time beyond 512 MPI processes. A similar trend is observed for the higher-resolution case G=7,Np=2G = 7, N_p = 2, where communication overtakes computation beyond approximately 4096 processes. As a result, deviations from ideal strong scaling in the total runtime are observed for both configurations.

The three-dimensional results are shown in Figure 6(c) and (d) for the configurations G=3,Np=2G = 3, N_p = 2 and G=4,Np=2G = 4, N_p = 2, respectively. Here, the number of MPI processes ranges from 160 to 20,400, corresponding to 4 to 425 compute nodes. The largest configuration uses 48 processes per node, while the remaining cases use 40 processes per node. Similar trends are observed as in the two-dimensional case: the total computation time initially scales well with increasing process count, but communication costs associated with ghost-value exchanges increasingly dominate at scale. For the G=3,Np=2G = 3, N_p = 2 configuration, communication becomes the primary contributor to runtime beyond approximately 5120 processes, while for the larger G=4,Np=2G = 4, N_p = 2 configuration this transition occurs at around 20,400 processes.

A further insight is obtained by examining the individual compute partitions. The computation times for part-0 and part-2, which correspond to interior elements, decrease sharply once the number of interior elements per process becomes very small. In this regime, the workload shifts almost entirely to PE-boundary elements handled in part-1, substantially reducing opportunities for computation–communication overlap. Consequently, the remaining element-local computations become dominated by fixed overheads rather than floating-point work. However, the time spent in vector compression remains consistently small across all configurations. Since vector compression primarily targets continuous finite-element discretizations and contributes negligibly to the overall runtime in the present DG setting, it is omitted from further performance analysis. Overall, these profiling results clearly demonstrate that, beyond a problem-dependent process count, communication and synchronization associated with ghost-value exchanges overwhelm computation and become the dominant bottleneck, ultimately limiting the scalability of the synchronous DG solver.

Accuracy

This subsection presents a comparative accuracy study of three implementations: the synchronous algorithm (SA) based on the standard discontinuous Galerkin (DG) method, and two communication-avoiding algorithm (CAA) variants based on the asynchronous discontinuous Galerkin (ADG) method using standard and asynchrony-tolerant (AT) numerical fluxes. The comparison is carried out using the two-dimensional isentropic vortex test case, for which an analytical solution is available, enabling a quantitative error assessment. The computational framework employs DG basis functions of degree NpN_p in space coupled with a low-storage explicit Runge-Kutta scheme of order q=Np+1q = N_p + 1 for time integration. We denote this combination as the DG(NpN_p)-LSERKqq scheme. The corresponding asynchronous formulations are referred to as ADG(NpN_p)-LSERKqq when the standard numerical flux is used, and ADG(NpN_p)-ATqq-LSERKqq when the AT flux is employed.

It is important to note that asynchronous DG schemes are subject to more restrictive stability limits compared to their synchronous counterparts, as discussed in detail in [32]. In particular, the maximum stable CFL number (σ\sigma) decreases as the allowable communication delay LL increases. However, the use of AT fluxes improves the stability properties of ADG schemes. For example, for a CFL value of σ=0.05\sigma= 0.05, the ADG(2)-LSERK3 scheme with L=4L = 4 becomes unstable, whereas, for the same LL and CFL value, the ADG(2)-AT3-LSERK3 scheme remains stable. For all accuracy studies presented here, we therefore fix the maximum allowable delay to L=4L = 4. A conservative CFL number of σ=0.05\sigma= 0.05 is used for the DG and ADG-AT schemes, which lies well within the stability limits of the ADG(2)-AT3-LSERK3 formulation. For the ADG scheme with standard fluxes, a smaller CFL value of σ=0.03\sigma= 0.03 is employed to ensure stability.

Figure 7 shows the density fields obtained using the DG(2)-LSERK3, ADG(2)-LSERK3, and ADG(2)-AT3-LSERK3 schemes at tfinal=4.0t_{\mathrm{final}} = 4.0, together with their corresponding error distributions. The simulations use a mesh refinement level of G=4G = 4, resulting in 4096 elements distributed across 256 MPI processes. While the density fields produced by all three schemes appear visually indistinguishable, the error distribution for the ADG(2)-LSERK3 scheme exhibits noticeably larger errors compared to the other two cases. This observation indicates that the use of standard numerical fluxes within the CAA framework leads to a degradation in accuracy. Whereas, incorporating AT fluxes reduces the error magnitude to a level comparable to that of the synchronous DG solver.

Image not extracted

Figure 7. Density field (a-c) and corresponding error distribution (d-f) at tfinal=4.0t_{\mathrm{final}} = 4.0 for the two-dimensional isentropic vortex test case using synchronous DG(2)-RK3, ADG(2)-RK3, and ADG(2)-AT3-RK3 schemes (left to right). Simulation parameters: G=4G = 4 (NE=4096N_{E} = 4096, 147,456 DoFs), P=256P = 256, σ=0.05\sigma= 0.05 for DG(2)-RK3 and ADG(2)-AT3-RK3, σ=0.03\sigma= 0.03 for ADG(2)-RK3, and maximum allowable delay L=4L = 4.

To quantify these observations, we compute the L2L_2-norm errors for density, momentum, and energy over a sequence of successively refined meshes. We first compare the ADG(NpN_p)-LSERKqq scheme using the standard flux with the corresponding synchronous DG(NpN_p)-LSERKqq scheme. The synchronous implementation is expected to achieve an accuracy of O(hNp+1)O(h^{N_p+1}), whereas the ADG implementation with standard fluxes is theoretically limited to first-order accuracy due to the use of delayed boundary data [32]. Figure 8 and Figure 9 show the error convergence for Np=1N_p = 1 and Np=2N_p = 2, respectively, obtained using P=256P = 256 MPI processes and L=4L = 4. The slopes of the error curves for the synchronous DG schemes (solid black lines) correspond to second- and third-order accuracy for DG(1)-LSERK2 and DG(2)-LSERK3, respectively. In comparison, the ADG schemes with standard fluxes (dashed orange lines) exhibit a convergence rate of approximately first order in both cases, in agreement with theoretical predictions.

Image not extracted

Figure 8. Convergence of L2L_2-norm errors with grid refinement for the compressible Euler equations: (a) density, (b) momentum, and (c) energy for the two-dimensional isentropic vortex test case. Solid black lines denote the synchronous DG(1)-LSERK2 scheme, dashed orange lines denote the ADG(1)-LSERK2 scheme, and dash-dotted blue lines denote the ADG(1)-AT2-LSERK2 scheme. Black dashed reference lines indicate slopes of −1-1 and −2-2. The xx-axis shows the number of elements in one direction ranging from 32 (G=3G = 3) to 256 (G=6G = 6). Simulation parameters: tfinal=4.0t_{\mathrm{final}} = 4.0, σ=0.05\sigma= 0.05 for SA (DG) and CAA-AT (ADG-AT), and 0.03 for CAA (ADG), P=256P = 256, and L=4L = 4.

Image not extracted

Figure 9. Convergence of L2L_2-norm errors with grid refinement for the compressible Euler equations: (a) density, (b) momentum, and (c) energy for the two-dimensional isentropic vortex test case. Solid black lines denote the synchronous DG(2)-LSERK3 scheme, dashed orange lines denote the ADG(2)-LSERK3 scheme, and dash-dotted blue lines denote the ADG(2)-AT3-LSERK3 scheme. Black dashed reference lines indicate slopes of −1-1 and −3-3. The xx-axis shows the number of elements in one direction ranging from 32 (G=3G = 3) to 256 (G=6G = 6). Simulation parameters: tfinal=4.0t_{\mathrm{final}} = 4.0, σ=0.05\sigma= 0.05 for SA (DG) and CAA-AT (ADG-AT), and 0.03 for CAA (ADG), P=256P = 256, and L=4L = 4.

Figure 8 and Figure 9 also show the corresponding results for the ADG schemes with second- and third-order AT fluxes (dash-dotted blue lines). Both approaches are expected to achieve an accuracy of O(hNp+1)O(h^{N_p+1}) for basis functions of degree NpN_p [32]. In both cases, the error curves for the synchronous DG scheme and the ADG scheme with AT fluxes overlap closely and exhibit identical slopes, demonstrating second- and third-order convergence for the ADG(1)-AT2-LSERK2 and ADG(2)-AT3-LSERK3 schemes, respectively. These results confirm that the use of AT fluxes successfully restores the optimal order of accuracy of the asynchronous DG method while enabling communication avoidance.

We next assess the accuracy of the asynchronous DG method for the three-dimensional flow around a cylinder test case. Figure 10 presents the pressure field at tfinal=1.0t_{\mathrm{final}} = 1.0 obtained using the DG(2)-LSERK3, ADG(2)-LSERK3, and ADG(2)-AT3-LSERK3 schemes, along with the corresponding contour plots at a set of equidistant pressure levels. The simulations use a mesh refinement level of G=3G = 3, resulting in 204,800 elements distributed across 320 MPI processes. For a fair comparison and to consistently visualize flow features in the three-dimensional domain, we use a common CFL value of σ=0.03\sigma= 0.03 for all three schemes. While the pressure fields produced by all three schemes in parts (a), (b), and (c) appear visually indistinguishable, differences become apparent in the contour plots. In particular, the ADG(2)-LSERK3 scheme (Figure 10(e)) exhibits localized distortions in the shock structures compared to the synchronous DG solution, indicating a loss of accuracy due to the use of delayed boundary data. These regions are highlighted by red ellipses. In contrast, the ADG(2)-AT3-LSERK3 scheme (Figure 10(f)) closely matches the synchronous solution, with nearly identical shock patterns and contour structures.

Pressure field and corresponding contour plots

Figure 10. Pressure field (a–c) and the corresponding contour plots for a set of equidistant pressure levels (d–f) at tfinal=1.0t_{\mathrm{final}} = 1.0 for the three-dimensional flow around a cylinder test case obtained using synchronous DG(2)-RK3 (top row), ADG(2)-RK3 (middle row), and ADG(2)-AT3-RK3 (bottom row) schemes. Simulation parameters: G=3G = 3 (NE=204,800N_{\mathrm{E}} = 204{,}800, 27.6M DoFs), P=320P = 320, σ=0.03\sigma= 0.03, and maximum allowable delay L=4L = 4.

To further highlight these differences, we next analyze the magnitude of the density gradient obtained using the three schemes, shown in Figure 11. The simulations use the same configuration as in Figure 10. While the global fields in parts (a), (b), and (c) appear largely similar, noticeable differences emerge in the downstream region, particularly in the interval x∈[1.6,1.8]x \in[1.6, 1.8]. In this region, the ADG(2)-LSERK3 solution exhibits slight distortions in the flow structures compared to the synchronous DG solution. To examine these differences more closely, parts (d), (e), and (f) present slices in the yzyz-plane at x=1.8x = 1.8. The discrepancies are clearly visible in Figure 11(e), where the high-gradient regions near the top corners are significantly under-resolved compared to the corresponding structures in the synchronous solution in part (d) of the figure. Whereas, the ADG(2)-AT3-LSERK3 result in Figure 11(f) closely matches Figure 11(d), with nearly identical gradient distributions. This again demonstrates that the use of AT fluxes effectively mitigates the errors introduced by delayed PE-boundary data in the asynchronous DG method.

Magnitude of the density gradient and corresponding slices for three-dimensional flow around a cylinder

Figure 11. Magnitude of the density gradient ∥∇ρ∥\lVert\nabla\rho\rVert (a–c) and the corresponding slices at x=1.80x = 1.80 (d–f) at tfinal=1.0t_{\mathrm{final}} = 1.0 for the three-dimensional flow around a cylinder test case obtained using synchronous DG(2)-RK3 (top row), ADG(2)-RK3 (middle row), and ADG(2)-AT3-RK3 (bottom row) schemes. Simulation parameters: G=3G = 3 (NE=204,800N_E = 204{,}800, 27.6M DoFs), P=320P = 320, σ=0.03\sigma= 0.03, and maximum allowable delay L=4L = 4.

Computational performance

This subsection evaluates the computational performance and scalability of the asynchronous DG solver based on the communication-avoiding algorithm (CAA) with asynchrony-tolerant (AT) fluxes. We focus on strong-scaling behavior and quantify performance improvements relative to the synchronous algorithm (SA) implemented in deal.II. All results in this section use the CAA-AT formulation, which preserves high-order accuracy and therefore represents the practically relevant asynchronous approach. Strong-scaling experiments are performed for both two- and three-dimensional test cases. For the two-dimensional isentropic vortex test case, we consider mesh refinement levels ranging from G=4G = 4, Np=2N_p = 2 (4096 elements and 147,456 DoFs) to G=7G = 7, Np=2N_p = 2 (262,144 elements and 9.4M DoFs). For the three-dimensional flow around a cylinder test case, we use configurations from G=1G = 1, Np=2N_p = 2 (3200 elements and 432,000 DoFs) to G=4G = 4, Np=2N_p = 2 (1.6M elements and 221.2M DoFs). The number of MPI processes ranges from 128 (4 compute nodes) to 16,416 (342 compute nodes) in two dimensions, and from 160 (4 compute nodes) to 20,400 (425 compute nodes) in three dimensions. The two-dimensional simulations are advanced for 5000 time steps, whereas the three-dimensional simulations use 1000 time steps. Each configuration is executed five times, and for every run the solver records the average time per operation over the entire simulation. The reported timings are obtained by taking the mean of these average times across the five independent runs for each configuration and process count. To assess performance, we report the speedup

speedup=TSATCAA\text{speedup} = \frac{T_{\mathrm{SA}}}{T_{\mathrm{CAA}}}

and the parallel efficiency

parallel efficiency=T(P0)T(P)P0P\text{parallel efficiency} = \frac{T(P_0)}{T(P)}\frac{P_0}{P}

where T(P)T(P) denotes the total runtime on PP processes and P0P_0 is the reference process count.

Figure 12 shows the speedup of the communication-avoiding algorithm with AT fluxes (CAA-AT) relative to the synchronous algorithm (SA) for the two-dimensional configuration G=6G = 6, Np=2N_p = 2 (65,536 elements and 2.4M DoFs). In these experiments, the problem size is kept fixed while the number of MPI processes is increased from 128 to 16,416. A speedup greater than one indicates that the asynchronous solver outperforms the synchronous baseline. The curves correspond to different values of the maximum allowable delay LL, ranging from L=2L = 2 to L=10L = 10. For all values of LL, the CAA-AT approach achieves a speedup greater than one across moderate to large process counts, demonstrating consistent performance gains over the synchronous solver. Moreover, the speedup increases with both the process count and the delay parameter LL. As the process count increases, the fraction of elements located on process boundaries grows, and the runtime of the synchronous solver becomes increasingly dominated by communication and synchronization overhead due to frequent ghost-value exchanges and global synchronization. Consequently, larger values

Figure 12. Effect of communication delays on the speedup of the communication-avoiding algorithm with AT fluxes (CAA-AT: ADG(2)-AT3-LSERK3) relative to the synchronous algorithm (SA: DG(2)-LSERK3), obtained from strong-scaling experiments for the two-dimensional isentropic vortex test case with configuration G=6G = 6, Np=2N_p = 2 (NE=65,536N_E = 65{,}536, 2.4M DoFs).

of LL lead to significantly higher speedups at extreme scales, as communication is performed less frequently and synchronization overhead is reduced. These results clearly demonstrate that increasing the maximum allowable delay enhances performance by effectively mitigating communication costs. Based on this observation, the remainder of the performance study focuses on the case L=10L = 10, which provides the highest speedup among the tested configurations. In this setting, communication is skipped for nine time steps and subsequently performed for Np+1N_p + 1 time steps to satisfy the AT-flux requirements.

Figure 13 compares the performance of the synchronous algorithm (SA) and the communication-avoiding algorithm with AT fluxes (CAA-AT) for multiple mesh resolutions in two and three dimensions using a fixed maximum allowable delay of L=10L = 10. In all cases, strong-scaling experiments are performed with a fixed problem size while increasing the number of MPI processes. The black dashed horizontal lines denote unit speedup, corresponding to equal performance of SA and CAA-AT. We first examine the two-dimensional results shown in Figure 13(a), which include four mesh refinement levels: G=4G = 4 (147,456 DoFs, red), G=5G = 5 (589,824 DoFs, green), G=6G = 6 (2.4M DoFs, blue), and G=7G = 7 (9.4M DoFs, magenta). For the lowest resolution case (G=4G = 4), the achievable speedup is limited due to the small amount of work per process. Nevertheless, CAA-AT consistently outperforms SA, achieving a speedup of approximately 1.4× at the largest scale. As the mesh resolution increases, the performance benefit of CAA-AT becomes more pronounced. For the G=5G = 5 and G=6G = 6 cases, the speedup at extreme scale increases further, with the largest improvement observed for G=6G = 6, where CAA-AT is approximately 1.9× faster than SA at 16,416 processes. Overall, CAA-AT delivers substantial performance gains across all two-dimensional configurations.

Figure 13. Speedup of the communication-avoiding algorithm with AT fluxes (CAA-AT: ADG(2)-AT3-LSERK3) relative to the synchronous algorithm (SA: DG(2)-LSERK3), obtained from strong-scaling experiments. (a) Two-dimensional isentropic vortex test case: 147,456 DoFs (red), 589,824 DoFs (green), 2.4M DoFs (blue), and 9.4M DoFs (magenta). (b) Three-dimensional flow around a cylinder test case: 432,000 DoFs (red), 3.5M DoFs (green), 27.6M DoFs (blue), and 221.2M DoFs (magenta). Results are shown for a maximum allowable delay L=10L = 10. The dashed horizontal line denotes unit speedup for reference.

Figure 13(b) presents the corresponding three-dimensional results for the configurations G=1G = 1 (432,000 DoFs, red), G=2G = 2 (3.5M DoFs, green), G=3G = 3 (27.6M DoFs, blue), and G=4G = 4 (221.2M DoFs, magenta). Similar trends are observed in three dimensions, where CAA-AT consistently outperforms SA across all mesh resolutions and process counts. The advantage of the communication-avoiding approach becomes increasingly evident at large scales, with speedups growing from approximately 1.26× for smaller problem sizes to about 1.6× for the largest configuration (G=4G = 4) at 20,400 processes. These results highlight the improved effectiveness of CAA-AT in communication-dominated regimes.

Table 4 further quantifies these trends through parallel efficiency measurements for the three-dimensional test case. Across all problem sizes, the CAA-AT approach consistently achieves higher parallel efficiency than the synchronous solver, with the gap widening at larger process counts. In particular, for the largest configuration (221.2M DoFs), CAA-AT maintains a parallel efficiency of approximately 0.53 at 20,400 processes, compared to only 0.13 for the synchronous algorithm. These results confirm that the communication-avoiding algorithm significantly improves scalability in the communication-dominated regime.

PPParallel efficiency
432,000 DoFs3.5M DoFs27.6M DoFs221.2M DoFs
SACAA-ATSACAA-ATSACAA-ATSACAA-AT
1601.001.001.001.001.001.001.001.00
3200.610.620.890.890.981.010.980.98
6400.510.520.800.880.890.960.910.90
1280––0.650.720.700.790.730.77
2560––0.600.700.510.630.560.71
5120––0.390.610.380.530.410.72
10240––––0.220.320.260.53
20400––––0.110.200.130.53

Table 4. Parallel efficiencies for the three-dimensional flow around a cylinder test case for the synchronous algorithm (SA: DG(2)-LSERK3) and the communication-avoiding algorithm with AT fluxes (CAA-AT: ADG(2)-AT3-LSERK3). Efficiencies are computed relative to the smallest process count (P0=160P_0 = 160).

To better explain the observed performance improvements, we examine the relative contributions of computation and communication to the overall runtime. This analysis demonstrates that the improved scalability of CAA-AT directly results from its ability to reduce communication overheads at scale. Figure 14 presents a breakdown of four key runtime components: the time required to initiate communication, Tstart_commT_{\mathrm{start\_comm}}; the synchronization time associated with completing communication, Tfinish_commT_{\mathrm{finish\_comm}}; the total element-local computation time, TcompT_{\mathrm{comp}}, defined as the sum of part-0, part-1, and part-2 computations; and the overall execution time, Ttotal=Tcomp+TcommT_{\mathrm{total}} = T_{\mathrm{comp}} + T_{\mathrm{comm}}. In the figure, red curves correspond to the synchronous algorithm (SA), while blue curves represent the communication-avoiding algorithm with AT fluxes (CAA-AT).

Figure 14. Strong scaling of total communication and computation times for (a) the two-dimensional isentropic vortex test case performed on up to 16,416 processes across 342 compute nodes, and (b) the three-dimensional flow around a cylinder test case performed on up to 20,400 processes across 425 compute nodes, using the synchronous algorithm (SA: DG(2)-LSERK3, red) and the communication-avoiding algorithm with AT fluxes (CAA-AT: ADG(2)-AT3-LSERK3, blue). Simulation parameters: number of time steps = 5000 for 2D and 1000 for 3D; refinement levels: (a) G=7G = 7 (9.4M DoFs) and (b) G=4G = 4 (221.2M DoFs).

Figure 14(a) presents the breakdown for the two-dimensional test case for the configuration G=7G = 7, Np=2N_p = 2 (9.4M DoFs). The total computation time exhibits near-ideal scaling across the entire range of MPI processes, decreasing almost linearly as the process count increases. However, communication-related costs do not follow such linear reductions. For the synchronous solver, the synchronization time Tfinish_commT_{\mathrm{finish\_comm}} becomes comparable to, and eventually exceeds the computation time at large process counts (beyond 2048 MPI processes), with the gap widening steadily at extreme scales. The CAA-AT approach, while exhibiting similar trends, consistently maintains substantially lower communication costs than SA. In particular, the growth of Tfinish_commT_{\mathrm{finish\_comm}} is significantly delayed compared to the synchronous solver, allowing computation to remain the dominant cost over a larger range of process counts (up to 8192 MPI processes). As a result, the total execution time TtotalT_{\mathrm{total}} for CAA-AT follows the ideal scaling trend more closely than that of SA.

Figure 14(b) presents the corresponding results for the three-dimensional test case with configuration G=4G = 4, Np=2N_p = 2 (221.2M DoFs). Similar behavior is observed, with computation initially scaling well and communication becoming dominant at larger process counts. For the synchronous solver, the synchronization time increases sharply at extreme scale (beyond 10,240 MPI processes), leading to a clear deviation of the total runtime from ideal scaling. Whereas, CAA-AT maintains significantly lower synchronization costs and continues to follow the expected scaling trend more closely. Consequently, CAA-AT achieves consistently lower total runtime than SA across the entire range of process counts.

Figure 15 presents a further operation-level breakdown for the extreme three-dimensional configuration (G=4G = 4, Np=2N_p = 2 with P=20,400P = 20, 400 processes). The bar plot compares the synchronous algorithm (SA) and the communication-avoiding algorithm with AT fluxes (CAA-AT) across individual runtime components, including communication costs, element-local computation, and the total time spent in the time-stepping loop. A clear reduction in communication and synchronization time is observed for CAA-AT, while the computation time remains comparable between the two methods. Overall, these results confirm that the communication-avoiding asynchronous DG method effectively mitigates synchronization overheads and sustains favorable strong-scaling behavior at extreme process counts, thereby enabling improved performance of high-order DG solvers on large-scale parallel systems.

Figure 15. Breakdown of communication and computation times for the three-dimensional configuration G=4G = 4 (221.2M DoFs) at P=20,400P = 20, 400.

Conclusions and discussion

The scalability of high-order discretizations for time-dependent partial differential equations remains fundamentally constrained by communication and synchronization overheads on modern massively parallel computing systems. In particular, discontinuous Galerkin (DG) methods, while attractive for their accuracy, geometric flexibility, and local conservation properties, typically rely on frequent global or neighborhood-level communication at every stage of time integration. At extreme scales, such synchronization requirements significantly limit strong scaling and overall efficiency. Motivated by these challenges, this work has investigated the practical realization of mathematically asynchronous computing within a DG solver for compressible flow simulations. Building on recent developments in asynchronous discontinuous Galerkin (ADG) methods, we implemented communication-avoiding algorithms (CAA) within the deal.II finite-element library, leveraging the matrix-free DG solver provided in step-76. The key idea underlying this approach is to relax communication and synchronization requirements at a mathematical level by allowing controlled delays in the exchange of ghost values between processing elements (PEs). Within this framework, inter-process communication is performed only periodically, while the solver continues advancing in time using previously communicated data. This approach enables substantial overlap of computation and communication and reduces the frequency of global synchronization points that otherwise dominate execution time at scale.

A central challenge of asynchronous DG schemes is the degradation of accuracy caused by the use of delayed solution values in numerical flux evaluations at PE boundaries. Consistent with prior theoretical analysis, our results confirm that ADG schemes combined with standard numerical fluxes are restricted to first-order accuracy, irrespective of the polynomial degree of the basis functions. To overcome this limitation, we incorporated asynchrony-tolerant (AT) numerical fluxes, which reconstruct high-order accurate fluxes using a linear combination of previously computed fluxes from multiple time levels. The CAA with an appropriate AT flux preserves local conservation and restores the formal order of accuracy of the DG discretization in the presence of communication delays. The accuracy of the proposed ADG schemes was validated using a two-dimensional isentropic vortex test case with an analytical solution. For polynomial degrees Np=1N_p = 1 and 2, the ADG method with AT fluxes achieved second- and third-order convergence, respectively, matching the accuracy of the synchronous DG solver. In contrast, the ADG method with standard fluxes exhibited only first-order convergence, in agreement with theoretical predictions. These results demonstrate that AT fluxes are essential for maintaining accuracy in asynchronous DG formulations.

To assess scalability, we performed extensive strong-scaling experiments for both two- and three-dimensional compressible Euler test cases on a modern CPU-based supercomputing system. Detailed profiling of the synchronous solver revealed that ghost-value synchronization, particularly the completion of nonblocking MPI communication, becomes the dominant bottleneck beyond moderate process counts. While element-local DG operator evaluations scale favorably, the increasing ratio of PE-boundary elements and the loss of interior work severely limit further performance gains. The communication-avoiding algorithm-based ADG solver with AT fluxes substantially alleviates these limitations. By increasing the maximum allowable communication delay, the CAA-AT approach reduces the frequency of synchronization and keeps communication costs significantly lower than computation over a wider range of process counts. For sufficiently large configurations considered, the ADG solver with AT fluxes achieved speedups of approximately 1.9×1.9{\times} in two dimensions and 1.6×1.6{\times} in three dimensions relative to the baseline synchronous DG solver. Operation-level breakdowns further confirmed that these gains arise primarily from suppressing synchronization overheads, while maintaining comparable computational costs per time step.

It is important to note that asynchronous DG schemes impose stricter stability constraints than their synchronous counterparts, with the allowable CFL number decreasing as the maximum communication delay LL increases. The use of asynchrony-tolerant (AT) fluxes partially alleviates this limitation by recovering stability of the ADG method. In this work, we restricted attention to second- and third-order accurate schemes and selected conservative CFL values to ensure stable execution across all configurations. In general, increasing the delay parameter LL leads to more restrictive CFL limits. However, alternative implementations of the ADG method can mitigate this effect. In particular, the synchronization-avoiding algorithm discussed briefly before, in which communication is performed regularly while relaxing global synchronization, can effectively limit the impact of large delays. Previous studies have shown that communication delays in such settings follow a Poisson distribution with a mean close to unity, indicating that large delays occur infrequently in practice. This behavior allows the use of less restrictive CFL values compared to fully communication-avoiding implementations. While stricter stability limits may constrain the achievable time step in some applications, they are not restrictive for problems involving stiff source terms, such as chemically reacting flows, where the time step is already dictated by fast chemical time scales. In such regimes, the additional stability constraints of the ADG method become largely irrelevant, and the improved scalability of communication-avoiding schemes can be fully exploited. Further improvements in the stability and robustness of asynchronous DG schemes, including adaptive control of the delay parameter LL and hybrid communication–synchronization strategies, remain active areas of research and will be explored in future work.

Several promising directions emerge from the present study. First, extending the current implementation to higher-order accurate schemes requires adopting more sophisticated coupling approaches between multi-level AT fluxes and multi-stage Runge-Kutta integrators. While this work employed a naive coupling sufficient for second- and third-order accuracy, high-order formulations offer the potential for improved efficiency at scale and require further investigation. Second, the integration of ADG methods with fully coupled massively parallel reacting-flow solvers represents a particularly attractive application area, where the communication-avoiding approach can deliver significant performance benefits without compromising stability. Finally, extending the present approach to more complex physical models, including viscous and multi-physics systems on unstructured meshes as well as high-dimensional plasma equations such as the Vlasov equation, where the surface-to-volume ratio is significantly larger and communication overheads become particularly severe, will further broaden the applicability of asynchronous DG methods.

In summary, this work demonstrates that the asynchronous DG method with asynchrony-tolerant fluxes can be successfully integrated into large-scale, matrix-free DG solvers using different communication-avoiding approaches and deliver substantial scalability improvements for compressible flow simulations. By mitigating synchronization bottlenecks while preserving accuracy and conservation, the proposed approach represents an important step toward the development of efficient DG-based solvers for next-generation exascale computing platforms.

Acknowledgments

The authors benefited from discussions with Phani Motamarri, Martin Kronbichler, Katharina Kormann and Michał Wichrowski. Special acknowledgment is also due to the Council of Scientific and Industrial Research (CSIR), India, for awarding the doctoral fellowship to SKG.

References

  1. [1]J. S. Hesthaven, T. Warburton, Nodal Discontinuous Galerkin Methods: Algorithms, Analysis, and Applications, Springer Publishing Company, Incorporated, 1st edition, 2007.DOI
  2. [2]D. Arndt, N. Fehn, G. Kanschat, K. Kormann, M. Kronbichler, P. Munch, W. Wall, J. Witte, ExaDG: High-Order Discontinuous Galerkin for the Exa-Scale, pp. 189–224.
  3. [3]P. Bastian, M. Altenbernd, N.-A. Dreier, C. Engwer, J. Fahlke, R. Fritze, M. Geveler, D. Gödd​eke, O. Iliev, O. Ippisch, J. Mohring, S. Müthing, M. Ohlberger, D. Rübbock, N. Shegunov, S. Turek, Exa-Dune—Flexible PDE Solvers, Numerical Methods and Applications, in: H.-J. Bungartz, S. Reiz, B. Uekermann, P. Neumann, W. E. Nagel (Eds.), Software for Exascale Computing - SPPEXA 2016-2019, Springer International Publishing, Cham, 2020, pp. 225–269.
  4. [4]M. Blind, M. Gao, D. Kempf, P. Kopper, M. Kurz, A. Schwarz, A. Beck, Towards Exascale CFD Simulations Using the Discontinuous Galerkin Solver FLEXI, preprint, arXiv: 2306.12891, 2023.arxiv.org/abs/2306.12891
  5. [5]P. Munch, K. Kormann, M. Kronbichler, hyper.deal: An Efficient, Matrix-free Finite-element Library for High-dimensional Partial Differential Equations, ACM Trans. Math. Softw. 47 (2021).arxiv.org/abs/2002.08110
  6. [6]B. Cockburn, J. Gopalakrishnan, R. Lazarov, Unified hybridization of discontinuous Galerkin, mixed, and continuous Galerkin methods for second order elliptic problems, SIAM Journal on Numerical Analysis 47 (2009) 1319–1365.DOI
  7. [7]X. Roca, C. Nguyen, J. Peraire, Scalable parallelization of the hybridized discontinuous Galerkin method for compressible flow, in: 21st AIAA Computational fluid dynamics conference, p. 2939.
  8. [8]J.-L. Lions, Y. Maday, G. Turinici, Résolution d’EDP par un schéma en temps parallèle, Comptes Rendus de l’Académie des Sciences - Series I - Mathematics 332 (2001) 661–668.
  9. [9]K. Burrage, Parallel Methods for ODEs, Advances in computational mathematics, Baltzer Science Publishers, 1997.DOI
  10. [10]M. J. Gander, S. Vandewalle, Analysis of the parareal time-parallel time-integration method, SIAM Journal on Scientific Computing 29 (2007-01-01).DOI
  11. [11]Y. Xia, J. Lou, H. Luo, J. Edwards, F. Mueller, OpenACC acceleration of an unstructured CFD solver based on a reconstructed discontinuous Galerkin method for compressible flows, International Journal for Numerical Methods in Fluids 78 (2015) 123–139.DOI
  12. [12]A. C. Kirby, D. J. Mavriplis, GPU-Accelerated Discontinuous Galerkin Methods: 30x Speedup on 345 Billion Unknowns, in: 2020 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7.arxiv.org/abs/2006.15698
  13. [13]M. Kronbichler, K. Kormann, Fast matrix-free evaluation of discontinuous Galerkin finite element operators, ACM Transactions on Mathematical Software (TOMS) 45 (2019) 1–40.DOI
  14. [14]N. Fehn, W. A. Wall, M. Kronbichler, A matrix-free high-order discontinuous Galerkin compressible Navier-Stokes solver: A performance comparison of compressible and incompressible formulations for turbulent incompressible flows, International Journal for Numerical Methods in Fluids 89 (2019) 71–102.DOI
  15. [15]D. Still, N. Fehn, W. A. Wall, M. Kronbichler, Matrix-Free Evaluation Strategies for Continuous and Discontinuous Galerkin Discretizations on Unstructured Tetrahedral Grids, preprint, arXiv: 2509.10226, 2025.arxiv.org/abs/2509.10226
  16. [16]R. Schussnig, N. Fehn, P. Munch, M. Kronbichler, Matrix-free higher-order finite element methods for hyperelasticity, Computer Methods in Applied Mechanics and Engineering 435 (2025) 117600.DOI
  17. [17]P. C. Africa, D. Arndt, W. Bangerth, B. Blais, M. Fehling, R. Gassmöller, T. Heister, L. Heltai, S. Kinneweg, M. Kronbichler, M. Maier, P. Munch, M. Schrester-Fleischhacker, J. P. Thiele, B. Turcksin, D. Wells, V. Yushutin, The deal.II library, Version 9.6, Journal of Numerical Mathematics 32 (2024) 369–380.
  18. [18]P. Bastian, M. Blatt, A. Dedner, N.-A. Dreier, C. Engwer, R. Fritze, C. Gräser, C. Grüninger, D. Kempf, B. Klöfkorn, M. Ohlberger, O. Sander, The Dune framework: Basic concepts and recent developments, Computers & Mathematics with Applications 81 (2021) 75–112. Development and Application of Open-source Software for Problems with Numerical PDEs.arxiv.org/abs/1909.13672
  19. [19]N. Krais, A. Beck, T. Bolemann, H. Frank, D. Flad, G. Gassner, F. Hindenlang, M. Hoffmann, T. Kuhn, M. Sonntag, C.-D. Munz, FLEXI: A high order discontinuous Galerkin framework for hyperbolic–parabolic conservation laws, Computers & Mathematics with Applications 81 (2021) 186–219. Development and Application of Open-source Software for Problems with Numerical PDEs.arxiv.org/abs/1910.02858
  20. [20]D. Amitai, A. Averbuch, S. Itzikowitz, E. Turkel, Asynchronous and corrected-asynchronous finite difference solutions of PDEs on MIMD multiprocessors, Numerical Algorithms 6 (1994) 275–296.DOI
  21. [21]D. Amitai, A. Averbuch, M. Israeli, S. Itzikowitz, E. Turkel, A survey of asynchronous finite-difference methods for parabolic PDEs on multiprocessors, Applied Numerical Mathematics 12 (1993) 27–45.DOI
  22. [22]D. A. Donzis, K. Aditya, Asynchronous finite-difference schemes for partial differential equations, Journal of Computational Physics 274 (2014) 370 – 392.DOI
  23. [23]A. Mittal, S. Girimaji, Proxy-equation paradigm: A strategy for massively parallel asynchronous computations, Physical Review E 96 (2017).DOI
  24. [24]K. Aditya, D. A. Donzis, Poster: Asynchronous Computing for Partial Differential Equations at Extreme Scales, in: Proceedings of the 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC ’12, IEEE Computer Society, Washington, DC, USA, 2012, p. 1444.DOI
  25. [25]K. Aditya, D. A. Donzis, High-order asynchrony-tolerant finite difference schemes for partial differential equations, Journal of Computational Physics 350 (2017) 550 – 572.DOI
  26. [26]S. K. Goswami, V. J. Matthew, K. Aditya, Implementation of low-storage Runge-Kutta time integration schemes in scalable asynchronous partial differential equation solvers, Journal of Computational Physics 477 (2023) 111922.DOI
  27. [27]K. Aditya, T. Gysi, G. Kwasniewski, T. Hoefler, D. A. Donzis, J. H. Chen, A scalable weakly-synchronous algorithm for solving partial differential equations, preprint, arXiv: 1911.05769, 2019.arxiv.org/abs/1911.05769
  28. [28]K. Kumari, D. A. Donzis, Direct numerical simulations of turbulent flows using high-order asynchrony-tolerant schemes: Accuracy and performance, Journal of Computational Physics 419 (2020) 109626.DOI
  29. [29]K. Kumari, E. Cleary, S. Desai, D. A. Donzis, J. H. Chen, K. Aditya, Evaluation of finite difference based asynchronous partial differential equations solver for reacting flows, Journal of Computational Physics 477 (2023) 111906.DOI
  30. [30]A. K. Arumugam, S. K. Goswami, N. R. Vadlamani, K. Aditya, Accuracy and scalability of asynchronous compressible flow solver for transitional flows, preprint, arXiv: 2506.03027, 2025.arxiv.org/abs/2506.03027
  31. [31]S. K. Goswami, K. Aditya, An asynchronous discontinuous-Galerkin method for solving PDEs at extreme scales, in: AIAA AVIATION 2022 Forum, p. 4165.DOI
  32. [32]S. K. Goswami, K. Aditya, An asynchronous discontinuous Galerkin method for massively parallel PDE solvers, Computer Methods in Applied Mechanics and Engineering 430 (2024) 117218.DOI
  33. [33]A. K. Arumugam, K. Aditya, An asynchronous discontinuous Galerkin method for combustion simulations, preprint, arXiv: 2501.01747, 2025.arxiv.org/abs/2501.01747
  34. [34]M. Kronbichler, P. Munch, D. Schneider, deal.II: The step-76 tutorial program, https://www.dealii.org/current/doxygen/deal.II/step_76.html, 2020.
  35. [35]C. A. Kennedy, M. H. Carpenter, R. Lewis, Low-storage, explicit Runge–Kutta schemes for the compressible Navier–Stokes equations, Applied Numerical Mathematics 35 (2000) 177–219.
  36. [36]M. Schäfer, S. Turek, F. Durst, E. Krause, R. Rannacher, Benchmark Computations of Laminar Flow Around a Cylinder, Vieweg+Teubner Verlag, Wiesbaden, pp. 547–566.

Paper details

vlnc:W_kw2d3fsn
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents