betaFeedback

WorksPhysics and Astronomy

Accuracy and scalability of asynchronous compressible flow solver for transitional flows

Abstract

To overcome the communication bottlenecks observed in state-of-the-art parallel time-dependent flow solvers at extreme scales, an asynchronous computing approach that relaxes communication and synchronization at a mathematical level was previously developed. This approach preserves high-order accuracy of computations near processing element boundaries using asynchrony-tolerant (AT) schemes while significantly improving the scalability. The numerical properties of the AT schemes were studied based on simple linear and nonlinear partial differential equations (PDEs) in previous works. Allowing asynchrony in numerical schemes can minimize communication overheads in a parallel setting in two ways: one that avoids communication over a few predetermined time steps, and the other that initiates communications without enforcing synchronization. In this study, the asynchronous algorithms are incorporated into the high-order compressible flow solver COMP-SQUARE, which solves practically relevant flow problems in complex geometries in a multi-block framework. The numerical efficacy and scalability of the two asynchronous algorithms are demonstrated for three test cases: isentropic advection of a vortex, the Taylor-Green vortex, and a much more sensitive case of the flow transitioning on a NACA0012 airfoil. Speed-ups of up to 4×4\times with respect to the baseline synchronous algorithm are observed in the scaling experiments performed on up to 18,432 cores. The results of this study demonstrate the applicability of AT schemes on established CFD solvers to improve scalability at extreme scales as the scientific computing environment moves to the exascale era.

Introduction

Direct numerical simulations (DNS) of practically relevant flow problems are computationally expensive and, therefore, require massive parallel computing resources. Current state-of-the-art DNS solvers often use high-order finite-difference schemes to approximate spatial derivatives and multistage schemes such as Runge-Kutta or multi-step schemes such as Adams-Bashforth for temporal integration [1, 2, 3]. While advancing in time, stencil operations are performed at grid points where the spatial derivatives are computed based on data from neighboring points. Implementing such methods in a parallel setting requires domain decomposition with a number of processing elements (PEs). This introduces data dependency on neighboring PEs, which enforces the need for data communication among PEs. These point-to-point communications are performed from the PE boundary points into the corresponding ghost/buffer points of neighboring PEs and are locally synchronized at every stage of the time integration.

At extreme scales, the communication and synchronization of data incur significant overheads, posing a major bottleneck to scalability, and rendering the standard synchronous approach prohibitively expensive. To minimize these communication overheads, several efforts have been made at different levels across hardware and software stacks [4]. At a mathematical level, different asynchronous approaches for partial differential equation (PDE) solvers were introduced in [5, 6, 7, 8, 9], where data synchronization between PEs is relaxed, and the solvers are allowed to proceed with computations using delayed data at PE boundaries regardless of the status of communications. The asynchronous finite difference method proposed in [7, 8] provides a general framework to analyze numerical properties with asynchrony, and shows that standard finite difference schemes are at most first-order accurate when delayed values are used in stencils. This issue was addressed by the development of asynchrony-tolerant (AT) schemes that use wider spatio-temporal stencils at PE boundaries and provide arbitrary orders of accuracy [10]. The asynchronous finite difference method can be implemented in solvers based on two parallel algorithms: first, the communication avoiding algorithm (CAA), which relaxes communications over regular intervals during time advancement, and second, the synchronization avoiding algorithm (SAA), which initiates communications at every time step but does not enforce explicit synchronization of data [11, 12]. The efficacy and scalability of these algorithms have been demonstrated using canonical problems, such as Burgers’ turbulence [11, 13] and compressible isotropic turbulence [12]. For instance, asynchronous algorithms achieved a maximum speed-up of 6.68×6.68\times relative to the synchronous algorithm at an extreme scale of 27,000 cores [13]. It should also be noted that when multistage time integration schemes, such as low-storage explicit Runge-Kutta (LSERK), are used, asynchronous algorithms can eliminate communication completely in the intermediate stages [13]. Furthermore, the numerical accuracy of the asynchronous finite difference method has been assessed on one-dimensional chemically reacting flows [14]. The asynchronous approach has also been extended to the discontinuous Galerkin (DG) method [15] and used to solve compressible Euler equations [16] and reacting flow equations [17].

The errors incurred at the PE boundaries due to the use of delayed data were observed to introduce high-frequency numerical oscillations. These numerical oscillations can potentially alter the flow structures and important features of transitional and turbulent flows. In this regard, the efficacy of high-order AT schemes in restricting these numerical errors to insignificant amounts must be investigated. To this end, the two asynchronous algorithms are implemented in the high-order compressible flow solver COMP-SQUARE to investigate the accuracy and scalability of the asynchronous approach on sensitive test cases involving transition to turbulence. COMP-SQUARE is a well established code that solves a wide range of practically relevant flow problems in complex geometries, such as transitional and turbulent boundary layers [18, 19], separated flows in aircraft engine intakes [20, 21, 22], airfoils [23] and cavities [24]. The remainder of this paper is organized as follows. Solver details, including the governing equations, standard numerical schemes, and parallelization method, are provided in § 2. The concept of asynchronous computing is described in § 3 followed by the parallel implementation of asynchronous solvers in § 4. The numerical experiments and results from three test cases, isentropic vortex advection, Taylor-Green vortex, and transitional flow over an airfoil, are detailed in § 5. Finally, the conclusions are presented in § 6.

COMP-SQUARE solver details

The high-order structured multiblock compressible flow solver COMP-SQUARE solves the unsteady non-dimensional mass, momentum, and energy conservation equations on the general curvilinear coordinates given by Eq. (1). Here U={ρ,ρu,ρv,ρw,ρEt}U=\{\rho,\rho u,\rho v,\rho w,\rho E_t\} is the vector of the conserved variables and J=∂(ξ,η,ζ)/∂(x,y,z)J=\partial(\xi,\eta,\zeta)/\partial(x,y,z) is the coordinate transformation Jacobian, ρ\rho is the density, uu, vv and ww are the components of the velocity, and EtE_t is the total specific energy. The advective fluxes F^\hat{F}, G^\hat{G}, H^\hat{H} and the viscous fluxes F^v\hat{F}_v, G^v\hat{G}_v, H^v\hat{H}_v are defined in the Appendix Appendix § 9 for completeness.

∂∂t(UJ)+∂F^∂ξ+∂G^∂η+∂H^∂ζ=1Re[∂F^v∂ξ+∂G^v∂η+∂H^v∂ζ](1)\frac{\partial}{\partial t}\left(\frac{U}{J}\right)+\frac{\partial\hat{F}}{\partial\xi}+\frac{\partial\hat{G}}{\partial\eta}+\frac{\partial\hat{H}}{\partial\zeta}=\frac{1}{Re}\left[\frac{\partial\hat{F}_v}{\partial\xi}+\frac{\partial\hat{G}_v}{\partial\eta}+\frac{\partial\hat{H}_v}{\partial\zeta}\right] \tag*{(1)}

The solver computes spatial derivatives using high-order explicit and compact (up to sixth-order accurate) finite-difference schemes. Specifically, the spatial derivative ϕ′\phi' of a general function ϕ\phi is computed using the generalized stencil

αϕi−1′+ϕi′+αϕi+1′=bϕi+2−ϕi−24Δξ+aϕi+1−ϕi−12Δξ,(2)\alpha\phi'_{i-1}+\phi'_i+\alpha\phi'_{i+1}=b\frac{\phi_{i+2}-\phi_{i-2}}{4\Delta\xi}+a\frac{\phi_{i+1}-\phi_{i-1}}{2\Delta\xi}, \tag*{(2)}

where α=0\alpha=0 for explicit schemes [25], and aa and bb determine the order of accuracy. High-frequency numerical oscillations that arise due to nonlinear interactions in the solution are suppressed using high-order Padé type low-pass filters (up to tenth-order accuracy) with a filter coefficient αf\alpha_f where −0.5<αf≤0.5-0.5<\alpha_f\leq0.5 [26]. The solver is equipped with classical fourth-order Runge–Kutta and low-storage explicit Runge-Kutta (LSERK) schemes for time integration [27, 28, 29]. For an ss stage LSERK method, the equations to compute Un+1U^{n+1} from UnU^n are given in Eq. (3), where Q(m)Q^{(m)} is the vector of intermediate variables, AmA_m and BmB_m are the coefficients at the mmth stage, where m=1,2,…,sm = 1, 2, \ldots, s such that U(0)≡UnU^{(0)} \equiv U^n and U(s)≡Un+1U^{(s)} \equiv U^{n+1}. The values of the coefficients used in this study are presented in Appendix Appendix B.

Q(m)=AmQ(m−1)+ΔtR(U(m−1))U(m)=U(m−1)+BmQ(m)(3)\begin{aligned} Q^{(m)} &= A_m Q^{(m-1)} + \Delta t R(U^{(m-1)}) \\ U^{(m)} &= U^{(m-1)} + B_m Q^{(m)} \tag*{(3)} \end{aligned}

The solver performs multiblock decomposition of the computational domain to preserve high fidelity and is especially important in problems involving complex geometries, such as turbine blades [19] and engine intakes [22]. The code is parallelized to scale on multiple central processing units (CPUs) using non-blocking communication subroutines from the message passing interface (MPI) framework. OpenACC directives are also incorporated to accelerate computations on graphical processing units (GPUs). Nevertheless, all simulations reported in this paper exploit the multi-CPU version of the solver.

Asynchronous computing approach

In parallel DNS implementations, the computational domain is decomposed into several subdomains that are mapped to several processing elements (PEs). The physical grid points in each subdomain can be classified into two groups based on the data dependency on communication: interior points and PE boundary points. The spatial derivatives at the interior points are computed using finite-difference stencils that contain physical grid points within the PE. On the other hand, the stencils for computing spatial derivatives at PE boundary points include grid points from other PEs, requiring data movement among PEs. Data communication is performed by sending solution values from PE boundary points and copying them into the ghost/buffer points of neighboring PEs. These communications are initiated between PEs and explicitly synchronized at every stage of time integration, referring to this procedure as the standard synchronous approach. At extreme scales, such communications and synchronizations result in significant overheads and, in turn, severely degrade the parallel performance.

Based on the asynchronous finite-difference method, the overhead associated with data communications can be reduced either by relaxing their synchronization or by periodically avoiding communication for some time steps. Thus, the solver can potentially perform computations using delayed/stale data in buffer points at the PE boundaries. In Figure 1(a), ϕin\phi_i^n is the function value at the left boundary point ii of PE 1 at time level nn, whereas ϕi−1n−k~\phi_{i-1}^{n-\tilde{k}} is the function value at the buffer point of PE 0 at time level n−k~n-\tilde{k}. k~\tilde{k} is the delay, where (⋅)(\cdot) denotes its random nature due to factors like performance variation and network topology in the parallel setting. Such an asynchronous approach was shown to be numerically consistent and stable, but only exhibits first-order accuracy irrespective of the spatial order of the standard finite difference schemes [7, 8]. In a subsequent study, asynchrony-tolerant (AT) schemes for computations at PE boundaries were proposed, which preserve high-order accuracy while operating on delayed data in buffers [10]. Using these schemes, the spatial derivatives at the PE boundary points are approximated based on extended spatio-temporal stencils consisting of multiple delayed time levels at buffer points. Furthermore, when multistage Runge–Kutta (RK) schemes are used for time integration, communications/synchronizations are relaxed over the intermediate stages [13]. This requires additional asynchrony-tolerant corrections at the buffer points to account for the fractional time-step advancement between RK stages. It should be noted that these additional computations using AT schemes are performed only at PE boundaries, which are typically a small fraction of the total grid points in a subdomain. Standard finite difference schemes are used at interior points, Eq. (2). Before we outline the procedure for implementing the asynchronous method, the necessary AT schemes that are used near the PE boundaries are described in the following sections.

Image not extracted

Figure 1. (a) An illustration of communications with relaxed synchronization at the interface between two processing elements (PEs). The delay in time levels at the buffer points is k~\tilde{k}. (b) Pattern of LSERK2-CD2-AT2 stencil computations at a left PE boundary point.

Asynchrony-tolerant (AT) schemes

PE boundary points: Let us consider the ddth spatial derivative of a function ϕ(ξ,t)\phi(\xi,t) expressed in a general spatio-temporal stencil

∂dϕn∂ξd∣i=∑j=−J1J2∑l=0L−1c~jlϕi+jn−l+O(Δξq),(4)\left.\frac{\partial^d\phi^n}{\partial\xi^d}\right|_i = \sum_{j=-J_1}^{J_2}\sum_{l=0}^{L-1}\widetilde{c}_j^l\phi_{i+j}^{n-l} + O(\Delta\xi^q), \tag*{(4)}

where j∈{−J1,…,J2}j \in\{-J_1,\ldots,J_2\}, J1J_1 and J2J_2 are the extents of the spatial stencil to the left and right of point ii, respectively, and l∈{0,…,L−1}l \in\{0,\ldots,L-1\} is the extent of the temporal stencil with delayed time levels n−ln-l bounded by a maximum allowable delay LL. O(Δξq)O(\Delta\xi^q) denotes that the approximation is qqth-order accurate in space. The Taylor series expansion for ϕi+jn−l\phi_{i+j}^{n-l} with respect to ϕin\phi_i^n on a general spatio-temporal stencil can be written as

ϕi+jn−l=∑λ=0∞∑θ=0∞ϕ(λ,θ)(jΔξ)λ(−lΔt)θλ!θ!,(5)\phi_{i+j}^{n-l} = \sum_{\lambda=0}^{\infty}\sum_{\theta=0}^{\infty}\phi^{(\lambda,\theta)}\frac{(j\Delta\xi)^\lambda(-l\Delta t)^\theta}{\lambda!\theta!}, \tag*{(5)}

where ϕ(λ,θ)\phi^{(\lambda,\theta)} denotes the partial derivative of ϕin\phi_i^n of order λ\lambda and θ\theta along ξ\xi and tt, respectively. In order to compute the spatial derivative based on such a hybrid stencil, necessary lower order terms must be eliminated upon substituting the Taylor series in Eq. (5) into the linear combination in Eq. (4). The lower order terms depend on the desired order of accuracy and the scaling exponent rr in the stability criterion (Δt∼Δξr)(\Delta t \sim\Delta\xi^r). The time integration scheme must be at least O(Δtq/r)O(\Delta t^{q/r}) accurate to obtain an overall accuracy of O(Δξq)O(\Delta\xi^q) in space. The coefficients c~jl\widetilde{c}_j^l in Eq. (4) can be determined by retaining the ddth-order derivative term and eliminating appropriate lower order terms with the constraints

∑j=−J1J2∑l=0L−1c~jl(jΔξ)λ(−lΔt)θλ!θ!={1for (λ,θ)=(d,0),0for λ+rθ<d+q;  (λ,θ)≠(d,0).(6)\sum_{j=-J_1}^{J_2}\sum_{l=0}^{L-1}\widetilde{c}_j^l\frac{(j\Delta\xi)^\lambda(-l\Delta t)^\theta}{\lambda!\theta!} = \begin{cases} 1 & \text{for }(\lambda,\theta)=(d,0),\\ 0 & \text{for }\lambda+r\theta<d+q;\;(\lambda,\theta)\ne(d,0). \end{cases} \tag*{(6)}

These constraints result in a linear system of equations to solve for the unknown coefficients c~jl\widetilde{c}_j^l that correspond to delayed time levels {n−k~,n−k~−1,…,n−k~−C+1}\{n-\widetilde{k}, n-\widetilde{k}-1,\ldots,n-\widetilde{k}-C+1\}, where k~\widetilde{k} and CC denote the latest delay and the number of time levels, respectively. For example, consider the schematic of the stencil at the left boundary of a PE shown in Figure 1(b). Let us approximate the first-order derivative using a second-order accurate scheme at the left PE boundary point ii. Stability criterion based on the advective Courant-Friedrichs-Lewy (CFL) Δt∼Δξr\Delta t \sim\Delta\xi^r (r=1r=1) is used. The unknown coefficients c~jl\widetilde{c}_j^l can be determined by solving Eq. (6) based on the required parameters (d,q,r)(d,q,r). The spatial derivative at the zeroth stage at time level nn at the left PE boundary point ii (denoted by gray filled circle) is computed based on a stencil that contains the solutions from time levels n−k~n-\widetilde{k}, n−k~−1n-\widetilde{k}-1 and n−k~−2n-\widetilde{k}-2 at the buffer point i−1i-1 (denoted by white circle). The AT scheme for the spatial derivative of ϕ\phi at the grid point ii can be approximated in the general form

∂ϕ∂ξ∣in≈c~(k~)Φ(k~),(7)\left.\frac{\partial\phi}{\partial\xi}\right|_i^n \approx\widetilde{\mathbf{c}}(\widetilde{k})\boldsymbol{\Phi}(\widetilde{k}), \tag*{(7)}

where c~(k~)=[c~k~,c~k~+1,…,c~k~+C−1]T\widetilde{\mathbf{c}}(\widetilde{k}) = [\widetilde{c}^{\widetilde{k}},\widetilde{c}^{\widetilde{k}+1},\ldots,\widetilde{c}^{\widetilde{k}+C-1}]^T is the vector containing the unknown coefficients and Φ(k~)\boldsymbol{\Phi}(\widetilde{k}) is the vector of spatial derivatives of ϕ\phi with delayed data at buffer points. For the parameters (d,q,r)=(1,2,1)(d,q,r)=(1,2,1), the second-order asynchrony-tolerant (AT2) scheme corresponds to

c~(k~)=[12(k~2+3k~+2)−12(k~2+2k~)12(k~2+k~)],Φ(k~)=[a(ϕi+1n−ϕi−1n−k~)2Δξa(ϕi+1n−ϕi−1n−k~−1)2Δξa(ϕi+1n−ϕi−1n−k~−2)2Δξ].\tilde{\mathbf{c}}(\tilde{k}) = \begin{bmatrix} \frac{1}{2}\left(\tilde{k}^{2}+3\tilde{k}+2\right)\\ -\frac{1}{2}\left(\tilde{k}^{2}+2\tilde{k}\right)\\ \frac{1}{2}\left(\tilde{k}^{2}+\tilde{k}\right) \end{bmatrix}, \quad \mathbf{\Phi}(\tilde{k}) = \begin{bmatrix} a\frac{\left(\phi_{i+1}^{n}-\phi_{i-1}^{n-\tilde{k}}\right)}{2\Delta\xi}\\ a\frac{\left(\phi_{i+1}^{n}-\phi_{i-1}^{n-\tilde{k}-1}\right)}{2\Delta\xi}\\ a\frac{\left(\phi_{i+1}^{n}-\phi_{i-1}^{n-\tilde{k}-2}\right)}{2\Delta\xi} \end{bmatrix}.

As illustrated in Figure 1(b), the spatial derivatives of flux functions at the left PE boundary point ii are computed using solutions from multiple delayed time levels at buffer point i−1i-1. Similarly, for a fourth-order central difference approximation, the parameters are (d,q,r)=(1,4,1)(d,q,r)=(1,4,1). The AT4 scheme for the spatial derivative at the left PE boundary point can be expressed in the same form as Eq. (7), with

c~(k~)=[124(k~4+10k~3+35k~2+50k~+24)−16(k~4+9k~3+25k~2+24k~)14(k~4+8k~3+19k~2+12k~)−16(k~4+7k~3+14k~2+8k~)124(k~4+6k~3+11k~2+6k~)],Φ(k~)=[bϕi+2n−ϕi−2n−k~4Δξ+aϕi+1n−ϕi−1n−k~2Δξbϕi+2n−ϕi−2n−k~−14Δξ+aϕi+1n−ϕi−1n−k~−12Δξbϕi+2n−ϕi−2n−k~−24Δξ+aϕi+1n−ϕi−1n−k~−22Δξbϕi+2n−ϕi−2n−k~−34Δξ+aϕi+1n−ϕi−1n−k~−32Δξbϕi+2n−ϕi−2n−k~−44Δξ+aϕi+1n−ϕi−1n−k~−42Δξ].\tilde{\mathbf{c}}(\tilde{k}) = \begin{bmatrix} \frac{1}{24}\left(\tilde{k}^{4}+10\tilde{k}^{3}+35\tilde{k}^{2}+50\tilde{k}+24\right)\\ -\frac{1}{6}\left(\tilde{k}^{4}+9\tilde{k}^{3}+25\tilde{k}^{2}+24\tilde{k}\right)\\ \frac{1}{4}\left(\tilde{k}^{4}+8\tilde{k}^{3}+19\tilde{k}^{2}+12\tilde{k}\right)\\ -\frac{1}{6}\left(\tilde{k}^{4}+7\tilde{k}^{3}+14\tilde{k}^{2}+8\tilde{k}\right)\\ \frac{1}{24}\left(\tilde{k}^{4}+6\tilde{k}^{3}+11\tilde{k}^{2}+6\tilde{k}\right) \end{bmatrix}, \quad \mathbf{\Phi}(\tilde{k}) = \begin{bmatrix} b\frac{\phi_{i+2}^{n}-\phi_{i-2}^{n-\tilde{k}}}{4\Delta\xi} +a\frac{\phi_{i+1}^{n}-\phi_{i-1}^{n-\tilde{k}}}{2\Delta\xi}\\ b\frac{\phi_{i+2}^{n}-\phi_{i-2}^{n-\tilde{k}-1}}{4\Delta\xi} +a\frac{\phi_{i+1}^{n}-\phi_{i-1}^{n-\tilde{k}-1}}{2\Delta\xi}\\ b\frac{\phi_{i+2}^{n}-\phi_{i-2}^{n-\tilde{k}-2}}{4\Delta\xi} +a\frac{\phi_{i+1}^{n}-\phi_{i-1}^{n-\tilde{k}-2}}{2\Delta\xi}\\ b\frac{\phi_{i+2}^{n}-\phi_{i-2}^{n-\tilde{k}-3}}{4\Delta\xi} +a\frac{\phi_{i+1}^{n}-\phi_{i-1}^{n-\tilde{k}-3}}{2\Delta\xi}\\ b\frac{\phi_{i+2}^{n}-\phi_{i-2}^{n-\tilde{k}-4}}{4\Delta\xi} +a\frac{\phi_{i+1}^{n}-\phi_{i-1}^{n-\tilde{k}-4}}{2\Delta\xi} \end{bmatrix}.

We note that the coefficient vector c~(k~)\tilde{\mathbf{c}}(\tilde{k}) remains the same for the right PE boundary point, while Φ(k~)\mathbf{\Phi}(\tilde{k}) is modified such that i+1i+1 and i+2i+2 are buffer points with delays whereas i−1i-1 and i−2i-2 are interior points. We also note that the spatial derivatives at PE boundary points obtained using AT schemes are based on an extended stencil in time alone, while the spatial extent remains the same relative to the standard central difference schemes. In general, the stencil could be expanded in both space and time. It should also be noted that introducing delays at PE boundaries gives rise to high frequency numerical oscillations, leading to stricter CFL conditions. However, the use of AT schemes improves the stability criterion to some extent compared to using standard schemes in the presence of delays [12].

Buffer points: In the absence of communications among PEs in the intermediate LSERK stages, asynchrony-tolerant (AT) schemes can be implemented at the buffer points in a naive manner by considering the fractional time step advancement at each RK stage in addition to the delayed levels across time steps (see Figure 1(b) for reference). However, this naive approach leads to poor accuracy (at most third-order accurate) as previously studied [13]. To overcome this issue, buffer points are updated at each RK stage using the delayed solution values by replacing the spatial derivative in Eq. (3) with a temporal derivative as shown in Eq. (8).

Qbuf(m)=AmQbuf(m−1)+ΔtRbuf(m−1),m=1,2,…,s=AmQbuf(m−1)+Δt∂U∂t∣buf(m−1).(8)\begin{aligned} Q_{\mathrm{buf}}^{(m)}=A_m Q_{\mathrm{buf}}^{(m-1)}+\Delta t R_{\mathrm{buf}}^{(m-1)},\qquad m=1,2,\ldots,s \\ &=A_m Q_{\mathrm{buf}}^{(m-1)}+\Delta t\left.\frac{\partial U}{\partial t}\right|_{\mathrm{buf}}^{(m-1)} . \tag*{(8)} \end{aligned}

The temporal derivative in Eq. (8) is to be computed at every RK stage using delayed solution values such that the necessary lower order terms are eliminated to retain higher order accuracy. The temporal derivative of a function ϕ\phi at stage e≡m−1e\equiv m-1 and at grid point ii can be written in terms of function values at previous time levels as expressed in Eq. (9).

∂ϕ∂t∣i(e)=∑l=L1L2c~lϕin−l,ϕin−l=∑θ=0∞(−lΔt)θθ!∂θϕ∂tθ∣in.(9)\left.\frac{\partial\phi}{\partial t}\right|_{i}^{(e)} =\sum_{l=L_1}^{L_2}\tilde{c}_{l}\phi_{i}^{n-l},\qquad \phi_{i}^{n-l}=\left.\sum_{\theta=0}^{\infty}\frac{(-l\Delta t)^{\theta}}{\theta!}\frac{\partial^{\theta}\phi}{\partial t^{\theta}}\right|_{i}^{n}. \tag*{(9)}

The Taylor series expansion of the LSERK scheme is expressed in Eq. (10), where v0e=1v_0^e=1, v1e=∑j=1eBj+∑j=2eAjBj+∑j=2e−1∑j=j+1eAjAj+1Bj+1+…+Be∑j=2eAjv_1^e=\sum_{j=1}^{e}B_j+\sum_{j=2}^{e}A_jB_j+\sum_{j=2}^{e-1}\sum_{j=j+1}^{e}A_jA_{j+1}B_{j+1}+\ldots+B_e\sum_{j=2}^{e}A_j, ..., and vee=∏j=1eBjv_e^e=\prod_{j=1}^{e}B_j.

ϕi(e)=v0eϕin+v1eΔt∂ϕ∂t∣in+…+vee(Δt)e∂eϕ∂te∣in.(10)\phi_i^{(e)}=v_0^e\phi_i^n+v_1^e\Delta t\left.\frac{\partial\phi}{\partial t}\right|_i^n+\ldots+v_e^e(\Delta t)^e\left.\frac{\partial^e\phi}{\partial t^e}\right|_i^n. \tag*{(10)}

The derivative of Eq. (10) with respect to time is

∂ϕ(e)∂t∣i=v0e∂ϕ∂t∣in+v1eΔt∂2ϕ∂t2∣in+…+vee(Δt)e∂e+1ϕ∂te+1∣in.(11)\left.\frac{\partial\phi^{(e)}}{\partial t}\right|_i=v_0^e\left.\frac{\partial\phi}{\partial t}\right|_i^n+v_1^e\Delta t\left.\frac{\partial^2\phi}{\partial t^2}\right|_i^n+\ldots+v_e^e(\Delta t)^e\left.\frac{\partial^{e+1}\phi}{\partial t^{e+1}}\right|_i^n. \tag*{(11)}

Equations 9 and 11 are equivalent and the coefficients are compared, giving the necessary constraints to satisfy O(Δξq)O(\Delta\xi^q) accuracy, as shown in Eq. (13).

∑j=0evje(Δt)j∂j+1ϕ∂tj+1=∑l=L1L2c~ l∑θ=0∞(−lΔt)θθ!∂θϕ∂tθ.(12)\sum_{j=0}^{e}v_j^e(\Delta t)^j\frac{\partial^{j+1}\phi}{\partial t^{j+1}}=\sum_{l=L_1}^{L_2}\tilde{c}^{\,l}\sum_{\theta=0}^{\infty}\frac{(-l\Delta t)^\theta}{\theta!}\frac{\partial^\theta\phi}{\partial t^\theta}. \tag*{(12)}
∑l=L1L2c~ l(−lΔt)θθ!={vje(Δt)jfor θ=j+1, j=0,…,e0for θ=0 and e+1<θ≤qr+1.(13)\sum_{l=L_1}^{L_2}\tilde{c}^{\,l}\frac{(-l\Delta t)^\theta}{\theta!}= \begin{cases} v_j^e(\Delta t)^j & \text{for }\theta=j+1,\ j=0,\ldots,e\\ 0 & \text{for }\theta=0\text{ and }e+1<\theta\leq\frac{q}{r}+1. \end{cases} \tag*{(13)}

Buffer update schemes: Equation 13 is solved for a given pair (q,r)(q,r) to obtain the coefficients of the AT buffer update schemes at the intermediate stages of the LSERK method. The formulae for computing the temporal derivative of ϕ\phi are listed in Appendix § 11 for the two-stage LSERK2 and the five-stage LSERK4 methods. We note that buffer updates are performed until the (s−1)(s-1)th stage which corresponds to e=s−2e=s-2.

Buffer extrapolation schemes: At the zeroth stage of the LSERK method, the solution at a buffer point is approximated as a polynomial extrapolation of solutions from multiple delayed time levels. The second-order accurate extrapolation scheme for ϕi(0)\phi_i^{(0)} in terms of the delayed function values is

ϕi(0)=12(k~2+3k~+2)ϕin−k~−(k~2+2k~)ϕin−k~−1+12(k~2+k~)ϕin−k~−2.(14)\phi_i^{(0)}=\frac{1}{2}(\tilde{k}^2+3\tilde{k}+2)\phi_i^{n-\tilde{k}}-(\tilde{k}^2+2\tilde{k})\phi_i^{n-\tilde{k}-1}+\frac{1}{2}(\tilde{k}^2+\tilde{k})\phi_i^{n-\tilde{k}-2}. \tag*{(14)}

Similarly, the fourth-order accurate extrapolation scheme for ϕi(0)\phi_i^{(0)} is

ϕi(0)=124(k~4+10k~3+35k~2+50k~+24)ϕin−k~−16(k~4+9k~3+25k~2+24k~)ϕin−k~−1+14(k~4+8k~3+19k~2+12k~)ϕin−k~−2−16(k~4+7k~3+14k~2+8k~)ϕin−k~−3+124(k~4+6k~3+11k~2+6k~)ϕin−k~−4.(15)\begin{aligned} \phi_i^{(0)}={}&\frac{1}{24}(\tilde{k}^4+10\tilde{k}^3+35\tilde{k}^2+50\tilde{k}+24)\phi_i^{n-\tilde{k}}-\frac{1}{6}(\tilde{k}^4+9\tilde{k}^3+25\tilde{k}^2+24\tilde{k})\phi_i^{n-\tilde{k}-1}\\ &+\frac{1}{4}(\tilde{k}^4+8\tilde{k}^3+19\tilde{k}^2+12\tilde{k})\phi_i^{n-\tilde{k}-2}-\frac{1}{6}(\tilde{k}^4+7\tilde{k}^3+14\tilde{k}^2+8\tilde{k})\phi_i^{n-\tilde{k}-3}\\ &+\frac{1}{24}(\tilde{k}^4+6\tilde{k}^3+11\tilde{k}^2+6\tilde{k})\phi_i^{n-\tilde{k}-4}. \tag*{(15)} \end{aligned}

Implementation of AT schemes

In this section, we explain the implementation of the LSERK-AT method that provides O(Δξq)O(\Delta\xi^q) accurate solutions in space. Let us consider the schematic of the LSERK2-CD2-AT2 method shown in Figure 1(b) for reference. CD2 denotes the standard second-order central difference scheme used at interior points while AT2 denotes the second-order asynchrony-tolerant schemes used at PE boundaries. Here, the interior points (denoted by black, filled circles) are independent of communications between PEs and therefore the spatial derivatives at these points can be computed using standard finite difference methods. The computation of spatial derivatives at PE boundary points (denoted by gray, filled circles) does indeed depend on data from neighboring PEs which is stored at the buffer points (denoted by white circles). In the first stage of the LSERK-AT method, the PE boundary points are updated using AT schemes. As discussed in § 3.1, Eq. (7) can be used to approximate the spatial derivative based on delayed data at buffer points. Meanwhile, the solution values U(0)U^{(0)} at buffer points are approximated to the current fractional time step with a polynomial extrapolation using delayed buffer data. In the subsequent stages, the buffer points are updated based on the modified update scheme in Eq. (8) that replaces spatial derivatives with temporal derivatives. These update schemes are specific to the LSERK scheme used, as they selectively eliminate lower order terms from the Taylor series such that the solution is O(Δtq/r)O(\Delta t^{q/r}) accurate after ss stages. Since the buffer points are updated using this accurate approach in the intermediate stages, the spatial derivatives at the PE boundary points are computed using standard finite difference schemes. We note that for an ss stage LSERK scheme, the AT buffer updates are performed until the (s−1)(s-1)th stage. The asynchronous solver requires additional memory to store the multi-level buffers of size C×NbufC \times N_{\mathrm{buf}} where CC and NbufN_{\mathrm{buf}} denote the number of delayed time levels and number of buffer points in the subdomain, respectively. It may appear that performing these AT corrections introduces additional computational overhead, however, these computations are fairly localized and exhibit high arithmetic intensity. In previous studies, the asynchronous solvers are observed to incur a small fraction of computational overhead but provide significant gains in communications [13].

Parallel implementation

The baseline synchronous algorithm is first described in this section for reference. The relaxation in communication/synchronization using asynchronous computing approach can be realized using two parallel algorithms. First, the communication avoiding algorithm, where communications are performed in a synchronous manner but once every few time steps. Second, the synchronization avoiding algorithm, where data communications are initiated but the explicit synchronization is not enforced.

Synchronous algorithm (SA)

In the standard synchronous algorithm (SA) based on Runge-Kutta time integration schemes, an additional stage loop within the main time loop is incorporated to advance the RK stages. As described in Algorithm 1, upon updating the solution at each RK stage, data communication between PEs is performed, where the solutions at the PE boundary points are copied to a send buffer and a send request is initiated using the MPI_Isend subroutine. Meanwhile, the neighboring PE initiates the receive request using the MPI_Irecv subroutine. The synchronization of communications is ensured by calling the blocking MPI_Wait subroutine, and the received data are copied to the buffer points. The algorithm proceeds to the next RK stage of time advancement only after all buffer points are updated. An additional optimization for communication overheads in this synchronous algorithm can include the overlap between the computations at the interior points and communications near the PE boundaries.

Algorithm 1 Synchronous algorithm (SA)
1: for n=1,…,nstepsn = 1, \ldots, nsteps do
2: for m=1,…,sm = 1, \ldots, s do
3: U(m)=f(U(m−1))U^{(m)} = f(U^{(m-1)})
4: Usend←UbndU_{\mathrm{send}} \leftarrow U_{\mathrm{bnd}}
5: Initiate MPI_Isend and MPI_Irecv requests
6: Call blocking MPI_Wait
7: Ubuf←UrecvU_{\mathrm{buf}} \leftarrow U_{\mathrm{recv}}
8: end for
9: end for

Table 1.

Communication avoiding algorithm (CAA)

The communication avoiding algorithm (CAA) reduces data movement among PEs by skipping communication for a finite number of time steps over regular intervals. As explained in Algorithm 2, the synchronous algorithm is carried out for the first CC time steps to populate the buffer points with data of CC time levels, followed by the asynchronous algorithm for the remaining time steps. The spatial derivatives at the PE boundaries are computed using the AT schemes and multi-level buffer data. In the CAA, communications are performed based on the condition associated with n%Ln\%L where nn and LL denote the nnth time step and the maximum allowable delay, respectively. At every time step, the latest delay is incremented by one level (k~←k~+1\tilde{k} \leftarrow\tilde{k} + 1) across all PEs. If 0≤n%L≤C−10 \le n\%L \le C - 1, the solutions at the PE boundary points are copied to an auxiliary send buffer with dimension CC in time. If n%L==C−1n\%L == C - 1, the auxiliary buffer data are communicated to the receiving buffers, and the delay is reset (k~←0\tilde{k} \leftarrow0). This results in uniform and periodic behavior of delays across all PEs.

Algorithm 2 Communication avoiding algorithm (CAA)
1: for n=1,…,Cn = 1, \ldots, C do ▷ Synchronous loop for first CC steps
2: for m=1,…,sm = 1, \ldots, s do
3: U(m)=f(U(m−1))U^{(m)} = f(U^{(m-1)})
4: Usend←UbndU_{\mathrm{send}} \leftarrow U_{\mathrm{bnd}}
5: Initiate MPI_Isend and MPI_Irecv requests
6: Call blocking MPI_Wait
7: Ubuf←UrecvU_{\mathrm{buf}} \leftarrow U_{\mathrm{recv}}
8: end for
9: end for
10: for n=C+1,…,nstepsn = C + 1, \ldots, nsteps do ▷ Asynchronous loop
11: for m=1,…,sm = 1, \ldots, s do
12: Ubuf(m)=f(Ubuf(m−1),Ubufn−k~,…,Ubufn−k~−C+1)U_{\mathrm{buf}}^{(m)} = f(U_{\mathrm{buf}}^{(m-1)}, U_{\mathrm{buf}}^{n-\tilde{k}}, \ldots, U_{\mathrm{buf}}^{n-\tilde{k}-C+1})
13: U(m)=f(U(m−1))U^{(m)} = f(U^{(m-1)})
14: end for
15: Update delay: k~=k~+1\tilde{k} = \tilde{k} + 1
16: if 0≤n%L≤C−10 \le n\%L \le C - 1 then
17: Usend←UbndU_{\mathrm{send}} \leftarrow U_{\mathrm{bnd}}
18: end if
19: if n%L==C−1n\%L == C - 1 then
20: Initiate MPI_Isend and MPI_Irecv requests
21: Call blocking MPI_Wait
22: Ubuf←UrecvU_{\mathrm{buf}} \leftarrow U_{\mathrm{recv}}
23: Reset delay: k~=0\tilde{k} = 0
24: end if
25: end for

Table 2.

Synchronization avoiding algorithm (SAA)

The synchronization avoiding algorithm (SAA) relaxes the communication overhead by not enforcing synchronization, such that the solver proceeds with computations regardless of the status of their completion. As shown in Algorithm 3, the first CC time steps are executed in a synchronous manner to populate the buffer points, followed by the asynchronous loop, where non-blocking MPI_Isend and MPI_Irecv requests are initiated at every time step. The status of the communication requests and the latest delay are obtained using the MPI_Test subroutine. If the latest delay reaches the maximum allowable value LL or if the buffer point data are not from consecutive time levels, all existing communications are locally synchronized using MPI_Wait, and the delay is reset. The behavior of delays in SAA depends on several factors, such as workload balancing among PEs, the network topology of the architecture, and the communication library used. These factors result in stochastic and non-uniform behavior of delays across PEs.

Algorithm 3 Synchronization avoiding algorithm (SAA)

  1. for n=1,…,Cn = 1, \ldots, C do ▷ Synchronous loop for first CC steps

  2. for m=1,…,sm = 1, \ldots, s do

  3. U(m)=f(U(m−1))U^{(m)} = f(U^{(m-1)})

  4. Usend←UbndU_{\mathrm{send}} \leftarrow U_{\mathrm{bnd}}

  5. Initiate MPI_Isend and MPI_Irecv requests

  6. Call blocking MPI_Wait

  7. Ubuf←UrecvU_{\mathrm{buf}} \leftarrow U_{\mathrm{recv}}

  8. end for

  9. end for

  10. for n=C+1,…,nstepsn = C + 1, \ldots, n_{\mathrm{steps}} do ▷ Asynchronous loop

  11. for m=1,…,sm = 1, \ldots, s do

  12. Ubuf(m)=f(Ubuf(m−1),Ubufn−k~,…,Ubufn−k~−C+1)U_{\mathrm{buf}}^{(m)} = f(U_{\mathrm{buf}}^{(m-1)}, U_{\mathrm{buf}}^{n-\tilde{k}}, \ldots, U_{\mathrm{buf}}^{n-\tilde{k}-C+1})

  13. U(m)=f(U(m−1))U^{(m)} = f(U^{(m-1)})

  14. end for

  15. Initiate MPI_Isend and MPI_Irecv requests

  16. Check communication status using MPI_Test and get latest delay k~\tilde{k}

  17. if k~>L−1\tilde{k} > L - 1 OR time levels not consecutive then

  18. Synchronize communication using MPI_Wait

  19. Reset delay k~=0\tilde{k} = 0

  20. end if

  21. end for

The performance evaluations of the above parallel algorithms are carried out on two supercomputers. First, PARAM Pravega at SERC, IISc where each compute node is built using two Intel Xeon Cascade 8268 2.9 GHz processors with 24 cores each and contains 192 GB RAM, connected via Infiniband interconnect. Second, PARAM Rudra at IUAC, Delhi where each compute node is built using two Intel Xeon Gold 6240R 2.4 GHz processors with 24 cores each and contains 192 GB RAM, connected via Infiniband interconnect.

Results

The results of the numerical experiments of the three test cases are presented in this section. The two-dimensional isentropic vortex advection has an exact solution and is therefore a suitable problem for verifying the order of accuracy of the asynchronous solver. The three-dimensional Taylor-Green vortex is a transitional flow problem that is widely solved to assess the accuracy of high-order schemes. The breakdown of large-scale structures to smaller scales is sensitive to velocity gradients, and the efficacy of asynchronous solvers in capturing these small-scale structures is investigated. These two test cases are also considered to demonstrate the scalability of asynchronous algorithms by performing strong and weak scaling studies. The third test case is the transitional flow over a NACA0012 airfoil and is considered for evaluating the effect of numerical errors introduced at near wall structures that could perturb the location of the transition to turbulence. Scalability is demonstrated using a strong scaling study to assess the performance under non-uniform workload balancing, commonly associated with simulations of practically relevant problems. For reference, the algorithms used in the numerical experiments are as follows: synchronous algorithm (SA), communication avoiding algorithm (CAA-AS) and synchronization avoiding algorithm (SAA-AS) that use standard schemes in asynchrony, and communication avoiding algorithm (CAA-AT) and synchronization avoiding algorithm (SAA-AT) that implement asynchrony-tolerant schemes at PE boundaries. The numerical schemes used for the algorithms are listed in Table 1.

CaseAlgorithmTime steppingSpatial derivativesBuffer point updatesOverall order of accuracy
InteriorPE boundary
1SALSERK2CD2CD2-2
2CAA-AS, SAA-ASLSERK2CD2CD2-1
3CAA-AT, SAA-ATLSERK2CD2AT2AT22
4SALSERK4CD4CD4-4
5CAA-AS, SAA-ASLSERK4CD4CD4-1
6CAA-AT, SAA-ATLSERK4CD4AT4AT44

Table 1 (PDF p. 10). List of schemes in the three algorithms used in numerical experiments, with the expected overall orders of accuracy.

Isentropic vortex advection

The two-dimensional isentropic vortex advection is an inviscid flow problem with periodic boundary conditions. The computational domain is defined as −8≤x≤8-8 \le x \le8 and −8≤y≤8-8 \le y \le8, with the Mach number of the driving flow M=0.1M = 0.1. The vortical flow field is initialized as a Gaussian profile centered at (xc,yc)(x_c,y_c) [26], with the primitive quantities defined as

u=1−β(y−yc)R2exp⁡(−Ω22)v=β(x−xc)R2exp⁡(−Ω22)p∞−p=ρβ22R2exp⁡(−Ω2)Ω2=(x−xc)2+(y−yc)2R2,(16)\begin{aligned} u &= 1 - \frac{\beta(y-y_c)}{R^2}\exp\left(-\frac{\Omega^2}{2}\right) \\ v &= \frac{\beta(x-x_c)}{R^2}\exp\left(-\frac{\Omega^2}{2}\right) \\ p_\infty-p &= \frac{\rho\beta^2}{2R^2}\exp(-\Omega^2) \\ \Omega^2 &= \frac{(x-x_c)^2+(y-y_c)^2}{R^2}, \tag*{(16)} \end{aligned}

where uu, vv, pp and RR denote the xx and yy velocity components, pressure, and vortex core radius, respectively. The non-dimensional vortex strength parameter β/(U∞R)\beta/(U_\infty R) is chosen as 0.02. Figure 2(a) plots the contours of the vorticity magnitude at t=0t = 0 on the domain that is divided into three PEs in the xx direction, with PE boundaries denoted by the white dashed lines. The flow is evolved for an entire advection cycle (t=16t = 16), ensuring that the vortex crosses all three PE boundary interfaces, including the periodic interface, and returns to its initial position. A maximum allowable delay L=8L = 8 is considered for the asynchronous numerical simulations. Figure 2(b), (Figure 2c), and (Figure 2d) show the contours of the error in the vorticity magnitude (computed against the analytical solution) corresponding to SA, CAA-AS, and CAA-AT, respectively. Time steps of 2×10−32 \times10^{-3}, 1×10−31 \times10^{-3} and 1×10−31 \times10^{-3} are used with SA, CAA and SAA, respectively, at Nx=60N_x = 60 and are proportionately reduced along with Δx\Delta x to satisfy the stability criterion. The error contours for CAA-AT are very similar to those of SA, whereas larger errors are evident for CAA-AS, particularly near the PE boundaries. To quantify the overall accuracy, the mean error ⟨E⟩\langle E\rangle in density, which is obtained by averaging the RMS error over all grid points, across multiple independent simulations to account for the randomness in the delays, at various grid resolutions is considered. From the numerical experiments, the second- (schemes in cases 1 and 3 in Table 1) and fourth-order (schemes in cases 4 and 6 in Table 1) of CAA-AT and SAA-AT are verified, as shown in the error convergence plot in Figure 3(a). However, the errors for CAA-AS and SAA-AS (cases 2 and 5 in Table 1) collapse to the first-order, thereby confirming the earlier predictions [8].

Image not extracted

Figure 2. (a) Vorticity magnitude contours of initial flow field. Instantaneous contours of errors in vorticity magnitude after one advection cycle for (b) synchronous algorithm (SA), (c) communication avoiding algorithm with standard schemes (CAA-AS), and (d) communication avoiding algorithm with asynchrony-tolerant schemes (CAA-AT), solved with Nx=Ny=240N_x=N_y=240 and L=8L=8 using the schemes in cases 4, 5, and 6, respectively, in Table 1. The dashed white lines represent processing element (PE) boundaries.

Image not extracted

Figure 3. (a) Order convergence plot for density error for all the cases in Table 1, and (b) energy spectra of vv velocity profile at the y=0y=0 line for cases 4, 5, and 6 in Table 1 with Nx=240N_x=240 and L=8L=8. The different lines indicate: SA (solid purple), CAA-AS (dash-dotted red), SAA-AS (dashed blue), CAA-AT (dash-dotted green) and SAA-AT (dashed yellow).

The accuracy of the numerical solutions is further examined by plotting the energy spectra since the errors incurred due to communication delays at the PE boundaries are localized in space and are expected to affect the solution at high wavenumbers. In Figure 3(b), plotted against the wave number κx\kappa_x are the energy spectra (v^2(κx))(\hat{v}^2(\kappa_x)) of vv-velocity along the y=0y = 0 line obtained from the numerical solutions computed using the cases 4, 5 and 6 in Table 1. In general, the physical spectra remain approximately constant at lower wavenumbers and begin decaying as κx\kappa_x increases. For the baseline SA, the physical spectrum is captured until κx≈30\kappa_x \approx30 and the energy accumulated at higher wavenumbers is due to round-off errors. In comparison, the spectra of CAA-AS and SAA-AS start deviating from those of SA at a smaller wavenumber κx≈15\kappa_x \approx15 and contain similar levels of energy, which are more than 10 orders of magnitude higher than those of SA. These deviations are a consequence of high-frequency numerical errors in the solution due to asynchrony at the PE boundaries. Meanwhile, the spectrum of SAA-AT deviates from that of SA at κx≈20\kappa_x \approx20 with slightly less energy contributions at higher wavenumbers. CAA-AT resolves the solution accurately until κx≈25\kappa_x \approx25 and contains similar higher wavenumber behavior to SA. The difference between the spectra of SAA-AT and CAA-AT can

Image not extracted

Figure 4. Plots of error in uu-velocity after t=4t=4 for the isentropic vortex advection solved on a 204822048^2 grid against (a) maximum allowable delay LL on 1024 PEs and (b) number of PEs PP with L=4L=4 for CAA-AT corresponding to case 3 in Table 1. Dashed lines in (a) and (b) represent best fit cubic polynomial in LL, and reference linear polynomial respectively.

be attributed to the randomness of the communication delays in SAA-AT, which agrees with the results of previous studies [12].

Unlike the synchronous algorithm, the error in the asynchronous algorithms depends on simulation parameters such as the maximum allowable delay (LL) and number of PEs (PP), in addition to the resolution. The error scaling relation ⟨Eˉ⟩∝(P/N)Δxq∑m=1Cγmk~m\langle\bar{E}\rangle\propto(P/N)\Delta x^q \sum_{m=1}^{C}\gamma_m\tilde{k}^m was derived in a previous study based on theoretical analyses [10], where k~\tilde{k} and γm\gamma_m denote the mean delay and the coefficient of its mmth moment, respectively. To validate this error scaling, simulations are performed on a 204822048^2 grid with two-dimensional domain decomposition, till t=4t = 4 using Case 3 in Table 1, considering different values of PP and LL. For CAA-AT, the delays are periodic and follow a uniform distribution. In the case considered here, the number of delayed time levels in the AT stencil is C=3C = 3 and the error therefore scales as a cubic polynomial in LL and linearly with PP. Figure 4(a) plots the variation of uu-velocity error against LL with a fixed value of P=1024P = 1024, while Fig. 4(b) plots the variation of error against PP with a fixed value of L=4L = 4. A cubic polynomial is fit to the plot in Fig. 4(a) while a line is fit to the plot in Fig. 4(b), agreeing with the predictions.

Image not extracted

Figure 5. Strong scaling results of isentropic vortex advection on a 16384216384^2 grid with cases 4 and 6 in Table 1. (a) Total execution time, and (b) split between time taken for computation (solid lines) and communication (dashed lines) against number of PEs, for SA (blue), CAA-AT (red) and SAA-AT (green). L=10L = 10 for CAA-AT and SAA-AT.

The computational performance of the asynchronous algorithms is evaluated using a strong scaling study, in which the overall problem size is kept fixed and the number of PEs is increased. Here, a two-dimensional grid of size 16384×1638416384 \times16384 is decomposed into subdomains along both xx and yy directions ensuring uniform workload distribution. The execution times along with the contribution from computations and communications are recorded by solving for 100 time steps using the cases 4 and 6 in Table 1, with a maximum allowable delay of 10. This procedure is repeated over 10 independent trials and averaged to account for performance variation. Figure 5(a) plots the total execution time against the number of CPU cores. The speed-ups of CAA-AT and SAA-AT relative to SA at each scale are shown in the figure. It appears that the total execution time of SA deviates early from ideal scaling (denoted by the black dashed line with slope -1) and plateaus at 8192 cores, while both CAA-AT and SAA-AT continue to remain close to the ideal scaling line. In fact, both asynchronous algorithms achieve a speed-up of up to 3× compared with the baseline SA-based solver at the extreme scale. This can be explained through the distribution of computation and communication times for different numbers of cores, as shown in Fig. 5(b). For all three cases, the computation time drops linearly with an increase in the PEs. Note that CAA-AT and SAA-AT incur a small additional overhead in computation compared to SA. Meanwhile, the communication cost of SA is significantly greater than that of the two asynchronous algorithms. Furthermore, the communication time overwhelms the computation time at 2048 cores for SA, where the scaling of the total execution time plateaus in part (a) of the figure. These plots demonstrate the significantly reduced communication costs associated with the asynchronous algorithms, resulting in improved scalability.

Taylor-Green vortex

The Taylor-Green vortex (TGV) is a widely considered benchmark problem for assessing the numerical accuracy of high-order schemes. This test case contains key physical phenomena such as vortex dynamics, transition to turbulence and turbulent dissipation, facilitated by an initial periodic flow defined in Eq. (17) based on [30, 31].

u=U0sin⁡(x/L0)cos⁡(y/L0)cos⁡(z/L0)v=U0cos⁡(x/L0)sin⁡(y/L0)cos⁡(z/L0)w=0p=ρ0U02[1γM02+116(cos⁡(2x/L0)+cos⁡(2y/L0))(cos⁡(2z/L0)+2)](17)\begin{aligned} u &= U_0 \sin(x/L_0)\cos(y/L_0)\cos(z/L_0) \\ v &= U_0 \cos(x/L_0)\sin(y/L_0)\cos(z/L_0) \\ w &= 0 \\ p &= \rho_0 U_0^2 \left[\frac{1}{\gamma M_0^2}+\frac{1}{16}\left(\cos(2x/L_0)+\cos(2y/L_0)\right)\left(\cos(2z/L_0)+2\right)\right] \tag*{(17)} \end{aligned}

The domain has dimensions Lx×Ly×Lz=2π×2π×2πL_x \times L_y \times L_z = 2\pi\times2\pi\times2\pi with periodic boundary conditions. The Mach number and Reynolds number are taken to be M0=0.1M_0 = 0.1 and Re=ρ0U0L0/μ0=1600Re = \rho_0 U_0 L_0/\mu_0 = 1600, to validate with the studies carried out in the literature [30, 31]. A 5123512^3 grid is divided uniformly in all three directions into 512 PEs and is solved using the SA, CAA-AS, and CAA-AT algorithms with schemes in cases 4, 5, and 6 from Table 1, respectively. Tenth-order implicit filtering with a filter coefficient αf=0.499\alpha_f = 0.499 is used to prevent numerical oscillations and ensure the stability of the solver. The maximum allowable delays (LL) of values 2 and 5 are considered for the CAA-AS and CAA-AT, respectively. Note that because AT schemes provide better stability relative to standard schemes with delays (AS), a greater LL can be imposed. The simulations are performed in the following manner: the standard synchronous algorithm (SA) is used over the time interval 0≤t/tc≤80 \leq t/t_c \leq8 with a time step 1.0×10−31.0 \times10^{-3}, where tc=L0/U0t_c = L_0/U_0 is the characteristic convective time. The checkpoint at t/tc=8t/t_c = 8 is used to initialize the flow, which is solved with SA, CAA-AS, and CAA-AT using a smaller time step 1.0×10−41.0 \times10^{-4} due to the stability constraints associated with the asynchronous computing approach, particularly when standard schemes are employed with delays. The time interval to perform the comparison study is chosen to be 8≤t/tc≤128 \leq t/t_c \leq12, where the enstrophy reaches its peak value due to the vortex interactions and the transition to turbulence.

The volume averaged kinetic energy, K=1/(2ρ0V)∫Vρuiui dVK = 1/(2\rho_0 V)\int_V \rho u_i u_i\,dV, and enstrophy, ϵ=1/(2ρ0V)∫Vρωiωi dV\epsilon= 1/(2\rho_0 V)\int_V \rho\omega_i \omega_i\,dV, are computed over this time interval, where ωi\omega_i is vorticity and VV is the volume of the domain. The evolution of KK and ϵ\epsilon are plotted in Figure 6(a) and (b), respectively, for SA, CAA-AS, and CAA-AT, along with the results from pseudo-spectral method-based solutions [32] for reference. CAA-AS is largely in agreement with SA, except for minor deviations in the evolution of enstrophy and a sudden initial spike in kinetic energy. On the other hand, CAA-AT is in excellent agreement with SA throughout the simulation period, and both are similar to the reference results.

Image not extracted

Figure 6. Time evolution of volume-averaged (a) kinetic energy and (b) enstrophy on 5123512^3 grid for the Taylor-Green vortex using cases 4, 5, and 6 in Table 1. Maximum allowable delay LL is 2 and 5 for CAA-AS and CAA-AT respectively.

We recall again that errors due to communication delays are introduced in a localized manner near PE boundaries; hence, it is important to inspect the flow structures with particular emphasis on their behavior near PE boundaries. The solution at t/tc=9t/t_c = 9 is considered for the analysis because, as shown in Figure 6(b), the enstrophy at this instant is close to its maximum. The pressure contours in Figure 7(a) highlight the spurious pressure waves throughout the domain in CAA-AS, which can be attributed to the communication delays. The pressure fluctuations in the global flow field are expected to affect the velocity fields and their gradients. However, the contours of CAA-AT are accurately represented without any visible spurious oscillations and are similar to those of SA. Figure 7(b) plots the magnified contours of the vorticity magnitude around a key vortical structure at the x=0x = 0 plane that has been studied extensively in the literature [32]. A close inspection of the contour lines near the PE boundaries (denoted by the dashed black line at the left boundary) shows distortion of the vortical structures in the case of CAA-AS. The contours of CAA-AT, on the other hand, are in good agreement with those of SA, highlighting the effect of AT schemes in reducing numerical errors due to communication delays, especially near the PE boundaries.

Image not extracted

Figure 7. (a) Global contours of pressure, and (b) magnified contours of vorticity magnitude around a vortical structure at the x=0x = 0 plane, at t/tc=9t/t_c = 9 with SA, CAA-AS and CAA-AT corresponding to cases 4, 5 and 6 in Table 1. Maximum allowable delay LL is 2 and 5 for CAA-AS and CAA-AT respectively. The dashed black lines represent processing element (PE) boundaries.

The scalability of the asynchronous algorithms in the case of a three-dimensional domain decomposition is demonstrated using strong and weak scaling experiments with the TGV case. In the strong scaling study, grids resolutions of 2563256^3, 5123512^3 and 102431024^3 are decomposed into subdomains along all three directions and solved on different core counts. In the weak scaling study, the number of grid points per PE is kept fixed, whereas the number of PEs is increased. Ideally, the overall problem is expected to be solved at a constant time irrespective of the number of PEs. Two configurations, 32332^3 and 64364^3 grid points per PE, are considered in this study. Figure 8 plots the total execution time recorded for 100 time steps against the number of PEs for strong and weak scaling. A maximum allowable delay L=10L = 10 is considered for CAA-AT and SAA-AT in both studies. The experiments are performed over 10 independent trials, and the average execution time is reported to account for the variation in performance. In strong scaling (see Figure 8(a)), the standard SA begins to deviate from ideal scaling (denoted by dashed black lines of −1-1 slope) and plateaus out at 2048 cores, whereas CAA-AT and SAA-AT lie closer to ideal scaling at the same scale. Moreover, the execution time for SA increases at extreme scales due to the overwhelming communication/synchronization overheads, as discussed in the scaling analysis in § 5.1. Speed-ups of 4.26×4.26\times and 3.54×3.54\times relative to SA are observed for CAA-AT and SAA-AT, respectively, at the extreme scale of 8192 cores. Similarly, in the weak scaling plot in Figure 8(b), SA starts deviating significantly from the ideal line at scales of 512 and 1024 cores for 32332^3 and 64364^3 grid points per core, respectively, whereas the asynchronous algorithms again scale better than SA for both the configurations. Here, CAA-AT and SAA-AT achieve maximum speed-up of 2.91×2.91\times and 2.64×2.64\times, respectively, relative to SA. In both strong and weak scaling, CAA-AT achieves slightly better speed-ups compared to SAA-AT at the extreme scales, which is consistent with the trend observed in previous studies [13].

Image not extracted

Figure 8. (a) Strong scaling results for resolutions of 2563256^3, 5123512^3 and 102431024^3, and (b) weak scaling results for resolutions of 32332^3 and 64364^3 per PE, for the Taylor-Green vortex, with SA, CAA-AT and SAA-AT (see cases 4 and 6 in Table 1). Maximum allowable delay L=10L = 10 for CAA-AT and SAA-AT.

Flow over NACA0012 airfoil

In addition to the canonical test cases discussed in the previous sections, we now consider a practically relevant case: flow around a NACA0012 airfoil oriented at an angle of attack. A chord Reynolds number Rec=cU∞/ν∞=2×105Re_c = cU_\infty/\nu_\infty= 2 \times10^5 is considered, where cc, U∞U_\infty and ν∞\nu_\infty are the chord length, freestream velocity and viscosity at freestream conditions. Mach number M=0.1M = 0.1 and angle of attack 4∘4^\circ are considered to validate with the results from the high-fidelity simulations carried out in [33]. The 3D computational domain is represented by a C-grid containing approximately 17 million grid points, and is divided into 60 PEs along the streamwise and wall-normal directions, imposing periodicity in the spanwise direction. Inflow and outflow conditions are imposed at the far-field boundaries and a no-slip adiabatic boundary condition is imposed on the wall. Numerical simulations are carried out using SA and CAA-AT (with L=1L = 1) corresponding to cases 4 and 6, respectively, in Table 1. Figure 9(a) and Figure 9(b) respectively show the instantaneous iso-surfaces of QQ (= 10) for SA and CAA-AT, colored by the uu-velocity. Some of the PE boundaries are highlighted as translucent planes normal to the airfoil surface. The boundary layer on the airfoil suction surface is laminar near the leading edge and separates at x/c≈0.3x/c \approx0.3, resulting in the formation of full-span

Image not extracted

Figure 9. Instantaneous iso-surfaces of Q=10Q = 10 for the flow around NACA0012 airfoil at AoA 4° with (a) SA and (b) CAA-AT (cases 4 and 6, respectively, in Table 1), colored by contours of u/U∞u/U_{\infty}. Maximum allowable delay L=1L = 1 for CAA-AT.

Kelvin-Helmholtz rollers. Transition to turbulence is driven via the secondary instabilities developing over these KH rollers, resulting in small coherent structures and hairpin vortices. Notice that the iso-surfaces corresponding to CAA-AT are qualitatively identical to those of SA, indicating that the asynchronous solver consistently captures the separation-induced transition over the suction surface of the airfoil. Figure 10(a,Figure 10b) further compares the contours of the pressure coefficient and the turbulent kinetic energy obtained from the flow field averaged in time and span. Here, colored contours and lines represent the predictions from SA and CAA-AT, respectively. The pressure coefficient (cpc_p) on the airfoil surface and the streamwise variation of the maximum turbulent kinetic energy (TKEmax⁡_{\max}) are compared in Figure 10(c,Figure 10d). CAA-AT is in excellent agreement with SA in capturing the pressure distribution, especially near the transition region, whereas good agreement is observed in the case of TKE. It should be noted that both schemes predict a consistent transition location and TKEmax⁡_{\max}. This demonstrates the robustness of the CAA-AT schemes in predicting sensitive phenomena like transition, while achieving higher scalability than SA.

Image not extracted

Figure 10. Contours of (a) pressure coefficient (cpc_p) and (b) turbulent kinetic energy (TKE, magnified near the suction surface) obtained from time and span averaged fields. The colored contours in the background and the overlaid dashed lines correspond to SA and CAA-AT, respectively. (c) cpc_p distribution on the airfoil surface, compared against [33], and (d) streamwise profile of maximum TKE along wall-normal direction (TKEmax⁡\mathrm{TKE}_{\max}).

From a computational performance standpoint, the domain decomposition of complex geometries, like the airfoil, leads to nonuniform workload distribution among PEs. These workload imbalances incur additional overheads and lead to performance variation among PEs. The performance of the asynchronous solver needs to be assessed in such practical conditions. Strong scaling experiments with two resolutions, containing 2.08 billion and 18 billion grid points with two-dimensional decomposition, are performed on the PARAM Pravega and PARAM Rudra supercomputers. The benchmarks are performed on up to 18,432 PEs by evolving for 100 time steps over 10 independent trials to quantify the performance variation, considering a maximum allowable delay of 5 for the asynchronous solvers. As predicted, the strong scaling results of both CAA-AT and SAA-AT lie closer to the ideal scaling line compared to SA. The asynchronous algorithms report speed-ups of 2.47×\times and 2.73×\times on PARAM Pravega (Figure 11(a)) and PARAM Rudra (Figure 11(b)), respectively, relative to SA. For SAA-AT, the behavior of communication delays plays a major role in reducing the synchronization overheads among PEs. Here, we quantify the behavior of delays using a probability distribution across different PE counts on both supercomputers as shown by the plots in Figure 12. The overall trend is that smaller values of delay are more likely than larger delays, with the mean delay lying between 1 and 2 time levels. At higher PE counts, data movement increases in the network, thereby increasing the probability of incurring higher delays. An interesting comparison can be made between the two architectures on the correlation between the delay distributions and the scaling graphs. At the extreme scale (6144 PEs), we observe higher probabilities for larger delays in the case of PARAM Rudra (Figure 12(b)) compared to PARAM Pravega (Figure 12(a)), leading to a greater deviation in the execution time for the former (Figure 11(b)) compared to the latter (Figure 11(a)). This demonstrates the dependence of communication delays on factors like workload distribution and the architecture.

Image not extracted

Figure 11. Strong scaling graphs for two resolutions of the airfoil case on (a) PARAM Pravega and (b) PARAM Rudra supercomputers, for SA, CAA-AT and SAA-AT (see cases 4 and 6 in Table 1). Maximum allowable delay L=5L = 5 for CAA-AT and SAA-AT.

Figure 12. Probability distribution of delays for the airfoil case on (a) PARAM Pravega and (b) PARAM Rudra supercomputers, for SAA-AT (see case 6 in Table 1). Maximum allowable delay L=5L = 5 for SAA-AT.

Conclusions

The asynchronous computing approach has recently been explored to address the communication overheads in parallel PDE solvers. These communication overheads are considered to be the major bottleneck to scalability in state-of-the-art DNS solvers. This provides the motivation to assess the efficacy of the asynchronous computing approach in accurately capturing flow properties while reducing the communication overheads in CFD solvers. In this study, high-order asynchrony-tolerant (AT) schemes that relax communication/synchronization at a mathematical level using delayed data at PE boundaries were implemented into the multi-block structured compressible flow solver COMP-SQUARE using two algorithms that introduce asynchrony in a distributed parallel setting. These algorithms also skip communications at intermediate stages of time advancement when Runge-Kutta schemes are used. The communication avoiding algorithm (CAA) relaxes communication by performing communications periodically once every few time steps, characterized by a uniform periodic delay across all PEs. The synchronization avoiding algorithm (SAA), on the other hand, relaxes the synchronization by initiating communication at every time step without enforcing explicit synchronization. Numerical experiments were performed using three test cases to evaluate the efficacy of the algorithms. First, the canonical problem of isentropic vortex advection was solved to demonstrate the behavior of numerical errors due to communication delays and the overall order of accuracy of the AT schemes. The computational performance of the asynchronous algorithms was compared against the baseline synchronous algorithm (SA) using a strong scaling analysis, where the considerable speed-ups of CAA-AT and SAA-AT relative to SA are demonstrated. This is a direct consequence of relaxed communication/synchronization. The second test case is the three-dimensional Taylor-Green vortex problem, which is a decaying vortex field transitioning into a turbulent flow. Important flow features were compared between SA, CAA-AS, and CAA-AT, showing the emergence of spurious numerical oscillations for CAA-AS (standard scheme used with delayed data) and excellent agreement between CAA-AT and SA. The computational performance was assessed by performing strong and weak scaling studies, and speed-ups of up to 4×4\times were achieved for the asynchronous solver relative to the SA-based solver. Thirdly, a more practical case of flow around a NACA0012 airfoil was considered. The asynchronous solver accurately captures the separation induced transition as well as the small scale structures in the turbulent boundary layer, suggesting that the high frequency numerical errors due to communication delays are reduced using the higher-order AT schemes near PE boundaries. Further, average flow features such as pressure and turbulent kinetic energy distribution were compared. The pressure distribution obtained with CAA-AT was very similar to that of SA, while a good agreement was observed in turbulent kinetic energy. The computational performance was assessed using strong scaling experiments carried out on two different architectures, achieving up to 2.73×2.73\times speed-up. Furthermore, the probability distribution of delays was also obtained to understand the stochastic behavior of delays in SAA-AT. The findings of this study demonstrate the applicability of asynchronous computing approach in contemporary scalable CFD codes to accelerate numerical simulations that can leverage the computing abilities of future exascale machines.

Acknowledgements

The authors thank C.P. Abdul Gafoor for his help in setting up the airfoil test case. KA acknowledges the financial support from the MoE-STARS grant and the ANRF Core Research Grant. The authors thank the National Supercomputing Mission, India, P.G. Senapathy Centre, IIT Madras and SERC, IISc for providing the computing resources. The authors thank the National PARAM Supercomputing Facility (NPSF) for providing the computing resources on the PARAM Rudra cluster under the National Supercomputing Mission.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Flux terms in governing equations

The unsteady conservation of mass, momentum, and energy equations are non-dimensionalized and cast in strong conservative form on general curvilinear coordinates (x,y,z)→(ξ,η,ζ)(x,y,z) \to(\xi,\eta,\zeta), as detailed in [34, 26].

∂∂t(UJ)+∂F^∂ξ+∂G^∂η+∂H^∂ζ=1Re[∂F^v∂ξ+∂G^v∂η+∂H^v∂ζ],(18)\frac{\partial}{\partial t}\left(\frac{U}{J}\right)+\frac{\partial\hat{F}}{\partial\xi}+\frac{\partial\hat{G}}{\partial\eta}+\frac{\partial\hat{H}}{\partial\zeta}=\frac{1}{Re}\left[\frac{\partial\hat{F}_{v}}{\partial\xi}+\frac{\partial\hat{G}_{v}}{\partial\eta}+\frac{\partial\hat{H}_{v}}{\partial\zeta}\right], \tag*{(18)}

where U={ρ,ρu,ρv,ρw,ρEt}U=\{\rho,\rho u,\rho v,\rho w,\rho E_{t}\} is the vector of the conserved variables and J=∂(ξ,η,ζ)/∂(x,y,z)J=\partial(\xi,\eta,\zeta)/\partial(x,y,z) is the coordinate transformation Jacobian. The advective fluxes F^\hat{F}, G^\hat{G}, and H^\hat{H} are

F^=[ρU^ρuU^+ξ^xpρvU^+ξ^ypρwU^+ξ^zpρ(Et+p)U^],G^=[ρV^ρuV^+η^xpρvV^+η^ypρwV^+η^zpρ(Et+p)V^],H^=[ρW^ρuW^+ζ^xpρvW^+ζ^ypρwW^+ζ^zpρ(Et+p)W^].(19)\hat{F}=\begin{bmatrix} \rho\hat{U}\\ \rho u\hat{U}+\hat{\xi}_{x}p\\ \rho v\hat{U}+\hat{\xi}_{y}p\\ \rho w\hat{U}+\hat{\xi}_{z}p\\ \rho(E_{t}+p)\hat{U} \end{bmatrix},\quad \hat{G}=\begin{bmatrix} \rho\hat{V}\\ \rho u\hat{V}+\hat{\eta}_{x}p\\ \rho v\hat{V}+\hat{\eta}_{y}p\\ \rho w\hat{V}+\hat{\eta}_{z}p\\ \rho(E_{t}+p)\hat{V} \end{bmatrix},\quad \hat{H}=\begin{bmatrix} \rho\hat{W}\\ \rho u\hat{W}+\hat{\zeta}_{x}p\\ \rho v\hat{W}+\hat{\zeta}_{y}p\\ \rho w\hat{W}+\hat{\zeta}_{z}p\\ \rho(E_{t}+p)\hat{W} \end{bmatrix}. \tag*{(19)}

Here, ξ^x=J−1∂ξ/∂x\hat{\xi}_{x}=J^{-1}\partial\xi/\partial x is the metric term, U^\hat{U}, V^\hat{V} and W^\hat{W} are the contravariant components of velocity, pp is the static pressure, and EtE_{t} is the total specific energy.

U^=ξ^xu+ξ^yv+ξ^zwV^=η^xu+η^yv+η^zwW^=ζ^xu+ζ^yv+ζ^zwEt=Tγ(γ−1)M∞2+12(u2+v2+w2).\begin{aligned} \hat{U}&=\hat{\xi}_{x}u+\hat{\xi}_{y}v+\hat{\xi}_{z}w\\ \hat{V}&=\hat{\eta}_{x}u+\hat{\eta}_{y}v+\hat{\eta}_{z}w\\ \hat{W}&=\hat{\zeta}_{x}u+\hat{\zeta}_{y}v+\hat{\zeta}_{z}w\\ E_{t}&=\frac{T}{\gamma(\gamma-1)M_{\infty}^{2}}+\frac{1}{2}(u^{2}+v^{2}+w^{2}). \end{aligned}

The viscous fluxes F^v\hat{F}_{v}, G^v\hat{G}_{v}, and H^v\hat{H}_{v} involve second-order derivatives and are written as

F^v=1J[0ξ^iτi1ξ^iτi2ξ^iτi3ξ^ibi],G^v=1J[0η^iτi1η^iτi2η^iτi3η^ibi],H^v=1J[0ζ^iτi1ζ^iτi2ζ^iτi3ζ^ibi].(20)\hat{F}_{v}=\frac{1}{J}\begin{bmatrix} 0\\ \hat{\xi}_{i}\tau_{i1}\\ \hat{\xi}_{i}\tau_{i2}\\ \hat{\xi}_{i}\tau_{i3}\\ \hat{\xi}_{i}b_{i} \end{bmatrix},\quad \hat{G}_{v}=\frac{1}{J}\begin{bmatrix} 0\\ \hat{\eta}_{i}\tau_{i1}\\ \hat{\eta}_{i}\tau_{i2}\\ \hat{\eta}_{i}\tau_{i3}\\ \hat{\eta}_{i}b_{i} \end{bmatrix},\quad \hat{H}_{v}=\frac{1}{J}\begin{bmatrix} 0\\ \hat{\zeta}_{i}\tau_{i1}\\ \hat{\zeta}_{i}\tau_{i2}\\ \hat{\zeta}_{i}\tau_{i3}\\ \hat{\zeta}_{i}b_{i} \end{bmatrix}. \tag*{(20)}

where the stress tensor τij\tau_{ij} and heat flux vector bib_{i} are

τij=μ(∂ξk∂xj∂ui∂ξk+∂ξk∂xi∂uj∂ξk)−23μδij∂ξl∂xk∂uk∂ξl,\tau_{ij}=\mu\left(\frac{\partial\xi_{k}}{\partial x_{j}}\frac{\partial u_{i}}{\partial\xi_{k}}+\frac{\partial\xi_{k}}{\partial x_{i}}\frac{\partial u_{j}}{\partial\xi_{k}}\right)-\frac{2}{3}\mu\delta_{ij}\frac{\partial\xi_{l}}{\partial x_{k}}\frac{\partial u_{k}}{\partial\xi_{l}},
bi=ujτij+μ(γ−1)RePrM∞2∂T∂xi∂ξl∂xl.(21)b_{i}=u_{j}\tau_{ij}+\frac{\mu}{(\gamma-1)RePrM_{\infty}^{2}}\frac{\partial T}{\partial x_{i}}\frac{\partial\xi_{l}}{\partial x_{l}}. \tag*{(21)}

Standard finite-difference and Runge-Kutta schemes used in COMP-SQUARE

Let us recall the general expression to obtain the spatial derivative of a function ϕ\phi,

αϕi−1′+ϕi′+αϕi+1′=bϕi+2−ϕi−24Δξ+aϕi+1−ϕi−12Δξ.\alpha\phi'_{i-1}+\phi'_i+\alpha\phi'_{i+1}=b\frac{\phi_{i+2}-\phi_{i-2}}{4\Delta\xi}+a\frac{\phi_{i+1}-\phi_{i-1}}{2\Delta\xi}.

The parameters α\alpha, aa, and bb for the spatial schemes used in this paper are as follows.

  • CD2: α=0\alpha=0, a=1a=1, b=0b=0

  • CD4: α=0\alpha=0, a=4/3a=4/3, b=−1/3b=-1/3

The low-storage explicit Runge-Kutta (LSERK) update equations are of the form

Q(m)=AmQ(m−1)+ΔtR(U(m−1))Q^{(m)}=A_mQ^{(m-1)}+\Delta tR(U^{(m-1)})
U(m)=U(m−1)+BmQ(m).U^{(m)}=U^{(m-1)}+B_mQ^{(m)}.

where UU is the vector of conserved variables and QQ is the intermediate vector. The coefficients AmA_m and BmB_m for the LSERK2 and LSERK4 schemes, respectively, are Am∈{0,−1/2}A_m\in\{0,-1/2\}, Bm∈{1/2,1}B_m\in\{1/2,1\} and as given in Table B.2.

mmAmA_mBmB_m
100.09761835
2−0.4812317-0.48123170.41225329
3−1.0495626-1.04956260.44021696
4−1.6025296-1.60252961.42631146
5−1.7782672-1.77826720.19787605

Table B.2. Coefficients for the five-stage LSERK4 method, adopted from [29].

Asynchrony-tolerant (AT) update schemes for buffer points

For the LSERK2-CD2-AT2 method, the solution at a buffer point must be updated at stage m=1m=1 alone, which requires the computation of ∂ϕ/∂t∣i(0)\left.\partial\phi/\partial t\right|_i^{(0)} based on the extended stencil in time. Solving Eq. (13) in § 3.1 for the unknown coefficients results in the approximation

∂ϕ∂t∣i(0)=2k~+32Δtϕin−k~−2k~+2Δtϕin−k~−1+2k~+12Δtϕin−k~−2.\left.\frac{\partial\phi}{\partial t}\right|_i^{(0)}=\frac{2\widetilde{k}+3}{2\Delta t}\phi_i^{n-\widetilde{k}}-\frac{2\widetilde{k}+2}{\Delta t}\phi_i^{n-\widetilde{k}-1}+\frac{2\widetilde{k}+1}{2\Delta t}\phi_i^{n-\widetilde{k}-2}.

On the other hand, for the LSERK4-CD4-AT4 method, buffer updates should be performed until the fourth stage (m=4m=4) and the expressions obtained for the temporal derivative in Eq. (8) are listed in Table C.3.

DerivativeExpression
(∂ϕ/∂t)i(0)(\partial\phi/\partial t)_{i}^{(0)}112Δt(2k~3+15k~2+35k~+25)ϕin−k~\frac{1}{12\Delta t}\left(2\tilde{k}^{3}+15\tilde{k}^{2}+35\tilde{k}+25\right)\phi_{i}^{n-\tilde{k}}
(∂ϕ/∂t)i(0)(\partial\phi/\partial t)_{i}^{(0)}−16Δt(4k~3+27k~2+52k~+24)ϕin−k~−1-\frac{1}{6\Delta t}\left(4\tilde{k}^{3}+27\tilde{k}^{2}+52\tilde{k}+24\right)\phi_{i}^{n-\tilde{k}-1}
(∂ϕ/∂t)i(0)(\partial\phi/\partial t)_{i}^{(0)}+12Δt(2k~3+12k~2+19k~+6)ϕin−k~−2+\frac{1}{2\Delta t}\left(2\tilde{k}^{3}+12\tilde{k}^{2}+19\tilde{k}+6\right)\phi_{i}^{n-\tilde{k}-2}
(∂ϕ/∂t)i(0)(\partial\phi/\partial t)_{i}^{(0)}−16Δt(4k~3+21k~2+28k~+8)ϕin−k~−3-\frac{1}{6\Delta t}\left(4\tilde{k}^{3}+21\tilde{k}^{2}+28\tilde{k}+8\right)\phi_{i}^{n-\tilde{k}-3}
(∂ϕ/∂t)i(0)(\partial\phi/\partial t)_{i}^{(0)}+112Δt(2k~3+9k~2+11k~+3)ϕin−k~−4+\frac{1}{12\Delta t}\left(2\tilde{k}^{3}+9\tilde{k}^{2}+11\tilde{k}+3\right)\phi_{i}^{n-\tilde{k}-4}
(∂ϕ/∂t)i(1)(\partial\phi/\partial t)_{i}^{(1)}112Δt(2k~3+(15+6ν11)k~2+(35+30ν11)k~+25+35ν11)ϕin−k~\frac{1}{12\Delta t}\left(2\tilde{k}^{3}+(15+6\nu_{1}^{1})\tilde{k}^{2}+(35+30\nu_{1}^{1})\tilde{k}+25+35\nu_{1}^{1}\right)\phi_{i}^{n-\tilde{k}}
(∂ϕ/∂t)i(1)(\partial\phi/\partial t)_{i}^{(1)}−16Δt(4k~3+(27+12ν11)k~2+(52+54ν11)k~+24+52ν11)ϕin−k~−1-\frac{1}{6\Delta t}\left(4\tilde{k}^{3}+(27+12\nu_{1}^{1})\tilde{k}^{2}+(52+54\nu_{1}^{1})\tilde{k}+24+52\nu_{1}^{1}\right)\phi_{i}^{n-\tilde{k}-1}
(∂ϕ/∂t)i(1)(\partial\phi/\partial t)_{i}^{(1)}+12Δt(2k~3+(12+6ν11)k~2+(19+24ν11)k~+6+19ν11)ϕin−k~−2+\frac{1}{2\Delta t}\left(2\tilde{k}^{3}+(12+6\nu_{1}^{1})\tilde{k}^{2}+(19+24\nu_{1}^{1})\tilde{k}+6+19\nu_{1}^{1}\right)\phi_{i}^{n-\tilde{k}-2}
(∂ϕ/∂t)i(1)(\partial\phi/\partial t)_{i}^{(1)}−16Δt(4k~3+(21+12ν11)k~2+(28+42ν11)k~+8+28ν11)ϕin−k~−3-\frac{1}{6\Delta t}\left(4\tilde{k}^{3}+(21+12\nu_{1}^{1})\tilde{k}^{2}+(28+42\nu_{1}^{1})\tilde{k}+8+28\nu_{1}^{1}\right)\phi_{i}^{n-\tilde{k}-3}
(∂ϕ/∂t)i(1)(\partial\phi/\partial t)_{i}^{(1)}+112Δt(2k~3+(9+6ν11)k~2+(11+18ν11)k~+3+11ν11)ϕin−k~−4+\frac{1}{12\Delta t}\left(2\tilde{k}^{3}+(9+6\nu_{1}^{1})\tilde{k}^{2}+(11+18\nu_{1}^{1})\tilde{k}+3+11\nu_{1}^{1}\right)\phi_{i}^{n-\tilde{k}-4}
(∂ϕ/∂t)i(2)(\partial\phi/\partial t)_{i}^{(2)}112Δt(2k~3+(15+6ν12)k~2+(35+30ν12+12ν22)k~+25+35ν12+30ν22)ϕin−k~\frac{1}{12\Delta t}\left(2\tilde{k}^{3}+(15+6\nu_{1}^{2})\tilde{k}^{2}+(35+30\nu_{1}^{2}+12\nu_{2}^{2})\tilde{k}+25+35\nu_{1}^{2}+30\nu_{2}^{2}\right)\phi_{i}^{n-\tilde{k}}
(∂ϕ/∂t)i(2)(\partial\phi/\partial t)_{i}^{(2)}−16Δt(4k~3+(27+12ν12)k~2+(52+54ν12+24ν22)k~+24+52ν12+54ν22)ϕin−k~−1-\frac{1}{6\Delta t}\left(4\tilde{k}^{3}+(27+12\nu_{1}^{2})\tilde{k}^{2}+(52+54\nu_{1}^{2}+24\nu_{2}^{2})\tilde{k}+24+52\nu_{1}^{2}+54\nu_{2}^{2}\right)\phi_{i}^{n-\tilde{k}-1}
(∂ϕ/∂t)i(2)(\partial\phi/\partial t)_{i}^{(2)}+12Δt(2k~3+(12+6ν12)k~2+(19+24ν12+12ν22)k~+6+19ν12+24ν22)ϕin−k~−2+\frac{1}{2\Delta t}\left(2\tilde{k}^{3}+(12+6\nu_{1}^{2})\tilde{k}^{2}+(19+24\nu_{1}^{2}+12\nu_{2}^{2})\tilde{k}+6+19\nu_{1}^{2}+24\nu_{2}^{2}\right)\phi_{i}^{n-\tilde{k}-2}
(∂ϕ/∂t)i(2)(\partial\phi/\partial t)_{i}^{(2)}−16Δt(4k~3+(21+12ν12)k~2+(28+42ν12+24ν22)k~+8+28ν12+42ν22)ϕin−k~−3-\frac{1}{6\Delta t}\left(4\tilde{k}^{3}+(21+12\nu_{1}^{2})\tilde{k}^{2}+(28+42\nu_{1}^{2}+24\nu_{2}^{2})\tilde{k}+8+28\nu_{1}^{2}+42\nu_{2}^{2}\right)\phi_{i}^{n-\tilde{k}-3}
(∂ϕ/∂t)i(2)(\partial\phi/\partial t)_{i}^{(2)}+112Δt(2k~3+(9+6ν12)k~2+(11+18ν12+12ν22)k~+3+11ν12+18ν22)ϕin−k~−4+\frac{1}{12\Delta t}\left(2\tilde{k}^{3}+(9+6\nu_{1}^{2})\tilde{k}^{2}+(11+18\nu_{1}^{2}+12\nu_{2}^{2})\tilde{k}+3+11\nu_{1}^{2}+18\nu_{2}^{2}\right)\phi_{i}^{n-\tilde{k}-4}
(∂ϕ/∂t)i(3)(\partial\phi/\partial t)_{i}^{(3)}112Δt(2k~3+(15+6ν13)k~2+(35+30ν13+12ν23)k~+25+35ν13+30ν23+12ν33)ϕin−k~\frac{1}{12\Delta t}\left(2\tilde{k}^{3}+(15+6\nu_{1}^{3})\tilde{k}^{2}+(35+30\nu_{1}^{3}+12\nu_{2}^{3})\tilde{k}+25+35\nu_{1}^{3}+30\nu_{2}^{3}+12\nu_{3}^{3}\right)\phi_{i}^{n-\tilde{k}}
(∂ϕ/∂t)i(3)(\partial\phi/\partial t)_{i}^{(3)}−16Δt(4k~3+(27+12ν13)k~2+(52+54ν13+24ν23)k~+24+52ν13+54ν23+24ν33)ϕin−k~−1-\frac{1}{6\Delta t}\left(4\tilde{k}^{3}+(27+12\nu_{1}^{3})\tilde{k}^{2}+(52+54\nu_{1}^{3}+24\nu_{2}^{3})\tilde{k}+24+52\nu_{1}^{3}+54\nu_{2}^{3}+24\nu_{3}^{3}\right)\phi_{i}^{n-\tilde{k}-1}
(∂ϕ/∂t)i(3)(\partial\phi/\partial t)_{i}^{(3)}+12Δt(2k~3+(12+6ν13)k~2+(19+24ν13+12ν23)k~+6+19ν13+24ν23+12ν33)ϕin−k~−2+\frac{1}{2\Delta t}\left(2\tilde{k}^{3}+(12+6\nu_{1}^{3})\tilde{k}^{2}+(19+24\nu_{1}^{3}+12\nu_{2}^{3})\tilde{k}+6+19\nu_{1}^{3}+24\nu_{2}^{3}+12\nu_{3}^{3}\right)\phi_{i}^{n-\tilde{k}-2}
(∂ϕ/∂t)i(3)(\partial\phi/\partial t)_{i}^{(3)}−16Δt(4k~3+(21+12ν13)k~2+(28+42ν13+24ν23)k~+8+28ν13+42ν23+24ν33)ϕin−k~−3-\frac{1}{6\Delta t}\left(4\tilde{k}^{3}+(21+12\nu_{1}^{3})\tilde{k}^{2}+(28+42\nu_{1}^{3}+24\nu_{2}^{3})\tilde{k}+8+28\nu_{1}^{3}+42\nu_{2}^{3}+24\nu_{3}^{3}\right)\phi_{i}^{n-\tilde{k}-3}
(∂ϕ/∂t)i(3)(\partial\phi/\partial t)_{i}^{(3)}+112Δt(2k~3+(9+6ν13)k~2+(11+18ν13+12ν23)k~+3+11ν13+18ν23+12ν33)ϕin−k~−4+\frac{1}{12\Delta t}\left(2\tilde{k}^{3}+(9+6\nu_{1}^{3})\tilde{k}^{2}+(11+18\nu_{1}^{3}+12\nu_{2}^{3})\tilde{k}+3+11\nu_{1}^{3}+18\nu_{2}^{3}+12\nu_{3}^{3}\right)\phi_{i}^{n-\tilde{k}-4}

Table C.3. Temporal derivatives of function ϕ\phi at buffer points based on the delay k~\tilde{k}, evaluated at (m−1)(m-1)th stage in the LSERK4-CD4-AT4.

References

  1. [1]J. H. Chen, A. Choudhary, B. D. Supinski, M. Devries, E. R. Hawkes, S. Klasky, W. K. Liao, K. L. Ma, J. Mellor-Crummey, N. Podhorszki, R. Sankaran, S. Shende, C. S. Yoo, Terascale direct numerical simulations of turbulent combustion using S3D, Computational Science and Discovery 2 (2009).
  2. [2]K. W. Jenkins, S. R. Cant, Direct numerical simulation of turbulent flame kernels, in: D. Knight, L. Sakell (Eds.), Recent Advances in DNS and LES, Springer Netherlands, Dordrecht, 1999, pp. 191–202.DOI
  3. [3]O. Desjardins, G. Blanquart, G. Balarac, H. Pitsch, High order conservative finite difference scheme for variable density low Mach number turbulent flows, Journal of Computational Physics 227 (2008) 7125–7159.
  4. [4]T. C. Germann, Co-design in the exascale computing project, The International Journal of High Performance Computing Applications 35 (2021) 503–507.DOI
  5. [5]D. Amitai, S. Itzikowitz, Parallel adaptive and time-stabilizing schemes for constant-coefficient parabolic PDE’s, 1992.DOI
  6. [6]D. Amitai, A. Averbuch, S. Itzikowitz, E. Turkel, Asynchronous and corrected-asynchronous finite difference solutions of PDEs on MIMD multiprocessors, 1994.DOI
  7. [7]A. Konduri, D. A. Donzis, Poster: Asynchronous Computing for Partial Differential Equations at Extreme Scales, in: 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, 2012, pp. 1444–1444.
  8. [8]D. A. Donzis, K. Aditya, Asynchronous finite-difference schemes for partial differential equations, Journal of Computational Physics 274 (2014) 370–392.DOI
  9. [9]A. Mittal, S. Girimaji, Proxy-equation paradigm: A strategy for massively parallel asynchronous computations, Physical Review E 96 (2017).DOI
  10. [10]K. Aditya, D. A. Donzis, High-order asynchrony-tolerant finite difference schemes for partial differential equations, Journal of Computational Physics 350 (2017) 550–572.
  11. [11]K. Aditya, T. Gysi, G. Kwasniewski, T. Hoefler, D. A. Donzis, J. H. Chen, A scalable weakly-synchronous algorithm for solving partial differential equations, 2019.arxiv.org/abs/1911.05769
  12. [12]K. Kumari, D. A. Donzis, Direct numerical simulations of turbulent flows using high-order asynchrony-tolerant schemes: Accuracy and performance, Journal of Computational Physics 419 (2020).DOI
  13. [13]S. K. Goswami, V. J. Matthew, K. Aditya, Implementation of low-storage Runge-Kutta time integration schemes in scalable asynchronous partial differential equation solvers, Journal of Computational Physics 477 (2023).DOI
  14. [14]K. Kumari, E. Cleary, S. Desai, D. A. Donzis, J. H. Chen, K. Aditya, Evaluation of finite difference based asynchronous partial differential equation solvers, Journal of Computational Physics 477 (2023).
  15. [15]S. K. Goswami, K. Aditya, An asynchronous discontinuous-Galerkin method for solving PDEs at extreme scales, 2022.DOI
  16. [16]S. K. Goswami, K. Aditya, An asynchronous discontinuous-Galerkin method for massively parallel PDE solvers, Computer Methods in Applied Mechanics and Engineering 430 (2024).DOI
  17. [17]A. K. Arumugam, K. Aditya, An asynchronous discontinuous-Galerkin method for combustion simulations, 2025.arxiv.org/abs/2501.01747
  18. [18]N. R. Vadlamani, P. G. Tucker, P. Durbin, Distributed Roughness Effects on Transitional and Turbulent Boundary Layers, Flow, Turbulence and Combustion 100 (2018) 627–649.DOI
  19. [19]A. S. Malathi, M. Nardini, A. Vaid, N. R. Vadlamani, R. D. Sandberg, Profile Loss Reduction of High-Lift Turbine Blades with Rough and Ribbed Surfaces, Journal of Turbomachinery 145 (2023).
  20. [20]N. R. Vadlamani, P. G. Tucker, Eddy resolving simulations of intake under crosswinds, in: M. V. Salvetti, V. Armenio, J. Fröhlich, B. J. Geurts, H. Kuersten (Eds.), Direct and Large-Eddy Simulation XI, Springer International Publishing, Cham, 2019, pp. 523–529.DOI
  21. [21]J. Tyacke, N. R. Vadlamani, W. Trojak, R. Watson, Y. Ma, P. G. Tucker, Turbomachinery simulation challenges and the future, 2019.DOI
  22. [22]A. I. Arasu, S. S. Sarvankar, N. R. Vadlamani, Distortion Control in Intakes Subject to Strong Crosswinds Using Steady Vortex Generator Jets, AIAA Journal (2024) 1–16.DOI
  23. [23]Y. Lin, R. Vadlamani, M. Savill, P. Tucker, Wall-resolved large eddy simulation for aeroengine aeroacoustic investigation, in: Aeronautical Journal, volume 121, Cambridge University Press, 2017, pp. 1032–1050.DOI
  24. [24]G. Nampelly, A. S. Malathi, A. Vaid, N. R. Vadlamani, S. Rengarajan, K. Kontis, Surface Roughness Effects on Cavity Flows, Flow, Turbulence and Combustion 109 (2022) 1215–1239.DOI
  25. [25]S. K. Lele, Compact Finite Difference Schemes with Spectral-like Resolution, Journal of Computational Physics 103 (1992) 42.
  26. [26]M. R. Visbal, D. V. Gaitonde, On the use of higher-order finite-difference schemes on curvilinear and deforming meshes, Journal of Computational Physics 181 (2002) 155–185.DOI
  27. [27]J. H. Williamson, Low-storage Runge-Kutta schemes, Journal of Computational Physics (1980) 3548–56.DOI
  28. [28]C. A. Kennedy, M. H. Carpenter, R. M. Lewis, Low-storage, explicit Runge-Kutta schemes for the compressible Navier-Stokes equations, Applied Numerical Mathematics 35 (2000) 177–219.
  29. [29]M. H. Carpenter, C. A. Kennedy, Fourth-order 2N-storage Runge-Kutta schemes, Technical Report, NASA Technical Memorandum 109112, 1994.
  30. [30]L. T. Diosady, S. M. Murman, Case 3.3: Taylor-Green Vortex Evolution, Technical Report, NASA Ames Research Center, 2015.
  31. [31]J. DeBonis, Solutions of the Taylor-Green Vortex Problem Using High-Resolution Explicit Finite Difference Methods, 2013.DOI
  32. [32]W. M. van Rees, A. Leonard, D. I. Pullin, P. Koumoutsakos, A comparison of vortex and pseudo-spectral methods for the simulation of periodic vortical flows at high Reynolds numbers, Journal of Computational Physics 230 (2011) 2794–2805.DOI
  33. [33]M. R. Visbal, D. J. Garmann, Analysis of dynamic stall on a pitching airfoil using high-fidelity large-eddy simulations, in: AIAA Journal, volume 56, American Institute of Aeronautics and Astronautics Inc., 2018, pp. 46–63.DOI
  34. [34]D. V. Gaitonde, M. R. Visbal, High-Order Schemes for Navier-Stokes Equations: Algorithm and Implementation Into FDL3DI, 1998.DOI

Paper details

vlnc:W_ar56vgh5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents