betaFeedback

WorksPhysics and Astronomy

PyFR: An Open Source Framework for Solving Advection-Diffusion Type Problems on Streaming Architectures using the Flux Reconstruction Approach

Abstract

High-order numerical methods for unstructured grids combine the superior accuracy of high-order spectral or finite difference methods with the geometric flexibility of low-order finite volume or finite element schemes. The Flux Reconstruction (FR) approach unifies various high-order schemes for unstructured grids within a single framework. Additionally, the FR approach exhibits a significant degree of element locality, and is thus able to run efficiently on modern streaming architectures, such as Graphical Processing Units (GPUs). The aforementioned properties of FR mean it offers a promising route to performing affordable, and hence industrially relevant, scale-resolving simulations of hitherto intractable unsteady flows within the vicinity of real-world engineering geometries. In this paper we present PyFR, an open-source Python based framework for solving advection-diffusion type problems on streaming architectures using the FR approach. The framework is designed to solve a range of governing systems on mixed unstructured grids containing various element types. It is also designed to target a range of hardware platforms via use of an in-built domain specific language based on the Mako templating engine. The current release of PyFR is able to solve the compressible Euler and Navier-Stokes equations on grids of quadrilateral and triangular elements in two dimensions, and hexahedral elements in three dimensions, targeting clusters of CPUs, and NVIDIA GPUs. Results are presented for various benchmark flow problems, single-node performance is discussed, and scalability of the code is demonstrated on up to 104 NVIDIA M2090 GPUs. The software is freely available under a 3-Clause New Style BSD license (see www.pyfr.org).

Program Description

Authors F. D. Witherden, A. M. Farrington, P. E. Vincent

Program title PyFR v0.1.0

Licensing provisions New Style BSD license

Programming language Python, CUDA and C

Computer Variable, up to and including GPU clusters

Operating system Recent version of Linux/UNIX

RAM Variable, from hundreds of megabytes to gigabytes

Number of processors used Variable, code is multi-GPU and multi-CPU aware through a combination of MPI and OpenMP

External routines/libraries Python 2.7, numpy, PyCUDA, mpi4py, SymPy, Mako

Nature of problem Compressible Euler and Navier-Stokes equations of fluid dynamics; potential for any advection-diffusion type problem.

Solution method High-order flux reconstruction approach suitable for curved, mixed, unstructured grids.

Unusual features Code makes extensive use of symbolic manipulation and runtime code generation through a domain specific language.

Running time Many small problems can be solved on a recent workstation in minutes to hours.

Nomenclature

Throughout we adopt a convention in which dummy indices on the right hand side of an expression are summed. For example Cijk=AijlBilk≡∑lAijlBilkC_{ijk}=A_{ijl}B_{ilk}\equiv\sum_l A_{ijl}B_{ilk} where the limits are implied from the surrounding context. All indices are assumed to be zero-based.

Functions.

Indices.

Domains.

Ωe\Omega_eAll elements in Ω\Omega of type eeAdornments and suffixes.
Ω^e\hat{\Omega}_eA standard element of type ee□~\tilde{\square}A quantity in transformed space
∂Ω^e\partial\hat{\Omega}_eBoundary of Ω^e\hat{\Omega}_e□^\hat{\square}A vector quantity of unit magnitude
Ωen\Omega_{en}Element nn of type ee in Ω\Omega□T\square^{T}Transpose
∣Ωe∣|\Omega_e|Number of elements of type ee□(u)\square^{(u)}A quantity at a solution point
Expansions.□(f)\square^{(f)}A quantity at a flux point
P\mathcal{P}Polynomial order□(f⊥)\square^{(f_\perp)}A normal quantity at a flux point
NDN_DNumber of spatial dimensionsOperators.
NVN_VNumber of field variablesCα\mathcal{C}_{\alpha}Common solution at an interface
ℓeρ\ell_{e\rho}Nodal basis polynomial ρ\rho for element type eeF~α\tilde{\mathcal{F}}_{\alpha}Common normal flux at an interface
x,y,zx,y,zPhysical coordinates
x~,y~,z~\tilde{x},\tilde{y},\tilde{z}Transformed coordinates
Men\mathcal{M}_{en}Transformed to physical mapping

Introduction

There is an increasing desire amongst industrial practitioners of computational fluid dynamics (CFD) to undertake high-fidelity scale-resolving simulations of transient compressible flows within the vicinity of complex geometries. For example, to improve the design of next generation unmanned aerial vehicles (UAVs), there exists a need to perform simulations—at Reynolds numbers 10410^{4}–10710^{7} and Mach numbers M∼0.1M \sim0.1–1.01.0—of highly separated flow over deployed spoilers/air-brakes; separated flow within serpentine intake ducts; acoustic loading in weapons bays; and flow over entire UAV configurations at off-design conditions. Unfortunately, current generation industry-standard CFD software based on first- or second-order accurate Reynolds Averaged Navier-Stokes (RANS) approaches is not well suited to performing such simulations. Henceforth, there has been significant interest in the potential of high-order accurate methods for unstructured mixed grids, and whether they can offer an efficient route to performing scale-resolving simulations within the vicinity of complex geometries. Popular examples of high-order schemes for unstructured mixed grids include the discontinuous Galerkin (DG) method, first introduced by Reed and Hill [1], and the spectral difference (SD) methods originally proposed under the moniker ‘staggered-gird Chebyshev multidomain methods’ by Kopriva and Kolias in 1996 [2] and later popularised by Sun et al. [3]. In 2007 Huynh proposed the flux reconstruction (FR) approach [4]; a unifying framework for high-order schemes for unstructured grids that incorporates both the nodal DG schemes of [5] and, at least for a linear flux function, any SD scheme. In addition to offering high-order accuracy on unstructured mixed grids, FR schemes are also compact in space, and thus when combined with explicit time marching offer a significant degree of element locality. As such, explicit high-order FR schemes are characterised by a large degree of structured computation.

Over the past two decades improvements in the arithmetic capabilities of processors have significantly outpaced advances in random access memory. Algorithms which have traditionally been compute bound—such as dense matrix-vector products—are now limited instead by the bandwidth to/from memory. This is epitomised in Figure 1. Whereas the processors of two decades ago had FLOPS-per-byte of ∼0.2\sim0.2 more recent chips have ratios upwards of ∼4\sim4. This disparity is not limited to just conventional CPUs. Massively parallel accelerators and co-processors such as the NVIDIA K20X and Intel Xeon Phi 5110P have ratios of 5.24 and 3.16, respectively.

Figure 1. Trends in the peak floating point performance (double precision) and memory bandwidth of sever-class Intel processors from 1994–2013. The quotient of these two measures yields the FLOPS-per-byte of a processor. Data courtesy of Jan Treibig.

A concomitant of this disparity is that modern hardware architectures are highly dependent on a combination of high speed caches and/or shared memory to maintain throughput. However, for an algorithm to utilise these efficiently its memory access pattern must exhibit a degree of either spatial or temporal locality. To a first-order approximation the spatial locality of a method is inversely proportional to the amount of memory indirection. On an unstructured grid indirection arises whenever there is coupling between elements. This is potentially a problem for discretisations whose stencil is not compact. Coupling also arises in the context of implicit time stepping schemes. Implementations are therefore very often bound by memory bandwidth. As a secondary trend we note that the manner in which FLOPS are realised has also changed. In the early 1990s commodity CPUs were predominantly scalar with a single core of execution. However in 2013 processors with eight or more cores are not uncommon. Moreover, the cores on modern processors almost always contain vector processing units. Vector lengths up to 256-bits, which permit up to four double precision values to be operated on at once, are not uncommon. It is therefore imperative that compute-bound algorithms are amenable to both multithreading and vectorisation. A versatile means of accomplishing this is by breaking the computation down into multiple, necessarily independent, streams. By virtue of their independence these streams can be readily divided up between cores and vector lanes. This leads directly to the concept of stream processing. We will refer to architectures amenable to this form of parallelisation as streaming architectures.

A corollary of the above discussion is that compute intensive discretisations which can be formulated within the stream processing paradigm are well suited to acceleration on current—and likely future—hardware platforms. The FR approach combined with explicit time stepping is an archetypical of this.

Our objective in this paper is to present PyFR, an open-source Python based framework for solving advection-diffusion type problems on streaming architectures using the FR approach. The framework is designed to solve a range of governing systems on mixed unstructured grids containing various element types. It is also designed to target a range of hardware platforms via use of an in-built domain specific language derived from the Mako templating engine. The current release of PyFR is able to solve the compressible Euler and Navier-Stokes equations on unstructured grids of quadrilateral and triangular elements in two-dimensions, and unstructured grids of hexahedral elements in three-dimensions, targeting clusters of CPUs, and NVIDIA GPUs. The paper is structured as follows. In section 2 we provide a overview of the FR approach for advection-diffusion type problems on mixed unstructured grids. In section 3 we proceed to describe our implementation strategy, and in section 4 we present the Euler and Navier-Stokes equations, which are solved by the current release of PyFR. The framework is then validated in section 5, single-node performance is discussed in section 6, and scalability of the code is demonstrated on up to 10410^4 NVIDIA M2090 GPUs in section 7. Finally, conclusions are drawn in section 8.

Flux Reconstruction

A brief overview of the FR approach for solving advection-diffusion type problems is given below. Extended presentations can be found elsewhere [4, 6, 7, 8, 9, 10, 11, 12, 13, 14].

Consider the following advection-diffusion problem inside an arbitrary domain Ω\Omega in NDN_D dimensions

∂uα∂t+∇⋅fα=0,(1)\frac{\partial u_\alpha}{\partial t} + \nabla\cdot\mathbf{f}_\alpha= 0, \tag*{(1)}

where 0≤α<NV0 \le\alpha< N_V is the field variable index, uα=uα(x,t)u_\alpha= u_\alpha(\mathbf{x},t) is a conserved quantity, fα=fα(u,∇u)\mathbf{f}_\alpha= \mathbf{f}_\alpha(\mathbf{u},\nabla\mathbf{u}) is the flux of this conserved quantity and x=xi∈RND\mathbf{x} = x_i \in\mathbb{R}^{N_D}. In defining the flux we have taken u\mathbf{u} in its unscripted form to refer to all of the NVN_V field variables and ∇u\nabla\mathbf{u} to be an object of length ND×NVN_D \times N_V consisting of the gradient of each field variable. We start by rewriting (1) as a first order system

∂uα∂t+∇⋅fα(u,q)=0,(2)\frac{\partial u_{\alpha}}{\partial t}+\nabla\cdot\mathbf{f}_{\alpha}(u,\mathbf{q})=0, \tag*{(2)}
qα−∇uα=0,(3)\mathbf{q}_{\alpha}-\nabla u_{\alpha}=0, \tag*{(3)}

where q\mathbf{q} is an auxiliary variable. Here, as with ∇u\nabla u, we have taken q\mathbf{q} in its unsubscripted form to refer to the gradients of all of the field variables.

Take E\mathcal{E} to be the set of available element types in RND\mathbb{R}^{N_D}. Examples include quadrilaterals and triangles in two dimensions and hexahedra, prisms, pyramids and tetrahedra in three dimensions. Consider using these various elements types to construct a conformal mesh of the domain such that

Ω=⋃e∈EΩeandΩe=⋃n=0∣Ωe∣−1Ωenand⋂e∈E⋂n=0∣Ωe∣−1Ωen=∅,\Omega=\bigcup_{e\in\mathcal{E}}\Omega_e \qquad\text{and}\qquad \Omega_e=\bigcup_{n=0}^{|\Omega_e|-1}\Omega_{en} \qquad\text{and}\qquad \bigcap_{e\in\mathcal{E}}\bigcap_{n=0}^{|\Omega_e|-1}\Omega_{en}=\varnothing,

where Ωe\Omega_e refers to all of the elements of type ee inside of the domain, ∣Ωe∣|\Omega_e| is the number of elements of this type in the decomposition, and nn is an index running over these elements with 0≤n<∣Ωe∣0\le n<|\Omega_e|. Inside each element Ωen\Omega_{en} we require that

∂uenα∂t+∇⋅fenα=0,(4)\frac{\partial u_{en\alpha}}{\partial t}+\nabla\cdot\mathbf{f}_{en\alpha}=0, \tag*{(4)}
qenα−∇uenα=0.(5)\mathbf{q}_{en\alpha}-\nabla u_{en\alpha}=0. \tag*{(5)}

It is convenient, for reasons of both mathematical simplicity and computational efficiency, to work in a transformed space. We accomplish this by introducing, for each element type, a standard element Ω^e\widehat{\Omega}_e which exists in a transformed space, x~=x~i\widetilde{\mathbf{x}}=\widetilde{x}_i. Next, assume the existence of a mapping function for each element such that

xi=Meni(x~),x=Men(x~),x_i=\mathcal{M}_{eni}(\widetilde{\mathbf{x}}),\qquad\mathbf{x}=\mathcal{M}_{en}(\widetilde{\mathbf{x}}),
x~i=Meni−1(x),x~=Men−1(x),\widetilde{x}_i=\mathcal{M}_{eni}^{-1}(\mathbf{x}),\qquad\widetilde{\mathbf{x}}=\mathcal{M}_{en}^{-1}(\mathbf{x}),

along with the relevant Jacobian matrices

Jen=Jenij=∂Meni∂x~j,Jen=det⁡Jen,\mathbf{J}_{en}=\mathbf{J}_{enij}=\frac{\partial\mathcal{M}_{eni}}{\partial\widetilde{x}_j},\qquad J_{en}=\det\mathbf{J}_{en},
Jen−1=Jeni−1=∂Meni−1∂xj,Jen−1=det⁡Jen−1=1Jen.\mathbf{J}_{en}^{-1}=\mathbf{J}_{eni}^{-1}=\frac{\partial\mathcal{M}_{eni}^{-1}}{\partial x_j},\qquad J_{en}^{-1}=\det\mathbf{J}_{en}^{-1}=\frac{1}{J_{en}}.

These definitions provide us with a means of transforming quantities to and from standard element space. Taking the transformed solution, flux, and gradients inside each element to be

u~enα=u~enα(x~,t)=Jen(x~)uenα(Men(x~),t),(6)\widetilde{u}_{en\alpha}=\widetilde{u}_{en\alpha}(\widetilde{\mathbf{x}},t)=J_{en}(\widetilde{\mathbf{x}})u_{en\alpha}(\mathcal{M}_{en}(\widetilde{\mathbf{x}}),t), \tag*{(6)}
f~enα=f~enα(x~,t)=Jen(x~)Jen−1(Men(x~))fenα(Men(x~),t),(7)\widetilde{\mathbf{f}}_{en\alpha}=\widetilde{\mathbf{f}}_{en\alpha}(\widetilde{\mathbf{x}},t)=J_{en}(\widetilde{\mathbf{x}})\mathbf{J}_{en}^{-1}(\mathcal{M}_{en}(\widetilde{\mathbf{x}}))\mathbf{f}_{en\alpha}(\mathcal{M}_{en}(\widetilde{\mathbf{x}}),t), \tag*{(7)}
q~enα=q~enα(x~,t)=JenT(x~)qenα(Men(x~),t),(8)\widetilde{\mathbf{q}}_{en\alpha}=\widetilde{\mathbf{q}}_{en\alpha}(\widetilde{\mathbf{x}},t)=\mathbf{J}_{en}^{T}(\widetilde{\mathbf{x}})\mathbf{q}_{en\alpha}(\mathcal{M}_{en}(\widetilde{\mathbf{x}}),t), \tag*{(8)}

and letting ∇~=∂/∂x~i\widetilde{\nabla} = \partial/\partial\widetilde{x}_i, it can be readily verified that

∂u~enα∂t+Jen−1∇~⋅f~enα=0,(9)\frac{\partial\widetilde{u}_{en\alpha}}{\partial t} + J_{en}^{-1}\widetilde{\nabla}\cdot\widetilde{\mathbf{f}}_{en\alpha}=0, \tag*{(9)}
q~enα−∇~uenα=0,(10)\widetilde{\mathbf{q}}_{en\alpha}-\widetilde{\nabla}u_{en\alpha}=0, \tag*{(10)}

as required. We note here the decision to multiply the first equation through by a factor of Jen−1J_{en}^{-1}. Doing so has the effect of taking u~en↦uen\widetilde{u}_{en}\mapsto u_{en} which allows us to work in terms of the physical solution. This is more convenient from a computational standpoint.

We next proceed to associate a set of solution points with each standard element. For each type e∈Ee\in\mathcal{E} take {x~eρ(u)}\{\widetilde{x}_{e\rho}^{(u)}\} to be the chosen set of points where 0≤ρ<Ne(u)(℘)0\leq\rho<N_{e}^{(u)}(\wp). These points can then be used to construct a nodal basis set {ℓeρ(u)(x~)}\{\ell_{e\rho}^{(u)}(\widetilde{x})\} with the property that ℓeρ(u)(x~eσ(u))=δρσ\ell_{e\rho}^{(u)}(\widetilde{x}_{e\sigma}^{(u)})=\delta_{\rho\sigma}. To obtain such a set we first take {ψeσ(x~)}\{\psi_{e\sigma}(\widetilde{x})\} to be any basis which spans a selected order ℘\wp polynomial space defined inside Ω^e\widehat{\Omega}_e. Next we compute the elements of the generalised Vandermonde matrix Veρσ=ψeρ(x~eσ(u))\mathcal{V}_{e\rho\sigma}=\psi_{e\rho}(\widetilde{x}_{e\sigma}^{(u)}). With these a nodal basis set can be constructed as ℓeρ(u)(x~)=Veρσ−1ψeσ(x~)\ell_{e\rho}^{(u)}(\widetilde{x})=\mathcal{V}_{e\rho\sigma}^{-1}\psi_{e\sigma}(\widetilde{x}). Along with the solution points inside of each element we also define a set of flux points on ∂Ω^e\partial\widehat{\Omega}_e. We denote the flux points for a particular element type as {x~eρ(f)}\{\widetilde{x}_{e\rho}^{(f)}\} where 0≤ρ<Ne(f)(℘)0\leq\rho<N_e^{(f)}(\wp). Let the set of corresponding normalised outward-pointing normal vectors be given by {n^eρ(f)}\{\widehat{\mathbf{n}}_{e\rho}^{(f)}\}. It is critical that each flux point pair along an interface share the same coordinates in physical space. For a pair of flux points eρne\rho n and e′ρ′n′e'\rho'n' at a non-periodic interface this can be formalised as Men(x~eρ(f))=Me′n′(x~e′ρ′(f))\mathcal{M}_{en}(\widetilde{x}_{e\rho}^{(f)})=\mathcal{M}_{e'n'}(\widetilde{x}_{e'\rho'}^{(f)}). A pictorial illustration of this can be seen in Figure 2.

Figure 2. Solution points (blue circles) and flux points (orange squares) for a triangle and quadrangle in physical space. For the top edge of the quadrangle the normal vectors have been plotted. Observe how the flux points at the interface between the two elements are co-located.

The first step in the FR approach is to go from the discontinuous solution at the solution points to the discontinuous solution at the flux points

ueρnα(f)=ueρnα(u)ℓeρ(u)(x~eσ(f)),(11)u_{e\rho n\alpha}^{(f)}=u_{e\rho n\alpha}^{(u)}\ell_{e\rho}^{(u)}(\widetilde{x}_{e\sigma}^{(f)}), \tag*{(11)}

where ueρnα(u)u_{e\rho n\alpha}^{(u)} is an approximate solution of field variable α\alpha inside of the nnth element

of type ee at solution point x~eρn(u)\widetilde{x}_{e\rho n}^{(u)}. This can then be used to compute a common solution

Cαueρnα(f)=Cαuegρnα(f)=Cα(ueρnα(f),uegρnα(f)),(12)\mathcal{C}_{\alpha}u_{e\rho n\alpha}^{(f)}=\mathcal{C}_{\alpha}u_{e g\rho n\alpha}^{(f)}=\mathcal{C}_{\alpha}\left(u_{e\rho n\alpha}^{(f)},u_{e g\rho n\alpha}^{(f)}\right), \tag*{(12)}

where Cα(uL,uR)\mathcal{C}_{\alpha}(u_L,u_R) is a scalar function that given two values at a point returns a common value. Here we have taken egρne g\rho n to be the element type, flux point number and element number of the adjoining point at the interface. Since grids in FR are permitted to be unstructured the relationship between eρne\rho n and egρne g\rho n is indirect. This necessitates the use of a lookup table. As the common solution function is permitted to perform upwinding or downwinding of the solution it is in general the case that Cα(uegρnα(f),ueρnα(f))≠Cα(ueρnα(f),uegρnα(f))\mathcal{C}_{\alpha}(u_{e g\rho n\alpha}^{(f)},u_{e\rho n\alpha}^{(f)}) \ne\mathcal{C}_{\alpha}(u_{e\rho n\alpha}^{(f)},u_{e g\rho n\alpha}^{(f)}). Hence, it is important that each flux point pair only be visited once with the same common solution value assigned to both Cαueρnα(f)\mathcal{C}_{\alpha}u_{e\rho n\alpha}^{(f)} and Cαuegρnα(f)\mathcal{C}_{\alpha}u_{e g\rho n\alpha}^{(f)}.

Further, associated with each flux point is a vector correction function geρ(f)(x~)\mathbf{g}_{e\rho}^{(f)}(\widetilde{\mathbf{x}}) constrained such that

n~eσ(f)⋅geρ(f)(x~eσ(f))=δρσ,(13)\widetilde{\mathbf{n}}_{e\sigma}^{(f)}\cdot\mathbf{g}_{e\rho}^{(f)}(\widetilde{\mathbf{x}}_{e\sigma}^{(f)})=\delta_{\rho\sigma}, \tag*{(13)}

with a divergence that sits in the same polynomial space as the solution. Using these fields we can express the solution to (10) as

q~eσnα(u)=[n~eρ(f)⋅∇~⋅geρ(f)(x~){Cαuegρnα(f)−ueρnα(f)}+ueρnα(u)∇~ℓ~eρ(u)(x~)]x~=x~eσ(u),(14)\widetilde{\mathbf{q}}_{e\sigma n\alpha}^{(u)}= \left[ \widetilde{\mathbf{n}}_{e\rho}^{(f)}\cdot\widetilde{\nabla}\cdot\mathbf{g}_{e\rho}^{(f)}(\widetilde{\mathbf{x}}) \left\{\mathcal{C}_{\alpha}u_{e g\rho n\alpha}^{(f)}-u_{e\rho n\alpha}^{(f)}\right\} +u_{e\rho n\alpha}^{(u)}\widetilde{\nabla}\widetilde{\ell}_{e\rho}^{(u)}(\widetilde{\mathbf{x}}) \right]_{\widetilde{\mathbf{x}}=\widetilde{\mathbf{x}}_{e\sigma}^{(u)}}, \tag*{(14)}

where the term inside the curly brackets is the ‘jump’ at the interface and the final term is an order ℘−1\wp-1 approximation of the gradient obtained by differentiating the discontinuous solution polynomial. Following the approaches of Kopriva [15] and Sun et al. [3] we can now compute physical gradients as

qeσnα(u)=Jeσn−T(u)q~eσnα(u),(15)\mathbf{q}_{e\sigma n\alpha}^{(u)}=\mathbf{J}_{e\sigma n}^{-T(u)}\widetilde{\mathbf{q}}_{e\sigma n\alpha}^{(u)}, \tag*{(15)}
qeσnα(f)=ℓeρ(u)(x~eσ(f))qeρnα(u),(16)\mathbf{q}_{e\sigma n\alpha}^{(f)}=\ell_{e\rho}^{(u)}(\widetilde{\mathbf{x}}_{e\sigma}^{(f)})\mathbf{q}_{e\rho n\alpha}^{(u)}, \tag*{(16)}

where Jeσn−T(u)=Jen−T(x~eσ(u))\mathbf{J}_{e\sigma n}^{-T(u)}=\mathbf{J}_{en}^{-T}(\widetilde{\mathbf{x}}_{e\sigma}^{(u)}). Having solved the auxiliary equation we are now able to evaluate the transformed flux

f~eρnα(u)=Jeρn(u)Jeρn−1(u)fα(ueρn(u),qeρn(u)),(17)\widetilde{\mathbf{f}}_{e\rho n\alpha}^{(u)}=\mathbf{J}_{e\rho n}^{(u)}\mathbf{J}_{e\rho n}^{-1(u)}\mathbf{f}_{\alpha}\left(\mathbf{u}_{e\rho n}^{(u)},\mathbf{q}_{e\rho n}^{(u)}\right), \tag*{(17)}

where Jeρn(u)=det⁡Jen(x~eρ(u))J_{e\rho n}^{(u)}=\det\mathbf{J}_{en}(\widetilde{\mathbf{x}}_{e\rho}^{(u)}). This can be seen to be a collocation projection of the flux. With this it is possible to compute the normal transformed flux at each of the flux points

f~eσnα(f)=ℓeρ(f)(x~eσ(f))n~eσ(f)⋅f~eρnα(u).(18)\widetilde{f}_{e\sigma n\alpha}^{(f)}=\ell_{e\rho}^{(f)}(\widetilde{\mathbf{x}}_{e\sigma}^{(f)})\widetilde{n}_{e\sigma}^{(f)}\cdot\widetilde{\mathbf{f}}_{e\rho n\alpha}^{(u)}. \tag*{(18)}

Considering the physical normals at the flux points we see that

neσn(f)=neσn(f)n^eσ(f)=Jeσn−T(f)n~eσ(f),(19)\mathbf{n}_{e\sigma n}^{(f)}=n_{e\sigma n}^{(f)}\widehat{\mathbf{n}}_{e\sigma}^{(f)}=\mathbf{J}_{e\sigma n}^{-T(f)}\widetilde{\mathbf{n}}_{e\sigma}^{(f)}, \tag*{(19)}

which is the outward facing normal vector in physical space where neσn(f)>0n_{e\sigma n}^{(f)}>0 is defined as the magnitude. As the interfaces between two elements conform we must have n^eσn(f)=−n^eσˉn(f)\hat{\mathbf{n}}^{(f)}_{e\sigma n}=-\hat{\mathbf{n}}^{(f)}_{e\bar{\sigma}n}. With these definitions we are now in a position to specify an expression for the common normal flux at a flux point pair as

F~αf~eσnα(f1)=−F~αfeσnα(f1)=F~α(ueσn(f),ueσˉn(f),qeσn(f),qeσˉn(f),n^eσn(f)).(20)\widetilde{\mathcal{F}}_\alpha\widetilde{f}^{(f1)}_{e\sigma n\alpha}=-\widetilde{\mathcal{F}}_\alpha f^{(f1)}_{e\sigma n\alpha}=\widetilde{\mathcal{F}}_\alpha\left(u^{(f)}_{e\sigma n},u^{(f)}_{e\bar{\sigma}n},q^{(f)}_{e\sigma n},q^{(f)}_{e\bar{\sigma}n},\hat{\mathbf{n}}^{(f)}_{e\sigma n}\right). \tag*{(20)}

The relationship F~αf~eσnα(f1)=−F~αfeσˉnα(f1)\widetilde{\mathcal{F}}_\alpha\widetilde{f}^{(f1)}_{e\sigma n\alpha}=-\widetilde{\mathcal{F}}_\alpha f^{(f1)}_{e\bar{\sigma}n\alpha} arises from the desire for the resulting numerical scheme to be conservative; a net outward flux from one element must be balanced by a corresponding inward flux on the adjoining element. It follows that Fα(uL,uR,qL,qR,n^L)=−Fα(uR,uL,qR,qL,−n^L)\mathcal{F}_\alpha(u_L,u_R,q_L,q_R,\hat{\mathbf{n}}_L)=-\mathcal{F}_\alpha(u_R,u_L,q_R,q_L,-\hat{\mathbf{n}}_L). The common normal fluxes in (20) can now be taken into transformed space via

F~αf~eσnα(f1)=Jeσn(f)neσn(f)F~αfeσnα(f1),(21)\widetilde{\mathcal{F}}_\alpha\widetilde{f}^{(f1)}_{e\sigma n\alpha}=J^{(f)}_{e\sigma n}n^{(f)}_{e\sigma n}\widetilde{\mathcal{F}}_\alpha f^{(f1)}_{e\sigma n\alpha}, \tag*{(21)}
F~αf~eσˉnα(f1)=Jeσˉn(f)neσˉn(f)F~αfeσˉnα(f1),(22)\widetilde{\mathcal{F}}_\alpha\widetilde{f}^{(f1)}_{e\bar{\sigma}n\alpha}=J^{(f)}_{e\bar{\sigma}n}n^{(f)}_{e\bar{\sigma}n}\widetilde{\mathcal{F}}_\alpha f^{(f1)}_{e\bar{\sigma}n\alpha}, \tag*{(22)}

where Jeσn(f)=det⁡Jen(x~eσ(f))J^{(f)}_{e\sigma n}=\det J_{en}(\tilde{\mathbf{x}}^{(f)}_{e\sigma}).

It is now possible to compute an approximation for the divergence of the continuous flux. The procedure is directly analogous to the one used to calculate the transformed gradient in (14)

(∇~⋅f~)eρnα(u)=[∇~⋅geσ(f)(x~){F~αf~eσnα(f1)−f~eσnα(f1)}+f~eνnα(u)⋅∇~ℓeν(u)(x~)]x~=x~eρ(u),(23)\left(\widetilde{\nabla}\cdot\widetilde{\mathbf{f}}\right)^{(u)}_{e\rho n\alpha}=\left[\widetilde{\nabla}\cdot\mathbf{g}^{(f)}_{e\sigma}(\tilde{\mathbf{x}})\left\{\widetilde{\mathcal{F}}_\alpha\widetilde{f}^{(f1)}_{e\sigma n\alpha}-\widetilde{f}^{(f1)}_{e\sigma n\alpha}\right\}+\widetilde{f}^{(u)}_{e\nu n\alpha}\cdot\widetilde{\nabla}\ell^{(u)}_{e\nu}(\tilde{\mathbf{x}})\right]_{\tilde{\mathbf{x}}=\tilde{\mathbf{x}}^{(u)}_{e\rho}}, \tag*{(23)}

which can then be used to obtain a semi-discretised form of the governing system

∂ueρnα(u)∂t=−Jeρn−1(u)(∇~⋅f~)eρnα(u),(24)\frac{\partial u^{(u)}_{e\rho n\alpha}}{\partial t}=-J^{-1(u)}_{e\rho n}\left(\widetilde{\nabla}\cdot\widetilde{\mathbf{f}}\right)^{(u)}_{e\rho n\alpha}, \tag*{(24)}

where Jeρn−1(u)=det⁡Jen−1(x~eρ(u))=1/Jeρn(u)J^{-1(u)}_{e\rho n}=\det J^{-1}_{en}(\tilde{\mathbf{x}}^{(u)}_{e\rho})=1/J^{(u)}_{e\rho n}. This semi-discretised form is simply a system of ordinary differential equations in tt and can be solved using one of a number of schemes, e.g. a classical fourth order Runge-Kutta (RK4) scheme.

Implementation

Overview

PyFR is a Python based implementation of the FR approach described in section section 2. It is designed to be compact, efficient, and platform portable. Key functionality is summarised in Table 1.

δij\delta_{ij}Kronecker deltannElement number
det⁡A\det\mathbf{A}Matrix determinantα\alphaField variable number
dim⁡A\dim\mathbf{A}Matrix dimensionsi,j,ki,j,kSummation indices
ρ,σ,ν\rho,\sigma,\nuSummation indices

Table 1.

Dimensions2D, 3D
ElementsTriangles, Quadrilaterals, Hexahedra
Spatial ordersArbitrary
Time steppersEuler, RK4, DOPRI5
PrecisionsSingle, Double
PlatformsCPUs via C/OpenMP, Nvidia GPUs via CUDA
CommunicationMPI
Governing SystemsEuler, Compressible Navier-Stokes

Table 1. Key functionality of PyFR.

The majority of operations within an FR step can be cast in terms of matrix-matrix multiplications, as detailed in Appendix A. All remaining operations (e.g. flux evaluations) are point-wise, concerning themselves with either a single solution point inside of an element or two collocating flux points at an interface. Hence, in broad terms, there are five salient aspects of an FR implementation, specifically i.) definition of the constant operator matrices detailed in Appendix A, ii.) specification of the state matrices detailed in Appendix A, iii.) implementation of matrix multiply kernels, iv.) implementation of point-wise kernels, and finally v.) handling of distributed memory parallelism and scheduling of kernel invocations. Details regarding how each of the above were achieved in PyFR are presented below.

Definition of Constant Operator Matrices

Setup of the seven constant operator matrices detailed in Appendix A requires evaluation of various polynomial expressions, and their derivatives, at solution/flux points within each type of standard element. Although conceptually simple, such operations can be cumbersome to code. To keep the codebase compact PyFR makes extensive use of symbolic manipulation via SymPy [16], which brings computer algebra facilities similar to those found in Maple and Mathematica to Python. SymPy has built-in support for most common polynomials and can readily evaluate such expressions to arbitrary precision. Efficiency of the setup phase is not critical, since the operations are only performed once at start-up. Since efficiency is not critical, platform portability is effectively achieved by running such operations on the host CPU in all cases.

Specification of State Matrices

In specifying the state matrices detailed in Appendix A there is a degree of freedom regarding how the field variables of each element are packed along a row. The packing of field variables can be characterised by considering the distance, Δj\Delta_j (in columns) between two subsequent field variables for a given element. The case of Δj=1\Delta_j = 1 corresponds to the array of structures (AoS) packing whereas the choice of Δj=∣Ωe∣\Delta_j = \lvert\Omega_e\rvert leads to the structure of arrays (SoA) packing. A hybrid approach wherein Δj=k\Delta_j = k with kk being constant results in the AoSoA(kk) approach. An implementation is free to chose between any of these counting patterns so long as it is consistent. For simplicity PyFR uses the SoA packing order across all platforms.

Matrix Multiplication Kernels

PyFR defers matrix multiplication to the GEMM family of sub-routines provided a suitable Basic Linear Algebra Subprograms (BLAS) library. BLAS is available for virtually all platforms and optimised versions are often maintained by the hardware vendors themselves (e.g. cuBLAS for Nvidia GPUs). This approach greatly facilitates development of efficient and platform portable code. We note, however, that the matrix sizes encountered in PyFR are not necessarily optimal from a GEMM perspective. Specifically, GEMM is optimised for the multiplication of large square matrices, whereas the constant operator matrices in PyFR are ‘small and square’ with 10–100 rows/columns, and the state matrices are ‘short and fat’ with 10–100 rows and 10 000–100 000 columns. Moreover, we note that the constant operator matrices are know a priori, and do not change in time. This a priori knowledge could, in theory, be leveraged to design bespoke matrix multiply kernels that are more efficient than GEMM. Development of such bespoke kernels will be a topic of future research - with results easily integrated into PyFR as an optional replacement for GEMM.

Point-Wise Kernels

Point-wise kernels are specified using a domain specific language implemented in PyFR atop of the Mako templating engine [17]. The templated kernels are then interpreted at runtime, converted to low-level code, compiled, linked and loaded. Currently the templating engine can generate C/OpenMP to target CPUs, and CUDA (via the PyCUDA wrapper [18]) to target Nvidia GPUs. Use of a domain specific language avoids implementation of each point-wise kernel for each target platform; keeping the codebase compact and platform portable. Runtime code generation also means it is possible to instruct the compiler to emit binaries which are optimised for the current hardware architecture. Such optimisations can result in anything up to a fourfold improvement in performance when compared with architectural defaults.

As an example of a point-wise kernel we consider the evaluation of the right hand side of (24), which reads −Jep−1 (u)(∇~⋅f~)epnα(u)-J_{ep}^{-1\,(u)}\left(\widetilde{\nabla}\cdot\widetilde{\mathbf{f}}\right)_{epn\alpha}^{(u)}. The operation consists of a point-wise multiplication between the negative reciprocal of the Jacobian and the transformed divergence of the flux at each solution point. Figure 3 shows how such a kernel can be expressed in the domain specific language of PyFR. There are several points of note. Firstly, the kernel is purely scalar in nature. This is by design; in PyFR point-wise kernels need only prescribe the point-wise operation to be applied. Important choices such as how to vectorise a given operation or how to gather data from memory are all delegated to templating engine. Secondly, we note it is possible to utilise Python when generating the main body of kernels. This capability is showcased on lines four, five and six where it is used to unroll a for loop over each of the field variables. Finally, we also highlight the use of an abstract data type fpdtypet\mathit{fpdtype}_{t} for floating point variables which permits a single set of kernels to be used for both single and double precision operation. Generated CUDA source

Figure 3. An example of an extrinsic kernel in PyFR. The template variable nvarsnvars is taken to be the number of field variables, NvN_v. The kernel arguments tdivtconftdivtconf and rcpdjacrcpdjac correspond to ∇~⋅f~\widetilde{\nabla}\cdot\widetilde{\mathbf{f}} and J−1J^{-1} respectively with the operation being performed in-place.

<%pyfr:kernel name='negdivconf' ndim='2'
    tdivtconf='inout fpdtype_t[${str(nvars)}]'
    rcpdjac='in fpdtype_t'>
% for i in range(nvars):
    tdivtconf[${i}] *= -rcpdjac;
% endfor
</%pyfr:kernel>

for this kernel can be seen in Figure 4, and the equivalent C kernel can be found in Figure 5.

Generated CUDA sourceVector (PDF)
__global__ void
negdivconf(int _ny, int _nx,
           const fpdtype_t* __restrict__ rcpdjac_v,
           int ldrcpdjac,
           fpdtype_t* __restrict__ tdivtconf_v,
           int ldtdivtconf)
{
    int _x = blockIdx.x*blockDim.x + threadIdx.x;

    for (int _y = 0; _y < _ny && _x < _nx; ++_y)
    {
        fpdtype_t rcpdjac, tdivtconf[4];

        // Load rcpdjac
        rcpdjac = rcpdjac_v[ldrcpdjac*_y + _x];

        // Load tdivtconf
        tdivtconf[0] = tdivtconf_v[ldtdivtconf*_y + _x + _nx*0];
        tdivtconf[1] = tdivtconf_v[ldtdivtconf*_y + _x + _nx*1];
        tdivtconf[2] = tdivtconf_v[ldtdivtconf*_y + _x + _nx*2];
        tdivtconf[3] = tdivtconf_v[ldtdivtconf*_y + _x + _nx*3];

        tdivtconf[0] *= -rcpdjac;
        tdivtconf[1] *= -rcpdjac;
        tdivtconf[2] *= -rcpdjac;
        tdivtconf[3] *= -rcpdjac;

        // Store tdivtconf
        tdivtconf_v[ldtdivtconf*_y + _x + _nx*0] = tdivtconf[0];
        tdivtconf_v[ldtdivtconf*_y + _x + _nx*1] = tdivtconf[1];
        tdivtconf_v[ldtdivtconf*_y + _x + _nx*2] = tdivtconf[2];
        tdivtconf_v[ldtdivtconf*_y + _x + _nx*3] = tdivtconf[3];
    }
}

Figure 4. Generated CUDA source for the template in Figure 3 for when NV=4N_V = 4.

Generated OpenMP annotated C source codeVector (PDF)
static PYFR_NOINLINE void
negdivconf_inner(int _nx,
                 const fpdtype_t *__restrict__ rcpdjac_v,
                 fpdtype_t *__restrict__ tdivtconf_v0,
                 fpdtype_t *__restrict__ tdivtconf_v1,
                 fpdtype_t *__restrict__ tdivtconf_v2,
                 fpdtype_t *__restrict__ tdivtconf_v3)
{
    PYFR_ALIGNED(rcpdjac_v);
    PYFR_ALIGNED(tdivtconf_v0);
    PYFR_ALIGNED(tdivtconf_v1);
    PYFR_ALIGNED(tdivtconf_v2);
    PYFR_ALIGNED(tdivtconf_v3);

    for (int _x = 0; _x < _nx; _x++)
    {
        fpdtype_t rcpdjac, tdivtconf[4];

        // Load rcpdjac
        rcpdjac = rcpdjac_v[_x];

        // Load tdivtconf
        tdivtconf[0] = tdivtconf_v0[_x];
        tdivtconf[1] = tdivtconf_v1[_x];
        tdivtconf[2] = tdivtconf_v2[_x];
        tdivtconf[3] = tdivtconf_v3[_x];

        tdivtconf[0] *= -rcpdjac;
        tdivtconf[1] *= -rcpdjac;
        tdivtconf[2] *= -rcpdjac;
        tdivtconf[3] *= -rcpdjac;

        // Store tdivtconf
        tdivtconf_v0[_x] = tdivtconf[0];
        tdivtconf_v1[_x] = tdivtconf[1];
        tdivtconf_v2[_x] = tdivtconf[2];
        tdivtconf_v3[_x] = tdivtconf[3];
    }
}

void
negdivconf(int _ny, int _nx,
           const fpdtype_t* __restrict__ rcpdjac_v,
           int lsdrcpdjac,
           fpdtype_t* __restrict__ tdivtconf_v,
           int lsdtdivtconf)
{
    #pragma omp parallel for
    for (int _y = 0; _y < _ny; _y++)
        negdivconf_inner(_nx, rcpdjac_v + _y*lsdrcpdjac,
                         tdivtconf_v + (_y*4 + 0)*lsdtdivtconf,
                         tdivtconf_v + (_y*4 + 1)*lsdtdivtconf,
                         tdivtconf_v + (_y*4 + 2)*lsdtdivtconf,
                         tdivtconf_v + (_y*4 + 3)*lsdtdivtconf);
}

Figure 5. Generated OpenMP annotated C source code for the template in Figure 3 for when NV=4N_V = 4. The somewhat unconventional structure is necessary to ensure that the kernel is properly vectorised across a range of compilers.

Distributed Memory Parallelism and Scheduling

PyFR is capable of operating on heigh performance computing clusters utilising distributed memory parallelism. This is accomplished through the Message Passing Interface (MPI). All MPI functionality is implemented at the Python level through the mpi4py [19] wrapper. To enhance the scalability of the code care has been taken to ensure that all requests are persistent, point-to-point and non-blocking. Further, the format of data that is shared between ranks has been made backend independent. It is therefore possible to deploy PyFR on heterogeneous clusters consisting of both conventional CPUs and accelerators.

The arrangement of kernel calls required to solve an advection-diffusion problem can be seen in Figure 6. Our primary objective when scheduling kernels was to maximise the potential for overlapping communication with computation. In order to help achieve this the common interface solution, U~α\widetilde{U}_{\alpha}, and common interface flux, F~α\widetilde{F}_{\alpha}, kernels have been broken apart into two separate kernels; suffixed in the figure by INT and MPI. PyFR is therefore able to perform a significant degree of rank-local computation while the relevant ghost states are being exchanged.

Figure 6. Flow diagram showing the stages required to compute −∇⋅f-\nabla\cdot\mathbf{f}. Symbols correspond to those of Appendix A. For simplicity arguments referencing constant data have been omitted. Memory indirection is indicated by red underlines. Synchronisation points are signified by black horizontal lines. Dotted lines correspond to data reuse.

Our secondary objective when scheduling kernels was to minimise the amount of temporary storage required during the evaluation of −∇⋅f-\nabla\cdot\mathbf{f}. Such optimisations are critical within the context of accelerators which often have an order of magnitude less memory than a contemporary platform. In order to help achieve this U(u)\mathbf{U}^{(u)}, R~(u)\widetilde{\mathbf{R}}^{(u)}, and R(u)\mathbf{R}^{(u)} are allowed to alias. By permitting the same storage location to be used for both the inputted solution and the outputted flux divergence it is possible to reduce the storage requirements of the RK schemes. Another opportunity for memory reuse is in the transformed flux function where the incoming gradients, Q(u)\mathbf{Q}^{(u)}, can be overwritten with the transformed flux, F~(u)\widetilde{\mathbf{F}}^{(u)}. A similar approach can be used in the common interface flux function whereby U(f)\mathbf{U}^{(f)} can updated in-place with the entires of D~(f)\widetilde{\mathbf{D}}^{(f)} which holds the transformed common normal flux. Moreover, C(f)\mathbf{C}^{(f)} is also able to utilise the same storage as the somewhat larger Q(f)\mathbf{Q}^{(f)} array. These optimisations allow PyFR to process over 100 000 curved, unstructured, hexahedral

elements at ℘=3\wp= 3 inside of a 5 GiB memory footprint.

Governing Systems

Overview

PyFR is a framework for solving various advection-diffusion type problems. In the current release of PyFR two specific governing systems can be solved, specifically the Euler equations for inviscid compressible flow, and the compressible Navier-Stokes equations for viscous compressible flow. Details regarding both are given below.

Euler Equations

Using the framework introduced in section 2 the three dimensional Euler equations can be expressed in conservative form as

u=(ρρvxρvyρvzE),f=f(inv)=(ρvxρvyρvzρvx2+pρvyvxρvzvxρvxvyρvy2+pρvzvyρvxvzρvyvzρvz2+pvx(E+p)vy(E+p)vz(E+p)),(25)u = \begin{pmatrix} \rho\\ \rho v_x \\ \rho v_y \\ \rho v_z \\ E \end{pmatrix}, \qquad \mathbf{f} = \mathbf{f}^{(\mathrm{inv})} = \begin{pmatrix} \rho v_x & \rho v_y & \rho v_z \\ \rho v_x^2 + p & \rho v_y v_x & \rho v_z v_x \\ \rho v_x v_y & \rho v_y^2 + p & \rho v_z v_y \\ \rho v_x v_z & \rho v_y v_z & \rho v_z^2 + p \\ v_x(E+p) & v_y(E+p) & v_z(E+p) \end{pmatrix}, \tag*{(25)}

with uu and f\mathbf{f} together satisfying (1). In the above ρ\rho is the mass density of the fluid, v=(vx,vy,vz)T\mathbf{v} = (v_x, v_y, v_z)^T is the fluid velocity vector, EE is the total energy per unit volume and pp is the pressure. For a perfect gas the pressure and total energy can be related by the ideal gas law

E=pγ−1+12ρ∥v∥2,(26)E = \frac{p}{\gamma- 1} + \frac{1}{2}\rho\lVert\mathbf{v} \rVert^2, \tag*{(26)}

with γ=Cp/Cv\gamma= C_p/C_v.

With the fluxes specified all that remains is to prescribe a method for computing the common normal flux, Fα\mathcal{F}_\alpha, at interfaces as defined in (20). This can be accomplished using an approximate Riemann solver for the Euler equations. There exist a variety of such solvers as detailed in [20]. A description of those implemented in PyFR can be found in Appendix B.

Compressible Navier-Stokes Equations

The compressible Navier-Stokes equations can be viewed as an extension of the Euler equations via the inclusion of viscous terms. Within the framework outlined above the flux now takes the form of f=f(inv)−f(vis)\mathbf{f} = \mathbf{f}^{(\mathrm{inv})} - \mathbf{f}^{(\mathrm{vis})} where

f(vis)={000TxxTyxTzxTxyTyyTzyTxzTyzTzzviTix+Δ∂xTviTiy+Δ∂yTviTiz+Δ∂zT}.(27)\mathbf{f}^{(\mathrm{vis})} = \left\{ \begin{array}{ccc} 0 & 0 & 0 \\ \mathcal{T}_{xx} & \mathcal{T}_{yx} & \mathcal{T}_{zx} \\ \mathcal{T}_{xy} & \mathcal{T}_{yy} & \mathcal{T}_{zy} \\ \mathcal{T}_{xz} & \mathcal{T}_{yz} & \mathcal{T}_{zz} \\ v_i\mathcal{T}_{ix} + \Delta\partial_x T & v_i\mathcal{T}_{iy} + \Delta\partial_y T & v_i\mathcal{T}_{iz} + \Delta\partial_z T \end{array} \right\}. \tag*{(27)}

In the above we have defined Δ=μCp/Pr\Delta= \mu C_p/Pr where μ\mu is the dynamic viscosity and PrPr is the Prandtl number. The components of the stress-energy tensor are given by

Tij=μ(∂ivj+∂jvi)−23μδij∇⋅v.(28)\mathcal{T}_{ij} = \mu(\partial_i v_j + \partial_j v_i) - \frac{2}{3}\mu\delta_{ij}\nabla\cdot\mathbf{v}. \tag*{(28)}

Using the ideal gas law the temperature can be expressed as

T=1Cv1γ−1pρ,(29)T = \frac{1}{C_v}\frac{1}{\gamma- 1}\frac{p}{\rho}, \tag*{(29)}

with partial derivatives thereof being given according to the quotient rule.

Since the Navier-Stokes equations are an advection-diffusion type system it is necessary to both compute a common solution (Cα\mathcal{C}_\alpha of (12)) at element boundaries and augment the inviscid Riemann solver to handle the viscous part of the flux. A popular approach is the LDG method as presented in [5, 13]. In this approach the common solution is given ∀α\forall\alpha according to

C(uL,uR)=(12−β)uL+(12+β)uR,(30)\mathcal{C}(u_L,u_R) = \left(\frac{1}{2}-\beta\right)u_L + \left(\frac{1}{2}+\beta\right)u_R, \tag*{(30)}

where β\beta controls the degree of upwinding/downwinding. The common normal interface flux is then prescribed, once again ∀α\forall\alpha, according to

F(uL,uR,qL,qR,n^L)=F(inv)−F(vis),(31)\mathcal{F}(u_L,u_R,\mathbf{q}_L,\mathbf{q}_R,\hat{\mathbf{n}}_L) = \mathcal{F}^{(\mathrm{inv})} - \mathcal{F}^{(\mathrm{vis})}, \tag*{(31)}

where F(inv)\mathcal{F}^{(\mathrm{inv})} is a suitable inviscid Riemann solver (see Appendix B) and

F(vis)=n^L⋅{(12+β)fL(vis)+(12−β)fR(vis)}+τ(uL−uR),(32)\mathcal{F}^{(\mathrm{vis})} = \hat{\mathbf{n}}_L \cdot\left\{\left(\frac{1}{2}+\beta\right)\mathbf{f}^{(\mathrm{vis})}_L + \left(\frac{1}{2}-\beta\right)\mathbf{f}^{(\mathrm{vis})}_R\right\} + \tau(u_L-u_R), \tag*{(32)}

with τ\tau being a penalty parameter, fL(vis)=f(vis)(uL,qL)\mathbf{f}^{(\mathrm{vis})}_L = \mathbf{f}^{(\mathrm{vis})}(u_L,\mathbf{q}_L), and fR(vis)=f(vis)(uR,qR)\mathbf{f}^{(\mathrm{vis})}_R = \mathbf{f}^{(\mathrm{vis})}(u_R,\mathbf{q}_R). We observe here that if the common solution is upwinded then the common normal flux will be downwinded. Generally, β=±1/2\beta= \pm1/2 as this results in the numerical scheme having a compact stencil and 0≤τ≤10 \leq\tau\leq1.

Presentation in Two Dimensions

The above prescriptions of the Euler and Navier-Stokes equations are valid for the case of ND=3N_D = 3. The two dimensional formulation can be recovered by deleting the fourth rows in the definitions of u\mathbf{u}, f(inv)\mathbf{f}^{(\mathrm{inv})} and f(vis)\mathbf{f}^{(\mathrm{vis})} along with the third columns of f(inv)\mathbf{f}^{(\mathrm{inv})} and f(vis)\mathbf{f}^{(\mathrm{vis})}. Vectors are now two dimensional with the velocity being given by v=(vx,vy)T\mathbf{v} = (v_x,v_y)^T.

Validation

Euler Equations: Euler Vortex Super Accuracy

Various authors [4, 10] have shown FR schemes exhibit so-called ‘super accuracy’ (an order of accuracy greater than the expected ℘+1\wp+ 1). To confirm PyFR can achieve super accuracy for the Euler equations a square domain Ω=[−20,20]2\Omega= [-20,20]^2 was decomposed into four structured quad meshes with spacings of h=1/3h = 1/3, h=2/7h = 2/7, h=1/4h = 1/4, and h=2/9h = 2/9. Initial conditions were taken to be those of an isentropic Euler vortex in a free-stream

ρ(x,t=0)={1−S2M2(γ−1)exp⁡2f8π2}1γ−1,(33)\rho(\mathbf{x},t=0)=\left\{1-\frac{S^2M^2(\gamma-1)\exp2f}{8\pi^2}\right\}^{\frac{1}{\gamma-1}}, \tag*{(33)}
v(x,t=0)=Syexp⁡f2πRx^+{1−Sxexp⁡f2πR}y^,(34)\mathbf{v}(\mathbf{x},t=0)=\frac{S y\exp f}{2\pi R}\hat{\mathbf{x}}+\left\{1-\frac{S x\exp f}{2\pi R}\right\}\hat{\mathbf{y}}, \tag*{(34)}
p(x,t=0)=ργγM2,(35)p(\mathbf{x},t=0)=\frac{\rho^\gamma}{\gamma M^2}, \tag*{(35)}

where f=(1−x2−y2)/2R2f=(1-x^2-y^2)/2R^2, S=13.5S=13.5 is the strength of the vortex, M=0.4M=0.4 is the free-stream Mach number, and R=1.5R=1.5 is the radius. All meshes were configured with periodic boundary conditions along boundaries of constant xx. Along boundaries of constant yy the dynamical variables were fixed according to

ρ(x=xx^±20y^,t)=1,\rho(\mathbf{x}=x\hat{\mathbf{x}}\pm20\hat{\mathbf{y}},t)=1,
v(x=xx^±20y^,t)=y^,\mathbf{v}(\mathbf{x}=x\hat{\mathbf{x}}\pm20\hat{\mathbf{y}},t)=\hat{\mathbf{y}},
p(x=xx^±20y^,t)=1γM2,p(\mathbf{x}=x\hat{\mathbf{x}}\pm20\hat{\mathbf{y}},t)=\frac{1}{\gamma M^2},

which are simply the limiting values of the initial conditions. Strictly speaking these conditions, on account of the periodicity, result in the modelling of an infinite array of coupled vortices. The impact of this is mitigated by the observation that the exponentially decaying vortex has a characteristic radius which is far smaller than the extent of the domain. Neglecting these effects the analytic solution of the system is a time tt is simply a translation of the initial conditions.

Using the analytical solution we can define an L2L^2 error as

σ(t)2=∫−22∫−22[ρδ(x+Δy(t)y^,t)−ρ(x,t=0)]2 d2x,(36)\sigma(t)^2=\int_{-2}^{2}\int_{-2}^{2}\left[\rho^\delta(\mathbf{x}+\Delta_y(t)\hat{\mathbf{y}},t)-\rho(\mathbf{x},t=0)\right]^2\,\mathrm{d}^2\mathbf{x}, \tag*{(36)}

where ρδ(x,t)\rho^\delta(\mathbf{x},t) is the numerical mass density, ρ(x,t=0)\rho(\mathbf{x},t=0) the analytic mass density, and Δy(t)\Delta_y(t) is the ordinate corresponding to the centre of the vortex at a time tt and accounts for the fact that the vortex is translating in a free stream velocity of unity in the yy direction. Restricting the region of consideration to a small box centred around the origin serves to further mitigate against the effects of vortices coupling together. The initial mass density along with the [−2,−2]×[2,2][-2,-2]\times[2,2] region used to evaluate the error can be seen in Figure 7. At times, tct_c, when the vortex is centred on the box the error can be readily computed by integrating over each element inside the box and summing the residuals

Figure 7. Initial density profile for the vortex in Ω\Omega. The black box shows the region where the error is calculated.

σ(tc)2=∬Ωe[ρiδ(x~,tc)−ρ(Mi(x~),0)]2Ji(x~) d2x~,(37)\sigma(t_c)^2 = \iint_{\Omega_e} \left[\rho_i^\delta(\tilde{x},t_c)-\rho(\mathcal{M}_i(\tilde{x}),0)\right]^2 J_i(\tilde{x})\,\mathrm{d}^2\tilde{x}, \tag*{(37)}

where, ρiδ(x~,tc)\rho_i^\delta(\tilde{x},t_c) is the approximate mass density inside of the ith element, and Ji(x~)J_i(\tilde{x}) the associated Jacobian. These integrals can be approximated by applying Gaussian quadrature

σ(tc)2≈Ji(x~j)[ρiδ(x~j,tc)−ρ(Mi(x~j),0)]2ωj=h24[ρiδ(x~j,tc)−ρ(Mi(x~j),0)]2ωj,(38)\begin{aligned} \sigma(t_c)^2 &\approx J_i(\tilde{x}_j)\left[\rho_i^\delta(\tilde{x}_j,t_c)-\rho(\mathcal{M}_i(\tilde{x}_j),0)\right]^2\omega_j \\ &= \frac{h^2}{4}\left[\rho_i^\delta(\tilde{x}_j,t_c)-\rho(\mathcal{M}_i(\tilde{x}_j),0)\right]^2\omega_j, \tag*{(38)} \end{aligned}

where {x~j}\{\tilde{x}_j\} are abscissa and {ωj}\{\omega_j\} the weights of a rule determined for integration inside of a standard quadrilateral. So long as the rule used is of a suitable strength then this will be a very good approximation of the true L2L^2 error.

Following [10] the initial conditions were laid onto the mesh using a collocation projection with ℘=3\wp= 3. The simulation was then run with three different flux reconstruction schemes: DG, SD, and HU as defined in [10]. Solution points were placed at a tensor product construction of Gauss-Legendre quadrature points. Common interface fluxes were computed using a Rusanov Riemann solver. To advance the solutions in time a classical fourth order Runge-Kutta method (RK4) was used. The time step was taken to be Δt=0.00125\Delta t = 0.00125 with t=0..1800t = 0..1800 with solutions written out to disk every 32 000 steps. The order of accuracy of the scheme at a particular time can be determined by plotting log⁡σ\log\sigma against log⁡h\log h and performing a least-squares fit through the four data points. The order is given by the gradient of the fit. A plot of order of accuracy against time for the three schemes can be seen in Figure 8. We note that the order of accuracy changes as a function of time. This is due to the fact that the error is actually of the form σ(t)=σp+σso(t)\sigma(t) = \sigma_{\mathrm{p}} + \sigma_{\mathrm{so}}(t) where σp\sigma_{\mathrm{p}} is a constant projection error and σso\sigma_{\mathrm{so}} is a time-dependent spatial operator error. The projection error arises as a consequence of the forth order collocation projection of the initial conditions onto the mesh. Over time the spatial operator error grows in magnitude and eventually dominates. Only when σso(t)≫σp\sigma_{\mathrm{so}}(t) \gg\sigma_{\mathrm{p}} can the true order of the method be observed. The results here can be seen to be in excellent agreement with those of [10].

Figure 8. Spatial super accuracy observed for a ℘=3\wp= 3 simulation using DG, SD and HU as defined in [10].

Compressible Navier-Stokes Equations: Couette Flow

Consider the case in which two parallel plates of infinite extent are separated by a distance HH in the yy direction. We treat both plates as isothermal walls at a temperature TwT_w and permit the top plate to move at a velocity vwv_w in the xx direction with respect to the bottom plate. For simplicity we shall take the ordinate of the bottom plate as zero. In the case of a constant viscosity μ\mu the Navier-Stokes equations admit an analytical solution in which

ρ(ϕ)=γγ−12p2CpTw+Pr⁡vw2ϕ(1−ϕ)(39)\rho(\phi) = \frac{\gamma}{\gamma- 1}\frac{2p}{2C_pT_w + \operatorname{Pr}v_w^2\phi(1-\phi)} \tag*{(39)}
v(ϕ)=vwϕx^(40)\mathbf{v}(\phi) = v_w\phi\hat{x} \tag*{(40)}
p=pc(41)p = p_c \tag*{(41)}

where ϕ=y/H\phi= y/H and pcp_c is a constant pressure. The total energy is given by the ideal gas law of (26). On a finite domain the Couette flow problem can be modelled through the imposition of periodic boundary conditions. For a two dimensional mesh periodicity is enforced in xx whereas for three dimensional meshes it is enforced in both xx and zz. To validate the Navier-Stokes solver in PyFR we take γ=1.4\gamma= 1.4, Pr=0.72Pr = 0.72, μ=0.417\mu= 0.417, Cp=1005 J K−1C_p = 1005\ \mathrm{J\,K}^{-1}, H=1 mH = 1\ \mathrm{m}, Tw=300 KT_w = 300\ \mathrm{K}, pc=1×105 Pap_c = 1 \times10^5\ \mathrm{Pa}, and vw=69.445 m s−1v_w = 69.445\ \mathrm{m\,s}^{-1}. These values correspond to a Mach number of 0.2 and a Reynolds number of 200. The plates were modelled as no-slip isothermal walls as detailed in subsection C.4 of Appendix C. A plot of the resulting energy profile can be seen in Figure 9. Constant initial conditions are taken as ρ=⟨ρ(ϕ)⟩\rho= \langle\rho(\phi) \rangle, v=vwx^\mathbf{v} = v_w \hat{\mathbf{x}}, and p=pcp = p_c. Using the analytical solution we again define an L2L^2 error as

Converged steady state energy profile for the two dimensional Couette flow problem

Figure 9. Converged steady state energy profile for the two dimensional Couette flow problem.

σ(t)2=∫Ω[Eδ(x,t)−E(x)]2 dNDx(42)\sigma(t)^2 = \int_{\Omega} \left[E^\delta(\mathbf{x},t) - E(\mathbf{x})\right]^2 \,\mathrm{d}^{N_D}\mathbf{x} \tag*{(42)}
=∫Ωei[Eeiδ(x~,t)−E(Mei(x~))]2Jei(x~) dNDx~(43)= \int_{\Omega_{e_i}} \left[E^\delta_{e_i}(\tilde{\mathbf{x}},t) - E(\mathcal{M}_{e_i}(\tilde{\mathbf{x}}))\right]^2 J_{e_i}(\tilde{\mathbf{x}})\,\mathrm{d}^{N_D}\tilde{\mathbf{x}} \tag*{(43)}
≈[Eeiδ(x~ej,t)−E(Mei(x~ej))]2Jei(x~ej)ωej,(44)\approx\left[E^\delta_{e_i}(\tilde{\mathbf{x}}_{e_j},t) - E(\mathcal{M}_{e_i}(\tilde{\mathbf{x}}_{e_j}))\right]^2 J_{e_i}(\tilde{\mathbf{x}}_{e_j})\omega_{e_j}, \tag*{(44)}

where Ω\Omega is the computational domain, Eδ(x,t)E^\delta(\mathbf{x},t) is the numerical total energy, and E(x)E(\mathbf{x}) the analytic total energy. In the third step we have approximated each integral by a quadrature rule with abscissa {x~ej}\{\tilde{\mathbf{x}}_{e_j}\} and weights {ωej}\{\omega_{e_j}\} inside of an element type ee. Couette flow is a steady state problem and so in the limit of t→∞t \to\infty the numerical total energy should converge to a solution. Starting from a constant initial condition the L2L^2 error was computed every 0.1 time units. The simulation was said to have converged when σ(t)/σ(t+0.1)≤1.01\sigma(t)/\sigma(t + 0.1) \le1.01 where σ\sigma is the L2L^2 error. We will denote the time at which this occurs by t∞t_\infty.

Once the system has converged for a range of meshes it is possible to compute the order of accuracy of the scheme. For a given P\mathcal{P} this is the slope (plus or minus a standard error) of a linear least squares fit of log⁡h∼log⁡σ(t∞)\log h \sim\log\sigma(t_\infty) where hh is an approximation of the characteristic grid spacing. The expected order of accuracy is P+1\mathcal{P} + 1. In all simulations inviscid fluxes were computed using the Rusanov approach and the LDG parameters were taken to be β=1/2\beta= 1/2 and τ=0.1\tau= 0.1. All simulations were performed with DG correction functions and at double precision. Inside tensor product elements Gauss-Legendre solution and flux points were employed. Triangular elements utilised Williams-Shunn solution points and Gauss-Legendre flux points.

Two dimensional unstructured mixed mesh. For the two dimensional test cases the computational domain was taken to be [−1,1]×[0,1][-1,1] \times[0,1]. This domain was then meshed with both triangles and quadrilaterals at four different refinement levels. The Couette flow problem described above was then solved on each of these meshes. Experimental L2L^2 errors and orders of accuracy can be seen in Table 2. We note that in all cases the expected order of accuracy was obtained.

eeElement type

Table 2.

σ(t∞)\sigma(t_{\infty}) / J m−3\mathrm{J}\,\mathrm{m}^{-3}
TrisQuads℘=1\wp= 1℘=2\wp= 2℘=3\wp= 3℘=4\wp= 4
281.26×1021.26 \times10^{2}5.77×10−15.77 \times10^{-1}5.54×10−35.54 \times10^{-3}6.62×10−56.62 \times10^{-5}
6223.56×1013.56 \times10^{1}1.40×10−11.40 \times10^{-1}6.72×10−46.72 \times10^{-4}3.91×10−63.91 \times10^{-6}
10372.08×1012.08 \times10^{1}4.35×10−24.35 \times10^{-2}2.54×10−42.54 \times10^{-4}8.16×10−78.16 \times10^{-7}
16561.46×1011.46 \times10^{1}3.52×10−23.52 \times10^{-2}1.09×10−41.09 \times10^{-4}4.62×10−74.62 \times10^{-7}
Order2.21±0.122.21 \pm0.122.99±0.322.99 \pm0.323.97±0.053.97 \pm0.055.20±0.385.20 \pm0.38

Table 2. L2L^2 energy error and orders of accuracy for the Couette flow problem on four mixed meshes. The mesh spacing was approximated as h∼NE−1/2h \sim N_E^{-1/2} where NEN_E is the total number of elements in the mesh.

Three dimensional extruded hexahedral mesh. For this three dimensional case the computational domain was taken to be [−1,1]×[0,1]×[0,1][-1,1] \times[0,1] \times[0,1]. Meshes were constructed through first generating a series of unstructured quadrilateral meshes in the xx-yy plane. A three layer extrusion was then performed on this meshes to yield

Figure 10. Unstructured mixed element meshes used for the two dimensional Couette flow problem.

a series of hexahedral meshes. Experimental L2L^2 errors and orders of accuracy for these meshes can be seen in Table 3.

Ω\OmegaSolution domain

Table 3.

Hexes℘=1\wp= 1℘=2\wp= 2℘=3\wp= 3
783.35×1013.35 \times10^{1}5.91×10−25.91 \times10^{-2}7.28×10−47.28 \times10^{-4}
1951.23×1011.23 \times10^{1}1.87×10−21.87 \times10^{-2}1.15×10−41.15 \times10^{-4}
4056.15×1006.15 \times10^{0}5.49×10−35.49 \times10^{-3}2.72×10−52.72 \times10^{-5}
Order2.06±0.082.06 \pm0.082.87±0.242.87 \pm0.243.99±0.033.99 \pm0.03

Table 3. L2L^2 energy errors and orders of accuracy for the Couette flow problem on three extruded hexahedral meshes. On account of the extrusion h∼NE−1/2h \sim N_E^{-1/2} where NEN_E is the total number of elements in the mesh.

Three dimensional unstructured hexahedral mesh. As a further test a domain of dimension [0,1]3[0, 1]^3 was considered. This domain was meshed using completely unstructured hexahedra. Three levels of refinement were used resulting in meshes with 96, 536 and 1004 elements. A cutaway of the most refined mesh can be seen in Figure 11. Experimental L2L^2 errors and the resulting orders of accuracy are presented in Table 4. Despite the fully unstructured nature of the mesh the expected order of accuracy was again obtained in all cases. We do, however, note the higher standard errors associated with these results.

Cutaway of the unstructured hexahedral mesh

Figure 11. Cutaway of the unstructured hexahedral mesh with 1004 elements.

HexesP=1\mathcal{P}=1P=2\mathcal{P}=2P=3\mathcal{P}=3
961.91×1011.91 \times10^{1}4.32×10−24.32 \times10^{-2}5.83×10−45.83 \times10^{-4}
5368.20×1008.20 \times10^{0}9.11×10−39.11 \times10^{-3}6.89×10−56.89 \times10^{-5}
10043.82×1003.82 \times10^{0}3.22×10−33.22 \times10^{-3}2.04×10−52.04 \times10^{-5}
Order1.93±0.461.93 \pm0.463.19±0.483.19 \pm0.484.16±0.444.16 \pm0.44

Table 4. L2L^{2} energy errors and orders of accuracy for the Couette flow problem on three unstructured hexahedral meshes. Mesh spacing was taken as h∼NE−1/3h \sim N_{E}^{-1/3} where NEN_{E} is the total number of elements in the mesh. The table entries are σ(t∞)/J m−3\sigma(t_{\infty})/\mathrm{J\,m^{-3}}.

Compressible Navier-Stokes Equations: Flow Over a Cylinder

In order to demonstrate the ability of PyFR to solve the unsteady Navier-Stokes equations flow over a cylinder at Reynolds number 3900 and Mach number M=0.2M=0.2 was simulated. A cylinder of radius 1/21/2 was placed at (0,0)(0,0) inside of a domain of dimension [−18,30]×[−10,10]×[0,3.2][-18,30] \times[-10,10] \times[0,3.2]. This domain was then meshed in the xx-yy plane with 4661 quadratically curved quadrilateral elements. Next, this grid was extruded along the zz-axis to yield a total of 46610 hexahedra. The grid, which can be seen in Figure 12, was partitioned into four pieces. Along surfaces of y=±10y=\pm10 and x=−18x=-18 the inflow boundary condition of subsection C.2 in Appendix C was imposed. Along the surface of x=30x=30 the outflow condition of subsection C.3 in Appendix C was used. Periodic conditions were imposed in the zz direction. On the surface of the cylinder the no-slip isothermal wall condition of subsection C.4 in Appendix C was imposed. The free-stream conditions were taken to be ρ=1\rho=1, v=x^\mathbf{v}=\hat{x}, and p=1/γM2p=1/\gamma M^{2}. These were also used as the initial conditions for the simulation. DG correction functions were used with the LDG parameters being β=1/2\beta=1/2 and τ=0.1\tau=0.1. The ratio of specific heats was taken as γ=1.4\gamma=1.4 and the Prandtl number as Pr=0.72\mathrm{Pr}=0.72.

Figure 12. Cross section in the xx-yy plane of the cylinder mesh. Colours indicate the partition to which the elements belong.

The simulation was run with P=4\mathcal{P}=4 with four NVIDIA K20c GPUs. It contained some 29×10629 \times10^{6} degrees of freedom. Isosurfaces of density captured after the turbulent wake had fully developed can be seen in Figure 13.

Isosurfaces of density around the cylinder

Figure 13. Isosurfaces of density around the cylinder.

Single Node Performance

The single node performance of PyFR has been evaluated on an NVIDIA M2090 GPU. This accelerator has a theoretical peak double precision floating point performance of 665 GFLOP/s, and when ECC is disabled the theoretical peak memory bandwidth is 177 GB/s. As points of reference we observe that cuBLAS (CUDA 5.5) is able to obtain 407 GFLOP/s when multiplying a pair of 4096×40964096 \times4096 matrices on this hardware, and the maximum device bandwidth obtainable by the bandwidth test application included with the CUDA SDK is 138.9 GiB/s when ECC is disabled.

We shall refer to these values as realisable peaks.

To conduct the evaluation a fully periodic cuboidal domain was meshed with 50 176 hexahedral elements. The double precision Navier-Stokes solver of PyFR was then run on this mesh at orders ℘=2,3,4\wp= 2, 3, 4 with β=1/2\beta= 1/2. In conducting the analysis kernels were grouped into one of three categories: matrix multiplications (DGEMM), point-wise kernels with direct memory access patterns (PD) and point-wise kernels with some level of indirect memory access (PI). Indirection arises in the computation of Fα\mathcal{F}_{\alpha} in (12) and Fα\mathcal{F}_{\alpha} in (20) and occurs as a consequence of the unstructured nature of PyFR. The resulting breakdowns of wall-clock time, memory bandwidth and floating point operations can be seen in Table 5. It is clear that he majority of floating point operations are concentrated inside the calls to DGEMM with the point-wise operations are heavily memory bandwidth bound. Of this bandwidth some ∼15%\sim15\% was ascribed to register spillage above and beyond that which can be absorbed by the L1 cache.

Order
℘=2\wp= 2℘=3\wp= 3℘=4\wp= 4
Wall time / %
DGEMM55.766.281.4
PD24.921.512.8
PI19.412.35.8
Bandwidth / GiB/s
PD125.5125.0124.8
PI124.8124.3124.2
Arithmetic / GFLOP/s
DGEMM205.3368.1305.4
PD0.70.70.7
PI0.90.80.9

Table 5. Single GPU performance of PyFR for the Navier-Stokes equations when run on an NVIDIA M2090 with ECC disabled. As the memory bandwidth requirements of DGEMM are dependent on the accumulation strategy adopted by the implementation these values have been omitted.

The high fraction of peak bandwidth obtained by the indirect kernels can be attributed to three factors. Firstly, the constant data required for calculations at ????, such as n^ϵσn(f)\hat{n}_{\epsilon\sigma n}^{(f)} and Jϵσnn(f)J_{\epsilon\sigma nn}^{(f)}, is ordered to ensure direct (coalesced) access. Secondly, at start-up PyFR attempts to determine an iteration ordering over the various flux-point pairs that will minimise the number of cache misses.

Many of the memory accesses are therefore are near-coalesced. Thirdly and finally we highlight the impressive latency-hiding capabilities of the CUDA programming model.

In line with expectations the proportion of time spent performing matrix-matrix multiplications increases as a function of order. When going from ϱ=2\varrho= 2 to ϱ=3\varrho= 3 a significant portion of the additional compute is offset by the improved performance of cuBLAS. However, when ϱ=4\varrho= 4 the performance of these kernels in absolute terms can be seen to regress slightly. This contributes to the greatly increased fraction of wall-clock time spent inside of these kernels. Nevertheless, the achieved rate of 305.4GFLOP/s is still over 75% of the realisable peak. Also in line with expectations is the invariance of the arithmetic performance of the point-wise kernels with respect to order. As the order is varied all that changes is the number of points to be processed with the operation itself remaining identical.

Scalability

The scalability of PyFR has been evaluated on the Emerald GPU cluster. It is housed at the STFC Rutherford Appleton Laboratory and based around 60 HP SL390 nodes with three NVIDIA M2090 GPUs and 24 HP SL390 nodes with eight NVIDIA M2090 GPUs. Nodes are connected by QDR InfiniBand.

For simplicity all runs herein were performed on the eight GPU nodes. As a starting point a domain of dimension [−16,16]×[−16,16]×[0,1.75][-16, 16] \times[-16, 16] \times[0, 1.75] was meshed isotropically with NE=114 688N_E = 114\,688 structured hexahedral elements. The mesh was configured with completely periodic boundary conditions. When run with the Navier-Stokes solver in PyFR with ϱ=3\varrho= 3 the mesh gives a working set of ∼4720\sim4720 MiB. This is sufficient to 90% load an M2090 which when ECC is enabled has ∼5250\sim5250 MiB memory available to the user. When examining the scalability of a code there are two commonly used metrics. The first of these is weak scalability in which the size of the target problem is increased in proportion to the number of ranks NN with NE∝NN_E \propto N. For a code with perfect weak scalability the runtime should remain unchanged as more ranks are added. The second metric is strong scalability wherein the problem size is fixed and the speedup compared to a single rank is assessed. Perfect strong scalability implies that the runtime scales as 1/N1/N.

For the domain outlined above weak scalability was evaluated by increasing the dimensions of the domain according to [−16,16]×N[−16,16]×[0,1.75][-16, 16] \times N[-16, 16] \times[0, 1.75]. This extension permitted the domain to be trivially decomposed along the yy-axis. The resulting runtimes for 1≤N≤1041 \leq N \leq104 can be seen in Table 6. We note that in the N=104N = 104 case that the simulation consisted of some 3.8×1093.8 \times10^{9} degrees of freedom with a working set of ∼485\sim485 GiB.

# M2090s1248163264104
Runtime1.001.001.011.011.011.011.011.01

Table 6. Weak scalability of PyFR for the Navier-Stokes equations with ϱ=3\varrho= 3. Runtime is normalised with respect to a single NVIDIA M2090 GPU.

To study the strong scalability the initial domain was partitioned along the xx- and yy-axes. Each partition contained exactly NE/NSN_E/N_S. The resulting speedups for 1≤N≤321 \le N \le32 can be seen in Table 7. Up to eight GPUs scalability can be seen to be near perfect. Beyond this the relationship begins to break down. When N=32N = 32 an improvement of 26 can be observed. However, in this case each GPU is loaded to less than 3% and so the result is to be expected.

# M2090s12481632
Speedup1.002.033.967.4814.0726.18

Table 7. Strong scalability of PyFR for the Navier-Stokes equations with φ=3\varphi= 3. The speedup is relative to a single NVIDIA M2090 GPU.

Conclusions

In this paper we have described PyFR, an open source Python based framework for solving advection-diffusion type problems on streaming architectures. The structure and ethos of PyFR has been explained including our methodology for targeting multiple hardware platforms. We have shown that PyFR exhibits spatial super accuracy when solving the 2D Euler equations and the expected order of accuracy when solving Couette flow problem on a range of grids in 2D and 3D. Qualitative results for unsteady 3D viscous flow problems on curved grids have also been presented. Performance of PyFR has been validated on an NVIDIA M2090 GPU in three dimensions. It has been shown that the compute bound kernels are able to obtain between 50% and 90% of realisable peak FLOP/s whereas the bandwidth bound point-wise kernels are, across the board, able to obtain in excess of 89% realisable peak bandwidth. The scalability of PyFR has been demonstrated in the strong sense up to 32 NVIDIA M2090s and in the weak sense up to 104 NVIDIA M2090s when solving the 3D Navier-Stokes equations.

Acknowledgements

The authors would like to thank the Engineering and Physical Sciences Research Council for their support via two Doctoral Training Grants and an Early Career Fellowship (EP/K027379/1). The authors would also like to thank the e-Infrastructure South Centre for Innovation for granting access to the Emerald supercomputer, and NVIDIA for donation of three K20c GPUs.

Matrix Representation

It is possible to cast the majority of operations in an FR step as matrix-matrix multiplications of the form

C←c1AB+c2C,(45)\mathbf{C} \leftarrow c_1\mathbf{A}\mathbf{B} + c_2\mathbf{C}, \tag*{(45)}

where c1,2∈Rc_{1,2} \in\mathbb{R} are constants, A\mathbf{A} is a constant operator matrix, and B\mathbf{B} and C\mathbf{C} are state matrices. To accomplish this we start by introducing the following constant operator matrix

(Me0)σρ=ℓeρ(u)(x~(f)),dim⁡Me0=Ne(f)×Ne(u),\left(\mathbf{M}^{0}_{e}\right)_{\sigma\rho} = \ell^{(u)}_{e\rho}\left(\tilde{\mathbf{x}}^{(f)}\right), \qquad\dim\mathbf{M}^{0}_{e} = N^{(f)}_{e} \times N^{(u)}_{e},

and the following state matrices

(Ue(u))ρ(nα)=ueρnα(u),dim⁡Ue(u)=Ne(u)×NV∣Ωe∣,\left(\mathbf{U}^{(u)}_{e}\right)_{\rho(n\alpha)} = u^{(u)}_{e\rho n\alpha}, \qquad\dim\mathbf{U}^{(u)}_{e} = N^{(u)}_{e} \times N_{V}|\Omega_{e}|,
(Ue(f))σ(nα)=ueσnα(f),dim⁡Ue(f)=Ne(f)×NV∣Ωe∣.\left(\mathbf{U}^{(f)}_{e}\right)_{\sigma(n\alpha)} = u^{(f)}_{e\sigma n\alpha}, \qquad\dim\mathbf{U}^{(f)}_{e} = N^{(f)}_{e} \times N_{V}|\Omega_{e}|.

In specifying the state matrices there is a degree of freedom associated with how the NVN_{V} field variables for each element are packed along a row of the matrix, with the possible packing choices being discussed in subsection 3.3. Using these matrices we are able to reformulate Equation 6 as

Ue(f)=Me0Ue(u).(46)\mathbf{U}^{(f)}_{e} = \mathbf{M}^{0}_{e}\mathbf{U}^{(u)}_{e}. \tag*{(46)}

In order to apply a similar procedure to Equation 9 we let

(Me4)ρσ=[∇~ ℓeρ(u)(x~)]x~=x~eσ(u),dim⁡Me4=NDNe(u)×Ne(u),\left(\mathbf{M}^{4}_{e}\right)_{\rho\sigma} = \left[\widetilde{\nabla}\,\ell^{(u)}_{e\rho}(\widetilde{\mathbf{x}})\right]_{\widetilde{\mathbf{x}}=\widetilde{\mathbf{x}}^{(u)}_{e\sigma}}, \qquad\dim\mathbf{M}^{4}_{e} = N_{D}N^{(u)}_{e} \times N^{(u)}_{e},
(Me6)ρσ=[n^eρ(f)⋅∇~⋅geρ(f)(x~)]x~=x~eσ(f),dim⁡Me6=NDNe(u)×Ne(f),\left(\mathbf{M}^{6}_{e}\right)_{\rho\sigma} = \left[\widehat{\mathbf{n}}^{(f)}_{e\rho}\cdot\widetilde{\nabla}\cdot\mathbf{g}^{(f)}_{e\rho}(\widetilde{\mathbf{x}})\right]_{\widetilde{\mathbf{x}}=\widetilde{\mathbf{x}}^{(f)}_{e\sigma}}, \qquad\dim\mathbf{M}^{6}_{e} = N_{D}N^{(u)}_{e} \times N^{(f)}_{e},
(Ce(f))ρ(nα)=Cαueρnα(f),dim⁡Ce(f)=Ne(f)×NV∣Ωe∣,\left(\mathbf{C}^{(f)}_{e}\right)_{\rho(n\alpha)} = \mathcal{C}_{\alpha}u^{(f)}_{e\rho n\alpha}, \qquad\dim\mathbf{C}^{(f)}_{e} = N^{(f)}_{e} \times N_{V}|\Omega_{e}|,
(Q~e(u))σ(nα)=q~eσnα(u),dim⁡Q~e(u)=NDNe(u)×NV∣Ωe∣.\left(\widetilde{\mathbf{Q}}^{(u)}_{e}\right)_{\sigma(n\alpha)} = \widetilde{q}^{(u)}_{e\sigma n\alpha}, \qquad\dim\widetilde{\mathbf{Q}}^{(u)}_{e} = N_{D}N^{(u)}_{e} \times N_{V}|\Omega_{e}|.

Here it is important to qualify assignments of the form Aij=x\mathbf{A}_{ij}=\mathbf{x} where x\mathbf{x} is a NDN_{D} component vector. As above there is a degree of freedom associated with the packing. With the benefit of foresight we take the stride between subsequent elements of x\mathbf{x} in a matrix column to be either Δi=Ne(u)\Delta i=N^{(u)}_{e} or Δi=Ne(f)\Delta i=N^{(f)}_{e} depending on the context. With these matrices Equation 9 reduces to

Q~e(u)=Me6Ce(f)−Ue(f)+Me4Ue(u)=Me6(Ce(f)−Me0Ue(u))+Me4Ue(u)=Me6Ce(f)+[Me4−Me6Me0]Ue(u).(47)\begin{aligned} \widetilde{\mathbf{Q}}^{(u)}_{e} &= \mathbf{M}^{6}_{e}\mathbf{C}^{(f)}_{e} - \mathbf{U}^{(f)}_{e} + \mathbf{M}^{4}_{e}\mathbf{U}^{(u)}_{e} \\ &= \mathbf{M}^{6}_{e}\left(\mathbf{C}^{(f)}_{e} - \mathbf{M}^{0}_{e}\mathbf{U}^{(u)}_{e}\right) + \mathbf{M}^{4}_{e}\mathbf{U}^{(u)}_{e} \\ &= \mathbf{M}^{6}_{e}\mathbf{C}^{(f)}_{e} + \left[\mathbf{M}^{4}_{e} - \mathbf{M}^{6}_{e}\mathbf{M}^{0}_{e}\right]\mathbf{U}^{(u)}_{e}. \tag*{(47)} \end{aligned}

Applying the procedure to Equation 11 we take

Me5=diag⁡(Me0,…,Me0),dim⁡Me5=NDNe(f)×NDNe(f),\mathbf{M}^{5}_{e} = \operatorname{diag}\left(\mathbf{M}^{0}_{e},\ldots,\mathbf{M}^{0}_{e}\right), \qquad\dim\mathbf{M}^{5}_{e} = N_{D}N^{(f)}_{e} \times N_{D}N^{(f)}_{e},
(Qe(u))σ(nα)=qeσnα(u),dim⁡Qe(u)=NDNe(u)×NV∣Ωe∣,\left(\mathbf{Q}^{(u)}_{e}\right)_{\sigma(n\alpha)} = q^{(u)}_{e\sigma n\alpha}, \qquad\dim\mathbf{Q}^{(u)}_{e} = N_{D}N^{(u)}_{e} \times N_{V}|\Omega_{e}|,
(Qe(f))σ(nα)=qeσnα(f),dim⁡Qe(f)=NDNe(f)×NV∣Ωe∣.\left(\mathbf{Q}^{(f)}_{e}\right)_{\sigma(n\alpha)} = q^{(f)}_{e\sigma n\alpha}, \qquad\dim\mathbf{Q}^{(f)}_{e} = N_{D}N^{(f)}_{e} \times N_{V}|\Omega_{e}|.

hence

Qe(f)=Me5Qe(u)(48)\mathbf{Q}_{e}^{(f)}=\mathbf{M}_{e}^{5}\mathbf{Q}_{e}^{(u)} \tag*{(48)}

where we note the block diagonal structure of Me5\mathbf{M}_{e}^{5}. This is a direct consequence of the above choices for Δi\Delta_i. Finally, to rewrite (23) we write

(Me1)ρσ=[∇~ φ~ρ(u)(x~)]x~=x~σ(u)T,dim⁡Me1=Ne(u)×NDNe(u),(Me2)ρσ=[f~eρ(u)(x~eσ(f))n^eσ(f)]T,dim⁡Me2=Ne(f)×NDNe(u),(Me3)ρσ=[∇~⋅geσ(u)(x~)]x~=x~ρ(u),dim⁡Me3=Ne(u)×Nee(f),(D~e(f))σ(nα)=G~σα(f1),dim⁡D~e(f)=Ne(f)×NV∣Ωe∣,(F~e(u))ρ(nα)=f~eρnα(u),dim⁡F~e(u)=NDNe(u)×NV∣Ωe∣,(R~e(u))ρ(nα)=(∇~⋅f~)eρnα(u),dim⁡R~e(u)=Ne(u)×NV∣Ωe∣,\begin{aligned} \left(\mathbf{M}_{e}^{1}\right)_{\rho\sigma}&=\left[\widetilde{\nabla}\,\widetilde{\varphi}_{\rho}^{(u)}\left(\widetilde{\mathbf{x}}\right)\right]_{\widetilde{\mathbf{x}}=\widetilde{\mathbf{x}}_{\sigma}^{(u)}}^{\mathsf{T}}, && \dim\mathbf{M}_{e}^{1}=N_{e}^{(u)}\times N_{D}N_{e}^{(u)},\\ \left(\mathbf{M}_{e}^{2}\right)_{\rho\sigma}&=\left[\widetilde{f}_{e\rho}^{(u)}\left(\widetilde{\mathbf{x}}_{e\sigma}^{(f)}\right)\widehat{\mathbf{n}}_{e\sigma}^{(f)}\right]^{\mathsf{T}}, && \dim\mathbf{M}_{e}^{2}=N_{e}^{(f)}\times N_{D}N_{e}^{(u)},\\ \left(\mathbf{M}_{e}^{3}\right)_{\rho\sigma}&=\left[\widetilde{\nabla}\cdot\mathbf{g}_{e\sigma}^{(u)}\left(\widetilde{\mathbf{x}}\right)\right]_{\widetilde{\mathbf{x}}=\widetilde{\mathbf{x}}_{\rho}^{(u)}}, && \dim\mathbf{M}_{e}^{3}=N_{e}^{(u)}\times N_{e}^{e(f)},\\ \left(\widetilde{\mathbf{D}}_{e}^{(f)}\right)_{\sigma(n\alpha)}&=\widetilde{\mathcal{G}}_{\sigma\alpha}^{(f_{1})}, && \dim\widetilde{\mathbf{D}}_{e}^{(f)}=N_{e}^{(f)}\times N_{V}\lvert\Omega_{e}\rvert,\\ \left(\widetilde{\mathbf{F}}_{e}^{(u)}\right)_{\rho(n\alpha)}&=\widetilde{f}_{e\rho n\alpha}^{(u)}, && \dim\widetilde{\mathbf{F}}_{e}^{(u)}=N_{D}N_{e}^{(u)}\times N_{V}\lvert\Omega_{e}\rvert,\\ \left(\widetilde{\mathbf{R}}_{e}^{(u)}\right)_{\rho(n\alpha)}&=\left(\widetilde{\nabla}\cdot\widetilde{\mathbf{f}}\right)_{e\rho n\alpha}^{(u)}, && \dim\widetilde{\mathbf{R}}_{e}^{(u)}=N_{e}^{(u)}\times N_{V}\lvert\Omega_{e}\rvert, \end{aligned}

and after substitution of (18) for G~eσnα(f1)\widetilde{\mathcal{G}}_{e\sigma n\alpha}^{(f_{1})} obtain

R~e(u)=Me3{D~e(f)−Me2F~e(u)+Me1F~e(u)=Me3D~e(f)+{Me1−Me3Me2}F~e(u).(49)\begin{aligned} \widetilde{\mathbf{R}}_{e}^{(u)}&=\mathbf{M}_{e}^{3}\{\widetilde{\mathbf{D}}_{e}^{(f)}-\mathbf{M}_{e}^{2}\widetilde{\mathbf{F}}_{e}^{(u)}+\mathbf{M}_{e}^{1}\widetilde{\mathbf{F}}_{e}^{(u)}\\ &=\mathbf{M}_{e}^{3}\widetilde{\mathbf{D}}_{e}^{(f)}+\left\{\mathbf{M}_{e}^{1}-\mathbf{M}_{e}^{3}\mathbf{M}_{e}^{2}\right\}\widetilde{\mathbf{F}}_{e}^{(u)}. \tag*{(49)} \end{aligned}

Approximate Riemann Solvers

Overview

In the following section we take uLu_L and uRu_R to be the two discontinuous solution states at an interface and n^L\hat{\mathbf{n}}_L to be the normal vector associated with the first state. For convenience we take fL(inv)=f(inv)(uL)\mathbf{f}_L^{(\mathrm{inv})}=\mathbf{f}^{(\mathrm{inv})}(u_L), and fR(inv)=f(inv)(uR)\mathbf{f}_R^{(\mathrm{inv})}=\mathbf{f}^{(\mathrm{inv})}(u_R) with inviscid fluxes being prescribed by (25).

Rusanov

Also known as the local Lax-Friedrichs method a Rusanov type Riemann solver imposes inviscid numerical interface fluxes according to

G(inv)=n^L2⋅{fL(inv)+fR(inv)}+s2(uL−uR),(50)\mathcal{G}^{(\mathrm{inv})}=\frac{\hat{\mathbf{n}}_L}{2}\cdot\left\{\mathbf{f}_L^{(\mathrm{inv})}+\mathbf{f}_R^{(\mathrm{inv})}\right\}+\frac{s}{2}(u_L-u_R), \tag*{(50)}

where ss is an estimate of the maximum wave speed

s=γ(pL+pR)ρL+ρR+12∣n^L⋅(vL+vR)∣.(51)s=\sqrt{\frac{\gamma(p_L+p_R)}{\rho_L+\rho_R}}+\frac{1}{2}\left|\hat{\mathbf{n}}_L\cdot(\mathbf{v}_L+\mathbf{v}_R)\right|. \tag*{(51)}

Boundary Conditions

Overview

To incorporate boundary conditions into the FR approach we introduce a set of boundary interface types b∈Bb \in\mathcal{B}. At a boundary interface there is only a single flux point: that which belongs to the element whose edge/face is on the boundary. Associated with each boundary type are a pair of functions Cα(b)(uL)\mathcal{C}^{(b)}_{\alpha}(u_L) and Fα(b)(uL,qL,n^L)\mathcal{F}^{(b)}_{\alpha}(u_L,\mathbf{q}_L,\hat{\mathbf{n}}_L) where uLu_L, qL\mathbf{q}_L, and n^L\hat{\mathbf{n}}_L are the solution, solution gradient and unit normals at the relevant flux point. These functions prescribe the common solutions and normal fluxes, respectively.

Instead of directly imposing solutions and normal fluxes it is oftentimes more convenient for a boundary to instead provide ghost states. In its simplest formulation Cα(b)=Cα(uL,B(b)uL)\mathcal{C}^{(b)}_{\alpha}=\mathcal{C}_{\alpha}(u_L,\mathcal{B}^{(b)}u_L) and Fα(b)=Fα(uL,B(b)uL,qL,B(b)qL,n^L)\mathcal{F}^{(b)}_{\alpha}=\mathcal{F}_{\alpha}(u_L,\mathcal{B}^{(b)}u_L,\mathbf{q}_L,\mathcal{B}^{(b)}\mathbf{q}_L,\hat{\mathbf{n}}_L) where B(b)uL\mathcal{B}^{(b)}u_L is the ghost solution state and B(b)qL\mathcal{B}^{(b)}\mathbf{q}_L is the ghost solution gradient. It is straightforward to extend this prescription to allow for the provisioning of different ghost solution states for Cα\mathcal{C}_{\alpha} and Fα\mathcal{F}_{\alpha} and to permit B(b)qL\mathcal{B}^{(b)}\mathbf{q}_L to be a function of uLu_L in addition to qL\mathbf{q}_L.

Supersonic Inflow

The supersonic inflow condition is parameterised by a free-stream density ρf\rho_f, velocity vf\mathbf{v}_f, and pressure pfp_f.

B(inv)uL=B(ldg)uL={(ρfρfvfpf/(γ−1)+ρf2∥vf∥2),(52)\mathcal{B}^{(\mathrm{inv})}u_L=\mathcal{B}^{(\mathrm{ldg})}u_L= \left\{ \begin{pmatrix} \rho_f\\ \rho_f\mathbf{v}_f\\ p_f/(\gamma-1)+\frac{\rho_f}{2}\lVert\mathbf{v}_f\rVert^2 \end{pmatrix} \right., \tag*{(52)}
B(ldg)qL=0,(53)\mathcal{B}^{(\mathrm{ldg})}\mathbf{q}_L=0, \tag*{(53)}

Subsonic Outflow

Subsonic outflow boundaries are parameterised by a free-stream pressure pfp_f.

B(inv)uL=B(ldg)uL={(ρLρLvLpf/(γ−1)+ρL2∥vL∥2),(54)\mathcal{B}^{(\mathrm{inv})}u_L=\mathcal{B}^{(\mathrm{ldg})}u_L= \left\{ \begin{pmatrix} \rho_L\\ \rho_L\mathbf{v}_L\\ p_f/(\gamma-1)+\frac{\rho_L}{2}\lVert\mathbf{v}_L\rVert^2 \end{pmatrix} \right., \tag*{(54)}
B(ldg)qL=0,(55)\mathcal{B}^{(\mathrm{ldg})}\mathbf{q}_L=0, \tag*{(55)}

No-slip Isothermal Wall

The no-slip isothermal wall condition depends on the wall temperature CpTwC_pT_w and the wall velocity vw\mathbf{v}_w. Usually vw=0\mathbf{v}_w=0.

B(inv)uL=ρL{12vw−vLCpTw/γ+12∥2vw−vL∥2(56)\mathcal{B}^{(\mathrm{inv})}\mathbf{u}_{L}=\rho_{L} \left\{ \begin{array}{c} 1\\ 2\mathbf{v}_{w}-\mathbf{v}_{L}\\ C_{p}T_{w}/\gamma+\frac{1}{2}\lVert2\mathbf{v}_{w}-\mathbf{v}_{L}\rVert^{2} \end{array} \right. \tag*{(56)}
B(ldg)uL=ρL{1vwCpTw/γ+12∥vw∥2(57)\mathcal{B}^{(\mathrm{ldg})}\mathbf{u}_{L}=\rho_{L} \left\{ \begin{array}{c} 1\\ \mathbf{v}_{w}\\ C_{p}T_{w}/\gamma+\frac{1}{2}\lVert\mathbf{v}_{w}\rVert^{2} \end{array} \right. \tag*{(57)}
B(ldg)qL=qL,(58)\mathcal{B}^{(\mathrm{ldg})}\mathbf{q}_{L}=\mathbf{q}_{L}, \tag*{(58)}

References

  1. [1]WH Reed and TR Hill. Triangular mesh methods for the neutron transport equation. Technical Report LA-UR-73-479, Los Alamos Scientific Laboratory, 1973.
  2. [2]David A Kopriva and John H Kolias. A conservative staggered-grid Chebyshev multidomain method for compressible flows. Journal of computational physics, 125(1):244–261, 1996.DOI
  3. [3]Yuzhi Sun, Zhi Jian Wang, and Yen Liu. High-order multidomain spectral difference method for the Navier-Stokes equations on unstructured hexahedral grids. Communications in Computational Physics, 2(2):310–333, 2007.DOI
  4. [4]HT Huynh. A flux reconstruction approach to high-order schemes including discontinuous Galerkin methods. AIAA paper, 4079:2007, 2007.DOI
  5. [5]Jan S Hesthaven and Tim Warburton. Nodal discontinuous Galerkin methods: algorithms, analysis, and applications, volume 54. Springer Verlag New York, 2008.DOI
  6. [6]PE Vincent, P Castonguay, and A Jameson. A new class of high-order energy stable flux reconstruction schemes. Journal of Scientific Computing, 47(1):50–72, 2011.DOI
  7. [7]P Castonguay, PE Vincent, and A Jameson. A new class of high-order energy stable flux reconstruction schemes for triangular elements. Journal of Scientific Computing, 2011.
  8. [8]Patrice Castonguay, PE Vincent, and Antony Jameson. Application of high-order energy stable flux reconstruction schemes to the Euler equations. In 49th AIAA Aerospace Sciences Meeting, volume 686, 2011.DOI
  9. [9]A Jameson, PE Vincent, and P Castonguay. On the non-linear stability of flux reconstruction schemes. Journal of Scientific Computing, 50(2):434–445, 2011.DOI
  10. [10]PE Vincent, P Castonguay, and A Jameson. Insights from von Neumann analysis of high-order flux reconstruction schemes. Journal of Computational Physics, 230(22):8134–8154, 2011.DOI
  11. [11]Patrice Castonguay, Peter E Vincent, and Antony Jameson. A new class of high-order energy stable flux reconstruction schemes for triangular elements. Journal of Scientific Computing, 51(1):224–256, 2012.
  12. [12]DM Williams, P Castonguay, PE Vincent, and A Jameson. Energy stable flux reconstruction schemes for advection-diffusion problems on triangles. Journal of Computational Physics, 2013.DOI
  13. [13]P Castonguay, DM Williams, PE Vincent, and A Jameson. Energy stable flux reconstruction schemes for advection-diffusion problems. Computer Methods in Applied Mechanics and Engineering, 2013.DOI
  14. [14]D.M. Williams and A. Jameson. Energy stable flux reconstruction schemes for advection-diffusion problems on tetrahedra. Journal of Scientific Computing, pages 1–39, 2013.DOI
  15. [15]David A Kopriva. A staggered-grid multidomain spectral method for the compressible navier–stokes equations. Journal of Computational Physics, 143(1):125–158, 1998.DOI
  16. [16]SymPy Development Team. Sympy: Python library for symbolic mathematics, 2013.
  17. [17]Michael Bayer. Mako: Templates for python, 2013.
  18. [18]Andreas Klöckner, Nicolas Pinto, Yunsup Lee, Bryan Catanzaro, Paul Ivanov, and Ahmed Fasih. Pycuda and pypopencl: A scripting-based approach to gpu run-time code generation. Parallel Comput., 38(3):157–174, 2012.DOI
  19. [19]Lisandro Dalcin. mpi4py: Mpi for python, 2013.
  20. [20]Eleuterio F Toro. Riemann solvers and numerical methods for fluid dynamics: a practical introduction. Springer, 2009.

Paper details

vlnc:W_d56eqk5c
Version
v1 · latest
Published
Added
License
CC-BY-3.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents