WorksPhysics and Astronomy
High-order thread-safe lattice Boltzmann model for HPC turbulent flow simulations
Abstract
We present a highly-optimized thread-safe lattice Boltzmann model in which the non-equilibrium part of the distribution function is locally reconstructed via recursivity of Hermite polynomials. Such a procedure allows the explicit incorporation of non-equilibrium moments of the distribution up to the order supported by the lattice. Thus, the proposed approach increases accuracy and stability at low viscosities without compromising performances and amenability to parallelization with respect to standard lattice Boltzmann models. The high-order thread-safe LB is tested on two types of turbulent flows, namely the turbulent channel flow at and the axisymmetric turbulent jet at , it delivers results in excellent agreement with reference data (both DNS, theory, and experiments) and a) achieves peak performances ( and an arithmetic intensity of on single GPU) by significantly reducing the memory footprint, b) retains the algorithmic simplicity of standard lattice Boltzmann computing and c) allows to perform stable simulations at vanishingly low viscosities. Our findings open attractive prospects for high-performance simulations of realistic turbulent flows on GPU-based architectures. Such expectations are confirmed by the excellent agreement among lattice Boltzmann, experimental, and DNS reference data.
Introduction
During the last couple of decades, the lattice Boltzmann method (LB hereafter) has met with massive success in the computational fluid dynamics community as a hydrodynamic solver in kinetic disguise due to its strongly appealing computational features combined with its conceptual and practical simplicity1,2. Indeed, at variance with standard hydrodynamic solvers in which the convective term (where is the fluid velocity, and Greek subscripts indicate Cartesian components) is non-linear and non-local at a time, the LB splits non-linearity and non-locality between the two backbones operators, namely the collision (a local and non-linear operator) and the streaming step (linear and non-local and exact at machine precision). In particular, the possibility to advect the set of discrete particle distribution functions along linear characteristics is a fundamental feature when dealing with flows in which the velocity streamlines may become ”wild,” as it occurs in chaotic, turbulent flows[3, 4]. Nonetheless, the lattice Boltzmann is known to suffer from drawbacks related to different aspects, where two of the major ones are the following:
From the computational standpoint, the LB possesses a relatively low operational (or arithmetic) intensity, in the range FLOP/byte (floating point operations needed to compute a kernel per byte of memory accessed), depending on the collision strategy and the floating point representation employed[5]. Such a low operational intensity prevents exploiting the ideal peak performance that processors and graphical accelerators could potentially deliver.
The LB model, in particular the widely used version based on the BGK single relaxation collisional operator[6], is known to suffer from the onset of instabilities at low viscosity (over-relaxed regimes) due to the relaxation of non-hydrodynamic ghost moments[7] which, in turn, stem from the lack of sufficient isotropy and moment representation of low order lattices. Alternative formulations based on more complex collisional operators have been proposed[8, 9] to deal with such shortcomings, albeit at the expense of simplicity and performances of the single relaxation time LB. These drawbacks limit the use of BGK-like LB models to the simulation of low Reynolds number values and creeping flows since their capability to deal with turbulent flows can only be guaranteed upon increasing the grid resolution.
In this work, we propose an LB model built upon a recently developed thread-safe version[10] and a higher order variant of the regularized LB[11], which is aimed at optimizing the performances on shared memory architecture typical of graphic processing units and to broaden the range of applicability of BGK-based LB to low viscosity fluid flows. The high-order thread-safe LB is tested on two types of turbulent flows, namely the turbulent channel flow at and the axisymmetric turbulent jet at . The results are in excellent agreement with reference data (both DNS, theory, and experiments) and confirm the capability of the approach to perform accurate high-performance computer simulations of turbulent flows of scientific and engineering interest.
Method
In this section, an in-depth description of the proposed computational framework based on the cross-fertilization between a recently developed thread-safe LB strategy[10] and a high-order variant of the regularized LB exploiting the recursivity of Hermite polynomials[11, 12, 13] is provided. Before proceeding, we shortly recap the salient features of the lattice Boltzmann equation in the single-relaxation time approximation (a thorough review of the LB equation can be found at Ref.[14, 15] [2]).
We start from the following discrete version of the Boltzmann equation
where and denote lattice position and time step, respectively, is a set of distribution functions, is an index spanning the discrete velocity vectors of the lattice and is a set of discrete thermodynamic equilibria obtained via a second-order expansion in Mach number of the continuous Maxwell-Boltzmann distribution. Also, is the fluid density, and is the fluid momentum. An analytical, truncated expression of is given by
where is a normalized set of weights and is the sound speed of the model. The standard LB algorithm consists of two main steps, namely a collision towards a local Maxwell-Boltzmann equilibrium occurring at a characteristic rate (right-hand side of Eq. (1)) and a free-flight of the set of distributions along predefined lattice directions. The first operation is local and non-linear in the macroscopic velocity, while the second one is a linear non-local operation, exact at machine precision (no loss of information during advection). The main hydrodynamic fields of interest, i.e., density , linear momentum and momentum flux tensor , can be calculated from the kinetic moments of the distribution functions by linear and local summations[15]. Finally, through a multi-scale expansion in the Knudsen number[16], it is possible to prove that the lattice Boltzmann equation recovers, in the continuum limit, a set of partial differential equations governing the conservation of mass and momentum of a volume of fluid, in the limit of weak compressibility (small Mach numbers)
where is the pressure, and is the kinematic viscosity.
Thread-safe reconstruction of post-collision distributions
The main idea behind the thread-safe (fused streaming-collision) LB algorithm comes from observing that, in a lattice BGK relaxation process, any post-collision distribution can be expressed as a weighted sum of the equilibrium and non-equilibrium parts of the probability distribution function (pdf), as follows1:
The above equality stems from the decomposition of the full lattice distribution into an equilibrium and non-equilibrium part, as .
A direct consequence of Eq. (5) is the possibility of reconstructing the set of distribution functions by decoupling pre and post-collisional states, with evident advantages for the implementation of highly efficient LB models on shared-memory architectures (SMA) present in graphic processing units (GPUs), as previously shown in[17, 18]. Indeed, Eq. (5) can be easily incorporated within actual lattice Boltzmann codes to eliminate data dependencies that arise during non-local read and write operations. Such data dependencies hinder an efficient implementation of thread-safe LB models on SMA unless more complex streaming strategies[19, 20, 21], which inevitably compromise the underlying simplicity of the LB, are employed.
Thus, a natural extension of Eq. (5) calls for an explicit inclusion of the streaming step as follows:
It is worth noting that Eq. (6) is a thread-safe operation on SMA provided that is directly calculated from macroscopic hydrodynamic quantities which, as noted above, in the LB framework can be efficiently computed as statistical moments of the set of PDFs, up to an order dictated by the symmetry and isotropy properties of the lattice. Such a task can be performed by resorting to the regularization procedure[7, 22, 23, 24, 25], where the set is defined as a projection of the hydrodynamic moments , and onto suitable Hermite basis.
More precisely, using Hermite polynomials and Gauss-Hermite quadratures, one can write where is the n-th order Hermite basis and the corresponding Hermite expansion coefficients. Both and are rank tensors. In standard regularized LB models, the non-equilibrium set of distributions contains information from hydrodynamic moments up to order 2 (i.e. the Navier-Stokes level) and is filtered from non-hydrodynamic (ghost) contributions stemming from higher order fluxes[25, 26]. Thus, the set of non-equilibrium distributions can be compactly written as:
Since, at the continuum level, the non-equilibrium pdfs contain information of non-equilibrium moments at all orders, they can be re-projected onto suitable Hermite basis to include a larger number of hydrodynamic moments, up to the order of isotropy of the lattice.
With the above relation in mind, the lattice kinetics exposed in Eq. (5) can be implemented by storing, at each time step and for each lattice node, three macroscopic quantities, namely, one scalar (), three vector components () and six components of a symmetric, tensor of rank two (). Thus, the full stream and collision step becomes
In other words, the thread-safe LB, rather than explicitly streaming distributions along the lattice directions, aims at reconstructing the post-streamed/post-collided distribution via a scattered reading of the neighboring macroscopic hydrodynamic fields, which are then employed to rebuild the complete set of distributions (see Figure 1). It is interesting to note that Eq.(8) clearly allows for the implementation of the lattice Boltzmann on unstructured grids and meshless domains due to the possibility of interpolating macroscopic fields (rather than the full set of PDFs) to reconstruct the distributions off-lattice. Such extension is currently ongoing and will be the subject of separate works.
Figure 1. Sketch of the thread-safe LB update algorithm. Rather than explicitly streaming distributions along the lattice directions, the algorithm reconstructs the post-streamed distribution using the neighboring macroscopic hydrodynamic fields, which are employed to rebuild the full set of distributions.
As anticipated in the introduction, a fundamental feature of the thread-safe approach lies in its simplicity with respect to other existing single-distribution, in-place streaming approaches, such as EsoTwist[20, 21] and the Bailey algorithm[19], which require the resort to high-level programming due to the need of explicitly swapping pointers to preserve data locality. Indeed, since the post-collision distributions lose any connection with their pre-collision values, the streaming step is replaced by a scattered reconstruction of the post-collisionals, performed at neighboring points along the lattice direction upstream. From a computational standpoint, such an approach is particularly beneficial, as it avoids the onset of race conditions that may jeopardize the correct memory access in shared memory.
GPU architectures10. Moreover, the thread-safe reconstruction allows for a straightforward porting of LB codes on GPU, either by resorting to CUDA native language or by employing directive-based procedures, in the second case, with virtually no need to modify the structure of the serial code. The latter strategy has been employed in this work.
We highlight that, in separate contexts, the use of regularized post-collisionals was proven to be instrumental in accessing off-equilibrium hydrodynamic regimes simply out of reach for standard, second-order isotropic lattice Boltzmann models and increasing the stability and accuracy of LB models to high inertia-over-dissipation hydrodynamic regimes, typical of turbulent flows[11, 22, 23, 27].
In the next subsection, we present a thread-safe framework in which third-order discrete equilibria are included where the set of non-equilibrium distributions is expanded to account for non-equilibrium statistical moments up to order three. This is the highest moment isotropy order supported by the standard fourth-order isotropic lattices, i.e., the sketched in Figure 2.
Figure 2. Graphical representation of the D3Q27 lattice. Red arrows stand for the velocity group having magnitude , the blue one and the green arrows
High-order regularized lattice Boltzmann based on recursivity of Hermite polynomials
The Hermite relation represents a hierarchy of terms with which the non-equilibrium part of the distribution functions can be thoroughly described, provided that the lattice in use supports an exact moment representation up to order .
As discussed in the previous section, the distribution can be split into its equilibrium and non-equilibrium parts, which can be represented through Hermite polynomials:
The equilibrium and non-equilibrium Hermite coefficients are defined as:
and
As far as the equilibrium Hermite coefficients are concerned, one can construct the −th order coefficient by exploiting the following relation[29]:
being , and . The D3Q27 (and the D2Q9 counterpart in 2D) correctly recovers macroscopic moments up to order three ()[14] [1]. Thus, by employing the relation (11) expanded up to [29], higher order hydrodynamic equilibria become
These third-order discrete Maxwell-Boltzmann equilibrium distributions allow, when equipped to a fourth-order isotropic lattice, to recover hydrodynamic moments up to order 3, i.e.,
which can be interpreted as a flux of momentum flux. It is important to recall that, despite such a higher-order moment having no counterpart in the Navier-Stokes equation, the use of extended equilibria increases both the stability and accuracy of an LB solver. Moreover, as shown in [13, 29], it is also possible to define recursive relations for the non-equilibrium Hermite coefficients, which can be compactly written as:
Recalling that , one can explicitly reconstruct the non-equilibrium set of distributions by incorporating non-equilibrium hydrodynamic moments up to the third-order
and thus finally obtain the post-collision and streaming distributions via recursive thread-safe LB:
It is worth noting that the cross-fertilization between recursivity and thread-safe strategy permits the implementation of a higher-order LB model by sensibly reducing the memory footprint without altering the simplicity of the LB scheme based on the AB (flip-flop) streaming strategy. Indeed, in an actual implementation of a high-order thread-safe LB, one needs to store just one set of distributions (i.e., one set per fluid component) plus, in three dimensions, 10 arrays for the macroscopic variables, namely, density, linear momentum, and non-equilibrium momentum flux tensor. The higher-order Hermite coefficients can then be computed locally and on the fly while performing the thread-safe fused stream-and-collide algorithm.
Thread-safe implementation of the non-equilibrium extrapolation boundary conditions
In this section, we present a novel strategy to implement boundary conditions based on the concept of non-equilibrium extrapolation[30], together with the possibility to approximate missing data at the boundary through Grad’s reconstruction[31]. Such implementation can be naturally embedded in the thread-safe LB framework and employed to model different boundary conditions accurately, such as Dirichlet-like (no-slip inflow, imposed velocity) and Neumann-like (outflow, zero-gradient).
The non-equilibrium extrapolation method is based on the observation that the post-collision distribution at a boundary node can be rewritten as:
where denotes a boundary node. As in the regularized model, it is possible to assume that , where is the Knudsen number. Since, at time , the hydrodynamic quantities are known at a neighboring fluid node (), can be determined as:
By observing that the boundary node is at a distance from the nearest bulk node, one can write , thus can be approximated by performing a first order extrapolation as
The above implies that the non-equilibrium distribution at the boundary is approximated at second-order accuracy.
Finally, Eq. (19) can be rewritten as
where the equilibrium distribution is computed by imposing the hydrodynamic variables at the boundary (either momentum or density/pressure), while the non-equilibrium part is extrapolated from the neighboring bulk nodes. The post-collision distribution at the boundary node is then streamed back within the fluid domain.
These boundary conditions can be naturally incorporated within the thread-safe LB framework. Indeed, considering that the non-equilibrium distribution of Eq. (21) can be replaced by an explicit reconstruction through Hermite projection, Eq. (22) can be rewritten as:
Thus, the non-equilibrium part of the (unknown) distributions incoming from the boundary nodes is readily reconstructed by employing the algorithm reported in Eq. (23), while the equilibrium part is evaluated from the macroscopic variables defined at the boundary node (see Figure 3).
Figure 3. Thread-safe, explicit non-equilibrium reconstruction of the boundary condition. The non-equilibrium part of the incoming (unknown) distributions from the boundary nodes is reconstructed using the algorithm reported in eq.23. The equilibrium part is computed by imposing macroscopic variables at the boundary node.
It is important to recall that the standard implementation of Eq. (19) requires two sets of distributions to be stored in memory in order for the non-equilibrium PDFs at the boundary to be safely evaluated on SMA. In addition, the explicit reconstruction of reported above can be naturally incorporated within the proposed computational framework, thus allowing the use of a data structure compliant with the thread-safe strategy and halving the memory footprint with respect to the standard fused AB scheme. Finally, the modified non-equilibrium boundary condition, whose calculation is based on the explicit evaluation of the non-equilibrium distributions from Hermite polynomials, naturally filters out the ghost moments (transported by the streaming step in a BGK LB), which are known to jeopardize the stability and accuracy of LB models operating in both under and over-relaxed
regimes. For the reason above, the modified boundary condition is expected to perform well in turbulent flow applications, where LB generally operates at vanishing-low viscosities.
Another important advantage of the present approach lies in the possibility of imposing boundary conditions straightforwardly on pressure and momentum, as in standard Navier-Stokes solvers, while approximating the momentum flux tensor (containing information on the deviatoric part of the stress tensor) with its value in the nearest-neighbor bulk node. This essentially means that the boundary conditions for the velocity and/or pressure do not need to be translated into conditions for the density distribution functions, a procedure that can be cumbersome for problems involving complex boundaries. A full assessment of the computational features of the novel thread-safe non-equilibrium boundary conditions is currently ongoing and will be the subject of a separate publication.
Benchmarks
In this section, we test the accuracy of the high-order thread-safe LB against two types of turbulent flows, namely the turbulent flow in a straight, smooth channel at and the axisymmetric turbulent jet at . All the simulations have been run in single precision on the Leonardo cluster (single GPU).
Turbulent channel flow at
The first test case, referred to as turbulent channel flow, is the flow between two parallel smooth walls driven by a constant upstream-downstream pressure gradient[32], with periodic boundary conditions along the directions parallel to the walls (i.e., along the and directions). Due to the periodicity of the momentum along the streamwise direction (i.e., coordinate), the pressure gradient driving the flow can be mimicked by imposing a constant body force (i.e., gravity) acting along the same direction. Also, no-slip boundary conditions are enforced at the top and bottom walls (i.e., at and , standing for the half-width of the channel, see top panel of Figure 4). At sufficiently high Reynolds numbers, the transition to turbulence is observed, and the flow becomes fully three-dimensional and chaotic. The results investigated hereafter refer to a regime of statistically steady turbulence, where the turbulent quantities, obtained as first and second order statistics of the hydrodynamic quantities (i.e. the three components of the velocity field , ), reach the steady state in a statistical sense.
Figure 4. Top panel: Instantaneous velocity field of the turbulent channel flow at (the color bar denotes the velocity values in wall units). Bottom panels: The plots show the space-time averaged velocity profiles (made non-dimensional by the friction velocity ) for two different grid resolutions (left plot , grid nodes, right plot , ). The legend of the symbols is shown in the plots.
Lattice Boltzmann results have been compared to reference data obtained from a Chebyshev pseudo-spectral method [33] applied to the direct numerical simulations of the turbulent channel flow at friction Reynolds number , where is the friction velocity defined as . Here is the constant acceleration imposed on the flow and the wall friction. Such value of the friction Reynolds number is high enough to guarantee a developed turbulent regime within the channel.
The simulations have been run on a computational domain where and . Two different resolutions have been employed, namely and lattice units, delivering values of the grid step size in wall units and (being ). In particular, as reported in [34] for the case of a fully developed channel flow, a value of should guarantee a fully resolved simulation. Nonetheless, the thread-safe LB has allowed us to perform stable simulations at resolutions as low as (not reported in this work). The value of the acceleration imposed to the fluid has been fixed to ((), while the viscosity can be inferred from the and is for and for (hereafter, all the dimensions are reported in lattice units[35]). The force has been implemented as in[36], through the local forcing term directly added during the thread-safe stream-collision step.
Mean velocity, Reynolds stresses and Skewness
The first statistic investigated is the mean velocity profile. In Figure 4 we report the mean streamwise velocity profiles (scaled by the friction velocity ) obtained for two different resolutions (Figure 4a, ) and (Figure 4b, ). The LB results are compared with the DNS pseudo-spectral results reported in[33] (plusses), the near-wall linear solution and the logarithmic law in bulk (dashed lines), being the Von Karman constant and for smooth walls.
The near-wall linear behavior is correctly reproduced at both resolutions. In contrast, the simulation at slightly overestimates the DNS solution in bulk, as can be observed from Figure 4a, being it described by a log-law with , a value larger than the
one predicted for turbulent channel flows in the presence of smooth walls. As expected, by increasing the resolution, DNS data can be reproduced accurately, as shown in Figure 4b. This is not surprising since, as reported in [34], a uniform grid step size is required for fully resolved simulations, while is a safe value that guarantees stable simulations. Nonetheless, as far as the mean velocity profile is concerned, the thread-safe high-order LB keeps on providing quite satisfactory results even at coarser resolutions.
We now focus on the second-order (turbulent kinetic energy components and Reynolds stresses) and third-order (skewness) velocity statistics. The first one is defined as:
where the diagonal components () stand for the root mean square values of the turbulent kinetic energy components, while the off-diagonal terms are the Reynolds stresses.
The second one, i.e., the skewness of individual velocity components, is defined as follows:
In order to obtain converged statistics, the simulations have been run long enough (several millions of cycles) to reach a statistically steady state.
In Figure 5, the agreement between the reference and the LB results is noticeable, even for the coarser simulation (panel (a)), where a larger discrepancy appears near the peak of the r.m.s x-component of the turbulent kinetic energy. Nonetheless, the thread-safe LB provides reliable results despite the discretization in use ruling out the possibility of capturing the turbulent structures in full throughout all the relevant scales. Such a discrepancy is, as expected, much reduced in the simulation performed at as clearly visible in panel (b), thus confirming that the TSLB simulation in the DNS regime allows to correctly reproduce the salient features of the turbulent channel flow, in agreement with reference data.
Figure 5. Mean turbulent kinetic energy components and Reynolds stresses. LB data are compared against the pseudo-spectral results reported in [33]. In panel (a) , in panel (b) .
To conclude, Figure 6 shows a comparison between the z-distributions of the skewness of the components of the fluctuating velocity as predicted by the present LB model and the pseudo-spectral simulations of [33]. Once again, higher-order statistics are captured with a good degree of accuracy, as clearly visible in the figure. We argue that the small deviations from the reference data could be attributed to the different domain sizes employed in this work (both in the streamwise and crossflow directions) and the sampling procedure, which may affect the averages.
Figure 6. The plot shows the values of the three components of the skewness at of the fluctuating velocity distributions. LB data are compared against the reference pseudo-spectral results.
Axisymmetric turbulent jet at
The case considered in this section is the space-time evolution of a round turbulent jet featuring an inlet Reynolds number , being (inlet velocity), (diameter of the round nozzle) and corresponding to a relaxation time (). Considering these numbers, the convective Mach number results in , well below the threshold value 0.6, which triggers non-negligible effects due to compressibility. Thus, comparing the present simulations against incompressible jet experiments reported in [37] and [38] is reasonable. The simulation has been performed on a Cartesian (uniform) grid of computational nodes. Inflow boundary condition has been imposed at the inlet of the jet via the thread-safe non-equilibrium boundary conditions described in section II C, while periodic boundary conditions are enforced along both crossflow directions (i.e., along and ). We also impose an outflow condition at by extrapolating the linear momentum and the values of the components of the momentum flux tensor from neighboring nodes adjacent to the boundary nodes. Moreover, to reduce the effect of back-propagating pressure waves from the outlet, we enforced a sponge layer by increasing the viscosity near the exit as follows:
being , and grid points[39].
Finally, for the inlet velocity we set a uniform profile with a superimposed, time-varying Gaussian noise[40], needed to trigger the onset of instability at the outlet and aimed at mimicking the fully turbulent flow inside the round inlet pipe (not simulated to reduce the computational burden).
The results of the simulation, such as mean velocity and stresses, are compared with experimental data of Hussein et al.[37] and of Panchapakesan and Lumley[38].
Mean velocity, turbulent kinetic energy and Reynolds stresses
Figure 7b shows the centerline velocity as a function of the streamwise distance, taken from , which is the end of the developing region[34]. The mean axial velocity obeys a linear relation of the kind
Figure 7. Panel (a) shows the instantaneous velocity field (streamwise component) of the turbulent jet at . Panel (b) and (c) show the inverse of centerline velocity (made non-dimensional by the inlet velocity) and the spreading radius of the jet vs the distance from the inlet. Both plot reports the straight line behaviors occurring in the fully developed region of the jet[34]. Panel (d) reports the mean axial velocity profiles at several locations downstream of the nozzle. The profiles fall onto the same curve, denoting the self-similarity of the mean axial velocity in turbulent incompressible jets[34].
where is a constant measuring the decay rate and is the location of the virtual origin. The value of is , in agreement with the ones reported in[37] (),[41]() and[38] ().
In Figure 7c, we show the variation of the jet radius , i.e. the radial distance from the axis where the mean axial velocity is equal to half the value assumed on the axis of the jet, as a function of the distance from the inlet (starting from the end of the developing region). As expected, the jet spreads linearly with a characteristic rate , in excellent agreement with experimental predictions[34, 37]. Panel (d) of Figure 7 displays the mean axial velocity profiles at several locations downstream of the nozzle. As one can see, all profiles. fall onto the same curve, denoting the self-similarity of the mean axial velocity profile in turbulent incompressible jets. The velocity profiles are compared against experimental data reported in[37] (plusses symbols) and an exponential fit . The best fit is obtained with , in close agreement with the values reported in[38, 42].
The above scaling laws and self-similar behavior of the mean velocity profile represent a zeroth order test to benchmark against the thread-safe LB. Below, we proceed to compare the numerical prediction of the turbulent stress components against available experimental data[37, 38].
In Figure 8, plots of the turbulent kinetic energy components (streamwise and radial) and the Reynolds stresses taken at several locations downstream of the nozzle after the developing region are provided. The second-order moments are made non-dimensional by the mean square centerline velocity . As recalled above, the LB results have been superposed to the experimental data reported in[37] (black plusses) and[38] (red crosses).
Figure 8. Panel (a) and (b) show the axial and radial rms values of the fluctuating velocity components, while panel (c) reports the Reynolds stresses (all made non-dimensional by the centerline axial velocity), taken at different sections along the jet axis. The LB results (filled circles) are compared against the experiments of Hussein et al.[37] and Panchapakesan and Lumley[38]
The present results agree very well with the experiments discussed in Hussein et al., as evidenced by the overlapping between simulation and experimental data sets. In particular, it is interesting to note that the LB simulations predict the position of the peak of the Reynolds stress around while they zeroed at , values close to those observed in incompressible turbulent jets experiments (see[34]).
A note on the implementation and the performances on GPU of the High-order thread-safe LB
The codes have been run on two different accelerators: GeForce RTX 3090 and Nvidia A100 (on Leonardo supercomputer). The former features 24 Gbytes of RAM shared by 10496 Cuda cores with a peak performance TeraFLOPS in single precision and a memory bandwidth of GB/s, while the latter has 64 Gbytes of RAM shared by 6912 Cuda cores with a peak performance TeraFLOPS in single precision with a memory bandwidth of .
The performances of the GPU porting implementations have been measured in billions of lattice update per second (GLUPS), defined as:
where are the number of lattice nodes along the three spatial dimensions, is the number of simulation time steps and is the run wall-clock time (in seconds) needed to perform the simulation.
The computational performances have been analyzed by running the pressure-driven flow benchmark on a domain having grid points.
The code has been ported on GPU by exploiting OpenACC directives that, as shown below, allow peak performances in line with the state-of-the-art LB computing with virtually no extra cost in terms of coding. Implementation details can be found in[10], and the actual formulas for the high-order equilibria and non-equilibrium distributions are reported in the appendix. Moreover, the interested reader is referred to the publicly available code shared on Git-Hub (see code availability section).
In the case of the pressure-driven flow, we reached a peak performance of GLUPS on RTX3090 and on the A100 mounted on Leonardo. We then inspected the roofline model of the stream-collide kernels, which exposes an operational intensity FLOP/byte achieving almost the ideal performance (bounded by the memory-bandwidth roof of the plot) of Teraflop/s, booth larger than those reported in[10] ( flop/byte and Teraflops/s) and of other state-of-the-art lattice Boltzmann HPC codes[21, 43, 44]. Thus, the larger number of operations needed to compute the high order equilibria and non-equilibrium distributions results in an increase of the number of floating point operations per unit time while the transferred amount information remains unchanged. In turn, the arithmetic intensity of the algorithm increases as well allowing to approach to the ideal peak performance delivered by the device. Lastly, it is worth noting that while state-of-the-art implementations are usually based on standard BGK collisional operator, our approach exploits a high-order regularized procedure aimed at increasing the stability and accuracy of the LB, thus allowing to span a broader and lower range of viscosities, as low as the ones reachable via MRT-like models, along with a sensible reduction of the memory footprint with respect to the standard LB model based on the AB streaming strategy.
Conclusions
This work presented a high-order, thread-safe implementation of the lattice Boltzmann model for HPC simulations of turbulent flows, with the aim of exploiting in full the memory bandwidth of GPU-based architectures. Our results show an outstanding performance while retaining the coding simplicity of standard lattice Boltzmann implementations, with a sensible reduction of the memory footprint if compared to fused streaming-collision strategies and with no need to resort to more complex and exotic streaming algorithms[20, 21]. Moreover, a new boundary condition based on the concept of non-equilibrium extrapolation method and Grad’s reconstruction of missing data[31] has been developed, that allows for an efficient implementation of Dirichlet and boundary conditions by imposing conditions at the boundaries directly on the macropscopic fields of interest. The above HPC implementation has been benchmarked against two different case studies, namely the turbulent flow in a straight channel at and the axisymmetric turbulent jet at . The results confirm the accuracy of the high-order TSLB and, in turn, its ability to predict the behavior of highly turbulent flows. Future work will be focused on the possibility of scaling the single-GPU performances of the thread-safe LB on multi-GPU architectures with the aim of scaling up to very large-scale simulations of turbulent flows calling for the computational power of existing pre-exascale supercomputers and upcoming exascale infrastructures.
Acknowledgments
A.M. and M.L. acknowledges the CINECA Computational Grant IsCb3 ”recTSLB”, IsCb4 ”LLBfast” and IsCa9 ”3DMPILB” under the ISCRA initiative, for the availability of high performance computing resources and support. S.S. acknowledges funding from ERC-PoC2 grant No. 101081171 (DropTrack).
Code availability
The code is freely available at https://github.com/andreamontessori/entropicThreadsafe
References
- [3]K. R. Sreenivasan, Reviews of Modern Physics 71, S383 (1999).
- [4]K. Sreenivasan and M. Gad-el Hak, “Frontiers in experimental fluid mechanics,” (1989).DOI
- [5]S. Succi, G. Amati, M. Bernaschi, G. Falcucci, M. Lauricella, and A. Montessori, Computer & Fluids 181, 107 (2019).
- [6]P. L. Bathnagar, E. P. Gross, and M. Krook, Physical Review 94, 511 (1954).
- [7]A. Montessori, G. Falcucci, P. Prestininzi, M. La Rocca, and S. Succi, Physical Review E 89, 053317 (2014).
- [8]D. d’Humières, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences 360, 437 (2002).
- [9]M. Geier, A. Greiner, and J. G. Korvink, Physical Review E 73, 066705 (2006).
- [10]A. Montessori, M. Lauricella, A. Tiribocchi, M. Durve, M. La Rocca, G. Amati, F. Bonaccorso, and S. Succi, Journal of Computational Science 74, 102165 (2023).
- [11]O. Malaspinas, arXiv preprint arXiv:1505.06900 (2015).arxiv.org/abs/1505.06900
- [12]H. Grad, Communications on pure and applied mathematics 2, 331 (1949).
- [13]H. Grad, Communications on Pure and Applied Mathematics 2, 325 (1949).
- [14]T. Krüger, H. Kusumaatmaja, A. Kuzmin, O. Shardt, G. Silva, and E. M. Viggen, Springer International Publishing 10, 4 (2017).
- [15]S. Succi, The Lattice Boltzmann equation: For complex states of flowing matter (Oxford University Press, 2018).
- [16]S. Chapman and T. G. Cowling, The mathematical theory of non-uniform gases: an account of the kinetic theory of viscosity, thermal conduction and diffusion in gases (Cambridge university press, 1990).
- [17]J. Mattila, J. Hyväluoma, J. Timonen, and T. Rossi, Computers & Mathematics with Applications 55, 1514 (2008).
- [18]J. Latt, C. Coreixas, and J. Beny, Plos one 16, e0250306 (2021).
- [19]P. Bailey, J. Myre, S. D. Walsh, D. J. Lilja, and M. O. Saar, in 2009 international conference on parallel processing (IEEE, 2009) pp. 550–557.
- [20]M. Geier and M. Schönherr, Computation 5, 19 (2017).
- [21]M. Lehmann, Computation 10, 92 (2022).
- [22]A. Montessori, P. Prestininzi, M. La Rocca, and S. Succi, Physical Review E 92, 043308 (2015).
- [23]A. Montessori, M. Lauricella, M. La Rocca, S. Succi, E. Stolovicki, R. Ziblat, and D. Weitz, Computers & Fluids 167, 33 (2018).
- [24]J. Latt and B. Chopard, Mathematics and Computers in Simulation 72, 165 (2006).
- [25]R. Zhang, X. Shan, and H. Chen, Physical Review E 74, 046703 (2006).
- [26]P. J. Dellar, Physica A: Statistical Mechanics and its Applications 362, 132 (2006).
- [27]A. Montessori, M. Lauricella, N. Tirelli, and S. Succi, Journal of Fluid Mechanics 872, 327 (2019).
- [29]X. Shan, X.-F. Yuan, and H. Chen, Journal of Fluid Mechanics 550, 413 (2006).
- [30]Z. Guo, C. Zheng, and B. Shi, Physics of fluids 14, 2007 (2002).
- [31]S. Chikatamarla, S. Ansumali, and I. Karlin, Europhysics Letters 74, 215 (2006).
- [32]I. Marusic, B. J. McKeon, P. A. Mon kewitz, H. M. Nagib, A. J. Smits, and K. R. Sreeni-vasan, Physics of fluids 22 (2010).
- [33]J. Kim, P. Moin, and R. Moser, Journal of fluid mechanics 177, 133 (1987).
- [34]S. B. Pope, Turbulent Flows (Oxford University Press, 2000).DOI
- [35]J. Latt, Freely available online at http://lbmethod. org /_media/howtos: lbunits. pdf (2008).
- [36]L.-S. Luo, Physical Review E 62, 4982 (2000).
- [37]H. J. Hussein, S. P. Capp, and W. K. George, Journal of Fluid Mechanics 258, 31 (1994).
- [38]N. R. Panchapakesan and J. L. Lumley, Journal of Fluid Mechanics 246, 197 (1993).
- [39]X. Xue, H.-D. Yao, and L. Davidson, Physics of Fluids 34 (2022).
- [40]Z. Wang, P. He, Y. Lv, J. Zhou, J. Fan, and K. Cen, Flow, turbulence and combustion 84, 669 (2010).
- [41]I. Wygnanski and H. Fiedler, Journal of Fluid Mechanics 38, 577 (1969).
- [42]B. Boersma and S. Lele, Center for Turbulence Research, Annual Research Briefs (1999).
- [43]M. Lehmann, L. M. Oehlschlägel, F. P. Häusl, A. Held, and S. Gekle, Microplastics and Nanoplastics 1, 1 (2021).
- [44]J. Latt, O. Malaspinas, D. Kontaxakis, A. Parmigiani, D. Lagrava, F. Brogi, M. B. Bel-gacem, Y. Thorimbert, S. Leclaire, S. Li, et al., Computers & Mathematics with Applica-tions 81, 334 (2021).