betaFeedback

WorksPhysics and Astronomy

accLB: A High-Performance Lattice Boltzmann Code for Multiphase Turbulence on Multi-Gpu Architectures

Abstract

In this work, we present accLB, a high-performance Fortran-based lattice Boltzmann (LB) solver tailored to multiphase turbulent flows on multi-GPU architectures. The code couples a conservative phase-field formulation of the Allen-Cahn equation with a thread-safe, regularized LB method to capture complex interface dynamics. Designed from the ground up for HPC environments, accLB employs MPI for distributed memory parallelism and OpenACC for GPU acceleration, achieving excellent portability and scalability on leading pre-exascale systems such as Leonardo and LUMI. Benchmark tests demonstrate strong and weak scaling efficiencies on multiple GPUs. Physical validation includes direct numerical simulations of homogeneous isotropic turbulence (HIT). Further, we examine bubble-laden HIT and observe a transition to a −3-3 energy scaling, as in experiments and theoretical predictions, due to bubble-induced dissipation, along with enhanced small-scale intermittency. These results highlight accLB as a robust and scalable platform for the simulation of multiphase turbulence in extreme computational regimes.

Introduction

Multiphase turbulent flows play a central role in a wide array of natural phenomena and industrial processes, from cloud formation and ocean–atmosphere interactions to chemical reactors and spray combustion [20, 39, 14, 38]. Accurately modeling these flows is particularly challenging due to the complex interplay between turbulence and dispersed phases, which span a vast range of spatial and temporal scales [4, 7]. Even single-phase turbulence exhibits multiscale dynamics, from the large energy-containing eddies down to the small-scale dissipative motions at the Kolmogorov scale. In multiphase systems, the complexity deepens as interface phenomena push the smallest relevant scales down to the molecular level.

Capturing these coupled multiscale features with high fidelity requires direct numerical simulations (DNS), which are notoriously computationally demanding—especially when resolving deformable fluid interfaces. As such, the development of scalable numerical solvers tailored for high-performance computing (HPC) platforms is essential to advance our understanding of turbulent multiphase flows.

Methods

Navier-Stokes equations for multiphase flows with interfaces

The multiphase system analyzed in this study is described by the continuity and momentum equations, which are numerically solved using the lattice Boltzmann method (LBM). These equations read:

∂tρ+∂α(ρuα)=0,(1)\partial_t \rho+ \partial_\alpha(\rho u_\alpha) = 0, \tag*{(1)}
∂t(ρuα)+∂β(ρuαuβ)=−∂αp+∂β(ρν[∂βuα+∂αuβ])+Fαs,(2)\partial_t(\rho u_\alpha) + \partial_\beta(\rho u_\alpha u_\beta) = -\partial_\alpha p + \partial_\beta\left(\rho\nu[\partial_\beta u_\alpha+ \partial_\alpha u_\beta]\right) + F_\alpha^s, \tag*{(2)}

where uαu_\alpha denotes the velocity field, ρ\rho the fluid density, pp the macroscopic pressure, ν\nu the kinematic viscosity, and FαsF_\alpha^s the surface tension force. Greek indices represent Cartesian tensor components.

The surface tension force FαsF_\alpha^s is expressed as the divergence of the capillary stress tensor, Fαs=∂β(γ(δαβ−nαnβ))F_\alpha^s = \partial_\beta\left(\gamma(\delta_{\alpha\beta} - n_\alpha n_\beta)\right), where nαn_\alpha is the local interface normal and γ\gamma the surface tension coefficient.

Allen-Cahn Equation for Interface Tracking

The interface between the two fluid phases is tracked using a conservative phase-field formulation based on the Allen-Cahn equation. The scalar order parameter ϕ\phi, which varies smoothly between 0 and 1 across the interface, evolves according to:

∂tϕ+uα∂αϕ=D∂α∂αϕ−κ∂α(ϕ(1−ϕ)nα),(3)\partial_t \phi+ u_\alpha\partial_\alpha\phi= D \partial_\alpha\partial_\alpha\phi- \kappa\partial_\alpha\left(\phi(1-\phi)n_\alpha\right), \tag*{(3)}

where DD denotes the interface diffusivity and κ=4D/δ\kappa= 4D/\delta, with δ\delta representing the interface width. The interface is implicitly located at the iso-surface ϕ=ϕ0=0.5\phi= \phi_0 = 0.5. The equilibrium profile across an interface centered at x0x_0 is given by [21], ϕ(x,y,z)=ϕ0±ϕH−ϕL2tanh⁡(∣x−x0∣+∣y−y0∣+∣z−z0∣δ)\phi(x,y,z) = \phi_0 \pm\frac{\phi_H-\phi_L}{2}\tanh\left(\frac{|x-x_0|+|y-y_0|+|z-z_0|}{\delta}\right).

This phase-field equation is tightly coupled to the hydrodynamic fields via the advection term.

To accurately resolve interface dynamics in the presence of strong advection, the convective term in Eq. (4) is discretized using a fifth-order Weighted Essentially Non-Oscillatory (WENO-5) scheme [19]. This high-order scheme minimizes numerical dissipation while capturing sharp gradients and reducing spurious oscillations, making it well-suited for turbulent multiphase flows.

Thread-safe lattice Boltzmann model

The LBM provides an efficient solver for the Navier-Stokes equations, particularly for complex or multiphase flows. In this work, we employ a thread-safe and regularized LBM framework, as described in [29, 23].

The second-order accurate discrete lattice Boltzmann equation over a set of qq velocity directions is given by [40, 22]:

fi(xα+ciαΔt,t+Δt)=fi(xα,t)+ω(fieq−fi)+(1−ω2)Si,(4)f_i(x_\alpha+c_{i\alpha}\Delta t,t+\Delta t)=f_i(x_\alpha,t)+\omega\left(f_i^{eq}-f_i\right)+\left(1-\frac{\omega}{2}\right)S_i, \tag*{(4)}

where ω\omega is a relaxation rate, fif_i are the distribution functions, fieqf_i^{eq} their equilibrium counterparts, and SiS_i denotes an external forcing term incorporated through the Guo forcing scheme [16]:

Si=wi(ciα−uαcs2+ciαciβuβcs4)Fα,(5)S_i=w_i\left(\frac{c_{i\alpha}-u_\alpha}{c_s^2}+\frac{c_{i\alpha}c_{i\beta}u_\beta}{c_s^4}\right)F_\alpha, \tag*{(5)}

being FαF_\alpha the total body force acting on the fluid.

The relaxation time, τ=1/ω\tau=1/\omega, controls the kinematic viscosity through the relation ν=cs2(τ−0.5)\nu=c_s^2(\tau-0.5) [40, 22, 28].

Defining fineq=fi−fieq+0.5Sif_i^{neq}=f_i-f_i^{eq}+0.5S_i and substituting in Eq. (4), the post-collision update can be reformulated as [11, 12]:

fi(xα+ciαΔt,t+Δt)=fieq+(1−ω)fineq+12Si.(6)f_i(x_\alpha+c_{i\alpha}\Delta t,t+\Delta t)=f_i^{eq}+(1-\omega)f_i^{neq}+\frac{1}{2}S_i. \tag*{(6)}

In the thread-safe implementation, both equilibrium and non-equilibrium components are reconstructed directly from macroscopic fields, avoiding direct access to full distribution functions and preventing race conditions due to non-local memory operations [29, 30]. The equilibrium distribution reads, fieq=wi(p∗+ciαuαcs2+(ciαciβ−cs2δαβ)uαuβ2cs4)f_i^{eq}=w_i\left(p^*+\frac{c_{i\alpha}u_\alpha}{c_s^2}+\frac{(c_{i\alpha}c_{i\beta}-c_s^2\delta_{\alpha\beta})u_\alpha u_\beta}{2c_s^4}\right), where p∗=p/(ρcs2)p^*=p/(\rho c_s^2) is the dimensionless pressure. The non-equilibrium contribution is reconstructed up to second order as fineq=ciαciβa1,αβ(2)2cs4f_i^{neq}=\frac{c_{i\alpha}c_{i\beta}a_{1,\alpha\beta}^{(2)}}{2c_s^4}, where a1,αβ(2)a_{1,\alpha\beta}^{(2)} denotes the second order Hermite coefficient of non-equilibrium [25]. Macroscopic variables are obtained from the distribution functions using p∗(xα,t)=∑ifi(xα,t)p^*(x_\alpha,t)=\sum_i f_i(x_\alpha,t) and uα(xα,t)=∑ifi(xα,t)ciα+12∑iSiu_\alpha(x_\alpha,t)=\sum_i f_i(x_\alpha,t)c_{i\alpha}+\frac{1}{2}\sum_i S_i: The second order Hermite coefficient, a1,αβ(2)a_{1,\alpha\beta}^{(2)}, is assessed by the sum over the discrete nonequilibrium populations fineqf_i^{neq} weighted on the second order Hermite polynomial Hi,αβ(2)=ci,αci,β−cs2δαβ\mathcal{H}_{i,\alpha\beta}^{(2)} = c_{i,\alpha}c_{i,\beta} - c_s^2\delta_{\alpha\beta} obtaining:

a1,αβ(2)(xα,t)=∑ifineq(xα,t)(ci,αci,β−cs2δαβ).(7)a_{1,\alpha\beta}^{(2)}(x_\alpha,t)=\sum_i f_i^{neq}(x_\alpha,t)(c_{i,\alpha}c_{i,\beta}-c_s^2\delta_{\alpha\beta}). \tag*{(7)}

where, as shown in Ref. [25], apart from higher-order smallness in the Mach number (Ma), the second order coefficient macroscopically reads: a1,αβ(2)=−cs2[∂α(uβ)+∂β(uα)]+O(Ma3)a_{1,\alpha\beta}^{(2)}=-c_s^2[\partial_\alpha(u_\beta)+\partial_\beta(u_\alpha)]+O(\mathrm{Ma}^3).

Extension for high-density and viscosity contrasts

To accurately simulate multiphase flows with strong density and viscosity contrasts, additional force contributions are introduced into the momentum equation. The total force acting on the system is given by [10]:

Fα=Fαs+Fαp+Fαv,(8)F_\alpha=F_\alpha^s+F_\alpha^p+F_\alpha^v, \tag*{(8)}

where FαsF_\alpha^s is the surface tension force, computed via a phase-field-based formulation[21], Fαs=−γκ∇ϕ∣∇ϕ∣F_\alpha^s=-\gamma\kappa\nabla\phi\lvert\nabla\phi\rvert. The pressure-induced force reads, Fαp=−p∗cs2∂αρF_\alpha^p=-p^*c_s^2\partial_\alpha\rho, where ρ=ϕρl+(1−ϕ)ρg\rho=\phi\rho_l+(1-\phi)\rho_g is the local mixture density. Given that the pressure is defined as p=p∗ρcs2p=p^*\rho c_s^2, its gradient contains implicit terms that require explicit correction via FαpF_\alpha^p. Lastly, the viscous correction force is given by Fαv=−νωcs2Δt[∑i(fi−fieq)ciαciβ]∂αρF_\alpha^v=-\frac{\nu_\omega}{c_s^2\Delta t}\left[\sum_i(f_i-f_i^{eq})c_{i\alpha}c_{i\beta}\right]\partial_\alpha\rho:

Spatial derivatives appearing in the force terms and phase-field evolution equation are discretized using lattice-based stencils, ∂αΨ=1cs2∑iwiΨ(xα+ciα)ciα\partial_\alpha\Psi=\frac{1}{c_s^2}\sum_iw_i\Psi(x_\alpha+c_{i\alpha})c_{i\alpha} and ∂α∂βΨ=1cs2(∑i≠0wiΨ(xα+ciα)−w0Ψ(xα))\partial_\alpha\partial_\beta\Psi=\frac{1}{c_s^2}\left(\sum_{i\ne0}w_i\Psi(x_\alpha+c_{i\alpha})-w_0\Psi(x_\alpha)\right)

This modeling framework enables robust and efficient simulations of multiphase flows with large physical property mismatches while maintaining numerical stability and thread safety through the use of regularization techniques.

Implementation

The accLB code is developed in Fortran 95, utilizing modules to keep the code organized and reduce clutter. Variables related to specific features (such as fluids and phase-field) or methods (like time integrators) are grouped into separate modules for better structure. Portability and readability are key design goals of accLB, ensuring the code can run across multiple architectures and is accessible for contributors despite the complexity of the underlying model.

From its inception, accLB has been designed with parallel computing platforms in mind. It uses the Message Passing Interface (MPI), the standard for inter-node communication. However, a corresponding serial version of accLB is available and can be selected during compilation.

The parallelization of the hydrodynamic solver in accLB is based on domain decomposition [15], where the simulation domain is divided into one-, two-, or three-dimensional blocks distributed across MPI tasks. The block decomposition is selected at run-time, without the need to recompile the code, allowing users to optimize performance based on the global grid size and the number of available compute cores. For cubic domains, three-dimensional decomposition typically yields the best performance by minimizing the surface area involved in inter-process communication. The current implementation employs a halo region of two lattice spacings, which is necessary to calculate the derivative terms in the WENO5 scheme. This halo width is flexible and could be easily extended in future developments to accommodate more advanced algorithms.

The accLB code leverages OpenACC directives to enable execution on GPU devices, allowing for efficient off-loading of compute-intensive tasks. This approach facilitates parallel acceleration with minimal code modifications, ensuring portability across different hardware architectures. By utilizing OpenACC, the code can fully utilize available GPU resources to enhance performance significantly, particularly for large-scale simulations. OpenACC also ensures broad compatibility across various GPU platforms, including devices from NVIDIA, AMD, and other vendors that support the standard. This flexibility allows the code to remain portable and adaptable to evolving hardware environments without requiring extensive rewrites. Finally, OpenACC enables compilation and execution on purely CPU-based architectures as well. The code exploits a hybrid shared-distributed memory parallelization strategy in such cases, combining OpenACC and MPI to scale across multiple CPU cores and nodes efficiently. This unified parallel framework allows direct performance comparison across different high-performance computing platforms, such as CPU-based versus GPU-based clusters, aiding in performance benchmarking and resource optimization.

The time-marching implementation of the lattice-Boltzmann equation Eq. (4) exploits the reformulation given in Eq. (6) to facilitate completely independent computations at each computational node, effectively eliminating race conditions. This design includes the thread-safe paradigm introduced by Montessori et al. [31, 30, 29]. This approach enhances data locality and removes memory dependencies during the propagation step by reconstructing the post-collision distribution functions through Hermite projection. Such a strategy is particularly advantageous for parallel computing architectures as it ensures safe and efficient execution across multiple processing units without the need for complex synchronization mechanisms. Moreover, this method significantly reduces the memory footprint, enabling the simulation of large-scale fluid dynamics problems with improved performance and stability, achieving a balance between computational efficiency and the accuracy required for simulating complex fluid behaviors. Thus, the primary variables in accLB include the 27 distribution functions alongside with the ten recovered macroscopic fields (dimensionless pressure p∗p^{*}, velocity vector uαu_{\alpha}, and six Hermite coefficient terms a1,αβ(2)a^{(2)}_{1,\alpha\beta} of the corresponding symmetric tensor). The distribution functions can be stored using one of two competing memory layouts: array of structures (AoS) or structure of arrays (SoA), regardless of the underlying memory order (row-major as in C or column-major as in Fortran) [41]. The accLB code employs the SoA layout, which has demonstrated superior performance over the AoS configuration. In particular, the SoA layout enables a unit-stride access pattern during the streaming step, ensuring coalesced memory access on GPU architectures. This choice significantly improves memory efficiency, reducing bandwidth-related bottlenecks and making it the optimal strategy for high-performance implementations.

Performances

To evaluate performance, we simulate two different benchmark cases: 1) a single-component fluid and 2) a two-component system containing a single droplet of heavier density dispersed in a lighter continuous phase (density contrast equal to 100), both with a kinematic viscosity equal to 0.005 in lattice units.

In the present work, we report both strong and weak scaling to assess the parallel performance of the code, where strong scaling provides information on how the execution time decreases with an increasing number of GPUs for a fixed problem size, reflecting the efficiency in accelerating a single workload. Instead, weak scaling analyzes how well the code maintains a constant execution time when the problem size grows proportionally with the number of GPUs, indicating scalability to larger workloads. The two scaling analyses provide a comprehensive view of parallel efficiency and the ability of accLB to handle increasingly demanding simulations.

Hence, the simulation domain is fixed as a cubic box with 512 lattice points per side in the strong scaling benchmarks. In the weak scaling simulations, we consider a cubic sub-domain of side 512 lattice points for each GPU device (fixed sub-domain size), adopting a linear decomposition along the z-axis. Thus, we obtain an overall simulation box with sides 512, 512, and 512⋅np512 \cdot n_p with npn_p denoting the number of GPU devices for the weak scaling simulations.

Efficiency is assessed using the Giga Lattice Updates Per Second (GLUPS) metric, defined as:

GLUPS=LxLyLz109ts,(9)\mathrm{GLUPS} = \frac{L_x L_y L_z}{10^{9}t_s}, \tag*{(9)}

where tst_s is the run (wall-clock) time (in seconds) per single time step iteration, while LxL_x, LyL_y, and LzL_z are the domain sizes.

In the present work, we define the speedup, SpS_p, in terms of GLUPS ratio as follows:

Sp=GLUPSpGLUPSs,(10)S_p = \frac{\mathrm{GLUPS}_p}{\mathrm{GLUPS}_s}, \tag*{(10)}

where GLUPSs\mathrm{GLUPS}_{s} denotes the Giga Lattice Updates Per Second for the code running on a single GPU and GLUPSp\mathrm{GLUPS}_{p} represents the Giga Lattice Updates Per Second for the code running on npn_{p} GPU cards. It is worth highlighting that the above definition of SpS_{p} differs from the typical expression given in the literature [17]. Finally, the parallel efficiency, EpE_{p}, reads:

Ep=Spnp.(11)E_{p} = \frac{S_{p}}{n_{p}}. \tag*{(11)}

The benchmark simulations were conducted on two different GPU-based architectures: the pre-exascale Leonardo HPC system at CINECA in Italy, equipped with four NVIDIA Ampere™ A100 Tensor Core GPU devices per computing node, each GPU card with 64 GB of HBM2e memory, and the LUMI pre-exascale supercomputer, hosted at CSC in Finland, based on the HPE Cray EX architecture, each compute node equipped with four AMD Instinct™ MI250X GPUs, where each GPU device contains two Graphics Compute Dies (GCDs), with each die providing 64 GB of HBM2e memory, for a total of 128 GB per GPU. The latter architecture yields eight GPU compute units per node, enabling highly parallel and memory-intensive workloads.

In all the benchmarks executed on the pre-exascale Leonardo HPC, the source code was compiled using the Nvidia Compiler, version 24.3, in conjunction with OpenMPI 4.1.6, an open-source implementation of the Message Passing Interface (MPI) standard. Instead, for the benchmark runs on the LUMI pre-exascale HPC, we used the HPE Cray Fortran Compiler, version 17.0.1, along with the Cray-provided MPI implementation based on MPICH, version 8.1.29.

Strong scaling results are presented in Figure 1 in terms of the speedup, SpS_{p} and parallel efficiency, EpE_{p}, for the single- and double-component systems, evaluated with the MPI decomposition that yields the best performance in terms of GLUPS over the two different GPU-based HPC infrastructures. Note that each AMD Instinct™ MI250X card provides two GPU units, and the performance estimators are evaluated as a function of the number of GPU units. The benchmark data indicate that the impact of parallel communication increases as the computational sub-domain of each GPU device diminishes.

Speed-up and parallel-efficiency plots for one- and two-component cases

Figure 1. Panels (a) and (b) show the speed up SpS_p and parallel efficiency EpE_p for the single component case measured in strong and weak scaling on the NVIDIA Ampere™ A100 and AMD Instinct™ MI250 GPU-based HPC infrastructures. Panels (c) and (d) report the corresponding data for the two-component benchmark case.

To better illustrate the effect of communication overhead, we also report weak scaling results, where the sub-domain size per MPI process (i.e., per GPU device) remains constant. In Figure 1, we show weak scaling performance using a fixed cubic sub-domain of side 512 lattice points per GPU. This scaling study employs a linear decomposition along the zz-axis and demonstrates excellent parallel efficiency, obtaining a peak performance value of 154.94 GLUPS for single-component and 105.48 GLUPS for two-component systems on 64 GPU devices in weak-scaling on NVIDIA Ampere™ A100 GPU-based cluster, versus 92.63 GLUPS and 65.67 GLUPS for the single- and double-component systems, respectively, for the AMD Instinct™ MI250 GPU-based cluster.

Results

Single-Component Homogeneous Isotropic Turbulence: Energy and Pressure Spectra and Single Point Statistics

We evaluate the accuracy of the accLBacc\mathrm{LB} code [30, 23] in performing fully developed turbulent flow simulations by analyzing single-component homogeneous isotropic turbulence (HIT), focusing on energy and pressure spectra along with selected single-point statistics. We simulate three separate cases of increasing Taylor-scale Reynolds numbers, Reλ=150Re_{\lambda}=150, 230 and 370, on a 5123512^{3} uniform Cartesian grid with unit spacing, Δx=1\Delta x=1.

The case at Reλ=150Re_{\lambda}=150 can be effectively considered as a Direct Numerical Simulation (DNS), where the Kolmogorov length scale is approximately equal to the grid spacing, Δx\Delta x, ensuring adequate resolution of the smallest turbulent scales. Conversely, the simulations at Reλ=230Re_{\lambda}=230 and 370 lie beyond the strict DNS regime, with a Kolmogorov scale of approximately 0.4Δx0.4\Delta x and 0.15Δx0.15\Delta x, respectively.

The homogeneous isotropic turbulent state in all three cases is maintained through large-scale energy injection via the ABC (Arnold–Beltrami–Childress) forcing method. This deterministic, divergence-free forcing scheme introduces a time-independent velocity field that acts as a steady source of energy at the largest scales. Specifically, the forcing takes the form:

f(x)=A(sin⁡(kfz)+cos⁡(kfy)sin⁡(kfx)+cos⁡(kfz)sin⁡(kfy)+cos⁡(kfx))\mathbf{f}(\mathbf{x}) = A \begin{pmatrix} \sin(k_f z) + \cos(k_f y) \\ \sin(k_f x) + \cos(k_f z) \\ \sin(k_f y) + \cos(k_f x) \end{pmatrix}

where AA is the forcing amplitude and kfk_f the forcing wavenumber. The ABC forcing guarantees a sustained turbulent regime by balancing viscous dissipation with energy input, without introducing significant anisotropy, thereby providing an efficient and widely adopted method to achieve statistically stationary HIT in periodic domains [34].

Figure 2(a) presents the energy spectra for the cases under scrutiny, compared to the classical −5/3-5/3 slope predicted for the inertial subrange [35, 13]. The observed agreement serves as initial validation of the thread-safe LB solver’s capability to reproduce energy transfer mechanisms in single-component turbulent flows.

Figure 2. Panel (a) shows the energy spectra for ReλRe_\lambda, 150, 230 and 370. The expected −5/3-5/3 power-law in the inertial range is well captured across a broad range of wavenumbers. Panel (b) displays the corresponding pressure spectra, which follow the −7/3-7/3 scaling in the inertial range.

In Figure 2(b), the pressure spectra are plotted and compared to the theoretical −7/3-7/3 scaling law [42]. In both panels, the simulation at higher Reynolds number exhibits a broader spectrum, indicative of an extended inertial range and enhanced scale separation. As expected, deviations from the −5/3-5/3 and −7/3-7/3 power laws emerge at large scales, near the Taylor length scale, which is marked by vertical dashed lines in the plots.

Figure 3 shows single-point statistical features of the homogeneous isotropic turbulent field under examination.

Probability density function and structure functions

Figure 3. Panel (a): Normalized probability density function (PDF) of the longitudinal velocity gradients. Panel (b): Second-order structure function, compared with the theoretical r2r^2 scaling in the dissipation range and r2/3r^{2/3} Kolmogorov scaling in the inertial range. Panel (c): Third-order structure function, compared with the theoretical r2r^2 scaling in the dissipation range and r2/3r^{2/3} Kolmogorov scaling in the inertial range.

Panel (a) displays the probability density function (PDF) of the longitudinal velocity gradient at increasing Taylor Reynolds numbers.

As shown in panel (a), the PDFs deviate significantly from a Gaussian distribution and show negative skewness, S≈0.42S \approx0.42 (S=⟨(∂u/∂x)3⟩/⟨(∂u/∂x)2⟩3/2S = \langle(\partial u/\partial x)^3\rangle/\langle(\partial u/\partial x)^2\rangle^{3/2}), and a flatness coefficient F≈5.25F \approx5.25 (F=⟨(∂u/∂x)4⟩/⟨(∂u/∂x)2⟩2F = \langle(\partial u/\partial x)^4\rangle/\langle(\partial u/\partial x)^2\rangle^2), both in line with prior studies [18]. In the physics of HIT, negative skewness points to the presence of long-tails in the statistical distribution of the velocity increments, highlighting the presence of anisotropic effects at small scales, typically caused by persistent vortex stretching, while the elevated flatness signals the presence of intermittent rare-events, i.e., intense fluctuations in the velocity gradients [26].

Panel (b) reports the second-order structure function, S2(r)=⟨[u(xi+ri)−u(xi)]2⟩S_2(r) = \langle[u(x_i+r_i)-u(x_i)]^2\rangle, benchmarked against two theoretical power laws [26]: a r2r^2 scaling at small rr, typical of the dissipation range, and the Kolmogorov r2/3r^{2/3} scaling in the inertial range, representative of energy transfer across larger scales. Panel (c) reports the value of the third-order invariant S3(r)S_3(r) across scales. In classical Kolmogorov theory, S3(r)S_3(r) follows a linear scaling in the inertial range (i.e., S3(r)=−45εrS_3(r) = -\frac{4}{5}\varepsilon r, where ε\varepsilon is the energy dissipation rate), implying that the energy transfer rate is scale invariant within the inertial range, i.e., that the turbulence exhibits a form of self-similarity across scales [35, 26, 5]. Both statistics show agreement with the expected theoretical behavior and tend to deviate from the theoretical behavior as rr increases to values close to the integral length scale [18].

Bubble-Laden Homogeneous Isotropic Turbulence: Effect of the dispersed phase on turbulence modulation

In this section, we investigate the effect of a dispersed, buoyant gas phase on the turbulence. Specifically, we examine how bubbles influence the energy spectrum at selected wavelengths and explore this behavior across a range of Taylor Reynolds numbers. The bubble volume fraction is fixed at 6%6\% of the total domain, which remains 5123512^3, as in the single-phase case (see Figure 4, reporting two snapshots of the bubble and velocity field taken at the statistical steady state). The computed values of ReλRe_\lambda for the cases under consideration are 65, 86 and 114. The surface tension coefficient is set to γ=0.01\gamma= 0.01 (in lattice units) and is kept constant across all simulations. The turbulent Weber number, defined as WeT=urms2λρL/γWe_T = u_{\mathrm{rms}}^2 \lambda\rho_L/\gamma is of order one in all simulations (in the formula urmsu_{\mathrm{rms}} is the root mean square value of the velocity fluctuations, λ\lambda the Taylor length scale and ρL\rho_L the bulk density). The turbulent Froude number, FrT=urms/gLIFr_T = u_{\mathrm{rms}}/\sqrt{gL_I}, where LIL_I is the integral length scale, is approximately 2, indicating that while bubbles are influenced by turbulence, their buoyancy remains sufficiently strong to impose a preferential upward motion.

Instantaneous snapshots of the bubble field and velocity field

Figure 4. Instantaneous snapshots of the bubble field panel (a) and velocity field (b) at Reλ=114Re_{\lambda}=114 and 6% bubble volume fraction. Snapshots are taken at the statistical steady state.

As shown in Figure 5, the energy spectra reveal a clear transition in scaling behavior with increasing ReλRe_{\lambda}, from the classical −5/3-5/3 Kolmogorov scaling to a steeper −3-3 power law, typical of 2D turbulence[1]. This trend aligns with experimental observations [27] and theoretical predictions [36]. The emergence of the steeper spectrum is a distinct signature of bubble-induced dissipation at intermediate wavenumbers. Notably, this effect is only observed in buoyant bubble–liquid systems, suggesting that buoyancy plays a critical role in enabling new pathways of energy transfer within homogeneous turbulence. The transition occurs around k=10k = 10 (corresponding to r≈35r \approx35), approximately one-third of the integral length scale.

Figure 5. Energy spectra for bubble-laden HIT at different ReλRe_{\lambda}. The transition from −5/3-5/3 to −3-3 scaling, observed experimentally in [27], is captured in the intermediate wavenumber range.

Risso [36] analytically demonstrated that the −3-3 scaling in dispersed multiphase flows arises due to the presence of strong, statistically independent, and uniformly distributed bursts within the turbulent field. These bursts can be interpreted as intermittent events driven by the complex dynamics of bubbles interacting with the carrier phase. As bubbles deform, break up, and re-coalesce, they continuously reorganize the surrounding flow, thus altering the statistical properties of HIT. This behavior is clearly illustrated in Figure 6, which compares the probability density functions (PDFs) of velocity increments in the single-phase and bubble-laden cases. The simulation parameters are otherwise identical, though the resulting Taylor Reynolds numbers differ slightly due to the enhanced activity at small scales caused by the dispersed phase [7].

Figure 6. Comparison of velocity increment PDFs between single-phase and bubble-laden HIT. The enhanced tails in the latter case indicate increased intermittency induced by bubble dynamics. In the figure, circles stand for the bubble-laden HIT while plusses for the single component case

While both PDFs display the expected non-Gaussian features of homogeneous isotropic turbulence, the presence of bubbles significantly amplifies the tails of the distribution. This broadening indicates a higher frequency of intense velocity fluctuations, a hallmark of intermittency. The more pronounced tails in the bubble-laden case point to localized, energetic bursts and rare events arising from bubble dynamics. These findings underscore how dispersed bubbles modify the fine-scale structure of turbulence, intensifying gradients and increasing deviations from the Gaussian statistics typically seen in single-phase flows. The presence of strong intermittent events is further supported by the increase of the flatness factor by a factor 3, from approximately 5 to 16.

Conclusions

In this work we have presented accLB, a high-performance, thread-safe Lattice Boltzmann solver specifically tailored for the simulation of multiphase turbulent flows on modern GPU-based supercomputing architectures. By coupling a conservative phase-field formulation of the Allen–Cahn equation with a regularized LBM, accLB have shown to accurately capture the dynamics of fluid interfaces in the presence of turbulence and large density and viscosity contrasts.

The code architecture exploits hybrid parallelism via MPI and OpenACC, offering excellent portability and scalability across heterogeneous hardware platforms. Benchmark tests performed on the EuroHPC pre-exascale systems Leonardo and LUMI demonstrate strong and weak scaling performance up to 64 parallel GPUs, with sustained throughput exceeding 150 GLUPS in large-scale runs.

The physical capabilities of accLB are validated through simulations of single-phase and bubble-laden homogeneous isotropic turbulence (HIT). In the single-phase case, we recover the classical energy and pressure spectra scalings and turbulence invariants across scales. For the bubble-laden scenarios, our simulations capture the emergence of a −3-3 scaling in the energy spectrum, as predicted by theory and observed in experiments, along with amplified small-scale intermittency. These findings confirm that accLB is a reliable tool to investigate turbulence modulation in complex multiphase systems.

In summary, accLB provides a scalable and accurate platform for high-fidelity turbulence simulations in multiphase flows, enabling the investigation of complex physical phenomena across a broad range of scales with high computational efficiency.

Acknowledgements

A.M. acknowledges fundings from the Italian Government through the PRIN (Progetti di Rilevante Interesse Nazionale) Grant (MOBIOS) ID: 2022N4ZNH3 -CUP: F53C24001000006 and computational support of CINECA through the ISCRA B project DODECA (HP10BUBFIL). M.L. acknowledges the support of the Italian National Group for Mathematical Physics (GNFM-INdAM). M.L. and S.S. acknowledge the support from the European Research Council under the ERCPoC Grant No. 101187935 (LBFAST). D.I. acknowledges financial support from the Research Council of Finland (Grant number: 354620). We acknowledge CSC, Finland for awarding this project access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium through CSC, Finland, Benchmark Access mode.

References

  1. [1]R Benzi and S Succi. Two-dimensional turbulence with the lattice boltzmann equation. Journal of Physics A: Mathematical and General, 23 (1):L1, 1990.DOI
  2. [4]Luca Brandt and Filippo Coletti. Particle-laden turbulence: Progress and perspectives. Annual Review of Fluid Mechanics, 54(Volume 54, 2022):159–189, 2022. ISSN 1545-4479.DOI
  3. [5]M Briscolini, P Santangelo, S Succi, and R Benzi. Extended self-similarity in the numerical simulation of three-dimensional homogeneous flows. Physical Review E, 50(3):R1745, 1994.pubmed.ncbi.nlm.nih.gov/9962259
  4. [7]Marco Crialesi-Esposito, Sergio Chibbaro, and Luca Brandt. The interaction of droplet dynamics and turbulence cascade. Communications Physics, 6(1):5, 2023.
  5. [10]Abbas Fakhari, Travis Mitchell, Christopher Leonardi, and Diogo Bolster. Improved locality of the phase-field lattice-boltzmann model for immiscible fluids at high density ratios. Physical Review E, 96(5):053301, 2017.DOI
  6. [11]Yongliang Feng, Pierre Boivin, Jérôme Jacob, and Pierre Sagaut. Hybrid recursive regularized lattice boltzmann simulation of humid air with application to meteorological flows. Physical Review E, 100(2):023304, 2019.DOI
  7. [12]Yongliang Feng, Pierre Boivin, Jérôme Jacob, and Pierre Sagaut. Hybrid recursive regularized thermal lattice boltzmann model for high subsonic compressible flows. Journal of Computational Physics, 394:82–99, 2019.DOI
  8. [13]Uriel Frisch and Andrei Nikolaevich Kolmogorov. Turbulence: the legacy of AN Kolmogorov. Cambridge university press, 1995.
  9. [14]Wojciech W Grabowski and Lian-Ping Wang. Growth of cloud droplets in a turbulent environment. Annual review of fluid mechanics, 45(1):293–324, 2013.DOI
  10. [15]William D Gropp. Parallel computing and domain decomposition. In Fifth International Symposium on Domain Decomposition Methods for Partial Differential Equations, pages 349–361. Pub by Soc for Industrial and Applied Mathematics Publ, 1992.
  11. [16]Zhaoli Guo, Chuguang Zheng, and Baochang Shi. Discrete lattice effects on the forcing term in the lattice boltzmann method. Physical review E, 65(4):046308, 2002.pubmed.ncbi.nlm.nih.gov/12006014
  12. [17]John L Hennessy and David A Patterson. Computer architecture: a quantitative approach. Elsevier, 2011.
  13. [18]Takashi Ishihara, Toshiyuki Gotoh, and Yukio Kaneda. Study of high–reynolds number isotropic turbulence by direct numerical simulation. Annual Review of Fluid Mechanics, 41:165–180, 2009. doi: 10.1146/annurev.fluid.010908.165203.DOI
  14. [19]Guang-Shan Jiang and Chi-Wang Shu. Efficient implementation of weighted eno schemes. Journal of Computational Physics, 126(1):202–228, 1996. ISSN 0021-9991. doi: https://doi.org/10.1006/jcph.1996.0130. URL https://www.sciencedirect.com/science/article/pii/S0021999196901308.
  15. [20]B. Jähne and H. Haußecker. Air-water gas exchange. Annual Review of Fluid Mechanics, 30(Volume 30, 1998):443–468, 1998. ISSN 1545-4479.DOI
  16. [21]Jeong Kim. A continuous surface tension force formulation for diffuse-interface models. Journal of computational physics, 204(2):784–804, 2005.
  17. [22]Timm Krüger, Halim Kusumaatmaja, Alexandr Kuzmin, Orest Shardt, Goncalo Silva, and Erlen Magnus Viggen. The lattice boltzmann method. Springer International Publishing, 10(978-3-14–15, 2017.
  18. [23]Marco Lauricella, Adriano Tiribocchi, Sauro Succi, Luca Brandt, Aritra Mukherjee, Michele La Rocca, and Andrea Montessori. Thread-safe multiphase lattice boltzmann model for droplet and bubble dynamics at high density and viscosity contrasts. arXiv preprint arXiv:2501.00846, 2025.arxiv.org/abs/2501.00846
  19. [25]Orestis Malaspinas. Increasing stability and accuracy of the lattice boltzmann scheme: recursivity and regularization. arXiv preprint arXiv:1505.06900, 2015.
  20. [26]WD McComb. Theory of turbulence. Reports on Progress in Physics, 58(10):1117, 1995.
  21. [27]Julian Martinez Mercado, Daniel Chehata Gomez, Dennis Van Gils, Chao Sun, and Detlef Lohse. On bubble clustering and energy spectra in pseudo-turbulence. Journal of fluid mechanics, 650:287–306, 2010.
  22. [28]Andrea Montessori and Giacomo Falcucci. Lattice Boltzmann modeling of complex flows for engineering applications. Morgan and Claypool Publishers, 2018.DOI
  23. [29]Andrea Montessori, Marco Lauricella, Adriano Tiribocchi, Mihir Durve, Michele La Rocca, Giorgio Amati, Fabio Bonaccorso, and Sauro Succi. Thread-safe lattice boltzmann for high-performance computing on gpus. Journal of Computational Science, 74:102165, 2023.
  24. [30]Andrea Montessori, Michele La Rocca, Giorgio Amati, Marco Lauricella, Adriano Tiribocchi, and Sauro Succi. High-order thread-safe lattice boltzmann model for high performance computing turbulent flow simulations. Physics of Fluids, 36(3), 2024.DOI
  25. [31]Andrea Montessori, Luiz A Hegele, and Marco Lauricella. A thread-safe lattice boltzmann model for multicomponent turbulent jet simulations. AIAA Journal, 63(3):1005–1012, 2025.DOI
  26. [34]O. Podvigina and A. Pouquet. On the non-linear stability of the 1:1:1 abc flow. Physica D: Nonlinear Phenomena, 75(4):471–508, 1994. ISSN 0167-2789. doi: https://doi.org/10.1016/0167-2789(94)00031-X. URL https://www.sciencedirect.com/science/article/pii/016727899400031X.
  27. [35]Stephen B. Pope. Turbulent Flows. Cambridge University Press, 2000.DOI
  28. [36]Frédéric Risso. Theoretical model for k-3 spectra in dispersed multiphase flows. Physics of fluids, 23(1), 2011.DOI
  29. [38]Gaetano Sardina, Francesco Picano, Luca Brandt, and Rodrigo Caballero. Continuous growth of droplet size variance due to condensation in turbulent clouds. Phys. Rev. Lett., 115:184501, Oct 2015.arxiv.org/abs/1501.07051
  30. [39]Laurier L. Schramm, Elaine N. Stasiuk, and D. Gerrard Marangoni. Surfactants and their applications. Annu. Rep. Prog. Chem., Sect. C: Phys. Chem., 99:3–48, 2003.
  31. [40]S. Succi. The Lattice Boltzmann equation: For complex states of flowing matter. Oxford University Press, 2018.
  32. [41]S. Succi, G. Amati, M. Bernaschi, G. Falcucci, M. Lauricella, and A. Montessori. Towards exascale lattice boltzmann computing. Computer and Fluids, 181:107–115, 2019. doi: https://doi.org/10.1016/j.compfluid.2019.01.005.
  33. [42]Sipei Zhao, Eva Cheng, Xiaojun Qiu, Ian Burnett, and Jacob Chia chun Liu. Pressure spectra in turbulent flows in the inertial and the dissipation ranges. The Journal of the Acoustical Society of America, 140:4178, 2016. doi: 10.1121/1.4968881.DOI

Paper details

vlnc:W_2bhfs2dy
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents