betaFeedback

WorksPhysics and Astronomy

A GPU-accelerated simulation of rapid intensification of a tropical cyclone with observed heating

Abstract

This paper presents a limited-area atmospheric simulation of a tropical cyclone accelerated using GPUs. The OpenACC directive-based programming model is used to port the atmospheric model to the GPU. The GPU implementation of the main functions and kernels is discussed. The GPU-accelerated code produces high-fidelity simulations of a realistic tropical cyclone forced by observational latent heating. Performance tests show that the GPU-accelerated code yields energy-efficient simulations and scales well in both the strong and weak limit.

Introduction

Tropical cyclones (TCs) are a natural hazard that significantly impact human life and economies. Despite their huge impact, TC evolution is a physical process that is poorly understood currently. In particular, the prediction of TC intensity has been a challenging problem because of complexities in the multiscale interactions of convective-scale processes and synoptic-scale environment [4]. This is true especially when TCs undergo rapid intensification (RI), where TCs drastically intensify in a short period of time – typically change in wind speed of around 30 knots within 24 hours. To explore and study the RI mechanisms, we perform high-fidelity TC simulations. Modeling and numerical simulations of TCs can enhance our understanding of the underlying geophysical processes and may improve the accuracy of forecasts and risk assessment. However, one of the main challenges in the modeling of TC evolution is the high computational cost to resolve the fine-scale features needed for accurate prediction of TC intensity. To achieve computationally efficient simulations, we port the limited-area atmospheric model to the graphics processing unit (GPU) using a multi-CPU multi-GPU approach.

A nonhydrostatic atmospheric model is used to simulate TC evolutions. The code we use to perform these simulations is called xNUMA, a new version of the NUMA (Nonhydrostatic Unified Model of the Atmosphere) code [14, 8]. The xNUMA code is designed specifically for a multiscale modeling framework (MMF) for atmospheric flows, and is written in modern Fortran using an object-oriented approach. Because all the spatial discretization and time-integration methods are encapsulated in the form of objects, it is capable of instantiating multiple simulations simultaneously during execution.

Previously, the NUMA nonhydrostatic atmospheric model, which employs element-based Galerkin methods, was ported to GPUs [1 b] using OCCA [17]. Various time integration methods, including explicit and semi-implicit schemes, were ported and demonstrated good scalability for numerical weather problems. However, using a hardware-agnostic application programming interface (API) like OCCA increases development complexity. It requires writing an entirely separate parallel branch of code in different languages, as well as modifying data structures and kernel execution patterns. In this work, we explore a hardware-specific directive-based approach to simplify offloading while maintaining performance.

In porting xNUMA to GPUs, we chose to approach the problem using a directive based implementation. The main advantage of directive-based approaches is the ability to easily make large GPU coding contributions to the code base while still maintaining a correct and performant CPU code base. In this work, we ported xNUMA to GPUs using the OpenACC programming standard [21] as implemented in the NVIDIA compiler [20].

High-order Galerkin methods are attractive for hurricane research due to their low numerical dissipation. An earlier study [10] reveals that high-order methods better capture the vortex response to asymmetric thermal perturbations in hurricane RI due to reduced damping of heat and kinetic energies. Recently, an adaptive mesh refinement technique has been incorporated into the high-order Galerkin method to efficiently capture the fine features of hurricanes (Tissauoi et al. 2024).

High-order Galerkin methods have been shown to be performant on GPUs and well-suited for modern

Corresponding author: Soonpil Kang

Email: [email protected] high-performance computing architectures. The Center for Efficient Exascale Discretizations (CEED) project documented applications of these methods in Kolev et al. [16]. The finite element solver MFEM has been ported to GPUs (Vargas et al. [26]; Andrej et al. [3]) by integrating the RAJA abstraction layer, which facilitates access to various backend programming models for parallelization of loop kernels. The spectral element Navier-Stokes solver Nek5000 has been ported to GPUs using OpenACC (Otero et al. [22]) and OCCA (Fischer et al. [5]). These works commonly adopted a matrix-free approach for GPU acceleration and reported performance improvements as the polynomial order of the basis functions increased.

The remainder of the paper is organized as follows. Section 2 describes the nonhydrostatic atmospheric model. In this section, the governing equations, spatial, and temporal discretizations are summarized. Section 3 presents the porting and optimization of the code for single and multiple GPU systems. Section 4 presents the numerical setup for tropical cyclone simulations. Section 5 shows the performance test results. The conclusions are drawn in Section 6.

Nonhydrostatic atmospheric model

Governing equations

Let us consider a fixed spatial domain Ω\Omega and a time interval (0,tf](0,t_f]. The governing equations for the nonhydrostatic model of the atmosphere can be written as follows:

∂ρ′∂t+∇⋅((ρ0+ρ′)u)=0,(1)\frac{\partial\rho'}{\partial t}+\nabla\cdot\left((\rho_0+\rho')u\right)=0, \tag*{(1)}
∂u∂t+u⋅∇u+1ρ0+ρ′∇p′+ρ′ρ0+ρ′gk=Hu,(2)\frac{\partial u}{\partial t}+u\cdot\nabla u+\frac{1}{\rho_0+\rho'}\nabla p'+\frac{\rho'}{\rho_0+\rho'}gk=\mathbf{H}_u, \tag*{(2)}
∂θ′∂t+u⋅∇(θ0+θ′)=Sθ+Hθ,(3)\frac{\partial\theta'}{\partial t}+u\cdot\nabla(\theta_0+\theta')=S_\theta+H_\theta, \tag*{(3)}

where ρ0\rho_0 is the reference density, ρ′\rho' is the density perturbation, uu is the velocity vector, θ0\theta_0 is the reference potential temperature, θ′\theta' represents the potential temperature perturbation, pp is the pressure, gg is the gravitational acceleration, kk is the upward-pointing unit vector, SθS_\theta is the heat source, and Hq\mathbf{H}_q is the hyper-diffusion operator for the variable qq, which we define in Section 2.4. Equations (1a)–(1c) represent local mass conservation, momentum balance, and thermodynamics, respectively. We assume that the reference fields are in hydrostatic balance and dependent only on the vertical coordinate zz, i.e., dp0dz=−ρ0g\frac{dp_0}{dz}=-\rho_0g.

The prognostic variables for the system (1) are q=(ρ′,u,θ′)Tq=(\rho',u,\theta')^T, where the superscript TT denotes the transpose operator. Eq. (1) can be rewritten more concisely in the compact form as

∂q∂t=S(q),(4)\frac{\partial q}{\partial t}=S(q), \tag*{(4)}

where S(q)S(q) contains all terms in the equations apart from the time derivatives.

The pressure of dry air is calculated using the thermodynamic equation of state,

p=pa(ρRθpa)γ(5)p=p_a\left(\frac{\rho R\theta}{p_a}\right)^\gamma \tag*{(5)}

where pap_a is the atmospheric pressure at the ground, RR is the specific gas constant of dry air, and γ=cp/cv\gamma=c_p/c_v with the specific heat at constant pressure cpc_p and constant volume cvc_v.

Element-based Galerkin method

The element-based Galerkin method decomposes the spatial domain Ω⊂Rd\Omega\subset\mathbb{R}^d (d=2d=2 or 3) into NeN_e disjoint elements Ωe\Omega_e such as Ω=⋃e=1NeΩe\Omega=\bigcup_{e=1}^{N_e}\Omega_e. In our model, the Ωe\Omega_e are either quadrilaterals (in 2D) or hexahedra (in 3D). Within each element Ωe\Omega_e, the prognostic vector qq is approximated as a finite-dimensional projection qNq_N using basis functions ψj(x)\psi_j(\boldsymbol{x}) and element-wise nodal coefficients qj(e)(t)q_j^{(e)}(t) such that

qN(x,t)=∑j=1MNψj(x)qj(e)(t),(6)q_N(\boldsymbol{x},t)=\sum_{j=1}^{M_N}\psi_j(\boldsymbol{x})q_j^{(e)}(t), \tag*{(6)}

where NN denotes the order of the basis functions, MN=(N+1)dM_N=(N+1)^d is the number of nodes per element, and the superscript (e)(e) denotes element-wise quantities. The basis functions are constructed as tensor products of NN-th order Lagrange polynomials hαh_\alpha associated with the Legendre-Gauss-Lobatto (LGL) points, given by

ψj(ξ,η,ζ)=hα(ξ)⊗hβ(η)⊗hγ(ζ),(7)\psi_j(\xi,\eta,\zeta)=h_\alpha(\xi)\otimes h_\beta(\eta)\otimes h_\gamma(\zeta), \tag*{(7)}

where α,β,γ∈{1,…,N+1}\alpha,\beta,\gamma\in\{1,\ldots,N+1\}, and (ξ,η,ζ)(\xi,\eta,\zeta) are the element coordinates in the reference domain E=[−1,1]dE=[-1,1]^d that are mapped from the physical coordinate x\boldsymbol{x} via metric terms.

For a continuous Galerkin (CG) method, let us consider a finite-dimensional Sobolev space VN\mathcal{V}_N defined as

VN={ψ∣ψ∈H1(Ω) and ψ∈PN(Ωe)},(8)\mathcal{V}_N=\{\psi\mid\psi\in H^1(\Omega)\text{ and }\psi\in\mathcal{P}_N(\Omega_e)\}, \tag*{(8)}

where PN(Ωe)\mathcal{P}_N(\Omega_e) is the set of all polynomials of degree less than or equal to NN on the element Ωe\Omega_e. The discrete weak form of the governing equation in (2) is obtained by multiplying it by a test function and integrating over the domain, which is stated as follows: find qN∈VNq_N\in\mathcal{V}_N such that for all ψ∈VN\psi\in\mathcal{V}_N,

∑e=1Ne∫Ωeψi∂qN∂t dΩe=∑e=1Ne∫ΩeψiS(qN) dΩe.(9)\sum_{e=1}^{N_e}\int_{\Omega_e}\psi_i\frac{\partial q_N}{\partial t}\,d\Omega_e=\sum_{e=1}^{N_e}\int_{\Omega_e}\psi_iS(q_N)\,d\Omega_e. \tag*{(9)}

In 3D, the integral over the domain Ω\Omega is computed using a quadrature rule based on the LGL points as follows:

∫Ω(⋅) dΩ=∑e=1Ne∫Ωe(⋅) dΩe=∑e=1Ne∑i=1Nξ∑j=1Nη∑k=1Nζwijk(e)Jijk(e)(⋅),(10)\begin{aligned} \int_\Omega(\cdot)\,d\Omega=\sum_{e=1}^{N_e}\int_{\Omega_e}(\cdot)\,d\Omega_e \\ &=\sum_{e=1}^{N_e}\sum_{i=1}^{N_\xi}\sum_{j=1}^{N_\eta}\sum_{k=1}^{N_\zeta}w_{ijk}^{(e)}J_{ijk}^{(e)}(\cdot), \tag*{(10)} \end{aligned}

where NξN_\xi, NηN_\eta, and NζN_\zeta are the orders of the polynomial basis functions in each direction, and wijk(e)w_{ijk}^{(e)} and Jijk(e)J_{ijk}^{(e)} denote the weight and Jacobian determinant associated with the LGL points, respectively.

The discrete weak form in (7) yields a matrix-vector form written as

MIJdqJdt=RI(qN),(11)M_{IJ}\frac{dq_J}{dt}=R_I(q_N), \tag*{(11)}

where I,J∈{1,⋯ ,Np}I,J \in\{1,\cdots,N_p\} and NpN_p is the total number of global grid points, and the standard summation convention is assumed unless otherwise specified. The global mass matrix MIJM_{IJ} and the right-hand side (RHS) vector RIR_I are constructed by applying the global assembly or direct stiffness summation (DSS) operator ⋀e=1Ne\bigwedge_{e=1}^{N_e} to the element-wise mass matrices and RHS vectors as follows:

MIJ=⋀e=1Ne∫Ωeψi(x)ψj(x) dΩe,(12)M_{IJ}=\bigwedge_{e=1}^{N_e}\int_{\Omega_e}\psi_i(\boldsymbol{x})\psi_j(\boldsymbol{x})\,\mathrm{d}\Omega_e, \tag*{(12)}
RI(qN)=⋀e=1Ne∫Ωeψi(x)S(qN) dΩe,(13)R_I(q_N)=\bigwedge_{e=1}^{N_e}\int_{\Omega_e}\psi_i(\boldsymbol{x})S(q_N)\,\mathrm{d}\Omega_e, \tag*{(13)}

where the DSS operator performs summation through mapping from local indices to global indices as (i,e)→I(i,e)\to I and (j,e)→J(j,e)\to J. We choose the integration points to be co-located with the interpolation points (inexact integration), which yields accurate numerical integration for N≥4N\geq4 [6]. This choice makes the mass matrix diagonal, i.e., MIM_I, due to the cardinal property of the basis functions, simplifying inversion. Consequently, we obtain

dqIdt=MI−1RI(qN).(14)\frac{\mathrm{d}q_I}{\mathrm{d}t}=M_I^{-1}R_I(q_N). \tag*{(14)}

Eq. (14) is used to calculate the numerical solutions in the simulations.

Implicit-explicit (IMEX) time integration

We integrate the discretized equation (14) in time using an implicit-explicit (IMEX) time integrator [8], which splits the operator S(q)S(q) in the governing equations as follows:

∂q∂t=[S(q)−L(q)]+L(q).(15)\frac{\partial q}{\partial t}=[S(q)-\mathcal{L}(q)]+\mathcal{L}(q). \tag*{(15)}

The terms in the square bracket account for slower advective waves and are discretized explicitly. The remaining terms in the linear operator L(q)\mathcal{L}(q) represent fast waves, such as acoustic and gravity waves, and are discretized implicitly. This IMEX strategy enables the use of larger time steps by eliminating the tight CFL restriction due to the fast waves. In this work, we employ the second-order additive Runge-Kutta (ARK2) scheme proposed in [8], which belongs to a family of linear multistage schemes. This time integrator is also used in [7] referred to as ARK(2,3,2)b.

We solve the linear system that arises from the implicit part of the IMEX formulation using the iterative Krylov subspace GMRES [23] in a fully matrix-free fashion. In this approach, the linearized operator is computed on the fly with no information precomputed or stored. The only memory required is to store the Krylov vectors, which is typically fewer than 20 in the simulations presented.

The complexity of the algorithm is analyzed in our earlier work. Detailed information can be found in our previous paper [13].

Tensor-based Hyper-Diffusion

We introduce numerical hyper-diffusion by adding the following term to the right-hand side of the continuous governing equations

Hq=(−1)α+1(∇⋅τ∇)αq,(16)\mathbf{H}q=(-1)^{\alpha+1}(\nabla\cdot\boldsymbol{\tau}\nabla)^\alpha q, \tag*{(16)}

where qq is the state variable, τ\boldsymbol{\tau} the viscosity tensor, and α\alpha is the order. In this work, we employed α=2\alpha=2, which yields fourth order hyper-diffusion. In this formulation, we adopt the tensor form of viscosity proposed in [9], but modify the original formulation by multiplying the viscosity tensor τ\boldsymbol{\tau} by each application of the Laplacian operator. This modification benefits the memory storage of metric terms. The viscosity tensor τ\boldsymbol{\tau} is defined through the eigendecomposition as follows:

τ=JE(ν1λ1−1000ν2λ2−1000ν3λ3−1)(JE)T,(17)\boldsymbol{\tau}=\mathbf{J}\mathbf{E} \begin{pmatrix} \nu_1\lambda_1^{-1} & 0 & 0\\ 0 & \nu_2\lambda_2^{-1} & 0\\ 0 & 0 & \nu_3\lambda_3^{-1} \end{pmatrix} (\mathbf{J}\mathbf{E})^{\mathsf{T}}, \tag*{(17)}

where J=∂x∂ξ\mathbf{J}=\frac{\partial\boldsymbol{x}}{\partial\boldsymbol{\xi}} is the Jacobian matrix of the isoparametric mapping, (ν1,ν2,ν3)(\nu_1,\nu_2,\nu_3) are the viscosity parameters that act in the principal axes of the metric tensor G=J−1J−T\mathbf{G}=\mathbf{J}^{-1}\mathbf{J}^{-\mathsf{T}}, and (λ1−1,λ2−1,λ3−1)(\lambda_1^{-1},\lambda_2^{-1},\lambda_3^{-1}) are the eigenvalues of G\mathbf{G} in ascending order (λ1>λ2>λ3)(\lambda_1>\lambda_2>\lambda_3). The matrix E\mathbf{E} contains the normalized eigenvectors (e1,e2,e3)(\boldsymbol{e}_1,\boldsymbol{e}_2,\boldsymbol{e}_3) of G\mathbf{G} in its columns, i.e., E=[e1∣e2∣e3]\mathbf{E}=[\boldsymbol{e}_1|\boldsymbol{e}_2|\boldsymbol{e}_3]. E\mathbf{E} is orthonormal because G\mathbf{G} is symmetric. The viscosity parameters νi\nu_i have physical dimensions L2/α/TαL^{2/\alpha}/T^\alpha, while kinematic (physical) hyperviscosity νp\nu_p has dimensions L2α/TL^{2\alpha}/T. The viscosity parameters can be scaled by the element size Δx\Delta x and time-step Δt\Delta t as follows

νi≡(ci)1/α(Δxi)2(Δt)1/α(i=1,…,3)(18)\nu_i\equiv(c_i)^{1/\alpha}\frac{(\Delta x_i)^2}{(\Delta t)^{1/\alpha}}\qquad(i=1,\ldots,3) \tag*{(18)}

where ci=(c1,c2,c3)c_i=(c_1,c_2,c_3) are the dimensionless coefficients that independently control the amount of numerical hyper-diffusion in the principal directions. Further details on our hyper-diffusion model are provided in our paper [7].

GPU implementation

Base code

The xNUMA code is written in modern Fortran and the base CPU code is parallelized using a Message Passing Interface (MPI) implementation. The simulator utilizes the dynamical core of the nonhydrostatic unified model of the atmosphere, referred to as NUMA [14, 8]. Each simulator object contains objects for input, spatial discretization, time integration, and solvers. Consequently, all the data and functions that constitute a simulation are members of the simulator objects.

The simulator is implemented via distributed memory parallelism using MPI. We subdivide the computational domain for limited-area simulations into multiple subdomains of equal size using non-overlapping element-based partitioning. In this configuration, each element is exclusively assigned to a single partition, and each partition is assigned to an MPI process (or core). As a result, workload is well balanced among MPI processes. Figure 1 illustrates three types of layouts for domain decomposition. We chose to keep the grid columns within a partition for a potential use of column-based microphysics model. The inter-process communications are required for the global DSS operation over the partitions. The partitioning of a grid is similar to what we implemented for the standard model. Details about the computational stencil for the element-based Galerkin method and the global DSS process can be found in [14].

Layouts of domain decomposition for limited-area models

Figure 1. Layouts of domain decomposition for limited-area models. (Nproc,xN_{\mathrm{proc},x} and Nproc,yN_{\mathrm{proc},y} are the number of processors in the xx and yy directions.)

OpenACC implementation

In this work, we ported the time integration task, a main component of the simulation, to GPUs using OpenACC directives. Figure 2 illustrates the simulation workflow on the CPU and GPU. Initially, the simulation is set up on the CPUs by reading input parameters, creating the grid, and computing the initial conditions. Subsequently, all data is transferred to GPUs using OpenACC directives to copy data from the CPU to the GPU. On the GPUs, the simulation advances in time by calculating the RHS vector, performing GMRES iterations, and updating the state vector. To output XML outputs and diagnostics, we transfer the GPU value for the state vector to the CPU, again using OpenACC directives. For this work, we have included all data management directives explicitly rather than using NVIDIA compiler implemented unified or managed memory approaches. These latter memory approaches can simplify porting, but can require either code refactoring (managed memory) or bleeding edge software/hardware (unified memory) to function properly. Explicit data management gives us minimal code change requirements while providing the ability to maximize performance.

Simulation workflow on the CPU and GPU

Figure 2. Simulation workflow on the CPU and GPU.

For communication across GPUs, we utilize OpenACC interoperability features that enable MPI communication directly between GPUs. The host_data construct makes GPU data addresses accessible on the CPU, and arrays listed in the use_device clause can be passed to CPU functions. If the hardware does not support CUDA-aware MPI or GPUDirect, standard network communication methods, such as Transmission Control Protocol (TCP), will be used.

RHS function

Algorithm 1 presents pseudocode for the function that computes the RHS vector, ported to the GPU using OpenACC directives. In the code, NeN_e is the number of elements, while NξN_{\xi}, NηN_{\eta}, and NζN_{\zeta} are the number of points within an element in each respective direction. NvarN_{\mathrm{var}} denotes the number of variables at each node. The function intma maps local nodal indices to global nodal indices. The RHS function consists of four kernels: global-to-local, gradient, divergence, and local-to-global kernels. Each kernel is parallelized using the parallel loop directive and optimized with gang and vector elements, which represent coarse-grain and fine-grain levels of parallelism, respectively. Vector parallelism operates within each gang in single instruction multiple threads (SIMT) lanes. The quadruple-nested loops over spectral elements and their nodal points are collapsed into a single loop using the collapse clause, which allows the compiler to maximize parallelism by mapping the nested loop to an unnested parallelizable loop with Ne×Nξ×Nη×NζN_e \times N_{\xi} \times N_{\eta} \times N_{\zeta} independent iterations. Despite also being a parallelizable loop, we found that allowing the innermost loop to be run sequentially maximized performance — possibly due to memory or register access constraints.

Gradient kernel

Algorithm 2 presents pseudocode for the function that computes the gradient fields of the prognostic variables. This function is the most computationally intensive part of the code and is executed when evaluating the right-hand side vector or the linearized operator. In the code, qξq_{\xi}, qηq_{\eta}, and qζq_{\zeta} represent the differentiation of the state vector with respect to the reference coordinates, ∂q/∂ξ\partial q/\partial\xi, ∂q/∂η\partial q/\partial\eta, and ∂q/∂ζ\partial q/\partial\zeta, respectively.

Assembly function

Algorithm 3 presents pseudocode for the function that performs the direct stiffness summation (DSS) to assemble the global RHS vector. This part of the code requires communication across ranks. The function consists of three major kernels. The first gathering kernels assemble the local RHS vector through overlapping nodes, and therefore requires avoiding race conditions among threads, which is achieved using the atomic update directive. Secondly, the values of the RHS vector at the grid interface are exchanged through MPI communications. The host_data directive enables the use of device data within the CPU code, MPI calls in this algorithm, by passing the device address of the data to the CPU.

Algorithm 1 RHS function.

function COMPUTE_RHS
  !$acc data present(q, q_e, f, rhs)

  !Kernel: Global to Local
  !$acc parallel loop gang vector collapse(4)
  for e = 1 : N_e, k = 1 : N_\zeta, j = 1 : N_\eta, i = 1 : N_\xi do
    I = intma(i, j, k, e)
    !$acc loop seq
    for m = 1 : N_var do
      q_e(i, j, k, e, m) = q(m, I)
    end for
  end for

  !Kernel: Gradient
  Compute local gradient operator

  !Kernel: Divergence
  Compute local divergence operator

  !Kernel: Local to Global
  !$acc parallel loop gang vector collapse(4)
  for e = 1 : N_e, k = 1 : N_\zeta, j = 1 : N_\eta, i = 1 : N_\xi do
    !$acc loop seq
    for m = 1 : N_var do
      Evaluate f(\nabla q(i, j, k, e, m))
    end for
    I = intma(i, j, k, e)
    !$acc loop seq
    for m = 1 : N_var do
      rhs(m, I) = f(\nabla q(i, j, k, e, m))
    end for
  end for

  !$acc end data
end function

Algorithm 2 Gradient kernel.

function COMPUTE_GRADIENT
  !$acc data present(q_\xi, q_\eta, q_\zeta)

  !$acc parallel loop gang vector collapse(4)
  for e = 1 : N_e, k = 1 : N_\zeta, j = 1 : N_\eta, i = 1 : N_\xi do
    for m = 1 : N_var do
      hold1 = 0.0
      hold2 = 0.0
      hold3 = 0.0
      !$acc loop reduction(+ : hold1, hold2, hold3)
      for l = 1 : N_\xi do
        hold1 = hold1 + \frac{\partial \phi}{\partial \xi}(l, i) \times q(l, j, k, e, m)
        hold2 = hold2 + \frac{\partial \phi}{\partial \eta}(l, j) \times q(i, l, k, e, m)
        hold3 = hold3 + \frac{\partial \phi}{\partial \zeta}(l, k) \times q(i, j, l, e, m)
      end for
      q_\xi(i, j, k, e, m) = hold1
      q_\eta(i, j, k, e, m) = hold2
      q_\zeta(i, j, k, e, m) = hold3
    end for
  end for

  !$acc end data
end function

Simulation setup

Numerical setup

We simulate a tropical cyclone case using the GPU-accelerated code. The computational domain is a 3D box with horizontal dimensions of 800 km and a vertical height of 20 km, defined as Ω=[−400,400]×[−400,400]×[0,20]\Omega= [-400, 400] \times[-400, 400] \times[0, 20] km. Doubly periodic boundary conditions are applied at the lateral boundaries, and a no-flux boundary condition is enforced at the bottom surface. To absorb upward gravity waves and prevent reflections, an implicit sponge layer [15, 13] with a thickness of 4 km is imposed at the top of the domain.

Algorithm 3 DSS function.

function PERFORM_DSS
  !$acc data present(rhscg, rhs)

  !Kernel: Gathering
  !$acc parallel loop independent collapse(4) private(ip_cg, ip)
  for e = 1 : N_e, k = 1 : N_\zeta, j = 1 : N_\eta, i = 1 : N_\xi do
    ip_cg = intma_cg(i, j, k, e)
    ip = intma(i, j, k, e)
    for m = 1 : N_var do
      !$acc atomic update
      rhscg(m, ip_cg) = rhscg(m, ip_cg) + rhs(m, ip)
    end for
  end for

  !Kernel: Communication
  for i = 1 : num_neighbors do
    Pack send_data
    !$acc host_data use_device(recv_data)
    MPI_Irecv(recv_data, num_send_recv, i_dest)
    !$acc end host_data

    !$acc host_data use_device(send_data)
    MPI_Isend(send_data, num_send_recv, i_dest)
    !$acc end host_data
    MPI_Waitall()
    Add recv_data to rhscg
  end for

  !Kernel: Scattering
  !$acc parallel loop independent collapse(4) private(ip_cg, ip)
  for e = 1 : N_e, k = 1 : N_\zeta, j = 1 : N_\eta, i = 1 : N_\xi do
    ip_cg = intma_cg(i, j, k, e)
    ip = intma(i, j, k, e)
    for m = 1 : N_var do
      rhs(m, ip) = rhscg(m, ip_cg)
    end for
  end for

  !$acc end data
end function

Several values for horizontal grid spacing are tested, with Δx∈{0.5,1,2,4}\Delta x \in\{0.5, 1, 2, 4\} km when using fifth-order basis functions (N=5N = 5), while the vertical spacing is fixed at Δz=167\Delta z = 167 m. For different orders of the basis function, the grid spacing is adjusted as close as possible to these values. The simulation runs for 6 hours with a time step size of Δt=0.5\Delta t = 0.5 seconds.

The values for the reference variables are taken from the background sounding provided in Jordan [12]. Figure 3 presents the profiles of reference potential temperature and reference pressure with respect to height. In the simulation, background wind is not considered.

Reference potential temperature and pressure profiles

Figure 3. Soundings for tropical cyclone simulations: reference potential temperature θ0\theta_0 and reference pressure p0p_0.

The tropical cyclone is initiated by an idealized vortex used in Guimond et al. [10]. This vortex is modified from the original form [18, 19] to enforce zero velocities at the lateral boundaries. In this vortex, the azimuthal-mean tangential velocity is defined as

u‾θ(r,z)=V(r)exp⁡(−zσσD1σ−1)exp⁡[−(rD2)6],(19)\overline{u}_{\theta}(r,z)=V(r)\exp\left(-\frac{z^{\sigma}}{\sigma D_{1}^{\sigma}-1}\right)\exp\left[-\left(\frac{r}{D_{2}}\right)^{6}\right], \tag*{(19)}

where VV is the surface tangential velocity, σ=2\sigma=2, D1=5823D_{1}=5823 m, and D2=200D_{2}=200 km. The surface tangential velocity is calculated by integrating a Gaussian distribution of vorticity with a peak of 1.5×10−3 s−11.5\times10^{-3}\ \mathrm{s}^{-1} at the vortex center, and a maximum wind speed of 21.5 m/s at a radius of 50 km.

The RI of tropical cyclones is driven by the vortex response to latent heating injected into the boundary layer. In this research, we incorporate the observational latent heating into the source term SθS_{\theta} in Eq. 1c. A time-varying 3D observational heating profile was obtained from airborne Doppler radar measurements during the RI of Hurricane Guillermo (1997) [11]. Figure 4 presents snapshots of the asymmetric spatial distribution of latent heating at different time points. Over the duration of 6 hours, 11 sampling points of latent heating are used to interpolate the heating source term.

Figure 4. Latent heating in the region [−60,60]×[−60,60]×[0,20][-60,60]\times[-60,60]\times[0,20] km.

Computing system

The performance of the GPU code for tropical cyclone simulations is evaluated on the Delta system, which was ranked No. 265 in the November 2024 TOP 500 list (top500.org 2024). Delta is an advanced computing resource maintained by the National Center for Supercomputing Applications (NCSA) at the University of Illinois, and supported by the National Science Foundation (NSF). While Delta provides access to both NVIDIA A40 GPUs and A100 GPUs, we only use A100 GPUs for performance tests in this study. Delta features 100 A100 GPU nodes, and each node is equipped with 64 AMD EPYC 7763 (Milan) CPUs and 4 A100 GPUs. The compute nodes are connected via a low-latency high-bandwidth HPE/Cray Slingshot interconnect.

The parallel code is compiled using NVIDIA HPC compilers (version 22.5), which include OpenMPI compiler wrappers. All performance tests in this study are conducted using double-precision calculations.

Results

Numerical results

Figure 5 and Figure 6 show the velocity magnitude and potential temperature perturbation, respectively, on the yy-zz plane at the center of the domain. In the initial state at t=0t=0 hours, a single idealized vortex defined by Eq. (19) is located at the center near the surface, with no thermal perturbation. By injecting thermal energy into the dynamical system, a strong updraft develops in the eyewall and leads to asymmetric and complex vortex structures. Figure 7 shows the vortical structures in the eye and spiral bands of the tropical cyclone at t=6t=6 hours.

Figure 5. Velocity magnitude at the middle yy-zz plane. (The domain is vertically stretched by a factor of 10 for visual purposes.)

Figure 6. Potential temperature perturbation at the middle yy-zz plane. (The domain is vertically stretched by a factor of 10 for visual purposes.)

Vortical structures in the eye and spiral bands of the tropical cyclone at $t=6$ hours

Figure 7. Vortical structures in the eye and spiral bands of the tropical cyclone at t=6t=6 hours. (Vortices are visualized using the QQ-criterion.)

Figure 8 shows the velocity magnitude on the horizontal plane at various heights. Uneven temperature distribution near the surface significantly contributes to the asymmetric structure of the tropical cyclone, where the extent of asymmetry varies with height.

Velocity magnitude at various heights: (a) $z = 8$ km, (b) $z = 10$ km, (c) $z = 12$ km, and (d) $z = 14$ km

Figure 8. Velocity magnitude at t=6t = 6 hours in the horizontal region [−20,20]×[−20,20][-20, 20] \times[-20, 20] km at various heights.

In order to quantify the magnitude of the vortex, we use the index of maximum tangential velocity and denote it as the RI velocity. In this work, the RI velocity is defined as the maximum of azimuthal-averaged tangential velocity as follows:

uRI=max⁡(u‾θ(r,z)),(20)u_{\mathrm{RI}}=\max\left(\overline{u}_{\theta}(r,z)\right), \tag*{(20)}

where

u‾θ(r,z)=12π∫02πuθ(r,θ,z) dθ(21)\overline{u}_{\theta}(r,z)=\frac{1}{2\pi}\int_{0}^{2\pi}u_{\theta}(r,\theta,z)\,\mathrm{d}\theta \tag*{(21)}
uθ=∥uθ∥(22)u_{\theta}=\lVert\mathbf{u}_{\theta}\rVert \tag*{(22)}
uθ=(I−er⊗er−ez⊗ez)u.(23)\mathbf{u}_{\theta}=(I-\mathbf{e}_{r}\otimes\mathbf{e}_{r}-\mathbf{e}_{z}\otimes\mathbf{e}_{z})\mathbf{u}. \tag*{(23)}

Here, er\mathbf{e}_{r} and ez\mathbf{e}_{z} are the unit vectors in the radial and vertical directions, respectively, which are calculated as er=r/∥r∥\mathbf{e}_{r}=\mathbf{r}/\lVert\mathbf{r}\rVert, r=(x−xc,y−yc,0)\mathbf{r}=(x-x_{c},y-y_{c},0), and ez=(0,0,1)\mathbf{e}_{z}=(0,0,1). The vertically varying centroid of the TC, xc=(xc,yc,z)\mathbf{x}_{c}=(x_{c},y_{c},z), is estimated as the weighted center of the velocity magnitude field:

xc=∫x∫y∥u∥x dxdy∫x∫y∥u∥ dxdy,(24)x_{c}=\frac{\int_{x}\int_{y}\lVert\mathbf{u}\rVert x\,\mathrm{d}x\mathrm{d}y}{\int_{x}\int_{y}\lVert\mathbf{u}\rVert\,\mathrm{d}x\mathrm{d}y}, \tag*{(24)}

where LxL_{x} and LyL_{y} are the lengths of the domain along the xx and yy directions, respectively.

We propose a simplified calculation of azimuthally averaged velocity, as given in Eq. (21). The azimuthally averaged velocity is approximated as the arithmetic mean of points that fall within a narrow circular strip at each radial distance r1r_1 and height z1z_1. This circular strip is defined as Ωstrip(r1,z1)={(r,θ,z)∣r1−ε2<r<r1+ε2,0≤θ≤2π,z=z1}\Omega_{\mathrm{strip}}(r_1,z_1)=\{(r,\theta,z)\mid r_1-\frac{\varepsilon}{2}<r<r_1+\frac{\varepsilon}{2},0\leq\theta\leq2\pi,z=z_1\}, where ε\varepsilon is the width of the strip. The averaged tangential velocity is evaluated as

uˉθ(r,z)≈1Np∑i=1Npuθ(ri,θi,zi), for (ri,θi,zi)∈Ωstrip(r,z),(25)\bar{u}_{\theta}(r,z)\approx\frac{1}{N_p}\sum_{i=1}^{N_p}u_{\theta}(r_i,\theta_i,z_i),\ \text{for }(r_i,\theta_i,z_i)\in\Omega_{\mathrm{strip}}(r,z), \tag*{(25)}

where NpN_p is the number of points that belong to the strip.

Figure 9 presents the time evolution of the RI velocity at various grid resolutions, compared with the maximum horizontal velocity. The RI velocity is calculated using 50 equi-width strips to average the tangential velocity. At all grid resolutions, the RI velocity exhibits a significant increase exceeding 20 m/s from its initial state. This is higher than the typical RI threshold of 13 m/s, which indicates that the occurrence of the RI event is effectively captured. The RI velocity produces a similar curve across all the resolutions and converges when the grid resolution is higher than Δx=2\Delta x=2 km, whereas, the maximum horizontal velocity shows larger variance between the profiles of different resolutions.

Maximum horizontal velocity and RI velocity at resolutions $\Delta x = \{0.5, 1, 2, 4\}$ km

Figure 9. Maximum horizontal velocity (umax⁡u_{\max}) and RI velocity (uRIu_{\mathrm{RI}}) at resolutions Δx={0.5,1,2,4}\Delta x = \{0.5, 1, 2, 4\} km.

Speedup and energy consumption

The execution time and energy consumption are compared between a CPU-based simulation using 128 AMD and 4 A100 processors. The test grid consists of 80×80×2480\times80\times24 fifth-order elements, which results in 165,888,000 degrees of freedom (DOF). Table 1 summarizes the comparison between the two simulations. The maximum power consumptions are assumed based on the hardware specifications.

CPUGPU
Max. Power Consumption (W)280 (per node)300 (per GPU)
Number of Nodes21
Number of Processors1284
Execution Time Per Step (seconds)5.5630.536
Energy Consumption Per Step (J)3115.3643.2

Table 1. Comparison of total energy consumption during 100 time steps between CPUs (AMD EPYC 7763) and GPUs (NVIDIA A100).

Execution times are measured for the time integration loop, i.e., after the simulation setup stage in Figure 2, over 100 time steps from the initial state. The CPU run employs the base code, originally optimized for CPUs. The GPU-accelerated code achieves approximately 10.4× speedup over the CPU code for the current configuration. This speedup is likely due to the higher FLOP capacity of the GPU and reduced communication overhead from the smaller number of processors.

The energy consumption per time step in the GPU simulation is approximately 4.84× lower than that of the CPU simulation. Although a single GPU consumes more power than a single CPU node, the total energy consumption is significantly reduced in the GPU run due to 10.4× speedup. This indicates that the GPU implementation offers greater energy efficiency compared to the purely CPU-based approach.

approach. To emphasize this point, Table 1 shows that 4 GPUs are 4.84× more efficient than 128 CPUs; therefore, we require 1284×4.84=154\frac{128}{4} \times4.84 = 154 CPUs to match a single GPU. To perform equivalent simulations that we performed on 256 GPUs would require nearly 40,000 CPUs (6× the CPU capacity of Delta).

Scalability

To evaluate the scalability of the GPU implementation, we measure the strong and weak scaling performance.

Figure 10 shows the strong scaling of the GPU implementation. The grid is composed of 160×160×24160 \times160 \times24 fifth-order elements, which results in a total of 663,552,000 DOF. Strong scalability tends to degrade as the number of GPUs increases. This can be attributed to higher GPU memory access latency, and its effects become relatively more significant as each rank holds a smaller portion of the problem. It is noteworthy that GPUs are the most performant when they are fully saturated.

Strong scaling of the GPU implementation

Figure 10. Strong scaling of the GPU implementation.

For weak scaling, the problem size is increased proportionally to the number of GPUs. To achieve this, the grid resolution is doubled while keeping the number of elements per GPU at 40×40×2440 \times40 \times24 (41,472,000 DOF), and the temporal resolution constant at Δt=0.5\Delta t = 0.5 seconds (the largest problem shown has 2.65 billion DOF). The order of basis function is N=5N = 5 in all direction. Figure 11 shows that the wall clock time per time step for the weak scaling test. The execution time per step converges as the number of GPUs increases. This increasing trend is due to more GMRES iterations being required for the implicit solve in larger problem sizes. This result highlights the high performance and scalability of the code.

Weak scaling of the GPU implementation

Figure 11. Weak scaling of the GPU implementation.

Kernel performance

We evaluate the performance of the main kernels to compute the RHS vector in Algorithm 1 using the roofline model. This model uses the metrics of double-precision floating point operations rates in GFLOPS/s, bandwidth in GB/s, and the arithmetic intensity (GFLOPS/GB). The measured peak performance and bandwidth of NVIDIA A100 GPUs in the test are 7,329 GFLOPS/s and 1,506 GB/s, respectively.

Figure 12 shows the GFLOP/s performance, arithmetic intensity, and roofline plots for various orders of the basis function. The gradient and divergence kernels are computationally intensive and achieve higher performance as the order of the basis function increases. The measured performance for the gradient kernel ranges from 9.5% to 12.3% of the peak performance, while the divergence kernel achieves between 4.1% and 6.2%. The lower performance of the divergence kernel compared to the gradient kernel is attributed to the governing equation set given in Eq. (1), which requires gradient operations for five variables (p′p^\prime, uu, θ\theta) whereas the divergence operation is applied only to the momentum, ρu\rho u. As a result, the divergence kernel involves fewer FLOP computations and becomes slightly more memory-intensive. Higher arithmetic intensity results in higher performance, which implies the potential of high-order methods to better exploit GPUs capabilities. In contrast, the performance of the global-to-local and local-to-global kernels remains independent of the basis function order, 7.1% and 3.2% of the peak performance, respectively, as they simply loop over the grid points and their associated DOF.

GFLOP/s performance, arithmetic intensity, and roofline model for four main kernels in the RHS function

Figure 12. GFLOP/s performance, arithmetic intensity, and roofline model for four main kernels in the RHS function.

Conclusions

In this work, we ported the nonhydrostatic atmospheric model xNUMA to GPUs using the OpenACC programming model. This approach facilitates simple maintenance of the code. The entire time-integration process, including implicit and explicit steps, is performed on the GPUs. A matrix-free implementation of the implicit solution is well-suited for GPU computations.

The model successfully simulates realistic tropical cyclones on GPUs and captures the RI process. The simplified calculation of azimuthally averaged velocity yields converged results of RI across various horizontal grid spacings.

The GPU-accelerated code yields an energy-efficient simulation, achieving a 4.84×4.84 \times improvement over the CPU version. The GPU implementation demonstrates both strong and weak scaling, as well as efficient kernel performance on NVIDIA A100 GPUs. We expect that this research will help accelerate high-fidelity simulations for numerical weather prediction.

Our future work includes the following:

  • Using the initial state obtained from observational data to produce more realistic tropical cyclones.

  • Adding a precipitation microphysics model to simulate moist dynamics of tropical cyclones.

  • Accelerating the multiscale modeling framework [13] using GPUs.

Acknowledgements

This work was supported by the Office of Naval Research under Grant No. N0001419WX00721. F. X. Giraldo was also supported by the National Science Foundation under grant AGS-1835881. This work was performed when Soonpil Kang held a National Academy of Sciences’ National Research Council Fellowship at the Naval Postgraduate School. Part of Soonpil Kang’s work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. LLNL-JRNL-2004409. This work used Delta at the National Center for Supercomputing Applications (NCSA) through allocation MTH240030 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation (NSF).

References

  1. [1]Abdi DS, Giraldo FX, Constantinescu EM, Carr LE, Wilcox LC and Warburton TC (2019a) Acceleration of the Implicit–Explicit nonhydrostatic unified model of the atmosphere on manycore processors. The International Journal of High Performance Computing Applications 33(2): 242–267.
  2. [3]Andrej J, Atallah N, Bäcker JP, Camier JS, Copeland D, Dobrev V, Dudouit Y, Duswald T, Keith B, Kim D et al. (2024) High-performance finite elements with MFEM. The International Journal of High Performance Computing Applications : 1094320241261981.DOI
  3. [4]Fischer MS, Tang BH and Corbosiero KL (2019) A climatological analysis of tropical cyclone rapid intensification in environments of upper-tropospheric troughs. Monthly Weather Review 147(10): 3693–3719.DOI
  4. [5]Fischer P, Kerkemeier S, Min M, Lan YH, Phillips M, Rathnayake T, Merzari E, Emboulides A, Karakus A, Chalmers N et al. (2022) Nekrs, a gpu-accelerated spectral element navier–stokes solver. Parallel Computing 114: 102982.arxiv.org/abs/2104.05829
  5. [6]Giraldo FX (1998) The Lagrange–Galerkin spectral element method on unstructured quadrilateral grids. Journal of Computational Physics 147(1): 114–146.DOI
  6. [7]Giraldo FX, de Bragança Alves FA, Kelly JF, Kang S and Reinecke PA (2024) A performance study of horizontally explicit vertically implicit (HEVI) time-integrators for non-hydrostatic atmospheric models. Journal of Computational Physics : 113275.
  7. [8]Giraldo FX, Kelly JF and Constantinescu EM (2013) Implicit-explicit formulations of a three-dimensional nonhydrostatic unified model of the atmosphere (NUMA). SIAM Journal on Scientific Computing 35(5): B1162–B1194.DOI
  8. [9]Guba O, Taylor M, Ullrich P, Overfelt J and Levy M (2014) The spectral element method (SEM) on variable-resolution grids: Evaluating grid sensitivity and resolution-aware numerical viscosity. Geoscientific Model Development 7(6): 2803–2816.DOI
  9. [10]Guimond SR, Reisner JM, Marras S and Giraldo FX (2016) The impacts of dry dynamic cores on asymmetric hurricane intensification. Journal of the Atmospheric Sciences 73(12): 4661–4684.DOI
  10. [11]Hasan MB, Guimond SR, Yu M, Reddy S and Giraldo FX (2022) The effects of numerical dissipation on hurricane rapid intensification with observational heating. Journal of Advances in Modeling Earth Systems 14(8): e2021MS002897.
  11. [12]Jordan CL (1958) Mean soundings for the west indies area. Journal of Atmospheric Sciences 15(1): 91–97.doi.org/10.1175/1520-0469(1958)015<0091:msftwi>2.0.co;2
  12. [13]Kang S, Kelly JF, Austin AP and Giraldo FX (2024) Multiscale modeling framework using element-based galerkin methods for moist atmospheric limited-area simulations. arXiv preprint arXiv:2407.05927.arxiv.org/abs/2407.05927
  13. [14]Kelly JF and Giraldo FX (2012) Continuous and discontinuous Galerkin methods for a scalable three-dimensional nonhydrostatic atmospheric model: Limited-area mode. Journal of Computational Physics 231(24): 7988–8008.DOI
  14. [15]Klemp J, Dudhia J and Hassiotis A (2008) An upper gravity-wave absorbing layer for NWP applications. Monthly Weather Review 136(10): 3987–4004.
  15. [16]Kolev T, Fischer P, Min M, Dongarra J, Brown J, Dobrev V, Warburton T, Tomov S, Shephard MS, Abdelfattah A et al. (2021) Efficient exascale discretizations: High-order finite element methods. The International Journal of High Performance Computing Applications 35(6): 527–552.
  16. [17]Medina DS, St-Cyr A and Warburton T (2014) Occa: A unified approach to multi-threading languages. arXiv preprint arXiv:1403.0968.arxiv.org/abs/1403.0968
  17. [18]Nolan DS and Grasso LD (2003) Nonhydrostatic, three-dimensional perturbations to balanced, hurricane-like vortices. part II: Symmetric response and nonlinear simulations. Journal of the Atmospheric sciences 60(22): 2717–2745.doi.org/10.1175/1520-0469(2003)060<2717:ntptbh>2.0.co;2
  18. [19]Nolan DS, Moon Y and Stern DP (2007) Tropical cyclone intensification from asymmetric convection: Energetics and efficiency. Journal of the Atmospheric Sciences 64(10): 3377–3405.DOI
  19. [20]NVIDIA Corporation (2025) NVIDIA HPC Compilers Reference Guide. URL https://docs.nvidia.com/hpc-sdk/compilers/hpc-compilers-ref-guide/index.html.
  20. [21]OpenACC Organization (2020) The openacc application programming interface, version 3.1. Available at https://www.openacc.org.
  21. [22]Otero E, Gong J, Min M, Fischer P, Schlatter P and Laure E (2019) Openacc acceleration for the pn–pn–2 algorithm in nek5000. Journal of Parallel and Distributed Computing 132: 69–78.DOI
  22. [23]Saad Y and Schultz MH (1986) Gmres: A generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM Journal on scientific and statistical computing 7(3): 856–869.DOI
  23. [26]Vargas A, Stitt TM, Weiss K, Tomov VZ, Camier JS, Kolev T and Rieben RN (2022) Matrix-free approaches for GPU acceleration of a high-order finite element hydrodynamics application using MFEM, Umpire, and RAJA. The International Journal of High Performance Computing Applications 36(4): 492–509.arxiv.org/abs/2112.07075

Paper details

vlnc:W_ahnkvmjz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents