betaFeedback

WorksPhysics and Astronomy

Implicit-Explicit Methods for the GX Gyrokinetics Code

Abstract

Gyrokinetic simulations are crucial for understanding turbulent transport in magnetic confinement fusion devices. Unfortunately, these simulations are typically extremely expensive, particularly when kinetically evolving electrons due to their significantly greater thermal speed compared to that of ions. In this work, we present new implicit-explicit timestepping methods for the GX gyrokinetics code to accelerate these simulations where the ion and electron streaming terms as well as the electron mirror terms are treated implicitly, while the remaining terms are treated explicitly. These methods include a lower-order Lie-Trotter splitting method as well as a higher-order additive Runge-Kutta method. Our schemes take advantage of the sparsity of the GX equations, which allows us to efficiently invert the linear operators that arise from the implicit scheme without consuming significant amounts of limited GPU memory. We demonstrate our new integration scheme for both linear (electrostatic and electromagnetic) and nonlinear (electrostatic) simulations in general toroidal geometry, and show that our methods can achieve up to the maximum theoretical linearly stable timestep for these simulations while maintaining a significant improvement in wall-clock time.

Introduction

Tokamaks and stellarators are two of the leading nuclear fusion concepts, and use strong magnetic fields to confine the fusion plasma. Assuming that the plasma is macroscopically stable, tokamaks and stellarators still leak energy and particles due to various transport processes [24, 25]. The largest contributor to these losses is often turbulent transport. Driven by steep temperature and density gradients, turbulent fluctuations rapidly mix the hot and cold regions of the plasma, preventing the plasma from maintaining higher pressures and limiting the overall fusion power gain [31, 40, 15].

One of the most widely-used models for simulating magnetized turbulence is gyrokinetics [3, 17, 2, 11]. Numerous gyrokinetic codes [30, 16, 26, 10, 6, 32] have been developed in the past few decades that can accurately match experimental measurements of turbulent losses as well as temperature and density profiles. Unfortunately, running these gyrokinetic codes can be computationally expensive, often requiring hundreds to thousands of CPU or GPU-hours. Including kinetic electrons exacerbates this cost, as their higher thermal speed (about 60 times greater than a deuterium ion of the same temperature) enforces a stricter Courant–Friedrichs–Lewy (CFL) condition that reduces the maximum allowed timestep. To avoid this issue, an adiabatic electron response is often used, particularly when studying the ion-temperature gradient mode (Coppi et al. 1967; Cowley et al. 1991). However, correctly resolving the electron dynamics is crucial for understanding other important microinstabilities like the trapped-electron mode [37], the kinetic-ballooning mode (Tang et al. 1980), and the microtearing mode (Hazeltine et al. 1975; Guttenfelder et al. 2011).

To overcome this computational restriction, codes like GS2 (Kotschenreuther et al. 1995a; Dorland et al. 2000) and stella (Barnes et al. 2019) have employed implicit or implicit-explicit (IMEX) time-integration methods. GS2 is fully linearly implicit, while stella treats the stiff electron terms implicitly. Here, linearly implicit means that only the linear part of the gyrokinetic equation is treated implicitly, while the nonlinear term appears explicitly and can be evaluated separately through operator splitting. Among the linear terms, only a subset of them are stiff due to the higher electron thermal speed. Alternatively, in CGYRO (Candy et al. 2016), one of the velocity coordinates is truncated, relaxing the CFL constraint. In this paper, we focus on the GX gyrokinetic code (Mandell et al. 2018, 2024). GX is a GPU-native code that solves the radially-local δf\delta f form of gyrokinetics by expanding the distribution function in a Hermite-Laguerre basis in velocity space, allowing for coarser velocity resolution. For our scheme, we treat implicitly the terms related to parallel streaming along field lines for both ions and electrons, as well as the term due to acceleration due to the mirror force for electrons.

Although being GPU-native substantially improves performance, the limited memory on GPUs can be very constraining for implicit methods. Inverting the linear operators that arise in implicit methods often requires significant storage. Indeed, as we explain in § 4.1, the memory cost of the implicit methods implemented in GS2 and stella can make them nearly infeasible to use for the GPU-native GX. However, the pseudo-spectral formulation used in GX yields equations that are sparse in velocity-space. We take advantage of this sparsity to present new algorithms to efficiently invert the implicit operators. We also employ physics-based preconditioners and iterative methods to further reduce the memory cost. We note that a similar Hermite–Laguerre code, GKX, has been recently developed that also utilizes IMEX timestepping [27], but it inverts the implicit operator exactly, using a direct factorization that is significantly more memory expensive than our low-memory approximate inversions.

The paper is organized as follows: In § 2, we introduce the GX code, and the resulting equations that need to be solved. Section 3 describes the time-integration schemes that we use in this work. Our methods to invert the implicit operators are described in § 4, and simulation examples are shown in § 5. The conclusions follow in § 6.

The GX Gyrokinetics Code

Our notation follows that of Mandell et al. (2018, 2024). GX solves the radially local δf\delta f formulation of gyrokinetics. In this approximation, the distribution function for species ss is split as Fs=F0s+δfsF_s = F_{0s} + \delta f_s, where δfs\delta f_s is a perturbation to an equilibrium distribution function F0sF_{0s} that is assumed to be Maxwellian:

F0s=FMs=ns(2πvth,s2)3/2exp⁡(−v∥2/2+μBvth,s2)(1)F_{0s}=F_{Ms}=\frac{n_s}{\left(2\pi v_{\mathrm{th},s}^{2}\right)^{3/2}}\exp\left(-\frac{v_{\parallel}^{2}/2+\mu B}{v_{\mathrm{th},s}^{2}}\right) \tag*{(1)}

where nsn_s is the equilibrium density of species ss, the thermal speed is vth,s=Ts/Msv_{\mathrm{th},s}=\sqrt{T_s/M_s}, and TsT_s and MsM_s are the equilibrium temperature and mass of species ss, respectively. GX uses as velocity-space coordinates the velocity parallel to the background magnetic field v∥v_{\parallel} and μB\mu B. The magnetic moment μ\mu is μ=v⊥2/2B\mu=v_{\perp}^{2}/2B, where v⊥v_{\perp} is the speed perpendicular to the magnetic field, and BB is the magnitude of the local magnetic field. Such velocity-space coordinates are useful for complex stellarator geometries that may have several magnetic wells along a field line. The perturbation is further split into non-Boltzmann and Boltzmann parts

δfs=hs−qsΦTsFMs=gs+qs⟨χ⟩RTsFMs−qsΦTsFMs,(2)\delta f_s=h_s-\frac{q_s\Phi}{T_s}F_{Ms}=g_s+\frac{q_s\langle\chi\rangle_R}{T_s}F_{Ms}-\frac{q_s\Phi}{T_s}F_{Ms}, \tag*{(2)}

where qsq_s is the charge of species ss, χ=Φ−v⋅A/c\chi=\Phi-\mathbf{v}\cdot\mathbf{A}/c with cc the speed of light (both GX and this paper are written in Gaussian units), and Φ\Phi and A\mathbf{A} are the electrostatic and magnetic vector potential, respectively.

The non-Boltzmann part hsh_s and the auxiliary distribution function gs=hs−qs⟨χ⟩RFMs/Tsg_s=h_s-q_s\langle\chi\rangle_R F_{Ms}/T_s satisfy the gyrokinetic equation

∂gs∂t+[v∥b^+⟨vχ⟩R+vds]⋅∇hs−μ(b^⋅∇B)∂hs∂v∥=−⟨vχ⟩R⋅∇ ⁣∂∂v∥FMs∣E+C(hs),(3)\frac{\partial g_s}{\partial t}+\left[v_{\parallel}\hat{\mathbf{b}}+\langle\mathbf{v}_{\chi}\rangle_R+\mathbf{v}_{ds}\right]\cdot\nabla h_s-\mu\left(\hat{\mathbf{b}}\cdot\nabla B\right)\frac{\partial h_s}{\partial v_{\parallel}} = -\langle\mathbf{v}_{\chi}\rangle_R\cdot\nabla\! \left.\vphantom{\frac{\partial}{\partial v_{\parallel}}}F_{Ms}\right|_E+C(h_s), \tag*{(3)}

where ⟨vχ⟩R=cb^×∇⟨χ⟩R/B\langle\mathbf{v}_{\chi}\rangle_R=c\hat{\mathbf{b}}\times\nabla\langle\chi\rangle_R/B is the gyroaveraged fluctuating drift velocity, vds=b^×(v∥2b^⋅∇b^+μ∇B)/Ωs\mathbf{v}_{ds}=\hat{\mathbf{b}}\times\left(v_{\parallel}^{2}\hat{\mathbf{b}}\cdot\nabla\hat{\mathbf{b}}+\mu\nabla B\right)/\Omega_s contains the magnetic curvature and ∇B\nabla B drifts, and Ωs=qsB/(Msc)\Omega_s=q_sB/(M_sc) is the cyclotron frequency. The collision operator is represented by C(hs)C(h_s).

The auxiliary distribution function gsg_s was used to eliminate the time derivative of the gyroaveraged fields. These fields are self-consistently determined using the following field equations

∑sqs2nsTsΦ=∑sqs∫d3v ⟨hs⟩r,(4)\sum_s\frac{q_s^2n_s}{T_s}\Phi=\sum_s q_s\int\mathrm{d}^3v\,\langle h_s\rangle_r, \tag*{(4)}
−∇⊥2A∥=4πc∑sqs∫d3v v∥⟨hs⟩r,(5)-\nabla_{\perp}^{2}A_{\parallel}=\frac{4\pi}{c}\sum_s q_s\int\mathrm{d}^3v\,v_{\parallel}\langle h_s\rangle_r, \tag*{(5)}

and

∇⊥2δB∥=−4πB∇⊥∇⊥:∑sMs∫d3v ⟨v⊥v⊥hs⟩r.(6)\nabla_{\perp}^{2}\delta B_{\parallel}=-\frac{4\pi}{B}\nabla_{\perp}\nabla_{\perp}:\sum_s M_s\int\mathrm{d}^3v\,\langle\mathbf{v}_{\perp}\mathbf{v}_{\perp}h_s\rangle_r. \tag*{(6)}

Once all quantities are normalized (see Appendix § 8 for normalizations), the non-dimensional gyrokinetic equation is

∂gs∂t+[vtsv∥b^+⟨vχ⟩R+TsZsvds]⋅∇hs−vtsμ(b^⋅∇B)∂hs∂v∥=−⟨vχ⟩R⋅∇ ⁣∂∂v∥FMs∣E+C(hs).(7)\frac{\partial g_s}{\partial t}+\left[v_{\mathrm{ts}}v_{\parallel}\hat{\mathbf{b}}+\langle\mathbf{v}_{\chi}\rangle_R+\frac{T_s}{Z_s}\mathbf{v}_{ds}\right]\cdot\nabla h_s-v_{\mathrm{ts}}\mu\left(\hat{\mathbf{b}}\cdot\nabla B\right)\frac{\partial h_s}{\partial v_{\parallel}} = -\langle\mathbf{v}_{\chi}\rangle_R\cdot\nabla\! \left.\vphantom{\frac{\partial}{\partial v_{\parallel}}}F_{Ms}\right|_E+C(h_s). \tag*{(7)}

and the non-dimensional field equations are

∑sZs2nsτsΦ=∑sZsns∫d3v ⟨hs⟩r,(8)\sum_s \frac{Z_s^2 n_s}{\tau_s}\Phi= \sum_s Z_s n_s \int\mathrm{d}^3v\,\langle h_s\rangle_r, \tag*{(8)}
−∇⊥2A∥=βref2∑sZsnsvts∫d3v v∥⟨hs⟩r,(9)-\nabla_\perp^2 A_\parallel= \frac{\beta_{\mathrm{ref}}}{2}\sum_s Z_s n_s v_{ts}\int\mathrm{d}^3v\,v_\parallel\langle h_s\rangle_r, \tag*{(9)}

and

∇⊥2δB∥=−βref2B∇⊥∇⊥:∑snsτs∫d3v ⟨v⊥v⊥hs⟩r,(10)\nabla_\perp^2\delta B_\parallel= -\frac{\beta_{\mathrm{ref}}}{2B}\nabla_\perp\nabla_\perp:\sum_s n_s\tau_s\int\mathrm{d}^3v\,\langle\mathbf{v}_\perp\mathbf{v}_\perp h_s\rangle_r, \tag*{(10)}

where βref≡8πnrefTref/BN2\beta_{\mathrm{ref}}\equiv8\pi n_{\mathrm{ref}}T_{\mathrm{ref}}/B_N^2 is the ratio of the plasma to magnetic pressure for the reference species and BNB_N is a reference magnetic field strength. These equations are solved in a radially-local flux-tube spatial domain that follows a single field line on a single magnetic flux surface.

In GX the distribution functions gsg_s and hsh_s are both expanded in a Fourier-Hermite-Laguerre series:

gs=∑k⊥∑ℓ∑m(−1)ℓ(2π)3m!Gℓ,ms(k⊥,z)Lℓ(μB)Hem(v∥)e−v∥2/2−μBeik⊥⋅R,(11)g_s=\sum_{\mathbf{k}_\perp}\sum_{\ell}\sum_m\frac{(-1)^\ell}{\sqrt{(2\pi)^3m!}}G^s_{\ell,m}(\mathbf{k}_\perp,z)L_\ell(\mu B)\mathrm{He}_m(v_\parallel)e^{-v_\parallel^2/2-\mu B}e^{i\mathbf{k}_\perp\cdot\mathbf{R}}, \tag*{(11)}
hs=∑k⊥∑ℓ∑m(−1)ℓ(2π)3m!Hℓ,ms(k⊥,z)Lℓ(μB)Hem(v∥)e−v∥2/2−μBeik⊥⋅R,(12)h_s=\sum_{\mathbf{k}_\perp}\sum_{\ell}\sum_m\frac{(-1)^\ell}{\sqrt{(2\pi)^3m!}}H^s_{\ell,m}(\mathbf{k}_\perp,z)L_\ell(\mu B)\mathrm{He}_m(v_\parallel)e^{-v_\parallel^2/2-\mu B}e^{i\mathbf{k}_\perp\cdot\mathbf{R}}, \tag*{(12)}

where the spectral coefficients Gℓ,msG^s_{\ell,m} and Hℓ,msH^s_{\ell,m} are related by

Gℓ,ms(k⊥,z)=Hℓ,ms−ZsτsJsΦδm0+ZsvtsτsJsA∥δm1−(Jℓs+Jℓ−1s)δB∥Bδm0.(13)G^s_{\ell,m}(\mathbf{k}_\perp,z)=H^s_{\ell,m}-\frac{Z_s}{\tau_s}\mathcal{J}^s\Phi\delta_{m0}+\frac{Z_sv_{ts}}{\tau_s}\mathcal{J}^s A_\parallel\delta_{m1}-(\mathcal{J}^s_\ell+\mathcal{J}^s_{\ell-1})\frac{\delta B_\parallel}{B}\delta_{m0}. \tag*{(13)}

A Fourier representation is used for the spatial dimensions perpendicular to the mean magnetic field (k⊥\mathbf{k}_\perp), while Hermite and Laguerre representations are used for the parallel (v∥v_\parallel) and perpendicular (μB\mu B) velocity domains. A discrete Fourier collocation grid is used for the spatial direction parallel to the field (zz) so that the parallel derivative ∇∥\nabla_\parallel can be evaluated spectrally. Substituting this expansion into the gyrokinetic equation and taking integrals over velocity space yields differential equations that describe the evolution of the spectral coefficients Gℓ,msG^s_{\ell,m} and Hℓ,msH^s_{\ell,m}:

∂Gℓ,ms∂t+Nℓ,ms+vts∇∥(m+1Hℓ,m+1s+mHℓ,m−1s)+vts[−(ℓ+1)m+1Hℓ,m+1s−ℓm+1Hℓ−1,m+1s+ℓmHℓ,m−1s+(ℓ+1)mHℓ+1,m−1s]∇∥ln⁡B+iτsZsωdκ[(m+1)(m+2)Hℓ,m+2s+(2m+1)Hℓ,ms+m(m−1)Hℓ,m−2s]+iτsZsωdB[(ℓ+1)Hℓ+1,ms+(2ℓ+1)Hℓ,ms+ℓHℓ−1,ms]=Dℓ,ms+Cℓ,ms.(14)\begin{aligned} \frac{\partial G^s_{\ell,m}}{\partial t} &+\mathcal{N}^s_{\ell,m} +v_{ts}\nabla_\parallel\left(\sqrt{m+1}H^s_{\ell,m+1}+\sqrt{m}H^s_{\ell,m-1}\right)\\ &+v_{ts}\left[-(\ell+1)\sqrt{m+1}H^s_{\ell,m+1}-\ell\sqrt{m+1}H^s_{\ell-1,m+1}\right.\\ &\qquad\left.+\ell\sqrt{m}H^s_{\ell,m-1}+(\ell+1)\sqrt{m}H^s_{\ell+1,m-1}\right]\nabla_\parallel\ln B\\ &+\mathrm{i}\frac{\tau_s}{Z_s}\omega_d^\kappa\left[\sqrt{(m+1)(m+2)}H^s_{\ell,m+2}+(2m+1)H^s_{\ell,m}\right.\\ &\qquad\left.+\sqrt{m(m-1)}H^s_{\ell,m-2}\right]\\ &+\mathrm{i}\frac{\tau_s}{Z_s}\omega_d^B\left[(\ell+1)H^s_{\ell+1,m}+(2\ell+1)H^s_{\ell,m}+\ell H^s_{\ell-1,m}\right]\\ &=\mathcal{D}^s_{\ell,m}+\mathcal{C}^s_{\ell,m}. \tag*{(14)} \end{aligned}

In this equation, Nℓ,ms\mathcal{N}^s_{\ell,m}, Dℓ,ms\mathcal{D}^s_{\ell,m}, and Cℓ,ms\mathcal{C}^s_{\ell,m} represent the nonlinear, drive, and collision terms, respectively. The terms coloured red and blue have a prefactor of vtsv_{ts}. For electrons, vtsv_{ts} is approximately 60 times larger than for (deuterium) ions, making these the stiff terms that we will treat implicitly. The red terms correspond physically to parallel streaming along the field line, while the blue terms correspond to the mirror force from the inhomogeneity of the magnetic field strength.

The field equations for Φ\Phi, A∥A_\parallel, and δB∥\delta B_\parallel are

∑sZs2nsτs(1−∑ℓ=0Nℓ(Jℓs)2)Φ−∑sZsns∑ℓ=0NℓJℓs(Jℓs+Jℓ−1s)δB∥B=∑sZsns∑ℓ=0NℓJℓsGℓ,0s(15)\sum_s \frac{Z_s^2 n_s}{\tau_s}\left(1-\sum_{\ell=0}^{N_\ell}(\mathcal{J}_\ell^s)^2\right)\Phi-\sum_s Z_s n_s\sum_{\ell=0}^{N_\ell}\mathcal{J}_\ell^s(\mathcal{J}_\ell^s+\mathcal{J}_{\ell-1}^s)\frac{\delta B_\parallel}{B} = \sum_s Z_s n_s\sum_{\ell=0}^{N_\ell}\mathcal{J}_\ell^s G_{\ell,0}^s \tag*{(15)}
(k⊥2+βref2∑sZs2nsms∑ℓ=0Nℓ(Jℓs)2)A∥=βref2∑sZsnsvts∑ℓ=0NℓJℓsGℓ,1s(16)\left(k_\perp^2+\frac{\beta_{\mathrm{ref}}}{2}\sum_s\frac{Z_s^2 n_s}{m_s}\sum_{\ell=0}^{N_\ell}(\mathcal{J}_\ell^s)^2\right)A_\parallel = \frac{\beta_{\mathrm{ref}}}{2}\sum_s Z_s n_s v_{ts}\sum_{\ell=0}^{N_\ell}\mathcal{J}_\ell^s G_{\ell,1}^s \tag*{(16)}

and

βref2B2∑sZsns∑ℓ=0NℓJℓs(Jℓs+Jℓ−1s)Φ+(1+βref2B2∑snsτs∑ℓ=0Nℓ(Jℓs+Jℓ−1s)2)δB∥B=−βref2B2∑snsτs∑ℓ=0Nℓ(Jℓs+Jℓ−1s)Gℓ,0s(17)\frac{\beta_{\mathrm{ref}}}{2B^2}\sum_s Z_s n_s\sum_{\ell=0}^{N_\ell}\mathcal{J}_\ell^s(\mathcal{J}_\ell^s+\mathcal{J}_{\ell-1}^s)\Phi+ \left(1+\frac{\beta_{\mathrm{ref}}}{2B^2}\sum_s n_s\tau_s\sum_{\ell=0}^{N_\ell}(\mathcal{J}_\ell^s+\mathcal{J}_{\ell-1}^s)^2\right)\frac{\delta B_\parallel}{B} = -\frac{\beta_{\mathrm{ref}}}{2B^2}\sum_s n_s\tau_s\sum_{\ell=0}^{N_\ell}(\mathcal{J}_\ell^s+\mathcal{J}_{\ell-1}^s)G_{\ell,0}^s \tag*{(17)}

where Jℓs(k⊥2ρs2)\mathcal{J}_\ell^s(k_\perp^2\rho_s^2) is the Laguerre projection of the Bessel function. In practice, Eq. (13) is used to replace Hℓ,mH_{\ell,m} for Gℓ,mG_{\ell,m} in Eq. (14). The resulting equation is then advanced for Gℓ,mG_{\ell,m} and the fields are resolved to obtain the updated Hℓ,mH_{\ell,m}. The details on these derivations can be found in Mandell et al. [33, 32]. Finally, we note that while by default GX includes δB∥\delta B_\parallel, in this work we neglect parallel magnetic field fluctuations, i.e. δB∥=0\delta B_\parallel=0. Ultimately, the implicit algorithms presented can be generalized to include δB∥\delta B_\parallel in a relatively straightforward manner.

Time-Integration Schemes

In this work, we use two types of time-integration schemes: a Lie-Trotter splitting scheme (Boyd [8]) similar to the one used in stella (Barnes et al. [6]), as well as additive Runge-Kutta methods (Cooper & Sayfy [12]).

Lie-Trotter

We begin with the Lie-Trotter scheme. For simplicity, we write Eq. (14) as

∂G∂t=A[G]+BG+CG(18)\frac{\partial G}{\partial t}=A[G]+\mathbf{B}G+\mathbf{C}G \tag*{(18)}

where GG is a vector containing all of the spectral coefficients of all species and the linear operators B\mathbf{B} and C\mathbf{C} represent the stiff streaming and mirror operators

(BG)ℓ,m=vts∇∥(m+1Hℓ,m+1s+mHℓ,m−1s)(19)(\mathbf{B}G)_{\ell,m}=v_{ts}\nabla_\parallel\left(\sqrt{m+1}H_{\ell,m+1}^s+\sqrt{m}H_{\ell,m-1}^s\right) \tag*{(19)}

and

(CG)ℓ,m=vts[−(ℓ+1)m+1Hℓ,m+1s−ℓm+1Hℓ−1,m+1s+ℓmHℓ,m−1s+(ℓ+1)mHℓ+1,m−1s]∇∥ln⁡B,(20)(\mathbf{C}G)_{\ell,m}=v_{ts}\left[-(\ell+1)\sqrt{m+1}H_{\ell,m+1}^s-\ell\sqrt{m+1}H_{\ell-1,m+1}^s+\ell\sqrt{m}H_{\ell,m-1}^s+(\ell+1)\sqrt{m}H_{\ell+1,m-1}^s\right]\nabla_\parallel\ln B, \tag*{(20)}

respectively. The remaining terms, including the nonlinear term, are contained in the operator AA. In a Lie-Trotter splitting, we advance the spectral coefficients GnG^n at time tnt^n to Gn+1G^{n+1} at tn+1t^{n+1} by evaluating each operator sequentially:

G‾n=Gn+ΔtA[Gn],(21)\overline{G}^{n}=G^n+\Delta t A[G^n], \tag*{(21)}
G‾‾n=G‾n+ΔtBG‾n,(22)\overline{\overline{G}}^{n}=\overline{G}^{n}+\Delta t B\overline{G}^{n}, \tag*{(22)}
Gn+1=G‾‾n+ΔtCGn+1.(23)G^{n+1}=\overline{\overline{G}}^{n}+\Delta t C G^{n+1}. \tag*{(23)}

For the explicit operator AA, we solve for the intermediate G‾n\overline{G}^{n} using GnG^n at the current timestep. However, the equations containing BB and CC use the second, intermediate, and future stage, and this results in two matrix equations that must be solved. Furthermore, this method does not require storing extra copies of GG, which makes it very memory-efficient.

While in the above equations, the explicit operator is advanced using a Forward Euler scheme, we in fact follow Barnes et al. [6] and use the standard third-order Strong-Stability Preserving Runge-Kutta (SSPRK3) method from Gottlieb et al. [20]. The implicit advances are performed using the implicit method from Giraldo et al. [19], which is second-order accurate. We find that the use of a second-order implicit scheme is important for accurately capturing transitions between different types of linear instabilities in toroidal geometry even though the entire scheme taken as a whole is first order. Strang-splitting is commonly used to obtain second-order accuracy at a higher computational cost. Nevertheless, it has been observed that the asymptotic convergence rate may only be realized for sufficiently small timesteps, and that the associated error constant can be large ([36]). This motivates the use of higher-order IMEX methods.

Additive Runge-Kutta

In the additive Runge-Kutta (ARK) methods, we solve for the solution at the next timestep Gn+1G^{n+1} using a series of stages

y(i)=Gn+Δt∑jaijA[y(j)]+Δt∑ja~ijB∗y(j),(24)y^{(i)}=G^n+\Delta t\sum_j a_{ij}A\left[y^{(j)}\right]+\Delta t\sum_j\widetilde{a}_{ij}B^*y^{(j)}, \tag*{(24)}

and

Gn+1=Gn+Δt∑jbjA[y(j)]+Δt∑jb~jB∗y(j).(25)G^{n+1}=G^n+\Delta t\sum_j b_j A\left[y^{(j)}\right]+\Delta t\sum_j\widetilde{b}_j B^*y^{(j)}. \tag*{(25)}

The operator AA is the explicit operator as for the Lie-Trotter scheme, but now B∗B^* contains both the streaming and mirror terms

(B∗G)ℓ,m=vts[∇∥(m+1Hℓ,m+1s+mHℓ,m−1s)+(−(ℓ+1)m+1Hℓ,m+1s−ℓm+1Hℓ−1,m+1s+ℓmHℓ,m−1s+(ℓ+1)mHℓ+1,m−1s)∇∥ln⁡B].(26)(B^*G)_{\ell,m}=v_{ts}\left[\nabla_\parallel\left(\sqrt{m+1}H^s_{\ell,m+1}+\sqrt{m}H^s_{\ell,m-1}\right) +\left(-(\ell+1)\sqrt{m+1}H^s_{\ell,m+1}-\ell\sqrt{m+1}H^s_{\ell-1,m+1} +\ell\sqrt{m}H^s_{\ell,m-1}+(\ell+1)\sqrt{m}H^s_{\ell+1,m-1}\right)\nabla_\parallel\ln B\right]. \tag*{(26)}

There are many different choices for coefficients aija_{ij}, a~ij\widetilde{a}_{ij}, bib_i and b~i\widetilde{b}_i in the published literature. For this work, we use the coefficients from the ‘ARK3(2)4L[2]SA–ESDIRK’ method of Kennedy & Carpenter [28]. This method contains 4 stages and is overall 3rd order accurate. Although the scheme has a higher order of accuracy than the Lie-Trotter method, it requires storing the intermediate solutions y(i)\mathbf{y}^{(i)} and performing three linear solves per step.

A key challenge for implicit gyrokinetic simulations is that the implicit operators act on both the plasma distribution function and the electromagnetic fields (which are both contained in Hℓ,mH_{\ell,m}), which are themselves obtained from integral operators of the distribution function. This coupling makes the inversion significantly more difficult. In the next section, we describe several algorithms designed to address this difficulty.

Solving the Implicit Time-Advance Equations

In this section, we describe our methods to invert the parallel streaming and mirror force operators for our implicit methods. In §4.1, we discuss Kotschenreuther’s method, which is used to invert the streaming operator in GS2 and stella. Due to the large memory requirements of this method, we introduce new methods in §4.2 and §4.3 that approximately invert the streaming operator. Then in §4.4, we present our method for inverting the mirror term. Sections 4.5 and 4.6 discuss the hypercollision and magnetic/curvature drift terms and their stiffness. Due to approximations made in inverting the streaming and mirror operators, in §4.7 and §4.8 we discuss simple iterative methods to improve the accuracy of our methods.

Kotschenreuther’s Method

We start by briefly describing Kotschenreuther’s method [30] for inverting the linear operators in the gyrokinetic equation. The Kotschenreuther method is used to invert all the linear terms in GS2, as well as the streaming terms for stella.

Projecting the parallel streaming terms onto the Hermite-Laguerre basis yields the equation

∂Gℓ,ms∂t+vts∇∥(m+1Gℓ,m+1s+mGℓ,m−1s+ZsτsJℓsΦδm−1,0)=0,(27)\frac{\partial G_{\ell,m}^{s}}{\partial t}+v_{ts}\nabla_{\parallel}\left(\sqrt{m+1}G_{\ell,m+1}^{s}+\sqrt{m}G_{\ell,m-1}^{s}+\frac{Z_s}{\tau_s}\mathcal{J}_{\ell}^{s}\Phi\delta_{m-1,0}\right)=0, \tag*{(27)}

where for simplicity we have ignored the drive and collision terms. We have also assumed only electrostatic perturbations. Discretizing in time using a Backwards Euler scheme, we must solve an equation of the form

Gℓ,ms,n+1+Δt vts∇∥(m+1Gℓ,m+1s,n+1+mGℓ,m−1s,n+1)+Δt Zsτsvtsδm1∇∥(JℓsΦn+1)=Gℓ,ms,n,(28)G_{\ell,m}^{s,n+1}+\Delta t\,v_{ts}\nabla_{\parallel}\left(\sqrt{m+1}G_{\ell,m+1}^{s,n+1}+\sqrt{m}G_{\ell,m-1}^{s,n+1}\right)+\Delta t\,\frac{Z_s}{\tau_s}v_{ts}\delta_{m1}\nabla_{\parallel}\left(\mathcal{J}_{\ell}^{s}\Phi^{n+1}\right)=G_{\ell,m}^{s,n}, \tag*{(28)}

where the nn and n+1n+1 indices indicate the current and future timesteps, respectively. In GX, the above equation is then Fourier transformed in the zz domain, resulting in the equation

G^ℓ,ms,n+1+Δt vtsik∥(m+1G^ℓ,m+1s,n+1+mG^ℓ,m−1s,n+1)+Δt Zsτsvtsδm1ik∥F[JℓsΦn+1]=G^ℓ,ms,n,(29)\widehat{G}_{\ell,m}^{s,n+1}+\Delta t\,v_{ts}ik_{\parallel}\left(\sqrt{m+1}\widehat{G}_{\ell,m+1}^{s,n+1}+\sqrt{m}\widehat{G}_{\ell,m-1}^{s,n+1}\right)+\Delta t\,\frac{Z_s}{\tau_s}v_{ts}\delta_{m1}ik_{\parallel}\mathcal{F}\left[\mathcal{J}_{\ell}^{s}\Phi^{n+1}\right]=\widehat{G}_{\ell,m}^{s,n}, \tag*{(29)}

where the hats and F[ ]\mathcal{F}[\ ] denote Fourier transforms.

We separate the solution G^ℓ,ms\widehat{G}_{\ell,m}^{s} into a homogeneous and inhomogeneous part G^ℓ,ms=\widehat{G}_{\ell,m}^{s} = G^ℓ,mh,s+G^ℓ,mi,s\hat{G}^{h,s}_{\ell,m}+\hat{G}^{i,s}_{\ell,m} satisfying

G^ℓ,mh,s,n+1+Δtvtsik∥(m+1G^ℓ,m+1h,s,n+1+mG^ℓ,m−1h,s,n+1)+ΔtZsτsvtsδm1ik∥F[JℓsΦn+1]=0,(30)\hat{G}^{h,s,n+1}_{\ell,m}+\Delta t v_{ts} i k_{\parallel}\left(\sqrt{m+1}\hat{G}^{h,s,n+1}_{\ell,m+1}+\sqrt{m}\hat{G}^{h,s,n+1}_{\ell,m-1}\right) +\Delta t\frac{Z_s}{\tau_s}v_{ts}\delta_{m1}i k_{\parallel}\mathcal{F}\left[\mathcal{J}_{\ell}^{s}\Phi^{n+1}\right]=0, \tag*{(30)}

and

G^ℓ,mi,s,n+1+Δtvtsik∥(m+1G^ℓ,m+1i,s,n+1+mG^ℓ,m−1i,s,n+1)=G^ℓ,ms,n.(31)\hat{G}^{i,s,n+1}_{\ell,m}+\Delta t v_{ts} i k_{\parallel}\left(\sqrt{m+1}\hat{G}^{i,s,n+1}_{\ell,m+1}+\sqrt{m}\hat{G}^{i,s,n+1}_{\ell,m-1}\right)=\hat{G}^{s,n}_{\ell,m}. \tag*{(31)}

We can straightforwardly solve Eq. (31) for G^ℓ,mi,s\hat{G}^{i,s}_{\ell,m} using a tri-diagonal solver. Specifically, Eq. (31) is tridiagonal with respect to the mm indices, and the solver concurrently solves the equation across the remaining parameters. To solve for the homogeneous part, we need to first find an expression for Φ\Phi. Note that Φ\Phi can be expressed as

[∑sZs2nsτs(1−∑ℓ=0Nℓ(Jℓs)2)]Φ=Φi+(∑sZsns∑ℓJℓsMℓs)Φ,(32)\left[\sum_s \frac{Z_s^2 n_s}{\tau_s}\left(1-\sum_{\ell=0}^{N_\ell}(\mathcal{J}_{\ell}^{s})^2\right)\right]\Phi = \Phi^i+\left(\sum_s Z_s n_s\sum_{\ell}\mathcal{J}_{\ell}^{s}\mathbf{M}_{\ell}^{s}\right)\Phi, \tag*{(32)}

where Φi\Phi^i is the contribution to Φ\Phi from the inhomogeneous solution, and Mℓs=δG0h,s(zi)/δΦj\mathbf{M}_{\ell}^{s}=\delta G_{0}^{h,s}(z_i)/\delta\Phi_j is called the ‘response matrix’. Mℓ,i,js\mathbf{M}_{\ell,i,j}^{s} is the response at ziz_i, in Gℓ,0h,s(zi)G_{\ell,0}^{h,s}(z_i) due to a unit perturbation in Φ\Phi at zjz_j. The response matrix can be calculated for each zz, kxk_x, and kyk_y using another series of tridiagonal matrix solves. For each solve, we set Φ=δij\Phi=\delta_{ij}, and multiply it by Jℓs\mathcal{J}_{\ell}^{s} before Fourier transforming. From there, we can solve for Φ\Phi by solving

(I−D−1∑sZsns∑ℓJℓsM)Φ=Φi,(33)\left(I-D^{-1}\sum_s Z_s n_s\sum_{\ell}\mathcal{J}_{\ell}^{s}\mathbf{M}\right)\Phi=\Phi^i, \tag*{(33)}

where D=∑sZs2ns/τs(1−∑ℓ=0Nℓ(Jℓs)2)D=\sum_s Z_s^2 n_s/\tau_s\left(1-\sum_{\ell=0}^{N_\ell}(\mathcal{J}_{\ell}^{s})^2\right). We can then substitute Φ\Phi into the original equation and solve for Gℓ,msG_{\ell,m}^{s}.

Although exact, the major issue with using Kotschenreuther’s method for GX is the memory requirement. For NzN_z points along the field-line and NxN_x and NyN_y number of kxk_x and kyk_y wavenumbers, there are NxNyN_xN_y matrices of dimension NzN_z, each of which needs to be inverted to solve for Φ\Phi. However, when including boundary conditions for the flux-tube, the dimension of each matrix is now NzNlinksN_zN_{\mathrm{links}}, where NlinksN_{\mathrm{links}} is the number of flux tubes at different kxk_x that are linked together. For a moderate resolution nonlinear simulation, Nz≈32N_z\approx32, Nx≈Ny≈256N_x\approx N_y\approx256, and Nlinks≈10N_{\mathrm{links}}\approx10. As GX uses single-precision complex values for all of its data, the total memory required could then reach approximately 100 GB, which would be well beyond the amount of memory most GPUs today contain, especially if the same GPU must also contain all of the moments and fields. These matrices will also be dense, and so cannot be efficiently handled with sparse matrix solvers. Finally, the need to compute and store the response matrices makes this method less amenable to matrix-free iterative methods. This motivates the need for alternative approaches such as the ones described in this paper.

Parallel Streaming

As above, we would like to solve an equation of the form

G^ℓ,ms,n+1+Δtvtsik∥(m+1G^ℓ,m+1s,n+1+mG^ℓ,m−1s,n+1)+ΔtZsτsvtsδm1ik∥F[JℓsΦn+1]=G^ℓ,ms,n.(34)\hat{G}^{s,n+1}_{\ell,m}+\Delta t v_{ts} i k_{\parallel}\left(\sqrt{m+1}\hat{G}^{s,n+1}_{\ell,m+1}+\sqrt{m}\hat{G}^{s,n+1}_{\ell,m-1}\right) +\Delta t\frac{Z_s}{\tau_s}v_{ts}\delta_{m1}i k_{\parallel}\mathcal{F}\left[\mathcal{J}_{\ell}^{s}\Phi^{n+1}\right]=\hat{G}^{s,n}_{\ell,m}. \tag*{(34)}

Without Φ\Phi, the equation is tridiagonal in the Hermite moments mm. However, the term F[JℓsΦ]\mathcal{F}[\mathcal{J}_{\ell}^{s}\Phi] involves convolving Φ\Phi, which makes the equation difficult to solve. Instead of directly inverting, we will use an iterative scheme to solve this equation.

Assuming only a single ion species and electrons, the equation for Φ\Phi is

Φ=Zini∑ℓJℓiGℓ0i∑sZs2nsτs(1−∑ℓ(Jℓs)2)+Zene∑ℓJℓeGℓ0e∑sZs2nsτs(1−∑ℓ(Jℓs)2).(35)\Phi= \frac{Z_i n_i \sum_{\ell}\mathcal{J}_{\ell}^{i}G_{\ell0}^{i}}{\sum_s \frac{Z_s^2 n_s}{\tau_s}\left(1-\sum_{\ell}(\mathcal{J}_{\ell}^{s})^2\right)}+\frac{Z_e n_e \sum_{\ell}\mathcal{J}_{\ell}^{e}G_{\ell0}^{e}}{\sum_s \frac{Z_s^2 n_s}{\tau_s}\left(1-\sum_{\ell}(\mathcal{J}_{\ell}^{s})^2\right)}. \tag*{(35)}

To simplify this equation, we first take a drift-kinetic (k⊥ρi≪1k_{\perp}\rho_i \ll1) limit. This simply results in neglecting all finite-Larmor-radius (FLR) terms in the equation for Φ\Phi by setting J0s=1\mathcal{J}_{0}^{s}=1 and Jℓ>0s=0\mathcal{J}_{\ell>0}^{s}=0 except for in the polarization term ∑sZs2ns/τs(1−∑ℓ=0Ne(Jℓs)2)\sum_s Z_s^2 n_s/\tau_s\left(1-\sum_{\ell=0}^{N_e}(\mathcal{J}_{\ell}^{s})^2\right). The Jℓs\mathcal{J}_{\ell}^{s} multiplying Φ\Phi in Eq. (34) is also set to 1. The equation for Φ\Phi is then

Φ≈ZiniG0,0i∑sZs2nsτs(1−∑ℓ(Jℓs)2)+ZeneG0,0e∑sZs2nsτs(1−∑ℓ(Jℓs)2).(36)\Phi\approx\frac{Z_i n_i G_{0,0}^{i}}{\sum_s \frac{Z_s^2 n_s}{\tau_s}\left(1-\sum_{\ell}(\mathcal{J}_{\ell}^{s})^2\right)}+\frac{Z_e n_e G_{0,0}^{e}}{\sum_s \frac{Z_s^2 n_s}{\tau_s}\left(1-\sum_{\ell}(\mathcal{J}_{\ell}^{s})^2\right)}. \tag*{(36)}

However, the denominator is still a function of the parallel coordinate, and would still result in a convolution after Fourier-transforming. To avoid this, we approximate the denominator by its minimum over zz:

Φ=ZiniG0,0iDm+ZeneG0,0eDm,(37)\Phi= \frac{Z_i n_i G_{0,0}^{i}}{D_m}+\frac{Z_e n_e G_{0,0}^{e}}{D_m}, \tag*{(37)}

where DmD_m is

Dm=min⁡z[∑sZs2nsτs(1−∑ℓ(Jℓs)2)].(38)D_m=\min_z\left[\sum_s\frac{Z_s^2 n_s}{\tau_s}\left(1-\sum_{\ell}(\mathcal{J}_{\ell}^{s})^2\right)\right]. \tag*{(38)}

As described in Appendix § 9, we make this choice to capture the highest frequency wave in the system. An iterative method described in §4.7 will then help accurately capture the remaining modes.

Substituting this approximation for Φ\Phi into Eq. (34) yields

G^ℓ,ms,n+1+Δtvtsik∥(m+1G^ℓ,m+1s,n+1+mG^ℓ,m−1s,n+1)+Δtvtsik∥Zsτs(ZiniG^0,0i,n+1Dm+ZeneG^0,0e,n+1Dm)δℓ0δm1=G^ℓ,ms,n.(39)\widehat{G}_{\ell,m}^{s,n+1}+\Delta t v_{ts}ik_{\parallel}\left(\sqrt{m+1}\widehat{G}_{\ell,m+1}^{s,n+1}+\sqrt{m}\widehat{G}_{\ell,m-1}^{s,n+1}\right) +\Delta t v_{ts}ik_{\parallel}\frac{Z_s}{\tau_s}\left(\frac{Z_i n_i\widehat{G}_{0,0}^{i,n+1}}{D_m}+\frac{Z_e n_e\widehat{G}_{0,0}^{e,n+1}}{D_m}\right)\delta_{\ell0}\delta_{m1} =\widehat{G}_{\ell,m}^{s,n}. \tag*{(39)}

This equation is now nearly tridiagonal, with simple corrections from Φ\Phi. The resulting matrix system can be written as

(BT+u⊗v)G=r,(40)(B_T+u\otimes v)G=r, \tag*{(40)}

where the matrix BTB_T contains the tridiagonal bands, and the outer product of uu and vv contains the approximation to Φ\Phi. The expressions for uu and vv are

u=ik∥Δt[0 Zivtiτiδℓ0δm1 0 Zevteτeδℓ0δm1 0],(41)u=ik_{\parallel}\Delta t\left[0\ \frac{Z_i v_{ti}}{\tau_i}\delta_{\ell0}\delta_{m1}\ 0\ \frac{Z_e v_{te}}{\tau_e}\delta_{\ell0}\delta_{m1}\ 0\right], \tag*{(41)}

and

v=[ZiniDmδℓ0δm0 0 ZeneDmδℓ0δm0 0],(42)v=\left[\frac{Z_i n_i}{D_m}\delta_{\ell0}\delta_{m0}\ 0\ \frac{Z_e n_e}{D_m}\delta_{\ell0}\delta_{m0}\ 0\right], \tag*{(42)}

where the bolded 0\mathbf{0} indicates consecutive zeros.

Since Φ\Phi is a low-rank correction, we can solve the above system using the Sherman-Morrison formula, as detailed in Appendix § 10. A visualization of this banded structure is shown in Figure 1.

Visualization of the banded structure of the streaming matrix

Figure 1. A visualization of the banded structure of the streaming matrix, with only electrostatic perturbations.

It should be noted that we did not have to take a long-wavelength approximation for Φ\Phi, and it would still be a low-rank correction. However, by taking a long-wavelength approximation, the intermediate solutions needed for the Sherman-Morrison formula are only of dimension nsLMNzn_sLMN_z, where nsn_s is the number of species, LL is the number of Laguerre moments, and MM is the number of Hermite moments. If we did not take a long-wavelength approximation, then the size would be nsLMNzNyNxn_sLMN_zN_yN_x. This memory saving is important given the limited memory on GPUs. Overall, this method involves only a small number of tri-diagonal solves, which can be computed in linear time without extra storage, as well as a series of cheap vector operations where the vectors are also low in memory. Furthermore, as we explain in §4.7, this approximation is only used as a preconditioner for an iterative method. For the simulations in this work, we only focus on ion-scale turbulence in which k⊥ρi≲1k_\perp\rho_i \lesssim1, and the iterative method converges rapidly. However, it is possible that shorter wavelength instabilities would require a more accurate preconditioner, and this will be addressed in future work.

When simulating multiple kxk_x wavenumbers in the presence of magnetic shear, s^≠0\hat{s} \ne0, enforcing periodicity in xx results in the well-known ‘twist-and-shift’ boundary conditions [7, 34]. Modes with differing kxk_x (within the grid) are coupled at the end of the zz domain via the formula

Hℓ,m(kx,ky,z)=Hℓ,m(kx+δk,ky,z+2π),(43)H_{\ell,m}(k_x,k_y,z) = H_{\ell,m}(k_x+\delta k,k_y,z+2\pi), \tag*{(43)}

where δk=2πkys^\delta k = 2\pi k_y\hat{s}. One can interpret this formula as simulating an extended flux-tube that contains all the coupled kxk_x’s. The above algorithm remains mostly unchanged in this new flux-tube, except that the Fourier transforms in zz are taken at fixed kyk_y along this extended flux tube. Similarly, we take DmD_m to be the minimum along the extended flux-tube:

Dm(ky)=min⁡∀kx,zD(z).(44)D_m(k_y) = \min_{\forall k_x,z} D(z). \tag*{(44)}

Streaming with A∥A_{\parallel}

When including A∥A_{\parallel}, the parallel streaming terms become

∂Gℓ,ms∂t+vts∇∥[m+1Gℓ,m+1s+mGℓ,m−1s+ZsτsJℓsΦδm1−ZsvtsτsJℓsA∥(δm0+2δm2)]=r.(45)\frac{\partial G^{s}_{\ell,m}}{\partial t}+v_{ts}\nabla_{\parallel}\left[\sqrt{m+1}G^{s}_{\ell,m+1}+\sqrt{m}G^{s}_{\ell,m-1}+\frac{Z_{s}}{\tau_{s}}\mathcal{J}^{s}_{\ell}\Phi\delta_{m1}-\frac{Z_{s}v_{ts}}{\tau_{s}}\mathcal{J}^{s}_{\ell}A_{\parallel}\left(\delta_{m0}+\sqrt{2}\delta_{m2}\right)\right]=r. \tag*{(45)}

Like with Φ\Phi, we take a drift-kinetic limit. In this limit A∥A_{\parallel} is simplified to

A∥=βref2F(ZinivtiG0,1i+ZenevteG0,1e),(46)A_{\parallel}=\frac{\beta_{\mathrm{ref}}}{2F}\left(Z_{i}n_{i}v_{ti}G^{i}_{0,1}+Z_{e}n_{e}v_{te}G^{e}_{0,1}\right), \tag*{(46)}

and FF is

F=k∥2+βref2∑sZs2nsms∑ℓ=0Ne(Jℓs)2.(47)F=k_{\parallel}^{2}+\frac{\beta_{\mathrm{ref}}}{2}\sum_{s}\frac{Z_{s}^{2}n_{s}}{m_{s}}\sum_{\ell=0}^{N_{e}}\left(\mathcal{J}^{s}_{\ell}\right)^{2}. \tag*{(47)}

As described in Appendix § 9, we maximize rather than minimize FF:

FM=max⁡z[k∥2+βref2∑sZs2nsms∑ℓ=0Ne(Jℓs)2].(48)F_{M}=\max_{z}\left[k_{\parallel}^{2}+\frac{\beta_{\mathrm{ref}}}{2}\sum_{s}\frac{Z_{s}^{2}n_{s}}{m_{s}}\sum_{\ell=0}^{N_{e}}\left(\mathcal{J}^{s}_{\ell}\right)^{2}\right]. \tag*{(48)}

Our final approximation for A∥A_{\parallel} is thus

A∥=βref2FM(ZinivtiG0,1i+ZenevteG0,1e).(49)A_{\parallel}=\frac{\beta_{\mathrm{ref}}}{2F_{M}}\left(Z_{i}n_{i}v_{ti}G^{i}_{0,1}+Z_{e}n_{e}v_{te}G^{e}_{0,1}\right). \tag*{(49)}

The final equation that we solve is

(BT+u1⊗v1+u2⊗v2)G=r,(50)\left(B_{T}+\boldsymbol{u}_{1}\otimes\boldsymbol{v}_{1}+\boldsymbol{u}_{2}\otimes\boldsymbol{v}_{2}\right)G=r, \tag*{(50)}

where u1\boldsymbol{u}_{1} and v1\boldsymbol{v}_{1} are the same as in Eqs. (41) and (42), and u2\boldsymbol{u}_{2} and v2\boldsymbol{v}_{2} are given by

u2=−ik∥Δt[Zivti2τiδℓ0δm0  0  2Zivti2τiδℓ0δm2  0Zevte2τeδℓ0δm0  0  2Zevte2τeδℓ0δm2  0],(51)\boldsymbol{u}_{2}=-ik_{\parallel}\Delta t\left[ \frac{Z_{i}v_{ti}^{2}}{\tau_{i}}\delta_{\ell0}\delta_{m0}\ \ 0\ \ \sqrt{2}\frac{Z_{i}v_{ti}^{2}}{\tau_{i}}\delta_{\ell0}\delta_{m2}\ \ \mathbf{0} \atop \frac{Z_{e}v_{te}^{2}}{\tau_{e}}\delta_{\ell0}\delta_{m0}\ \ 0\ \ \sqrt{2}\frac{Z_{e}v_{te}^{2}}{\tau_{e}}\delta_{\ell0}\delta_{m2}\ \ \mathbf{0} \right], \tag*{(51)}

and

v2=[0  βref2FMZinivtiδℓ0δm1  0  βref2FMZenevteδℓ0δm1  0].(52)\boldsymbol{v}_{2}=\left[ 0\ \ \frac{\beta_{\mathrm{ref}}}{2F_{M}}Z_{i}n_{i}v_{ti}\delta_{\ell0}\delta_{m1}\ \ \mathbf{0}\ \ \frac{\beta_{\mathrm{ref}}}{2F_{M}}Z_{e}n_{e}v_{te}\delta_{\ell0}\delta_{m1}\ \ \mathbf{0} \right]. \tag*{(52)}

Mirror Force

For the mirror terms, since there is no ∇∥\nabla_{\parallel} operator, we do not perform any parallel Fourier transforms. As a result, we do not make approximations for the fields and instead directly solve the equations that arise from our implicit scheme. Furthermore, we will only treat the electron mirror term implicitly. After discretizing in time, the equation involving the electron mirror term is of the form

Gℓ,me,n+1+Δtvte[−(ℓ+1)m+1Hℓ,m+1e,n+1−ℓm+1Hℓ−1,m+1e,n+1+ℓmHℓ,m−1e,n+1+(ℓ+1)mHℓ+1,m−1e,n+1]∇∥ln⁡B=Gℓ,me,n.(53)G^{e,n+1}_{\ell,m}+\Delta t v_{te}\left[-(\ell+1)\sqrt{m+1}H^{e,n+1}_{\ell,m+1}-\ell\sqrt{m+1}H^{e,n+1}_{\ell-1,m+1}+\ell\sqrt{m}H^{e,n+1}_{\ell,m-1}+(\ell+1)\sqrt{m}H^{e,n+1}_{\ell+1,m-1}\right]\nabla_{\parallel}\ln B=G^{e,n}_{\ell,m}. \tag*{(53)}

The matrix we invert is again sparse and banded, with low rank corrections from the fields that are handled using the Sherman-Morrison formula. The banded pattern is shown in Figure 2a. There are 5 bands in the matrix: a tridiagonal structure with two additional bands at an offset of M−1M-1 from the main diagonal.

Figure 2. The visualizations of the banded structure of the mirror (a) and magnetic drift (c) terms. The middle figure (b) shows the structure of the mirror matrix that is used in the MAGMA banded matrix routines.

To take advantage of this sparsity pattern, we utilize direct solvers for banded matrices recently developed in the MAGMA linear algebra package [1]. In this solver, the original input matrix of size LM×LMLM \times LM is stored in banded format into a matrix of size (2KL+KU+1)×LM(2K_{\mathrm{L}}+K_{\mathrm{U}}+1)\times LM, where KLK_{\mathrm{L}} and KUK_{\mathrm{U}} are the lower and upper bandwidth, respectively. Row KL+KU+1K_{\mathrm{L}}+K_{\mathrm{U}}+1 of this new matrix contains the main diagonal, while the rows above and below contain the upper and lower diagonals. This structure is shown in Figure 2b. The full mirror matrix contains 4L2M24L^2M^2 entries for 2 species, whereas this condensed matrix only contains (3M−2)(2LM)∼6LM2(3M-2)(2LM)\sim6LM^2 entries, and so results in about a factor of LL reduction in memory.

To analyse how Φ\Phi enters into the mirror terms, we rewrite each Hℓ,meH^e_{\ell,m} in terms of Gℓ,m+1eG^e_{\ell,m+1}

Hℓ,m+1e=Gℓ,m+1e−1τeJℓeΦδm+1,0,(54)H^e_{\ell,m+1}=G^e_{\ell,m+1}-\frac{1}{\tau_e}\mathcal{J}^e_{\ell}\Phi\delta_{m+1,0}, \tag*{(54)}
Hℓ−1,m+1e=Gℓ−1,m+1e−1τeJℓ−1eΦδm+1,0,(55)H^e_{\ell-1,m+1}=G^e_{\ell-1,m+1}-\frac{1}{\tau_e}\mathcal{J}^e_{\ell-1}\Phi\delta_{m+1,0}, \tag*{(55)}
Hℓ,m−1e=Gℓ,m−1e−1τeJℓeΦδm−1,0,(56)H^e_{\ell,m-1}=G^e_{\ell,m-1}-\frac{1}{\tau_e}\mathcal{J}^e_{\ell}\Phi\delta_{m-1,0}, \tag*{(56)}
Hℓ+1,m−1e=Gℓ+1,m−1e−1τeJℓ+1eΦδm−1,0.(57)H^e_{\ell+1,m-1}=G^e_{\ell+1,m-1}-\frac{1}{\tau_e}\mathcal{J}^e_{\ell+1}\Phi\delta_{m-1,0}. \tag*{(57)}

The only contributions are from δm−1,0\delta_{m-1,0} or δm,1\delta_{m,1}. However, from Eq. (14) these terms have prefactors that vanish when taking a drift-kinetic limit, allowing us to neglect Φ\Phi when inverting. The drift-kinetic limit is particularly appropriate as we only treat the electron mirror terms implicitly while focusing on ion scale turbulence.

Repeating this exercise with A∥A_{\parallel} yields

Hℓ,m+1e=Gℓ,m+1e+vteτeJℓeA∥δm+1,1,(58)H^e_{\ell,m+1}=G^e_{\ell,m+1}+\frac{v_{\mathrm{te}}}{\tau_e}\mathcal{J}^e_{\ell}A_{\parallel}\delta_{m+1,1}, \tag*{(58)}
Hℓ−1,m+1e=Gℓ−1,m+1e+vteτeJℓ−1eA∥δm+1,1,(59)H^e_{\ell-1,m+1}=G^e_{\ell-1,m+1}+\frac{v_{\mathrm{te}}}{\tau_e}\mathcal{J}^e_{\ell-1}A_{\parallel}\delta_{m+1,1}, \tag*{(59)}
Hℓ,m−1e=Gℓ,m−1e+vteτeJℓeA∥δm−1,1,(60)H^e_{\ell,m-1}=G^e_{\ell,m-1}+\frac{v_{\mathrm{te}}}{\tau_e}\mathcal{J}^e_{\ell}A_{\parallel}\delta_{m-1,1}, \tag*{(60)}
Hℓ+1,m−1e=Gℓ+1,m−1e+vteτeJℓ+1eA∥δm−1,1.(61)H^e_{\ell+1,m-1}=G^e_{\ell+1,m-1}+\frac{v_{\mathrm{te}}}{\tau_e}\mathcal{J}^e_{\ell+1}A_{\parallel}\delta_{m-1,1}. \tag*{(61)}

When taking a drift-kinetic limit and including prefactors, we only need include the A∥A_{\parallel} contribution to the Hℓ,m+1eH^e_{\ell,m+1} and Hℓ−1,m+1eH^e_{\ell-1,m+1} terms. Substituting in A∥A_{\parallel}, the electron mirror term is then

vte[−(ℓ+1)m+1Gℓ,m+1e−ℓm+1Gℓ−1,m+1e+ℓmGℓ,m−1e+(ℓ+1)mGℓ+1,m−1e−vteτeA∥δℓ0δm0−vteτeA∥δℓ1δm0]∇∥ln⁡B.(62)v_{\mathrm{te}}\left[-(\ell+1)\sqrt{m+1}G^e_{\ell,m+1}-\ell\sqrt{m+1}G^e_{\ell-1,m+1}+\ell\sqrt{m}G^e_{\ell,m-1}+(\ell+1)\sqrt{m}G^e_{\ell+1,m-1}-\frac{v_{\mathrm{te}}}{\tau_e}A_{\parallel}\delta_{\ell0}\delta_{m0}-\frac{v_{\mathrm{te}}}{\tau_e}A_{\parallel}\delta_{\ell1}\delta_{m0}\right]\nabla_{\parallel}\ln B. \tag*{(62)}

In A∥A_{\parallel} there are both ion and electron contributions A∥=A∥i+A∥eA_{\parallel}=A^i_{\parallel}+A^e_{\parallel}. However, as we are only treating the electron mirror term implicitly, at each step of our timestepping schemes, the ion term has already been advanced explicitly. Therefore, A∥iA^i_{\parallel} can be moved to the RHS of the equation. We are only left with the electron contribution

vte[−(ℓ+1)m+1Gℓ,m+1e−ℓm+1Gℓ−1,m+1e+ℓmGℓ,m−1e+(ℓ+1)mGℓ+1,m−1e−vteτeA∥eδℓ0δm0−vteτeA∥eδℓ1δm0]∇∥ln⁡B.(63)v_{\mathrm{te}}\left[-(\ell+1)\sqrt{m+1}G^e_{\ell,m+1}-\ell\sqrt{m+1}G^e_{\ell-1,m+1}+\ell\sqrt{m}G^e_{\ell,m-1}+(\ell+1)\sqrt{m}G^e_{\ell+1,m-1}-\frac{v_{\mathrm{te}}}{\tau_e}A^e_{\parallel}\delta_{\ell0}\delta_{m0}-\frac{v_{\mathrm{te}}}{\tau_e}A^e_{\parallel}\delta_{\ell1}\delta_{m0}\right]\nabla_{\parallel}\ln B. \tag*{(63)}

Finally, the equation we would solve is

(Bband+u3⊗v3)Ge=r,(64)\left(\mathbf{B}_{\mathrm{band}}+\mathbf{u}_3\otimes\mathbf{v}_3\right)\mathbf{G}^{e}=\mathbf{r}, \tag*{(64)}

where Bband\mathbf{B}_{\mathrm{band}} is a banded matrix formed from the coefficients of Gℓ,meG_{\ell,m}^{e}, Gℓ,m+1eG_{\ell,m+1}^{e}, Gℓ,m−1eG_{\ell,m-1}^{e}, Gℓ−1,m+1eG_{\ell-1,m+1}^{e}, and Gℓ+1,m−1eG_{\ell+1,m-1}^{e}, and u3\mathbf{u}_3 and v3\mathbf{v}_3 are

u3=−vte∇∥ln⁡BΔt[vteτeδℓ0δm0  0  vteτeδℓ1δm0  0],(65)\mathbf{u}_3=-v_{te}\nabla_{\parallel}\ln B\Delta t\left[\frac{v_{te}}{\tau_e}\delta_{\ell0}\delta_{m0}\ \ 0\ \ \frac{v_{te}}{\tau_e}\delta_{\ell1}\delta_{m0}\ \ 0\right], \tag*{(65)}

and

v3=[0  −βref2Fnevteδℓ0δm1  0].(66)\mathbf{v}_3=\left[0\ \ -\frac{\beta_{\mathrm{ref}}}{2F}n_ev_{te}\delta_{\ell0}\delta_{m1}\ \ 0\right]. \tag*{(66)}

Hypercollisions

Since the parallel streaming terms are Fourier transformed in GX, there is no inherent parallel dissipation that would normally arise from upwinding in other gyrokinetic codes. Furthermore, in the absence of sophisticated closures, additional dissipation is needed in velocity space to dampen the higher ℓ\ell or mm modes. As a result, GX is normally run with a hypercollision operator which is derived in Mandell et al. [32]. This operator takes the form

\left(\frac{\partial G_{\ell,m}}{\partial t}\right)_{\mathrm{hyp\mathchar`-coll}}=-\nu_{\mathrm{hyp}}m^{p}G_{\ell,m},\qquad(m>2), \tag*{(67)}

where the collision frequency νhyp\nu_{\mathrm{hyp}} is

νhyp=2.5fhypp+1/2mmax⁡p+1/2∣k∥∣vts.(68)\nu_{\mathrm{hyp}}=2.5f_{\mathrm{hyp}}\frac{p+1/2}{m_{\max}^{p+1/2}}\lvert k_{\parallel}\rvert v_{ts}. \tag*{(68)}

Due to the factor of vtsv_{ts}, this term can become stiff for electrons. However, this term is straightforward to handle alongside the parallel streaming terms, as it is also evaluated spectrally in zz. Since this term is only on the main diagonal, it can be directly included in the tridiagonal solver routine.

Remaining Terms

The remaining terms that are not evolved implicitly correspond physically to the magnetic and curvature drifts, drive terms from the temperature and density gradients, and the collision operator. Although all of these terms are usually about the same order of magnitude between ions and electrons, the magnetic and curvature drift terms can become stiff due to the prefactor of ωdκ\omega_d^{\kappa} or ωd∇B\omega_d^{\nabla B} which are

iωdκ=(1/B)b^×(b^⋅∇b^)⋅ik⊥,(69)\mathrm{i}\omega_d^{\kappa}=(1/B)\hat{\mathbf{b}}\times(\hat{\mathbf{b}}\cdot\nabla\hat{\mathbf{b}})\cdot\mathrm{i}\mathbf{k}_{\perp}, \tag*{(69)}

and

iωd∇B=(1/B2)b^×∇B⋅ik⊥.(70)\mathrm{i}\omega_d^{\nabla B}=(1/B^2)\hat{\mathbf{b}}\times\nabla B\cdot\mathrm{i}\mathbf{k}_{\perp}. \tag*{(70)}

If the geometry of the fusion device has particularly strong curvature or ∇B\nabla B, or if one wants to investigate electron scale turbulence, the drift terms can dominate the CFL condition instead of the streaming or mirror terms. Indeed, we show in the next section that these drift terms will limit the maximum stable timestep for the tokamak cases. In practice, it is straightforward to treat these terms implicitly together with the mirror terms. However, the k⊥k_{\perp} dependence of ωdκ\omega_d^{\kappa} and ωd∇B\omega_d^{\nabla B} results in NxNyNzN_xN_yN_z matrices that would need to be stored (as opposed to just NzN_z mirror matrices). This memory cost is prohibitively expensive on a single GPU, even for banded matrices. While we present some preliminary results of implicit evaluation of these terms in Appendix § 11, we leave a full implementation to future work.

Stationary Iteration

The approximations introduced in § 4.2 and §4.3 to invert the streaming operator can introduce significant errors that degrade both the accuracy and stability of the numerical method. To address these issues, we apply a simple stationary iteration to improve the solution at each implicit step.

Specifically, at each IMEX timestep, we solve a linear system of the form

LG=r,(71)\mathbf{L}\mathbf{G} = \mathbf{r}, \tag*{(71)}

where L\mathbf{L} is the linear operator and r\mathbf{r} is the right-hand side. Rather than inverting L\mathbf{L} directly, we introduce a preconditioner P≈L\mathbf{P} \approx\mathbf{L}, and rewrite the equation as

(L−P+P)G=r,(72)(\mathbf{L} - \mathbf{P} + \mathbf{P})\mathbf{G} = \mathbf{r}, \tag*{(72)}

which can be rearranged into a fixed-point form:

PG=(P−L)G+r.(73)\mathbf{P}\mathbf{G} = (\mathbf{P} - \mathbf{L})\mathbf{G} + \mathbf{r}. \tag*{(73)}

This motivates the following iterative update (where the subscript index indicates the iteration step number, not the time index):

Gn+1=(I−P−1L)Gn+P−1r.(74)\mathbf{G}_{n+1} = (\mathbf{I} - \mathbf{P}^{-1}\mathbf{L})\mathbf{G}_n + \mathbf{P}^{-1}\mathbf{r}. \tag*{(74)}

This scheme converges when the spectral radius of I−P−1L\mathbf{I} - \mathbf{P}^{-1}\mathbf{L} is less than one. When the preconditioner P\mathbf{P} is a good approximation to L\mathbf{L}, then P−1L≈I\mathbf{P}^{-1}\mathbf{L} \approx\mathbf{I}, and convergence is typically rapid. In the cases of § 4.2 and §4.3, B\mathbf{B} is the linear operator, and the derived simplified matrices represent the preconditioner.

While this approach is simple and effective, more advanced iterative solvers such as GMRES or BiCGSTAB may provide more robust convergence. These methods, along with a suite of IMEX integrators, are available in the SUNDIALS library [23, 4, 18, 35]. Interfacing GX with SUNDIALS will be the subject of future work. Importantly, if the current algorithm converges using basic stationary iteration, significant performance improvements can be expected by switching to Krylov methods.

Operator-Split Preconditioning

The ARK methods used in this work require that the parallel streaming and magnetic mirror terms be combined and inverted together. However, the magnetic mirror terms are not Fourier-transformed and thus cannot be easily combined with the parallel streaming operator into a single matrix. To address this, we apply an operator-splitting approximation.

Ideally, we would invert a matrix of the form

I−ΔtB−ΔtC,(75)\mathbf{I} - \Delta t\mathbf{B} - \Delta t\mathbf{C}, \tag*{(75)}

where B\mathbf{B} represents the parallel streaming operator and C\mathbf{C} represents the mirror force operator (or similar). Rather than constructing and inverting the combined matrix, we approximate the inverse by sequentially applying each operator:

(I−ΔtB)(I−ΔtC)=I−ΔtB−ΔtC+(Δt)2BC.(76)(\mathbf{I} - \Delta t\mathbf{B})(\mathbf{I} - \Delta t\mathbf{C}) = \mathbf{I} - \Delta t\mathbf{B} - \Delta t\mathbf{C} + (\Delta t)^2 \mathbf{B}\mathbf{C}. \tag*{(76)}

This approximation introduces a local error of second order in Δt\Delta t due to the (Δt)2BC(\Delta t)^2 \mathbf{B}\mathbf{C} term. To reduce this error, we employ an ‘outer’ stationary iteration that simultaneously includes both the streaming and mirror operators in the iteration loop, using the operator split approximation as the preconditioner.

For the examples shown in this paper, we find that this combined iteration eliminates the need for a separate stationary iteration for the streaming term alone, thereby providing a simple approximation for the full implicit solve.

Numerical Results

In this section, we present results from several simulations that test our implicit methods. The simulations in §5.1 are conducted in slab geometry to test the implicit treatment of the parallel streaming terms. The simulations in §5.2 are performed in toroidal geometry to test the combined inversion of the parallel streaming and mirror terms. We initially focus on evaluating the accuracy of the implicit methods before giving a more detailed discussion of the timings in §5.4.

Slab Geometry

To test our preconditioner for the parallel streaming terms, we start in slab geometry, where the mirror and drift terms are zero. The geometry also includes finite magnetic shear so that the perpendicular wavenumbers k⊥k_\perp vary along the field-line.

The results in Figure 3a show the linear growth rates and frequencies in this slab geometry. The magnetic shear is s^=(1/q) dq/dψ=0.8\hat{s} = (1/q) \, \mathrm{d}q/\mathrm{d}\psi= 0.8, which is comparable to shear in the core of a tokamak. The simulations using the Lie-Trotter and ARK methods employ a timestep of 25 and 50 times larger, respectively, than that used in the explicit scheme. As seen in the plot, the agreement between all three methods is excellent. Although the Lie-Trotter method is numerically stable with a 50 times greater timestep, there is worse agreement.

Linear growth rates and frequencies for the explicit RK3, Lie-Trotter, and ARK methods in slab geometry

Figure 3. The linear growth rates and frequencies for the explicit RK3 (blue), the Lie-Trotter (orange), and the ARK (green) methods in slab geometry. The left plot is for the electrostatic case, while the right plot is at β=4%\beta= 4\%. The ‘x’ indicates the ratio of the IMEX timestep to the explicit timestep.

To test the electromagnetic implicit scheme, we set β=4%\beta= 4\% and reran the simulations with the same magnetic geometry. The results are shown in Figure 3b. As in the electrostatic case, the IMEX methods achieve good accuracy even when including electromagnetic fluctuations.

Finally, we run an electrostatic nonlinear simulation, and plot the resulting turbulent ion heat fluxes in Figure 4. The plot shows excellent agreement in the heat fluxes between the RK3 and ARK methods. However, even at a reduced timestep of only 25 times the explicit method, the Lie-Trotter method does not recover a similar average heat flux.

Ion heat flux for the nonlinear electrostatic slab case

Figure 4. The ion heat flux for the nonlinear electrostatic slab case between the explicit RK3 (blue), Lie-Trotter (orange), and ARK (green) methods. The electron heat flux in this case is negligible, and so not plotted. The ‘x’ indicates the ratio of the IMEX timestep to the explicit timestep.

Tokamak Geometry

In this section, we compare the different schemes in tokamak-like geometry. The geometry we use is the ‘Cyclone Base Case’ (CBC), which is commonly used to benchmark gyrokinetic codes ([15]).

The results in Figure 5 show the linear growth rates and frequencies for each simulation. Unlike in slab geometry, in tokamak geometry the implicit methods cannot use a 60 times larger timestep than that of the explicit method. As will be explained in §5.4, the magnetic and curvature drift terms dominate the CFL constraint for the implicit methods. In this case, we can only use a 7–10 times greater timestep. Nevertheless, at this timestep, we achieve good agreement between all of the schemes for both the electrostatic and electromagnetic simulations.

Linear growth rates and frequencies for the explicit RK3, Lie-Trotter, and ARK methods

Figure 5. The linear growth rates and frequencies for the explicit RK3 (blue), the Lie-Trotter (orange), and the ARK (green) methods for the cyclone base-case. The left plot is for an electrostatic simulation, while the right is at β=1.5%\beta= 1.5\%. The ‘x’ indicates the ratio of the IMEX timestep to the explicit timestep.

As a final test, we run nonlinear electrostatic simulations of the CBC. In this case, due to the magnetic/curvature drifts and the nonlinear term, the IMEX methods can only take up to a 4 times larger timestep. The resulting heat fluxes are shown in Figure 6. As seen in the plot, there is good agreement in the heat fluxes between the methods.

Ion and electron heat fluxes for the electrostatic cyclone base-case

Figure 6. The ion (solid) and electron (dashed) heat fluxes for the electrostatic cyclone base-case for the explicit RK3 (blue), the Lie-Trotter (orange), and the ARK (green) methods. The ‘x’ indicates the ratio of the IMEX timestep to the explicit timestep.

Convergence of the Iterative Method for the Streaming Terms

To analyse the convergence of the iterative method used to invert the parallel streaming operator, we perform the electrostatic linear and nonlinear slab simulations with different numbers of stationary iterations using the ARK method. Figure 7a shows the relative error in the growth rate at ky=0.4k_y = 0.4 versus the number of iterations. The errors decrease rapidly, reaching about a 0.1%0.1\% error at 4 iterations. The decrease in the error follows approximately a 1/N21/N^2 scaling, where NN is the number of iterations. Figure 7b shows the nonlinear heat flux traces for different numbers of iterations. We observe the steady-state heat flux from the implicit simulations consistently converges towards the heat flux from the explicit simulation as the number of iterations increases. Overall, this demonstrates that our simple preconditioned iterative solver is effective, and could have potential for further improvement using more sophisticated iterative methods.

Growth rate error and nonlinear heat fluxes

Figure 7. (a) The growth rate error at ky=0.4k_y = 0.4 versus the number of iterations for the iterative method for the streaming terms. The blue curve is the actual error, while the orange dashed line is N−2N^{-2} that the error seems to follow. (b) The nonlinear heat fluxes for the ARK method for different number of iterations. The average heat flux eventually matches that from the explicit method at 5 iterations.

Timings

Table 1 shows the timings of the different simulations shown in the previous sections. For the slab cases, the Lie-Trotter and ARK methods maintain a speedup of over an order of magnitude. This speedup reduces in the CBC cases in part due to the CFL condition being set by the explicit drift terms. Following the procedure in Mandell et al. [32], we expect the CFL condition to be set by a combination of the drift and ion mirror terms

ωmax⁡=ωdrifts,max+ωion,mirror,max.(77)\omega_{\max} = \omega_{\mathrm{drifts,max}} + \omega_{\mathrm{ion,mirror,max}}. \tag*{(77)}

The maximum drift frequency can be estimated as

ωdrifts,max∼ωd,max⁡κv∥,max⁡2+ωd,max⁡∇B(μB)max⁡,(78)\omega_{\mathrm{drifts,max}} \sim\omega^{\kappa}_{d,\max} v_{\parallel,\max}^{2} + \omega^{\nabla B}_{d,\max} (\mu B)_{\max}, \tag*{(78)}

where v∥,max⁡v_{\parallel,\max} and (μB)max⁡(\mu B)_{\max} scale like

v∥,max⁡∼2He,c,max,(79)v_{\parallel,\max} \sim2\mathrm{H}_{\mathrm{e,c,max}}, \tag*{(79)}

and

(μB)max⁡∼Lc,max,(80)(\mu B)_{\max} \sim L_{\mathrm{c,max}}, \tag*{(80)}

and He,c,max\mathrm{H}_{\mathrm{e,c,max}} and Lc,maxL_{\mathrm{c,max}} are the maximum values of the Gauss-Hermite and Gauss-Laguerre collocation points. For the linear CBC simulations in this work, ωdrifts,max≈200k⊥\omega_{\mathrm{drifts,max}} \approx200k_{\perp}, while ωion,mirror,max≈1.5\omega_{\mathrm{ion,mirror,max}} \approx1.5. Since ωdrifts,max\omega_{\mathrm{drifts,max}} scales with the perpendicular wavenumber k⊥k_{\perp}, the maximum timestep in the electromagnetic CBC simulation was slightly larger than in the electrostatic simulation, as it had a smaller maximum kyk_y. For ky≈1k_y \approx1, we expect a maximum stable timestep of about 1/ωmax⁡≈5.0×10−3 vti/a1/\omega_{\max} \approx5.0 \times10^{-3}\ v_{\mathrm{ti}}/a.

In comparison, the streaming terms would contribute a term that scales like

ωstream,max∼(b^⋅∇z)kz,max⁡v∥,max⁡vts.(81)\omega_{\mathrm{stream,max}} \sim\left(\hat{b} \cdot\nabla_z\right) k_{z,\max}v_{\parallel,\max}v_{ts}. \tag*{(81)}

For electrons, this term is approximately ωstream,max≈1980\omega_{\mathrm{stream,max}} \approx1980, making the maximum timestep 1/ωstream,max≈5.0×10−4 vti/a1/\omega_{\mathrm{stream,max}} \approx5.0 \times10^{-4}\,v_{ti}/a. Therefore, we expect that the implicit methods can only use about a 10 times greater timestep than the explicit method, which is demonstrated in the CBC simulations.

Finally, we find that the most expensive operation in our IMEX schemes is inverting the matrices for the mirror operator. This cost increases as we increase the number of iterations, particularly for the ARK scheme. To demonstrate this effect, we run the CBC simulations again with the ARK method, but treat the mirror term explicitly. As shown in Figure 8 and Table 1, we can achieve comparable accuracy and maximum stable timestep at a reduced wall-time. Therefore, future work will investigate different sparse linear algebra libraries to reduce this cost. Nevertheless, for the nonlinear case, we can still achieve approximately a factor of 2 speedup.

Linear growth rates and frequencies for the explicit RK3 and ARK methods

Figure 8. The linear growth rates and frequencies for the explicit RK3 (blue) and the ARK methods for the cyclone base-case. The left plot is for an electrostatic simulation, while the right is at β=1.5%\beta= 1.5\%. The ‘x’ indicates the ratio of the IMEX timestep to the explicit timestep. In these cases, the ARK method evaluates the mirror term explicitly.

SchemeΔt (vti/a)\Delta t\ (v_{ti}/a)Δt\Delta t FactorWall Time (min)Speedup
Sheared Slab (ES)
RK38.0×10−48.0 \times10^{-4}18.681.00
Lie-Trotter (3 stream iters)2.0×10−22.0 \times10^{-2}250.7910.99
ARK (3 stream iters)4.0×10−24.0 \times10^{-2}500.5216.69
Sheared Slab (EM)
RK38.0×10−48.0 \times10^{-4}18.191.00
Lie-Trotter (3 stream iters)2.0×10−22.0 \times10^{-2}250.968.53
ARK (3 stream iters)4.0×10−24.0 \times10^{-2}500.6313.0
Sheared Slab (Nonlinear, ES)
RK38.0×10−48.0 \times10^{-4}1489.491.00
Lie-Trotter (8 stream iters)2.0×10−22.0 \times10^{-2}2539.8812.27
ARK (5 stream iters)4.0×10−24.0 \times10^{-2}5023.5620.78
CBC (ES)
RK37.0×10−47.0 \times10^{-4}15.851.00
Lie-Trotter (2 stream iters)5.0×10−35.0 \times10^{-3}7.141.493.93
ARK (1 outer iters)5.0×10−35.0 \times10^{-3}7.141.573.73
ARK (1 stream iters, explicit mirror)5.0×10−35.0 \times10^{-3}7.141.254.68
Lie-Trotter (4 stream iters, implicit drifts)1.0×10−31.0 \times10^{-3}14.291.055.57
CBC (EM)
RK39.0×10−49.0 \times10^{-4}12.561.00
Lie-Trotter (3 stream iters)7.0×10−37.0 \times10^{-3}7.781.122.29
ARK (2 outer iters)9.0×10−39.0 \times10^{-3}10.01.242.06
ARK (2 stream iters, explicit mirror)9.0×10−39.0 \times10^{-3}10.00.693.71
CBC (Nonlinear, ES)
RK31.0×10−31.0 \times10^{-3}1617.971.00
Lie-Trotter (3 stream iters)4.0×10−34.0 \times10^{-3}4.0217.832.84
ARK (2 outer iters)4.0×10−34.0 \times10^{-3}4.0331.581.86

Table 1. Simulation timings for each scheme and configuration. ES and EM denote electrostatic and electromagnetic simulations, respectively.

Conclusions

In this work, we developed implicit-explicit methods to accelerate kinetic-electron simulations for the GX gyrokinetics code. Since GX is GPU-native, we have designed our methods to be low-memory by taking advantage of banded and low-rank structure of the GX equations to efficiently invert the implicit operators. We showed that our methods can achieve up to the maximum theoretical timestep while still maintaining an improvement in the wall-clock time for both linear and nonlinear simulations. However, this scheme uses approximate iterative methods. While we show that the iterative method for the streaming terms is efficient and accurate, the operator-split preconditioned iterative method can be more expensive due to the cost of direct banded solvers. These methods are also less effective in cases where the drift terms dominate the CFL condition.

There are several directions for possible future work. We first aim to implement these methods within the SUNDIALS framework. Doing so should improve the relatively simple iterative methods shown here, and improve the efficiency of the ARK methods. The

SUNDIALS library also includes robust methods for error-checking and adaptive timesteps. Additionally, we will include a full implementation of the magnetic and curvature drift terms in the implicit operator, which would further relax the CFL constraint.

Acknowledgments

P. S. K. would like to acknowledge F. Parra Diaz for several helpful conversations. This work was supported in part by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Scientific Discovery through Advanced Computing (SciDAC) Program through the Computational Evaluation and Design of Actuators for Core-Edge Integration (CEDA) project, and through the Frameworks, Algorithms and Software Technologies for Mathematics (FASTMath) Institute. This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Department of Energy Computational Science Graduate Fellowship under Award Number DE-SC0024386. P. S. K. used ChatGPT (OpenAI) and Claude (Anthropic) to aid in software devel-

kx,ky,x,yk_x, k_y, x, y∇,z\nabla, zv∥v_{\parallel}μ\muω,t\omega, tψ\psi
ρref\rho_{\mathrm{ref}}aNa_Nvth,sv_{\mathrm{th},s}vth,s2/BNv_{\mathrm{th},s}^{2}/B_Nvth,ref/aNv_{\mathrm{th},\mathrm{ref}}/a_NaN2BNa_N^{2}B_N

caption: Table 2. Normalizing the quantities for the independent variables. Taken from Mandell et al. [32].

:::

hhΦ\PhiA∥A_{\parallel}δB∥\delta B_{\parallel}
ρ∗n0/vth,ref3\rho_{\ast}n_0/v_{\mathrm{th},\mathrm{ref}}^{3}ρ∗Tref/qref\rho_{\ast}T_{\mathrm{ref}}/q_{\mathrm{ref}}ρ∗(ρrefBN)\rho_{\ast}(\rho_{\mathrm{ref}}B_N)ρ∗BN\rho_{\ast}B_N

Table 3. Normalizing quantities for dependent variables. Taken from [32].

opment and debugging and in revising text. P. S. K reviewed any AI-assisted code and text, and the manuscript has been reviewed by the authors, who take full responsibility for its content.

Normalizations

The normalizations used in this work are the same as in Mandell et al. [32] and are summarized in tables 2 and 3. The normalizing quantities in table 2 are arbitrary, but have common choices. For example, the normalizing length aNa_N is typically either the minor or major radius. The reference magnetic field BNB_N is typically taken to be the vacuum magnetic field at the magnetic axis or is defined in terms of the enclosed toroidal magnetic flux. The reference thermal speed vth,refv_{\mathrm{th},\mathrm{ref}} is defined in terms of a reference temperature and mass vth,ref=Tref/Mrefv_{\mathrm{th},\mathrm{ref}} = \sqrt{T_{\mathrm{ref}}/M_{\mathrm{ref}}}. The corresponding reference gyroradius and cyclotron frequency are then ρref≡vth,ref/Ωref\rho_{\mathrm{ref}} \equiv v_{\mathrm{th},\mathrm{ref}}/\Omega_{\mathrm{ref}} and Ωref=qrefBN/Mrefc\Omega_{\mathrm{ref}} = q_{\mathrm{ref}}B_N/M_{\mathrm{ref}}c, respectively. Specific species quantities such as the temperature, mass, thermal speed, charge, and gyroradius are then normalized with respect to these reference values: τs=Ts/Tref\tau_s = T_s/T_{\mathrm{ref}}, ms=Ms/Mrefm_s = M_s/M_{\mathrm{ref}}, vts=vth,s/vth,refv_{ts} = v_{\mathrm{th},s}/v_{\mathrm{th},\mathrm{ref}}, Zs=qs/qrefZ_s = q_s/q_{\mathrm{ref}}, and ρs=(vth,s/Ωs)/ρref\rho_s = (v_{\mathrm{th},s}/\Omega_s)/\rho_{\mathrm{ref}}.

Long-Wavelength Dispersion Relation In Slab Geometry

In unsheared slab geometry, we consider a simplified limit with no gradients or collisions, stationary ions, and isothermal electrons. In the long-wavelength limit, the GX equations for the electron moments reduce to

∂G0e∂t+vte∇∥(G1e+A∥)=0,(82)\frac{\partial G_0^e}{\partial t} + v_{te}\nabla_{\parallel}\left(G_1^e + A_{\parallel}\right) = 0, \tag*{(82)}

and

∂G1e∂t+vte∇∥(G0e−Φ)=0,(83)\frac{\partial G_1^e}{\partial t} + v_{te}\nabla_{\parallel}\left(G_0^e - \Phi\right) = 0, \tag*{(83)}

with the corresponding field equations given by

Φ=−G0e(k⊥ρi)2,(84)\Phi= -\frac{G_0^e}{\left(k_{\perp}\rho_i\right)^2}, \tag*{(84)}

and

A∥=−β^vteG1ek⊥2+β^(85)A_{\parallel}=-\frac{\hat{\beta}v_{te}G_{1}^{e}}{k_{\perp}^{2}+\hat{\beta}} \tag*{(85)}

where we define β^=βmi/2me\hat{\beta}=\beta m_{i}/2m_{e}.

Fourier transforming and combining the moment and field equations yields the dispersion relation:

ω2=vte2k∥2(1+1k⊥2)(1−β^k⊥2+β^).(86)\omega^{2}=v_{te}^{2}k_{\parallel}^{2}\left(1+\frac{1}{k_{\perp}^{2}}\right)\left(1-\frac{\hat{\beta}}{k_{\perp}^{2}+\hat{\beta}}\right). \tag*{(86)}

This simplifies to the standard dispersion relation for the kinetic Alfvén wave:

ω2=vte2k∥2(1+k⊥2k⊥2+β^).(87)\omega^{2}=v_{te}^{2}k_{\parallel}^{2}\left(\frac{1+k_{\perp}^{2}}{k_{\perp}^{2}+\hat{\beta}}\right). \tag*{(87)}

In the electrostatic limit (β^=0\hat{\beta}=0), this further reduces to the ωH\omega_{H} mode:

ωH2=vte2k∥21+k⊥2k⊥2.(88)\omega_{H}^{2}=v_{te}^{2}k_{\parallel}^{2}\frac{1+k_{\perp}^{2}}{k_{\perp}^{2}}. \tag*{(88)}

This expression shows that the mode frequency increases significantly as k⊥→0k_{\perp}\to0. Therefore, to account for the fastest waves, we take the smallest k⊥k_{\perp} along the field line in constructing our preconditioner. Notably, explicit gyrokinetic codes often retain a small but nonzero β\beta even for nominally electrostatic simulations in order to avoid this high-frequency mode.

For more general geometry and nonzero β^\hat{\beta}, we approximate the dispersion relation as

ω2=vte2k∥2(1+1(k⊥2)m)(1−β^(k⊥2+β^)M),(89)\omega^{2}=v_{te}^{2}k_{\parallel}^{2}\left(1+\frac{1}{(k_{\perp}^{2})_{m}}\right)\left(1-\frac{\hat{\beta}}{(k_{\perp}^{2}+\hat{\beta})_{M}}\right), \tag*{(89)}

where (k⊥2)m(k_{\perp}^{2})_{m} and (k⊥2+β^)M(k_{\perp}^{2}+\hat{\beta})_{M} represent specific values chosen along the field line. Again, k⊥ρk_{\perp}\rho is minimized, while k⊥2+β^k_{\perp}^{2}+\hat{\beta} is maximized to result in the maximum possible frequency

Sherman-Morrison Formula

The matrices we invert in this work are of the form

(A+u⊗v)G=r,(90)(A+u\otimes v)G=r, \tag*{(90)}

where uu and vv contain information about the fields. The Sherman-Morrison formula states that the solution to this equation is

G=y−(v⋅y1+v⋅z)z,(91)G=y-\left(\frac{v\cdot y}{1+v\cdot z}\right)z, \tag*{(91)}

where yy and zz are solutions to the equations

Ay=r,Az=u.(92)Ay=r,\qquad Az=u. \tag*{(92)}

When including multiple fields, like Φ\Phi and A∥A_{\parallel}, the equation becomes

(A+u1⊗v1+u2⊗v2)G=r.(93)(A+u_{1}\otimes v_{1}+u_{2}\otimes v_{2})G=r. \tag*{(93)}

Following the procedure in Tatsuno & Dorland [39] and Barnes et al. [5], we rewrite the above equation as

(A1+u2⊗v2)G=r,(94)(\mathbf{A}_1+\mathbf{u}_2\otimes\mathbf{v}_2)\mathbf{G}=\mathbf{r}, \tag*{(94)}

where A1=A+u1⊗v1\mathbf{A}_1=\mathbf{A}+\mathbf{u}_1\otimes\mathbf{v}_1. The solution is then

G=y2−(v2⋅y21+v2⋅z2)z2,(95)\mathbf{G}=\mathbf{y}_2-\left(\frac{\mathbf{v}_2\cdot\mathbf{y}_2}{1+\mathbf{v}_2\cdot\mathbf{z}_2}\right)\mathbf{z}_2, \tag*{(95)}

where y2\mathbf{y}_2 and z2\mathbf{z}_2 are solutions to the equations

A1y2=r,A1z2=u2.(96)\mathbf{A}_1\mathbf{y}_2=\mathbf{r},\quad\mathbf{A}_1\mathbf{z}_2=\mathbf{u}_2. \tag*{(96)}

To solve for y2\mathbf{y}_2 and z2\mathbf{z}_2, we apply the Sherman-Morrison formula a second time

y2=y1−(v1⋅y11+v1⋅z1)z1,(97)\mathbf{y}_2=\mathbf{y}_1-\left(\frac{\mathbf{v}_1\cdot\mathbf{y}_1}{1+\mathbf{v}_1\cdot\mathbf{z}_1}\right)\mathbf{z}_1, \tag*{(97)}

and

z2=z′−(v1⋅z′1+v1⋅z1)z1,(98)\mathbf{z}_2=\mathbf{z}'-\left(\frac{\mathbf{v}_1\cdot\mathbf{z}'}{1+\mathbf{v}_1\cdot\mathbf{z}_1}\right)\mathbf{z}_1, \tag*{(98)}

where y1\mathbf{y}_1, z1\mathbf{z}_1, and z′\mathbf{z}' are solutions to

Ay1=r,Az1=u1,Az′=u2.(99)\mathbf{A}\mathbf{y}_1=\mathbf{r},\quad\mathbf{A}\mathbf{z}_1=\mathbf{u}_1,\quad\mathbf{A}\mathbf{z}'=\mathbf{u}_2. \tag*{(99)}

Implicit Treatment of the Magnetic and Curvature Drift Terms

When expanded in terms of the Hermite-Laguerre basis functions, the magnetic and curvature drift terms are

iτsZsωd∇B[(ℓ+1)Hℓ+1,ms+(2ℓ+1)Hℓ,ms+ℓHℓ−1,ms],(100)\mathrm{i}\frac{\tau_s}{Z_s}\omega_d^{\nabla B}\left[(\ell+1)H^s_{\ell+1,m}+(2\ell+1)H^s_{\ell,m}+\ell H^s_{\ell-1,m}\right], \tag*{(100)}

and

iτsZsωdκ[(m+1)(m+2)Hℓ,m+2s+(2m+1)Hℓ,ms+m(m−1)Hℓ,m−2s].(101)\mathrm{i}\frac{\tau_s}{Z_s}\omega_d^\kappa\left[\sqrt{(m+1)(m+2)}H^s_{\ell,m+2}+(2m+1)H^s_{\ell,m}+\sqrt{m(m-1)}H^s_{\ell,m-2}\right]. \tag*{(101)}

The matrix corresponding to this term can be written as

Bdrifts=Bdrifts,band+uΦ⊗vΦ+uA∥⊗vA∥,(102)\mathbf{B}_{\mathrm{drifts}}=\mathbf{B}_{\mathrm{drifts,band}}+\mathbf{u}_{\Phi}\otimes\mathbf{v}_{\Phi}+\mathbf{u}_{A_\parallel}\otimes\mathbf{v}_{A_\parallel}, \tag*{(102)}

where Bdrifts,band\mathbf{B}_{\mathrm{drifts,band}} is a banded matrix of bandwidth MM. The field terms are contained in the vectors uΦ\mathbf{u}_{\Phi}, vΦ\mathbf{v}_{\Phi}, uA∥\mathbf{u}_{A_\parallel}, and vA∥\mathbf{v}_{A_\parallel}, whose entries are defined as

(uΦ)s,ℓ,m=iωd∇Bδm0[(ℓ+1)Jℓ+1+(2ℓ+1)Jℓ+ℓJℓ−1]+iωdκ[(2m+1)Jℓδm0],(103)(\mathbf{u}_{\Phi})_{s,\ell,m}=\mathrm{i}\omega_d^{\nabla B}\delta_{m0}\left[(\ell+1)\mathcal{J}_{\ell+1}+(2\ell+1)\mathcal{J}_{\ell}+\ell\mathcal{J}_{\ell-1}\right] +\mathrm{i}\omega_d^\kappa\left[(2m+1)\mathcal{J}_{\ell}\delta_{m0}\right], \tag*{(103)}
(vΦ)s,ℓ,m=Zsns∑sZs2nsτs(1−∑ℓ=0Nℓ(Jℓs)2)Jℓsδm0,(104)(\mathbf{v}_{\Phi})_{s,\ell,m}= \frac{Z_s n_s}{\sum_s\frac{Z_s^2n_s}{\tau_s}\left(1-\sum_{\ell=0}^{N_\ell}(\mathcal{J}^s_\ell)^2\right)} \mathcal{J}^s_\ell\delta_{m0}, \tag*{(104)}
(uA∥)s,ℓ,m=−iωd∇Bvtsδm1[(ℓ+1)Jℓ+1+(2ℓ+1)Jℓ+ℓJℓ−1]−iωdκvte[(2m+1)Jℓδm1+m(m−2)Jℓδm3],(105)(\mathbf{u}_{A_\parallel})_{s,\ell,m}=-\mathrm{i}\omega_d^{\nabla B}v_{ts}\delta_{m1}\left[(\ell+1)\mathcal{J}_{\ell+1}+(2\ell+1)\mathcal{J}_{\ell}+\ell\mathcal{J}_{\ell-1}\right] -\mathrm{i}\omega_d^\kappa v_{te}\left[(2m+1)\mathcal{J}_{\ell}\delta_{m1}+\sqrt{m(m-2)}\mathcal{J}_{\ell}\delta_{m3}\right], \tag*{(105)}

and

(vA∥)s,ℓ,m=βref2(k⊥2+βref2∑sZs2nsms∑ℓ=0Nℓ(Jℓs)2)ZsnsvtsJℓsδm1.(106)(\mathbf{v}_{A_\parallel})_{s,\ell,m}= \frac{\beta_{\mathrm{ref}}}{2\left(k_\perp^2+\frac{\beta_{\mathrm{ref}}}{2}\sum_s\frac{Z_s^2n_s}{m_s}\sum_{\ell=0}^{N_\ell}(\mathcal{J}^s_\ell)^2\right)} Z_s n_s v_{ts}\mathcal{J}^s_\ell\delta_{m1}. \tag*{(106)}

Since this term is evaluated in real space in zz, Bdrifts\mathbf{B}_{\mathrm{drifts}} can be combined with the mirror matrix in the implicit methods.

To test the implicit treatment of the magnetic drifts, we again run the linear electrostatic CBC simulation. We use the Lie-Trotter method, but the split for the mirror term now also contains the drift terms. For both the mirror and drift terms, both ions and electrons are treated implicitly, and we retain full FLR-effects in the implicit operator. A plot of the linear growth rates and frequencies is in Figure 9. In this case, we are able to double the timestep while maintaining good accuracy. The maximum stable timestep is larger, but at the cost of further reduced accuracy. We do not show results using the ARK method, as we find that the operator-split preconditioner requires an increased number of iterations at a larger timestep to remain stable, which offsets the advantage of the increased timestep.

Linear growth rates and frequencies for the explicit RK3 and Lie-Trotter methods

Figure 9. The linear growth rates and frequencies for the explicit RK3 (blue) method and the Lie-Trotter method (orange) scheme for the electrostatic cyclone base-case. The magnetic and curvature drift terms are treated implicitly in this case.

References

  1. [1]Abdelfattah, Ahmad, Tomov, Stanimire, Luszczek, Piotr, Anzt, Hartwig & Dongarra, Jack 2023 GPU-based LU Factorization and Solve on Batches of Matrices with Band Structure. In ACM International Conference Proceedings Series, pp. 1672–1679. Association for Computing Machinery.DOI
  2. [2]Abel, I G, Plunk, G G, Wang, E, Barnes, M, Cowley, S C, Dorland, W & Schekochihin, A A 2013 Multiscale gyrokinetics for rotating tokamak plasmas: fluctuations, transport and energy flows. Reports on Progress in Physics 76 (11), 116201.DOI
  3. [3]Antonsen Jr., Thomas M & Lane, Barton 1980 Kinetic equations for low frequency instabilities in inhomogeneous plasmas. The Physics of Fluids 23 (6), 1205–1214.DOI
  4. [4]Balos, Cody J, Gardner, David J, Woodward, Carol S & Reynolds, Daniel R 2021 Enabling GPU accelerated computing in the SUNDIALS time integration library. Parallel Computing 108, 102836.DOI
  5. [5]Barnes, M, Abel, I G, Dorland, W, Ernst, D R, Hammett, G W, Ricci, P, Rogers, B N, Schekochihin, A A & Tatsuno, T 2009 Linearized model Fokker–Planck collision operators for gyrokinetic simulations. II. Numerical implementation and tests. Physics of Plasmas 16 (7), 072107.DOI
  6. [6]Barnes, M., Parra, F. I. & Landreman, M. 2019 stella: An operator-split, implicit–explicit δf-gyrokinetic code for general magnetic field configurations. Journal of Computational Physics 391, 365–380.DOI
  7. [7]Beer, Michael Alan 1995 Gyrofluid Models of Turbulent Transport in Tokamaks. PhD thesis.
  8. [8]Boyd, John P. 2001 Chebyshev and Fourier spectral methods. Dover Publications.
  9. [10]Candy, Jeff & Waltz, R. E. 2003 An Eulerian gyrokinetic-Maxwell solver. Journal of Computational Physics 186 (2), 545–581.DOI
  10. [11]Catto, P J 1978 Linearized gyro-kinetics. Plasma Physics 20 (7), 719–722.DOI
  11. [12]Cooper, G J & Sayfy, A 1980 Additive methods for the numerical solution of ordinary differential equations. Mathematics of Computation 35 (152), 1159–1172.
  12. [15]Dimits, A M, Bateman, G, Beer, M A, Cohen, B I, Dorland, W, Hammett, G W, Kim, C, Kinsey, J E, Kotschenreuther, M, Kritz, A H, Lao, L L, Mandrekas, J, Nevins, W M, Parker, S E, Redd, A J, Shumaker, D E, Sydora, R & Weiland, J 2000 Comparisons and physics basis of tokamak transport models and turbulence simulations. Physics of Plasmas 7 (3), 969–983.
  13. [16]Dorland, W., Jenko, F., Kotschenreuther, M. & Rogers, B. N. 2000 Electron Temperature Gradient Turbulence. Physical Review Letters 85 (26), 5579–5582.pubmed.ncbi.nlm.nih.gov/11136051
  14. [17]Frieman, E A & Chen, Liu 1982 Nonlinear gyrokinetic equations for low-frequency electromagnetic waves in general plasma equilibria. The Physics of Fluids 25 (3), 502–508.
  15. [18]Gardner, David J, Reynolds, Daniel R, Woodward, Carol S & Balos, Cody J 2022 Enabling New Flexibility in the SUNDIALS Suite of Nonlinear and Differential/Algebraic Equation Solvers. ACM Trans. Math. Softw. 48 (3).DOI
  16. [19]Giraldo, F X, Kelly, J F & Constantinescu, E M 2013 Implicit-Explicit Formulations of a Three-Dimensional Nonhydrostatic Unified Model of the Atmosphere (NUMA). SIAM Journal on Scientific Computing 35 (5), B1162–B1194.DOI
  17. [20]Gottlieb, Sigal, Shu, Chi-Wang & Tadmor, Eitan 2001 Strong Stability-Preserving High-Order Time Discretization Methods. SIAM Review 43 (1), 89–112.DOI
  18. [23]Hindmarsh, Alan C, Brown, Peter N, Grant, Keith E, Lee, Steven L, Serban, Radu, Shumaker, Dan E & Woodward, Carol S 2005 SUNDIALS: Suite of nonlinear and differential/algebraic equation solvers. ACM Trans. Math. Softw. 31 (3), 363–396.
  19. [24]Hinton, F L & Hazeltine, R D 1976 Theory of plasma transport in toroidal confinement systems. Reviews of Modern Physics 48 (2), 239–308.DOI
  20. [25]Horton, W 1999 Drift waves and transport. Reviews of Modern Physics 71 (3), 735–778.DOI
  21. [26]Jenko, F, Dorland, W, Kotschenreuther, M & Rogers, B N 2000 Electron temperature gradient driven turbulence. Physics of Plasmas 7 (5), 1904–1910.DOI
  22. [27]Jorge, Rogerio 2026 GKX: gyrokinetic turbulence and sensitivities in JAX.
  23. [28]Kennedy, Christopher A. & Carpenter, Mark H. 2003 Additive Runge–Kutta schemes for convection–diffusion–reaction equations. Applied Numerical Mathematics 44 (1-2), 139–181.DOI
  24. [30]Kotschenreuther, Mike, Rewoldt, G. & Tang, W. M. 1995b Comparison of initial value and eigenvalue codes for kinetic toroidal plasma instabilities. Computer Physics Communications 88 (2-3), 128–140.
  25. [31]Liewer, Paulett C 1985 Measurements of microturbulence in tokamaks and comparisons with theories of turbulence and anomalous transport. Nuclear Fusion 25 (5), 543.DOI
  26. [32]Mandell, N R, Dorland, W, Abel, I, Gaur, R, Kim, P, Martin, M & Qian, T 2024 GX: a GPU-native gyrokinetic turbulence code for tokamak and stellarator design. Journal of Plasma Physics 90 (4), 905900402.arxiv.org/abs/2209.06731
  27. [33]Mandell, N R, Dorland, W & Landreman, M 2018 Laguerre-Hermite pseudo-spectral velocity formulation of gyrokinetics. J. Plasma Phys 84 (1), 905840108.DOI
  28. [34]Martin, M F, Landreman, M, Xanthopoulos, P, Mandell, N R & Dorland, W 2018 The parallel boundary condition for turbulence simulations in low magnetic shear devices. Plasma Physics and Controlled Fusion 60 (9), 095008.DOI
  29. [35]Reynolds, Daniel R., Gardner, David J., Woodward, Carol S & Chinomona, Rujeko 2023 ARKODE: A Flexible IVP Solver Infrastructure for One-step Methods. ACM Trans. Math. Softw. 49 (2).DOI
  30. [36]Ropp, David L., Shadid, John N. & Ober, Curtis C. 2004 Studies of the accuracy of time integration methods for reaction–diffusion equations. Journal of Computational Physics 194 (2), 544–574.DOI
  31. [37]Rosenbluth, Marshall N 1968 Low-Frequency Limit of Interchange Instability. The Physics of Fluids 11 (4), 869–872.DOI
  32. [39]Tatsuno, T & Dorland, W 2008 Secondary tearingmode in the nonlinear evolution of magnetorotational instability. Astronomische Nachrichten 329 (7), 688–700.DOI
  33. [40]Wootton, A J, Carreras, B A, Matsumoto, H, McGuire, K, Peebles, W A, Ritz, Ch. P, Terry, P W & Zweben, S J 1990 Fluctuations and anomalous transport in tokamaks. Physics of Fluids B: Plasma Physics 2 (12), 2879–2903.DOI

Paper details

vlnc:W_hd5ktdj2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

ChatGPT (OpenAI), Claude (Anthropic)

View statement
Looking for related papers…
Contents