WorksPhysics and Astronomy
Implicit-Explicit Methods for the GX Gyrokinetics Code
Abstract
Gyrokinetic simulations are crucial for understanding turbulent transport in magnetic confinement fusion devices. Unfortunately, these simulations are typically extremely expensive, particularly when kinetically evolving electrons due to their significantly greater thermal speed compared to that of ions. In this work, we present new implicit-explicit timestepping methods for the GX gyrokinetics code to accelerate these simulations where the ion and electron streaming terms as well as the electron mirror terms are treated implicitly, while the remaining terms are treated explicitly. These methods include a lower-order Lie-Trotter splitting method as well as a higher-order additive Runge-Kutta method. Our schemes take advantage of the sparsity of the GX equations, which allows us to efficiently invert the linear operators that arise from the implicit scheme without consuming significant amounts of limited GPU memory. We demonstrate our new integration scheme for both linear (electrostatic and electromagnetic) and nonlinear (electrostatic) simulations in general toroidal geometry, and show that our methods can achieve up to the maximum theoretical linearly stable timestep for these simulations while maintaining a significant improvement in wall-clock time.
Introduction
Tokamaks and stellarators are two of the leading nuclear fusion concepts, and use strong magnetic fields to confine the fusion plasma. Assuming that the plasma is macroscopically stable, tokamaks and stellarators still leak energy and particles due to various transport processes [24, 25]. The largest contributor to these losses is often turbulent transport. Driven by steep temperature and density gradients, turbulent fluctuations rapidly mix the hot and cold regions of the plasma, preventing the plasma from maintaining higher pressures and limiting the overall fusion power gain [31, 40, 15].
One of the most widely-used models for simulating magnetized turbulence is gyrokinetics [3, 17, 2, 11]. Numerous gyrokinetic codes [30, 16, 26, 10, 6, 32] have been developed in the past few decades that can accurately match experimental measurements of turbulent losses as well as temperature and density profiles. Unfortunately, running these gyrokinetic codes can be computationally expensive, often requiring hundreds to thousands of CPU or GPU-hours. Including kinetic electrons exacerbates this cost, as their higher thermal speed (about 60 times greater than a deuterium ion of the same temperature) enforces a stricter Courant–Friedrichs–Lewy (CFL) condition that reduces the maximum allowed timestep. To avoid this issue, an adiabatic electron response is often used, particularly when studying the ion-temperature gradient mode (Coppi et al. 1967; Cowley et al. 1991). However, correctly resolving the electron dynamics is crucial for understanding other important microinstabilities like the trapped-electron mode [37], the kinetic-ballooning mode (Tang et al. 1980), and the microtearing mode (Hazeltine et al. 1975; Guttenfelder et al. 2011).
To overcome this computational restriction, codes like GS2 (Kotschenreuther et al. 1995a; Dorland et al. 2000) and stella (Barnes et al. 2019) have employed implicit or implicit-explicit (IMEX) time-integration methods. GS2 is fully linearly implicit, while stella treats the stiff electron terms implicitly. Here, linearly implicit means that only the linear part of the gyrokinetic equation is treated implicitly, while the nonlinear term appears explicitly and can be evaluated separately through operator splitting. Among the linear terms, only a subset of them are stiff due to the higher electron thermal speed. Alternatively, in CGYRO (Candy et al. 2016), one of the velocity coordinates is truncated, relaxing the CFL constraint. In this paper, we focus on the GX gyrokinetic code (Mandell et al. 2018, 2024). GX is a GPU-native code that solves the radially-local form of gyrokinetics by expanding the distribution function in a Hermite-Laguerre basis in velocity space, allowing for coarser velocity resolution. For our scheme, we treat implicitly the terms related to parallel streaming along field lines for both ions and electrons, as well as the term due to acceleration due to the mirror force for electrons.
Although being GPU-native substantially improves performance, the limited memory on GPUs can be very constraining for implicit methods. Inverting the linear operators that arise in implicit methods often requires significant storage. Indeed, as we explain in § 4.1, the memory cost of the implicit methods implemented in GS2 and stella can make them nearly infeasible to use for the GPU-native GX. However, the pseudo-spectral formulation used in GX yields equations that are sparse in velocity-space. We take advantage of this sparsity to present new algorithms to efficiently invert the implicit operators. We also employ physics-based preconditioners and iterative methods to further reduce the memory cost. We note that a similar Hermite–Laguerre code, GKX, has been recently developed that also utilizes IMEX timestepping [27], but it inverts the implicit operator exactly, using a direct factorization that is significantly more memory expensive than our low-memory approximate inversions.
The paper is organized as follows: In § 2, we introduce the GX code, and the resulting equations that need to be solved. Section 3 describes the time-integration schemes that we use in this work. Our methods to invert the implicit operators are described in § 4, and simulation examples are shown in § 5. The conclusions follow in § 6.
The GX Gyrokinetics Code
Our notation follows that of Mandell et al. (2018, 2024). GX solves the radially local formulation of gyrokinetics. In this approximation, the distribution function for species is split as , where is a perturbation to an equilibrium distribution function that is assumed to be Maxwellian:
where is the equilibrium density of species , the thermal speed is , and and are the equilibrium temperature and mass of species , respectively. GX uses as velocity-space coordinates the velocity parallel to the background magnetic field and . The magnetic moment is , where is the speed perpendicular to the magnetic field, and is the magnitude of the local magnetic field. Such velocity-space coordinates are useful for complex stellarator geometries that may have several magnetic wells along a field line. The perturbation is further split into non-Boltzmann and Boltzmann parts
where is the charge of species , with the speed of light (both GX and this paper are written in Gaussian units), and and are the electrostatic and magnetic vector potential, respectively.
The non-Boltzmann part and the auxiliary distribution function satisfy the gyrokinetic equation
where is the gyroaveraged fluctuating drift velocity, contains the magnetic curvature and drifts, and is the cyclotron frequency. The collision operator is represented by .
The auxiliary distribution function was used to eliminate the time derivative of the gyroaveraged fields. These fields are self-consistently determined using the following field equations
and
Once all quantities are normalized (see Appendix § 8 for normalizations), the non-dimensional gyrokinetic equation is
and the non-dimensional field equations are
and
where is the ratio of the plasma to magnetic pressure for the reference species and is a reference magnetic field strength. These equations are solved in a radially-local flux-tube spatial domain that follows a single field line on a single magnetic flux surface.
In GX the distribution functions and are both expanded in a Fourier-Hermite-Laguerre series:
where the spectral coefficients and are related by
A Fourier representation is used for the spatial dimensions perpendicular to the mean magnetic field (), while Hermite and Laguerre representations are used for the parallel () and perpendicular () velocity domains. A discrete Fourier collocation grid is used for the spatial direction parallel to the field () so that the parallel derivative can be evaluated spectrally. Substituting this expansion into the gyrokinetic equation and taking integrals over velocity space yields differential equations that describe the evolution of the spectral coefficients and :
In this equation, , , and represent the nonlinear, drive, and collision terms, respectively. The terms coloured red and blue have a prefactor of . For electrons, is approximately 60 times larger than for (deuterium) ions, making these the stiff terms that we will treat implicitly. The red terms correspond physically to parallel streaming along the field line, while the blue terms correspond to the mirror force from the inhomogeneity of the magnetic field strength.
The field equations for , , and are
and
where is the Laguerre projection of the Bessel function. In practice, Eq. (13) is used to replace for in Eq. (14). The resulting equation is then advanced for and the fields are resolved to obtain the updated . The details on these derivations can be found in Mandell et al. [33, 32]. Finally, we note that while by default GX includes , in this work we neglect parallel magnetic field fluctuations, i.e. . Ultimately, the implicit algorithms presented can be generalized to include in a relatively straightforward manner.
Time-Integration Schemes
In this work, we use two types of time-integration schemes: a Lie-Trotter splitting scheme (Boyd [8]) similar to the one used in stella (Barnes et al. [6]), as well as additive Runge-Kutta methods (Cooper & Sayfy [12]).
Lie-Trotter
We begin with the Lie-Trotter scheme. For simplicity, we write Eq. (14) as
where is a vector containing all of the spectral coefficients of all species and the linear operators and represent the stiff streaming and mirror operators
and
respectively. The remaining terms, including the nonlinear term, are contained in the operator . In a Lie-Trotter splitting, we advance the spectral coefficients at time to at by evaluating each operator sequentially:
For the explicit operator , we solve for the intermediate using at the current timestep. However, the equations containing and use the second, intermediate, and future stage, and this results in two matrix equations that must be solved. Furthermore, this method does not require storing extra copies of , which makes it very memory-efficient.
While in the above equations, the explicit operator is advanced using a Forward Euler scheme, we in fact follow Barnes et al. [6] and use the standard third-order Strong-Stability Preserving Runge-Kutta (SSPRK3) method from Gottlieb et al. [20]. The implicit advances are performed using the implicit method from Giraldo et al. [19], which is second-order accurate. We find that the use of a second-order implicit scheme is important for accurately capturing transitions between different types of linear instabilities in toroidal geometry even though the entire scheme taken as a whole is first order. Strang-splitting is commonly used to obtain second-order accuracy at a higher computational cost. Nevertheless, it has been observed that the asymptotic convergence rate may only be realized for sufficiently small timesteps, and that the associated error constant can be large ([36]). This motivates the use of higher-order IMEX methods.
Additive Runge-Kutta
In the additive Runge-Kutta (ARK) methods, we solve for the solution at the next timestep using a series of stages
and
The operator is the explicit operator as for the Lie-Trotter scheme, but now contains both the streaming and mirror terms
There are many different choices for coefficients , , and in the published literature. For this work, we use the coefficients from the ‘ARK3(2)4L[2]SA–ESDIRK’ method of Kennedy & Carpenter [28]. This method contains 4 stages and is overall 3rd order accurate. Although the scheme has a higher order of accuracy than the Lie-Trotter method, it requires storing the intermediate solutions and performing three linear solves per step.
A key challenge for implicit gyrokinetic simulations is that the implicit operators act on both the plasma distribution function and the electromagnetic fields (which are both contained in ), which are themselves obtained from integral operators of the distribution function. This coupling makes the inversion significantly more difficult. In the next section, we describe several algorithms designed to address this difficulty.
Solving the Implicit Time-Advance Equations
In this section, we describe our methods to invert the parallel streaming and mirror force operators for our implicit methods. In §4.1, we discuss Kotschenreuther’s method, which is used to invert the streaming operator in GS2 and stella. Due to the large memory requirements of this method, we introduce new methods in §4.2 and §4.3 that approximately invert the streaming operator. Then in §4.4, we present our method for inverting the mirror term. Sections 4.5 and 4.6 discuss the hypercollision and magnetic/curvature drift terms and their stiffness. Due to approximations made in inverting the streaming and mirror operators, in §4.7 and §4.8 we discuss simple iterative methods to improve the accuracy of our methods.
Kotschenreuther’s Method
We start by briefly describing Kotschenreuther’s method [30] for inverting the linear operators in the gyrokinetic equation. The Kotschenreuther method is used to invert all the linear terms in GS2, as well as the streaming terms for stella.
Projecting the parallel streaming terms onto the Hermite-Laguerre basis yields the equation
where for simplicity we have ignored the drive and collision terms. We have also assumed only electrostatic perturbations. Discretizing in time using a Backwards Euler scheme, we must solve an equation of the form
where the and indices indicate the current and future timesteps, respectively. In GX, the above equation is then Fourier transformed in the domain, resulting in the equation
where the hats and denote Fourier transforms.
We separate the solution into a homogeneous and inhomogeneous part satisfying
and
We can straightforwardly solve Eq. (31) for using a tri-diagonal solver. Specifically, Eq. (31) is tridiagonal with respect to the indices, and the solver concurrently solves the equation across the remaining parameters. To solve for the homogeneous part, we need to first find an expression for . Note that can be expressed as
where is the contribution to from the inhomogeneous solution, and is called the ‘response matrix’. is the response at , in due to a unit perturbation in at . The response matrix can be calculated for each , , and using another series of tridiagonal matrix solves. For each solve, we set , and multiply it by before Fourier transforming. From there, we can solve for by solving
where . We can then substitute into the original equation and solve for .
Although exact, the major issue with using Kotschenreuther’s method for GX is the memory requirement. For points along the field-line and and number of and wavenumbers, there are matrices of dimension , each of which needs to be inverted to solve for . However, when including boundary conditions for the flux-tube, the dimension of each matrix is now , where is the number of flux tubes at different that are linked together. For a moderate resolution nonlinear simulation, , , and . As GX uses single-precision complex values for all of its data, the total memory required could then reach approximately 100 GB, which would be well beyond the amount of memory most GPUs today contain, especially if the same GPU must also contain all of the moments and fields. These matrices will also be dense, and so cannot be efficiently handled with sparse matrix solvers. Finally, the need to compute and store the response matrices makes this method less amenable to matrix-free iterative methods. This motivates the need for alternative approaches such as the ones described in this paper.
Parallel Streaming
As above, we would like to solve an equation of the form
Without , the equation is tridiagonal in the Hermite moments . However, the term involves convolving , which makes the equation difficult to solve. Instead of directly inverting, we will use an iterative scheme to solve this equation.
Assuming only a single ion species and electrons, the equation for is
To simplify this equation, we first take a drift-kinetic () limit. This simply results in neglecting all finite-Larmor-radius (FLR) terms in the equation for by setting and except for in the polarization term . The multiplying in Eq. (34) is also set to 1. The equation for is then
However, the denominator is still a function of the parallel coordinate, and would still result in a convolution after Fourier-transforming. To avoid this, we approximate the denominator by its minimum over :
where is
As described in Appendix § 9, we make this choice to capture the highest frequency wave in the system. An iterative method described in §4.7 will then help accurately capture the remaining modes.
Substituting this approximation for into Eq. (34) yields
This equation is now nearly tridiagonal, with simple corrections from . The resulting matrix system can be written as
where the matrix contains the tridiagonal bands, and the outer product of and contains the approximation to . The expressions for and are
and
where the bolded indicates consecutive zeros.
Since is a low-rank correction, we can solve the above system using the Sherman-Morrison formula, as detailed in Appendix § 10. A visualization of this banded structure is shown in Figure 1.

Figure 1. A visualization of the banded structure of the streaming matrix, with only electrostatic perturbations.
It should be noted that we did not have to take a long-wavelength approximation for , and it would still be a low-rank correction. However, by taking a long-wavelength approximation, the intermediate solutions needed for the Sherman-Morrison formula are only of dimension , where is the number of species, is the number of Laguerre moments, and is the number of Hermite moments. If we did not take a long-wavelength approximation, then the size would be . This memory saving is important given the limited memory on GPUs. Overall, this method involves only a small number of tri-diagonal solves, which can be computed in linear time without extra storage, as well as a series of cheap vector operations where the vectors are also low in memory. Furthermore, as we explain in §4.7, this approximation is only used as a preconditioner for an iterative method. For the simulations in this work, we only focus on ion-scale turbulence in which , and the iterative method converges rapidly. However, it is possible that shorter wavelength instabilities would require a more accurate preconditioner, and this will be addressed in future work.
When simulating multiple wavenumbers in the presence of magnetic shear, , enforcing periodicity in results in the well-known ‘twist-and-shift’ boundary conditions [7, 34]. Modes with differing (within the grid) are coupled at the end of the domain via the formula
where . One can interpret this formula as simulating an extended flux-tube that contains all the coupled ’s. The above algorithm remains mostly unchanged in this new flux-tube, except that the Fourier transforms in are taken at fixed along this extended flux tube. Similarly, we take to be the minimum along the extended flux-tube:
Streaming with
When including , the parallel streaming terms become
Like with , we take a drift-kinetic limit. In this limit is simplified to
and is
As described in Appendix § 9, we maximize rather than minimize :
Our final approximation for is thus
The final equation that we solve is
where and are the same as in Eqs. (41) and (42), and and are given by
and
Mirror Force
For the mirror terms, since there is no operator, we do not perform any parallel Fourier transforms. As a result, we do not make approximations for the fields and instead directly solve the equations that arise from our implicit scheme. Furthermore, we will only treat the electron mirror term implicitly. After discretizing in time, the equation involving the electron mirror term is of the form
The matrix we invert is again sparse and banded, with low rank corrections from the fields that are handled using the Sherman-Morrison formula. The banded pattern is shown in Figure 2a. There are 5 bands in the matrix: a tridiagonal structure with two additional bands at an offset of from the main diagonal.
Figure 2. The visualizations of the banded structure of the mirror (a) and magnetic drift (c) terms. The middle figure (b) shows the structure of the mirror matrix that is used in the MAGMA banded matrix routines.
To take advantage of this sparsity pattern, we utilize direct solvers for banded matrices recently developed in the MAGMA linear algebra package [1]. In this solver, the original input matrix of size is stored in banded format into a matrix of size , where and are the lower and upper bandwidth, respectively. Row of this new matrix contains the main diagonal, while the rows above and below contain the upper and lower diagonals. This structure is shown in Figure 2b. The full mirror matrix contains entries for 2 species, whereas this condensed matrix only contains entries, and so results in about a factor of reduction in memory.
To analyse how enters into the mirror terms, we rewrite each in terms of
The only contributions are from or . However, from Eq. (14) these terms have prefactors that vanish when taking a drift-kinetic limit, allowing us to neglect when inverting. The drift-kinetic limit is particularly appropriate as we only treat the electron mirror terms implicitly while focusing on ion scale turbulence.
Repeating this exercise with yields
When taking a drift-kinetic limit and including prefactors, we only need include the contribution to the and terms. Substituting in , the electron mirror term is then
In there are both ion and electron contributions . However, as we are only treating the electron mirror term implicitly, at each step of our timestepping schemes, the ion term has already been advanced explicitly. Therefore, can be moved to the RHS of the equation. We are only left with the electron contribution
Finally, the equation we would solve is
where is a banded matrix formed from the coefficients of , , , , and , and and are
and
Hypercollisions
Since the parallel streaming terms are Fourier transformed in GX, there is no inherent parallel dissipation that would normally arise from upwinding in other gyrokinetic codes. Furthermore, in the absence of sophisticated closures, additional dissipation is needed in velocity space to dampen the higher or modes. As a result, GX is normally run with a hypercollision operator which is derived in Mandell et al. [32]. This operator takes the form
where the collision frequency is
Due to the factor of , this term can become stiff for electrons. However, this term is straightforward to handle alongside the parallel streaming terms, as it is also evaluated spectrally in . Since this term is only on the main diagonal, it can be directly included in the tridiagonal solver routine.
Remaining Terms
The remaining terms that are not evolved implicitly correspond physically to the magnetic and curvature drifts, drive terms from the temperature and density gradients, and the collision operator. Although all of these terms are usually about the same order of magnitude between ions and electrons, the magnetic and curvature drift terms can become stiff due to the prefactor of or which are
and
If the geometry of the fusion device has particularly strong curvature or , or if one wants to investigate electron scale turbulence, the drift terms can dominate the CFL condition instead of the streaming or mirror terms. Indeed, we show in the next section that these drift terms will limit the maximum stable timestep for the tokamak cases. In practice, it is straightforward to treat these terms implicitly together with the mirror terms. However, the dependence of and results in matrices that would need to be stored (as opposed to just mirror matrices). This memory cost is prohibitively expensive on a single GPU, even for banded matrices. While we present some preliminary results of implicit evaluation of these terms in Appendix § 11, we leave a full implementation to future work.
Stationary Iteration
The approximations introduced in § 4.2 and §4.3 to invert the streaming operator can introduce significant errors that degrade both the accuracy and stability of the numerical method. To address these issues, we apply a simple stationary iteration to improve the solution at each implicit step.
Specifically, at each IMEX timestep, we solve a linear system of the form
where is the linear operator and is the right-hand side. Rather than inverting directly, we introduce a preconditioner , and rewrite the equation as
which can be rearranged into a fixed-point form:
This motivates the following iterative update (where the subscript index indicates the iteration step number, not the time index):
This scheme converges when the spectral radius of is less than one. When the preconditioner is a good approximation to , then , and convergence is typically rapid. In the cases of § 4.2 and §4.3, is the linear operator, and the derived simplified matrices represent the preconditioner.
While this approach is simple and effective, more advanced iterative solvers such as GMRES or BiCGSTAB may provide more robust convergence. These methods, along with a suite of IMEX integrators, are available in the SUNDIALS library [23, 4, 18, 35]. Interfacing GX with SUNDIALS will be the subject of future work. Importantly, if the current algorithm converges using basic stationary iteration, significant performance improvements can be expected by switching to Krylov methods.
Operator-Split Preconditioning
The ARK methods used in this work require that the parallel streaming and magnetic mirror terms be combined and inverted together. However, the magnetic mirror terms are not Fourier-transformed and thus cannot be easily combined with the parallel streaming operator into a single matrix. To address this, we apply an operator-splitting approximation.
Ideally, we would invert a matrix of the form
where represents the parallel streaming operator and represents the mirror force operator (or similar). Rather than constructing and inverting the combined matrix, we approximate the inverse by sequentially applying each operator:
This approximation introduces a local error of second order in due to the term. To reduce this error, we employ an ‘outer’ stationary iteration that simultaneously includes both the streaming and mirror operators in the iteration loop, using the operator split approximation as the preconditioner.
For the examples shown in this paper, we find that this combined iteration eliminates the need for a separate stationary iteration for the streaming term alone, thereby providing a simple approximation for the full implicit solve.
Numerical Results
In this section, we present results from several simulations that test our implicit methods. The simulations in §5.1 are conducted in slab geometry to test the implicit treatment of the parallel streaming terms. The simulations in §5.2 are performed in toroidal geometry to test the combined inversion of the parallel streaming and mirror terms. We initially focus on evaluating the accuracy of the implicit methods before giving a more detailed discussion of the timings in §5.4.
Slab Geometry
To test our preconditioner for the parallel streaming terms, we start in slab geometry, where the mirror and drift terms are zero. The geometry also includes finite magnetic shear so that the perpendicular wavenumbers vary along the field-line.
The results in Figure 3a show the linear growth rates and frequencies in this slab geometry. The magnetic shear is , which is comparable to shear in the core of a tokamak. The simulations using the Lie-Trotter and ARK methods employ a timestep of 25 and 50 times larger, respectively, than that used in the explicit scheme. As seen in the plot, the agreement between all three methods is excellent. Although the Lie-Trotter method is numerically stable with a 50 times greater timestep, there is worse agreement.

Figure 3. The linear growth rates and frequencies for the explicit RK3 (blue), the Lie-Trotter (orange), and the ARK (green) methods in slab geometry. The left plot is for the electrostatic case, while the right plot is at . The ‘x’ indicates the ratio of the IMEX timestep to the explicit timestep.
To test the electromagnetic implicit scheme, we set and reran the simulations with the same magnetic geometry. The results are shown in Figure 3b. As in the electrostatic case, the IMEX methods achieve good accuracy even when including electromagnetic fluctuations.
Finally, we run an electrostatic nonlinear simulation, and plot the resulting turbulent ion heat fluxes in Figure 4. The plot shows excellent agreement in the heat fluxes between the RK3 and ARK methods. However, even at a reduced timestep of only 25 times the explicit method, the Lie-Trotter method does not recover a similar average heat flux.

Figure 4. The ion heat flux for the nonlinear electrostatic slab case between the explicit RK3 (blue), Lie-Trotter (orange), and ARK (green) methods. The electron heat flux in this case is negligible, and so not plotted. The ‘x’ indicates the ratio of the IMEX timestep to the explicit timestep.
Tokamak Geometry
In this section, we compare the different schemes in tokamak-like geometry. The geometry we use is the ‘Cyclone Base Case’ (CBC), which is commonly used to benchmark gyrokinetic codes ([15]).
The results in Figure 5 show the linear growth rates and frequencies for each simulation. Unlike in slab geometry, in tokamak geometry the implicit methods cannot use a 60 times larger timestep than that of the explicit method. As will be explained in §5.4, the magnetic and curvature drift terms dominate the CFL constraint for the implicit methods. In this case, we can only use a 7–10 times greater timestep. Nevertheless, at this timestep, we achieve good agreement between all of the schemes for both the electrostatic and electromagnetic simulations.

Figure 5. The linear growth rates and frequencies for the explicit RK3 (blue), the Lie-Trotter (orange), and the ARK (green) methods for the cyclone base-case. The left plot is for an electrostatic simulation, while the right is at . The ‘x’ indicates the ratio of the IMEX timestep to the explicit timestep.
As a final test, we run nonlinear electrostatic simulations of the CBC. In this case, due to the magnetic/curvature drifts and the nonlinear term, the IMEX methods can only take up to a 4 times larger timestep. The resulting heat fluxes are shown in Figure 6. As seen in the plot, there is good agreement in the heat fluxes between the methods.

Figure 6. The ion (solid) and electron (dashed) heat fluxes for the electrostatic cyclone base-case for the explicit RK3 (blue), the Lie-Trotter (orange), and the ARK (green) methods. The ‘x’ indicates the ratio of the IMEX timestep to the explicit timestep.
Convergence of the Iterative Method for the Streaming Terms
To analyse the convergence of the iterative method used to invert the parallel streaming operator, we perform the electrostatic linear and nonlinear slab simulations with different numbers of stationary iterations using the ARK method. Figure 7a shows the relative error in the growth rate at versus the number of iterations. The errors decrease rapidly, reaching about a error at 4 iterations. The decrease in the error follows approximately a scaling, where is the number of iterations. Figure 7b shows the nonlinear heat flux traces for different numbers of iterations. We observe the steady-state heat flux from the implicit simulations consistently converges towards the heat flux from the explicit simulation as the number of iterations increases. Overall, this demonstrates that our simple preconditioned iterative solver is effective, and could have potential for further improvement using more sophisticated iterative methods.

Figure 7. (a) The growth rate error at versus the number of iterations for the iterative method for the streaming terms. The blue curve is the actual error, while the orange dashed line is that the error seems to follow. (b) The nonlinear heat fluxes for the ARK method for different number of iterations. The average heat flux eventually matches that from the explicit method at 5 iterations.
Timings
Table 1 shows the timings of the different simulations shown in the previous sections. For the slab cases, the Lie-Trotter and ARK methods maintain a speedup of over an order of magnitude. This speedup reduces in the CBC cases in part due to the CFL condition being set by the explicit drift terms. Following the procedure in Mandell et al. [32], we expect the CFL condition to be set by a combination of the drift and ion mirror terms
The maximum drift frequency can be estimated as
where and scale like
and
and and are the maximum values of the Gauss-Hermite and Gauss-Laguerre collocation points. For the linear CBC simulations in this work, , while . Since scales with the perpendicular wavenumber , the maximum timestep in the electromagnetic CBC simulation was slightly larger than in the electrostatic simulation, as it had a smaller maximum . For , we expect a maximum stable timestep of about .
In comparison, the streaming terms would contribute a term that scales like
For electrons, this term is approximately , making the maximum timestep . Therefore, we expect that the implicit methods can only use about a 10 times greater timestep than the explicit method, which is demonstrated in the CBC simulations.
Finally, we find that the most expensive operation in our IMEX schemes is inverting the matrices for the mirror operator. This cost increases as we increase the number of iterations, particularly for the ARK scheme. To demonstrate this effect, we run the CBC simulations again with the ARK method, but treat the mirror term explicitly. As shown in Figure 8 and Table 1, we can achieve comparable accuracy and maximum stable timestep at a reduced wall-time. Therefore, future work will investigate different sparse linear algebra libraries to reduce this cost. Nevertheless, for the nonlinear case, we can still achieve approximately a factor of 2 speedup.

Figure 8. The linear growth rates and frequencies for the explicit RK3 (blue) and the ARK methods for the cyclone base-case. The left plot is for an electrostatic simulation, while the right is at . The ‘x’ indicates the ratio of the IMEX timestep to the explicit timestep. In these cases, the ARK method evaluates the mirror term explicitly.
| Scheme | Factor | Wall Time (min) | Speedup | |
| Sheared Slab (ES) | ||||
| RK3 | 1 | 8.68 | 1.00 | |
| Lie-Trotter (3 stream iters) | 25 | 0.79 | 10.99 | |
| ARK (3 stream iters) | 50 | 0.52 | 16.69 | |
| Sheared Slab (EM) | ||||
| RK3 | 1 | 8.19 | 1.00 | |
| Lie-Trotter (3 stream iters) | 25 | 0.96 | 8.53 | |
| ARK (3 stream iters) | 50 | 0.63 | 13.0 | |
| Sheared Slab (Nonlinear, ES) | ||||
| RK3 | 1 | 489.49 | 1.00 | |
| Lie-Trotter (8 stream iters) | 25 | 39.88 | 12.27 | |
| ARK (5 stream iters) | 50 | 23.56 | 20.78 | |
| CBC (ES) | ||||
| RK3 | 1 | 5.85 | 1.00 | |
| Lie-Trotter (2 stream iters) | 7.14 | 1.49 | 3.93 | |
| ARK (1 outer iters) | 7.14 | 1.57 | 3.73 | |
| ARK (1 stream iters, explicit mirror) | 7.14 | 1.25 | 4.68 | |
| Lie-Trotter (4 stream iters, implicit drifts) | 14.29 | 1.05 | 5.57 | |
| CBC (EM) | ||||
| RK3 | 1 | 2.56 | 1.00 | |
| Lie-Trotter (3 stream iters) | 7.78 | 1.12 | 2.29 | |
| ARK (2 outer iters) | 10.0 | 1.24 | 2.06 | |
| ARK (2 stream iters, explicit mirror) | 10.0 | 0.69 | 3.71 | |
| CBC (Nonlinear, ES) | ||||
| RK3 | 1 | 617.97 | 1.00 | |
| Lie-Trotter (3 stream iters) | 4.0 | 217.83 | 2.84 | |
| ARK (2 outer iters) | 4.0 | 331.58 | 1.86 |
Table 1. Simulation timings for each scheme and configuration. ES and EM denote electrostatic and electromagnetic simulations, respectively.
Conclusions
In this work, we developed implicit-explicit methods to accelerate kinetic-electron simulations for the GX gyrokinetics code. Since GX is GPU-native, we have designed our methods to be low-memory by taking advantage of banded and low-rank structure of the GX equations to efficiently invert the implicit operators. We showed that our methods can achieve up to the maximum theoretical timestep while still maintaining an improvement in the wall-clock time for both linear and nonlinear simulations. However, this scheme uses approximate iterative methods. While we show that the iterative method for the streaming terms is efficient and accurate, the operator-split preconditioned iterative method can be more expensive due to the cost of direct banded solvers. These methods are also less effective in cases where the drift terms dominate the CFL condition.
There are several directions for possible future work. We first aim to implement these methods within the SUNDIALS framework. Doing so should improve the relatively simple iterative methods shown here, and improve the efficiency of the ARK methods. The
SUNDIALS library also includes robust methods for error-checking and adaptive timesteps. Additionally, we will include a full implementation of the magnetic and curvature drift terms in the implicit operator, which would further relax the CFL constraint.
Acknowledgments
P. S. K. would like to acknowledge F. Parra Diaz for several helpful conversations. This work was supported in part by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Scientific Discovery through Advanced Computing (SciDAC) Program through the Computational Evaluation and Design of Actuators for Core-Edge Integration (CEDA) project, and through the Frameworks, Algorithms and Software Technologies for Mathematics (FASTMath) Institute. This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Department of Energy Computational Science Graduate Fellowship under Award Number DE-SC0024386. P. S. K. used ChatGPT (OpenAI) and Claude (Anthropic) to aid in software devel-
caption: Table 2. Normalizing the quantities for the independent variables. Taken from Mandell et al. [32].
:::
Table 3. Normalizing quantities for dependent variables. Taken from [32].
opment and debugging and in revising text. P. S. K reviewed any AI-assisted code and text, and the manuscript has been reviewed by the authors, who take full responsibility for its content.
Normalizations
The normalizations used in this work are the same as in Mandell et al. [32] and are summarized in tables 2 and 3. The normalizing quantities in table 2 are arbitrary, but have common choices. For example, the normalizing length is typically either the minor or major radius. The reference magnetic field is typically taken to be the vacuum magnetic field at the magnetic axis or is defined in terms of the enclosed toroidal magnetic flux. The reference thermal speed is defined in terms of a reference temperature and mass . The corresponding reference gyroradius and cyclotron frequency are then and , respectively. Specific species quantities such as the temperature, mass, thermal speed, charge, and gyroradius are then normalized with respect to these reference values: , , , , and .
Long-Wavelength Dispersion Relation In Slab Geometry
In unsheared slab geometry, we consider a simplified limit with no gradients or collisions, stationary ions, and isothermal electrons. In the long-wavelength limit, the GX equations for the electron moments reduce to
and
with the corresponding field equations given by
and
where we define .
Fourier transforming and combining the moment and field equations yields the dispersion relation:
This simplifies to the standard dispersion relation for the kinetic Alfvén wave:
In the electrostatic limit (), this further reduces to the mode:
This expression shows that the mode frequency increases significantly as . Therefore, to account for the fastest waves, we take the smallest along the field line in constructing our preconditioner. Notably, explicit gyrokinetic codes often retain a small but nonzero even for nominally electrostatic simulations in order to avoid this high-frequency mode.
For more general geometry and nonzero , we approximate the dispersion relation as
where and represent specific values chosen along the field line. Again, is minimized, while is maximized to result in the maximum possible frequency
Sherman-Morrison Formula
The matrices we invert in this work are of the form
where and contain information about the fields. The Sherman-Morrison formula states that the solution to this equation is
where and are solutions to the equations
When including multiple fields, like and , the equation becomes
Following the procedure in Tatsuno & Dorland [39] and Barnes et al. [5], we rewrite the above equation as
where . The solution is then
where and are solutions to the equations
To solve for and , we apply the Sherman-Morrison formula a second time
and
where , , and are solutions to
Implicit Treatment of the Magnetic and Curvature Drift Terms
When expanded in terms of the Hermite-Laguerre basis functions, the magnetic and curvature drift terms are
and
The matrix corresponding to this term can be written as
where is a banded matrix of bandwidth . The field terms are contained in the vectors , , , and , whose entries are defined as
and
Since this term is evaluated in real space in , can be combined with the mirror matrix in the implicit methods.
To test the implicit treatment of the magnetic drifts, we again run the linear electrostatic CBC simulation. We use the Lie-Trotter method, but the split for the mirror term now also contains the drift terms. For both the mirror and drift terms, both ions and electrons are treated implicitly, and we retain full FLR-effects in the implicit operator. A plot of the linear growth rates and frequencies is in Figure 9. In this case, we are able to double the timestep while maintaining good accuracy. The maximum stable timestep is larger, but at the cost of further reduced accuracy. We do not show results using the ARK method, as we find that the operator-split preconditioner requires an increased number of iterations at a larger timestep to remain stable, which offsets the advantage of the increased timestep.

Figure 9. The linear growth rates and frequencies for the explicit RK3 (blue) method and the Lie-Trotter method (orange) scheme for the electrostatic cyclone base-case. The magnetic and curvature drift terms are treated implicitly in this case.
References
- [1]Abdelfattah, Ahmad, Tomov, Stanimire, Luszczek, Piotr, Anzt, Hartwig & Dongarra, Jack 2023 GPU-based LU Factorization and Solve on Batches of Matrices with Band Structure. In ACM International Conference Proceedings Series, pp. 1672–1679. Association for Computing Machinery.DOI
- [2]Abel, I G, Plunk, G G, Wang, E, Barnes, M, Cowley, S C, Dorland, W & Schekochihin, A A 2013 Multiscale gyrokinetics for rotating tokamak plasmas: fluctuations, transport and energy flows. Reports on Progress in Physics 76 (11), 116201.DOI
- [3]Antonsen Jr., Thomas M & Lane, Barton 1980 Kinetic equations for low frequency instabilities in inhomogeneous plasmas. The Physics of Fluids 23 (6), 1205–1214.DOI
- [4]Balos, Cody J, Gardner, David J, Woodward, Carol S & Reynolds, Daniel R 2021 Enabling GPU accelerated computing in the SUNDIALS time integration library. Parallel Computing 108, 102836.DOI
- [5]Barnes, M, Abel, I G, Dorland, W, Ernst, D R, Hammett, G W, Ricci, P, Rogers, B N, Schekochihin, A A & Tatsuno, T 2009 Linearized model Fokker–Planck collision operators for gyrokinetic simulations. II. Numerical implementation and tests. Physics of Plasmas 16 (7), 072107.DOI
- [6]Barnes, M., Parra, F. I. & Landreman, M. 2019 stella: An operator-split, implicit–explicit δf-gyrokinetic code for general magnetic field configurations. Journal of Computational Physics 391, 365–380.DOI
- [7]Beer, Michael Alan 1995 Gyrofluid Models of Turbulent Transport in Tokamaks. PhD thesis.
- [8]Boyd, John P. 2001 Chebyshev and Fourier spectral methods. Dover Publications.
- [10]Candy, Jeff & Waltz, R. E. 2003 An Eulerian gyrokinetic-Maxwell solver. Journal of Computational Physics 186 (2), 545–581.DOI
- [11]Catto, P J 1978 Linearized gyro-kinetics. Plasma Physics 20 (7), 719–722.DOI
- [12]Cooper, G J & Sayfy, A 1980 Additive methods for the numerical solution of ordinary differential equations. Mathematics of Computation 35 (152), 1159–1172.
- [15]Dimits, A M, Bateman, G, Beer, M A, Cohen, B I, Dorland, W, Hammett, G W, Kim, C, Kinsey, J E, Kotschenreuther, M, Kritz, A H, Lao, L L, Mandrekas, J, Nevins, W M, Parker, S E, Redd, A J, Shumaker, D E, Sydora, R & Weiland, J 2000 Comparisons and physics basis of tokamak transport models and turbulence simulations. Physics of Plasmas 7 (3), 969–983.
- [16]Dorland, W., Jenko, F., Kotschenreuther, M. & Rogers, B. N. 2000 Electron Temperature Gradient Turbulence. Physical Review Letters 85 (26), 5579–5582.pubmed.ncbi.nlm.nih.gov/11136051
- [17]Frieman, E A & Chen, Liu 1982 Nonlinear gyrokinetic equations for low-frequency electromagnetic waves in general plasma equilibria. The Physics of Fluids 25 (3), 502–508.
- [18]Gardner, David J, Reynolds, Daniel R, Woodward, Carol S & Balos, Cody J 2022 Enabling New Flexibility in the SUNDIALS Suite of Nonlinear and Differential/Algebraic Equation Solvers. ACM Trans. Math. Softw. 48 (3).DOI
- [19]Giraldo, F X, Kelly, J F & Constantinescu, E M 2013 Implicit-Explicit Formulations of a Three-Dimensional Nonhydrostatic Unified Model of the Atmosphere (NUMA). SIAM Journal on Scientific Computing 35 (5), B1162–B1194.DOI
- [20]Gottlieb, Sigal, Shu, Chi-Wang & Tadmor, Eitan 2001 Strong Stability-Preserving High-Order Time Discretization Methods. SIAM Review 43 (1), 89–112.DOI
- [23]Hindmarsh, Alan C, Brown, Peter N, Grant, Keith E, Lee, Steven L, Serban, Radu, Shumaker, Dan E & Woodward, Carol S 2005 SUNDIALS: Suite of nonlinear and differential/algebraic equation solvers. ACM Trans. Math. Softw. 31 (3), 363–396.
- [24]Hinton, F L & Hazeltine, R D 1976 Theory of plasma transport in toroidal confinement systems. Reviews of Modern Physics 48 (2), 239–308.DOI
- [25]Horton, W 1999 Drift waves and transport. Reviews of Modern Physics 71 (3), 735–778.DOI
- [26]Jenko, F, Dorland, W, Kotschenreuther, M & Rogers, B N 2000 Electron temperature gradient driven turbulence. Physics of Plasmas 7 (5), 1904–1910.DOI
- [27]Jorge, Rogerio 2026 GKX: gyrokinetic turbulence and sensitivities in JAX.
- [28]Kennedy, Christopher A. & Carpenter, Mark H. 2003 Additive Runge–Kutta schemes for convection–diffusion–reaction equations. Applied Numerical Mathematics 44 (1-2), 139–181.DOI
- [30]Kotschenreuther, Mike, Rewoldt, G. & Tang, W. M. 1995b Comparison of initial value and eigenvalue codes for kinetic toroidal plasma instabilities. Computer Physics Communications 88 (2-3), 128–140.
- [31]Liewer, Paulett C 1985 Measurements of microturbulence in tokamaks and comparisons with theories of turbulence and anomalous transport. Nuclear Fusion 25 (5), 543.DOI
- [32]Mandell, N R, Dorland, W, Abel, I, Gaur, R, Kim, P, Martin, M & Qian, T 2024 GX: a GPU-native gyrokinetic turbulence code for tokamak and stellarator design. Journal of Plasma Physics 90 (4), 905900402.arxiv.org/abs/2209.06731
- [33]Mandell, N R, Dorland, W & Landreman, M 2018 Laguerre-Hermite pseudo-spectral velocity formulation of gyrokinetics. J. Plasma Phys 84 (1), 905840108.DOI
- [34]Martin, M F, Landreman, M, Xanthopoulos, P, Mandell, N R & Dorland, W 2018 The parallel boundary condition for turbulence simulations in low magnetic shear devices. Plasma Physics and Controlled Fusion 60 (9), 095008.DOI
- [35]Reynolds, Daniel R., Gardner, David J., Woodward, Carol S & Chinomona, Rujeko 2023 ARKODE: A Flexible IVP Solver Infrastructure for One-step Methods. ACM Trans. Math. Softw. 49 (2).DOI
- [36]Ropp, David L., Shadid, John N. & Ober, Curtis C. 2004 Studies of the accuracy of time integration methods for reaction–diffusion equations. Journal of Computational Physics 194 (2), 544–574.DOI
- [37]Rosenbluth, Marshall N 1968 Low-Frequency Limit of Interchange Instability. The Physics of Fluids 11 (4), 869–872.DOI
- [39]Tatsuno, T & Dorland, W 2008 Secondary tearingmode in the nonlinear evolution of magnetorotational instability. Astronomische Nachrichten 329 (7), 688–700.DOI
- [40]Wootton, A J, Carreras, B A, Matsumoto, H, McGuire, K, Peebles, W A, Ritz, Ch. P, Terry, P W & Zweben, S J 1990 Fluctuations and anomalous transport in tokamaks. Physics of Fluids B: Plasma Physics 2 (12), 2879–2903.DOI
