WorksPhysics and Astronomy
Numerical relativity simulations of compact binaries: comparison of cell- and vertex-centered adaptive meshes
Abstract
Given the compact binary evolution problem of numerical relativity, in the finite-difference, block-based, adaptive mesh refinement context, choices must be made on how evolved fields are to be discretized. In GR-Athena++, the space-time solver was previously fixed to be vertex-centered. Here, our recent extensions to a cell-centered treatment, are described. Simplifications in the handling of variables during the treatment of general relativistic magneto-hydrodynamical (GRMHD) evolution are found. A novelty is that performance comparison for the two choices of grid sampling is made within a single code-base. In the case of a binary black hole inspiral-merger problem, by evolving geometric fields on vertex-centers, an average speed increase is observed, when compared against cell-centered sampling. The opposite occurs in the GRMHD setting. A binary neutron star inspiral-merger-collapse problem, representative of typical production simulations is considered. We find that cell-centered sampling for the space-time solver improves performance, by a similar factor.
Introduction
Fashioning accurate description of astrophysically sourced, binary constituent encounters, under extreme conditions, is a pressing challenge for the numerical relativity (NR) community. Multimessenger detection efforts [1, 2, 3] benefit when supplemented by such data. Observational efforts of next-generation gravitational-wave detectors [4, 5, 6, 7] will also greatly benefit from higher fidelity reference numerical data, that spans a larger range of parameter space (or e.g. models [8] suitably informed by). To confront simulation challenges, sophisticated general relativistic, magneto-hydrodynamical (GRMHD) techniques (see e.g. [9]), and microphysics descriptions [10, 11] have been pursued. In tandem, the development of code infrastructure, and algorithms, with the aim of improved performance scaling on modern high-performance computing (HPC) resources, has also received recent, particular attention. Some examples of modern NR codes under active development with these goals include: bamps [12, 13], Dendro-GR [14], Einstein Toolkit [15] (GRaM-X [16]), GRChombo [17], NMesh [18], SpECTRE [19], and SPHINCS_BSSN [20]. A variety of numerical methods [21, 22, 23], algorithmic approaches [24, 25, 26, 27], and NR formulations [28, 30, 31, 32, 33, 34, 35] [29] are employed, to list but a few prominent examples. In spite of this, the actual time taken (HPC resource consumed) in the calculation of numerical evolutions, particularly for extreme scenarios that require highly resolved physical scales, can require weeks, or months [36]. Clearly, even minor performance optimizations, of a few percent, can have a significant impact on such HPC wait times, together with broader, sustainability considerations [37].
In this work, we focus on finite-difference based evolution of the Z4c formulation [31, 32] of NR, as coupled to the general relativistic, hydrodynamical (GRHD) formulation1 of [38]. This is precisely the setting treated by our code GR-Athena++ [39, 40] (see also [41]). In brief, this is our extension to the astrophysical (radiation), GRMHD code Athena++ [42, 43, 44], we have geared toward evolution with dynamical space-time. Building upon the underlying octree, block-based adaptive-mesh-refinement (AMR) infrastructure, with addition of our Z4c based space-time solver both in vacuum [39], and for GRMHD with dynamical space-time [40], has led to demonstrable scaling inefficiencies in excess of 80% on (pre-)exascale HPC infrastructure.
Specifically, here, we aim to address a seemingly innocuous question: should we select the geometric fields, that is, the dynamical fields of Z4c, to be discretized on vertex-centers (VC) or cell-centers? Our initial choice was based on considerations for binary black hole evolution, where transfer of data between differing levels of refinement, is less computationally expensive for VC in contrast to cell-centers. On the other hand, in consideration of GR(M)HD, typically hydrodynamical sampling is on cell-centers (see e.g. BAM [45], GRHydro [46], and WhiskyTHC [47]). This choice is in part due to greater amenability of the underlying hydrodynamical treatment as based on conservative, high-resolution-shock-capture (HRSC) methods (see e.g. [48]). Athena++ also handles hydrodynamical description on cell-centers. Having previously fixed geometric fields to VC, discretized fields must consequently be transferred between grids, which incurs an overhead. A priori it is not necessarily clear as to how large this is. Our aim here is to investigate switching to a cell-centered treatment of the space-time solver in GR-Athena++, thus providing an answer. Such a code-internal investigation does not appear to have been previously performed. This gap we fill here. This paper proceeds as follows. In §II we briefly explain the computational domain structure and the variable discretization strategy. Subsequently, we present a simplified refinement strategy, tailored to the binary merger problem, and independent of geometric field sampling selection. In §III we demonstrate the robustness of the technique through numerically solving BBH and BNS inspiral-merger problems, representative of typical production simulations. Section §IV concludes.
Method
GR-Athena++ [39, 40] builds upon Athena++ [42, 43, 44] and thus inherits overall features of the latter framework. In brief, problems are formulated over a target computational domain (the Mesh) which is partitioned into a collection of sub-domains (MeshBlock objects). We have where may be thought of as an ordered parametrization of multi-dimensional coordinates to a space-filling curve [26, 27]. For a -dimensional problem the overall extent of together with suitable boundary data on must be provided. Following this, the underlying discretization is fixed through specification of Mesh and MeshBlock sampling and , respectively, where must divide component-wise. In the case that (local) refinement is desired, MeshBlock partitioning into new MeshBlock objects is recursively performed. This involves halving size along each axis at a fixed number of samples, which doubles the resolution at each step. The aforementioned is done while maintaining an (at-most) refinement ratio between neighboring . While the logical-rectilinear structure here described may then be further mapped to suitable curvilinear coordinates, for simplicity we focus on the Cartesian context.
Cell-centered extension
In GR-Athena++ [39, 40] we extended sampling support to enable vertex-centered (VC) discretization of field variables together with infrastructure enabling high-order approximations to be carried out. In this work we focus on analogous extensions in description of geometric quantities on cell-centers (CX). A variable where () is CX discretized to over
where and . This yields total samples. In practice the interval is extended by an additional “ghost” nodes either-side. This extension facilitates evaluation of e.g. derivative approximations, transfer of field data between neighboring MeshBlock objects that may differ in the level of refinement, and/or imposition of boundary conditions.
In order to populate ghost nodes, communication is required. In the case that is discretized over a Mesh decomposed at a fixed, common level, under CX sampling, this entails that nearest-neighbor MeshBlock objects can be utilized to populate ghost-layers through a straightforward copy. Suppose instead that sub-domains differ in level . In this situation, analogous to the VC case, each sub-domain (where prefix indicates level) as described by a MeshBlock, features a fundamental and an additional coarse representation of the sampled variable. This latter is one level above at . Thus if is the number of physical points on a MeshBlock extended by ghost nodes, then the complementary coarse representation features physical points extended by coarse ghosts. The situation is illustrated (locally) in Figure 1.
Figure 1. One-dimensional illustration of inter-MeshBlock communication given a refined local Mesh structure. Each MeshBlock features a fundamental representation of a sampled field depicted through square nodes which is extended in each direction through ghost nodes (circles) that facilitate communication. Additionally a complementary coarse sampling is present which is explicitly depicted for data on level and suppressed on level . Here , , and . Prior to all communication data in filled square nodes is available; the task is thus to fill data at locations indicated by shaded circles and diamonds. Nodes are assigned MeshBlock-local labels for ease of discussion.
The general communication strategy and ordering of operations closely follows and extends that of Athena++. This minimizes the amount of infrastructure changes required. Synchronization of data across levels may be understood and implemented by considering the transfer operations: fine to coarse (restriction ), and coarse to fine (prolongation ). As an example consider synchronization of ghost-layer data between and as depicted in Figure 1. Initially data on the fundamental grid of over the nodes (is, is+1, . . . , ie) is known. This is first restricted to a subset of the complementary coarse grid of over the nodes (cime+1, . . . , cime+); see green diamonds in the figure. Subsequently data at the nodes (ie + 1, . . . , ipe) of may be filled through communication or a copy (green circles in figure). Conversely using data from at the nodes (ie − + 1, . . . , ie) allows for filling of (cims, . . . , cime) on (purple circles) through communication or a copy. Finally, ghost layer nodes (ims, . . . , is − 1) of the fundamental grid over may be filled (blue circles) through based on data local to the MeshBlock. The procedure is analogous for the pair and .
Comparing node positions in Figure 1 across differing levels of refinement clearly shows that nodes do not align but are fully staggered5. This is one aspect where CX differs from VC sampling [39]. The coarsening and refining of sampled field data via and respectively, is based on polynomial interpolation under CX and VC.
Due to the inter-level node staggering occurring in CX, a non-trivial evaluation of interpolants for every target node is required. This is in contrast to the case of VC where a subset of operations reduces to injection (copy) [39]. The distinction heralds a performance implication as will be demonstrated in §III A and §III B.
We close this subsection with a collection of technical points concerning implementation aspects.
The formal order of accuracy for the and operations on CX is controlled by the number of nodes selected. We fix this choice based on selection so to match the approximation order with that of the finite-differencing.
Given a target node, source data entering an interpolant describing CX has a directional bias due to grid staggering. This can be understood by noting that the source data nodes are not symmetrically placed about the location of the target node, but offset. Consider at a desired formal order of accuracy, the initial coarse, complementary restriction needed when communicating data to a finer neighboring MeshBlock. There may be insufficient nodes available to evaluate with symmetrically spaced source data. In this situation, we again bias the stencil selection.
As is well-known, floating point (FP) arithmetic is not necessarily associative: for FP numbers . If care is not taken in how numerical expressions are ordered, errors can e.g. lead to catastrophic cancellations, or result in spurious symmetry breaking during simulations [49]. A combination of Kahan-Babushka-Neumaier compensated summation [50], and explicit symmetrization of arithmetic operations involving averaging can partially mitigate this [49]. This can be optionally activated for interpolant evaluation. We have not however found it necessary for the simulations presented here.
There are now two styles of cell-centered sampling where discretized field variables may be registered and stored: that inherited from Athena++, which we denote CC, and the newly introduced CX. The principal distinction is that for a Mesh featuring MeshBlock objects on distinct levels, the and operators are distinct: for CC there is minmod based slope-limiting [51], whereas for CX we utilize the Barycentric Lagrange approach [52]. Special care has been taken to ensure that distinct fields may be suitably registered as either CC or CX with their respective, logic available concurrently.
The VC double-restriction operation described in [39] we have found to not be strictly required. Its
removal means that when utilizing vertex-centered sampling we now have . This allows for smaller MeshBlock size, which offers greater flexibility in tailoring resolution adapted regions.
Refinement strategy
In order to resolve dynamical features over widely disparate spatial scales we utilize adaptive mesh refinement (AMR). Our purpose here is to fix notation, and to point to two simple, minimal criteria that will facilitate consistency tests of the new infrastructure. The aforementioned feature evolution of initial data describing a binary black hole (BBH) (§III A), and a binary neutron star (§III B). For a comprehensive investigation of AMR criteria in GR-Athena++, and effect on waveform quality for BBH evolution, see [53].
The AMR conditions of GR-Athena++ are quite flexible. A user-specified function is defined for each MeshBlock; this function returns a flag that controls whether to (de)-refine. Consider puncture-based evolution [45] of a BBH system. Here a black hole center is described through a puncture. The position of a puncture can be computed by augmenting the dynamical system evolved through an additional ODE based on gauge information [54]. In short, this is one variety of a so-called tracker. This allows us to construct a dynamically adjusted, octree-based, pseudo-box-in-box (pBIB) refined Mesh as detailed in [39] (or -norm refinement in the language of [53]).
Let us suppose the Mesh is the Cartesian coordinatized, computational domain . A parameter controlling the maximal level of refinement is specified. A hierarchy of MeshBlock objects is then constructed, which increases in resolution as a puncture is approached (see e.g. Figure 6). On the finest resolution level the grid spacing will be [39]. As an alternative to pBIB, we can exploit information from the trackers and directly impose a desired region to be at fixed resolution (equivalently level). Denote a spherical region of radius , centered at as . We center such a region on a puncture and impose a desired local level of refinement . This we write as .
Figure 6. Mesh structure and values of under at in the orbital plane. Each quadrant depicts distinct resolutions (AMR choices) for the same physical calibration problem configuration. Counter-clockwise, starting from the upper right quadrant we have pBIB AMR with . The lower right quadrant depicts evolution with the dynamical spherical AMR criterion, and fixed. Each grey box (MeshBlock) is sampled with points along each axis. For the cases involving the pBIB AMR criterion observe that reflection of outward propagating dynamical spherical features in is present in the vicinity of a change in refinement level. See text for further discussion.
In a similar vein, consider a dynamical, user selected, control field featuring a local extrema . Another type of tracker is possible to construct based on following the time development of through . This allows for the region surrounding the extrema to be captured as its position changes in time. If multiple such regions are registered, occur at different , and happen to intersect, then we take the finest level as the target level. This is the principal condition that we employ for AMR during BNS evolution.
Validation
Our first goal in this section is validating the extended cell-centered implementation in the context of vacuum binary black hole evolution. The space-time solver implemented in GR-Athena++ is based on the Z4c formulation [31, 32]. Here the space-time is constructed based on the evolution of the dynamical, geometric variables . The definition and meaning of these variables is standard and may be found, together with implementation details, in [39]. Previously, motivated by AMR efficiency considerations in the vacuum context, we have chosen these to evolve over VC grid sampling. We compactly denote this through . This work details a relaxation of this restriction, extending functionality to allow for CX sampling, i.e. sampling and evolving directly as . We detail tests of this in §III A.
Secondly, we investigate the binary neutron star (BNS) inspiral-merger problem. Our recently implemented GRMHD treatment in GR-Athena++ is based on the formulation of [38]. For full details, and conventions, see [40]. Here we recall key ingredients of constructing the matter side of the evolution as based on selecting . For simplicity, we restrict our exposition to the GRHD sector. For the hydrodynamics, the (conserved) field variables are , and satisfy a balance law. Recall that this is evolved based on conservative, finite-volume, HRSC techniques [48]. The natural choice of sampling, that also leverages extant Athena++ infrastructure is cell-centered, i.e. we take . We also work with a set of complementary, primitive variables . The relation is non-linearly implicit, and is evaluated through numerical inversion [55]. In this work the conservative to primitive variable mapping is handled through the use of RePrimAnd [56]. This inversion requires geometric fields to be available on the same grid sampling as the matter. Hence an intergrid interpolation is required.
In order to evolve the quantities via HRSC we require assembly of fluxes at cell-interfaces (that is, face-centers (FC)). In this work, this is achieved based on the reconstruction of to FC with the fifth order method of [57] denoted WENO5Z. Flux assembly also requires a subset of and hence another interpolation. The interface fluxes are combined according to the Local-Lax-Friedrichs (LLF) scheme (see e.g. [58]). Finally the sources appearing in the hydrodynamical balance law are assembled, which again requires . On the other hand, the gravitational matter density, momenta, and stress quantities that recouple the system to the space-time solver, must be provided on VC. This entails the interpolation of hydrodynamical quantities to VC. It is clear from the above summary, that by instead taking the evolved geometric variables as from the start, fewer transfers of geometric data between grids would be required. We further validate our code functionality, for BNS, in §III B.
In addition to ensuring consistency of the implementation, and that problems are solved correctly, a practical performance assessment is needed. We demonstrate that execution speed, and suitable choice of () is contingent on the class of physical problem simulated.
Binary Black Holes
We assess the performance of our code by focusing on the well-known calibration binary black hole (BBH) problem of [45]. The Z4c system is coupled to the moving puncture gauge using the -slicing condition [59] and the Gamma driver shift [60, 61], with parameters as described in [39]. Initial data is generated through the use of the TwoPunctures6 library based on [62]. We model two initial non-spinning black holes of bare-mass , located on the -axis, with , and initial momenta directed along the -axis, . These parameter choices lead to an evolution featuring orbits to merger, which occurs at an evolution time .
In these tests the Courant-Friedrich-Lewy (CFL) is fixed at . This choice determines the global time-step taken on each MeshBlock based on the finest grid-spacing (occuring in the vicinity of a puncture and denoted ). Kreiss-Oliger dissipation is selected as throughout. The Z4c constraint damping parameters are taken as (see [39]). Unless otherwise stated, in this section we take , that is, the number of ghosts on the fundamental grid is set equal to the number on the complementary coarse representation (see Figure 1). We select , which induces a formal order of spatial approximation. Time-evolution is performed using the explicit order RK4()4[2S] [63].
The overall grid configurations based on pBIB refinement (see §II B) are summarized in Tab. (I). In principle
| #MB | ||
| 96 | 3.125 | 5144 |
| 128 | 2.344 | 6336 |
| 160 | 1.875 | 18640 |
| 192 | 1.563 | 22392 |
caption: Properties of grid configurations utilized for vacuum tests. The total number of MeshBlock objects (#MB) at the initial time is indicated. In all cases the Mesh is the Cartesian coordinatized domain , where is selected. This entails that grid spacing is uniform along each spatial axis. In each case we select the maximal refinement through . The MeshBlock sampling is selected as . Resultant puncture resolutions are also provided.
:::
the calibration BBH problem possesses a underlying bitant symmetry (i.e. reflection across the plane to which the co-orbit and merger is confined). This allows for an approximate halving of the total computational resources required for a given problem. When imposing bitant symmetry, #MB as listed in Tab. I is roughly halved, and in order to keep the grid spacing fixed, we halve the Mesh sampling parameter along the -axis to . As an initial consistency check, we select the geometric sampling as and the grid of Tab. I. We perform a comparable run with , for which we inspect the puncture tracker evolution in Figure 2. Excellent agreement can be seen.
Figure 2. Tracker evolution during calibration BBH inspiral and merger. The one-parameter families describe where punctures are located. Consistent behaviour for cell-centered extended (CX) and vertex-centered (VC) choices of sampling is observed. Parameters: , and .
The BBH evolution considered here should result in an evolution with a strict symmetry about the plane. During numerical evolution, however, this condition is not necessarily preserved. The component of a puncture can potentially slowly drift out of the plane. The behaviour is demonstrated in Figure 3 for full 3D runs. In the case of , based on the structure of the level-to-level transfer operators, we do not observe, nor expect, strict FP symmetry preservation. Instead, during the full 3D run we find a very minor drift developing, where the -component of the puncture reaches . We also verify in Figure 3 that may indeed be independently selected, so as to increase the formal order of accuracy in the prolongation operation for . We have similarly verified this for . As this does not appear to substantially change simulation quality for the short evolution here, we proceed with runs that fix . One reason for this behaviour, is that the TwoPunctures library does not allow for explicit enforcement of an underlying symmetry about . Thus, when interpolating an initial data (ID) set to a Cartesian grid, the result violates the bitant property at the FP level. We have taken special care to allow for symmetrization of the prepared ID by rewriting the interpolation process. The purpose here is to provide a test on preservation of FP symmetry across various elements of the infrastructure, such as, e.g. operator approximants (finite-differencing kernels, interpolation, etc.).
Figure 3. Evolution of the -component of the tracker during calibration BBH inspiral and merger. The CX and VC sampling choices are compared at fixed resolution. Parenthesis indicate ghost node selection. The “bit.ID” set evolved with the choice has ID explicitly symmetrized. All runs of this figure are performed over full 3D grids, regardless of ID symmetrization. Parameters: , and . See text for further discussion.
In the case of , explicit bitant symmetrization of the ID does lead to an evolution with exact confinement of the punctures to the plane. This is in contrast to runs in Figure 3.
Having passed a tracker-based consistency check, we now check gravitational waveforms (GW). To do this, we construct the Weyl scalar [45], and then interpolate onto geodesic spheres [64]. Subsequently, we project through numerical quadrature evaluation, onto spin-weighted spherical harmonics [65] of spin-weight . Our convention differs from the reference by a Condon-Shortley phase of . The above allows us to extract . The modes of the gravitational wave strain satisfy . The strain is then given by the mode-sum:
Following the convention of the LIGO algorithms library [66] we set:
and the gravitational-wave instantaneous frequency is defined through:
Note that the peak amplitude of the mode of is conventionally taken as the time of merger. We denote this .
The quantity is illustrated for the Calibration BBH in Figure 4. Rather than just imposing ID bitant symmetrization, symmetry reduction on the 3D grid is also imposed for a subset of runs. The waveforms continue to overlap consistently. This provides another important consistency test on implementation details.
Figure 4. Consistent behaviour between and for calibration BBH waveforms with and without bitant symmetry reduction. Black profiles depict the amplitude of the given mode based on calculations involving bitant symmetry reduction. Parameters: , and ; extraction radius .
Beyond verification of qualitative behaviour, we next focus on the gravitational strain, impose bitant symmetry reduction, and assess convergence properties based on Appendix A. The result, for constructing , based on time-domain integration (see e.g. [67]), is shown in Figure 5. A order compatible convergence trend is observed for the phase . This is present for both , and . We have also verified that behaviour persists, consistently between grid sampling choices, for e.g. the mode.
Figure 5. The mode of the gravitational strain (top-panel) together with convergence properties in the associated phase for CX and VC sampling in the middle and lower panels respectively. All quantities are taken as functions of the Schwarzschild tortoise coordinate . We assess order compatible convergence. Indicated in a common vertical black line across panels is . We select calculations involving bitant symmetry reduction based on the coarse (c), medium (m), and fine (f) resolution triplet induced through selecting . Extraction performed at .
As described in §II B AMR conditions may be freely selected. In the approach to a puncture location, the conformal factor approaches a local minimum (see e.g. [68]). Indeed, we have numerically verified that the trackers shown in Figure 2 can also be constructed based on such local minimum. On account of the underlying dynamical (approximate) symmetries of the problem, together with their impact on conserved quantities [69], and GW extraction taking place on spheres, an approximately matched AMR condition is motivated. To this end, spherically matched Mesh refinement is induced by fixing levels based on the collection . That is, two spherical regions of radius , with follow both punctures. A central fixed sphere of radius , and is set to improve resolution in the wave-zone. This is compared against pBIB in Figure 6 and Figure 7.
Figure 7. Zoom of the strong-field region of Figure 6. Trackers are overlaid where lighter color indicates increasing . Observe that the finest resolution in both runs is the same, however the region on the finest level is larger in the bottom right (dynamical spherical region) run. Note: color scalings are distinct between this figure and Figure 6.
Let us consider the behaviour of in the vicinity of a change in refinement level as shown in Figure 6. It is observed that small, spurious reflection occurs, from the underlying rectilinear grid structure. In contrast, by comparing the two runs, we see that approximate spherically matched AMR partially mitigates these spurious reflections. The magnitude of the effect can also be seen to converge away when comparing .
The change in refinement strategy also has an effect on GW phasing, which we show in Figure 8. At early times the amplitude and phasing error of the spherically matched AMR with is comparable to that of a pBIB run at . This can be attributed to a combination of increased wave-zone resolution and better symmetry matching of the AMR (See also the versus AMR discussion of [53]).
Figure 8. Comparison of differences in the amplitude of mode of the gravitational strain (top-panel) and phase (bottom-panel), for runs at different resolutions and refinement strategy, under . We consider pBIB AMR with corresponding to . A single run with spherically matched AMR at is denoted with . Vertical black line across panels indicates . Extraction performed at .
Another potential advantage of running the spherically matched AMR is a reduction in execution time. Running the spherically matched AMR at in bitant results in initially, where is as in Tab.I. Compared to pBIB AMR under bitant symmetry reduction at , it results in a computational resource reduction of when coarser time-step at fixed CFL is taken into account.
Finally, we provide a rough characterization of performance differences between and through a short timing test. To this aim we evolve the problem for the calibration BBH on the pBIB grid configurations of Tab.I. The initial checkpoint is taken at a time of , where the AMR condition has refined the Mesh. Timings are as summarized in Tab.II. We see that for pure vacuum, the space-time solver has a distinct advantage when run with the choice in terms of speed.
| (2 dp) | /Core | ||||
| 96 | 26.48 | 35.17 | 0.71 | 2404 | |
| 128 | 18.01 | 21.98 | 0.77 | 3280 | |
| 160 | 10.41 | 10.80 | 0.93 | 8774 | |
| 192 | 8.49 | 9.03 | 0.87 | 11490 |
caption: One hour timing test on SuperMUC-NG of LRZ utilizing 16 nodes. The timing ratio is defined as . The number of MeshBlock objects is provided at . Bitant symmetry reduction has been performed. The AMR is based on the pBIB criterion.
:::
Indeed, we observe an approximate average later final evolution time, across , when compared to the choice. This we attribute principally to a difference in complexity of the operations. A cautionary remark is however in order: the MeshBlock/CPU saturation has not been taken into account, which requires a certain threshold for efficiency [39, 40]. With regard to performance, these timing results nonetheless strongly suggest that is the better option for BBH.
Binary Neutron Stars
In the case of binary neutron star (BNS) evolution, in addition to the space-time evolution, we need to address the evolution of the hydrodynamical variables. These variables satisfy a balance law of the form [38]. The source term does
not involve derivatives of , but does contain geometric fields and their derivatives. Here the finite-volume, HRSC methods we employ lead us to take the evolved on cell-centers. In the approximation step, primitive variables are assembled and reconstructed to FC [58]. This process requires geometric data on FC. Information about the evolved fluid variable state, must be available on the grid used for the space-time solver. If we select the required intergrid transfer operations can be summarized as: , , and finally . On the other hand, selecting means we only need .
The intergrid transfer operations summarized above are handled through the use of symmetric Lagrange interpolation. Specifically, during evaluation, nearest neighbor nodes either side of a target interpolation point are used along a salient axis. We have found it beneficial, for stability reasons, to fix in a subset of operations. This we refer to as a hybrid mode. In this mode is only allowed to vary freely for derivative assembly entering , and primitives that recouple to the space-time solver.
As initial consistency checks for geometry-matter field coupling and the above operations we consider static star tests in Appendix A. A more representative setting of binary neutron star (BNS) evolution problems is required. For this, we consider a quasi-circular, equal-mass BNS inspiral-merger problem. Here irrotational, constraint-satisfying initial data is provided by Lorene [70]. Specifically, as a validation problem, we consider the G2_I14vs14_D4R33_45km binary7 which features baryon mass and gravitational mass at an initial separation of km. The Arnowitt-Deser-Misner (ADM) mass of the binary is , the angular momentum , and the initial orbital frequency is Hz. For these tests we utilize an ideal gas equation-of-state (see Appendix A) with . The evolution of this BNS system proceeds for three orbits prior to the formation of a massive remnant, thereafter leading to gravitational collapse.
The computational domain is again selected as , where km. The AMR is induced through tracking the local minimum of the lapse , as positioned on each constituent of the BNS system. Resolution levels are fixed according to . To perform a resolution study, we take a sequence of . This translates into a sequence of finest resolution levels of m. The MeshBlock sampling is selected as . For the time-evolution we make use of the SSPRK(3,3) method of [71], with a CFL of , and . A tenuous atmosphere is added, where . We do not make use of thresholding [40]. For the RePrimAnd-based primitive recovery we set the tolerance to . In this section no symmetry reductions are performed.
We select and evolve the initial data with the aforementioned parameters. An additional run featuring with and all else fixed is also performed for internal comparison. In each case we find approximately three orbits prior to the formation of a massive remnant. A snapshot of the fluid rest mass-density at this point () is shown in Figure 9 comparing the four runs. We observe that the spiral arm structure is
Figure 9. Mesh structure and values of fluid rest mass-density [g cm] shortly post-merger in the orbital plane for the BNS inspiral problem described in the text. Counter-clockwise, starting from the upper right quadrant we have evolution with CX sampling with . The lower right quadrant depicts evolution under VC sampling for geometric fields with . Each grey box (MeshBlock) is sampled with . The AMR criterion for each run has driven the distribution of refined region resolution towards an origin-centered approximate sphere. Trackers that follow local minima of are overlaid. Darker tracker color indicates increasing .
qualitatively similar for the two runs at , which provides an initial consistency check. The structure becomes sharper and more compact at higher . The extrema tracking based AMR can also be seen to guide increased resolution towards regions that suitably follow the binary constituents.
Quantitative scalar monitors are shown in Figure 10. We find that the time of collapse falls in the range for selection. This depends on the selection of . Importantly, we observe consistency to within 1% when comparing a verification run with based on . Violation in baryon mass conservation grows to a plateau of at merger for each run. This remains approximately constant until the
Figure 10. Relative change of (global) maximum density (upper panel), baryon mass conservation (middle), and minimum of the lapse (lower panel), for the BNS model problem. A resolution triplet induced by the choices and selection of is depicted, with a single run at and shown for reference. Collapse times, for each run, are signalled by: post-merger, global peak in the monitor, blow up of the mass conservation monitor, and confirmed by reduction of lapse below . Vertical black line depicts merger time (), as based on the peak amplitude associated with the run (C.f. Figure 11). See text for further discussion. Note: legend common to all panels.
point of gravitational collapse. Again we see that both choices of grid sampling for the geometric fields lead to consistent results.
Gravitational waveforms are the next item to verify. To construct the mode of the strain, for the BNS system, we use fixed frequency integration (FFI) [72]. The FFI proceeds by transforming the signal to the frequency representation, applying a low-frequency cutoff , and integrating. We select . The result is shown in Figure 11. Based on the run, we find that the time of merger (retarded coordinate) is . This translates into . Indeed, all CX runs fall within a band of . This indicates strong robustness of the simulations performed. The merger time for the waveform is within 3% of that of the CX at . Similarly, the morphology of the instantaneous frequency for CX when cross-compared against VC is seen to be alike. With the method described in Appendix A we also assess convergence. As can be seen in Figure 11 we have a strong indication for a second order trend in the convergence of the associated phase, as is increased.

Figure 11. Gravitational strain (2, 2) mode extracted at for binary neutron star run of §III B. (Top): real part of the strain with for is depicted. For the amplitude is also shown. In the case of we find (). We observe consistent behaviour at for a comparable run based on VC sampling. (Middle): Instantaneous frequency is shown. (Top, middle): vertical dashed lines indicate time of merger for the different data sets. (Bottom): Phase differences for the CX resolution triplet suggest convergence (to merger) as compatible with a second order trend.
Overall, based on this discussion, when running with , good agreement is found with here, and our prior results in [40]. This provides confidence in the correctness of our implementation, and the validity of the strategy outlined in §II B.
To close this section, as in the BBH case, we provide a rough characterization of performance differences between and through a short timing test. The initial checkpoint is taken at a time of with timings summarized in Tab. III.
| (2 dp) | #MB | #MB/Core | |||
| 64 | 170.45 | 151.02 | 1.13 | 3676 | |
| 96 | 70.84 | 61.11 | 1.17 | 6768 | |
| 128 | 34.78 | 28.36 | 1.27 | 12384 |
Table III. One hour timing test on SuperMUC-NG of LRZ utilizing 16 nodes. The timing ratio is defined as . The number of MeshBlock objects is provided at . No symmetry reduction has been performed. The AMR is based on spherical regions criteria.
We see that for BNS, the space-time solver has a distinct advantage when run with the choice in terms of speed. Indeed, we observe an approximate average later evolution time, across , when compared to the choice. This is in direct contrast to the BBH case (Tab. II). While it is the case that the level-to-level operators become more involved for CX, when contrasted against VC, the reduction in the number of intergrid transfer operations appears to compensate. Indeed, it appears that the gap between CX and VC ( of Tab. III) increases with increasing . This is also in contrast to the BBH case, where the gap appeared to decrease. Overall, we see that CX achieves later final evolved times in the BNS problem. Hence we conclude that is the favoured choice, for this class of problem.
in contrast to the BBH case, where the gap appeared to decrease. Overall, we see that CX achieves later final evolved times in the BNS problem. Hence we conclude that is the favoured choice, for this class of problem.
Conclusion
In this work we have proposed a new, cell-centered (CX) variable treatment within the context of our space-time solver embedded within GR-Athena++. This has allowed us to solve the Z4c system in mode, in a fashion consistent with our earlier work on vertex-centers, i.e. mode. Tests for both vacuum and GRHD problems demonstrated robust functionality for either grid sampling. The choice of for GR(M)HD yields a substantial simplification in treatment of discretized variables.
To validate the new treatment, together with implementation details within the code, first consistency tests based on BBH evolution were performed. A step-by-step strategy was taken which featured, inter alia, a resolution study. This allowed us to assess a trend compatible with order convergence in the error of the phase of the (2, 2) mode of the gravitational wave strain, when working with evolution of . This is consistent with our prior work [39]. In order to provide insight as to which grid sampling strategy performs better, for BBH evolution, we considered a one hour timing test on SuperMUC-NG of the Leibniz-Rechenzentrum Munich. It was found that , on average, is faster. This we attribute to the differences in how the level-to-level refinement operators are evaluated for the two grid samplings. Motivated by the “symmetry-seeking” property of the -AMR previously investigated in [53], we proposed, a simplified, spherical-region based AMR strategy. This was demonstrated in the BBH validation problem to potentially partially mitigate some rectilinear grid imprinting.
Following this, a quasi-circular, equal-mass, BNS inspiral, merger, and gravitational collapse problem was illustrated in the context of GRHD. We again utilized the (2, 2) mode of the gravitational wave strain as a consistency diagnostic. A trend compatible with order convergence was assessed in a resolution study when working with . This trend is consistent with that observed in [40] when working with . With regard to performance, in contrast to the BBH case, it was found that is, on average, faster for the runs considered. This we attribute to a reduction in the total number of intergrid interpolation operations required. For these tests, we also leveraged the simplified, spherical-region based AMR strategy. While we do not explicitly demonstrate GRMHD runs in this work, we have found another potential advantage of working with . First tests, involving the initial data of the BNS simulation presented here, allowed for evolution with magnetic fields through merger and collapse, without apparent horizonbased excision. This contrasts against the situation with ZVC where we have found such excision techniques to be crucial.
Acknowledgments
BD and SB acknowledges support by the EU Horizon under ERC Consolidator Grant, no. InspiReM-101043372. PH acknowledges funding from the National Science Foundation under Grant No. PHY-2116686. EG, DR acknowledge funding from the National Science Foundation under grant No. AST-2108467. EG acknowledges funding from an Institute for Gravitation and Cosmology fellowship. JF, DR acknowledge U.S. Department of Energy, Office of Science, Division of Nuclear Physics under Award Number(s) DE-SC0021177. DR acknowledge support from NASA under award No. 80NSSC21K1720 and from the Sloan Foundation. The authors are indebted to A. Celentano’s PRĪSENCŌLĪNENSĪĀINCĪUSŌL.
The numerical simulations were performed on the national HPE Apollo Hawk at the High Performance Computing Center Stuttgart (HLRS). The authors acknowledge HLRS for funding this project by providing access to the supercomputer HPE Apollo Hawk under the grant numbers INTRHYGUE/44215 and MAG-NETIST/44288. Simulations were also performed on SuperMUC NG at the Leibniz-Rechenzentrum (LRZ) Munich. The authors acknowledge the Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for funding this project by providing computing time on the GCS Supercomputer SuperMUC-NG at LRZ (allocations pn68wi and pn36jo). Simulations were also performed on TACC’s Frontera (NSF LRAC allocation PHY23001) and on Perlmutter. This research used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. Postprocessing and development run were performed on the ARA cluster at Friedrich Schiller University Jena. The ARA cluster is funded in part by DFG grants INST 275/334-1 FUGG and INST 275/363-1 FUGG, and ERC Starting Grant, grant agreement no. BinGraSp-714626.
References
Assessing convergence with a static star
Evolving a stable, static star in the general relativistic setting provides for a simple, sanity check on code correctness. To this end initial data (ID) is prepared based on the assumption of a spherically symmetric, self-gravitating matter distribution at equilibrium satisfying the Tolman-Oppenheimer-Volkoff (TOV) equations (see e.g. [73]). For ID preparation a polytropic EOS is assumed with adiabatic index , and polytropic constant as in [31]. The stable star model is taken to have a central density , gravitational mass , and circumferential radius . During evolution we make use of the more general EOS with . We perform an evolution of this data to a final time of . This is done for both ZVC and ZCX choices, for a variety of intergrid interpolation orders (see §III B). For these runs we have taken a CFL of , Kreiss-Oliger dissipation , and set artificial atmosphere to . The Z4c damping parameters are taken as . Scalar diagnostic quantities are shown in Figure 12.

Figure 12. Comparison of scalar quantities in stable star evolution (TOV test) on a fixed grid configuration involving CX and VC sampling where various varieties of intergrid interpolation of §III B are utilized. The Mesh and MeshBlock samplings are and respectively. The computational domain is taken as with a central, cubic, region refined with . The region completely contains the star. As intergrid interpolation order is increased it can be seen (upper panel) that central density is better preserved, with oscillation magnitude of reduced. Hamiltonian constraint violation (lower panel) is also reduced as interpolation order is increased. We find common mass conservation (middle panel) properties after regardless of interpolation order, or geometric sampling. Vertical grey line indicates first time of causal contact of with the origin.
To assess solution convergence we consider numerical evolution repeated at a triplet of coarse, medium, and fine resolutions that satisfy . The convergence rate of an approximation to the corresponding field data may be investigated by comparing differences of solutions at distinct resolutions. For one finds based on Taylor expansion that where we have introduced the so-called convergence factor:
The simulations featuring CX and VC of Figure 12 show the smallest deviation from the initial value during evolution, together with the smallest constraint violation. We construct a triplet of resolutions for convergence assessment by repeating the runs with . Based on the refinement structure this entails on the finest level, covering the star, for each run. Behaviour for the central density, together with the Hamiltonian constraint for our convergence assessment is shown in Figure 13. As can be seen we observe order, and order compatible trends for the aforementioned quantities respectively. Based on the consistency between the qualitative features of the CX and VC scalar diagnostics depicted in Figure 12, together with consistent order of convergence demonstrated in Figure 13 we consider the TOV test to pass.
Figure 13. Convergence of scalar quantities in the TOV test for a fixed grid configuration involving CX and VC sampling. Overall grid structure is as in Figure 12 where resolution triplets are constructed based selecting the level of refinement through . In dashed lines we depict the rescaled coarse-medium difference according to of Eq. (4). As additionally verified through least-squares fitting on we find: i.e. nd order convergence for (upper panel); for the Hamiltonian constraint (lower panel).
References
- [1]B. P. Abbott et al. (Virgo, LIGO Scientific), Phys. Rev. Lett. 119, 161101 (2017), arXiv:1710.05832 [gr-qc].
- [2]A. Goldstein et al., Astrophys. J. 848, L14 (2017), arXiv:1710.05446 [astro-ph.HE].DOI
- [3]V. Savchenko et al., Astrophys. J. 848, L15 (2017), arXiv:1710.05449 [astro-ph.HE].DOI
- [4]T. Akutsu et al., arXiv:2009.09305 [astro-ph, physics:gr-qc] (2020), arXiv:2009.09305 [astro-ph, physics:gr-qc].arxiv.org/abs/2009.09305
- [5]e. a. Amaro-Seoane, arXiv:1702.00786 [astro-ph] (2017), arXiv:1702.00786 [astro-ph].
- [6]M. Punturo, M. Abernathy, F. Acernese, B. Allen, N. Andersson, et al., Class.Quant.Grav. 27, 194002 (2010).
- [7]B. P. Abbott et al. (LIGO Scientific), Class. Quant. Grav. 34, 044001 (2017), arXiv:1607.08697 [astro-ph.IM].DOI
- [8]A. Nagar et al., Phys. Rev. D98, 104052 (2018), arXiv:1806.01772 [gr-qc].DOI
- [9]J. A. Font, Living Rev. Rel. 11, 7 (2007).
- [10]M. Shibata, K. Kiuchi, Y.-i. Sekiguchi, and Y. Suwa, Prog.Theor.Phys. 125, 1255 (2011), arXiv:1104.3937 [astro-ph.HE].DOI
- [11]D. Radice, S. Bernuzzi, A. Perego, and R. Haas, Mon. Not. Roy. Astron. Soc. 512, 1499 (2022), arXiv:2111.14858 [astro-ph.HE].DOI
- [12]D. Hilditch, A. Weyhausen, and B. Brügmann, Phys. Rev. D93, 063006 (2016), arXiv:1504.04732 [gr-qc].DOI
- [13]M. Bugner, T. Dietrich, S. Bernuzzi, A. Weyhausen, and B. Brügmann, Phys. Rev. D94, 084004 (2016), arXiv:1508.07147 [gr-qc].DOI
- [14]M. Fernando, D. Neilsen, H. Lim, E. Hirschmann, and H. Sundar, (2018), 10.1137/18M1196972, arXiv:1807.06128 [gr-qc].DOI
- [15]F. Löfmler et al., Class. Quant. Grav. 29, 115001 (2012), arXiv:1111.3344 [gr-qc].
- [17]K. Clough, P. Figueras, H. Finkel, M. Kunesch, E. A. Lim, and S. Tunyasuvunakool, (2015), arXiv:1503.03436 [gr-qc].DOI
- [18]W. Tichy, L. Ji, A. Adhikari, A. Rashti, and M. Pirog, Classical and Quantum Gravity 40, 025004 (2023), arxiv:2212.06340 [gr-qc].DOI
- [19]L. E. Kidder et al., J. Comput. Phys. 335, 84 (2017), arXiv:1609.00098 [astro-ph.HE].DOI
- [20]S. Rosswog and P. Diener, Class. Quant. Grav. 38, 115002 (2021), arXiv:2012.13954 [gr-qc].DOI
- [21]P. Grandclement and J. Novak, Living Rev. Rel. 12, 1 (2009), arXiv:0706.2286 [gr-qc].DOI
- [22]G. Doulis, F. Atteneder, S. Bernuzzi, and B. Brügmann, Phys. Rev. D 106, 024001 (2022), arXiv:2202.08839 [gr-qc].DOI
- [23]R. Alfieri, S. Bernuzzi, A. Perego, and D. Radice, Journal of Low Power Electronics and Applications Special Issue ”Energy Aware Scientific Computing on Low Power and Heterogeneous Architectures”, 8 (2018), 10.3390/jlpea8020015.
- [24]Q. F. Stout, D. L. De Zeeuw, T. I. Gombosi, C. P. T. Groth, H. G. Marshall, and K. G. Powell, in Proceedings of the 1997 ACM/IEEE Conference on Supercomputing ’97 (Association for Computing Machinery, New York, NY, USA, 1997) pp. 1–10.
- [25]M. J. Berger and J. Oliger, J.Comput.Phys. 53, 484 (1984).
- [26]C. Burstedde, J. Holke, and T. Isaac, Foundations of Computational Mathematics 19, 843 (2019).
- [27]G. M. Morton, A computer oriented geodetic data base and a new Tech. Rep. (1966).
- [28]T. Nakamura, K. Oohara, and Y. Kojima, Prog. Theor. M. Shibata and T. Nakamura, Phys. Rev. D52, 5428 (1995).
- [30]T. W. Baumgarte and S. L. Shapiro, Phys. Rev. D59, 024007 (1999), arXiv:gr-qc/9810065.DOI
- [31]S. Bernuzzi and D. Hilditch, Phys. Rev. D81, 084003 (2010), arXiv:0912.2920 [gr-qc].DOI
- [32]D. Hilditch, S. Bernuzzi, M. Thierfelder, Z. Cao, W. Tichy, and B. Bruegmann, Phys. Rev. D88, 084057 (2013), arXiv:1212.2901 [gr-qc].DOI
- [33]H. Friedrich, Communications in Mathematical Physics 100, 525 (1985).
- [34]F. Pretorius, Phys. Rev. Lett. 95, 121101 (2005), arXiv:gr-qc/0507014.DOI
- [35]L. Lindblom, M. A. Scheel, L. E. Kidder, R. Owen, and O. Rinne, Class.Quant.Grav. 23, S447 (2006), arXiv:gr-qc/0512093 [gr-qc].DOI
- [36]C. O. Lousto and J. Healy, Phys. Rev. Lett. 125, 191102 (2020), arXiv:2006.04818 [gr-qc].DOI
- [37]J. Lannelongue, H.-E. Aronson, A. Bateman, E. Birney, T. Caplan, M. Juckes, J. McEntyre, A. D. Morris, G. Reilly, and M. Inouye, Nature Computational Science 3, 514 (2023).
- [38]F. Banyuls, J. A. Font, J. M. A. Ibanez, J. M. A. Marti, and J. A. Miralles, Astrophys. J. 476, 221 (1997).
- [39]B. Daszuta, F. Zappa, W. Cook, D. Radice, S. Bernuzzi, and V. Morozova, Astrophys. J. Supp. 257, 25 (2021), arXiv:2101.08289 [gr-qc].DOI
- [40]W. Cook, B. Daszuta, J. Fields, P. Hammond, S. Albanesi, F. Zappa, S. Bernuzzi, and D. Radice, (2023), arXiv:2311.04989 [gr-qc].arxiv.org/abs/2311.04989
- [41]B. Daszuta and W. Cook, “GR-Athena++: Magnetohydrodynamical evolution with dynamical space-time,” (2024), arXiv:2406.05126 [astro-ph, physics:gr-qc].arxiv.org/abs/2406.05126
- [42]C. J. White, J. M. Stone, and C. F. Gammie, The Astrophysical Journal Supplement Series 225, 22 (2016), arXiv:1511.00943.DOI
- [43]K. G. Felker and J. M. Stone, Journal of Computational Physics 375, 1365 (2018).
- [44]J. M. Stone, K. Tomida, C. J. White, and K. G. Felker, The Astrophysical Journal Supplement Series 249, 4 (2020).
- [45]B. Brügmann, J. A. Gonzalez, M. Hannam, S. Husa, U. Sperhake, et al., Phys.Rev. D77, 024027 (2008), arXiv:gr-qc/0610128 [gr-qc].DOI
- [46]P. Mösta, B. C. Mundim, J. A. Faber, R. Haas, S. C. Noble, et al., Class.Quant.Grav. 31, 015005 (2014), arXiv:1304.5544 [gr-qc].
- [47]D. Radice, L. Rezzolla, and F. Galeazzi, Class.Quant.Grav. 31, 075012 (2014), arXiv:1312.5004 [gr-qc].DOI
- [48]M. Thierfelder, S. Bernuzzi, and B. Brügmann, Phys.Rev. D84, 044012 (2011), arXiv:1104.4751 [gr-qc].DOI
- [49]N. Fleischmann, S. Adami, and N. A. Adams, Computers & Fluids 89, 94 (2019).
- [50]A. Neumaier, ZAMM - Journal of Applied Mathematics and Mechanics / Zeitschrift für Angewandte Mathematik und Mechanik 54, 39 (1974).
- [51]J. M. Stone, K. Tomida, C. J. White, and K. G. Felker, Astrophys. J. Suppl. 249, 4 (2020), arXiv:2005.06651 [astro-ph.IM].DOI
- [52]J.-P. Berrut and L. N. Trefethen, SIAM Review 46, 501 (2004).
- [53]A. Rashti, M. Bhattacharyya, D. Radice, B. Daszuta, W. Cook, and S. Bernuzzi, Class. Quant. Grav. 41, 095001 (2024), arXiv:2312.05438 [gr-qc].DOI
- [54]M. Campanelli, C. O. Lousto, P. Marronetti, and Y. Zlochower, Phys. Rev. Lett. 96, 111101 (2006), arXiv:gr-qc/0511048.DOI
- [55]S. C. Noble, C. F. Gammie, J. C. McKinney, and L. Del Zanna, Astrophys. J. 641, 626 (2006), arXiv:astro-ph/0512420.DOI
- [56]W. Kastaun, J. V. Kalanin, and R. Ciolfi, Phys. Rev. D 103, 023018 (2021), arXiv:2005.01821 [gr-qc].DOI
- [57]R. Borges, M. Carmona, B. Costa, and W. S. Don, Journal of Computational Physics 227, 3191 (2008).
- [58]L. D. Zanna and N. Bucciantini, Astronomy & Astrophysics 390, 1177 (2002).
- [59]C. Bona, J. Massó, J. Stela, and E. Seidel, in The Seventh Marcel Grossmann Meeting: On Recent Developments, edited by R. T. Jantzen, G. M. Keiser, and R. Ruffini (World Scientific, Singapore, 1996).
- [60]M. Alcubierre, B. Brügmann, P. Diener, M. Koppitz, D. Pollney, et al., Phys.Rev. D67, 084023 (2003), arXiv:gr-qc/0206072 [gr-qc].DOI
- [61]J. R. van Meter, J. G. Baker, M. Koppitz, and D.-I. Choi, Phys. Rev. D73, 124011 (2006), arXiv:gr-qc/0605030.DOI
- [62]M. Ansorg, B. Brügmann, and W. Tichy, Phys. Rev. D70, 064011 (2004), arXiv:gr-qc/0404056.DOI
- [63]D. I. Ketcheson, Journal of Computational Physics 229, 1763 (2010).
- [64]N. Wang and J.-L. Lee, SIAM Journal of Scientific Computing 33, 2536 (2011).
- [65]J. N. Goldberg, A. J. MacFarlane, E. T. Newman, F. Rohrlich, and E. C. G. Sudarshan, J. Math. Phys. 8, 2155 (1967).
- [66]LIGO Scientific Collaboration, “LIGO Algorithm Library - LALSuite,” free software (GPL) (2018).DOI
- [67]S. Albanesi, A. Rashti, F. Zappa, R. Gamba, W. Cook, B. Daszuta, S. Bernuzzi, A. Nagar, and D. Radice, “Scattering and dynamical capture of two black holes: Synergies between numerical and analytical methods,” (2024), arXiv:2405.20398 [gr-qc].DOI
- [68]M. Thierfelder, S. Bernuzzi, D. Hilditch, B. Brügmann, and L. Rezzolla, Phys.Rev. D83, 064022 (2011), arXiv:1012.3703 [gr-qc].DOI
- [69]V. Mewes, Y. Zlochower, M. Campanelli, T. W. Baumgarte, Z. B. Etienne, F. G. Lopez Armengol, and F. Cipolletta, Phys. Rev. D 101, 104007 (2020), arXiv:2002.06225 [gr-qc].DOI
- [70]E. Gourgoulhon, P. Grandclement, K. Taniguchi, J.-A. Marcq, and S. Bonazzola, Phys.Rev. D63, 064029 (2001), arXiv:gr-qc/0007028 [gr-qc].DOI
- [71]S. Gottlieb, D. I. Ketcheson, and C.-W. Shu, Journal of Scientific Computing 38, 251 (2009).
- [72]C. Reisswig and D. Pollney, Class.Quant.Grav. 28, 195015 (2011), arXiv:1006.1632 [gr-qc].DOI
- [73]J. A. Font, M. A. Miller, W.-M. Suen, and M. Tobias, Phys. Rev. D61, 044011 (2000), gr-qc/9811015.










