betaFeedback

WorksPhysics and Astronomy

Numerical relativity simulations of compact binaries: comparison of cell- and vertex-centered adaptive meshes

Abstract

Given the compact binary evolution problem of numerical relativity, in the finite-difference, block-based, adaptive mesh refinement context, choices must be made on how evolved fields are to be discretized. In GR-Athena++, the space-time solver was previously fixed to be vertex-centered. Here, our recent extensions to a cell-centered treatment, are described. Simplifications in the handling of variables during the treatment of general relativistic magneto-hydrodynamical (GRMHD) evolution are found. A novelty is that performance comparison for the two choices of grid sampling is made within a single code-base. In the case of a binary black hole inspiral-merger problem, by evolving geometric fields on vertex-centers, an average ∼20%\sim 20\% speed increase is observed, when compared against cell-centered sampling. The opposite occurs in the GRMHD setting. A binary neutron star inspiral-merger-collapse problem, representative of typical production simulations is considered. We find that cell-centered sampling for the space-time solver improves performance, by a similar factor.

Introduction

Fashioning accurate description of astrophysically sourced, binary constituent encounters, under extreme conditions, is a pressing challenge for the numerical relativity (NR) community. Multimessenger detection efforts [1, 2, 3] benefit when supplemented by such data. Observational efforts of next-generation gravitational-wave detectors [4, 5, 6, 7] will also greatly benefit from higher fidelity reference numerical data, that spans a larger range of parameter space (or e.g. models [8] suitably informed by). To confront simulation challenges, sophisticated general relativistic, magneto-hydrodynamical (GRMHD) techniques (see e.g. [9]), and microphysics descriptions [10, 11] have been pursued. In tandem, the development of code infrastructure, and algorithms, with the aim of improved performance scaling on modern high-performance computing (HPC) resources, has also received recent, particular attention. Some examples of modern NR codes under active development with these goals include: bamps [12, 13], Dendro-GR [14], Einstein Toolkit [15] (GRaM-X [16]), GRChombo [17], NMesh [18], SpECTRE [19], and SPHINCS_BSSN [20]. A variety of numerical methods [21, 22, 23], algorithmic approaches [24, 25, 26, 27], and NR formulations [28, 30, 31, 32, 33, 34, 35] [29] are employed, to list but a few prominent examples. In spite of this, the actual time taken (HPC resource consumed) in the calculation of numerical evolutions, particularly for extreme scenarios that require highly resolved physical scales, can require weeks, or months [36]. Clearly, even minor performance optimizations, of a few percent, can have a significant impact on such HPC wait times, together with broader, sustainability considerations [37].

In this work, we focus on finite-difference based evolution of the Z4c formulation [31, 32] of NR, as coupled to the general relativistic, hydrodynamical (GRHD) formulation1 of [38]. This is precisely the setting treated by our code GR-Athena++ [39, 40] (see also [41]). In brief, this is our extension to the astrophysical (radiation), GRMHD code Athena++ [42, 43, 44], we have geared toward evolution with dynamical space-time. Building upon the underlying octree, block-based adaptive-mesh-refinement (AMR) infrastructure, with addition of our Z4c based space-time solver both in vacuum [39], and for GRMHD with dynamical space-time [40], has led to demonstrable scaling inefficiencies in excess of 80% on (pre-)exascale HPC infrastructure.

Specifically, here, we aim to address a seemingly innocuous question: should we select the geometric fields, that is, the dynamical fields of Z4c, to be discretized on vertex-centers (VC) or cell-centers? Our initial choice was based on considerations for binary black hole evolution, where transfer of data between differing levels of refinement, is less computationally expensive for VC in contrast to cell-centers. On the other hand, in consideration of GR(M)HD, typically hydrodynamical sampling is on cell-centers (see e.g. BAM [45], GRHydro [46], and WhiskyTHC [47]). This choice is in part due to greater amenability of the underlying hydrodynamical treatment as based on conservative, high-resolution-shock-capture (HRSC) methods (see e.g. [48]). Athena++ also handles hydrodynamical description on cell-centers. Having previously fixed geometric fields to VC, discretized fields must consequently be transferred between grids, which incurs an overhead. A priori it is not necessarily clear as to how large this is. Our aim here is to investigate switching to a cell-centered treatment of the space-time solver in GR-Athena++, thus providing an answer. Such a code-internal investigation does not appear to have been previously performed. This gap we fill here. This paper proceeds as follows. In §II we briefly explain the computational domain structure and the variable discretization strategy. Subsequently, we present a simplified refinement strategy, tailored to the binary merger problem, and independent of geometric field sampling selection. In §III we demonstrate the robustness of the technique through numerically solving BBH and BNS inspiral-merger problems, representative of typical production simulations. Section §IV concludes.

Method

GR-Athena++ [39, 40] builds upon Athena++ [42, 43, 44] and thus inherits overall features of the latter framework. In brief, problems are formulated over a target computational domain Ω\Omega (the Mesh) which is partitioned into a collection of sub-domains Ωi\Omega_i (MeshBlock objects). We have Ω=⨆i∈ZΩi\Omega= \bigsqcup_{i\in Z}\Omega_i where ZZ may be thought of as an ordered parametrization of multi-dimensional coordinates to a space-filling curve [26, 27]. For a dd-dimensional problem the overall extent of Ω\Omega together with suitable boundary data on ∂Ω\partial\Omega must be provided. Following this, the underlying discretization is fixed through specification of Mesh and MeshBlock sampling NM=(NM1,…,NMd)N_M = (N_{M_1}, \ldots, N_{M_d}) and NB=(NB1,…,NBd)N_B = (N_{B_1}, \ldots, N_{B_d}), respectively, where NBN_B must divide NMN_M component-wise2^2. In the case that (local) refinement is desired, MeshBlock partitioning into 2d2^d new MeshBlock objects is recursively performed. This involves halving size along each axis at a fixed number of samples, which doubles the resolution at each step. The aforementioned is done while maintaining an (at-most) 2:12:1 refinement ratio between neighboring {Ωi}i∈Z\{\Omega_i\}_{i\in Z}. While the logical-rectilinear structure here described may then be further mapped to suitable curvilinear coordinates, for simplicity we focus on the Cartesian context.

Cell-centered extension

In GR-Athena++ [39, 40] we extended sampling support to enable vertex-centered (VC) discretization of field variables together with infrastructure enabling high-order approximations to be carried out. In this work we focus on analogous extensions in description of geometric quantities on cell-centers (CX). A variable V(x)\mathcal{V}(x) where x∈[a,b]x \in[a,b] (a<ba < b) is CX discretized3^3 to Vi+1/2\mathcal{V}_{i+1/2} over

xi+1/2:=a+(i+1/2)δxx_{i+1/2} := a + (i+1/2)\delta x where δx:=(b−a)/NB\delta x := (b-a)/N_B and i=0,…,NB−1i = 0,\ldots,N_B-1. This yields NBN_B total samples. In practice the interval is extended by an additional NgN_g “ghost” nodes either-side4^4. This extension facilitates evaluation of e.g. derivative approximations, transfer of field data between neighboring MeshBlock objects that may differ in the level of refinement, and/or imposition of boundary conditions.

In order to populate ghost nodes, communication is required. In the case that V\mathcal{V} is discretized over a Mesh decomposed at a fixed, common level, under CX sampling, this entails that nearest-neighbor MeshBlock objects can be utilized to populate ghost-layers through a straightforward copy. Suppose instead that sub-domains differ in level ll. In this situation, analogous to the VC case, each sub-domain lΩi^l\Omega_i (where prefix ll indicates level) as described by a MeshBlock, features a fundamental and an additional coarse representation of the sampled variable. This latter is one level above at l−1l-1. Thus if NBN_B is the number of physical points on a MeshBlock extended by 2Ng2N_g ghost nodes, then the complementary coarse representation features NB/2N_B/2 physical points extended by 2Ng2N_g coarse ghosts. The situation is illustrated (locally) in Figure 1.

Figure 1. One-dimensional illustration of inter-MeshBlock communication given a refined local Mesh structure. Each MeshBlock features a fundamental representation of a sampled field depicted through square nodes which is extended in each direction through ghost nodes (circles) that facilitate communication. Additionally a complementary coarse sampling is present which is explicitly depicted for data on level ll and suppressed on level l−1l-1. Here NB=12N_B = 12, Ng=2N_g = 2, and Ncg=3N_{cg} = 3. Prior to all communication data in filled square nodes is available; the task is thus to fill data at locations indicated by shaded circles and diamonds. Nodes are assigned MeshBlock-local labels for ease of discussion.

The general communication strategy and ordering of operations closely follows and extends that of Athena++. This minimizes the amount of infrastructure changes required. Synchronization of data across levels may be understood and implemented by considering the transfer operations: fine to coarse (restriction R\mathcal{R}), and coarse to fine (prolongation P\mathcal{P}). As an example consider synchronization of ghost-layer data between l−1ΩA^{l-1}\Omega_A and lΩB^{l}\Omega_B as depicted in Figure 1. Initially data on the fundamental grid of lΩB^{l}\Omega_B over the nodes (is, is+1, . . . , ie) is known. This is first restricted to a subset of the complementary coarse grid of lΩB^{l}\Omega_B over the nodes (cime+1, . . . , cime+NgN_g); see green diamonds in the figure. Subsequently data at the nodes (ie + 1, . . . , ipe) of l−1ΩA^{l-1}\Omega_A may be filled through communication or a copy (green circles in figure). Conversely using data from l−1ΩA^{l-1}\Omega_A at the nodes (ie − NcgN_{cg} + 1, . . . , ie) allows for filling of (cims, . . . , cime) on lΩB^{l}\Omega_B (purple circles) through communication or a copy. Finally, ghost layer nodes (ims, . . . , is − 1) of the fundamental grid over lΩB^{l}\Omega_B may be filled (blue circles) through P\mathcal{P} based on data local to the MeshBlock. The procedure is analogous for the pair lΩC^{l}\Omega_C and l−1ΩD^{l-1}\Omega_D.

Comparing node positions in Figure 1 across differing levels of refinement clearly shows that nodes do not align but are fully staggered5. This is one aspect where CX differs from VC sampling [39]. The coarsening and refining of sampled field data via R\mathcal{R} and P\mathcal{P} respectively, is based on polynomial interpolation under CX and VC.

Due to the inter-level node staggering occurring in CX, a non-trivial evaluation of interpolants for every target node is required. This is in contrast to the case of VC where a subset of operations reduces to injection (copy) [39]. The distinction heralds a performance implication as will be demonstrated in §III A and §III B.

We close this subsection with a collection of technical points concerning implementation aspects.

  • The formal order of accuracy for the R\mathcal{R} and P\mathcal{P} operations on CX is controlled by the number of nodes selected. We fix this choice based on NgN_g selection so to match the approximation order with that of the finite-differencing.

  • Given a target node, source data entering an interpolant describing CX P\mathcal{P} has a directional bias due to grid staggering. This can be understood by noting that the source data nodes are not symmetrically placed about the location of the target node, but offset. Consider at a desired formal order of accuracy, the initial coarse, complementary restriction needed when communicating data to a finer neighboring MeshBlock. There may be insufficient nodes available to evaluate with symmetrically spaced source data. In this situation, we again bias the stencil selection.

  • As is well-known, floating point (FP) arithmetic is not necessarily associative: (a+b)+c≠a+(b+c)(a+b)+c \ne a+(b+c) for FP numbers {a,b,c}\{a, b, c\}. If care is not taken in how numerical expressions are ordered, errors can e.g. lead to catastrophic cancellations, or result in spurious symmetry breaking during simulations [49]. A combination of Kahan-Babushka-Neumaier compensated summation [50], and explicit symmetrization of arithmetic operations involving averaging can partially mitigate this [49]. This can be optionally activated for interpolant evaluation. We have not however found it necessary for the simulations presented here.

  • There are now two styles of cell-centered sampling where discretized field variables may be registered and stored: that inherited from Athena++, which we denote CC, and the newly introduced CX. The principal distinction is that for a Mesh featuring MeshBlock objects on distinct levels, the R\mathcal{R} and P\mathcal{P} operators are distinct: for CC there is minmod based slope-limiting [51], whereas for CX we utilize the Barycentric Lagrange approach [52]. Special care has been taken to ensure that distinct fields may be suitably registered as either CC or CX with their respective, logic available concurrently.

  • The VC double-restriction operation described in [39] we have found to not be strictly required. Its

removal means that when utilizing vertex-centered sampling we now have NB≥max⁡(4,Ng)N_{\mathrm{B}} \ge\max(4,\mathcal{N}_{g}). This allows for smaller MeshBlock size, which offers greater flexibility in tailoring resolution adapted regions.

Refinement strategy

In order to resolve dynamical features over widely disparate spatial scales we utilize adaptive mesh refinement (AMR). Our purpose here is to fix notation, and to point to two simple, minimal criteria that will facilitate consistency tests of the new infrastructure. The aforementioned feature evolution of initial data describing a binary black hole (BBH) (§III A), and a binary neutron star (§III B). For a comprehensive investigation of AMR criteria in GR-Athena++, and effect on waveform quality for BBH evolution, see [53].

The AMR conditions of GR-Athena++ are quite flexible. A user-specified function is defined for each MeshBlock; this function returns a flag that controls whether to (de)-refine. Consider puncture-based evolution [45] of a BBH system. Here a black hole center is described through a puncture. The position of a puncture xp(t)x_{\mathrm{p}}(t) can be computed by augmenting the dynamical system evolved through an additional ODE based on gauge information [54]. In short, this is one variety of a so-called tracker. This allows us to construct a dynamically adjusted, octree-based, pseudo-box-in-box (pBIB) refined Mesh as detailed in [39] (or L∞L_{\infty}-norm refinement in the language of [53]).

Let us suppose the Mesh is the Cartesian coordinatized, computational domain Ω:=[−D,D]3\Omega:= [-D,D]^3. A parameter controlling the maximal level of refinement NLN_{\mathrm{L}} is specified. A hierarchy of MeshBlock objects is then constructed, which increases in resolution as a puncture is approached (see e.g. Figure 6). On the finest resolution level the grid spacing will be δx=2D/(NM2NL−1)\delta x = 2D/(N_{\mathrm{M}}2^{N_{\mathrm{L}}-1}) [39]. As an alternative to pBIB, we can exploit information from the trackers and directly impose a desired region to be at fixed resolution (equivalently level). Denote a spherical region of radius RR, centered at pp as SR(p)S_{R}(p). We center such a region on a puncture and impose a desired local level of refinement ll. This we write as Sl,R(xp(t))S_{l,R}(x_{\mathrm{p}}(t)).

Figure 6. Mesh structure and values of γ~xx\tilde{\gamma}_{xx} under ZCX\mathcal{Z}_{\mathrm{CX}} at t≃110Mt \simeq110M in the orbital plane. Each quadrant depicts distinct resolutions (AMR choices) for the same physical calibration problem configuration. Counter-clockwise, starting from the upper right quadrant we have pBIB AMR with NM∈{96,128,192}N_M \in\{96, 128, 192\}. The lower right quadrant depicts evolution with the dynamical spherical AMR criterion, and NM=96N_M = 96 fixed. Each grey box (MeshBlock) is sampled with NB=16N_B = 16 points along each axis. For the cases involving the pBIB AMR criterion observe that reflection of outward propagating dynamical spherical features in γ~xx\tilde{\gamma}_{xx} is present in the vicinity of a change in refinement level. See text for further discussion.

In a similar vein, consider a dynamical, user selected, control field χ\chi featuring a local extrema χ∗\chi^{*}. Another type of tracker is possible to construct based on following the time development of χ∗(t)\chi^{*}(t) through Sl,R(χ∗)S_{l,R}(\chi^{*}). This allows for the region surrounding the extrema to be captured as its position changes in time. If multiple such regions are registered, occur at different ll, and happen to intersect, then we take the finest level as the target level. This is the principal condition that we employ for AMR during BNS evolution.

Validation

Our first goal in this section is validating the extended cell-centered implementation in the context of vacuum binary black hole evolution. The space-time solver implemented in GR-Athena++ is based on the Z4c formulation [31, 32]. Here the space-time is constructed based on the evolution of the dynamical, geometric variables Z:=(χ,γ~ij,K^,A~ij,Θ,Γ~i)\mathcal{Z} := (\chi,\tilde{\gamma}_{ij},\hat{K},\tilde{A}_{ij},\Theta,\tilde{\Gamma}^{i}). The definition and meaning of these variables is standard and may be found, together with implementation details, in [39]. Previously, motivated by AMR efficiency considerations in the vacuum context, we have chosen these to evolve over VC grid sampling. We compactly denote this through ZVC\mathcal{Z}_{\mathrm{VC}}. This work details a relaxation of this restriction, extending functionality to allow for CX sampling, i.e. sampling and evolving directly as ZCX\mathcal{Z}_{\mathrm{CX}}. We detail tests of this in §III A.

Secondly, we investigate the binary neutron star (BNS) inspiral-merger problem. Our recently implemented GRMHD treatment in GR-Athena++ is based on the formulation of [38]. For full details, and conventions, see [40]. Here we recall key ingredients of constructing the matter side of the evolution as based on selecting ZVC\mathcal{Z}_{\mathrm{VC}}. For simplicity, we restrict our exposition to the GRHD sector. For the hydrodynamics, the (conserved) field variables are QA:=(D,Sj,τ)\mathcal{Q}_{A} := (D,S_{j},\tau), and satisfy a balance law. Recall that this is evolved based on conservative, finite-volume, HRSC techniques [48]. The natural choice of sampling, that also leverages extant Athena++ infrastructure is cell-centered, i.e. we take QA,CC\mathcal{Q}_{A,\mathrm{CC}}. We also work with a set of complementary, primitive variables RA:=(ρ,u~i,p)\mathcal{R}_{A} := (\rho,\tilde{u}^{i},p). The relation QA(R)\mathcal{Q}_{A}(\mathcal{R}) is non-linearly implicit, and is evaluated through numerical inversion [55]. In this work the conservative to primitive variable mapping is handled through the use of RePrimAnd [56]. This inversion requires geometric fields to be available on the same grid sampling as the matter. Hence an intergrid interpolation ZVC→ZCC\mathcal{Z}_{\mathrm{VC}} \to\mathcal{Z}_{\mathrm{CC}} is required.

In order to evolve the quantities QA,CC\mathcal{Q}_{A,\mathrm{CC}} via HRSC we require assembly of fluxes at cell-interfaces (that is, face-centers (FC)). In this work, this is achieved based on the reconstruction of RA,CC\mathcal{R}_{A,\mathrm{CC}} to FC with the fifth order method of [57] denoted WENO5Z. Flux assembly also requires a subset of ZFC\mathcal{Z}_{\mathrm{FC}} and hence another interpolation. The interface fluxes are combined according to the Local-Lax-Friedrichs (LLF) scheme (see e.g. [58]). Finally the sources appearing in the hydrodynamical balance law are assembled, which again requires ZCC\mathcal{Z}_{\mathrm{CC}}. On the other hand, the gravitational matter density, momenta, and stress quantities that recouple the system to the space-time solver, must be provided on VC. This entails the interpolation of hydrodynamical quantities to VC. It is clear from the above summary, that by instead taking the evolved geometric variables as ZCX\mathcal{Z}_{\mathrm{CX}} from the start, fewer transfers of geometric data between grids would be required. We further validate our code functionality, for BNS, in §III B.

In addition to ensuring consistency of the implementation, and that problems are solved correctly, a practical performance assessment is needed. We demonstrate that execution speed, and suitable choice of ZVC\mathcal{Z}_{\mathrm{VC}} (ZCX\mathcal{Z}_{\mathrm{CX}}) is contingent on the class of physical problem simulated.

Binary Black Holes

We assess the performance of our code by focusing on the well-known calibration binary black hole (BBH) problem of [45]. The Z4c system is coupled to the moving puncture gauge using the 1+log⁡1+\log-slicing condition [59] and the Gamma driver shift [60, 61], with parameters as described in [39]. Initial data is generated through the use of the TwoPunctures6 library based on [62]. We model two initial non-spinning black holes of bare-mass m±=0.483 Mm_{\pm}=0.483\,M, located on the xx-axis, with x±1(t=0)=±3.257 Mx^{1}_{\pm}(t=0)=\pm3.257\,M, and initial momenta directed along the yy-axis, p±2(t=0)=∓0.133 Mp^{2}_{\pm}(t=0)=\mp0.133\,M. These parameter choices lead to an evolution featuring ∼3\sim3 orbits to merger, which occurs at an evolution time t∼170 Mt\sim170\,M.

In these tests the Courant-Friedrich-Lewy (CFL) is fixed at C1=0.2C_{1}=0.2. This choice determines the global time-step taken on each MeshBlock based on the finest grid-spacing (occuring in the vicinity of a puncture and denoted δxp\delta x_{p}). Kreiss-Oliger dissipation is selected as σD=0.02\sigma_{\mathrm{D}}=0.02 throughout. The Z4c constraint damping parameters are taken as (κ1,κ2)=(0.02,0)(\kappa_{1},\kappa_{2})=(0.02,0) (see [39]). Unless otherwise stated, in this section we take Ng=Ncg\mathcal{N}_{g}=\mathcal{N}_{cg}, that is, the number of ghosts on the fundamental grid is set equal to the number on the complementary coarse representation (see Figure 1). We select Ng=4\mathcal{N}_{g}=4, which induces a formal 6th6^{\mathrm{th}} order of spatial approximation. Time-evolution is performed using the explicit 4th4^{\mathrm{th}} order RK4()4[2S] [63].

The overall grid configurations based on pBIB refinement (see §II B) are summarized in Tab. (I). In principle

NM\mathcal{N}_{M}δxp×10−2[M]\delta x_{p}\times10^{-2}[M]#MB
963.1255144
1282.3446336
1601.87518640
1921.56322392

caption: Properties of grid configurations utilized for vacuum tests. The total number of MeshBlock objects (#MB) at the initial time is indicated. In all cases the Mesh is the Cartesian coordinatized domain Ω:=[−D,D]3\Omega:=[-D,D]^{3}, where D=1536MD=1536M is selected. This entails that grid spacing is uniform along each spatial axis. In each case we select the maximal refinement through NL=11\mathcal{N}_{L}=11. The MeshBlock sampling is selected as NB=16\mathcal{N}_{B}=16. Resultant puncture resolutions δxp\delta x_{p} are also provided.

:::

the calibration BBH problem possesses a underlying bitant symmetry (i.e. reflection across the z=0z=0 plane to which the co-orbit and merger is confined). This allows for an approximate halving of the total computational resources required for a given problem. When imposing bitant symmetry, #MB as listed in Tab. I is roughly halved, and in order to keep the grid spacing fixed, we halve the Mesh sampling parameter along the zz-axis to NM/2\mathcal{N}_{M}/2. As an initial consistency check, we select the geometric sampling as ZVC\mathcal{Z}_{\mathrm{VC}} and the NM=96\mathcal{N}_{M}=96 grid of Tab. I. We perform a comparable run with ZCX\mathcal{Z}_{\mathrm{CX}}, for which we inspect the puncture tracker evolution in Figure 2. Excellent agreement can be seen.

Figure 2. Tracker evolution during calibration BBH inspiral and merger. The one-parameter families x±(t)x_{\pm}(t) describe where punctures are located. Consistent behaviour for cell-centered extended (CX) and vertex-centered (VC) choices of sampling is observed. Parameters: NM=96\mathcal{N}_{M}=96, and NB=16\mathcal{N}_{B}=16.

The BBH evolution considered here should result in an evolution with a strict symmetry about the z=0z=0 plane. During numerical evolution, however, this condition is not necessarily preserved. The zz component of a puncture can potentially slowly drift out of the plane. The behaviour is demonstrated in Figure 3 for full 3D runs. In the case of ZCX\mathcal{Z}_{\mathrm{CX}}, based on the structure of the level-to-level transfer operators, we do not observe, nor expect, strict FP symmetry preservation. Instead, during the full 3D run we find a very minor drift developing, where the zz-component of the puncture reaches ∼O(10−10)\sim O(10^{-10}). We also verify in Figure 3 that Ncg\mathcal{N}_{cg} may indeed be independently selected, so as to increase the formal order of accuracy in the prolongation operation for P\mathcal{P}. We have similarly verified this for ZCX\mathcal{Z}_{\mathrm{CX}}. As this does not appear to substantially change simulation quality for the short evolution here, we proceed with runs that fix Ng=Ncg=4N_g = N_{cg} = 4. One reason for this behaviour, is that the TwoPunctures library does not allow for explicit enforcement of an underlying symmetry about z=0z = 0. Thus, when interpolating an initial data (ID) set to a Cartesian grid, the result violates the bitant property at the FP level. We have taken special care to allow for symmetrization of the prepared ID by rewriting the interpolation process. The purpose here is to provide a test on preservation of FP symmetry across various elements of the infrastructure, such as, e.g. operator approximants (finite-differencing kernels, interpolation, etc.).

Figure 3. Evolution of the zz-component of the tracker x−(t)\mathbf{x}_{-}(t) during calibration BBH inspiral and merger. The CX and VC sampling choices are compared at fixed resolution. Parenthesis (Ng,Ncg)(N_g, N_{cg}) indicate ghost node selection. The “bit.ID” set evolved with the choice ZCX\mathcal{Z}_{\mathrm{CX}} has ID explicitly symmetrized. All runs of this figure are performed over full 3D grids, regardless of ID symmetrization. Parameters: NM=96N_M = 96, and NB=16N_B = 16. See text for further discussion.

In the case of ZVC\mathcal{Z}_{\mathrm{VC}}, explicit bitant symmetrization of the ID does lead to an evolution with exact confinement of the punctures to the z=0z = 0 plane. This is in contrast to runs in Figure 3.

Having passed a tracker-based consistency check, we now check gravitational waveforms (GW). To do this, we construct the Weyl scalar Ψ4\Psi_4 [45], and then interpolate onto geodesic spheres [64]. Subsequently, we project through numerical quadrature evaluation, onto spin-weighted spherical harmonics [65] sYℓm{}_{s}Y_{\ell m} of spin-weight s=−2s = -2. Our convention differs from the reference by a Condon-Shortley phase of (−1)m(-1)^m. The above allows us to extract ψℓm\psi_{\ell m}. The (ℓ,m)(\ell,m) modes of the gravitational wave strain hh satisfy ψℓm=h¨ℓm\psi_{\ell m} = \ddot{h}_{\ell m}. The strain is then given by the mode-sum:

R(h+−ih×)=∑ℓ=2∞∑m=−ℓℓhℓm(t) −2Yℓm(ϑ,φ).(1)R(h_{+} - ih_{\times}) = \sum_{\ell=2}^{\infty}\sum_{m=-\ell}^{\ell} h_{\ell m}(t)\,{}_{-2}Y_{\ell m}(\vartheta,\varphi). \tag*{(1)}

Following the convention of the LIGO algorithms library [66] we set:

Rhℓm=Aℓmexp⁡(−iϕℓm),(2)Rh_{\ell m} = A_{\ell m}\exp(-i\phi_{\ell m}), \tag*{(2)}

and the gravitational-wave instantaneous frequency is defined through:

ωℓm=ddtϕℓm.(3)\omega_{\ell m} = \frac{d}{dt}\phi_{\ell m}. \tag*{(3)}

Note that the peak amplitude of the (2,2)(2,2) mode of hh is conventionally taken as the time of merger. We denote this umergeru_{\mathrm{merger}}.

The quantity ψ22\psi_{22} is illustrated for the Calibration BBH in Figure 4. Rather than just imposing ID bitant symmetrization, symmetry reduction on the 3D grid is also imposed for a subset of runs. The waveforms continue to overlap consistently. This provides another important consistency test on implementation details.

Figure 4. Consistent behaviour between ZVC\mathcal{Z}_{\mathrm{VC}} and ZCX\mathcal{Z}_{\mathrm{CX}} for calibration BBH waveforms with and without bitant symmetry reduction. Black profiles depict the amplitude of the given mode based on calculations involving bitant symmetry reduction. Parameters: NM=96N_M = 96, and NB=16N_B = 16; extraction radius R=85MR = 85M.

Beyond verification of qualitative behaviour, we next focus on the gravitational strain, impose bitant symmetry reduction, and assess convergence properties based on Appendix A. The result, for constructing hℓmh_{\ell m}, based on time-domain integration (see e.g. [67]), is shown in Figure 5. A 6th6^{\mathrm{th}} order compatible convergence trend is observed for the phase ϕ22\phi_{22}. This is present for both ZVC\mathcal{Z}_{\mathrm{VC}}, and ZCX\mathcal{Z}_{\mathrm{CX}}. We have also verified that behaviour persists, consistently between grid sampling choices, for e.g. the (4,4)(4,4) mode.

Figure 5. The (2,2)(2, 2) mode of the gravitational strain (top-panel) together with convergence properties in the associated phase for CX and VC sampling in the middle and lower panels respectively. All quantities are taken as functions of the Schwarzschild tortoise coordinate uu. We assess 6th6^{\mathrm{th}} order compatible convergence. Indicated in a common vertical black line across panels is umerger≃158 Mu_{\mathrm{merger}} \simeq158\,M. We select calculations involving bitant symmetry reduction based on the coarse (c), medium (m), and fine (f) resolution triplet induced through selecting NM∈{128,160,192}N_M \in\{128, 160, 192\}. Extraction performed at R=85MR = 85M.

As described in §II B AMR conditions may be freely selected. In the approach to a puncture location, the conformal factor χ\chi approaches a local minimum (see e.g. [68]). Indeed, we have numerically verified that the trackers shown in Figure 2 can also be constructed based on such local minimum. On account of the underlying dynamical (approximate) symmetries of the problem, together with their impact on conserved quantities [69], and GW extraction taking place on spheres, an approximately matched AMR condition is motivated. To this end, spherically matched Mesh refinement is induced by fixing levels based on the collection {S11,3(x±⋆),S6,100(0)}\{S_{11,3}(x_{\pm}^{\star}), S_{6,100}(0)\}. That is, two spherical regions of radius 3M3M, with NL=11N_L = 11 follow both punctures. A central fixed sphere of radius 100M100M, and NL=6N_L = 6 is set to improve resolution in the wave-zone. This is compared against pBIB in Figure 6 and Figure 7.

Figure 7. Zoom of the strong-field region of Figure 6. Trackers are overlaid where lighter color indicates increasing tt. Observe that the finest resolution in both NM=96N_M=96 runs is the same, however the region on the finest level is larger in the bottom right (dynamical spherical region) run. Note: color scalings are distinct between this figure and Figure 6.

Let us consider the behaviour of γ~xx\tilde{\gamma}_{xx} in the vicinity of a change in refinement level as shown in Figure 6. It is observed that small, spurious reflection occurs, from the underlying rectilinear grid structure. In contrast, by comparing the two NM=96N_M = 96 runs, we see that approximate spherically matched AMR partially mitigates these spurious reflections. The magnitude of the effect can also be seen to converge away when comparing NM=192N_M = 192.

The change in refinement strategy also has an effect on GW phasing, which we show in Figure 8. At early times the amplitude and phasing error of the spherically matched AMR with NM=96N_M = 96 is comparable to that of a pBIB run at NM=160N_M = 160. This can be attributed to a combination of increased wave-zone resolution and better symmetry matching of the AMR (See also the L2L_2 versus L∞L_\infty AMR discussion of [53]).

Figure 8. Comparison of differences in the amplitude AA of (2,2)(2,2) mode of the gravitational strain (top-panel) and phase ϕ\phi (bottom-panel), for runs at different resolutions and refinement strategy, under ZCX\mathcal{Z}_{\rm CX}. We consider pBIB AMR with (c,m,f)(c,m,f) corresponding to NM∈{96,160,192}N_M \in\{96,160,192\}. A single run with spherically matched AMR at NM=96N_M=96 is denoted with c(†)c(\dagger). Vertical black line across panels indicates umergeru_{\rm merger}. Extraction performed at R=100MR=100M.

Another potential advantage of running the spherically matched AMR is a reduction in execution time. Running the spherically matched AMR at NM=96N_M = 96 in bitant results in #MB=4532\#\mathrm{MB} = 4532 initially, where δxp\delta x_p is as in Tab.I. Compared to pBIB AMR under bitant symmetry reduction at NM=160N_M = 160, it results in a computational resource reduction of ∼3.4×\sim3.4\times when coarser time-step δt\delta t at fixed CFL is taken into account.

Finally, we provide a rough characterization of performance differences between ZCX\mathcal{Z}_{\mathrm{CX}} and ZVC\mathcal{Z}_{\mathrm{VC}} through a short timing test. To this aim we evolve the problem for the calibration BBH on the pBIB grid configurations of Tab.I. The initial checkpoint is taken at a time of t0=5 Mt_0 = 5\,M, where the AMR condition has refined the Mesh. Timings are as summarized in Tab.II. We see that for pure vacuum, the space-time solver has a distinct advantage when run with the choice ZVC\mathcal{Z}_{\mathrm{VC}} in terms of speed.

NMN_MtF(CX)t_F(\mathrm{CX})tF(VC)t_F(\mathrm{VC})rtr_t (2 dp)#MB\#\mathrm{MB}#MB\#\mathrm{MB}/Core
9626.4835.170.712404∼3\sim3
12818.0121.980.773280∼4\sim4
16010.4110.800.938774∼11\sim11
1928.499.030.8711490∼15\sim15

caption: One hour timing test on SuperMUC-NG of LRZ utilizing 16 nodes. The timing ratio rr is defined as rt:=(tF(CX)−t0)/(tF(VC)−t0)r_t := (t_F(\mathrm{CX}) - t_0)/(t_F(\mathrm{VC}) - t_0). The number of MeshBlock objects is provided at t0t_0. Bitant symmetry reduction has been performed. The AMR is based on the pBIB criterion.

:::

Indeed, we observe an approximate average ∼20%\sim20\% later final evolution time, across NMN_M, when compared to the ZCX\mathcal{Z}_{\mathrm{CX}} choice. This we attribute principally to a difference in complexity of the R/P\mathcal{R}/\mathcal{P} operations. A cautionary remark is however in order: the MeshBlock/CPU saturation has not been taken into account, which requires a certain threshold for efficiency [39, 40]. With regard to performance, these timing results nonetheless strongly suggest that ZVC\mathcal{Z}_{\mathrm{VC}} is the better option for BBH.

Binary Neutron Stars

In the case of binary neutron star (BNS) evolution, in addition to the space-time evolution, we need to address the evolution of the hydrodynamical variables. These variables satisfy a balance law of the form ∂t[QA]+∂iFAi[Q]=SA[Q]\partial_t[Q_A]+\partial_i\mathcal{F}^{i}_A[Q]=\mathcal{S}_A[Q] [38]. The source term SA[Q]\mathcal{S}_A[Q] does

not involve derivatives of QAQ_A, but does contain geometric fields and their derivatives. Here the finite-volume, HRSC methods we employ lead us to take the evolved QAQ_A on cell-centers. In the ∂iFAi[Q]\partial_i\mathcal{F}^{i}_A[Q] approximation step, primitive variables RA[Q]\mathcal{R}_A[Q] are assembled and reconstructed to FC [58]. This process requires geometric data on FC. Information about the evolved fluid variable state, must be available on the grid used for the space-time solver. If we select ZVC\mathcal{Z}_{\rm VC} the required intergrid transfer operations can be summarized as: ZVC→CC\mathcal{Z}_{\rm VC}\to\mathrm{CC}, ZVC→FC\mathcal{Z}_{\rm VC}\to\mathrm{FC}, and finally RA,CC→VC\mathcal{R}_{A,\mathrm{CC}}\to\mathrm{VC}. On the other hand, selecting ZCX\mathcal{Z}_{\rm CX} means we only need ZCX→FC\mathcal{Z}_{\rm CX}\to\mathrm{FC}.

The intergrid transfer operations summarized above are handled through the use of symmetric Lagrange interpolation. Specifically, during evaluation, NIN_I nearest neighbor nodes either side of a target interpolation point are used along a salient axis. We have found it beneficial, for stability reasons, to fix NI=1N_I=1 in a subset of operations. This we refer to as a hybrid mode. In this mode NIN_I is only allowed to vary freely for Z\mathcal{Z} derivative assembly entering SA[Q]\mathcal{S}_A[Q], and primitives RA,CC→VC\mathcal{R}_{A,\mathrm{CC}}\to\mathrm{VC} that recouple to the space-time solver.

As initial consistency checks for geometry-matter field coupling and the above operations we consider static star tests in Appendix A. A more representative setting of binary neutron star (BNS) evolution problems is required. For this, we consider a quasi-circular, equal-mass BNS inspiral-merger problem. Here irrotational, constraint-satisfying initial data is provided by Lorene [70]. Specifically, as a validation problem, we consider the G2_I14vs14_D4R33_45km binary7 which features baryon mass Mb=3.2500M⊙M_b=3.2500M_{\odot} and gravitational mass M=3.0297M⊙M=3.0297M_{\odot} at an initial separation of 4545 km. The Arnowitt-Deser-Misner (ADM) mass of the binary is MADM=2.9984M⊙M_{\rm ADM}=2.9984M_{\odot}, the angular momentum JADM=8.8354M⊙2 G/cJ_{\rm ADM}=8.8354M_{\odot}^{2}\,\mathrm{G}/c, and the initial orbital frequency is f0=294f_0=294 Hz. For these tests we utilize an ideal gas equation-of-state (see Appendix A) with K=123.64K=123.64. The evolution of this BNS system proceeds for three orbits prior to the formation of a massive remnant, thereafter leading to gravitational collapse.

The computational domain is again selected as Ω:=[−D,D]3\Omega:=[-D,D]^3, where D=1536M⊙≃2268D=1536M_{\odot}\simeq2268 km. The AMR is induced through tracking the local minimum of the lapse α\alpha, as positioned on each constituent of the BNS system. Resolution levels are fixed according to {S8,15(α∗±),S7,30(α∗±),S6,110(0)}\{S_{8,15}(\alpha_*^\pm),S_{7,30}(\alpha_*^\pm),S_{6,110}(0)\}. To perform a resolution study, we take a sequence of NM∈{64,96,128}N_M\in\{64,96,128\}. This translates into a sequence of finest resolution levels of (554,369,277)(554,369,277) m. The MeshBlock sampling is selected as NB=16N_B=16. For the time-evolution we make use of the SSPRK(3,3) method of [71], with a CFL of C1=0.2C_1=0.2, and σ0=0.5\sigma_0=0.5. A tenuous atmosphere is added, where ρatm=10−16\rho_{\rm atm}=10^{-16}. We do not make use of thresholding [40]. For the RePrimAnd-based primitive recovery we set the tolerance to 10−1010^{-10}. In this section no symmetry reductions are performed.

We select ZCX\mathcal{Z}_{\mathrm{CX}} and evolve the initial data with the aforementioned parameters. An additional run featuring ZVC\mathcal{Z}_{\mathrm{VC}} with NM=64N_M = 64 and all else fixed is also performed for internal comparison. In each case we find approximately three orbits prior to the formation of a massive remnant. A snapshot of the fluid rest mass-density at this point (t≃8.8 mst \simeq8.8\,\mathrm{ms}) is shown in Figure 9 comparing the four runs. We observe that the spiral arm structure is

Figure 9. Mesh structure and values of fluid rest mass-density ρ\rho [g cm−3^{-3}] shortly post-merger in the orbital plane for the BNS inspiral problem described in the text. Counter-clockwise, starting from the upper right quadrant we have evolution with CX sampling with NM∈{64,96,128}N_M \in\{64, 96, 128\}. The lower right quadrant depicts evolution under VC sampling for geometric fields with NM=64N_M = 64. Each grey box (MeshBlock) is sampled with NB=16N_B = 16. The AMR criterion for each run has driven the distribution of refined region resolution towards an origin-centered approximate sphere. Trackers that follow local minima of α(t)\alpha(t) are overlaid. Darker tracker color indicates increasing tt.

qualitatively similar for the two runs at NM=64N_M = 64, which provides an initial consistency check. The structure becomes sharper and more compact at higher NMN_M. The extrema tracking based AMR can also be seen to guide increased resolution towards regions that suitably follow the binary constituents.

Quantitative scalar monitors are shown in Figure 10. We find that the time of collapse falls in the range t∈[16.5,20.2] mst \in[16.5, 20.2]\,\mathrm{ms} for ZCX\mathcal{Z}_{\mathrm{CX}} selection. This depends on the selection of NMN_M. Importantly, we observe consistency to within 1% when comparing a verification run with NM=64N_M = 64 based on ZVC\mathcal{Z}_{\mathrm{VC}}. Violation in baryon mass conservation grows to a plateau of ∼O(10−6)\sim O(10^{-6}) at merger for each run. This remains approximately constant until the

Figure 10. Relative change of (global) maximum density ρc\rho_c (upper panel), baryon mass conservation MbM_b (middle), and minimum of the lapse α\alpha (lower panel), for the BNS model problem. A resolution triplet induced by the choices NM∈{64,96,128}N_M \in\{64, 96, 128\} and selection of ZCX\mathcal{Z}_{\mathrm{CX}} is depicted, with a single run at NM=64N_M = 64 and ZVC\mathcal{Z}_{\mathrm{VC}} shown for reference. Collapse times, for each run, are signalled by: post-merger, global peak in the ρc\rho_c monitor, blow up of the mass conservation monitor, and confirmed by reduction of lapse below ∼0.2\sim0.2. Vertical black line depicts merger time (t≃8.4 mst \simeq8.4\,\mathrm{ms}), as based on the peak amplitude h22h_{22} associated with the NM=128N_M = 128 run (C.f. Figure 11). See text for further discussion. Note: legend common to all panels.

point of gravitational collapse. Again we see that both choices of grid sampling for the geometric fields lead to consistent results.

Gravitational waveforms are the next item to verify. To construct the (2,2)(2, 2) mode of the strain, for the BNS system, we use fixed frequency integration (FFI) [72]. The FFI proceeds by transforming the signal to the frequency representation, applying a low-frequency cutoff f0f_0, and integrating. We select f0=0.002M−1f_0 = 0.002M^{-1}. The result is shown in Figure 11. Based on the NM=128N_M = 128 run, we find that the time of merger (retarded coordinate) is umerger≃1583.2 Mu_{\mathrm{merger}} \simeq1583.2\,M. This translates into tmerger≃1699.8 M≃7.9 mst_{\mathrm{merger}} \simeq1699.8\,M \simeq7.9\,\mathrm{ms}. Indeed, all CX runs fall within a band of ∼0.2 ms\sim0.2\,\mathrm{ms}. This indicates strong robustness of the simulations performed. The merger time for the ZVC\mathcal{Z}_{\mathrm{VC}} waveform is within 3% of that of the CX at NM=96N_M = 96. Similarly, the morphology of the instantaneous frequency for CX when cross-compared against VC is seen to be alike. With the method described in Appendix A we also assess convergence. As can be seen in Figure 11 we have a strong indication for a second order trend in the convergence of the h22h_{22} associated phase, as NMN_M is increased.

Gravitational strain, instantaneous frequency, and phase differences for the binary neutron star run

Figure 11. Gravitational strain (2, 2) mode extracted at R=100MR = 100M for binary neutron star run of §III B. (Top): real part of the strain with ZCX\mathcal{Z}_{\mathrm{CX}} for NM∈{64,96,128}N_M \in\{64, 96, 128\} is depicted. For NM=64N_M = 64 the amplitude is also shown. In the case of NM=128N_M = 128 we find umerger≃1583.2Mu_{\mathrm{merger}} \simeq1583.2 M (tmerger≃1699.8Mt_{\mathrm{merger}} \simeq1699.8 M). We observe consistent behaviour at NM=64N_M = 64 for a comparable run based on VC sampling. (Middle): Instantaneous frequency is shown. (Top, middle): vertical dashed lines indicate time of merger for the different data sets. (Bottom): Phase differences for the CX resolution triplet suggest convergence (to merger) as compatible with a second order trend.

Overall, based on this discussion, when running with ZCX\mathcal{Z}_{\mathrm{CX}}, good agreement is found with ZVC\mathcal{Z}_{\mathrm{VC}} here, and our prior results in [40]. This provides confidence in the correctness of our implementation, and the validity of the strategy outlined in §II B.

To close this section, as in the BBH case, we provide a rough characterization of performance differences between ZCX\mathcal{Z}_{\mathrm{CX}} and ZVC\mathcal{Z}_{\mathrm{VC}} through a short timing test. The initial checkpoint is taken at a time of t0=5Mt_0 = 5 M with timings summarized in Tab. III.

NMN_MtF(cx)t_F(\mathrm{cx})tF(vc)t_F(\mathrm{vc})rtr_t (2 dp)#MB#MB/Core
64170.45151.021.133676∼5\sim5
9670.8461.111.176768∼9\sim9
12834.7828.361.2712384∼16\sim16

Table III. One hour timing test on SuperMUC-NG of LRZ utilizing 16 nodes. The timing ratio rtr_t is defined as rt:=(tF(cx)−t0)/(tF(vc)−t0)r_t := (t_F(\mathrm{cx}) - t_0)/(t_F(\mathrm{vc}) - t_0). The number of MeshBlock objects is provided at t0t_0. No symmetry reduction has been performed. The AMR is based on spherical regions criteria.

We see that for BNS, the space-time solver has a distinct advantage when run with the choice ZCX\mathcal{Z}_{\mathrm{CX}} in terms of speed. Indeed, we observe an approximate average ∼17%\sim17\% later evolution time, across NMN_M, when compared to the ZVC\mathcal{Z}_{\mathrm{VC}} choice. This is in direct contrast to the BBH case (Tab. II). While it is the case that the level-to-level operators become more involved for CX, when contrasted against VC, the reduction in the number of intergrid transfer operations appears to compensate. Indeed, it appears that the gap between CX and VC (rtr_t of Tab. III) increases with increasing NMN_M. This is also in contrast to the BBH case, where the gap appeared to decrease. Overall, we see that CX achieves later final evolved times in the BNS problem. Hence we conclude that ZCX\mathcal{Z}_{\mathrm{CX}} is the favoured choice, for this class of problem.

in contrast to the BBH case, where the gap appeared to decrease. Overall, we see that CX achieves later final evolved times in the BNS problem. Hence we conclude that ZCX\mathcal{Z}_{\mathrm{CX}} is the favoured choice, for this class of problem.

Conclusion

In this work we have proposed a new, cell-centered (CX) variable treatment within the context of our space-time solver embedded within GR-Athena++. This has allowed us to solve the Z4c system in ZCX\mathcal{Z}_{\mathrm{CX}} mode, in a fashion consistent with our earlier work on vertex-centers, i.e. ZVC\mathcal{Z}_{\mathrm{VC}} mode. Tests for both vacuum and GRHD problems demonstrated robust functionality for either grid sampling. The choice of ZCX\mathcal{Z}_{\mathrm{CX}} for GR(M)HD yields a substantial simplification in treatment of discretized variables.

To validate the new treatment, together with implementation details within the code, first consistency tests based on BBH evolution were performed. A step-by-step strategy was taken which featured, inter alia, a resolution study. This allowed us to assess a trend compatible with 6th6^\mathrm{th} order convergence in the error of the phase of the (2, 2) mode of the gravitational wave strain, when working with evolution of ZCX\mathcal{Z}_{\mathrm{CX}}. This is consistent with our prior work [39]. In order to provide insight as to which grid sampling strategy performs better, for BBH evolution, we considered a one hour timing test on SuperMUC-NG of the Leibniz-Rechenzentrum Munich. It was found that ZVC\mathcal{Z}_{\mathrm{VC}}, on average, is ∼20%\sim20\% faster. This we attribute to the differences in how the level-to-level refinement operators are evaluated for the two grid samplings. Motivated by the “symmetry-seeking” property of the L2L_2-AMR previously investigated in [53], we proposed, a simplified, spherical-region based AMR strategy. This was demonstrated in the BBH validation problem to potentially partially mitigate some rectilinear grid imprinting.

Following this, a quasi-circular, equal-mass, BNS inspiral, merger, and gravitational collapse problem was illustrated in the context of GRHD. We again utilized the (2, 2) mode of the gravitational wave strain as a consistency diagnostic. A trend compatible with 2nd2^\mathrm{nd} order convergence was assessed in a resolution study when working with ZCX\mathcal{Z}_{\mathrm{CX}}. This trend is consistent with that observed in [40] when working with ZVC\mathcal{Z}_{\mathrm{VC}}. With regard to performance, in contrast to the BBH case, it was found that ZCX\mathcal{Z}_{\mathrm{CX}} is, on average, ∼17%\sim17\% faster for the runs considered. This we attribute to a reduction in the total number of intergrid interpolation operations required. For these tests, we also leveraged the simplified, spherical-region based AMR strategy. While we do not explicitly demonstrate GRMHD runs in this work, we have found another potential advantage of working with ZCX\mathcal{Z}_{\mathrm{CX}}. First tests, involving the initial data of the BNS simulation presented here, allowed for evolution with magnetic fields through merger and collapse, without apparent horizonbased excision. This contrasts against the situation with ZVC where we have found such excision techniques to be crucial.

Acknowledgments

BD and SB acknowledges support by the EU Horizon under ERC Consolidator Grant, no. InspiReM-101043372. PH acknowledges funding from the National Science Foundation under Grant No. PHY-2116686. EG, DR acknowledge funding from the National Science Foundation under grant No. AST-2108467. EG acknowledges funding from an Institute for Gravitation and Cosmology fellowship. JF, DR acknowledge U.S. Department of Energy, Office of Science, Division of Nuclear Physics under Award Number(s) DE-SC0021177. DR acknowledge support from NASA under award No. 80NSSC21K1720 and from the Sloan Foundation. The authors are indebted to A. Celentano’s PRĪSENCŌLĪNENSĪĀINCĪUSŌL.

The numerical simulations were performed on the national HPE Apollo Hawk at the High Performance Computing Center Stuttgart (HLRS). The authors acknowledge HLRS for funding this project by providing access to the supercomputer HPE Apollo Hawk under the grant numbers INTRHYGUE/44215 and MAG-NETIST/44288. Simulations were also performed on SuperMUC NG at the Leibniz-Rechenzentrum (LRZ) Munich. The authors acknowledge the Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for funding this project by providing computing time on the GCS Supercomputer SuperMUC-NG at LRZ (allocations pn68wi and pn36jo). Simulations were also performed on TACC’s Frontera (NSF LRAC allocation PHY23001) and on Perlmutter. This research used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. Postprocessing and development run were performed on the ARA cluster at Friedrich Schiller University Jena. The ARA cluster is funded in part by DFG grants INST 275/334-1 FUGG and INST 275/363-1 FUGG, and ERC Starting Grant, grant agreement no. BinGraSp-714626.

References

Assessing convergence with a static star

Evolving a stable, static star in the general relativistic setting provides for a simple, sanity check on code correctness. To this end initial data (ID) is prepared based on the assumption of a spherically symmetric, self-gravitating matter distribution at equilibrium satisfying the Tolman-Oppenheimer-Volkoff (TOV) equations (see e.g. [73]). For ID preparation a polytropic EOS p=KρΓp = K\rho^{\Gamma} is assumed with adiabatic index Γ=2\Gamma= 2, and polytropic constant K=100K = 100 as in [31]. The stable star model is taken to have a central density ρc=1.280×10−3\rho_{c} = 1.280 \times10^{-3}, gravitational mass M=1.400M = 1.400, and circumferential radius R=9.586R = 9.586. During evolution we make use of the more general EOS p=(Γ−1)ρϵp = (\Gamma- 1)\rho\epsilon with ϵ=KρΓ−1/(Γ−1)\epsilon= K\rho^{\Gamma-1}/(\Gamma- 1). We perform an evolution of this data to a final time of T=1500T = 1500. This is done for both ZVC and ZCX choices, for a variety of intergrid interpolation orders (see §III B). For these runs we have taken a CFL of C1=0.25C_{1} = 0.25, Kreiss-Oliger dissipation σD=0.5\sigma_{D} = 0.5, and set artificial atmosphere to ρatm=10−18\rho_{\mathrm{atm}} = 10^{-18}. The Z4c damping parameters are taken as (κ1,κ2)=(0.02,0)(\kappa_{1}, \kappa_{2}) = (0.02, 0). Scalar diagnostic quantities are shown in Figure 12.

Comparison of scalar quantities in stable star evolution

Figure 12. Comparison of scalar quantities in stable star evolution (TOV test) on a fixed grid configuration involving CX and VC sampling where various varieties of intergrid interpolation of §III B are utilized. The Mesh and MeshBlock samplings are NM=64N_{M} = 64 and NB=16N_{B} = 16 respectively. The computational domain is taken as Ω:=[−1024,1024]3\Omega:= [-1024, 1024]^{3} with a central, cubic, region Ωr:=[−10,10]3\Omega_{r} := [-10, 10]^{3} refined with NL=7N_{L} = 7. The region Ωr\Omega_{r} completely contains the star. As intergrid interpolation order is increased it can be seen (upper panel) that central density is better preserved, with oscillation magnitude of ρc(t)\rho_{c}(t) reduced. Hamiltonian constraint violation (lower panel) is also reduced as interpolation order is increased. We find common mass conservation (middle panel) properties after ∼t=200\sim t = 200 regardless of interpolation order, or geometric sampling. Vertical grey line indicates first time of causal contact of ∂Ω\partial\Omega with the origin.

To assess solution convergence we consider numerical evolution repeated at a triplet of coarse, medium, and fine resolutions (δxc,δxm,δxf)(\delta x_{c}, \delta x_{m}, \delta x_{f}) that satisfy δxc>δxm>δxf\delta x_{c} > \delta x_{m} > \delta x_{f}. The convergence rate of an approximation to the corresponding field data F\mathcal{F} may be investigated by comparing differences of solutions δFab:=Fa−Fb\delta\mathcal{F}_{ab} := \mathcal{F}_{a} - \mathcal{F}_{b} at distinct resolutions. For δxn\delta x^{n} one finds based on Taylor expansion that δFcm≃δFmf×Qn(c,m,m,f)\delta\mathcal{F}_{cm} \simeq\delta\mathcal{F}_{mf} \times Q_{n}(c,m,m,f) where we have introduced the so-called convergence factor:

Qn(a,b,c,d):=δxan−δxbnδxcn−δxdn.(4)Q_{n}(a,b,c,d) := \frac{\delta x_{a}^{n} - \delta x_{b}^{n}}{\delta x_{c}^{n} - \delta x_{d}^{n}}. \tag*{(4)}

The NI=2N_{I} = 2 simulations featuring CX and VC of Figure 12 show the smallest deviation from the initial value ρc\rho_{c} during evolution, together with the smallest constraint violation. We construct a triplet of resolutions for convergence assessment by repeating the runs with NL∈{5,6}N_{L} \in\{5, 6\}. Based on the refinement structure this entails (δxc,δxm,δxf)=(δxc,δxc/2,δxc/4)(\delta x_{c}, \delta x_{m}, \delta x_{f}) = (\delta x_{c}, \delta x_{c}/2, \delta x_{c}/4) on the finest level, covering the star, for each run. Behaviour for the central density, together with the Hamiltonian constraint for our convergence assessment is shown in Figure 13. As can be seen we observe ∼2nd\sim2^{\mathrm{nd}} order, and ∼5th\sim5^{\mathrm{th}} order compatible trends for the aforementioned quantities respectively. Based on the consistency between the qualitative features of the CX and VC scalar diagnostics depicted in Figure 12, together with consistent order of convergence demonstrated in Figure 13 we consider the TOV test to pass.

Figure 13. Convergence of scalar quantities in the TOV test for a fixed grid configuration involving CX and VC sampling. Overall grid structure is as in Figure 12 where resolution triplets are constructed based selecting the level of Ωr\Omega_{r} refinement through NL∈{5,6,7}N_{L} \in\{5, 6, 7\}. In dashed lines we depict the rescaled coarse-medium difference according to QnQ_{n} of Eq. (4). As additionally verified through least-squares fitting on t∈[128,950]t \in[128, 950] we find: n∼2n \sim2 i.e. ∼2\sim2nd order convergence for ρc\rho_{c} (upper panel); n∼5n \sim5 for the Hamiltonian constraint (lower panel).

References

  1. [1]B. P. Abbott et al. (Virgo, LIGO Scientific), Phys. Rev. Lett. 119, 161101 (2017), arXiv:1710.05832 [gr-qc].
  2. [2]A. Goldstein et al., Astrophys. J. 848, L14 (2017), arXiv:1710.05446 [astro-ph.HE].DOI
  3. [3]V. Savchenko et al., Astrophys. J. 848, L15 (2017), arXiv:1710.05449 [astro-ph.HE].DOI
  4. [4]T. Akutsu et al., arXiv:2009.09305 [astro-ph, physics:gr-qc] (2020), arXiv:2009.09305 [astro-ph, physics:gr-qc].arxiv.org/abs/2009.09305
  5. [5]e. a. Amaro-Seoane, arXiv:1702.00786 [astro-ph] (2017), arXiv:1702.00786 [astro-ph].
  6. [6]M. Punturo, M. Abernathy, F. Acernese, B. Allen, N. Andersson, et al., Class.Quant.Grav. 27, 194002 (2010).
  7. [7]B. P. Abbott et al. (LIGO Scientific), Class. Quant. Grav. 34, 044001 (2017), arXiv:1607.08697 [astro-ph.IM].DOI
  8. [8]A. Nagar et al., Phys. Rev. D98, 104052 (2018), arXiv:1806.01772 [gr-qc].DOI
  9. [9]J. A. Font, Living Rev. Rel. 11, 7 (2007).
  10. [10]M. Shibata, K. Kiuchi, Y.-i. Sekiguchi, and Y. Suwa, Prog.Theor.Phys. 125, 1255 (2011), arXiv:1104.3937 [astro-ph.HE].DOI
  11. [11]D. Radice, S. Bernuzzi, A. Perego, and R. Haas, Mon. Not. Roy. Astron. Soc. 512, 1499 (2022), arXiv:2111.14858 [astro-ph.HE].DOI
  12. [12]D. Hilditch, A. Weyhausen, and B. Brügmann, Phys. Rev. D93, 063006 (2016), arXiv:1504.04732 [gr-qc].DOI
  13. [13]M. Bugner, T. Dietrich, S. Bernuzzi, A. Weyhausen, and B. Brügmann, Phys. Rev. D94, 084004 (2016), arXiv:1508.07147 [gr-qc].DOI
  14. [14]M. Fernando, D. Neilsen, H. Lim, E. Hirschmann, and H. Sundar, (2018), 10.1137/18M1196972, arXiv:1807.06128 [gr-qc].DOI
  15. [15]F. Löfmler et al., Class. Quant. Grav. 29, 115001 (2012), arXiv:1111.3344 [gr-qc].
  16. [17]K. Clough, P. Figueras, H. Finkel, M. Kunesch, E. A. Lim, and S. Tunyasuvunakool, (2015), arXiv:1503.03436 [gr-qc].DOI
  17. [18]W. Tichy, L. Ji, A. Adhikari, A. Rashti, and M. Pirog, Classical and Quantum Gravity 40, 025004 (2023), arxiv:2212.06340 [gr-qc].DOI
  18. [19]L. E. Kidder et al., J. Comput. Phys. 335, 84 (2017), arXiv:1609.00098 [astro-ph.HE].DOI
  19. [20]S. Rosswog and P. Diener, Class. Quant. Grav. 38, 115002 (2021), arXiv:2012.13954 [gr-qc].DOI
  20. [21]P. Grandclement and J. Novak, Living Rev. Rel. 12, 1 (2009), arXiv:0706.2286 [gr-qc].DOI
  21. [22]G. Doulis, F. Atteneder, S. Bernuzzi, and B. Brügmann, Phys. Rev. D 106, 024001 (2022), arXiv:2202.08839 [gr-qc].DOI
  22. [23]R. Alfieri, S. Bernuzzi, A. Perego, and D. Radice, Journal of Low Power Electronics and Applications Special Issue ”Energy Aware Scientific Computing on Low Power and Heterogeneous Architectures”, 8 (2018), 10.3390/jlpea8020015.
  23. [24]Q. F. Stout, D. L. De Zeeuw, T. I. Gombosi, C. P. T. Groth, H. G. Marshall, and K. G. Powell, in Proceedings of the 1997 ACM/IEEE Conference on Supercomputing ’97 (Association for Computing Machinery, New York, NY, USA, 1997) pp. 1–10.
  24. [25]M. J. Berger and J. Oliger, J.Comput.Phys. 53, 484 (1984).
  25. [26]C. Burstedde, J. Holke, and T. Isaac, Foundations of Computational Mathematics 19, 843 (2019).
  26. [27]G. M. Morton, A computer oriented geodetic data base and a new Tech. Rep. (1966).
  27. [28]T. Nakamura, K. Oohara, and Y. Kojima, Prog. Theor. M. Shibata and T. Nakamura, Phys. Rev. D52, 5428 (1995).
  28. [30]T. W. Baumgarte and S. L. Shapiro, Phys. Rev. D59, 024007 (1999), arXiv:gr-qc/9810065.DOI
  29. [31]S. Bernuzzi and D. Hilditch, Phys. Rev. D81, 084003 (2010), arXiv:0912.2920 [gr-qc].DOI
  30. [32]D. Hilditch, S. Bernuzzi, M. Thierfelder, Z. Cao, W. Tichy, and B. Bruegmann, Phys. Rev. D88, 084057 (2013), arXiv:1212.2901 [gr-qc].DOI
  31. [33]H. Friedrich, Communications in Mathematical Physics 100, 525 (1985).
  32. [34]F. Pretorius, Phys. Rev. Lett. 95, 121101 (2005), arXiv:gr-qc/0507014.DOI
  33. [35]L. Lindblom, M. A. Scheel, L. E. Kidder, R. Owen, and O. Rinne, Class.Quant.Grav. 23, S447 (2006), arXiv:gr-qc/0512093 [gr-qc].DOI
  34. [36]C. O. Lousto and J. Healy, Phys. Rev. Lett. 125, 191102 (2020), arXiv:2006.04818 [gr-qc].DOI
  35. [37]J. Lannelongue, H.-E. Aronson, A. Bateman, E. Birney, T. Caplan, M. Juckes, J. McEntyre, A. D. Morris, G. Reilly, and M. Inouye, Nature Computational Science 3, 514 (2023).
  36. [38]F. Banyuls, J. A. Font, J. M. A. Ibanez, J. M. A. Marti, and J. A. Miralles, Astrophys. J. 476, 221 (1997).
  37. [39]B. Daszuta, F. Zappa, W. Cook, D. Radice, S. Bernuzzi, and V. Morozova, Astrophys. J. Supp. 257, 25 (2021), arXiv:2101.08289 [gr-qc].DOI
  38. [40]W. Cook, B. Daszuta, J. Fields, P. Hammond, S. Albanesi, F. Zappa, S. Bernuzzi, and D. Radice, (2023), arXiv:2311.04989 [gr-qc].arxiv.org/abs/2311.04989
  39. [41]B. Daszuta and W. Cook, “GR-Athena++: Magnetohydrodynamical evolution with dynamical space-time,” (2024), arXiv:2406.05126 [astro-ph, physics:gr-qc].arxiv.org/abs/2406.05126
  40. [42]C. J. White, J. M. Stone, and C. F. Gammie, The Astrophysical Journal Supplement Series 225, 22 (2016), arXiv:1511.00943.DOI
  41. [43]K. G. Felker and J. M. Stone, Journal of Computational Physics 375, 1365 (2018).
  42. [44]J. M. Stone, K. Tomida, C. J. White, and K. G. Felker, The Astrophysical Journal Supplement Series 249, 4 (2020).
  43. [45]B. Brügmann, J. A. Gonzalez, M. Hannam, S. Husa, U. Sperhake, et al., Phys.Rev. D77, 024027 (2008), arXiv:gr-qc/0610128 [gr-qc].DOI
  44. [46]P. Mösta, B. C. Mundim, J. A. Faber, R. Haas, S. C. Noble, et al., Class.Quant.Grav. 31, 015005 (2014), arXiv:1304.5544 [gr-qc].
  45. [47]D. Radice, L. Rezzolla, and F. Galeazzi, Class.Quant.Grav. 31, 075012 (2014), arXiv:1312.5004 [gr-qc].DOI
  46. [48]M. Thierfelder, S. Bernuzzi, and B. Brügmann, Phys.Rev. D84, 044012 (2011), arXiv:1104.4751 [gr-qc].DOI
  47. [49]N. Fleischmann, S. Adami, and N. A. Adams, Computers & Fluids 89, 94 (2019).
  48. [50]A. Neumaier, ZAMM - Journal of Applied Mathematics and Mechanics / Zeitschrift für Angewandte Mathematik und Mechanik 54, 39 (1974).
  49. [51]J. M. Stone, K. Tomida, C. J. White, and K. G. Felker, Astrophys. J. Suppl. 249, 4 (2020), arXiv:2005.06651 [astro-ph.IM].DOI
  50. [52]J.-P. Berrut and L. N. Trefethen, SIAM Review 46, 501 (2004).
  51. [53]A. Rashti, M. Bhattacharyya, D. Radice, B. Daszuta, W. Cook, and S. Bernuzzi, Class. Quant. Grav. 41, 095001 (2024), arXiv:2312.05438 [gr-qc].DOI
  52. [54]M. Campanelli, C. O. Lousto, P. Marronetti, and Y. Zlochower, Phys. Rev. Lett. 96, 111101 (2006), arXiv:gr-qc/0511048.DOI
  53. [55]S. C. Noble, C. F. Gammie, J. C. McKinney, and L. Del Zanna, Astrophys. J. 641, 626 (2006), arXiv:astro-ph/0512420.DOI
  54. [56]W. Kastaun, J. V. Kalanin, and R. Ciolfi, Phys. Rev. D 103, 023018 (2021), arXiv:2005.01821 [gr-qc].DOI
  55. [57]R. Borges, M. Carmona, B. Costa, and W. S. Don, Journal of Computational Physics 227, 3191 (2008).
  56. [58]L. D. Zanna and N. Bucciantini, Astronomy & Astrophysics 390, 1177 (2002).
  57. [59]C. Bona, J. Massó, J. Stela, and E. Seidel, in The Seventh Marcel Grossmann Meeting: On Recent Developments, edited by R. T. Jantzen, G. M. Keiser, and R. Ruffini (World Scientific, Singapore, 1996).
  58. [60]M. Alcubierre, B. Brügmann, P. Diener, M. Koppitz, D. Pollney, et al., Phys.Rev. D67, 084023 (2003), arXiv:gr-qc/0206072 [gr-qc].DOI
  59. [61]J. R. van Meter, J. G. Baker, M. Koppitz, and D.-I. Choi, Phys. Rev. D73, 124011 (2006), arXiv:gr-qc/0605030.DOI
  60. [62]M. Ansorg, B. Brügmann, and W. Tichy, Phys. Rev. D70, 064011 (2004), arXiv:gr-qc/0404056.DOI
  61. [63]D. I. Ketcheson, Journal of Computational Physics 229, 1763 (2010).
  62. [64]N. Wang and J.-L. Lee, SIAM Journal of Scientific Computing 33, 2536 (2011).
  63. [65]J. N. Goldberg, A. J. MacFarlane, E. T. Newman, F. Rohrlich, and E. C. G. Sudarshan, J. Math. Phys. 8, 2155 (1967).
  64. [66]LIGO Scientific Collaboration, “LIGO Algorithm Library - LALSuite,” free software (GPL) (2018).DOI
  65. [67]S. Albanesi, A. Rashti, F. Zappa, R. Gamba, W. Cook, B. Daszuta, S. Bernuzzi, A. Nagar, and D. Radice, “Scattering and dynamical capture of two black holes: Synergies between numerical and analytical methods,” (2024), arXiv:2405.20398 [gr-qc].DOI
  66. [68]M. Thierfelder, S. Bernuzzi, D. Hilditch, B. Brügmann, and L. Rezzolla, Phys.Rev. D83, 064022 (2011), arXiv:1012.3703 [gr-qc].DOI
  67. [69]V. Mewes, Y. Zlochower, M. Campanelli, T. W. Baumgarte, Z. B. Etienne, F. G. Lopez Armengol, and F. Cipolletta, Phys. Rev. D 101, 104007 (2020), arXiv:2002.06225 [gr-qc].DOI
  68. [70]E. Gourgoulhon, P. Grandclement, K. Taniguchi, J.-A. Marcq, and S. Bonazzola, Phys.Rev. D63, 064029 (2001), arXiv:gr-qc/0007028 [gr-qc].DOI
  69. [71]S. Gottlieb, D. I. Ketcheson, and C.-W. Shu, Journal of Scientific Computing 38, 251 (2009).
  70. [72]C. Reisswig and D. Pollney, Class.Quant.Grav. 28, 195015 (2011), arXiv:1006.1632 [gr-qc].DOI
  71. [73]J. A. Font, M. A. Miller, W.-M. Suen, and M. Tobias, Phys. Rev. D61, 044011 (2000), gr-qc/9811015.

Paper details

vlnc:W_rsgzhbnt
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents