betaFeedback

WorksPhysics and Astronomy

Relaxation-driven topological domains in moiré materials

Abstract

Spatial control of topology is highly desirable for realizing tunable quantum functionalities in materials. Moiré superlattices formed by twisting van der Waals heterostructures provide a natural platform for spatially modulated electronic phases, yet the emergence of tunable topological domains in these systems remains largely unexplored. Here we show that structural relaxation in twisted bilayer BiSb drives the formation of a distinct moiré topological phase, characterized by coexisting topologically nontrivial (Z2_2 = 1) and trivial (Z2_2 = 0) domains within a single moiré unit cell. As the twist angle is reduced, relaxation-induced modulation of the interlayer separation stabilizes an expanding network of topological regions embedded within trivial backgrounds of the moiré unit cell. The resulting internal domain boundaries host topologically-protected gapless 1D edge states that are directly visible in our simulated scanning-tunnelling microscopy maps.Furthermore, we demonstrate that the real-space topological domain structure and associated gapless edge states can be reversibly tuned by an out-of-plane electric field. Together, these results establish twisted BiSb as a promising platform for programmable topological domain patterning, where intrinsic networks of edge channels can be continuously tuned and electrically reconfigured in moiré materials.

Introduction

Introduction. Twisted two-dimensional van der Waals (vdW) heterostructures offer a highly versatile and tunable platform for exploring a wide range of emergent quantum phenomena, including strongly correlated insulating phases [1, 2], the fractional anomalous quantum Hall effect [3, 4], superconductivity [5, 6], and generalized Wigner crystals [7, 8, 9]. When two layers are twisted by an angle θ\theta, the interference of their lattice periodicities produces a moiré superlattice characterized by spatially varying atomic registries [10, 11]. These local variations induce periodic modulations of interlayer hybridization and stacking energies across the supercell, producing an energy landscape that is highly sensitive to local atomic configuration. Such variations strongly renormalize the electronic structure, leading to domain-dependent properties that can be precisely tuned by adjusting the twist angle. Thus, twisted 2D vdW heterostructures provide an exciting and highly controllable platform for realizing materials in which contrasting quantum phases can coexist within the same system maintaining their key features [12, 13, 14].

In this context, a promising direction is the design of materials with spatially varying local topological character. Unlike conventional 2D topological insulators, where edge states are confined only to sample boundaries, moiré systems can host internal topological interfaces arising purely from local variations in stacking. This gives rise to self-organized networks of conducting 1D edge states along domain boundaries; these states are analogous to lithographically-patterned topological circuitry, but are instead intrinsically defined by the moiré geometry, ensuring high symmetry and uniformity. Previously, these topological mosaic networks were found to be especially important in the description of Landau levels [15]. Without a magnetic field, such in-plane topological networks provide a natural platform for planar device architectures, where conducting 1D channels can be electrically tuned and reconfigured within the same material.

Although topological mosaic phases have been proposed theoretically in twisted materials, existing studies [16, 17, 18, 19] have largely neglected the role of structural relaxation, despite its central importance in determining the local electronic structure of moiré materials [20, 21, 22, 23, 24]. Structural relaxation fundamentally reconstructs moiré superlattices by expanding energetically favorable stacking regions and compressing unfavorable registries into narrow domain walls [22, 25]. This process induces pronounced spatial variations in interlayer separation and electronic hybridization, leading to an emergent real-space topological patterning in moiré materials.

Here we show that structural relaxation can itself drive the emergence of real-space topological domain patterning in twisted bilayer BiSb heterostructures. Although an isolated BiSb monolayer is topologically trivial [26, 27, 28], twisting inverted BiSb layers generates a moiré superlattice in which relaxation induces a substantial modulation of the interlayer spacing and hybridization (see Figure 1). This gives rise to a rich network of topologically trivial (Z2=0Z_2 = 0) and nontrivial (Z2=1Z_2 = 1) domains within a single moiré supercell. These topological domains persist even at relatively large twist angles (e.g., 21.78∘21.78^\circ). The resulting domain boundaries host robust, gapless edge states that are directly visible in simulated scanning-tunnelling microscopy (STM) maps through enhanced local density of states (LDOS). We further demonstrate that the area of the topological domains can be continuously tuned by twist angle, while an out-of-plane electric field enables reversible control of the edge-state network. Overall, our results establish that topology in moiré systems is not globally uniform, instead it emerges as a real-space texture governed by the interplay of twist, lattice relaxation, and spin-orbit coupling.

Schematic of the moiré supercell, structural relaxation, variation of interlayer separation, and topological domains

Figure 1. (a) Schematic of the moiré supercell formed by twisting two BiSb layers by an angle θ\theta. (b) BiSb–SbBi bilayer at a 13.17∘13.17^\circ twist angle, illustrating how structural relaxation leads to spatially varying interlayer separation dd. (c) 2D phase map of dd across the moiré supercell, where red and blue indicate regions of lower and higher interlayer separation, respectively. The minimum and maximum values of dd correspond to high-symmetry stacking configurations AB (Bi atoms in the two layers aligned) and AA (Sb atoms in the two layers aligned), respectively. In the AA region, enhanced Sb-pz orbital repulsion leads to an increased interlayer spacing. (d) Schematic illustrating the role of twist angle and an out-of-plane electric field in controlling topological domain size: decreasing the twist angle enlarges the Z2=1Z_2 = 1 domains, while an applied electric field can drive a transition to a fully Z2=0Z_2 = 0 state.

Results

Topological properties of twisted BiSb-SbBi bilayers

The BiSb monolayer crystallizes in the p3m1p3m1 symmetry (layer group no. 69). Due to the lack of inversion symmetry, it exhibits strong Rashba spin–orbit coupling (RSOC) (αR=1.9\alpha_R = 1.9 eVÅ) [28]. When two BiSb monolayers are stacked in an inverted configuration, the resulting BiSb–SbBi bilayer adopts pˉ3m1\bar{p}3m1 symmetry (layer group no. 72). In this bilayer, inversion symmetry is restored, giving rise to a hidden Rashba effect [29, 30], where RSOC exists locally within each layer but cancels out globally due to inversion symmetry. While the monolayer is topologically trivial (Z2=0Z_2 = 0), the untwisted bilayer is topologically nontrivial (Z2=1Z_2 = 1) [28].

Twisted bilayer configurations were generated by rotating one BiSb monolayer relative to the other at specific twist angles. In this work, we explore three commensurate angles: 21.78∘21.78^\circ, 13.17∘13.17^\circ, and 9.43∘9.43^\circ, chosen based on the commensurate moiré condition, cos⁡θ=m2+4mn+n22(m2+mn+n2)\cos\theta= \frac{m^2+4mn+n^2}{2(m^2+mn+n^2)}, where mm and nn are positive integers. The results for the 13.17∘13.17^\circ and 9.43∘9.43^\circ twist angles are presented in the main text, while the 21.78∘21.78^\circ case is included in the Supplementary Information. For all three twist angles, the relaxed moiré supercells crystallize in the p312p312 symmetry (layer group no. 67), containing 28, 76, and 128 atoms for the 21.78∘21.78^\circ, 13.17∘13.17^\circ, and 9.43∘9.43^\circ twist angles, respectively. While inversion symmetry is broken in the moiré supercell, the threefold rotational symmetry is preserved.

Role of structural relaxation in determining the local topological character of twisted BiSb-SbBi bilayers

During structural relaxation, the twisted bilayers undergo significant lattice reconstruction, resulting in spatially modulated atomic registries across the moiré supercell. Along the long diagonal of the supercell, two high-symmetry stacking configurations are observed: AB, where Bi atoms from the two layers align, and AA, where Sb atoms from both layers lie directly on top of each other, as shown in Figure 2(a) and 2(d). The regions separating these two domains exhibit intermediate stacking configurations that smoothly interpolate between the two high-symmetry configurations. As shown in Figure 2(b) and Figure 2(e), the optimized structures exhibit a continuous modulation of the interlayer separation dd, which directly determines the local topological character, as reflected in the Z2\mathbb{Z}_2 phase diagrams [Figure 2(c) and Figure 2(f)]. Notably, at larger twist angles such as 60°, which correspond to Bernal stacking, the optimized bilayer remains topologically trivial and does not exhibit any spatially varying topological character (see Supplementary Fig. S2). This is a concrete example of a topological phase transition that is purely driven by twisting.

Figure 2. Optimized crystal structures of twisted BiSb–SbBi bilayers at (a) 13.17° and (d) 9.43° twist angles, with AB and AA stacking regions highlighted in blue and yellow, respectively. 2D phase maps of interlayer separation dd for 2×2×12 \times2 \times1 supercells at (b) 13.17° and (e) 9.43° twists. 2D maps showing the spatial variation of the Z2\mathbb{Z}_2 invariant for the bilayers at (c) 13.17° and (f) 9.43° twist angles.

To elucidate the spatial variation of dd across the moiré supercell and its influence on the topological properties, we analyze stacking configurations of the untwisted bilayer, including the AA and AB arrangements based on sampling the disregistry index of the layers [21, 31, 32] by constructing a 15×1515 \times15 grid generated through systematic relative shifts of the top layer with respect to the bottom layer along random in-plane directions, i.e., (x,y)→(x+aux,y+auy)(x, y) \rightarrow(x+a u_x, y+a u_y), where aa represents the lattice constant. Among all configurations, AB stacking is the most energetically favorable, whereas AA stacking is the least favorable, lying approximately 0.5 eV higher in energy. The relative energies of all intermediate configurations are displayed as a 2D colormap in Supplementary Figure 1(a).

The interlayer separation between the two BiSb layers is highly sensitive to the local stacking configuration. In the AA stacking, the vertical alignment of Sb atoms leads to significant overlap of the out-of-plane Sb–pzp_z orbitals [as shown in Figure 1(d)], resulting in strong interlayer repulsion and, consequently, a large interlayer separation of approximately 3.7 Å. In contrast, the AB stacking, where Sb atoms are laterally displaced, minimizes orbital overlap, reducing interlayer repulsion and yielding a smaller separation of about 2.1 Å; see Supplementary Fig. 1(b) for details.

Owing to the strong Sb–pzp_z orbital repulsion, the electronic states near the Fermi level are primarily contributed by the Bi–pzp_z orbitals. Since the stacking configuration strongly influences the interlayer separation, it directly governs the degree of hybridization and interlayer electron tunneling between the Bi–pzp_z orbitals. As demonstrated in our previous work [28], the interplay between this interlayer tunneling and the SOC strength determines the topological character of the system. Consequently, the AB stacking, which exhibits the smallest interlayer spacing, shows the strongest electron tunneling and exhibits a topologically nontrivial phase (Z2=1\mathbb{Z}_2 = 1). In contrast, AA stacking, characterized by the largest interlayer separation, exhibits reduced tunneling and remains topologically trivial (Z2=0\mathbb{Z}_2 = 0) in their optimized configuration.

These stacking-dependent variations in dd and Z2\mathbb{Z}_2 are directly reflected in the moiré supercells. In unrelaxed twisted bilayers, the interlayer separation is uniform throughout the supercell [Figure 1(b)]. However, structural relaxation induces spatial variations in the interlayer distance due to lattice reconstruction, resulting in locally distinct stacking configurations. For instance, the corners of the moiré supercell correspond to AB stacking (blue-shaded regions), while the yellow-shaded regions correspond to AA stacking. Since each stacking configuration exhibits a characteristic interlayer separation, so the relaxed supercell shows an in-plane modulation of interlayer separation. Figure 2 illustrates this variation for moiré supercells with twist angles of 13.17° and 9.43°. AB-stacked regions exhibit the smallest interlayer separation, whereas AA-stacked regions have the largest. Overall, the interlayer distance across the supercell ranges approximately from 2.1 Å to 3.2 Å.

Leveraging the previously established dependence of the Z2Z_2 topological character on both stacking pattern and interlayer distance, we constructed a spatial Z2Z_2 phase diagram for twist angles of 13.17° and 9.43°, shown in Figure 2(c) and Figure 2(f). As expected, AA stacking regions with the largest interlayer separation are topologically trivial (Z2=0Z_2 = 0, shown in gray), whereas AB domains with the smallest interlayer separation exhibit a nontrivial phase (Z2=1Z_2 = 1, blue regions).

Notably, reducing the twist angle from 13.17° to 9.43° increases the Z2=1Z_2 = 1 domain fraction from ∼49% to approximately 70% of the total area. This arises because, during structural relaxation, the 9.43° twisted bilayer significantly maximizes the AB domains in the optimized structure, owing to the lower energy of the AB stacking configuration. This suggests that the size of the topological domains in BiSb-SbBi bilayers can be effectively controlled by adjusting the twist angle.

Real space mapping of local edge states in the moiré supercell. To explore an experimentally feasible method for probing the spatially distributed topological domains in the moiré supercell, we simulated STM images for all three twist angles. The results for 13.17° and 9.43° (Figure 3) are shown in the main text, whereas the 21.78° case is shown in the Supplementary Information. Figure 3(a) shows the simulated STM images of the twisted bilayer at 13.17°, probed at different chemical potentials.

Figure 3. Electronic band structures with SOC for twisted BiSb-SbBi bilayers at (a) 13.17° and (b) 9.43°. The right panel of (a) shows simulated STM images for a 2×2×12 \times2 \times1 supercell of the 13.17° twisted BiSb-SbBi bilayer, probing electronic states near the Fermi level (EF±0.1E_{\mathrm{F}} \pm0.1 eV, red highlighted region) and deeper occupied states (EF−0.6E_{\mathrm{F}} - 0.6 eV to EF−0.8E_{\mathrm{F}} - 0.8 eV, green highlighted region) at a tip height of 4.5 Å. The right panel of (b) shows STM images for the 9.43° case probing states near the Fermi level (EF±0.15E_{\mathrm{F}} \pm0.15 eV) and deeper occupied states (EF−0.8E_{\mathrm{F}} - 0.8 eV to EF−1.0E_{\mathrm{F}} - 1.0 eV). Edge states are indicated by white dotted lines. Bright yellow dots mark the position of Bi atoms in the top layer, located closest to the STM tip.

For the 13.17° twist angle, probing near the Fermi level reveals highly local density of states (bright yellow) along the boundaries of the Z2=1Z_2 = 1 domains at their interfaces with Z2=0Z_2 = 0 regions, corresponding to edge states localized at the boundaries of topologically nontrivial domains. In contrast, probing further below EFE_{\mathrm{F}} excludes these edge states, leaving only bulk trivial states, so no high-LDOS features appear along the boundaries of the nontrivial regions. This establishes STM as a direct probe of emergent edge states in moiré-induced topological domains. A similar behavior is observed for the 9.43° twist angle. Enhanced LDOS near EFE_{\mathrm{F}} (EF±0.15E_{\mathrm{F}} \pm0.15 eV) traces the domain-wall edge states, while at lower energies (EF−0.8E_{\mathrm{F}} - 0.8 eV to EF−1.0E_{\mathrm{F}} - 1.0 eV) the edge-state signature vanishes due to the dominance of bulk states. However, at the larger twist angle of 60° corresponding to Bernal stacking, no such edge-state-induced LDOS enhancement is observed, consistent with its topologically trivial character (see Fig. S2 in the Supplementary Information).

Tunability of topological character through external electric field. An external electric field can serve as a tunable knob to control the topological character of the moiré supercell. To investigate this effect, a vertical displacement field was applied individually to each of the shifted configurations. Since the moiré supercell is effectively a collection of different local stacking configurations, analyzing the electric-field response of the shifted configurations provides an intuitive understanding of the tunability of the topological phase in twisted systems. Figure 4 shows the variation of the topological character of different stacking configurations under an applied electric field.

Image not extracted

Figure 4. Variation of the topological character of different stacking configurations under an external out-of-plane electric field. The centre of the hexagon corresponds to AB stacking, and other points represent randomly shifted configurations relative to AB. xx and yy axes denote shifts along the a1\mathbf{a}_1 and a2\mathbf{a}_2 lattice vectors (fractional units). Blue regions indicate Z2=1Z_2 = 1, while gray regions indicate Z2=0Z_2 = 0.

As shown in Figure 4(a), at zero field, the AB domain is topologically nontrivial, while the remaining stacking patterns exhibit a mixture of trivial and nontrivial domains. Consequently, the moiré supercell hosts both Z2=1Z_2 = 1 and Z2=0Z_2 = 0 domains in the absence of an applied field. Increasing the field strength to approximately 0.10–0.15 eV/Å renders all stacking configurations trivial, corresponding to a complete OFF state of the conducting edge channels in the moiré supercell. Upon further increasing the field, nontrivial domains reemerge, effectively turning the edge channels back to the ON state.

It is worth noting that the electric field plays a dual role here: (i) it enhances the out-of-plane interlayer tunneling, driving the topological phase transition [28], and (ii) it shifts the bands near the Fermi level that are responsible for topological band inversion. As these bands are pushed away from the Fermi level, the band inversion vanishes and the system transitions into a topologically trivial phase.

Continuum modelling of emergent topological networks and real-space imaging of gapless edge states. The complex nature of the multi-domain textures revealed in Figure 2-Figure 3 raises the question of the underlying topological network structure in moiré matter. Given the numerical difficulty in accurately determining the topological properties from a full ab-initio calculation, here we provide a conceptual model that captures all the essential physics emerging from full DFT calculations. We propose a minimal continuum model reflecting the strong SOC and the natural composition of Bi-pzp_z orbitals in the low energy sector of the (untwisted) system [28]. Writing σ=(σx,σy,σz)\boldsymbol{\sigma} = (\sigma_x, \sigma_y, \sigma_z) for the orbital pseudospin Pauli matrices generated by the Bi-pzp_z orbitals localized on individual layers [28], we construct a spatially modulated 2D analogue of the Bernevig-Hughes-Zhang (BHZ) model (variant of the Kane-Mele model) on the continuum, [33, 34]:

H(r)=ℏv σ⋅k 1^+M(r) σz,k=−i∇.(1)H(\mathbf{r}) = \hbar v\,\boldsymbol{\sigma}\cdot\mathbf{k}\,\hat{1} + M(\mathbf{r})\,\sigma_z,\qquad\mathbf{k} = -i\nabla. \tag*{(1)}

The local Z2Z_2 of the gapped phase is set by sgn⁡M\operatorname{sgn} M: M>0M > 0 trivial, M<0M < 0 inverted, by convention.

Across an isolated M=0M = 0 contour, a single chiral Dirac mode appears in a single time-reversal fixed sector, while its partner adds the counter-propagating mode in the second sector. Together, they form a helical Kramers pair localized within a characteristic length scale

ξ(r)≃ℏv∣M(r)∣(2)\xi(\mathbf{r}) \simeq\frac{\hbar v}{\lvert M(\mathbf{r})\rvert} \tag*{(2)}

at the domain wall [35]. The observation of a sharp edge-state network implies ξ≪aM\xi\ll a_M, where aMa_M is the moiré periodicity. By construction in Figure 3, aM≈2a_M \approx2 nm. Using ℏv=120\hbar v = 120 meV·nm along with a typical mass scale ∣M∣∼1\lvert M\rvert\sim1 eV, which is consistent with the strength of interorbital repulsion in this system [28], we obtain ξ∼0.1\xi\sim0.1 nm, placing the system well within the sharp edge-state network regime.

We parameterize M(r)M(\mathbf{r}) using first-shell harmonics on the triangular moiré lattice with reciprocal vectors b1,b2\mathbf{b}_1,\mathbf{b}_2, and b3=−(b1+b2)\mathbf{b}_3 = -(\mathbf{b}_1 + \mathbf{b}_2):

M(r)=M0+VC∑i=13cos⁡(bi⋅r)+VS∑i=13sin⁡(bi⋅r).(3)M(\mathbf{r}) = M_0 + V_C \sum_{i=1}^{3}\cos(\mathbf{b}_i\cdot\mathbf{r}) + V_S \sum_{i=1}^{3}\sin(\mathbf{b}_i\cdot\mathbf{r}). \tag*{(3)}

This expression represents the most general first-shell scalar field preserving C3C_3 symmetry on a triangular lattice. We note that VCV_C is generically present for any moiré lattice with a triangular unit cell. The chiral (and inversion symmetry breaking) properties of the twisting are encoded in VSV_S. To leading order, VS/VC∝θV_S/V_C \propto\theta.

Structural relaxation, captured by our first-principles calculations, strongly renormalizes the values of VCV_C and VSV_S, and naturally incorporates the mirror-symmetry breaking between different stacking regions. As a result, the AB and BA domains become inequivalent and acquire distinct local Z2Z_2 character. At the same time, the smooth spatial variation of M(r)M(\mathbf{r}) enables multiple helical edge channels to merge at the boundaries of topologically trivial and nontrivial domains. Within this framework of M(r)M(\mathbf{r}), M0M_0 preserves full rotational symmetry, VcV_c retains C6vC_{6v} symmetry, and VsV_s lowers the point group to C3vC_{3v}. The competition between the relative magnitude of these terms governs both the symmetry breaking induced by twisting and the resulting topological networks in moiré matter.

We now proceed to solve Eq. (1) by diagonalizing the Hamiltonian in the moiré basis. Letting the wavefunction be ψk,n(r)=∑G,σcG,σk,nei(k+G)⋅r∣l⟩\psi_{\mathbf{k},n}(\mathbf{r})=\sum_{\mathbf{G},\sigma}c_{\mathbf{G},\sigma}^{k,n}e^{i(\mathbf{k}+\mathbf{G})\cdot\mathbf{r}}|l\rangle, where ll is the layer basis encoded by σ\sigma. The Hamiltonian becomes:

[H(k)](Gl),(G′l′)=δGG′ℏv[(k+G)⋅σ]ll′+MG−G′[σz]ll′(4)\left[H(\mathbf{k})\right]_{(\mathbf{G}l),(\mathbf{G}'l')}=\delta_{\mathbf{G}\mathbf{G}'}\hbar v\left[(\mathbf{k}+\mathbf{G})\cdot\boldsymbol{\sigma}\right]_{ll'}+M_{\mathbf{G}-\mathbf{G}'}[\sigma_z]_{ll'} \tag*{(4)}

and the LDOS is:

ρ(r,E)=1Nk∑k∑n∣ψk,n(r)∣2δ(E−Ek,n).\rho(\mathbf{r},E)=\frac{1}{N_k}\sum_{\mathbf{k}}\sum_n|\psi_{\mathbf{k},n}(\mathbf{r})|^2\delta(E-E_{\mathbf{k},n}).

In Figure 5, we plot the diagonalization of Eq. (4) for particular parameters chosen to reproduce the DFT results of Figure 3 (see caption). We find an expected inverted mass distribution and a honeycomb-like pattern reflecting the underlying C3vC_{3v} symmetry of the moiré lattice. This mass distribution, contains a component which flips sign with reversing the twist direction, directly revealing its chiral origin [15].

Figure 5. Mass distribution M(r)M(\mathbf{r}) and LDOS for the effective model of Eq. (1) which reproduce the full DFT calculations. Parameters are: Vc=−100V_c=-100, Vs=±6V_s=\pm6, and M0=−75M_0=-75 (all meV). For visualization purposes we take aM=14a_M=14 nm. ℏv=120\hbar v=120 meV nm. (a-b) Mass distribution as a function of position showing a clear inversion (blue regions) and trivial domain walls (faint red). (a) and (b) show +θ+\theta and −θ-\theta twists, respectively. (c-d) Corresponding LDOS maps showing the inversion-odd partners. Bright yellow spots indicate the topological edge states found near E=0E=0.

its chiral origin [15]. We then compute the LDOS, near E≈0E\approx0, and plot it in Figure 5(c)-(Figure 5d). We observe pronounced one-dimensional edge states which emerge precisely along the boundaries separating topologically nontrivial and trivial domains, whereas the domains themselves remain gapped and exhibit zero DOS (purple hue). The observed asymmetry between inequivalent sites originates from the relative magnitude of VsV_s and VcV_c, which produces a slightly larger positive (trivial) mass in regions where the edge states percolate. This behaviour is consistent with a network-scattering picture in which edge modes preferentially localize along boundaries with the strongest mass inversion, reminiscent of the Chalker-Coddington model [15]. Importantly, the LDOS maps display a clear contrast under reversal of the twist chirality, providing a direct real-space probe of the local moiré chirality.

Taken together, these results demonstrate that scanning probe measurements can be used to reconstruct the spatial mass profile M(r)M(\mathbf{r}) from LDOS data, similar to recent proposals for mapping electrostatic potentials in moiré systems [36, 37].

Having established the real-space topological structure, we next analyze the topology of the moiré bands in momentum space. Within the continuum framework, the evaluation of the Z2Z_2 invariant in the full Brillouin zone becomes numerically tractable. Supplementary Fig. S6 presents the moiré band structure together with the corresponding Chern numbers and the Z2Z_2 invariants for states filled up to a particular Fermi level. We identify an extended energy window with topological bands below the charge neutrality point, spanning an energy range of approximately VcV_c below EFE_F. We then show the existence of a broad trivial sector of occupied bands. These results match the observed coexistence of topological and trivial regions identified in DFT calculations shown in Figure 3(b). Combined with gate-tunable STM measurements, our framework provides an innovative route to reconstructing the topological landscape of moiré bands and extracting the underlying parameters governing the spatial mass texture M(r)M(\mathbf{r}).

Discussion

Discussion. Our results establish structural relaxation as a fundamental mechanism that reshapes topology in twisted vdW heterostructures. Rather than producing a uniform topological phase, the interplay of moiré reconstruction, interlayer hybridization, and spin-orbit coupling generates a spatially textured topological landscape in which topologically nontrivial and trivial regions coexist within a single moiré unit cell. This establishes a “topological mosaic” in which the local stacking environment directly controls the sign and magnitude of an effective mass term M(r)M(\mathbf{r}), making topology intrinsically real-space dependent. In particular, the spatial modulation of M(r)M(\mathbf{r}) demonstrates how local stacking environments can act as nanoscale switches between distinct topological regimes, emphasizing the need for real-space approaches when characterizing topology in moiré systems.

A caveat that is worth noting in pristine BiSb bilayers is that these are semimetallic in nature, with some topologically trivial electron/hole pockets lying near the Fermi level. However, as we demonstrate in the Supplementary Figure S5, this semimetallicity can be systematically reduced through chemical substitution with more electronegative elements such as iodine or fluorine, which open a band gap and drive the system toward a semiconducting regime. More broadly, the coexistence of nontrivial topology with metallic or lightly doped regimes is of significant interest, as it provides a potential platform for emergent correlated phases, including topological superconductivity [38], ferroelectricity [39], and magnetism [40].

Conclusions

Conclusions. We have demonstrated that twisting and structural relaxation in BiSb bilayers provide a route to programmable topological matter in moiré systems. Starting from two individually trivial monolayers (Z2=0Z_2 = 0), inversion-stacked and twisted bilayers develop a spatially varying topological phase landscape characterized by coexisting topologically nontrivial (Z2=1Z_2 = 1) and trivial (Z2=0Z_2 = 0) domains. Relaxation-driven modulation of interlayer separation plays a central role in stabilizing this phase separation in moiré matter.

Our continuum BHZ-type description, informed by first-principles calculations, reproduces the real-space topology and establishes a direct connection between STM observables and the underlying spatially varying mass term M(r)M(\mathbf{r}). This will further extend recent methods for the determination of topology through scanning probe measurements [41, 42]. The ability to tune domain size via twist angle and reversibly control edge-state connectivity using an electric field demonstrates a high degree of external programmability. More broadly, our results identify relaxation as a powerful design principle for engineering reconfigurable topological networks in vdW materials, opening pathways toward intrinsically patterned topological circuits and planar device architectures.

Methods

First-principles density functional theory (DFT) calculations were performed for the moiré supercells corresponding to three twist angles: 21.78∘21.78^\circ, 13.17∘13.17^\circ, and 9.43∘9.43^\circ using the projector augmented-wave (PAW) method, as implemented in the Vienna Ab initio Simulation Package (VASP) [43, 44]. The kinetic energy cutoff for the plane-wave basis set was set to 650 eV. The electronic energy convergence criterion was 10−710^{-7} eV, and the Hellmann–Feynman residual forces on each atom were converged to 10−310^{-3} eV/Å. For structural relaxations of the moiré supercell, a Γ\Gamma-centered 1×1×11 \times1 \times1 kk-mesh was used, while a denser 8×8×18 \times8 \times1 mesh was employed for partial charge density calculations. To study the stacking patterns corresponding to different atomic registries (i.e., AA, AB, etc.), a 12×12×112 \times12 \times1 kk-mesh was used. The exchange–correlation potential was treated within the generalized gradient approximation (GGA) as parameterized by Perdew, Burke, and Ernzerhof for solids (PBEsol) [45]. van der Waals (vdW) interactions were included using the DFT-D3 [46] approximation. A vacuum spacing of 15 Å was maintained along the out-of-plane direction. The PAW pseudopotentials included contributions from five valence electrons for both Bi (6s26p36s^26p^3) and Sb (5s25p35s^25p^3) atoms. VASPKIT [47] was used to simulate STM images by post-processing the partial charge density data.

Wannierization of the individual stacking configurations was performed using maximally localized Wannier functions, as implemented in Wannier90 [48], with initial projections taken from the ss, pxp_x, pyp_y, and pzp_z orbitals of Bi and Sb. The Z2Z_2 topological invariants were subsequently computed from the Wannier tight-binding Hamiltonian using WannierTools [49]. To study the effect of an external vertical electric field, a displacement field was applied to the Wannier tight-binding Hamiltonian,

Hαβ(k)=∑Rtαβ(R)eik⋅R,(5)H_{\alpha\beta}(\mathbf{k}) = \sum_{\mathbf{R}} t_{\alpha\beta}(\mathbf{R})e^{i\mathbf{k}\cdot\mathbf{R}}, \tag*{(5)}

where Ri=na1+ma2\mathbf{R}_i = n\mathbf{a}_1 + m\mathbf{a}_2 denotes lattice vectors, tαβt_{\alpha\beta} are hopping amplitudes in the Wannier basis, and αβ\alpha\beta represent orbital indices. Assuming the Wannier orbitals are well localized along the zz axis, the effect of the vertical displacement field EE was introduced as

Hαβ(k)=H0αβ(k)+δαβRαzE,(6)H^{\alpha\beta}(\mathbf{k}) = H_0^{\alpha\beta}(\mathbf{k}) + \delta_{\alpha\beta}R^z_\alpha E, \tag*{(6)}

where RαzR^z_\alpha is the zz coordinate of the α\alpha Wannier orbital and H0H_0 is the unperturbed Hamiltonian in Eq. (5).

Acknowledgements

A.B. and S.S. acknowledge support from the U.S. Department of Energy, Office of Science, Office of Fusion Energy Sciences, Quantum Information Science program under Award No. DE-SC-0020340. Authors also acknowledge support from the Furtch Research Fund at the University of Rochester. D.K. is supported by an Abrahams postdoctoral fellowship of the Center for Materials Theory, Rutgers University and the Zuckerman STEM fellowship. D.K. thanks the hospitality of the Aspen Center for Physics, which is supported by National Science Foundation grant PHY-2210452, where a part of this work was carried out. A.B. and S.S. thank the Pittsburgh Supercomputer Center (Bridges2) supported by the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.

References

References

  1. [1]Yuya Shimazaki, Ido Schwartz, Kenji Watanabe, Takashi Taniguchi, Martin Kroner, and Ataç İmamoğlu, “Strongly correlated electrons and hybrid excitons in a moiré heterostructure,” Nature 580, 472–477 (2020).
  2. [2]Lei Wang, En-Min Shih, Augusto Ghiotto, Lede Xian, Daniel A Rhodes, Cheng Tan, Martin Claassen, Dante M Kennes, Yusong Bai, Bumho Kim, et al., “Correlated electronic phases in twisted bilayer transition metal dichalcogenides,” Nature materials 19, 861–866 (2020).DOI
  3. [3]Heonjoon Park, Jiaqi Cai, Eric Anderson, Yinong Zhang, Jiayi Zhu, Xiaoyu Liu, Chong Wang, William Holtzmann, Chaowei Hu, Zhaoyu Liu, et al., “Observation of fractionally quantized anomalous hall effect,” Nature 622, 74–79 (2023).DOI
  4. [4]Jiaqi Cai, Eric Anderson, Chong Wang, Xiaowei Zhang, Xiaoyu Liu, William Holtzmann, Yinong Zhang, Fengren Fan, Takashi Taniguchi, Kenji Watanabe, et al., “Signatures of fractional quantum anomalous hall states in twisted mote2,” Nature 622, 63–68 (2023).DOI
  5. [5]Yiyu Xia, Zhongdong Han, Kenji Watanabe, Takashi Taniguchi, Jie Shan, and Kin Fai Mak, “Superconductivity in twisted bilayer wse2,” Nature 637, 833–838 (2025).
  6. [6]Yinjie Guo, Jordan Pack, Joshua Swann, Luke Holtzman, Matthew Cothrine, Kenji Watanabe, Takashi Taniguchi, David G Mandrus, Katayun Barmak, James Hone, et al., “Superconductivity in 5.0° twisted bilayer wse2,” Nature 637, 839–845 (2025).
  7. [7]You Zhou, Jiho Sung, Elise Brutschea, Ilya Esterlis, Yao Wang, Giovanni Scuri, Ryan J Gelly, Hoseok Heo, Takashi Taniguchi, Kenji Watanabe, et al., “Bilayer wigner crystals in a transition metal dichalcogenide heterostructure,” Nature 595, 48–52 (2021).DOI
  8. [8]Hongyuan Li, Shaowei Li, Emma C Regan, Danqing Wang, Wenyu Zhao, Salman Kahn, Kentaro Yumigeta, Mark Blei, Takashi Taniguchi, Kenji Watanabe, et al., “Imaging two-dimensional generalized wigner crystals,” Nature 597, 650–654 (2021).DOI
  9. [9]Emma C Regan, Danqing Wang, Chenhao Jin, M Iqbal Bakti Utama, Beini Gao, Xin Wei, Sihan Zhao, Wenyu Zhao, Zuocheng Zhang, Kentaro Yumigeta, et al., “Mott and generalized wigner crystal states in wse2/ws2 moiré superlattices,” Nature 579, 359–363 (2020).
  10. [10]Cameron JR Duncan, Amalya C Johnson, Indrajit Maity, Angel Rubio, Matthew Gordon, Adam C Bartnik, Michael Kaeningk, William H Li, Matthew B Andorf, Chad A Pennington, et al., “Photoinduced twist and untwist of moiré superlattices,” Nature, 1–6 (2025).
  11. [11]Indrajit Maity, Prabal K Maiti, HR Krishnamurthy, and Manish Jain, “Reconstruction of moiré lattices in twisted transition metal dichalcogenide bilayers,” Physical Review B 103, L121102 (2021).
  12. [12]Astrid Weston, Eli G Castanon, Vladimir Enaldiev, Fabio Ferreira, Shubhadeep Bhattacharjee, Shuigang Xu, Héctor Corte-León, Zefei Wu, Nicholas Clark, Alex Summerfield, et al., “Interfacial ferroelectricity in marginally twisted 2d semiconductors,” Nature nanotechnology 17, 390–395 (2022).DOI
  13. [13]Matthew R Rosenberger, Hsun-Jen Chuang, Madeleine Phillips, Vladimir P Oleshko, Kathleen M McCreary, Saujan V Sivaram, C Stephen Hellberg, and Berend T Jonker, “Twist angle-dependent atomic reconstruction and moiré patterns in transition metal dichalcogenide heterostructures,” ACS nano 14, 4550–4558 (2020).DOI
  14. [14]VV Enaldiev, Fabio Ferreira, SJ Magorrian, and Vladimir I Fal’ko, “Piezoelectric networks and ferroelectric domains in twistronic superlattices in ws2/mos2 and wse2/mose2 bilayers,” 2D Materials 8, 025030 (2021).
  15. [15]JT Chalker and PD Coddington, “Percolation, quantum tunnelling and the integer hall effect,” Journal of Physics C: Solid State Physics 21, 2665–2679 (1988).
  16. [16]Qingjun Tong, Hongyi Yu, Qizhong Zhu, Yong Wang, Xiaodong Xu, and Wang Yao, “Topological mosaics in moiré superlattices of van der waals heterobilayers,” Nature Physics 13, 356–362 (2017).DOI
  17. [17]Ikuma Tateishi and Motoaki Hirayama, “Quantum spin hall effect from multiscale band inversion in twisted bilayer bi 2 (te 1- x se x) 3,” Physical Review Research 4, 043045 (2022).
  18. [18]Pablo San-Jose and Elsa Prada, “Helical networks in twisted bilayer graphene under interlayer bias,” Phys. Rev. B 88, 121408(R) (2013).DOI
  19. [19]Dmitry K. Efetov and Allan H. MacDonald, “Helical network model for twisted bilayer graphene,” Phys. Rev. B 98, 035404 (2018).
  20. [20]Stephen Carr, Sh iang Fang, and Efthimios Kaxiras, “Electronic-structure methods for twisted moiré layers,” Nature Reviews Materials 5, 748–763 (2020).DOI
  21. [21]Stephen Carr, Daniel Massatt, Steven B Torrisi, Paul Cazeaux, Mitchell Luskin, and Efthimios Kaxiras, “Relaxation and domain formation in incommensurate two-dimensional heterostructures,” Physical Review B 98, 224102 (2018).DOI
  22. [22]Nguyen N. T. Nam and Mikito Koshino, “Lattice relaxation and energy band modulation in twisted bilayer graphene,” Phys. Rev. B 96, 075311 (2017).
  23. [23]Mikito Koshino and Nguyen NT Nam, “Effective continuum model for relaxed twisted bilayer graphene and moiré electron-phonon interaction,” Physical Review B 101, 195425 (2020).
  24. [24]Naoto Nakatsuji, Takuto Kawakami, and Mikito Koshino, “Multiscale lattice relaxation in general twisted trilayer graphenes,” Physical Review X 13, 041007 (2023).DOI
  25. [25]Jingtian Shi, Gaurav Chaudhary, Allan H. MacDonald, and Ivan Martin, “Spontaneous twirls and structural frustration in moiré materials,” Phys. Rev. Lett. 136, 026101 (2026).DOI
  26. [26]Sobhit Singh and Aldo H. Romero, “Giant tunable rashba spin splitting in a two-dimensional bisb monolayer and in bisb/aln heterostructures,” Phys. Rev. B 95, 165444 (2017).DOI
  27. [27]Daniel Kaplan, Ady Stern, and Binghai Yan, “Even integer quantum hall effect in materials with hidden spin texture,” (2024), arXiv:2406.08348 [cond-mat.mes-hall].
  28. [28]Arjyama Bordoloi, Daniel Kaplan, and Sobhit Singh, “Engineering stacking-induced topological phase transitions in bilayer heterostructures,” 2D Materials 13, 015006 (2026).DOI
  29. [29]Jonathon Mark Riley, F Mazzola, M Dendzik, M Michiardi, T Takayama, Lewis Bawden, C Granerød, Mats Leandersson, T Balasubramanian, M Hoesch, et al., “Direct observation of spin-polarized bulk bands in an inversion-symmetric semiconductor,” Nature Physics 10, 835–839 (2014).DOI
  30. [30]Linding Yuan, Qihang Liu, Xiuwen Zhang, Jun-Wei Luo, Shu-Shen Li, and Alex Zunger, “Uncovering and tailoring hidden Rashba spin–orbit splitting in centrosymmetric crystals,” Nature Communications 10, 906 (2019).
  31. [31]Stephen Carr, Daniel Massatt, Shiang Fang, Paul Cazeaux, Mitchell Luskin, and Efthimios Kaxiras, “Twistronics: Manipulating the electronic properties of two-dimensional layered structures through their twist angle,” Physical Review B 95, 075420 (2017).
  32. [32]Paul Cazeaux, Drake Clark, Rebecca Engelke, Philip Kim, and Mitchell Luskin, “Relaxation and domain wall structure of bilayer moiré systems,” Journal of Elasticity 154, 443–466 (2023).
  33. [33]B Andrei Bernevig, Taylor L Hughes, and Shou-Cheng Zhang, “Quantum spin hall effect and topological phase transition in hgte quantum wells,” science 314, 1757–1761 (2006).
  34. [34]C. L. Kane and E. J. Mele, “Quantum Spin hall effect in graphene,” Physical Review Letters 95, 226801 (2005).
  35. [35]Roman Jackiw and Cláudio Rebbi, “Solitons with fermion number 1/2,” Physical Review D 13, 3398 (1976).
  36. [36]Eric Seewald, Sanat Ghosh, Nishchal Verma, John Cenker, Yinan Dong, Birui Yang, Amit Basu, Takashi Taniguchi, Kenji Watanabe, Mandar M Deshmukh, et al., “Mapping the moiré potential in multi-layer rhombohedral graphene,” arXiv preprint arXiv:2510.09548 (2025).arxiv.org/abs/2510.09548
  37. [37]Dahlia R Klein, Uri Zondiner, Amit Keren, John Birkbeck, Alon Inbar, Jiewen Xiao, Yuval Zamir, Mariia Sidorova, Mohammed M Al Ezzi, Liangtao Peng, et al., “Imaging the sub-moiré potential using an atomic single electron transistor,” Nature , 1–7 (2026).
  38. [38]Daniele Guerci, Daniel Kaplan, Julian Ingham, JH Pixley, and Andrew J Millis, “Topological superconductivity from repulsive interactions in twisted wse _2,” arXiv preprint arXiv:2408.16075 (2024).arxiv.org/abs/2408.16075
  39. [39]Xiao-Wei Zhang, Chong Wang, Xiaoyu Liu, Yueyao Fan, Ting Cao, and Di Xiao, “Polarization-driven band topology evolution in twisted mote2 and wse2,” Nature Communications 15, 4223 (2024).DOI
  40. [40]Liangtao Peng, Christophe De Beule, Yiyang Lai, Du Li, Li Yang, EJ Mele, and Shaffique Adam, “Itinerant magnetism in twisted bilayer wse 2 and mote 2,” Physical Review B 113, L161119 (2026).
  41. [41]Madisen Holbrook, Julian Ingham, Daniel Kaplan, Luke N Holtzman, Brenna Biermann, Bowen Hou, Nicholas Olsen, Luca Nashabeh, Yiliu Li, Song Liu, et al., “Real-space imaging of the band topology of transition metal dichalcogenides,” Nature Physics , 1–6 (2026).
  42. [42]Dumitru Călugăru, Yi Jiang, Haojie Guo, Sandra Sajan, Yongsong Wang, Haoyu Hu, Jiabin Yu, B Andrei Bernevig, Fernando de Juan, and Miguel M Ugeda, “Observation of an obstructed atomic band in a transition metal dichalcogenide,” Nature Physics , 1–6 (2026).
  43. [43]G. Kresse and J. Furthmüller, “Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set,” Phys. Rev. B 54, 11169–11186 (1996).
  44. [44]G. Kresse and J. Furthmüller, “Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set,” Comput. Mater. Sci. 6, 15–50 (1996).
  45. [45]John P. Perdew, Adrienn Ruzsinszky, Gábor I. Csonka, Oleg A. Vydrov, Gustavo E. Scuseria, Lucian A. Constantin, Xiaolan Zhou, and Kieron Burke, “Restoring the density-gradient expansion for exchange in solids and surfaces,” Phys. Rev. Lett. 100, 136406 (2008).
  46. [46]Stefan Grimme, Jens Antony, Stephan Ehrlich, and Helge Krieg, “A consistent and accurate ab initio parametrization of density functional dispersion correction (dft-d) for the 94 elements h-pu,” J. Chem. Phys. 132, 154104 (2010).
  47. [47]Wei Wang, Nan Xu, Jin-Cheng Liu, Gang Tang, and Wen-Tong Geng, “VASPkit: A user-friendly interface facilitating high-throughput computing and analysis using VASP code,” Computer Physics Communications 267, 108033 (2021).
  48. [48]Arash A Mostofi, Jonathan R Yates, Giovanni Pizzi, Young-Su Lee, Ivo Souza, David Vanderbilt, and Nicola Marzari, “An updated version of wannier90: A tool for obtaining maximally-localised wannier functions,” Computer Physics Communications 185, 2309–2310 (2014).
  49. [49]QuanSheng Wu, ShengNan Zhang, Hai-Feng Song, Matthias Troyer, and Alexey A. Soluyanov, “Wannier-tools: An open-source software package for novel topological materials,” Comput. Phys. Commun. 224, 405–416 (2018).

Paper details

vlnc:W_2d4g5aj2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents