WorksPhysics and Astronomy
Approximate SU(4) spin models on triangular and honeycomb lattices in twisted AB-Stacked WSe homo-bilayer
Abstract
In this paper, we derive lattice models for the narrow moiré bands of the AB-stacked twisted WSe homobilayer through continuum model and Wannier orbital construction. Previous work has shown that an approximate SU(4) Hubbard model may be realized by combining spin and layer because inter-layer tunneling is suppressed due to spin conservation. However, Rashba spin-orbit coupling (SOC) was ignored in the previous analysis. Here, we show that a Rashba SOC of reasonable magnitude can induce a finite but very small inter-layer hopping in the final lattice Hubbard model. At total filling , we derive a spin-layer model on a triangular lattice in the large-U limit where the inter-layer tunneling contributes as a sublattice-dependent transverse Ising field for the layer pseudospin. We then show that the Mott insulator is also captured by an approximate SU(4) spin model, but now on honeycomb lattice. We comment on the possibility of a Dirac spin liquid (DSL) and competing phases due to SU(4) anisotropy terms.
Introduction
Recent experimental progresses in moiré superlattices[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22] make it possible to study spin physics in a Hubbard model on a moiré superlattice[23, 24, 25, 26, 27, 28, 29, 30, 31]. With these highly tunable systems, it is natural to ask whether we can realize unconventional spin phases which are not magnetically ordered. One route to disordering magnetic order is through adding frustration[32, 33, 34, 35, 36, 37, 38, 39, 40] or charge fluctuation[41, 42] to a spin model. Another route is to consider an SU() spin model with , where the magnetic order is generally suppressed and a spin liquid phase may be stabilized even in the simplest Heisenberg model[43, 44]. However, it is challenging to realize an SU() spin model in real systems. Graphene moiré system may simulate an SU(4) spin model[45]. We also note a recent proposal of SU(8) model in twisted bilayer graphene[46]. One issue for the graphene moiré systems is that there is generically a large valley contrasting flux[47]. In this paper, we focus on the moiré superlattices formed by transition metal dichalcogenides (TMDs), where the valley contrasting flux is usually negligible[23], except in twisted AA stacked TMD homo-bilayer with a large displacement field[26].
With TMD layers, one can realize a moiré bilayer to simulate an SU(4) Hubbard model by combining the spin and layer degree of freedom[48]. If the inter-layer tunneling is suppressed and inter-layer distance is much smaller than the moiré lattice constant, there is an approximate SU(4) symmetry. There are two different ways to suppress the inter-layer tunneling to realize a moiré bilayer. The first one is to insert an insulating barrier such as a hexagon boron nitride (hBN) layer in the middle. However, a thick hBN will lead to a smaller inter-layer repulsion compared to the intra-layer repulsion and reduces the U(4) symmetry to [49]. Although interesting phases can still arise in this less symmetric case[49], in this paper we are interested in improving the SU(4) symmetry. The second realization of a moiré bilayer is the twisted AB-stacked transition metal dichalcogenide (TMD) homo-bilayer[48], which was recently realized in an experiment[50]. AB stacking is generated from the standard AA stacking through rotating one TMD layer by relative to the other layer. Therefore, the two TMD layers at the same valley now have opposite spin splittings. The highest valence bands have opposite spin in the two layers at the same valley (see Figure 1(a)). As a result, the inter-layer tunneling is forbidden due to the spin conservation[51].

Figure 1. (a) Illustration of spin-valley locking of the low-energy valence bands in AB stacking. (b) Illustration of spin-valley locking of the low-energy valence bands in AA stacking. (c) Schematics of the moiré pattern formed by AB-stacked twisted bilayer TMD . High symmetry regions are highlighted with circles. ‘MX’ means the metal atoms in the top layer are aligned with the chalcogen atoms in the bottom layer, and other areas are defined similarly. There is a symmetry which rotates the system by 180° around the x axis. (d) Moiré pattern of AA-stacked twisted bilayer . There is a symmetry now.
However, in the analysis above, we ignore the Rashba spin-orbit coupling (SOC). With a Rashba SOC, spin is not a good quantum number anymore and one may imagine a finite inter-layer tunneling, similar to what is happening in the AB-stacked TMD hetero-bilayer[52, 53]. In this paper we analyze the effect of the Rashba SOC in the AB stacked TMD homo-bilayer. It is known that Rashba SOC usually arises in hetero-structures with a finite vertical electric field[54]. This is indeed expected for TMD hetero-bilayer. However, for TMD homo-bilayer, there is no vertical electric field if the displacement field is zero. Hence, the Rashba SOC may be suppressed. When , there is indeed a symmetry which exchanges the two layers and forbids the layer symmetric Rashba SOC. The only allowed Rashba SOC term must be opposite in the two layers. It is unclear to us how this layer-opposite Rashba SOC can arise microscopically. We conjecture its value is negligible. But to obtain an upper bound of the inter-layer tunneling, we simply add layer-opposite Rashba SOC of a large magnitude (comparable to the value in a hetero-bilayer) to the continuum model and then extract an on-site inter-layer hopping term in low-energy lattice Hubbard model through Wannier orbital construction. The inter-layer hopping turns out to be quite small (at order of ) even with large layer-opposite Rashba SOC.
At total filling , the Mott insulator in the large- limit is described by an approximate SU(4) model. The small inter-layer hopping results in a sublattice dependent transverse Ising field in the layer pseudospin space in a convenient gauge. At zero magnetic field, previous work showed that the ground state of the SU(4) Heisenberg model on a triangular lattice has a plaquette-ordered ground state[51].
These authors contributed equally.; [email protected] † These authors contributed equally.; bz [email protected] ‡ [email protected]
We expect that the plaquette-ordered phase is stable with a small transverse Ising field. We can polarize the real spin with a large magnetic field in the z-direction and get a spin-1/2 XXZ model for the layer pseudospin[55]. We analyze the phase diagram with an additional displacement field and a small sublattice dependent transverse Ising field originating from the inter-layer tunneling. We find a 1/3 plateau in layer polarization when varying the displacement field , which is stable against the small transverse Ising field.
We then turn to the total filling . Naively, one may expect a Mott insulator on a triangular lattice where each site is occupied by two particles, resulting in a representation of the approximate symmetry[45]. However, a careful analysis shows that the two particles in a unit cell prefer to stay in A, B sublattice of a honeycomb lattice to reduce the on-site Hubbard [25]. An appropriate lattice model for turns out to be an approximate Hubbard model on a honeycomb lattice, which we derive through Wannier orbital construction of the lowest two moiré bands (each band consists of four flavors coming from valleys and layers). Then the Mott insulator at is described by an approximate spin model on a honeycomb lattice, which may host a Dirac spin liquid as indicated by previous calculations[56, 57, 58] and theory[59]. We discuss the effect of anisotropy terms and possible competing phases.
Continuum model with Rashba SOC
Continuum model and symmetry
We consider a twisted TMD homo-bilayer with a small twist angle . Due to the lack of inversion symmetry within each TMD layer, there are two inequivalent stacking patterns at zero twist angle. In the AA stacking, each atom of one TMD layer aligns with the corresponding one in the other layer. In the AB stacking, one rotates one … the spin conservation. This property still holds at a small twist angle relative to the AB stacking. Another difference between the twisted AB stacking and AA stacking TMD homo-bilayer is in their symmetries. AA stacking has a symmetry at zero displacement field which rotates the system by around the -axis and exchanges the two layers, while AB stacking’s corresponding symmetry is which rotates the system by around the -axis and exchanges the two layers (see Figure 1(c)(Figure 1d)). As we will see, the different symmetries in the two stacking patterns lead to different Hamiltonians and properties.
Due to the twist, the valence bands from the valley and are folded into a mini moiré Brillouin zone (MBZ) as shown in Figure 2. We construct a continuum model including 8 bands from spin, valley and layer. We use , to label the Pauli matrices in layer, valley and spin subspace respectively. We define , where is the annihilation operator of a hole and denotes the top layer and the bottom layer. We can then define , where is the absolute momentum relative to the same origin for both layers and both valleys. It is also convenient to define a relative momentum , where is the momentum of the valley at layer . In the following we use for simplicity.
Figure 2. The blue and orange hexagons are the Brillouin zones of the top and bottom layer in the hole picture, respectively. Twisting them with an angle creates the moiré Brillouin zone depicted by the black hexagon.
The Hamiltonian for the continuum model in the hole picture is then:
where is the effective hole mass, is the voltage between the two layers, is the layer symmetric Rashba SOC induced by an external electric field, is a layer-opposite Rashba SOC (LORSOC) allowed by symmetry (explained below) at , which has been studied in bilayer graphene[60], is the Ising spin-orbit coupling constant. s are the reciprocal vectors of the MBZ, and is an interlayer tunneling strength parameter. We choose to be , where nm ( is the twist angle) is the moiré superlattice constant, and all the other G vectors are generated by 60-degree counterclockwise rotations of (see Figure 2).
At , there is a symmetry which acts as: . It enforces , so there is only the layer-opposite Rashba term at . Also the moiré Hamiltonian is fully constrained by the symmetry and another rotation symmetry which acts as . is defined relative to the valleys K and K′ of the corresponding layer. We note that the symmetry guarantees that the moiré potentials have the same phase in the two layers for the same . In contrast, for the AA stacking, the symmetry requires the phase to be opposite for the two layers at the same . This is crucial for the approximate SU(4) symmetry at for the twisted AB stacked homo-bilayer.
The LORSOC in Eq. (1) is allowed at by symmetry. It is not clear that how it can arise microscopically, but it should be quite small given that there is no vertical electric field between the two TMD layers at . Typical Rashba SOC in systems with strong mirror symmetry breaking is around 30 meVÅ [61]. We believe in our system at is significantly smaller. However, in the following we will use a at the order of 10 meVÅ to obtain an upper bound of inter-layer hybridization. The layer isotropic Rashba SOC should be proportional to . Based on previous first-principle calculations in bilayer MXY[62], we use as an estimation, where is a proportionality constant and 0.7nm is the estimated inter-layer distance.
We compute the band structure from Eq. (1) with degrees, [63], meV, , meV[64], meV[65], meVÅ. We plot the first two moiré bands in the valley K with several values of in Figure 3. Even when the LORSOC is as large as 20meVÅ, we can see that the bands from the two layers are almost decoupled, indicating a very small inter-layer tunneling. We note that the used here is overestimated. Thus, we expect the inter-layer hybridization to be even weaker in realistic system.

Figure 3. Dispersion relations of the first two moiré bands from the same valley at different displacement field . The colors of the bands indicate layer polarization . (a) mV, meV · Å (b) mV, meV · Å (c) mV, meV · Å (d) mV, meV · Å. In all of the plots we set and include a layer-opposite Rashba SOC meV · Å.
Lattice model and inter-layer hopping when
From the band structure calculation, we can see that the bands of the two layers are almost decoupled even with the Rashba SOC included. If we construct a lattice model for
a band at one layer and one valley, it should be a simple tight-binding model on a triangular lattice. The question is how large the inter-layer tunneling is. To estimate it, we construct Wannier orbitals for the lowest two bands (from the two layers) at each valley on a triangular lattice. The two valleys are related by the time reversal symmetry, so we can only focus on one valley. We obtain an extended Hubbard model (see Appendix A for details):
where refers to the nearest neighbors, is the real nearest-neighbour intra-layer hopping, is the on-site inter-layer tunneling. Because spin is not a good quantum number, we ignore the spin index. should be locked to valley, so one can view valley as the usual spin in the familiar Hubbard model. is the density at layer (summed over valleys), , so that the flux through each triangle is as shown in Figure 4. and are the intra-layer and inter-layer on-site repulsion, and are the intra-layer and inter-layer repulsion among nearest neighbours. We use , to be the triangular lattice primitive vectors, is the moiré

Figure 4. Illustration of the hopping phase for in Eq. (2). We have . Along the arrow we have . Neighbouring triangles have opposite flux .
superlattice constant. Note that we are free to choose the phase of . In this paper, we let it be real.
We plot in Figure 5 (a) and in Figure 5 (b) at with several as functions of the twist angle . The on-site inter-layer tunneling is smaller than 1% of the nearest neighbour intra-layer hopping at for meV · Å and . We can see that decreases with , mainly because increases with the twist angle. We also want to check whether this result depends on the choice of in our model. We provide a plot of as a function of in Figure 6. One can see that the choice of does not affect the order of magnitude of .
Figure 5. (a)Nearest-neighbour intra-layer hopping vs the twist angle at D = 0, = 142°. is independent of . (b)Ratio of on-site inter-layer tunneling to nearest-neighbour intra-layer hopping vs the twist angle at D = 0 with different LORSOC constants . = 142°.
Figure 6. Ratio of on-site inter-layer tunneling to nearest-neighbour intra-layer hopping vs at D = 0, , .
A more convenient gauge
It is convenient to gauge away the phase of the nearest neighbor hopping. Let us define . The flux in each triangle can be gauged away by the transformation: , , . Note that is a corner of the MBZ.
After the gauge transformation, we reach a new lattice model:
where .
If we ignore the term and set , the above model has a symmetry. The cost of the gauge transformation is to add a position dependent phase on the inter-layer on-site hopping. In the following we will use this gauge so that the approximate SU(4) symmetry is explicit.
Approximate SU(4) spin model and the effect of inter-layer hopping at
Approximate SU(4) spin model
In a typical system, is about 0.01 0.1[66], so we can obtain a spin model through a Schrieffer-Wolff transformation of Eq. (3) in [67] for the Mott insulating phase at total filling . We define a layer pseudo-spin operator and a spin-valley operator . The effective spin model is:
where , , . , are anisotropy terms breaking SU(4) spin symmetry due to the distance between the two layers. , are estimated to be 0.2 and 0.3 at [48]. When , the above model is an SU(4) Heisenberg model of which the ground state was shown to be in a plaquette order[48]. It is interesting to note that the inter-layer tunneling term acts as a sublattice-dependent transverse Ising field in the layer pseudo-spin space.
Layer pseudo-spin magnetism and effect of transverse Ising field
Now, we study the effect of the sublattice dependent transverse Ising field. For simplicity, we apply a strong magnetic field to polarize the valleys in order to neglect in Eq. (4). Then, we only need to consider an XXZ model with a transverse Ising field coupled to the layer pseudospin :
,
,
,
,
One can see that the potential difference now plays the role of a Zeeman field in the layer pseudo-spin space. The inter-layer tunneling acts as a sublattice dependent transverse Ising field. Without the transverse Ising field, this is an XXZ model with a Zeeman field, which has been studied [68, 69, 70]. For example, it is known that there is a plateau in the magnetization when increasing the Zeeman field . For our system, this plateau will be manifested in the layer polarization . The effect of a uniform transverse Ising field has been studied[71]. But there is no discussion on the sublattice-dependent transverse Ising field in Eq. (5). We plot the directions of the field’s projection in the x-y plane in Figure 7.

Figure 7. Illustration of the sublattice-dependent effective transverse Ising field from the inter-layer tunneling in the space. Here label three sublattices. labels the direction of the transverse Ising field.
We compute the polarization curves of Eq. (5) numerically using the standard linear spin wave theory (see Appendix § 2.2 for details). We use because , . We show the polarization curves with the transverse Ising field in Figure 8. The plateau is clearly visible with . We believe that is smaller than in typical systems because the inter-layer tunneling is found to be of and . Therefore, we propose to search for the plateau in layer polarization by increasing the displacement field under a strong magnetic field in experiments[50].

Figure 8. Layer polarization vs displacement field at different sublattice dependent transverse Ising field . We fix .
Then, we study the phase diagram of the above model by varying and while fixing . The result is shown in Figure 9 (see Appendix § 2.2 for details). We find that the plateau survives if . When is weak, the phases are similar to an XXZ model in a z-direction magnetic field[70], but an umbrella phase replaces the co-planar V-shape phase. When is strong, the vectors form a 120-degree structure in the x-y plane. When a displacement field is applied, the vectors form an umbrella structure. In usual twisted bilayer TMDs, we only realize the bottom part of the phase diagram because is small.

Figure 9. Phase diagram of Eq. (5) when . is the effective sublattice dependent x-y plane magnetic field originating from the inter-layer hopping . is the voltage between two layers and is an effective magnetic field in the z-direction. The dashed lines mark first-order phase transitions where jumps and solid lines mark second-order phase transitions. The line was studied in [70]. Note that the umbrella structure becomes co-planar at on the blue line. In experiments of the twisted AB stacked TMD homo-bilayer[50], one only has access to the bottom part where is small.
V Hubbard model on a honeycomb lattice when
The single orbital Hubbard model in the above is valid only when . When , the additional hole may want to enter another orbital in a different position inside the moiré unit cell to avoid the large Hubbard U due to the double occupancy[25]. To capture this effect, we need to keep at least another orbital from a higher band. As we have demonstrated that the inter-layer tunneling is quite small, in the following we ignore the Rashba SOC in Eq. (1). It is also convenient to shift the valence band maximum of the two layers to the point of the MBZ for each valley. This is equivalent to the gauge transformation we did to reach Eq. (3) in the previous section. Then we have two degenerate bands from the two layers at the same valley. In together we have four fold degeneracy for each moiré band coming from the combination of layer and valley.
In the hole picture, we use the two lowest bands (the blue and yellow bands in Figure 10(a)) to build a lattice model. It turns out that the Wannier orbitals[72] of the these two bands are on the two sublattices of a honeycomb lattice (see Appendix C for details). This leads to a four-flavor Hubbard model:
Figure 10. (a) The first four energy bands when and in AB stacking case. (b) Illustration of a honeycomb lattice. The red dots and the blue dots represent sublattice A and B respectively.
where is the creation operator of localized Wannier orbitals (see Appendix C). Here labels the layer and labels the valley. , , represents nearest neighbor, next nearest neighbor, next next nearest neighbor, respectively. represents the site on A,B sublattice, which is defined in Figure 10(b). The interaction parameters , , , are calculated via projecting the Coulomb repulsion onto the Wannier orbitals, the details are in Appendix C.
The values of the parameters are listed in Table I for a few twist angles. Because of the sublattice potential term, holes occupy the sublattice when , leading to a single orbital Hubbard model on the triangular lattice formed by B only, which reduces to the model used in the previous section. However, when , the additional holes prefer to enter if . This happens when the dielectric constant is smaller than a threshold. The boundary between the two regions is plot in Figure 11(a). In the following we focus on the region II. There is already experimental evidence that the region II is realized in the real experiment[50].
Figure 11. (a) The phase diagram of the four-flavor Hubbard model when . The two regions: (I) The additional holes still prefer to stay on sublattice B. (II) The additional holes go to sublattice A. (b) Dependence of and on the twist angle . Here, we use as an example.
| 2 | 19.15 | 5.26 | 0.788 | -0.764 | -0.587 | 479.22 | 825.44 | 405.27 | 640.10 | 276.13 | 259.27 |
| 3 | 14.25 | 12.58 | 2.926 | -1.328 | -1.945 | 694.20 | 1005.68 | 544.28 | 729.95 | 389.02 | 354.83 |
| 4 | 9.86 | 22.27 | 6.513 | -2.205 | -3.866 | 873.12 | 1174.04 | 639.25 | 798.28 | 489.45 | 431.29 |
| 5 | 6.18 | 34.36 | 11.38 | -3.466 | -6.216 | 1046.88 | 1342.77 | 717.40 | 856.04 | 583.76 | 496.37 |
Table I. Dependence of the kinetic and interaction parameters on different twist angle . The unit of is ° and that of energy is meV. We assume that the distance between the top and the bottom layer is 0.7nm.
Now we have a Mott insulator on a honeycomb lattice at the total filling . The resulting spin model in the large limit is:
where stands for nearest neighbor AB bond of the honeycomb lattice. , , . and are plotted as functions of the twist angle in Figure 11. We only keep nearest-neighbor hopping because is usually larger enough than , which can be seen from Table I. Strictly speaking there should still be an inter-layer hopping term similar to the term in Eq. (3). We will ignore it because it is very small as discussed in Sec. III.
If we set , the above model is a SU(4) Heisenberg model on honeycomb lattice. Previous works have suggested that the ground state is a Dirac spin liquid with flux ansatz[56, 57, 58] with Dirac fermions. Here 4 is from the four flavors and each flavor hosts two Dirac fermions. If is large enough, the ground state must have on one sublattice and on the other sublattice. It is interesting to study the phase transition between the Dirac spin liquid and this layer pseudo-spin density wave state through tuning . The monopole operator[59] in the Dirac spin liquid may also play an important role in the potential phase transitions. We leave to future work to carefully study the phase diagram and possible quantum criticalities in Eq. (7).
Conclusion
In this paper, we derive a lattice model for the twisted AB stacked TMD homo-bilayer through continuum model calculation and Wannier orbital construction. Without Rashba SOC, we have an approximate SU(4) Hubbard model. We consider the effect of the Rashba SOC and find that it generates a small inter-layer tunneling, meaning that the violation of the SU(4) symmetry due to inter-layer tunneling is small. At total filling , we derive an approximate SU(4) spin model which reduces to an XXZ model for the layer pseudospin with an additional sublattice-dependent transverse Ising field in a strong magnetic field. We study the phase diagram and find a 1/3 plateau in the layer polarization curve when increasing the displacement field . When , the two holes prefer to stay in the A, B sublattices of a honeycomb lattice in a certain regime. This results in an approximate SU(4) spin model on a honeycomb lattice, which may host a Dirac spin liquid ground state. The lattice models derived in this work offer a starting point to explore the various strongly correlated physics in twisted AB-stacked TMD homo-bilayer.
Acknowledgments
We thank Junyi Zhang for helpful discussions. This work is supported by a startup fund from the Johns Hopkins University. The numerical simulation was carried out at the Advanced Research Computing at Hopkins (ARCH) core facility (rockfish.jhu.edu), which is supported by the National Science Foundation (NSF) grant number OAC 1920103.
References
Construction of the tight-binding model
We want to build a two-layer tight-binding model of the lowest moiré band which is four-fold degenerate and consists of two valleys and two layers. As the two valleys are decoupled and are related by the time reversal symmetry, we only focus on the valley K. We start from the bands of Eq. (1) which can be obtained from numerical diagonalization. We choose the lowest two bands from the two layers at valley K:
where are band indices. If there is no Rashba SOC, these two bands are from the two layers and are decoupled. With a finite Rashba SOC, the band indices are not layer indices anymore. To build a two-layer model, we define a layer polarization operator and project it into the lowest two bands. We transform to a new basis where is diagonal at every momentum . Then in this new basis we reach a two-band Hamiltonian:
Note that are invariant under a change of phase , but are not. We fix the relative phase by enforcing that . is obtained by projecting into the lowest two bands, similar to . Figure 12 shows that our gauge is smooth and we only need to consider .

Figure 12. at as a function of lattice coordinates at twist angle . is clearly dominant.
Consider the term . We plug in the Wannier functions and get . Let , , where are two Bravais lattice vectors of the triangular lattice. The top-to-bottom-layer hopping term in the tight-binding model can now be written as . We only keep on-site and nearest-neighbour terms.
Spin-Wave Theory
We want to study the quantum phase diagram of Eq. (5) with respect to the effective magnetic field. It is well-known that on a triangular lattice, the XXZ model has a three-sublattice magnetic structure[68, 69, 70]. We assume that the basic magnetic structure is similar here. We choose the order parameter to be the average sublattice polarization which we can use spin wave theory[73, 74] to compute.
The main idea of spin wave theory is viewing the expectation of the pseudo-spin as a classical vector and using three bosonic fields a,b,c for the three sublattices of a triangular lattice to do a Holstein-Primakoff (H-P) transformation such that , , , , and is the magnitude of , where we have set , taken the large- limit and only kept the leading order term. This is called linear spin wave theory. In this approximation, the magnons are free. In the H-P transformation, we only keep terms of order and and quadratic in the a,b,c fields. We then use the Fourier series , to find the Hamiltonian in the momentum space. Note that is in the Brillouin zone of only one of the sublattices. In the end, we set as an approximation.
In each sublattice, the vector is in a different direction. Therefore, in Eq. (5) it is convenient to transform the vectors to their local frames.
Rotation of frame
Let the three sublattices be a,b,c. Firstly, we can rotate the x-y plane so that the direction of the magnetic field’s projection in the x-y plane becomes the new x’-axis:
The problem is now only in the x’-z plane. Then, we use spherical coordinates to parameterize the vectors.
For convenience, we remove all the primes of the vectors in the local frame in Eq. (10) below. Please keep in mind that all the axis labels below belong to the local frames. After the rotations, Eq. (5) becomes
where
, are defined similarly, . We have defined
We have discarded terms that do not contribute to the leading-order ground state fluctuations such as . We choose , , . Note that equation B1 has an a,b,c permutation symmetry, so our choice of a,b,c in this paper is without loss of generality.
Hamiltonian
After the rotations above, H-P transformation, and Fourier transform, we get
where is the classical Hamiltonian
are spherical coordinates and minimize (We use scipy.optimize to perform the optimization.),
, and
, , , , .
Note that and can be simplified with the angle sum and difference identities.
Sublattice polarization
Our goal is to compute the average sublattice polarization. The sublattice polarization of , for example, is given by the ground state expectation of , which can be obtained from the Bogoliubov transformation that diagonalizes M. We follow [75] to obtain the Bogoliubov transformation matrix U such that , where is a diagonal basis for M. Once we have U, the layer pseudo-spin reduction due to quantum fluctuation for sublattice l is
, if , if , and if [74]. Finally, the average sublattice layer polarization in the -direction is
Phase diagram
We explain how we plot Figure 9 here. First, we set to study the model’s behavior near the limit.
As Figure 13 shows, the transition between the Y-shape phase and the plateau happens near and the transition between the plateau and the umbrella phase happens near . The optimized , confirming the umbrella phase. Then around , the system is saturated. Then we use to extend the boundaries. All the boundaries are determined similarly with curves similar to the ones in Figure 9 and Figure 13. We find that the Y-shape phase disappears between and .

Figure 13. Layer polarization vs displacement field at different sublattice dependent transverse Ising field near phase boundaries. .
The layer polarization clearly jumps at first-order phase transitions but not at second-order ones.
C Construction of the honeycomb lattice model
In the main text, we constructed an effective lattice model to describe the first 2 energy bands (each is four fold degenerate coming from the spin and the layer). The appropriate Wannier orbitals for the construction are[47]:
where is an unitary matrix, is the creation operator of the band’s Bloch wave function. can be determined by projecting the original Bloch orbitals to well-localized wave functions[72]. We first calculate . Here, and are the Bloch state and the trial state respectively. Then, . As for the trial wave function, we choose it to be:
where and are determined by the center of the original Bloch state. If we consider the case with Rashba coupling and include first two bands, i.e., , the centers of Wannier orbitals form a honeycomb lattice as required by the symmetry. Their positions correspond to the center of the original Bloch states, which can be obtained by calculating the eigenvalue of the operator. We observed that the eigenvalues of the first band are at . For the second band, they are respectively. This constrains the Wannier orbitals of the two bands to be in different positions within a moiré unit cell. We find that the kinetic and the interaction parameters in Eq. (6) can be calculated by projecting the energy band and the Coulomb interaction onto the new basis. We take and as an example, suppose that the wannier function localized at in sublattice is , then and are:
where is the dispersion relation of the band. In reality we first express the interaction in the momentum space with form factors and then do the Fourier transformation, following the same procedure as in Ref. 47.
References
- [1]Y. Cao, V. Fatemi, A. Demir, S. Fang, S. L. Tomarken, J. Y. Luo, J. D. Sanchez-Yamagishi, K. Watanabe, T. Taniguchi, E. Kaxiras, et al., Correlated insulator behaviour at half-filling in magic-angle graphene superlattices, Nature 556, 80 (2018).DOI
- [2]Y. Cao, V. Fatemi, S. Fang, K. Watanabe, T. Taniguchi, E. Kaxiras, and P. Jarillo-Herrero, Unconventional superconductivity in magic-angle graphene superlattices, Nature 556, 43 (2018).DOI
- [3]G. Chen, A. L. Sharpe, P. Gallagher, I. T. Rosen, E. Fox, L. Jiang, B. Lyu, H. Li, K. Watanabe, T. Taniguchi, et al., Signatures of gate-tunable superconductivity in trilayer graphene/boron nitride moiré superlattice, arXiv e-prints , arXiv (2019).DOI
- [4]G. Chen, L. Jiang, S. Wu, B. Lyu, H. Li, B. L. Chittari, K. Watanabe, T. Taniguchi, Z. Shi, J. Jung, et al., Evidence of a gate-tunable Mott insulator in a trilayer graphene moiré superlattice, Nature Physics 15, 237 (2019).DOI
- [5]M. Yankowitz, S. Chen, H. Polshyn, Y. Zhang, K. Watanabe, T. Taniguchi, D. Graf, A. F. Young, and C. R. Dean, Tuning superconductivity in twisted bilayer graphene, Science 363, 1059 (2019).
- [6]G. Chen, A. L. Sharpe, E. J. Fox, Y.-H. Zhang, S. Wang, L. Jiang, B. Lyu, H. Li, K. Watanabe, T. Taniguchi, et al., Tunable correlated chern insulator and ferromagnetism in a moiré superlattice, Nature 579, 56 (2020).DOI
- [7]X. Liu, P. Stepanov, W. Yang, M. Xie, M. A. Aamir, I. Das, C. Urgell, K. Watanabe, T. Taniguchi, G. Zhang, et al., Superconductors, orbital magnets and correlated states in magic-angle bilayer graphene, Nature 574, 653 (2019).
- [8]Y. Cao, D. Rodan-Legrain, O. Rubies-Bigorda, J. M. Park, K. Watanabe, T. Taniguchi, and P. Jarillo-Herrero, Tunable correlated states and spin-polarized phases in twisted bilayer–bilayer graphene, Nature 583, 215 (2020).DOI
- [9]X. Liu, Z. Hao, E. Khalaf, J. Y. Lee, Y. Ronen, H. Yoo, D. H. Najafabadi, K. Watanabe, T. Taniguchi, A. Vishwanath, et al., Tunable spin-polarized correlated states in twisted double bilayer graphene, Nature 583, 221 (2020).
- [10]C. Shen, Y. Chu, Q. Wu, N. Li, S. Wang, Y. Zhao, J. Tang, J. Liu, J. Tian, K. Watanabe, et al., Correlated states in twisted double bilayer graphene, Nature Physics , 1 (2020).
- [11]H. Polshyn, J. Zhu, M. A. Kumar, Y. Zhang, F. Yang, C. L. Tschirhart, M. Serlin, K. Watanabe, T. Taniguchi, A. H. MacDonald, et al., Electrical switching of magnetic order in an orbital Chern insulator, Nature 588, 66 (2020).DOI
- [12]S. Chen, M. He, Y.-H. Zhang, V. Hsieh, Z. Fei, K. Watanabe, T. Taniguchi, D. H. Cobden, X. Xu, C. R. Dean, et al., Electrically tunable correlated and topological states in twisted monolayer–bilayer graphene, Nature Physics 17, 374 (2021).
- [13]A. L. Sharpe, E. J. Fox, A. W. Barnard, J. Finney, K. Watanabe, T. Taniguchi, M. Kastner, and D. Goldhaber-Gordon, Emergent ferromagnetism near three-quarters filling in twisted bilayer graphene, Science 365, 605 (2019).DOI
- [14]M. Serlin, C. Tschirhart, H. Polshyn, Y. Zhang, J. Zhu, K. Watanabe, T. Taniguchi, L. Balents, and A. Young, Intrinsic quantized anomalous hall effect in a moiré heterostructure, Science 367, 900 (2020).
- [15]Y. Tang, L. Li, T. Li, Y. Xu, S. Liu, K. Barmak, K. Watanabe, T. Taniguchi, A. H. MacDonald, J. Shan, et al., Simulation of hubbard model physics in wse 2/ws 2 moiré superlattices, Nature 579, 353 (2020).DOI
- [16]E. C. Regan, D. Wang, C. Jin, M. I. B. Utama, B. Gao, X. Wei, S. Zhao, W. Zhao, Z. Zhang, K. Yumigeta, et al., Mott and generalized wigner crystal states in wse 2/ws 2 moiré superlattice, Nature 579, 359 (2020).
- [17]L. Wang, E.-M. Shih, A. Ghiotto, L. Xian, D. A. Rhodes, C. Tan, M. Claassen, D. M. Kennes, Y. Bai, B. Kim, et al., Correlated electronic phases in twisted bilayer transition metal dichalcogenides, Nature materials 19, 861 (2020).DOI
- [18]Y. Shimazaki, I. Schwartz, K. Watanabe, T. Taniguchi, M. Kroner, and A. Imamoğlu, Strongly correlated electrons and hybrid excitons in a moiré heterostructure, Nature 580, 472 (2020).
- [19]J. Gu, L. Ma, S. Liu, K. Watanabe, T. Taniguchi, J. C. Hone, J. Shan, and K. F. Mak, Dipolar excitonic insulator in a moire lattice, Nature physics 19, 395–400 (2022).
- [20]Z. Zhang, E. C. Regan, D. Wang, W. Zhao, S. Wang, M. Sayyad, K. Yumigeta, K. Watanabe, T. Taniguchi, S. Tongay, et al., Correlated interlayer exciton insulator in double layers of monolayer wse2 and moir’e wse2/wse2, Nature physics 18, 1214–1220 (2022).DOI
- [21]T. Li, S. Jiang, L. Li, Y. Zhang, K. Kang, J. Zhu, K. Watanabe, T. Taniguchi, D. Chowdhury, L. Fu, et al., Continuous mott transition in semiconductor moiré superlattices, Nature 597, 350 (2021).DOI
- [22]A. Ghiotto, E.-M. Shih, G. S. Pereira, D. A. Rhodes, B. Kim, J. Zang, A. J. Millis, K. Watanabe, T. Taniguchi, J. C. Hone, et al., Quantum criticality in twisted transition metal dichalcogenides, Nature 597, 345 (2021).
- [23]F. Wu, T. Lovorn, E. Tutuc, and A. H. MacDonald, Hubbard model physics in transition metal dichalcogenide moiré bands, Physical review letters 121, 026402 (2018).DOI
- [24]F. Wu, T. Lovorn, E. Tutuc, I. Martin, and A. H. Macdonald, Topological insulators in twisted transition metal dichalcogenide homobilayers, Physical Review Letters 122, 10.1103/PhysRevLett.122.086402 (2019).
- [25]Y. Zhang, N. F. Yuan, and L. Fu, Moiré quantum chemistry: charge transfer in transition metal dichalcogenide superlattices, Physical Review B 102, 201115 (2020).DOI
- [26]H. Pan, F. Wu, and S. D. Sarma, Band topology, hubbard model, heisenberg model, and dzialoshinskii-moriya interaction in twisted bilayer wse 2, Physical Review Research 2, 033087 (2020).DOI
- [27]H. Pan, F. Wu, and S. D. Sarma, Quantum phase diagram of a moiré-hubbard model, Physical Review B 102, 201104 (2020).DOI
- [28]H. Pan and S. D. Sarma, Interaction-driven filling-induced metal-insulator transitions in 2d moiré lattices, Physical Review Letters 127, 096802 (2021).DOI
- [29]H. Pan and S. D. Sarma, Interaction range and temperature dependence of symmetry breaking in strongly correlated two-dimensional moiré transition metal dichalcogenide bilayers, Physical Review B 105, 041109 (2022).DOI
- [30]J. Zang, J. Wang, J. Cano, and A. J. Millis, Hartree-fock study of the moiré hubbard model for twisted bilayer transition metal dichalcogenides, Physical Review B 104, 075150 (2021).DOI
- [31]J. Zang, J. Wang, J. Cano, A. Georges, and A. J. Millis, Dynamical mean-field theory of moiré bilayer transition metal dichalcogenides: phase diagram, resistivity, and quantum criticality, Physical Review X 12, 021064 (2022).DOI
- [32]R. Kaneko, S. Morita, and M. Imada, Gapless spin-liquid phase in an extended spin 1/2 triangular heisenberg model, Journal of the Physical Society of Japan 83, 093707 (2014).DOI
- [33]Z. Zhu and S. R. White, Spin liquid phase of the s= 1 2 j 1- j 2 heisenberg model on the triangular lattice, Physical Review B 92, 041105 (2015).
- [34]W.-J. Hu, S.-S. Gong, W. Zhu, and D. Sheng, Competing spin-liquid states in the spin-1 2 heisenberg model on the triangular lattice, Physical Review B 92, 140403 (2015).
- [35]S. Saadatmand and I. McCulloch, Symmetry fractionalization in the topological phase of the spin-1 2 j 1- j 2 triangular heisenberg model, Physical Review B 94, 121111 (2016).DOI
- [36]Y. Iqbal, W.-J. Hu, R. Thomale, D. Poilblanc, and F. Becca, Spin liquid nature in the heisenberg j 1- j 2 triangular antiferromagnet, Physical Review B 93, 144411 (2016).
- [37]A. Wietek and A. M. Läuchli, Chiral spin liquid and quantum criticality in extended s= 1 2 heisenberg models on the triangular lattice, Physical Review B 95, 035141 (2017).
- [38]K. Misumi, T. Kaneko, and Y. Ohta, Mott transition and magnetism of the triangular-lattice hubbard model with next-nearest-neighbor hopping, Physical Review B 95, 075124 (2017).DOI
- [39]S.-S. Gong, W. Zhu, J.-X. Zhu, D. N. Sheng, and K. Yang, Global phase diagram and quantum spin liquids in a spin-1 2 triangular antiferromagnet, Physical Review B 96, 075116 (2017).
- [40]Y.-F. Jiang and H.-C. Jiang, Nature of quantum spin liquids of the s= 1/2 heisenberg antiferromagnet on the triangular lattice: A parallel dmrg study, arXiv preprint arXiv:2203.10216 (2022).DOI
- [41]O. I. Motrunich, Variational study of triangular lattice spin-1/2 model with ring exchanges and spin liquid state in κ-(et) 2 cu 2 (cn) 3, Physical Review B 72, 045105 (2005).
- [42]A. Szasz, J. Motruk, M. P. Zaletel, and J. E. Moore, Chiral spin liquid phase of the triangular lattice hubbard model: a density matrix renormalization group study, Physical Review X 10, 021042 (2020).DOI
- [43]M. Hermele, V. Gurarie, and A. M. Rey, Mott insulators of ultracold fermionic alkaline earth atoms: Unconstrained magnetism and chiral spin liquid, Physical Review Letters 103, 135301 (2009).
- [44]M. Hermele and V. Gurarie, Topological liquids and valence cluster states in two-dimensional su (n) magnets, Physical Review B 84, 174441 (2011).DOI
- [45]Y.-H. Zhang and D. Mao, Spin liquids and pseudogap metals in the su (4) hubbard model in a moiré superlattice, Physical Review B 101, 035122 (2020).DOI
- [46]Y.-Z. Chou and S. D. Sarma, Kondo lattice model in magic-angle twisted bilayer graphene, arXiv preprint arXiv:2211.15682 (2022).DOI
- [47]Y.-H. Zhang and T. Senthil, Bridging hubbard model physics and quantum hall physics in trilayer graphene/h- bn moiré superlattice, Physical Review B 99, 205150 (2019).DOI
- [48]Y. H. Zhang, D. N. Sheng, and A. Vishwanath, Su(4) chiral spin liquid, exciton supersolid, and electric detection in moiré bilayers, Physical Review Letters 127, 10.1103/PhysRevLett.127.247701 (2021).
- [49]Y.-H. Zhang, Doping a mott insulator with excitons in a moiré bilayer: Fractional superfluid, neutral fermi surface, and mott transition, Physical Review B 106, 10.1103/physrevb.106.195120 (2022).DOI
- [50]Y. Xu, K. Kang, K. Watanabe, T. Taniguchi, K. F. Mak, and J. Shan, A tunable bilayer hubbard model in twisted wse2, Nature Nanotechnology 17, 934 (2022).DOI
- [51]Y.-H. Zhang, D. Sheng, and A. Vishwanath, Su (4) chiral spin liquid, exciton supersolid, and electric detection in moiré bilayers, Physical review letters 127, 247701 (2021).DOI
- [52]T. Li, S. Jiang, B. Shen, Y. Zhang, L. Li, Z. Tao, T. Devakul, K. Watanabe, T. Taniguchi, L. Fu, et al., Quantum anomalous hall effect from intertwined moiré bands, Nature 600, 641 (2021).DOI
- [53]Y. Zhang, T. Devakul, and L. Fu, Spin-textured chern bands in ab-stacked transition metal dichalcogenide bilayers, Proceedings of the National Academy of Sciences 118, e2112673118 (2021).DOI
- [54]A. Manchon, H. C. Koo, J. Nitta, S. M. Frolov, and R. A. Duine, New perspectives for rashba spin-orbit coupling, Nature Materials 14, 871 (2015).DOI
- [55]Y. Zeng, N. Wei, and A. H. MacDonald, Layer pseudospin magnetism in a transition metal dichalcogenide double-moiré system, Physical Review B 106, 165105 (2022).DOI
- [56]P. Corboz, M. Lajkó, A. M. Läuchli, K. Penc, and F. Mila, Spin-orbital quantum liquid on the honeycomb lattice, Physical Review X 2, 041013 (2012).
- [57]X.-P. Yao, R. L. Luo, and G. Chen, Intertwining su (n) symmetry and frustration on a honeycomb lattice, Physical Review B 105, 024401 (2022).DOI
- [58]H.-K. Jin, W. Natori, and J. Knolle, Twisting the dirac cones of the su (4) spin-orbital liquid on the honeycomb lattice, arXiv preprint arXiv:2207.02239 (2022).DOI
- [59]V. Calvera and C. Wang, Theory of dirac spin-orbital liquids: monopoles, anomalies, and applications to su(4) honeycomb models, arXiv preprint arXiv:2103.13405 (2021).arxiv.org/abs/2103.13405
- [60]X. Zhai, Layered opposite rashba spin-orbit coupling in bilayer graphene: Loss of spin chirality, symmetry breaking, and topological transition, Physical Review B 105, 10.1103/PhysRevB.105.205429 (2022).DOI
- [61]B. T. Zhou, K. Taguchi, Y. Kawaguchi, Y. Tanaka, and K. T. Law, Spin-orbit coupling induced valley hall effects in transition-metal dichalcogenides, Communications Physics 2, 10.1038/s42005-019-0127-7 (2019).DOI
- [62]A. Rezavand and N. Ghobadi, Stacking-dependent rashba spin-splitting in janus bilayer transition metal dichalcogenides: The role of in-plane strain and out-of-plane electric field, Physica E: Low-Dimensional Systems and Nanostructures 132, 10.1016/j.physe.2021.114768 (2021).DOI
- [63]B. Fallahazad, H. C. Movva, K. Kim, S. Larentis, T. Taniguchi, K. Watanabe, S. K. Banerjee, and E. Tutuc, Shubnikov-de haas oscillations of high-mobility holes in monolayer and bilayer wse2: Landau level degeneracy, effective mass, and negative compressibility, Physical Review Letters 116, 10.1103/PhysRevLett.116.086601 (2016).DOI
- [64]Z. Bi and L. Fu, Excitonic density wave and spin-valley superfluid in bilayer transition metal dichalcogenide, Nature Communications 12, 10.1038/s41467-020-20802-z (2021).
- [65]D. Xiao, G. B. Liu, W. Feng, X. Xu, and W. Yao, Coupled spin and valley physics in monolayers of mos 2 and other group-vi dichalcogenides, Physical Review Letters 108, 10.1103/PhysRevLett.108.196802 (2012).
- [66]F. Wu, T. Lovorn, E. Tutuc, and A. H. MacDonald, Hubbard model physics in transition metal dichalcogenide moiré bands, Physical Review Letters 121, 10.1103/PhysRevLett.121.026402 (2018).DOI
- [67]A. MacDonald, S. Girvin, and D. Yoshioka, t/u expansion for the hubbard model, Physical Review B 37 (1988).
- [68]S. Miyashita, magnetic properties of heisenberg antiferromagnet on a triangular lattice, Journal of the Physical Society of Japan 55, 3605 (1986).
- [69]D. Sellmann, X. F. Zhang, and S. Eggert, Phase diagram of the antiferromagnetic xxz model on the triangular lattice, Physical Review B 91, 10.1103/PhysRevB.91.081104 (2015).
- [70]D. Yamamoto, G. Marmorini, and I. Danshita, Quantum phase diagram of the triangular-lattice xxz model in a magnetic field, Physical Review Letters 112, 10.1103/PhysRevLett.112.127203 (2013).
- [71]D. Yamamoto, G. Marmorini, M. Tabata, K. Sakakura, and I. Danshita, Magnetism driven by the interplay of fluctuations and frustration in the easy-axis triangular xxz model with transverse fields, Physical Review B 100, 10.1103/PhysRevB.100.140410 (2019).DOI
- [72]N. Marzari, A. A. Mostofi, J. R. Yates, I. Souza, and D. Vanderbilt, Maximally localized wannier functions: Theory and applications, Reviews of Modern Physics 84, 1419 (2012).
- [73]A. V. Chubukov, S. Sachdev, and T. Senthil, Large-s expansion for quantum antiferromagnets on a triangular lattice, Journal of Physics: Condensed Matter 6, 8891 (1994).DOI
- [74]J. Y. Gan, F. C. Zhang, and Z. B. Su, Spin wave theory for antiferromagnetic xxz spin model on a triangle lattice in the presence of an external magnetic field, Physical Review B 67, 10.1103/PhysRevB.67.144427 (2003).DOI
- [75]J. Colpa, Diagonalization of the quadratic boson hamiltonian, Physica , 327 (1978).
