Nematic Excitonic Insulator in Transition Metal Dichalcogenide Moiré Heterobilayers.

We study the effect of interelectron Coulomb interactions on the displacement field induced topological phase transition in transition metal dichalcogenide moiré heterobilayers. We find a nematic excitonic insulator phase that breaks the moiré superlattice's threefold rotational symmetry and preempts the topological phase transition in both AA and AB stacked heterobilayers when the interlayer tunneling is weak, or when the Coulomb interaction is not strongly screened. The nematicity originates from the frustration between the nontrivial spatial structure of the interlayer tunneling, which is crucial to the existence of the topological Chern band, and the interlayer coherence induced by the Coulomb interaction that favors uniformity in layer pseudospin orientations. We construct a unified effective two-band model that captures the physics near the band inversion and applies to both AA and AB stacked heterobilayers. Within the two-band model the competition between the nematic excitonic insulator phase and the Chern insulator phase can be understood as the switching of the energetic order between the s-wave and the p-wave excitons upon increasing the interlayer tunneling.