Chiral spin density wave order on the frustrated honeycomb and bilayer triangle lattice hubbard model at half-filling.

Abstract

We study the Hubbard model on the frustrated honeycomb lattice with nearest-neighbor hopping t_{1} and second nearest-neighbor hopping t_{2}, which is isomorphic to the bilayer triangle lattice, using the SU(2)-invariant slave boson theory. We show that the Coulomb interaction U induces antiferromagnetic (AF) chiral spin density wave (χSDW) order in a wide range of κ=t_{2}/t_{1} where both the two-sublattice AF order at small κ and the decoupled three-sublattice 120° order at large κ are strongly frustrated, leading to three distinct phases with different anomalous Hall responses. We find a continuous transition from a χSDW semimetal with the anomalous Hall effect to a topological chiral Chern insulator exhibiting the quantum anomalous Hall effect, followed by a discontinuous transition to a χSDW insulator with a zero total Chern number but an anomalous ac Hall effect. The χSDW is likely a generic phase of strongly correlated and highly frustrated hexagonal lattice electrons.