Magnetic ordering phenomena of interacting quantum spin Hall models

Abstract

The two-dimensional Hubbard model defined for topological band structures exhibiting a quantum spin Hall effect poses fundamental challenges in terms of phenomenological characterization and microscopic classification. In the limit of infinite coupling $U$ at half filling, the spin model Hamiltonians resulting from a strong-coupling expansion show various forms of magnetic ordering phenomena depending on the underlying spin-orbit coupling terms. We investigate the infinite-$U$ limit of the Kane-Mele-Hubbard model with $z$-axis intrinsic spin-orbit coupling as well as its generalization to a generically multidirectional spin-orbit term which has been claimed to account for the physical scenario in monolayer ${\text{Na}}_{2}{\text{IrO}}_{3}$. We find that the axial spin symmetry which is kept in the former but broken in the latter has a fundamental impact on the magnetic phase diagram as we vary the spin-orbit coupling strength. While the Kane-Mele spin model shows a continuous evolution from conventional honeycomb N\'eel to $XY$ antiferromagnetism which avoids the frustration imposed by the increased spin-orbit coupling, the multidirectional spin-orbit term induces a commensurate to incommensurate transition at intermediate coupling strength, and yields a complex spiral state with a 24 site unit cell in the limit of infinite spin-orbit coupling. From our findings, we conjecture that in the case of broken axial spin symmetry there is a large propensity for an additional phase at sufficiently large spin-orbit coupling and intermediate $U$.