Order-by-disorder and spin-orbital liquids in a distorted Heisenberg-Kitaev model

Abstract

The microscopic modeling of spin-orbit entangled $j=1/2$ Mott insulators such as the layered hexagonal Iridates Na$_2$IrO$_3$ and Li$_2$IrO$_3$ has spurred an interest in the physics of Heisenberg-Kitaev models. Here we explore the effect of lattice distortions on the formation of the collective spin-orbital states which include not only conventionally ordered phases but also gapped and gapless spin-orbital liquids. In particular, we demonstrate that in the presence of spatial anisotropies of the exchange couplings conventionally ordered states are formed through an order-by-disorder selection which is not only sensitive to the type of exchange anisotropy but also to the relative strength of the Heisenberg and Kitaev couplings. The spin-orbital liquid phases of the Kitaev limit -- a gapless phase in the vicinity of spatially isotropic couplings and a gapped Z$_2$ phase for a dominant spatial anisotropy of the exchange couplings -- show vastly different sensitivities to the inclusion of a Heisenberg exchange. While the gapless phase is remarkably stable, the gapped Z$_2$ phase quickly breaks down in what might be a rather unconventional phase transition driven by the simultaneous condensation of its elementary excitations.