Nesting instability of gapless U(1) spin liquids with spinon Fermi pockets in two dimensions

Abstract

Quantum spin liquids are exotic states of matter that may be realized in frustrated quantum magnets and feature fractionalized excitations and emergent gauge fields. Here, we consider a gapless U(1) spin liquid with spinon Fermi pockets in two spatial dimensions. Such a state appears to be the most promising candidate to describe the exotic field-induced behavior observed in numerical simulations of the antiferromagnetic Kitaev honeycomb model. A similar such state may also be responsible for the recently-reported quantum oscillations of the thermal conductivity in the field-induced quantum paramagnetic phase of $\alpha$-RuCl$_3$. We consider the regime close to a Lifshitz transition, at which the spinon Fermi pockets shrink to small circles around high-symmetry points in the Brillouin zone. By employing renormalization group and mean-field arguments, we demonstrate that interactions lead to a gap opening in the spinon spectrum at low temperatures, which can be understood as a nesting instability of the spinon Fermi surface. This leads to proliferation of monopole operators of the emergent U(1) gauge field and confinement of spinons. While signatures of fractionalization may be observable at finite temperatures, the gapless U(1) spin liquid state with nested spinon Fermi pockets is ultimately unstable at low temperatures towards a conventional long-range-ordered ground state, such as a valence bond solid. Implications for Kitaev materials in external magnetic fields are discussed.