Unconventional pairing and electronic dimerization instabilities in the doped Kitaev-Heisenberg model

Abstract

We study the quantum many-body instabilities of the t−JK−JH Kitaev-Heisenberg Hamiltonian on the honeycomb lattice as a minimal model for a doped spin-orbit Mott insulator. This spin-1/2 model is believed to describe the magnetic properties of the layered transition-metal oxide Na2IrO3. We determine the ground state of the system with finite charge-carrier density from the functional renormalization group (fRG) for correlated fermionic systems. To this end, we derive fRG flow equations adapted to the lack of full spin-rotational invariance in the fermionic interactions, here represented by the highly frustrated and anisotropic Kitaev exchange term. Additionally employing a set of the Ward identities for the Kitaev-Heisenberg model, the numerical solution of the flow equations suggests a rich phase diagram emerging upon doping charge carriers into the ground-state manifold (Z2 quantum spin liquids and magnetically ordered phases). We corroborate superconducting triplet p-wave instabilities driven by ferromagnetic exchange and various singlet pairing phases. For filling δ>1/4, the p-wave pairing gives rise to a topological state with protected Majorana edge modes. For antiferromagnetic Kitaev and ferromagnetic Heisenberg exchanges, we obtain bond-order instabilities at van Hove filling supported by nesting and density-of-states enhancement, yielding dimerization patterns of the electronic degrees of freedom on the honeycomb lattice. Further, our flow equations are applicable to a wider class of model Hamiltonians.11 MoreReceived 1 May 2014Revised 30 June 2014DOI:https://doi.org/10.1103/PhysRevB.90.045135©2014 American Physical Society