Competing spin-liquid states in the spin-12Heisenberg model on the triangular lattice

Abstract

We study the spin-$1/2$ Heisenberg model on the triangular lattice with the antiferromagnetic first ($J_1$) and second ($J_2$) nearest-neighbor interactions using density matrix renormalization group. By studying the spin correlation function, we find a $120^{\circ}$ magnetic order phase for $J_2 \lesssim 0.07 J_1$ and a stripe antiferromagnetic phase for $J_2 \gtrsim 0.15 J_1$. Between these two phases, we identify a spin liquid region characterized by the exponential decaying spin and dimer correlations, as well as the large spin singlet and triplet excitation gaps on finite-size systems. We find two near degenerating ground states with distinct properties in two sectors, which indicates more than one spin liquid candidates in this region. While the sector with spinon is found to respect the time reversal symmetry, the even sector without a spinon breaks such a symmetry for finite-size systems. Furthermore, we detect the signature of the fractionalization by following the evolution of different ground states with inserting spin flux into the cylinder system. Moreover, by tuning the anisotropic bond coupling, we explore the nature of the spin liquid phase and find the optimal parameter region for the gapped $Z_2$ spin liquid.