Phase diagram of a bilinear-biquadratic spin-1 model on the triangular lattice from density matrix renormalization group simulations

Abstract

We investigate a highly frustrated spin-1 model on the triangular lattice, with nearest- and next-nearest-neighbor antiferromagnetic $S\ifmmode\cdot\else\textperiodcentered\fi{}S$ interactions and nearest-neighbor ${(S\ifmmode\cdot\else\textperiodcentered\fi{}S)}^{2}$ interactions. Using the density matrix renormalization group technique, we find three magnetically ordered phases, namely ${120}^{\ensuremath{\circ}}$ spiral order, stripe order, and tetrahedral order, as well as two spin nematic phases: ferroquadrupolar and antiferroquadrupolar. While our data could be consistent with a spin liquid phase between the ${120}^{\ensuremath{\circ}}$ spiral and antiferroquadrupolar orders, the more likely scenario is a direct continuous transition between these two orders.