Three-dimensional nematic spin liquid in a stacked triangular lattice 6H-B structure

Abstract

Recently, a number of experiments indicate the possible presence of spin-liquid phases in quantum magnets with spin-$\frac{1}{2}$ and spin-1 moments sitting on triangular-lattice-based structures in Ba${}_{3}$CuSb${}_{2}$O${}_{9}$ and Ba${}_{3}$NiSb${}_{2}$O${}_{9}$, respectively. In relation to these experiments, several theoretical proposals have been made for spin-liquid phases and spin-liquid-like behaviors on the stacked triangular lattice. While the crystal structures of these materials are currently under debate, it is nonetheless interesting to understand possible spin-liquid phases on such frustrated lattices. In this work, we apply Schwinger boson mean-field theory and projective symmetry group analysis to investigate spin-liquid phases on the fully three-dimensional 6$H$-B structure, in contrast to previous works that considered two-dimensional systems. We find that a nematic ${Z}_{2}$ spin-liquid phase, where the lattice-rotational symmetry is spontaneously broken, is the most promising spin-liquid phase that is consistent with spiral magnetic ordering in the classical limit. We discuss the implications of our results to future theoretical and experimental works.