Theory of the spin-nematic to spin-Peierls quantum phase transition in ultracold spin-1 atoms in optical lattices

Abstract

We present a theory of the anisotropy-tuned quantum phase transition between spin-nematic and spin-Peierls phases in $S=1$ systems with significant biquadratic exchange interactions. Based on quantum Monte Carlo studies on finite-size systems, it has been proposed that this phase transition is second order with new deconfined fractional excitations that are absent in either of the two phases. The possibility of a weak first-order transition, however, cannot be ruled out. To elucidate the nature of the transition, we construct a large-$N$ $\text{SO}(3N)$ model for this phase transition and find in the $N\ensuremath{\rightarrow}\ensuremath{\infty}$ limit that the transition is generically of first-order. Furthermore, we find a critical point in the one-dimensional (1D) limit, where two transition lines, separating spin-nematic, ferromagnetic, and spin-Peierls phases, meet. Our study indicates that the spin-nematic phase is absent in 1D, while its correlation length diverges at the critical point. Predictions for $^{23}\text{N}\text{a}$ atoms trapped in an optical lattice, where the nematic to spin-Peierls quantum phase transition naturally arises are discussed.