Mott insulating phases and quantum phase transitions of interacting spin-32fermionic cold atoms in optical lattices at half filling

Abstract

We study various Mott insulating phases of interacting spin-$\frac{3}{2}$ fermionic ultracold atoms in two-dimensional square optical lattices at half filling. Using a generalized one-band Hubbard model with hidden SO(5) symmetry, we identify two distinct symmetry breaking phases: the degenerate antiferromagnetic spin-dipole--spin-octupole ordering and spin-quadrupole ordering, depending on the sign of the spin-dependent interaction. These two competing orders exhibit very different symmetry properties, low energy excitations, and topological characterizations. Near the SU(4) symmetric point, a quantum critical state with a $\ensuremath{\pi}$-flux phase may emerge due to strong quantum fluctuations, leading to spin algebraic correlations and gapless excitations.