Reentrant metal-insulator transition and competing magnetic interactions on a triangular lattice with second nearest-neighbor hopping

Abstract

The $120^{\circ}$ antiferromagnetism (AFM) is widely believed as the magnetic ground state of the triangular systems because of the geometrical frustration. The emergence of novel magnetism, such as the row-wise AFM in Mn/Cu(111) and Sn/Si(111), reveals the importance of the longer-range hopping on magnetic competitions in realistic material systems. By utilizing advanced many-body techniques, we systematically studied the isotropic triangular Hubbard model with second nearest-neighbor hopping $t^{\prime}$, including both the single- and the two-particle responses. We found that both electronic and magnetic phase transitions show a clear dependence on $t^{\prime}/t$. Consequently, we observed a remarkable reentrance of the metal-insulator transition and a crossover between the $120^{\circ}$- and the row-wise AFM. The Fermi surface (FS) shows two distinct structures with the nesting vectors consistent with the magnetic correlations. When $t^{\prime}$ evolves from 0 to 1, the correlated Fermi surface demonstrates a Lifschitz transition between the two nesting structures, and exotic phases like the featureless insulating state can be realized. Our work sheds light on the engineering of electronic and magnetic correlations of correlated triangular surfaces via longer-range hopping. The rich phase diagram and the high degree of tunability make the triangular lattice with longer-range hopping a more realistic platform to study the emergent magnetic competitions.