This article investigates the distributed attitude consensus tracking control for spacecraft formation flying with unknown external disturbances and model uncertainties. First, a terminal sliding mode disturbance observer (TSMDO) is constructed to estimate the generalized disturbances including external disturbances and model uncertainties. The finite-time convergence of the estimation errors using TSMDO is analyzed. Second, a variable structure control law is developed to avoid introducing initial errors of the TSMDO. Third, a novel adaptive nonsingular fast terminal sliding mode (ANFTSM) control law based on TSMDO is proposed to ensure the convergence of attitude tracking errors to zero. Based on theoretical analysis, the finite-time stability can be guaranteed by Lyapunov theory. Finally, the effectiveness of the developed control law is verified via numerical simulations.