In this paper, a sliding-mode disturbance observer (SMDO)-based finite-time control scheme is proposed for a quadrotor under uncertainties and external disturbances. The SMDO is used to estimate and compensate the unknown time-varying disturbance with minor chattering and rapid convergence. However, a priori bounded uncertainty may impose a priori constraint on the system state before obtaining closed-loop stability. To address this issue, an adaptive SMDO (ASMDO) with a nested adaptive structure is further introduced. As for controller design, a fast nonsingular integral terminal sliding mode controller (FNITSMC) is proposed to ensure finite-time convergence and zero tracking error for the quadrotor attitude control. Finally, effectiveness of proposed method is verified through simulations and experiments.