This article proposes a simple robust adaptive control architecture with minimum parameter estimation laws for trajectory tracking of a quadrotor subjected to parametric variations and environmental disturbances. This simple control architecture aims to achieve highly accurate, robust, and fast trajectory tracking of the quadrotor in a short time with low computational cost. Firstly, the quadrotor model is divided into altitude, attitude, and position subsystems for which appropriate control methods are designed without prior knowledge of the upper bound of external disturbances. A simple adaptive fractional-order sliding mode control (AFSMC) is designed to enhance the tracking of the altitude subsystem and estimate the upper bound of the disturbances. Then, a simple adaptive backstepping control (ABC) is developed for the horizontal position to generate the required roll and pitch orientations. The adaptation laws not only estimate the upper bound of the disturbances but also adjust the controller gains thereby enhancing the robustness of the ABC. A nonsingular fast terminal sliding mode control (NFTSMC) is incorporated with a finite-time disturbance observer (FDO) to accurately suppress the disturbances, and follow the target rotation angles within a short finite-time. Simulation results showed that the compounded control structure ensures accurate, fast, and robust tracking. The AFSMC can achieve the desired altitude with a settling time of 0.31 s and root mean square error (RMSE) of 0.0454 m. The ABC can attain the target horizontal position coordinates with a settling time of (0.47 s, 0.47 s) and RMSE of (0.0370 m, 0.0518 s). The NFTSMC-FDO can achieve the desired attitude angles with a settling time of (0.11 s, 0.13 s, 0.52 s) and RMSE of (0.0098 rad, 0.0092 rad, 0.0935 rad). Performance comparisons with existing control methods in terms of settling time and RMSE demonstrated that the proposed control architecture is superior.