In this paper, a novel control scheme, which includes an improved fast finite-time adaptive backstepping controller based on a newly designed adaptive smooth disturbance observer, is presented for the attitude tracking of a small unmanned helicopter system subject to lumped disturbances. First, an adaptive smooth disturbance observer (ASDO) is proposed to approximate the lumped disturbances, which owns the characteristics of smooth output, fast finite-time convergence, and adaptability to disturbances. Then, a finite-time backstepping control protocol is constructed to achieve the attitude tracking objective. By virtue of our newly proposed inequality, a singularity-free virtual control law is designed. To tackle the “explosion of complexity” problem, a fast finite-time command filter (FFTCF) is utilized to estimate the virtual control signals and their derivatives. In addition, an auxiliary dynamic system is introduced to attenuate the impact of the errors caused by FFTCF estimation. Moreover, an adaptive law with σ-modification term is developed to compensate the ASDO approximation error. Theoretical analysis proves that all signals of the closed-loop system are fast finite-time bounded, while the attitude tracking errors fast converge to a small region of the origin in finite time. Finally, two contrastive numerical simulations are carried out to validate the effectiveness and superiority of the designed control scheme.