With the widespread application of unmanned aerial vehicles, harsh environments require higher control performance. Numerous existing works offer solutions for handling external disturbances during flight; however, the timeliness of the disturbance compensation is rarely discussed. This study focuses on trajectory tracking under external wind disturbances, specifically under two types of interference: average wind and wind shear. After compensating for the disturbance, a novel antidisturbance sliding mode control is designed based on reference model, to realize accurate, robust, and chatterless trajectory tracking under wind disturbance. Firstly, we addressed the limitations of conventional extended state observers, nonderived forms, and inadequate use of information by introducing state compensation function observers to enhance estimation accuracy for states and disturbances that are challenging to measure directly. Moreover, the disturbance variation is estimated using tracking differentiator and then compensated in the controller to enhance antidisturbance sensitivity. In addition, the effectiveness and stability of the proposed observer and controller are analyzed. Finally, a two-step verification is presented, including simulation with input disturbances and actual flight under two wind field types; multiple industrial fans are used to generate the average wind and wind shear during the experiment. Two robust control methods, state-of-the-art active disturbance rejection control and sliding mode control, are also implemented for comparison, and the results indicate that more than 61.2% accuracy improvement over comparison methods is achieved in simulation and actual experiments.