In this paper, we propose an immersion and invariance-based sliding mode controller for a tilt tri-rotor unmanned aerial vehicle subjects to parameter perturbation, unmodeled dynamics, and external disturbances. The control scheme is divided into three parts, including the disturbance observer, the attitude controller, and the control allocation. Firstly, to alleviate the chattering and improve the robustness for attitude control, the observer using immersion and invariance theory is developed to estimate the disturbance. Note that the observer can relax the requirement of disturbance upper bound and guarantee the convergence of the estimation error. Secondly, to improve the dynamic response capability, a sliding mode attitude controller with an adaptive switch function is designed based on the disturbance observer. Thirdly, a hierarchical control allocation algorithm is proposed. The performance improvement is illustrated by comparing with other sliding mode controllers. Simulations and flight experiments are conducted to verify the effectiveness and applicability of the proposed control scheme.