For quadrotor control applications, it is necessary to rely on attitude angle changes to indirectly achieve the position trajectory tracking purpose. Several existing literature studies omit the non-negligible attitude transients in the position controller design for this kind of cascade system. The result leads to the position tracking performance not being as good as expected. In fact, the transient behavior of the attitude tracking response cannot be ignored. Therefore, the closed-loop stability of the attitude loop as well as the position tracking should be considered simultaneously. In this study, the flight controller design of the position and attitude control loops is presented based on an integral backstepping control algorithm. This control algorithm relies on the derivatives of the associated virtual control laws for implementation. Examining existing literature, the derivatives of the virtual control law are realized approximated by numerical differentiations. Nevertheless, in practical scenarios, the numerical differentiations will cause the chattering phenomenon of control signals in the presence of unavoidable measurement noise. The noise-induced control signals may further cause damage to the actuators or even diverge the system response. To address this issue, the analytic form for the derivative of the virtual control law is derived. The time derivative virtual control law is analyzed and split into the disturbance-independent compensable and disturbance-dependent non-compensable terms. By utilizing the compensable term, the control chattering due to the differentiation of the noise can be avoided significantly. The simulation results reveal that the proposed control algorithm has a better position tracking performance than the traditional dual-loop control scheme. Meanwhile, a relatively smooth control signal can be obtained for a realistic control algorithm realization. Simulations are provided to illustrate the position tracking issue of a quadrotor and to demonstrate the effectiveness of the proposed compromised control scheme.