ABSTRACT This study investigates the 3D trajectory tracking of a quadrotor-type unmanned aerial vehicle (UAV) using a visual localization system integrated in the feedback control loop. First, adaptive stereo visual odometry (SVO) is proposed to solve the UAV localization problem. The proposed solution mainly employs adaptive iterative closest point speeded-up robust features. Second, a vision-based trajectory tracking algorithm is implemented using a backstepping controller, whose parameters are optimized using the particle swarm optimization algorithm. Finally, to avoid the limitations of visual localization (e.g. dark environment, uniform area, and dynamic scene), the visual pose is supported by the inertial pose obtained using a fuzzy adaptive Kalman filter (FAKF). Based on the number and average depth of the estimated 3D points, the FAKF algorithm adaptively tunes the extended Kalman filter parameters. The proposed algorithms are validated using simulation and experimental data. Many scenarios are considered with different trajectories. Good performance is achieved, confirming the efficiency of the proposed approach.