This article studies the trajectory tracking and active vibration control of flexible space manipulator without velocity measurements. Variable-speed control moment gyros (VSCMGs) are adopted as actuators for vibration suppression. Based on the singular perturbation method, the dynamics model of the manipulator system considering the uncertainties and disturbances is decoupled into two reduced-order subsystems: the slow subsystem and the fast subsystem. For the slow subsystem, neural network (NN) is used to approximate the unknown nonlinear functions. An NN-based adaptive integral sliding-mode observer is designed to estimate some unknown velocity variables. An observer-based slow controller is proposed. Considering the uncertainties and the unmeasured modal information of the fast system, a modified extended state observer (ESO) is designed for the estimation of the uncertainties and unknown fast variables. On the basis of the modified ESO, an adaptive feedback controller is proposed for the flexible motion. Lyapunov theory is used to prove the stability of the proposed controllers. The weighted robust pseudoinverse steering laws are designed for VSCMGs. The effectiveness of the proposed observer-based hybrid controller is verified via numerical simulations.