The strong nonlinear dynamics of tail-sitter vertical takeoff and landing (VTOL) unmanned aerial vehicles (UAVs) have consistently constrained their applications. Particularly, the widely-varying flight speed and attitude during the transition flights pose substantial challenges for the controller design. In order to gain a clear understanding of the nonlinear transition flights, the long- and short- period dynamic characteristics of tail-sitter UAVs are investigated for the first time. The results reveal that the short-period mode exhibits either sluggish response or excessive overshoot in different transition stages, primarily due to the widely-varying damping ratio and natural angular frequency. Even worse, the long-period mode can diverge due to positive poles. To address these issues, a systematic controller is proposed. First, the transition trajectory is optimized and the optimal control inputs are employed in a feedforward manner to trim the nominal forces and moments. On this basis, a discrete-time linear quadratic regulation (LQR) controller with a predefined decay rate is developed to position the desired poles, and a novel angular acceleration estimation method is introduced to compensate for unmodeled dynamics. Simulations under different uncertainties indicate that the proposed control method performs better in both transition trajectory tracking and uncertainty suppression, compared to the PID and incremental nonlinear dynamic inversion (INDI) controllers.