This paper focuses on the system identification of a small unmanned helicopter in hover or low-speed flight conditions. A novel genetic algorithm including chaotic optimization operation named chaos-genetic algorithm (CGA) is proposed to identify the linear helicopter model. Based on the input-output data collected from real flight tests, the identification performance of CGA is compared with those calculated by the traditional genetic algorithm (TGA) and the prediction error method (PEM). The accuracy of the identified model is verified by simulation in time domain. Additionally, the small unmanned helicopter is stabilized by a linear quadratic Gaussian (LQG) regulator based on the proposed identified model. In the automatic flight experiments, the achievement of automatic take-off and landing, hovering performance within a 1.2 m diameter circle and point-to-point horizontal polyline flight also demonstrates the accuracy of the identified model and the effectiveness of the proposed method.