In this paper, the nonlinear mapping relationship between characteristic parameters of failures and failure types is realized by using neural network through extracting characteristic variables of failures during operation of the gear. Aiming at the problems of neutral network such as slow convergence speed and existence of local minima, the neural network is optimized and the ant colony neural network is established by using the ant colony algorithm to realize rapid and accurate determination of failure status of a gear from characteristic parameters of failures. In addition, validity of the established model is verified through experiments.