This paper presents an optimized gear fault identification system using genetic algorithm (GA) to investigate the type of gear failures of a complex gearbox system using artificial neural networks (ANNs) with a well-designed structure suited for practical implementations due to its short training duration and high accuracy. For this purpose, slight-worn, medium-worn, and broken-tooth of a spur gear of the gearbox system were selected as the faults. In fault simulating, two very similar models of worn gear have been considered with partial difference for evaluating the preciseness of the proposed algorithm. Moreover, the processing of vibration signals has become much more difficult because a full-of-oil complex gearbox system has been considered to record raw vibration signals. Raw vibration signals were segmented into the signals recorded during one complete revolution of the input shaft using tachometer information and then synchronized using piecewise cubic hermite interpolation to construct the sample signals with the same length. Next, standard deviation of wavelet packet coefficients of the vibration signals considered as the feature vector for training purposes of the ANN. To ameliorate the algorithm, GA was exploited to optimize the algorithm so as to determine the best values for “mother wavelet function”, “decomposition level of the signals by means of wavelet analysis”, and “number of neurons in hidden layer” resulted in a high-speed, meticulous two-layer ANN with a small-sized structure. This technique has been eliminated the drawbacks of the type of mother function for fault classification purpose not only in machine condition monitoring, but also in other related areas. The small-sized proposed network has improved the stability and reliability of the system for practical purposes.