Higher order cumulant (HOC) is a new theory and method of modern signal analysis. Empirical mode decomposition (EMD) is a new time-frequency analysis method for analysis of non-stationary and nonlinear processes. This paper discusses the characteristics of both HOC and EMD, investigates their related technology and advantage in the process of dealing with nonlinear coupling characteristics of random and non-stationary signals. Then through combining with the merits of HOC and EMD, it proposes a new method for multi-damage feature extraction and fault diagnosis of gear systems. For restraining system noise and extracting the fault features of signals, the acquired signal is decomposed into a three-layer signal which has different frequency bands, and then every layer is given to a HOC analysis. Six kinds of signal are analyzed including signal without fault, signal with tooth root short crack, signal with tooth root long crack, signal with pitch circle short crack, signal with pitch circle long crack and signal with tooth wear, which were acquired at four different speeds: 300 r/min, 900 r/min, 1200 r/min and 1500 r/min. The results demonstrate that this method not only can be used to identify various faults (including multi-fault) in the low-speed and high-speed running conditions, but can also be used to identify the damage level of some faults. On this basis, combining the advantages of MATLAB and Visual C++ (VC++), a virtual instrument (VI) system of gear damage detection and diagnosis is developed through a mixed programming method. The interface is designed with VC++, the calculation of test data, signal processing and graphical display are completed by MATLAB. Using COM module technology, it calls the *.m file conversion program in VC++. A different kind and multifunctional gear fault diagnosis software system is successfully empoldered, which possesses some functions, including the introduction of gear vibration signals, signal processing, feature extraction, graphics display, fault detection and diagnosis. The system can be used to detect damage and diagnose faults in gear systems as proven by the results of the validation testing.