Simulation study of the failure of gearbox system using virtual prototype technology and analysis of its vibration signal is a foreword topic in recent years. This paper established a rigid-flexible coupling model of gearbox system using SolidWorks, ANSYS and ADAMS. Then, make simulation under the following three conditions: normal, gear fault and bearing outer ring fault. Next, make time domain and frequency domain analysis of the simulated fault signal using MATLAB. Simulation results are consistent with gear vibration mechanism and signal modulation theory, which verified correctness and feasibility of virtual prototype technology in gearbox fault simulation. Lay foundation for gearbox fault diagnosis.