When the rolling bearing raceway is tilted, the contact load of the raceway increases, which may greatly increase the occurrence of raceway defects. However, the evolution law of the raceway defect extension process on the system dynamic characteristics is not clear when the bearing is tilted. In this paper, considering the influence of rough surface caused by the extension of bearing defects, a mathematical model of raceway extension defects which is more consistent with the actual morphology is proposed. The model can be used to evaluate the damage degree of the bearing raceway of rotating machinery. Taking the two-span rotor-bearing-pedestal system of the test rig as the research object, a nonlinear dynamic model of the system is established considering the nonlinear supporting force of bearings and pedestal, shaft coupling looseness and other nonlinear factors. The effectiveness of the bearing fault characterization model and the system dynamics model is verified by experiments. On this basis, the influence of tilt on bearing fault and the change rule of raceway defect extension on bearing contact characteristics and system dynamic characteristics are discussed. The results indicate that raceway tilt aggravates system vibration caused by the bearing defect. With the increase of the defect range, the impact vibration characteristics weaken, and the roughness of the raceway surface in the defect area has a great influence. However, with the increase of defect depth, the influence law is just the opposite. The research results are expected to provide theoretical support for bearing fault diagnosis of rotating machinery systems.