Abstract In this study, the effects of the evolution of bearing outer race defect size and increase in speed on the vibration characteristics of a shaft-bearing system under unbalanced conditions are simulated and analyzed. A two degrees-of-freedom mathematical model is presented for a ball bearing used in an unbalanced shaft-bearing system. The contact stiffness between the races and the balls is considered as a series of springs is incorporated in the model. Hertzian contact deformation theory is used to obtain the contact stiffness. This model considers the contact deformation between the balls and the races, the additional displacement between the balls and the inner race due to radial clearance and due to defect geometry. The maximum possible radial displacement of the ball into the defect is calculated analytically using the groove radius, ball radius, and defect diameter. The rectangular function is used for modeling the defect. matlab codes are developed for modeling the bearing and for solving the differential equations of motion using the Runge–Kutta method. The vibration responses (peak and root-mean-square (RMS) values) obtained by modeling and by experimentation show similar vibration characteristics. The investigation shows that the values of statistical parameters initially increase with the increase in defect size and then decrease with a further increase in defect size. While peak and RMS increase with the increase in speed, crest factor and kurtosis decrease with the increase in speed. Peak is more sensitive for diagnosing spalls on outer race and its evolution. This study helps as an effective diagnosis of antifriction bearings having spalls on the outer race under unbalanced conditions.