Bolted joint structure is widely utilized in the aeroengine compress rotor systems with convenience in assembly and maintenance. Due to the inherent structure discontinuity and stick-slip behavior at the mating interface, stiffness softening and frictional energy dissipation would be generated at the mating interface, resulting in complex vibration features of the rotor system. In this work, to describe the stick-slip behavior at the mating interface of the rotor system, an Iwan-based model is introduced to describe the contact state of asperities at the multi-scale level. The corresponding model identification method is proposed based on the partial hysteresis curve of the bolted joint. After that, a dynamic model of the bolted joint rotor system is established, and its dynamic response is obtained using the Runge–Kutta method. The effect of preload and friction coefficient on the nonlinear vibration of the bolted joint rotor is evaluated in detail based on the dynamic model. The result indicates that the stiffness softening would lead to the critical speed decreasing and aggravating the vibration response of the rotor system, while the frictional energy dissipation would restrain the response amplitude and induce the self-excited vibration within the super-critical speed range. The higher preload or friction coefficient has little effect on the motion state of the rotor system within the sub-critical speed range and makes system vibration stable between the 1st- and 2nd-order critical speed while unstable at the super-critical speed range. This research helps understand the vibration features of bolted joint rotor systems under the condition of stick-slip behaviors at the mating interface.