Hysteresis behavior at the mating interface can induce nonlinear stiffness and frictional dissipation, which may lead to a complex vibration character, self-excited vibration, and even the rubbing fault for the bolted joint rotor system. The present work aims to investigate the effect mechanism of hysteresis behavior on the rotor dynamics and the mechanism of bolted joint performance change induced by a rubbing fault. Firstly, the bolted joint model considering the hysteresis behavior at the mating interface is proposed based on the Iwan model. Then, a non-dimensional dynamic model of the bolted joint rotor system with rubbing fault is established by combining the lumped-mass method and fixed-point rub-impact force model. After that, the nonlinear response of the bolted joint rotor system considering the hysteresis behavior at the mating interface and the effect of rubbing fault on the dynamic performance of such a rotor system is explored. Finally, an experimental verification is carried out by adopting a rotor system test rig with a different stiffness rubbing device. The result shows that the rubbing fault will reduce the critical speed by changing the equivalent average stiffness of the bolted joint and aggravate the self-excited vibration of the system, leading to a stronger nonlinear performance of the rotor system. The outcome of the present work can provide valuable insights into vibration prediction, health detection, and rubbing fault diagnosis for a rotor system with a bolted joint.