The vibration response and nonlinear dynamic behavior of a rotor-bearing-brush seal system were investigated with a new seal force model of the brush seal. The nonlinear oil–film force model was adopted based on a short bearing assumption. The dimensionless equation of motion was solved using the fourth order Runge–Kutta method. The effects of key parameters including rotor speed, installation spacing of the brush seal, disk eccentricity, disk mass, and journal mass on the nonlinear dynamic characteristics of rotor-bearing-brush seal system were determined and compared under different operating conditions with a bifurcation diagram, time history, axis orbit, poincaré map, frequency spectrum, and spectrum cascade. The results showed that the system response contained various nonlinear phenomena, such as periodic motion, multi-periodic motion, and quasi-periodic motion. The interaction of the rotor speed, installation spacing of the brush seal, disk eccentricity, disk mass, and journal mass could seriously affect the stability and working condition of the system. This study provides a theoretical support for the selection of key design parameters and further understanding of the nonlinear characteristics of rotor-bearing-seal systems with a brush seal.