Considering the two kinds of nonlinear constraints of rolling mill hydraulic cylinder, spring force and friction force, the vibration model of rolling mill system is established. The amplitude frequency response equations are obtained by using the average method. Comparing the time history curves of vertical vibration displacement of rolling mill system under the nonlinear spring force and friction force, the amplitude frequency characteristic curves are simulated. The external excitation amplitude is viewed as the bifurcation parameter, and the system bifurcation response changing with the external excitation amplitude is analyzed. The influence of the external excitation amplitude on the system stability is studied. The results indicate that the increase of the nonlinear spring force makes the rolling mill system’s unstable area to become wider, and the influence on the rolling mill system of nonlinear friction force behaves as the damping characteristics; the vibration of rolling mill system is alternating between the periodic, period-doubling, and the chaotic motion. The research results provide a theoretical support for restraining the vibration of the rolling mill system in the actual production process.