Rolling mill is the core equipment in modern iron and steel industry, and the reliability of its mill roll system (MRS) is the key to ensure the rolling process with high precision, high speed, continuity and stability. However, the MRS possesses some features such as high nonlinearity, time variability and strong coupling. The vertical vibration easily happens in its working process. Nevertheless, the mathematical model of MRS is difficult to be established and hard to be solved. In this paper, the nonlinear dynamics theory and modern signal processing method were introduced to solve this difficult problem. A two degree of freedom (DOF) nonlinear vertical vibration model of MRS was established. And the model was analytically solved by using complexification averaging (CA) method. The solution error of CA method was thoroughly analyzed. Moreover, the CA method was improved by combining the fast empirical mode decomposition (FEMD) method. Research results indicate that the improved CA method presents a significant advantage in improving the solution precision, and can be used to solve strong nonlinear vibration system (NVS) with two DOF.