As the core equipment of metal shaping processing, the vibration problem of rolling mill seriously affects the product quality and production efficiency. To address the problem of the limited prediction of mill vibration due to small sample data in non-steady states, we propose to predict dynamic mechanical parameters (rolling force and rolling torque) based on the TCN-LSTM step strategy transfer model. The prediction accuracies of the TCN-LSTM step strategy transfer model under 1000 sets of training data reach 95.8% and 93.2%, respectively, and at the same time, it saves the time of training and regulation of the deep learning model and can better meet the needs of online prediction. The horizontal-torsion hot rolling vibration model is further established to quantitatively analyze the relationship between process parameters and mill vibration. The experimental results show that with the vibration suppression measures of 10% reduction of the rolling speed and 10% reduction of the entrance thickness, the horizontal vibration displacement amplitude is reduced from 0.687 × 10−4 m to 0.229 × 10−4 m, and the rotational displacement amplitude is reduced from 0.0273 m to 0.0117 m, which effectively suppresses the vibration of the mill and improves the stability of the rolling process.