To reduce errors in sub-synchronous oscillation (SSO) modal identification and improve the accuracy and noise resistance of the traditional Prony algorithm, this paper focuses on SSOs caused by the integration of doubly fed induction generators (DFIGs) with series compensation into the grid. A novel SSO modal identification method based on the hippopotamus optimization–variational mode decomposition (HO-VMD) and singular value decomposition–regularized total least squares–Prony (SVD-RTLS-Prony) algorithms is proposed. First, the energy ratio function is used for real-time monitoring of the system to identify oscillation signals. Then, to address the limitations of the VMD algorithm, the HO algorithm’s excellent optimization capabilities were utilized to improve the VMD algorithm, leading to preliminary denoising. Finally, the SVD-RTLS-improved Prony algorithm was employed to further suppress noise interference and extract oscillation characteristics, allowing for the accurate identification of SSO modes. The performance of the proposed method was evaluated using theoretical and practical models on the Matlab and PSCAD simulation platforms. The results indicate that the algorithms effectively perform denoising and accurately identify the characteristics of SSO signals, confirming its effectiveness, accuracy, superiority, and robustness against interference.