Vibration sources in multi-rotor gas turbines under time-varying speed conditions are strongly coupled in the time–frequency domain, which prevents further mining of dynamic information from measured signals. Hence, a novel scheme of vibration source identification and separation is proposed to obtain individual source information. Firstly, the order and resonance components are identified by order analysis, and precise order numbers are further determined by computed order tracking. Then, in four typical vibration source coupling scenarios, the performances of Vold-Kalman filter (VKF), several typical adaptive mode decomposition methods and derivative methods are comparatively studied using Jeffcott rotor simulation system. Finally, the optimal source separation based on VKF and variational mode decomposition/extraction (VMD/VME) is determined. Furthermore, the settings of VKF filter bandwidth are studied quantitatively, and the effects of VMD and VME penalty factors are analyzed. In conclusion, the proposed scheme effectively identifies and accurately separates vibration sources from the test bed and gas turbine datasets, which provides important evidences for vibration monitoring and control as well as vibration transmission analysis of gas turbines.