To improve the accuracy of the diagnosis of the composite failure of the rotor system under underdetermined conditions, a single-channel blind source separation method based on an adaptive multiscale generalized morphology filter (AMGMF) and an adaptive fast ensemble empirical mode decomposition (AFEEMD) algorithm is proposed. First, AMGMF is used to filter out the background noise in the signal and enhance the signal-to-noise ratio (SNR), and then, the filtered signal is decomposed by the AFEEMD algorithm to improve the accuracy of signal decomposition and reduce computation time. Second, the singular value decomposition-based method is used to estimate the number of source signals, and the main components highly correlated with the observed signal are selected according to the number of source signals. Finally, the effective separation of source signals is achieved by the fast independent component analysis (FastICA) algorithm. Simulation tests and multiple rotor fault signals are used to demonstrate the feasibility and effectiveness of the proposed method. Experimental results show that the proposed method has better separation performance than the existing EEMD-PCA-FastICA method.