Abstract Rotating machinery signals are usually intricate and often nonstationary. The analysis of such signals is further complicated because of the spectral overlaps between constituent frequency components under time-varying running conditions. Most conventional signal processing methods are unable to effectively extract the embedded meaningful information. Adaptive mode decomposition (AMD) methods are flexible to describe complex signals, free from the limitations inherent in conventional basis expansion and thus more suited to unveiling the underlying physical nature. However, as those methods work in a way similar to filter banks, i.e. via rectangular partition of time-frequency plane, they are effective only for signals with constituent frequency components that are almost constant and parallel to each other. As such, it would be inappropriate to apply such methods when signal constituent frequency components vary considerably over time and particularly overlap in frequency domain. To address this issue, this paper proposes a general framework by exploiting the unique capability of generalized demodulation to transform an arbitrary time-varying instantaneous frequency into a constant frequency. In doing so, many state-of-the-art AMD methods (including empirical mode composition, local mean decomposition, intrinsic time-scale decomposition, local characteristic scale decomposition, Hilbert vibration decomposition, empirical wavelet transform, variational mode decomposition, and adaptive local iterative filtering) can be extended to analyze highly nonstationary complex signals with spectral overlaps. The principle and advantage of generalized adaptive mode decomposition (GAMD) are illustrated through numerical simulation. The GAMD’s performance in monitoring different rotating machinery components (including a planetary gearbox, an aircraft engine rolling bearing and a hydraulic turbine rotor) is demonstrated and validated under time-varying conditions.