Abstract Planetary gearboxes often experience multiple component failures during service, which can accelerate the degradation and failure of industrial equipment. Accurate separation and identification of multiple faults is an important means of ensuring the safe and stable operation of equipment. However, different faults can interact with each other, along with the influence of background noise, making it challenging to accurately extract faults with relatively weak energy among multiple faults. This difficulty leads to the problems of potential misdiagnosis and underdiagnosis. To address this issue, an adaptive mode decomposition method based on fault feature orientation (AMD-FF) is proposed in this paper. Initially, a fault impact indicator (FII) is constructed based on period-weighted kurtosis of envelope spectral and correlated combination negentropy to effectively characterize the impulsiveness and periodicity of fault features. Furthermore, with the objective of maximizing the FII, an adaptive decomposition of the original signal is designed based on blind convolution theory using a finite-impulse response filter group. Subsequently, a variable weight particle swarm optimization is employed to adaptively optimize the key decomposition parameters. Finally, the data of industrial-grade planetary gear transmission test rig are collected to validate the proposed method for compound fault diagnosis of planetary gearboxes. The results indicate that the AFMD-FF can effectively separate and extract compound faults in planetary gearboxes, demonstrating superior fault separation and diagnostic performance compared to the fault mode decomposition (FMD) and adaptive FMD. This method offers a novel approach to diagnosing compound faults in rotating equipment in industrial scenarios.