Aiming at improving the accuracy of planetary gearbox fault diagnosis, an integrated scheme based on dimensionality reduction method and deep belief networks (DBNs) is presented in this paper. Firstly, the acquired vibration signals are decomposed into mono-component called intrinsic mode functions (IMFs) through ensemble empirical mode decomposition (EEMD), and then Teager–Kaiser energy operator (TKEO) is used to track the instantaneous amplitude (IA) and instantaneous frequency (IF) of a mono-component amplitude modulation (AM) and frequency modulation (FM) signal. Secondly, a high dimensional feature set is constructed through extracting statistical features from six different signal groups. Then, an integrated dimensionality reduction method combining feature selection and feature extraction techniques is proposed to yield a more sensitive and lower dimensional feature set, which not only reduces the computation burden for fault diagnosis but also improves the separability of the samples by integrating the label information. Further, the low dimensional feature set is fed into DBNs classifier to identify the fault types using the optimal parameters selected by particle swarm optimization algorithm (PSO). Finally, two independent cases study of planetary gearbox fault diagnosis are carried out on test rig, and the results show that the proposed method provides higher accuracy in comparison with the existing methods.