Fault coupling and fault override are common phenomena when faults occur in different parts of the planetary gearbox. Labeled compound fault samples are very rare or even unavailable in industrial scenarios. Thus, it is a challenging task to decouple the compound fault and detect the compound fault with unequal severity only utilizing single faults. In this paper, an untrained planetary gearbox compound fault diagnosis method based on Adaptive Learning Variational Mode Decomposition (ALVMD) and Dual Scale Squeeze-and-Excitation Convolutional Neural Network (DSSECNN) is proposed. An ALVMD algorithm is established to enhance weak fault characteristics and enrich the diversity of sample information. The DSSECNN intelligent fault diagnosis model is presented, and the lightweight SENet can enhance the sensitivity of the model to channel features. Moreover, the mapping relationship between the compound fault mode and single faults is revealed through the proposed probability formula, which can reduce the dependence of the training model on compound fault samples. Experiments have been performed on the gearbox fault experiment bench in the laboratory to verify the effectiveness and generalization performance of the method. Besides, wind turbine gearbox test results show that the ALVMD-DSSECNN can achieve the highest average accuracy of 98.83% compared with other related methods in variable working conditions, which has a certain guiding significance for practical engineering.