Abstract Given the complex and harsh operating conditions of gear transmission systems, gearboxes are prone to compound faults. These faults, involving multiple types, often couple together, causing the weak fault pulse characteristics to be entirely masked by strong environmental noise. This significantly complicates the extraction of relevant fault information from the gearbox. To address these issues, this paper proposes a method for extracting compound fault features based on improved symplectic geometric mode decomposition (SGMD) and optimized cyclic bispectrum deconvolution (CYCBD). Firstly, considering the periodic impact characteristics of different fault types, morphological envelope cyclic bispectrum is proposed to cluster the initial components obtained from SGMD decomposition, thereby adaptively separating different fault features contained in the compound fault signals. An adaptive filter length search strategy is subsequently introduced to optimize the CYCBD by deconvolving each initial fault component, thereby eliminating the interference caused by complex transmission paths and substantial environmental noise, which, in turn, enhances weak periodic fault pulses. Following this, the enhanced signals are subjected to envelope demodulation to extract fault characteristic frequencies, enabling the identification of various types of faults. The effectiveness and feasibility of the proposed method are demonstrated through both simulation signals and actual experimental data related to gearbox compound faults. Compared with existing methods, the proposed method demonstrates superior performance in identifying weak compound faults under strong environmental noise.