Weak fault detection is a complex and challenging task when two or more faults (i.e. a compound fault) with discordant severity occur in different parts of a gearbox. The weak fault features are prone to be submerged by the severe fault features and strong background noise, which easily lead to a missed diagnosis. To solve this problem, a novel diagnosis method combining multi-symplectic geometry mode decomposition (MSGMD) and multipoint optimal minimum entropy deconvolution adjusted (MOMEDA) is proposed for gearbox compound fault in this paper. Specifically, different fault components are separated by an improved symplectic geometry mode decomposition (SGMD), namely the MSGMD method. The weak fault features are enhanced by the MOMEDA. In the process of research, a new scheme of selecting the key parameters for MOMEDA is proposed, which is a key step in the practical application of MOMEDA. Compared with SGMD, the proposed MSGMD has two main improvements, including suppressing mode mixing and preventing the generation of pseudo-components. Compared with the original method of selecting parameters based on multipoint kurtosis, the proposed MOMEDA parameter-selecting scheme has the merits of higher accuracy and greater precision. The analysis results of two cases, of simulation and an experiment signal, reveal that the MSGMD–MOMEDA method can accurately diagnose gearbox compound fault even under strong background noise.