Abstract The transient electromagnetic method (TEM) is now commonly used in the detection of coal mined-out areas, but it is susceptible to various sources of noise during field site probing, significantly impacting the accuracy of detection. Variational mode decomposition (VMD) has gained widespread adoption for eliminating noise in TEM signals. However, a single VMD may not effectively address both full-band noise and residual Gaussian white noise. To address this limitation, the singular value decomposition (SVD) and wavelet threshold denoising (WTD) are introduced correspondingly. Furthermore, this paper tackles challenges related to parameters selection in VMD and traditional optimization algorithms by proposing the integration of dung beetle optimization (DBO) enhanced with circle chaotic mapping, random walk strategy, and crisscross optimization algorithm (CRCDBO). Consequently, a new combined TEM noise reduction method, VMD-SVD-WTD based on CRCDBO is presented. Through comprehensive simulation tests and comparative analyses, the signal-to-noise ratio of this method is improved by 12.79% compared with VMD and the RMSE is 0.0011, which demonstrate the superiority and effectiveness of VMD-SVD-WTD based on CRCDBO in noise reduction. Meanwhile, this method is applied to an iron ore mine in Shanxi, which can accurately monitor the location of the mined-out area and has good application value.