Purpose This paper aims to address the challenges in hydroacoustic signal detection, signal distortion and target localization caused by baseline drift. The authors propose a combined algorithm that integrates short-time Fourier transform (STFT) detection, smoothness priors approach (SPA), attitude calibration and direction of arrival (DOA) estimation for micro-electro-mechanical system vector hydrophones. Design/methodology/approach Initially, STFT method screens target signals with baseline drift in low signal-to-noise ratio environments, facilitating easier subsequent processing. Next, SPA is applied to the screened target signal, effectively removing the baseline drift, and combined with filtering to improve the signal-to-noise ratio. Then, vector channel amplitudes are corrected using attitude correction with 2D compass data. Finally, the absolute target azimuth is estimated using the minimum variance distortion-free response beamformer. Findings Simulation and experimental results demonstrate that the SPA outperforms high-pass filtering in removing baseline drift and is comparable to the effectiveness of variational mode decomposition, with significantly shorter processing times, making it more suitable for real-time applications. The detection performance of the STFT method is superior to instantaneous correlation detection and sample entropy methods. The final DOA estimation achieves an accuracy within 2°, enabling precise target azimuth estimation. Originality/value To the best of the authors’ knowledge, this study is the first to apply SPA to baseline drift removal in hydroacoustic signals, significantly enhancing the efficiency and accuracy of signal processing. It demonstrates the method’s outstanding performance in the field of underwater signal processing. In addition, it confirms the reliability and feasibility of STFT for signal detection in the presence of baseline drift.