To address the difficulty of extracting weak spectral lines from signals received by passive sonar, we propose a spectral line extraction scheme based on an improved hidden Markov model (HMM). A new state transition probability based on spectral line features is proposed that solves the problem of state transition probability relying on prior information in traditional HMMs. Using a peak detection algorithm and a parallel processing framework reduces computation. We employ the boxplot method to remove the outliers from the spectral lines caused by strong noise and compensate for them. By improving the forward–backward probability calculation method through a peak penalty factor, we manage adjacent spectral lines prone to be missed by traditional HMMs. Lastly, we use dynamic sliding windows to determine a spectral line’s birth and death. Data verification by simulations and sea tests show that our algorithm extracts spectral lines better and with a smaller error, accurately detects the birth and death of spectral lines, and is faster than traditional HMM algorithms.