Speckle noise significantly impairs the accuracy of vibration measurements for non-cooperative targets in frequency-scanning interferometry (FSI). To restore the real vibration, this paper proposes a comprehensive vibration demodulation algorithm based on the FSI. We established an FSI vibration model accounting for speckle noise. The extraction of instantaneous frequency from the interference signal is performed using a complex-shifted Morlet wavelet. An autoregressive differential model, incorporating a smooth weight function, is applied for bidirectional prediction to correct speckle noise. Finally, the true vibration of the target is restored through Kalman filtering. We validated the algorithm’s accuracy using a Monte Carlo strategy. The experimental results show that the surface vibration of black aluminum oxide alloy and the axial runout of the DC motor during operation was successfully captured, and the time–frequency analysis results showed that this method has strong robustness against dense speckle noise.