For the estimation of frequency response function (FRF) using the window-based spectral estimation method, the error models of the FRF estimation are introduced, including window interpolation error and transient error models, and an error model-based iterative compensation (EMIC) method is then proposed to improve the FRF estimation accuracy. The specific iteration steps of the proposed EMIC method are presented, fast computations are adopted in the steps as far as possible; and that, iteration convergence and noise uncertainty are analyzed. The analyses reveal that, the proposed EMIC method is almost always convergent as long as the error of the FRF pre-estimation is small, intercepting enough data and using a window which has smaller and fewer side DFT bins as well as smaller front-end value for FRF estimation can result in better iteration convergence; whereas, the proposed EMIC method will slightly increase the noise uncertainty of FRF estimation compared with that without iterative compensation because it removes the window interpolation effects on noise just like removing the error. To verify the effectiveness and practicability of the proposed EMIC method, model simulation and actual sensor data verification are implemented, and multi existing methods are employed for comparison. The results show that, the proposed EMIC method can largely decrease the error of the FRF estimated by the window-based spectral estimation method, it has the best maximum error restraining capabilities especially in the resonance frequency band and relatively better comprehensive error restraining capabilities especially when the data is not long enough compared with the mentioned existing methods, also, the time consumption of the proposed EMIC method is acceptable and is usually seconds or less; and that, the theoretical analyses, model simulation results and actual data verification results are consistent with each other. Consequently, the proposed EMIC method is proven to be effective and practical for improving the FRF estimation of a stable LTI system from its step or impulse response data.