This paper proposes an improved speech enhancement algorithm based on Wiener-Filtering, which addresses the problems of speech distortion and musical noise. The proposed algorithm adopts the masking properties of human auditory system on calculating the gain of spectrum point, in order that the signal in the enhanced speech whose energy is lower than the threshold will not be decreased further and the less distortion will be brought to enhanced speech by the trade-off between the noise elimination and speech signal distortion. What’s more, in order to eliminate the “musical noise”, a spectrum-shaping technology using averaging method between adjacent frames is adopted. And to guarantee the real-time application, two-stage moving-average strategy is adopted. The computer simulation results show that the proposed algorithm is superior to the traditional Wiener method in the low CPU cost, real-time statistics, the reduction of the speech distortion and residual musical noise.