Speech processing in hearing aid often operates in noisy environments. So noise reduction algorithm is very important for hearing aid. This paper reviews conventional wiener filter algorithm and points out its remained problems----musical residual noise and low speech intelligibility for low input signal-to-noise ratios (SNR). To solve the problems, the proposed algorithm divides the frequency (0-8000 Hz) into 16 bands in Mel-frequency scale and processes the signal separately to remove musical residual noise. Meanwhile, a new Voice Activity Detector (VAD) applied for digital hearing aid is also presented. Based on the accurate judgment of speech frame, the filter processes the speech frame and non-speech frame in different ways. The method is realized in matlab to test output SNR and PESQ (Perceptual Evaluation of Speech Quality). It is also realized in android app for real-time test. Both of the experimental results demonstrate the advances of the proposed algorithm.