AbstractThe DPCM system is often used for encoding images, but particularly in low‐rate encoding methods, two types of noise of different nature (e.g., overload noise and granular noise) are introduced. In this paper a tree encoding scheme is applied to image signals and a design technique for an encoding filter is shown which is a combination of a least‐mean‐square filter derived from linear prediction theory and a nonlinear smoothing filter. The purpose of this nonlinear filter is to decrease the above‐mentioned noise. Its construction is like that of an adaptive quantizer in which one of the preassigned output levels is selected, keeping in view the previously encoded symbols. The linear smoothing filter used in tree encoding of speech is a special case of this sytem. Moreover, the conditional mean output level for the present encoding symbol matches the output level of the Lloyd‐Max quantizer for the Laplacian density function. The results obtained by applying this scheme to a head and shoulder image having clear boundaries are compared with the results without using a nonlinear smoothing filter and improvement of about 2 dB in SNR is obtained when the encoding rate is 1 bit/pixel.