Background subtraction is an extensively used approach to localize the moving object in a video se- quence. However, detecting an object under the spatiotem- poral behavior of background such as rippling of water, moving curtain and illumination change or low resolution is not a straightforward task. To deal with the above-men- tioned problem, we address a background maintenance scheme based on the updating of background pixels by estimating the current spatial variance along the temporal line. The work is focused to immune the variation of local motion in the background. Finally, the most suitable label assignment to the motion field is estimated and optimized by using iterated conditional mode (ICM) under a Mar- kovian frame- work. Performance evaluation and compari- sons with the other well-known background subtraction methods show that the proposed method is unaffected by the problem of aperture distortion, ghost image, and high frequency noise.