Jittery image is visually abnormal in jags of edge and loss of coherence. The problem of image dejittering is challenging to resolve due to the ubiquitous blur and/or noise in jittery data. In this paper, we devote to the pixel-jitter (possibly blurry) image recovery on the perspective of spatially-varying mixed noise removal. By viewing jittery image as the corruption of ideal image with outliers and spatially-varying Gaussian noise, we proposed a two-phase (including filtering and diffusing phases) image dejittering approach. In the filtering phase, outliers posed by jitters around edges are inspected by median filters. In the diffusing phase, structure tensor based anisotropic diffusion is exploited to reduce the perturbations in piecewise smooth image regions. Upon the spectral decomposition of structure tensor, the variational model in diffusing phase can be solved by some state-of-the-art optimization methods. Numerical simulations on synthetic and real jittery data demonstrate the compelling performance of the proposed approach. The Matlab source codes of the proposed approach are available at the repositories of https://github.com/WenxingZhang.