The approach described in this research is an exemplar‐based inpainting problem that combines a two‐stage structure tensor and image sparse representation to fill in any missing pixels. An important step is to select the filling order and local intensity smoothness, as well as to ensure that the structure is not destroyed. We employ a two‐stage structure tensor‐based priority for the filling order: finding the candidate patches and determining the appropriate weight of each candidate patch under the constraint of local patch consistency, then applying a blend of a sparse linear combination of candidate patches to fill in the missing region of the image. In addition, this technique may also be used for object removal. The proposed method yields results that are visually natural and qualitative.