Image upscaling employs various polynomial techniques, including bicubic and Lanczos, to minimize computing load for various real-time applications. Polynomial interpolation, on the other hand, results in blurring artifacts in high resolution (HR) images due to edge degradation. Many other interpolation approaches, including edge-based and dictionary learning-based schemes, result in blurring in the image’s high frequency (HF) region. As a result, a novel post-processing technique is proposed that uses the Lanczos method to upscale the low resolution (LR) image. To eliminate blur along the image’s edge, the edge component of the interpolated image is retrieved. Adaptive sharpening is used to make the edges more prominent, as predicting pixels in the high variance region is challenging. After the edge is extracted, the smooth region is filtered with an optimized directional anisotropic diffusion (ODAD) filter to preserve texture details and geometric regularity. The sharpened edge image is aggregated with the ODAD filtered image to generate an improved HR (IHR) image. The IHR is then scaled up and down to reduce blurring even further, resulting in a blurred HR image (BHR). The residual image is obtained by subtracting the BHR from the IHR images, which represent the undetected HF. Finally, the residual image and the IHR image are combined to generate the final restored image, which includes all prediction loss details. The results of the experiments show that the proposed algorithm outperforms existing methodologies, particularly in terms of visuality and parameter-wise measurement.