Abstract
Image restoration based on the Deep Convolutional Neural Network (CNN) based image restoration has demonstrated promising results in many sub-tasks, such as image super-resolution, compression artifacts removal, image denoising, and image enhancement. Compared to many CNN-based high-level vision tasks that predict sparse probabilities of each class, the CNN for image restoration requires dense pixel-level predictions with precise intensity-level values. Therefore, a minimum number of spatial pooling (or down-sampling) operations is required to maintain the image details. Therefore, designing a fast or lightweight model for image restoration is a difficult problem and is even more critical when the spatial resolution of an image becomes larger. In this paper, we propose a family of networks called Subpixel Prediction Networks (SPNs) that predict reshaped and spatially down-sampled block-wise tensors instead of raw images with full resolution. Under this scheme, spatial downsampling decreases the restoration performance less while making the network faster. We propose a novel Subpixel Block Attention (SBA) module that re-calibrates blockwise features to diminish blockwise discontinuity to increase the performance further. The experimental results reveal that these networks demonstrate good trade-offs between speed (number of computations) and restoration performance in the three image restoration tasks: image compression artifacts removal, color image denoising, and image enhancement.
Highlights
Image restoration is a representative ill-posed problem with a long history
After the great success of deep Convolutional Neural Networks (CNNs), many studies have solved image restoration problems, such as image super-resolution [1]–[5], image compression artifacts removal [6]–[10], image denoising [8], [11]–[13], and image enhancement [14] using the deep CNNs in an end-to-end manner
To increase the performance further, we propose several modifications from the baseline: 1) advanced training schemes and optimized network structures, 2) the novel subpixel block attention (SBA) module to diminish the discontinuity between subpixel blocks, and 3) the Subpixel Prediction Networks (SPNs) in the frequency domain for the image compression artifacts removal task
Summary
Image restoration is a representative ill-posed problem with a long history. After the great success of deep Convolutional Neural Networks (CNNs), many studies have solved image restoration problems, such as image super-resolution [1]–[5], image compression artifacts removal [6]–[10], image denoising [8], [11]–[13], and image enhancement [14] using the deep CNNs in an end-to-end manner. For other restoration tasks that maintain the original resolution, the de-subpixel and subpixel convolutional based method [25] is proposed to reduce the computation (Fig. 1 (c)). To increase the performance further, we propose several modifications from the baseline: 1) advanced training schemes and optimized network structures, 2) the novel subpixel block attention (SBA) module to diminish the discontinuity between subpixel blocks, and 3) the SPN in the frequency domain (spatial-to-frequency networks) for the image compression artifacts removal task. This paper extends our previous conference paper [26] with the following additional contributions: 1) we generalize the task-specific spatial-to-frequency networks into the family of SPNs, 2) we refactor the network details, training dataset, and training algorithms to increase the performance, 3) we add additional analysis of memory-computation trade-offs, 4) we propose a novel SBA module, and 5) we generalize the SPNs for diverse image restoration tasks, such as color image denoising and image enhancement in various datasets.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
