Image Super-resolution Task Research Articles

SRNMSM: A Deep Light-Weight Image Super Resolution Network Using Multi-Scale Spatial and Morphological Feature Generating Residual Blocks

Generating features representing the textures and structures of an image is very important characteristic of a super resolution network. Morphological operations are the nonlinear mathematical operations that can process signals focusing on their structures and textures. In this paper, we propose a novel residual block to generate and process morphological features and fuse them with the conventional spatial features, in order to produce a very rich and highly representational set of residual feature maps. The proposed residual block is then used in a deep convolutional neural network for the task of image super resolution. It is shown that the capability of the proposed block in generating and using the morphological features can significantly improve the super resolution performance of a deep network. The super resolution network employing the proposed residual block is shown to outperform the state-of-the-art low-complexity image super resolution networks on various benchmark datasets.

An efficient and lightweight image super-resolution with feature supplement network

Despite the great success of deep learning approaches in image super-resolution tasks in recent years, a large proportion of the approaches emphasize designing complex network structures to continuously deepen the network to achieve superior reconstruction performance. However, some shortcomings need to be addressed, such as the large number of parameters required for training the complex network structure, which makes the model difficult to optimize and train. To address the above problems, we propose an accurate and lightweight neural network which is called the feature supplement network (FSN) to achieve image reconstruction. Specifically, we design a novel residual feature supplement structure(RFSS), which is composed of some residual feature supplement groups (RFSGs) and layer feature attention module (LFAM). Each RFSG contains several heterogeneous residual channel attention blocks (HRCAB) with different levels of skip connections. In RFSS, the stacked RFSG is able to extract features efficiently with limited parameters. Meanwhile, LFAM further improves the feature representation capability of the network because it is capable of capturing more discriminative features from the original image by considering the correlation between the extracted features of different depths. Experiments of various models conducted on benchmark datasets show that our proposed method not only can achieve comparable results compared to the state-of-the-art methods, but also effectively recover high-quality images, even when using limited parameters than the existing method for image super-resolution. The codes of our proposed structure-FSN are accessible on https://github.com/ReckonerInheritor/RFSN.

DTS-Net: Depth-to-Space Networks for Fast and Accurate Semantic Object Segmentation.

We propose Depth-to-Space Net (DTS-Net), an effective technique for semantic segmentation using the efficient sub-pixel convolutional neural network. This technique is inspired by depth-to-space (DTS) image reconstruction, which was originally used for image and video super-resolution tasks, combined with a mask enhancement filtration technique based on multi-label classification, namely, Nearest Label Filtration. In the proposed technique, we employ depth-wise separable convolution-based architectures. We propose both a deep network, that is, DTS-Net, and a lightweight network, DTS-Net-Lite, for real-time semantic segmentation; these networks employ Xception and MobileNetV2 architectures as the feature extractors, respectively. In addition, we explore the joint semantic segmentation and depth estimation task and demonstrate that the proposed technique can efficiently perform both tasks simultaneously, outperforming state-of-art (SOTA) methods. We train and evaluate the performance of the proposed method on the PASCAL VOC2012, NYUV2, and CITYSCAPES benchmarks. Hence, we obtain high mean intersection over union (mIOU) and mean pixel accuracy (Pix.acc.) values using simple and lightweight convolutional neural network architectures of the developed networks. Notably, the proposed method outperforms SOTA methods that depend on encoder–decoder architectures, although our implementation and computations are far simpler.

Deep Unfolding Network for Spatiospectral Image Super-Resolution

In this paper, we explore the spatiospectral image super-resolution (SSSR) task, <i>i.e.</i>, joint spatial and spectral super-resolution, which aims to generate a high spatial resolution hyperspectral image (HR-HSI) from a low spatial resolution multispectral image (LR-MSI). To tackle such a severely ill-posed problem, one straightforward but inefficient way is to sequentially perform a single image super-resolution (SISR) network followed by a spectral super-resolution (SSR) network in a two-stage manner or reverse order. In this paper, we propose a model-based deep learning network for SSSR task, named unfolding spatiospectral super-resolution network (US3RN), which not only uses closed-form solutions to solve SISR subproblem and SSR subproblem, but also has extremely small parameters (only 295 K). In specific, we reformulate the image degradation and incorporate the spatiospectral super-resolution (SSSR) model, which takes the observation models of SISR and SSR into consideration. Then we solve the model-based energy function via the alternative direction multiplier method (ADMM) technique. Finally, we unfold the iterative process of the ADMM algorithm into a multistage network. Therefore, US3RN combines the merits of interpretability and generality of model-based methods with the advantages of learning-based methods. The experimental results show that, compared with the two-step method, US3RN achieves better results both quantitatively and qualitatively, while sharply reducing the number of parameters and FLOPs. Source code will be available at <uri>https://github.com/junjun-jiang/US3RN</uri>.

Transformer-Based Multistage Enhancement for Remote Sensing Image Super-Resolution

Convolutional neural networks have made a great breakthrough in recent remote sensing image super-resolution (SR) tasks. Most of these methods adopt upsampling layers at the end of the models to perform enlargement, which ignores feature extraction in the high-dimension space, and thus, limits SR performance. To address this problem, we propose a new SR framework for remote sensing images to enhance the high-dimensional feature representation after the upsampling layers. We name the proposed method as a transformer-based enhancement network (TransENet), where transformers are introduced to exploit features at different levels. The core of the TransENet is a transformer-based multistage enhancement structure, which can be combined with traditional SR frameworks to fuse multiscale high-/low-dimension features. Specifically, in this structure, the encoders aim to embed the multilevel features in the feature extraction part and the decoders are used to fuse these encoded embeddings. Experimental results demonstrate that our proposed TransENet can improve super-resolved results and obtain superior performance over several state-of-the-art methods.

SRUNet: Stacked Reversed U-Shape Network for Lightweight Single Image Super-Resolution

In recent years, for relieving the heavy computational cost, lightweight models have been successfully applied to single image super-resolution (SISR) task. However, most lightweight models adopt low-resolution images as input and apply transposed convolution or subpixel convolution only in the tail of the model to reconstruct the super-resolution image which neglect to fully utilize the multi-scale features. To resolve this problem, we propose a stacked reversed U-shape network (SRUNet) to further extract and utilize the multi-scale features at different resolutions. In detail, SRUNet consists of shallow feature extraction (SFE), stacked reversed U-shape module (SRUM), multi-scale backward fusion module (MSBFM) and feature refinement module (FRM). Instead of upsampling the feature map at the tail of the model, we perform the upsampling and downsampling operation progressively and iteratively by the stacked reversed U-shape module to extract richer multi-scale features. Furthermore, for archiving better use of multi-scale features, a scale-wise dense connection with residual channel-wise attention and multi-scale backward fusion is added to the network. The fused super-resolved features are refined by the feature refinement module and reconstructed to the image. Extensive experiments demonstrate that our model can achieve competitive performance compared with the state-of-the-art methods. When scaling factor is 4, SRUNet achieves the highest SSIM performance in all benchmarks and it takes 29.2ms to process per image in Urban100 dataset.

Learning Deep Resonant Prior for Hyperspectral Image Super-Resolution

Hyperspectral image super-resolution (HSISR) task has been widely studied, and significant progress has been made by leveraging the deep convolution neural network (CNN) techniques. Nevertheless, the scarcity of training images hinders the research progress of HSISR task. Moreover, the differences in imaging conditions and the number of spectral bands among different datasets, make it very difficult to construct a unified deep neural network. In this paper, we first present a non-training based HSISR method based on deep prior knowledge, which captures the image prior to restore the high resolution image by using the intrinsic characteristics of CNN. Then, we append a special network input processing module onto the HSI super-resolution network to automatically adjust the structure of the input so that the choice of network structure is no longer limited, while the network design focuses on exploiting the spatial information of hyperspectral images and the correlation between spectral bands, making the method more suitable for HSISR tasks and greatly extending its applications. Extensive experiment results on the hyperspectral image datasets illustrate the effectiveness of the proposed method, and we have got comparable results with the state-of-the-art methods while requiring no training samples.

Learning Discrete Representations From Reference Images for Large Scale Factor Image Super-Resolution.

Image super-resolution (SR) task aims to recover high-resolution (HR) images from degraded low-resolution (LR) images, which has achieved great progress due to the recent advances of deep neural networks. Due to severe information loss of the LR images, it is more challenging to reconstruct high quality HR images at large scale factors, i. e., higher than 4× . Traditional reference image based SR methods usually perform patch matching to locate detailed texture from HR reference images which could provide fine details from similar image contents. But it suffers from difficulties in achieving good matching in the largely downscaled image space or feature space due to the ill-posed nature between LR and HR mapping. In this paper, we tackle this problem by exploiting fine details contained in reference HR images. Inspired by vector quantization (VQ), we propose a simple yet effective auto-encoder convolutional neural network (CNN) module to learn discrete representations of images. Furthermore, we propose to progressively learn pairs of cross-scale discrete feature representations using paired LR and HR reference images. The coarser scale of the discrete representation is responsible for encoding the global image structure while the paired finer scale of the discrete representation takes charge of capturing missing details in the finer image scale. During inference, continuous features of the test LR image are used as queries to retrieve finer scale discrete representations (value) by searching the nearest coarser scale discrete representations (key). Then, the queries and retrieved values are combined to progressively recover the HR image. Experimental results indicate that when compared with the state-of-the-art image SR models, the proposed method can achieve advanced performance in terms of both objective quality and subjective quality. The code will be available on URL: https://github.com/sunwj/refsr.

Image Super-Resolution Using Generative Adversarial Networks with Learned Degradation Operators

Image super-resolution is a research endeavour that has gained notoriety in computer vision. The research goal is to increase the spatial dimensions of an image using corresponding low-resolution and high-resolution image pairs to enhance the perceptual quality. The challenge of maintaining such perceptual quality lies in developing appropriate algorithms that learn to reconstruct higher-quality images from their lower-resolution counterparts. Recent methods employ deep learning algorithms to reconstruct textural details prevalent in low-resolution images. Since corresponding image pairs are non-trivial to collect, researchers attempt super-resolution by creating synthetic low-resolution representations of high-resolution images. Unfortunately, such methods employ ineffective downscaling operations to achieve synthetic low-resolution images. These methods fail to generalize well on real-world images that may suffer different degradations. A different angle is offered to solve the task of image super-resolution by investigating the plausibility of learning the degradational operation using generative adversarial networks. A two-stage generative adversarial network along with two architectural variations is proposed to solve the task of real-world super-resolution from low-resolution images with unknown degradations. It is demonstrated that learning to downsample images in a weakly supervised manner is an impactful and viable approach for super-resolution.

A Non-Local Enhanced Network for Image Restoration

Non-local modules have been widely studied in image restoration (IR) tasks since they can learn long-range dependencies to enhance local features. However, most existing non-local modules still focus on extracting long-range dependencies within a single image or feature map. On the other hand, most IR methods simply employ a single type of non-local module in the network. A combination of various types of non-local modules to enhance local features can be more effective. In this paper, we propose a batch-wise non-local module to explore richer non-local dependencies within images. Furthermore, we combine various non-local extractors (different attention modules) with the proposed batch-wise non-local module as the Enhanced Batch-wise Non-local Attentive module (EBNA). Besides exploring richer non-local information, we build the Non-local and Local Information extracting Block (NLIB), in which we combine the EBNA with DEformable-Convolution Block (DECB) to utilize richer non-local and adaptive local information. Finally, We embed the NLIB within a U-net-like structure and build the Non-local Enhanced Network (NLENet). Extensive experiments on synthetic image denoising, real image denoising, JPEG artifacts removal, and real image super resolution tasks demonstrate that our proposed network achieves state-of-the-art performance on several IR benchmark datasets.

Dual Learning-Based Graph Neural Network for Remote Sensing Image Super-Resolution

High-resolution (HR) remote sensing imagery plays a critical role in remote sensing image interpretation, and single image super-resolution (SISR) reconstruction technology is becoming increasingly valuable and significant. The state-of-the-art deep-learning-based SISR methods have demonstrated remarkable advantages, while reconstructing complex texture details still remains a big challenge. Besides, as a typical ill-posed inverse problem, how to determine the optimal solution is another important topic. To address these problems, in this work, a dual learning-based graph neural network (DLGNN) is proposed, in which the GNN is utilized to consider the self-similarity patches in remote sensing imagery by aggregating cross-scale neighboring feature patches, and dual learning strategy is adopted to refine the reconstruction results by constraining the mapping process in terms of the loss function, transferring the typical ill-posed problem to a well-posed one. Abundant experiments on 3K VEHICLE&#x005F;SR datasets and Massachusetts Roads demonstrate the validity and outstanding performance for remote sensing image super-resolution tasks compared with other state-of-the-art super-resolution construction methods. Code is available at https://github.com/CUG-RS/DLGNN.

Cross View Capture for Stereo Image Super-Resolution

Stereo image super-resolution exploits additional features from cross view image pairs for high resolution (HR) image reconstruction. Recently, several new methods have been proposed to investigate cross view features along epipolar lines to enhance the visual perception of recovered HR images. Despite the impressive performance of these methods, global contextual features from cross view images are left unexplored. In this paper, we propose a cross view capture network (CVCnet) for stereo image super-resolution by using both global contextual and local features extracted from both views. Specifically, we design a cross view block to capture diverse feature embeddings from the views in stereo vision. In addition, a cascaded spatial perception module is proposed to redistribute each location in feature maps according to the weight it occupies to make the extraction of features more effective. Extensive experiments demonstrate that our proposed CVCnet outperforms the state-of-the-art image super-resolution methods to achieve the best performance for stereo image super-resolution tasks. The source code is available at https://github.com/xyzhu1/CVCnet.

From Artifact Removal to Super-Resolution

Deep-learning-based super-resolution methods have been extensively studied and achieved significant performance with deep convolutional neural networks. However, the results still suffer from the ringing effect, especially in satellite image super-resolution tasks, due to the loss of image details in the satellite degradation process. In this paper, we build a novel satellite super-resolution framework by decomposing a high-resolution image into three components, i.e., low-resolution, artifact, and high-frequency information. Specifically, we propose an artifact removal network with a self-adaption difference convolution (SDC) to fully exploit the structure prior in the low-resolution image and predict the artifact map. Considering that the artifact map and the high-frequency map share a similar pattern, we introduce the supervised structure correction block (SSC) that establishes a bridge between the high-frequency generation process and the artifact removal process. Experimental results on satellite images demonstrate that the proposed method owns an improved tradeoff between the performance and the computational cost compared to existing state-of-the-art satellite and natural super-resolution methods. The source code is available at https://github.com/jiaming-wang/ARSRN.

Dynamic High-Pass Filtering and Multi-Spectral Attention for Image Super-Resolution.

Deep convolutional neural networks (CNNs) have pushed forward the frontier of super-resolution (SR) research. However, current CNN models exhibit a major flaw: they are biased towards learning low-frequency signals. This bias becomes more problematic for the image SR task which targets reconstructing all fine details and image textures. To tackle this challenge, we propose to improve the learning of high-frequency features both locally and globally and introduce two novel architectural units to existing SR models. Specifically, we propose a dynamic highpass filtering (HPF) module that locally applies adaptive filter weights for each spatial location and channel group to preserve high-frequency signals. We also propose a matrix multi-spectral channel attention (MMCA) module that predicts the attention map of features decomposed in the frequency domain. This module operates in a global context to adaptively recalibrate feature responses at different frequencies. Extensive qualitative and quantitative results demonstrate that our proposed modules achieve better accuracy and visual improvements against state-of-the-art methods on several benchmark datasets.

Channel rearrangement multi-branch network for image super-resolution

In image super-resolution task, the existing convolutional neural network methods proceed to increase the number of network layers and filters to achieve better performance. Nevertheless, such approaches not only lead to a dramatic increase in network parameters, but also restrict the practical application of super-resolution methods. In order to address this problem, we propose a lightweight model called Channel Rearrangement Multi-Branch Network (CRMBN) for super-resolution of single images. Specifically, we design a channel rearrangement algorithm based on the total variation of the feature maps. It is inspired by the fact that the feature maps with high total variation contain more high-frequency details, which are valuable for super-resolution reconstruction. Therefore, for the super-resolution networks, the more feature maps with high total variation, the better performance of reconstruction. The rearranged channels are divided into two groups, one with high total variation and the other with the opposite. In order to reduce the parameters while mining for more details, we send the features with low total variation to the deep network. Finally, high total variation features and deep features are fused to reconstruct high-resolution image. The experimental results show that compared with other lightweight models, the proposed method can offer a favorable improvement in terms of PSNR and SSIM.

Cross-Dimension Attention Guided Self-Supervised Remote Sensing Single-Image Super-Resolution

In recent years, the application of deep learning has achieved a huge leap in the performance of remote sensing image super-resolution (SR). However, most of the existing SR methods employ bicubic downsampling of high-resolution (HR) images to obtain low-resolution (LR) images and use the obtained LR and HR images as training pairs. This supervised method that uses ideal kernel (bicubic) downsampled images to train the network will significantly degrade performance when used in realistic LR remote sensing images, usually resulting in blurry images. The main reason is that the degradation process of real remote sensing images is more complicated. The training data cannot reflect the SR problem of real remote sensing images. Inspired by the self-supervised methods, this paper proposes a cross-dimension attention guided self-supervised remote sensing single-image super-resolution method (CASSISR). It does not require pre-training on a dataset, only utilizes the internal information reproducibility of a single image, and uses the lower-resolution image downsampled from the input image to train the cross-dimension attention network (CDAN). The cross-dimension attention module (CDAM) selectively captures more useful internal duplicate information by modeling the interdependence of channel and spatial features and jointly learning their weights. The proposed CASSISR adapts well to real remote sensing image SR tasks. A large number of experiments show that CASSISR has achieved superior performance to current state-of-the-art methods.

A Regularization by Denoising super-resolution method based on genetic algorithms

Increasing the resolution of an image is an actual and extensively studied problem in image processing. Recently, Regularization by Denoising (RED) showing that any inverse problem can be handled by sequentially applying image denoising steps, including the image super-resolution (SR) task, which facilitate the resolution of the encountered optimization problem. In this paper, we propose a new configuration of genetic algorithms to resolve the super-resolution problem using a Non-Local Means filter as a denoiser function with a rigorous proof of the existence of a unique minimizer. In fact, since the SR algorithms always skip the complex spatial interactions within images, a more consistent model is then needed. The use of the genetic algorithms with the RED techniques guaranteed, in high intensity of noise and blur, the convergence to the globally optimal solution. As a result, the proposed algorithm shows efficient and consistent results, in terms of edges and feature preservation, compared with other SR approaches.

Multi-scale Xception based depthwise separable convolution for single image super-resolution.

The main target of Single image super-resolution is to recover high-quality or high-resolution image from degraded version of low-quality or low-resolution image. Recently, deep learning-based approaches have achieved significant performance in image super-resolution tasks. However, existing approaches related with image super-resolution fail to use the features information of low-resolution images as well as do not recover the hierarchical features for the final reconstruction purpose. In this research work, we have proposed a new architecture inspired by ResNet and Xception networks, which enable a significant drop in the number of network parameters and improve the processing speed to obtain the SR results. We are compared our proposed algorithm with existing state-of-the-art algorithms and confirmed the great ability to construct HR images with fine, rich, and sharp texture details as well as edges. The experimental results validate that our proposed approach has robust performance compared to other popular techniques related to accuracy, speed, and visual quality.

An efficient image super resolution model with dense skip connections between complex filter structures in Generative Adversarial Networks

Generative Adversarial Network (GAN) based models have gained a lot of popularity due to their outstanding performance in image super resolution tasks. However, these networks have few inherent problems; such as, high computational complexity, large depth and vanishing gradient. Present work proposes a novel GAN based architecture, namely, Super Resolution with Inception Network (SRINet) to solve the above mentioned problems. Generator architecture of SRINet uses complex filter structure rather than the linear filter structure to increase the depth and width of network without increasing the computational cost. Complex filter settings, present in the architecture, helps it to attain locally distributed information along with hierarchical global information in an image. Hence, the proposed method approximates the most favorable sparse structures to foster the learning capability of network. Additionally, progressive two stage upscaling approach with dense skip connections are introduced in the generator architecture. This technique helps the proposed network to learn precise mapping to generate an output image from the low resolution image. To measure visual quality of an image, we use human visual system based visual information fidelity metric. Proposed method outperforms all the state-of-the-art methods qualitatively (perceptually) and quantitatively on other GAN based methods.

Deep Image Prior for Super Resolution of Noisy Image

Single image super-resolution task aims to reconstruct a high-resolution image from a low-resolution image. Recently, it has been shown that by using deep image prior (DIP), a single neural network is sufficient to capture low-level image statistics using only a single image without data-driven training such that it can be used for various image restoration problems. However, super-resolution tasks are difficult to perform with DIP when the target image is noisy. The super-resolved image becomes noisy because the reconstruction loss of DIP does not consider the noise in the target image. Furthermore, when the target image contains noise, the optimization process of DIP becomes unstable and sensitive to noise. In this paper, we propose a noise-robust and stable framework based on DIP. To this end, we propose a noise-estimation method using the generative adversarial network (GAN) and self-supervision loss (SSL). We show that a generator of DIP can learn the distribution of noise in the target image with the proposed framework. Moreover, we argue that the optimization process of DIP is stabilized when the proposed self-supervision loss is incorporated. The experiments show that the proposed method quantitatively and qualitatively outperforms existing single image super-resolution methods for noisy images.