Network For Single Image Super-resolution Research Articles

Multi-scale convolutional attention network for lightweight image super-resolution

Convolutional neural network (CNN) based methods have recently achieved extraordinary performance in single image super-resolution (SISR) tasks. However, most existing CNN-based approaches increase the model’s depth by stacking massive kernel convolutions, bringing expensive computational costs and limiting their application in mobile devices with limited resources. Furthermore, large kernel convolutions are rarely used in lightweight super-resolution designs. To alleviate the above problems, we propose a multi-scale convolutional attention network (MCAN), a lightweight and efficient network for SISR. Specifically, a multi-scale convolutional attention (MCA) is designed to aggregate the spatial information of different large receptive fields. Since the contextual information of the image has a strong local correlation, we design a local feature enhancement unit (LFEU) to further enhance the local feature extraction. Extensive experimental results illustrate that our proposed MCAN can achieve better performance with lower model complexity compared with other state-of-the-art lightweight methods.

Single-Image Super-Resolution Challenges: A Brief Review

Single-image super-resolution (SISR) is an important task in image processing, aiming to achieve enhanced image resolution. With the development of deep learning, SISR based on convolutional neural networks has also gained great progress, but as the network deepens and the task of SISR becomes more complex, SISR networks become difficult to train, which hinders SISR from achieving greater success. Therefore, to further promote SISR, many challenges have emerged in recent years. In this review, we briefly review the SISR challenges organized from 2017 to 2022 and focus on the in-depth classification of these challenges, the datasets employed, the evaluation methods used, and the powerful network architectures proposed or accepted by the winners. First, depending on the tasks of the challenges, the SISR challenges can be broadly classified into four categories: classic SISR, efficient SISR, perceptual extreme SISR, and real-world SISR. Second, we introduce the datasets commonly used in the challenges in recent years and describe their characteristics. Third, we present the image evaluation methods commonly used in SISR challenges in recent years. Fourth, we introduce the network architectures used by the winners, mainly to explore in depth where the advantages of their network architectures lie and to compare the results of previous years’ winners. Finally, we summarize the methods that have been widely used in SISR in recent years and suggest several possible promising directions for future SISR.

Learnable Nonlocal Contrastive Network for Single Image Super-Resolution

Single image super-resolution (SISR) aims to recover a high-resolution image from a single low-resolution image. In recent years, SISR methods based on deep convolutional neural networks have achieved remarkable success, and some methods further improve the performance of the SISR model by introducing nonlocal attention into the model. However, most SISR methods that introduce nonlocal attention focus on more complex attention mechanisms and only use fixed functions for measurement when exploring image similarity. In addition, the model penalizes the algorithm in terms of loss when the output predicted by the model does not match the target data, even if this output is a potentially valid solution. To this end, we propose learnable nonlocal contrastive attention (LNLCA), which flexibly aggregates image features while maintaining linear computational complexity. Then, we introduce the adaptive target generator (ATG) model to address the problem of the single model training mode. Based on LNLCA, we construct a learnable nonlocal contrastive network (LNLCN). The experimental results demonstrate the effectiveness of the algorithm, which produces reconstructed images with more natural texture details.

Fast Non-Local Attention network for light super-resolution

Although convolutional neural network-based methods have achieved significant performance improvement for Single Image Super-Resolution (SISR), their vast computational cost hinders real-world environment application. Thus, the interest in light networks for SISR is rising. Since existing SISR light models mainly focus on extracting fine local features using convolution operation, they have a limitation in that networks hardly capture global information. To capture the long-range dependency, Non-Local (NL) attention and Transformers have been explored in the SISR task. However, they are still suffering from a balancing problem between performance and computational cost. In this paper, we propose Fast Non-Local attention NETwork (FNLNET) for a super light SISR, which can capture the global representation. To acquire global information, we propose The Fast Non-Local Attention (FNLA) module that has low computational complexity while capturing global representation that reflects long-distance relationships between patches. Then, FNLA requires only 16 times lower computational cost than conventional NL networks while improving performance. In addition, we propose a powerful module called Global Self-Intension Mining (GSIM) that fuses the multi-information resources such as local, and global representation. Our FNLNET shows outstanding performance with fewer parameters and computational costs in the experiments on the benchmark datasets against state-of-the-art light SISR models.

Image super-resolution reconstruction algorithm based on EDSR and improved SRGAN

Nowadays Image super-resolution reconstruction algorithm has become the dominant research aspect of computer vision, which is about generate high resolution composite image using low resolution original image. SRCNN(Super-Resolution Convolutional Neural Network) algorithm is the pioneer to apply deep learning to image super-resolution reconstruction, while SRGAN(Super-Resolution using Generative Adversarial Network) further explores the fusion of deep learning domain and image super-resolution reconstruction on grounds of SRCNN algorithm, and uses generative adversarial network to fit the extra information after image resolution improvement. Aiming at the defects of SRResNet generation network in SRGAN algorithm, this paper tries to improve the generator in its generation adversarial network by referring to the idea of ENSR(Enhanced Deep Residual Networks for Single Image Super-Resolution) algorithm, and obtains obvious effect. The improved algorithm has a significant improvement in performance.

Single-image super-resolution via dynamic multi-task learning and multi-level mutual feature fusion

In recent years, deep neural networks have exhibited promising performance in the single-image super-resolution (SISR) task by designing complex network structures from the low-resolution (LR) images to the high-resolution (HR) images and learning the mapping function. However, the SISR task is a widely known ill-posed problem. Since it may map one LR image input to multiple HR images, there are always undesired structural distortions in the recovered SR images. We creatively propose a SISR network with dynamic multi-task learning and multi-level mutual feature fusion (DMMFSR), alleviating structural distortion problems while maintaining more edge structure. Specifically, we utilize a DML block (DMLB) to train low-frequency and high-frequency image information extraction networks so the two networks can mutually promote each other. In addition, we also use a multi-level mutual feature fusion block to fuse image and edge features at different levels. Extensive experiments demonstrate the effectiveness of our proposed DMMFSR. Meanwhile, our model achieves comparable peak signal-to-noise ratio and structural similarity scores on multiple SR benchmarks compared with state-of-the-art SR methods.

Dual-branch feature learning network for single image super-resolution

Dual-branch feature learning network for single image super-resolution

An efficient lightweight network for single image super-resolution

The outstanding performance of convolutional neural networks (CNNs) shown in single image super-resolution (SISR) strongly depends on network’s depth, which hampers its application in low-power computing devices. In this paper, a lightweight and efficient network (LESR) is proposed for SISR by constructing the shallow feature extraction block (SFBlock), the cascaded sparse mask blocks (SMBlocks) and the feature fusion block (FFBlock). The SFBlock efficiently extracts global informative features from the original low resolution image using sparse self-attention, SMBlock skips the redundant computation in extracted features, and more meaningful information is distilled for the sequential reconstruction block by the FFBlock. In addition, a recently proposed activation function called ACON-C is used to replace the ReLU function to ease the training difficulty. Extensive experiments show that our proposed network performs better than most advanced lightweight SISR algorithms with comparable parameters and less FLOPs on benchmark database for ×2/3/4 SISR.

A lightweight generative adversarial network for single image super-resolution

A lightweight generative adversarial network for single image super-resolution

Post-trained convolution networks for single image super-resolution

A new method is proposed to increase the accuracy of the state-of-the-art single image super-resolution (SISR) using novel training procedure. The proposed method, named post-trained convolutional neural network (CNN), is carried out stochastic dual simplex algorithm (SDSA) in the last reconstruction layer. The method utilizes contextual information to update the last reconstruction layer of CNN. The extracted contextual information is projected to the last reconstructed layer by optimized weights and the bias is managed through SDSA. Post-trained CNN is applied to the very deep super-resolution (VDSR) method to show its performance. The quantitative and visual results demonstrate that the proposed post-trained VDSR (PTVDSR) exhibits excellent and competitive performance when compared with the VDSR and other super-resolution methods.

SIT-SR 3D: Self-supervised slice interpolation via transfer learning for 3D volume super-resolution

We present SIT-SR 3D, a novel self-supervised method for 3D single image super-resolution (SISR). Scaling 2D SISR networks to 3D SISR requires code redesign, high computing resources, and 3D ground-truth. However, we circumvent this by (1) using a pre-trained 2D SISR for indirect supervision and (2) using a novel consistency loss to learn frame interpolation. Any pre-trained state of the art 2D SISR method can replace the 2D SISR used in SIT-SR 3D, thus transferring the merits of 2D to 3D and ensuring modularity. We trained two end-to-end 3D baselines in a supervised setting; a 3D RRDBNet trained only with L1 loss and a 3D ESRGAN trained with adversarial and perceptual loss. We show that the proposed pipeline's self-supervised version is qualitatively better than the baselines. When trained in a supervised setting, SIT-SR 3D achieves better PSNR than its counterparts. Furthermore, our pipeline uses fewer parameters compared to the baselines. We demonstrate our results on an open-source digital rock CT dataset. Our code and pre-trained models will be made publicly available.

Lightweight Asymmetric Convolutional Distillation Network for Single Image Super-Resolution

Recent single image super-resolution methods based on various complex deep neural networks have achieved remarkable success. However, these methods require a large amount of computational overhead while improving performance, and thus are difficult to apply to mobile devices in real-world scenarios. In this letter, we design an efficient asymmetric convolutional distillation block (ACDB). Especially in this block, introducing an asymmetric convolution block (ACB) and reusing shallow distillation features can effectively improve the performance of the model and reduce the model complexity. Our model achieves efficient performance while maintaining low complexity.>

Joint Wavelet Sub-Bands Guided Network for Single Image Super-Resolution

Joint Wavelet Sub-Bands Guided Network for Single Image Super-Resolution

Wavelet detail perception network for single image super-resolution

Single image super-resolution (SR) is an important topic in computer vision because of its ability to generate high-resolution (HR) images. Traditional SR methods do not pay attention to high-frequency detail perception in the reconstruction process, resulting in unrealistic high-frequency details of images. To address the problem of over-smoothing of details, a novel wavelet detail perception network (WDPNet) is proposed in this study. Different from traditional SR methods that directly restore high-resolution images, the proposed WDPNet decomposes images into low-frequency and high-frequency sub-images by wavelet transform and then uses different models to train these sub-images. Moreover, low-frequency structures are also provided to the high-frequency model to further recover and enhance high-frequency details through the proposed low-to-high information delivery (L2HID) and detail perception enhancement (DPE) mechanisms. Finally, the low-frequency and high-frequency models are fused and weighted to different degrees to enhance image details further. Compared with the state-of-the-art methods, the experimental results show that the proposed WDPNet achieves better performance and visual results in image detail perception.

Adaptable physics-based super-resolution for electron backscatter diffraction maps

In computer vision, single-image super-resolution (SISR) has been extensively explored using convolutional neural networks (CNNs) on optical images, but images outside this domain, such as those from scientific experiments, are not well investigated. Experimental data is often gathered using non-optical methods, which alters the metrics for image quality. One such example is electron backscatter diffraction (EBSD), a materials characterization technique that maps crystal arrangement in solid materials, which provides insight into processing, structure, and property relationships. We present a broadly adaptable approach for applying state-of-art SISR networks to generate super-resolved EBSD orientation maps. This approach includes quaternion-based orientation recognition, loss functions that consider rotational effects and crystallographic symmetry, and an inference pipeline to convert network output into established visualization formats for EBSD maps. The ability to generate physically accurate, high-resolution EBSD maps with super-resolution enables high-throughput characterization and broadens the capture capabilities for three-dimensional experimental EBSD datasets.

A deep recursive multi-scale feature fusion network for image super-resolution

Recently, Convolutional Neural Networks (CNNs) have achieved great success in Single Image Super-Resolution (SISR). In particular, the recursive networks are now widely used. However, existing recursion-based SISR networks can only make use of multi-scale features in a layer-wise manner. In this paper, a Deep Recursive Multi-Scale Feature Fusion Network (DRMSFFN) is proposed to address this issue. Specifically, we propose a Recursive Multi-Scale Feature Fusion Block (RMSFFB) to make full use of multi-scale features. Besides, a Progressive Feature Fusion (PFF) technique is proposed to take advantage of the hierarchical features from the RMSFFB in a global manner. At the reconstruction stage, we use a deconvolutional layer to upscale the feature maps to the desired size. Extensive experimental results on benchmark datasets demonstrate the superiority of the proposed DRMSFFN in comparison with the state-of-the-art methods in both quantitative and qualitative evaluations.

Pyramid Attention “Zero-Shot” Network for Single-Image Superresolution

Single-image superresolution (SISR) is one of the requisite image processing methods used to reconstructs a high-resolution (HR) image from a low-resolution (LR) observation. Existing SISR methods mostly rely on supervised learning and are confined to specific training data, where implicit conformity matching of the LR images with their high-resolution (HR) counterparts is conducted. However, real LR images rarely obey these constraints, and the degradation process is much more intricate and unknown. In this paper, we proposed a pyramid attention zero-shot (PAZS) network for SISR that sufficiently explores the information hidden in an image itself by learning the patch distribution at a different scale of the image. The proposed pyramid generative model learns the patch distribution in the internal patch learning stage. Then, based on the learned intrinsic property, we explore the corresponding superresolution (SR) image by integrating the intrinsic information into a self-attention mechanism with a progressive generation style at test time. This mechanism consists of internal-external attention and a cross-scale guided fusion module as the connecting passageway. Thus, the PAZS architecture maintains both the global structure and the fine textures of the SR image. This allows SR to be performed without other training datasets and can be adapted to different settings for each image. Experiments on diverse datasets demonstrate that the proposed method outperforms other methods based on single-image training.

A very lightweight and efficient image super-resolution network

Deep convolutional neural networks significantly improve the performance of single image super-resolution (SISR). Generally, larger networks (i.e., deeper and wider) have better performance. However, larger networks require higher computing and storage costs, which limit their application on resource-constrained devices. Lightweight SISR networks with fewer parameters and smaller computational workloads are highly desirable. The key challenge is to obtain a better balance of model complexity and performance. In this paper, we propose a very lightweight and efficient SISR network. Our main contributions include: (1) Propose a frequency grouping fusion block (FGFB), which can better fuse high-/low-frequency feature information; (2) Propose a multi-way attention block (MWAB), which can exploit the multiple different clues of the feature information; (3) Propose a lightweight residual concatenation block (LRCB), which can combine the advantages of the residual connection and the channel concatenation; (4) Propose a lightweight convolutional block (LConv) for image super-resolution, which can significantly reduce the number of parameters; (5) Propose a progressive interactive group convolution (PIGC), which is more effective than the conventional group convolution. Extensive experimental results demonstrate that our method is significantly superior to other state-of-the-art methods currently available, with a better balance between model complexity and performance.

Dual attention mechanism network for lung cancer images super-resolution

Background and objectiveCurrently, the morbidity and mortality of lung cancer rank first among malignant tumors worldwide. Improving the resolution of thin-slice CT of the lung is particularly important for the early diagnosis of lung cancer screening. MethodsAiming at the problems of network training difficulty and low utilization of feature information caused by the deepening of network layers in super-resolution (SR) reconstruction technology, we propose the dual attention mechanism network for single image super-resolution (SISR). Firstly, the feature of a low-resolution image is extracted directly to retain the feature information. Secondly, several independent dual attention mechanism modules are constructed to extract high-frequency details. The introduction of residual connections can effectively solve the gradient disappearance caused by network deepening, and long and short skip connections can effectively enhance the data features. Furthermore, a hybrid loss function speeds up the network's convergence and improves image SR restoration ability. Finally, through the upsampling operation, the reconstructed high-resolution image is obtained. ResultsThe results on the Set5 dataset for 4 × enlargement show that compared with traditional SR methods such as Bicubic, VDSR, and DRRN, the average PSNR/SSIM is increased by 3.33 dB / 0.079, 0.41 dB / 0.007 and 0.22 dB / 0.006 respectively. The experimental data fully show that DAMN can better restore the image contour features, obtain higher PSNR, SSIM, and better visual effect. ConclusionThrough the DAMN reconstruction method, the image quality can be improved without increasing radiation exposure and scanning time. Radiologists can enhance their confidence in diagnosing early lung cancer, provide a basis for clinical experts to choose treatment plans, formulate follow-up strategies, and benefit patients in the early stage.

Single image super-resolution based on directional variance attention network

Recent advances in single image super-resolution (SISR) explore the power of deep convolutional neural networks (CNNs) to achieve better performance. However, most of the progress has been made by scaling CNN architectures, which usually raise computational demands and memory consumption. This makes modern architectures less applicable in practice. In addition, most CNN-based SR methods do not fully utilize the informative hierarchical features that are helpful for final image recovery. In order to address these issues, we propose a directional variance attention network (DiVANet), a computationally efficient yet accurate network for SISR. Specifically, we introduce a novel directional variance attention (DiVA) mechanism to capture long-range spatial dependencies and exploit inter-channel dependencies simultaneously for more discriminative representations. Furthermore, we propose a residual attention feature group (RAFG) for parallelizing attention and residual block computation. The output of each residual block is linearly fused at the RAFG output to provide access to the whole feature hierarchy. In parallel, DiVA extracts most relevant features from the network for improving the final output and preventing information loss along the successive operations inside the network. Experimental results demonstrate the superiority of DiVANet over the state of the art in several datasets, while maintaining relatively low computation and memory footprint. The code is available at https://github.com/pbehjatii/DiVANet.