Network For Image Super-resolution Research Articles

Optimization of semi-supervised generative adversarial network models: a survey

PurposeWith the development of intelligent technology, deep learning has made significant progress and has been widely used in various fields. Deep learning is data-driven, and its training process requires a large amount of data to improve model performance. However, labeled data is expensive and not readily available.Design/methodology/approachTo address the above problem, researchers have integrated semi-supervised and deep learning, using a limited number of labeled data and many unlabeled data to train models. In this paper, Generative Adversarial Networks (GANs) are analyzed as an entry point. Firstly, we discuss the current research on GANs in image super-resolution applications, including supervised, unsupervised, and semi-supervised learning approaches. Secondly, based on semi-supervised learning, different optimization methods are introduced as an example of image classification. Eventually, experimental comparisons and analyses of existing semi-supervised optimization methods based on GANs will be performed.FindingsFollowing the analysis of the selected studies, we summarize the problems that existed during the research process and propose future research directions.Originality/valueThis paper reviews and analyzes research on generative adversarial networks for image super-resolution and classification from various learning approaches. The comparative analysis of experimental results on current semi-supervised GAN optimizations is performed to provide a reference for further research.

Infrared Image Super-Resolution Network Utilizing the Enhanced Transformer and U-Net.

Infrared images hold significant value in applications such as remote sensing and fire safety. However, infrared detectors often face the problem of high hardware costs, which limits their widespread use. Advancements in deep learning have spurred innovative approaches to image super-resolution (SR), but comparatively few efforts have been dedicated to the exploration of infrared images. To address this, we design the Residual Swin Transformer and Average Pooling Block (RSTAB) and propose the SwinAIR, which can effectively extract and fuse the diverse frequency features in infrared images and achieve superior SR reconstruction performance. By further integrating SwinAIR with U-Net, we propose the SwinAIR-GAN for real infrared image SR reconstruction. SwinAIR-GAN extends the degradation space to better simulate the degradation process of real infrared images. Additionally, it incorporates spectral normalization, dropout, and artifact discrimination loss to reduce the potential image artifacts. Qualitative and quantitative evaluations on various datasets confirm the effectiveness of our proposed method in reconstructing realistic textures and details of infrared images.

A very lightweight image super-resolution network

Recently, ConvNeXt and blueprint separable convolution (BSConv) constructed from standard ConvNet modules have demonstrated competitive performance in advanced computer vision tasks. This paper proposes an efficient model (BCRN) based on BSConv and the ConvNeXt residual structure for single image super-resolution, which achieves superior performance with very low parametric numbers. Specifically, the residual block (BCB) of the BCRN utilizes the ConvNeXt residual structure and BSConv to significantly reduce the number of parameters. Within the residual block, enhanced spatial attention and contrast-aware channel attention modules are simultaneously introduced to prioritize valuable features within the network. Multiple residual blocks are then stacked to form the backbone network, with Dense connections utilized between them to enhance feature utilization. Our model boasts extremely low parameters compared to other state-of-the-art lightweight models, while experimental results on benchmark datasets demonstrate its excellent performance. The code will be available at https://github.com/kptx666/BCRN.

DHBSR: A deep hybrid representation-based network for blind image super resolution

Image super resolution involves enhancing the spatial resolution of low-quality images and improving their visual quality. As in many real-life situations, the image degradation process is unknown, performing the task of image super resolution in a blind manner is of paramount importance. Deep neural networks provide high performances for the task of blind image super resolution, in view of their end-to-end learning capability between the low-resolution images and their ground truth versions. Generally speaking, deep blind image super resolution networks initially estimate the parameters of the image degradation process, such as blurring kernel, and then use them for super-resolving the low-resolution images. In this paper, we develop a novel deep learning-based scheme for the task of blind image super resolution, in which the idea of leveraging the hybrid representations is utilized. Specifically, we employ the deterministic and stochastic representations of the blurring kernel parameters to train a deep blind super resolution network in an effective manner. The results of extensive experiments prove the effectiveness of various ideas used in the development of the proposed deep blind image super resolution network.

Super-resolution imaging of telescopic systems based on optical-neural network joint optimization

Abstract Optical telescopes are an important tool for acquiring optical information about
distant objects, and resolution is an important indicator that measures the ability to observe
object details. However, due to the effects of system aberration, atmospheric vortex, and
other factors, the observation image of ground-based telescopes is often degraded, resulting in
reduced resolution. This paper proposes an optical-neural network joint optimization method
to improve the resolution of the observed image by co-optimizing the point spread function
(PSF) of the telescopic system and the image super-resolution network. To improve the speed
of image reconstruction, we designed a generative adversarial net (LCR-GAN) with light
parameters, which is much faster than the latest unsupervised networks. To reconstruct the
PSF trained by the network in the optical path, a phase mask is introduced. It improves the
image reconstruction effect of LCR-GAN by reconstructing the point spread function that
best matches the network. The results of simulation and verification experiments show that
compared with the pure deep learning method, the super-resolution image reconstructed by
this method is rich in detail and easier to distinguish stars or stripes.

Frequency-Separated Attention Network for Image Super-Resolution

The use of deep convolutional neural networks has significantly improved the performance of super-resolution. Employing deeper networks to enhance the non-linear mapping capability from low-resolution (LR) to high-resolution (HR) images has inadvertently weakened the information flow and disrupted long-term memory. Moreover, overly deep networks are challenging to train, thus failing to exhibit the expressive capability commensurate with their depth. High-frequency and low-frequency features in images play different roles in image super-resolution. Networks based on CNNs, which should focus more on high-frequency features, treat these two types of features equally. This results in redundant computations when processing low-frequency features and causes complex and detailed parts of the reconstructed images to appear as smooth as the background. To maintain long-term memory and focus more on the restoration of image details in networks with strong representational capabilities, we propose the Frequency-Separated Attention Network (FSANet), where dense connections ensure the full utilization of multi-level features. In the Feature Extraction Module (FEM), the use of the Res ASPP Module expands the network’s receptive field without increasing its depth. To differentiate between high-frequency and low-frequency features within the network, we introduce the Feature-Separated Attention Block (FSAB). Furthermore, to enhance the quality of the restored images using heuristic features, we incorporate attention mechanisms into the Low-Frequency Attention Block (LFAB) and the High-Frequency Attention Block (HFAB) for processing low-frequency and high-frequency features, respectively. The proposed network outperforms the current state-of-the-art methods in tests on benchmark datasets.

Light field image super-resolution based on dual learning and deep Fourier channel attention.

Light field (LF) imaging has gained significant attention in the field of computational imaging due to its unique capability to capture both spatial and angular information of a scene. In recent years, super-resolution (SR) techniques based on deep learning have shown considerable advantages in enhancing LF image resolution. However, the inherent challenges of obtaining rich structural information and reconstructing complex texture details persist, particularly in scenarios where spatial and angular information are intricately interwoven. This Letter introduces a novel, to the best of our knowledge, approach for Disentangling LF Image SR Network (DLISN) by leveraging the synergy of dual learning and Fourier channel attention (FCA) mechanisms. Dual learning strategies are employed to enhance reconstruction results, addressing limitations in model generalization caused by the difficulty in acquiring paired datasets in real-world LF scenarios. The integration of FCA facilitates the extraction of high-frequency information associated with different structures, contributing to improved spatial resolution. Experimental results consistently demonstrate superior performance in enhancing the resolution of LF images.

CVGSR: Stereo image Super-Resolution with Cross-View guidance

The ability to capture the complementary information of stereo images is critical to the development of stereo image super-resolution. Most existing studies have attempted to integrate reliable stereo correspondence along the polar directions through parallax attention and various fusion strategies. However, most of these approaches ignore the large parallax differences in stereo images, resulting in poor performance of convolutional-based parallax attention in capturing the long-range dependencies between images. In this paper, we propose a novel cross-view guided stereo image super-resolution network (CVGSR) for reconstructing high-resolution stereo image pairs with rich texture details by fully exploiting the complementary nature of stereo image pairs. Specifically, we first deploy a cross-view interaction module (CVIM) to explore intra/cross-view dependencies from local to global to compensate for the incomplete compatibility of information between the left and right views. This module uses a progressive cross-guiding strategy to better merge features from occluded and non-occluded regions. Based on this, an efficient attention Transformer (EAT) is improved to activate more input information and further mine the cross-view complementarity. Furthermore, we design a texture loss to optimize the visual perceptual quality of reconstructed images with sharp boundaries and rich texture details. Extensive experiments on four stereo image datasets demonstrate that the proposed CVGSR achieves a competitive and excellent performance.

Multi-level feature interaction image super-resolution network based on convolutional nonlinear spiking neural model

Image super-resolution (ISR) is designed to recover lost detail information from low-resolution images, resulting in high-quality and high-definition high-resolution images. In the existing single ISR (SISR) methods based on convolutional neural networks (CNN), however, most of the models cannot effectively combine global and local information and are also easy to ignore the correlation between different hierarchical feature information. To address these problems, this study proposes a multi-level feature interactive image super-resolution network, which is constructed by the convolutional units inspired by nonlinear spiking mechanism in nonlinear spiking neural P systems, including shallow feature processing, deep feature extraction and fusion, and reconstruction modules. The different omni domain self-attention blocks are introduced to extract global information in the deep feature extraction and fusion stage and formed a feature enhancement module having a Transformer structure using a novel convolutional unit for extracting local information. Furthermore, to adaptively fuse features between different hierarchies, we design a multi-level feature fusion module, which not only can adaptively fuse features between different hierarchies, but also can better interact with contextual information. The proposed model is compared with 16 state-of-the-art or baseline models on five benchmark datasets. The experimental results show that the proposed model not only achieves good reconstruction performance, but also strikes a good balance between model parameters and performance.

Local Means Binary Networks for Image Super-Resolution.

The success of modern single image super-resolution (SISR) algorithms is inspired by the development of deep convolutional neural networks (CNNs). However, these CNN-based methods require considerable computation and complexity, making it impossible for these methods to perform real-time calculations in edge devices. Thus, lightweight model design has become a development trend in the super-resolution field, including pruning, quantization, and other methods. The 1-bit quantization is an extreme lightweight method which can reduce the calculation amount of the model in an extreme manner and is friendly to hardware such as edge devices. Most existing binary quantization approaches lead to a large information loss during forward propagation, especially in detailed color information (e.g., edge, texture, and contrast). The loss of color information makes modern binary methods unsuitable for SISR tasks. We think the loss occurs because these methods typically utilize a uniform threshold to quantize the weights and activations. Thus, in this article, we thoroughly analyze the difference between normal classification tasks and SISR tasks, and present a binarization scheme based on local means. The proposed method can maintain more detailed information in feature maps using dynamic thresholds during quantization. Specifically, each value in the full precision activations has a corresponding threshold during the quantization process, and those thresholds are determined by the full precision values of the surroundings. In addition, a gradient approximator is introduced to adaptively optimize the gradient for updating binary weights. We then verify the effectiveness of our method for training binary networks on several SISR benchmarks including VDSR and SRResNet. Experimental results show that the proposed method can outperform the state-of-the-art algorithms to obtain binary networks for image super-resolution with better peak signal-to-noise ratio (PSNR) values and visual quality.

Image Super-resolution Networks based on Structure Reparameterized Convolution

The traditional image super resolution method based on convolutional neural network generally brings two problems: First, the model has only a single scale of receptive field, which cannot use the image information in a smaller range; Second, the size of the feature map will become smaller and smaller in the process of continuous convolution, and it is necessary to continuously carry out edge zeroing operation to maintain the original size, which leads to the loss of part of the edge information. To solve the above two problems, we propose a network structure based on structure re-parameterized convolution, which sets a small convolution kernel in parallel beside a large convolution kernel in the same layer, trains the two cores simultaneously, and finally merges the two cores. The experimental results show that in this way, we make the large convolution kernel could capture smaller information, which not only improves the high-frequency details of the reconstructed image but also avoids frequent edge filling to reduce the information density, and effectively speeds up the reasoning speed after the model is re-parameterized. Compared with the advanced methods, our method achieves good results.

A blind image super-resolution network guided by kernel estimation and structural prior knowledge

The goal of blind image super-resolution (BISR) is to recover the corresponding high-resolution image from a given low-resolution image with unknown degradation. Prior related research has primarily focused effectively on utilizing the kernel as prior knowledge to recover the high-frequency components of image. However, they overlooked the function of structural prior information within the same image, which resulted in unsatisfactory recovery performance for textures with strong self-similarity. To address this issue, we propose a two stage blind super-resolution network that is based on kernel estimation strategy and is capable of integrating structural texture as prior knowledge. In the first stage, we utilize a dynamic kernel estimator to achieve degradation presentation embedding. Then, we propose a triple path attention groups consists of triple path attention blocks and a global feature fusion block to extract structural prior information to assist the recovery of details within images. The quantitative and qualitative results on standard benchmarks with various degradation settings, including Gaussian8 and DIV2KRK, validate that our proposed method outperforms the state-of-the-art methods in terms of fidelity and recovery of clear details. The relevant code is made available on this link as open source.

SwinDPSR: Dual-Path Face Super-Resolution Network Integrating Swin Transformer

Whether to use face priors in the face super-resolution (FSR) methods is a symmetry problem.Various face priors are used to describe the overall and local face features, making the generation of super-resolution face images expensive and laborious. FSR methods that do not require any prior information tend to focus too much on the local features of the face, ignoring the modeling of global information. To solve this problem, we propose a dual-path facial image super-resolution network (SwinDPSR) fused with Swin Transformer. The network does not require additional face priors, and it learns global face shape and local face components through two independent branches. In addition, the channel attention ECA module is used to aggregate the global and local face information in the above dual-path sub-networks, which can generate corresponding high-quality face images. The results of face super-resolution reconstruction experiments on public face datasets and a real-scene face dataset show that SwinDPSR is superior to previous advanced methods both in terms of visual effects and objective indicators. The reconstruction results are evaluated with four evaluation metrics: peak signal-to-noise ratio (PSNR), structural similarity (SSIM), learned perceptual image patch similarity (LPIPS), and mean perceptual score (MPS).

TARN: a lightweight two-branch adaptive residual network for image super-resolution

TARN: a lightweight two-branch adaptive residual network for image super-resolution

Residual SwinV2 transformer coordinate attention network for image super resolution

Swin Transformers have been designed and used in various image super-resolution (SR) applications. One of the recent image restoration methods is RSTCANet, which combines Swin Transformer with Channel Attention. However, for some channels of images that may carry less useful information or noise, Channel Attention cannot automatically learn the insignificance of these channels. Instead, it tries to enhance their expression capability by adjusting the weights. It may lead to excessive focus on noise information while neglecting more essential features. In this paper, we propose a new image SR method, RSVTCANet, based on an extension of Swin2SR. Specifically, to effectively gather global information for the channel of images, we modify the Residual SwinV2 Transformer blocks in Swin2SR by introducing the coordinate attention for each two successive SwinV2 Transformer Layers (S2TL) and replacing Multi-head Self-Attention (MSA) with Efficient Multi-head Self-Attention version 2 (EMSAv2) to employ the resulting residual SwinV2 Transformer coordinate attention blocks (RSVTCABs) for feature extraction. Additionally, to improve the generalization of RSVTCANet during training, we apply an optimized RandAugment for data augmentation on the training dataset. Extensive experimental results show that RSVTCANet outperforms the recent image SR method regarding visual quality and measures such as PSNR and SSIM.

Dual-space high-frequency learning for transformer-based MRI super-resolution

Background and ObjectiveMagnetic resonance imaging (MRI) can provide rich and detailed high-contrast information of soft tissues, while the scanning of MRI is time-consuming. To accelerate MR imaging, a variety of Transformer-based single image super-resolution methods are proposed in recent years, achieving promising results thanks to their superior capability of capturing long-range dependencies. Nevertheless, most existing works prioritize the design of transformer attention blocks to capture global information. The local high-frequency details, which are pivotal to faithful MRI restoration, are unfortunately neglected. MethodsIn this work, we propose a high-frequency enhanced learning scheme to effectively improve the awareness of high frequency information in current Transformer-based MRI single image super-resolution methods. Specifically, we present two entirely plug-and-play modules designed to equip Transformer-based networks with the ability to recover high-frequency details from dual spaces: 1) in the feature space, we design a high-frequency block (Hi-Fe block) paralleled with Transformer-based attention layers to extract rich high-frequency features; while 2) in the image intensity space, we tailor a high-frequency amplification module (HFA) to further refine the high-frequency details. By fully exploiting the merits of the two modules, our framework can recover abundant and diverse high-frequency information, rendering faithful MRI super-resolved results with fine details. ResultsWe integrated our modules with six Transformer-based models and conducted experiments across three datasets. The results indicate that our plug-and-play modules can enhance the super-resolution performance of all foundational models to varying degrees, surpassing the capabilities of existing state-of-the-art single image super-resolution networks. ConclusionComprehensive comparison of super-resolution images and high-frequency maps from various methods, clearly demonstrating that our module possesses the capability to restore high-frequency information, showing huge potential in clinical practice for accelerated MRI reconstruction.

Non-local self-attention network for image super-resolution

Non-local self-attention network for image super-resolution

Edge fusion back projection GAN for large scale face super resolution

Face super-resolution is an important low-level vision task that has wide applications. Existing deep-learning-based face super-resolution (SR) methods often optimize the image super-resolution network by directly minimizing the pixel or feature level distance between the synthetic low-resolution face and the ground truth face image. These methods usually generate blur or over smooth results and lack of high-frequency face details. These are especially true for making high super-resolution of faces. Say for example, a super-resolution of 16x, only 0.4 % of the reference data points are available. To address the problem, we propose a novel network with edge fusion, back projection, and GAN prior (EFBPGAN) which can significantly improve the visual quality and generate realistic faces. To further make use of the spatial information and keep the structural consistency, we have developed new edge fusion and spatial fusion modules. We also propose a back projection extensive based coarse to fine SR pipeline to suppress the distortion and artifacts caused by GAN. Much experimental work has been done, results of which show that our proposed EFBPGAN can outperform the state-of-the-art approaches not only on numerical metrics but also on subjective visual evaluations.

Multi-Scale Cross-Attention Fusion Network Based on Image Super-Resolution

In recent years, deep convolutional neural networks with multi-scale features have been widely used in image super-resolution reconstruction (ISR), and the quality of the generated images has been significantly improved compared with traditional methods. However, in current image super-resolution network algorithms, these methods need to be further explored in terms of the effective fusion of multi-scale features and cross-domain application of attention mechanisms. To address these issues, we propose a novel multi-scale cross-attention fusion network (MCFN), which optimizes the feature extraction and fusion process in structural design and modular innovation. In order to make better use of the attention mechanism, we propose a Pyramid Multi-scale Module (PMM) to extract multi-scale information by cascading. This PMM is introduced in MCFN and is mainly constructed by multiple multi-scale cross-attention modules (MTMs). To fuse the feature information of PMMs efficiently in both channel and spatial dimensions, we propose the cross-attention fusion module (CFM). In addition, an improved integrated attention enhancement module (IAEM) is inserted at the network’s end to enhance the correlation of high-frequency feature information between layers. Experimental results show that the algorithm significantly improves the reconstructed images’ edge information and texture details, and the benchmark dataset’s performance evaluation shows comparable performance to current state-of-the-art techniques.

Low-Dose CT Image Super-resolution Network with Noise Inhibition Based on Feedback Feature Distillation Mechanism.

Low-dose computed tomography (LDCT) has been widely used in medical diagnosis. In practice, doctors often zoom in on LDCT slices for clearer lesions and issues, while, a simple zooming operation fails to suppress low-dose artifacts, leading to distorted details. Therefore, numerous LDCT super-resolution (SR) methods have been proposed to promote the quality of zooming without the increase of the dose in CT scanning. However, there are still some drawbacks that need to be addressed in existing methods. First, the region of interest (ROI) is not emphasized due to the lack of guidance in the reconstruction process. Second, the convolutional blocks extracting fix-resolution features fail to concentrate on the essential multi-scale features. Third, a single SR head cannot suppress the residual artifacts. To address these issues, we propose an LDCT CT joint SR and denoising reconstruction network. Our proposed network consists of global dual-guidance attention fusion modules (GDAFMs) and multi-scale anastomosis blocks (MABs). The GDAFM directs the network to focus on ROI by fusing the extra mask guidance and average CT image guidance, while the MAB introduces hierarchical features through anastomosis connections to leverage multi-scale features and promote the feature representation ability. To suppress radial residual artifacts, we optimize our network using the feedback feature distillation mechanism (FFDM) which shares the backbone to learn features corresponding to the denoising task. We apply the proposed method to the 3D-IRCADB and PANCREAS datasets to evaluate its ability on LDCT image SR reconstruction. The experimental results compared with state-of-the-art methods illustrate the superiority of our approach with respect to peak signal-to-noise (PSNR), structural similarity (SSIM), and qualitative observations. Our proposed LDCT joint SR and denoising reconstruction network has been extensively evaluated through ablation, quantitative, and qualitative experiments. The results demonstrate that our method can recover noise-free and detail-sharp images, resulting in better reconstruction results. Code is available at https://github.com/neu-szy/ldct_sr_dn_w_ffdm .