Network For Image Super-resolution Research Articles

Thermal Image Super-Resolution Based on Lightweight Dynamic Attention Network for Infrared Sensors.

Infrared sensors capture infrared rays radiated by objects to form thermal images. They have a steady ability to penetrate smoke and fog, and are widely used in security monitoring, military, etc. However, civilian infrared detectors with lower resolution cannot compare with megapixel RGB camera sensors. In this paper, we propose a dynamic attention mechanism-based thermal image super-resolution network for infrared sensors. Specifically, the dynamic attention modules adaptively reweight the outputs of the attention and non-attention branches according to features at different depths of the network. The attention branch, which consists of channel- and pixel-wise attention blocks, is responsible for extracting the most informative features, while the non-attention branch is adopted as a supplement to extract the remaining ignored features. The dynamic weights block operates with 1D convolution instead of the full multi-layer perceptron on the global average pooled features, reducing parameters and enhancing information interaction between channels, and the same structure is adopted in the channel attention block. Qualitative and quantitative results on three testing datasets demonstrate that the proposed network can superior restore high-frequency details while improving the resolution of thermal images. And the lightweight structure of the proposed network with lower computing cost can be practically deployed on edge devices, effectively improving the imaging perception quality of infrared sensors.

An efficient multi-scale learning method for image super-resolution networks

The image super-resolution (SR) operation holds multiple solutions with the one-to-many mapping from low-resolution (LR) to high-resolution (HR) space. However, the SR of different scales for the same image is usually regarded as independent tasks in the existing SR networks. Therefore, these networks are inflexible to effectively utilize feature learning experience and require much more computing time to recover HR images in higher resolutions. Recent arbitrary scale SR methods still cannot solve these problems. To efficiently and effectively recover HR images, this paper presents an efficient multi-scale learning method for image SR networks based on a novel self-generating (SG) mechanism. This method (briefly named SG-SR) utilizes the feature learning results of SR networks to generate upscale filters by using the novel SG upscale module, which is proposed to replace the traditional upscale module. For each scale factor, the SG upscale module provides the corresponding amount of the spatial weights to filter the LR tensor and then converts filtered tensors with the original tensor to corresponding HR images. The proposed method is evaluated through extensive experiments and compared with state-of-the-art (SOTA) methods on widely used benchmark datasets. The experimental results show that our method has superior performance compared with SOTA methods, and the SG upscale module can improve the performance of existing SR networks effectively. What is more, our module has a much less calculation cost than the other upscale modules.

Dense-Gated Network for Image Super-Resolution

Dense-Gated Network for Image Super-Resolution

Real-Time Video Super-Resolution with Spatio-Temporal Modeling and Redundancy-Aware Inference.

Video super-resolution aims to generate high-resolution frames from low-resolution counterparts. It can be regarded as a specialized application of image super-resolution, serving various purposes, such as video display and surveillance. This paper proposes a novel method for real-time video super-resolution. It effectively exploits spatial information by utilizing the capabilities of an image super-resolution model and leverages the temporal information inherent in videos. Specifically, the method incorporates a pre-trained image super-resolution network as its foundational framework, allowing it to leverage existing expertise for super-resolution. A fast temporal information aggregation module is presented to further aggregate temporal cues across frames. By using deformable convolution to align features of neighboring frames, this module takes advantage of inter-frame dependency. In addition, it employs a hierarchical fast spatial offset feature extraction and a channel attention-based temporal fusion. A redundancy-aware inference algorithm is developed to reduce computational redundancy by reusing intermediate features, achieving real-time inferring speed. Extensive experiments on several benchmarks demonstrate that the proposed method can reconstruct satisfactory results with strong quantitative performance and visual qualities. The real-time inferring ability makes it suitable for real-world deployment.

Lightweight subpixel sampling network for image super-resolution

Lightweight subpixel sampling network for image super-resolution

Global Aligned Structured Sparsity Learning for Efficient Image Super-Resolution.

Efficient image super-resolution (SR) has witnessed rapid progress thanks to novel lightweight architectures or model compression techniques (e.g., neural architecture search and knowledge distillation). Nevertheless, these methods consume considerable resources or/and neglect to squeeze out the network redundancy at a more fine-grained convolution filter level. Network pruning is a promising alternative to overcome these shortcomings. However, structured pruning is known to be tricky when applied to SR networks because the extensive residual blocks demand the pruned indices of different layers to be the same. Besides, the principled determination of proper layerwise sparsities remains challenging too. In this paper, we present Global Aligned Structured Sparsity Learning (GASSL) to resolve these problems. GASSL has two major components: Hessian-Aided Regularization (HAIR) and Aligned Structured Sparsity Learning (ASSL). HAIR is a regularization-based sparsity auto-selection algorithm with Hessian considered implicitly. A proven proposition is introduced to justify its design. ASSL is for physically pruning SR networks. Particularly, a new penalty term Sparsity Structure Alignment (SSA) is proposed to align the pruned indices of different layers. With GASSL, we design two new efficient single image SR networks of different architecture genres, pushing the efficiency envelope of SR models one step forward. Extensive results demonstrate the merits of GASSL over other recent counterparts.

Residual shuffle attention network for image super-resolution

Residual shuffle attention network for image super-resolution

BCSR: toward arbitrarily oriented text image super-resolution via adaptive Bezier curve network

Although existing super-resolution networks based on deep learning have obtained good results, it is still challenging to achieve an ideal visual effect for irregular texts, especially spatially deformed ones. In this paper, we propose a robust Bezier Curve-based image super-resolution network (BCSR), which can efficiently handle the degradation caused by deformations. Firstly, the arbitrarily shaped text is adaptively fitted by a parameterized Bezier curve, aiming to convert a curved text box into an annotated text box. Then, we design a BezierAlign layer to calibrate between the extracted features and the input image. By importing the extracted text prior information, the accuracy of the super-resolution network can be significantly improved. It is worth highlighting that we propose a kind of text prior loss that enables the text prior image and the super-resolution text image to achieve cooperation enhancement. Extensive experiments on several standard scene text datasets demonstrate that our proposed model achieves desirable objective evaluation results and further immensely helps downstream tasks related to text recognition, especially in text instances with multi-orientation and curved shapes.

FSR: A General Frequency-Oriented Framework to Accelerate Image Super-resolution Networks

Deep neural networks (DNNs) have witnessed remarkable achievement in image super-resolution (SR), and plenty of DNN-based SR models with elaborated network designs have recently been proposed. However, existing methods usually require substantial computations by operating in spatial domain. To address this issue, we propose a general frequency-oriented framework (FSR) to accelerate SR networks by considering data characteristics in frequency domain. Our FSR mainly contains dual feature aggregation module (DFAM) to extract informative features in both spatial and transform domains, followed by a four-path SR-Module with different capacities to super-resolve in the frequency domain. Specifically, DFAM further consists of a transform attention block (TABlock) and a spatial context block (SCBlock) to extract global spectral information and local spatial information, respectively, while SR-Module is a parallel network container that contains four to-be-accelerated branches. Furthermore, we propose an adaptive weight strategy for a trade-off between image details recovery and visual quality. Extensive experiments show that our FSR can save FLOPs by almost 40% while reducing inference time by 50% for other SR methods (e.g., FSRCNN, CARN, SRResNet and RCAN). Code is available at https://github.com/THU-Kingmin/FSR.

Context-aware lightweight remote-sensing image super-resolution network.

In recent years, remote-sensing image super-resolution (RSISR) methods based on convolutional neural networks (CNNs) have achieved significant progress. However, the limited receptive field of the convolutional kernel in CNNs hinders the network's ability to effectively capture long-range features in images, thus limiting further improvements in model performance. Additionally, the deployment of existing RSISR models to terminal devices is challenging due to their high computational complexity and large number of parameters. To address these issues, we propose a Context-Aware Lightweight Super-Resolution Network (CALSRN) for remote-sensing images. The proposed network primarily consists of Context-Aware Transformer Blocks (CATBs), which incorporate a Local Context Extraction Branch (LCEB) and a Global Context Extraction Branch (GCEB) to explore both local and global image features. Furthermore, a Dynamic Weight Generation Branch (DWGB) is designed to generate aggregation weights for global and local features, enabling dynamic adjustment of the aggregation process. Specifically, the GCEB employs a Swin Transformer-based structure to obtain global information, while the LCEB utilizes a CNN-based cross-attention mechanism to extract local information. Ultimately, global and local features are aggregated using the weights acquired from the DWGB, capturing the global and local dependencies of the image and enhancing the quality of super-resolution reconstruction. The experimental results demonstrate that the proposed method is capable of reconstructing high-quality images with fewer parameters and less computational complexity compared with existing methods.

Cross-View Attention Interaction Fusion Algorithm for Stereo Super-Resolution

In the process of stereo super-resolution reconstruction, in addition to the richness of the extracted feature information directly affecting the texture details of the reconstructed image, the texture details of the corresponding pixels between stereo image pairs also have an important impact on the reconstruction accuracy in the process of network learning. Therefore, aiming at the information interaction and stereo consistency of stereo image pairs, a cross-view attention interaction fusion stereo super-resolution algorithm is proposed. Firstly, based on parallax attention mechanism and triple attention mechanism, an attention stereo fusion module is constructed. The attention stereo fusion module is inserted between different levels of two single image super-resolution network branches, and the attention weight is calculated through the cross dimensional interaction of the three branches. It makes full use of the ability of single image super-resolution network to extract single view information and further maintaining the stereo consistency between stereo image pairs. Then, an enhanced cross-view interaction strategy including three fusion methods is proposed. Specifically, the vertical sparse fusion method is used to integrate the interior view information of different levels in the two single image super-resolution sub branches, the horizontal dense fusion method is used to connect the adjacent attention stereo fusion modules and the constraint between stereo image consistency is further strengthened in combination with the feature fusion method. Finally, the experimental results on Flickr 1024, Middlebury and KITTI benchmark datasets show that the proposed algorithm is superior to the existing stereo image super-resolution methods in quantitative measurement and qualitative visual quality while maintaining the stereo consistency of image pairs.

Fast CNN enhancement using channel attention and residual networks for image super-resolution

Single image super-resolution (SISR) refers to the process of reconstructing a high-resolution (HR) image from a low-resolution (LR) input image. Deep learning super-resolution algorithms have widely been used to solve SISR tasks. However, the demanding computation cost and memory occupation incurred through training the deep learning models has been hindering its real-world application. In this paper, we rebuild FSRCNN and apply it to solve SISR tasks. Firstly, we change the original training dataset to RealSR, a larger dataset consisting of real-world images. Secondly, channel attention and residual blocks have been applied to the mapping layers and important parameters including learning rate and optimizer have been reset. Thirdly, we change the cost function from loss to loss and replace the activation function from parametric rectified linear unit (PReLU) to exponential linear unit (ELU), to verify the discrepancies between different loss functions and activation functions. Finally, we compare the rebuilt models with the official FSRCNN based on the Peak signal-to-noise ratio(PSNR) and thestructural similarity index measure(SSIM) on three common test datasets. The original model achieves better performance on all the test datasets across different scale factors while the rebuilt models show better generalization capability. Our analyses illustrate that residual blocks can slightly promote model performance while different loss functions and activation functions do not generate an evident impact on the rebuilt model.

GCPAN: an adaptive global cross-scale prior attention network for image super-resolution

GCPAN: an adaptive global cross-scale prior attention network for image super-resolution

Deep primitive convolutional neural network for image super resolution

Deep primitive convolutional neural network for image super resolution

HIFGAN: A High-Frequency Information-Based Generative Adversarial Network for Image Super-Resolution

Since the neural network was introduced into the super-resolution (SR) field, many SR deep models have been proposed and have achieved excellent results. However, there are two main drawbacks: one is that the methods based on the best peak-signal-to-noise ratio (PSNR) do not have enough comfortable visual quality; the other is that although the SR models based on generative adversarial network (GAN) have satisfactory visual quality, the structure of the reconstructed image has apparent defects. Therefore, according to the characteristics that human eyes are sensitive to high-frequency components in images, this article proposes an improved image SRGAN model based on high-frequency information fusion (HIFGAN). It builds a feature extraction network for high-frequency information fusion by designing a lightweight spatial attention module and improving the network architecture of enhanced super-resolution GAN (ESRGAN). It makes the generator in the GAN network have better feature recovery ability, reduces the dependence of the later training on the decider and loss function, and makes the generated image structure more consistent with the real situation. In addition, we build a high-frequency loss function to optimize the training of the generator network. Detailed experimental results show that HIFGAN performs excellently in both objective criterion evaluation and subjective visual effect. Compared with the state-of-the-art GAN-based SR networks, the reconstructed image by our model is more precise and complete in texture details.

Spectral Correlation and Spatial High–Low Frequency Information of Hyperspectral Image Super-Resolution Network

Hyperspectral images (HSIs) generally contain tens or even hundreds of spectral segments within a specific frequency range. Due to the limitations and cost of imaging sensors, HSIs often trade spatial resolution for finer band resolution. To compensate for the loss of spatial resolution and maintain a balance between space and spectrum, existing algorithms were used to obtain excellent results. However, these algorithms could not fully mine the coupling relationship between the spectral domain and spatial domain of HSIs. In this study, we presented a spectral correlation and spatial high–low frequency information of a hyperspectral image super-resolution network (SCSFINet) based on the spectrum-guided attention for analyzing the information already obtained from HSIs. The core of our algorithms was the spectral and spatial feature extraction module (SSFM), consisting of two key elements: (a) spectrum-guided attention fusion (SGAF) using SGSA/SGCA and CFJSF to extract spectral–spatial and spectral–channel joint feature attention, and (b) high- and low-frequency separated multi-level feature fusion (FSMFF) for fusing the multi-level information. In the final stage of upsampling, we proposed the channel grouping and fusion (CGF) module, which can group feature channels and extract and merge features within and between groups to further refine the features and provide finer feature details for sub-pixel convolution. The test on the three general hyperspectral datasets, compared to the existing hyperspectral super-resolution algorithms, suggested the advantage of our method.

A sparse lightweight attention network for image super-resolution

A sparse lightweight attention network for image super-resolution

Relation-consistency graph convolutional network for image super-resolution

Relation-consistency graph convolutional network for image super-resolution

HighBoostNet: a deep light-weight image super-resolution network using high-boost residual blocks

HighBoostNet: a deep light-weight image super-resolution network using high-boost residual blocks

RSHAN: Image super-resolution network based on residual separation hybrid attention module

Transformer has become one of the main architectures in deep learning, showing impressive performance in various vision tasks, especially for image super-resolution (SR). However, due to the usage of high-resolution input images, most current Transformer-based image super-resolution models have a large number of parameters and high computational complexity. Moreover, some components employed in the Transformer may be redundant for SR tasks, which may limit the SR performance. In this work, we propose an efficient and concise model for image super-resolution tasks termed Residual Separation Hybrid Attention Network (RSHAN), which aims to solve the problems of redundant components and insufficient ability to extract high-frequency information of Transformer. Specifically, we present the Residual Separation Hybrid Attention Module (RSHAM) which fuses the local features extracted by the convolutional neural network (CNN) branch and the long-range dependencies extracted by Transformers to improve the performance of RSHAN. Extensive experiments on numerous benchmark datasets show that the proposed method outperforms state-of-the-art SR methods by up to 0.11 dB in peak signal-to-noise ratio (PSNR) metric, while the computational complexity and the inference time is reduced by 5% and 10%, respectively.