Infrared Image Super-resolution Research Articles

Infrared Image Super-Resolution Network Utilizing the Enhanced Transformer and U-Net.

Infrared images hold significant value in applications such as remote sensing and fire safety. However, infrared detectors often face the problem of high hardware costs, which limits their widespread use. Advancements in deep learning have spurred innovative approaches to image super-resolution (SR), but comparatively few efforts have been dedicated to the exploration of infrared images. To address this, we design the Residual Swin Transformer and Average Pooling Block (RSTAB) and propose the SwinAIR, which can effectively extract and fuse the diverse frequency features in infrared images and achieve superior SR reconstruction performance. By further integrating SwinAIR with U-Net, we propose the SwinAIR-GAN for real infrared image SR reconstruction. SwinAIR-GAN extends the degradation space to better simulate the degradation process of real infrared images. Additionally, it incorporates spectral normalization, dropout, and artifact discrimination loss to reduce the potential image artifacts. Qualitative and quantitative evaluations on various datasets confirm the effectiveness of our proposed method in reconstructing realistic textures and details of infrared images.

SwinIBSR: Towards real-world infrared image super-resolution

Traditional infrared image super-resolution (SR) methods often fall short in real-world scenarios, particularly when dealing with images degraded by various forms of noise and artifacts. Addressing this gap, this paper presents SwinIBSR, a blind SR approach optimized for real-world infrared imaging conditions. SwinIBSR introduces a practical degradation model, specifically designed to simulate the complex degradation processes encountered in real-world infrared imaging. Additionally, a mixed training strategy using both infrared and visible light datasets greatly enhances the network’s generalization capabilities, leading to the production of high-quality SR images. Building upon the Transformer-based architecture of SwinIR, SwinIBSR is adept at learning intricate mappings from low-resolution to high-resolution images. Our comprehensive experiments validate the effectiveness of SwinIBSR, showcasing its outstanding performance in processing real-world infrared images and its notable superiority over existing methods regarding visual quality. This work contributes to the field of infrared image SR, offering insights in enhancing SR techniques for practical applications.

LKFormer: large kernel transformer for infrared image super-resolution

LKFormer: large kernel transformer for infrared image super-resolution

Infrared image super-resolution based on spatially variant blur kernel calibration

Infrared image super-resolution based on spatially variant blur kernel calibration

Gradient residual attention network for infrared image super-resolution

Image super-resolution techniques overcome the limitations of physical limits on infrared imaging systems and allow higher-resolution images of targets to be obtained on top of existing systems. In this paper, we proved that infrared images are more reconstructed than visible images based on data compression. We propose a CNN-based Gradient Residual Attention Network (GRAN) for infrared image super-resolution. Specifically, the residual dense module (RDB) is used to acquire depth features and the gradient operator (GO) gains fine-grained detail features. Meanwhile, the 3D attention block (3DAB) learns features' channel and spatial correlation to selectively capture more useful informative features. The experimental results show that the proposed method performs well compared to popular image super-resolution methods in terms of quantitative metrics and visual quality.

Real-infraredSR: real-world infrared image super-resolution via thermal imager.

Infrared image super-resolution technology aims to overcome the pixel size limitation of the infrared focal plane array for higher resolution images. Due to the real-world images with different resolutions having more complex degradation processes than mathematical calculation, most existing super-resolution methods using the synthetic data obtained by bicubic interpolation achieve unsatisfactory reconstruction performance in real-world scenes. To solve this, this paper innovatively proposes an infrared real-world dataset with different resolutions based on a refrigerated thermal detector and the infrared zoom lens, enabling the network to acquire more realistic details. We obtain images under different fields of view by adjusting the infrared zoom lens and then achieve the scale and luminance alignment of high and low-resolution (HR-LR) images. This dataset can be used for infrared image super-resolution, with an up-sampling scale of two. In order to learn complex features of infrared images efficiently, an asymmetric residual block structure is proposed to effectively reduce the number of parameters and improve the performance of the network. Finally, to solve the slight misalignment problem in the pre-processing stage, contextual loss and perceptual loss are introduced to improve the visual performance. Experiments show that our method achieves superior results both in reconstruction effect and practical value for single infrared image super-resolution in real scenarios.

DASR: Dual-Attention Transformer for infrared image super-resolution

The infrared image super-resolution (SR) method successfully overcomes the hardware limitations of infrared cameras that reconstruct higher-quality images with improved efficiency and cost-effectiveness. However, existing infrared SR methods do not take into account the specificity of infrared images and are primarily designed for small-scale factors. In this paper, we first investigate the domain differences between infrared and visible images and the impact of these differences on super-resolution tasks. It was found that compared with visible SR, the infrared SR will require more global edge structure information rather than local texture information reconstructed by previous CNN-based methods mainly. To address this disparity, we propose a novel infrared SR model, named DASR, which incorporates a Transformer with spatial and channel dual-attention mechanisms. In DASR, spatial attention captures both local and global information through window-based and cross-window contextual long-range interactions, while channel attention captures channel-wise global information through cross-channel interactions. With this new Transformer architecture, our method effectively extracts spatial and channel global information, which cannot be captured by the local receptive field of convolution, thus making it more suitable for infrared SR. Extensive experiments on benchmark datasets have indicated that our method outperforms the state-of-the-art infrared SR methods with less number of parameters and lower computational complexity while having the best visual results.

Meta transfer learning-based super-resolution infrared imaging

We propose an infrared image super-resolution method with meta-transfer learning and a lightweight network. We design a lightweight network to learn the map between low-resolution and high-resolution infrared images. We train the network with an external dataset and use meta-transfer learning with an internal dataset that makes the network drop to a sensitive and transferable point. We build an infrared imaging system with an infrared module. The designed network is implemented on a personal computer and the SR image is reconstructed by the trained network. The main contribution of this paper is to adopt a lightweight network and meta-transfer learning method, which obtains infrared super-resolution images with better visual effects. Both numerical and experimental results show that the proposed method achieves the infrared image super-resolution, and the performance of the proposed method is superior to four state-of-art image super-resolution methods. The proposed method has practical application in the image super-resolution of mobile infrared devices.

Visible-Assisted Infrared Image Super-Resolution Based on Spatial Attention Residual Network

Infrared images have a wide range of applications in military and civilian fields, including night vision, surveillance, and robotics. However, the most commonly used infrared images are low-resolution (LR), which lack texture details, and existing infrared image super-resolution (SR) algorithms are limited by the lack of spatial information utilization. To solve the above problems, a spatial attention residual network (SAResNet) is proposed. Specifically, the network consists of spatial attention residual block (SARB) with several short skip connections (SSCs). The SARB contains 20 spatial attention blocks (SAB), which adaptively adjusts weights of different spatial regions by considering interdependence between spatial features. Meanwhile, the visible images are considered as complementary sources; thus, a visible-assisted training strategy is designed for the infrared SR process, promoting details preservation. Furthermore, the spatial attention (SA) mechanism is utilized, which focuses more on spatial characteristics of the image and refines the main objects and target boundaries. Experimentally, the proposed method, SAResNet, is compared with existing SR methods, and the effectiveness of the proposed method is demonstrated based on both quantity and quality analyses.

Maritime Infrared Image Super-Resolution Using Cascaded Residual Network and Novel Evaluation Metric

Infrared (IR) cameras have been important surveillance sensors for autonomous surface vessels; however, their detection ranges are limited by low resolution. In this study, we collect maritime IR images, analyze the characteristics of those images, and develop datasets for training and testing. Then, a new maritime IR image super-resolution network, maritime infrared super-resolution using cascaded residual network, is developed to reconstruct IR images using a scale of 4. Moreover, different loss functions have different effects on output images; a loss function is set to be a combination of three loss functions, including mean absolute error, mean squared error, and perceptual loss. Peak signal-to-noise ratio and structural similarity index measure cannot effectively describe super-resolution performance. As the novel evaluation metric, Canny edge detection method is used because edges are important for human and target detection algorithms. Finally, experiments are conducted and the results demonstrate that the developed residual network can achieve high-quality reconstructed maritime IR images.

Infrared Image Super-Resolution via Progressive Compact Distillation Network

Deep convolutional neural networks are capable of achieving remarkable performance in single-image super-resolution (SISR). However, due to the weak availability of infrared images, heavy network architectures for insufficient infrared images are confronted by excessive parameters and computational complexity. To address these issues, we propose a lightweight progressive compact distillation network (PCDN) with a transfer learning strategy to achieve infrared image super-resolution reconstruction with a few samples. We design a progressive feature residual distillation (PFDB) block to efficiently refine hierarchical features, and parallel dilation convolutions are utilized to expand PFDB’s receptive field, thereby maximizing the characterization power of marginal features and minimizing the network parameters. Moreover, the bil-global connection mechanism and the difference calculation algorithm between two adjacent PFDBs are proposed to accelerate the network convergence and extract the high-frequency information, respectively. Furthermore, we introduce transfer learning to fine-tune network weights with few-shot infrared images to obtain infrared image mapping information. Experimental results suggest the effectiveness and superiority of the proposed framework with low computational load in infrared image super-resolution. Notably, our PCDN outperforms existing methods on two public datasets for both ×2 and ×4 with parameters less than 240 k, proving its efficient and excellent reconstruction performance.

Generative Adversarial Network-Based Edge-Preserving Superresolution Reconstruction of Infrared Images

The convolutional neural network has achieved good results in the superresolution reconstruction of single-frame images. However, due to the shortcomings of infrared images such as lack of details, poor contrast, and blurred edges, superresolution reconstruction of infrared images that preserves the edge structure and better visual quality is still challenging. Aiming at the problems of low resolution and unclear edges of infrared images, this work proposes a two-stage generative adversarial network model to reconstruct realistic superresolution images from four times downsampled infrared images. In the first stage of the generative adversarial network, it focuses on recovering the overall contour information of the image to obtain clear image edges; the second stage of the generative adversarial network focuses on recovering the detailed feature information of the image and has a stronger ability to express details. The infrared image superresolution reconstruction method proposed in this work has highly realistic visual effects and good objective quality evaluation results.

Infrared Image Super-Resolution via Transfer Learning and PSRGAN

Recent advances in single image super-resolution (SISR) demonstrate the power of deep learning for achieving better performance. Because it is costly to recollect the training data and retrain the model for infrared (IR) image super-resolution, the availability of only a few samples for restoring IR images presents an important challenge in the field of SISR. To solve this problem, we first propose the progressive super-resolution generative adversarial network (PSRGAN) that includes the main path and branch path. The depthwise residual block (DWRB) is used to represent the features of the IR image in the main path. Then, the novel shallow lightweight distillation residual block (SLDRB) is used to extract the features of the readily available visible image in the other path. Furthermore, inspired by transfer learning, we propose the multistage transfer learning strategy for bridging the gap between different high-dimensional feature spaces that can improve the PSGAN performance. Finally, quantitative and qualitative evaluations of two public datasets show that PSRGAN can achieve better results compared to the SR methods.

A lightweight iterative error reconstruction network for infrared image super-resolution in smart grid

Thermal infrared (IR) images are widely used in smart grids for numerous applications. These applications prefer high-resolution (HR) IR images since HR IR images benefit the performance. However, HR IR imaging devices are extremely expensive. To save the cost of upgrading imaging devices, an iterative error reconstruction network (IERN) is proposed to improve the resolution of IR images. We first achieve efficient dense connections based on linearly compressive skip links. Slightly sacrificing the performance, the efficient dense connections can markedly reduce the parameters and computations of the vanilla dense connections. Then, an iterative error reconstruction mechanism is proposed to boost the performance, which enables IERN to restore many more textures and edges. Specifically, an initial SR image, high-level features, and up-sampled features are obtained firstly. Secondly, a SR error image is acquired by reconstructing the errors between the initial high-level features and the back-projected features from the up-sampled features. Thirdly, a new SR image is obtained by adding the SR error image to the initial SR image. Iterating the above process, the final SR image is achieved when the number of iterations reaches to the iteration threshold. Experimental results reveal the superiority of the proposed method over state-of-the-art methods.

Infrared image super-resolution via discriminative dictionary and deep residual network

Image super-resolution successfully overcomes the limitation of infrared imaging system and satisfies the increasing demand for high-resolution infrared images. In this paper, we propose a novel infrared image super-resolution method via discriminative dictionary and deep residual network, which is benefited from the advantages of both compressive sensing and deep learning for more accurate reconstruction results. A discriminative dictionary learning method is introduced to explicitly learn a shared sub-dictionary with a set of cluster-specific sub-dictionaries from the training dataset to better characterize the specific representation of each image. Besides, multiple constraints are integrated to preserve the intrinsic relationship and structure information among images in the learned dictionary. Moreover, to further compensate for the limitation of dictionary representation capacity, a deep residual network is constructed to capture more high-frequency details of infrared images. Both quantitative and qualitative evaluations demonstrate that the proposed method could achieve favorable performances against other state-of-the-art methods.

RGB-IR Cross Input and Sub-Pixel Upsampling Network for Infrared Image Super-Resolution.

Deep learning-based image super-resolution has shown significantly good performance in improving image quality. In this paper, the RGB-IR cross input and sub-pixel upsampling network is proposed to increase the spatial resolution of an Infrared (IR) image by combining it with a color image of higher spatial resolution obtained with a different imaging modality. Specifically, this is accomplished by fusion of the features map of two RGB-IR inputs in the reconstruction of an infrared image. To improve the accuracy of feature extraction, deconvolution is replaced by sub-pixel convolution to upsample image in the network. Then, the guided filter layer is introduced for image denoising of IR images, and it can preserve the image detail. In addition, the experimental dataset, which is collected by us, contains large numbers of RGB images and corresponding IR images with the same scene. Experimental results on our dataset and other datasets demonstrate that the method is superior to existing methods in accuracy and visual improvement.

Infrared Image Super-resolution Using Improved Convolutional Neural Network

Infrared Image Super-resolution Using Improved Convolutional Neural Network

Infrared Image Super-Resolution Reconstruction Based on Quaternion and High-Order Overlapping Group Sparse Total Variation.

Owing to the limitations of imaging principles and system imaging characteristics, infrared images generally have some shortcomings, such as low resolution, insufficient details, and blurred edges. Therefore, it is of practical significance to improve the quality of infrared images. To make full use of the information on adjacent points, preserve the image structure, and avoid staircase artifacts, this paper proposes a super-resolution reconstruction method for infrared images based on quaternion total variation and high-order overlapping group sparse. The method uses a quaternion total variation method to utilize the correlation between adjacent points to improve image anti-noise ability and reconstruction effect. It uses the sparsity of a higher-order gradient to reconstruct a clear image structure and restore smooth changes. In addition, we performed regularization by using the denoising method, alternating direction method of multipliers, and fast Fourier transform theory to improve the efficiency and robustness of our method. Our experimental results show that this method has excellent performance in objective evaluation and subjective visual effects.

Cascaded Deep Networks With Multiple Receptive Fields for Infrared Image Super-Resolution

Infrared images have a wide range of military and civilian applications, including night vision, surveillance, and robotics. However, high-resolution infrared detectors are difficult to fabricate and their manufacturing cost is expensive. In this paper, we present a cascaded architecture of deep neural networks with multiple receptive fields to increase the spatial resolution of infrared images by a large scale factor ( $\times 8$ ). Instead of reconstructing a high-resolution image from its low-resolution version using a single complex deep network, the key idea of our approach is to set up a mid-point (scale $\times 2$ ) between scale $\times 1$ and $\times 8$ such that lost information can be divided into two components. Lost information within each component contains similar patterns thus can be more accurately recovered even using a simpler deep network. In our proposed cascaded architecture, two consecutive deep networks with different receptive fields are jointly trained through a multi-scale loss function. The first network with a large receptive field is applied to recover large-scale structure information, while the second one uses a relatively smaller receptive field to reconstruct small-scale image details. Our proposed method is systematically evaluated using realistic infrared images. Compared with state-of-the-art super-resolution methods, our proposed cascaded approach achieves improved reconstruction accuracy using significantly fewer parameters.

Infrared Image Super-Resolution Reconstruction Based on Quaternion Fractional Order Total Variation with Lp Quasinorm

Owing to the limitations of the imaging principle as well as the properties of imaging systems, infrared images often have some drawbacks, including low resolution, a lack of detail, and indistinct edges. Therefore, it is essential to improve infrared image quality. Considering the information of neighbors, a description of sparse edges, and by avoiding staircase artifacts, a new super-resolution reconstruction (SRR) method is proposed for infrared images, which is based on fractional order total variation (FTV) with quaternion total variation and the L p quasinorm. Our proposed method improves the sparsity exploitation of FTV, and efficiently preserves image structures. Furthermore, we adopt the plug-and-play alternating direction method of multipliers (ADMM) and the fast Fourier transform (FFT) theory for the proposed method to improve the efficiency and robustness of our algorithm; in addition, an accelerated step is adopted. Our experimental results show that the proposed method leads to excellent performances in terms of an objective evaluation and the subjective visual effect.