Original Low-resolution Image Research Articles

Multi-scale transformer network for super-resolution of visible and thermal air images

Reference image-based Super-Resolution (RefSR) is introduced to improve the quality of a Low-resolution (LR) input image by leveraging the additional information provided by a High-Resolution (HR) reference image (Ref). While existing RefSR methods focus on thermal or visible flows separately, they often struggle to enhance the resolution of small objects such as Mini/Micro UAVs (Unmanned Aerial Vehicle) due to the resolution disparities between the input and reference images. To cope with these challenges when dealing with UAV early detection in context of video surveillance, we propose ThermoVisSR, a multiscale texture transformer for enhancing the Super-Resolution (SR) of visible and thermal images of Mini/Micro UAVs. Our approach tries to reconstruct the fine details of these objects while preserving their approximation (the body form and color of the different scene objects) already contained in the LR image. Hence, our model is divided up into two streams dealing separately with approximation and detail reconstruction. In the first one, we introduce a Convolution Neural Network (CNN) fusion backbone to extract the Low-Frequency (LF) approximation from the original LR image pairs. In the second one and to extract the details from the Ref image, our approach involves blending features from both visible and thermal sources to make the most of what each offer. Subsequently, we introduce the High-Frequency Texture Transformer (HFTT) across various resolutions of the merged features to ensure an accurate correspondence matching and significant transfer of High-Frequency (HF) patches from Ref to LR images. Moreover, to adapt the injection to the different bands well, we incorporate the separable software decoder (SSD) into the HFTT allowing to capture channel-specific details during the reconstruction phase. We validated our approach using a newly created dataset of Air images of Mini/Micro UAVs. Experimental results demonstrate that the proposed model consistently outperforms the state-of-the-art approaches on both qualitative and quantitative assessments.

A General Image Super-Resolution Reconstruction Technique for Walnut Object Detection Model

Object detection models are commonly used in yield estimation processes in intelligent walnut production. The accuracy of these models in capturing walnut features largely depends on the quality of the input images. Without changing the existing image acquisition devices, this study proposes a super-resolution reconstruction module for drone-acquired walnut images, named Walnut-SR, to enhance the detailed features of walnut fruits in images, thereby improving the detection accuracy of the object detection model. In Walnut-SR, a deep feature extraction backbone network called MDAARB (multilevel depth adaptive attention residual block) is designed to capture multiscale information through multilevel channel connections. Additionally, Walnut-SR incorporates an RRDB (residual-in-residual dense block) branch, enabling the module to focus on important feature information and reconstruct images with rich details. Finally, the CBAM (convolutional block attention module) attention mechanism is integrated into the shallow feature extraction residual branch to mitigate noise in shallow features. In 2× and 4× reconstruction experiments, objective evaluation results show that the PSNR and SSIM for 2× and 4× reconstruction reached 24.66 dB and 0.8031, and 19.26 dB and 0.4991, respectively. Subjective evaluation results indicate that Walnut-SR can reconstruct images with richer detail information and clearer texture features. Comparative experimental results of the integrated Walnut-SR module show significant improvements in mAP50 and mAP50:95 for object detection models compared to detection results using the original low-resolution images.

Bayesian decision based fusion algorithm for remote sensing images

Remote sensing image fusion is dedicated to obtain a high-resolution multispectral (HRMS) image without spatial or spectral distortion compared to the single source image. In this paper, a novel fusion algorithm based on Bayesian estimation for remote sensing images is proposed from the new perspective of risk decisions. In this study, an observation model based on Bayesian estimation for remote sensing image fusion is constructed. Three categories of probabilities including prior, conditional and posterior probabilities are calculated after an intensity-hue-saturation (IHS) transformation is applied to the original low-resolution MS image. To obtain the desired HRMS image, with the corrected posterior probability, a fusion rule based on Bayesian decisions is designed to estimate which pixels to select from the panchromatic (PAN) image and the intensity component of the MS image. The selected pixels constitute a new component that will participate in an IHS inverse transformation to yield the fused image. Extensive experiments were performed on the Pleiades, WorldView-3, and IKONOS datasets, and the results demonstrate the effectiveness of the proposed method.

LPSRGAN: Generative adversarial networks for super-resolution of license plate image

This paper proposes a super-resolution algorithm for reconstructing license plate images based on generative adversarial networks (GAN) for improving the recognition rate of low-resolution license plate images. To this end, this paper first designs a new image degradation model, called the n-stage random combination degradation model (n-RCD), which extends the traditional degradation model horizontally and vertically to better simulate the features of low-resolution license plate images in natural scenes. Subsequently, to address the problem that existing GAN models cannot adapt to the complex degradation space in natural scenes, this paper proposes a new network structure called LPSRGAN, which optimizes the SRGAN model from the generator and discriminator to improve its super-resolution capability. In addition, this paper also proposes a perceptual optical character recognition (OCR) loss for license plate images, which defines the connectionist temporal classification (CTC) loss of the output of the OCR network with character label values to better preserve the character details and features of the license plate images. Finally, the experiments reported in this paper show that the proposed algorithm improves the recognition rate of the original low-resolution license plate image by 12.48%, and the recognition rate of the reconstructed image reaches 93.90%. Based on this, this paper also tests the algorithm on actual captured data, and the results show that it performs well in natural scenes, demonstrating the algorithm has broad application prospects.

The super-resolution reconstruction algorithm of multi-scale dilated convolution residual network.

Aiming at the problems of traditional image super-resolution reconstruction algorithms in the image reconstruction process, such as small receptive field, insufficient multi-scale feature extraction, and easy loss of image feature information, a super-resolution reconstruction algorithm of multi-scale dilated convolution network based on dilated convolution is proposed in this paper. First, the algorithm extracts features from the same input image through the dilated convolution kernels of different receptive fields to obtain feature maps with different scales; then, through the residual attention dense block, further obtain the features of the original low resolution images, local residual connections are added to fuse multi-scale feature information between multiple channels, and residual nested networks and jump connections are used at the same time to speed up deep network convergence and avoid network degradation problems. Finally, deep network extraction features, and it is fused with input features to increase the nonlinear expression ability of the network to enhance the super-resolution reconstruction effect. Experimental results show that compared with Bicubic, SRCNN, ESPCN, VDSR, DRCN, LapSRN, MemNet, and DSRNet algorithms on the Set5, Set14, BSDS100, and Urban100 test sets, the proposed algorithm has improved peak signal-to-noise ratio and structural similarity, and reconstructed images. The visual effect is better.

EVALUATION OF PSNR VALUE FOR IMAGE SUPER-RESOLUTION USING DEEP LEARNING

Deep learning is a branch of study that simulates the human brain and allows learning from large data. In recent years, deep learning techniques have become more prevalent in a range of applications. These techniques have been widely employed in image processing, especially for object detection, image augmentation, and super-resolution. Single-image super-resolution, which converts low-resolution images into high-resolution ones, is a significant application of deep learning. Outstanding perceptual and antagonistic outcomes have been achieved using multiple single-image super-resolution models. In this study, several noteworthy methods, including Super Resolution with Convolutional Neural Network, Super Resolution Residual Network, Super Resolution Generative Adversarial Network, and Enhanced Super Resolution Generative Adversarial Network are used for single image super-resolution. Each layer in these deep learning models carries out certain operations on low-resolution images to transform them into high-resolution equivalents. As a result, many high-resolution images can be produced from a single low-resolution image. We created a dataset of random images for our research. The original low-resolution images are compared to the high-resolution images that were created using a deep learning technique. Peak Signal-to-Noise Ratio and the Structural Similarity Index are used in the comparison. Additionally, the evaluation process includes computing the ground truth value. The highest similarity indexed is achieved by VGG54 with 96% of similarity.

An assisted multi-frame approach for super-resolution in hyperspectral images of rock samples

Imaging spectroscopy is decisive for accurately characterizing rock samples for geological and mineral applications, since slight differences in mineral composition can be easily recognized in spectral signatures. Although, instrument limitations prevent high spatial resolution pixels scanned along hundreds of spectral channels, which considerably reduces the data capabilities. Multi frame Super-Resolution (SR) techniques can overcome this issue by retrieving high-quality images using purely computational methods. However, these approaches still have important drawbacks because of inherent uncertainties when estimating the spatial variation between frames, which strongly penalize the spectral information, essential for lithology. In this paper we propose an assisted framework to provide fully controlled motion parameters of the imaging process, skipping the intricate registration task. Essentially, we use a stepping device to successively change the position of rock samples for each new frame acquisition. With accurate knowledge about motion parameters, we were able to overcome the uncertainties regarding spatial registration between frames. Therefore, a reasonable number of equations connecting the low-resolution frames and the super resolved image could be used to solve the optimization problem. Extensive experiments proved that the proposed assisted method achieved best performances (both qualitatively and quantitatively) compared with similar approaches. Moreover, detailed spectral analysis showed almost absolute consistency between the original low-resolution image and the super-resolved result, which is crucial to guarantee reliable mineral characterization.

An efficient lightweight network for single image super-resolution

The outstanding performance of convolutional neural networks (CNNs) shown in single image super-resolution (SISR) strongly depends on network’s depth, which hampers its application in low-power computing devices. In this paper, a lightweight and efficient network (LESR) is proposed for SISR by constructing the shallow feature extraction block (SFBlock), the cascaded sparse mask blocks (SMBlocks) and the feature fusion block (FFBlock). The SFBlock efficiently extracts global informative features from the original low resolution image using sparse self-attention, SMBlock skips the redundant computation in extracted features, and more meaningful information is distilled for the sequential reconstruction block by the FFBlock. In addition, a recently proposed activation function called ACON-C is used to replace the ReLU function to ease the training difficulty. Extensive experiments show that our proposed network performs better than most advanced lightweight SISR algorithms with comparable parameters and less FLOPs on benchmark database for ×2/3/4 SISR.

A generative adversarial network model fused with a self‐attention mechanism for the super‐resolution reconstruction of ancient murals

AbstractThe problem that the texture details of low‐resolution (LR) digital images of ancient murals are ambiguous persists. To solve this problem, this study proposes a super‐resolution (SR) reconstruction method for fuzzy murals based on a generative adversarial network with self‐attention (SA). The network uses a blur kernel and realistic noise data to add blur and noise, respectively, to a high‐resolution (HR) image to obtain an original LR image. Then, a feature image with the same size as that of the input image is obtained through a SA module. Finally, the feature information extracted from the image is input into the high‐resolution image space by using a subpixel convolution layer to realize the image enlargement process from an LR to an HR. Experiments evaluate the proposed approach both objectively and subjectively. The objective evaluation results show that compared with other SR reconstruction algorithms, the proposed algorithm's peak signal‐to‐noise ratio (PSNR) is increased by 0.04 to 3.78 dB on average, and its structural similarity is increased by 0.002 to 0.191. A subjective perception evaluation shows that the developed algorithm can better reconstruct the texture details of murals, thus better meeting the visual perception needs of the public. The method proposed in this study can satisfactorily reconstruct the texture details of murals, which may provide technical guidance for the development of mural protection plans. Furthermore, it may be of certain practical significance for the SR reconstruction of ancient murals.

Remote Sensing Image Super-Resolution via Residual-Dense Hybrid Attention Network

Nowadays, remote sensing datasets with long temporal coverage generally have a limited spatial resolution, most of the existing research uses the single image super-resolution (SISR) method to reconstruct high-resolution (HR) images. However, due to the lack of information in low-resolution (LR) images and the ill-posed nature of SISR, it is difficult to reconstruct the fine texture of HR images under large-scale magnification factors (e.g., four times). To address this problem, we propose a new reference-based super-resolution method called a Residual-Dense Hybrid Attention Network (R-DHAN), which uses the rich texture information in the reference image to make up for the deficiency of the original LR image. The proposed SR model employs Super-Resolution by Neural Texture Transfer (SRNTT) as a backbone. Based on this structure, we propose a dense hybrid attention block (DHAB) as a building block of R-DHAN. The DHAB fuses the input and its internal features of current block. While making full use of the feature information, it uses the interdependence between different channels and different spatial dimensions to model and obtains a strong representation ability. In addition, a hybrid channel-spatial attention mechanism is introduced to focus on important and useful regions to better reconstruct the final image. Experiments show that compared with SRNTT and some classical SR techniques, the proposed R-DHAN method performs well in quantitative evaluation and visual quality.

Super‐resolution of brain tumor MRI images based on deep learning

To explore and evaluate the performance of MRI-based brain tumor super-resolution generative adversarial network (MRBT-SR-GAN) for improving the MRI image resolution in brain tumors. A total of 237 patients from December 2018 and April 2020 with T2-fluid attenuated inversion recovery (FLAIR) MR images (one image per patient) were included in the present research to form the super-resolution MR dataset. The MRBT-SR-GAN was modified from the enhanced super-resolution generative adversarial networks (ESRGAN) architecture, which could effectively recover high-resolution MRI images while retaining the quality of the images. The T2-FLAIR images from the brain tumor segmentation (BRATS) dataset were used to evaluate the performance of MRBT-SR-GAN contributed to the BRATS task. The super-resolution T2-FLAIR images yielded a 0.062 dice ratio improvement from 0.724 to 0.786 compared with the original low-resolution T2-FLAIR images, indicating the robustness of MRBT-SR-GAN in providing more substantial supervision for intensity consistency and texture recovery of the MRI images. The MRBT-SR-GAN was also modified and generalized to perform slice interpolation and other tasks. MRBT-SR-GAN exhibited great potential in the early detection and accurate evaluation of the recurrence and prognosis of brain tumors, which could be employed in brain tumor surgery planning and navigation. In addition, this technique renders precise radiotherapy possible. The design paradigm of the MRBT-SR-GAN neural network may be applied for medical image super-resolution in other diseases with different modalities as well.

Lightweight single image super-resolution with attentive residual refinement network

In recent years, deep convolutional neural network (CNN) based single image super-resolution (SISR) methods have been demonstrated impressive performance in terms of quantitative metrics and visual effects. Most CNN-based SISR methods can learn the complex non-linear mapping between low-resolution (LR) images and their corresponding high-resolution (HR) images due to the powerful representation capabilities of deep convolutional neural networks. However, as the depth and width of the SISR networks increase, the parameters of SISR networks will increase dramatically, leading to huge computational cost and large memory consumption, making them impractical in real-world applications. To attack the above issues, we propose an accurate and lightweight deep convolutional neural network, named Attentive Residual Refinement Network (ARRFN), to recover the high-resolution image from the original low-resolution image directly for SISR. In general, our proposed ARRFN consists of three parts, a feature extraction block, a stack of attentive residual refinement blocks (ARRFB), and a multi-scale separable upscaling module (MSSU), respectively. Specifically, our ARRFB consists of two branches, a regular residual learning branch and an attentive residual refinement branch. The former conducts regular residual learning by two residual blocks while the latter refines the residual information from the two residual blocks of the former branch with an attentive residual mechanism to further enhance the representation capabilities of the network. Furthermore, a multi-scale separable upsampling module (MSSU) is proposed to replace the regular upsampling operation for better SR results. Extensive experiments on several standard benchmarks show that the proposed method outperforms state-of-the-art SR methods in terms of quantitative metrics, visual quality, memory footprint, and inference time.

B2GAN: Bidirectional-branch generative adversarial network for text image super-resolution with structure preservation

Text image super-resolution (SR) is one of the most hot topic research in computer vision. The traditional convolutional neural networks usually over-smooth fine structures of SR images while the popular generative adversarial network (GAN) based SR methods may produce undesirable SR results with distorted edges. To cope with these problems, this paper proposes a bidirectional-branch generative adversarial network (B2GAN) for text image SR. The proposed B2GAN consists of two parts: one is a generator, the other is a discriminator. In the generator, a forward branch is used for mapping the low-resolution (LR) image to its corresponding SR image to generate photo-realistic images, while a reverse branch is responsible for mapping the generated SR image to a low-resolution image which is viewed as its original LR image. A loss function is proposed to train a pleasing generator and guarantee a stable convergence to the original LR image. In the discriminator, The VGG16 and the adversarial loss are employed. In the experiments, we establish two text image datasets including Chinese and multi-language characters which are used to test the effectiveness of the proposed B2GAN method. Quantitative associated with qualitative experimental results validate that the proposed B2GAN method can achieve competitive performance and even outperform the state-of-the-arts.

SAR image super-resolution reconstruction based on cross-resolution discrimination

High-resolution (HR) synthetic aperture radar (SAR) images play an important role in people’s daily life and military applications. However, due to the interference of speckle noise, the texture details of the SAR images become relatively blurred. The fine texture details can be reconstructed by increasing the resolution of the SAR images. Generative adversarial networks achieve high performance in image super-resolution (SR) reconstruction, but the existing generative adversarial networks only pay attention to the discrimination of HR images without that of the low-resolution (LR) images. If the reconstructed HR image is sufficiently realistic, the LR image obtained from downsampled super-resolved images should also be the same as the original LR image. To take advantage of the LR image, an SAR image SR reconstruction algorithm based on cross-resolution discrimination (CRD) using teacher–student network is proposed. First, the teacher discriminator network (TD-Net) discriminates the HR images, which enriches the reconstructed HR images with more high-frequency texture details. Second, the student discriminator network (SD-Net) discriminates the LR images, which enables the reconstructed HR images to be accurately downsampled to the original LR image. Finally, the TD-Net guides the training of the SD-Net by transmitting distillation knowledge to the SD-Net, which further improves the discriminative performance of the SD-Net. Experiments on the SAR image dataset demonstrate that the performance of the proposed CRD algorithm is better than other algorithms when both the objective evaluations and subjective effects are considered.

3D dense convolutional neural network for fast and accurate single MR image super-resolution

Super-resolution (SR) MR image reconstruction has shown to be a very promising direction to improve the spatial resolution of low-resolution (LR) MR images. In this paper, we presented a novel MR image SR method based on a dense convolutional neural network (DDSR), and its enhanced version called EDDSR. There are three major innovations: first, we re-designed dense modules to extract hierarchical features directly from LR images and propagate the extracted feature maps through dense connections. Therefore, unlike other CNN-based SR MR techniques that upsample LR patches in the initial phase, our methods take the original LR images or patches as input. This effectively reduces computational complexity and speeds up SR reconstruction. Second, a final deconvolution filter in our model automatically learns filters to fuse and upscale all hierarchical feature maps to generate HR MR images. Using this, EDDSR can perform SR reconstructions at different upscale factors using a single model with one stride fixed deconvolution operation. Third, to further improve SR reconstruction accuracy, we exploited a geometric self-ensemble strategy. Experimental results on three benchmark datasets demonstrate that our methods, DDSR and EDDSR, achieved superior performance compared to state-of-the-art MR image SR methods with less computational load and memory usage.

Image super-resolution using multi-granularity perception and pyramid attention networks

Recently, single image super-resolution (SISR) has been widely applied in the fields of multimedia and computer vision communities and obtained remarkable performance. However, most current methods ignore to utilize multi-granularity features of low-resolution (LR) image to further improve the SISR performance. And the channel and spatial features obtained from original LR images are treated equally, resulting in unnecessary computations for abundant uninformative features, thereby hindering the representational ability of super-resolution (SR) models. In this paper, we present a novel Multi-Granularity Pyramid Attention Network (MGPAN) which fully exploits the multi-granularity perception and attention mechanisms to improve the quality of reconstructed images. We design a multi-branch dilated convolution layer with varied kernels corresponding to receptive fields of different sizes to modulate multi-granularity features for adaptively capturing more important information. Moreover, a novel spatial pyramid pooling attention (SPPA) module is constructed to integrate the channel-wise and multi-scale spatial information, which is beneficial to compute the response values from the multi-scale regions of each neuron, and then establish the accurate mapping from low to high dimensional solution space. Besides, for long-short-term information preservation and information flow enhancement, we adopt the short, long, and global skip connection structures to concatenate and fuse the states of each module, which can improve the SR network performance effectively. Extensive experiments on several standard benchmark datasets show that the proposed MGPAN can provide state-of-the-art or even better performance in both quantitative and qualitative measurements.

Multi-scale channel network based on filer pruning for image super-resolution

Image super-resolution is an important image restoration technology, which is widely used in optical system, optical communication and other fields. Deep convolutional neural networks (CNNs) have made a great breakthrough in accuracy of single image super-resolution. Nevertheless, as the depth and width of the networks increase, CNNs methods face the challenge of memory consumption and computational complexity. To solve this problem, in this paper, we introduce a multi-scale channel network (MSCN) based on filter pruning to reconstruct the high-resolution (HR) image from the original low-resolution (LR) image. Firstly, we propose a multi-scale channel block (MSCB) to obtain the fine features based on the convolution kernels with different receptive fields. Channel split operator is used in MSCB to reduce the model parameters. The wider MSCN with more filters is trained to achieve a better performance. Then, we calculate the rank of feature maps and prune filters with low-rank feature information. Finally, the compressed multi-scale channel network with fewer parameters can achieve similar accuracies. The experimental results validate the effectiveness of our approaches.

MADNet: A Fast and Lightweight Network for Single-Image Super Resolution.

Recently, deep convolutional neural networks (CNNs) have been successfully applied to the single-image super-resolution (SISR) task with great improvement in terms of both peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). However, most of the existing CNN-based SR models require high computing power, which considerably limits their real-world applications. In addition, most CNN-based methods rarely explore the intermediate features that are helpful for final image recovery. To address these issues, in this article, we propose a dense lightweight network, called MADNet, for stronger multiscale feature expression and feature correlation learning. Specifically, a residual multiscale module with an attention mechanism (RMAM) is developed to enhance the informative multiscale feature representation ability. Furthermore, we present a dual residual-path block (DRPB) that utilizes the hierarchical features from original low-resolution images. To take advantage of the multilevel features, dense connections are employed among blocks. The comparative results demonstrate the superior performance of our MADNet model while employing considerably fewer multiadds and parameters.

Spatial Downscaling Based on Spectrum Analysis for Soil Freeze/Thaw Status Retrieved From Passive Microwave

Seasonal variations of frozen soil can be effectively monitored at regional scales by using passive microwave remote sensing techniques. However, low spatial resolution of passive microwave remote sensing considerably constrains its application at local scales. Therefore, an effective spatial downscaling approach for frozen soil monitoring based on passive microwave remote sensing is essential to improve the application of passive microwave remote sensing for frozen soil monitoring at both regional and local scales. The present study is aimed to develop an effective spatial downscaling approach with spectrum analysis for frozen soil monitoring based on passive microwave remote sensing. The feasibility of the proposed spatial downscaling approach was investigated and discussed with the field <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">in situ</i> field observations data in northeastern China. The result obtained revealed a quite similar relationship of power spectral density (PSD) and spatial frequency with that between original low spatial resolution and high spatial resolution images in the frequency domain. The amplitude information in unresolved higher spatial resolution image thus can be estimated approximated by the relationship of the PSD with spatial frequency of original low spatial resolution image, whereas the phase information can be extracted by some traditional methods, such as resampling (RES) or geographically weighted regression (GWR) method. The spatial downscaling approach based on spectrum analysis can not only take spatial heterogeneity into account but also reveals the spatial characteristics of the surface soil freeze/thaw status. In addition, it was found that the phase information determined the spatial heterogeneity of downscaled results of surface soil frozen/thaw status.

Perception consistency ultrasound image super-resolution via self-supervised CycleGAN

Due to the limitations of sensors, the transmission medium, and the intrinsic properties of ultrasound, the quality of ultrasound imaging is always not ideal, especially its low spatial resolution. To remedy this situation, deep learning networks have been recently developed for ultrasound image super-resolution (SR) because of the powerful approximation capability. However, most current supervised SR methods are not suitable for ultrasound medical images because the medical image samples are always rare, and usually, there are no low-resolution (LR) and high-resolution (HR) training pairs in reality. In this work, based on self-supervision and cycle generative adversarial network, we propose a new perception consistency ultrasound image SR method, which only requires the LR ultrasound data and can ensure the re-degenerated image of the generated SR one to be consistent with the original LR image, and vice versa. We first generate the HR fathers and the LR sons of the test ultrasound LR image through image enhancement, and then make full use of the cycle loss of LR–SR–LR and HR–LR–SR and the adversarial characteristics of the discriminator to promote the generator to produce better perceptually consistent SR results. The evaluation of PSNR/IFC/SSIM, inference efficiency and visual effects under the benchmark CCA-US and CCA-US datasets illustrate our proposed approach is effective and superior to other state-of-the-art methods.