Super-resolution Convolutional Neural Network Research Articles

DownScaleBench for developing and applying a deep learning based urban climate downscaling- first results for high-resolution urban precipitation climatology over Austin, Texas

Cities need climate information to develop resilient infrastructure and for adaptation decisions. The information desired is at the order of magnitudes finer scales relative to what is typically available from climate analysis and future projections. Urban downscaling refers to developing such climate information at the city (order of 1 – 10 km) and neighborhood (order of 0.1 – 1 km) resolutions from coarser climate products. Developing these higher resolution (finer grid spacing) data needed for assessments typically covering multiyear climatology of past data and future projections is complex and computationally expensive for traditional physics-based dynamical models. In this study, we develop and adopt a novel approach for urban downscaling by generating a general-purpose operator using deep learning. This ‘DownScaleBench’ tool can aid the process of downscaling to any location. The DownScaleBench has been generalized for both in situ (ground- based) and satellite or reanalysis gridded data. The algorithm employs an iterative super-resolution convolutional neural network (Iterative SRCNN) over the city. We apply this for the development of a high-resolution gridded precipitation product (300 m) from a relatively coarse (10 km) satellite-based product (JAXA GsMAP). The high-resolution gridded precipitation datasets is compared against insitu observations for past heavy rain events over Austin, Texas, and shows marked improvement relative to the coarser datasets relative to cubic interpolation as a baseline. The creation of this Downscaling Bench has implications for generating high-resolution gridded urban meteorological datasets and aiding the planning process for climate-ready cities.

Unsupervised real image super-resolution via knowledge distillation network

Super-resolution convolutional neural networks recently have demonstrated high-quality restoration for single images. Despite existing methods have achieved remarkable performance based on synthetic datasets, the performance is poor on real-world or natural data. To address this issue, zero-shot super-resolution (ZSSR) has been proposed for adaptive learning. However, ZSSR is unable to keep the simulated degradation process consistent with the degradation kernel of the real degradation process. Furthermore, the learned mapping of ZSSR is different from the desired mapping. In this paper, an unsupervised image super-resolution via knowledge distillation network (USRKDN) is proposed. Specifically, the proposed degradation module generates an image-specific degradation kernel and corresponding degenerated images. Moreover, the knowledge distillation module is proposed to solve the issue that the mapping cannot be completely equivalent, which transfers the learned map by knowledge distillation. The full convolution module is also explored to help the reconstruction of information. Extensive experimental results on synthetic and real datasets demonstrate the effectiveness of USRKDN. In addition, USRKDN is proven to be good at reconstructing image details in real scenes, which provides an effective method for generating information learning tasks with fewer samples.

Improved deep learning method for accurate flow field reconstruction from sparse data

The analysis of flow field is essential for the fluid dynamics and the internal mechanism of hydraulic machinery. The use of deep learning technology has revolutionized the field of fluid dynamics by not only reducing computational power and time, but also improving the quality of experimental measurements and numerical simulations. However, certain numerical computing errors can lead to inaccurate simulation results with inadequate data. Fortunately, deep learning technology can accurately reconstruct distorted results, making flow mechanism research more convenient. In this study, a novel super-resolution convolutional neural network with a u-shaped architecture (SRUnet) model was developed by combining the benefits of super-resolution convolutional neural network (SRCNN) and convolutional neural network with the u-shaped architecture (U-net) to address the challenge of reconstructing turbulence flow field images with low resolution. The SRUnet model was validated with small datasets and was shown to deliver high-precision results when applied to reconstruct the velocity flow field of the two-dimensional (2D) cylinder wake flow. Furthermore, the SRUnet model's training and testing performance surpasses that of SRCNN and U-net models, producing more under-resolved details. When compared to the quality of the velocity flow field reconstruction of SRCNN and U-net models with the same hyper-parameter values, the SRUnet model produced 5.35% and 14.22% higher average values of the peak signal-to-noise ratio (PSNR) when low-resolution velocity field images were used as model inputs for the testing dataset reconstruction. Additionally, the average values of the structural similarity index metric (SSIM) were 1.09% and 3.33% higher than those of SRCNN and U-net models. When super-low resolution flow field images were used as model inputs for the testing dataset reconstruction, the average values of the PSNR metric were 8.31% and 2.76% higher than those of SRCNN and U-net models, respectively. The average values of the SSIM metric were also 5.81% and 2.25% higher than those of SRCNN and U-net models, respectively.

Design of a Low-Power Super-Resolution Architecture for Virtual Reality Wearable Devices

Head-mounted displays (HMDs) have made a virtual reality (VR) accessible to a widespread consumer market, introducing a revolution in many applications. Among the limitations of current HMD technology, the need for generating high-resolution images and streaming them at adequate frame rates is one of the most critical. Super-resolution (SR) convolutional neural networks (CNNs) can be exploited to alleviate timing and bandwidth bottlenecks of video streaming by reconstructing high-resolution images locally (i.e., near the display). However, such techniques involve a significant amount of computations that makes their deployment within area-/power-constrained wearable devices often unfeasible. This research work originated from the consideration that the human eye can capture details with high acuity only within a certain region, called the fovea. Therefore, we designed a custom hardware architecture able to reconstruct high-resolution images by treating foveal region (FR) and peripheral region (PR) through accurate and inaccurate operations, respectively. Hardware experiments demonstrate the effectiveness of our proposal: a customized fast SR CNN (FSRCNN) accelerator realized as described here and implemented on a 28-nm process technology is able to process up to 214 ultrahigh definition frames/s, while consuming just 0.51 pJ/pixel without compromising the perceptual visual quality, thus achieving a 55% energy reduction and a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times 14$ </tex-math></inline-formula> times higher throughput rate, with respect to state-of-the-art competitors.

Dual-aware transformer network for single-image super-resolution

The advent of deep learning has enabled image super-resolution (SR) technology to achieve tremendous success. The recent exploration between convolutional neural network (CNN) and transformer has produced an impressive performance. However, the huge model size and heavy computational cost occupied by most approaches cannot be ignored. Additionally, the global and local modeling of image features is not well utilized and explored, which is important for texture content reconstruction. In this study, we present an efficient and lightweight network based on CNN and transformer for image SR, dubbed dual-aware transformer network (DATN). Specifically, a dual-aware fusion module (DAFM), consisting of spatial-aware adaptive block (SAAB) and global-aware fusion block (GAFB), learns local and global features in a complementary manner for yielding abundant content details. SAAB delivers the captured spatial information to GAFB in parallel, whereas GAFB establishes long-range spatial dependencies among diverse features to boost the network’s ability for recovering texture details. Further, in the upsampling stage, our proposed transformer-empowered upsampling module aggregates global information from the transformer module to generate location-wise reassembly kernels for content-aware upsampling. In such way, feature-rich information is highlighted adaptively to reconstruct high-quality images with maximum efficiency. Experimental results reveal that our proposed DATN achieves high qualitative performance while maintaining low computational demands, surpassing most state-of-the-art SR networks.

Super-Resolution of Dental Panoramic Radiographs Using Deep Learning: A Pilot Study.

Using super-resolution (SR) algorithms, an image with a low resolution can be converted into a high-quality image. Our objective was to compare deep learning-based SR models to a conventional approach for improving the resolution of dental panoramic radiographs. A total of 888 dental panoramic radiographs were obtained. Our study involved five state-of-the-art deep learning-based SR approaches, including SR convolutional neural networks (SRCNN), SR generative adversarial network (SRGAN), U-Net, Swin for image restoration (SwinIr), and local texture estimator (LTE). Their results were compared with one another and with conventional bicubic interpolation. The performance of each model was evaluated using the metrics of mean squared error (MSE), peak signal-to-noise ratio (PNSR), structural similarity index (SSIM), and mean opinion score by four experts (MOS). Among all the models evaluated, the LTE model presented the highest performance, with MSE, SSIM, PSNR, and MOS results of 7.42 ± 0.44, 39.74 ± 0.17, 0.919 ± 0.003, and 3.59 ± 0.54, respectively. Additionally, compared with low-resolution images, the output of all the used approaches showed significant improvements in MOS evaluation. A significant enhancement in the quality of panoramic radiographs can be achieved by SR. The LTE model outperformed the other models.

Underwater optical image enhancement based on super-resolution convolutional neural network and perceptual fusion.

Underwater optical images often have serious quality degradations and distortions, which hinders the development of underwater optics and vision systems. Currently, there are two mainstream solutions: non-learning based and learning-based. Both have their advantages and disadvantages. To fully integrate the advantages of both, we propose an enhancement method based on superresolution convolutional neural network (SRCNN) and perceptual fusion. First, we introduce a weighted fusion BL estimation model with a saturation correction factor (SCF-BLs fusion), the accuracy of image prior information is improved effectively. Next, a refined underwater dark channel prior (RUDCP) is proposed, which combines guided filtering and an adaptive reverse saturation map (ARSM) to restore the image, which not only preserves edge details but also avoids the interference of artificial light. Then, the SRCNN fusion adaptive contrast enhancement is proposed to enhance the colour and contrast. Finally, to further enhance image quality, we employ efficient perceptual fusion to blend the different resulting outputs. Extensive experiments demonstrate that our method has outstanding visual results in underwater optical image dehazing, color enhancement and is artefact- and halo-free.

Deep Learning for Super-Resolution Channel Estimation in Reconfigurable Intelligent Surface Aided Systems

Reconfigurable intelligent surface (RIS) enables the configuration of the propagation environment. Channel estimation is an essential task in realizing the RIS-aided communication system. A RIS-aided multi-user multiple-input multiple-output (MIMO) orthogonal frequency division multiplexing (OFDM) communication system involves cascaded channels with high dimensions and sophisticated statistics. Thus, implementing the optimal minimum mean square error (MMSE) with the integration computation is infeasible in practice. To accurately estimate channels with high accuracy in a RIS-aided multi-user MIMO-OFDM system, we model the channel state information (CSI) estimation as an image super-resolution (SR) problem to recover and denoise the channel matrix. Particularly, a convolutional neural network based on a super-resolution convolutional neural network (SRCNN) and denoising convolutional neural network (DnCNN), named SRDnNet, is then proposed. By taking estimated channels at pilot positions as a low-resolution image, the enhanced SRCNN can fully exploit the features of inputs to learn a suitable interpolation method and generate the coarse estimation of the channel matrix. The denoising model DnCNN with an element-wise subtraction structure can exploit features of the additive noise and recover channel coefficients from the coarse channel matrix. The simulation results demonstrate the effectiveness and excellent performance of the proposed SRDnNet.

Improving the brain image resolution of generalized q-sampling MRI revealed by a three-dimensional CNN-based method.

Understanding neural connections facilitates the neuroscience and cognitive behavioral research. There are many nerve fiber intersections in the brain that need to be observed, and the size is between 30 and 50 nanometers. Improving image resolution has become an important issue for mapping the neural connections non-invasively. Generalized q-sampling imaging (GQI) was used to reveal the fiber geometry of straight and crossing. In this work, we attempted to achieve super-resolution with a deep learning method on diffusion weighted imaging (DWI). A three-dimensional super-resolution convolutional neural network (3D SRCNN) was utilized to achieve super-resolution on DWI. Then, generalized fractional anisotropy (GFA), normalized quantitative anisotropy (NQA), and the isotropic value of the orientation distribution function (ISO) mapping were reconstructed using GQI with super-resolution DWI. We also reconstructed the orientation distribution function (ODF) of brain fibers using GQI. With the proposed super-resolution method, the reconstructed DWI was closer to the target image than the interpolation method. The peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) were also significantly improved. The diffusion index mapping reconstructed by GQI also had higher performance. The ventricles and white matter regions were much clearer. This super-resolution method can assist in postprocessing low-resolution images. With SRCNN, high-resolution images can be effectively and accurately generated. The method can clearly reconstruct the intersection structure in the brain connectome and has the potential to accurately describe the fiber geometry on a subvoxel scale.

Intelligent Image Super-Resolution for Vehicle License Plate in Surveillance Applications

Vehicle license plate images are often low resolution and blurry because of the large distance and relative motion between the vision sensor and vehicle, making license plate identification arduous. The extensive use of expensive, high-quality vision sensors is uneconomical in most cases; thus, images are initially captured and then translated from low resolution to high resolution. For this purpose, several traditional techniques such as bilinear, bicubic, super-resolution convolutional neural network, and super-resolution generative adversarial network (SRGAN) have been developed over time to upgrade low-quality images. However, most studies in this area pertain to the conversion of low-resolution images to super-resolution images, and little attention has been paid to motion de-blurring. This work extends SRGAN by adding an intelligent motion-deblurring method (termed SRGAN-LP), which helps to enhance the image resolution and remove motion blur from the given images. A comprehensive and new domain-specific dataset was developed to achieve improved results. Moreover, maintaining higher quantitative and qualitative results in comparison to the ground truth images, this study upscales the provided low-resolution image four times and removes the motion blur to a reasonable extent, making it suitable for surveillance applications.

Fast Image Super-Resolution Using Particle Swarm Optimization-Based Convolutional Neural Networks.

Image super-resolution based on convolutional neural networks (CNN) is a hot topic in image processing. However, image super-resolution faces significant challenges in practical applications. Improving its performance on lightweight architectures is important for real-time super-resolution. In this paper, a joint algorithm consisting of modified particle swarm optimization (SMCPSO) and fast super-resolution convolutional neural networks (FSRCNN) is proposed. In addition, a mutation mechanism for particle swarm optimization (PSO) was obtained. Specifically, the SMCPSO algorithm was introduced to optimize the weights and bias of the CNNs, and the aggregation degree of the particles was adjusted adaptively by a mutation mechanism to ensure the global searching ability of the particles and the diversity of the population. The results showed that SMCPSO-FSRCNN achieved the most significant improvement, being about 4.84% better than the FSRCNN model, using the BSD100 data set at a scale factor of 2. In addition, a chest X-ray super-resolution images classification test experiment was conducted, and the experimental results demonstrated that the reconstruction ability of this model could improve the classification accuracy by 13.46%; in particular, the precision and recall rate of COVID-19 were improved by 45.3% and 6.92%, respectively.

Energy Reduction with Super-Resolution Convolutional Neural Network for Ultrasound Tomography

This study addresses the issue of energy optimization by investigating solutions for the reduction of energy consumption in the diagnostics and monitoring of technological processes. The implementation of advanced process control is identified as a key approach for achieving energy savings and improving product quality, process efficiency, and production flexibility. The goal of this research is to develop a cost-effective system with a minimal number of ultrasound sensors, thus reducing the energy consumption of the overall system. To accomplish this, a novel method for obtaining high-resolution reconstruction in transmission ultrasound tomography (t-UST) is proposed. The method involves utilizing a convolutional neural network to take low-resolution measurements as input and output high-resolution sinograms that are used for tomography image reconstruction. This approach allows for the construction of a super-resolution sinogram by utilizing information hidden in the low-resolution measurement. The model is trained on simulation data and validated on real measurement data. The results of this technique demonstrate significant improvement compared to state-of-the-art methods. The study also highlights that UST measurements contain more information than previously thought, and this hidden information can be extracted and utilized with the use of machine learning techniques to further improve image quality and object recognition.

A New Deep Spiking Architecture for Reconstruction of Compressed Data in Cognitive Radio Networks

Cognitive Radio (CR) offers a spectrum sharing solution to handle the massive amount of devices operating in the same spectrum. In this work a sub-Nyquist compressive sensing technique is proposed that allows secondary users to sense and utilize idle spectrum. Reconstruction of compressed sparse data is achieved through a dual stage sophisticated reconstruction algorithm. The reconstruction uses a classical fast Orthogonal Matching Persuit (OMP), followed by a new spiking deep Residual neural Network (ResNet) architecture. The proposed architecture is obtained through a novel distributed conversion technique that is proposed to convert deep architectures to a spiking neural networks. The reconstructed data is compared in terms of Peak Signal-to-Noise Ratio (PSNR), Mean Square Error (MSE) and Structural Similarity (SSIM) to the compressed data and the ground truth. Super Resolution Convolutional Neural Network (SRCNN) and a Deep ResNet are also used for reconstruction. The proposed algorithm outperforms SRCNN and the unconverted ResNet, specially at low Channel SNR (CSNR). In addition, the proposed algorithm results in a 68% reduction in both storage and energy requirements, which makes it suitable for implementation on User Equipment (UE).

Research on Image Sharpness Enhancement Technology based on Depth Learning

Image technology is widely used in security, traffic, monitoring, and other social activities. However, these images carrying detailed information will have feature distortion due to various external physical factors in social ingestion, transmission, and storage, resulting in poor image quality and clarity. Resolution determines the definition of an image. Super-resolution reconstruction is the process of transforming a low-resolution picture into a high-resolution image. To enhance the image clarity, this experiment introduces the advantages and disadvantages of the Super Resolution Convolutional Network (SRCNN) and Fast Super Resolution Convolutional Neural Network (FSRCNN) model and then constructs an image super resolution method based on DSRCNN. The algorithm consists of two sub-network blocks, an enhancement block and a purification block. The model first uses two Convolutional Neural Networks (CNN) to obtain complementary low-frequency information that improves the model's learning ability; next, it employs an enhancement block to fuse the image features of two paths via residual operation and sub-pixel convolution to prevent the loss of low-resolution image information; finally, it employs a feature purification block to refine high-frequency information that more accurately represents the predicted high-quality image. It is found that the PSNR and SSIM of the DSRCNN model can reach 33.43dB and 0.9157dB, respectively.

An Anti-Steganalysis HEVC Video Steganography With High Performance Based on CNN and PU Partition Modes

The steganography research of videos leads to excellent communication methods for transmitting secret message, and high efficiency video coding(HEVC) video is one popular steganographic carrier. This article proposes a prediction unit(PU) based wide residual-net steganography(PWRN) for HEVC videos. The visual quality distortion of modifying PUs is theoretically analyzed, which illustrates that modifying PUs only has a little negative effect on visual quality. Therefore, the data hiding method in this article allows to modify all types of PUs except for <inline-formula><tex-math notation="LaTeX">$2N\times 2N$</tex-math></inline-formula> to each other according to the secret data. In this way, high embedding efficiency is achieved, and the PU distributions in stego-videos can be kept similar to those of cover-videos, which is essential for resisting steganalysis. Meanwhile, a super-resolution convolutional neural network(CNN) with wide residual-net filter(WRNF) is proposed to replace the in-loop filter in HEVC for reconstructing I-pictures, which results in more precisely predicted P-pictures, and it further leads to less bitrate cost and better visual quality of stego-videos. The experimental results show that the proposed PWRN successfully resists the latest PU-targeted steganalysis algorithms, and compared with the state-of-the-art work, PWRN has achieved the lowest bitrate cost and the highest visual quality under the same capacity.

Bandwidth Improvement in Ultrasound Image Reconstruction Using Deep Learning Techniques.

Ultrasound (US) imaging is a medical imaging modality that uses the reflection of sound in the range of 2-18 MHz to image internal body structures. In US, the frequency bandwidth (BW) is directly associated with image resolution. BW is a property of the transducer and more bandwidth comes at a higher cost. Thus, methods that can transform strongly bandlimited ultrasound data into broadband data are essential. In this work, we propose a deep learning (DL) technique to improve the image quality for a given bandwidth by learning features provided by broadband data of the same field of view. Therefore, the performance of several DL architectures and conventional state-of-the-art techniques for image quality improvement and artifact removal have been compared on in vitro US datasets. Two training losses have been utilized on three different architectures: a super resolution convolutional neural network (SRCNN), U-Net, and a residual encoder decoder network (REDNet) architecture. The models have been trained to transform low-bandwidth image reconstructions to high-bandwidth image reconstructions, to reduce the artifacts, and make the reconstructions visually more attractive. Experiments were performed for 20%, 40%, and 60% fractional bandwidth on the original images and showed that the improvements obtained are as high as 45.5% in RMSE, and 3.85 dB in PSNR, in datasets with a 20% bandwidth limitation.

Deep learning-based computed tomographic image super-resolution via wavelet embedding

Effort to realize high-resolution medical images have been made steadily. In particular, super resolution technology based on deep learning is making excellent achievement in computer vision recently. In this study, we developed a model that can dramatically increase the spatial resolution of medical images using deep learning technology, and we try to demonstrate the superiority of proposed model by analyzing it quantitatively. We simulated the computed tomography images with various detector pixel size and tried to restore the low-resolution image to high resolution image. We set the pixel size to 0.5, 0.8 and 1 mm2 for low resolution image and the high-resolution image, which were used for ground truth, was simulated with 0.25 mm2 pixel size. The deep learning model that we used was a fully convolution neural network based on residual structure. The result image demonstrated that proposed super resolution convolution neural network improve image resolution significantly. We also confirmed that PSNR and MTF was improved up to 38% and 65% respectively. The quality of the prediction image is not significantly different depending on the quality of the input image. In addition, the proposed technique not only increases image resolution but also has some effect on noise reduction. In conclusion, we developed deep learning architectures for improving image resolution of computed tomography images. We quantitatively confirmed that the proposed technique effectively improves image resolution without distorting the anatomical structures.

Multi-view and multi-scale super-resolution method of logging curves based on fractal theory.

With the increasing development of unconventional reservoirs around the world, there is an increasing need to enhance the level of geological characterization. Obviously, since the well loggings are one of the most important data to obtain a fine-scale reservoir model, obtaining well loggings with sufficiently high vertical resolution has always been an important issue and challenge. Therefore, due to its low cost and less time-consuming, employing an advanced signal processing technique to enhance the vertical resolution of loggings has always been a research hotspot in the relevant literature. However, non-homogeneity of the target reservoir is not taken into account in the traditional methods such as vertical resolution matching and spline function interpolation. Furthermore, for state-of-the-art methods that employ different versions of shallow or deep machine learning models, how to adequately exploit the multi-scale shape and temporal information in the employed loggings has been a very challenging problem. To address the above problems, by combining the fractal theory with the long short-term memory network technique, a novel multi-view and multi-scale logging resolution enhancing method was proposed in this paper to make full use of the self-shape-similarity and temporal correlation information in the logging data. Experimental results show that, compared with bi-cubic linear interpolation, sparse representation, super-resolution convolutional neural network and random forest, more promising supper-resolution results can be obtained using the proposed method.

MUSTFN: A spatiotemporal fusion method for multi-scale and multi-sensor remote sensing images based on a convolutional neural network

• A deep learning spatiotemporal fusion method for large area remote sensing images. • Multi-constraint loss function to improve model accuracy. • Multi-scale features to avoid the influence of different sensor registration errors on fusion results. • Model is suitable for drastically changing remote sensing image fusion scenarios. • Model has strong transferability for remote sensing images at different times and locations. Spatiotemporal data fusion is a commonly-used and well-proven technique to enhance the application potential of multi-source remote sensing images. However, most existing methods have trouble in generating quality fusion results when areas covered by the images undergoes rapid land cover changes or images have substantial registration errors. While deep learning algorithms have demonstrated their capabilities for imagery fusion, it is challenging to apply deep-learning-based fusion methods in regions that experiences persistent cloud covers and have limited cloud-free imagery observations. To address these challenges, we developed a Multi-scene Spatiotemporal Fusion Network (MUSTFN) algorithm based on a Convolutional Neural Network (CNN). Our approach uses multi-level features to fuse images at different resolutions acquired by multiple sensors. Furthermore, MUSTFN uses the multi-scale features to overcome the effects of geometric registration errors between different images. Additionally, a multi-constrained loss function is proposed to improve the accuracy of imagery fusion over large areas and solve fusion and gap-filling problems simultaneously by utilizing cloud-contaminated images with the fine-tuning method. Compared with several commonly-used methods, our proposed MUSTFN performs better in fusing the 30-m Landsat-7 images and 500-m MODIS images over a small area that has undergone large changes (the average relative Mean Absolute Errors (rMAE) of the first four bands are 6.8% by MUSTFN as compared to 14.1% by the Enhanced Spatial and Temporal Adaptive Reflectance Fusion Model (ESTARFM), 12.8% by the Flexible Spatiotemporal Data Fusion (FSDAF), 8.4% by the Extended Super-Resolution Convolutional Neural Network (ESRCNN), 8.1% by the Spatiotemporal Fusion Using a Generative Adversarial Network (STFGAN)). In particularly for images at different resolutions with different registration accuracies (e.g., 16-m Chinese GaoFen-1 and 500-m MODIS), MUSTFN achieved fusion results of good quality with an average rMAE of 9.3% in spectral reflectance at the first four bands. Finally, we demonstrated the applicability of MUSTFN (average rMAE of 9.18%) when fusing long-term Landsat-8 composite images and MODIS images over a large region (830 km × 600 km). Overall, our results suggest the effectiveness of MUSTFN to address the challenges in imagery fusion, including rapid land cover changes between image acquisition dates, geometric misregistration between images and limited availabilities of cloud-free images. The program of MUSTFN is freely available at: https://github.com/qpyeah/MUSTFN .

An infrared image detection of power equipment based on super‐resolution reconstruction and YOLOv4

Infrared imaging technology is widely used in the fault diagnosis of power equipment. The power equipment images obtained by infrared thermal cameras generally have the characteristics of low resolution, low contrast, similar geometric features, and complex image backgrounds. The traditional infrared image detection methods have problems of low detection accuracy, low real-time performance, and missed and fault detection, all of which will lead to great difficulties in the automatic identification process of infrared images of power equipment. This paper adopted an algorithm based on You Only Look Once (YOLO) v4 to realize the detection of electrical equipment under infrared images. First of all, the authors utilized the Fast Super-Resolution Convolutional Neural Network (FSRCNN) algorithm to perform super-resolution reconstruction on the infrared images of power equipment to achieve image enhancement. Moreover, the lightweight network Mobilenetv3 was selected as the backbone network of YOLOv4 to improve the running speed by reducing network parameters. Finally, the performance of the YOLOv4 algorithm was compared to verify the effectiveness of the algorithm proposed in this paper. The experimental results showed that the proposed method had strong generalization ability and reliable detection of infrared targets of various types of power equipment, with the mean average precision (mAP) reaching 91.3%.