Super-resolution Reconstruction Research Articles

Multispectral and hyperspectral images fusion based on subspace representation and nonlocal low-rank regularization

ABSTRACT Multispectral image (MSI) and hyperspectral image (HSI) fusion is a popular method for HSI super-resolution reconstruction (HSI-SR). MSI-HSI fusion problem is ill-posed and demands several image priors or regularization terms to solve accurately, which is a challenging issue. In this paper, we propose an MSI-HSI fusion model via subspace representation and nonlocal low-rank regularization (SRNLRR). SRNLRR model incorporates the global spectral correlations and spatial nonlocal similarities of HSI to improve the fusion results, where the priors complement each other. First, we use the mode-n tensor-matrix product to project latent high spatial resolution HSI (HR-HSI) into spectral subspace, which can capture spectral low-rankness and reduce computational complexity. Then, based on low-rank representation (LRR) and nonlocal processing strategy, we design a spatial nonlocal LRR regularization (spa-NLRR) and a spectral global LRR regularization (spe-GLRR). These two regularizations analyze the spatial nonlocal similarities and spectral global correlations from intermediate-level vision. Finally, we use the residual regularization program to obtain more image information and input it into the fusion model. We use alternate minimization (AM) methods to optimize the SRNLRR model and employ the alternating direction method (ADM) on spatial/spectral LRR learning. Comparison experiments between the SRNLRR method and six advanced methods on three HSI datasets indicate the superiority of our method.

Super-Resolution Microscopy with Dense Grid After Interpolation

The conventional camera image’s pixel size of super-resolution (SR) microscopy is almost the point spread function’s standard deviation, and the grid of a SR image is 1/8 of the pixel size in conventional compressed sensing-based SR microscopy. Here, based on smaller grid size and smaller pixel size, we proposed and generated different measurement matrices, and then compared and analyzed the SR reconstruction results based on the interpolated conventional camera image and different measurement matrices. The quality of the measurement matrix is related to the interpolation’s size. The larger the interpolation’s size, the better its performance. The quality of SR reconstruction depends not only on the measurement matrix’s performance, but also on the grid size. It is found that dense grid based on the size of interpolation equal to 2 can help to obtain the best SR reconstruction in simulation experiments when added Gaussian noise is lower.

A hybrid deep learning scheme for MRI-based preliminary multiclassification diagnosis of primary brain tumors.

The diagnosis and treatment of brain tumors have greatly benefited from extensive research in traditional radiomics, leading to improved efficiency for clinicians. With the rapid development of cutting-edge technologies, especially deep learning, further improvements in accuracy and automation are expected. In this study, we explored a hybrid deep learning scheme that integrates several advanced techniques to achieve reliable diagnosis of primary brain tumors with enhanced classification performance and interpretability. This study retrospectively included 230 patients with primary brain tumors, including 97 meningiomas, 66 gliomas and 67 pituitary tumors, from the First Affiliated Hospital of Yangtze University. The effectiveness of the proposed scheme was validated by the included data and a commonly used data. Based on super-resolution reconstruction and dynamic learning rate annealing strategies, we compared the classification results of several deep learning models. The multi-classification performance was further improved by combining feature transfer and machine learning. Classification performance metrics included accuracy (ACC), area under the curve (AUC), sensitivity (SEN), and specificity (SPE). In the deep learning tests conducted on two datasets, the DenseNet121 model achieved the highest classification performance, with five-test accuracies of 0.989 ± 0.006 and 0.967 ± 0.013, and AUCs of 0.999 ± 0.001 and 0.994 ± 0.005, respectively. In the hybrid deep learning tests, LightGBM, a promising classifier, achieved accuracies of 0.989 and 0.984, which were improved from the original deep learning scheme of 0.987 and 0.965. Sensitivities for both datasets were 0.985, specificities were 0.988 and 0.984, respectively, and relatively desirable receiver operating characteristic (ROC) curves were obtained. In addition, model visualization studies further verified the reliability and interpretability of the results. These results illustrated that deep learning models combining several advanced technologies can reliably improve the performance, automation, and interpretability of primary brain tumor diagnosis, which is crucial for further brain tumor diagnostic research and individualized treatment.

Iterative Adaptive Based Multi-Polarimetric SAR Tomography of the Forested Areas

Synthetic aperture radar tomography (TomoSAR) is an extension of synthetic aperture radar (SAR) imaging. It introduces the synthetic aperture principle into the elevation direction to achieve three-dimensional (3-D) reconstruction of the observed target. Compressive sensing (CS) is a favorable technology for sparse elevation recovery. However, for the non-sparse elevation distribution of the forested areas, if CS is selected to reconstruct it, it is necessary to utilize some orthogonal bases to first represent the elevation reflectivity sparsely. The iterative adaptive approach (IAA) is a non-parametric algorithm that enables super-resolution reconstruction with minimal snapshots, eliminates the need for hyperparameter optimization, and requires fewer iterations. This paper introduces IAA to tomographicinversion of the forested areas and proposes a novel multi-polarimetric-channel joint 3-D imaging method. The proposed method relies on the characteristics of the consistent support of the elevation distribution of different polarimetric channels and uses the L2-norm to constrain the IAA-based 3-D reconstruction of each polarimetric channel. Compared with typical spectral estimation (SE)-based algorithms, the proposed method suppresses the elevation sidelobes and ambiguity and, hence, improves the quality of the recovered 3-D image. Compared with the wavelet-based CS algorithm, it reduces computational cost and avoids the influence of orthogonal basis selection. In addition, in comparison to the IAA, it demonstrates greater accuracy in identifying the support of the elevation distribution in forested areas. Experimental results based on BioSAR 2008 data are used to validate the proposed method.

Image Super-Resolution Reconstruction in Sports Scenarios and Its Application in Motion Analysis

Image Super-Resolution Reconstruction in Sports Scenarios and Its Application in Motion Analysis

Super-resolution Deep Learning Reconstruction Cervical Spine 1.5T MRI: Improved Interobserver Agreement in Evaluations of Neuroforaminal Stenosis Compared to Conventional Deep Learning Reconstruction.

The aim of this study was to investigate whether super-resolution deep learning reconstruction (SR-DLR) is superior to conventional deep learning reconstruction (DLR) with respect to interobserver agreement in the evaluation of neuroforaminal stenosis using 1.5T cervical spine MRI. This retrospective study included 39 patients who underwent 1.5T cervical spine MRI. T2-weighted sagittal images were reconstructed with SR-DLR and DLR. Three blinded radiologists independently evaluated the images in terms of the degree of neuroforaminal stenosis, depictions of the vertebrae, spinal cord and neural foramina, sharpness, noise, artefacts and diagnostic acceptability. In quantitative image analyses, a fourth radiologist evaluated the signal-to-noise ratio (SNR) by placing a circular or ovoid region of interest on the spinal cord, and the edge slope based on a linear region of interest placed across the surface of the spinal cord. Interobserver agreement in the evaluations of neuroforaminal stenosis using SR-DLR and DLR was 0.422-0.571 and 0.410-0.542, respectively. The kappa values between reader 1 vs. reader 2 and reader 2 vs. reader 3 significantly differed. Two of the three readers rated depictions of the spinal cord, sharpness, and diagnostic acceptability as significantly better with SR-DLR than with DLR. Both SNR and edge slope (/mm) were also significantly better with SR-DLR (12.9 and 6031, respectively) than with DLR (11.5 and 3741, respectively) (p < 0.001 for both). In conclusion, compared to DLR, SR-DLR improved interobserver agreement in the evaluations of neuroforaminal stenosis using 1.5T cervical spine MRI.

Enhanced Video Super-Resolution Network towards Compressed Data

Video super-resolution (VSR) algorithms aim at recovering a temporally consistent high-resolution (HR) video from its corresponding low-resolution (LR) video sequence. Due to the limited bandwidth during video transmission, most available videos on the internet are compressed. Nevertheless, few existing algorithms consider the compression factor in practical applications. In this paper, we propose an enhanced VSR model towards compressed videos, termed as ECVSR, to simultaneously achieve compression artifacts reduction and SR reconstruction end-to-end. ECVSR contains a motion-excited temporal adaption network (METAN) and a multi-frame SR network (SRNet). The METAN takes decoded LR video frames as input and models inter-frame correlations via bidirectional deformable alignment and motion-excited temporal adaption, where temporal differences are calculated as motion prior to excite the motion-sensitive regions of temporal features. In SRNet, cascaded recurrent multi-scale blocks (RMSB) are employed to learn deep spatio-temporal representations from adapted multi-frame features. Then, we build a reconstruction module for spatio-temporal information integration and HR frame reconstruction, which is followed by a detail refinement module for texture and visual quality enhancement. Extensive experimental results on compressed videos demonstrate the superiority of our method for compressed VSR. Code will be available at https://github.com/lifengcs/ECVSR .

Multi-scale perceptual YOLO for automatic detection of clue cells and trichomonas in fluorescence microscopic images

Vaginitis is a common disease among women and has a high recurrence rate. The primary diagnosis method is fluorescence microscopic inspection, but manual inspection is inefficient and can lead to false detection or missed detection. Automatic cell identification and localization in microscopic images are necessary. For vaginitis diagnosis, clue cells and trichomonas are two important indicators and are difficult to be detected because of the different scales and image characteristics. This study proposes a Multi-Scale Perceptual YOLO (MSP-YOLO) with super-resolution reconstruction branch to meet the detection requirements of clue cells and trichomonas. Based on the scales and image characteristics of clue cells and trichomonas, we employed a super-resolution reconstruction branch to the detection network. This branch guides the detection branch to focus on subtle feature differences. Simultaneously, we proposed an attention-based feature fusion module that is injected with dilated convolutional group. This module makes the network pay attention to the non-centered features of the large target clue cells, which contributes to the enhancement of detection sensitivity. Experimental results show that the proposed detection network MSP-YOLO can improve sensitivity without compromising specificity. For clue cell and trichomoniasis detection, the proposed network achieved sensitivities of 0.706 and 0.910, respectively, which were 0.218 and 0.051 higher than those of the baseline model. In this study, the characteristics of the super-resolution reconstruction task are used to guide the network to effectively extract and process image features. The novel proposed network has an increased sensitivity, which makes it possible to detect vaginitis automatically.

Dual-branch feature encoding framework for infrared images super-resolution reconstruction

Infrared thermal imaging is a passive non-contact detection and identification technology, which is not subject to electromagnetic infection and good concealment, is widely used in military and commercial fields. However, due to the limitations of the existing infrared imaging system mechanisms, the spatial resolution of the acquired infrared images is low and the edge details are blurred, which in turn leads to poor performance in downstream missions based on infrared images. In this paper, in order to better solve the above problems, we propose a new super-resolution reconstruction framework for infrared images, called DBFE, which extracts and retains abundant structure and textual information for robust infrared image high-resolution reconstruction with a novel structure-textual encoder module. Extensive experiment demonstrates that our proposed method achieves significantly superior contraband high-resolution reconstruction results on the multiple dataset compared to progressive methods for high resolution infrared image reconstruction, effectively proving the practicability of the method proposed in this paper.

SwinDPSR: Dual-Path Face Super-Resolution Network Integrating Swin Transformer

Whether to use face priors in the face super-resolution (FSR) methods is a symmetry problem.Various face priors are used to describe the overall and local face features, making the generation of super-resolution face images expensive and laborious. FSR methods that do not require any prior information tend to focus too much on the local features of the face, ignoring the modeling of global information. To solve this problem, we propose a dual-path facial image super-resolution network (SwinDPSR) fused with Swin Transformer. The network does not require additional face priors, and it learns global face shape and local face components through two independent branches. In addition, the channel attention ECA module is used to aggregate the global and local face information in the above dual-path sub-networks, which can generate corresponding high-quality face images. The results of face super-resolution reconstruction experiments on public face datasets and a real-scene face dataset show that SwinDPSR is superior to previous advanced methods both in terms of visual effects and objective indicators. The reconstruction results are evaluated with four evaluation metrics: peak signal-to-noise ratio (PSNR), structural similarity (SSIM), learned perceptual image patch similarity (LPIPS), and mean perceptual score (MPS).

Infrared image super-resolution reconstruction based on residual fast fourier transform

Infrared image super-resolution reconstruction based on residual fast fourier transform

A Super-Resolution Reconstruction Model for Remote Sensing Image Based on Generative Adversarial Networks

In current times, reconstruction of remote sensing images using super-resolution is a prominent topic of study. Remote sensing data have a complex spatial distribution. Compared with natural pictures, remote sensing pictures often contain subtler and more complicated information. Most super-resolution reconstruction algorithms cannot restore all the information contained in remote sensing images when reconstructing them. The content of some areas in the reconstructed images may be too smooth, and some areas may even have color changes, resulting in lower quality reconstructed images. In response to the problems presenting in current reconstruction algorithms about super-resolution, this article proposes the SRGAN-MSAM-DRC model (SRGAN model with multi-scale attention mechanism and dense residual connection). This model roots in generative adversarial networks and incorporates multi-scale attention mechanisms and dense residual connections into the generator. Furthermore, residual blocks are incorporated into the discriminator. We use some remote sensing image datasets of real-world data to evaluate this model, and the results indicate the SRGAN-MSAM-DRC model has shown enhancements in three evaluation metrics for reconstructed images about super-resolution. Compared to the basic SRGAN model, the SSIM (structural similarity), PSNR (peak signal-to-noise ratio), and IE (image entropy) increase by 5.0%, 4.0%, and 4.1%, respectively. From the results, we know the quality of the reconstructed images of remote sensing using the SRGAN-MSAM-DRC model is better than basic SRGAN model, and verifies that the model has good applicability and performance in reconstruction of remote sensing images using super-resolution.

Accurate blood glucose level monitoring using microwave imaging

Painless and non-invasive detection techniques are needed to replace finger-prick blood collection for people with diabetes. A first-of-its-kind, noninvasive, and continuous blood glucose level (BGL) detection method based on microwave imaging is introduced in this paper. This method avoids the complex task of frequency choice for the design of electromagnetic sensors. A radar-based microwave imaging technology combined with an improved very-deep super-resolution (VDSR-BL) method is presented to obtain high-resolution (HR) microwave images. After image super-resolution reconstruction by VDSR-BL, the peak signal-to-noise ratio and structural similarity index of HR images reach 35.4461 dB and 0.9761, respectively. Then, an ensemble learning strategy based on support vector regression and random forest algorithms is proposed to identify HR microwave images for BGL estimation. The developed detection system has been verified on the medium under tests with different glucose solutions. The final detection results obtain a root mean squared error of 0.1394 mg ml−1 and a mean absolute relative difference of 8.02%, which show good accuracy with clinical acceptance. Meanwhile, we also conducted human trials. A high correlation coefficient (R) of 0.9254 was achieved between the results of microwave imaging and invasive BGL. Together, these results show that microwave imaging offers a promising new approach for noninvasive BGL monitoring.

Single image multi-scale enhancement for rock Micro-CT super-resolution using residual U-Net

Micro-CT, also known as X-ray micro-computed tomography, has emerged as the primary instrument for pore-scale properties study in geological materials. Several studies have used deep learning to achieve super-resolution reconstruction in order to balance the trade-off between resolution of CT images and field of view. Nevertheless, most existing methods only work with single-scale CT scans, ignoring the possibility of using multi-scale image features for image reconstruction. In this study, we proposed a super-resolution approach via multi-scale fusion using residual U-Net for rock micro-CT image reconstruction (MS-ResUnet). The residual U-Net provides an encoder-decoder structure. In each encoder layer, several residual sequential blocks and improved residual blocks are used. The decoder is composed of convolutional ReLU residual blocks and residual chained pooling blocks. During the encoding-decoding method, information transfers between neighboring multi-resolution images are fused, resulting in richer rock characteristic information. Qualitative and quantitative comparisons of sandstone, carbonate, and coal CT images demonstrate that our proposed algorithm surpasses existing approaches. Our model accurately reconstructed the intricate details of pores in carbonate and sandstone, as well as clearly visible coal cracks.

Improving the spatial resolution of solar images using super-resolution diffusion generative adversarial networks

High-spatial-resolution solar images contribute to the study of small-scale structures on the Sun. The Helioseismic and Magnetic Imager (HMI) conducts continuous full-disk observations of the Sun at a fixed cadence, accumulating a wealth of observational data. However, the spatial resolution of HMI images is not sufficient to analyze the small-scale structures of solar activity. We present a new super-resolution (SR) method based on generative adversarial networks (GANs) and denoising diffusion probabilistic models (DDPMs) that can increase the spatial resolution of HMI images by a factor four. We propose a method called super-resolution diffusion GANs (SDGAN), which combines GANs and DDPMs for the SR reconstruction of HMI images. SDGAN progressively maps low-resolution (LR) images to high-resolution (HR) images through a conditional denoising process. It employs conditional GANs to simulate the denoising distribution and optimizes model results using nonsaturating adversarial loss and perceptual loss. This approach enables fast and high-quality reconstruction of solar images. We used high-spatial-resolution images from the Goode Solar Telescope (GST) as HR images and created a data set consisting of paired images from HMI and GST. We then used this data set to train SDGAN for the purpose of reconstructing HMI images with four times the original spatial resolution. The experimental results demonstrate that SDGAN can obtain high-quality HMI reconstructed images with just four denoising steps.

Target Image Processing based on Super-resolution Reconstruction and Machine Learning Algorithm

This article proposes a target image processing method based on super-resolution reconstruction and machine learning algorithms, which solves the low-resolution problem in medical images during imaging. This method uses nonlocal autoregressive learning based on a medical image super-resolution reconstruction method. The autoregressive model is introduced into the sparse representation-based medical image super-resolution reconstruction model by utilizing medical image data inherent nonlocal similarity characteristics. At the same time, a clustering algorithm is used to obtain a classification dictionary, improving experimental efficiency. The experimental results show that ten randomly selected CT/MR images are used as test images, and each image's peak signal-to-noise ratio and structural similarity values are calculated separately. Compared with other methods, the method proposed in this paper is significantly better and can achieve ideal results, with the highest value being 31.49. This method demonstrates the feasibility of using super-resolution reconstruction and machine learning algorithms in medical image resolution.

Super-Resolution for Land Surface Temperature Retrieval Images via Cross-Scale Diffusion Model Using Reference Images

Geothermal resources are efficient, clean, and renewable energy sources. Using high-resolution images captured by remote sensing satellites for temperature retrieval and searching for geothermal anomaly areas is an efficient method. However, obtaining land surface temperature retrieval images requires multiple steps of calculation, which can result in a great loss of image information and resolution. Therefore, the super-resolution reconstruction of LST retrieval images is currently a challenge in geothermal resource exploration. Although the current super-resolution methods for LST retrieval images can appropriately restore image quality, the overall restoration of the surface temperature information in the region is still not ideal. We propose a cross-scale reference image super-resolution model based on a diffusion model using deep learning technology. First, we propose the Pre-Super-Resolution Network (PreNet), which can improve both indices and the visual effect of images. Second, to reduce the white noise in the super-resolution images, we propose the Cross-Scale Reference Image Attention Mechanism (CSRIAM). The introduction of this mechanism greatly reduces noise in the images and improves the overall image quality. Compared to previous methods, we improved both experimental indices such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), etc., and vision quality, and optimized the recovery of geothermal anomalies. Through our experimental results, we found that the CS-Diffusion model has a very strong ability to restore the image quality of the LST retrieval. After restoring its image quality, we can make a positive contribution to subsequent geothermal resource exploration.

Dynamic Unstructured Pruning Neural Network Image Super-resolution Reconstruction

Dynamic Unstructured Pruning Neural Network Image Super-resolution Reconstruction

3D whole heart k-space-based super-resolution cardiac T1 mapping using rotated stacks

Objective. To provide three-dimensional (3D) whole-heart high-resolution isotropic cardiac T1 maps using a k-space-based through-plane super-resolution reconstruction (SRR) with rotated multi-slice stacks. Approach. Due to limited SNR and cardiac motion, often only 2D T1 maps with low through-plane resolution (4–8 mm) can be obtained. Previous approaches used SRR to calculate 3D high-resolution isotropic cardiac T1 maps. However, they were limited to the ventricles. The proposed approach acquires rotated stacks in long-axis orientation with high in-plane resolution but low through-plane resolution. This results in radially overlapping stacks from which high-resolution T1 maps of the whole heart are reconstructed using a k-space-based SRR framework considering the complete acquisition model. Cardiac and residual respiratory motion between different breath holds is estimated and incorporated into the reconstruction. The proposed approach was evaluated in simulations and phantom experiments and successfully applied to ten healthy subjects.Main results. 3D T1 maps of the whole heart were obtained in the same acquisition time as previous methods covering only the ventricles. T1 measurements were possible even for small structures, such as the atrial wall. The proposed approach provided accurate (P > 0.4; R 2 > 0.99) and precise T1 values (SD of 64.32 ± 22.77 ms in the proposed approach, 44.73 ± 31.9 ms in the reference). The edge sharpness of the T1 maps was increased by 6.20% and 4.73% in simulation and phantom experiments, respectively. Contrast-to-noise ratios between the septum and blood pool increased by 14.50% in in vivo measurements with a k-space compared to an image-space-based SRR. Significance. The proposed approach provided whole-heart high-resolution 1.3 mm isotropic T1 maps in an overall acquisition time of approximately three minutes. Small structures, such as the atrial and right ventricular walls, could be visualized in the T1 maps.

Sparse learning model with embedded RIP conditions for turbulence super-resolution reconstruction

In practical engineering scenarios, constraints arising from sensor placement, quantity, and the limitations of current testing technologies often lead to turbulence data characterized by low resolution and irregular structures. Turbulence super-resolution reconstruction is crucial for extracting finer details from irregularly structured, low-resolution measurement data, thereby facilitating comprehensive flow field analyses. This article introduces a sparse learning model (Embedding Restricted Isometry Property Autoencoder (RIP-AE)) for achieving super-resolution reconstruction of flow fields by obtaining compressed representations and sparse transform domain information. In this model, we embed the Restricted Isometry Property (RIP) condition to ensure the effectiveness of compressed representations and sparse transform domains, thereby enhancing the super-resolution reconstruction accuracy of the model. To assess the performance of the RIP-AE model, we utilized data sets of flow around a cylinder at different Reynolds numbers generated by unsteady Reynolds-Averaged Navier–Stokes simulations. We sequentially validate the effectiveness of the RIP condition at different Reynolds numbers (ranging from 1,000 to 500,000) and compare the reconstruction results of the RIP-AE model with those of the downsampled skip-connection/multi-scale (DSC/MS) and MS-AE models under different sampling ratios. The results indicate that the RIP-AE model excels in terms of L2 relative error and demonstrates the capability to achieve high-precision flow field reconstruction even under high sampling ratios.