Super-resolution Reconstruction Research Articles

Image Super-Resolution Reconstruction by Huber Regularization and Tailored Finite Point Method

Image Super-Resolution Reconstruction by Huber Regularization and Tailored Finite Point Method

Inter-scanner super-resolution of 3D cine MRI using a transfer-learning network for MRgRT

Objective. Deep-learning networks for super-resolution (SR) reconstruction enhance the spatial-resolution of 3D magnetic resonance imaging (MRI) for MR-guided radiotherapy (MRgRT). However, variations between MRI scanners and patients impact the quality of SR for real-time 3D low-resolution (LR) cine MRI. In this study, we present a personalized super-resolution (psSR) network that incorporates transfer-learning to overcome the challenges in inter-scanner SR of 3D cine MRI. Approach: Development of the proposed psSR network comprises two-stages: (1) a cohort-specific SR (csSR) network using clinical patient datasets, and (2) a psSR network using transfer-learning to target datasets. The csSR network was developed by training on breath-hold and respiratory-gated high-resolution (HR) 3D MRIs and their k-space down-sampled LR MRIs from 53 thoracoabdominal patients scanned at 1.5 T. The psSR network was developed through transfer-learning to retrain the csSR network using a single breath-hold HR MRI and a corresponding 3D cine MRI from 5 healthy volunteers scanned at 0.55 T. Image quality was evaluated using the peak-signal-noise-ratio (PSNR) and the structure-similarity-index-measure (SSIM). The clinical feasibility was assessed by liver contouring on the psSR MRI using an auto-segmentation network and quantified using the dice-similarity-coefficient (DSC). Results. Mean PSNR and SSIM values of psSR MRIs were increased by 57.2% (13.8–21.7) and 94.7% (0.38–0.74) compared to cine MRIs, with the reference 0.55 T breath-hold HR MRI. In the contour evaluation, DSC was increased by 15% (0.79–0.91). Average time consumed for transfer-learning was 90 s, psSR was 4.51 ms per volume, and auto-segmentation was 210 ms, respectively. Significance. The proposed psSR reconstruction substantially increased image and segmentation quality of cine MRI in an average of 215 ms across the scanners and patients with less than 2 min of prerequisite transfer-learning. This approach would be effective in overcoming cohort- and scanner-dependency of deep-learning for MRgRT.

Super-resolution reconstruction of single image for latent features

AbstractSingle-image super-resolution (SISR) typically focuses on restoring various degraded low-resolution (LR) images to a single high-resolution (HR) image. However, during SISR tasks, it is often challenging for models to simultaneously maintain high quality and rapid sampling while preserving diversity in details and texture features. This challenge can lead to issues such as model collapse, lack of rich details and texture features in the reconstructed HR images, and excessive time consumption for model sampling. To address these problems, this paper proposes a Latent Feature-oriented Diffusion Probability Model (LDDPM). First, we designed a conditional encoder capable of effectively encoding LR images, reducing the solution space for model image reconstruction and thereby improving the quality of the reconstructed images. We then employed a normalized flow and multimodal adversarial training, learning from complex multimodal distributions, to model the denoising distribution. Doing so boosts the generative modeling capabilities within a minimal number of sampling steps. Experimental comparisons of our proposed model with existing SISR methods on mainstream datasets demonstrate that our model reconstructs more realistic HR images and achieves better performance on multiple evaluation metrics, providing a fresh perspective for tackling SISR tasks.

Lightweight interactive feature inference network for single-image super-resolution

The emergence of convolutional neural network (CNN) and transformer has recently facilitated significant advances in image super-resolution (SR) tasks. However, these networks commonly construct complex structures, having huge model parameters and high computational costs, to boost reconstruction performance. In addition, they do not consider the structural prior well, which is not conducive to high-quality image reconstruction. In this work, we devise a lightweight interactive feature inference network (IFIN), complementing the strengths of CNN and Transformer, for effective image SR reconstruction. Specifically, the interactive feature aggregation module (IFAM), implemented by structure-aware attention block (SAAB), Swin Transformer block (SWTB), and enhanced spatial adaptive block (ESAB), serves as the network backbone, progressively extracts more dedicated features to facilitate the reconstruction of high-frequency details in the image. SAAB adaptively recalibrates local salient structural information, and SWTB effectively captures rich global information. Further, ESAB synergetically complements local and global priors to ensure the consistent fusion of diverse features, achieving high-quality reconstruction of images. Comprehensive experiments reveal that our proposed networks attain state-of-the-art reconstruction accuracy on benchmark datasets while maintaining low computational demands. Our code and results are available at: https://github.com/wwaannggllii/IFIN.

Super resolution reconstruction of background oriented schlieren displacement field based on the convolution neural network with the gradient loss function

Abstract In order to refine the displacement field of the background oriented schlieren method, a super-resolution method based on deep learning has been newly proposed in the study, and were compared with the bicubic interpolation. The gradient loss functions were firstly introduced into the hybrid downsampled skip-connection/multi-scale (DSC/MS) model to improve the reconstruction effect. The reconstruction effects of the new loss functions were compared with that of the traditional MSE loss function. The results showed that the Laplace operator with average pooling had better performance than the origin loss function in all the indexes including PSNR, MSE, MSE of the gradient and the max MSE. In these four indexes, MSE of the gradient and the max MSE performed especially better than the others, where the MSE of the gradient reduced from 3.90E-05 to 3.30E-05 and the max MSE reduced from 0.392 to 0.360.

Parallel compressive super-resolution imaging with wide field-of-view based on physics enhanced network

Parallel compressive super-resolution imaging has attracted increasing attention in recent years. However, the super-resolution quality depends on modulation masks and reconstruction algorithms. A deep-learning method provides an efficient solution, but in wide field-of-view (FOV) scenarios, the differences between optical transfer functions (OTFs) of each pixel increase the system complexity and limit its practical application. This study proposed a wide FOV parallel compressive super-resolution imaging approach based on a physics-enhanced network. First, the network and modulation masks of an arbitrary 128 × 128-pixel region were trained; then, the trained network was fine-tuned for the rest of the 128 × 128-pixel regions in the entire wide FOV, which effectively eliminated the OTF variability. Numerical simulations and practical experiments demonstrated that through the proposed approach, super-resolution images of 1020 × 1500 pixels can be reconstructed from 272 × 400-pixel low-resolution measurements using only three designed masks, with the resolution enhanced 3.75×3.75 times and the peak signal-to-noise ratio improved by 89.4% compared to the results of the previous compressed sensing algorithm. Besides, the training time was dramatically reduced by 95.5-fold compared with the traditional training strategy for each region alone. This approach decreases the imaging complexity of wide FOV and achieves the high-quality super-resolution reconstruction under few trained masks, thus we believe it can promote rapid imaging for super-resolution and a wide FOV ranging from infrared to terahertz.

Lightweight Super-Resolution Generative Adversarial Network for SAR Images

Due to a unique imaging mechanism, Synthetic Aperture Radar (SAR) images typically exhibit degradation phenomena. To enhance image quality and support real-time on-board processing capabilities, we propose a lightweight deep generative network framework, namely, the Lightweight Super-Resolution Generative Adversarial Network (LSRGAN). This method introduces Depthwise Separable Convolution (DSConv) in residual blocks to compress the original Generative Adversarial Network (GAN) and uses the SeLU activation function to construct a lightweight residual module (LRM) suitable for SAR image characteristics. Furthermore, we combine the LRM with an optimized Coordinated Attention (CA) module, enhancing the lightweight network’s capability to learn feature representations. Experimental results on spaceborne SAR images demonstrate that compared to other deep generative networks focused on SAR image super-resolution reconstruction, LSRGAN achieves compression ratios of 74.68% in model storage requirements and 55.93% in computational resource demands. In this work, we significantly reduce the model complexity, improve the quality of spaceborne SAR images, and validate the effectiveness of the SAR image super-resolution algorithm as well as the feasibility of real-time on-board processing technology.

RMSRGAN: A Real Multispectral Imagery Super-Resolution Reconstruction for Enhancing Ginkgo Biloba Yield Prediction

Multispectral remote sensing data with abundant spectral information can be used to compute vegetation indices to improve the accuracy of Ginkgo biloba yield prediction. The limited spatial resolution of multispectral cameras restricts the detail capture over wide farmland, but super-resolution (SR) reconstruction methods can enhance image quality. However, most existing SR models have been trained on images processed from downsampled high-resolution (HR) images, making them less effective in reconstructing real low-resolution (LR) images. This study proposes a GAN-based super-resolution reconstruction method (RMSRGAN) for multispectral remote sensing images of Ginkgo biloba trees in real scenes. A U-Net-based network is employed instead of the traditional discriminator. Convolutional block attention modules (CBAMs) are incorporated into the Residual-in-Residual Dense Blocks (RRDBs) of the generator and the U-Net of the discriminator to preserve image details and texture features. An unmanned aerial vehicle (UAV) equipped with a multispectral camera was employed to capture field multispectral remote sensing images of Ginkgo biloba trees at different spatial resolutions. Four matching HR and LR datasets were created from these images to train RMSRGAN. The proposed model outperforms the traditional models by achieving superior results in both quantitative evaluation metrics (peak signal-to-noise ratio (PSNR) is 32.490, 31.085, 27.084, 26.819, and structural similarity index (SSIM) is 0.894, 0.881, 0.832, 0.818, respectively) and qualitative evaluation visualization. Furthermore, the efficiency of our proposed method was tested by generating individual vegetation indices (VIs) from images taken before and after reconstruction to predict the yield of Ginkgo biloba. The results show that the SR images exhibit better R2 and RMSE values than LR images. These findings show that RMSRGAN can improve the spatial resolution of real multispectral images, increasing the accuracy of Ginkgo biloba yield prediction and providing more effective and accurate data support for crop management.

Improved stent sharpness evaluation with Super-Resolution deep learning reconstruction in coronary computed tomography angiography.

This study aimed to assess the impact of super-resolution deep learning reconstruction (SR-DLR) on coronary computed tomography angiography (CCTA) image quality and blooming artifacts from coronary artery stents in comparison to conventional methods, including hybrid iterative reconstruction (HIR) and deep learning-based reconstruction (DLR). A retrospective analysis included sixty-six CCTA patients from July to November 2022. Major coronary arteries were evaluated for image noise, signal-to-noise ratio (SNR), and contrast-to-noise ratio (CNR). Stent sharpness was quantified using 10-90% edge rise slope (ERS) and 10-90% edge rise distance (ERD). Qualitative analysis employed a 5-point scoring system to assess overall image quality, image noise, vessel wall, and stent structure. SR-DLR demonstrated significantly lower image noise compared to HIR and DLR. SNR and CNR were notably higher in SR-DLR. Stent ERS was significantly improved in SR-DLR, with mean ERD values of 0.70 ± 0.20 mm for SR-DLR, 1.13 ± 0.28 mm for HIR, and 0.85 ± 0.26 mm for DLR. Qualitatively, SR-DLR scored higher in all categories. SR-DLR produces images with lower image noise, leading to improved overall image quality, compared with HIR and DLR. SR-DLR is a valuable image reconstruction algorithm for enhancing the spatial resolution and sharpness of coronary artery stents without being constrained by hardware limitations. The overall image quality was significantly higher in SR-DLR, resulting in sharper coronary artery stents compared to HIR and DLR.

Enhancing unsupervised rock CT image super-resolution with non-local attention

Traditional low-resolution rock core images fail to capture the details and features of rocks, limiting the accurate analysis of rock properties. Therefore, researchers have proposed supervised and unsupervised digital rock super-resolution reconstruction methods. Supervised methods require paired data and manual annotation, while unsupervised methods generate high-resolution images through self-learning that are suitable for real-world scenarios with a lack of paired data. However, the current unsupervised methods still face challenges in terms of consistency and feature capturing. To address this problem, a novel unsupervised single-image super-resolution model called NL-CycleGAN is proposed. This model employs a non-local attention-guided CycleGAN framework to effectively capture low-level pixel variations between unpaired source and target images while preserving the overall tone. To evaluate the performance of NL-CycleGAN, we conduct both quantitative and qualitative tests using the DeepRock-SR dataset. In terms of quantitative evaluation, our method achieved the lowest perceptual loss (LPIPS) metric compared to existing methods. Additionally, our model obtained the highest performance in peak signal-to-noise ratio (PSNR), and multi-scale structural similarity index measure (MS-SSIM). In qualitative testing, the comparison images clearly show that our model accurately captures the intricate details of pores in carbonate and sandstone, as well as the complex crack patterns in coal. In conclusion, our model contributes to improving rock resolution and provides more accurate image processing techniques for rock property analysis and geological research.

Multispectral and hyperspectral images fusion based on subspace representation and nonlocal low-rank regularization

ABSTRACT Multispectral image (MSI) and hyperspectral image (HSI) fusion is a popular method for HSI super-resolution reconstruction (HSI-SR). MSI-HSI fusion problem is ill-posed and demands several image priors or regularization terms to solve accurately, which is a challenging issue. In this paper, we propose an MSI-HSI fusion model via subspace representation and nonlocal low-rank regularization (SRNLRR). SRNLRR model incorporates the global spectral correlations and spatial nonlocal similarities of HSI to improve the fusion results, where the priors complement each other. First, we use the mode-n tensor-matrix product to project latent high spatial resolution HSI (HR-HSI) into spectral subspace, which can capture spectral low-rankness and reduce computational complexity. Then, based on low-rank representation (LRR) and nonlocal processing strategy, we design a spatial nonlocal LRR regularization (spa-NLRR) and a spectral global LRR regularization (spe-GLRR). These two regularizations analyze the spatial nonlocal similarities and spectral global correlations from intermediate-level vision. Finally, we use the residual regularization program to obtain more image information and input it into the fusion model. We use alternate minimization (AM) methods to optimize the SRNLRR model and employ the alternating direction method (ADM) on spatial/spectral LRR learning. Comparison experiments between the SRNLRR method and six advanced methods on three HSI datasets indicate the superiority of our method.

Super-Resolution Microscopy with Dense Grid After Interpolation

The conventional camera image’s pixel size of super-resolution (SR) microscopy is almost the point spread function’s standard deviation, and the grid of a SR image is 1/8 of the pixel size in conventional compressed sensing-based SR microscopy. Here, based on smaller grid size and smaller pixel size, we proposed and generated different measurement matrices, and then compared and analyzed the SR reconstruction results based on the interpolated conventional camera image and different measurement matrices. The quality of the measurement matrix is related to the interpolation’s size. The larger the interpolation’s size, the better its performance. The quality of SR reconstruction depends not only on the measurement matrix’s performance, but also on the grid size. It is found that dense grid based on the size of interpolation equal to 2 can help to obtain the best SR reconstruction in simulation experiments when added Gaussian noise is lower.

Efficient blind super-resolution imaging via adaptive degradation-aware estimation

Single image super-resolution (SISR) has achieved prominent success based on deep learning. However, most SISR methods based on the specific degradation pattern, e.g., bicubic interpolation with a bulky network structure, are unsuitable for real-world images reconstructed on edge devices. To mitigate the above limitations, unlike the majority of blind super-resolution methods, which consist of a separate complex degradation kernel projection network and a SR reconstruction network, we propose an end-to-end lightweight multi-degradation oriented network that not only can estimate degradation kernel but also reconstructs HR image fast and accurately. The core components of the designed network adopt a lightweight SR model with region non-local feature similarity learning, i.e., Region Non-Local Feature Block (RNLFB), to establish region-wise global feature correlation. Concretely, more and fewer RNLFBs are used to learn more sophisticated feature representation and relatively simple degradation representation. Then, a degradation kernel projector is equipped to fulfill adaptive degradation-aware estimation from the degradation representation. With the help of the encoded degradation kernel, the reconstruction model can learn to restore a high-resolution image from the feature representation. By doing so, the degradation kernel estimation task and the HR image reconstruction task can be accomplished without designing two complex networks separately, which guides more stable and effective network training under low computational resource costs. Extensive experiments on synthetic and real-world datasets demonstrate that our method can fulfill precise degradation kernel prediction and HR image reconstruction compared with other state-of-the-art SR methods with lower model complexity.

A hybrid deep learning scheme for MRI-based preliminary multiclassification diagnosis of primary brain tumors.

The diagnosis and treatment of brain tumors have greatly benefited from extensive research in traditional radiomics, leading to improved efficiency for clinicians. With the rapid development of cutting-edge technologies, especially deep learning, further improvements in accuracy and automation are expected. In this study, we explored a hybrid deep learning scheme that integrates several advanced techniques to achieve reliable diagnosis of primary brain tumors with enhanced classification performance and interpretability. This study retrospectively included 230 patients with primary brain tumors, including 97 meningiomas, 66 gliomas and 67 pituitary tumors, from the First Affiliated Hospital of Yangtze University. The effectiveness of the proposed scheme was validated by the included data and a commonly used data. Based on super-resolution reconstruction and dynamic learning rate annealing strategies, we compared the classification results of several deep learning models. The multi-classification performance was further improved by combining feature transfer and machine learning. Classification performance metrics included accuracy (ACC), area under the curve (AUC), sensitivity (SEN), and specificity (SPE). In the deep learning tests conducted on two datasets, the DenseNet121 model achieved the highest classification performance, with five-test accuracies of 0.989 ± 0.006 and 0.967 ± 0.013, and AUCs of 0.999 ± 0.001 and 0.994 ± 0.005, respectively. In the hybrid deep learning tests, LightGBM, a promising classifier, achieved accuracies of 0.989 and 0.984, which were improved from the original deep learning scheme of 0.987 and 0.965. Sensitivities for both datasets were 0.985, specificities were 0.988 and 0.984, respectively, and relatively desirable receiver operating characteristic (ROC) curves were obtained. In addition, model visualization studies further verified the reliability and interpretability of the results. These results illustrated that deep learning models combining several advanced technologies can reliably improve the performance, automation, and interpretability of primary brain tumor diagnosis, which is crucial for further brain tumor diagnostic research and individualized treatment.

Iterative Adaptive Based Multi-Polarimetric SAR Tomography of the Forested Areas

Synthetic aperture radar tomography (TomoSAR) is an extension of synthetic aperture radar (SAR) imaging. It introduces the synthetic aperture principle into the elevation direction to achieve three-dimensional (3-D) reconstruction of the observed target. Compressive sensing (CS) is a favorable technology for sparse elevation recovery. However, for the non-sparse elevation distribution of the forested areas, if CS is selected to reconstruct it, it is necessary to utilize some orthogonal bases to first represent the elevation reflectivity sparsely. The iterative adaptive approach (IAA) is a non-parametric algorithm that enables super-resolution reconstruction with minimal snapshots, eliminates the need for hyperparameter optimization, and requires fewer iterations. This paper introduces IAA to tomographicinversion of the forested areas and proposes a novel multi-polarimetric-channel joint 3-D imaging method. The proposed method relies on the characteristics of the consistent support of the elevation distribution of different polarimetric channels and uses the L2-norm to constrain the IAA-based 3-D reconstruction of each polarimetric channel. Compared with typical spectral estimation (SE)-based algorithms, the proposed method suppresses the elevation sidelobes and ambiguity and, hence, improves the quality of the recovered 3-D image. Compared with the wavelet-based CS algorithm, it reduces computational cost and avoids the influence of orthogonal basis selection. In addition, in comparison to the IAA, it demonstrates greater accuracy in identifying the support of the elevation distribution in forested areas. Experimental results based on BioSAR 2008 data are used to validate the proposed method.

Image Super-Resolution Reconstruction in Sports Scenarios and Its Application in Motion Analysis

Image Super-Resolution Reconstruction in Sports Scenarios and Its Application in Motion Analysis

Super-resolution Deep Learning Reconstruction Cervical Spine 1.5T MRI: Improved Interobserver Agreement in Evaluations of Neuroforaminal Stenosis Compared to Conventional Deep Learning Reconstruction.

The aim of this study was to investigate whether super-resolution deep learning reconstruction (SR-DLR) is superior to conventional deep learning reconstruction (DLR) with respect to interobserver agreement in the evaluation of neuroforaminal stenosis using 1.5T cervical spine MRI. This retrospective study included 39 patients who underwent 1.5T cervical spine MRI. T2-weighted sagittal images were reconstructed with SR-DLR and DLR. Three blinded radiologists independently evaluated the images in terms of the degree of neuroforaminal stenosis, depictions of the vertebrae, spinal cord and neural foramina, sharpness, noise, artefacts and diagnostic acceptability. In quantitative image analyses, a fourth radiologist evaluated the signal-to-noise ratio (SNR) by placing a circular or ovoid region of interest on the spinal cord, and the edge slope based on a linear region of interest placed across the surface of the spinal cord. Interobserver agreement in the evaluations of neuroforaminal stenosis using SR-DLR and DLR was 0.422-0.571 and 0.410-0.542, respectively. The kappa values between reader 1 vs. reader 2 and reader 2 vs. reader 3 significantly differed. Two of the three readers rated depictions of the spinal cord, sharpness, and diagnostic acceptability as significantly better with SR-DLR than with DLR. Both SNR and edge slope (/mm) were also significantly better with SR-DLR (12.9 and 6031, respectively) than with DLR (11.5 and 3741, respectively) (p < 0.001 for both). In conclusion, compared to DLR, SR-DLR improved interobserver agreement in the evaluations of neuroforaminal stenosis using 1.5T cervical spine MRI.

Enhanced Video Super-Resolution Network towards Compressed Data

Video super-resolution (VSR) algorithms aim at recovering a temporally consistent high-resolution (HR) video from its corresponding low-resolution (LR) video sequence. Due to the limited bandwidth during video transmission, most available videos on the internet are compressed. Nevertheless, few existing algorithms consider the compression factor in practical applications. In this paper, we propose an enhanced VSR model towards compressed videos, termed as ECVSR, to simultaneously achieve compression artifacts reduction and SR reconstruction end-to-end. ECVSR contains a motion-excited temporal adaption network (METAN) and a multi-frame SR network (SRNet). The METAN takes decoded LR video frames as input and models inter-frame correlations via bidirectional deformable alignment and motion-excited temporal adaption, where temporal differences are calculated as motion prior to excite the motion-sensitive regions of temporal features. In SRNet, cascaded recurrent multi-scale blocks (RMSB) are employed to learn deep spatio-temporal representations from adapted multi-frame features. Then, we build a reconstruction module for spatio-temporal information integration and HR frame reconstruction, which is followed by a detail refinement module for texture and visual quality enhancement. Extensive experimental results on compressed videos demonstrate the superiority of our method for compressed VSR. Code will be available at https://github.com/lifengcs/ECVSR .

Multi-scale perceptual YOLO for automatic detection of clue cells and trichomonas in fluorescence microscopic images

Vaginitis is a common disease among women and has a high recurrence rate. The primary diagnosis method is fluorescence microscopic inspection, but manual inspection is inefficient and can lead to false detection or missed detection. Automatic cell identification and localization in microscopic images are necessary. For vaginitis diagnosis, clue cells and trichomonas are two important indicators and are difficult to be detected because of the different scales and image characteristics. This study proposes a Multi-Scale Perceptual YOLO (MSP-YOLO) with super-resolution reconstruction branch to meet the detection requirements of clue cells and trichomonas. Based on the scales and image characteristics of clue cells and trichomonas, we employed a super-resolution reconstruction branch to the detection network. This branch guides the detection branch to focus on subtle feature differences. Simultaneously, we proposed an attention-based feature fusion module that is injected with dilated convolutional group. This module makes the network pay attention to the non-centered features of the large target clue cells, which contributes to the enhancement of detection sensitivity. Experimental results show that the proposed detection network MSP-YOLO can improve sensitivity without compromising specificity. For clue cell and trichomoniasis detection, the proposed network achieved sensitivities of 0.706 and 0.910, respectively, which were 0.218 and 0.051 higher than those of the baseline model. In this study, the characteristics of the super-resolution reconstruction task are used to guide the network to effectively extract and process image features. The novel proposed network has an increased sensitivity, which makes it possible to detect vaginitis automatically.

Dual-branch feature encoding framework for infrared images super-resolution reconstruction

Infrared thermal imaging is a passive non-contact detection and identification technology, which is not subject to electromagnetic infection and good concealment, is widely used in military and commercial fields. However, due to the limitations of the existing infrared imaging system mechanisms, the spatial resolution of the acquired infrared images is low and the edge details are blurred, which in turn leads to poor performance in downstream missions based on infrared images. In this paper, in order to better solve the above problems, we propose a new super-resolution reconstruction framework for infrared images, called DBFE, which extracts and retains abundant structure and textual information for robust infrared image high-resolution reconstruction with a novel structure-textual encoder module. Extensive experiment demonstrates that our proposed method achieves significantly superior contraband high-resolution reconstruction results on the multiple dataset compared to progressive methods for high resolution infrared image reconstruction, effectively proving the practicability of the method proposed in this paper.