Super-resolution Neural Network Research Articles

Super-resolution neural networks improve the spatiotemporal resolution of adaptive MRI-guided radiation therapy

BackgroundMagnetic resonance imaging (MRI) offers superb non-invasive, soft tissue imaging of the human body. However, extensive data sampling requirements severely restrict the spatiotemporal resolution achievable with MRI. This limits the modality’s utility in real-time guidance applications, particularly for the rapidly growing MRI-guided radiation therapy approach to cancer treatment. Recent advances in artificial intelligence (AI) could reduce the trade-off between the spatial and the temporal resolution of MRI, thus increasing the clinical utility of the imaging modality.MethodsWe trained deep learning-based super-resolution neural networks to increase the spatial resolution of real-time MRI. We developed a framework to integrate neural networks directly onto a 1.0 T MRI-linac enabling real-time super-resolution imaging. We integrated this framework with the targeting system of the MRI-linac to demonstrate real-time beam adaptation with super-resolution-based imaging. We tested the integrated system using large publicly available datasets, healthy volunteer imaging, phantom imaging, and beam tracking experiments using bicubic interpolation as a baseline comparison.ResultsDeep learning-based super-resolution increases the spatial resolution of real-time MRI across a variety of experiments, offering measured performance benefits compared to bicubic interpolation. The temporal resolution is not compromised as measured by a real-time adaptation latency experiment. These two effects, an increase in the spatial resolution with a negligible decrease in the temporal resolution, leads to a net increase in the spatiotemporal resolution.ConclusionsDeployed super-resolution neural networks can increase the spatiotemporal resolution of real-time MRI. This has applications to domains such as MRI-guided radiation therapy and interventional procedures.

A hyperspectral surgical microscope with super-resolution reconstruction for intraoperative image guidance.

Hyperspectral imaging (HSI) is an emerging imaging modality in medical applications, especially for intraoperative image guidance. A surgical microscope improves surgeons' visualization with fine details during surgery. The combination of HSI and surgical microscope can provide a powerful tool for surgical guidance. However, to acquire high-resolution hyperspectral images, the long integration time and large image file size can be a burden for intraoperative applications. Super-resolution reconstruction allows acquisition of low-resolution hyperspectral images and generates high-resolution HSI. In this work, we developed a hyperspectral surgical microscope and employed our unsupervised super-resolution neural network, which generated high-resolution hyperspectral images with fine textures and spectral characteristics of tissues. The proposed method can reduce the acquisition time and save storage space taken up by hyperspectral images without compromising image quality, which will facilitate the adaptation of hyperspectral imaging technology in intraoperative image guidance.

A novel approach to recognition of Alzheimer's and Parkinson's diseases: random subspace ensemble classifier based on deep hybrid features with a super-resolution image.

Artificial intelligence technologies have great potential in classifying neurodegenerative diseases such as Alzheimer's and Parkinson's. These technologies can aid in early diagnosis, enhance classification accuracy, and improve patient access to appropriate treatments. For this purpose, we focused on AI-based auto-diagnosis of Alzheimer's disease, Parkinson's disease, and healthy MRI images. In the current study, a deep hybrid network based on an ensemble classifier and convolutional neural network was designed. First, a very deep super-resolution neural network was adapted to improve the resolution of MRI images. Low and high-level features were extracted from the images processed with the hybrid deep convolutional neural network. Finally, these deep features are given as input to the k-nearest neighbor (KNN)-based random subspace ensemble classifier. A 3-class dataset containing publicly available MRI images was utilized to test the proposed architecture. In experimental works, the proposed model produced 99.11% accuracy, 98.75% sensitivity, 99.54% specificity, 98.65% precision, and 98.70% F1-score performance values. The results indicate that our AI system has the potential to provide valuable diagnostic assistance in clinical settings.

DWSR: an architecture optimization framework for adaptive super-resolution neural networks based on meta-heuristics

Despite recent advancements in super-resolution neural network optimization, a fundamental challenge remains unresolved: as the number of parameters is reduced, the network’s performance significantly deteriorates. This paper presents a novel framework called the Depthwise Separable Convolution Super-Resolution Neural Network Framework (DWSR) for optimizing super-resolution neural network architectures. The depthwise separable convolutions are introduced to reduce the number of parameters and minimize the impact on the performance of the super-resolution neural network. The proposed framework uses the RUNge Kutta optimizer (RUN) variant (MoBRUN) as the search method. MoBRUN is a multi-objective binary version of RUN, which balances multiple objectives when optimizing the neural network architecture. Experimental results on publicly available datasets indicate that the DWSR framework can reduce the number of parameters of the Residual Dense Network (RDN) model by 22.17% while suffering only a minor decrease of 0.018 in Peak Signal-to-Noise Ratio (PSNR), the framework can reduce the number of parameters of the Enhanced SRGAN (ESRGAN) model by 31.45% while losing only 0.08 PSNR. Additionally, the framework can reduce the number of parameters of the HAT model by 5.38% while losing only 0.02 PSNR.

SuperAdjoint: Super-resolution neural networks in adjoint-based error estimation

Numerical simulations and optimisation methods, such as mesh adaptation, rely on the accurate and inexpensive use of error estimation methods. Adjoint-based error estimation is the most accurate method, and generally the most costly. A strong contributor to this cost is the need to compute a higher resolution adjoint solution. Here, it is proposed to use super-resolution neural networks to super-resolve a fine adjoint solution from a lower-cost coarse adjoint solution: a superAdjoint. The method is compared to reference error estimators on an unsteady Burgers’ equation using the method of manufactured solutions. Two forms of the superAdjoint were implemented, a twice and a four times refining super-resolution neural network. These were used to demonstrate both the computational cost reduction and the potential for the reduction of the storage footprint of the primal problem. The first, referred to as 2×CNN, was able to reconstruct the spatially enriched adjoint solution, thus providing a robust and inexpensive local output error. The second, the 4×CNN, was able to demonstrate the reconstruction ability of super-resolution neural networks for higher upscaling factors. This was leveraged in order to subsample the primal solution in space, thus reducing substantially the storage footprint of the discrete primal solution. Both superAdjoints could achieve the desired level of accuracy when compared to the reconstruction of the refined adjoint-based error estimate. Moreover, superAdjoint was shown to be able to generalise to a new QoI that was untrained for. This gives great confidence in the use of super-resolution neural networks for the reduction of both computational cost and storage requirements of adjoint-based error estimation, and goal-oriented mesh adaptation.

Deep Hierarchical Super Resolution for Scientific Data.

We present a novel technique for hierarchical super resolution (SR) with neural networks (NNs), which upscales volumetric data represented with an octree data structure to a high-resolution uniform grid with minimal seam artifacts on octree node boundaries. Our method uses existing state-of-the-art SR models and adds flexibility to upscale input data with varying levels of detail across the domain, instead of only uniform grid data that are supported in previous approaches. The key is to use a hierarchy of SR NNs, each trained to perform 2× SR between two levels of detail, with a hierarchical SR algorithm that minimizes seam artifacts by starting from the coarsest level of detail and working up. We show that our hierarchical approach outperforms baseline interpolation and hierarchical upscaling methods, and demonstrate the usefulness of our proposed approach across three use cases including data reduction using hierarchical downsampling+SR instead of uniform downsampling+SR, computation savings for hierarchical finite-time Lyapunov exponent field calculation, and super-resolving low-resolution simulation results for a high-resolution approximation visualization.

HASR-TAI: Hybrid model-based interpretable network and super-resolution network for thermoacoustic imaging

Microwave induced thermoacoustic tomography has shown promise for noninvasive and non-ionizing early tumor detection. Nowadays, thermoacoustic reconstruction methods based on deep learning have achieved good and time-efficient results. However, both deep learning methods based on the initial thermoacoustic image and end-to-end methods lack interpretability due to the black-box property of neural networks. In this Letter, we propose an interpretable end-to-end network structure comprising an unrolling part and a super-resolution part. In the unrolling part, a deep unfolding network interprets the iterations of the model-based algorithm based on compressed sensing as layers of the network. Subsequently, a fast and efficient super-resolution neural network maps the low-resolution image into the super-resolution space. Two breast models with different sizes of tumor targets are used for validation. By comparing with the traditional method and the deep learning method, the proposed method demonstrates superior performance in image quality and imaging time. Moreover, the parameters in the network hold physical significance, offering the potential for the interpretable end-to-end network in thermoacoustic imaging.

Enhancing identification of digital rock images using super-resolution deep neural network

Identification of pore and minerals from digital rock images plays an important role in characterization of oil and gas reservoirs. Traditional methods usually require high costs of time and labor. With the rapid development of artificial intelligence, deep learning methods have been applied to geosciences. Among them, semantic segmentation neural networks are used to identify rock images, but traditional semantic segmentation networks require high-resolution input images to improve performance and can only generate masks with the same resolutions as the input images, which again requires high-performance equipment and increases computational and equipment costs. In this paper, we propose an approach to enhancing identification of digital rock images with super-resolution neural network (SSRN) which enables low-resolution input images to generate high-resolution representations. We use SSRN to train the 10-classification thin-section images, and compare it with traditional deep learning methods. It indicates that SSRN can effectively improve the performance of the generated high-resolution masks.

Super-Resolution Neural Network Models Comparison Metod for Brain MRI Images Based on PSNR and SSIM Metrics

The diagnosis of many diseases is largely possible thanks to MRI(Magnetic Resonance Imaging). This technology allows to study internal organs of the patient: the brain, spine, bones, joints, vessels and etc. The resolution of the MRI image is limited due to various factors: movement of the patient during the scan, the continuous movement of internal organs. The higher the quality of the MRI image, the longer it takes to scan. For more accurate diagnostics it is possible to increase resolution of the yielded images. This is achieved by using SISR(Single Image Super Resolution) algorithms, which allow you to obtain images with increased resolution from a single input image. In this paper the idea of the image super-resolution algorithms is presented, various forms of the problem and solutions to it are provided. The advantages of the SISR algorithms are described. The relevance of this task in the field of medical MRI images is explained. Metrics for comparing image quality PSNR and SSIM are given and described. A dataset for testing is presented. The stage of data preparation is described: the principle of selecting images from a set of datasets, converting data into the required format, compressing images to obtain input data for selected neural network models. The PSNR, SSIM metrics of two neural network models mDCSRN and FAWDN are measured on equally prepared input data. The comparison results are presented in the form of images and averaged data for the entire sample is stored in the table.

228 × 304 200-mW lidar based on a single-point global-depth d-ToF sensor and RGB-guided super-resolution neural network.

The cutting-edge imaging system exhibits low output resolution and high power consumption, presenting challenges for the RGB-D fusion algorithm. In practical scenarios, aligning the depth map resolution with the RGB image sensor is a crucial requirement. In this Letter, the software and hardware co-design is considered to implement a lidar system based on the monocular RGB 3D imaging algorithm. A 6.4 × 6.4-mm2 deep-learning accelerator (DLA) system-on-chip (SoC) manufactured in a 40-nm CMOS is incorporated with a 3.6-mm2 TX-RX integrated chip fabricated in a 180-nm CMOS to employ the customized single-pixel imaging neural network. In comparison to the RGB-only monocular depth estimation technique, the root mean square error is reduced from 0.48 m to 0.3 m on the evaluated dataset, and the output depth map resolution matches the RGB input.

Image Resizing for Object Detection: A Learnable Downsampler–Upsampler Pair with Differentiable Image Entropy Estimation

In recent years, super-resolution neural networks have achieved good results in restoring super-resolution images from low-resolution ones. However, most subsequent tasks based on super-resolution images such as object detection are done by the computer. Considering this situation, we propose a learnable downsampler–upsampler pair, which can realize both the downscaling process and the upscaling process by neural networks, and is jointly trained with the YOLOV5 network to optimize the object detection task. Thus, different from existing super-resolution networks, the entire downsampler–upsampler pair is optimized for machine perception. In addition, to further reduce the size of the downsampled images, we also propose a differentiable method for estimating image entropy and add it to the loss function. We verify the effectiveness of our method on the pothole dataset and use scale factors 2× and 4× to prove that our method is capable of diverse resizing levels. The experimental results show that using our learnable downsampler–upsampler pair as a resizing method can highly improve the detection performance compared with other resizing techniques.

Digital rock resolution enhancement and detail recovery with multi attention neural network

High-quality digital rock images are essential for subsequent high-precision numerical simulations. But limited by the imaging capability of computed tomography (CT), high resolution digital rock images with wide imaging field of view (FOV) cannot be acquired simultaneously. To cope with this constraint, we propose a novel Multi Attention Super-Resolution Neural Network (MASR) that enhances the resolution of images with wide FOV. Considering that textures and edges are more crucial in digital rocks, MASR introduces the component attention mechanism of Component Divide-and-Conquer Super-Resolution (CDCSR) model. By redesigning the hourglass network with spatial and channel attention mechanisms, proposing a spatial attention-based mask module, and optimizing the component attention mask calculation process, MASR delivers higher information utilization with fewer parameters and faster training than CDCSR. In addition, we optimize the depth of MASR to trade off speed and super-resolution quality. Furthermore, we retrain several state-of-the-art models. Through quantitative evaluations and qualitative visualizations, it is verified that MASR can recover sharper edges while removing noise, and obtain digital rock images with superior quality and reliability. The pixel-wise relative errors of MASR reconstructions are reduced by 15%–26% over bicubic interpolation method. Our codes and pretrained model are publicly available at https://github.com/MHDXing/MASR-for-Digital-Rock-Images.

A Novel Multi-Task Learning Network Based on Melanoma Segmentation and Classification with Skin Lesion Images.

Melanoma is known worldwide as a malignant tumor and the fastest-growing skin cancer type. It is a very life-threatening disease with a high mortality rate. Automatic melanoma detection improves the early detection of the disease and the survival rate. In accordance with this purpose, we presented a multi-task learning approach based on melanoma recognition with dermoscopy images. Firstly, an effective pre-processing approach based on max pooling, contrast, and shape filters is used to eliminate hair details and to perform image enhancement operations. Next, the lesion region was segmented with a VGGNet model-based FCN Layer architecture using enhanced images. Later, a cropping process was performed for the detected lesions. Then, the cropped images were converted to the input size of the classifier model using the very deep super-resolution neural network approach, and the decrease in image resolution was minimized. Finally, a deep learning network approach based on pre-trained convolutional neural networks was developed for melanoma classification. We used the International Skin Imaging Collaboration, a publicly available dermoscopic skin lesion dataset in experimental studies. While the performance measures of accuracy, specificity, precision, and sensitivity, obtained for segmentation of the lesion region, were produced at rates of 96.99%, 92.53%, 97.65%, and 98.41%, respectively, the performance measures achieved rates for classification of 97.73%, 99.83%, 99.83%, and 95.67%, respectively.

Cycle-Consistency-Based Uncertainty Quantification of Neural Networks in Inverse Imaging Problems

Uncertainty estimation is critical for numerous deep neural network (DNN) applications and has drawn increasing attention from researchers. In this study, we demonstrated an uncertainty quantification approach for DNNs used in inverse problems based on cycle consistency. We built forward–backward cycles using the available physical forward model and a trained DNN solving the inverse problem at hand and accordingly derived uncertainty estimators through regression analysis on the consistency of these forward–backward cycles. We theoretically analyzed the cycle consistency metrics and derived their relationship with the uncertainty, bias, and robustness of neural network inference. To demonstrate the effectiveness of these cycle-consistency-based uncertainty estimators, we classified corrupted and out-of-distribution input image data using widely used image deblurring and super-resolution neural networks as test beds. Our blind tests demonstrated that our method surpassed other models in detecting previously unseen data corruption and distribution shifts. This study provides a simple-to-implement and rapid uncertainty quantification method that can be universally applied to various neural networks used to solve inverse problems.

Hybrid Post-Training Quantization for Super-Resolution Neural Network Compression

Quantization is a widely adopted technique to reduce the storage cost of neural networks. However, existing methods primarily focus on minimizing the quantization error of neural network parameters without considering the correlation between the quantization error and performance of quantized neural networks. Motivated by this consideration, we propose a hybrid post-training quantization (HPTQ) method for super-resolution neural networks. Layer-wise quantization and piecewise quantization are integrated based on error sensitivity and the quantization error of parameters. In HPTQ, we utilize Taylor expansion to demonstrate that the performance distortion of quantized neural networks is a weighted average of parameter quantization errors with respect to gradients. To reduce the quantization error, we apply uniform and clustered quantization to parameters in dense and sparse regions, respectively. Furthermore, we allocate larger bit-widths to layers with higher error sensitivity indicated by gradients. Numerical experiments show that the super-resolution neural networks perform better under the proposed quantization approach compared to existing quantization methods.

상호증류를 활용한 SRGAN의 성능 개선

This paper proposes a method to improve the performance of SRGAN, a generative adversarial network (GAN)-based super-resolution neural network model, by applying mutual distillation. The proposed method adds a discriminator for mutual distillation to SRGAN, allowing the generator of each SRGAN to indirectly share the knowledge of the other SRGANs’ generators through adversarial learning with the discriminators added to the other SRGANs. Experiments using two SRGANs were conducted to analyze the performance of the proposed method, and the experiments confirmed that SRGANs produced super-resolution images with improved qualitative and quantitative image quality by applying mutual distillation. In addition, it was confirmed that the performance varies depending on the amount of knowledge transferred through mutual distillation.

Multi-Scale Histogram-Based Probabilistic Deep Neural Network for Super-Resolution 3D LiDAR Imaging

LiDAR (Light Detection and Ranging) imaging based on SPAD (Single-Photon Avalanche Diode) technology suffers from severe area penalty for large on-chip histogram peak detection circuits required by the high precision of measured depth values. In this work, a probabilistic estimation-based super-resolution neural network for SPAD imaging that firstly uses temporal multi-scale histograms as inputs is proposed. To reduce the area and cost of on-chip histogram computation, only part of the histogram hardware for calculating the reflected photons is implemented on a chip. On account of the distribution rule of returned photons, a probabilistic encoder as a part of the network is first proposed to solve the depth estimation problem of SPADs. By jointly using this neural network with a super-resolution network, 16× up-sampling depth estimation is realized using 32 × 32 multi-scale histogram outputs. Finally, the effectiveness of this neural network was verified in the laboratory with a 32 × 32 SPAD sensor system.

Abstract 10054: Super Resolution Deep Learning Reconstruction for Detection of In-Stent Restenosis

Introduction: Evaluation of in-stent-restenosis (ISR), especially of small stents, remains a challenge for computed tomography (CT) angiography. Hypothesis: To quantitatively define stent strut thickness and lumen vessel diameter at the stent site by deep learning reconstruction and compare this with conventional reconstruction strategies. Methods: We employed ultra-high-resolution CT (URCT) data to train a super resolution neural network (Precise IQ engine, or PIQE) and apply the network to current standard resolution CT (Forward projected model-based Iterative Reconstruction SoluTion, FIRST). We examined 166 stents in 85 consecutive patients who underwent CT and invasive coronary angiography (ICA) within three months of each other from 2019-2021 after percutaneous coronary intervention with coronary stent placement. The diagnostic accuracy for the presence of ISR was defined as percent diameter stenosis > 50% on ICA. All images were reconstructed with PIQE and FIRST and assessed by two blinded cardiovascular imagers. Results: Compared to FIRST, the full width at half maximum of the lumen was larger, and that of the strut was smaller in PIQE. Image quality score was significantly better in PIQE than in FIRST (4.2±1.1 vs. 2.7±1.2, P <0.01). The sensitivity, specificity, PPV, NPV, and accuracy for detection of ISR in PIQE were better than the FIRST (86.1, 89.5, 50.0, 98.1, and 89.2 vs. 80.6, 77.4, 30.2, 97.0, and 77.7, respectively); the differences were particularly pronounced in a group with < 3.0 mm stent diameter. Conclusions: PIQE provides superior image quality and diagnostic accuracy for ISR, even in stents with < 3.0 mm diameter.

Cascade neural approximating for few-shot super-resolution photoacoustic angiography

High-resolution photoacoustic angiography images are reconstructed from undersampled images with the help of a super-resolution deep neural network, enhancing the ability of the photoacoustic angiography systems to image dynamic processes in living tissues. However, image degradations are difficult to estimate due to a lack of knowledge of the point spread function and noise sources, resulting in poor generalization capability of the trained super-resolution model. In this work, a high-order residual cascade neural network was developed to reconstruct high-resolution vascular images, which is a neural approximating approach used to remove image degradations of photoacoustic angiography. To handle overfitting in training super-resolution model with a limited dataset, we proposed a BicycleGAN based image synthesis method in data preparation, achieving a strong regularization by forging realistic photoacoustic vascular images that act to essentially increase the training dataset. The quantitative analysis of the reconstructed results shows that the high-order residual cascade neural network surpassed the other residual super-resolution neural networks. Most importantly, we demonstrated that the generalized model could be achieved despite the limited training dataset, promising to be a methodology for few-shot super-resolution photoacoustic angiography.

Deep-learning-based super-resolution reconstruction of high-speed imaging in fluids

In many fluid experiments, we can only obtain low-spatial high-temporal resolution flow images and high-spatial low-temporal resolution flow images due to the limitation of high-speed imaging systems. To solve this problem, we proposed a degradation and super-resolution attention model (D-SRA) using unsupervised machine learning to super-resolution reconstruct high resolution (HR) time-resolved fluid images from coarse data. Unlike the prior research to increase the resolution of coarse data artificially generated by simple bilinear down-sampling, our model that consists of a degradation neural network and a super-resolution neural network aims to learn the mappings between experimental low-resolution data and corresponding HR data. What is more, channel and spatial attention modules are also adopted in D-SRA to facilitate the restoration of abundant and critical details of flow fields. The proposed model is validated by two high-speed schlieren experiments of under-expanded impinging supersonic jets. The comprehensive capability of D-SRA is statistically analyzed based on the synthetic unpaired schlieren images. The spatial-resolution of coarse images can be successfully augmented by 42 times and 82 times with most physical details recovered perfectly, which outperforms the existing method. The D-SRA also exhibits considerable generalization and robustness against unknown-degenerated schlieren images. Moreover, the practicability of the proposed method is also further explored on real unpaired jets schlieren images. It is convincingly demonstrated that the present study successfully surpasses the performance limitations of high-speed cameras and has significant applications in various fluid experiments to obtain flow images with high spatial and temporal resolution.