Deep Learning-based Super-resolution Research Articles

Deep learning-based super-resolution of structural brain MRI at 1.5T: application to quantitative volume measurement.

This study investigated the feasibility of using deep learning-based super-resolution (DL-SR) technique on low-resolution (LR) images to generate high-resolution (HR) MR images with the aim of scan time reduction. The efficacy of DL-SR was also assessed through the application of brain volume measurement (BVM). In vivo brain images acquired with 3D-T1W from various MRI scanners were utilized. For model training, LR images were generated by downsampling the original 1mm-2mm isotropic resolution images. Pairs of LR and HR images were used for training 3D residual dense net (RDN). For model testing, actual scanned 2mm isotropic resolution 3D-T1W images with one-minute scan time were used. Normalized root-mean-square error (NRMSE), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM) were used for model evaluation. The evaluation also included brain volume measurement, with assessments of subcortical brain regions. The results showed that DL-SR model improved the quality of LR images compared with cubic interpolation, as indicated by NRMSE (24.22% vs 30.13%), PSNR (26.19 vs 24.65), and SSIM (0.96 vs 0.95). For volumetric assessments, there were no significant differences between DL-SR and actual HR images (p > 0.05, Pearson's correlation > 0.90) at seven subcortical regions. The combination of LR MRI and DL-SR enables addressing prolonged scan time in 3D MRI scans while providing sufficient image quality without affecting brain volume measurement.

Deep Learning-Based Super Resolution Applied to Finite Element Analysis of Fused Deposition Modeling 3D Printing

Deep Learning-Based Super Resolution Applied to Finite Element Analysis of Fused Deposition Modeling 3D Printing

Deep-learning-based super-resolution for accelerating chemical exchange saturation transfer MRI.

Chemical exchange saturation transfer (CEST) MRI is a molecular imaging tool that provides physiological information about tissues, making it an invaluable tool for disease diagnosis and guided treatment. Its clinical application requires the acquisition of high-resolution images capable of accurately identifying subtle regional changes in vivo, while simultaneously maintaining a high level of spectral resolution. However, the acquisition of such high-resolution images is time consuming, presenting a challenge for practical implementation in clinical settings. Among several techniques that have been explored to reduce the acquisition time in MRI, deep-learning-based super-resolution (DLSR) is a promising approach to address this problem due to its adaptability to any acquisition sequence and hardware. However, its translation to CEST MRI has been hindered by the lack of the large CEST datasets required for network development. Thus, we aim to develop a DLSR method, named DLSR-CEST, to reduce the acquisition time for CEST MRI by reconstructing high-resolution images from fast low-resolution acquisitions. This is achieved by first pretraining the DLSR-CEST on human brain T1w and T2w images to initialize the weights of the network and then training the network on very small human and mouse brain CEST datasetsto fine-tune the weights. Using the trained DLSR-CEST network, the reconstructed CEST source images exhibited improved spatial resolution in both peak signal-to-noise ratio and structural similarity index measure metrics at all downsampling factors (2-8). Moreover, amide CEST and relayed nuclear Overhauser effect maps extrapolated from the DLSR-CEST source images exhibited high spatial resolution and low normalized root mean square error, indicating a negligible loss in Z-spectrum information. Therefore, our DLSR-CEST demonstrated a robust reconstruction of high-resolution CEST source images from fast low-resolution acquisitions, thereby improving the spatial resolution and preserving most Z-spectrum information.

HoloSR: deep learning-based super-resolution for real-time high-resolution computer-generated holograms.

This study presents HoloSR, a novel deep learning-based super-resolution approach designed to produce high-resolution computer-generated holograms from low-resolution RGBD images, enabling the real-time production of realistic three-dimensional images. The HoloSR combines the enhanced deep super-resolution network with resize and convolution layers, facilitating the direct generation of high-resolution computer-generated holograms without requiring additional interpolation. Various upscaling scales, extending up to ×4, are evaluated to assess the performance of our method. Quantitative metrics such as structural similarity and peak signal-to-noise ratio are employed to measure the quality of the reconstructed images. Our simulation and experimental results demonstrate that HoloSR successfully achieves super-resolution by generating high-resolution holograms from low-resolution RGBD inputs with supervised and unsupervised learning.

Deep Learning Super-Resolution Technique Based on Magnetic Resonance Imaging for Application of Image-Guided Diagnosis and Surgery of Trigeminal Neuralgia.

This study aimed to implement a deep learning-based super-resolution (SR) technique that can assist in the diagnosis and surgery of trigeminal neuralgia (TN) using magnetic resonance imaging (MRI). Experimental methods applied SR to MRI data examined using five techniques, including T2-weighted imaging (T2WI), T1-weighted imaging (T1WI), contrast-enhancement T1WI (CE-T1WI), T2WI turbo spin-echo series volume isotropic turbo spin-echo acquisition (VISTA), and proton density (PD), in patients diagnosed with TN. The image quality was evaluated using the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). High-quality reconstructed MRI images were assessed using the Leksell coordinate system in gamma knife radiosurgery (GKRS). The results showed that the PSNR and SSIM values achieved by SR were higher than those obtained by image postprocessing techniques, and the coordinates of the images reconstructed in the gamma plan showed no differences from those of the original images. Consequently, SR demonstrated remarkable effects in improving the image quality without discrepancies in the coordinate system, confirming its potential as a useful tool for the diagnosis and surgery of TN.

Elevating Chest X-ray Image Super-Resolution with Residual Network Enhancement.

Chest X-ray (CXR) imaging plays a pivotal role in diagnosing various pulmonary diseases, which account for a significant portion of the global mortality rate, as recognized by the World Health Organization (WHO). Medical practitioners routinely depend on CXR images to identify anomalies and make critical clinical decisions. Dramatic improvements in super-resolution (SR) have been achieved by applying deep learning techniques. However, some SR methods are very difficult to utilize due to their low-resolution inputs and features containing abundant low-frequency information, similar to the case of X-ray image super-resolution. In this paper, we introduce an advanced deep learning-based SR approach that incorporates the innovative residual-in-residual (RIR) structure to augment the diagnostic potential of CXR imaging. Specifically, we propose forming a light network consisting of residual groups built by residual blocks, with multiple skip connections to facilitate the efficient bypassing of abundant low-frequency information through multiple skip connections. This approach allows the main network to concentrate on learning high-frequency information. In addition, we adopted the dense feature fusion within residual groups and designed high parallel residual blocks for better feature extraction. Our proposed methods exhibit superior performance compared to existing state-of-the-art (SOTA) SR methods, delivering enhanced accuracy and notable visual improvements, as evidenced by our results.

Arbitrary scale super-resolution diffusion model for brain MRI images

Given the constraints posed by hardware capacity, scan duration, and patient cooperation, the reconstruction of magnetic resonance imaging (MRI) images emerges as a pivotal aspect of medical imaging research. Currently, deep learning-based super-resolution (SR) methods have been widely discussed in medical image processing due to their ability to reconstruct high-quality, high resolution (HR) images from low resolution (LR) inputs. However, most existing MRI SR methods are designed for specific magnifications and cannot generate MRI images at arbitrary scales, which hinders the radiologists from fully visualizing the lesions. Moreover, current arbitrary scale SR methods often suffer from issues like excessive smoothing and artifacts. In this paper, we propose an Arbitrary Scale Super-Resolution Diffusion Model (ASSRDM), which combines implicit neural representation with the denoising diffusion probabilistic model to achieve arbitrary-scale, high-fidelity medical images SR. Moreover, we formulate a continuous resolution regulation mechanism, comprising a multi-scale LR guidance network and a scaling factor. The scaling factor finely adjusts the resolution and dynamically influences the weighting of LR details and synthesized features in the final output. This capability allows the model to seamlessly adapt to the requirements of continuous resolution adjustments. Additionally, the multi-scale LR guidance network provides the denoising block with multi-resolution LR features to enrich texture information and restore high-frequency details. Extensive experiments conducted on the IXI and fastMRI datasets demonstrate that our ASSRDM exhibits superior performance compared to existing techniques and has tremendous potential in clinical practice.

Kiosk 9R-TC-01 - Deep Learning-based Super Resolution for 3D Whole-heart Coronary MR Angiography

Kiosk 9R-TC-01 - Deep Learning-based Super Resolution for 3D Whole-heart Coronary MR Angiography

딥러닝 기반 초해상화 모델을 이용한 드론 사진의 해상도 향상 연구

딥러닝 기반 초해상화 모델을 이용한 드론 사진의 해상도 향상 연구

Deep learning in computed tomography super resolution using multi-modality data training.

One of the limitations in leveraging the potential of artificial intelligence in X-ray imaging is the limited availability of annotated training data. As X-ray and CT shares similar imaging physics, one could achieve cross-domain data sharing, so to generate labeled synthetic X-ray images from annotated CT volumes as digitally reconstructed radiographs (DRRs). To account for the lower resolution of CT and the CT-generated DRRs as compared to the real X-ray images, we propose the use of super-resolution (SR) techniques to enhance the CT resolution before DRRgeneration. As spatial resolution can be defined by the modulation transfer function kernel in CT physics, we propose to train a SR network using paired low-resolution (LR) and high-resolution (HR) images by varying the kernel's shape and cutoff frequency. This is different to previous deep learning-based SR techniques on RGB and medical images which focused on refining the sampling grid. Instead of generating LR images by bicubic interpolation, we aim to create realistic multi-detector CT (MDCT) like LR images from HR cone-beam CT (CBCT)scans. We propose and evaluate the use of a SR U-Net for the mapping between LR and HR CBCT image slices. We reconstructed paired LR and HR training volumes from the same CT scans with small in-plane sampling grid size of . We used the residual U-Net architecture to train two models. SRUN : trained with kernel-based LR images, and SRUN : trained with bicubic downsampled data as baseline. Both models are trained on one CBCT dataset (n = 13 391). The performance of both models was then evaluated on unseen kernel-based and interpolation-based LR CBCT images (n = 10 950), and also on MDCT images (n = 1392). Five-fold cross validation and ablation study were performed to find the optimal hyperparameters. Both SRUN and SRUN models show significant improvements (p-value 0.05) in mean absolute error (MAE), peak signal-to-noise ratio (PSNR) and structural similarity index measures (SSIMs) on unseen CBCT images. Also, the improvement percentages in MAE, PSNR, and SSIM by SRUN is larger than SRUN . For SRUN , MAE is reduced by 14%, and PSNR and SSIMs increased by 6 and 8%, respectively. To conclude, SRUN outperforms SRUN , which the former generates sharper images when tested with kernel-based LR CBCT images as well as cross-modality LR MDCTdata. Our proposed method showed better performance than the baseline interpolation approach on unseen LR CBCT. We showed that the frequency behavior of the used data is important for learning the SR features. Additionally, we showed cross-modality resolution improvements to LR MDCT images. Our approach is, therefore, a first and essential step in enabling realistic high spatial resolution CT-generated DRRs for deep learningtraining.

Pixel-Level Degradation for Text Image Super-Resolution and Recognition

In the realm of image reconstruction, deep learning-based super-resolution (SR) has established itself as a prevalent technique, particularly in the domain of text image restoration. This study aims to address notable deficiencies in existing research, including constraints imposed by restricted datasets and challenges related to model generalization. Specifically, the goal is to enhance the super-resolution network’s reconstruction of scene text image super-resolution and utilize the generated degenerate dataset to alleviate issues associated with poor generalization due to the sparse scene text image super-resolution dataset. The methodology employed begins with the degradation of images from the MJSynth dataset, using a stochastic degradation process to create eight distinct degraded versions. Subsequently, a blank image is constructed, preserving identical dimensions to the low-resolution image, with each pixel sourced randomly from the corresponding points across the eight degraded images. Following several iterations of training via Finetune, the LR-HR method is applied to the TextZoom dataset. The pivotal metric for assessment is optical character recognition (OCR) accuracy, recognized for its fundamental role in gauging the pragmatic effectiveness of this approach. The experimental findings reveal a notable enhancement in OCR accuracy when compared to the TBSRN model, yielding improvements of 2.4%, 2.3%, and 4.8% on the TextZoom dataset. This innovative approach, founded on pixel-level degradation, not only exhibits commendable generalization capabilities but also demonstrates resilience in confronting the intricate challenges inherent to text image super-resolution.

Impact of using sinogram domain data in the super-resolution of CT images on diagnostic information.

In recent times, deep-learning-based super-resolution (DL-SR) techniques for computed tomography (CT) images have shown outstanding results in terms of full-reference image quality (FR-IQ) metrics (e.g., root mean square error and structural similarity index metric), which assesses IQ by measuring its similarity to the high-resolution (HR) image. In addition, IQ can be evaluated via task-based IQ (Task-IQ) metrics that evaluate the ability to perform specific tasks. Ironically, most proposed image domain-based SR techniques are not possible to improve a Task-IQ metric, which assesses the amount of information related todiagnosis. In the case of CT imaging systems, sinogram domain data can be utilized for SR techniques. Therefore, this study aims to investigate the impact of utilizing sinogram domain data on diagnostic information restorationability. We evaluated three DL-SR techniques: using image domain data (Image-SR), using sinogram domain data (Sinogram-SR), and using sinogram as well as image domain data (Dual-SR). For Task-IQ evaluation, the Rayleigh discrimination task was used to evaluate diagnostic ability by focusing on the resolving power aspect, and an ideal observer (IO) can be used to perform the task. In this study, we used a convolutional neural network (CNN)-based IO that approximates the IO performance. We compared the IO performances of the SR techniques according to the data domain to evaluate the discriminative information restorationability. Overall, the low-resolution (LR) and SR exhibit lower IO performances compared with that of HR owing to their degraded discriminative information when detector binning is used. Next, between the SR techniques, Image-SR does not show superior IO performances compared to the LR image, but Sinogram-SR and Dual-SR show superior IO performances than the LR image. Furthermore, in Sinogram-SR, we confirm that FR-IQ and IO performance are positively correlated. These observations demonstrate that sinogram domain upsampling improves the representation ability for discriminative information in the image domain compared to the LR andImage-SR. Unlike Image-SR, Sinogram-SR can improve the amount of discriminative information present in the image domain. This demonstrates that to improve the amount of discriminative information on the resolving power aspect, it is necessary to employ sinogram domainprocessing.

Frequency-aware optical coherence tomography image super-resolution via conditional generative adversarial neural network.

Optical coherence tomography (OCT) has stimulated a wide range of medical image-based diagnosis and treatment in fields such as cardiology and ophthalmology. Such applications can be further facilitated by deep learning-based super-resolution technology, which improves the capability of resolving morphological structures. However, existing deep learning-based method only focuses on spatial distribution and disregards frequency fidelity in image reconstruction, leading to a frequency bias. To overcome this limitation, we propose a frequency-aware super-resolution framework that integrates three critical frequency-based modules (i.e., frequency transformation, frequency skip connection, and frequency alignment) and frequency-based loss function into a conditional generative adversarial network (cGAN). We conducted a large-scale quantitative study from an existing coronary OCT dataset to demonstrate the superiority of our proposed framework over existing deep learning frameworks. In addition, we confirmed the generalizability of our framework by applying it to fish corneal images and rat retinal images, demonstrating its capability to super-resolve morphological details in eye imaging.

Deep learning-based super-resolution for harmful algal bloom monitoring of inland water

ABSTRACTInland water frequently occurs during harmful algal blooms (HABs), rendering it challenging to comprehend the spatiotemporal features of algal dynamics. Recently, remote sensing has been applied to effectively detect the algal spatiotemporal behaviors in expensive water bodies. However, image sensor resolution limitation can render the understanding of spatiotemporal features of relatively small water bodies challenging. In addition, few studies have improved the resolution of remote sensing images to investigate inland water quality, owing to the image sensor resolution limitations. Therefore, this study applied deep learning-based Super-resolution for transforming satellite imagery of 20 m to airborne imagery of 5 m. After performing atmospheric correction for the acquired images, we adopted super-resolution (SR) methodologies using a super-resolution convolutional neural network (SRCNN) and super-resolution generative adversarial networks (SRGAN) to estimate the Chlorophyll-a (Chl-a) concentration in the Geum River of South Korea. Both methods generated SR images with water reflectance at 665, 705, and 740 nm. Then, two band-ratio algorithms at 665 and 740 nm wavelengths were applied to the reflectance images to estimate the Chl-a concentration maps. The SRCNN model outperformed SRGAN and bicubic interpolation with peak signal-to-noise ratios (PSNR), mean square errors (MSE), and structural similarity index measures (SSIM) for the validation dataset of 24.47 (dB), 0.0074, and 0.74, respectively. SR maps from the SRCNN provided more detailed spatial information on Chl-a in the Geum River compared to the information obtained from satellite images. Therefore, these findings showed the potential of deep learning-based SR algorithms by providing further information according to the algal dynamics for inland water management with remote sensing images.

Physics-informed deep learning for T2-deblurred superresolution turbo spin echo MRI.

Deep learning superresolution (SR) is a promising approach to reduce MRI scan time without requiring custom sequences or iterative reconstruction. Previous deep learning SR approaches have generated low-resolution training images by simple k-space truncation, but this does not properly model in-plane turbo spin echo (TSE) MRI resolution degradation, which has variable T2 relaxation effects in different k-space regions. To fill this gap, we developed a T2 -deblurred deep learning SR method for the SR of 3D-TSE images. A SR generative adversarial network was trained using physically realistic resolution degradation (asymmetric T2 weighting of raw high-resolution k-space data). For comparison, we trained the same network structure on previous degradation models without TSE physics modeling. We tested all models for both retrospective and prospective SR with 3 × 3 acceleration factor (in the two phase-encoding directions) of genetically engineered mouse embryo model TSE-MR images. The proposed method can produce high-quality 3 × 3 SR images for a typical 500-slice volume with 6-7 mouse embryos. Because 3 × 3 SR was performed, the image acquisition time can be reduced from 15 h to 1.7 h. Compared to previous SR methods without TSE modeling, the proposed method achieved the best quantitative imaging metrics for both retrospective and prospective evaluations and achieved the best imaging-quality expert scores for prospective evaluation. The proposed T2 -deblurring method improved accuracy and image quality of deep learning-based SR of TSE MRI. This method has the potential to accelerate TSE image acquisition by a factor of up to 9.

Mouse brain MR super-resolution using a deep learning network trained with optical imaging data.

The resolution of magnetic resonance imaging is often limited at the millimeter level due to its inherent signal-to-noise disadvantage compared to other imaging modalities. Super-resolution (SR) of MRI data aims to enhance its resolution and diagnostic value. While deep learning-based SR has shown potential, its applications in MRI remain limited, especially for preclinical MRI, where large high-resolution MRI datasets for training are often lacking. In this study, we first used high-resolution mouse brain auto-fluorescence (AF) data acquired using serial two-photon tomography (STPT) to examine the performance of deep learning-based SR for mouse brain images. We found that the best SR performance was obtained when the resolutions of training and target data were matched. We then applied the network trained using AF data to MRI data of the mouse brain, and found that the performance of the SR network depended on the tissue contrast presented in the MRI data. Using transfer learning and a limited set of high-resolution mouse brain MRI data, we were able to fine-tune the initial network trained using AF to enhance the resolution of MRI data. Our results suggest that deep learning SR networks trained using high-resolution data of a different modality can be applied to MRI data after transfer learning.

Exploring microstructure and petrophysical properties of microporous volcanic rocks through 3D multiscale and super-resolution imaging

Digital rock physics offers powerful perspectives to investigate Earth materials in 3D and non-destructively. However, it has been poorly applied to microporous volcanic rocks due to their challenging microstructures, although they are studied for numerous volcanological, geothermal and engineering applications. Their rapid origin, in fact, leads to complex textures, where pores are dispersed in fine, heterogeneous and lithified matrices. We propose a framework to optimize their investigation and face innovative 3D/4D imaging challenges. A 3D multiscale study of a tuff was performed through X-ray microtomography and image-based simulations, finding that accurate characterizations of microstructure and petrophysical properties require high-resolution scans (≤ 4 μm/px). However, high-resolution imaging of large samples may need long times and hard X-rays, covering small rock volumes. To deal with these limitations, we implemented 2D/3D convolutional neural network and generative adversarial network-based super-resolution approaches. They can improve the quality of low-resolution scans, learning mapping functions from low-resolution to high-resolution images. This is one of the first efforts to apply deep learning-based super-resolution to unconventional non-sedimentary digital rocks and real scans. Our findings suggest that these approaches, and mainly 2D U-Net and pix2pix networks trained on paired data, can strongly facilitate high-resolution imaging of large microporous (volcanic) rocks.

Knowledge Distillation for Optical Flow-Based Video Super-resolution

Recently, deep learning-based super-resolution (SR) models have been used to improve SR performance by equipping preprocessing networks with baseline SR networks. In particular, in video SR, which creates a high-resolution (HR) image with multiple frames, optical flow extraction is accompanied by a preprocessing process. These preprocessing networks work effectively in terms of quality, but at the cost of increased network parameters, which increase the computational complexity and memory consumption for SR tasks with restricted resources. One well-known approach is the knowledge distillation (KD) method, which can transfer the original model’s knowledge to a lightweight model with less performance degradation. Moreover, KD may improve SR quality with reduced model parameters. In this study, we propose an effective KD method that can effectively reduce the original SR model parameters and even improve network performance. The experimental results demonstrated that our method achieved a better PSNR than the original state-of-the-art SR network despite having fewer parameters.

Super-Resolution of Dental Panoramic Radiographs Using Deep Learning: A Pilot Study.

Using super-resolution (SR) algorithms, an image with a low resolution can be converted into a high-quality image. Our objective was to compare deep learning-based SR models to a conventional approach for improving the resolution of dental panoramic radiographs. A total of 888 dental panoramic radiographs were obtained. Our study involved five state-of-the-art deep learning-based SR approaches, including SR convolutional neural networks (SRCNN), SR generative adversarial network (SRGAN), U-Net, Swin for image restoration (SwinIr), and local texture estimator (LTE). Their results were compared with one another and with conventional bicubic interpolation. The performance of each model was evaluated using the metrics of mean squared error (MSE), peak signal-to-noise ratio (PNSR), structural similarity index (SSIM), and mean opinion score by four experts (MOS). Among all the models evaluated, the LTE model presented the highest performance, with MSE, SSIM, PSNR, and MOS results of 7.42 ± 0.44, 39.74 ± 0.17, 0.919 ± 0.003, and 3.59 ± 0.54, respectively. Additionally, compared with low-resolution images, the output of all the used approaches showed significant improvements in MOS evaluation. A significant enhancement in the quality of panoramic radiographs can be achieved by SR. The LTE model outperformed the other models.

Application of deep learning-based super-resolution to T1-weighted postcontrast gradient echo imaging of the chest

ObjectivesA deep learning-based super-resolution for postcontrast volume-interpolated breath-hold examination (VIBE) of the chest was investigated in this study. Aim was to improve image quality, noise, artifacts and diagnostic confidence without change of acquisition parameters.Materials and methodsFifty patients who received VIBE postcontrast imaging of the chest at 1.5 T were included in this retrospective study. After acquisition of the standard VIBE (VIBES), a novel deep learning-based algorithm and a denoising algorithm were applied, resulting in enhanced images (VIBEDL). Two radiologists qualitatively evaluated both datasets independently, rating sharpness of soft tissue, vessels, bronchial structures, lymph nodes, artifacts, cardiac motion artifacts, noise levels and overall diagnostic confidence, using a Likert scale ranging from 1 to 4. In the presence of lung lesions, the largest lesion was rated regarding sharpness and diagnostic confidence using the same Likert scale as mentioned above. Additionally, the largest diameter of the lesion was measured.ResultsThe sharpness of soft tissue, vessels, bronchial structures and lymph nodes as well as the diagnostic confidence, the extent of artifacts, the extent of cardiac motion artifacts and noise levels were rated superior in VIBEDL (all P < 0.001).There was no significant difference in the diameter or the localization of the largest lung lesion in VIBEDL compared to VIBES. Lesion sharpness as well as detectability was rated significantly better by both readers with VIBEDL (both P < 0.001).ConclusionThe application of a novel deep learning-based super-resolution approach in T1-weighted VIBE postcontrast imaging resulted in an improvement in image quality, noise levels and diagnostic confidence as well as in a shortened acquisition time.