Super-resolution Approach Research Articles

An assisted multi-frame approach for super-resolution in hyperspectral images of rock samples

Imaging spectroscopy is decisive for accurately characterizing rock samples for geological and mineral applications, since slight differences in mineral composition can be easily recognized in spectral signatures. Although, instrument limitations prevent high spatial resolution pixels scanned along hundreds of spectral channels, which considerably reduces the data capabilities. Multi frame Super-Resolution (SR) techniques can overcome this issue by retrieving high-quality images using purely computational methods. However, these approaches still have important drawbacks because of inherent uncertainties when estimating the spatial variation between frames, which strongly penalize the spectral information, essential for lithology. In this paper we propose an assisted framework to provide fully controlled motion parameters of the imaging process, skipping the intricate registration task. Essentially, we use a stepping device to successively change the position of rock samples for each new frame acquisition. With accurate knowledge about motion parameters, we were able to overcome the uncertainties regarding spatial registration between frames. Therefore, a reasonable number of equations connecting the low-resolution frames and the super resolved image could be used to solve the optimization problem. Extensive experiments proved that the proposed assisted method achieved best performances (both qualitatively and quantitatively) compared with similar approaches. Moreover, detailed spectral analysis showed almost absolute consistency between the original low-resolution image and the super-resolved result, which is crucial to guarantee reliable mineral characterization.

MR image reconstruction using iterative up and downsampling network

Magnetic resonance imaging (MRI) is one of the most significant modalities in medical imaging. It suffers from rather lengthy acquisition times, which lead to prolonged patient restriction, patient discomfort, and imaging artifacts. Hence, satisfactory MRI reconstruction from undersampled data sequences constitutes an important research problem. With the advances in deep learning (DL), a plethora of new models have been proposed to solve the MRI reconstruction problem using deep networks. On the other hand, single image super-resolution (SR) is another well-studied field that benefited from the success of DL, and it has applications in various imaging modalities. SR is the process of recovering a high-resolution image from a low-resolution image. SR models work on low-resolution images to recover missing details. MR image reconstruction on the other hand is a battle to eliminate the aliasing artifacts which originate from data downsampling. The motivation for the proposed work is based on the premise that SR approaches can possibly get adapted to MR image reconstruction. Hence in this study, inspired by the great success of deep SR networks, we customize an architecture introduced in SR setting to MRI reconstruction. This novel approach uses the iterative up and downsampling framework labeled as Iterative Up and Down Network (IUDN) for MRI reconstruction. We design two variants of the proposed network with different number of scale factors. We present extensive simulations for the proposed architecture using multiple k-space undersampling ratios. The simulation results indicate cutting edge MRI reconstruction performance for the proposed models. The networks were trained on fastMRI dataset and tested on both fastMRI and IXI datasets to show the robustness of the method. The proposed models achieved improved results in terms of PSNR and visual quality of the reconstructed image when compared to some recent state-of-the-art solutions for MRI reconstruction.

Single image super-resolution approaches in medical images based-deep learning: a survey

Single image super-resolution approaches in medical images based-deep learning: a survey

Improved Generative Adversarial Network for Super-Resolution Reconstruction of Coal Photomicrographs.

Analyzing the photomicrographs of coal and conducting maceral analysis are essential steps in understanding the coal's characteristics, quality, and potential uses. However, due to limitations of equipment and technology, the obtained coal photomicrographs may have low resolution, failing to show clear details. In this study, we introduce a novel Generative Adversarial Network (GAN) to restore high-definition coal photomicrographs. Compared to traditional image restoration methods, the lightweight GAN-based network generates more explicit and realistic results. In particular, we employ the Wide Residual Block to eliminate the influence of artifacts and improve non-linear fitting ability. Moreover, we adopt a multi-scale attention block embedded in the generator network to capture long-range feature correlations across multiple scales. Experimental results on 468 photomicrographs demonstrate that the proposed method achieves a peak signal-to-noise ratio of 31.12 dB and a structural similarity index of 0.906, significantly higher than state-of-the-art super-resolution reconstruction approaches.

Physics-informed deep learning for T2-deblurred superresolution turbo spin echo MRI.

Deep learning superresolution (SR) is a promising approach to reduce MRI scan time without requiring custom sequences or iterative reconstruction. Previous deep learning SR approaches have generated low-resolution training images by simple k-space truncation, but this does not properly model in-plane turbo spin echo (TSE) MRI resolution degradation, which has variable T2 relaxation effects in different k-space regions. To fill this gap, we developed a T2 -deblurred deep learning SR method for the SR of 3D-TSE images. A SR generative adversarial network was trained using physically realistic resolution degradation (asymmetric T2 weighting of raw high-resolution k-space data). For comparison, we trained the same network structure on previous degradation models without TSE physics modeling. We tested all models for both retrospective and prospective SR with 3 × 3 acceleration factor (in the two phase-encoding directions) of genetically engineered mouse embryo model TSE-MR images. The proposed method can produce high-quality 3 × 3 SR images for a typical 500-slice volume with 6-7 mouse embryos. Because 3 × 3 SR was performed, the image acquisition time can be reduced from 15 h to 1.7 h. Compared to previous SR methods without TSE modeling, the proposed method achieved the best quantitative imaging metrics for both retrospective and prospective evaluations and achieved the best imaging-quality expert scores for prospective evaluation. The proposed T2 -deblurring method improved accuracy and image quality of deep learning-based SR of TSE MRI. This method has the potential to accelerate TSE image acquisition by a factor of up to 9.

Sarcomeric network analysis of ex vivo cultivated human atrial appendage tissue using super-resolution microscopy

Investigating native human cardiac tissue with preserved 3D macro- and microarchitecture is fundamental for clinical and basic research. Unfortunately, the low accessibility of the human myocardium continues to limit scientific progress. To overcome this issue, utilizing atrial appendages of the human heart may become highly beneficial. Atrial appendages are often removed during open-heart surgery and can be preserved ex vivo as living tissue with varying durability depending on the culture method. In this study, we prepared living thin myocardial slices from left atrial appendages that were cultured using an air-liquid interface system for overall 10 days. Metabolic activity of the cultured slices was assessed using a conventional methyl thiazolyl tetrazolium (MTT) assay. To monitor the structural integrity of cardiomyocytes within the tissue, we implemented our recently described super-resolution microscopy approach that allows both qualitative and quantitative in-depth evaluation of sarcomere network based on parameters such as overall sarcomere content, filament size and orientation. Additionally, expression of mRNAs coding for key structural and functional proteins was analyzed by real-time reverse transcription polymerase chain reaction (qRT-PCR). Our findings demonstrate highly significant disassembly of contractile apparatus represented by degradation of alpha -actinin filaments detected after three days in culture, while metabolic activity was constantly rising and remained high for up to seven days. However, gene expression of crucial cardiac markers strongly decreased after the first day in culture indicating an early destructive response to ex vivo conditions. Therefore, we suggest static cultivation of living myocardial slices derived from left atrial appendage and prepared according to our protocol only for short-termed experiments (e.g. medicinal drug testing), while introduction of electro-mechanical stimulation protocols may offer the possibility for long-term integrity of such constructs.

Cerebrovascular super-resolution 4D Flow MRI - Sequential combination of resolution enhancement by deep learning and physics-informed image processing to non-invasively quantify intracranial velocity, flow, and relative pressure.

The development of cerebrovascular disease is tightly coupled to regional changes in intracranial flow and relative pressure. Image-based assessment using phase contrast magnetic resonance imaging has particular promise for non-invasive full-field mapping of cerebrovascular hemodynamics. However, estimations are complicated by the narrow and tortuous intracranial vasculature, with accurate image-based quantification directly dependent on sufficient spatial resolution. Further, extended scan times are required for high-resolution acquisitions, and most clinical acquisitions are performed at comparably low resolution (>1mm) where biases have been observed with regard to the quantification of both flow and relative pressure. The aim of our study was to develop an approach for quantitative intracranial super-resolution 4D Flow MRI, with effective resolution enhancement achieved by a dedicated deep residual network, and with accurate quantification of functional relative pressures achieved by subsequent physics-informed image processing. To achieve this, our two-step approach was trained and validated in a patient-specific in-silico cohort, showing good accuracy in estimating velocity (relative error: 15.0±0.1%, mean absolute error (MAE): 0.07±0.06m/s, and cosine similarity: 0.99±0.06 at peak velocity) and flow (relative error: 6.6±4.7%, root mean square error (RMSE): 0.56mL/s at peak flow), and with the coupled physics-informed image analysis allowing for maintained recovery of functional relative pressure throughout the circle of Willis (relative error: 11.0±7.3%, RMSE: 0.3±0.2mmHg). Furthermore, the quantitative super-resolution approach is applied to an in-vivo volunteer cohort, effectively generating intracranial flow images at <0.5mm resolution and showing reduced low-resolution bias in relative pressure estimation. Our work thus presents a promising two-step approach to non-invasively quantify cerebrovascular hemodynamics, being applicable to dedicated clinical cohorts in the future.

A review of image super-resolution approaches based on deep learning methods

A review of image super-resolution approaches based on deep learning methods

Super-Resolution Approach to Enhance Bone Marrow Trephine Image in the Classification of Classical Myeloproliferative Neoplasms

Many diseases require histopathology images to characterise biological components or study cell and tissue architectures. The histopathology images are also essential in supporting disease classification, including myeloproliferative neoplasms (MPN). Despite significant developments to improve the diagnostic tools, morphological assessment from histopathology images obtained by bone marrow trephine (BMT) remains crucial to confirm MPN subtypes. However, the assessment outcome is challenging due to subjective characteristics that are hard to replicate due to its inter-observer variability. Apart from that, image processing may reduce the quality of the BMT images and affect the diagnosis result. This study has developed a classification system for classical MPN subtypes: polycythemia vera (PV), essential thrombocythemia (ET), and primary myelofibrosis (MF). It was done by reconstructing low-resolution images of BMT using a super-resolution approach to address the issue. Identified low-resolution images from calculating Laplacian variance were reconstructed using a super-resolution convolution neural network (SRCNN) to transform into rich information of high-resolution images. Original BMT images and reconstructed BMT images using the SRCNN dataset were fed into a CNN classifier, and the classifier’s output for both datasets was compared accordingly. Based on the result, the dataset consisting of the reconstructed images showed better output with 92% accuracy, while the control images gave 88% accuracy. In conclusion, the high quality of histopathology images substantially impacts disease process classification, and the reconstruction of low-resolution images has improved the classification output.

Super-Resolution Optical Imaging of Bacterial Cells

We review different super-resolution microscopy approaches used by microbiologists to investigate bacterial cell morphology and functions. We evidence the resolution achieved and focus on fluorophore selection and other critical imaging requirements. Some recent applications are described, which have defined novel insights into bacterial cellular structures. In particular, we evidenced the pivotal role that super-resolution microscopy can play in a near future to deal with antimicrobial resistance.

A Hybrid Approach for retinal image super-resolution

Experts require large high-resolution retinal images to detect tiny abnormalities, such as microaneurysms or issues of vascular branches. However, these images often suffer from low quality (e.g., resolution) due to poor imaging device configuration and misoperations. Many works utilized Convolutional Neural Network-based (CNN) methods for image super-resolution. The authors focused on making these models more complex by adding layers and various blocks. It leads to additional computational expenses and obstructs the application in real-life scenarios. Thus, this paper proposes a novel, lightweight, deep-learning super-resolution method for retinal images. It comprises a Vision Transformer (ViT) encoder and a convolutional neural network decoder. To our best knowledge, this is the first attempt to use a transformer-based network to solve the issue of accurate retinal image super-resolution. A progressively growing super-resolution training technique is applied to increase the resolution of images by factors of 2, 4, and 8. The prominent architecture remains constant thanks to the adaptive patch embedding layer, which does not lead to additional computational expense due to increased up-scaling factors. This patch embedding layer includes 2-dimensional convolution with specific values of kernel size and strides that depend on the input shape. This strategy has removed the need to append additional super-resolution blocks to the model. The proposed method has been evaluated through quantitative and qualitative measures. The qualitative analysis also includes vessel segmentation of super-resolved and ground truth images. Experimental results indicate that the proposed method outperforms the current state-of-the-art methods.

Mish-DCTGAN based combined image super-resolution and deblurring approach for blurry license plates

Mish-DCTGAN based combined image super-resolution and deblurring approach for blurry license plates

Super-resolution of license plate images using attention modules and sub-pixel convolution layers

Recent years have seen significant developments in the field of License Plate Recognition (LPR) through the integration of deep learning techniques and the increasing availability of training data. Nevertheless, reconstructing license plates (LPs) from low-resolution (LR) surveillance footage remains challenging. To address this issue, we introduce a Single-Image Super-Resolution (SISR) approach that integrates attention and transformer modules to enhance the detection of structural and textural features in LR images. Our approach incorporates sub-pixel convolution layers (also known as PixelShuffle) and a loss function that uses an Optical Character Recognition (OCR) model for feature extraction. We trained the proposed architecture on synthetic images created by applying heavy Gaussian noise to high-resolution LP images from two public datasets, followed by bicubic downsampling. As a result, the generated images have a Structural Similarity Index Measure (SSIM) of less than 0.10. Our results show that our approach for reconstructing these low-resolution synthesized images outperforms existing ones in both quantitative and qualitative measures. Our code is publicly available at .

Lightweight Implicit Blur Kernel Estimation Network for Blind Image Super-Resolution

Blind image super-resolution (Blind-SR) is the process of leveraging a low-resolution (LR) image, with unknown degradation, to generate its high-resolution (HR) version. Most of the existing blind SR techniques use a degradation estimator network to explicitly estimate the blur kernel to guide the SR network with the supervision of ground truth (GT) kernels. To solve this issue, it is necessary to design an implicit estimator network that can extract discriminative blur kernel representation without relying on the supervision of ground-truth blur kernels. We design a lightweight approach for blind super-resolution (Blind-SR) that estimates the blur kernel and restores the HR image based on a deep convolutional neural network (CNN) and a deep super-resolution residual convolutional generative adversarial network. Since the blur kernel for blind image SR is unknown, following the image formation model of blind super-resolution problem, we firstly introduce a neural network-based model to estimate the blur kernel. This is achieved by (i) a Super Resolver that, from a low-resolution input, generates the corresponding SR image; and (ii) an Estimator Network generating the blur kernel from the input datum. The output of both models is used in a novel loss formulation. The proposed network is end-to-end trainable. The methodology proposed is substantiated by both quantitative and qualitative experiments. Results on benchmarks demonstrate that our computationally efficient approach (12x fewer parameters than the state-of-the-art models) performs favorably with respect to existing approaches and can be used on devices with limited computational capabilities.

Structured Illumination Microscopy Improves Spot Detection Performance in Spatial Transcriptomics.

Spatial biology is a rapidly growing research field that focuses on the transcriptomic or proteomic profiling of single cells within tissues with preserved spatial information. Imaging-based spatial transcriptomics uses epifluorescence microscopy, which has shown remarkable results for the identification of multiple targets in situ. Nonetheless, the number of genes that can be reliably visualized is limited by the diffraction of light. Here, we investigate the effect of structured illumination (SIM), a super-resolution microscopy approach, on the performance of single-gene transcript detection in spatial transcriptomics experiments. We performed direct mRNA-targeted hybridization in situ sequencing for multiple genes in mouse coronal brain tissue sections. We evaluated spot detection performance in widefield and confocal images versus those with SIM in combination with 20×, 25× and 60× objectives. In general, SIM increases the detection efficiency of gene transcript spots compared to widefield and confocal modes. For each case, the specific fold increase in localizations is dependent on gene transcript density and the numerical aperture of the objective used, which has been shown to play an important role, especially for densely clustered spots. Taken together, our results suggest that SIM has the capacity to improve spot detection and overall data quality in spatial transcriptomics.

Loop stacking organizes genome folding from TADs to chromosomes

Although population-level analyses revealed significant roles for CTCF and cohesin in mammalian genome organization, their contributions at the single-cell level remain incompletely understood. Here, we used a super-resolution microscopy approach to measure the effects of removal of CTCF or cohesin in mouse embryonic stem cells. Single-chromosome traces revealed cohesin-dependent loops, frequently stacked at their loop anchors forming multi-way contacts (hubs), bridging across TAD boundaries. Despite these bridging interactions, chromatin in intervening TADs was not intermixed, remaining separated in distinct loops around the hub. At the multi-TAD scale, steric effects from loop stacking insulated local chromatin from ultra-long range (>4 Mb) contacts. Upon cohesin removal, the chromosomes were more disordered and increased cell-cell variability in gene expression. Our data revise the TAD-centric understanding of CTCF and cohesin and provide a multi-scale, structural picture of how they organize the genome on the single-cell level through distinct contributions to loop stacking.

SsFPN: Scale Sequence (S2) Feature-Based Feature Pyramid Network for Object Detection.

Object detection is a fundamental task in computer vision. Over the past several years, convolutional neural network (CNN)-based object detection models have significantly improved detection accuracyin terms of average precision (AP). Furthermore, feature pyramid networks (FPNs) are essential modules for object detection models to consider various object scales. However, the AP for small objects is lower than the AP for medium and large objects. It is difficult to recognize small objects because they do not have sufficient information, and information is lost in deeper CNN layers. This paper proposes a new FPN model named ssFPN (scale sequence (S2) feature-based feature pyramid network) to detect multi-scale objects, especially small objects. We propose a new scale sequence (S2) feature that is extracted by 3D convolution on the level of the FPN. It is defined and extracted from the FPN to strengthen the information on small objects based on scale-space theory. Motivated by this theory, the FPN is regarded as a scale space and extracts a scale sequence (S2) feature by three-dimensional convolution on the level axis of the FPN. The defined feature is basically scale-invariant and is built on a high-resolution pyramid feature map for small objects. Additionally, the deigned S2 feature can be extended to most object detection models based on FPNs. We also designed a feature-level super-resolution approach to show the efficiency of the scale sequence (S2) feature. We verified that the scale sequence (S2) feature could improve the classification accuracy for low-resolution images by training a feature-level super-resolution model. To demonstrate the effect of the scale sequence (S2) feature, experiments on the scale sequence (S2) feature built-in object detection approach including both one-stage and two-stage models were conducted on the MS COCO dataset. For the two-stage object detection models Faster R-CNN and Mask R-CNN with the S2 feature, AP improvements of up to 1.6% and 1.4%, respectively, were achieved. Additionally, the APS of each model was improved by 1.2% and 1.1%, respectively. Furthermore, the one-stage object detection models in the YOLO series were improved. For YOLOv4-P5, YOLOv4-P6, YOLOR-P6, YOLOR-W6, and YOLOR-D6 with the S2 feature, 0.9%, 0.5%, 0.5%, 0.1%, and 0.1% AP improvements were observed. For small object detection, the APS increased by 1.1%, 1.1%, 0.9%, 0.4%, and 0.1%, respectively. Experiments using the feature-level super-resolution approach with the proposed scale sequence (S2) feature were conducted on the CIFAR-100 dataset. By training the feature-level super-resolution model, we verified that ResNet-101 with the S2 feature trained on LR images achieved a 55.2% classification accuracy, which was 1.6% higher than for ResNet-101 trained on HR images.

TESR: Two-Stage Approach for Enhancement and Super-Resolution of Remote Sensing Images

Remote Sensing (RS) images are usually captured at resolutions lower than those required. Deep Learning (DL)-based super-resolution (SR) architectures are typically used to increase the resolution artificially. In this study, we designed a new architecture called TESR (Two-stage approach for Enhancement and super-resolution), leveraging the power of Vision Transformers (ViT) and the Diffusion Model (DM) to increase the resolution of RS images artificially. The first stage is the ViT-based model, which serves to increase resolution. The second stage is an iterative DM pre-trained on a larger dataset, which serves to increase image quality. Every stage is trained separately on the given task using a separate dataset. The self-attention mechanism of the ViT helps the first stage generate global and contextual details. The iterative Diffusion Model helps the second stage enhance the image’s quality and generate consistent and harmonic fine details. We found that TESR outperforms state-of-the-art architectures on super-resolution of remote sensing images on the UCMerced benchmark dataset. Considering the PSNR/SSIM metrics, TESR improves SR image quality as compared to state-of-the-art techniques from 34.03/0.9301 to 35.367/0.9449 in the scale ×2. On a scale of ×3, it improves from 29.92/0.8408 to 32.311/0.91143. On a scale of ×4, it improves from 27.77/0.7630 to 31.951/0.90456. We also found that the Charbonnier loss outperformed other loss functions in the training of both stages of TESR. The improvement was by a margin of 21.5%/14.3%, in the PSNR/SSIM, respectively. The source code of TESR is open to the community.

Exploring microstructure and petrophysical properties of microporous volcanic rocks through 3D multiscale and super-resolution imaging

Digital rock physics offers powerful perspectives to investigate Earth materials in 3D and non-destructively. However, it has been poorly applied to microporous volcanic rocks due to their challenging microstructures, although they are studied for numerous volcanological, geothermal and engineering applications. Their rapid origin, in fact, leads to complex textures, where pores are dispersed in fine, heterogeneous and lithified matrices. We propose a framework to optimize their investigation and face innovative 3D/4D imaging challenges. A 3D multiscale study of a tuff was performed through X-ray microtomography and image-based simulations, finding that accurate characterizations of microstructure and petrophysical properties require high-resolution scans (≤ 4 μm/px). However, high-resolution imaging of large samples may need long times and hard X-rays, covering small rock volumes. To deal with these limitations, we implemented 2D/3D convolutional neural network and generative adversarial network-based super-resolution approaches. They can improve the quality of low-resolution scans, learning mapping functions from low-resolution to high-resolution images. This is one of the first efforts to apply deep learning-based super-resolution to unconventional non-sedimentary digital rocks and real scans. Our findings suggest that these approaches, and mainly 2D U-Net and pix2pix networks trained on paired data, can strongly facilitate high-resolution imaging of large microporous (volcanic) rocks.

Medical image super-resolution: A review, recent trends and techniques

The quality and resolution of medical images significantly impact the accuracy of clinical decisions. Preprocessing of medical images plays a crucial role in the performance of computer-aided diagnosis (CAD) systems, which helps to improve the quality of input images, remove noise, and enhance contrast. Medical images are often characterized by low resolution due to different constraints such as hardware limits and image acquisition time. To address this issue, researchers have proposed various super-resolution approaches to improve the resolution of medical imagesand therefore better image quality. Hence, the super-resolution (SR) process aims to reconstruct a high-resolution (HR) image from its low-resolution (LR) counterpart. Several techniques are suggested in the literature including interpolation-based methods, reconstruction-based methods and learning-based methods. In recent years, deep learning-based methods have shownpromising results in various medical image applications while preserving important features and structures. In this article, we present a review of different studies that have proven to be useful in medical image super-resolution including classical computer vision methods and deep learning based-methods.