Super-resolution Scheme Research Articles

Bayesian Nonlocal Patch Tensor Factorization for Hyperspectral Image Super-Resolution.

The synthesis of high-resolution (HR) hyperspectral image (HSI) by fusing a low-resolution HSI with a corresponding HR multispectral image has emerged as a prevalent HSI super-resolution (HSR) scheme. Recent researches have revealed that tensor analysis is an emerging tool for HSR. However, most off-the-shelf tensor-based HSR algorithms tend to encounter challenges in rank determination and modeling capacity. To address these issues, we construct nonlocal patch tensors (NPTs) and characterize low-rank structures with coupled Bayesian tensor factorization. It is worth emphasizing that the intrinsic global spectral correlation and nonlocal spatial similarity can be simultaneously explored under the proposed model. Moreover, benefiting from the technique of automatic relevance determination, we propose a hierarchical probabilistic framework based on Canonical Polyadic (CP) factorization, which incorporates a sparsity-inducing prior over the underlying factor matrices. We further develop an effective expectation-maximization-type optimization scheme for framework estimation. In contrast to existing works, the proposed model can infer the latent CP rank of NPT adaptively without tuning parameters. Extensive experiments on synthesized and real datasets illustrate the intrinsic capability of our model in rank determination as well as its superiority in fusion performance.

A Super-Resolution Scheme for Multichannel Radar Forward-Looking Imaging Considering Failure Channels and Motion Error

To obtain high-resolution images of the objects in front of platform, a super-resolution scheme for multichannel radar forward-looking imaging considering failure channels and motion error is proposed in this study. In the scheme, a failure channel detection method based on the correlation of pulse-compressed data of different channels is presented first, and then a revised steering matrix considering failure channels and motion error is constructed. Finally, the echo data are processed by the iterative adaptive approach (IAA) with the revised steering matrix. Simulation results are given to illustrate the effectiveness of the proposed scheme when dealing with failure channels and motion error.

Development of a Super-Resolution Scheme for Pediatric Magnetic Resonance Brain Imaging Through Convolutional Neural Networks

Pediatric medical imaging represents a real challenge for physicians, as children who are patients often move during the examination, and it causes the appearance of different artifacts in the images. Thus, it is not possible to obtain good quality images for this target population limiting the possibility of evaluation and diagnosis in certain pathological conditions. Specifically, magnetic resonance imaging (MRI) is a technique that requires long acquisition times and, therefore, demands the use of sedation or general anesthesia to avoid the movement of the patient, which is really damaging in this specific population. Because ALARA (as low as reasonably achievable) principles should be considered for all imaging studies, one of the most important reasons for establishing novel MRI imaging protocols is to avoid the harmful effects of anesthesia/sedation. In this context, ground-breaking concepts and novel technologies, such as artificial intelligence, can help to find a solution to these challenges while helping in the search for underlying disease mechanisms. The use of new MRI protocols and new image acquisition and/or pre-processing techniques can aid in the development of neuroimaging studies for children evaluation, and their translation to pediatric populations. In this paper, a novel super-resolution method based on a convolutional neural network (CNN) in two and three dimensions to automatically increase the resolution of pediatric brain MRI acquired in a reduced time scheme is proposed. Low resolution images have been generated from an original high resolution dataset and used as the input of the CNN, while several scaling factors have been assessed separately. Apart from a healthy dataset, we also tested our model with pathological pediatric MRI, and it successfully recovers the original image quality in both visual and quantitative ways, even for available examples of dysplasia lesions. We hope then to establish the basis for developing an innovative free-sedation protocol in pediatric anatomical MRI acquisition.

Image super-resolution: A comprehensive review, recent trends, challenges and applications

Super resolution (SR) is an eminent system in the field of computer vison and image processing to improve the visual perception of the poor-quality images. The key objective of image super resolution is to address the limitations of imaging systems mainly due to hardware problems and requirements for clinical processing of medical imaging using post-processing operations. Numerous super resolution strategies have been put-forward in the computer vision community to improve and achieve high-resolution images over the years. In the past few years, there has been a significant advancement in image super-resolution algorithms. This paper aims to provide the detailed survey on recent advancements in image super-resolution in terms of traditional, deep learning and the latest transformer-based algorithms. The in-depth taxonomy of broadly classified super-resolution techniques within these categories has been broadly discussed. An extensive survey has been carried out on deep learning techniques in terms of parameters, architecture, network complexity, depth, learning rate, framework, optimization, and loss function. Furthermore, we also address some of the significant parameters such as problem definition, evaluation metrics, publicly benchmarks datasets, loss functions and applications. In addition, we have performed an experimental analysis and comparison of various benchmark algorithms on publicly available datasets both qualitively and quantitively. Lastly, we conclude our survey by emphasizing some of the prospective future directions and open issues that the community need to address in the future.

A Proposed Video Super-Resolution Strategy using Wavelet Multi-Scale Convolutional Neural Networks

A Proposed Video Super-Resolution Strategy using Wavelet Multi-Scale Convolutional Neural Networks

Downscaling atmospheric chemistry simulations with physically consistent deep learning

Abstract. Recent advances in deep convolutional neural network (CNN)-based super resolution can be used to downscale atmospheric chemistry simulations with substantially higher accuracy than conventional downscaling methods. This work both demonstrates the downscaling capabilities of modern CNN-based single image super resolution and video super-resolution schemes and develops modifications to these schemes to ensure they are appropriate for use with physical science data. The CNN-based video super-resolution schemes in particular incur only 39 % to 54 % of the grid-cell-level error of interpolation schemes and generate outputs with extremely realistic small-scale variability based on multiple perceptual quality metrics while performing a large (8×10) increase in resolution in the spatial dimensions. Methods are introduced to strictly enforce physical conservation laws within CNNs, perform large and asymmetric resolution changes between common model grid resolutions, account for non-uniform grid-cell areas, super-resolve lognormally distributed datasets, and leverage additional inputs such as high-resolution climatologies and model state variables. High-resolution chemistry simulations are critical for modeling regional air quality and for understanding future climate, and CNN-based downscaling has the potential to generate these high-resolution simulations and ensembles at a fraction of the computational cost.

Smart quantum statistical imaging beyond the Abbe-Rayleigh criterion

The wave nature of light imposes limits on the resolution of optical imaging systems. For over a century, the Abbe-Rayleigh criterion has been utilized to assess the spatial resolution limits of imaging instruments. Recently, there has been interest in using spatial projective measurements to enhance the resolution of imaging systems. Unfortunately, these schemes require a priori information regarding the coherence properties of “unknown” light beams and impose stringent alignment conditions. Here, we introduce a smart quantum camera for superresolving imaging that exploits the self-learning features of artificial intelligence to identify the statistical fluctuations of unknown mixtures of light sources at each pixel. This is achieved through a universal quantum model that enables the design of artificial neural networks for the identification of photon fluctuations. Our protocol overcomes limitations of existing superresolution schemes based on spatial mode projections, and consequently provides alternative methods for microscopy, remote sensing, and astronomy.

Super-Resolution Ultrasound Imaging Scheme Based on a Symmetric Series Convolutional Neural Network.

In this paper, we propose a symmetric series convolutional neural network (SS-CNN), which is a novel deep convolutional neural network (DCNN)-based super-resolution (SR) technique for ultrasound medical imaging. The proposed model comprises two parts: a feature extraction network (FEN) and an up-sampling layer. In the FEN, the low-resolution (LR) counterpart of the ultrasound image passes through a symmetric series of two different DCNNs. The low-level feature maps obtained from the subsequent layers of both DCNNs are concatenated in a feed forward manner, aiding in robust feature extraction to ensure high reconstruction quality. Subsequently, the final concatenated features serve as an input map to the latter 2D convolutional layers, where the textural information of the input image is connected via skip connections. The second part of the proposed model is a sub-pixel convolutional (SPC) layer, which up-samples the output of the FEN by multiplying it with a multi-dimensional kernel followed by a periodic shuffling operation to reconstruct a high-quality SR ultrasound image. We validate the performance of the SS-CNN with publicly available ultrasound image datasets. Experimental results show that the proposed model achieves a high-quality reconstruction of the ultrasound image over the conventional methods in terms of peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), while providing compelling SR reconstruction time.

Depth Map Super-Resolution via Cascaded Transformers Guidance

Depth information captured by affordable depth sensors is characterized by low spatial resolution, which limits potential applications. Several methods have recently been proposed for guided super-resolution of depth maps using convolutional neural networks to overcome this limitation. In a guided super-resolution scheme, high-resolution depth maps are inferred from low-resolution ones with the additional guidance of a corresponding high-resolution intensity image. However, these methods are still prone to texture copying issues due to improper guidance by the intensity image. We propose a multi-scale residual deep network for depth map super-resolution. A cascaded transformer module incorporates high-resolution structural information from the intensity image into the depth upsampling process. The proposed cascaded transformer module achieves linear complexity in image resolution, making it applicable to high-resolution images. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art techniques for guided depth super-resolution.

Isotropic super-resolution light-sheet microscopy of dynamic intracellular structures at subsecond timescales.

Long-term visualization of the dynamic interactions between intracellular structures throughout the three-dimensional space of whole live cells is essential to better understand their functions, but this task remains challenging due to the limitations of existing three-dimensional fluorescence microscopy techniques, such as an insufficient axial resolution, low volumetric imaging rate and photobleaching. Here, we present the combination of a progressive deep-learning super-resolution strategy with a double-ring-modulated selective plane illumination microscopy design capable of visualizing the dynamics of intracellular structures in live cells for hours at an isotropic spatial resolution of roughly 100 nm in three dimensions at speeds up to roughly 17 Hz. Using this approach, we reveal the complex spatial relationships and interactions between endoplasmic reticulum (ER) and mitochondria throughout live cells, providing new insights into ER-mediated mitochondrial division. We also examined the motion of Drp1 oligomers involved in mitochondrial fission and revealed the dynamic interactions between Drp1 and mitochondria in three dimensions.

2-D canonical correlation analysis based image super-resolution scheme for facial emotion recognition

2-D canonical correlation analysis based image super-resolution scheme for facial emotion recognition

Plug-and-Play video super-resolution using edge-preserving filtering

Video super-resolution objective is based on high resolution video frame reconstruction after downsampling. Such technology overcomes the inherent resolution limitations in videos. The mentioned task considers the inverse problem of original high resolution video frame recovery using prior information and reasonable suppositions. Motivated by the notable results of learning-based super-resolution strategies in obtaining high resolution outcomes from low resolution inputs, we leverage student-t mixture model as a promising reconstruction tool regarding video super-resolution. Student-t mixture model has a heavy tail which makes it robust and suitable for video frame patch prior and a rich mixture model in terms of log likelihood for information retrieval. Furthermore, in order to overcome the potential data uncertainties, edge-preserving filtering is applied to detect and preserve the video frame areas where the light sharply changes. For this purpose, we consider video frames in the framework of patches which will be selected on each frame based on their high amount of information. Afterwards, we use Plug-and-Play structure in order to apply student-t mixture prior model along with edge preserving filtering for representing the video frame patch prior in super-resolution algorithm. Finally, the proposed algorithm is evaluated empirically over 5 video frame sets, escalator, fountain, tree, wave and traffic. In addition, the results are compared with eight other state of the art super-resolution methods including SAN, RCAN, RDN, ANR, A+, SRCNN, FSRCNN-s and SelfExSR and it is proved that the proposed framework generally provides the best results in comparison with other techniques over four different super-resolution scales.

Multiple Frame Splicing and Degradation Learning for Hyperspectral Imagery Super-Resolution

Hyperspectral Imagery (HSI) is an emerging remote sensing technology to discriminate different remote sensing objects. However, the HSI spatial resolution is relatively low due to the trade-off in restricted physical hardware and various imaging conditions, restricting the subsequent object detection applications. At present, the Single Hyperspectral Super Resolution (SHSR) strategy has encountered the bottleneck on more precise details extraction, and the Fusion Hyperspectral Image Super Resolution (FHSR) strategy must need extra RGB/multispectral information which is not suitable for general HSI usage. Also, both types of current strategies focus less on the multiple degradation causes of low spatial resolution. In this paper, a step forward in designing a novel framework of multiple frame splicing strategy to greatly improve the SHSR quality, and applying multiple HSI degradation models to better fit the real degradation circumstance. Specifically, the framework is an end-to-end Super Resolution (SR) network that supersedes a single up-sampling module and removes complex attention residual model due to the same size of multiple splicing low-resolution input samples with high-resolution outputs. The effective framework will alleviate the vague at higher multiples, and accelerate the training convergence. Based on this framework, multiple degradation low-resolution samples can be simultaneously combined to fit better for the blind super-resolution result. Concretely, the degradation focus on the blur, noise, compression, and their combinations to simulate the real degradation. Experimental results on three different hyperspectral datasets demonstrate that the proposed MFSDM algorithm can significantly enhance the details in the recovered high-resolution hyperspectral images, and outperforms the state-of-the-art SHSR methods.

Bayesian Hyperspectral Image Super-Resolution in the Presence of Spectral Variability

Synthesizing high resolution (HR) hyperspectral image (HSI) by merging a low resolution (LR) HSI with corresponding HR multispectral image (MSI) has become a promising HSI super-resolution scheme. Most existing HSI-MSI fusion methods are effective to some extent, while several challenges remain. First, the spectral response of a given material exhibits considerable variability due to different acquisition time and conditions, however variations in spectral signatures are often neglected. Second, a majority of off-the-shelf methods require predefined degradation operators, which can be unavailable in practice. To tackle above issues, we introduce a novel fusion approach with Bayesian framework. Specifically, we regard the up-sampled LR-HSI as the low frequency component of the underlying HR-HSI. We characterize the texture features of high and low frequency components respectively, which can enlarge modeling capacity and bypass the absence of degradation operators. Furthermore, we depict the relative smoothness of reflectance spectra with Gaussian Process. Extensive experiments on synthesized and real datasets illustrate the superiority of the proposed strategy in terms of fusion performance and robustness to spectral variability.

Dual-Resolution Local Attention Unfolding Network for Optical Remote Sensing Image Super-Resolution

Single image super-resolution technology based on deep learning is widely applied in remote sensing. In recent years, the deep unfolding super-resolution strategy has been proposed, which combines the neural networks with traditional optimization-based algorithms, making the neural networks interpretable and achieving high performance. However, the typical deep unfolding algorithms usually treat different kinds of blurring kernels in the same way, so the algorithms cannot take advantage of the properties of blurring kernels, limiting the algorithm’s performance. To design a super-resolution network that can fully use the properties of Gaussian blurring kernels, a dual-resolution local attention unfolding network (DLANet) is proposed. Based on the Gaussian blurring functions, a low-resolution (LR) space branch is designed to supplement the high-resolution (HR) space branch. Specifically, for Gaussian blurring kernels, the closer the pixel is to the center, the greater the weight is. It means that the pixel points retained after downsampling will contain more information about the original corresponding pixel points, and it could be easier to estimate their original pixel values. So we design two branches. The HR branch completes the estimation of the whole image, and the LR branch only estimates the points retained after downsampling. To better complete the feature fusion of the two branches, we propose a row-column decoupling local attention module. This module can retain more information when fuse features and the row-column decoupling strategy can reduce computational complexity. Comprehensive experiments demonstrate the superiority of our method over the current state-of-the-art on remote sensing datasets.

Deep-ASL enhancement technique in arterial spin labeling MRI – A novel approach for the error reduction of partial volume correction technique with linear regression algorithm

Arterial Spin Labelling MRI is a noninvasive quantitative imaging technique for measuring Cerebral Blood Flow (CBF) that plays a vital role in diagnosing different neurological disorders. Limited signal-to-noise ratio and significant partial volume effect due to the low resolution of ASL images make an accurate CBF estimation difficult. This work proposes a deep learning based ASL enhancement algorithm (Deep-ASL ENHANCE), based on the principle of single image super resolution and multi-loss joint strategy with two reconstruction modules and one weighted fusion module that employ residual dense block as the basic building block. Lack of huge amount of low quality and high quality images for training this deep learning network, is addressed by generating simulated ASL images from structural images of ADNI2. The experiment is conducted and results are evaluated on a simulated dataset in terms of different metrics such as RMSE, PSNR and SSIM. The model is also validated using clinical ASL images with the help of two independent radiologists and the results are compared using Visual Quality Score (VQS). The deep learning model trained by using simulated ASL images shows more promising results on clinical ASL data. The effectiveness of using Deep-ASL ENHANCE as a preprocessing step to the partial volume correction technique with Linear Regression algorithm (LR) has been investigated using RMSE score and it is found that CBF quantification accuracy is improved compared to the standalone LR algorithm.

A diagnostic information based framework for super-resolution and quality assessment of retinal OCT images.

High-resolution (HR) retinal optical coherence tomography (OCT) images are preferred by the ophthalmologists to diagnose retinal diseases. These images can be obtained by dense scanning of the target retinal region during acquisition. However, a dense scanning increases the image acquisition time and introduces motion artefacts, which corrupt diagnostic information. Therefore, researchers have a growing interest in developing image processing techniques to recover HR images from low-resolution (LR) OCT images. In this paper, we present an automated super-resolution (SR) scheme using diagnostic information weighted sparse representation framework to reconstruct HR images from LR OCT images. The proposed method performs fast and reliable reconstruction of the LR images. We also propose a 2D- variational mode decomposition (VMD) based OCT diagnostic distortion measure (QOCT) to quantify diagnostic distortion in the reconstructed OCT images. The SR method is evaluated on clinical grade OCT images with the proposed diagnostic distortion measure along with the conventional non-diagnostic measures like the contrast to noise ratio (CNR), the equivalent number of looks (ENL) and the peak signal to noise ratio (PSNR). The results show an average CNR of 4.07, ENL of 58.96 and PSNR of 27.72dB. An average score of 1.53 is obtained using the proposed diagnostic distortion measure. Experimental results quantify that the proposed QOCT metric can effectively capture diagnostic distortion.

A method for optimal linear super-resolution image restoration

In this paper, we propose a super-resolution (pixel grid refinement) method for digital images. It is based on the linear filtering of a zero-padded discrete signal. We introduce a continuous-discrete observation model to create a reconstruction system. The proposed observation model is typical of real-world imaging systems - an initially continuous signal first undergoes linear (dynamic) distortions and then is subjected to sampling and the effect of additive noise. The proposed method is optimal in the sense of mean square recovery error minimization. In the theoretical part of the article, a general scheme of the linear super-resolution of the signal is presented and expressions for the impulse and frequency responses of the optimal reconstruction system are derived. An expression for the error of such restoration is also derived. For the sake of brevity, the entire description is presented for one-dimensional signals, but the obtained results can easily be generalized for the case of two-dimensional images. The experimental section of the paper is devoted to the analysis of the super-resolution reconstruction error depending on the parameters of the observation model. The significant superiority of the proposed method in terms of the reconstruction error is demonstrated in comparison with linear interpolation, which is usually used to refine the grid of image pixels.

SRNHARB: A deep light-weight image super resolution network using hybrid activation residual blocks

The quality of the images in all image-based applications and specially in computer vision applications is very crucial. Hence, design of a light-weight high-performance image super resolution scheme that enhances the quality of the acquired images is crucial for satisfactory functioning of such applications. Design of most of image super resolution schemes ignore extracting and processing of the negative-valued features of the images. In this paper, a novel light-weight residual block, which efficiently extracts and processes both the positive and negative-valued features, is proposed. This new residual block is capable of producing a richer set of features in order to improve the super resolution performance of the network using a set of such blocks. The network using the new residual blocks is shown to yield a performance superior to those of the existing light-weight super resolution networks using other types of residual blocks.

Wavelet-based super resolution using pansharpened multispectral images

Several remote sensing applications require high-spatial-high-spectral resolution multispectral (MS) images. However, most MS sensors provide low-spatial-high-spectral resolution MS images together with high-spatial-low-spectral resolution panchromatic (PAN) bands. In order to increase the spatial resolution of MS bands to the resolution of PAN images and to obtain high-spatial/spectral resolution MS bands, either MS and PAN images are fused (i.e., pansharpening) or super resolution (SR) is performed using MS bands only. Nevertheless, existing methods do not utilize the available temporal and spatial information together. In this paper, we propose a multiframe SR algorithm using high-spatial/spectral resolution MS images (i.e., pansharpened), taking advantage of both spatial and temporal data, in order to exceed the spatial resolution of the available PAN bands. We first employ a wavelet-based pansharpening method on a set of MS and PAN images captured at different times. Then, we utilize these pansharpened MS bands in a wavelet-based multiframe SR scheme. The proposed method reveals the inter-wavelet-subband relationship of multitemporal images for SR. We demonstrate our results with comparisons on a Landsat 7 ETM+ dataset.