Robust Super-resolution Research Articles

Degradation regression with uncertainty for blind super-resolution

Some recent blind super-resolution (SR) efforts focus on designing complex degradation models to better simulate real-world degradations. The paired high-resolution (HR) & low-resolution (LR) samples synthesized by these models can cover a large degradation space, which helps train a robust SR model in real scenarios. However, these diverse synthetic samples may render the SR model degradation-unaware and prevent it from achieving optimal results on LR images with specific degradations. Alternatively, another category of methods is proposed to estimate specific degradations in the given application and then tailor a degradation-aware SR model accordingly. Nonetheless, degradation estimation is an ill-posed problem and accurate estimation is quite challenging. Towards these issues, we propose a probabilistic degradation estimator (PDE) which can predict the degradation as a certain distribution rather than a single point. Specifically, we develop an intersection over union (IoU) based degradation regression loss with uncertainty, which could lead PDE to shrink the possible degradation space of the test LR image. This enables the degradation model to synthesize more degradation-specific training samples and further improve SR performance. In this way, our PDE can alleviate degradation redundancy compared with degradation-unaware methods and is more robust to the degradation estimation error than previous degradation-aware methods. Extensive experiments show that the proposed PDE can help the SR model produce better results on both synthetic and real-world images.

A Robust Super-Resolution Gridless Imaging Framework for UAV-Borne SAR Tomography

A Robust Super-Resolution Gridless Imaging Framework for UAV-Borne SAR Tomography

Learning Structure for Concrete Crack Detection Using Robust Super-Resolution with Generative Adversarial Network

To develop countermeasures against the inevitable aging of tunnels, accurate inspection is critical for ensuring stable tunnel services. Tunnels are large-scale infrastructures, whereas the cracks in such tunnels are small-scale objects, necessitating the development of methods capable of rapidly detecting cracks in a wide field of view. Therefore, this study proposes a new learning structure based on a segmentation network. This structure includes super-resolution and generative adversarial networks, which contribute to improved detection performance with robustness to input data. Furthermore, a method for the effective construction of training data for the application of the proposed structure is presented. Subsequently, the performance of the method over 1,606 crack images with randomly degraded qualities is evaluated. The proposed structure presents improved crack intersection over union and F1-scores of 63.686% and 77.811%, respectively, and low variances of 0.9008 and 0.5015 compared to the original structure. The results presented herein indicate the possible application of the proposed accurate condition inspection technology to tunnel maintenance in the future.

Blind Super-Resolution of 3D MRI via Unsupervised Domain Transformation.

High-resolution medical images can be effectively used for clinical diagnosis. However, the acquisition of high-resolution images is difficult and often limited by medical instruments. Super-resolution (SR) methods provide a solution, where high-resolution (HR) images can be reconstructed from low-resolution (LR) ones. Most of existing deep neural networks for 3D SR medical images trained in a non-blind process, where LR images are directly degraded from HR data via a pre-determined downscale method. Such approaches rely heavily on the assumed degradation model, resulting in inevitable deviations in real clinical practice. Blind super-resolution, as a more attractive research line for this field, aims to generate HR images from LR inputs containing unknown degradation. Towards generalizing SR models for diverse types of degradation, we propose a robust blind SR of 3D medical images in an unsupervised manner with domain correction and upscaling treatment. First, a CycleGAN-based architecture is implemented to generate the LR data from the source domain to the target one for domain correction. Then, an upscaling network is learned via pre-determined HR-LR couples for reconstruction. The proposed framework is able to automatically learn noisy and blurry correction kernels for unpaired 3D SR magnetic resonance images (MRI). Our method achieves better and more robust performances in reconstruction of HR images from LR MRI with multiple unknown degradation processes, and show its superiority to other state-of-the-art supervised models and cycle-consistency based methods, especially in severe distortion cases.

Semi-Cycled Generative Adversarial Networks for Real-World Face Super-Resolution.

Real-world face super-resolution (SR) is a highly ill-posed image restoration task. The fully-cycled Cycle-GAN architecture is widely employed to achieve promising performance on face SR, but is prone to produce artifacts upon challenging cases in real-world scenarios, since joint participation in the same degradation branch will impact final performance due to huge domain gap between real-world and synthetic LR ones obtained by generators. To better exploit the powerful generative capability of GAN for real-world face SR, in this paper, we establish two independent degradation branches in the forward and backward cycle-consistent reconstruction processes, respectively, while the two processes share the same restoration branch. Our Semi-Cycled Generative Adversarial Networks (SCGAN) is able to alleviate the adverse effects of the domain gap between the real-world LR face images and the synthetic LR ones, and to achieve accurate and robust face SR performance by the shared restoration branch regularized by both the forward and backward cycle-consistent learning processes. Experiments on two synthetic and two real-world datasets demonstrate that, our SCGAN outperforms the state-of-the-art methods on recovering the face structures/details and quantitative metrics for real-world face SR. The code will be publicly released at https://github.com/HaoHou-98/SCGAN.

RZSR: Reference-Based Zero-Shot Super-Resolution With Depth Guided Self-Exemplars

Recent methods for single image super-resolution (SISR) have demonstrated outstanding performance in generating high-resolution (HR) images from low-resolution (LR) images. However, most of these methods show their superiority using synthetically generated LR images, and their generalizability to real-world images is often not satisfactory. In this paper, we pay attention to two well-known strategies developed for robust super-resolution (SR), i.e., reference-based SR (RefSR) and zero-shot SR (ZSSR), and propose an integrated solution, called reference-based zero-shot SR (RZSR). Following the principle of ZSSR, we train an image-specific SR network at test time using training samples extracted only from the input image itself. To advance ZSSR, we obtain reference image patches with rich textures and high-frequency details which are also extracted only from the input image using cross-scale matching. To this end, we construct an internal reference dataset and retrieve reference image patches from the dataset using depth information. Using LR patches and their corresponding HR reference patches, we train a RefSR network that is embodied with a non-local attention module. Experimental results demonstrate the superiority of the proposed RZSR compared to the previous ZSSR methods and robustness to unseen images compared to other fully supervised SISR methods.

Deep robust residual network for super-resolution of 2D fetal brain MRI

Spatial resolution is a key factor of quantitatively evaluating the quality of magnetic resonance imagery (MRI). Super-resolution (SR) approaches can improve its spatial resolution by reconstructing high-resolution (HR) images from low-resolution (LR) ones to meet clinical and scientific requirements. To increase the quality of brain MRI, we study a robust residual-learning SR network (RRLSRN) to generate a sharp HR brain image from an LR input. Due to the Charbonnier loss can handle outliers well, and Gradient Difference Loss (GDL) can sharpen an image, we combined the Charbonnier loss and GDL to improve the robustness of the model and enhance the texture information of SR results. Two MRI datasets of adult brain, Kirby 21 and NAMIC, were used to train and verify the effectiveness of our model. To further verify the generalizability and robustness of the proposed model, we collected eight clinical fetal brain MRI 2D data for evaluation. The experimental results have shown that the proposed deep residual-learning network achieved superior performance and high efficiency over other compared methods.

Deep Gradient Prior Regularized Robust Video Super-Resolution

This paper proposes a robust multi-frame video super-resolution (SR) scheme to obtain high SR performance under large upscaling factors. Although the reference low-resolution frames can provide complementary information for the high-resolution frame, an effective regularizer is required to rectify the unreliable information from the reference frames. As the high-frequency information is mostly contained in the image gradient field, we propose to learn the gradient-mapping function between the high-resolution (HR) and the low-resolution (LR) image to regularize the fusion of multiple frames. In contrast to the existing spatial-domain networks, we train a deep gradient-mapping network to learn the horizontal and vertical gradients. We found that adding the low-frequency information (mainly from the LR image) to the gradient-learning network can boost the performance of the network. A forward and backward motion field prior is used to regularize the estimation of the motion flow between frames. For robust SR reconstruction, a weighting scheme is proposed to exclude the outlier data. Visual and quantitative evaluations on benchmark datasets demonstrate that our method is superior to many state-of-the-art methods and can recover better details with less artifacts.

Modified Rider Optimization-Based V Channel Magnification for Enhanced Video Super Resolution

In video Super Resolution (SR), the problem of cost expense concerning the attainment of enhanced spatial resolution, computational complexity and difficulties in motion blur makes video SR a complex task. Moreover, maintaining temporal consistency is crucial to achieving an efficient and robust video SR model. This paper plans to develop an intelligent SR model for video frames. Initially, the video frames in RGB format will be transformed into HSV. In general, the improvement in video frames is done in V-channel to achieve High-Resolution (HR) videos. In order to enhance the RGB pixels, the current window size is enhanced to high-dimensional window size. As a novelty, this paper intends to formulate a high-dimensional matrix with enriched pixel intensity in V-channel to produce enhanced HR video frames. Estimating the enriched pixels in the high-dimensional matrix is complex, however in this paper, it is dealt in a significant way by means of a certain process: (i) motion estimation (ii) cubic spline interpolation and deblurring or sharpening. As the main contribution, the cubic spline interpolation process is enhanced via optimization in terms of selecting the optimal resolution factor and different cubic spline parameters. For optimal tuning, this paper introduces a new modified algorithm, which is the modification of the Rider Optimization Algorithm (ROA) named Mean Fitness-ROA (MF-ROA). Once the HR image is attained, it combines the HSV and converts to RGB, which obtains the enhanced output RGB video frame. Finally, the performance of the proposed work is compared over other state-of-the-art models with respect to BRISQUE, SDME and ESSIM measures, and proves its superiority over other models.

Coarse-to-Fine CNN for Image Super-Resolution

Deep convolutional neural networks (CNNs) have been popularly adopted in image super-resolution (SR). However, deep CNNs for SR often suffer from the instability of training, resulting in poor image SR performance. Gathering complementary contextual information can effectively overcome the problem. Along this line, we propose a coarse-to-fine SR CNN (CFSRCNN) to recover a high-resolution (HR) image from its low-resolution version. The proposed CFSRCNN consists of a stack of feature extraction blocks (FEBs), an enhancement block (EB), a construction block (CB) and, a feature refinement block (FRB) to learn a robust SR model. Specifically, the stack of FEBs learns the long- and short-path features, and then fuses the learned features by expending the effect of the shallower layers to the deeper layers to improve the representing power of learned features. A compression unit is then used in each FEB to distill important information of features so as to reduce the number of parameters. Subsequently, the EB utilizes residual learning to integrate the extracted features to prevent from losing edge information due to repeated distillation operations. After that, the CB applies the global and local LR features to obtain coarse features, followed by the FRB to refine the features to reconstruct a high-resolution image. Extensive experiments demonstrate the high efficiency and good performance of our CFSRCNN model on benchmark datasets compared with state-of-the-art SR models. The code of CFSRCNN is accessible on <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/hellloxiaotian/CFSRCNN</uri> .

Fast single image super-resolution using estimated low-frequency k-space data in MRI

PurposeSingle image super-resolution (SR) is highly desired in many fields but obtaining it is often technically limited in practice. The purpose of this study was to propose a simple, rapid and robust single image SR method in magnetic resonance (MR) imaging (MRI). MethodsThe idea is based on the mathematical formulation of the intrinsic link in k-space between a given (modulus) low-resolution (LR) image and the desired SR image. The method consists of two steps: 1) estimating the low-frequency k-space data of the desired SR image from a single LR image; 2) reconstructing the SR image using the estimated low-frequency and zero-filled high-frequency k-space data. The method was evaluated on digital phantom images, physical phantom MR images and real brain MR images, and compared with existing SR methods. ResultsThe proposed SR method exhibited a good robustness by reaching a clearly higher PSNR (25.77dB) and SSIM (0.991) averaged over different noise levels in comparison with existing edge-guided nonlinear interpolation (EGNI) (PSNR=23.78dB, SSIM=0.983), zero-filling (ZF) (PSNR=24.09dB, SSIM=0.985) and total variation (TV) (PSNR=24.54dB, SSIM=0.987) methods while presenting the same order of computation time as the ZF method but being much faster than the EGNI or TV method. The average PSNR or SSIM over different slice images of the proposed method (PSNR=26.33 dB or SSIM=0.955) was also higher than the EGNI (PSNR=25.07dB or SSIM=0.952), ZF (PSNR=24.97dB or SSIM=0.950) and TV (PSNR=25.70dB or SSIM=0.953) methods, demonstrating its good robustness to variation in anatomical structure of the images. Meanwhile, the proposed method always produced less ringing artifacts than the ZF method, gave a clearer image than the EGNI method, and did not exhibit any blocking effect presented in the TV method. In addition, the proposed method yielded the highest spatial consistency in the inter-slice dimension among the four methods. ConclusionsThis study proposed a fast, robust and efficient single image SR method with high spatial consistency in the inter-slice dimension for clinical MR images by estimating the low-frequency k-space data of the desired SR image from a single spatial modulus LR image.

Super-Resolution of Dynamic Scenes Using Sampling Rate Diversity.

In earlier work, we proposed a super-resolution (SR) method that required the availability of two low resolution (LR) sequences corresponding to two different sampling rates, where images from one sequence were used as a basis to represent the polyphase components (PPCs) of the high resolution (HR) image, while the other LR sequences provided the reference LR image (to be super-resolved). The (simple) algorithm implemented by Salem and Yagle is only applicable when the scene is static. In this paper, we recast our approach to SR as a two-stage example-based algorithm to process dynamic scenes. We employ feature selection to create, from the LR frames, local LR dictionaries to represent PPCs of HR patches. To enforce sparsity, we implement Gaussian generative models as an efficient alternative to L1-norm minimization. Estimation errors are further reduced using what we refer to as the anchors, which are based on the relationship between PPCs corresponding to different sampling rates. In the second stage, we revert to simple single frame SR (applied to each frame), using HR dictionaries extracted from the super-resolved sequence of the previous stage. The second stage is thus a reiteration of the sparsity coding scheme, using only one LR sequence, and without involving PPCs. The ability of the modified algorithm to super-resolve challenging LR sequences reintroduces sampling rate diversity as a prerequisite of robust multiframe SR.

Single Molecule Localisation Microscopy with sCMOS Cameras

Single-molecule localisation microscopy (SMLM) is rapidly becoming a widely used method to achieve super-resolution at the nanometre scale. The development of improved camera technologies has been increasingly important in enabling robust super-resolution of the PALM and STORM type. Here we investigate the performance of scientific complementary metal-oxide semiconductor (sCMOS) cameras in SMLM for nanometre scale imaging. With sCMOS cameras characteristics such as gain, offset and read noise are pixel dependent which can introduce positional bias when localising images of single molecules to determine their localisation. We investigate an algorithm based on a weighted least squares approach to correct this bias and show that it behaves stably in SMLM with typical background photon levels. Our algorithm includes criteria to exclude pixels with high dark currents or excessive read noise from the fitting process. Once signal levels are sufficiently high the pixel dependent properties have little impact on the localisation results and we identify signal levels where using complex algorithms may be unwarranted. We introduce practical strategies to simplify determination of camera maps and provide an efficient software implementation to allow for seamless bias-corrected SMLM with sCMOS cameras. With these algorithms, the characterisation of camera properties can be used to achieve a ∼22% improvement of the localisation precision over standard, un-corrected, algorithms. By comparing the localisation results from both sCMOS and EMCCD cameras, we confirm that sCMOS cameras can perform as well as EMCCDs while exhibiting distinct advantages, such as 20 times larger field of view and faster frame rates. We used an sCMOS based setup to obtain super-resolution images of a range of biological samples and conclude that sCMOS cameras are an attractive alternative to EMCCDs for SMLM.

A combined total variation and bilateral filter approach for image robust super resolution

In this paper, we consider the image super-resolution (SR) reconstitution problem. The main goal consists of obtaining a high-resolution (HR) image from a set of low-resolution (LR) ones. For that, we propose a novel approach based on a regularized criterion. The criterion is composed of the classical generalized total variation (TV) but adding a bilateral filter (BTV) regularizer. The main goal of our approach consists of the derivation and the use of an efficient combined deblurring and denoising stage that is applied on the high-resolution image. We demonstrate the existence of minimizers of the combined variational problem in the bounded variation space, and we propose a minimization algorithm. The numerical results obtained by our approach are compared with the classical robust super-resolution (RSR) algorithm and the SR with TV regularization. They confirm that the proposed combined approach allows to overcome efficiently the blurring effect while removing the noise.

Robust super-resolution by fusion of interpolated frames for color and grayscale images

Multi-frame super-resolution (SR) processing seeks to overcome undersampling issues that can lead to undesirable aliasing artifacts. The key to effective multi-frame SR is accurate subpixel inter-frame registration. This accurate registration is challenging when the motion does not obey a simple global translational model and may include local motion. SR processing is further complicated when the camera uses a division-of-focal-plane (DoFP) sensor, such as the Bayer color filter array. Various aspects of these SR challenges have been previously investigated. Fast SR algorithms tend to have difficulty accommodating complex motion and DoFP sensors. Furthermore, methods that can tolerate these complexities tend to be iterative in nature and may not be amenable to real-time processing. In this paper, we present a new fast approach for performing SR in the presence of these challenging imaging conditions. We refer to the new approach as Fusion of Interpolated Frames (FIF) SR. The FIF SR method decouples the demosaicing, interpolation, and restoration steps to simplify the algorithm. Frames are first individually demosaiced and interpolated to the desired resolution. Next, FIF uses a novel weighted sum of the interpolated frames to fuse them into an improved resolution estimate. Finally, restoration is applied to deconvolve the modeled system PSF. The proposed FIF approach has a lower computational complexity than most iterative methods, making it a candidate for real-time implementation. We provide a detailed description of the FIF SR method and show experimental results using synthetic and real datasets in both constrained and complex imaging scenarios. The experiments include airborne grayscale imagery and Bayer color array images with affine background motion plus local motion.

Sub pixel location identification using super resolved multilooking CHRIS data

Abstract. CHRIS /Proba is a multiviewing hyperspectral sensor that monitors the earth in five different zenith angles +55°, +36°, nadir, −36° and −55° with a spatial resolution of 17 m and within a spectral range of 400–1050 nm in mode 3. These multiviewing images are suitable for constructing a super resolved high resolution image that can reveal the mixed pixel of the hyperspectral image. In the present work, an attempt is made to find the location of various features constituted within the 17m mixed pixel of the CHRIS image using various super resolution reconstruction techniques. Four different super resolution reconstruction techniques namely interpolation, iterative back projection, projection on to convex sets (POCS) and robust super resolution were tried on the −36, nadir and +36 images to construct a super resolved high resolution 5.6 m image. The results of super resolution reconstruction were compared with the scaled nadir image and bicubic convoluted image for comparision of the spatial and spectral property preservance. A support vector machine classification of the best super resolved high resolution image was performed to analyse the location of the sub pixel features. Validation of the obtained results was performed using the spectral unmixing fraction images and the 5.6 m classified LISS IV image.

Robust Super-Resolution TOA Estimation against Doppler Shift for Vehicle Tracking

This letter proposes a robust super-resolution TOA estimation algorithm against Doppler shift for vehicle tracking. The proposed method is developed to make use of the dual shift invariant structure to eliminate the Doppler induced bias from TOA estimation. To verify the effectiveness of the proposed method, we compare the performance of the proposed method with that of conventional algorithms, such as MUSIC and ESPRIT.

A robust super-resolution approach with sparsity constraint in acoustic imaging

Acoustic imaging is a standard technique for mapping acoustic source powers and positions from limited observations on microphone sensors, which often causes an ill-conditioned inverse problem. In this article, we firstly improve the forward model of acoustic power propagation by considering background noises at the sensor array, and the propagation uncertainty caused by wind tunnel effects. We then propose a robust super-resolution approach via sparsity constraint for acoustic imaging in strong background noises. The sparsity parameter is adaptively derived from the sparse distribution of source powers. The proposed approach can jointly reconstruct source powers and positions, as well as the background noise power. Our approach is compared with the conventional beamforming, deconvolution and sparse regularization methods by simulated, wind tunnel data and hybrid data respectively. It is feasible to apply the proposed approach for effectively mapping monopole sources in wind tunnel tests.

Fast super-resolution using an adaptive Wiener filter with robustness to local motion

We present a new adaptive Wiener filter (AWF) super-resolution (SR) algorithm that employs a global background motion model but is also robust to limited local motion. The AWF relies on registration to populate a common high resolution (HR) grid with samples from several frames. A weighted sum of local samples is then used to perform nonuniform interpolation and image restoration simultaneously. To achieve accurate subpixel registration, we employ a global background motion model with relatively few parameters that can be estimated accurately. However, local motion may be present that includes moving objects, motion parallax, or other deviations from the background motion model. In our proposed robust approach, pixels from frames other than the reference that are inconsistent with the background motion model are detected and excluded from populating the HR grid. Here we propose and compare several local motion detection algorithms. We also propose a modified multiscale background registration method that incorporates pixel selection at each scale to minimize the impact of local motion. We demonstrate the efficacy of the new robust SR methods using several datasets, including airborne infrared data with moving vehicles and a ground resolution pattern for objective resolution analysis.

Robust super resolution of compressed video

This paper presents a robust algorithm to recover high-frequency information from compressed low-resolution (LR) video sequences. Previous super-resolution (SR) approaches have succeeded in resolution enhancement when the motion in the LR sequence is simple. However, when the motion is complex, new artifacts will be introduced in the SR processing. To solve this problem, we develop a robust Bayesian SR algorithm with two steps. We first isolate the frames individually to get their corresponding initial SR solutions within the Bayesian framework. Secondly, with a robust cost function to reject outliers and noise, final SR images are achieved with multiple LR frames. In the mean time, we impose the constraint that the distribution of high-resolution (HR) image gradient should be equal to one of the corresponding decompressed LR images to sharpen the edges of the results. As a result of these steps, we are able to produce high-quality deblurred results, which show a suppressing of high-frequency artifacts and less ringing artifacts, with a higher peak signal-to-noise ratio (PSNR).