Multi-frame Super-resolution Method Research Articles

Robust Multi-Frame Super-Resolution Based on Adaptive Half-Quadratic Function and Local Structure Tensor Weighted BTV.

It is difficult to improve image resolution in hardware due to the limitations of technology and too high costs, but most application fields need high resolution images, so super-resolution technology has been produced. This paper mainly uses information redundancy to realize multi-frame super-resolution. In recent years, many researchers have proposed a variety of multi-frame super-resolution methods, but it is very difficult to preserve the image edge and texture details and remove the influence of noise effectively in practical applications. In this paper, a minimum variance method is proposed to select the low resolution images with appropriate quality quickly for super-resolution. The half-quadratic function is used as the loss function to minimize the observation error between the estimated high resolution image and low-resolution images. The function parameter is determined adaptively according to observation errors of each low-resolution image. The combination of a local structure tensor and Bilateral Total Variation (BTV) as image prior knowledge preserves the details of the image and suppresses the noise simultaneously. The experimental results on synthetic data and real data show that our proposed method can better preserve the details of the image than the existing methods.

Multi-Frame Labeled Faces Database: Towards Face Super-Resolution from Realistic Video Sequences

Forensically trained facial reviewers are still considered as one of the most accurate approaches for person identification from video records. The human brain can utilize information, not just from a single image, but also from a sequence of images (i.e., videos), and even in the case of low-quality records or a long distance from a camera, it can accurately identify a given person. Unfortunately, in many cases, a single still image is needed. An example of such a case is a police search that is about to be announced in newspapers. This paper introduces a face database obtained from real environment counting in 17,426 sequences of images. The dataset includes persons of various races and ages and also different environments, different lighting conditions or camera device types. This paper also introduces a new multi-frame face super-resolution method and compares this method with the state-of-the-art single-frame and multi-frame super-resolution methods. We prove that the proposed method increases the quality of face images, even in cases of low-resolution low-quality input images, and provides better results than single-frame approaches that are still considered the best in this area. Quality of face images was evaluated using several objective mathematical methods, and also subjective ones, by several volunteers. The source code and the dataset were released and the experiment is fully reproducible.

Multi-frame super resolution via deep plug-and-play CNN regularization

Abstract Because of the ill-posedness of multi-frame super resolution (MSR), the regularization method plays an important role in the MSR field. Various regularization terms have been proposed to constrain the image to be estimated. However, artifacts also exist in the estimated image due to the artificial tendency in the manually designed prior model. To solve this problem, we propose a novel regularization-based MSR method with learned prior knowledge. By using the variable splitting technique, the fidelity term and regularization term are separated. The fidelity term is associated with an “ L 2 {L^{2}} - L 2 {L^{2}} ” form sub-problem. Meanwhile, the sub-problem respect to regularization term is a denoising problem, which can be solved by denoisers learned from a deep convolutional neural network. Different from the traditional regularization methods which employ hand-crafted image priors, in this paper the image prior model is replaced by learned prior implicitly. The two sub-problems are solved alternately and iteratively. The proposed method cannot only handle complex degradation model, but also use the learned prior knowledge to guide the reconstruction process to avoid the artifacts. Both the quantitative and qualitative results demonstrate that the proposed method gains better quality than the state-of-the-art methods.

Novel image restoration method based on multi‐frame super‐resolution for atmospherically distorted images

In this study, the authors propose a novel multi-frame super-resolution method using frame selection and multiple fusions for atmospherically distorted, zoomed-in, image-quality enhancement. When a small part of the image captured by placing a target several kilometres away from the fixed camera is enlarged, the quality of the part becomes poor owing to low resolution, spatial deformations and noise that are mainly caused by long distance and atmospheric turbulence. Thus, the authors propose an adaptive frame selection method that selects only a few frames with small blur based on the corresponding images with relatively clear edges. Further, they propose multiple fusion schemes to reconstruct the selected frames, thereby suppressing the influence of deformation. By converting all the frames into high-resolution based on each frame and integrating them, deformation and noise are effectively removed without high computation cost using the multiple fusion scheme. The proposed method, which enhances the quality of atmospherically distorted zoomed-in images, exhibits superior performance than the state-of-the-art image super-resolution methods with regard to high accuracy, efficiency and ease of implementation, ensuring that the proposed method is suitable for enhancing the quality of an image captured using a general digital camera or a smartphone.

Multi-Frame RGB/NIR Imaging for Low-Light Color Image Super-Resolution

We propose a method for super-resolution (SR) of low-resolution (LR) color images taken in low-light scenes. Our method is based on the multi-frame SR technique, which reconstructs a high-resolution color image by fusing multiple LR images that were taken at different camera positions so as to alter the alignment between scene details and pixels. Previous multi-frame SR methods have implicitly assumed that the LR images could be captured with less noise and blur. However, images taken in low-light scene will contain high level of noise and motion blur, making it difficult to achieve high quality SR. To overcome these problems, we utilize a single sensor that captures red, green, blue (RGB) and near-infrared (NIR) information. We capture multiple raw images of a low-light scene using our RGB/NIR single sensor with an NIR flash unit. Because a NIR flash can provide sufficient illumination for low-light scenes, the structural scene information that, contributes to an effective SR process, can be captured with high quality. With the help of the NIR information, we jointly perform deblurring, denoising and SR of LR RGB/NIR raw images. Our experiments demonstrate the effectiveness of the proposed method using synthetic and real RGB/NIR raw images.

Study of the Behavior of Super Resolution on Soft-Classified Output

Once the satellite sensor is in orbit, no hardware enhancement of the lens assembly can be done to improve spatial and spectral resolution. Super resolution (SR) as single frame or multi-frame can solve this problem up to a large extent. In this study, single- and multi-frame SR techniques were applied and tested on Worldview-2 datasets as well as on across-spatial datasets of LISS III and LISS IV. Study of soft classifier’s behavior on super-resolved images was performed through possibilistic C-means classifier. Quantitative methods based on calculation of peak signal-to-noise ratio, mean square error, root means square error, image quality index and qualitative methods of visual interpretation proved that both super-resolution methods remove outliers in an efficient way and resulted in images containing sharp edges. The single-frame super-resolution technique was found relatively inferior in terms of contrast and spatial resolution. Overall, multi-frame SR method outperformed other methods.

A Mixed Non‐local Prior Model for Image Super‐resolution Reconstruction

Generating high-resolution image from a set of degraded low-resolution images is a challenge problem in image processing. Due to the ill-posed nature of Super-resolution (SR), it is necessary to find an effective image prior model to make it well-posed. For this purpose, we propose a mixed non-local prior model by adaptively combining the non-local total variation and non-local H1 models, and establish a multi-frame SR method based on this mixed non-local prior model. The unknown Highresolution (HR) image, motion parameters and hyperparameters related to the new prior model and noise statistics are determined automatically, resulting in an unsupervised super-resolution method. Extensive experiments demonstrate the effectiveness of the proposed SR method, which can not only preserve image details better but also suppress noise better.

Video Super-Resolution via Bidirectional Recurrent Convolutional Networks.

Super resolving a low-resolution video, namely video super-resolution (SR), is usually handled by either single-image SR or multi-frame SR. Single-Image SR deals with each video frame independently, and ignores intrinsic temporal dependency of video frames which actually plays a very important role in video SR. Multi-Frame SR generally extracts motion information, e.g., optical flow, to model the temporal dependency, but often shows high computational cost. Considering that recurrent neural networks (RNNs) can model long-term temporal dependency of video sequences well, we propose a fully convolutional RNN named bidirectional recurrent convolutional network for efficient multi-frame SR. Different from vanilla RNNs, 1) the commonly-used full feedforward and recurrent connections are replaced with weight-sharing convolutional connections. So they can greatly reduce the large number of network parameters and well model the temporal dependency in a finer level, i.e., patch-based rather than frame-based, and 2) connections from input layers at previous timesteps to the current hidden layer are added by 3D feedforward convolutions, which aim to capture discriminate spatio-temporal patterns for short-term fast-varying motions in local adjacent frames. Due to the cheap convolutional operations, our model has a low computational complexity and runs orders of magnitude faster than other multi-frame SR methods. With the powerful temporal dependency modeling, our model can super resolve videos with complex motions and achieve well performance.

Multi-Frame Super-Resolution Reconstruction Based on Gradient Vector Flow Hybrid Field

In this paper, we propose a novel multi-frame super-resolution (SR) method, which is developed by considering image enhancement and denoising into the SR processing. For image enhancement, a gradient vector flow hybrid field (GVFHF) algorithm, which is robust to noise is first designed to capture the image edges more accurately. Then, through replacing the gradient of anisotropic diffusion shock filter (ADSF) by GVFHF, a GVFHF-based ADSF (GVFHF-ADSF) model is proposed, which can effectively achieve image denoising and enhancement. In addition, a difference curvature-based spatial weight factor is defined in the GVFHF-ADSF model to obtain an adaptive weight between denoising and enhancement in the flat and edge regions. Finally, a GVFHF-ADSF-based multi-frame SR method is presented by employing the GVFHF-ADSF model as a regularization term and the steepest descent algorithm is adopted to solve the inverse SR problem. Experimental results and comparisons with existing methods demonstrate that the proposed GVFHF-ADSF-based SR algorithm can effectively suppress both Gaussian and salt-and-pepper noise, meanwhile enhance edges of the reconstructed image.

Multiframe super-resolution based on half-quadratic prior with artifacts suppress

The multiframe super-resolution (SR) technique aims to obtain a high-resolution (HR) image by using a set of observed low-resolution (LR) images. In the reconstruction process, artifacts may be possibly produced due to the noise, especially in presence of stronger noise. In order to suppress artifacts while preserving discontinuities of images, in this paper a multiframe SR method is proposed by involving the reconstruction properties of the half-quadratic prior model together with the quadratic prior model using a convex combination. Moreover, by analyzing local features of the underlined HR image, these two prior models are combined by using an automatically calculated weight function, making both smooth and discontinuous pixels handled properly. A variational Bayesian inference (VBF) based algorithm is designed to efficiently and effectively seek the solution of the proposed method. With the VBF framework, motion parameters and hyper-parameters are all determined automatically, leading to an unsupervised SR method. The efficiency of the hybrid prior model is demonstrated theoretically and practically, which shows that our SR method can obtain better results from LR images even with stronger noise. Extensive experiments on several visual data have demonstrated the efficacy and superior performance of the proposed algorithm, which can not only preserve image details but also suppress artifacts.

Multispectral super-resolution of tropical cyclone imagery using sparsity-based approaches

ABSTRACTAn aperture synthesis system produces ringing at sharp edges and other transitions in the observed field. In this article, we have developed an efficient multispectral deconvolution method, based on the Split Bregman total variation minimization technique, and it was successful in reducing image ringing, blurring, and distortion, while sharpening the image and preserving the information content. We also present a multispectral multi-frame super-resolution method that is robust to image noise and noise in the point spread function (PSF) and leads to additional improvements in spatial resolution. The methodologies are based on current research in sparse optimization and compressed sensing, which lead to unprecedented efficiencies in solving image reconstruction problems.

Novel Graph Cuts Method for Multi-Frame Super-Resolution

In this letter, we propose a new graph cuts multi-frame super resolution method. The method is carried out in 3 steps. First, we project each high-resolution pixel $p$ onto the low-resolution images and select low-resolution pixels which fall within the zone of influence of $p$ . Second, we weigh the contribution of the low-resolution pixels via a soft switching function and add them to construct a virtual low resolution pixel. The high resolution image is then recovered after minimizing a Maximum a posteriori Markov Random Field (MAP-MRF) energy function. This is done by approximating our energy function to make it graph representable and minimize it with a graph cuts $\alpha $ -expansion algorithm. Experimental results show that our approach outperforms state-of-the-art methods.

Efficient Deconvolution and Super-Resolution Methods in Microwave Imagery

In this paper, we develop efficient deconvolution and super-resolution methodologies, and apply these techniques to reduce image blurring and distortion inherent in an aperture synthesis system. Such a system produces ringing at sharp edges and other transitions in the observed field. The conventional approach to suppressing sidelobes is to apply linear apodization, which has the undesirable side effect of degrading spatial resolution. We have developed an efficient total variation minimization technique based on Split Bregman deconvolution that reduces image ringing while sharpening the image and preserving information content. The model was generalized to include upsampling of deconvolved images to a higher resolution grid. Furthermore, a proposed multiframe super-resolution method is presented that is robust to image noise and noise in the point spread function, and leads to additional improvements in spatial resolution. Our super-resolution methodologies are based on current research in sparse optimization and compressed sensing, which lead to unprecedented efficiencies for solving image reconstruction problems.

A multi-frame super-resolution method based on the variable-exponent nonlinear diffusion regularizer

In this work, the authors have proposed a multi-frame super-resolution method that is based on the diffusion-driven regularization functional. The new regularizer contains a variable exponent that adaptively regulates its diffusion mechanism depending upon the local image features. In smooth regions, the method favors linear isotropic diffusion, which removes noise more effectively and avoids unwanted artifacts (blocking and staircasing). Near edges and contours, diffusion adaptively and significantly diminishes, and since noise is hardly visible in these regions, an image becomes sharper and resolute—with noise being largely reduced in flat regions. Empirical results from both simulated and real experiments demonstrate that our method outperforms some of the state-of-the-art classical methods based on the total variation framework.

A collaborative adaptive Wiener filter for multi-frame super-resolution

Factors that can limit the effective resolution of an imaging system may include aliasing from under-sampling, blur from the optics and external factors, and sensor noise. Image restoration and super-resolution (SR) techniques can be used to improve image resolution. One SR method, developed recently, is the adaptive Wiener filter (AWF) SR algorithm. This is a multi-frame SR method that combines registered temporal frames through a joint nonuniform interpolation and restoration process to provide a high-resolution image estimate. Variations of this method have been demonstrated to be effective for multi-frame SR, as well demosaicing RGB and polarimetric imagery. While the AWF SR method effectively exploits subpixel shifts between temporal frames, it does not exploit self similarity within the observed imagery. However, very recently, the current authors have developed a multi-patch extension of the AWF method. This new method is referred to as a collaborative AWF (CAWF). The CAWF method employs a finite size moving window. At each position, we identify the most similar patches in the image within a given search window about the reference patch. A single-stage weighted sum of all of the pixels in all of the similar patches is used to estimate the center pixel in the reference patch. Like the AWF, the CAWF can perform nonuniform interpolation, deblurring, and denoising jointly. The big advantage of the CAWF, versus the AWF, is the CAWF can also exploit self-similarity. This is particularly beneficial for treating low signal-to-noise ratio (SNR) imagery. To date, the CAWF has only been developed for Nyquist-sampled single-frame image restoration. In this paper, we extend the CAWF method for multi-frame SR. We provide a quantitative performance comparison between the CAWF SR and the AWF SR techniques using real and simulated data. We demonstrate that CAWF SR outperforms AWF SR, especially in low SNR applications.

Multi-Frame Super-Resolution of High Frequency with Spatially Weighted Bilateral Total Variance Regularization

Bayesian based Multi-Frame Super-Resolution (MF-SR) has been used as a popular and effective SR model. On the other hand, the texture region is not reconstructed sufficiently because it works on the spatial domain. In this study, the MF-SR method was extended to operate on the frequency domain to improve HF information as much as possible. For this, a spatially weighted bilateral total variation model was proposed as a regularization term for a Bayesian estimation. The experimental results showed that the proposed method can recover the texture region more realistically with reduced noise, compared to conventional methods.