Multi-frame Super-resolution Reconstruction Research Articles

A Hybrid Regularization-Based Multi-Frame Super-Resolution Using Bayesian Framework

The prime purpose for the image reconstruction of a multi-frame super-resolution is to reconstruct a higher-resolution image through incorporating the knowledge obtained from a series of relevant low-resolution images, which is useful in numerous fields. Nevertheless, super-resolution image reconstruction methods are usually damaged by undesirable restorative artifacts, which include blurring distortion, noises, and stair-casing effects. Consequently, it is always challenging to achieve balancing between image smoothness and preservation of the edges inside the image. In this research work, we seek to increase the effectiveness of multi-frame super-resolution image reconstruction by increasing the visual information and improving the automated machine perception, which improves human analysis and interpretation processes. Accordingly, we propose a new approach to the image reconstruction of multi-frame super-resolution, so that it is created through the use of the regularization framework. In the proposed approach, the bilateral edge preserving and bilateral total variation regularizations are employed to approximate a high-resolution image generated from a sequence of corresponding images with low-resolution to protect significant features of an image, including sharp image edges and texture details while preventing artifacts. The experimental results of the synthesized image demonstrate that the new proposed approach has improved efficacy both visually and numerically more than other approaches.

Regularized Multiframe Super-Resolution Image Reconstruction Using Linear and Nonlinear Filters

The primary goal of the multiframe super-resolution image reconstruction is to produce an image with a higher resolution by integrating information extracted from a set of corresponding images with low resolution, which is used in various fields. However, super-resolution image reconstruction approaches are typically affected by annoying restorative artifacts, including blurring, noise, and staircasing effect. Accordingly, it is always difficult to balance between smoothness and edge preservation. In this paper, we intend to enhance the efficiency of multiframe super-resolution image reconstruction in order to optimize both analysis and human interpretation processes by improving the pictorial information and enhancing the automatic machine perception. As a result, we propose new approaches that firstly rely on estimating the initial high-resolution image through preprocessing of the reference low-resolution image based on median, mean, Lucy-Richardson, and Wiener filters. This preprocessing stage is used to overcome the degradation present in the reference low-resolution image, which is a suitable kernel for producing the initial high-resolution image to be used in the reconstruction phase of the final image. Then, L2 norm is employed for the data-fidelity term to minimize the residual among the predicted high-resolution image and the observed low-resolution images. Finally, bilateral total variation prior model is utilized to restrict the minimization function to a stable state of the generated HR image. The experimental results of the synthetic data indicate that the proposed approaches have enhanced efficiency visually and quantitatively compared to other existing approaches.

Dark Channel Based Multiframe Super-Resolution Reconstruction

Multiframe super-resolution (MFSR) can obtain a high-resolution image from a set of low-resolution images. The performance of super-resolution is affected by the image prior information. The current super-resolution algorithms typically use total variation prior and its improved version, restoring the image edges well. However, it will produce artifacts and stair effects in the smooth region of the image. Therefore, we propose a dark channel-based MFSR algorithm to achieve edge-preserving and noise-suppressing. Firstly, the total variation prior is used to ensure the edge-preserving ability of the algorithm. Secondly, the dark channel prior is added to suppress artifacts and stair effects. Finally, the weights of the prior terms are iteratively adapted to obtain the final high-resolution image. Experiments show that the proposed algorithm can achieve a better result in objective and subjective visual evaluations.

Multi-Frame Super-Resolution Reconstruction Algorithm of Optical Remote Sensing Images Based on Double Regularization Terms and Unsupervised Learning

High-resolution images have always been in urgent need in the fields of surveying, mapping, military and civilian. In this paper, first, based on anisotropic nonlinear diffusion tensor, a diffusion tensor regularization term which can make full use of direction selection smoothing property was constructed. Based on the improved gradient vector field (GVF), a regularization term which can constrain the continuity of gradient vectors for high-resolution and low-resolution images was constructed. On the basis of these, a multi-frame super-resolution reconstruction algorithm based on double regularization terms was proposed and verified by simulation. Second, combining PCA with adaptive dictionary learning, two constraints of reconstruction regularity based on improved nonlocal means and kernel regression were proposed for experimental verification, and an improved K-means clustering algorithm for initial centre selection of spatial characteristic measure clustering was proposed to enhance the stability of the algorithm. Then high-resolution image generated by learning method was used as the initial input of multi-frame reconstruction of optical remote sensing images. The experimental results show that the reconstruction algorithm based on partial differential equation and unsupervised learning achieves both subjective and objective results for the realization of super-resolution reconstruction of optical remote sensing images.

Multi-channel Memristive Pulse Coupled Neural Network Based Multi-frame Images Super-resolution Reconstruction Algorithm

The high-resolution image is the prerequisite of information acquisition and precise analysis. Multi-frame super-resolution images reconstruction technologies are able to address many image degraded issues (caused by external shooting environment), such as detail information lost, blurred edges, and so forth. According to the nanoscale memristor, a Multi-channel Memristive Pulse Coupled Neural Network (MMPCNN) model is proposed. This model is able to simulate the adaptive-variable linking coefficient in pulse coupled neural network. Meanwhile, the proposed network is applied to the multi-frame super resolution reconstruction for fusing the registered low resolution images. Furthermore, the sparse coding based super resolution method is performed to improve the original high-resolution image. Finally, a series of computer experiments and the relevant subjective/objective analysis jointly illustrate the validity and effectiveness of the entire scheme.

Multi-frame super-resolution reconstruction based on mixed Poisson–Gaussian noise

In this paper, a non-blind multi-frame super-resolution (SR) model based on mixed Poisson–Gaussian noise (MPGSR) is proposed. Poisson noise arises from the stochastic nature of the photon-counting process. Readout noise and reset noise inherent to the readout circuitry can be modeled by an additive Gaussian noise. Therefore, a mixed Poisson–Gaussian noise model is more appropriate for real imaging system. Instead of deriving the data fidelity term from the perspective of error norms and the corresponding influence functions, we address the multi-frame SR problem based on a statistical noise model. The derived objective function is decomposed into sub-functions and solved by the alternating direction method of multipliers (ADMM) algorithm which allows using techniques of constrained optimization. The validation of the proposed MPGSR was performed quantitatively and qualitatively on natural and X-ray images. In comparison to the optimization-based and learning-based state-of-the-art methods, we have demonstrated the feasibility of MPGSR and the significance of applying a more appropriate noise model on the SR image reconstruction.

Multi-frame super-resolution reconstruction based on global motion estimation using a novel CNN descriptor

In this paper, we introduce a novel feature descriptor based on deep learning that trains a model to match the patches of images on scenes captured under different viewpoints and lighting conditions for Multi-frame super-resolution. The patch matching of images capturing the same scene in varied circumstances and diverse manners is challenging. We develop a model which maps the raw image patch to a low dimensional feature vector. As our experiments show, the proposed approach is much better than state-of-the-art descriptors and can be considered as a direct replacement of SURF. The results confirm that these techniques further improve the performance of the proposed descriptor. Then we propose an improved Random Sample Consensus algorithm for removing false matching points. Finally, we show that our neural network based image descriptor for image patch matching outperforms state-of-the-art methods on a number of benchmark datasets and can be used for image registration with high quality in multi-frame super-resolution reconstruction.

Error Reduction in Infrared Thermography by Multiframe Super-Resolution

Accurate temperature measurement techniques are critical for monitoring hotspots that induce thermal stresses in electronics packages. Infrared thermography is a popular nonintrusive method for emissivity mapping and measuring surface temperature distribution, but is often impeded by the low native resolution of the camera. A promising technique to mitigate these resolution limits is multiframe super-resolution, which uses multiple subpixel shifted images to generate a single high-resolution image. This study quantifies the error reduction offered by multiframe super-resolution to demonstrate the potential improvement for infrared imaging applications. The multiframe super-resolution reconstruction is implemented using an algorithm developed to interpolate the sub-pixel-shifted low-resolution images to a higher resolution grid. Experimental multiframe super-resolution temperature maps of an electronic component are measured to demonstrate the improvement in feature capture and reduction in aliasing effects. Furthermore, emissivity mapping of the component surface is conducted and demonstrates a dramatic improvement in the temperature correction by multiframe super-resolution. A sensitivity analysis is conducted to assess the effect of registration uncertainty on the multiframe super-resolution algorithm; simulated images are used to demonstrate the smoothing effect at sharp emissivity boundaries as well as improvement in the feature size capture based on the native camera resolution. These results show that, within the limitations of the technique, multiframe super-resolution can be an effective approach for improving the accuracy of emissivity-mapped temperature measurements.

An Extra-Rate Spatial Enhancement Constructed by MSRR using Regularized Technique and SSRR using High-Frequency Pre-Forecasting

Under many circumstances, a high spatial resolution image (HR) is greatly needed for modern applications nevertheless the HR image captured device is usually overpriced cost. Hence, Super Resolution Reconstruction (SRR) technique, which can reconstruct a HR image from a single LR image or many LR images by using algebraic formulation, is one of the modern research fields in digital image processing (DIP) and Computer Vision (CV). In this paper, an extra-rate spatial enhancement constructed by MSRR (Multi-frame Super Resolution Reconstruction) using regularized technique and SSRR (Single-frame Super Resolution Reconstruction) using high-frequency pre-forecasting is presented in order to enlarge up to 16x ratio rate. Initially, a group of captured images with low spatial resolution are mathematically fused by MSRR using regularized technique established on a recursive Maximum Likelihood and Tukey's Biweight norm in order to enlarge up to 4x ratio rate. Next, this 4x enhanced image is enlarged to be 16x spatial resolution image by SSRR established on the high-frequency pre-forecasting. In the verification experimentation section, the verification outcome demonstrates that the proposed spatial enhancement is successful for enlarging HR image with 16x ratio rate with finer quality.

Multiframe super resolution reconstruction method based on light field angular images

The plenoptic camera can directly obtain 4-dimensional light field information from a 2-dimensional sensor. However, based on the sampling theorem, the spatial resolution is greatly limited by the microlenses. In this paper, we present a method of reconstructing high-resolution images from the angular images. First, the ray tracing method is used to model the telecentric-based light field imaging process. Then, we analyze the subpixel shifts between the angular images extracted from the defocused light field data and the blur in the angular images. According to the analysis above, we construct the observation model from the ideal high-resolution image to the angular images. Applying the regularized super resolution method, we can obtain the super resolution result with a magnification ratio of 8. The results demonstrate the effectiveness of the proposed observation model.

A New Sparse Representation Framework for Reconstruction of an Isotropic High Spatial Resolution MR Volume From Orthogonal Anisotropic Resolution Scans.

In magnetic resonance (MR), hardware limitations, scan time constraints, and patient movement often result in the acquisition of anisotropic 3-D MR images with limited spatial resolution in the out-of-plane views. Our goal is to construct an isotropic high-resolution (HR) 3-D MR image through upsampling and fusion of orthogonal anisotropic input scans. We propose a multiframe super-resolution (SR) reconstruction technique based on sparse representation of MR images. Our proposed algorithm exploits the correspondence between the HR slices and the low-resolution (LR) sections of the orthogonal input scans as well as the self-similarity of each input scan to train pairs of overcomplete dictionaries that are used in a sparse-land local model to upsample the input scans. The upsampled images are then combined using wavelet fusion and error backprojection to reconstruct an image. Features are learned from the data and no extra training set is needed. Qualitative and quantitative analyses were conducted to evaluate the proposed algorithm using simulated and clinical MR scans. Experimental results show that the proposed algorithm achieves promising results in terms of peak signal-to-noise ratio, structural similarity image index, intensity profiles, and visualization of small structures obscured in the LR imaging process due to partial volume effects. Our novel SR algorithm outperforms the nonlocal means (NLM) method using self-similarity, NLM method using self-similarity and image prior, self-training dictionary learning-based SR method, averaging of upsampled scans, and the wavelet fusion method. Our SR algorithm can reduce through-plane partial volume artifact by combining multiple orthogonal MR scans, and thus can potentially improve medical image analysis, research, and clinical diagnosis.

A new denoising model for multi-frame super-resolution image reconstruction

Multi-frame image super-resolution (SR) aims to combine the sub-pixel information from a sequence of low-resolution (LR) images to build a high-resolution (HR) one. SR techniques usually suffers from annoying restoration artifacts such as noise, jagged edges, and staircasing effect. In this paper, we aim to increase the performance of SR reconstitution under a variational framework using adaptive diffusion-based regularization term. We propose a new tensor based diffusion regularization that takes the benefit from the diffusion model of Perona–Malik in the flat regions and use a nonlinear tensor derived from the diffusion process of Weickert filter near boundaries. Thus, the proposed SR approach can preserve important image features (sharp edges and corners) much better while avoiding artifacts. The synthetic and real experimental results show the effectiveness of the proposed regularisation compared to other methods in both quantitatively and visually.

Integrating the Missing Information Estimation into Multi-frame Super-Resolution

Multi-frame super-resolution image reconstruction aims to restore a high-resolution image by fusing a set of low-resolution images. The low-resolution images are usually subject to some degradation, such as warping, blurring, down-sampling, or noising, which causes substantial information loss in the low-resolution images, especially in the texture regions. The missing information is not well estimated using existing traditional methods. In this paper, having analyzed the observation model describing the degradation process starting with a high-resolution image and moving to the low-resolution images, we propose a more reasonable observation model that integrates the missing information into the super-resolution reconstruction. Our approach is fully formulated in a Bayesian framework using the Kullback---Leibler divergence. In this way, the missing information is estimated simultaneously with the high-resolution image, motion parameters, and hyper-parameters. Our proposed estimation of the missing information improves the quality of the reconstructed image. Experimental results presented in this paper show improved performance compared with that of existing traditional methods.

Comparative Experimental Exploration of Robust Norm Functions for Iterative Super Resolution Reconstructions under Noise Surrounding

In DIP (Digital Image Processing) research society, the multi-frame SRR (Super Resolution Reconstruction) algorithm has grown to be the momentous theme in the last ten years because of its cost effectiveness and its superior spectacle. Consequently, for a multi-frame SRR algorithm which is commonly comprised of a Bayesian ML (Maximum Likelihood) approach and a regularization technique into the unify SRR framework, numerous robust norm functions (which have both redescending and non-redescending influence functions) have been commonly comprised in the unify SRR framework for increasingly against noise or outlier. First, this paper presents the mathematical model of several iterative SRR based on Bayesian ML (Maximum Likelihood) approach and a regularization technique. Three groups of robust norm functions (a zero-redescending influence function (Tukey’s Biweight, Andrew’s Sine and Hampel), a nonzero-redescending influence function (Lorentzian, Leclerc, Geman&McClure, Myriad and Meridian) and a non-redescending influence function (Huber)) are mathematically incorporated into the SRR framework. The close form solutions of the SRR framework based on these robust norm functions have been concluded. Later, the experimental section utilizes two standard images of Lena and Susie (40th) for pilot studies and fraudulent noise patterns of noiseless, AWGN, Poisson, Salt&Pepper, and Speckle of several magnitudes are used to contaminate these two standard images. In order to acquire the maximum PSNR, the comparative experimental exploration has been done by comprehensively tailoring all experimental parameters such as step-size, regularization parameter, norm constant parameter.

A noise-suppressing and edge-preserving multiframe super-resolution image reconstruction method

Super-resolution technology is an approach that helps to restore high quality images and videos from degraded ones. The method stems from an ill-posed minimization problem, which is usually solved using the L2 norm and some regularization techniques. Most of the classical regularizing functionals are based on the Total Variation and the Perona–Malik frameworks, which suffer from undesirable artifacts (blocking and staircasing). To address these problems, we have developed a super-resolution framework that integrates an adaptive diffusion-based regularizer. The model is feature-dependent: linear isotropic in flat regions, a condition that regularizes an image uniformly and removes noise more effectively; and nonlinear anisotropic near boundaries, which helps to preserve important image features, such as edges and contours. Additionally, the new regularizing kernel incorporates a shape-defining parameter that can be automatically updated to ensure convexity and stability of the corresponding energy functional. Comparisons with other methods show that our method is superior and, more importantly, can achieve higher reconstruction factors.

Super resolution of aerial image by means of polyphase components reconstruction

Multi-frame super resolution reconstruction is a technology for obtaining a high resolution image from a set of blurred and aliased low resolution images. The most popular and widely used super resolution methods are motion based. However, the estimation of motion information (registration) is very difficult, computationally expensive and inaccurate, especially for aerial image. The sub-pixel registration error restricts the performance of the subsequent super resolution. Instead of trying to parameterize the motion estimation model, this paper proposes an image super resolution framework based on the polyphase components reconstruction algorithm and an improved steering kernel regression algorithm. Given an image observation model, a reversible 2D polyphase decomposition, which breaks down a high resolution image into polyphase components, is obtained. Though the assumption of diversity sampling, this paper adopts a fundamentally different approach, in which the low-resolution frames is used as the basis and the reference frame as the reference sub-polyphase component of the high resolution image for recovering the polyphase components of the high resolution image. The polyphase components, which fuse the low resolution frames with the complementary details, can be obtained by computing their expansion coefficients in terms of this basis using the available sub-polyphase components and then inversely transforming them into a high resolution image. This paper accomplishes this by formulating the problem as the maximum likelihood estimation, which guarantees a close-to-perfect solution. Furthermore, this paper proposes an improved steering kernel regression algorithm, to help restore the fusion image with mild blur and random noise. This paper adaptively refines the steering kernel regression function according to the local region context and structures. Thus, this new algorithm not only effectively combines denoising and deblurring together, but also preserves the edge information. Our framework develops an efficient and stable algorithm to tackle the huge size and ill-posedness of the super resolution problem, and improves the computational efficiency via avoiding registration and iterative computation. Several experimental results on synthetic data illustrate that our method outperforms the state-of-the-art methods in quantitative and qualitative comparisons. The proposed super resolution algorithm can indeed reconstruct high-frequency information which is otherwise unavailable in the single LR image. It can effectively suppress blur and noise, and produce visually pleasing resolution enhancement in aerial images.

Multiframe Superresolution Reconstruction Based on Self-Learning Method

One category of the superresolution algorithms widely used in practical applications is dictionary-based superresolution algorithms, which constructs a single high-resolution (HR) and high-clarity image from multiple low-resolution (LR) images. Despite the fact that general dictionary-based superresolution algorithms obtain redundant dictionaries from numerous HR-LR images, HR image distortion is unavoidable. To solve this problem, this paper proposes a multiframe superresolution reconstruction based on self-learning methods. First, multiple images from the same scene are selected to be both input and training images, and larger-scale images, which are also involved in the training set, are constructed from the learning dictionary. Then, different larger-scale images are constructed via repetition of the first step and the initial HR sets whose scale closely approximates that of the target HR image are finally obtained. Lastly, initial HR images are fused into one target HR image under the NLM idea, while the IBP idea is adopted to meet the global constraint. The simulation results demonstrate that the proposed algorithm produces more accurate reconstructions than those produced by other general superresolution algorithms, while, in real scene experiments, the proposed algorithm can run well and create clearer HR images from input images captured by cameras.

Multi-Frame Super-Resolution Reconstruction Algorithm Based on Diffusion Tensor Regularization Term

This paper presents a multi-frame super-resolution (SR) reconstruction algorithm based on diffusion tensor regularization term. Firstly, L1-norm structure is used as data fidelity term, anisotropic diffusion equation with directional smooth characteristics is introduced as a prior knowledge to optimize reconstruction result. Secondly, combined with shock filter, SR reconstruction energy functional is established. Finally, Euler-Lagrange equation based on nonlinear diffusion model is exported. Simulation results validate that the proposed algorithm enhances image edges and suppresses noise effectively, which proves better robustness.

Zernike-Moment-Based Image Super Resolution

Multiframe super-resolution (SR) reconstruction aims to produce a high-resolution (HR) image using a set of low-resolution (LR) images. In the process of reconstruction, fuzzy registration usually plays a critical role. It mainly focuses on the correlation between pixels of the candidate and the reference images to reconstruct each pixel by averaging all its neighboring pixels. Therefore, the fuzzy-registration-based SR performs well and has been widely applied in practice. However, if some objects appear or disappear among LR images or different angle rotations exist among them, the correlation between corresponding pixels becomes weak. Thus, it will be difficult to use LR images effectively in the process of SR reconstruction. Moreover, if the LR images are noised, the reconstruction quality will be affected seriously. To address or at least reduce these problems, this paper presents a novel SR method based on the Zernike moment, to make the most of possible details in each LR image for high-quality SR reconstruction. Experimental results show that the proposed method outperforms existing methods in terms of robustness and visual effects.

Multi-frame Image Super-resolution Reconstruction based on Sparse Representation and POCS

Super-resolution image reconstruction algorithms produce a high-resolution image from one or a set of low-resolution images of the desired scene. In this paper, we present a novel two-stage super-resolution (SR) algorithm combined sparse signal representation with the projection onto convex sets (POCS). In the first stage, inspired by recent results in sparse signal representation, we get a high-resolution intermediate image based on learning dictionary method for each low-resolution image of an input image sequence. In the second stage, by fusing these high-resolution intermediate images, a higher resolution image is generated based on POCS method. Experiment results show the effectiveness of our method and the improved performance over other SR algorithms.