Multi-frame Super-resolution Algorithm Research Articles

Novel multi-frame super resolution algorithm for X-ray diffraction (XRD)

The spatial distribution of signals in X-ray diffraction (XRD) may be very narrow, posing challenges in obtaining high angular resolution results through traditional methods. This article explores multi-frame super-resolution algorithms from the field of image processing, selecting and refining the best suited to the requirements of the crystal truncation rod (CTR) test. Consequently, a new algorithm with a corresponding methodology is proposed. This approach is subsequently applied to characterize the CTR signal of SrTiO3 single-crystal samples, effectively enhancing the clarity of blurry and rough CTR cross-section signals. The resolution with this method surpasses that obtained using the pixel size alone, which is beneficial for characterizing weak signals. By combining data in three dimensions, a significantly enhanced three-dimensional CTR signal is obtained. This method enables the extraction of otherwise difficult-to-obtain information and can be conveniently integrated into subsequent theoretical work. Moreover, these methods hold potential for application in other techniques that require improved resolution under similar conditions.

Exponential Fusion of Interpolated Frames Network (EFIF-Net): Advancing Multi-Frame Image Super-Resolution with Convolutional Neural Networks.

Convolutional neural networks (CNNs) have become instrumental in advancing multi-frame image super-resolution (SR), a technique that merges multiple low-resolution images of the same scene into a high-resolution image. In this paper, a novel deep learning multi-frame SR algorithm is introduced. The proposed CNN model, named Exponential Fusion of Interpolated Frames Network (EFIF-Net), seamlessly integrates fusion and restoration within an end-to-end network. Key features of the new EFIF-Net include a custom exponentially weighted fusion (EWF) layer for image fusion and a modification of the Residual Channel Attention Network for restoration to deblur the fused image. Input frames are registered with subpixel accuracy using an affine motion model to capture the camera platform motion. The frames are externally upsampled using single-image interpolation. The interpolated frames are then fused with the custom EWF layer, employing subpixel registration information to give more weight to pixels with less interpolation error. Realistic image acquisition conditions are simulated to generate training and testing datasets with corresponding ground truths. The observation model captures optical degradation from diffraction and detector integration from the sensor. The experimental results demonstrate the efficacy of EFIF-Net using both simulated and real camera data. The real camera results use authentic, unaltered camera data without artificial downsampling or degradation.

Fiber Bundle Image Reconstruction Using Convolutional Neural Networks and Bundle Rotation in Endomicroscopy.

Fiber-bundle endomicroscopy has several recognized drawbacks, the most prominent being the honeycomb effect. We developed a multi-frame super-resolution algorithm exploiting bundle rotation to extract features and reconstruct underlying tissue. Simulated data was used with rotated fiber-bundle masks to create multi-frame stacks to train the model. Super-resolved images are numerically analyzed, which demonstrates that the algorithm can restore images with high quality. The mean structural similarity index measurement (SSIM) improved by a factor of 1.97 compared with linear interpolation. The model was trained using images taken from a single prostate slide, 1343 images were used for training, 336 for validation, and 420 for testing. The model had no prior information about the test images, adding to the robustness of the system. Image reconstruction was completed in 0.03 s for 256 × 256 images indicating future real-time performance is within reach. The combination of fiber bundle rotation and multi-frame image enhancement through machine learning has not been utilized before in an experimental setting but could provide a much-needed improvement to image resolution in practice.

An Enhanced Fluid Registration for Image Multi-Frame Super Resolution

The main idea of multi-frame super resolution (SR) algorithms is to recover a single high-resolution image from a sequence of low resolution ones of the same object. The success of the SR approaches is often related to a well registration and restoration steps. Therefore, we propose a new approach based on fluid image registration and we use a second order partial differential equation (PDE) to treat both the registration and restoration steps that guarantees the success of SR algorithms. Since the registration step is usually a variational ill-posed model, a mathematical study is needed to check the existence of the solution to the regularized problem. Thus, we prove the existence and uniqueness of the well posed fluid image registration and assure also the existence of the used second order PDE in the restoration step. The results show that the proposed method is competitive with the existing methods.

An improved PDE-constrained optimization fluid registration for image multi-frame super resolution

The main idea of multi-frame super resolution (SR) algorithms is to recover a single high-resolution image from a sequence of low resolution ones of the same object. The success of the SR approaches is often related to a well registration and restoration steps. Therefore, we propose a new approach based on a partial differential equation (PDE)-constrained optimization fluid image registration and we use a fourth order PDE to treat both the registration and restoration steps that guarantee the success of SR algorithms. Since the registration step is usually a variational ill-posed model, a mathematical study is needed to check the existence of the solution to the regularized problem. Thus, we prove the existence and of the well posed fluid image registration and assure also the existence of the used second order PDE in the restoration step. The results show that the proposed method is competitive with the existing methods.

Research on Low-Resolution Image Fusion Algorithm Based on Deep Learning

The emergence of portable devices provides great convenience for image acquisition, but the image resolution is too low, which affects the identification and use of the image. Multiframe super-resolution algorithm makes a great contribution to extracting image features, but there is a problem with too much computation. Based on this, this paper proposes a low-resolution image fusion algorithm based on deep learning. Based on the introduction of extracting image features from low-resolution images, this paper proposes to fuse multiple low-resolution images, reduce the time of calculation error per frame, reduce the amount of calculation, extract feature information, and improve the alexnet network model to realize image extraction. The convolutional self-coding fusion network is used to realize image fusion and complete the reconstruction and fusion of low-resolution images. In the simulation analysis of the fusion algorithm, objective indexes such as network structure and loss function are selected to evaluate the effectiveness of the algorithm. This paper reduces the computational complexity of image acquisition and ensures that the amount of calculation can be reduced in each iteration. Analyze the effectiveness of the fusion algorithm, comprehensively evaluate the objective indicators, and verify the advantages of the algorithm through indicators such as gradient change and information entropy.

Wavelet-based super resolution using pansharpened multispectral images

Several remote sensing applications require high-spatial-high-spectral resolution multispectral (MS) images. However, most MS sensors provide low-spatial-high-spectral resolution MS images together with high-spatial-low-spectral resolution panchromatic (PAN) bands. In order to increase the spatial resolution of MS bands to the resolution of PAN images and to obtain high-spatial/spectral resolution MS bands, either MS and PAN images are fused (i.e., pansharpening) or super resolution (SR) is performed using MS bands only. Nevertheless, existing methods do not utilize the available temporal and spatial information together. In this paper, we propose a multiframe SR algorithm using high-spatial/spectral resolution MS images (i.e., pansharpened), taking advantage of both spatial and temporal data, in order to exceed the spatial resolution of the available PAN bands. We first employ a wavelet-based pansharpening method on a set of MS and PAN images captured at different times. Then, we utilize these pansharpened MS bands in a wavelet-based multiframe SR scheme. The proposed method reveals the inter-wavelet-subband relationship of multitemporal images for SR. We demonstrate our results with comparisons on a Landsat 7 ETM+ dataset.

NON-LINEAR MULTI-FRAME IMAGE DENOISING USING WEIGHTED NUCLEAR NORM MINIMIZATION

Abstract. We address the problem of constructing single low noise image from a sequence of multiple noisy images. We use the approach based on finding and averaging similar blocks in the image and extend it to multiple images. Unlike traditional multi-frame super-resolution algorithms, the block-matching approach does not require computationally expensive motion estimation for multi-frame image denoising. In this work, we use an algorithm based on weighted nuclear minimization for image denoising. The evaluation of the algorithm shows noticeable improvement of image quality when using multiple input images instead of single one. The improvement is the most noticeable in the areas with complex non-repeated structure.

Research on Multi-Frame Sea Surface Image Super Resolution Algorithm Based on Deep Learning

Research on Multi-Frame Sea Surface Image Super Resolution Algorithm Based on Deep Learning

Multi-Frame Super-Resolution Algorithm Based on a WGAN

Image super-resolution reconstruction has been widely used in remote sensing, medicine and other fields. In recent years, due to the rise of deep learning research and the successful application of convolutional neural networks in the image field, the super-resolution reconstruction technology based on deep learning has also achieved great development. However, there are still some problems that need to be solved. For example, the current mainstream image super-resolution algorithms based on single or multiple frames pursue high performance indicators such as PSNR and SSIM, while the reconstructed image is relatively smooth and lacks many high-frequency details. It is not conducive to application in a real environment. To address such problem, this paper proposes a super-resolution reconstruction model of sequential images based on Generative Adversarial Networks (GAN). The proposed approach combines the registration module to fuse adjacent frames, effectively use the detailed information in multiple consecutive frames, and enhances the spatio-temporality of low-resolution images in sequential images. While the GAN was used to improve the effect of image high-frequency texture detail reconstruction, WGAN was introduced to optimize model training. The reconstruction results not only improved the PSNR and SSIM indexes but also reconstructed more high-frequency detail textures. Finally, in order to further improve the perception effect, an additional registration loss item RLT is introduced in the GAN network perception loss. Through extensive experiments, it shows that the model proposed in this paper effectively obtains the information between the sequence images. When the PSNR and SSIM indicators are improve, it can reconstruct better high-frequency texture details than the current advanced multi-frame algorithms.

FAST AND ACCURATE MULTI-FRAME SUPER-RESOLUTION OF SATELLITE IMAGES

Abstract. Recent constellations of small satellites, such as Planet’s SkySats, offer new acquisition modes where very short videos or bursts of images are acquired instead of a single still image. Compared to sequences of multi-date images, these sequences of consecutive video frames yield a large redundancy of information within the range of seconds. This redundancy enables to increase the spatial resolution using multi-frame super-resolution algorithms. In this paper, we propose a novel super-resolution method based on a high-order spline interpolation model that combines multiple low-resolution frames to produce a high-resolution image. Moreover this method can be implemented efficiently on GPU to process entire images from real satellite acquisitions. Synthetic and real experiments show that the proposed method is able to recover fine details, and measurements of the resulting resolution indicate a gain of 10 cm / pixel with respect to Planet’s SkySat standard imagery products.

Multi-Frame Super-Resolution Reconstruction Algorithm of Optical Remote Sensing Images Based on Double Regularization Terms and Unsupervised Learning

High-resolution images have always been in urgent need in the fields of surveying, mapping, military and civilian. In this paper, first, based on anisotropic nonlinear diffusion tensor, a diffusion tensor regularization term which can make full use of direction selection smoothing property was constructed. Based on the improved gradient vector field (GVF), a regularization term which can constrain the continuity of gradient vectors for high-resolution and low-resolution images was constructed. On the basis of these, a multi-frame super-resolution reconstruction algorithm based on double regularization terms was proposed and verified by simulation. Second, combining PCA with adaptive dictionary learning, two constraints of reconstruction regularity based on improved nonlocal means and kernel regression were proposed for experimental verification, and an improved K-means clustering algorithm for initial centre selection of spatial characteristic measure clustering was proposed to enhance the stability of the algorithm. Then high-resolution image generated by learning method was used as the initial input of multi-frame reconstruction of optical remote sensing images. The experimental results show that the reconstruction algorithm based on partial differential equation and unsupervised learning achieves both subjective and objective results for the realization of super-resolution reconstruction of optical remote sensing images.

A multi-frame super-resolution based on new variational data fidelity term

The main idea of multi-frame super-resolution (SR) algorithm is to recover a single high-resolution (HR) image from a sequence of low resolution ones of the same scene. Since the restoration step of super-resolution algorithms is always an ill-posed problem, the choice of the fidelity term and the regularization are always crucial. In this paper, we propose a new variational SR framework based on an automatic selection of the weighting parameter that control the balance between the L1 and L2 fidelity terms, which handle different type of noise distributions. Concerning the regularization, we use the combined total variation (TV) and the total variation of the first derivatives (TV2) model with a new implementation of the Primal-dual algorithm to solve the corresponding discretized problem. The obtained results are compared with some competitive algorithms and confirm that the proposed method has much benefices over the others in avoiding some undesirable artifacts.

Application of Bayesian super-resolution imaging algorithm to micro–nano satellite images

The need for high-resolution imaging becomes particularly important in remote sensing image applications, such as ground-object identification. We introduce a Bayesian multiframe super-resolution algorithm that efficiently improves the imaging resolution of our micro–nano carbon satellite images. We begin by presenting a theoretical overview of the algorithm, and subsequently we conduct experiments in two phases. In the first phase, we compare the performance of our algorithm with those of other similar algorithms using a set of reference images. In the second practical application phase, we apply all the algorithms considered to panchromatic images obtained from our civilian micro–nano carbon satellite. Our results indicate that the proposed algorithm significantly outperforms the other algorithms, affording higher image resolution and greater image detail. We believe that our approach can contribute to the further development of satellite imaging systems.

Handheld multi-frame super-resolution

Compared to DSLR cameras, smartphone cameras have smaller sensors, which limits their spatial resolution; smaller apertures, which limits their light gathering ability; and smaller pixels, which reduces their signal-to-noise ratio. The use of color filter arrays (CFAs) requires demosaicing, which further degrades resolution. In this paper, we supplant the use of traditional demosaicing in single-frame and burst photography pipelines with a multiframe super-resolution algorithm that creates a complete RGB image directly from a burst of CFA raw images. We harness natural hand tremor, typical in handheld photography, to acquire a burst of raw frames with small offsets. These frames are then aligned and merged to form a single image with red, green, and blue values at every pixel site. This approach, which includes no explicit demosaicing step, serves to both increase image resolution and boost signal to noise ratio. Our algorithm is robust to challenging scene conditions: local motion, occlusion, or scene changes. It runs at 100 milliseconds per 12-megapixel RAW input burst frame on mass-produced mobile phones. Specifically, the algorithm is the basis of the Super-Res Zoom feature, as well as the default merge method in Night Sight mode (whether zooming or not) on Google's flagship phone.

Error Reduction in Infrared Thermography by Multiframe Super-Resolution

Accurate temperature measurement techniques are critical for monitoring hotspots that induce thermal stresses in electronics packages. Infrared thermography is a popular nonintrusive method for emissivity mapping and measuring surface temperature distribution, but is often impeded by the low native resolution of the camera. A promising technique to mitigate these resolution limits is multiframe super-resolution, which uses multiple subpixel shifted images to generate a single high-resolution image. This study quantifies the error reduction offered by multiframe super-resolution to demonstrate the potential improvement for infrared imaging applications. The multiframe super-resolution reconstruction is implemented using an algorithm developed to interpolate the sub-pixel-shifted low-resolution images to a higher resolution grid. Experimental multiframe super-resolution temperature maps of an electronic component are measured to demonstrate the improvement in feature capture and reduction in aliasing effects. Furthermore, emissivity mapping of the component surface is conducted and demonstrates a dramatic improvement in the temperature correction by multiframe super-resolution. A sensitivity analysis is conducted to assess the effect of registration uncertainty on the multiframe super-resolution algorithm; simulated images are used to demonstrate the smoothing effect at sharp emissivity boundaries as well as improvement in the feature size capture based on the native camera resolution. These results show that, within the limitations of the technique, multiframe super-resolution can be an effective approach for improving the accuracy of emissivity-mapped temperature measurements.

Single image super-resolution under multi-frame method

The multi-frame image super-resolution method utilizes a series of low-resolution images from the same scene to reconstruct the corresponding high-quality super-resolution image. However, the insufficient input low-resolution images make it incapable of reconstructing the high-resolution image from a single low-resolution image. In this work, we extend the framework of multi-frame image super-resolution to handle the single low-resolution image via rolling guidance filtering. And an improved version of diffusion-driven regularizer-based multi-frame image super-resolution algorithm is proposed and applied on passive millimeter-wave (PMMW) image super-resolution. Specifically, the joint filtering is first exploited to suppress the noise of single low-resolution noisy image. The rolling guidance method is exploited to generate the structurally multi-scale low-resolution images forming the basis of multi-frame image super-resolution. The generated image sequences are then fed to the nonlinear diffusion regularizer-based super-resolution algorithm. The two-directional total variation de-convolution is finally employed to remove the blur, producing a sharp and clear high-resolution image. Experiments demonstrate the effectiveness of the proposed method and show its superiority for the natural and PMMW images.

Preserving quality in minimum frame selection within multi-frame super-resolution

As there are data redundancies in successive frames in a multi-frame super resolution (SR) algorithm, one can expect that discarding some of these superfluous frames would have no impact on the quality of the high resolution (HR) output image. The present paper presents an efficient algorithm for selecting the proper combination of the minimum frames required for multi-frame SR algorithms so as to not only preserve the quality of the obtained HR output, but also reduce the SR procedure complexity and memory. To achieve this, the present study first seeks to prove that minimizing the spectral interference between the selected frames for SR procedure will result in maximizing the HR output power. Then, the criterion for measuring the Upper Bound on Spectral Interferences (UBSI) among the selected frames for SR procedure is presented; the formulation is expressed in such a way that it can be extended to global sub-pixel translations between frames. Our proposed frame selection algorithm evaluates all candidate combinations from input frames so that the best option capable of minimizing the UBSI can be selected. In order to evaluate our proposed frame selection algorithm, five well-known SR image reconstruction methods are applied both in four standard simulated images and in three well known real video sequences, employing two different procedures: Using our proposed frame selection algorithm and otherwise. The obtained results indicate that when our proposed frame selection algorithm is applied, the quality of the HR output images is preserved tantamount to considering all available frames. Besides, the computational complexity of the SR algorithms is dramatically reduced adopting the proposed frame selection algorithm, for the number of frames engaged in the SR is diminished. Also compared with the SR algorithms presented in the literature, our proposed frame selection method takes relatively negligible time to execute.

Sparse Representation-Based Multiple Frame Video Super-Resolution.

In this paper, we propose two multiple-frame super-resolution (SR) algorithms based on dictionary learning (DL) and motion estimation. First, we adopt the use of video bilevel DL, which has been used for single-frame SR. It is extended to multiple frames by using motion estimation with sub-pixel accuracy. We propose a batch and a temporally recursive multi-frame SR algorithm, which improves over single-frame SR. Finally, we propose a novel DL algorithm utilizing consecutive video frames, rather than still images or individual video frames, which further improves the performance of the video SR algorithms. Extensive experimental comparisons with the state-of-the-art SR algorithms verify the effectiveness of our proposed multiple-frame video SR approach.

Real-Time Enhancement of Dynamic Depth Videos with Non-Rigid Deformations.

We propose a novel approach for enhancing depth videos containing non-rigidly deforming objects. Depth sensors are capable of capturing depth maps in real-time but suffer from high noise levels and low spatial resolutions. While solutions for reconstructing 3D details in static scenes, or scenes with rigid global motions have been recently proposed, handling unconstrained non-rigid deformations in relative complex scenes remains a challenge. Our solution consists in a recursive dynamic multi-frame super-resolution algorithm where the relative local 3D motions between consecutive frames are directly accounted for. We rely on the assumption that these 3D motions can be decoupled into lateral motions and radial displacements. This allows to perform a simple local per-pixel tracking where both depth measurements and deformations are dynamically optimized. The geometric smoothness is subsequently added using a multi-level L1 minimization with a bilateral total variation regularization. The performance of this method is thoroughly evaluated on both real and synthetic data. As compared to alternative approaches, the results show a clear improvement in reconstruction accuracy and in robustness to noise, to relative large non-rigid deformations, and to topological changes. Moreover, the proposed approach, implemented on a CPU, is shown to be computationally efficient and working in real-time.