Multi-frame Super-resolution Research Articles

A novel theoretical analysis on optimal pipeline of multi-frame image super-resolution using sparse coding

Super-resolution is the process of obtaining a high-resolution (HR) image from one or more low-resolution (LR) images. Single image super-resolution (SISR) deals with one LR image while multi-frame super-resolution (MFSR) employs several LR ones to reach the HR output. MFSR pipeline consists of alignment, fusion, and reconstruction. We conduct a theoretical analysis using sparse coding (SC) and iterative shrinkage-thresholding algorithm to fill the gap of mathematical justification in the execution order of the optimal MFSR pipeline. Our analysis recommends executing alignment and fusion before the reconstruction stage (whether through deconvolution or SISR techniques). The suggested order ensures enhanced performance in terms of peak signal-to-noise ratio and structural similarity index. The optimal pipeline also reduces computational complexity compared to intuitive approaches that apply SISR to each input LR image. Also, we demonstrate the usefulness of SC in analysis of computer vision tasks such as MFSR, leveraging the sparsity assumption in natural images. Simulation results support the findings of our theoretical analysis, both quantitatively and qualitatively.

A Single-Frame and Multi-Frame Cascaded Image Super-Resolution Method.

The objective of image super-resolution is to reconstruct a high-resolution (HR) image with the prior knowledge from one or several low-resolution (LR) images. However, in the real world, due to the limited complementary information, the performance of both single-frame and multi-frame super-resolution reconstruction degrades rapidly as the magnification increases. In this paper, we propose a novel two-step image super resolution method concatenating multi-frame super-resolution (MFSR) with single-frame super-resolution (SFSR), to progressively upsample images to the desired resolution. The proposed method consisting of an L0-norm constrained reconstruction scheme and an enhanced residual back-projection network, integrating the flexibility of the variational model-based method and the feature learning capacity of the deep learning-based method. To verify the effectiveness of the proposed algorithm, extensive experiments with both simulated and real world sequences were implemented. The experimental results show that the proposed method yields superior performance in both objective and perceptual quality measurements. The average PSNRs of the cascade model in set5 and set14 are 33.413 dB and 29.658 dB respectively, which are 0.76 dB and 0.621 dB more than the baseline method. In addition, the experiment indicates that this cascade model can be robustly applied to different SFSR and MFSR methods.

A novel multi-frame image super-resolution model based on regularized nonlinear diffusion with Caputo time fractional derivative

In this work, we introduce an innovative fractional nonlinear parabolic model using a time-fractional order derivative, specifically employing the Caputo sense for fractional differentiation. This model aims to enhance traditional super-resolution models, particularly in the context of multi-frame image super-resolution. Additionally, we incorporate a regularized Perona–Malik diffusion mechanism to control the speed and direction of diffusion at each image location. We begin our study by exploring the theoretical solvability of our proposed model. Firstly, we employ the Faedo–Galerkin approach to establish the existence and uniqueness of a weak solution for an auxiliary fractional super-resolution model. Subsequently, we use the Schauder fixed point method to demonstrate the existence and uniqueness of a weak solution for our model. To validate the effectiveness of our model in the multi-frame super-resolution (SR) context, we conduct numerical experiments on images featuring diverse characteristics, including corners and edges, while applying various warping, decimation, and blurring matrices to the low-resolution (LR) images. We start the evaluation by introducing an adaptive discrete scheme tailored to the proposed model. To prove the robustness of our approach, we subject our images to varying levels of noise. Additionally, we perform simulations on real data (videos). The obtained high-resolution (HR) results demonstrate notable efficiency and robustness against noise, outperforming competitive models both visually and quantitatively.

Novel multi-frame super resolution algorithm for X-ray diffraction (XRD)

The spatial distribution of signals in X-ray diffraction (XRD) may be very narrow, posing challenges in obtaining high angular resolution results through traditional methods. This article explores multi-frame super-resolution algorithms from the field of image processing, selecting and refining the best suited to the requirements of the crystal truncation rod (CTR) test. Consequently, a new algorithm with a corresponding methodology is proposed. This approach is subsequently applied to characterize the CTR signal of SrTiO3 single-crystal samples, effectively enhancing the clarity of blurry and rough CTR cross-section signals. The resolution with this method surpasses that obtained using the pixel size alone, which is beneficial for characterizing weak signals. By combining data in three dimensions, a significantly enhanced three-dimensional CTR signal is obtained. This method enables the extraction of otherwise difficult-to-obtain information and can be conveniently integrated into subsequent theoretical work. Moreover, these methods hold potential for application in other techniques that require improved resolution under similar conditions.

Burst-Enhanced Super-Resolution Network (BESR).

Multi-frame super-resolution (MFSR) leverages complementary information between image sequences of the same scene to increase the resolution of the reconstructed image. As a branch of MFSR, burst super-resolution aims to restore image details by leveraging the complementary information between noisy sequences. In this paper, we propose an efficient burst-enhanced super-resolution network (BESR). Specifically, we introduce Geformer, a gate-enhanced transformer, and construct an enhanced CNN-Transformer block (ECTB) by combining convolutions to enhance local perception. ECTB efficiently aggregates intra-frame context and inter-frame correlation information, yielding an enhanced feature representation. Additionally, we leverage reference features to facilitate inter-frame communication, enhancing spatiotemporal coherence among multiple frames. To address the critical processes of inter-frame alignment and feature fusion, we propose optimized pyramid alignment (OPA) and hybrid feature fusion (HFF) modules to capture and utilize complementary information between multiple frames to recover more high-frequency details. Extensive experiments demonstrate that, compared to state-of-the-art methods, BESR achieves higher efficiency and competitively superior reconstruction results. On the synthetic dataset and real-world dataset of BurstSR, our BESR achieves PSNR values of 42.79 dB and 48.86 dB, respectively, outperforming other MFSR models significantly.

Exponential Fusion of Interpolated Frames Network (EFIF-Net): Advancing Multi-Frame Image Super-Resolution with Convolutional Neural Networks.

Convolutional neural networks (CNNs) have become instrumental in advancing multi-frame image super-resolution (SR), a technique that merges multiple low-resolution images of the same scene into a high-resolution image. In this paper, a novel deep learning multi-frame SR algorithm is introduced. The proposed CNN model, named Exponential Fusion of Interpolated Frames Network (EFIF-Net), seamlessly integrates fusion and restoration within an end-to-end network. Key features of the new EFIF-Net include a custom exponentially weighted fusion (EWF) layer for image fusion and a modification of the Residual Channel Attention Network for restoration to deblur the fused image. Input frames are registered with subpixel accuracy using an affine motion model to capture the camera platform motion. The frames are externally upsampled using single-image interpolation. The interpolated frames are then fused with the custom EWF layer, employing subpixel registration information to give more weight to pixels with less interpolation error. Realistic image acquisition conditions are simulated to generate training and testing datasets with corresponding ground truths. The observation model captures optical degradation from diffraction and detector integration from the sensor. The experimental results demonstrate the efficacy of EFIF-Net using both simulated and real camera data. The real camera results use authentic, unaltered camera data without artificial downsampling or degradation.

An assisted multi-frame approach for super-resolution in hyperspectral images of rock samples

Imaging spectroscopy is decisive for accurately characterizing rock samples for geological and mineral applications, since slight differences in mineral composition can be easily recognized in spectral signatures. Although, instrument limitations prevent high spatial resolution pixels scanned along hundreds of spectral channels, which considerably reduces the data capabilities. Multi frame Super-Resolution (SR) techniques can overcome this issue by retrieving high-quality images using purely computational methods. However, these approaches still have important drawbacks because of inherent uncertainties when estimating the spatial variation between frames, which strongly penalize the spectral information, essential for lithology. In this paper we propose an assisted framework to provide fully controlled motion parameters of the imaging process, skipping the intricate registration task. Essentially, we use a stepping device to successively change the position of rock samples for each new frame acquisition. With accurate knowledge about motion parameters, we were able to overcome the uncertainties regarding spatial registration between frames. Therefore, a reasonable number of equations connecting the low-resolution frames and the super resolved image could be used to solve the optimization problem. Extensive experiments proved that the proposed assisted method achieved best performances (both qualitatively and quantitatively) compared with similar approaches. Moreover, detailed spectral analysis showed almost absolute consistency between the original low-resolution image and the super-resolved result, which is crucial to guarantee reliable mineral characterization.

Joint Video Super-Resolution and Frame Interpolation via Permutation Invariance

We propose a joint super resolution (SR) and frame interpolation framework that can perform both spatial and temporal super resolution. We identify performance variation according to permutation of inputs in video super-resolution and video frame interpolation. We postulate that favorable features extracted from multiple frames should be consistent regardless of input order if the features are optimally complementary for respective frames. With this motivation, we propose a permutation invariant deep architecture that makes use of the multi-frame SR principles by virtue of our order (permutation) invariant network. Specifically, given two adjacent frames, our model employs a permutation invariant convolutional neural network module to extract "complementary" feature representations facilitating both the SR and temporal interpolation tasks. We demonstrate the effectiveness of our end-to-end joint method against various combinations of the competing SR and frame interpolation methods on challenging video datasets, and thereby we verify our hypothesis.

Multi-frame super-resolution of remote sensing images using attention-based GAN models

Multi-frame super-resolution (MFSR) of remote sensing (RS) imageries becomes a critical research topic with the launch of new satellites having video capturing capability and the advancement of artificial intelligence techniques. In this study, an attention-based Generative Adversarial Network (GAN) algorithm is proposed for the multi-frame remote sensing image super-resolution (MRSISR). Firstly, we introduced an attention module to the generator and designed a space-based net that worked on every single frame for better temporal information extraction. Secondly, we proposed a novel attention module for better spatial and spectral information extraction. Thirdly, we applied an attention-based discriminator for the discriminative ability improvement of the discriminator. We implemented several experiments with the state-of-the-art models and the proposed approach using SpaceNet7 and Jilin-1 datasets. We quantitatively and qualitatively compared the results of different multi-frame super-resolution models.

Multiframe Super-Resolution With Dual Pyramid Multiattention Network for Droplet Measurement

Accurate visual measurement of micrometer-scale flying droplets in inkjet printing remains a challenge due to low image resolution caused by severe image conditions. Multi-frame super resolution (MFSR) has the potential to break through the measurement bottleneck. However, most existing MFSR methods are not satisfactory in multi-frame information utilization, especially for fast-motion scenes, and they often suffer from detail loss. In this study, focusing on multi-frame information utilization and deep feature extraction, we propose a dual pyramid multi-attention network (DPMAN). First, a dual pyramid deformable alignment (DPDA) module is proposed to deal with diverse motion, which extracts explicit offsets to enhance deformable alignment and perform coarse-to-fine alignment. Then, a gated attention fusion (GAF) module is devised to adaptively aggregate the aligned features to emphasize favorable features. Finally, a residual self-attention reconstruction (RSAR) module based on the multi-stage aggregation self-attention architecture is proposed to extract finer deep features for detail restoration. Experimental results on three benchmark datasets demonstrate that DPMAN achieves state-of-the-art performance. DPMAN is applied to droplet image reconstruction and improves the measurement accuracy from 3.34% to 2.52%.

A Hybrid Regularization-Based Multi-Frame Super-Resolution Using Bayesian Framework

The prime purpose for the image reconstruction of a multi-frame super-resolution is to reconstruct a higher-resolution image through incorporating the knowledge obtained from a series of relevant low-resolution images, which is useful in numerous fields. Nevertheless, super-resolution image reconstruction methods are usually damaged by undesirable restorative artifacts, which include blurring distortion, noises, and stair-casing effects. Consequently, it is always challenging to achieve balancing between image smoothness and preservation of the edges inside the image. In this research work, we seek to increase the effectiveness of multi-frame super-resolution image reconstruction by increasing the visual information and improving the automated machine perception, which improves human analysis and interpretation processes. Accordingly, we propose a new approach to the image reconstruction of multi-frame super-resolution, so that it is created through the use of the regularization framework. In the proposed approach, the bilateral edge preserving and bilateral total variation regularizations are employed to approximate a high-resolution image generated from a sequence of corresponding images with low-resolution to protect significant features of an image, including sharp image edges and texture details while preventing artifacts. The experimental results of the synthesized image demonstrate that the new proposed approach has improved efficacy both visually and numerically more than other approaches.

Computational Super-Resolution Imaging With a Sparse Rotational Camera Array

A sparse camera array utilizes several identical cameras for multi-frame super-resolution. However, the influence of camera layout on super-resolution performance remains unclear. In conventional camera arrays, all cameras share a similar observation model, resulting in redundancy during information collection. This study presents a novel super-resolution framework that actively controls cameras in a sparse camera array to rotate to certain degrees. Through a detailed analysis of the forward imaging model, it is shown that the rotation of cameras uses the rotational asymmetry property of pixel layouts and shape. The proposed camera layout method provides more diverse and informative spatial sampling than a conventional camera array with only translations between cameras. A physics-enabled deep neural network was introduced for multi-frame super-resolution. With the explicitly defined gradients and affine models as guidance, it can effectively decode additional information collected by the rotational camera array and overcome the large-scale spatial feature alignment problem. The experimental results on synthetic and real prototype datasets validated the proposed theory and solution. We believe our study can inspire innovations for future task-specific camera array system design.

An Enhanced Fluid Registration for Image Multi-Frame Super Resolution

The main idea of multi-frame super resolution (SR) algorithms is to recover a single high-resolution image from a sequence of low resolution ones of the same object. The success of the SR approaches is often related to a well registration and restoration steps. Therefore, we propose a new approach based on fluid image registration and we use a second order partial differential equation (PDE) to treat both the registration and restoration steps that guarantees the success of SR algorithms. Since the registration step is usually a variational ill-posed model, a mathematical study is needed to check the existence of the solution to the regularized problem. Thus, we prove the existence and uniqueness of the well posed fluid image registration and assure also the existence of the used second order PDE in the restoration step. The results show that the proposed method is competitive with the existing methods.

Multi-Frame Blind Super-Resolution Based on Joint Motion Estimation and Blur Kernel Estimation

Multi-frame super-resolution makes up for the deficiency of sensor hardware and significantly improves image resolution by using the information of inter-frame and intra-frame images. Inaccurate blur kernel estimation will enlarge the distortion of the estimated high-resolution image. Therefore, multi-frame blind super resolution with unknown blur kernel is more challenging. For the purpose of reducing the impact of inaccurate motion estimation and blur kernel estimation on the super-resolved image, we propose a novel method combining motion estimation, blur kernel estimation and super resolution. The confidence weight of low-resolution images and the parameter value of the motion model obtained in image reconstruction are added to the modified motion estimation and blur kernel estimation. At the same time, Jacobian matrix, which can better describe the motion change, is introduced to further correct the error of motion estimation. Based on the results acquired from the experiments on synthetic data and real data, the superiority of the proposed method over others is obvious. The reconstructed high-resolution image retains the details of the image effectively, and the artifacts are greatly reduced.

An improved PDE-constrained optimization fluid registration for image multi-frame super resolution

The main idea of multi-frame super resolution (SR) algorithms is to recover a single high-resolution image from a sequence of low resolution ones of the same object. The success of the SR approaches is often related to a well registration and restoration steps. Therefore, we propose a new approach based on a partial differential equation (PDE)-constrained optimization fluid image registration and we use a fourth order PDE to treat both the registration and restoration steps that guarantee the success of SR algorithms. Since the registration step is usually a variational ill-posed model, a mathematical study is needed to check the existence of the solution to the regularized problem. Thus, we prove the existence and of the well posed fluid image registration and assure also the existence of the used second order PDE in the restoration step. The results show that the proposed method is competitive with the existing methods.

Research on Low-Resolution Image Fusion Algorithm Based on Deep Learning

The emergence of portable devices provides great convenience for image acquisition, but the image resolution is too low, which affects the identification and use of the image. Multiframe super-resolution algorithm makes a great contribution to extracting image features, but there is a problem with too much computation. Based on this, this paper proposes a low-resolution image fusion algorithm based on deep learning. Based on the introduction of extracting image features from low-resolution images, this paper proposes to fuse multiple low-resolution images, reduce the time of calculation error per frame, reduce the amount of calculation, extract feature information, and improve the alexnet network model to realize image extraction. The convolutional self-coding fusion network is used to realize image fusion and complete the reconstruction and fusion of low-resolution images. In the simulation analysis of the fusion algorithm, objective indexes such as network structure and loss function are selected to evaluate the effectiveness of the algorithm. This paper reduces the computational complexity of image acquisition and ensures that the amount of calculation can be reduced in each iteration. Analyze the effectiveness of the fusion algorithm, comprehensively evaluate the objective indicators, and verify the advantages of the algorithm through indicators such as gradient change and information entropy.

Simultaneous Measurement of Temperature and Strain in Electronic Packages Using Multiframe Super-Resolution Infrared Thermography and Digital Image Correlation

Abstract For micro-electronic components and systems, reliability under thermomechanical stress is of critical importance. Experimental characterization of hotspots and temperature gradients, which can lead to deformation in the component, relies on accurate mapping of the surface temperature. One method of noninvasively acquiring this data is through infrared (IR) thermography. However, IR thermography is often limited by the typically low resolution of such cameras. Additionally, the unique surface finish preparations required to infer physical deformation using digital image correlation (DIC) generally interfere with the ability to measure the temperature with IR thermography, which prefers a uniform high emissivity. This work introduces a one-shot technique for the simultaneous measurement of surface temperature and deformation using multiframe super-resolution-enhanced IR imaging combined with DIC analysis. Multiframe super-resolution processing uses several subpixel shifted images, interpolating the image set to extract additional information and create a single higher-resolution image. Measurement of physical deformation is incorporated using a test sample with a black background and low-emissivity speckle features, heated in a manner that induces a nonuniform temperature field and stretched to induce physical deformation. Through processing and filtering, data from the black surface regions used for surface temperature mapping are separated from the speckle features used to track deformation with DIC. This method allows DIC to be performed on the IR images, yielding a deformation field consistent with the applied tensioning. While both the low- and super-resolution data sets can be successfully processed with DIC, super-resolution helps to reduce noise in the extracted deformation fields. As for temperature measurement, using super-resolution is shown to allow for better removal of the speckle features and reduce noise, as quantified by a lower mean deviation from the spatial moving average.

Deep-Learning-Based Super-Resolution of Video Satellite Imagery by the Coupling of Multiframe and Single-Frame Models

Image super-resolution (SR) is an effective sol-ution to the limitation of the spatial resolution of video satellite images, which is caused by the degradation and compression in the imaging phase. For the processing of satellite videos, the commonly employed deep learning based single-frame SR (SFSR) framework have limited performance without using complementary information between the video frames. On the other side, the multi-frame SR (MFSR) can utilize temporal sub-pixel information to super-resolve the high-resolution (HR) imagery. However, although deeper and wider deep learning network provide powerful feature representations for SR methods, it has always been a challenge to accurately recon-struct the boundaries of ground objects in video satellite images. In this paper, to address these issues, we propose an edge-guided video super-resolution (EGVSR) framework for video satellite image SR, which couple MFSR model and edge-SFSR (E-SFSR) model in a unified network. The EGVSR framework is composed of an MFSR branch and an edge branch. The MFSR branch is used to extract the complementary features from the consecutive video frames. Concurrently, the edge branch acts as an SFSR model to translate the edge maps from the low-resolution modality to the HR one. At the final SR stage, the DBFM is built to focus on the promising inner repre-sentations of the features of the two branches and fuse them. Extensive experiments on video satellite imagery show that the proposed EGVSR method can achieve a superior performance when compared to the representative deep learning based SR methods.

Research on super-resolution reconstruction of remote sensing images: a comprehensive review

The super-resolution (SR) reconstruction of remote sensing images is a low-cost and efficient method to improve their resolution, and it is often used for further image analysis. To understand the development of SR reconstruction of remote sensing images and research hotspots and trends, we examined its history and reviewed existing methods categorized into traditional, learning-based, and deep-learning-based methods. To evaluate the reconstruction performance, we conducted experiments comparing various algorithms for the single- and multi-frame SR reconstruction of remote sensing images considering three datasets. The experimental results indicate the advantages and limitations of single- and multi-frame reconstruction, with the latter showing a higher performance. Finally, we provide directions for future development of this SR reconstruction.

Enhancing Lunar Reconnaissance Orbiter Images via Multi-Frame Super Resolution for Future Robotic Space Missions

This paper presents a novel application of a Multi-frame Super Resolution (MFSR) method for lunar surface imagery called Lunar HighRes-net (L-HRN). In this work, we adapted and used NASA's Lunar Reconnaissance Orbiter (LRO) image database to train the Deep Learning architecture for image super resolution. Additionally, we also gathered an artificial image dataset from our virtual Moon to improve the amount of input data in the neural network training process. The network's architecture follows a standard MFSR algorithm that was enhanced for this specific use case. The proposed MFSR method has been evaluated using the well-known peak signal-to-noise ratio (PSNR) metric against other generic super-resolution methods of the state of the art. This work aims to improve environmental knowledge about the lunar surface to enhance future autonomous robots capabilities on the surface of the Moon.