Multiframe Image Restoration Research Articles

SiamTrans: Zero-Shot Multi-Frame Image Restoration with Pre-trained Siamese Transformers

We propose a novel zero-shot multi-frame image restoration method for removing unwanted obstruction elements (such as rains, snow, and moire patterns) that vary in successive frames. It has three stages: transformer pre-training, zero-shot restoration, and hard patch refinement. Using the pre-trained transformers, our model is able to tell the motion difference between the true image information and the obstructing elements. For zero-shot image restoration, we design a novel model, termed SiamTrans, which is constructed by Siamese transformers, encoders, and decoders. Each transformer has a temporal attention layer and several self-attention layers, to capture both temporal and spatial information of multiple frames. Only self-supervisedly pre-trained on the denoising task, SiamTrans is tested on three different low-level vision tasks (deraining, demoireing, and desnowing). Compared with related methods, SiamTrans achieves the best performances, even outperforming those with supervised learning.

Multi-frame image restoration method for Luojia 1-01 night-light remote sensing images

ABSTRACT To address blur and noise issues caused by airlight and atmospheric scattering in nighttime imaging environment, we proposed a multi-frame image restoration method. First, the Luojia 1–01 night-light image degradation model was derived. Thereafter, the APSF (Atmospheric Point Spread Function) for night-light images was estimated. Improved dark channel prior and sparse constraint models were used to eliminate effects ofairlight and atmospheric scattering. Finally, a multi-frame sparse constraint model was established to eliminate image noise. The results show the proposed method is effective as it can reduce the image blur phenomenon, suppress image noise, and improve image quality.

Multi-frame image restoration method for novel rotating synthetic aperture imaging system

The novel rotating synthetic aperture (RSA) optical imaging system is an important development direction for future high-resolution optical remote sensing satellites in geostationary orbit. However, owing to the rotating rectangular pupil, the point spread function of the RSA system has an asymmetric spatial distribution, and the images obtained using the primary mirror from different rotation angles have nonuniform blur degradation. Moreover, platform vibration and pupil rotation have coupling effects on the RSA imaging, resulting in further radiometric and geometric quality degradation. To address these problems, the image degradation characteristics are first analyzed according to the imaging mechanism. Then, combined with the theory of mutual information, an image registration method is suggested by introducing the orientation gradient information. From this, a multi-frame image restoration model is proposed based on the directional gradient prior of the RSA system image. From the perspective of interpretation and application, when the aspect ratio is less than 3, the proposed inversion restoration method can achieve a satisfactory processing performance. This work can provide engineering application reference for the future space application of RSA imaging technology.

Split-Attention Multiframe Alignment Network for Image Restoration

Image registration (or image alignment), the problem of aligning multiple images with relative displacement, is a crucial step in many multiframe image restoration algorithms. To solve the problem that most existing image registration approaches can only align two images in one inference, we propose a split-attention multiframe alignment network (SAMANet). Pixel-level displacements between multiple images are first estimated at low-resolution scales and then refined gradually with the increase in feature resolution. To better integrate the interframe information, we present a split-attention module (SAM) and a dot-product attention module (DPAM), which can adaptively rescale the cost volume features and optical flow features according to the similarity between features from different images. The experimental results demonstrate the superiority of our SAMANet over state-of-the-art image registration methods in terms of both accuracy and robustness. To solve the “ghosting effect” caused by pixelwise registration, we designed two “ghost” removal modules: warping repetition detection module (WRDM) and attention fusion module (AFM). WRDM detects “ghost” regions during the image warping process without increasing the time complexity of the registration algorithm. AFM uses an attention mechanism to rescale the aligned images and enables the registration network and the subsequent image restoration networks to be trained jointly. To validate the strengths of the proposed approaches, we apply SAMANet, WRDM and AFM to three image/video restoration tasks. Extensive evaluations demonstrate that the proposed methods can enhance the performance of image restoration algorithms and outperform the other compared registration algorithms.

Variational Bayesian Approach to Multiframe Image Restoration.

Image restoration is a fundamental problem in the field of image processing. The key objective of image restoration is to recover clean images from images degraded by noise and blur. Recently, a family of new statistical techniques called variational Bayes (VB) has been introduced to image restoration, which enables us to automatically tune parameters that control restoration. While information from one image is often insufficient for high-quality restoration, however, current state-of-the-art methods of image restoration via VB approaches use only a single-degraded image to recover a clean image. In this paper, we propose a novel method of multiframe image restoration via a VB approach, which can achieve higher image quality while tuning parameters automatically. Given multiple degraded images, this method jointly estimates a clean image and other parameters, including an image warping parameter introduced for the use of multiple images, through Bayesian inference that we enable by making full use of VB techniques. Through various experiments, we demonstrate the effectiveness of our multiframe method by comparing it with single-frame one, and also show the advantages of our VB approach over non-VB approaches.

Multi-frame real image restoration based on double loops with alternative maximum likelihood estimation

A multi-frame blind restoration algorithm based on alternative maximum likelihood estimation using double loops is proposed to restore object images. An anisotropic regularization term is incorporated into the integral likelihood function to avoid over-smoothing of details of the restored images and improve the stability of image restoration. In order to make full use of the data relationship in the observed multi-frame images which contain the same objects with different point spread functions (PSF) in a short sequence, an iterative strategy based on double loops is employed in estimating the unknown different PSFs of the observed images. Experiments on synthetic and real turbulence-degraded images illustrate that the proposed algorithm is effective in restoring real turbulence-degraded images by using very few frames.

Restoration of turbulence degraded underwater images

The ability to image underwater is highly desired for scientific and military applications, including optical communications, submarine awareness, diver visibility, and mine detection. Underwater imaging is severely impaired by scattering and optical turbulence associated with refractive index fluctuations. This work introduces a novel approach to restoration of degraded underwater imagery based on a multi-frame correction technique developed for atmospheric distortions. The method represents synthesis of lucky-region fusion with nonlinear gain and optical flow-based image warping. The developed multiframe image restoration algorithm is tested on underwater imagery collected in a laboratory tank and in a field exercise. Reliance of image restoration on accuracy of the optical flow algorithm is revealed. The developed algorithm demonstrates significant resolution improvement of the restored image in comparison to any single frame or the mean of the underwater image sequence.

Efficient Fourier-Wavelet Super-Resolution

Super-resolution (SR) is the process of combining multiple aliased low-quality images to produce a high-resolution high-quality image. Aside from registration and fusion of low-resolution images, a key process in SR is the restoration and denoising of the fused images. We present a novel extension of the combined Fourier-wavelet deconvolution and denoising algorithm ForWarD to the multiframe SR application. Our method first uses a fast Fourier-base multiframe image restoration to produce a sharp, yet noisy estimate of the high-resolution image. Our method then applies a space-variant nonlinear wavelet thresholding that addresses the nonstationarity inherent in resolution-enhanced fused images. We describe a computationally efficient method for implementing this space-variant processing that leverages the efficiency of the fast Fourier transform (FFT) to minimize complexity. Finally, we demonstrate the effectiveness of this algorithm for regular imagery as well as in digital mammography.

Sparsity constrained regularization for multiframe image restoration

In this paper we present a new algorithm for restoring an object from multiple undersampled low-resolution (LR) images that are degraded by optical blur and additive white Gaussian noise. We formulate the multiframe superresolution problem as maximum a posteriori estimation. The prior knowledge that the object is sparse in some domain is incorporated in two ways: first we use the popular l(1) norm as the regularization operator. Second, we model wavelet coefficients of natural objects using generalized Gaussian densities. The model parameters are learned from a set of training objects, and the regularization operator is derived from these parameters. We compare the results from our algorithms with an expectation-maximization (EM) algorithm for l(1) norm minimization and also with the linear minimum-mean-squared error (LMMSE) estimator. Using only eight 4 x 4 pixel downsampled LR images the reconstruction errors of object estimates obtained from our algorithm are 5.5% smaller than by the EM method and 14.3% smaller than by the LMMSE method.

Optimal Registration Of Aliased Images Using Variable Projection With Applications To Super-Resolution

Accurate registration of images is the most important and challenging aspect of multiframe image restoration problems such as super-resolution. The accuracy of super-resolution algorithms is quite often limited by the ability to register a set of low-resolution images. The main challenge in registering such images is the presence of aliasing. In this paper, we analyse the problem of jointly registering a set of aliased images and its relationship to super-resolution. We describe a statistically optimal approach to multiframe registration which exploits the concept of variable projections to achieve very efficient algorithms. Finally, we demonstrate how the proposed algorithm offers accurate estimation under various conditions when standard approaches fail to provide sufficient accuracy for super-resolution.