Video Deblurring Research Articles

Face deblurring using dual camera fusion on mobile phones

Motion blur of fast-moving subjects is a longstanding problem in photography and very common on mobile phones due to limited light collection efficiency, particularly in low-light conditions. While we have witnessed great progress in image deblurring in recent years, most methods require significant computational power and have limitations in processing high-resolution photos with severe local motions. To this end, we develop a novel face deblurring system based on the dual camera fusion technique for mobile phones. The system detects subject motion to dynamically enable a reference camera, e.g., ultrawide angle camera commonly available on recent premium phones, and captures an auxiliary photo with faster shutter settings. While the main shot is low noise but blurry (Figure 1(a)), the reference shot is sharp but noisy (Figure 1(b)). We learn ML models to align and fuse these two shots and output a clear photo without motion blur (Figure 1(c)). Our algorithm runs efficiently on Google Pixel 6, which takes 463 ms overhead per shot. Our experiments demonstrate the advantage and robustness of our system against alternative single-image, multi-frame, face-specific, and video deblurring algorithms as well as commercial products. To the best of our knowledge, our work is the first mobile solution for face motion deblurring that works reliably and robustly over thousands of images in diverse motion and lighting conditions.

Deep Recurrent Neural Network with Multi-Scale Bi-directional Propagation for Video Deblurring

The success of the state-of-the-art video deblurring methods stems mainly from implicit or explicit estimation of alignment among the adjacent frames for latent video restoration. However, due to the influence of the blur effect, estimating the alignment information from the blurry adjacent frames is not a trivial task. Inaccurate estimations will interfere the following frame restoration. Instead of estimating alignment information, we propose a simple and effective deep Recurrent Neural Network with Multi-scale Bi-directional Propagation (RNN-MBP) to effectively propagate and gather the information from unaligned neighboring frames for better video deblurring. Specifically, we build a Multi-scale Bi-directional Propagation (MBP) module with two U-Net RNN cells which can directly exploit the inter-frame information from unaligned neighboring hidden states by integrating them in different scales. Moreover, to better evaluate the proposed algorithm and existing state-of-the-art methods on real-world blurry scenes, we also create a Real-World Blurry Video Dataset (RBVD) by a well-designed Digital Video Acquisition System (DVAS) and use it as the training and evaluation dataset. Extensive experimental results demonstrate that the proposed RBVD dataset effectively improve the performance of existing algorithms on real-world blurry videos, and the proposed algorithm performs favorably against the state-of-the-art methods on three typical benchmarks. The code is available at https://github.com/XJTU-CVLAB-LOWLEVEL/RNN-MBP.

Film and Video Quality Optimization Using Attention Mechanism-Embedded Lightweight Neural Network Model.

In filming, the collected video may be blurred due to camera shake and object movement, causing the target edge to be unclear or deforming the targets. In order to solve these problems and deeply optimize the quality of movie videos, this work proposes a video deblurring (VD) algorithm based on neural network (NN) model and attention mechanism (AM). Based on the scale recurrent network, Haar planar wavelet transform (WT) is introduced to preprocess the video image and to deblur the video image in the wavelet domain. Additionally, the spatial and channel AMs are fused into the overall network framework to improve the feature expression ability. Further, the residual inception spatial-channel attention (RISCA) mechanism is introduced to extract the multiscale feature information from video images. Meanwhile, skip spatial-channel attention (SSCA) accelerates the network training time to achieve a better VD effect. Finally, relevant experiments are designed, factoring in peak signal-to-noise ratio (PSNR) and structural similarity (SSI). The experimental findings corroborate that the proposed Haar and attention video deblurring (HAVD) outperforms multisize network Haar (MSNH) in PSNR and structural similarity (SSIM), improved by 0.10 dB and 0.005, respectively. Therefore, embedding the dual AMs can improve the model performance and optimize the video quality. This work provides technical support for solving the video distortion problems.

Multi-Attention Convolutional Neural Network for Video Deblurring

Video deblurring, which aims at restoring the sharp video from blurry video, is drawing increasing attention in the field of computer vision. In this paper, a method called Multi-Attention Convolutional Neural Network (MACNN) consisting of the temporal-spatial attention module, the frame channel attention module, and the feature extraction-reconstruction module is proposed. First, we use the temporal-spatial attention module and the frame channel attention module to capture features with temporal and spatial information existing across neighboring frames. Then, these captured features are fused and reconstructed to restore the sharp frame. Last but not least, we train MACNN together with a content loss and a perceptual loss in an end-to-end manner to recover realistic video details. Both quantitative and qualitative evaluation results on standard benchmarks demonstrate the proposed MACNN is superior to the state-of-the-art methods in terms of accuracy, efficiency, and visual effect.

Formula omitted] structure-prior assisted blur-intensity aware efficient video deblurring

In this paper, we propose a L0 Structure-Prior Assisted Blur-intensity Aware method for efficient video deblurring. To efficiently generate sharp frames, we utilize separate streams to estimate content and structure information, both two streams effectively extract features under dedicated constraints using light-weight architecture. In content estimation, to better extract features from frames with non-uniform blur distribution, we introduce a blur-intensity detection module to generate a blur-intensity mask. The mask reflects region-wise blur degrees similar to an attention map and provides preliminary information to the content decoder. For structure estimation, to generate frames with less structural distortion, we utilize L0 smoothed frames as supervision to estimate structure information. A pyramid module with a deformable convolution layer in the last scale is designed in this stream. The estimated content and structure information are then merged with the frame synthesis module to generate perceptually favorable frames. Experiments demonstrate that our method could achieve favorable results on both synthetic and real-world datasets in terms of objective metrics and perceptual quality, with fewer input frames and less time-consuming.

Video Deblurring with Deconvolution and Aggregation Networks

Video Deblurring with Deconvolution and Aggregation Networks

Geometry-Aware Deep Video Deblurring via Recurrent Feature Refinement.

Blurring in videos is a frequent phenomenon in real-world video data owing to camera shake or object movement at different scene depths. Hence, video deblurring is an ill-posed problem that requires understanding of geometric and temporal information. Traditional model-based optimization methods first define a degradation model and then solve an optimization problem to recover the latent frames with a variational model for additional external information, such as optical flow, segmentation, depth, or camera movement. Recent deep-learning-based approaches learn from numerous training pairs of blurred and clean latent frames, with the powerful representation ability of deep convolutional neural networks. Although deep models have achieved remarkable performances without the explicit model, existing deep methods do not utilize geometrical information as strong priors. Therefore, they cannot handle extreme blurring caused by large camera shake or scene depth variations. In this paper, we propose a geometry-aware deep video deblurring method via a recurrent feature refinement module that exploits optimization-based and deep-learning-based schemes. In addition to the off-the-shelf deep geometry estimation modules, we design an effective fusion module for geometrical information with deep video features. Specifically, similar to model-based optimization, our proposed module recurrently refines video features as well as geometrical information to restore more precise latent frames. To evaluate the effectiveness and generalization of our framework, we perform tests on eight baseline networks whose structures are motivated by the previous research. The experimental results show that our framework offers greater performances than the eight baselines and produces state-of-the-art performance on four video deblurring benchmark datasets.

Dual-stream spatio-temporal decoupling network for video deblurring

It is very important to obtain spatio-temporal information in video deblurring based on deep learning. The existing methods usually jointly learn the spatio-temporal information of blurred videos through single-stream networks, which inevitably limit spatio-temporal information learning and video deblurring performance of networks. Therefore, we propose a dual-stream spatio-temporal decoupling network (STDN), which can learn the spatio-temporal information of blurred videos more flexibly and efficiently with the decoupled temporal stream and spatial stream, for solving this problem. Firstly, in the temporal stream of STDN, we propose a video deblurring pipeline, that is motion compensation plus 3D CNNs, for solving the drawback of 3D CNNs that its receptive field cannot effectively cover the same but misplaced contents of different frames. Thus, the temporal stream can aggregate temporal information of frame sequences and handle inter-frame misalignments more effectively. Specifically, we design a novel deformable convolution compensation module (DCCM) to achieve motion compensation of this pipeline more accurately. Then, we develop a 3DConv module optimized by the designed temporal, spatial, and channel decoupling attention block, named the CTS, to achieve 3D CNNs of this pipeline. Secondly, we design a spatial stream in which two types of wide-activation residual modules are stacked, for learning more spatial features of the central frame to supplement the temporal stream. Finally, extensive experiments on the baseline datasets demonstrate that the proposed STDN has better performance than the latest methods. Remarkably, using the proposed temporal stream alone already can achieve competitive video deblurring performance than the existing methods.

Recurrent Video Deblurring with Blur-Invariant Motion Estimation and Pixel Volumes

For the success of video deblurring, it is essential to utilize information from neighboring frames. Most state-of-the-art video deblurring methods adopt motion compensation between video frames to aggregate information from multiple frames that can help deblur a target frame. However, the motion compensation methods adopted by previous deblurring methods are not blur-invariant, and consequently, their accuracy is limited for blurry frames with different blur amounts. To alleviate this problem, we propose two novel approaches to deblur videos by effectively aggregating information from multiple video frames. First, we present blur-invariant motion estimation learning to improve motion estimation accuracy between blurry frames. Second, for motion compensation, instead of aligning frames by warping with estimated motions, we use a pixel volume that contains candidate sharp pixels to resolve motion estimation errors. We combine these two processes to propose an effective recurrent video deblurring network that fully exploits deblurred previous frames. Experiments show that our method achieves the state-of-the-art performance both quantitatively and qualitatively compared to recent methods that use deep learning.

Learn from the past – sequentially one-to-one video deblurring network

With the growing availability of hand-held cameras in recent years, more and more images and videos are taken at any time and any place. However, they usually suffer from undesirable blur due to camera shake or object motion in the scene. In recent years, a few modern video deblurring methods are proposed and achieve impressive performance. However, they are still not suitable for practical applications as high computational cost or using future information as input. To address the issues, we propose a sequentially one-to-one video deblurring network (SOON) which can deblur effectively without any future information. It transfers both spatial and temporal information to the next frame by utilizing the recurrent architecture. In addition, we design a novel Spatio-Temporal Attention module to nudge the network to focus on the meaningful and essential features in the past. Extensive experiments demonstrate that the proposed method outperforms the state-of-the-art deblurring methods, both quantitatively and qualitatively, on various challenging real-world deblurring datasets. Moreover, as our method deblurs in an online manner and is potentially real-time, it is more suitable for practical applications.

Spatio-Temporal Learning for Video Deblurring based on Two-Stream Generative Adversarial Network

Video-deblurring has achieved excellent results by using deep learning approaches. How to capture the dynamic spatio-temporal information in the videos is crucial on deblurring. In this paper, we propose a two-stream DeblurGAN which combines a 3D stream with a 2D stream to deblur. The 3D convolution provides spatial and temporal invariance to restore the foreground of frames, while the 2D convolution is sufficient to deal with spatial features, given a relatively consistant background. Thus, our model takes advantage of the great processing power of the 3D stream to handle the foreground which usually contains more dynamical motion blur, and the advantage of the simplicity of the 2D stream to handle the mostly consistent background. We have the full advantage of combining both the 3D convolution and the 2D convolution. Then we take the two-stream model as the generator and adopt the adversarial learning. We test our model on the VideoDeblurring and GOPRO datasets, and compare with other methods which we have listed. Our method outperforms others in the Peak Signal-to-Noise Ratio (PSNR), especially shows a good performance handling the foreground with obvious motion blur.

A total variation and group sparsity-based algorithm for nuclear radiation-contaminated video restoration

ABSTRACT Nuclear radiation-contaminated video deblurring is an important issue of the robot vision system and has been widely studied. In this paper, a hybrid radiation-contaminated video frame enhancement algorithm is proposed that utilizes both intra-frame and inter-frame correlation by a two-stage strategy. In the first stage, total variation (TV) transformation are used to locate the spot areas, and then local TV is employed to restore spot areas. The preliminary deblurring result not only enhances the video frame and similar patch matching accuracy but also provides reliable estimates of filtering parameters. In the second stage, visual group technology and improved k-nearest neighbours (k-NN) method is used to select similar frames and reference patches respectively. The final enhanced video frame is obtained by a novel patch-based group sparse method. Experimental results clearly show that the proposed method outperforms other state-of-the-art methods in both quantitative evaluation indices and visual quality measurements.

Dataset and Network Structure: Towards Frames Selection for Fast Video Deblurring

Beyond the underlaying unrealistic presumptions in the existing video deblurring datasets and algorithms which presume that a naturally blurred video is fully blurred. In this work, we define a more realistic video frames averaging-based data degradation model by referring to a naturally blurred video as a partially blurred frames sequence, and use it to build REBVIDS, as a novel video deblurring dataset to close the gap between naturally blurred and synthetically blurred video training data, and to address most shortcomings of the existing datasets. We also present DeblurNet, a two phases training-based deep learning model for video deblurring, it consists of two main sub-modules; a Frame Selection Module and a Frame Deblurring Module. Compared to the recent learning-based approaches, its sub-modules have simpler network structures, with smaller number of training parameters, are easier to train and with faster inference. As naturally blurred videos are only partially blurred, the Frame Selection Module is in charge of selecting the blurred frames in a video sequence and forwarding them to the Frame Deblurring Module input, the Frame Deblurring Module in its turn will get them restored and recombine them according to the original order in a newly restored sequence beside their initially sharp neighbor frames. Extensive experimental results on several benchmarks demonstrate that DeblurNet performs favorably against the state-of-the-art, both quantitatively and qualitatively. DeblurNet proves its ability to trade between speed, computational cost and restoration quality. Besides its ability to restore video blurred frames with necessary edges and details, benefiting from its small size and its video frames selection integrated mechanism, it can speed up the inference phase by over ten times compared to existing approaches. This project dataset and code will be released soon and will be accessible through: https://github.com/nahliabdelwahed/Speed-up-video-deblurring-

Dynamic Scene Video Deblurring Using Robust Incremental Weighted Fourier Aggregation

Motion blur is an inevitable problem when shooting a video in motion, and it will cause serious degradation of images. In this letter, we propose a new robust incremental weighted Fourier aggregation algorithm for dynamic scene video deblurring. This is motivated by the fact that existing multiframe aggregation methods usually require a tedious iterative process to obtain sharp information in distant frames. We propose to perform alignment and aggregation once for each frame using the current frame and the previous frame result, which includes the sharp information of all the previous frames. Moreover, a criterion for evaluating the alignment of blurred images is proposed. Based on this, a robust aggregation approach is also proposed to help reduce artifacts caused by image patch alignment failure. Specifically, we decompose the current frame into a series of image patches, find consistent image patches in the deblurring results of the previous frame by a coarse-to-fine alignment algorithm, and aggregate them in the Fourier domain. Experiments show that our method can obtain improved deblurring results and reduce the computational complexity compared to the traditional aggregation method.

Concurrent Video Denoising and Deblurring for Dynamic Scenes

Dynamic scene video deblurring is a challenging task due to the spatially variant blur inflicted by independently moving objects and camera shakes. Recent deep learning works bypass the ill-posedness of explicitly deriving the blur kernel by learning pixel-to-pixel mappings, which is commonly enhanced by larger region awareness. This is a difficult yet simplified scenario because noise is neglected when it is omnipresent in a wide spectrum of video processing applications. Despite its relevance, the problem of concurrent noise and dynamic blur has not yet been addressed in the deep learning literature. To this end, we analyze existing state-of-the-art deblurring methods and encounter their limitations in handling non-uniform blur under strong noise conditions. Thereafter, we propose a first-to-date work that addresses blur- and noise-free frame recovery by casting the restoration problem into a multi-task learning framework. Our contribution is threefold: a) We propose R2-D4, a multi-scale encoder architecture attached to two cascaded decoders performing the restoration task in two steps. b) We design multi-scale residual dense modules, bolstered by our modulated efficient channel attention, to enhance the encoder representations via augmenting deformable convolutions to capture longer-range and object-specific context that assists blur kernel estimation under strong noise. c) We perform extensive experiments and evaluate state-of-the-art approaches on a publicly available dataset under different noise levels. The proposed method performs favorably under all noise levels while retaining a reasonably low computational and memory footprint.

Video Deblurring via Spatiotemporal Pyramid Network and Adversarial Gradient Prior

Video deblurring is to restore sharp frames from a blurry sequence. It is a challenging low-level vision task because the blur caused by camera shake, object motions and depth variations is heterogeneous in both spatial and temporal dimensions. Traditional methods usually work on a fixed spatiotemporal scale. However, the spatiotemporal scale of blurs in the video can vastly vary in the real-world situation. To address this challenge, we propose a Spatiotemporal Pyramid Network (SPN) to dynamically learn different spatiotemporal cues for video deblurring. Specifically, inside SPN, a spatiotemporal pyramid module is employed to effectively capture both spatial information and temporal information from the blurry sequence in a pyramid mode. An image reconstruction module constructs the sharp center frame through the obtained spatiotemporal information. Additionally, inspired by the statistical image prior and adversarial learning, we extend SPN and propose a Spatiotemporal Pyramid Generative Adversarial Network (SPGAN), which conducts adversarial discrimination in the gradient space. It helps the network produce more realistic sharp video frames. Experiments conducted on benchmarks demonstrate that the proposed methods achieve state-of-the-art results in terms of PSNR, SSIM and visual quality.

Recursive Neural Network for Video Deblurring

Video deblurring is still a challenging low-level vision task since spatio-temporal characteristics across both the spatial and temporal domains are difficult to model. In this article, to model the temporal information, we develop a non-local block which estimates inter-frame similarity and inter-frame difference. Specially, for modeling the spatial characteristics and restoring sharp frame details, we propose a recursive block that iteratively refines feature maps generated at the last iteration. In addition, a novel temporal loss function is introduced to ensure the temporal consistency of generated frames. Experimental results on public datasets demonstrate that our method achieves state-of-the-art performance both quantitatively and qualitatively.

Attention-based interpolation network for video deblurring

Video deblurring is a challenging low-level vision task due to variant blur artifacts caused by factors such as depth variations, high-speed movements and camera shakes. Although significant efforts have been devoted to addressing this task, two challenges of capturing temporal patterns and spatial topologies still remain. In this paper, an attention-based interframe compensation scheme is proposed to address the first challenge. The proposed scheme replaces frames in blurry sequences with newly restored frames, and estimates temporal patterns among the replaced sequence to restore the whole sequence. After each replacement, an attention block is employed to exploit dependencies among restored and blurry frames to capture stable temporal patterns. To tackle the second challenge, we propose an adaptive residual block that dynamically fuses multi-level features via learning location-specific weights. Comprehensive experimental results demonstrate that the proposed method achieves state-of-the-art performance in terms of accuracy, visual effect and model size.

Video Deblurring via Temporally and Spatially Variant Recurrent Neural Network

The camera shake and high-speed motion of objects often produce a blurry video. However, it is hard to recover sharp videos using existing single or multiple image deblurring methods, as the blur artifacts in blurry videos are both temporally and spatially varying. In this paper, we propose a temporally and spatially variant recurrent neural network for video deblurring, in which both temporally and spatially variants employ ConvGRU blocks and a weight generator to capture spatio-temporal features. Meanwhile, the proposed model is trained in an end-to-end manner, where the model input and output are set to the same number. Thus, our model does not reduce the number of frames both in training and testing stages, which is important in practical applications. Quantitative and qualitative evaluations on standard benchmark datasets demonstrate that the proposed method outperforms the current state-of-the-art methods.

Deep Video Deblurring Using Sharpness Features from Exemplars.

Video deblurring is a challenging problem as the blur in videos is usually caused by camera shake, object motion, depth variation, etc. Existing methods usually impose handcrafted image priors or use end-to-end trainable networks to solve this problem. However, using image priors usually leads to highly non-convex problems while directly using end-to-end trainable networks in a regression generates over-smoothes details in the restored images. In this paper, we explore the sharpness features from exemplars to help the blur removal and details restoration. We first estimate optical flow to explore the temporal information which can help to make full use of neighboring information. Then, we develop an encoder and decoder network and explore the sharpness features from exemplars to guide the network for better image restoration. We train the proposed algorithm in an end-to-end manner and show that using sharpness features from exemplars can help blur removal and details restoration. Both quantitative and qualitative evaluations demonstrate that our method performs favorably against state-of-the-art approaches on the benchmark video deblurring datasets and real-world images.