Motion Deblurring Research Articles

CrossZoom: Simultaneous Motion Deblurring and Event Super-Resolving.

Even though the collaboration between traditional and neuromorphic event cameras brings prosperity to frame-event based vision applications, the performance is still confined by the resolution gap crossing two modalities in both spatial and temporal domains. This paper is devoted to bridging the gap by increasing the temporal resolution for images, i.e., motion deblurring, and the spatial resolution for events, i.e., event super-resolving, respectively. To this end, we introduce CrossZoom, a novel unified neural Network (CZ-Net) to jointly recover sharp latent sequences within the exposure period of a blurry input and the corresponding High-Resolution (HR) events. Specifically, we present a multi-scale blur-event fusion architecture that leverages the scale-variant properties and effectively fuses cross-modal information to achieve cross-enhancement. Attention-based adaptive enhancement and cross-interaction prediction modules are devised to alleviate the distortions inherent in Low-Resolution (LR) events and enhance the final results through the prior blur-event complementary information. Furthermore, we propose a new dataset containing HR sharp-blurry images and the corresponding HR-LR event streams to facilitate future research. Extensive qualitative and quantitative experiments on synthetic and real-world datasets demonstrate the effectiveness and robustness of the proposed method.

Single Image Integrated Deblurring Algorithm in Non-Uniform Environment

Motion blurriness in an image caused by camera shake during exposure is unavoidable. It could lead to information loss and degradation in the image quality. Therefore, many researchers are dedicated to developing image deblurring techniques to recover clear images from blurred images. During the deblurring process, structural edges in images play a vital part in estimating the blur kernels. For images with rich textures, fine-scale edges become more apparent. This will cause vagueness in the image’s structural edges and affect the accuracy of the kernel estimation process. In this study, we propose a single-image motion deblurring by kernel estimation method combined with L0-Regularized Intensity and Gradient Prior, and enhanced Scale Aware Smoothing methods. Two types of non-uniform datasets are used, which are real and synthetic. While synthetic datasets are utilized to assess the consistency in performance across real and synthetic images, real datasets are used to portray the level of detail and variation of actual blurred images. The dataset is divided into five categories (people, nature, manmade, text, and night light). Two image quality metrics were selected: full-reference assessment, including learned perceptual image patch similarity (LPIPS), peak signal-to-noise ratio (PSNR), and structural similarity indexing method (SSIM) for synthetic datasets, and no-reference assessments, including blind/reference-less image spatial quality (BRISQUE), natural image quality evaluator (NIQE) and perception-based image quality evaluator (PIQE) for real datasets. According to the findings, the fusion method performs best in the text category, followed by manmade and in nature, then night light and poor in people. The proposed method not only removes fine-scale edges and preserves the boundary sharpness, but also improved the estimated blur kernel.

Motion-Oriented Hybrid Spiking Neural Networks for Event-Based Motion Deblurring

Motion-Oriented Hybrid Spiking Neural Networks for Event-Based Motion Deblurring

Motion Deblurring via Spatial-Temporal Collaboration of Frames and Events

Motion deblurring can be advanced by exploiting informative features from supplementary sensors such as event cameras, which can capture rich motion information asynchronously with high temporal resolution. Existing event-based motion deblurring methods neither consider the modality redundancy in spatial fusion nor temporal cooperation between events and frames. To tackle these limitations, a novel spatial-temporal collaboration network (STCNet) is proposed for event-based motion deblurring. Firstly, we propose a differential-modality based cross-modal calibration strategy to suppress redundancy for complementarity enhancement, and then bimodal spatial fusion is achieved with an elaborate cross-modal co-attention mechanism to weight the contributions of them for importance balance. Besides, we present a frame-event mutual spatio-temporal attention scheme to alleviate the errors of relying only on frames to compute cross-temporal similarities when the motion blur is significant, and then the spatio-temporal features from both frames and events are aggregated with the custom cross-temporal coordinate attention. Extensive experiments on both synthetic and real-world datasets demonstrate that our method achieves state-of-the-art performance. Project website: https://github.com/wyang-vis/STCNet.

DeblurSR: Event-Based Motion Deblurring under the Spiking Representation

We present DeblurSR, a novel motion deblurring approach that converts a blurry image into a sharp video. DeblurSR utilizes event data to compensate for motion ambiguities and exploits the spiking representation to parameterize the sharp output video as a mapping from time to intensity. Our key contribution, the Spiking Representation (SR), is inspired by the neuromorphic principles determining how biological neurons communicate with each other in living organisms. We discuss why the spikes can represent sharp edges and how the spiking parameters are interpreted from the neuromorphic perspective. DeblurSR has higher output quality and requires fewer computing resources than state-of-the-art event-based motion deblurring methods. We additionally show that our approach easily extends to video super-resolution when combined with recent advances in implicit neural representation.

Improving Diffusion-Based Image Restoration with Error Contraction and Error Correction

Generative diffusion prior captured from the off-the-shelf denoising diffusion generative model has recently attained significant interest. However, several attempts have been made to adopt diffusion models to noisy inverse problems either fail to achieve satisfactory results or require a few thousand iterations to achieve high-quality reconstructions. In this work, we propose a diffusion-based image restoration with error contraction and error correction (DiffECC) method. Two strategies are introduced to contract the restoration error in the posterior sampling process. First, we combine existing CNN-based approaches with diffusion models to ensure data consistency from the beginning. Second, to amplify the error contraction effects of the noise, a restart sampling algorithm is designed. In the error correction strategy, the estimation-correction idea is proposed on both the data term and the prior term. Solving them iteratively within the diffusion sampling framework leads to superior image generation results. Experimental results for image restoration tasks such as super-resolution (SR), Gaussian deblurring, and motion deblurring demonstrate that our approach can reconstruct high-quality images compared with state-of-the-art sampling-based diffusion models.

Image Restoration via Frequency Selection.

Image restoration aims to reconstruct the latent sharp image from its corrupted counterpart. Besides dealing with this long-standing task in the spatial domain, a few approaches seek solutions in the frequency domain by considering the large discrepancy between spectra of sharp/degraded image pairs. However, these algorithms commonly utilize transformation tools, e.g., wavelet transform, to split features into several frequency parts, which is not flexible enough to select the most informative frequency component to recover. In this paper, we exploit a multi-branch and content-aware module to decompose features into separate frequency subbands dynamically and locally, and then accentuate the useful ones via channel-wise attention weights. In addition, to handle large-scale degradation blurs, we propose an extremely simple decoupling and modulation module to enlarge the receptive field via global and window-based average pooling. Furthermore, we merge the paradigm of multi-stage networks into a single U-shaped network to pursue multi-scale receptive fields and improve efficiency. Finally, integrating the above designs into a convolutional backbone, the proposed Frequency Selection Network (FSNet) performs favorably against state-of-the-art algorithms on 20 different benchmark datasets for 6 representative image restoration tasks, including single-image defocus deblurring, image dehazing, image motion deblurring, image desnowing, image deraining, and image denoising.

Event-Based Shutter Unrolling and Motion Deblurring in Dynamic Scenes

Event-Based Shutter Unrolling and Motion Deblurring in Dynamic Scenes

Motion Deblur by Learning Residual From Events

Motion Deblur by Learning Residual From Events

An End-to-end Network for Rotary Motion Deblurring in the Polar Coordinate System

An End-to-end Network for Rotary Motion Deblurring in the Polar Coordinate System

Underwater Motion Deblurring Based on Cascaded Attention Mechanism

Underwater Motion Deblurring Based on Cascaded Attention Mechanism

Image Deblurring by Exploring In-Depth Properties of Transformer.

Image deblurring continues to achieve impressive performance with the development of generative models. Nonetheless, there still remains a displeasing problem if one wants to improve perceptual quality and quantitative scores of recovered image at the same time. In this study, drawing inspiration from the research of transformer properties, we introduce the pretrained transformers to address this problem. In particular, we leverage deep features extracted from a pretrained vision transformer (ViT) to encourage recovered images to be sharp without sacrificing the performance measured by the quantitative metrics. The pretrained transformer can capture the global topological relations (i.e., self-similarity) of image, and we observe that the captured topological relationships about the sharp image will change when blur occurs. By comparing the transformer features between recovered image and target one, the pretrained transformer provides high-resolution blur-sensitive semantic information, which is critical in measuring the sharpness of the deblurred image. On the basis of the advantages, we present two types of novel perceptual losses to guide image deblurring. One regards the features as vectors and computes the discrepancy between representations extracted from recovered image and target one in Euclidean space. The other type considers the features extracted from an image as a distribution and compares the distribution discrepancy between recovered image and target one. We demonstrate the effectiveness of transformer properties in improving the perceptual quality while not sacrificing the quantitative scores peak signal-to-noise ratio (PSNR) over the most competitive models, such as Uformer, Restormer, and NAFNet, on defocus deblurring and motion deblurring tasks. The code is available at https://github. com/erfect2020/TransformerPerceptualLoss.

An Efficient Multiscale Spatial Rearrangement MLP Architecture for Image Restoration.

The effective use of long-range information can yield improved network performance, which is very important for image restoration. Although local window-based models have linear complexity and can be feasibly applied to process high-resolution images, a single-scale window has a limited receptive field and is less efficient for encoding long-range context information. To address this issue, this paper presents a single-stage multiscale spatial rearrangement multilayer perceptron (MSSR-MLP) architecture that can obtain information at different scales within a local window. Specifically, we propose a simple and efficient spatial rearrangement module (SRM) that moves information outside the local window to the inside of the local window so that long-range dependencies can be modeled using only a window-based fully connected (FC) layer. The SRM can extend the local receptive field of a window-based FC layer without introducing additional parameters and FLOPs. Utilizing several spatial rearrangement modules with different step sizes, we design an efficient multiscale spatial rearrangement MLP architecture for image restoration. This design aggregates multiscale information to achieve improved restoration quality while maintaining a low computational cost. Extensive experiments conducted on several image restoration tasks demonstrate the efficiency and effectiveness of our method. For example, it requires only ~4.3% of the FLOPs needed by SwinIR for Gaussian gray image denoising, ~13.9% of the FLOPs needed by C2 PNet for single-image dehazing and ~18.9% of the FLOPs needed by MAXIM for single-image motion deblurring but achieves better performance on each of these restoration tasks.

Fourier Transform-Based U-Shaped Network for Single Image Motion Deblrring

Fourier Transform-Based U-Shaped Network for Single Image Motion Deblrring

Joint learning of motion deblurring and defocus deblurring networks with a real-world dataset

When recovering the sharp image from a blurry observation, moving objects, e.g., people and vehicles, usually attract more perceptual attention. However, existing motion deblurring methods primarily focus on removing the global camera motion blur, while neglecting the object motion blur. In this paper, we propose a joint learning framework for motion deblurring and defocus deblurring (JLMD), where a motion deblurring network can be trained for removing object motion blur, and a defocus deblurring network can be also obtained as a byproduct. Especially, we propose to circumvent the hardship in obtaining all-sharp ground truth for object motion blurry images by leveraging the complementary nature between object motion blur and defocus blur. The JLMD framework incorporates three key components. Firstly, a mask generation module is deployed to predict the boundary of motion blur. Secondly, a mutual supervision mechanism is elaborated, which enables the defocus blurry image and the motion blurry image to serve as the ground truth for each other. Thirdly, a content preservation loss is designed to encourage non-blurry regions to stay unchanged. Furthermore, we introduce a Defocus and Motion Deblurring Dataset (DMDD), which is the first real-world dataset specifically dedicated to object motion blurry images. Extensive experiments validate that our method is more effective in removing object motion blur than state-of-the-art methods, while it is at least comparable or superior in removing global camera motion blur and defocus blur. The source code and dataset are available at https://github.com/liyucs/JLMD.

A Motion Deblurring Network for Enhancing UAV Image Quality in Bridge Inspection

Unmanned aerial vehicles (UAVs) have been increasingly utilized for facility safety inspections due to their superior safety, cost effectiveness, and inspection accuracy compared to traditional manpower-based methods. High-resolution images captured by UAVs directly contribute to identifying and quantifying structural defects on facility exteriors, making image quality a critical factor in achieving accurate results. However, motion blur induced by external factors such as vibration, low light conditions, and wind during UAV operation significantly degrades image quality, leading to inaccurate defect detection and quantification. To address this issue, this research proposes a deblurring network using a Generative Adversarial Network (GAN) to eliminate the motion blur effect in UAV images. The GAN-based motion deblur network represents an image inpainting method that leverages generative models to correct blurry artifacts, thereby generating clear images. Unlike previous studies, this proposed approach incorporates deblur and blur learning modules to realistically generate blur images required for training the generative models. The UAV images processed using the motion deblur network are evaluated using a quality assessment method based on local blur map and other well-known image quality assessment (IQA) metrics. Moreover, in the experiment of crack detection utilizing the object detection system, improved detection results are observed when using enhanced images. Overall, this research contributes to improving the quality and accuracy of facility safety inspections conducted with UAV-based inspections by effectively addressing the challenges associated with motion blur effects in UAV-captured images.

Deep coded exposure: end-to-end co-optimization of flutter shutter and deblurring processing for general motion blur removal

Coded exposure photography is a promising computational imaging technique capable of addressing motion blur much better than using a conventional camera, via tailoring invertible blur kernels. However, existing methods suffer from restrictive assumptions, complicated preprocessing, and inferior performance. To address these issues, we proposed an end-to-end framework to handle general motion blurs with a unified deep neural network, and optimize the shutter’s encoding pattern together with the deblurring processing to achieve high-quality sharp images. The framework incorporates a learnable flutter shutter sequence to capture coded exposure snapshots and a learning-based deblurring network to restore the sharp images from the blurry inputs. By co-optimizing the encoding and the deblurring modules jointly, our approach avoids exhaustively searching for encoding sequences and achieves an optimal overall deblurring performance. Compared with existing coded exposure based motion deblurring methods, the proposed framework eliminates tedious preprocessing steps such as foreground segmentation and blur kernel estimation, and extends coded exposure deblurring to more general blind and nonuniform cases. Both simulation and real-data experiments demonstrate the superior performance and flexibility of the proposed method.

Plant Image Classification with Nonlinear Motion Deblurring Based on Deep Learning

Despite the significant number of classification studies conducted using plant images, studies on nonlinear motion blur are limited. In general, motion blur results from movements of the hands of a person holding a camera for capturing plant images, or when the plant moves owing to wind while the camera is stationary. When these two cases occur simultaneously, nonlinear motion blur is highly probable. Therefore, a novel deep learning-based classification method applied on plant images with various nonlinear motion blurs is proposed. In addition, this study proposes a generative adversarial network-based method to reduce nonlinear motion blur; accordingly, the method is explored for improving classification performance. Herein, experiments are conducted using a self-collected visible light images dataset. Evidently, nonlinear motion deblurring results in a structural similarity index measure (SSIM) of 73.1 and a peak signal-to-noise ratio (PSNR) of 21.55, whereas plant classification results in a top-1 accuracy of 90.09% and F1-score of 84.84%. In addition, the experiment conducted using two types of open datasets resulted in PSNRs of 20.84 and 21.02 and SSIMs of 72.96 and 72.86, respectively. The proposed method of plant classification results in top-1 accuracies of 89.79% and 82.21% and F1-scores of 84% and 76.52%, respectively. Thus, the proposed network produces higher accuracies than the existing state-of-the-art methods.

Multi-patch multi-scale model for motion deblurring with high-frequency information

Nowadays deep learning-based models have made encouraging accomplishments in the field of image motion deblurring, but this type of algorithms does not effectively use the known prior knowledge and the physical model of motion blur. So as to settle the above disputes, this paper suggests a multi-patch multi-scale input model that fuses high-frequency information, structural self-similarity (SSS), multi-scale channel spatial attention (MSCSA), and superpixel gradient loss for motion deblurring. The multi-patch multi-scale input model can fully extract the salient information, scale information, and high-frequency information of the image to restore the sharp image. The SSS module can take full advantage of the prior information of the structure self-similarity of the image. And the MSCSA module dynamically and selectively enhances relevant features and suppresses irrelevant features by fusing spatial, scale, and channel characteristics. Moreover, this paper advises a superpixel gradient loss function, which can remove motion blur and restore sharp images more effectively. And to increase the diversity of scenes, for the first time, we propose a motion blur dataset (Underwater-Blur) specifically for underwater scenes. Massive comparative experiments conducted on three public datasets of GoPro, RealBlur, RWBI, and the Underwater-Blur dataset prove that our model outperforms the state-of-the-art in terms of comprehensive performance.

Abnormal event detection model using an improved ResNet101 in context aware surveillance system

AbstractSurveillance system plays a significant role for achieving security monitoring in the place of crowd areas. Offline monitoring of these crowd activity is quite challenging because it requires huge number of human resources for attaining efficient tracking. For shortcoming these issue automated and intelligent based system must be developed for efficiently monitor crowd and detect abnormal activity. However the existing methods faces issues like irrelevant features, high cost and process complexity. In this current research context aware surveillance‐system utilising hybrid ResNet101‐ANN is developed for effective abnormal activity detection. For this proposed approach video acquired from surveillance camera is considered as input. Then, acquired video is segmented into multiple frames. After that pre‐processing techniques such as denoising using mean filter, motion deblurring, contrast enhancement using Histogram Equalisation and canny edge detection is applied in this segmented frames. Further, the pre‐processed frame is fetched into hybrid ResNet101‐ANN classifier for abnormal event classification. Here, ResNet101 is used for extracting the features from the frames and Artificial neural network which replaces the fully connected layer of ResNet101 us used to detect the abnormal activity. If once abnormal‐events detected the context aware services generate alert to the user for preventing abnormal‐activities. Accuracy, precision, recall, and error values reached for the proposed‐model on simulation were 0.98, 0.98, 0.98 and 0.017 respectively. Using this proposed model effective crowd monitoring and abnormal activity detection can be achieved.