Defocus Deblurring Research Articles

Learning dual-pixel alignment for defocus deblurring

Learning dual-pixel alignment for defocus deblurring

Deep Single Image Defocus Deblurring via Gaussian Kernel Mixture Learning.

This paper proposes an end-to-end deep learning approach for removing defocus blur from a single defocused image. Defocus blur is a common issue in digital photography that poses a challenge due to its spatially-varying and large blurring effect. The proposed approach addresses this challenge by employing a pixel-wise Gaussian kernel mixture (GKM) model to accurately yet compactly parameterize spatially-varying defocus point spread functions (PSFs), which is motivated by the isotropy in defocus PSFs. We further propose a grouped GKM (GGKM) model that decouples the coefficients in GKM, so as to improve the modeling accuracy with an economic manner. Afterward, a deep neural network called GGKMNet is then developed by unrolling a fixed-point iteration process of GGKM-based image deblurring, which avoids the efficiency issues in existing unrolling DNNs. Using a lightweight scale-recurrent architecture with a coarse-to-fine estimation scheme to predict the coefficients in GGKM, the GGKMNet can efficiently recover an all-in-focus image from a defocused one. Such advantages are demonstrated with extensive experiments on five benchmark datasets, where the GGKMNet outperforms existing defocus deblurring methods in restoration quality, as well as showing advantages in terms of model complexity and computational efficiency.

Defocus Blur Detection Attack via Mutual-Referenced Feature Transfer.

Benefiting from deep learning, defocus blur detection (DBD) has made prominent progress. Existing DBD methods generally study multiscale and multilevel features to improve performance. In this article, from a different perspective, we explore to generate confrontational images to attack DBD network. Based on the observation that defocus area and focus region in an image can provide mutual feature reference to help improve the quality of the confrontational image, we propose a novel mutual-referenced attack framework. Firstly, we design a divide-and-conquer perturbation image generation model, where the focus region attack image and defocus area attack image are generated respectively. Then, we integrate mutual-referenced feature transfer (MRFT) models to improve attack performance. Comprehensive experiments are provided to verify the effectiveness of our method. Moreover, related applications of our study are presented, e.g., sample augmentation to improve DBD and paired sample generation to boost defocus deblurring.

Prior and Prediction Inverse Kernel Transformer for Single Image Defocus Deblurring

Defocus blur, due to spatially-varying sizes and shapes, is hard to remove. Existing methods either are unable to effectively handle irregular defocus blur or fail to generalize well on other datasets. In this work, we propose a divide-and-conquer approach to tackling this issue, which gives rise to a novel end-to-end deep learning method, called prior-and-prediction inverse kernel transformer (P2IKT), for single image defocus deblurring. Since most defocus blur can be approximated as Gaussian blur or its variants, we construct an inverse Gaussian kernel module in our method to enhance its generalization ability. At the same time, an inverse kernel prediction module is introduced in order to flexibly address the irregular blur that cannot be approximated by Gaussian blur. We further design a scale recurrent transformer, which estimates mixing coefficients for adaptively combining the results from the two modules and runs the scale recurrent ``coarse-to-fine" procedure for progressive defocus deblurring. Extensive experimental results demonstrate that our P2IKT outperforms previous methods in terms of PSNR on multiple defocus deblurring datasets.

Image Restoration via Frequency Selection.

Image restoration aims to reconstruct the latent sharp image from its corrupted counterpart. Besides dealing with this long-standing task in the spatial domain, a few approaches seek solutions in the frequency domain by considering the large discrepancy between spectra of sharp/degraded image pairs. However, these algorithms commonly utilize transformation tools, e.g., wavelet transform, to split features into several frequency parts, which is not flexible enough to select the most informative frequency component to recover. In this paper, we exploit a multi-branch and content-aware module to decompose features into separate frequency subbands dynamically and locally, and then accentuate the useful ones via channel-wise attention weights. In addition, to handle large-scale degradation blurs, we propose an extremely simple decoupling and modulation module to enlarge the receptive field via global and window-based average pooling. Furthermore, we merge the paradigm of multi-stage networks into a single U-shaped network to pursue multi-scale receptive fields and improve efficiency. Finally, integrating the above designs into a convolutional backbone, the proposed Frequency Selection Network (FSNet) performs favorably against state-of-the-art algorithms on 20 different benchmark datasets for 6 representative image restoration tasks, including single-image defocus deblurring, image dehazing, image motion deblurring, image desnowing, image deraining, and image denoising.

Image Deblurring by Exploring In-Depth Properties of Transformer.

Image deblurring continues to achieve impressive performance with the development of generative models. Nonetheless, there still remains a displeasing problem if one wants to improve perceptual quality and quantitative scores of recovered image at the same time. In this study, drawing inspiration from the research of transformer properties, we introduce the pretrained transformers to address this problem. In particular, we leverage deep features extracted from a pretrained vision transformer (ViT) to encourage recovered images to be sharp without sacrificing the performance measured by the quantitative metrics. The pretrained transformer can capture the global topological relations (i.e., self-similarity) of image, and we observe that the captured topological relationships about the sharp image will change when blur occurs. By comparing the transformer features between recovered image and target one, the pretrained transformer provides high-resolution blur-sensitive semantic information, which is critical in measuring the sharpness of the deblurred image. On the basis of the advantages, we present two types of novel perceptual losses to guide image deblurring. One regards the features as vectors and computes the discrepancy between representations extracted from recovered image and target one in Euclidean space. The other type considers the features extracted from an image as a distribution and compares the distribution discrepancy between recovered image and target one. We demonstrate the effectiveness of transformer properties in improving the perceptual quality while not sacrificing the quantitative scores peak signal-to-noise ratio (PSNR) over the most competitive models, such as Uformer, Restormer, and NAFNet, on defocus deblurring and motion deblurring tasks. The code is available at https://github. com/erfect2020/TransformerPerceptualLoss.

Joint Defocus Deblurring and Superresolution Learning Network for Autonomous Driving

Joint Defocus Deblurring and Superresolution Learning Network for Autonomous Driving

Decoupling Image Deblurring Into Twofold: A Hierarchical Model for Defocus Deblurring

Decoupling Image Deblurring Into Twofold: A Hierarchical Model for Defocus Deblurring

Attacking Defocus Detection With Blur-Aware Transformation for Defocus Deblurring

Attacking Defocus Detection With Blur-Aware Transformation for Defocus Deblurring

WIG-Net: Wavelet-Based Defocus Deblurring with IFA and GCN

Although the existing deblurring methods for defocused images are capable of approximately recovering clear images, they still exhibit certain limitations, such as ringing artifacts and remaining blur. Along these lines, in this work, a novel deep-learning-based method for image defocus deblurring was proposed, which can be applied to medical images, traffic monitoring, and other fields. The developed approach is equipped with wavelet transform, an iterative filter adaptive module, and graph neural network and was specifically designed for handling defocused blur. Our network exhibits excellent properties in preserving the original information during the restoration of clear images, thereby enhancing its capability to spatially address varying blurriness and improving the quality of deblurring. From the acquired experimental results, the superiority of the introduced method in the context of image defocus deblurring compared to the majority of the existing algorithms was clearly demonstrated.

Joint learning of motion deblurring and defocus deblurring networks with a real-world dataset

When recovering the sharp image from a blurry observation, moving objects, e.g., people and vehicles, usually attract more perceptual attention. However, existing motion deblurring methods primarily focus on removing the global camera motion blur, while neglecting the object motion blur. In this paper, we propose a joint learning framework for motion deblurring and defocus deblurring (JLMD), where a motion deblurring network can be trained for removing object motion blur, and a defocus deblurring network can be also obtained as a byproduct. Especially, we propose to circumvent the hardship in obtaining all-sharp ground truth for object motion blurry images by leveraging the complementary nature between object motion blur and defocus blur. The JLMD framework incorporates three key components. Firstly, a mask generation module is deployed to predict the boundary of motion blur. Secondly, a mutual supervision mechanism is elaborated, which enables the defocus blurry image and the motion blurry image to serve as the ground truth for each other. Thirdly, a content preservation loss is designed to encourage non-blurry regions to stay unchanged. Furthermore, we introduce a Defocus and Motion Deblurring Dataset (DMDD), which is the first real-world dataset specifically dedicated to object motion blurry images. Extensive experiments validate that our method is more effective in removing object motion blur than state-of-the-art methods, while it is at least comparable or superior in removing global camera motion blur and defocus blur. The source code and dataset are available at https://github.com/liyucs/JLMD.

Learnable Blur Kernel for Single-Image Defocus Deblurring in the Wild

Recent research showed that the dual-pixel sensor has made great progress in defocus map estimation and image defocus deblurring. However, extracting real-time dual-pixel views is troublesome and complex in algorithm deployment. Moreover, the deblurred image generated by the defocus deblurring network lacks high-frequency details, which is unsatisfactory in human perception. To overcome this issue, we propose a novel defocus deblurring method that uses the guidance of the defocus map to implement image deblurring. The proposed method consists of a learnable blur kernel to estimate the defocus map, which is an unsupervised method, and a single-image defocus deblurring generative adversarial network (DefocusGAN) for the first time. The proposed network can learn the deblurring of different regions and recover realistic details. We propose a defocus adversarial loss to guide this training process. Competitive experimental results confirm that with a learnable blur kernel, the generated defocus map can achieve results comparable to supervised methods. In the single-image defocus deblurring task, the proposed method achieves state-of-the-art results, especially significant improvements in perceptual quality, where PSNR reaches 25.56 dB and LPIPS reaches 0.111.

Learning Single Image Defocus Deblurring with Misaligned Training Pairs

By adopting popular pixel-wise loss, existing methods for defocus deblurring heavily rely on well aligned training image pairs. Although training pairs of ground-truth and blurry images are carefully collected, e.g., DPDD dataset, misalignment is inevitable between training pairs, making existing methods possibly suffer from deformation artifacts. In this paper, we propose a joint deblurring and reblurring learning (JDRL) framework for single image defocus deblurring with misaligned training pairs. Generally, JDRL consists of a deblurring module and a spatially invariant reblurring module, by which deblurred result can be adaptively supervised by ground-truth image to recover sharp textures while maintaining spatial consistency with the blurry image. First, in the deblurring module, a bi-directional optical flow-based deformation is introduced to tolerate spatial misalignment between deblurred and ground-truth images. Second, in the reblurring module, deblurred result is reblurred to be spatially aligned with blurry image, by predicting a set of isotropic blur kernels and weighting maps. Moreover, we establish a new single image defocus deblurring (SDD) dataset, further validating our JDRL and also benefiting future research. Our JDRL can be applied to boost defocus deblurring networks in terms of both quantitative metrics and visual quality on DPDD, RealDOF and our SDD datasets.

Learning Enriched Features for Fast Image Restoration and Enhancement.

Given a degraded input image, image restoration aims to recover the missing high-quality image content. Numerous applications demand effective image restoration, e.g., computational photography, surveillance, autonomous vehicles, and remote sensing. Significant advances in image restoration have been made in recent years, dominated by convolutional neural networks (CNNs). The widely-used CNN-based methods typically operate either on full-resolution or on progressively low-resolution representations. In the former case, spatial details are preserved but the contextual information cannot be precisely encoded. In the latter case, generated outputs are semantically reliable but spatially less accurate. This paper presents a new architecture with a holistic goal of maintaining spatially-precise high-resolution representations through the entire network, and receiving complementary contextual information from the low-resolution representations. The core of our approach is a multi-scale residual block containing the following key elements: (a) parallel multi-resolution convolution streams for extracting multi-scale features, (b) information exchange across the multi-resolution streams, (c) non-local attention mechanism for capturing contextual information, and (d) attention based multi-scale feature aggregation. Our approach learns an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details. Extensive experiments on six real image benchmark datasets demonstrate that our method, named as MIRNet-v2, achieves state-of-the-art results for a variety of image processing tasks, including defocus deblurring, image denoising, super-resolution, and image enhancement. The source code and pre-trained models are available at https://github.com/swz30/MIRNetv2.

BaMBNet: A Blur-Aware Multi-Branch Network for Dual-Pixel Defocus Deblurring

Reducing the defocus blur that arises from the finite aperture size and short exposure time is an essential problem in computational photography. It is very challenging because the blur kernel is spatially varying and difficult to estimate by traditional methods. Due to its great breakthrough in low-level tasks, convolutional neural networks (CNNs) have been introduced to the defocus deblurring problem and achieved significant progress. However, previous methods apply the same learned kernel for different regions of the defocus blurred images, thus it is difficult to handle nonuniform blurred images. To this end, this study designs a novel blur-aware multi-branch network (Ba-MBNet), in which different regions are treated differentially. In particular, we estimate the blur amounts of different regions by the internal geometric constraint of the dual-pixel (DP) data, which measures the defocus disparity between the left and right views. Based on the assumption that different image regions with different blur amounts have different deblurring difficulties, we leverage different networks with different capacities to treat different image regions. Moreover, we introduce a meta-learning defocus mask generation algorithm to assign each pixel to a proper branch. In this way, we can expect to maintain the information of the clear regions well while recovering the missing details of the blurred regions. Both quantitative and qualitative experiments demonstrate that our BaMBNet outperforms the state-of-the-art (SOTA) methods. For the dual-pixel defocus deblurring (DPD)-blur dataset, the proposed BaMBNet achieves 1.20 dB gain over the previous SOTA method in term of peak signal-to-noise ratio (PSNR) and reduces learnable parameters by 85%. The details of the code and dataset are available at https://github.com/junjun-jiang/BaMBNet.

Defocus deblurring: a designed deep model based on CNN

With the flourishing development of deep learning in the field of computer vision, research of defocus deblurring based on it has gradually become a hotspot. However, most of the research focuses on defocus region detection or defocus map estimation, and algorithms for directly generating restoration images are less studied. Stressing on the problems of defocus deblurring, we propose a defocus deblurring deep model based on multi-scale information and convolution neural network. Concretely, we first perform an efficient and concise multi-scale information fusion by the selective receptive field module, thus the model can adapt to the scale sensitivity of the image defocusing region. We then use the residual channel attention module in the bottleneck module to extract the correlation features between channels, which enhances the effective channels and suppress the useless ones. Finally, a fusion objective function of edge loss and mean square loss is proposed to enhance the edge details of the image. Experimental results on a large-scale defocus deblurring dual-pixel dataset demonstrate that the proposed model has better performance than the traditional and existing deep-based methods. Comparing with the methods of the state of the art, the proposed model has a 0.44-DB improvement in PSNR metric.

High-quality blind defocus deblurring of multispectral images with optics and gradient prior.

This paper presents a blind defocus deblurring method that produces high-quality deblurred multispectral images. The high quality is achieved by two means: i) more accurate kernel estimation based on the optics prior by simulating the simple lens imaging, and ii) the gradient-based inter-channel correlation with the reference image generated by the content-adaptive combination of adjacent channels for restoring the latent sharp image. As a result, our method gains the prominence on both effectiveness and efficiency in deblurring defocus multispectral images with very good restoration on the obscure details. The experiments on some multispectral image datasets demonstrate the advantages of our method over state-of-the-art deblurring methods.

Analysis and design of coded apertures for defocus deblurring based on imaging system properties and optical features

As the physical size of single pixels in digital cameras grows smaller, the captured images are increasingly affected by defocused blurring and consequently valuable details are lost. Different aperture patterns have already been proposed to mitigate this problem based on presumed conditions, which maybe violated in practise. Sensor characteristics and current photometric scene properties have been largely ignored in the design of aperture patterns in the literature. In this study, a number of perceptually optimised coded apertures are introduced for defocused deblurring. These apertures are specifically designed considering illumination conditions, sensor specifications and human visual system characteristics. The designed patterns are compared with circular apertures of equal throughput and pinhole aperture. Experiments show signal-to-noise ratio (SNR) gains of up to 0.35 and 2 dB over circular and pinhole apertures, respectively. To study the trade-off between diffraction and deblurring gains, the proposed binary masks are enhanced by smoothing and morphological operations, which can yield non-binary and rounded binary patterns. The results of the authors’ study show that rounded binary patterns improve diffraction behaviour while maintaining the desired SNR level.

A Framework for Analysis of Computational Imaging Systems: Role of Signal Prior, Sensor Noise and Multiplexing.

Over the last decade, a number of computational imaging (CI) systems have been proposed for tasks such as motion deblurring, defocus deblurring and multispectral imaging. These techniques increase the amount of light reaching the sensor via multiplexing and then undo the deleterious effects of multiplexing by appropriate reconstruction algorithms. Given the widespread appeal and the considerable enthusiasm generated by these techniques, a detailed performance analysis of the benefits conferred by this approach is important. Unfortunately, a detailed analysis of CI has proven to be a challenging problem because performance depends equally on three components: (1) the optical multiplexing, (2) the noise characteristics of the sensor, and (3) the reconstruction algorithm which typically uses signal priors. A few recent papers [12], [30], [49] have performed analysis taking multiplexing and noise characteristics into account. However, analysis of CI systems under state-of-the-art reconstruction algorithms, most of which exploit signal prior models, has proven to be unwieldy. In this paper, we present a comprehensive analysis framework incorporating all three components. In order to perform this analysis, we model the signal priors using a Gaussian Mixture Model (GMM). A GMM prior confers two unique characteristics. First, GMM satisfies the universal approximation property which says that any prior density function can be approximated to any fidelity using a GMM with appropriate number of mixtures. Second, a GMM prior lends itself to analytical tractability allowing us to derive simple expressions for the `minimum mean square error' (MMSE) which we use as a metric to characterize the performance of CI systems. We use our framework to analyze several previously proposed CI techniques (focal sweep, flutter shutter, parabolic exposure, etc.), giving conclusive answer to the question: `How much performance gain is due to use of a signal prior and how much is due to multiplexing? Our analysis also clearly shows that multiplexing provides significant performance gains above and beyond the gains obtained due to use of signal priors.

Omnigradient Based Total Variation Minimization for Enhanced Defocus Deblurring of Omnidirectional Images

We propose a new method of image restoration for catadioptric defocus blur using omnitotal variation (Omni-TV) minimization based on omnigradient. Catadioptric omnidirectional imaging systems usually consist of conventional cameras and curved mirrors for capturing 360° field of view. The problem of catadioptric omnidirectional imaging defocus blur, which is caused by lens aperture and mirror curvature, becomes more severe when high resolution sensors and large apertures are used. In an omnidirectional image, two points near each other may not be close to one another in the 3D scene. Traditional gradient computation cannot be directly applied to omnidirectional image processing. Thus, omnigradient computing method combined with the characteristics of catadioptric omnidirectional imaging is proposed. Following this Omni-TV minimization is used as the constraint for deconvolution regularization, leading to the restoration of defocus blur in an omnidirectional image to obtain all sharp omnidirectional images. The proposed method is important for improving catadioptric omnidirectional imaging quality and promoting applications in related fields like omnidirectional video and image processing.