Defocus Map Research Articles

Spatially varying defocus map estimation from a single image based on spatial aliasing sampling method.

In current optical systems, defocus blur is inevitable due to the constrained depth of field. However, it is difficult to accurately identify the defocus amount at each pixel position as the point spread function changes spatially. In this paper, we introduce a histogram-invariant spatial aliasing sampling method for reconstructing all-in-focus images, which addresses the challenge of insufficient pixel-level annotated samples, and subsequently introduces a high-resolution network for estimating spatially varying defocus maps from a single image. The accuracy of the proposed method is evaluated on various synthetic and real data. The experimental results demonstrate that our proposed model outperforms state-of-the-art methods for defocus map estimation significantly.

A defocus and similarity attention-based cascaded network for multi-focus and misaligned image fusion

Multi-focus image fusion uses multiple images focused on different depths to generate a clear image covering the whole scene. The existing multi-focus image fusion methods do not consider the defocus successive variation in close-range photography and the camera shaking in sequence image shooting, and most methods cannot process multiple images simultaneously. We proposed an end-to-end deep learning network to generate the all-in-focus image from multi-focus and misaligned images to solve these problems. Specifically, taking multiple multi-focus and misaligned source images as input, our Defocus and Similarity Attention Fusion Network (DSAF-Net) first generates the corresponding defocus map through the Defocus-Net, and warps the source images and defocus maps to a unified camera view according to the optical flow estimated by the OpticalFlow-Net. Finally, our model applies a coarse-to-fine correspondence matching scheme to obtain the similarity weight maps, combined with the warped source images and defocus maps to fuse into a clear image. For training and testing our DSAF-Net, a multi-focus and misaligned cultural heritage photography dataset (WHU-MFM) is constructed using Blender Cycles renderer. Experiments results demonstrate that our method outperforms the state-of-the-art methods both qualitatively and quantitatively. DSAF-Net is available at https://github.com/PeimingCHEN/DSAF-Net. And WHU-MFM is available at https://github.com/PeimingCHEN/WHU-MFM-Dataset.

Learnable Blur Kernel for Single-Image Defocus Deblurring in the Wild

Recent research showed that the dual-pixel sensor has made great progress in defocus map estimation and image defocus deblurring. However, extracting real-time dual-pixel views is troublesome and complex in algorithm deployment. Moreover, the deblurred image generated by the defocus deblurring network lacks high-frequency details, which is unsatisfactory in human perception. To overcome this issue, we propose a novel defocus deblurring method that uses the guidance of the defocus map to implement image deblurring. The proposed method consists of a learnable blur kernel to estimate the defocus map, which is an unsupervised method, and a single-image defocus deblurring generative adversarial network (DefocusGAN) for the first time. The proposed network can learn the deblurring of different regions and recover realistic details. We propose a defocus adversarial loss to guide this training process. Competitive experimental results confirm that with a learnable blur kernel, the generated defocus map can achieve results comparable to supervised methods. In the single-image defocus deblurring task, the proposed method achieves state-of-the-art results, especially significant improvements in perceptual quality, where PSNR reaches 25.56 dB and LPIPS reaches 0.111.

Defocus to focus: Photo-realistic bokeh rendering by fusing defocus and radiance priors

We consider the problem of realistic bokeh rendering from a single all-in-focus image. Bokeh rendering mimics aesthetic shallow depth-of-field (DoF) in professional photography, but these visual effects generated by existing methods suffer from simple flat background blur and blurred in-focus regions, giving rise to unrealistic rendered results. In this work, we argue that realistic bokeh rendering should (i) model depth relations and distinguish in-focus regions, (ii) sustain sharp in-focus regions, and (iii) render physically accurate Circle of Confusion (CoC). To this end, we present a Defocus to Focus (D2F) framework to learn realistic bokeh rendering by fusing defocus priors with the all-in-focus image and by implementing radiance priors in layered fusion. Since no depth map is provided, we introduce defocus hallucination to integrate depth by learning to focus. The predicted defocus map implies the blur amount of bokeh and is used to guide weighted layered rendering. In layered rendering, we fuse images blurred by different kernels based on the defocus map. To increase the reality of the bokeh, we adopt radiance virtualization to simulate scene radiance. The scene radiance used in weighted layered rendering reassigns weights in the soft disk kernel to produce the CoC. To ensure the sharpness of in-focus regions, we propose to fuse upsampled bokeh images and original images. We predict the initial fusion mask from our defocus map and refine the mask with a deep network. We evaluate our model on a large-scale bokeh dataset. Extensive experiments show that our approach is capable of rendering visually pleasing bokeh effects in complex scenes. In particular, our solution receives the runner-up award in the AIM 2020 Rendering Realistic Bokeh Challenge.

Single image defocus map estimation through patch blurriness classification and its applications

Depth information is useful in many image processing applications. However, since taking a picture is a process of projection of a 3D scene onto a 2D imaging sensor, the depth information is embedded in the image. Extracting the depth information from the image is a challenging task. A guiding principle is that the level of blurriness due to defocus is related to the distance between the object and the focal plane. Based on this principle and the widely used assumption that Gaussian blur is a good model for defocus blur, we formulate the problem of estimating the spatially varying defocus blurriness as a Gaussian blur classification problem. We solve the problem by training a deep neural network to classify image patches into one of the 20 levels of blurriness. We have created a dataset of more than 500,000 image patches of size 32times 32 which does not require human labelling. The dataset is used to train and test several well-known network models. We find that MobileNetV2 is suitable for this application due to its low memory requirement and high accuracy. The trained model is used to determine the patch blurriness which is then refined by applying an iterative weighted guided filter. The result is a defocus map that carries the information of the degree of blurriness for each pixel. We compare the proposed method with state-of-the-art techniques and we demonstrate its successful applications in adaptive image enhancement and defocus magnification limited to images that present a clear distinction between defocus levels.

Three-dimensional measurement of the droplets out of focus in shadowgraphy systems via deep learning-based image-processing method

For the shadowgraphy techniques with a single camera, it is difficult to accurately obtain the shape, size, and depth location of the droplets out of focus due to the defocus blur. This paper proposed a deep learning-based method to recover the sharp images and infer the depth information from the defocused blur droplets images. The proposed model comprising of a defocus map estimation subnetwork and a defocus deblur subnetwork is optimized with a two-stage strategy. To train the networks, the synthetic blur data generated by the Gauss kernel method are utilized as the input data, which mimic the defocused images of droplets. The proposed approach has been assessed based on synthetic images and real sphere blur images. The results demonstrate that our method has satisfactory performance both in terms of depth location estimation and droplet size measurement, e.g., the diameter relative error is less than 5% and the location error is less than 1 mm for the sphere with a diameter of more than 1 mm. Moreover, the present model also exhibits considerable generalization and robustness against the transparent ellipsoid and the random background noise. A further application of the present model to the measurement of transparent water droplets generated by an injector is also explored and illustrates the practicability of the present model in real experiments. The present study indicates that the proposed learning-based method is promising for the three-dimensional (3D) measurement of spray droplets via a combination of shadowgraphy techniques using a single camera, which will greatly reduce experimental costs and complexity.

Defocus Image Deblurring Network With Defocus Map Estimation as Auxiliary Task.

Different from the object motion blur, the defocus blur is caused by the limitation of the cameras' depth of field. The defocus amount can be characterized by the parameter of point spread function and thus forms a defocus map. In this paper, we propose a new network architecture called Defocus Image Deblurring Auxiliary Learning Net (DID-ANet), which is specifically designed for single image defocus deblurring by using defocus map estimation as auxiliary task to improve the deblurring result. To facilitate the training of the network, we build a novel and large-scale dataset for single image defocus deblurring, which contains the defocus images, the defocus maps and the all-sharp images. To the best of our knowledge, the new dataset is the first large-scale defocus deblurring dataset for training deep networks. Moreover, the experimental results demonstrate that the proposed DID-ANet outperforms the state-of-the-art methods for both tasks of defocus image deblurring and defocus map estimation, both quantitatively and qualitatively. The dataset, code, and model is available on GitHub: https://github.com/xytmhy/DID-ANet-Defocus-Deblurring.

MA-GANet: A Multi-Attention Generative Adversarial Network for Defocus Blur Detection.

Background clutters pose challenges to defocus blur detection. Existing approaches often produce artifact predictions in background areas with clutter and relatively low confident predictions in boundary areas. In this work, we tackle the above issues from two perspectives. Firstly, inspired by the recent success of self-attention mechanism, we introduce channel-wise and spatial-wise attention modules to attentively aggregate features at different channels and spatial locations to obtain more discriminative features. Secondly, we propose a generative adversarial training strategy to suppress spurious and low reliable predictions. This is achieved by utilizing a discriminator to identify predicted defocus map from ground-truth ones. As such, the defocus network (generator) needs to produce 'realistic' defocus map to minimize discriminator loss. We further demonstrate that the generative adversarial training allows exploiting additional unlabeled data to improve performance, a.k.a. semi-supervised learning, and we provide the first benchmark on semi-supervised defocus detection. Finally, we demonstrate that the existing evaluation metrics for defocus detection generally fail to quantify the robustness with respect to thresholding. For a fair and practical evaluation, we introduce an effective yet efficient AUFβ metric. Extensive experiments on three public datasets verify the superiority of the proposed methods compared against state-of-the-art approaches.

Defocus deblurring: a designed deep model based on CNN

With the flourishing development of deep learning in the field of computer vision, research of defocus deblurring based on it has gradually become a hotspot. However, most of the research focuses on defocus region detection or defocus map estimation, and algorithms for directly generating restoration images are less studied. Stressing on the problems of defocus deblurring, we propose a defocus deblurring deep model based on multi-scale information and convolution neural network. Concretely, we first perform an efficient and concise multi-scale information fusion by the selective receptive field module, thus the model can adapt to the scale sensitivity of the image defocusing region. We then use the residual channel attention module in the bottleneck module to extract the correlation features between channels, which enhances the effective channels and suppress the useless ones. Finally, a fusion objective function of edge loss and mean square loss is proposed to enhance the edge details of the image. Experimental results on a large-scale defocus deblurring dual-pixel dataset demonstrate that the proposed model has better performance than the traditional and existing deep-based methods. Comparing with the methods of the state of the art, the proposed model has a 0.44-DB improvement in PSNR metric.

Joint Depth and Defocus Estimation From a Single Image Using Physical Consistency.

Estimating depth and defocus maps are two fundamental tasks in computer vision. Recently, many methods explore these two tasks separately with the help of the powerful feature learning ability of deep learning and these methods have achieved impressive progress. However, due to the difficulty in densely labeling depth and defocus on real images, these methods are mostly based on synthetic training dataset, and the performance of learned network degrades significantly on real images. In this paper, we tackle a new task that jointly estimates depth and defocus from a single image. We design a dual network with two subnets respectively for estimating depth and defocus. The network is jointly trained on synthetic dataset with a physical constraint to enforce the physical consistency between depth and defocus. Moreover, we design a simple method to label depth and defocus order on real image dataset, and design two novel metrics to measure accuracies of depth and defocus estimation on real images. Comprehensive experiments demonstrate that joint training for depth and defocus estimation using physical consistency constraint enables these two subnets to guide each other, and effectively improves their depth and defocus estimation performance on real defocused image dataset.

Depth Estimation from a Single Image Based on Cauchy Distribution Model

Most approaches to estimate a scene’s 3D depth from a single image often model the point spread function (PSF) as a 2D Gaussian function. However, those methods are suffered from some noises, and difficult to get a high quality of depth recovery. We presented a simple yet effective approach to estimate exactly the amount of spatially varying defocus blur at edges, based on a Cauchy distribution model for the PSF. The raw image was re-blurred twice using two known Cauchy distribution kernels, and the defocus blur amount at edges could be derived from the gradient ratio between the two re-blurred images. By propagating the blur amount at edge locations to the entire image using the matting interpolation, a full depth map was then recovered. Experimental results on several real images demonstrated both feasibility and effectiveness of our method, being a non-Gaussian model for DSF, in providing a better estimation of the defocus map from a single un-calibrated defocused image. These results also showed that our method was robust to image noises, inaccurate edge location and interferences of neighboring edges. It could generate more accurate scene depth maps than the most of existing methods using a Gaussian based DSF model.

AIFNet: All-in-Focus Image Restoration Network Using a Light Field-Based Dataset

Defocus blur often degrades the performance of image understanding, such as object recognition and image segmentation. Restoring an all-in-focus image from its defocused version is highly beneficial to visual information processing and many photographic applications, despite being a severely ill-posed problem. We propose a novel convolutional neural network architecture AIFNet for removing spatially-varying defocus blur from a single defocused image. We leverage light field synthetic aperture and refocusing techniques to generate a large set of realistic defocused and all-in-focus image pairs depicting a variety of natural scenes for network training. AIFNet consists of three modules: defocus map estimation, deblurring and domain adaptation. The effects and performance of various network components are extensively evaluated. We also compare our method with existing solutions using several publicly available datasets. Quantitative and qualitative evaluations demonstrate that AIFNet shows the state-of-the-art performance.

Defocus map estimation from a single image using improved likelihood feature and edge-based basis

Defocus map estimation (DME) is very useful in many computer vision applications and has drawn much attention in recent years. Edge-based DME methods can generate sharp defocus discontinuities but usually suffer from textures of the input image. Region-based methods are free of textures but cannot catch the defocus discontinuities very well. In this paper, we propose a DME method combining edge-based and region-based methods together to keep their respective advantages while eliminating the shortcomings. The combination is achieved via regression tree fields (RTF). In an RTF, the input feature and the linear basis are of vital importance. For our RTF, they are obtained as follows. (i) Two orthogonal gradient operators with the corresponding subsets of Gabor filters are employed in localized 2D frequency analysis to generate accurate likelihood, and the first K highest local maximums of likelihood are sent to an RTF as input feature. (ii) At the same time, the input image is processed by three edge-based methods and the results serve as the linear basis of RTF. The experiments demonstrate that the proposed method outperforms state-of-the-art DME methods. Moreover, the proposed method can be readily applied to defocused image deblurring and defocus blur detection.

Image attention retargeting using defocus map and bilateral filter

In this paper a method is proposed to enhance the saliency of selected region with respect to its surrounding to improve user’s attention. Bilateral filter is first used to make the ROI smoother to reduce the unwanted details in the defocus map. Hue component of ROI is adjusted and the outline is extracted using neighbouring data. The outline in resultant image is modified using hue component to direct viewer’s attention in a certain region. The proposed method is applied on different natural images and the results are compared with the state-of-the-art existing methods to prove the proposed method’s effectiveness.

Defocus blur estimation in calibrated multi-view images for 3D archaeological documentation

Defocus blur in images deteriorates the quality of image-based textures for 3D archaeological models. We propose a method to estimate the defocus blur in calibrated multi-view images based on photographic optics. Our method estimates the defocus blur maps without any modification to cameras or using any statistical model of images. After the focus distance of each image is estimated from the reconstructed 3D models with physical scale, the near distance dn and the far distance df of acceptable sharpness from the view point are calculated. We set the colour of the model vertex to white if the vertex’s distance from the view point is within the interval between dn and df, otherwise black. Finally we render the 3D model into a defocus map using the intrinsic and extrinsic parameters of the camera. The obtained defocus maps can be utilized as masks to generate an all-focused texture map for the reconstructed 3D archaeological model. Experimental results on real data demonstrate the effectiveness of the method.

A global approach to describe retinal defocus patterns.

The popularity of myopia treatments based on the peripheral defocus theory has risen. So far, little evidence has emerged around the questions which of these treatments are effective and why. In order to establish a framework that enables clinicians and researchers to acknowledge the possible interactions of different defocus patterns across the retina, different peripheral refractive errors (PRX) of subjects and different designs of optical treatments were evaluated. Dioptric defocus patterns on the retinal level have been obtained by merging the matrices of dioptric defocus maps of the visual field of different scenarios with individual peripheral refractive errors and different optical designs of multifocal contact lenses. The newly obtained matrices were statistically compared using a non-parametric test with familywise error algorithms and multi-comparison tests. Results show that asymmetric peripheral refractive error profiles (temporal or nasal positively skewed) appear to be less prone to be changed by the defocus imposition of multifocal contact lenses than those presenting symmetric patterns (relative peripheral myopia or hyperopia).

A Robust and Accurate Calibration Method for Out-of-focus Camera

Conventional camera calibration methods regard camera as ideal pinhole model and require well-focused images, which can't be satisfied for long-range photogrammetry or low depth-of-field lens. In this paper, we propose a novel active calibration method for out-of-focus camera using LCD monitor. Firstly, we estimate the defocus map by the temporal coded binary-shift patterns, which makes our method more accurate. Secondly, based on the defocus map, we encode LCD pixel's coordinates into phase-shift patterns with optimal frequency and step properties, and then deblur captured patterns. Finally, deblurred patterns are decoded to generate dense phases map to extract accurate feature points coordinates. Our method significantly improves camera calibrations robustness to lens' defocus, noises, glass refraction compared with state-of-art methods. Experimental results demonstrate that our method is superior to conventional methods whether camera is in- or out-of- focus.

Dioptric defocus maps across the visual field for different indoor environments.

One of the factors proposed to regulate the eye growth is the error signal derived from the defocus in the retina and actually, this might arise from defocus not only in the fovea but the whole visual field. Therefore, myopia could be better predicted by spatio-temporally mapping the 'environmental defocus' over the visual field. At present, no devices are available that could provide this information. A 'Kinect sensor v1' camera (Microsoft Corp.) and a portable eye tracker were used for developing a system for quantifying 'indoor defocus error signals' across the central 58° of the visual field. Dioptric differences relative to the fovea (assumed to be in focus) were recorded over the visual field and 'defocus maps' were generated for various scenes and tasks.

A Single Image Deblurring Algorithm for Nonuniform Motion Blur Using Uniform Defocus Map Estimation

One of the most common artifacts in digital photography is motion blur. When capturing an image under dim light by using a handheld camera, the tendency of the photographer’s hand to shake causes the image to blur. In response to this problem, image deblurring has become an active topic in computational photography and image processing in recent years. From the view of signal processing, image deblurring can be reduced to a deconvolution problem if the kernel function of the motion blur is assumed to be shift invariant. However, the kernel function is not always shift invariant in real cases; for example, in-plane rotation of a camera or a moving object can blur different parts of an image according to different kernel functions. An image that is degraded by multiple blur kernels is called a nonuniform blur image. In this paper, we propose a novel single image deblurring algorithm for nonuniform motion blur images that is blurred by moving object. First, a proposed uniform defocus map method is presented for measurement of the amounts and directions of motion blur. These blurred regions are then used to estimate point spread functions simultaneously. Finally, a fast deconvolution algorithm is used to restore the nonuniform blur image. We expect that the proposed method can achieve satisfactory deblurring of a single nonuniform blur image.

Defocus Map Estimation From a Single Image Based on Two-Parameter Defocus Model

Defocus map estimation (DME) is highly important in many computer vision applications. Nearly, all existing approaches for DME from a single image are based on a one-parameter defocus model, which does not allow for the variation of depth over edges. In this paper, a novel two-parameter model of defocused edges is proposed for DME from a single image. We can estimate the defocus amounts for each side of the edges through this proposed model, and the confidence that the edge is a pattern edge, where the depth remains the same over the edge, can be generated. Then, we modify the TV-L1 algorithm for structure-texture decomposition by taking advantage of this confidence to eliminate pattern edges while preserving structural ones. Finally, the defocus amounts estimated at the edge positions are used as initial values, and the structure component is employed as a guidance in the following Laplacian matting procedure to avoid the influence of pattern edges on the final defocus map. Experiment results show that the proposed method can effectively eliminate the influence of pattern edges compared with the state-of-art method. Furthermore, the estimated defocus map is feasible in applications of depth estimation and foreground/background segmentation.