Depth From Defocus Research Articles

An improved DFD method for three-dimensional displacement measurement of vision-based tactile sensor

Currently, traditional tactile sensors based on the principles of capacitance or piezoelectricity have complex structures and difficulty in obtaining tactile information. A vision-based tactile sensor is introduced which can realize visual measurement of three-dimensional displacement in this paper. The vision-based tactile sensor is mainly composed of an elastomer embedded with marker point array, a transparent acrylic plate, 8 LED lights and a micro monocular camera. The elastomer deforms when the tactile sensor contacts an object, and the micro monocular camera is used to capture the elastomer deformation and transmit it to the computer in the form of image, and then the three-dimensional displacement information is obtained by processing the image. In order to more accurately recover the missing dimensional information in the three-dimensional displacement detection of monocular camera, an improved DFD (Depth from Defocus) method based on finite element theory is proposed in this paper. It is verified by experiments that the improved DFD method proposed in this paper can measure the three-dimensional displacement information more accurately compared with the DFD method. In addition, an experiment is conducted to prove the robustness of the improved DFD method on the robotic gripper. The experimental results demonstrate that the three-dimensional displacement measurement method proposed in this paper can provide technical support for the design and development of vision-based tactile sensors.

Depth from Defocus Technique For High Number Density Particle Images

The Depth from Defocus (DFD) imaging technique is used to measure the size and number concentration of particles in dispersed two-phase flows, but until now it has primarily been applied to low concentration particle images. This study explores how the technique can be extended to handle overlapping images caused by neighboring particles, significantly broadening the application scope of the DFD technique and enabling measurements at higher particle number/volume concentrations. The processing algorithms are experimentally validated using a dedicated apparatus that can systematically vary particle size, shape, and degree of image overlap. Additionally, this study explores the use of Convolutional Neural Networks (CNN) for this task, comparing these results with those obtained using conventional analyses in terms of accuracy, tolerable concentration limits, and computational speed. This approach requires a large teaching dataset of images, which is only practical and feasible if the dataset can be synthetically generated. An image generation procedure for out-of-focus neighboring spherical particles, resulting in a known blurred image overlap, is therefore first developed. This procedure is validated using laboratory images with known particle size distribution, position, and image overlap before creating the teaching dataset. The trained processing scheme is then applied to both synthetic datasets and experimental data, allowing the evaluation of the technique’s limits in terms of image overlap and tolerable volume concentration, as a function of particle size distribution.

A Single Camera Calibration-Free Approach For Dispersion Size Measurement Using Depth From Defocus

The Depth from Defocus (DFD) imaging technique is used to measure the size and number concentration of particles in dispersed two-phase flows, but until now it has primarily been applied to low concentration particle images. This study explores how the technique can be extended to handle overlapping images caused by neighboring particles, significantly broadening the application scope of the DFD technique and enabling measurements at higher particle number/volume concentrations. The processing algorithms are experimentally validated using a dedicated apparatus that can systematically vary particle size, shape, and degree of image overlap. Additionally, this study explores the use of Convolutional Neural Networks (CNN) for this task, comparing these results with those obtained using conventional analyses in terms of accuracy, tolerable concentration limits, and computational speed. This approach requires a large teaching dataset of images, which is only practical and feasible if the dataset can be synthetically generated. An image generation procedure for out-of-focus neighboring spherical particles, resulting in a known blurred image overlap, is therefore first developed. This procedure is validated using laboratory images with known particle size distribution, position, and image overlap before creating the teaching dataset. The trained processing scheme is then applied to both synthetic datasets and experimental data, allowing the evaluation of the technique’s limits in terms of image overlap and tolerable volume concentration, as a function of particle size distribution.

Depth from Defocus technique for irregular particle images

The Depth from Defocus (DFD) imaging technique for measuring the size and number concentration of particles in dispersed two-phase flows has up to now been restricted to the processing of spherical particle images. This study examines how this technique can be extended to process irregular particle images, arising either from neighboring spherical particles which lead to overlapping images, or from the imaging of non-spherical particles. In either case, the application scope of the DFD technique is significantly expanded. On the one hand the ability to identify and process overlapping images allows measurements to be performed at higher number/volume concentrations. On the other hand, especially solid particles are often non-spherical, thus making the DFD technique attractive for these applications. For both cases, the processing algorithms are experimentally benchmarked against ground truth using a dedicated apparatus in which particles of known size, shape and degree of overlapping images can be systematically varied.

A Monocular Variable Magnifications 3D Laparoscope System Using Double Liquid Lenses.

During minimal invasive surgery (MIS), the laparoscope only provides a single viewpoint to the surgeon, leaving a lack of 3D perception. Many works have been proposed to obtain depth and 3D reconstruction by designing a new optical structure or by depending on the camera pose and image sequences. Most of these works modify the structure of the conventional laparoscopes and cannot provide 3D reconstruction of different magnification views. In this study, we propose a laparoscopic system based on double liquid lenses, which provide doctors with variable magnification rates, near observation, and real-time monocular 3D reconstruction. Our system composes of an optical structure that can obtain auto magnification change and autofocus without any physically moving element, and a deep learning network based on the Depth from Defocus (DFD) method, trained to suit inconsistent camera intrinsic situations and estimate depth from images of different focal lengths. The optical structure is portable and can be mounted on conventional laparoscopes. The depth estimation network estimates depth in real-time from monocular images of different focal lengths and magnification rates. Experiments show that our system provides a 0.68-1.44x zoom rate and can estimate depth from different magnification rates at 6fps. Monocular 3D reconstruction reaches at least 6mm accuracy. The system also provides a clear view even under 1mm close working distance. Ex-vivo experiments and implementation on clinical images prove that our system provides doctors with a magnified clear view of the lesion, as well as quick monocular depth perception during laparoscopy, which help surgeons get better detection and size diagnosis of the abdomen during laparoscope surgeries.

Learning local depth regression from defocus blur by soft-assignment encoding.

We present a novel, to the best of our knowledge, patch-based approach for depth regression from defocus blur. Most state-of-the-art methods for depth from defocus (DFD) use a patch classification approach among a set of potential defocus blurs related to a depth, which induces errors due to the continuous variation of the depth. Here, we propose to adapt a simple classification model using a soft-assignment encoding of the true depth into a membership probability vector during training and a regression scale to predict intermediate depth values. Our method uses no blur model or scene model; it only requires a training dataset of image patches (either raw, gray scale, or RGB) and their corresponding depth label. We show that our method outperforms both classification and direct regression on simulated images from structured or natural texture datasets, and on raw real data having optical aberrations from an active DFD experiment.

A Two-Camera Depth From Defocus Imaging System And Sensitivity Analysis

A single lens, two-camera imaging system using backlight photography has been previously proposed based on Depth from Defocus (DFD) for the measurement of size and three-dimensional position of particles or drops. It makes use of two images with different degrees of defocus blur captured simultaneously at different imaging planes. In the present study, the influence of system and particle parameters on the calibration functions and measurement uncertainties of such a system have been comprehensively investigated and experimentally verified. The parameters varied include the distance between the imaging planes, the image bit resolution, pixel resolution and the random image noise. Furthermore, the intensity and position of the backlighting and the type of particle and its relative refractive index have been varied. Experimental verification was conducted using calibration dot targets, polystyrene latex spheres, and pulverized coal powders. A sensitivity analysis was performed using relative errors or differences for the three quantities - calibration function, particle size and particle position. The study concludes with explicit recommendations on which quantities are most influential in determining measurement accuracy and therefore, must be carefully controlled. Using this imaging system, measurements were conducted for a mono-sized standard latex sphere clouds suspended in a water container, and the results of particle size distribution were compared with that obtained using a common single lens, one-camera imaging system with thresholding image processing. Subsequently, the system is used for measurement of a sparse spray. Comparison measurements using the laser diffraction technique and the analysis of focused images are provided,with relatively good quantitative agreement. The measurement depth range based on the DFD method exhibits a simple proportional relationship with particle size, which can be used for bias correction of the particle size distribution and number density. This leads to a highly resolved measurement volume depth; hence, an accurate estimation of the overall measurement volume, allowing very accurate volume number density distributions to be estimated. This constitutes a decisive advantage over existing optical techniques for particle size measurements.

Learning scene and blur model for active chromatic depth from defocus.

In this paper, we propose what we believe is a new monocular depth estimation algorithm based on local estimation of defocus blur, an approach referred to as depth from defocus (DFD). Using a limited set of calibration images, we directly learn image covariance, which encodes both scene and blur (i.e., depth) information. Depth is then estimated from a single image patch using a maximum likelihood criterion defined using the learned covariance. This method is applied here within a new active DFD method using a dense textured projection and a chromatic lens for image acquisition. The projector adds texture for low-textured objects, which is usually a limitation of DFD, and the chromatic aberration increases the estimated depth range with respect to a conventional DFD. Here, we provide quantitative evaluations of the depth estimation performance of our method on simulated and real data of fronto-parallel untextured scenes. The proposed method is then experimentally evaluated qualitatively using a 3D printed benchmark.

Simple method of acquiring high-quality light fields based on the chromatic aberration of only one defocused image pair.

Direct light field acquisition method using a lens array requires a complex system and has a low resolution. On the other hand, the light fields can be also acquired indirectly by back-projection of the focal stack images without lens array, providing a resolution as high as the sensor resolution. However, it also requires the bulky optical system design to fix field-of-view (FOV) between the focal stacks, and an additional device for sensor shifting. Also, the reconstructed light field is texture-dependent and low-quality because it uses either a high-pass filter or a guided filter for back-projection. This paper presents a simple light field acquisition method based on chromatic aberration of only one defocused image pair. An image with chromatic aberration has a different defocus distribution for each R, G, and B channel. Thus, the focal stack can be synthesized with structural similarity (SSIM) 0.96 from only one defocused image pair. Then this image pair is also used to estimate the depth map by depth-from-defocus (DFD) using chromatic aberration (chromatic DFD). The depth map obtained by chromatic DFD is used for high-quality light field reconstruction. Compared to existing light field indirect acquisition, the proposed method requires only one pair of defocused images and can clearly reconstruct light field images with Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) scores lowered by 17%-38% and with Perception-Based Image Quality Evaluator (PIQE) scores lowered by 19%-45%. A defocused image pair is acquired by our customized compact optical system consisting of only three lenses, including a varifocal lens. Image processing and image quality evaluation are all performed using MATLAB.

Performance model of depth from defocus with an unconventional camera.

In this paper, we present a generic performance model able to evaluate the accuracy of depth estimation using depth from defocus (DFD). This model only requires the sensor point spread function at a given depth to evaluate the theoretical accuracy of depth estimation. Hence, it can be used for any (un)conventional system, using either one or several images. This model is validated experimentally on two unconventional DFD cameras, using either a coded aperture or a lens with chromatic aberration. Then, we use the proposed model for the end-to-end design of a 3D camera using an unconventional lens with chromatic aberration, for the specific use-case of small unmanned aerial vehicle navigation.

Compact and fast depth sensor based on a liquid lens using chromatic aberration to improve accuracy.

Depth from defocus (DFD) obtains depth information using two defocused images, making it possible to obtain a depth map with high resolution equal to that of the RGB image. However, it is difficult to change the focus mechanically in real-time applications, and the depth range is narrow because it is inversely proportional to the depth accuracy. This paper presents a compact DFD system based on a liquid lens that uses chromatic aberration for real-time application and depth accuracy improvement. The electrical focus changing of a liquid lens greatly shortens the image-capturing time, making it suitable for real-time applications as well as helping with compact lens design. Depth accuracy can be improved by dividing the depth range into three channels using chromatic aberration. This work demonstrated the improvement of depth accuracy through theory and simulation and verified it through DFD system design and depth measurement experiments of real 3D objects. Our depth measurement system showed a root mean square error (RMSE) of 0.7 mm to 4.98 mm compared to 2.275 mm to 12.3 mm in the conventional method, for the depth measurement range of 30 cm to 70 cm. Only three lenses are required in the total optical system. The response time of changing focus by the liquid lens is 10 ms, so two defocused images for DFD can be acquired within a single frame period of real-time operations. Lens design and image processing were conducted using Zemax and MATLAB, respectively.

Autofocus Measurement for Electronic Components Using Deep Regression

Image-based focus measurement in the manufacturing process is important for tasks such as automated inspection, automated assembly, and autosoldering. To inspect or manipulate an object, the visual system must sense the best image position to ensure that all the details of the product can be viewed. For this reason, an autofocus system is required. Traditional autofocus systems based on depth-from-focus (DFF) or depth-from-defocus (DFD) rely on a series of images by moving the lens with a control motor to find the sharpest location. The processing time defers it for in-line, real-time industrial applications. In this article, we propose a deep neural network regressor for fast and accurate autofocus. The convolutional neural network (CNN) model is proposed and evaluated for the autofocus task on a variety of electronic surfaces, including wafers, liquid crystal display (LCD), and ball grid array (BGA). The proposed deep regression model requires only one single sensed image of the object to estimate the depth. The effect of environmental variations in the illumination is evaluated for the robustness of the proposed method. Experimental results indicate the proposed CNN Regressor can achieve fast and accurate results for the tested objects with repetitive and arbitrary patterns.

A new exponential wear model to analyze precision retention of guideway based on macro-micro multiscale method

At present, wear models and accuracy prediction of guideway are established based on elastoplastic mechanics of continuous media. These methods are limited to describe the process of accuracy degradation by using the material performance defined based on macroscopic hypothesis condition. In this paper, the multiscale method based on quasicontinuum medium(QM) principle is proposed to describe the degradation process of guideway linear precision with an exponential model. According to the wear distribution of the guideway surface with the micro-morphology evolution process, the measurement value of the guideway linear precision is modeled systematically. Then, using the quasicontinuum medium method instead of the continuum hypothesis, the exponential wear model of guideway is established. The exponential wear model uses the wear exponent to describe the wear status based on linear precision measurement rather than tremendous wear volume tests. By means of Depth From Defocus (DFD) method, the micro-topography information of the guideway surface is obtained. Therefore, the wear state of the guideway is verified under different loading conditions, and the validity of using the QM method to establish the guideway exponential wear model is verified.

Learning to autofocus based on Gradient Boosting Machine for optical microscopy

Autofocus is an essential part of modern high-throughput optical microscopic imaging system, and the traditional passive autofocus algorithms such as hill climbing search are heuristics and therefore are slow and less accuracy. Instead of using the heuristics, in this paper we treat the autofocus as a regression problem and propose to learn a Gradient Boosting Machine (GBM) to predict the direction and step size simultaneously. We first leverage Fourier optical theory to explore the feasibility of predicting the step size and direction simultaneously with only one regressor. And then, inspired by Depth from Defocus (DFD), we design novel basic and combined features for faster and better autofocus. Finally, we comprehensively evaluate our methods on a dataset consisting of 2000 annotated images corresponding to 20 benchmarks of cell images. Our Defocus of Focus (DFF)-based autofocus shows improved accuracy over previous work from 97.1% to 99.9%, and significantly reduces the number of steps, achieving a 51.48% relative improvement. Code and dataset will be made publicly available.

FPGA Design of Real-Time MDFD System Using High Level Synthesis

3D shape information is one of the very important clues in image processing and computer vision. Unlike traditional multi-input depth from defocus (DFD) technique, monocular DFD (MDFD) algorithm proposed by Hu and Haan can reconstruct 3D shape only from a single monocular defocus image with low computing complexity. In this paper, we present a real-time MDFD system implemented on the FPGA device. In order to reduce the FPGA design cost, vivado high level synthesis (VHLS) is applied to design the MDFD system. The system architecture on the basis of FIFO based convolution is first designed through C/C++ code that is further converted to the FPGA design by VHLS. Then the PIPELINE, LOOP_MERGE, and ARRAY_PARTITION directives are used to optimize the latency and interval of the proposed system. The performance and resource utilization of the whole system are evaluated by processing defocus images from the real scene with 640×480 pixel size. The system can process about 22 images at 20 MHz working frequency and keep the 93.29% depth accuracy on the 3D objects test, which achieves a real-time state-of-the-art MDFD system by comparing to other recent works.

Depth from defocus measurement method based on liquid crystal lens.

This paper describes a depth from defocus (DFD) method to recover the depth of scene from two images taken by a liquid crystal (LC) lens imaging system. The system is composed of a camera module and an LC lens. The system's focal length is electrically adjusted by the voltages that are applied on the LC lens. We use two images that are taken at maximum positive and negative, respectively, and the LC lens's optical power, to obtain the depth information via DFD. The principle is described and the experimental results are successfully obtained. The method is simple in that it needs not to involve any mechanical lens movements in the imaging system.

Turning a conventional camera into a 3D camera with an add-on.

We propose to add an optical component in front of a conventional camera to improve depth estimation performance of depth from defocus (DFD), an approach based on the relation between defocus blur and depth. The add-on overcomes ambiguity and the dead zone, which are the fundamental limitations of DFD with a conventional camera, by adding an optical aberration to the whole system that makes the blur unambiguous and measurable for each depth. We look into two optical components: the first one adds astigmatism and the other one chromatic aberration. In both cases, we present the principle of the add-on and experimental validations on real prototypes.

Depth from defocus (DFD) based on VFISTA optimization algorithm in micro/nanometer vision

In the three-dimensional (3D) morphological reconstruction of micro/nano-scale vision, the global depth from defocus algorithm (DFD) transforms the depth information of the scene into a dynamic optimization problem to solve. In order to improve the problem of dynamic optimization in the recovery process of global DFD, a variable-step-size fast iterative shrinkage-thresholding algorithm (VFISTA) is proposed. The traditional iterative shrinkage-thresholding algorithm (ISTA) is often used to solve this dynamic optimization problem in the global DFD method. The ISTA algorithm is an extension of the gradient descent method, which is close to the minimal value point of the optimization process, and the convergence speed is slow. What is more, the ISTA algorithm uses fixed step length in the iterative process, The search direction tend to be “orthogonal”, prone to “saw tooth” phenomenon, so close to the minimum point when the convergence rate is slower. First, the VFISTA algorithm joins the acceleration operator on the basis of the ISTA algorithm. Further, it combines linear search method to find the optimal iteration length, and changes the limit of the ISTA algorithm step fixed. Finally, it is applied to the depth measurement of defocus scene in micro/nanometer scale vision. The experimental results show that the proposed fast depth from defocus algorithm based on VFISTA has faster convergent speed. Moreover, the precision of the measurement is obviously improved in micro/nanometer scale vision.

Automatic selection of focal lengths in a Depth From Defocus measurement system based on liquid lenses

A simple and compact Depth-From-Defocus (DFD) setup, using telecentric illumination and liquid-lens based camera observation, was shown to perform well for 3D shape acquisition over extended measuring range. A further step to ameliorate the system performance is described in this paper. We focused on finding an algorithm to speed up the calibration step of the method, that automatically determines the minimum number of focal lengths to be used in the calibration and measurement procedure. As a result, the calibration is significantly shortened (up to 80% with respect to the original procedure), and the need to manually (and to some extent arbitrarily) select the focal length pairs is overcome. Measurement errors down to 0.73mm over the measurement depth range of 130mm, corresponding to 0.55% of the depth range are achieved, in total agreement with the original system.

A Fast 2D-to-3D Image Conversion System based on Depth from Defocus

Blur variation in 2D images caused by camera focus provides a suitable cue for depth estimation. Depth from defocus (DFD) technique calculates the blur amount in images considering that the depth and defocus blur are related to each other. Conventional DFD methods use only defocused images that might yield low-quality depth data and reconstructed infocused image. In this article, a new DFD methodology based on infocused and defocused images is proposed in which using an infocused image can solve the quality degradation problems. In this method, Subbarao’s DFD is combined with a novel edge blur estimation method to obtain an improved depth map. In addition, a saliency map mitigates the ill-posed problem of depth estimation in homogeneous regions. For real-time full high definition (Full-HD) image processing, a parallelized graphics processing unit (GPU) implementation is devised to improve execution speed.