Focal Stack Images Research Articles

All-in-Focus Image Generation from Microscope Focal Stack Images by Deep Learning

In the visual inspection of industrial products using a microscope with a shallow depth of field, it is difficult to capture an all-in-focus image. This paper presents a method to generate an all-in-focus image from a large focal stack, which can be applied to visual inspection. The proposed method adds two improvements to a deep learning method [M. Maximov et al., CVPR, (2020) 1068] that leverages defocus cues in depth estimation. First, it interpolates an index map (a collection of in-focus image indices at all pixels) after the forward pass. It generates an all-in-focus image using all images in focal stacks while reducing the number of images input to the network. Second, the network is trained with synthetic datasets to which a random texture with high-frequency components is added. It helps the network to learn the degree of defocus blur. Experiments using synthetic images and real microscope images show that interpolating the index map and adding texture improves the accuracy of all-in-focus image generation.

Light Field View Synthesis Using the Focal Stack and All-in-Focus Image.

Light field reconstruction and synthesis algorithms are essential for improving the lower spatial resolution for hand-held plenoptic cameras. Previous light field synthesis algorithms produce blurred regions around depth discontinuities, especially for stereo-based algorithms, where no information is available to fill the occluded areas in the light field image. In this paper, we propose a light field synthesis algorithm that uses the focal stack images and the all-in-focus image to synthesize a 9 × 9 sub-aperture view light field image. Our approach uses depth from defocus to estimate a depth map. Then, we use the depth map and the all-in-focus image to synthesize the sub-aperture views, and their corresponding depth maps by mimicking the apparent shifting of the central image according to the depth values. We handle the occluded regions in the synthesized sub-aperture views by filling them with the information recovered from the focal stack images. We also show that, if the depth levels in the image are known, we can synthesize a high-accuracy light field image with just five focal stack images. The accuracy of our approach is compared with three state-of-the-art algorithms: one non-learning and two CNN-based approaches, and the results show that our algorithm outperforms all three in terms of PSNR and SSIM metrics.

Focal Stack Image Compression Based on Basis-Quadtree Representation

In this paper, we propose an efficient compression scheme for focal stack images (FoSIs) based on a new basis-quadtree representation. In the new basis-quadtree representation, FoSIs are initially reorganized as co-located block groups in the depth dimension. In each group, selective basis blocks and adaptive quadtree partition are optimized to predict the focused or defocused co-located blocks by intra-group approximation. By solving a joint optimization problem, FoSIs can be sparsely represented by the optimal basis blocks, corresponding quadtree partition and approximation parameters, which will be compressed separately. Then, these basis blocks are stitched into several new frames (basis frames) according to their original locations and partition modes. Basis frames are compressed by our designed encoder, where the intra-group approximation is embedded into the high efficiency video coding (HEVC) encoder. Thus, the redundancies of basis blocks can be further eliminated. Finally, the approximation parameters are refined to suppress the amplified errors caused by introduced compression blur after basis frame coding. The refined parameters are compressed losslessly and multiplexed with the bitstream of the basis frames to ensure the reconstruction quality of FoSIs. Experiments on 12 test sequences demonstrate that the proposed scheme can obtain higher coding performance than the state-of-the-art comparison schemes. Specifically, the proposed scheme achieves up to 5.23 dB PSNR gains and 71.59% bitrate savings over the HEVC baseline scheme on sequences I03 and I05, respectively.

Light Field Reflection and Background Separation Network Based on Adaptive Focus Selection

Separating the background and the reflection layer is an ill-posed but meaningful task. Since the two layers often have different depths, Light Fields (LFs), which contain the depth information, show great potential to deal with this task. In this paper, we propose to fully exploit the difference between background and reflection layers from different kinds of images in LFs. Specifically, we fuse the complementary features from sub-aperture images, in which the overlaps of the two layers vary a lot. Then we propose the adaptive focus selection strategy and use the dynamic filter to select the appropriate focus. The refocused image is then obtained from the focal stack images, where the clearness of the two layers is different. Finally, the clean images are restored using a dual attention reconstruction module. Our model is the first one that successfully addresses the LF reflection separation task with an end-to-end deep neural network. The experiments show that the background and reflection layers are well separated, regardless of the relative intensity difference between the two layers on both synthetic and real-world datasets. The quantitative and qualitative comparison results show that our method achieves better performance than other state-of-the-art methods.

Focal stack based image forgery localization.

Image security is becoming an increasingly important issue due to advances in deep learning based image manipulations, such as deep image inpainting and deepfakes. There has been considerable work to date on detecting such image manipulations using improved algorithms, with little attention paid to the possible role that hardware advances may have for improving security. We propose to use a focal stack camera as a novel secure imaging device, to the best of our knowledge, that facilitates localizing modified regions in manipulated images. We show that applying convolutional neural network detection methods to focal stack images achieves significantly better detection accuracy compared to single image based forgery detection. This work demonstrates that focal stack images could be used as a novel secure image file format and opens up a new direction for secure imaging.

Noise-Resilient Depth Estimation for Light Field Images Using Focal Stack and FFT Analysis.

Depth estimation for light field images is essential for applications such as light field image compression, reconstructing perspective views and 3D reconstruction. Previous depth map estimation approaches do not capture sharp transitions around object boundaries due to occlusions, making many of the current approaches unreliable at depth discontinuities. This is especially the case for light field images because the pixels do not exhibit photo-consistency in the presence of occlusions. In this paper, we propose an algorithm to estimate the depth map for light field images using depth from defocus. Our approach uses a small patch size of pixels in each focal stack image for comparing defocus cues, allowing the algorithm to generate sharper depth boundaries. Then, in contrast to existing approaches that use defocus cues for depth estimation, we use frequency domain analysis image similarity checking to generate the depth map. Processing in the frequency domain reduces the individual pixel errors that occur while directly comparing RGB images, making the algorithm more resilient to noise. The algorithm has been evaluated on both a synthetic image dataset and real-world images in the JPEG dataset. Experimental results demonstrate that our proposed algorithm outperforms state-of-the-art depth estimation techniques for light field images, particularly in case of noisy images.

Simple method of acquiring high-quality light fields based on the chromatic aberration of only one defocused image pair.

Direct light field acquisition method using a lens array requires a complex system and has a low resolution. On the other hand, the light fields can be also acquired indirectly by back-projection of the focal stack images without lens array, providing a resolution as high as the sensor resolution. However, it also requires the bulky optical system design to fix field-of-view (FOV) between the focal stacks, and an additional device for sensor shifting. Also, the reconstructed light field is texture-dependent and low-quality because it uses either a high-pass filter or a guided filter for back-projection. This paper presents a simple light field acquisition method based on chromatic aberration of only one defocused image pair. An image with chromatic aberration has a different defocus distribution for each R, G, and B channel. Thus, the focal stack can be synthesized with structural similarity (SSIM) 0.96 from only one defocused image pair. Then this image pair is also used to estimate the depth map by depth-from-defocus (DFD) using chromatic aberration (chromatic DFD). The depth map obtained by chromatic DFD is used for high-quality light field reconstruction. Compared to existing light field indirect acquisition, the proposed method requires only one pair of defocused images and can clearly reconstruct light field images with Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) scores lowered by 17%-38% and with Perception-Based Image Quality Evaluator (PIQE) scores lowered by 19%-45%. A defocused image pair is acquired by our customized compact optical system consisting of only three lenses, including a varifocal lens. Image processing and image quality evaluation are all performed using MATLAB.

Gaussian-Wiener Representation and Hierarchical Coding Scheme for Focal Stack Images

Focal stack images (FoSIs) are a set of 2D images that captured one scene with serial focal depths. The redundancies of FoSIs mainly come from gradual focused depths changes rather than motion of objects. Conventional coding schemes cannot fully exploit such redundancies, leading to coding inefficiency. In this paper, we propose a new Gaussian-Wiener representation to model the gradual focused depths changes among FoSIs. In the representation, image degradation-restoration relations are utilized to describe the focus-defocus changing characteristics of FoSIs. Based on this representation, we propose a new hierarchical coding scheme for fully exploiting the inter-frame redundancies of FoSIs. In the scheme, a Gaussian-Wiener representation based inter prediction (GWR-IP) is presented by embedding Gaussian convolution and Wiener deconvolution into normal video encoder. Block-wise focus-defocus changing of FoSIs can be predicted in bi-directional manner by solving optimization problem. For higher coding efficiency, a Gaussian-Wiener representation based hierarchical prediction structure (GWR-HPS) is also designed and applied in the coding scheme. The proposed coding scheme is performed on 10 test sequences, including 5 synthetic scenes and 5 realistic scenes. Experimental results show that proposed coding scheme can obtain 2.640 dB PSNR gains and 51.830% bitrate savings on average of all test sequences in Low Delay P configuration, 2.123 dB PSNR gains and 43.975% bitrate savings in Low Delay B configuration, and 1.044 dB PSNR gains and 26.078% bitrate savings in Random Access configuration. Particularly, it achieves up to 65.544% bit rate savings and 3.901 dB PSNR increments for test sequence I09 in Low Delay P configuration. Furthermore, ablation test demonstrates that Gaussian representation contributes more on coding performance than Wiener representation and GWR-HPS.

Block-wise focal stack image representation for end-to-end applications.

In optical imaging systems, the depth of field (DoF) is generally constricted due to the nature of optical lens. The limited DoF produces partially focused images of the scene. Focal stack images (FoSIs) are a sequence of images that focused on serial depths of a scene. FoSIs are capable of extending DoF of optical systems and provide practical solutions for computational photography, macroscopic and microscopic imaging, interactive and immersive media. However, high volumes of data remains one of the biggest obstacles to the development of end-to-end applications. In order to solve this challenge, we propose a block-wise Gaussian based representation model for FoSIs and utilize this model to solve the problem of coding, reconstruction and rendering for end-to-end applications. Experimental results demonstrate the high efficiency of proposed representation model and the superior performance of proposed schemes.

All-in-Focused Image Combination in the Frequency Domain Using Light Field Images

All-in-focused image combination is a fusion technique used to acquire related data from a set of focused images at different depth levels, which suggests that one can determine objects in the foreground and background regions. When attempting to reconstruct an all-in-focused image, we need to identify in-focused regions from multiple input images captured with different focal lengths. This paper presents a new method to find and fuse the in-focused regions of the different focal stack images. After we apply the two-dimensional discrete cosine transform (DCT) to transform the focal stack images into the frequency domain, we utilize the sum of the updated modified Laplacian (SUML), enhancement of the SUML, and harmonic mean (HM) for calculating in-focused regions of the stack images. After fusing all the in-focused information, we transform the result back by using the inverse DCT. Hence, the out-focused parts are removed. Finally, we combine all the in-focused image regions and reconstruct the all-in-focused image.

A Light Field Imaging Based Measuring Method for Blast Furnace Burden Distribution

Accurate height and distribution of blast furnace burden is essential to improve coke-ore ratio, improve gas flow and reduce CO2 emission. However, conventional burden distribution measure method by mechanical sounding rod can only measure 2 or 3 fixed discontinues burden height information. This article uses a light field imaging system to estimate blast furnace burden distribution. At first, the light field camera is used to obtain 4D image of the burden surface, light position and angle information is utilized to refocus to obtain focal stack images. Then the depth of burden surface is obtained by finding out the refocused image to target depth based on laplacian operator. At last, the whole burden surface is reconstructed in Csharp/emgu platform to extract burden surface distribution and radial burden distribution.

Focal Stack Image Fusion Using Image Volume Reconstruction

Traditional camera can’t take photos with large depth of field (DOF) because of the lens limits. The region out of DOF often gets blurred due to narrow width of depth. Focal stack photography take a set of (three and above) pictures viewing the same scene with different focus point and fusion the pictures as one using computational algorithms in order to have all-in-focus photos. Some state-of-art focal stack algorithms seek the best-in-focus point among these stack photos and collect them into a new all-in-focus image. This method is derived from the assumption that at least one photo in the stack has the clear vision of the point of the scene. But what if none of these points are in focus because unfortunately focus point falls out of the object of the scene? Focal stack images not only provide us several pictures independently but also partial information about how the object is imaging at a certain depth. With this information, we seek a space called ‘image volume’ to search the best in-focus depth beyond the two dimensional images to avoid the failure mentioned above. We work through the whole scene space and find the accurate depth of every point in the scene. We experiment some pictures with focal stack example and show better results than traditional methods.

3D compressive spectral integral imaging.

A novel compressive 3D imaging spectrometer based on the coded aperture snapshot spectral imager (CASSI) is proposed. By inserting a microlens array (MLA) into the CASSI system, one can capture spectral data of 3D objects in a single snapshot without requiring 3D scanning. The 3D spatio-spectral sensing phenomena is modelled by computational integral imaging in tandem with compressive coded aperture spectral imaging. A set of focal stack images is reconstructed from a single compressive measurement, and presented as images focused on different depth planes where the objects are located. The proposed optical system is demonstrated with simulations and experimental results.

Robust All-in-focus Super-Resolution for Focal Stack Photography

We present an unconventional image super-resolution algorithm targeting focal stack images. Contrary to previous works, which align multiple images with sub-pixel accuracy for image super-resolution, we analyze the correlation among the differently focused narrow depth-of-field images in a focal stack to infer high-resolution details for image super-resolution. In order to accurately model the defocus kernels at different depths, we use a cubic interpolation to parameterize the projection of defocus kernels, and apply the radon transform to accurately reconstruct the defocus kernels at arbitrary depth. In the image super-resolution, we utilize the multi-image deconvolution method with a l1 -norm regularization to suppress noise and ringing artifacts. We have also extended the depth-of-field of our inputs to produce an all-in-focus super-resolution image. The effectiveness of our algorithm is demonstrated with the quantitative analysis using synthetic examples and the qualitative analysis using real-world examples.