Microlens Images Research Articles

Light field Laparoscope imaging model and calibration method based on flexible aperture-angular plane

With rapid developments in light field imaging, a great deal of attention has been given to its applications in industrial, medical and other fields due to its ability to perform three-dimensional reconstruction in single-shot. In these applications, Light field Laparoscope (LFL) is an important one, but it often suffers severe micro-lens image deformations that lead to incorrect LFL decoding, calibration and three-dimensional reconstruction. Based on the micro-lens image non-deformation constraint presented by us before, we propose the flexible aperture-angular plane to analyze the LFL imaging model, the modified microlens image non-deformation constraint for 3D LFL system and an advanced two-step calibration method to compute 3D LFL imaging parameters. Moreover, a 3D LFL imaging prototype is designed and calibrated. Experimental results show that microlens image deformations are avoided in this 3D LFL prototype, and the typical RMS re-projection error is about 0.06 pixels.

Design of a Novel Microlens Array and Imaging System for Light Fields.

Light field cameras are unsuitable for further acquisition of high-quality images due to their small depth of field, insufficient spatial resolution, and poor imaging quality. To address these issues, we proposed a novel four-focal-square microlens and light field system. A square aspheric microlens array with four orthogonal focal lengths was designed, in which the aperture of a single lens was 100 μm. The square arrangement improves pixel utilization, the four focal lengths increase the depth of field, and the aspheric improves image quality. The simulations demonstrate pixel utilization rates exceeding 90%, depth-of-field ranges 6.57 times that of a single focal length, and image quality is significantly improved. We have provided a potential solution for improving the depth of field and image quality of the light field imaging system.

Distortionless condition for microlens images in light field imaging

Distortionless condition for microlens images in light field imaging

Pseudoreference Subaperture Images and Microlens Image-Based Blind Light Field Image Quality Measurement

Light field image (LFI) records the radiation intensity and direction information of light in scene, offering powerful capabilities for computer vision. It is found that the distortion caused by LFI processing is obviously reflected in the two representations of LFI, that is, subaperture images (SAIs) and microlens images (MLI). Specifically, on the one hand, pseudoreference SAI (PSAI) of a distorted SAI, which is rendered with the adjacent distorted SAIs, can be used to measure the distortion of the involved SAIs. On the other hand, macropixel blocks (MPBs) of MLI can be adopted to measure LFI global angular distortion. Therefore, a PSAIs- and MLI-based blind LFI quality measurement (PM-BLFIQM) method is proposed in this article. First, the local light field (LLF) is defined to generate PSAI of the central distorted SAI in each LLF of LFI, and then a pseudo-full-reference quality measurement metric based on PSAIs is presented. Second, to measure the angular consistency of the distorted LFI, an MLI-based feature extraction scheme is proposed for MPBs and their mean removal version by using discrete cosine transform and singular value decomposition. Finally, the quality of the distorted LFI is predicted with the features extracted from LLFs and MLI. Experimental results demonstrate that both PSAI and MPB are effective for measuring the quality of distorted LFI. In addition, the results on three LFI databases show that the proposed method achieves higher Spearman rank-order correlation coefficient (SROCC) and Pearson linear correlation coefficient (PLCC) values, so it outperforms the state-of-the-art methods in terms of consistency with human visual perception.

Design of bionic compound eye system based on hexagonal closely spliced structure

Based on the visual advantages of bionic compound eyes, the research progress of bionic compound eyes was analyzed, and the imaging principle of compound eyes and the structure of some insect compound eyes were studied. According to the structure of the biological compound eye, a hexagonal closely spliced curved micro-lens array and image transfer system were designed. At the same time, in order to prevent the crosstalk of imaging beams between adjacent sub-eyes, an aperture array with a single aperture of 1.5 mm in length was designed to realize single-channel imaging of each sub-eye. According to the principle of pupil connection, the micro lens array and the conversion system were combined and optimized. The diameter of the entire compound eye was 8.66 mm, the field of view was 121°, the diameter of each sub-eye was 500 μm, and the field of view of the sub-eye was 6°. At 90 lp/mm, the MTF value of the compound eye system was greater than 0.3, and the RMS spot radius were less than the radius of the Airy disk, the imaging quality of the system meet the requirements of the design index. In order to met the requirements of 3D additive manufacturing process, the mechanical structure of the compound eye system was designed. The tolerance analysis result shows that the system structure meets the preparation requirements of 3D printing technology.

Lossless Compression of Plenoptic Camera Sensor Images

We propose a codec for the lossless compression of plenoptic camera sensor images. The proposed encoder starts by splitting the input lenslet image into rectangular patches, with each patch corresponding to a microlens image. The encoder and decoder exploit the correlation between the pixels in neighbor patches using a patch-by-patch prediction mechanism where each pixel of a patch has its own dedicated sparse predictor designed to utilize the most relevant pixels from the neighbor patch to the left. An intra-patch prediction mask together with the pixels from the neighbor left patch form the final prediction template. The encoder performs the design of sparse predictors by first finding the relevant regressors in the final template. The patches are classified into M classes according to two possible mechanisms (either based on depth information or based on Bayer mask colors), and the sparse predictor design is performed for each pair (class label; patch pixel index). A relevant context selection mirrors the selection of relevant regressors, thereby providing the arithmetic coding with skewed coding distributions at each context. We show examples of the application of the proposed methods on sensor images from the JPEG Pleno database, thus demonstrating the improved performance of the proposed methods compared to the existing predictive methods for encoding camera sensor images.

Micro-Lens Image Stack Upsampling for Densely-Sampled Light Field Reconstruction

Due to hardware restriction, it is costly to capture densely-sampled Light Fields (LFs) with high angular and spatial resolution, which becomes the main bottleneck of LFs development. In this paper, we propose a learning-based framework to synthesize novel views and reconstruct densely-sampled LFs from sparsely-sampled LFs. In the proposed framework, micro-lens image stacks and view image stacks are separately grouped, in which details in novel views are explored from spatial and angular domain. The two kinds of stacks contain epipolar information and 3D convolution layers are employed to effectively extract features that include structure information. Moreover, an innovative way is proposed to synthesize views by upsampling micro-lens image stacks using deconvolution layers. The parameters in decovolution layers provide the view position information and different interpolation and extrapolation tasks can be explicitly modeled. It is validated that this view synthesis module can be embedded in different frameworks and improve the related performances. Without precise depth estimation and view warping, the proposed method is mainly designed for reconstructing LFs with small baselines. Related experimental results show that the proposed model outperforms other state-of-the-art methods in terms of both visual and numerical evaluations. Furthermore, the consistency between synthesized views and the intrinsic structure information is well preserved in the proposed method.

Accurate calibration of standard plenoptic cameras using corner features from raw images.

Light field cameras capture spatial and angular information simultaneously. A scene point in the 3D space appears many times on the raw image, bringing challenges to light field camera calibration. This paper proposes a novel calibration method for standard plenoptic cameras by using corner features from raw images. We select appropriate micro-lens images on raw images and detect corner features on them. During calibration, we first build the relationship of corner features and points in object space by using a few intrinsic parameters and then perform a linear calculation of these parameters, which are further refined via a non-linear optimization. Experiments on Lytro and Lytro Illum cameras demonstrate that the accuracy and efficiency of the proposed method are superior to the state-of-the-art methods based on features of raw images.

Light field endoscope calibration based on virtual objective lens and virtual feature points

Three-dimensional light field imaging in laparoscopic surgery is an emerging technology, which has the potential to enable three-dimensional imaging. Calibration of the three-dimensional light field endoscope (3D LFE) is essential but challenging as the disparity is much smaller than that of the conventional light field camera. The geometrical model for 3D LFE is established, and a calibration method based on virtual objective lens and virtual feature points is proposed. First, the virtual objective lens is introduced and the parameters about it are calibrated using corner features in center subaperture images. Second, two types of virtual feature points are proposed to calibrate the parameters about the microlens array, one is on the black-and-white board line and the other is selectively determined but can be anywhere on the checkerboard. Moreover, the relationship between the virtual feature points mapping in the microlens image and the virtual feature points mapping in the central subaperture image is deduced to overcome tiny light field disparity. Experimental results verify the performance of our calibration method.

Longitudinal optical imaging technique to visualize progressive axonal damage after brain injury in mice reveals responses to different minocycline treatments

A high-resolution, three-dimensional, optical imaging technique for the murine brain was developed to identify the effects of different therapeutic windows for preclinical brain research. This technique tracks the same cells over several weeks. We conducted a pilot study of a promising drug to treat diffuse axonal injury (DAI) caused by traumatic brain injury, using two different therapeutic windows, as a means to demonstrate the utility of this novel longitudinal imaging technique. DAI causes immediate, sporadic axon damage followed by progressive secondary axon damage. We administered minocycline for three days commencing one hour after injury in one treatment group and beginning 72 hours after injury in another group to demonstrate the method’s ability to show how and when the therapeutic drug exerts protective and/or healing effects. Fewer varicosities developed in acutely treated mice while more varicosities resolved in mice with delayed treatment. For both treatments, the drug arrested development of new axonal damage by 30 days. In addition to evaluation of therapeutics for traumatic brain injury, this hybrid microlens imaging method should be useful to study other types of brain injury and neurodegeneration and cellular responses to treatment.

Light Field Saliency Detection with Deep Convolutional Networks.

Light field imaging presents an attractive alternative to RGB imaging because of the recording of the direction of the incoming light. The detection of salient regions in a light field image benefits from the additional modeling of angular patterns. For RGB imaging, methods using CNNs have achieved excellent results on a range of tasks, including saliency detection. However, it is not trivial to use CNN-based methods for saliency detection on light field images because these methods are not specifically designed for processing light field inputs. In addition, current light field datasets are not sufficiently large to train CNNs. To overcome these issues, we present a new Lytro Illum dataset, which contains 640 light fields and their corresponding ground-truth saliency maps. Compared to current publicly available light field saliency datasets [1], [2], our new dataset is larger, of higher quality, contains more variation and more types of light field inputs. This makes our dataset suitable for training deeper networks and benchmarking. Furthermore, we propose a novel end-to-end CNN-based framework for light field saliency detection. Specifically, we propose three novel MAC (Model Angular Changes) blocks to process light field micro-lens images. We systematically study the impact of different architecture variants and compare light field saliency with regular 2D saliency. Our extensive comparisons indicate that our novel network significantly outperforms state-of-the-art methods on the proposed dataset and has desired generalization abilities on other existing datasets.

Novel Immune Microlens Imaging for Detection of Antigen and Antibody

Detection and analysis of antigen-antibody reaction is one of the most critical detection techniques in the fields of medicine, biology, environmental science, and food safety. Traditional and classical methods for detecting antigen and antibody encounter many problems, such as time-consuming, high cost, and low accuracy. A novel immune microsphere imaging technique by the microlens is used to test the changes of refractive index before and after antigen-antibody reaction. It can quickly perform qualitative and quantitative determination for antigen-antibody reaction without any labeling, premodification, postwashing, and expensive enzymes. Here, we feature and discuss its principle and advantages, structure of a microlens immunoassay instrument, and potential in measuring clinical samples. It is promising to be developed for application to diagnosis of clinical diseases.

Light field-based face liveness detection with convolutional neural networks

Face recognition based-access systems have been used widely in security systems as the recognition accuracy can be quite high. However, these systems suffer from low robustness to spoofing attacks. To achieve a reliable security system, a well-defined face liveness detection technique is crucial. We present an approach for this problem by combining data of the light-field camera (LFC) and the convolutional neural networks in the detection process. The LFC can detect the depth of an object by a single shot, from which we derive meaningful features to distinguish the spoofing attack from the real face, through a single shot. We propose two features for liveness detection: the ray difference images and the microlens images. Experimental results based on a self-built light-field imaging database for three types of the spoofing attacks are presented. The experimental results show that the proposed system gives a lower average classification error (0.028) as compared with the method of using hand-crafted features and conventional imaging systems. In addition, the proposed system can be used to classify the type of the spoofing attack.

Binocular microlens imaging system based on micro fabrication technology and its application in vein-enhanced display

Vein display device can significantly increase the success rate of intravenous injection. However, traditional vein display devices only contain near-infrared image information, which inevitably loses skin color information. In this article, a binocular microlens was fabricated by micro-fabrication technology. Then, a smart binocular microlens imaging system is set up. By this system, the near-infrared image with venous distribution and visible light image with actual skin color can be obtained simultaneously. In addition, an improved image fusing algorithm is proposed. The image fusing result shows that the venous details can be clearly shown and the background color of the hand also can be well preserved. Therefore, the proposed scheme can provide more real vein information. As the smart structure of the binocular microlens, the dual-bands microlens imaging system has the potential for miniaturization and could integrate with intelligent glasses for assisting intravenous injection.

Technique of dual-wavelength micro-lens imaging which can eliminate thermal noise for accurate on-site concentration measurement

The present paper reports a novel dual-wavelength micro-lens imaging technique which uses micro-lens as the sensor and can eliminate thermal noise for on-site real time and continues accurate concentration detection in mixing or inhomogeneous solutions. By simply immersing micro-lenses into sample solution and deducing the refractive indices of the solution from their images taken with two wavelengths, the technique can eliminate random thermal noise for accurate solution concentration measurement. With the technique, the accuracy and precision of refractive index detection was improved to ±10−6. Detections on NaCl and glucose solutions demonstrated that the measured concentrations were just about 0.005%–0.01% away from true values for one centigrade temperature fluctuation, and can catch the temporal concentration variation at a speed of two measurements per second.

Micro-Lens-Based Matching for Scene Recovery in Lenslet Cameras.

Since a light-field camera is able to capture more information than a traditional camera, a lot of methods, such as depth estimation, image super-resolution, and view synthesis, are explored for recovering scene information. In this paper, we propose a novel framework for scene recovery based on lenslet-based light-field camera images. Instead of using traditional matching terms, we design a new micro-lens-based matching term to calculate structure information and recover several kinds of scene information simultaneously. On the one hand, inherent information in micro-lens images is selected to complement details in sub-aperture images. On the other hand, sub-aperture images are used to expand micro-lens images and synthesize new view images. A new micro-lens-based consistency metric is introduced for the matching term to handle occlusions in depth estimation and image reconstruction. The newly appeared and newly occluded areas in synthesized views are analyzed and recovered based on information from surrounding points. Experimental results show that the proposed depth estimation method outperforms state-of-the-art methods on both synthetic and lenslet-based light-field images, especially in low-texture and occlusion regions. Furthermore, the super-resolution and view synthesis methods are able to acquire view images with more details and less aliasing artifacts.

Easy and rapid multi-pass detection of antigen and antibody with micro-lens sensors

We introduce a micro-lens imaging method that can perform easy and rapid multi-pass detection of antigen/antibody (Ag/Ab) without the requirement of any labeling, expensive enzymes, pre-immobilization/modification, and post-washing. Our method detects Ag or Ab presenting in solutions in a quantitative or qualitative manner by using micro-lens as the sensor to monitor the refractive index variation of the solutions during the primary stage of Ag-Ab reaction. The detection can be taken rapidly and finished in two minutes, while requires very low sample volume (several micron liters) and its detection limit can be as low as ∼pg/mL. The method is also able to provide kinetic and thermodynamic parameters of Ag-Ab reactions. The detections of ten Ag-Ab systems and two kinds of clinical samples demonstrated that our method is of high sensitivity, accuracy, reliability and permitting on-site analysis.

Automated Robust Metric Calibration Algorithm for Multifocus Plenoptic Cameras

The concept of plenoptic cameras has existed since the early 1900s, with recent advances in computational power making them a viable tool for industrial measurement applications. Plenoptic cameras can give 3-D information for every point of the scene with one camera, one lens, and a single image. This is possible by placing a microlens array directly in front of the image sensor. The depth estimation is based on disparities observed in neighboring microlens images, similar to stereo camera approaches. To relate these so-called virtual depth units to metric distances, a camera calibration must be performed. This paper presents a robust, automated camera calibration technique, which introduces new ways to model a multifocus plenoptic camera.

Automatic calibration method for plenoptic camera

An automatic calibration method is proposed for a microlens-based plenoptic camera. First, all microlens images on the white image are searched and recognized automatically based on digital morphology. Then, the center points of microlens images are rearranged according to their relative position relationships. Consequently, the microlens images are located, i.e., the plenoptic camera is calibrated without the prior knowledge of camera parameters. Furthermore, this method is appropriate for all types of microlens-based plenoptic cameras, even the multifocus plenoptic camera, the plenoptic camera with arbitrarily arranged microlenses, or the plenoptic camera with different sizes of microlenses. Finally, we verify our method by the raw data of Lytro. The experiments show that our method has higher intelligence than the methods published before.

Geometric Calibration of Micro-Lens-Based Light Field Cameras Using Line Features.

We present a novel method for the geometric calibration of micro-lens-based light field cameras. Accurate geometric calibration is the basis of various applications. Instead of using sub-aperture images, we directly utilize raw images for calibration. We select appropriate regions in raw images and extract line features from micro-lens images in those regions. For the entire process, we formulate a new projection model of a micro-lens-based light field camera, which contains a smaller number of parameters than previous models. The model is transformed into a linear form using line features. We compute the initial solution of both the intrinsic and the extrinsic parameters by a linear computation and refine them via non-linear optimization. Experimental results demonstrate the accuracy of the correspondences between rays and pixels in raw images, as estimated by the proposed method.