Camera Exposure Research Articles

Learned Camera Gain and Exposure Control for Improved Visual Feature Detection and Matching

Successful visual navigation depends upon capturing images that contain sufficient useful information. In this letter, we explore a data-driven approach to account for environmental lighting changes, improving the quality of images for use in visual odometry (VO) or visual simultaneous localization and mapping (SLAM). We train a deep convolutional neural network model to predictively adjust camera gain and exposure time parameters such that consecutive images contain a maximal number of matchable features. The training process is fully self-supervised: our training signal is derived from an underlying VO or SLAM pipeline and, as a result, the model is optimized to perform well with that specific pipeline. We demonstrate through extensive real-world experiments that our network can anticipate and compensate for dramatic lighting changes (e.g., transitions into and out of road tunnels), maintaining a substantially higher number of inlier feature matches than competing camera parameter control algorithms.

Compensating over- and underexposure in optical target pose determination

Optical coded targets allow to determine the relative pose of a camera, on a metric scale, from one image only. Furthermore, they are easily and efficiently detected, opening to a wide range of applications in robotics and computer vision. In this work we describe the effect of pixel saturation and non-ideal lens Point Spread Function, causing the apparent position of the corners and the edges of the target to change as a function of the camera exposure time. This effect, which we call exposure bias, is frequent in over- or underexposed images and introduces a systematic error in the estimated camera pose. We propose an algorithm that is able to estimate and correct for the exposure bias exploiting specific geometric features of a common target design based on concentric circles. Through rigorous laboratory experiments carried out in a highly controlled environment, we demonstrate that the proposed algorithm is seven times more precise and three times more accurate in the target distance estimation than the algorithms available in the literature.

Using computer-aided image processing to estimate chemical composition of igneous rocks: A potential tool for large-scale compositional mapping

Digital cameras, particularly on smartphones, have led to the proliferation of amateur photographers. Of interest here is the use of smartphone cameras to conduct rapid, low-cost compositional mapping of geologic bedrock, such as plutons and batholiths, in combination with chemical analyses of rocks in the laboratory. This paper discusses some of the challenges in geochemical mapping using image analysis. We discuss methods for color calibration through a series of experiments under different light intensities and conditions (spectra). All indoor and outdoor experiments show good reproducibility, but suffer from biases imparted by different light intensities, light conditions, and camera exposure times. These biases can be greatly reduced with a linear color calibration method. Over-exposed and under-exposed images, however, cannot be fully calibrated, so we discuss methods that ensure images are properly exposed. We applied our method to 59 natural granitoid and mafic enclave samples of known chemical composition. Multivariate linear regression has been explored for relating calibrated rock images with chemical compositions. Among all the chromatic and texture features of rock images, we show that average gray levels strongly correlate with major oxide concentrations. Subtle variations in bulk composition can potentially be rapidly assessed using calibrated photographs of outcrops.

High-speed 3D shape measurement using a rotary mechanical projector.

In this paper, a fast rotary mechanical projector (RMP) is designed and manufactured for high-speed 3D shape measurement. Compared with the common high-speed projectors, RMP has a good performance in high-speed projection, which can obtain high quality projected fringes with shorter camera exposure time by using the error diffusion binary coding method and chrome plating technology. The magnitude, acceptability of systemic projection error is analyzed and quantified in detail. For the quantified error, the probability distribution function (PDF) algorithm is introduced to correct the error. Corrected projection error is reduced to more than one third of the original error. Subsequently, a monocular measurement system composed of the RMP and a single camera is constructed. The combination of the RMP device and PDF algorithm ensure the accuracy of a corresponding 3D shape measurement system. Experiments have demonstrated that the proposed solution has a good performance for the 3D measurement of high-speed scenes.

3-D shape reconstruction of non-uniform reflectance surface based on pixel intensity, pixel color and camera exposure time adaptive adjustment

High dynamic range 3-D shape measurement is a challenge. In this work, we propose a novel method to solve the 3-D shape reconstruction of high-reflection and colored surfaces. First, we propose a method to establish a fast pixel-level mapping between the projected image and the captured image. Secondly, we propose a color texture extraction method using a black-and-white (B/W) camera and a pixel-level projection color adjustment method. Third, we give an optimal projection fringe modulation/background intensity ratio. Fourth, we propose a method for estimating the reflectivity of the object surface and ambient light interference, and a method for adjusting the projection intensity at the pixel level and a method for estimating the optimal exposure time. Experiments show that, compared with the existing methods, the proposed method not only can obtain high-quality captured images, but also has higher measurement efficiency and wider application range.

Two-color differential dynamic microscopy for capturing fast dynamics.

Differential dynamic microscopy (DDM) is increasingly used in the fields of soft matter physics and biophysics to extract the dynamics of microscopic objects across a range of wavevectors by optical microscopy. Standard DDM is limited to detecting dynamics no faster than the camera frame rate. We report on an extension to DDM where we sequentially illuminate the sample with spectrally distinct light and image with a color camera. By pulsing blue and then red light separated by a lag time much smaller than the camera's exposure time, we are able to use this two-color DDM method to measure dynamics occurring much faster than the camera frame rate.

Optical Flow Monocular Visual-Inertial Odometry with Online Photometric Calibration

Kanade-Lucas-Tomasi (KLT) optical flow algorithm based on the brightness constancy assumption is widely used in visual simultaneous localization and mapping (SLAM) and visual odometry (VO). However, the automatic adjustment of camera exposure time, the attenuation factor of sensor irradiance caused by vignetting, and the nonlinear camera response function will cause the same feature point to have different brightness values on different image frames, thus breaking this assumption. Hence, we propose a gain-adaptive KLT optical flow algorithm with online photometric calibration, and on this basis, design a monocular visual-inertial odometry which is insensitive to brightness changes. This method can calibrate the photometric parameters online in real time, meet the assumption of constant brightness in practical applications, and make the algorithm more robust and accurate in the case of dynamic changes in brightness. Experimental results on the TUM Mono and EuRoC datasets show that the proposed algorithm can reliably calibrate the photometric parameters of any video sequence and perform well in the environment with varying brightness.

3D surface reconstruction of small height object based on thin structured light scanning

Aiming at the problem of 3D surface reconstruction of small height objects, a method based on the scanning principle with thin structured light is proposed. The laser fringe is produced by a laser light source. It scans the surface of small height object under the control of a precise motion control system, and is integrated with a stereo light microscope and two cameras to form a complete structured light profilometry system. This method is very suitable for 3D surface reconstruction of small height objects. In order to deal with the phenomenon of "cracking" and local uneven brightness in microscope fringe image, the fringe image sequence is captured under different camera exposure parameters, and the quality of fringe image is improved by image fusion. The location of pixels on the center line of laser stripes is detected by the Loess local second-order fitting method, and the positions of these pixels are smoothed and predicted by using the Lowess local linear fitting method. The deformation of the laser stripe center curve is calculated by constructing a baseline, and the 3D surface reconstruction of the object with small height is realized in the image space. Taking the small characters on the surface of Chinese coins and human hair as test samples, their 3D surfaces are constructed by using the method proposed in this paper. The reconstructed 3D surface shapes are highly consistent with the real 3D surface shapes of objects.

Simultaneous measurements of tissue blood flow and oxygenation using a wearable fiber-free optical sensor.

.Significance: There is an essential need to develop wearable multimodality technologies that can continuously measure both blood flow and oxygenation in deep tissues to investigate and manage various vascular/cellular diseases.Aim: To develop a wearable dual-wavelength diffuse speckle contrast flow oximetry (DSCFO) for simultaneous measurements of blood flow and oxygenation variations in deep tissues.Approach: A wearable fiber-free DSCFO probe was fabricated using 3D printing to confine two small near-infrared laser diodes and a tiny CMOS camera in positions for DSCFO measurements. The spatial diffuse speckle contrast and light intensity measurements at the two different wavelengths enable quantification of tissue blood flow and oxygenation, respectively. The DSCFO was first calibrated using tissue phantoms and then tested in adult forearms during artery cuff occlusion.Results: Phantom tests determined the largest effective source–detector distance (15 mm) and optimal camera exposure time (10 ms) and verified the accuracy of DSCFO in measuring absorption coefficient variations. The DSCFO detected substantial changes in forearm blood flow and oxygenation resulting from the artery occlusion, which meet physiological expectations and are consistent with previous study results.Conclusions: The wearable DSCFO may be used for continuous and simultaneous monitoring of blood flow and oxygenation variations in freely behaving subjects.

Improving Laparoscopic Techniques Using Speckle Imaging

Investigators reduce inaccuracies due to noise in laser speckle contrast imaging by using multiple camera exposures and demonstrate the method in study of a live animal.

High-speed compressed-sensing fluorescence lifetime imaging microscopy of live cells

We present high-resolution, high-speed fluorescence lifetime imaging microscopy (FLIM) of live cells based on a compressed sensing scheme. By leveraging the compressibility of biological scenes in a specific domain, we simultaneously record the time-lapse fluorescence decay upon pulsed laser excitation within a large field of view. The resultant system, referred to as compressed FLIM, can acquire a widefield fluorescence lifetime image within a single camera exposure, eliminating the motion artifact and minimizing the photobleaching and phototoxicity. The imaging speed, limited only by the readout speed of the camera, is up to 100 Hz. We demonstrated the utility of compressed FLIM in imaging various transient dynamics at the microscopic scale.

Six-pack holographic imaging for dynamic rejection of out-of-focus objects.

Six-pack holography is adapted to reject out-of-focus objects in dynamic samples, using a single camera exposure and without any scanning. By illuminating the sample from six different angles in parallel using a low-coherence source, out-of-focus objects are laterally shifted in six different directions when projected onto the focal plane. Then pixel-wise averaging of the six reconstructed images creates a significantly clearer image, with rejection of out-of-focus objects. Dynamic imaging results are shown for swimming microalgae and flowing microbeads, including numerical refocusing by Fresnel propagation. The averaged images reduced the contribution of out-of-focus objects by up to 83% in comparison to standard holograms captured using the same light source, further improving the system sectioning capabilities. Both simulation and experimental results are presented.

Blind Motion Deblurring Super-Resolution: When Dynamic Spatio-Temporal Learning Meets Static Image Understanding.

Single-image super-resolution (SR) and multi-frame SR are two ways to super resolve low-resolution images. Single-Image SR generally handles each image independently, but ignores the temporal information implied in continuing frames. Multi-frame SR is able to model the temporal dependency via capturing motion information. However, it relies on neighbouring frames which are not always available in the real world. Meanwhile, slight camera shake easily causes heavy motion blur on long-distance-shot low-resolution images. To address these problems, a Blind Motion Deblurring Super-Reslution Networks, BMDSRNet, is proposed to learn dynamic spatio-temporal information from single static motion-blurred images. Motion-blurred images are the accumulation over time during the exposure of cameras, while the proposed BMDSRNet learns the reverse process and uses three-streams to learn Bidirectional spatio-temporal information based on well designed reconstruction loss functions to recover clean high-resolution images. Extensive experiments demonstrate that the proposed BMDSRNet outperforms recent state-of-the-art methods, and has the ability to simultaneously deal with image deblurring and SR.

Application of Monocular Direct Vision Odometry in Augmented Reality

In recent years, the unlabeled augmented reality system has been gradually applied to various mobile devices, among which stable, accurate, and fast registration is the key to realizing this function. For this technique, this paper introduces camera exposure parameters and puts the data association and pose estimation into a unified nonlinear optimization problem. Moreover, the direct monocular vision odometer is transplanted into the augmented reality system through the position adjustment module. We compare it with the traditional visual odometry method that matches the feature points. The results show that this improved method can be used to track more quickly and build a more visual semi-dense point cloud map, which can be used to support the registration and tracking of virtual objects in augmented reality.

Application of Hybrid Monocular SLAM Method in Augmented Reality

In this paper, we design a hybrid (semi-direct) approach to simultaneous localization and mapping (SLAM) for monocular cameras and apply it to augmented reality (AR) for monocular cameras. We combine the advantagesof the direct method and the feature point method. We use both photometric bundle adjustment which is robust to camera exposure time and motion bundle adjustment which is geometrically robust based on feature points to do tracking process. This approach can maintain an intuitive direct local map as well as a reusable global sparse feature point map. Through the processing of point clouds, such as PCA plane detection and grid reconstruction, we greatly improve the effect of the augmented reality system.

3D-design-model-assisted shiny surface shape measurement in fringe projection profilometry

In fringe projection three-dimensional (3D) shape measurement, image saturation is an important influencing factor for shiny surfaces, for example, an aero-engine blade. This paper proposes a novel method assisted by the 3D design model to resolve image saturation. The proposed algorithm has the following characteristics: (a) Based on the high matching between the measured finished object and its 3D design model, the design model is transformed into the discrete point cloud with sufficient accuracy. (b) Based on the spatial coordinate transformation principle, a multi-coordinate system transformation method is proposed to convert discrete point cloud data from the design coordinate system to the world coordinate system. (c) Considering the influence of the actual measurement conditions on the measured object, the existing illumination model is improved by integrating the system’s greyscale nonlinear response, camera exposure time and sensitivity. (d) Combined with the captured grating images and the 3D design model, the parameters are accurately calibrated by the least-squares algorithm. (e) Using the calibrated reflection model, the optimal rotation angle is determined to compensate for the spatial points corresponding to the image saturation area, and finally, the entire measured surface is registered with point clouds in the calculated optimal rotation locations. Our method is used in the actual pressure surface measurement experiment. The experimental results show that our method can effectively measure the space points corresponding to the image saturation area in the optimal rotation angle and improve the measurement accuracy of shiny surfaces.

LCTF-based multispectral fluorescence imaging: System development and potential for real-time foreign object detection in fresh-cut vegetable processing

Ensuring the supply of safe and contaminant-free fresh-cut vegetables is of importance to consumers and suppliers worldwide. The present research aimed to develop an online inspection system, making use of fluorescence and color imaging for detection of potential foreign materials (FMs) unintentionally added to fresh-cut vegetables. A multispectral fluorescence imager incorporating a liquid crystal tunable filter (LCTF) was developed to sequentially acquire selected band images of fresh-cut vegetables. Custom-built software was developed to synchronize the LCTF with the camera exposure to collect the fluorescence images of selected spectral bands and to collect the digital color images of the same samples using a color camera. Image processing algorithms were developed for both fluorescence and color images to detect a wide range of potential FMs in two different fresh-cut vegetables, namely, cabbage and green onion. The obtained results showed that the combined use of the multispectral fluorescence technique and color imaging can detect most FMs with accuracy > 95%; however, non-fluorescent tiny dark FMs could not be detected. The potential of the developed online detection system was further tested in an industrial environment by installation in a fresh-cut processing unit. All hardware parts were enclosed in a black box and the software program was modified to programmatically control conveyor belt movement and facilitate real-time image processing for FM visualization. FMs mixed with fresh-cut vegetables were manually fed onto the conveyor belt; fluorescence and color images of fresh-cut vegetables were acquired simultaneously while moving through the conveyor belt. The developed online system performed almost identically in the industrial environment, and has the potential to scan an area as large as 24 × 24 cm2 in approximately 1.5 s. Thus, its application in commercial setting may be feasible.

Orientation mapping of graphene using 4D STEM-in-SEM

A scanning diffraction technique is implemented in the scanning electron microscope. The technique, referred to as 4D STEM-in-SEM (four-dimensional scanning transmission electron microscopy in the scanning electron microscope), collects a diffraction pattern from each point on a sample which is saved to disk for further analysis. The diffraction patterns are collected using an on-axis lens-coupled phosphor/CCD arrangement. Synchronization between the electron beam and the camera exposure is accomplished with off-the-shelf data acquisition hardware. Graphene is used as a model system to test the sensitivity of the instrumentation and develop some basic analysis techniques. The data show interpretable diffraction patterns from monolayer graphene with integration times as short as 0.5 ms with a beam current of 245 pA (7.65×105 incident electrons per pixel). Diffraction patterns are collected at a rate of ca. 100/s from the mm to nm length scales. Using a grain boundary as a ‘knife-edge’, the spatial resolution of the technique is demonstrated to be ≤5.6nm (edge-width 25 % to 75 %). Analysis of the orientation of the diffraction patterns yields an angular (orientation) precision of ≤0.19∘ (full width at half maximum) for unsupported monolayer graphene. In addition, it is demonstrated that the 4D datasets have the information content necessary to analyze complex and heterogeneous multilayer graphene films.

Phase contrast-based phase retrieval: a bridge between qualitative phase contrast and quantitative phase imaging by phase retrieval algorithms.

In the last five decades, iterative phase retrieval methods have drawn a lot of interest across the research community as a non-interferometric approach to recover quantitative phase distributions from one (or more) intensity measurement. However, in cases where a unique solution does exist, these methods often require oversampling and high computational resources, which limit the use of this approach in important applications. On the other hand, phase contrast methods are based on a single camera exposure, but provide only a qualitative description of the phase; thus, they are not useful for applications in which the quantitative phase description is needed. In this Letter, we establish a combined approach based on the two above-mentioned methods to overcome their respective drawbacks. We show that a modified phase retrieval algorithm easily converges to the correct solution by initializing the algorithm with a phase-induced intensity measurement, namely with a phase contrast image of the examined object. Accurate quantitative phase measurements for both binary and continuously varying phase objects are demonstrated to support the suggested system as a single-shot quantitative phase contrast microscope.

Three-Dimensional Shape Detection for Non Uniform Reflective Objects: Combination of Color Light Projection and Camera’s Exposure Adjustment

For non uniform reflective objects, a new three-dimensional (3-D) shape detection method that combines color light projection with camera exposure adjustment is proposed. First, the intensity of the projected fringes is set to maximum. Second, we combine red, green and blue light projection with the camera exposure adjustment to obtain fringe sequences with different intensity and contrast. Finally, the optimal set of fringes is extracted from raw fringes, and used for 3-D shape reconstruction. Experimental results show that the method can successfully reconstruct the 3-D shape of the specular and colored objects with minimum of MSE (mean square error) 0.125 mm.