Active Stereo Research Articles

Multiuser Stereoscopic Projection Techniques for CAVE-Type Virtual Reality Systems

Despite the development of increasingly popular head mounted displays, CAVE-type systems may still be considered one of the most immersive virtual reality systems with many advantages. However, a serious limitation of most CAVE-type systems is the generation of a three-dimensional (3-D) image from the perspective of only one person. This problem is significant because in some applications, the participants must cooperate with each other in the virtual world. This paper presents the adaptation of a one-user Cave Automatic Virtual Environment (CAVE) installation in the Immersive 3-D Visualization Lab at the Gdańsk University of Technology to a two-user stereoscopy system. Simultaneous use of two alternative one-user stereoscopies available in the I3DVL (a technique with spectrum separation—Infitec, and active stereo) and a simple electronic circuit have allowed us to transform the one-user stereoscopy CAVE installation to a two-user stereoscopic system. The experiments performed concentrated on several objective measurable parameters. The calculated crosstalk value was low, approximately 1%, which can be considered negligible and shows the proper operation of the proposed technique. Additionally, initial experiments based on the tested two-user application and related to user comfort in the developed two-user stereoscopy are discussed in this paper. However, this topic still needs further research. The proposed solutions are a cheap alternative to adapt the existing one-user CAVE-type systems which support two projection techniques to a two-user system.

REAL-TIME DEPTH MAP ESTIMATION FROM INFRARED STEREO IMAGES OF RGB-D CAMERAS

Abstract. RGB-D cameras are novel sensing systems that can rapidly provide accurate depth information for 3D perception, among which the type based on active stereo vision has been widely used. However, there are some problems exiting in use, such as the short measurement range and incomplete depth maps. This paper presents a robust and efficient matching algorithm based on semi-global matching to obtain more complete and accurate depth maps in real time. Considering characteristics of captured infrared speckle images, the Gaussian filter is performed firstly to restrain noise and enhance the relativity. It also adopts the idea of block matching for reliability, and a dynamic threshold selection of the block size is used to adapt to various situation. Moreover, several optimizations are applied to improve precision and reduce error. Through experiments on the Intel Realsense R200, the excellent capability of our proposed method is verified.

3D Sensors for Sewer Inspection: A Quantitative Review and Analysis.

Automating inspection of critical infrastructure such as sewer systems will help utilities optimize maintenance and replacement schedules. The current inspection process consists of manual reviews of video as an operator controls a sewer inspection vehicle remotely. The process is slow, labor-intensive, and expensive and presents a huge potential for automation. With this work, we address a central component of the next generation of robotic inspection of sewers, namely the choice of 3D sensing technology. We investigate three prominent techniques for 3D vision: passive stereo, active stereo, and time-of-flight (ToF). The Realsense D435 camera is chosen as the representative of the first two techniques wheres the PMD CamBoard pico flexx represents ToF. The 3D reconstruction performance of the sensors is assessed in both a laboratory setup and in an outdoor above-ground setup. The acquired point clouds from the sensors are compared with reference 3D models using the cloud-to-mesh metric. The reconstruction performance of the sensors is tested with respect to different illuminance levels and different levels of water in the pipes. The results of the tests show that the ToF-based point cloud from the pico flexx is superior to the output of the active and passive stereo cameras.

Probabilistic Multi-View Fusion of Active Stereo Depth Maps for Robotic Bin-Picking

The reliable fusion of depth maps from multiple viewpoints has become an important problem in many 3D reconstruction pipelines. In this work, we investigate its impact on robotic bin-picking tasks such as 6D object pose estimation. The performance of object pose estimation relies heavily on the quality of depth data. However, due to the prevalence of shiny surfaces and cluttered scenes, industrial grade depth cameras often fail to sense depth or generate unreliable measurements from a single viewpoint. To this end, we propose a novel probabilistic framework for scene reconstruction in robotic bin-picking. Based on active stereo camera data, we first explicitly estimate the uncertainty of depth measurements for mitigating the adverse effects of both noise and outliers. The uncertainty estimates are then incorporated into a probabilistic model for incrementally updating the scene. To extensively evaluate the traditional fusion approach alongside our own approach, we will release a novel representative dataset with multiple views for each bin and curated parts. Over the entire dataset, we demonstrate that our framework outperforms a traditional fusion approach by a 12.8% reduction in reconstruction error, and 6.1% improvement in detection rate. The dataset will be available at https://www.trailab.utias.utoronto.ca/robi .

A portable three-component displacement measurement technique using an unmanned aerial vehicle (UAV) and computer vision: A proof of concept

This study proposes a new remote sensing technique to measure three-component (3C) dynamic displacement of three-dimensional (3D) structures. A sensing system with a UAV platform and contact-free sensors (e.g., optical and infrared (IR) cameras) is employed to provide a portable and convenient alternative to conventional approaches that require sensor installation on a structure. The original contributions of this study include (1) integrating both optical and IR cameras with a UAV platform to measure dynamic structural response, and (2) developing new data post-processing algorithms (based on target identification, Direct Linear Transformation, and active stereo vision) to simultaneously extract the 3C displacement of a 3D structure from optical and IR videos, which presents a unique advantage compared to the existing UAV-based displacement measurement techniques that allow the measurements in only one or two directions using optical cameras or laser sensors. The efficacy of the proposed technique is validated through laboratory experiments.

A Calibration and 3-D Measurement Method for an Active Vision System With Symmetric Yawing Cameras

In this study, we propose a 3-D measurement method for an active stereo vision system with two symmetric yawing cameras. First, we establish a measurement model with five constant systematic parameters and five variables. Second, we present a design for a relatively simple and convenient calibration method to determine the constant systematic parameters for the active vision system. In our proposed method, the initial yaw angles of two cameras are calibrated with a minimum of three equal interval points in a common field of view (FOV). The initial pitch angles are calibrated with at least one point in this common FOV. The initial yaw angles are used in the following calibration processes of the baseline length and the optical offset. The baseline length is calibrated with respect to two such points with known horizontal distance. An offset used to compensate for the difference between the actual optical centers of the two cameras along their optical axes is also computed with at least one feature point with the cameras yawed at different angles. Third, an associated measurement method for objects in the common FOV is presented. The proposed method computes 3-D positions of feature points in the common FOV using these five constant systematic parameters and five variables. This active stereo vision system design does not require repeated calibration for varying yaw of the two cameras to conduct measurements once the five constant systematic parameters have been calibrated offline. We conduct comparative experiments, and the results verify the effectiveness of the proposed method.

The Use of Digital Image Art under Visual Sensing Technology for Art Education

The traditional art education model often cannot achieve good teaching results due to the development of digital technology. Art is integrated with technology based on three‐dimensional (3D) panoramic vision sensing technology, so as to study the application of the combination of visual sensing technology and digital image art in the field of art education. With the application of projection virtual game “Whac‐A‐Mole” as an example, the integration of panoramic color volume structured light generation technology with single emission point and omnidirectional visual imaging technology with single view is proposed; the mathematical model of the active vision system is established, and the camera model, projector model, object surface illumination model, and their relationship are studied. On this basis, the mathematical relationship between the color of the projected light source and the color of the corresponding imaging point is proposed; a light source color correction algorithm based on the two‐color reflection model is proposed. The object surface color is corrected by three‐channel reflectance after the light source color correction algorithm is used for correction. The results suggest that the 3D panoramic vision sensing technology makes the current active stereo vision develop from the visual perception of surface structure to the visual perception of volume structure and reduces the error rate of recognition from 44% to 11%. Moreover, the algorithm does not need to know the coding method of the projected light source and the surface material of the object in advance. The scene teaching combined with technology overcomes the limitations of the teaching site and brings students into a completely realistic teaching situation. The presentation of digital image art using visual sensing technology can inspire children’s imagination, combine education and games, and perform edutainment. Thereby, the application research of digital image art based on 3D panoramic visual sensing technology has an irreplaceable development prospect in the field of art education.

TSFE-Net: Two-Stream Feature Extraction Networks for Active Stereo Matching

In this paper, we propose TSFE-Net, two stream feature extraction networks for active stereo matching. First, we perform extra local contrast normalization (LCN) for dataset due to dependency between speckle intensity and distance. Second, we construct two stream feature extraction layers which consist of convolutional layers and deconvolutional layers in different scales to simultaneously learn the features of the original images and LCN images and aggregate context information to form the left and right features. Third, we convert the obtained depth map into disparity map in virtue of camera parameters to construct a supervised learning model. The TSFE-Net not only solves illumination effects between speckle intensity and distance but also reserves details of the original image. Our dataset are captured by RealSense D435 camera. We research extensive quantitative and qualitative evaluations based on a series of scenes, and achieve the end point error (EPE) accuracy of 0.335 on the TITAN XP platform only for valid pixel. The assessment results show that our network has the ability of real-time deep reconstruction for active pattern.

Fast heuristic method to detect people in frontal depth images

This paper presents a new method for detecting people using only depth images captured by a camera in a frontal position. The approach is based on first detecting all the objects present in the scene and determining their average depth (distance to the camera). Next, for each object, a 3D Region of Interest (ROI) is processed around it in order to determine if the characteristics of the object correspond to the biometric characteristics of a human head. The results obtained using three public datasets captured by three depth sensors with different spatial resolutions and different operation principle (structured light, active stereo vision and Time of Flight) are presented. These results demonstrate that our method can run in realtime using a low-cost CPU platform with a high accuracy, being the processing times smaller than 1 ms per frame for a 512 × 424 image resolution with a precision of 99.26% and smaller than 4 ms per frame for a 1280 × 720 image resolution with a precision of 99.77%.

RGB Line Pattern-Based Stereo Vision Matching for Single-Shot 3-D Measurement

Active stereo vision has been used for real-time measurement in both academic and industrial fields. Currently, it is still very challenging for active stereo vision to achieve the continuous and robust measurement of a dynamic object due to the lack of a robust pattern extraction and matching method. In this article, we introduce a novel red green blue (RGB) line pattern with coarse-to-fine features in which the intervals between the adjacent green lines are largest and the intervals between the adjacent red lines are smallest. The large interval represents the coarse feature of the pattern and the small interval represents the fine feature of the pattern. Accordingly, we propose a matched pixel difference modeling (MPDM) method to model the matching relationship between two camera views based on the designed pattern. The line pattern is extracted in the hue saturation value (HSV) color space by slope difference distribution (SDD)-based threshold selection. The coarse green lines in two camera views are matched according to the minimum distance principle after the centroid-based alignment. The matching relationship is then modeled based on the matched green lines by least squares (LS) method. With the matching model, the fine blue lines and the fine red lines are matched successively. After stereo matching, the 3-D point on the measured object is computed by the ray intersection method. Experimental results showed that the proposed method is robust in measuring the dynamic objects by single-shot.

200×200 µm2 structured light source.

3D structured illumination is important in high-speed 3D metrology where beam patterns are roughly categorized into multi-dot and fringe patterns. For example, large-scale multi-dot patterns are utilized for facial recognition in an iPhone X on the basis of an active stereo method, while fringe patterns are utilized in Grey code patterns or fringe projection profilometry including Fourier transform profilometry and the phase shifting profilometry, which is suitable for high-resolution measurement. Among these applications, the light sources include a combination of vertical-cavity surface-emitting lasers (VCSELs) and diffractive optical elements (DOEs), a projector, and so on. Recently, we demonstrated static arbitrary two-dimensional beam patterns without a zero-order beam from needle-tip sized integrable spatial-phase-modulating surface-emitting lasers (iPMSELs). Due to their compactness (they are one order of magnitude smaller than DOE), surface-emitting device, lack of zero-order beam, and ease of switching the beam patterns electrically, iPMSELs will be suitable as an ultra-compact light source for 3D metrology that not only downsizes the conventional light source but also contributes to 3D inspections in narrow spaces such as dental and endoscope examinations. In this context, we have examined two beam patterns (multi-dot and fringe) both without a zero-order beam by using the iPMSELs. In the former, we have demonstrated projection of large-scale dot patterns of more than 10,000 points, which is the same order of magnitude as points in a practical device from a 200×200-µm2 emitter. Since the emitter has approximately 1 mega scattering points, this structure enables 1-mega-pixel images in the wavenumber space, which are comparable to the images of a typical projector emitting several-mega-pixel images from several tens of centimeters. In the latter, we successfully shifted the fringe patterns, which is vital to applying the phase shifting profilometry, despite the superposition of the conjugate ±1st order beam patterns.

P‐71: Concept of Stereoscopic Full‐Screen Resolution Flicker‐Free Imaging of Two Three‐Dimensional Scenes Simultaneously for Two Observers Using a Single 120 Hz Amplitude‐Polarization Imager

The described method provides stereoscopic imaging of two different three‐dimensional (3D) scenes in parallel for two observers without flicker at 100‐120 Hz frame rate using a single flat amplitude‐polarization imager (API). During each of two working subcycles, the API screen reproduces simultaneously a pair of full‐screen resolution view images of two 3D scenes with the help of information‐dependent light polarization coding of two image elements in each screen pixel. Two observers use two different sets of liquid crystal (LC) shutters with input polarizers serving as complementary polarization decoders. The shutters are implemented in the form of two active stereo glasses or two active distant binocular filters (DBF).

Active stereo vision three-dimensional reconstruction by RGB dot pattern projection and ray intersection

Active stereo vision is important in reconstructing objects without obvious textures. However, it is still very challenging to extract and match the projected patterns from two camera views automatically and robustly. In this paper, we propose a new pattern extraction method and a new stereo vision matching method based on our novel structured light pattern. Instead of using the widely used 2D disparity to calculate the depths of the objects, we use the ray intersection to compute the 3D shapes directly. Experimental results showed that the proposed approach could reconstruct the 3D shape of the object significantly more robustly than state of the art methods that include the widely used disparity based active stereo vision method, the time of flight method and the structured light method. In addition, experimental results also showed that the proposed approach could reconstruct the 3D motions of the dynamic shapes robustly.

Rotational Offset Microlens Arrays for Highly Efficient Structured Pattern Projection

AbstractThis article reports the diffractive optical elements of rotational offset microlens arrays for generating highly efficient structured light patterns in active stereoscopic 3D imaging. The double‐side configuration of microlens arrays increases the uniformity of diffraction patterns by using two successive refractions on microlens surfaces. In addition, the rotational offsets between the frontside and the backside microlens arrays with rectangular or hexagonal arrangements control the contrast and the density of structured dot array patterns. Two layers of microlens arrays with high curvature and high fill‐factor are microfabricated on both sides of the glass wafer by using ultrathin fluorocarbon encapsulation, thermal reflow, and parylene gap filling. The rotational offset angles of 22.5°, 28°, and 37° in rectangular and 13.25°, 21.75°, and 27.75° in hexagonal arrangements efficiently provide diffractive dot arrays with high uniformity, contrast, and density by minimizing the overlapping of diffractive patterns. The active stereo imaging using rotational offset microlens arrays successfully acquire high‐resolution depth map for 3D plaster objects with smooth and curved surfaces. The structured light pattern projection using rotational offset microlens arrays will provide promising opportunities for compact advanced imaging systems in industrial, medical, or military applications.

Active Stereo 3-D Surface Reconstruction Using Multistep Matching

Precise 3-D surface reconstruction plays an important role in automated manipulation, industrial inspection, robotics, and so on. In this article, we present a novel 3-D surface reconstruction framework for stereo vision systems assisted with structured light projection. In the framework, a multistep matching scheme is proposed to establish a reliable correspondence between image pairs with high computation efficiency and accuracy. The successive matching steps can find the most precise correspondence through a step-by-step filtering procedure. To further enhance the precision, a correspondence refinement algorithm is presented. Phase maps with different frequencies are utilized as the code words for the multistep matching due to their high encoding accuracy and robustness to noise. This method does not require phase unwrapping or projector calibration, which improves the reconstruction precision and simplifies the operation. Selection strategies for the number of matching steps, the pattern frequencies, and the matching threshold are proposed. Furthermore, various 3-D reconstruction experiments are conducted using the proposed framework. Comparative experiments verify the advantages of the proposed framework compared with existing 3-D reconstruction methods regarding the accuracy and precision. The adaptability to scenarios with different motion speeds is demonstrated. Robustness and limitations of the framework are also revealed by conducting experiments in challenging scenarios. Note to Practitioners —This article is motivated by the precise 3-D surface reconstruction problem in automated robotic systems. In different scenarios, such as the reconstruction of the static objects or moving objects, the errors induced by sensor noise and motion should be taken into consideration. To enhance the measurement precision under these occasions, selection of pattern number and fringe frequencies has been a problem. To overcome these problems, this article proposes a novel framework for active stereo 3-D surface reconstruction. The framework utilizes multifrequency phase-shifting fringes to encode the reconstructed target. Then, a multistep matching method filters the candidates step by step to obtain the most precise corresponding pixel and avoid noise error accumulation. A refinement method is introduced to further improve the precision. Selection strategies of the number of matching steps, the fringe frequencies, and matching thresholds enable the 3-D reconstruction framework to be utilized on different occasions. In applications, limitations of the proposed method should be noted.

LiDAR-Camera Fusion for 3D Object Detection

This paper describes the development of a low-cost, lowpower, accurate sensor designed for precise, feedback control of an autonomous vehicle to a hitch. The solution that has been developed uses an active stereo vision system, combining classical stereo vision with a low cost, low power laser speckle projection system, which solves the correspondence problem experienced by classic stereo vision sensors. A third camera is added to the sensor for texture mapping. A model test of the hitching problem was developed using an RC car and a target to represent a hitch. A control system is implemented to precisely control the vehicle to the hitch. The system can successfully control the vehicle from within 35° of perpendicular to the hitch, to a final position with an overall standard deviation of 3.0 m m of lateral error and 1.5° of angular error.

Active Stereo Vision for Precise Autonomous Vehicle Control

The demand for object tracking (OT) applications has been increasing for the past few decades in many areas of interest: security, surveillance, intelligence gathering, and reconnaissance. Lately, newly-defined requirements for unmanned vehicles have enhanced the interest in OT. Advancements in machine learning, data analytics, and deep learning have facilitated the recognition and tracking of objects of interest; however, continuous tracking is currently a problem of interest to many research projects. This paper presents a system implementing a means to continuously track an object and predict its trajectory based on its previous pathway, even when the object is partially or fully concealed for a period of time. The system is composed of six main subsystems: Image Processing, Detection Algorithm, Image Subtractor, Image Tracking, Tracking Predictor, and the Feedback Analyzer. Combined, these systems allow for reasonable object continuity in the face of object concealment.

A Real-Time 3D Laparoscopic Imaging System: Design, Method, and Validation.

This paper aims to propose a 3D laparoscopic imaging system that can realize dense 3D reconstruction in real time. Based on the active stereo technique which yields high-density, accurate and robust 3D reconstruction by combining structured light and stereo vision, we design a laparoscopic system consisting of two image feedback channels and one pattern projection channel. Remote high-speed image acquisition and pattern generation lay the foundation for the real-time dense 3D surface reconstruction and enable the miniaturization of the laparoscopic probe. To enhance the reconstruction efficiency and accuracy, we propose a novel active stereo method by which the dense 3D point cloud is obtained using only five patterns, while most existing multiple-shot structured light techniques require [Formula: see text] patterns. In our method, dual-frequency phase-shifting fringes are utilized to uniquely encode the pixels of the measured targets, and a dual-codeword matching scheme is developed to simplify the matching procedure and achieve high-precision reconstruction. Compared with the existing structured light techniques, the proposed method shows better real-time efficiency and accuracy in both quantitative and qualitative ways. Ex-vivo experiments demonstrate the robustness of the proposed method to different biological organs and the effectiveness to lesions and deformations of the organs. Feasibility of the proposed system for real-time dense 3D reconstruction is verified in dynamic experiments. According to the experimental results, the system acquires 3D point clouds with a speed of 12 frames per second. Each frame contains more than 40,000 points, and the average errors tested on standard objects are less than 0.2mm. This paper provides a new real-time dense 3D reconstruction method for 3D laparoscopic imaging. The established prototype system has shown good performance in reconstructing surface of biological tissues.

A new way to calibrate an active optical stereo system consisting of a monocular camera and structured line illumination

A new way to calibrate an active optical stereo system consisting of a monocular camera and structured line illumination

Simultaneous shape and camera-projector parameter estimation for 3D endoscopic system using CNN-based grid-oneshot scan.

For effective in situ endoscopic diagnosis and treatment, measurement of polyp sizes is important. For this purpose, 3D endoscopic systems have been researched. Among such systems, an active stereo technique, which projects a special pattern wherein each feature is coded, is a promising approach because of simplicity and high precision. However, previous works of this approach have problems. First, the quality of 3D reconstruction depended on the stabilities of feature extraction from the images captured by the endoscope camera. Second, due to the limited pattern projection area, the reconstructed region was relatively small. In this Letter, the authors propose a learning-based technique using convolutional neural networks to solve the first problem and an extended bundle adjustment technique, which integrates multiple shapes into a consistent single shape, to address the second. The effectiveness of the proposed techniques compared to previous techniques was evaluated experimentally.