Catadioptric Stereo Research Articles

Catadioptric Stereo Optical Gas Imaging System for Scene Flow Computation of Gas Structures

The need for reducing greenhouse gas emissions from industrial facilities has motivated the development of new and more reliable techniques for gas leak localization and quantification. Optical gas imaging (OGI) systems offer a fast and intuitive approach by exploiting the absorption properties of gases, such as methane, in the infrared range. Based on this, several efforts have been made in the recent years for gaining additional quantitative information from the measurement data, e.g. gas concentration or gas velocity. In this work, a catadioptric stereo OGI system based on four planar mirrors and an OGI camera is proposed for computing spatio-temporal (also referred to as scene flow) information from stereo gas images. The proposed system represents a way to reduce the effects of temporal and radiometric differences and the cost and size of two-camera-based stereo OGI systems and leads to more reliable results under the tested conditions.

Online droplet monitoring in inkjet 3D printing using catadioptric stereo system

Inkjet 3D printing is becoming one of the most disruptive additive manufacturing technologies, due to its unique capability of precisely depositing micro-droplets of multi-functional materials. It has found widespread industrial applications in aerospace, energy and health areas by processing multi-functional metal-materials, nano-materials, and bio-materials. However, the current inkjet 3D printing system still suffers from a low production quality issue, due to low process reliability caused by the complex and dynamic droplet dispensing behavior. Due to the challenges in terms of efficiency, accuracy, and versatility, robust droplet monitoring and process inspection tools are still largely unavailable. To this end, a novel catadioptric stereo system is proposed for online droplet monitoring in an inkjet 3D printing process. In this system, a regular industrial CCD camera is coupled with a flat mirror and magnification lens system to capture the tiny droplet images to detect the droplet location in 3D space. A mathematical model is formulated to calculate the droplet location in 3D world space from 2D image space. A holistic hardware and software framework is constructed to evaluate the performance of the proposed system in terms of resolution, accuracy, efficiency, and versatility, both theoretically and experimentally. The results show that the proposed catadioptric stereo system can achieve single micron resolution and accuracy, which is one-order-of-magnitude higher than the 3D printing system itself. The proposed droplet location detection algorithm has low time complexity, and the detection efficiency can meet the online monitoring requirement. Multi-facet features including the droplet location and speed can be effectively detected by the presented technique. The proposed catadioptric stereo system is a promising online droplet monitoring tool and has tremendous potential to enable trustworthy quality assurance in inkjet 3D printing.

Edge points-based stereo matching approach for omnidirectional images

We address the problem of matching stereo-omnidirectional images acquired by a catadioptric stereo system. The proposed approach deals with omnidirectional stereo image sequences, which allows taking advantage of the temporal information in both rectification and stereo matching steps. First, spherical edge points detection is performed on the spherical images in order to compute the disparity map only on significant features, i.e., edge points. Second, a spherical rectification is performed on the three omnidirectional pairs that are involved in the disparity map computation at the current frame, such as the current pair, the current left image, and its preceding one, and the current right image and its preceding one. Third, dynamic programming-based stereo matching is applied in each of the three stereo pairs. The disparity maps computed at the preceding frame are exploited to deduce the disparity range that will be used by the dynamic programming. Fourth, the three disparity maps provided by the last step are combined together into one 360-deg disparity map. The proposed method has been tested on a high-resolution omnidirectional images captured with a stereo pair of catadioptric cameras, and the obtained results are satisfactory.

Real-Time Monocular Three-Dimensional Motion Tracking Using a Multithread Active Vision System

In this study, we developed a monocular stereo tracking system to be used as a marker-based, three-dimensional (3-D) motion capture system. This system aims to localize dozens of markers on multiple moving objects in real time by switching five hundred different views in 1 s. The ultrafast mirror-drive active vision used in our catadioptric stereo tracking system can accelerate a series of operations for multithread gaze control with video shooting, computation, and actuation within 2 ms. By switching between five hundred different views in 1 s, with real-time video processing for marker extraction, our system can function asJvirtual left and right pan-tilt tracking cameras, operating at 250/Jfps to simultaneously capture and processJpairs of 512 × 512 stereo images with different views via the catadioptric mirror system. We conducted several real-time 3-D motion experiments to capture multiple fast-moving objects with markers. The results demonstrated the effectiveness of our monocular 3-D motion tracking system.

Monocular Stereo Measurement Using High-Speed Catadioptric Tracking.

This paper presents a novel concept of real-time catadioptric stereo tracking using a single ultrafast mirror-drive pan-tilt active vision system that can simultaneously switch between hundreds of different views in a second. By accelerating video-shooting, computation, and actuation at the millisecond-granularity level for time-division multithreaded processing in ultrafast gaze control, the active vision system can function virtually as two or more tracking cameras with different views. It enables a single active vision system to act as virtual left and right pan-tilt cameras that can simultaneously shoot a pair of stereo images for the same object to be observed at arbitrary viewpoints by switching the direction of the mirrors of the active vision system frame by frame. We developed a monocular galvano-mirror-based stereo tracking system that can switch between 500 different views in a second, and it functions as a catadioptric active stereo with left and right pan-tilt tracking cameras that can virtually capture 8-bit color images each operating at 250 fps to mechanically track a fast-moving object with a sufficient parallax for accurate 3D measurement. Several tracking experiments for moving objects in 3D space are described to demonstrate the performance of our monocular stereo tracking system.

Optimization and Rapid Prototyping of Catadioptric Omnidirectional Stereo Sensors

Omnidirectional stereo imaging provides useful depth information for autonomous navigation. In principle, omnidirectional stereo images can be achieved at a low cost, using a single camera and two curved mirrors, but such systems are not widely deployed. Here we describe the optimization and rapid prototyping of a low-cost omnidirectional stereo sensor for a telepresence robot. We consider single-viewpoint and non-single-viewpoint designs. We present a new way of relaxing single-viewpoint constraints, while retaining the ability to vertically rectify images. We also present a method of optimizing the resulting design to minimize depth errors. However, we show that despite these steps, nonsingle-viewpoint designs produce stereo disparities over a more practical range of distances. The lack of a single viewpoint could potentially introduce distortions that affect stereo matching, but these distortions are removed by projecting rays through the mirror geometry. We also describe a new method for rapid prototyping of curved mirrors using 3D printing and vacuum forming.

Catadioptric Stereo Tracking for Three Dimensional Shape Measurement of MRI Guided Catheters.

The recent introduction of Magnetic Resonance Imaging (MRI)-actuated steerable catheters lays the ground work for increasing the efficacy of cardiac catheter procedures. The MRI, while capable of imaging the catheter for tracking and control, does not fulfill all of the needs required to identify and develop a complete catheter model. Specifially, the frequency response of the catheter must be identified to ensure stable control of the catheter system. This requires a higher frequency imaging than the MRI can achieve. This work uses a catadioptric stereo camera system consisting of a mirror and a single camera in order to track a MRI actuated catheter inside a MRI machine. The catadioptric system works in parallel to the MRI and is capable of recording the catheter at 60 fps for post processing. The accuracy of the catadioptric system is verified in imaging conditions that would be found inside the MRI. The stereo camera is then used to track a catheter as it is actuated inside the MRI.

Single Camera Stereo with Planar Mirrors

This paper presents a new system for rapidly acquiring stereo images using a single camera and a pair of planar mirrors (catadioptric stereo). Firstly, the camera used to capture images is calibrated with matlab toolbox. Secondly, the position and pose of the planar mirrors relative to the fixed, calibrated camera is estimated, and this procedure is accomplished by calculating the symmetry plane of the real and reflected image corners of a chessboard. Thirdly, the relative orientation of two reflected virtual cameras is obtained. Finally, Gaussian noise is added to the image corners of the chessboard to verify the performance of the established stereo system. Experimental results show the effectiveness and robustness of our system.

Fundamental matrix of planar catadioptric stereo systems

Planar mirror has been used successfully in catadioptric stereo systems to capture stereo images with a single camera. These planar catadioptric stereo (PCS) systems not only provide radiometric advantages over traditional two-camera stereo, but also reduce the complexity and cost of acquiring stereoscopic video. Although many papers have been published on the design of the PCS system, few studies have been conducted for the analysis of its epipolar geometry, that is, the fundamental matrix. In this study, the authors investigated features of two types of PCS system and verified that they both can be approximated by affine epipolar geometry. This affine property is either because of constraints on the structure of the PCS system or because of the narrow field of view (FOV) of the mirror. This affine model reduces the number of parameters in the fundamental matrix from seven in the conventional stereo system to only four in the PCS system. Experimental results based on various criteria verify that by using the affine model with fewer parameters, estimation of the fundamental matrix for a PCS system can be more robust, precise and much easier to implement. These significant advantages confirm the usefulness of an affine fundamental matrix model for the PCS systems.

3D Slit Scanning with Planar Constraints★

Abstract We present a planarity constraint and a novel three‐dimensional (3D) point reconstruction algorithm for a multiview laser range slit scanner. The constraint is based on the fact that all observed points on a projected laser line lie on the same plane of laser light in 3D. The parameters of the plane of laser light linearly parametrize a homography between a pair of images of the laser points. This homography can be recovered from point correspondences derived from epipolar geometry. The use of the planar constraint reduces outliers in the reconstruction and allows for the reconstruction of points seen in only one view. We derive an optimal reconstruction of points subject to the planar constraint and compare the accuracy to the suboptimal approach in prior work. We also construct a catadioptric stereo rig with high quality optical components to remove error due to camera synchronization and non‐uniform laser projection. The reconstruction results are compared to prior work that uses inexpensive optics and two cameras.

Epipolar geometry of catadioptric stereo systems with planar mirrors

In order to simplify the design and implementation of a stereo vision system, catadioptric instruments have been used to capture stereo images with a single camera. These catadioptric stereo systems not only provide radiometric advantages over traditional two-camera stereo, but also reduce the complexity and cost of acquiring stereoscopic video. Although much research has been done on the design of the catadioptric stereo system, little attention has been paid to analyze the planar catadioptric stereo (PCS) system based on the epipolar geometry. In this paper, we investigated characteristics of a selected PCS system and proved that it can be approximated by affine epipolar geometry. This affine model reduces the number of parameters in the fundamental matrix from seven in the conventional stereo system to only four in the PCS system. Experimental results verify that estimation of the fundamental matrix for a PCS system can be more robust, precise, and much easier to implement with the affine model. Furthermore, rectification of the image pair based on the affine fundamental matrix can achieve better performance with much less geometric distortion. These significant advantages confirm the usefulness of an affine fundamental matrix model for the selected PCS system.

Rectification of Stereoscopic Video for Planar Catadioptric Stereo Systems

Planar mirror has been used successfully in the catadioptric stereo systems to capture image pairs with only a single camera. These planar catadioptric stereo (PCS) systems not only provide radiometric advantages over traditional two camera stereo, but also reduce the complexity and cost of acquiring stereoscopic video. One problem with these PCS sensors is that image pairs taken by them are usually not rectified and can not be viewed directly or compressed efficiently. Although several conventional methods have been suggested for rectification, features of the PCS system are not employed to optimize the design and rectified results are not evaluated. In this paper, we investigated features of the PCS system and found that epipolar geometries of certain systems can be approximated by affine. On the basis of this affine property, an effective algorithm is proposed for rectifying image pairs they acquired. This new approach reduces the number of parameters to be estimated from 11 in the conventional method to only four. The greatly reduced number of parameters makes the new method more robust and easier to implement. A PCS system was selected for evaluation and experimental results showed that the proposed method not only outperformed the conventional approaches but also worked quite well both visually and quantitatively for all the image pairs under test

Stereoscopic demultiplexing of field-sequential video

Much research has been devoted to investigating the generation of stereoscopic video based on catadioptric stereo systems. These systems, which utilize a proper combination of mirrors and lens, can capture stereoscopic video using only one camera. The left and right images are usually multiplexed to generate an interlaced video where each image corresponds to one field. This field-sequential format unavoidably reduces vertical resolution of the image pair by half. We proposed a notion called stereoscopic demultiplexing (SD) to recover the original resolution of the image pairs from the field-sequential video. To increase the sharpness of the image and avoid jagged artifacts, interpolation along the edge direction is emphasized. Because the SD problem is similar to deinterlacing, several methods previously used in deinterlacing were evaluated, and a new edge adaptive interpolation (EAI) approach was proposed. This new method can estimate edge direction with higher accuracy and obtain better image quality. Several experimental results show that the proposed algorithm outperforms the other popular deinterlacing methods both visually and quantitatively. Since resolution of the display technology is higher and better than that of current catadioptric stereo systems, we believe that further improvement in the resolution of the acquisition device by software is necessary to make 3-D applications more economical and popular.

Panoramic stereo reconstruction using non-SVP optics

Omni-directional sensors are useful in obtaining a 360° field of view with a single lens camera. Omni-directional stereo imaging systems can be constructed using a single camera and a mirror consisting of two concentric, radially symmetric lobes. If the central camera-mirror axis is vertical, a stereo image containing imagery from all azimuth directions from two viewpoints can be captured within one image. A vertically oriented catadioptric optical system that employs a mirror with a radial profile other than parabolic or hyperbolic cannot maintain the straightness of non-vertical lines. However, non-parabolic or hyperbolic mirror profiles are desirable for resolution distribution, sensor size, and cost. These mirror shapes, including spherical mirrors, are said to be non-central or non-single view-point (SVP); without a virtual perspective point, they pose new feature extraction challenges. The projection of straight lines in the world do not map to straight lines on the pixel array for non-SVP panoramic images, nor in the general case map to any conventional conic. A method for processing the imagery from a panoramic non-SVP catadioptric stereo sensor to reconstruct a 3D model of polyhedral objects with horizontal and vertical line edges is introduced. Horizontal line segments are extracted using the Panoramic Hough Transform and vertical line segments are recognized as straight radial lines. These segments, and closed shapes they form, are matched between the two lobe views to estimate position in three-dimensions. A novel Hough transform technique is described, and practical considerations for its successful use are explored. Four general weaknesses with Hough transform techniques in general are identified, and solutions to address them in each case are given. Details of a complete system using the Panoramic Hough Transform are given to demonstrate where this transform must be supplemented with other methods for robust operation. Issues of reconstruction accuracy are addressed, and experimental results with synthetic and real images are presented.

Rectified catadioptric stereo sensors

It has been shown elsewhere how mirrors can be used to capture stereo images with a single camera, an approach termed catadioptric stereo. We present novel catadioptric sensors that use mirrors to produce rectified stereo images. The scanline correspondence of these images benefits real-time stereo by avoiding the computational cost and image degradation due to resampling when rectification is performed after image capture. First, we develop a theory which determines the number of mirrors that must be used and the constraints on those mirrors that must be satisfied to obtain rectified stereo images with a single camera. Then, we discuss in detail the use of both one and three mirrors. In addition, we show how the mirrors should be placed in order to minimize sensor size for a given baseline, an important design consideration. In order to understand the feasibility of building these sensors, we analyze rectification errors due to misplacement of the camera with respect to the mirrors.

Catadioptric Stereo Using Planar Mirrors

By using mirror reflections of a scene, stereo images can be captured with a single camera (catadioptric stereo). In addition to simplifying data acquisition single camera stereo provides both geometric and radiometric advantages over traditional two camera stereo. In this paper, we discuss the geometry and calibration of catadioptric stereo with two planar mirrors. In particular, we will show that the relative orientation of a catadioptric stereo rig is restricted to the class of planar motions thus reducing the number of external calibration parameters from 6 to 5. Next we derive the epipolar geometry for catadioptric stereo and show that it has 6 degrees of freedom rather than 7 for traditional stereo. Furthermore, we show how focal length can be recovered from a single catadioptric image solely from a set of stereo correspondences. To test the accuracy of the calibration we present a comparison to Tsai camera calibration and we measure the quality of Euclidean reconstruction. In addition, we will describe a real-time system which demonstrates the viability of stereo with mirrors as an alternative to traditional two camera stereo.