Stereo Image Sequences Research Articles

Disparity Surface Reconstruction Based on a Stereo Light Microscope and Laser Fringes.

Microscopic vision systems based on a stereo light microscope (SLM) are used in microscopic measuring fields. Conventional measuring methods output the disparity surface based on stereo matching methods; however, these methods require that stereo images contain sufficient distinguishing features. Moreover, matching results typically contain many mismatched points. This paper presents a novel method for disparity surface reconstruction by combining an SLM and laser measuring techniques. The surfaces of small objects are scanned by a laser fringe, and a stereo image sequence containing laser stripes is obtained. The central contours of the laser stripes are extracted, and central contours are derived for alignment. A disparity coordinate system is then defined and used to analyze the relationship between the motion direction and reference plane. Next, the method of aligning disparity contours is proposed. The results show that our method can achieve a precision of ±0.5 pixels and that the real and measured shapes described by the disparity surface are consistent based on our method. Our method is confirmed to perform much better than the conventional block-matching method. The disparity surface output obtained by our method can be used to measure the surface profiles of microscopic objects accurately.

Edge points-based stereo matching approach for omnidirectional images

We address the problem of matching stereo-omnidirectional images acquired by a catadioptric stereo system. The proposed approach deals with omnidirectional stereo image sequences, which allows taking advantage of the temporal information in both rectification and stereo matching steps. First, spherical edge points detection is performed on the spherical images in order to compute the disparity map only on significant features, i.e., edge points. Second, a spherical rectification is performed on the three omnidirectional pairs that are involved in the disparity map computation at the current frame, such as the current pair, the current left image, and its preceding one, and the current right image and its preceding one. Third, dynamic programming-based stereo matching is applied in each of the three stereo pairs. The disparity maps computed at the preceding frame are exploited to deduce the disparity range that will be used by the dynamic programming. Fourth, the three disparity maps provided by the last step are combined together into one 360-deg disparity map. The proposed method has been tested on a high-resolution omnidirectional images captured with a stereo pair of catadioptric cameras, and the obtained results are satisfactory.

Subjective quality assessment of stereo image compression based on stereoscopic fusion in binocular vision

Stereo image transmission needs a nominal doubling of the bandwidth of the video signal, but the additional savings in bandwidth can be obtained by utilizing properties of the human visual system. Embarking from human visual perception and binocular stereo imaging mechanism, this study proposes a new subjective assessment method of the quality of stereo image compression on the basis of the characteristics of human binocular vision and the psychophysics of stereoscopic fusion. Specifically, the present study explored the response of the human visual system to mixed-compressed stereo image sequences, in which one eye view was spatially and temporally compressed but the depth perception was unaffected. This method exploited the correlation between the left and right frames of the given sequence that can not only help achieving certain efficiency of stereo image transmission but also facilitate the consistency of the directly intuitive output evaluation value and the sensitivity of stereoscopic perception, which could also overcome the shortcomings of the traditional subjective quality evaluation method that restricted by subjective rating system, exhibits good feasibility and effectiveness.

Linear SFM: A hierarchical approach to solving structure-from-motion problems by decoupling the linear and nonlinear components

This paper presents a novel hierarchical approach to solving structure-from-motion (SFM) problems. The algorithm begins with small local reconstructions based on nonlinear bundle adjustment (BA). These are then joined in a hierarchical manner using a strategy that requires solving a linear least squares optimization problem followed by a nonlinear transform. The algorithm can handle ordered monocular and stereo image sequences. Two stereo images or three monocular images are adequate for building each initial reconstruction. The bulk of the computation involves solving a linear least squares problem and, therefore, the proposed algorithm avoids three major issues associated with most of the nonlinear optimization algorithms currently used for SFM: the need for a reasonably accurate initial estimate, the need for iterations, and the possibility of being trapped in a local minimum. Also, by summarizing all the original observations into the small local reconstructions with associated information matrices, the proposed Linear SFM manages to preserve all the information contained in the observations. The paper also demonstrates that the proposed problem formulation results in a sparse structure that leads to an efficient numerical implementation. The experimental results using publicly available datasets show that the proposed algorithm yields solutions that are very close to those obtained using a global BA starting with an accurate initial estimate. The C/C++ source code of the proposed algorithm is publicly available at https://github.com/LiangZhaoPKUImperial/LinearSFM.

Temporally Consistent Road Surface Profile Estimation Using Stereo Vision

Road surface profile (RSP) estimation is an important task to find the imperfections of a road surface, thereby improving ride quality. The RSP estimation has been recently studied using stereo vision owing to its affordable price. However, the existing methods provide noisy and temporally unstable results for real-world driving scenes because of noisy range measurements, noisy pitch angle estimates between the camera and the road surface, and interference from obstacles. This paper proposes a novel method for temporally consistent and robust RSP estimation to overcome these problems. The proposed method consists of three steps: free space estimation, digital elevation map (DEM) estimation, and RSP estimation. We first estimate a drivable area, i.e., free space. For robust and fast free space estimation, we propose an optimization-based non-parametric road surface modeling method and an integral disparity histogram-based free space estimation method. Then, the DEM of the road surface is estimated using range measurements on the free space to avoid obstacle interference. The DEM is temporally updated every frame using the moving average filter and the DEM reference grid update scheme. Owing to these strategies, the proposed method reduces the elevation estimation noise and the pitch angle error, and therefore provides the temporally consistent RSP. We demonstrate the superiority of the proposed method experimentally using stereo image sequences captured in real-world driving scenes.

Frequency-Domain Joint Motion and Disparity Estimation Using Steerable Filters

In this paper, the problem of joint disparity and motion estimation from stereo image sequences is formulated in the spatiotemporal frequency domain, and a novel steerable filter-based approach is proposed. Our rationale behind coupling the two problems is that according to experimental evidence in the literature, the biological visual mechanisms for depth and motion are not independent of each other. Furthermore, our motivation to study the problem in the frequency domain and search for a filter-based solution is based on the fact that, according to early experimental studies, the biological visual mechanisms can be modelled based on frequency-domain or filter-based considerations, for both the perception of depth and the perception of motion. The proposed framework constitutes the first attempt to solve the joint estimation problem through a filter-based solution, based on frequency-domain considerations. Thus, the presented ideas provide a new direction of work and could be the basis for further developments. From an algorithmic point of view, we additionally extend state-of-the-art ideas from the disparity estimation literature to handle the joint disparity-motion estimation problem and formulate an algorithm that is evaluated through a number of experimental results. Comparisons with state-of-the-art-methods demonstrate the accuracy of the proposed approach.

Dynamic Reconstruction of Deformable Soft-Tissue With Stereo Scope in Minimal Invasive Surgery

In minimal invasive surgery, it is important to rebuild and visualize the latest deformed shape of soft-tissue surfaces to mitigate tissue damages. This paper proposes an innovative Simultaneous Localization and Mapping (SLAM) algorithm for deformable dense reconstruction of surfaces using a sequence of images from a stereoscope. We introduce a warping field based on the Embedded Deformation (ED) nodes with 3D shapes recovered from consecutive pairs of stereo images. The warping field is estimated by deforming the last updated model to the current live model. Our SLAM system can: (1) Incrementally build a live model by progressively fusing new observations with vivid accurate texture. (2) Estimate the deformed shape of unobserved region with the principle As-Rigid-As-Possible. (3) Show the consecutive shape of models. (4) Estimate the current relative pose between the soft-tissue and the scope. In-vivo experiments with publicly available datasets demonstrate that the 3D models can be incrementally built for different soft-tissues with different deformations from sequences of stereo images obtained by laparoscopes. Results show the potential clinical application of our SLAM system for providing surgeon useful shape and texture information in minimal invasive surgery.

Scene Flow Estimation Based on Adaptive Anisotropic Total Variation Flow-Driven Method

Scene flow estimation based on disparity and optical flow is a challenging task. We present a novel method based on adaptive anisotropic total variation flow-driven method for scene flow estimation from a calibrated stereo image sequence. The basic idea is that diffusion of flow field in different directions has different rates, which can be used to calculate total variation and anisotropic diffusion automatically. Brightness consistency and gradient consistency constraint are employed to establish the data term, and adaptive anisotropic flow-driven penalty constraint is employed to establish the smoothness term. Similar to the optical flow estimation, there are also large displacement problems in the estimation of the scene flow, which is solved by introducing a hierarchical computing optimization. The proposed method is verified by using the synthetic dataset and the real scene image sequences. The experimental results show the effectiveness of the proposed algorithm.

Multiple moving targets positioning via local trajectory consistency using stereo geostationary infrared image sequences

Multiple moving targets positioning under observation by stereo infrared geostationary satellites is a challenging task. Due to the poor texture information of infrared (IR) images, as well as the difficulties of synchronous control and the continuous movement of the targets, the traditional photogrammetric stereo positioning methods are no longer effective. This paper introduces a new approach which considers the local trajectory consistency in solving the positioning problem of moving targets that follow a certain dynamic principle under the continuous observation by stereo IR geostationary satellites. The new method first detects the point targets from the IR image sequences by a widely used method based on a modified partial differential equation. Then, the constraints of the epipolar lines and the imaging times of the corresponding points are combined to match the corresponding trajectories. Finally, the image coordinates of the targets are obtained from the parametric function and their three dimensional coordinates are easily computed by forward intersection. Experimental results confirmed that the proposed method can achieve better positioning accuracy compared to the direct forward intersection method even when the stereo image sequences are synchronously observed. At the same time, the proposed method was found to be capable of effectively coping with the asynchronous situation.

Piecewise-Planar StereoScan: Sequential Structure and Motion Using Plane Primitives.

The article describes a pipeline that receives as input a sequence of stereo images, and outputs the camera motion and a Piecewise-Planar Reconstruction (PPR) of the scene. The pipeline, named Piecewise-Planar StereoScan (PPSS), works as follows: the planes in the scene are detected for each stereo view using semi-dense depth estimation; the relative pose is computed by a new closed-form minimal algorithm that only uses point correspondences whenever plane detections do not fully constrain the motion; the camera motion and the PPR are jointly refined by alternating between discrete optimization and continuous bundle adjustment; and, finally, the detected 3D planes are segmented in images using a new framework that handles low texture and visibility issues. PPSS is extensively validated in indoor and outdoor datasets, and benchmarked against two popular point-based SfM pipelines. The experiments confirm that plane-based visual odometry is resilient to situations of small image overlap, poor texture, specularity, and perceptual aliasing where the fast LIBVISO2 [1] pipeline fails. The comparison against VisualSfM+CMVS/PMVS [2] , [3] shows that, for a similar computational complexity, PPSS is more accurate and provides much more compelling and visually pleasant 3D models. These results strongly suggest that plane primitives are an advantageous alternative to point correspondences for applications of SfM and 3D reconstruction in man-made environments.

Measurement of Surface Changes in a Scaled-Down Landslide Model Using High-Speed Stereo Image Sequences

Construction of scaled-down landslide models is an important means for landslide study. The objective of this study is to develop an innovative non-contact photogrammetric system to meet the challenge for monitoring the fast surface deformation of a laboratory-simulated landslide, which can detect pre-failure events and the final failure, provide sectional and overall surface deformation patterns, and generate speed maps during the rapid slope failure. We proposed an event detector based on altered surface features in the slower slrc (single-lens-reflex camera) image sequence to detect pre-failure events, while a combined analysis tool used the surface velocity fields and deformation areas based on fast hscs (high-speed stereocamera system) stereo image sequences to reveal fast-changing landslide behavior during the short final failure. The introduced surface change detector uses the percentage of sliding block areas, percentage of changed features, and average speeds. It successfully detected four pre-failure local collapse events and the final slope failure; the extent of surface changes reached its maximum to accumulate energy 1.5seconds before the failure when the average speed of changed features achieved its peak of 0.8m/s. The developed system achieved a position accuracy of 3.8mm and a speed accuracy of 0.11m/s. The analysis result demonstrated a time period of 66minutes before the failure which is confirmed by significant signals from both imaging and contact sensors and is important for landslide early warning. A field implementation scheme in western China will be designed and realized in the near future.

Measurement of Surface Changes in a Scaled-Down Landslide Model Using High-Speed Stereo Image Sequences

Construction of scaled-down landslide models is an important means for landslide study. The objective of this study is to develop an innovative non-contact photogrammetric system to meet the challenge for monitoring the fast surface deformation of a laboratory-simulated landslide, which can detect pre-failure events and the final failure, provide sectional and overall surface deformation patterns, and generate speed maps during the rapid slope failure. We proposed an event detector based on altered surface features in the slower SLRC (single-lensreflex camera) image sequence to detect pre-failure events, while a combined analysis tool used the surface velocity fields and deformation areas based on fast HSCS (high-speed stereocamera system) stereo image sequences to reveal fast-changing landslide behavior during the short final failure. The introduced surface change detector uses the percentage of sliding block areas, percentage of changed features, and average speeds. It successfully detected four pre-failure local collapse events and the final slope failure; the extent of surface changes reached its maximum to accumulate energy 1.5 seconds before the failure when the average speed of changed features achieved its peak of 0.8 m/s. The developed system achieved a position accuracy of 3.8 mm and a speed accuracy of 0.11 m/s. The analysis result demonstrated a time period of 66 minutes before the failure which is confirmed by significant signals from both imaging and contact sensors and is important for landslide early warning. A field implementation scheme in western China will be designed and realized in the near future.

Study on clear stereo image pair acquisition method for small objects with big vertical size in SLM vision system.

Microscopic vision system with stereo light microscope (SLM) has been applied to surface profile measurement. If the vertical size of a small object exceeds the range of depth, its images will contain clear and fuzzy image regions. Hence, in order to obtain clear stereo images, we propose a microscopic sequence image fusion method which is suitable for SLM vision system. First, a solution to capture and align image sequence is designed, which outputs an aligning stereo images. Second, we decompose stereo image sequence by wavelet analysis theory, and obtain a series of high and low frequency coefficients with different resolutions. Then fused stereo images are output based on the high and low frequency coefficient fusion rules proposed in this article. The results show that Δw1 (Δw2 ) and ΔZ of stereo images in a sequence have linear relationship. Hence, a procedure for image alignment is necessary before image fusion. In contrast with other image fusion methods, our method can output clear fused stereo images with better performance, which is suitable for SLM vision system, and very helpful for avoiding image fuzzy caused by big vertical size of small objects.

Measurement of drag coefficients of non-spherical particles with a camera-based method

The current paper presents a novel experimental set-up which allows the automated determination of the drag coefficients of relatively large particles with complex shape. Typical examples of such types of particles are waste derived fuel (RDF) particles which are non-spherical and have a size up to a few centimeters. In contrast to conventional fossil fuel particles, where the particles may be considered as material points during the calculation of particle tracks in a reacting flow field, the spatial extent of RDF-particles and their lack of sphericity lead to pronounced self-induced movement and associated variations in the drag-coefficients.The experiments are based on a drop shaft equipped with two digital cameras. This allows to obtain time resolved stereo image sequences from which the settling velocity of particles, the self-induced velocity fluctuations and the corresponding drag and lift coefficients can be derived. As the system is automated, a large number of particles can be examined and statistical information on the distribution of drag coefficients can be obtained.In this publication, the methodology of these drop shaft measurements and their evaluation will be presented. Additionally drag coefficients of isometric spherical and non-spherical particle geometries (spheres, cubes, square plates and circular disks) were measured and compared with known correlations for drag coefficients. Probability density functions for the properties of typical RDF particles will be presented to highlight the potential of the new set-up.

Efficient Dense Rigid-Body Motion Segmentation and Estimation in RGB-D Video

Motion is a fundamental grouping cue in video. Many current approaches to motion segmentation in monocular or stereo image sequences rely on sparse interest points or are dense but computationally demanding. We propose an efficient expectation---maximization (EM) framework for dense 3D segmentation of moving rigid parts in RGB-D video. Our approach segments images into pixel regions that undergo coherent 3D rigid-body motion. Our formulation treats background and foreground objects equally and poses no further assumptions on the motion of the camera or the objects than rigidness. While our EM-formulation is not restricted to a specific image representation, we supplement it with efficient image representation and registration for rapid segmentation of RGB-D video. In experiments, we demonstrate that our approach recovers segmentation and 3D motion at good precision.

Modeling leaf growth of rosette plants using infrared stereo image sequences

In this paper, we present a novel multi-level procedure for finding and tracking leaves of a rosette plant, in our case up to 3 weeks old tobacco plants, during early growth from infrared-image sequences. This allows measuring important plant parameters, e.g. leaf growth rates, in an automatic and non-invasive manner. The procedure consists of three main stages: preprocessing, leaf segmentation, and leaf tracking. Leaf-shape models are applied to improve leaf segmentation, and further used for measuring leaf sizes and handling occlusions. Leaves typically grow radially away from the stem, a property that is exploited in our method, reducing the dimensionality of the tracking task. We successfully tested the method on infrared image sequences showing the growth of tobacco-plant seedlings up to an age of about 30days, which allows measuring relevant plant growth parameters such as leaf growth rate. By robustly fitting a suitably modified autocatalytic growth model to all growth curves from plants under the same treatment, average plant growth models could be derived. Future applications of the method include plant-growth monitoring for optimizing plant production in green houses or plant phenotyping for plant research.

Spacetime Stereo and 3D Flow via Binocular Spatiotemporal Orientation Analysis.

This paper presents a novel approach to recovering estimates of 3D structure and motion of a dynamic scene from a sequence of binocular stereo images. The approach is based on matching spatiotemporal orientation distributions between left and right temporal image streams, which encapsulates both local spatial and temporal structure for disparity estimation. By capturing spatial and temporal structure in this unified fashion, both sources of information combine to yield disparity estimates that are naturally temporal coherent, while helping to resolve matches that might be ambiguous when either source is considered alone. Further, by allowing subsets of the orientation measurements to support different disparity estimates, an approach to recovering multilayer disparity from spacetime stereo is realized. Similarly, the matched distributions allow for direct recovery of dense, robust estimates of 3D scene flow. The approach has been implemented with real-time performance on commodity GPUs using OpenCL. Empirical evaluation shows that the proposed approach yields qualitatively and quantitatively superior estimates in comparison to various alternative approaches, including the ability to provide accurate multilayer estimates in the presence of (semi)transparent and specular surfaces.

Dynamic Point Clustering with Line Constraints for Moving Object Detection in DAS

In this letter, we propose a robust dynamic point clustering method for detecting moving objects in stereo image sequences, which is essential for collision detection in driver assistance system. If multiple objects with similar motions are located in close proximity, dynamic points from different moving objects may be clustered together when using the position and velocity as clustering criteria. To solve this problem, we apply a geometric constraint between dynamic points using line segments. Based on this constraint, we propose a variable K-nearest neighbor clustering method and three cost functions that are defined between line segments and points. The proposed method is verified experimentally in terms of its accuracy, and comparisons are also made with conventional methods that only utilize the positions and velocities of dynamic points.

Determining the orientation of the observed object in threedimensional space using stereo vision methods

The task of matching image of an object with its template is central for many optoelectronic systems. Solution of the matching problem in three-dimensional space in contrast to the structural alignment in the image plane allows using a larger amount of information about the object for determining its orientation, which may increase the probability of correct matching. In the case of stereo vision methods for constructing a three-dimensional image of the object, it becomes possible to achieve invariance w.r.t. background and distance to the observed object. Only three of the orientation angle of the object relative to the camera are uncertain and require measurements. This paper proposes a method for determining the orientation angles of the observed object in three-dimensional space, which is based on the processing of stereo image sequences. Disparity map segmentation method that allows one to ensure the invariance of the background is presented. Quantitative estimates of the effectiveness of the proposed method are presented and discussed.

Congruence analysis of point clouds from unstable stereo image sequences

Abstract. This paper deals with the correction of exterior orientation parameters of stereo image sequences over deformed free-form surfaces without control points. Such imaging situation can occur, for example, during photogrammetric car crash test recordings where onboard high-speed stereo cameras are used to measure 3D surfaces. As a result of such measurements 3D point clouds of deformed surfaces are generated for a complete stereo sequence. The first objective of this research focusses on the development and investigation of methods for the detection of corresponding spatial and temporal tie points within the stereo image sequences (by stereo image matching and 3D point tracking) that are robust enough for a reliable handling of occlusions and other disturbances that may occur. The second objective of this research is the analysis of object deformations in order to detect stable areas (congruence analysis). For this purpose a RANSAC-based method for congruence analysis has been developed. This process is based on the sequential transformation of randomly selected point groups from one epoch to another by using a 3D similarity transformation. The paper gives a detailed description of the congruence analysis. The approach has been tested successfully on synthetic and real image data.