Stereo Image Sequences Research Articles

Localization of mobile robot in prior 3D LiDAR maps using stereo image sequence

Localization of mobile robot in prior 3D LiDAR maps using stereo image sequence

Damage Analysis and Quality Control of Carbon-Reinforced Concrete Beams Based on In Situ Computed Tomography Tests

Carbon-reinforced concrete (CRC) is increasingly utilized in construction, due to its unique properties, such as corrosion resistance, high-tensile strength, and durability. Understanding its behavior under different loads is crucial to ensuring its safe and effective use in various construction applications. In this study, three-point bending tests were performed in combination with large-scale in situ computed tomography (CT). This paper presents the related three- and four-dimensional evaluation methods, with emphasis on crack width and quality control. The focus was on large CRC beams, with cross-sectional sizes of up to 80 mm by 160 mm. Such dimensions require extremely high energy during a CT scan. Therefore, a new experimental setup with energies of up to 8 MeV was used in this study. However, such high energies posed new challenges to the analysis methods. Therefore, two methods (digital volume correlation and grayscale profile analysis) for accurate crack width estimation were adapted and applied to the 3D reconstructions. In addition, a photogrammetric stereo image sequence was acquired and analyzed, using digital image correlation to cross-validate the results derived from the 3D crack width estimates. The 3D CT images also played a key role in the quality control measures, including the localization of the carbon-reinforcement and the assessment of porosity within the concrete structure.

3D Displacement Measurement of Railway Bridge According to Cyclic Loads of Different Types of Railcars with Sequential Photogrammetry

In the early days of railroads in Korea, railway bridges were constructed as steel plate-girder structures, which are vulnerable to vibration and torsion. Many of these bridges have since been replaced with concrete-slab structures, which have high stability. Nevertheless, steel railway bridges still remain all over the country, and a lot of manpower and cost is being invested in the maintenance and repair of such bridges. Moreover, there have not been experimental analyses aiming to measure the cyclic loads that occur when a train enters. To ensure bridge safety, it is necessary to periodically inspect deformations. To this end, the present study proposed a sequential photogrammetric technique for measuring the deformation of a steel railway bridge for three types of railcars. Sequential stereo images of the bridge with multiple feature points are obtained using sequential photographing cameras, to determine the ground coordinates of each point as a function of time based on the space intersection from the relative orientation with coplanarity and the scale adjustment. All of these processes are performed through automated techniques using only the cameras and the targets. With this setup, the 3-dimensional dynamic motions of the bridge due to the cyclic loading of trains could be measured. In addition, the displacements by the proposed method were compared to those obtained with the 3D Laser tracker. The horizontal displacements errors did not exceed 0.5 mm and the vertical error was within 2.3 mm in root mean square error (RMSE) at camera-to-object distances of about 9 m.

Stereovision-Based Ego-Motion Estimation for Combine Harvesters.

Ego-motion estimation is a foundational capability for autonomous combine harvesters, supporting high-level functions such as navigation and harvesting. This paper presents a novel approach for estimating the motion of a combine harvester from a sequence of stereo images. The proposed method starts with tracking a set of 3D landmarks which are triangulated from stereo-matched features. Six Degree of Freedom (DoF) ego motion is obtained by minimizing the reprojection error of those landmarks on the current frame. Then, local bundle adjustment is performed to refine structure (i.e., landmark positions) and motion (i.e., keyframe poses) jointly in a sliding window. Both processes are encapsulated into a two-threaded architecture to achieve real-time performance. Our method utilizes a stereo camera, which enables estimation at true scale and easy startup of the system. Quantitative tests were performed on real agricultural scene data, comprising several different working paths, in terms of estimating accuracy and real-time performance. The experimental results demonstrated that our proposed perception system achieved favorable accuracy, outputting the pose at 10 Hz, which is sufficient for online ego-motion estimation for combine harvesters.

WHUVID: A Large-Scale Stereo-IMU Dataset for Visual-Inertial Odometry and Autonomous Driving in Chinese Urban Scenarios

In this paper, we present a challenging stereo-inertial dataset collected onboard a sports utility vehicle (SUV) for the tasks of visual-inertial odometry (VIO), simultaneous localization and mapping (SLAM), autonomous driving, object detection, and other computer vision techniques. We recorded a large set of time-synchronized stereo image sequences (2 × 1280 × 720 @ 30 fps RGB) and corresponding inertial measurement unit (IMU) readings (400 Hz) from a Stereolabs ZED2 camera, along with centimeter-level-accurate six-degree-of-freedom ground truth (100 Hz) from a u-blox GNSS-IMU navigation device with real-time kinematic correction signals. The dataset comprises 34 sequences recorded during November 2020 in Wuhan, the largest city of Central China. Further, the dataset contains abundant unique urban scenes and features of a complex modern metropolis, which have rarely appeared in previously released benchmarks. Results from milestone VIO/SLAM algorithms reveal that methods exhibiting excellent performance on established datasets such as KITTI and EuRoC perform unsatisfactorily when moved outside the laboratory to the real world. We expect our dataset to reduce this limitation by providing more challenging and diverse scenarios to the research community. The full dataset with raw and calibrated data is publicly available along with a lightweight MATLAB/Python toolbox for preprocessing and evaluation. The dataset can be downloaded in its entirety from the uniform resource locator (URL) we provide in the main text.

Air2Land: A deep learning dataset for unmanned aerial vehicle autolanding from air to land

AbstractIn this paper, a novel deep learning dataset, called Air2Land, is presented for advancing the state‐of‐the‐art object detection and pose estimation in the context of one fixed‐wing unmanned aerial vehicle autolanding scenarios. It bridges vision and control for ground‐based vision guidance systems having the multi‐modal data obtained by diverse sensors and pushes forward the development of computer vision and autopilot algorithms targeted at visually assisted landing of one fixed‐wing vehicle. The dataset is composed of sequential stereo images and synchronised sensor data, in terms of the flying vehicle pose and Pan‐Tilt Unit angles, simulated in various climate conditions and landing scenarios. Since real‐world automated landing data is very limited, the proposed dataset provides the necessary foundation for vision‐based tasks such as flying vehicle detection, key point localisation, pose estimation etc. Hereafter, in addition to providing plentiful and scene‐rich data, the developed dataset covers high‐risk scenarios that are hardly accessible in reality. The dataset is also open and available at https://github.com/micros‐uav/micros_air2land as well.The cover image is based on the Research Article Air2Land: A deep learning dataset for unmanned aerial vehicle autolanding from air to land by Tianjiang Hu et al., https://doi.org/10.1049/csy2.12045.

Quadruple tripatch-wise modular architecture-based real-time structure from motion

Structure from motion (SFM), a research hotspot in the intelligent transportation field (including autonomous driving, environmental perception and augmented reality (AR) for artificial intelligence terminals), can automatically recover ego-motion state estimations and 3D scene reconstructions from multiple images or video sequences. Most existing vision methods operate offline in indoor scenes, and their reconstruction accuracies greatly depend on the tracking lifetimes and accuracies of feature points. Reprojection matrix and redundant regression noise computations are exponential disasters for the calculation and reconstruction drift of large-scale scenes. This paper proposes a quadruple tripatch-wise modular architecture (QTMA) for autonomous vehicle stereo image sequences that decomposes rigid scenes into nonrigid motion-segmented pieces for reconstruction. An advanced energy function for salient image features is established by combining multiple feature types with weighted finite-element mesh. Closed quadruple annular matching and relocation are performed via multiresolution pyramid images. The proposed incremental integral-mapping calculation method and unique tree-like stacked storage containers prolong the tracking lifetimes of consecutive frames and ensure the spatiotemporal consistency and robustness of the homonymous image features in different subsequences. Experimental results verify the effectiveness of this architecture for different transportation scenes; the frame rate processing speed reaches 30 fps, the calculation accuracy regarding the path distance difference reaches 99.49%, and the estimation results regarding the maximum speeds of motion are closer to the ground truth. The translation error of the motion pose is 0.0136%, and the rotation error is 0.0035 [deg/m], which has more yaw stability than the existing state-of-the-art methods. Furthermore, in the reconstructed point cloud quality demonstration, the mean value of roughness is reduced by 40.127%, the mean value of density is improved by 27.701%, and the accuracy reaches 91.149% within a certain distance tolerance. This paper has significant theoretical research value and application potential for positioning, path tracking, and navigation in adaptive cruise control (ACC) and advanced driver assistance systems (ADAS).

Comprehensive observations on pendulum oscillation using stereo vision

We have developed a stereo vision system for observing real objects for experimental physics demonstration. The stereo vision observes objects from two different points of view. An observer can identify the distance and velocity of an object based on at least two different points of view. This paper demonstrates our comprehensive observation in a pendulum oscillation experiment to test the stereo vision system where a stereo camera is in a forward view position. The stereo camera produces stereo images. We use automatic object tracking techniques to obtain the 3D position of the pendulum in real space based on the sequential stereo images of the pendulum movement. The experimental results in a 3D graph that shows the trajectory of the pendulum oscillation. We confirm the common understanding of the oscillation movement by obtaining the graph of the pendulum oscillation as a function of time. We can obtain the oscillation motion parameters such as amplitude, angular frequency, frequency, period, and initial phase. Also, we can show the damping oscillation phenomena and the damping factor.

Spatiotemporal Correlation-Based Accurate 3D Face Imaging Using Speckle Projection and Real-Time Improvement

The reconstruction of 3D face data is widely used in the fields of biometric recognition and virtual reality. However, the rapid acquisition of 3D data is plagued by reconstruction accuracy, slow speed, excessive scenes and contemporary reconstruction-technology. To solve this problem, an accurate 3D face-imaging implementation framework based on coarse-to-fine spatiotemporal correlation is designed, improving the spatiotemporal correlation stereo matching process and accelerating the processing using a spatiotemporal box filter. The reliability of the reconstruction parameters is further verified in order to resolve the contention between the measurement accuracy and time cost. A binocular 3D data acquisition device with a rotary speckle projector is used to continuously and synchronously acquire an infrared speckle stereo image sequence for reconstructing an accurate 3D face model. Based on the face mask data obtained by the high-precision industrial 3D scanner, the relationship between the number of projected speckle patterns, the matching window size, the reconstruction accuracy and the time cost is quantitatively analysed. An optimal combination of parameters is used to achieve a balance between reconstruction speed and accuracy. Thus, to overcome the problem of a long acquisition time caused by the switching of the rotary speckle pattern, a compact 3D face acquisition device using a fixed three-speckle projector is designed. Using the optimal combination parameters of the three speckles, the parallel pipeline strategy is adopted in each core processing unit to maximise system resource utilisation and data throughput. The most time-consuming spatiotemporal correlation stereo matching activity was accelerated by the graphical processing unit. The results show that the system achieves real-time image acquisition, as well as 3D face reconstruction, while maintaining acceptable systematic precision.

Self-Supervised Monocular Depth Estimation With Extensive Pretraining

Although depth estimation is a key technology for three-dimensional sensing applications involving motion, active sensors such as LiDAR and depth cameras tend to be expensive and bulky. Here, we explore the potential of monocular depth estimation (MDE) using a self-supervised approach. MDE is a promising technology, but supervised learning suffers from a need for accurate ground-truth depth data. Recent studies have enabled self-supervised training on an MDE model with only monocular image sequences and image-reconstruction errors. We pretrained networks using multiple datasets, including monocular and stereo image sequences. The main challenges posed by the self-supervised MDE model were occlusions and dynamic objects. We proposed novel loss functions to handle these problems in the form of min-over-all and min-with-flow losses, both based on the per-pixel minimum reprojection error of Monodepth2 and extended to stereo images and optical flow. With extensive pretraining and novel losses, our model outperformed existing unsupervised approaches in quantitative depth estimation and the ability to distinguish small objects against a background, as evaluated by KITTI 2015.

Genetic algorithms and bundle adjustment for the enhancement of 3D reconstruction

In this paper, we present a new technique of tridimensional reconstruction from a sequence of uncalibrated stereo images taken with cameras having varying parameters. At first, our system allows to recover initial coordinates of a set of 3D points. In this context, we have used our method of self-calibration based on the use of unknown 3D scene with its image projections and genetic algorithms to estimate all intrinsic parameters. After that extrinsic parameters are estimated based on classical pose estimation algorithms. Matching points and estimated value of intrinsic and extrinsic parameters are used to estimate initial 3D model that helps us in the initialization step. In order to have a reliable and relevant 3D reconstruction the proposed method is based on good and new exploitation of bundle adjustment (without camera poses initialization) technique based on Levenberg-Marquardt optimization with the aim to estimate our optimal 3D model that has special features compared to the classical case because it masks the pose parameters estimation in the optimization process. Finally, 3D mesh of the 3D scene is constructed with Delaunay algorithm and the 2D image is projected on the 3D model to generate the texture mapping. Experiments is conducted on real data to achieve demonstrate the validity and the performance of the proposed approach in terms of convergence, simplicity, stability and reconstruction quality.

Reconstructing the Velocity and Deformation of a Rapid Landslide Using Multiview Video

AbstractNoncontact measurements of spatially varied ground surface deformation during landslide motion can provide important constraints on landslide mechanics. Here, we present and test a new method for extracting measurements of rapid landslide surface displacement and velocity (accelerations of approximately 1 m/s2) using sequences of stereo images obtained from a pair of inexpensive, stationary 4K video cameras with nominal frame rates of 29.97 Hz. The method combines elements of Structure from Motion with those of optical flow to extract data on 3‐D evolution of the ground surface during slope failure. We apply the method to an experiment at the U.S. Geological Survey debris‐flow flume in which a high‐speed, liquefying landslide was triggered by gradually adding water to a 6‐m3 prism of loosely packed sediment on a 31° slope. Strip‐scanning lidar measurements made during the experiment corroborate our video‐based measurements, but the latter encompassed the entire landslide surface and were much lower in cost. Our video‐based measurements enabled computation of depth‐integrated landslide dilation/contraction rates. The range of computed rates was within the ranges inferred from independent measurements of evolving pore water pressures and reasonable estimates of the hydraulic permeability of the sediment. Dilation and contraction rates play a crucial role in landslide mechanics. The dilation and contraction we observe contradict the incompressible flow assumption used in many studies that have employed noncontact methods to infer landslide properties.

Support vector machines based stereo matching method for advanced driver assistance systems

Stereo vision is a measurement method for finding correspondence between two or more input images in order to obtain a detailed 3D representation of a scene. This paper presents an approach for matching stereo sequences acquired by a stereo sensor mounted on an intelligent vehicle. The approach uses machine learning alongside spatio-temporal information to predict the matching results. This means that the obtained matching results during the previous frame are used as a training samples for a support vector machine classifier, as well as to derive disparity ranges for each scan-line, which are then used to predict the matching of the current frame. The distance to the hyper-plan computed by the SVM is used as a cost function to fill a 2D search space. Then, the dynamic programming algorithm is performed for matching edge points in the stereo pair. Experiments on both virtual and real stereo image sequences have been conducted, demonstrating satisfactory performance.

Unsupervised framework for depth estimation and camera motion prediction from video

Depth estimation from monocular video plays a crucial role in scene perception. The significant drawback of supervised learning models is the need for vast amounts of manually labeled data (ground truth) for training. To overcome this limitation, unsupervised learning strategies without the requirement for ground truth have achieved extensive attention from researchers in the past few years. This paper presents a novel unsupervised framework for estimating single-view depth and predicting camera motion jointly. Stereo image sequences are used to train the model while monocular images are required for inference. The presented framework is composed of two CNNs (depth CNN and pose CNN) which are trained concurrently and tested independently. The objective function is constructed on the basis of the epipolar geometry constraints between stereo image sequences. To improve the accuracy of the model, a left-right consistency loss is added to the objective function. The use of stereo image sequences enables us to utilize both spatial information between stereo images and temporal photometric warp error from image sequences. Experimental results on the KITTI and Cityscapes datasets show that our model not only outperforms prior unsupervised approaches but also achieving better results comparable with several supervised methods. Moreover, we also train our model on the Euroc dataset which is captured in an indoor environment. Experiments in indoor and outdoor scenes are conducted to test the generalization capability of the model.

Prediction of terrain occlusion in Chang'e-4 mission

This paper proposes a precise terrain occlusion prediction algorithm based on stereo image sequences from the China's Chang'e-4 Yutu-2 rover’ visual system, which includes an on orbit stereo camera calibration method and a camera pose estimation method. The main advantage of the proposed algorithm is that the above two methods consider the problem that the coefficient matrix and observation vector should also be considered with regard to the error during optimization. In this case, the total least squares solution is applied instead of the ordinary least squares solution. Experimental results show that the proposed algorithm has higher precision (with average differences in the altitude angles and azimuthal angles of −0.37 degrees and −0.29 degrees) and application stability than the ordinary least squares method. The proposed algorithm has been successfully used in Chang'e-4 mission operations.

Disparity Estimation Using Stereo Images With Different Focal Lengths

This study proposes a disparity estimation method for images captured by cameras with different focal lengths. Whereas a short focal length camera captures a close scene, a long focal length camera captures a far scene. The proposed method computes full disparity maps for both the close and far scenes. To achieve this, we introduce a stereo rectification method that works directly on images with different focal lengths and parameterizes the zoom levels between the two cameras. To evaluate the proposed method, we set up a stereo system with two cameras with different focal lengths and captured sequences of stereo images. In addition, we used KITTI, EISATS, and SceneFlow stereo datasets to simulate images with different focal lengths. Experimental results show that the proposed disparity estimation method successfully computed disparity maps for both near and far scenes, with significantly better performance than state-of-the-art monocular disparity estimation methods. Source code and experimental results are available online at https://github.com/comvisdinh/disparityestimation .

CRACK WIDTH MEASUREMENT FOR NON-PLANAR SURFACES BY TRIANGLE MESH ANALYSIS IN CIVIL ENGINEERING MATERIAL TESTING

Abstract. This publication concentrates on the photogrammetric crack width measurement of crack patterns of concrete probes under impact loading in high-speed stereo image sequences. The presented algorithm works for non-planar specimens with deformations that only appear tangential to the surface and the method is based on triangle mesh analysis. Experiments were conducted with cylindrical specimens with an impact load affecting parallel to the main axis of the cylinder.

Line-Based Stereo SLAM by Junction Matching and Vanishing Point Alignment

In this paper, we propose a stereo simultaneous localization and mapping (SLAM) method based on line segments. For the front-end module of SLAM, we designed a novel method based on the coplanar junction detection, description, and matching. Then the junctions along with their multi-scale rotated BRIEF descriptors are used in other SLAM modules, including line tracking, mapping, and loop closure. The line extraction and matching thread runs at 20 ~ 40Hz for stereo image sequences on a laptop, making it a practical front-end for line-based SLAM system. For the back-end module, a cost function is designed to minimize both of the reprojection error of line segments and alignment error of the vanishing points. The experimental results demonstrate that the proposed method exhibits more accurate localization and reconstruction than state-of-the-art line-based SLAM systems in line-rich environments.

Autonomous relative navigation for noncooperative target using stereo vision measurements

An autonomous relative navigation method based on stereo vision measurements is proposed for noncooperative targets without any prior information. Firstly, a target identification model using only stereo vision measurements is established. Based on the relative position vector computed from the stereo image sequence, the initial state vector of the noncooperative target and navigation filter covariance matrix are initialized by taken spacecraft attitude and camera installation information into account. Then, an extended Kalman filter considering state noise compensation as well as observation variance matrix and state noise matrix adaptive correction is formulated to solve the noncooperative relative navigation problem. Finally, a low Earth orbit, noncooperative target approaching navigation scenario is considered. The simulation results demonstrate the effectiveness of the proposed method for noncooperative autonomous relative navigation.

Patch-based Stereo Direct Visual Odometry Robust to Illumination Changes

In this paper, we present a patch-based direct visual odometry (DVO) that is robust to illumination changes at a sequence of stereo images. Illumination change violates the photo-consistency assumption and degrades the performance of DVO, thus, it should be carefully handled during minimizing the photometric error. Our approach divides an incoming image into several buckets, and patches inside each bucket own its unique affine illumination parameter to account for local illumination changes for which the global affine model fails to account, then it aligns small patches placed at temporal images. We do not distribute affine parameters to each patch since this yields huge computational load. Furthermore, we propose a prior weight as a function of the previous pose in a constant velocity model which implies that the faster a camera moves, the more likely it maintains the constant velocity model. Lastly, we verify that the proposed algorithm outperforms the global affine illumination model at the publicly available micro aerial vehicle and the planetary rover dataset which exhibit irregular and partial illumination changes due to the automatic exposure of the camera and the strong outdoor sunlight, respectively.