Multiple Stereo Research Articles

Highly accurate three-dimensional measurement of large structures using multiple stereo vision with improved two-step calibration algorithm

A multiple stereo-based three-dimensional (3D) measurement method of large-scale structures is presented, which replaces time-consuming and laborious 3D measurement procedure of laser-based conventional equipment. To have 3D reconstruction accuracy with millimeter-precision, two-step calibration algorithms are improved. Specifically, we use 2D epipolar constraint, using fundamental matrix determined by matched points, to new combination of objective function for intrinsic parameter calibration. We additionally propose an onsite structure parameter calibration algorithm using different epipolar constraint of measurement points in images of large ship block as well as reprojection error of checkerboard, to bypass the limitation of the coverage of checkerboard in large-scale measurement. The actual 3D measurement of the three ship blocks is conducted, and the 3D reconstruction accuracy is validated by comparing the 3D coordinates measured by total station. The experimental results showed that the mean 3D reconstruction error ranged from 6 to 14 mm.

A multi-modal garden dataset and hybrid 3D dense reconstruction framework based on panoramic stereo images for a trimming robot

Recovering an outdoor environment’s surface mesh is vital for an agricultural robot during task planning and remote visualization. Image-based dense 3D reconstruction is sensitive to large movements between adjacent frames and the quality of the estimated depth maps. Our proposed solution for these problems is based on a newly-designed panoramic stereo camera along with a hybrid novel software framework that consists of three fusion modules: disparity fusion, pose fusion, and volumetric fusion. The panoramic stereo camera with a pentagon shape consists of 5 stereo vision camera pairs to stream synchronized panoramic stereo images for the following three fusion modules. In the disparity fusion module, rectified stereo images produce the initial disparity maps using multiple stereo vision algorithms. Then, these initial disparity maps, along with the intensity images, are input into a disparity fusion network to produce refined disparity maps. Next, the refined disparity maps are converted into full-view (360°) point clouds or single-view (72°) point clouds for the pose fusion module. The pose fusion module adopts a two-stage global-coarse-to-local-fine strategy. In the first stage, each pair of full-view point clouds is registered by a global point cloud matching algorithm to estimate the transformation for a global pose graph’s edge, which effectively implements loop closure. In the second stage, a local point cloud matching algorithm is used to match single-view point clouds in different nodes. Next, we locally refine the poses of all corresponding edges in the global pose graph using three proposed rules, thus constructing a refined pose graph. The refined pose graph is optimized to produce a global pose trajectory for volumetric fusion. In the volumetric fusion module, the global poses of all the nodes are used to integrate the single-view point clouds into the volume to produce the mesh of the whole garden. The proposed framework and its three fusion modules are tested on a real outdoor garden dataset to show the superiority of the performance. The whole pipeline takes about 4 min on a desktop computer to process the real garden dataset, which is available at: https://github.com/Canpu999/Trimbot-Wageningen-SLAM-Dataset.

Learning Type-2 Fuzzy Logic for Factor Graph Based-Robust Pose Estimation With Multi-Sensor Fusion

Although a wide variety of high-performance state estimation techniques have been introduced recently, the robustness and extension to actual conditions of the estimation systems have been challenging. This paper presents a robust adaptive state estimation framework based on the Type-2 fuzzy inference system and factor graph optimization for autonomous mobile robots. We use the hybrid solution to connect the advantages of the tightly and loosely coupled technique by providing an inertial sensor and other extrinsic sensors such as LiDARs and cameras. In order to tackle the uncertainty input covariance and sensor failures problems, a learnable observation model is introduced by joining the Type-2 FIS and factor graph optimization. In particular, the use of Type-2 Takagi-Sugeno FIS can learn the uncertainty by using particle swarm optimization before adding the observation model to the factor graph. The proposed design consists of four parts: sensor odometry, up-sampling, FIS based-learning observation model, and factor graph-based smoothing. We evaluate our system by using a mobile robot platform equipped with a sensor setup of multiple stereo cameras, an IMU, and a LiDAR sensor. We imitate the LiDAR odometry in structure environments without needing other bulky motion capture systems to learn the observation model of the visual-inertial estimators. The experimental results are deployed in real-world environments to present the accuracy and robustness of the algorithm.

Variable Baseline and Flexible Configuration Stereo Vision Using Two Aerial Robots.

In this work, a new method for aerial robot remote sensing using stereo vision is proposed. A variable baseline and flexible configuration stereo setup is achieved by separating the left camera and right camera on two separate quadrotor aerial robots. Monocular cameras, one on each aerial robot, are used as a stereo pair, allowing independent adjustment of the pose of the stereo pair. In contrast to conventional stereo vision where two cameras are fixed, having a flexible configuration system allows a large degree of independence in changing the configuration in accordance with various kinds of applications. Larger baselines can be used for stereo vision of farther away targets while using a vertical stereo configuration in tasks where there would be a loss of horizontal overlap caused by a lack of suitable horizontal configuration. Additionally, a method for the practical use of variable baseline stereo vision is introduced, combining multiple point clouds from multiple stereo baselines. Issues from using an inappropriate baseline, such as estimation error induced by insufficient baseline, and occlusions from using too large a baseline can be avoided with this solution.

FPGA-based multi-view stereo system with flexible measurement setup

In recent years, stereoscopic image processing algorithms have gained importance for a variety of applications. To capture larger measurement volumes, multiple stereo systems are combined into a multi-view stereo (MVS) system. To reduce the amount of data and the data rate, calculation steps close to the sensors are outsourced to Field Programmable Gate Arrays (FPGAs) as upstream computing units. The calculation steps include lens distortion correction, rectification and stereo matching. In this paper a FPGA-based MVS system with flexible camera arrangement and partly overlapping field of view is presented. The system consists of four FPGA-based passive stereoscopic systems (Xilinx Zynq-7000 7020 SoC, EV76C570 CMOS sensor) and a downstream processing unit (Zynq Ultrascale ZU9EG SoC). This synchronizes the sensor near processing modules and receives the disparity maps with corresponding left camera image via HDMI. The subsequent computing unit calculates a coherent 3D point cloud. Our developed FPGA-based 3D measurement system captures a large measurement volume at 24 fps by combining a multiple view with eight cameras (using Semi-Global Matching for an image size of 640 px × 460 px, up to 256 px disparity range and with aggregated costs over 4 directions). The capabilities and limitation of the system are shown by an application example with optical non-cooperative surface.

Technical Consideration towards Robust 3D Reconstruction with Multi-View Active Stereo Sensors.

It is possible to construct cost-efficient three-dimensional (3D) or four-dimensional (4D) scanning systems using multiple affordable off-the-shelf RGB-D sensors to produce high-quality reconstructions of 3D objects. However, the quality of these systems’ reconstructions is sensitive to a number of factors in reconstruction pipelines, such as multi-view calibration, depth estimation, 3D reconstruction, and color mapping accuracy, because the successive pipelines to reconstruct 3D meshes from multiple active stereo sensors are strongly correlated with each other. This paper categorizes the pipelines into sub-procedures and analyze various factors that can significantly affect reconstruction quality. Thus, this paper provides analytical and practical guidelines for high-quality 3D reconstructions with off-the-shelf sensors. For each sub-procedure, this paper shows comparisons and evaluations of several methods using data captured by 18 RGB-D sensors and provide analyses and discussions towards robust 3D reconstruction. Through various experiments, it has been demonstrated that significantly more accurate 3D scans can be obtained with the considerations along the pipelines. We believe our analyses, benchmarks, and guidelines will help anyone build their own studio and their further research for 3D reconstruction.

L1B+: A PERFECT SENSOR LOCALIZATION MODEL FOR SIMPLE SATELLITE STEREO RECONSTRUCTION FROM PUSH-FRAME IMAGE STRIPS

Abstract. We propose a novel method to generate a single image product from a multi-image strip acquired by a push-frame satellite imaging system. The images of the push-frame strips are combined into a large scale mosaic simulating a perfect sensor geometry. The local camera models of the input images are leveraged to produce a new localization model that covers the output mosaic entirely. Among other applications, this simplifies the task of stereo reconstruction enormously: instead of treating multiple stereo pairs of small images, it is possible to reconstruct the entire area covered by the push-frame acquisition using a single pair of mosaics incorporating all the images. We test our method using strips of SkySat L1B scenes and denote the output images as L1B+. To evaluate the quality of the L1B+ images and their localization models, the stereo reconstructions obtained with L1B+ are compared with those obtained with L1B and with a lidar reference model.

Error Propagation in Satellite Multi-Image Geometry

This paper describes an investigation of the source of geospatial error in digital surface models (DSMs) constructed from multiple satellite images. In this study the uncertainty in surface geometry is separated into two spatial components; global error that affects the absolute position of the surface, and local error that varies from surface point to surface point. The global error component is caused by inaccuracy in the satellite imaging process, mainly due to uncertainty in the satellite position and orientation (pose) during image collection. A key result of the investigation is a new algorithm for determining the absolute geoposition of the DSM that takes into account the pose covariance of each satellite during image collection. This covariance information is used to weigh the evidence from each image in the computation of the global position of the DSM. The use of covariance information significantly decreases the overall uncertainty in global position. The paper also describes an approach to the prediction of local error in the DSM surface. The observed variance in surface position within a single stereo surface reconstruction defines the local horizontal error. The variance in the fused set of elevations from multiple stereo pairs at a single DSM location defines the local vertical error. These accuracy predictions are compared to ground truth provided by LiDAR scans of the same geographic region of interest.

Desymmetrisation of meso‐2,4‐Dimethyl‐8‐oxabicyclo[3.2.1]‐oct‐6‐ene‐3‐ol and its Application in Natural Product Syntheses

AbstractThe compounds containing chiral centers and different functional groups serve as magnificent building blocks for the preparation of various natural products that are having immense biological activity. “Dimethyl‐8‐oxa‐bicyclo[3.2.1]oct‐6‐en‐3‐ol” is one of the wonderful synthons to construct multiple stereo centers at a time during the asymmetric synthesis. In this account, we discuss our research efforts toward the synthesis of various simple and complex natural products from the past three decades (1995–2020) by using dimethyl‐8‐oxa‐bicyclo[3.2.1]oct‐6‐en‐3‐ol as a synthon. Moreover, the synthetic utility of this starting material was investigated and well demonstrated. Further, we executed the desymmetrization of dimethyl‐8‐oxa‐bicyclo[3.2.1]oct‐6‐en‐3‐ol by hydroboration to get different chiral centers. After obtaining the stereocenters, we could manage either the fragment, formal or total synthesis of natural products, by simple protection and deprotection sequence followed by C−C bond formation steps.

A strategic approach for synthesis of benzimidazo[2,1-b]thiazolidinone appended dispirooxindole hybrids via [3 + 2] cycloaddition using fluoro-ethanol as solvent

A series of new functionalized analogues of spirooxindole-pyrrolidines have been synthesized by comprising benzimidazo[2,1-b]thiazolidinone heterocyclic fused moiety via 1,3-dipolar cycloaddition reaction strategy under an efficient and eco-comfortable reaction conditions. In the present study a new dipolarophiles alkyl(Z)-2-(3-oxobenzo[4,5]imidazo[2,1-b]thiazol-2(3H)-ylidene)acetate reacts efficiently with non-stabilized 1,3-dipolar component azomethine ylide formed in situ from the reaction of substituted isatins and α-amino acid sarcosine to affording new complex heterocyclic hybrid i.e.,dispiro[benzimidazo[2,1-b]thiazole-2,3′-pyrrolidine-2′,3′'- indoline]-4′-carboxylates. The present benzimidazo[2,1-b]thiazolidinone grafted dispirooxindole core system formed by the creation of two new CC and one CN bonds with multiple stereo centers in one-step operation and reaction proceed with highly regio and exo- selective manner. Further, the present approach is high yielding protocol for synthesis of target cycloadducts in short reaction time using 2,2,2-trifluoroethanol (TFE) as an efficient reaction medium. The regio-/stereochemistry has been concluded by single crystal X-ray diffraction analysis.

Localization System for Lightweight Unmanned Aerial Vehicles in Inspection Tasks

This paper presents a localization system for Unmanned Aerial Vehicles (UAVs) especially designed to be used in infrastructure inspection, where the UAVs have to fly in challenging conditions, such as relatively high altitude (e.g., 15 m), eventually with poor or absent GNSS (Global Navigation Satellite System) signal reception, or the need for a BVLOS (Beyond Visual Line of Sight) operation in some periods. In addition, these infrastructure inspection applications impose the following requirements for the localization system: defect traceability, accuracy, reliability, and fault tolerance. Our system proposes a lightweight solution combining multiple stereo cameras with a robotic total station to comply with these requirements, providing full-state estimation (i.e., position, orientation, and linear and angular velocities) in a fixed and time-persistent reference frame. Moreover, the system can align and fuse all sensor measurements in real-time at high frequency. We have integrated this localization system in our aerial platform, and we have tested its performance for inspection in a real-world viaduct scenario, where the UAV has to operate with poor or absent GNSS signal at high altitude.

Rotation alignment of a camera-IMU system using a single affine correspondence.

We propose an accurate and easy-to-implement method on rotation alignment of a camera-inertial measurement unit (IMU) system using only a single affine correspondence in the minimal case. The known initial rotation angles between the camera and IMU are utilized; thus, the alignment model can be formulated as a polynomial equation system based on homography constraints by expressing the rotation matrix in a first-order approximation. By solving the equation system, we can recover the rotation alignment parameters. Furthermore, more accurate alignment results can be achieved with the joint optimization of multiple stereo image pairs. The proposed method does not require additional auxiliary equipment or a camera's particular motion. The experimental results on synthetic data and two real-world data sets demonstrate that our method is efficient and precise for the camera-IMU system's rotation alignment.

Enhanced Lunar Topographic Mapping Using Multiple Stereo Images Taken by Yutu-2 Rover with Changing Illumination Conditions

In a planetary-rover exploration mission, stereovision-based 3D reconstruction has been widely applied to topographic mapping of the planetary surface using stereo cameras onboard the rover. In this study, we propose an enhanced topographic mapping method based on multiple stereo images taken at the same rover location with changing illumination conditions. Key steps of the method include dense matching of stereo images, 3D point-cloud generation, point-cloud co-registration, and fusion. The final point cloud has more complete coverage and more details of the terrain than that conventionally generated from a single stereo pair. The effectiveness of the proposed method is verified by experiments using the Yutu-2 rover, in which two data sets were acquired by the navigation cameras at two locations and under changing illumination conditions. This method, which does not involve complex operations, has great potential for application in planetary-rover and lander missions.

A multi-BRIEF-descriptor stereo matching algorithm for binocular visual sensing of fillet welds with indistinct features

Binocular vision is employed in the present work for recognition and positioning of weld seams with the expectation of well handling the reconstruction of the welds for extracting the weld information. However, the indistinct or similar features for the surface of weldments make the stereo matching process in binocular vision difficult. A multi-BRIEF-descriptor stereo matching algorithm with the combination of epipolar constraint is proposed to improve the matching results and to reduce the time consumption. In the algorithm, the distinct edge information of the weldment is carefully detected and taken as the feature points, and the multiple stereo matching processes are conducted with different BRIEF (Binary Robust Independent Elementary Features) descriptors to eliminate the mismatches caused by the indistinct or similar features with fused information. Stereo matching results show that this approach has advantages in reducing the mismatches and obtaining more matched feature points compared with the feature matching methods with the descriptor of SURF (Speeded Up Robust Features), or ORB (Oriented FAST and Rotated BRIEF), although it consumes somewhat more time than that with SURF.

Chemistry of Unsymmetrical C1-Substituted Oxabenzonorbornadienes.

Oxabenzonorbornadiene (OBD) is a useful synthetic intermediate, which can be readily activated by transition metal complexes with great face selectivity due to its dual-faced nature and intrinsic angle strain on the alkene. To date, the understanding of transition-metal catalyzed reactions of OBD itself has burgeoned; however, this has not been the case for unsymmetrical OBDs. Throughout the development of these reactions, the nature of C1-substituent has proven to have a profound effect on both the reactivity and selectivity of the outcome of the reaction. Upon substitution, different modes of reactivity arise, contributing to the possibility of multiple stereo-, regio-, and in extreme cases, constitutional isomers, which can provide unique means of constructing a variety of synthetically useful cyclic frameworks. To maximize selectivity, an understanding of bridgehead substituent effects is crucial. To that end, this review outlines hitherto reported examples of bridgehead substituent effects on the chemistry of unsymmetrical C1-substituted OBDs.

DYNAMIC MONITORING METHOD OF ILLEGAL BUILDINGS USING SPATIOTEMPORAL BIG DATA BASED ON URBAN HIGH LYING ZONES

Abstract. China's urban illegal buildings are emerging in an endless stream with a large number. There is a wide demand for urban illegal buildings monitoring in urban management departments, including Beijing, Shanghai, Guangzhou and other regions where urban management is facing increasingly serious problems of illegal buildings. It is urgent to solve the common problem of "urban disease" caused by urban illegal buildings, and a new automatic monitoring method that can reduce the cost of urban management is urgently needed. This kind of automated monitoring method of illegal buildings has a wide market demand in Urban Management and Law Enforcement and the Ministry of Land and Resources. Existing technologies cannot realize long-term, autonomous, rapid and intelligent dynamic real-time monitoring of urban illegal buildings, which leads to the problem that illegal buildings’ behaviors cannot be stopped in time. There is a possibility to solve these problems using the ubiquitous network of base stations in cities to monitor illegal buildings.This paper proposes a dynamic monitoring method of illegal buildings using spatiotemporal big data based on urban high lying zones. Through the Spatiotemporal sensor network technology, the tilt-type stereo camera is set up at the high lying zones around the survey area. According to the real-time ambient temperature and humidity numerical data fed back by the temperature and humidity sensor, the tilt-type stereo camera uses intelligent time-lapse photogrammetry technology to obtain multiple stereo pairs. The tilted remote sensor transmits the multiple stereo pairs to the urban dynamic analysis service network using wireless transmission. The urban dynamic analysis service network will complete a series of analysis and processing operations without any human intervention, and then transmit the results of the analysis to the early-warning terminal successively through the base station, WIFI and other wireless transmission methods. Finally, the illegal building data is uploaded to the terminal. That is to say, the urban dynamic analysis service network can realize the intelligent, automatic analysis and processing of spatial analysis server and the operation of calling all database data and storing data.The method proposed in this paper uses the high lying zones around the survey area to expand the monitoring range, improve the accuracy of monitoring data, realize long-term real-time monitoring, and fully utilize the characteristics of Spatiotemporal sensing network technology intelligent, autonomous, wireless transmission, etc., significantly reducing labor. It greatly shortens the process from the emergence to the discovery of illegal buildings in cities. The workload of monitoring has improved the efficiency of dynamic monitoring and warehousing of illegal buildings data in cities.

Integrating multiview optical point clouds and multispectral images from ZiYuan-3 satellite remote sensing data to generate an urban digital surface model

The ability to accurately extract urban digital surface models (DSMs) at a reasonable cost is quite valuable. In this study, the ZiYuan-3 (ZY-3) satellite across-track and along-track modes were used to build stereo pairs and elevation models featuring different views. Image matching processing was used to generate optical point clouds for multiple stereo pairs, which were used to analyze the effects of point cloud fusion under different scenarios and optimize the ZY-3 multiview point cloud integration. Then, with the assistance of a high-precision airborne light detection and ranging point cloud, 12 indices, including the maximum, minimum, mean, and B10 to B90, were selected from the building roof optical point cloud in order to construct an urban building elevation model. The results were as follows: (1) The across-track optical point cloud formed by stereo pairs of ZY-3 nadir images is more suitable for urban DSM extraction due to the high spatial resolution and small base/height ratio of these images. (2) The B70 index performs best in the estimated urban building elevation model [ R 2 = 0.91, root mean square error (RMSE) = 5.59 m]; this model accurately identified 95.39% and 95.25% of the whole-building elevations in a narrow-scale test of 3170 buildings and a broad-scale test of 1,510,606 buildings, respectively. Moreover, the model R 2 and number of accurate building elevation recognitions gradually increased with increasing numbers of fused point clouds, while the RMSE decreased significantly. (3) A combined ZY-3 multiview point cloud fusion and multispectral data analysis technique delivered a more refined urban DSM and better characterization of building elevation under complex underlying surface scenarios than were obtained using traditional stereo pair technology. This research presents the methods and theory relating to the application of ∼2- to 6-m resolution satellite images for complex urban underlying surface scenarios.

Multiple data cost-based stereo matching method to generate dense disparity maps from images under radiometric variations

Stereo matching algorithms are capable of providing dense 3D information of the environment through two images taken simultaneously from a pair of cameras placed parallel to each other. Obtaining an accurate disparity map from a stereo image pair is a challenging task and also computationally expensive. If we take into consideration the environmental effect, then the difficulty of the task increases drastically. The authors try to overcome this problem by combining multiple stereo cost functions in the form of a linear equation. To reduce the computation time, a segmentation based cost aggregation method is followed in an attempt to produce an accurate disparity map even in the presence of radiometric variations in the images. The performance of the proposed algorithm is observed while varying the parameter ‘’ between the cost functions and the number of segments in the images. Different image pairs from the Middlebury stereo dataset were considered her

Multiple data cost-based stereo matching method to generate dense disparity maps from images under radiometric variations

Stereo matching algorithms are capable of providing dense 3D information of the environment through two images taken simultaneously from a pair of cameras placed parallel to each other. Obtaining an accurate disparity map from a stereo image pair is a challenging task and also computationally expensive. If we take into consideration the environmental effect, then the difficulty of the task increases drastically. The authors try to overcome this problem by combining multiple stereo cost functions in the form of a linear equation. To reduce the computation time, a segmentation based cost aggregation method is followed in an attempt to produce an accurate disparity map even in the presence of radiometric variations in the images. The performance of the proposed algorithm is observed while varying the parameter 'α' between the cost functions and the number of segments in the images. Different image pairs from the Middlebury stereo dataset were considered here.

Self-calibration approach to stereo cameras with radial distortion based on epipolar constraint.

In this paper, we propose a self-calibration approach to stereo cameras with radial distortion from stereo image pairs of a common 3D scene. Based on the epipolar constraint in the stereo image pair, the intrinsic and extrinsic parameters of stereo cameras are estimated synchronously with a minimum number of nine image point correspondences. It is significant within a random sample consensus (RANSAC) scheme to cope with the outliers of feature matches efficiently and robustly. Then the inliers of the stereo image pair that have been determined after RANSAC are used to optimize the calibration parameters of stereo cameras. Furthermore, more accurate calibration results can be achieved with the joint optimization of multiple stereo image pairs. Both synthetic and real data are used to evaluate the performance of the proposed method, demonstrating that our method can calibrate stereo cameras with radial distortions efficiently and accurately.