Global Bundle Adjustment Research Articles

Efficient Structure from Motion for Large-Size Videos from an Open Outdoor UAV Dataset.

Modern UAVs (unmanned aerial vehicles) equipped with video cameras can provide large-scale high-resolution video data. This poses significant challenges for structure from motion (SfM) and simultaneous localization and mapping (SLAM) algorithms, as most of them are developed for relatively small-scale and low-resolution scenes. In this paper, we present a video-based SfM method specifically designed for high-resolution large-size UAV videos. Despite the wide range of applications for SfM, performing mainstream SfM methods on such videos poses challenges due to their high computational cost. Our method consists of three main steps. Firstly, we employ a visual SLAM (VSLAM) system to efficiently extract keyframes, keypoints, initial camera poses, and sparse structures from downsampled videos. Next, we propose a novel two-step keypoint adjustment method. Instead of matching new points in the original videos, our method effectively and efficiently adjusts the existing keypoints at the original scale. Finally, we refine the poses and structures using a rotation-averaging constrained global bundle adjustment (BA) technique, incorporating the adjusted keypoints. To enrich the resources available for SLAM or SfM studies, we provide a large-size (3840 × 2160) outdoor video dataset with millimeter-level-accuracy ground control points, which supplements the current relatively low-resolution video datasets. Experiments demonstrate that, compared with other SLAM or SfM methods, our method achieves an average efficiency improvement of 100% on our collected dataset and 45% on the EuRoc dataset. Our method also demonstrates superior localization accuracy when compared with state-of-the-art SLAM or SfM methods.

YES-SLAM: YOLOv7-enhanced-semantic visual SLAM for mobile robots in dynamic scenes

In dynamic scenes, moving objects will cause a significant error accumulation in robot’s pose estimation, and might even lead to tracking loss. In view of these problems, this paper proposes a semantic visual simultaneous localization and mapping algorithm based on YOLOv7. First, a light-weight network YOLOv7 is employed to acquire the semantic information of different objects in the scene, and flood filling and edge-enhanced techniques are combined to accurately and quickly separate the dynamic feature points from the extracted feature point set. In this way, the obtained static feature points with high-confidence are used to achieve the accurate estimation of robot’s pose. Then, according to the semantic information of YOLOv7, the motion magnitude of the robot, and the number of dynamic feature points in camera’s field-of-view, a high-performance keyframe selection strategy is constructed. On this basis, a robust loop closure detection method is developed by introducing the semantic information into the bag-of-words model, and global bundle adjustment optimization is performed on all keyframes and map points to obtain a global consistent pose graph. Finally, YOLOv7 is further utilized to carry out semantic segmentation on the keyframes, remove the dynamic objects in its semantic mask, and combine the point cloud pre-processing and octree map to build a 3D navigation semantic map. A series of simulations on TUM dataset and a case study in real scene clearly demonstrated the performance superiority of the proposed algorithms.

A Robust Semi-Direct 3D SLAM for Mobile Robot Based on Dense Optical Flow in Dynamic Scenes.

Dynamic objects bring about a large number of error accumulations in pose estimation of mobile robots in dynamic scenes, and result in the failure to build a map that is consistent with the surrounding environment. Along these lines, this paper presents a robust semi-direct 3D simultaneous localization and mapping (SLAM) algorithm for mobile robots based on dense optical flow. First, a preliminary estimation of the robot's pose is conducted using the sparse direct method and the homography matrix is utilized to compensate for the current frame image to reduce the image deformation caused by rotation during the robot's motion. Then, by calculating the dense optical flow field of two adjacent frames and segmenting the dynamic region in the scene based on the dynamic threshold, the local map points projected within the dynamic regions are eliminated. On this basis, the robot's pose is optimized by minimizing the reprojection error. Moreover, a high-performance keyframe selection strategy is developed, and keyframes are inserted when the robot's pose is successfully tracked. Meanwhile, feature points are extracted and matched to the keyframes for subsequent optimization and mapping. Considering that the direct method is subject to tracking failure in practical application scenarios, the feature points and map points of keyframes are employed in robot relocation. Finally, all keyframes and map points are used as optimization variables for global bundle adjustment (BA) optimization, so as to construct a globally consistent 3D dense octree map. A series of simulations and experiments demonstrate the superior performance of the proposed algorithm.

High Precision Mesh-Based Drone Image Stitching Based on Salient Structure Preservation and Regular Boundaries

Addressing problems such as obvious ghost, dislocation, and distortion resulting from the traditional stitching method, a novel drone image-stitching method is proposed using mesh-based local double-feature bundle adjustment and salient structure preservation which aims to obtain more natural panoramas.The proposed method is divided into the following steps. First, reducing parallax error is considered from both global and local aspects. Global bundle adjustment is introduced to minimize global transfer error, and then the local mesh-based feature-alignment model is incorporated into the optimization framework to achieve more accurate alignment. Considering the sensitivity of human eyes to linear structure, the global linear structure that runs through the images obtained by segment fusion is introduced to prevent distortions and align matching line segments better. Rectangular panoramas usually have better visual effects. Therefore, regular boundary constraint combined with mesh-based shape-preserving transform can make the results more natural while preserving mesh geometry. Two new evaluation metrics are also developed to quantify the performance of linear structure preservation and the alignment difference of matching line segments. Extensive experiments show that our proposed method can eliminate parallax and preserve global linear structures better than other state-of-the-art stitching methods and obtain more natural-looking stitching results.

TOWARDS ROBUST INDOOR VISUAL SLAM AND DENSE RECONSTRUCTION FOR MOBILE ROBOTS

Abstract. Mobile robots are being increasingly employed in various indoor scenarios. The fundamental prerequisite is that the robot can reconstruct an accurate and complete map of the observed environment and estimate the track of its movements in this map. Current visual SLAM methods can perform this task reasonably well, but mostly in small spaces, such as a single room, and often tested in well-textured environments. In real-world applications of large indoor scenes, they lack robustness and fail to build a globally consistent map. To this end, we propose a novel system that can robustly solve the problem encountered by existing visual SLAM methods, such as weak texture and long-term drift. By combining information from a wheel odometer, the robot poses can be predicted smoothly in the absence of texture. The geometric cues are leveraged by aligning Truncated Signed Distance Function (TSDF) based submaps to minimize the long-term drift. To reconstruct a more complete and accurate dense map, we refine the sensor depth maps by taking advantage of color information and the optimization result of global bundle adjustment. As a result, the system can provide precise trajectory estimation and a globally consistent map for the downstream tasks. We validate the accuracy and robustness of the proposed method on both public and self-collected datasets and show the complementary nature of each module. Evaluation results based on high precision ground-truth show an improvement in the mean Absolute Trajectory Error (ATE) from 21 cm to 2 cm for the trajectory estimation, and the reconstructed map has a mean accuracy of 8 cm.

Optimization algorithm for high precision RGB-D dense point cloud 3D reconstruction in indoor unbounded extension area

Aimed at the problems existing in the present Red Green Blue-Depth (RGB-D) three-dimensional (3D) reconstruction algorithms in the unbounded extension area, such as low accuracy, inaccurate pose estimation, and more restrictions on data set shooting, an optimization algorithm for indoor unbounded RGB-D dense point cloud 3D reconstruction with high accuracy is proposed. The algorithm aims at obtaining better pose estimation during image construction. In the image preprocessing stage, normal direction information is given to each point cloud. In camera pose estimation, since perspective-n-points (PNPs) pose estimation is more accurate and has a smaller cumulative error than the traditional near-point iterative algorithm, this paper improves PNP pose estimation and puts it into the pose estimation algorithm. Direct average distribution of errors to achieve loop closure will affect the accuracy of pose estimation. In this study, Similarity Transformation of 3 Points was used to optimize the solution before global Bundle adjustment, enhancing the closed-loop performance of the algorithm. Experimental verification showed that the error of the proposed algorithm for indoor environment reconstruction was about 2 cm at macro and small scales, and the reconstruction error was less than 2%. It can be widely used for RGB-D 3D reconstruction of large indoor scenes and has high accuracy in pose estimation and mapping.

3FO: The Three-Frame-Only Approach for Fast and Accurate Monocular SLAM Initialization

The monocular simultaneous localization and mapping (SLAM) system is one of the most important among all visual or visual-inertial SLAM (VSLAM or VISLAM) systems due to its low cost, easy calibration and identification. Initialization is always crucial to bootstrap the monocular SLAM system. With the rapid growth of some quick-start required SLAM applications, e.g., augmented reality (AR) and unmanned aerial vehicles (UAVs), devising faster and more accurate initialization has become a central problem. Traditional initialization uses the first two frames to create landmarks and then computes camera poses for the subsequential frames, using a 3D-to-2D perspective-n-points (PnP) mechanism. In this paper, we propose a novel three-frame-only (3FO) initialization approach for the monocular SLAM system, which consists of two steps. In the first step, we use the first two frames to preinitialize poses and landmarks, and in the second step, we use the second and third frames to improve the preinitialization by using the scale consistency of the landmarks generated in the first step to filter out outliers and using inliers to generate more robust landmarks. In both steps, we use the pretrained multilayer perceptron (MLP) combined with homotopy continuation to solve the essential matrices. Finally, we perform a global bundle adjustment (BA) to refine the three camera poses and all the created landmarks. The proposed 3FO initialization approach is evaluated experimentally on the EuRoC benchmark data set with the initialization time and trajectory metrics. The results show that, compared to the traditional ORB-SLAM2 initialization, the 3FO approach reduces the initialization time by 5 times and improves the accuracy by 36.7% on average.

Information sparsification for visual-inertial odometry by manipulating Bayes tree

A new information sparsification method is developed for visual-inertial odometry (VIO) used for exploratory unmanned aerial vehicles (UAV's). While most of the existing methods are designed for global (and thus postprocessing) bundle adjustment (BA) problems, our method is specifically designed to run incrementally on a local BA formulation in real time. The majority of the existing state-of-the-art methods are designed for pose graph optimization (PGO). Unlike PGO problems where the only variables to be optimized are robot poses, a combination of pose, velocity, and inertial measurement unit (IMU) bias is used in VIO. Existing approaches to sparsification cannot be directly applied to this formulation. Therefore, we design a trick to split the dense information matrix into two parts: IMU information and purely visual information and by doing so, existing methods of sparsification can be reused and applied to the purely visual information, which accounts for most of the nonzero entries within the dense information matrix. In addition, we design a new way of summarizing the dense information matrix: using Bayes tree in an incremental smoothing framework for fast data retrieval. Results using public benchmark dataset EuRoC show that the marginalized and sparsified smoother achieves the primary goal of power saving in terms of shortened pipeline cycle time while maintaining absolute trajectory accuracies comparable to that of the complete formulation and better than those offered by fellow sparsification solutions.

Panoramic annular SLAM with loop closure and global optimization.

In this paper, we propose panoramic annular simultaneous localization and mapping (PA-SLAM), a visual SLAM system based on a panoramic annular lens. A hybrid point selection strategy is put forward in the tracking front end, which ensures repeatability of key points and enables loop closure detection based on the bag-of-words approach. Every detected loop candidate is verified geometrically, and the Sim(3) relative pose constraint is estimated to perform pose graph optimization and global bundle adjustment in the back end. A comprehensive set of experiments on real-world data sets demonstrates that the hybrid point selection strategy allows reliable loop closure detection, and the accumulated error and scale drift have been significantly reduced via global optimization, enabling PA-SLAM to reach state-of-the-art accuracy while maintaining high robustness and efficiency.

An improved feature matching ORB-SLAM algorithm

To solve the problems of low accuracy and poor real-time performance in traditional mobile robot vision simultaneous positioning and map construction (SLAM), the original algorithm was improved. First, the ORB features of adjacent images are extracted, and the PROSAC algorithm is used to achieve feature point matching. At the same time, the PROSAC algorithm is improved and optimized, and the execution time of the optimized PROSAC algorithm is significantly reduced; finally, based on the graph optimization model, a global Bundle Adjustment algorithm based on the largest common-view weight frame is proposed to achieve dense and sparse map creation. The algorithm is verified by Tum data set, and the experiment shows that the root mean square error has dropped significantly. The results on the data set effectively prove the effectiveness of the algorithm in this paper.

A Fisheye Image Matching Method Boosted by Recursive Search Space for Close Range Photogrammetry

Close range photogrammetry (CRP) with large field-of-view images has become widespread in recent years, especially in terrestrial mobile mapping systems (TMMS). However, feature-based matching (FBM) with omnidirectional images (e.g., fisheye) is challenging even for state-of-the-art methods, such as the scale-invariant feature transform (SIFT), because of the strong scale change from image to image. This paper proposes an approach to boost FBM techniques on fisheye images with recursive reduction of the search space based on epipolar geometry. The epipolar restriction is calculated with the equidistant mathematical model and the initial exterior orientation parameters (EOPs) determined with navigation sensors from TMMS. The proposed method was assessed with data sets acquired by a low-cost TMMS. The TMMS is composed of a calibrated poly-dioptric system (Ricoh Theta S) and navigation sensors aimed at outdoor applications. The assessments show that Ricoh Theta S position and attitude were estimated in a global bundle adjustment with a precision (standard deviation) of 4 cm and 0.3°, respectively, using as observations the detected matches from the proposed method. Compared with other methods based on SIFT extended to the omnidirectional geometry, our approach achieved compatible results for outdoor applications.

Automated Aerial Triangulation for UAV-Based Mapping

Accurate 3D reconstruction/modelling from unmanned aerial vehicle (UAV)-based imagery has become the key prerequisite in various applications. Although current commercial software has automated the process of image-based reconstruction, a transparent system, which can be incorporated with different user-defined constraints, is still preferred by the photogrammetric research community. In this regard, this paper presents a transparent framework for the automated aerial triangulation of UAV images. The proposed framework is conducted in three steps. In the first step, two approaches, which take advantage of prior information regarding the flight trajectory, are implemented for reliable relative orientation recovery. Then, initial recovery of image exterior orientation parameters (EOPs) is achieved through either an incremental or global approach. Finally, a global bundle adjustment involving Ground Control Points (GCPs) and check points is carried out to refine all estimated parameters in the defined mapping coordinate system. Four real image datasets, which are acquired by two different UAV platforms, have been utilized to evaluate the feasibility of the proposed framework. In addition, a comparative analysis between the proposed framework and the existing commercial software is performed. The derived experimental results demonstrate the superior performance of the proposed framework in providing an accurate 3D model, especially when dealing with acquired UAV images containing repetitive pattern and significant image distortions.

Metrological Characterization of a Vision-Based System for Relative Pose Measurements with Fiducial Marker Mapping for Spacecrafts

An improved approach for the measurement of the relative pose between a target and a chaser spacecraft is presented. The selected method is based on a single camera, which can be mounted on the chaser, and a plurality of fiducial markers, which can be mounted on the external surface of the target. The measurement procedure comprises of a closed-form solution of the Perspective from n Points (PnP) problem, a RANdom SAmple Consensus (RANSAC) procedure, a non-linear local optimization and a global Bundle Adjustment refinement of the marker map and relative poses. A metrological characterization of the measurement system is performed using an experimental set-up that can impose rotations combined with a linear translation and can measure them. The rotation and position measurement errors are calculated with reference instrumentations and their uncertainties are evaluated by the Monte Carlo method. The experimental laboratory tests highlight the significant improvements provided by the Bundle Adjustment refinement. Moreover, a set of possible influencing physical parameters are defined and their correlations with the rotation and position errors and uncertainties are analyzed. Using both numerical quantitative correlation coefficients and qualitative graphical representations, the most significant parameters for the final measurement errors and uncertainties are determined. The obtained results give clear indications and advice for the design of future measurement systems and for the selection of the marker positioning on a satellite surface.

Distributed Very Large Scale Bundle Adjustment by Global Camera Consensus.

The increasing scale of Structure-from-Motion is fundamentally limited by the conventional optimization framework for the all-in-one global bundle adjustment. In this paper, we propose a distributed approach to coping with this global bundle adjustment for very large scale Structure-from-Motion computation. First, we derive the distributed formulation from the classical optimization algorithm ADMM, Alternating Direction Method of Multipliers, based on the global camera consensus. Then, we analyze the conditions under which the convergence of this distributed optimization would be guaranteed. In particular, we adopt over-relaxation and self-adaption schemes to improve the convergence rate. After that, we propose to split the large scale camera-point visibility graph in order to reduce the communication overheads of the distributed computing. The experiments on both public large scale SfM data-sets and our very large scale aerial photo sets demonstrate that the proposed distributed method clearly outperforms the state-of-the-art method in efficiency and accuracy.

Linear SFM: A hierarchical approach to solving structure-from-motion problems by decoupling the linear and nonlinear components

This paper presents a novel hierarchical approach to solving structure-from-motion (SFM) problems. The algorithm begins with small local reconstructions based on nonlinear bundle adjustment (BA). These are then joined in a hierarchical manner using a strategy that requires solving a linear least squares optimization problem followed by a nonlinear transform. The algorithm can handle ordered monocular and stereo image sequences. Two stereo images or three monocular images are adequate for building each initial reconstruction. The bulk of the computation involves solving a linear least squares problem and, therefore, the proposed algorithm avoids three major issues associated with most of the nonlinear optimization algorithms currently used for SFM: the need for a reasonably accurate initial estimate, the need for iterations, and the possibility of being trapped in a local minimum. Also, by summarizing all the original observations into the small local reconstructions with associated information matrices, the proposed Linear SFM manages to preserve all the information contained in the observations. The paper also demonstrates that the proposed problem formulation results in a sparse structure that leads to an efficient numerical implementation. The experimental results using publicly available datasets show that the proposed algorithm yields solutions that are very close to those obtained using a global BA starting with an accurate initial estimate. The C/C++ source code of the proposed algorithm is publicly available at https://github.com/LiangZhaoPKUImperial/LinearSFM.

3D reconstruction system based on incremental structure from motion using a camera with varying parameters

In this paper, we present a flexible and fast system for multi-scale objects/scenes 3D reconstruction from uncalibrated images/video taken by a moving camera characterized by variable parameters. The proposed system is based on incremental structure from motion and good exploitation of bundle adjustment. At first, from two selected images, our system allows to recover, in a well-chosen reference, coordinates of a set of 3D points. In this context, we have proposed a new method of self-calibration based on the use of two unknown scene points with their image projections. After that, new images are inserted progressively using 3D information already obtained. Local bundle adjustment is used to adjust the new estimated entities. At some time, we introduce a global bundle adjustment to adjust as best as possible all estimated entities and to have an initial 3D model of quality covering an interesting part of the object/scene. This model will be used as reference for the insertion of the rest of images. The proposed system allows to obtain satisfactory results within a reasonable time.

Divide and conquer: A hierarchical approach to large-scale structure-from-motion

In this paper we present a novel pipeline for large-scale SfM. We first organise the images into a hierarchical tree built using agglomerative clustering. The SfM problem is then solved by reconstructing smaller image sets and merging them into a common frame of reference as we move up the tree in a bottom-up fashion. Such an approach drastically reduces the computational load for matching image pairs without sacrificing accuracy. It also makes the resulting sequence of bundle adjustment problems well-conditioned at all stages of reconstruction. We use motion averaging followed by global bundle adjustment for reconstruction of each individual cluster. Our 3D registration or alignment of partial reconstructions based on epipolar relationships is both robust and reliable and works well even when the available camera-point relationships are poorly conditioned. The overall result is a robust, accurate and efficient pipeline for large-scale SfM. We present extensive results that demonstrate these attributes of our pipeline on a number of large-scale, real-world datasets and compare with the state-of-the-art.

A GENERIC PROBABILISTIC MODEL AND A HIERARCHICAL SOLUTION FOR SENSOR LOCALIZATION IN NOISY AND RESTRICTED CONDITIONS

A generic probabilistic model, under fundamental Bayes’ rule and Markov assumption, is introduced to integrate the process of mobile platform localization with optical sensors. And based on it, three relative independent solutions, bundle adjustment, Kalman filtering and particle filtering are deduced under different and additional restrictions. We want to prove that first, Kalman filtering, may be a better initial-value supplier for bundle adjustment than traditional relative orientation in irregular strips and networks or failed tie-point extraction. Second, in high noisy conditions, particle filtering can act as a bridge for gap binding when a large number of gross errors fail a Kalman filtering or a bundle adjustment. Third, both filtering methods, which help reduce the error propagation and eliminate gross errors, guarantee a global and static bundle adjustment, who requires the strictest initial values and control conditions. The main innovation is about the integrated processing of stochastic errors and gross errors in sensor observations, and the integration of the three most used solutions, bundle adjustment, Kalman filtering and particle filtering into a generic probabilistic localization model. The tests in noisy and restricted situations are designed and examined to prove them.

Retinal slit lamp video mosaicking.

PurposeTo this day, the slit lamp remains the first tool used by an ophthalmologist to examine patient eyes. Imaging of the retina poses, however, a variety of problems, namely a shallow depth of focus, reflections from the optical system, a small field of view and non-uniform illumination. For ophthalmologists, the use of slit lamp images for documentation and analysis purposes, however, remains extremely challenging due to large image artifacts. For this reason, we propose an automatic retinal slit lamp video mosaicking, which enlarges the field of view and reduces amount of noise and reflections, thus enhancing image quality.MethodsOur method is composed of three parts: (i) viable content segmentation, (ii) global registration and (iii) image blending. Frame content is segmented using gradient boosting with custom pixel-wise features. Speeded-up robust features are used for finding pair-wise translations between frames with robust random sample consensus estimation and graph-based simultaneous localization and mapping for global bundle adjustment. Foreground-aware blending based on feathering merges video frames into comprehensive mosaics.ResultsForeground is segmented successfully with an area under the curve of the receiver operating characteristic curve of 0.9557. Mosaicking results and state-of-the-art methods were compared and rated by ophthalmologists showing a strong preference for a large field of view provided by our method.ConclusionsThe proposed method for global registration of retinal slit lamp images of the retina into comprehensive mosaics improves over state-of-the-art methods and is preferred qualitatively.

FUSION OF MULTI-VIEW AND MULTI-SCALE AERIAL IMAGERY FOR REAL-TIME SITUATION AWARENESS APPLICATIONS

Abstract. Manned aircraft has long been used for capturing large-scale aerial images, yet the high costs and weather dependence restrict its availability in emergency situations. In recent years, MAV (Micro Aerial Vehicle) emerged as a novel modality for aerial image acquisition. Its maneuverability and flexibility enable a rapid awareness of the scene of interest. Since these two platforms deliver scene information from different scale and different view, it makes sense to fuse these two types of complimentary imagery to achieve a quick, accurate and detailed description of the scene, which is the main concern of real-time situation awareness. This paper proposes a method to fuse multi-view and multi-scale aerial imagery by establishing a common reference frame. In particular, common features among MAV images and geo-referenced airplane images can be extracted by a scale invariant feature detector like SIFT. From the tie point of geo-referenced images we derive the coordinate of corresponding ground points, which are then utilized as ground control points in global bundle adjustment of MAV images. In this way, the MAV block is aligned to the reference frame. Experiment results show that this method can achieve fully automatic geo-referencing of MAV images even if GPS/IMU acquisition has dropouts, and the orientation accuracy is improved compared to the GPS/IMU based georeferencing. The concept for a subsequent 3D classification method is also described in this paper.