Traditional Bundle Adjustment Research Articles

Digital Surface Model Generation from Satellite Images Based on Double-Penalty Bundle Adjustment Optimization

Digital Surface Model (DSM) generation from high-resolution optical satellite images is an important topic of research in the remote sensing field. In optical satellite imaging systems, the attitude information of the cameras recorded by satellite sensors is often biased, which leads to errors in the Rational Polynomial Camera (RPC) model of satellite imaging. These errors in the RPC model can mislead the DSM generation. To solve the above problems, we propose an automatic DSM generation method from satellite images based on the Double-Penalty bundle adjustment (DPBA) optimization algorithm. In the proposed method, two penalty functions representing the camera’s attitude and the spatial 3D points, respectively, are added to the reprojection error model of the traditional bundle adjustment optimization algorithm. Instead of acting on images directly, the penalty functions are used to adjust the reprojection error model and improve the RPC parameters. We evaluate the performance of the proposed method using high-resolution satellite image pairs and multi-date satellite images. Through some experiments, we compare the accuracy and completeness of the DSM generated by the proposed method, the Satellite Stereo Pipeline (S2P) method, and the traditional bundle adjustment (BA) method. Compared to the S2P method, the experiment results of the satellite image pair indicate that the proposed method can significantly improve the accuracy and the completeness of the generated DSM by about 1–5 m and 20%–60% in most cases. Compared to the traditional BA method, the proposed method improves the accuracy and completeness of the generated DSM by about 0.01–0.05 m and 1%–3% in most cases. The experiment results can be a testament to the feasibility and effectiveness of the proposed method.

ADGEO: A new shore‐based approach to improving spatial accuracy when mapping water bodies using low‐cost drones

AbstractOver the last three decades, satellite imagery has been instrumental in mapping and monitoring water quality. However, satellites often have limitations due to image availability and cloud cover. Today, the spatial resolution of satellite images does not provide finer detail measurements essential for small‐scale water pollution management. Drones offer a complimentary platform capable of operating below cloud cover and acquiring very high spatial resolution datasets in near real‐time. Studies have shown that drone mapping over water can be done via the Direct Georeferencing approach. However, this method is only suitable for high‐end drones with accurate GNSS/IMU. Importantly, this limitation is exacerbated because of the difficulty in placing targets over water, which can be used to improve the accuracy after the survey. This study explored a new method called Assisted Direct Georeferencing which combines the benefits of traditional Bundle Adjustment with Direct Georeferencing. The performance of the approach was evaluated over a variety of different scenarios, demonstrating significant improvement in the planimetric accuracy. From the results, the method reduced the error in XY of drone imagery from MAE of 18.9 to 3.4 m. The result shows the potential of low‐cost drones with Assisted Direct Georeferencing in closing the gap to high‐end drones.

DeepSFM: Robust Deep Iterative Refinement for Structure From Motion.

Structure from Motion (SfM) is a fundamental computer vision problem which has not been well handled by deep learning. One of the promising solutions is to apply explicit structural constraint, e.g., 3D cost volume, into the neural network. Obtaining accurate camera poses from images alone can be challenging, especially with complicated environmental factors. Existing methods usually assume accurate camera poses from GT or other methods, which is unrealistic in practice and additional sensors are needed. In this work, we design a physical driven architecture, namely DeepSFM, inspired by traditional Bundle Adjustment, which consists of two cost volume based architectures to iteratively refine depth and pose. The explicit constraints on both depth and pose, when combined with the learning components, bring merit from both traditional BA and emerging deep learning technology. To speed up the learning and inference efficiency, we apply the Gated Recurrent Units (GRUs)-based depth and pose update modules with coarse to fine cost volumes on the iterative refinements. In addition, with the extended residual depth prediction module, our model can be adapted to dynamic scenes effectively. Extensive experiments on various datasets show that our model achieves state-of-the-art performance with superior robustness against challenging inputs.

DEVELOPING A THEORETICAL ASSESSMENT METHOD FOR AN ASSISTED DIRECT GEOREFERENCING APPROACH TO IMPROVE ACCURACY WHEN MAPPING OVER WATER: THE CONCEPT, POTENTIAL AND LIMITATIONS

Abstract. Drones offer a a unique survey platform that can operate below cloud cover and acquire very high spatial resolution datasets in near real-time. Studies have demonstrated that drones can be used for mapping over water using the Direct Georeferencing approach. However, this method is typically only feasible with high-end drones equipped with highly accurate GNSS/IMU systems. Moreover, placing targets over water to improve accuracy in post-processing can be challenging, further exacerbating this limitation. In this study, we developed an Assisted Direct Georeferencing method which combines the advantages of traditional Bundle Adjustment (BA) and Direct Georeferencing to overcome these challenges. Our approach utilizes BA over feature-rich segments of the drone trajectory, such as the shoreline, and DG in featureless areas, such as over water. To simulate a water-type environment or surface for our early tests, synthetic datasets have been created using Python for theoretical analysis. We then conducted a theoretical assessment of our approach under low and high variability attitude measurements. Our findings revealed that our methodology performs well under low variability attitude measurements, where wind conditions are close to optimal with an R-square value of 0.93. However, our model performs poorly under high variability attitude measurements, with an R-square value of only 0.028. These results suggest that Assisted Direct Georeferencing can serve as an alternative to high-end drones and Direct Georeferencing for water mapping applications in most standard. The findings from this theoretical assessment provide valuable insights into the achievable accuracy, error budgets, and limitations of the proposed model.

CONSTRAINED BUNDLE ADJUSTMENT FOR STRUCTURE FROM MOTION USING UNCALIBRATED MULTI-CAMERA SYSTEMS

Abstract. Structure from motion using uncalibrated multi-camera systems is a challenging task. This paper proposes a bundle adjustment solution that implements a baseline constraint respecting that these cameras are static to each other. We assume these cameras are mounted on a mobile platform, uncalibrated, and coarsely synchronized. To this end, we propose the baseline constraint that is formulated for the scenario in which the cameras have overlapping views. The constraint is incorporated in the bundle adjustment solution to keep the relative motion of different cameras static. Experiments were conducted using video frames of two collocated GoPro cameras mounted on a vehicle with no system calibration. These two cameras were placed capturing overlapping contents. We performed our bundle adjustment using the proposed constraint and then produced 3D dense point clouds. Evaluations were performed by comparing these dense point clouds against LiDAR reference data. We showed that, as compared to traditional bundle adjustment, our proposed method achieved an improvement of 29.38%.

Depth-based sparse bundle adjustment.

It is demonstrated in this paper that due to error model inconsistency, a certain degree of accuracy loss would be incurred to the estimated parameters when the traditional bundle adjustment method is directly applied to the scenario where a fraction of observations is implicitly error free (e.g., the reference image points in commonly used least squares matching refinement). To this end, a depth-based object point model and corresponding depth-based sparse bundle adjustment method are proposed in this paper, in which the position of an object point is represented by its 1D depth relative to its reference image. A corresponding projection model is derived, the sparse block structures of normal equations are studied depending on whether there are shared image parameters to be optimized or not, and corresponding sparse solutions of the normal equations and parameter covariance matrices are derived. Compared with the traditional sparse bundle adjustment method, simulated experiments demonstrate that our method matches the error model of the target scenario, and thus can avoid further accuracy loss. Moreover, both simulated and real data experiments demonstrate that our method can effectively improve computational efficiency.

SBAS: Salient Bundle Adjustment for Visual SLAM

Most visual simultaneous localization and mapping (SLAM) algorithms rely on geometric features, such as points, lines, and planes, and these algorithms treated all features as equally important in the optimization process, ignoring the semantic information of these features. However, the robust and rich semantic features should play a more important role, which is similar to the mechanism of visual saliency or visual attention. Therefore, this article aims to mimic this mechanism in the visual SLAM framework by using the saliency prediction model. To solve the center bias of the generic salient dataset, we proposed a method for recomputing saliency map by considering both geometric and semantic information. Then, we proposed the salient bundle adjustment (SBA) algorithm by using the value of saliency prediction map as the weight of the feature points in the traditional bundle adjustment (BA) approach. Finally, exhaustive experiments conducted with the state-of-the-art algorithm show that our proposed algorithm outperforms existing algorithms such as direct sparse odometry (DSO) and oriented brief (ORB)-SLAM3 in both indoor and outdoor environments.

Matching Confidence Constrained Bundle Adjustment for Multi-View High-Resolution Satellite Images

Bundle adjustment of multi-view satellite images is a powerful tool to align the orientations of all the images in a unified framework. However, the traditional bundle adjustment process faces a problem in detecting mismatches and evaluating low/medium/high-accuracy matches, which limits the final bundle adjustment accuracy, especially when the mismatches are several times more than the correct matches. To achieve more accurate bundle adjustment results, this paper formulates the prior knowledge of matching accuracy as matching confidences and proposes a matching confidence based bundle adjustment method. The core algorithm firstly selects several highest-confidence matches to initially correct orientations of all images, then detects and eliminates the mismatches under the initial orientation guesses and finally formulates both the matching confidences and the forward-backward projection errors as weights in an iterative bundle adjustment process for more accurate orientation results. We compared our proposed method with the famous RANSAC strategy as well as a state-of-the-art bundle adjustment method on the high-resolution multi-view satellite images. The experimental comparisons are evaluated by image checking points and ground control points, which shows that our proposed method is able to obtain more robust and more accurate mismatch detection results than the RANSAC strategy, even though the mismatches are four times more than the correct matches and it can also achieve more accurate orientation results than the state-of-the-art bundle adjustment method.

TOWARDS AN ACCURATE LOW-COST STEREO-BASED NAVIGATION OF UNMANNED PLATFORMS IN GNSS-DENIED AREAS

Abstract. While lightweight stereo vision sensors provide detailed and high-resolution information that allows robust and accurate localization, the computation demands required for such process is doubled compared to monocular sensors. In this paper, an alternative model for pose estimation of stereo sensors is introduced which provides an efficient and precise framework for investigating system configurations and maximize pose accuracies. Using the proposed formulation, we examine the parameters that affect accurate pose estimation and their magnitudes and show that for standard operational altitudes of ∼50 m, a five-fold improvement in localization is reached, from ∼0.4–0.5 m with a single sensor to less than 0.1 m by taking advantage of the extended field of view from both cameras. Furthermore, such improvement is reached using cameras with reduced sensor size which are more affordable. Hence, a dual-camera setup improves not only the pose estimation but also enables to use smaller sensors and reduce the overall system cost. Our analysis shows that even a slight modification in camera directions improves the positional accuracy further and yield attitude angle as accurate as ±6’ (compared to ±20’). The proposed pose estimation method relieves computational demands of traditional bundle adjustment processes and is easily integrated with other inertial sensors.

Monocular SLAM for large scale scenes

Monocular simultaneous localization and mapping (SLAM) methods easily accumulate error along with the growing map. Especially for a large scale scene, accumulating error may be too large to prevent the close of loops. Bundle adjustment can be employed to close loops and eliminate accumulation error. However, for a large scale scene, traditional bundle adjustment may have the bottleneck in terms of memory and efficiency, which limits its applications in practice. Based on our prior off-line SfM work, in this paper, we present a novel monocular SLAM system for large scale scenes. We modify and improve the key techniques to make them appropriate for real-time SLAM. Particularly, a novel online loop closure detection method based on non-consecutive track matching is proposed, which can automatically detect and efficiently match the common features among different subsequences. In addition, a segment-based bundle adjustment is employed to close the loop efficiently, which can perform an efficient global optimization in a limited memory space to effectively eliminate accumulation error. In the experiments, we made qualitative and quantitative comparison with two state-of-the-art SLAM systems using both our own dataset and KITTI dataset, which demonstrates the effectiveness of the proposed approach.

RBA: Reduced Bundle Adjustment for oblique aerial photogrammetry

This study proposes an efficient Bundle Adjustment (BA) model for oblique aerial photogrammetry to reduce the number of unknown parameters and the dimensions of a non-linear optimization problem. Instead of serving as independent exterior orientations, oblique camera poses are parameterized with nadir camera poses and constant relative poses between oblique and nadir cameras. New observation functions are created with image points as a function of the nadir pose and the relative pose parameters. With these observation functions, the problem of BA is defined as finding optimal unknown parameters by minimizing the total difference between estimated and observed image points. A Gauss-Newton optimization method is utilized to provide a solution for this least-square problem with a reduced normal equation, which plays a very critical role in the convergence and efficiency of BA. Compared with traditional BA methods, the number of unknown parameters and the dimension of the normal equations decrease, this approach dramatically reduces the computational complexity and memory cost especially for large-scale scenarios with a number of oblique images. Four synthetic datasets and a real dataset were used to check the validation and the accuracy of the proposed method. The accuracy of the proposed method is very close to that of the traditional BA method, but the efficiency can be significantly improved by the proposed method. For very large-scale scenarios, the proposed method can still address the limitation of memory and orientate all images captured by an oblique aerial multi-camera system.

The precise multimode GNSS positioning for UAV and its application in large scale photogrammetry

In this paper, we use multimode GNSS instead of single GPS constellation to resolve the three exterior line elements of each image. The principles of differential GNSS positioning and GNSS-supported aerial triangulation (AT) are presented and an implementation case is demonstrated. With multi-constellation system, the number of visible satellites is significantly increased and the geometry distribution of the satellites is well improved. The positioning accuracy and robustness are therefore getting better compared to GPS positioning. Experimental results show that differential GNSS has remarkable increment on the integer rate of ambiguity solution when GPS has few number and low elevation angle of satellites. The combined AT adjustment of GNSS resolution and 10 ground control points (GCPs) achieve the horizontal accuracy of ±18 cm and vertical accuracy of ±23 cm for the check points, which are comparable with traditional bundle adjustment with dense GCPs and better than GPS-supported AT. The achieved accuracy also satisfies the requirement for 1:500 topographic maps with the bonus of 84% GCPs eliminated. In conclusion, GNSS-supported AT is of feasibility and superiority for large scale Unmanned Aerial Vehicle-based photogrammetry.

An asymmetric re-weighting method for the precision combined bundle adjustment of aerial oblique images

Combined bundle adjustment is a fundamental step in the processing of massive oblique images. Traditional bundle adjustment designed for nadir images gives identical weights to different parts of image point observations made from different directions, due to the assumption that the errors in the observations follow the same Gaussian distribution. However, because of their large tilt angles, aerial oblique images have trapezoidal footprints on the ground, and their areas correspond to conspicuously different ground sample distances. The errors in different observations no longer conform to the above assumption, which leads to suboptimal bundle adjustment accuracy and restricts subsequent 3D applications. To model the distribution of the errors correctly for the combined bundle adjustment of oblique images, this paper proposes an asymmetric re-weighting method. The scale of each pixel is used to determine a re-weighting factor, and each pixel is subsequently projected onto the ground to identify another anisotropic re-weighting factor using the shape of its quadrangle. Next, these two factors are integrated into the combined bundle adjustment using asymmetric weights for the image point observations; greater weights are assigned to observations with fine resolutions, and those with coarse resolutions are penalized. This paper analyzes urban and rural images captured by three different five-angle camera systems, from both proprietary datasets and the ISPRS/EuroSDR benchmark. The results reveal that the proposed method outperforms the traditional method in both back-projected and triangulated precision by approximately 5–10% in most cases. Furthermore, the misalignments of point clouds generated by the different cameras are significantly alleviated after combined bundle adjustment.

Novel method of calibration with restrictive constraints for stereo-vision system

Regarding the calibration of a stereo vision measurement system, this paper puts forward a new bundle adjustment algorithm based on the stereo vision camera calibration method. Multiple-view geometric constraints and a bundle adjustment algorithm are used to optimize the inner and outer parameters of the camera accurately. A fixed relative constraint relationship between cameras is introduced. We have improved the normal equation construction process of the traditional bundle adjustment method, so that each iteration process occurs just outside the parameters of two images that are taken by a camera that has been optimized to better integrate two cameras bound together as one camera. The relationship between the fixed relative constraints can effectively increase the number of superfluous observations of the adjustment system and optimize higher accuracy while reducing the dimension of the normal matrix; it means that each iteration will reduce the time required. Simulation and actual experimental results show the superior performance of the proposed approach in terms of robustness and accuracy, and our approach also can be extended to stereo-vision system with more than two cameras.

Particle filtering methods for georeferencing panoramic image sequence in complex urban scenes

Georeferencing image sequences is critical for mobile mapping systems. Traditional methods such as bundle adjustment need adequate and well-distributed ground control points (GCP) when accurate GPS data are not available in complex urban scenes. For applications of large areas, automatic extraction of GCPs by matching vehicle-born image sequences with geo-referenced ortho-images will be a better choice than intensive GCP collection with field surveying. However, such image matching generated GCP’s are highly noisy, especially in complex urban street environments due to shadows, occlusions and moving objects in the ortho images. This study presents a probabilistic solution that integrates matching and localization under one framework. First, a probabilistic and global localization model is formulated based on the Bayes’ rules and Markov chain. Unlike many conventional methods, our model can accommodate non-Gaussian observation. In the next step, a particle filtering method is applied to determine this model under highly noisy GCP’s. Owing to the multiple hypotheses tracking represented by diverse particles, the method can balance the strength of geometric and radiometric constraints, i.e., drifted motion models and noisy GCP’s, and guarantee an approximately optimal trajectory. Carried out tests are with thousands of mobile panoramic images and aerial ortho-images. Comparing with the conventional extended Kalman filtering and a global registration method, the proposed approach can succeed even under more than 80% gross errors in GCP’s and reach a good accuracy equivalent to the traditional bundle adjustment with dense and precise control.

A new approach for the bundle adjustment problem with fixed constraints in stereo vision

A new approach for the bundle adjustment problem with fixed constraints in stereo vision is described in this paper. Since the direct application of traditional bundle adjustment fails to use the inner constraints completely which are maintained by fixating the orientation and the baseline between the left and right cameras. However, if the fixed constraints are applied to the traditional bundle adjustment, we refine only the left camera extrinsic parameters and 3D points for simplification in stereo pairs. The new method using the fixed constraints has superior theoretical 3D accuracy, and it can reduce the matrix dimension of the covariance matrix so that the total computation time is decreased. Experiments results using synthetic and real data have shown that our method is better than the traditional bundle adjustment algorithm in the 3D accuracy and the convergence rate.

Application of Bayesian Statistics in Photogrammetric Bundle Adjustment

The basic mathematical principle for bundle adjustment (BA) in photogrammetry is the Gauss-Markov Theorem within the framework of classical statistical inference. In the present article we try to show how Bayesian statistics can be applied in this field, leading to a so-called Bayesian bundle adjustment. The rigorous implementation of the Bayesian approach is derived and a comparison with the traditional BA both in theory and practice is accomplished. The empirical test results show that the Bayesian approach achieves almost the same accuracy as the conventional one, but with advantages as well as some difficulties to be discussed at the end of this paper. © 2011 Published by Elsevier Ltd.

Vast-scale Outdoor Navigation Using Adaptive Relative Bundle Adjustment

In this paper we describe a relative approach to simultaneous localization and mapping, based on the insight that a continuous relative representation can make the problem tractable at large scales. First, it is well known that bundle adjustment is the optimal non-linear least-squares formulation for this problem, in that its maximum-likelihood form matches the definition of the Cramer—Rao lower bound. Unfortunately, computing the maximum-likelihood solution is often prohibitively expensive: this is especially true during loop closures, which often necessitate adjusting all parameters in a loop. In this paper we note that it is precisely the choice of a single privileged coordinate frame that makes bundle adjustment costly, and that this expense can be avoided by adopting a completely relative approach. We derive a new relative bundle adjustment which, instead of optimizing in a single Euclidean space, works in a metric space defined by a manifold. Using an adaptive optimization strategy, we show experimentally that it is possible to solve for the full maximum-likelihood solution incrementally in constant time, even at loop closure. Our approach is, by definition, everywhere locally Euclidean, and we show that the local Euclidean estimate matches that of traditional bundle adjustment. Our system operates online in realtime using stereo data, with fast appearance-based loop closure detection. We show results on over 850,000 images that indicate the accuracy and scalability of the approach, and process over 330 GB of image data into a relative map covering 142 km of Southern England. To demonstrate a baseline sufficiency for navigation, we show that it is possible to find shortest paths in the relative maps we build, in terms of both time and distance. Query images from the web of popular landmarks around London, such as the London Eye or Trafalgar Square, are matched to the relative map to provide route planning goals.

DEM-Aided Block Adjustment for Satellite Images With Weak Convergence Geometry

To acquire the largest possible coverage for environmental monitoring, it is important in most situations that the overlapping areas and the convergent angles of respective satellite images be small. The traditional bundle adjustment method used in aerial photogrammetry may not be the most suitable for direct orientation modeling in situations characterized by weak convergence geometry. We propose and compare three block adjustment methods for the processing of satellite images using the digital elevation model (DEM) as the elevation control. The first of these methods is a revised traditional bundle adjustment approach. The second is based on the direct georeferencing approach. The third is a rational function model with sensor-oriented rational polynomial coefficients. A collocation technique is integrated into all three methods to improve the positioning accuracy. Experimental results indicate that using the DEM as an elevation control can significantly improve the geometric accuracy as well as the geometric discrepancies between images. This is the case for all three methods. Moreover, the geometric performance of the three methods is similar. There is a significant improvement in geometric consistency between overlapping SPOT images with respect to single image adjustment for steep areas.