Camera Pose Estimation Research Articles

Penguin colony georegistration using camera pose estimation and phototourism.

Satellite-based remote sensing and uncrewed aerial imagery play increasingly important roles in the mapping of wildlife populations and wildlife habitat, but the availability of imagery has been limited in remote areas. At the same time, ecotourism is a rapidly growing industry and can yield a vast catalog of photographs that could be harnessed for monitoring purposes, but the inherently ad-hoc and unstructured nature of these images make them difficult to use. To help address this, a subfield of computer vision known as phototourism has been developed to leverage a diverse collection of unstructured photographs to reconstruct a georeferenced three-dimensional scene capturing the environment at that location. Here we demonstrate the use of phototourism in an application involving Antarctic penguins, sentinel species whose dynamics are closely tracked as a measure of ecosystem functioning, and introduce a semi-automated pipeline for aligning and registering ground photographs using a digital elevation model (DEM) and satellite imagery. We employ the Segment Anything Model (SAM) for the interactive identification and segmentation of penguin colonies in these photographs. By creating a textured 3D mesh from the DEM and satellite imagery, we estimate camera poses to align ground photographs with the mesh and register the segmented penguin colony area to the mesh, achieving a detailed representation of the colony. Our approach has demonstrated promising performance, though challenges persist due to variations in image quality and the dynamic nature of natural landscapes. Nevertheless, our method offers a straightforward and effective tool for the georegistration of ad-hoc photographs in natural landscapes, with additional applications such as monitoring glacial retreat.

A novel online time calibration framework via double-stage EKF for visual-inertial odometry

Abstract There is an unavoidable time offset between the camera stream and the inertial measurement unit (IMU) data due to the sensor triggering and transmission delays, which will seriously affect the accuracy of visual-inertial odometry (VIO). A novel online time calibration framework via double-stage EKF for VIO is proposed in this paper. First, the first-stage complementary Kalman filter is constructed by adapting the complementary characteristics between the accelerometer and the gyroscope in the IMU, where the rotation result predicted by the gyroscope is corrected through the measurement of the accelerometer so that the IMU can output a more accurate initial pose. Second, the unknown time offset is added to the state vector of the VIO system. The estimated pose of IMU is used as the prediction information, and the reprojection error of multiple cameras on the same feature point is used as the constraint information. During the operation of the VIO system, the time offset is continuously calculated and superimposed on the IMU timestamp to obtain the data synchronized by the IMU and the camera. The Schur complement model is used to marginalize the camera state that carries less information in the system state, avoiding the loss of prior information between images, and improving the accuracy of camera pose estimation. Finally, the effectiveness of proposed algorithm is verified using the EuRoC dataset and the real experimental data.

Hierarchical Graph Neural Network: A Lightweight Image Matching Model with Enhanced Message Passing of Local and Global Information in Hierarchical Graph Neural Networks

Graph Neural Networks (GNNs) have gained popularity in image matching methods, proving useful for various computer vision tasks like Structure from Motion (SfM) and 3D reconstruction. A well-known example is SuperGlue. Lightweight variants, such as LightGlue, have been developed with a focus on stacking fewer GNN layers compared to SuperGlue. This paper proposes the h-GNN, a lightweight image matching model, with improvements in the two processing modules, the GNN and matching modules. After image features are detected and described as keypoint nodes of a base graph, the GNN module, which primarily aims at increasing the h-GNN’s depth, creates successive hierarchies of compressed-size graphs from the base graph through a clustering technique termed SC+PCA. SC+PCA combines Principal Component Analysis (PCA) with Spectral Clustering (SC) to enrich nodes with local and global information during graph clustering. A dual non-contrastive clustering loss is used to optimize graph clustering. Additionally, four message-passing mechanisms have been proposed to only update node representations within a graph cluster at the same hierarchical level or to update node representations across graph clusters at different hierarchical levels. The matching module performs iterative pairwise matching on the enriched node representations to obtain a scoring matrix. This matrix comprises scores indicating potential correct matches between the image keypoint nodes. The score matrix is refined with a ‘dustbin’ to further suppress unmatched features. There is a reprojection loss used to optimize keypoint match positions. The Sinkhorn algorithm generates a final partial assignment from the refined score matrix. Experimental results demonstrate the performance of the proposed h-GNN against competing state-of-the-art (SOTA) GNN-based methods on several image matching tasks under homography, estimation, indoor and outdoor camera pose estimation, and 3D reconstruction on multiple datasets. Experiments also demonstrate improved computational memory and runtime, approximately 38.1% and 26.14% lower than SuperGlue, and an average of about 6.8% and 7.1% lower than LightGlue. Future research will explore the effects of integrating more recent simplicial message-passing mechanisms, which concurrently update both node and edge representations, into our proposed model.

Robust visual SLAM algorithm based on target detection and clustering in dynamic scenarios.

We propose a visual Simultaneous Localization and Mapping (SLAM) algorithm that integrates target detection and clustering techniques in dynamic scenarios to address the vulnerability of traditional SLAM algorithms to moving targets. The proposed algorithm integrates the target detection module into the front end of the SLAM and identifies dynamic objects within the visual range by improving the YOLOv5. Feature points associated with the dynamic objects are disregarded, and only those that correspond to static targets are utilized for frame-to-frame matching. This approach effectively addresses the camera pose estimation in dynamic environments, enhances system positioning accuracy, and optimizes the visual SLAM performance. Experiments on the TUM public dataset and comparison with the traditional ORB-SLAM3 algorithm and DS-SLAM algorithm validate that the proposed visual SLAM algorithm demonstrates an average improvement of 85.70 and 30.92% in positioning accuracy in highly dynamic scenarios. In comparison to the DynaSLAM system using MASK-RCNN, our system exhibits superior real-time performance while maintaining a comparable ATE index. These results highlight that our pro-posed SLAM algorithm effectively reduces pose estimation errors, enhances positioning accuracy, and showcases enhanced robustness compared to conventional visual SLAM algorithms.

On Challenges of Monocular Pose Estimation for Endoluminal Navigation

Accurate endoscope navigation is vital for precise sampling and drug delivery during endoluminal procedures. Current systems rely on sensorized endoscopes for real-time bronchoscope tracking, but they are costly and require specialised settings. Camera pose estimation offers an alternative for endoscope motion tracking and navigation. In this study, we assess state-of-the-art monocular pose estimation methods for endoluminal navigation. We analyze multiple datasets and diverse feature tracking frameworks, including SIFT, ORB, BRISK, FREAK, AKAZE, SuperPoint, LoFTR, and EndoSfMLearner. Our findings reveal that, while modern deep learning-based feature detectors like SuperPoint and LoFTR yield denser features compared to traditional methods, their overall quality and reliability remain suboptimal. This insight is crucial for understanding the limitations of current monocular odometry in endoluminal environments. We also emphasize the challenges in generalizing learned frameworks developed for other odometry applications to endoscopic applications, stressing the need for more publicly available datasets. To fill this gap, we introduce a clinical bronchoscopy dataset with ground truth camera pose data. This dataset serves as a realistic benchmark for improving feature-based odometry in surgical navigation and encourages tailored framework development for learned pose estimation in endoluminal interventions.

Camera Pose Estimation Using a 3D Gaussian Splatting Radiance Field

Introduction. Accurate camera pose estimation is crucial for many applications ranging from robotics to virtual and augmented reality. The process of determining agents pose from a set of observations is called odometry. This work focuses on visual odometry, which utilizes only images from camera as the input data. The purpose of the paper is to demonstrate an approach for small-scale camera pose estimation using 3D Gaussians as the environment representation. Methods. Given the rise of neural volumetric representations for the environment reconstruction, this work relies on Gaussian Splatting algorithm for high-fidelity volumetric representation. Results. For a trained Gaussian Splatting model and the target image, unseen during training, we estimate its camera pose using differentiable rendering and gradient-based optimization methods. Gradients with respect to camera pose are computed directly from image-space per-pixel loss function via backpropagation. The choice of Gaussian Splatting as representation is particularly appealing because it allows for end-to-end estimation and removes several stages that are common for more classical algorithms. And differentiable rasterization as the image formation algorithm provides real-time performance which facilitates its use in real-world applications. Conclusions. This end-to-end approach greatly simplifies camera pose estimation, avoiding compounding errors that are common for multi-stage algorithms and provides a high-quality camera pose estimation. Keywords: radiance fields, scientific computing, odometry, slam, pose estimation, Gaussian Splatting, differentiable rendering.

Development of the Anthropomorphic Arm for Collaborative and Home Service Robot CHARMIE

Service robots are rapidly transitioning from concept to reality, making significant strides in development. Similarly, the field of prosthetics is evolving at an impressive pace, with both areas now being highly relevant in the industry. Advancements in these fields are continually pushing the boundaries of what is possible, leading to the increasing creation of individual arm and hand prosthetics, either as standalone units or combined packages. This trend is driven by the rise of advanced collaborative robots that seamlessly integrate with human counterparts in real-world applications. This paper presents an open-source, 3D-printed robotic arm that has been assembled and programmed using two distinct approaches. The first approach involves controlling the hand via teleoperation, utilizing a camera and machine learning-based hand pose estimation. This method details the programming techniques and processes required to capture data from the camera and convert it into hardware signals. The second approach employs kinematic control using the Denavit-Hartenbergmethod to define motion and determine the position of the end effector in 3D space. Additionally, this work discusses the assembly and modifications made to the arm and hand to create a cost-effective and practical solution. Typically, implementing teleoperation requires numerous sensors and cameras to ensure smooth and successful operation. This paper explores methods enabled by artificial intelligence (AI) that reduce the need for extensive sensor arrays and equipment. It investigates how AI-generated data can be translated into tangible hardware applications across various fields. The advancements in computer vision, combined with AI capable of accurately predicting poses, have the potential to revolutionize the way we control and interact with the world around us.

Accelerating Globally Optimal Consensus Maximization in Geometric Vision.

Branch-and-bound-based consensus maximization stands out due to its important ability of retrieving the globally optimal solution to outlier-affected geometric problems. However, while the discovery of such solutions caries high scientific value, its application in practical scenarios is often prohibited by its computational complexity growing exponentially as a function of the dimensionality of the problem at hand. In this work, we convey a novel, general technique that allows us to branch over an n-1 dimensional space for an n-dimensional problem. The remaining degree of freedom can be solved globally optimally within each bound calculation by applying the efficient interval stabbing technique. While each individual bound derivation is harder to compute owing to the additional need for solving a sorting problem, the reduced number of intervals and tighter bounds in practice lead to a significant reduction in the overall number of required iterations. Besides an abstract introduction of the approach, we present applications to four fundamental geometric computer vision problems: camera resectioning, relative camera pose estimation, point set registration, and rotation and focal length estimation. Through our exhaustive tests, we demonstrate significant speed-up factors at times exceeding two orders of magnitude, thereby increasing the viability of globally optimal consensus maximizers in online application scenarios.

Retraction Note: Fast and robust absolute camera pose estimation with known focal length

Retraction Note: Fast and robust absolute camera pose estimation with known focal length

PMA-Net: Progressive multi-stage adaptive feature learning for two-view correspondence

Establishing high-quality correspondences is a fundamental step for many computer vision tasks. Leveraging the local consistency of correct correspondences (i.e., inliers) has been a prevalent approach to distinguish them from incorrect correspondences (i.e., outliers). However, the random distribution of a significant proportion of outliers complicates the local neighborhood construction of inliers, making it inevitably contain many outliers, which compromises the reliability of the consistency information. In this paper, we propose a Progressive Multi-Stage Adaptive Feature Learning Network (PMA-Net) to solve this problem, which progressively and adaptively learns over multiple stages to remove random outliers embedded in the initial consistency. Specifically, we propose an Adaptive Dynamic Graph Construction (ADGC) module to construct a global topological graph with a high inlier ratio by calculating the affinity between the corresponding neighbors. Additionally, we also design a Feature Mapping Processing (FMP) block and Multi-Stage Prediction (MSP) block to improve the accuracy of subsequent local neighborhood information aggregation, as well as information loss due to channel dimension compression in the prediction phase. Experimental results in camera pose estimation, remote sensing image registration, and homography estimation demonstrate that the proposed PMA-Net performs better than other state-of-the-art methods on various public datasets. Code: https://github.com/XiaojieLi11/PMA-Net

TQU-SLAM Benchmark Dataset for Comparative Study to Build Visual Odometry Based on Extracted Features from Feature Descriptors and Deep Learning

The problem of data enrichment to train visual SLAM and VO construction models using deep learning (DL) is an urgent problem today in computer vision. DL requires a large amount of data to train a model, and more data with many different contextual and conditional conditions will create a more accurate visual SLAM and VO construction model. In this paper, we introduce the TQU-SLAM benchmark dataset, which includes 160,631 RGB-D frame pairs. It was collected from the corridors of three interconnected buildings comprising a length of about 230 m. The ground-truth data of the TQU-SLAM benchmark dataset were prepared manually, including 6-DOF camera poses, 3D point cloud data, intrinsic parameters, and the transformation matrix between the camera coordinate system and the real world. We also tested the TQU-SLAM benchmark dataset using the PySLAM framework with traditional features such as SHI_TOMASI, SIFT, SURF, ORB, ORB2, AKAZE, KAZE, and BRISK and features extracted from DL such as VGG, DPVO, and TartanVO. The camera pose estimation results are evaluated, and we show that the ORB2 features have the best results (Errd = 5.74 mm), while the ratio of the number of frames with detected keypoints of the SHI_TOMASI feature is the best (rd=98.97%). At the same time, we also present and analyze the challenges of the TQU-SLAM benchmark dataset for building visual SLAM and VO systems.

CamGNN: Cascade Graph Neural Network for Camera Re-Localization

In response to the inaccurate positioning of traditional camera relocation methods in scenes with large-scale or severe viewpoint changes, this study proposes a camera relocation method based on a cascaded graph neural network to achieve accurate scene relocation. Firstly, the NetVLAD retrieval method, which has advantages in image feature representation and similarity calculation, is used to retrieve the most similar images to a given query image. Then, the feature pyramid is employed to extract features at different scales of these images, and the features at the same scale are treated as nodes of the graph neural network to construct a single-layer graph neural network structure. Secondly, a top–down connection is used to cascade the single-layer graph structures, where the information of nodes in the previous graph is fused into a message node to improve the accuracy of camera pose estimation. To better capture the topological relationships and spatial geometric constraints between images, an attention mechanism is introduced in the single-layer graph structure, which helps to effectively propagate information to the next graph during the cascading process, thereby enhancing the robustness of camera relocation. Experimental results on the public dataset 7-Scenes demonstrate that the proposed method can effectively improve the accuracy of camera absolute pose localization, with average translation and rotation errors of 0.19 m and 6.9°, respectively. Compared to other deep learning-based methods, the proposed method achieves more than 10% improvement in both average translation and rotation accuracy, demonstrating highly competitive localization precision.

Vision-based algorithm for structural response measurement using movable camera and damage localization

Vibration-based Structural Health Monitoring (SHM) is widely recognized as a prominent approach for evaluating the safety and integrity of civil infrastructure. This research presents a novel algorithm for vibration measurement using a movable camera, effectively overcoming the issue of signal drift caused by camera motion. The proposed algorithm comprises three modules: optical flow computation, camera pose estimation, and motion compensation. Validation experiments by virtual images demonstrate accurate camera pose estimation and effective mitigation of drifting signals. The dynamic characteristics of the frame structure are extracted using a movable camera and the proposed algorithm. Several challenges are addressed, including accurately estimating the scale factor to compensate for the drifting signals. Experimental validation is conducted to verify the proposed approach. The results demonstrate the efficacy and promising potential of the proposed approach. Finally, this study demonstrates the practical application through damage localization experiments.

BCLNet: Bilateral Consensus Learning for Two-View Correspondence Pruning

Correspondence pruning aims to establish reliable correspondences between two related images and recover relative camera motion. Existing approaches often employ a progressive strategy to handle the local and global contexts, with a prominent emphasis on transitioning from local to global, resulting in the neglect of interactions between different contexts. To tackle this issue, we propose a parallel context learning strategy that involves acquiring bilateral consensus for the two-view correspondence pruning task. In our approach, we design a distinctive self-attention block to capture global context and parallel process it with the established local context learning module, which enables us to simultaneously capture both local and global consensuses. By combining these local and global consensuses, we derive the required bilateral consensus. We also design a recalibration block, reducing the influence of erroneous consensus information and enhancing the robustness of the model. The culmination of our efforts is the Bilateral Consensus Learning Network (BCLNet), which efficiently estimates camera pose and identifies inliers (true correspondences). Extensive experiments results demonstrate that our network not only surpasses state-of-the-art methods on benchmark datasets but also showcases robust generalization abilities across various feature extraction techniques. Noteworthily, BCLNet obtains significant improvement gains over the second best method on unknown outdoor dataset, and obviously accelerates model training speed.

Structure-aware neural radiance fields without posed camera

The neural radiance fields (NeRF) for realistic novel view synthesis require camera poses to be pre-acquired by a structure-from-motion (SfM) approach. This two-stage strategy is not convenient to use and degrades the performance because the error in the pose extraction can propagate to the view synthesis. We integrate pose extraction and view synthesis into a jointly optimized process so that they can benefit from each other. For network training, only images are given without pre-known camera poses. The camera poses are obtained by the depth-consistent constraint in which the identical feature in different views has the same world coordinates transformed from the local camera coordinates according to the extracted poses. The depth-consistent constraint is jointly optimized with the pixel color constraint. The poses are represented by a CNN-based deep network, whose input is the related frames. This joint optimization enables NeRF to be aware of the scene’s structure, resulting in improved generalization performance. Experiments on three datasets demonstrate the effectiveness of camera pose estimation and novel view synthesis. Code is available at https://github.com/XTU-PR-LAB/SaNerf.

A more reliable local-global-guided network for correspondence pruning

The correspondence pruning task relies on both local and global contexts, which are considered to be essential in inferring the probability of inliers. Many previous approaches seek to devise various structures to make effective use of them, but they either use only a plain structure or base it on their own hypothetical relationships, which leads to some limitations remaining to be improved. Derived from this, we propose our LG-Net including a simple yet effective LGA block and a well-designed GPA block to extract local and global information respectively. Specifically, the LGA block combines local topology into the neighborhood and enhances the representative ability by enabling the interaction of the adjacency neighbors with a simple Softmax operation. Meanwhile, the GPA block prefers correspondences with higher inlier-probability to restrict the interference of outliers. With the guide of relatively reliable prior, it will facilitate the robustness of gathering rich global contextual information. As a consequence, our LG-Net takes both local and global context into account to help successfully recover correspondences. Extensive experiments have demonstrated the better performance of our method comparing with existing state-of-the-art methods on camera pose estimation and homography estimation.

Spatial and temporal consistency learning for monocular 6D pose estimation

Monocular 6D pose estimation is a challenging task in the field of computer vision and robotics. Many previous works only input the cropped image of single object during training and inference, aiming to remove the noise from non-object regions. However, most of these methods ignore the viewpoint and spatial relationships of objects in the scene, which are crucial for accurate pose estimation of camera. To address this issue, this paper proposes a novel multi-view and multi-object based learning strategy for monocular 6D pose estimation, which involves the consistency of object coordinate for the same object at different viewpoints and the consistency of world coordinate for different objects in the same space. In the proposed method, the spatial and temporal groups are generated to trained the monocular 6D pose estimation network. Due to the camera motion, scene images taken at different times can be regarded as images captured from different viewpoints. Therefore, a temporal consistency loss is designed to constraint the relationship of the same object at different viewpoints, while a spatial consistency loss is designed to constraint the relationship of different objects at the same space. Finally, the proposed method is verified on the public datasets. Experimental results show that the proposed method is accurate, robust, and outperforms similar state-of-the-art approaches.

A Measure of Semantic Class Difference of Point Reprojection Pairs in Camera Pose Estimation

In this paper, we propose a new measure that evaluates the semantic errors of camera poses in visual odometry (VO) and visual simultaneous localization and mapping (VSLAM). Traditionally, VO/VSLAM methods have used photometric images to estimate camera poses, but they suffer from varying illumination and viewpoint changes. Thus, methods using semantic images have been an alternative to increase consistency, as semantic information has shown its robustness even in hostile environments. Our measure compares semantic classes of map point reprojection pairs between images to improve the camera pose estimation accuracy in VO/VSLAM. To evaluate the difference between semantic classes, we adopt the normalized information distance (NID) from information theory. Furthermore, we suggest a weight parameter to balance the existing error of VO/VSLAM with the semantic error introduced by our approach. Our experimental results, obtained from the VKITTI and KITTI benchmark datasets, show that the proposed semantic error measure reduces both the relative pose error (RPE) and absolute trajectory error (ATE) of camera pose estimation compared to the existing photometric image-based errors of indirect and direct VO/VSLAM.

Camera Pose Estimation and Relocalization Algorithm Based on Ground Texture

Camera Pose Estimation and Relocalization Algorithm Based on Ground Texture

Depth-Based Efficient PnP: A Rapid and Accurate Method for Camera Pose Estimation

Depth-Based Efficient PnP: A Rapid and Accurate Method for Camera Pose Estimation