Camera Ego-motion Research Articles

Full-scale structural displacement measurement with camera ego-motion compensation using RGB and LiDAR cameras

Displacement measurements based on computer vision techniques have shown significant potential in structural health monitoring. However, as the inevitable camera ego-motion results in a drift in the measured displacement, compensation for such errors must be performed carefully. Although an additional RGB camera is considered in conventional methods for error compensation, the need for the sub-target limits its application. This study presents a dual-camera system, consisting of RGB and light detection and ranging (LiDAR) cameras, that enables accurate camera ego-motion compensation for long-term structural displacement measurements. In particular, the LiDAR camera of the proposed system can measure the six-degrees-of-freedom (6-DOF) ego-motion of the camera body regardless of the sub-target and its elaborate design. The accuracy of the proposed method is evaluated by comparing it with that of a conventional RGB camera. The resulting displacement can be employed to monitor the long-term behavior of full-scale bridges.

Long-term displacement measurement system for bridge bearing capable of camera reposition

Bridge bearings are critical components that counteract stresses from thermal expansion and traffic loads. Their malfunction due to significant deviations from the intended positions can compromise structural safety. Thus, monitoring their displacement is essential. Previous studies utilized computer vision methods for measuring bearing displacement. However, field-installed cameras often experience movement, resulting in measurement inaccuracies. This paper introduces a computer vision-based method for measuring the six degrees of freedom (6-DOF) displacement of a bridge bearing while correcting for camera ego-motion. A pair of three-dimensional (3D) markers is affixed to both the top and bottom of the bearing. The 3D positions of these markers in relation to the camera are determined using the perspective-n-point method. Employing a formula introduced in this study, we calculate the relative pose between the markers, representing the 6-DOF bearing displacement with camera ego-motion correction. This method’s effectiveness is confirmed through both laboratory-scale and field experiments.

ARD-SLAM: Accurate and robust dynamic SLAM using dynamic object identification and improved multi-view geometrical approaches

In the evolving landscape of autonomous navigation, traditional Visual Simultaneous Localization and Mapping (SLAM) systems often encounter challenges in dynamic environments, primarily due to their reliance on assumptions of static surroundings. In response to these limitations, we introduce ARD-SLAM, a groundbreaking approach to dynamic SLAM that innovatively combines global dense optical tracking with sophisticated geometric methodologies. The core innovation of ARD-SLAM lies in its dynamic object identification technique, which harmoniously integrates geometric motion information with prospective motion data. This integration facilitates effective segmentation of moving objects, thereby substantially diminishing their impact on camera ego-motion estimation. ARD-SLAM is further enhanced by an advanced multi-view geometry method that emphasizes the selection of well-matched feature points. This approach is instrumental in efficiently managing dynamic scenarios while also reducing computational load. Rigorous testing on the TUM RGB-D and Bonn RGB-D benchmark datasets has established ARD-SLAM's superiority over established techniques like ORB-SLAM2/3, DynaSLAM, SD-SLAM, DGS-SLAM, and OVD-SLAM. Notably, ARD-SLAM achieves a substantial average reduction in Absolute Trajectory Error (ATE) by 86.1% and in Relative Pose Error (RPE) by 88.0% compared to ORB-SLAM3. The results from the Bonn RGB-D Dataset further underscore ARD-SLAM's effectiveness. Compared to other SLAM methods, ARD-SLAM shows remarkable improvements: 37.8% and 66.4% over DynaSLAM, 41.2% and 73.1% over DGS-SLAM, and 48.9% and 79.7% over OVD-SLAM in ATE and RPE metrics, respectively. This robust performance in dynamically changing environments solidifies ARD-SLAM as a significant advancement in SLAM technology, offering a more precise and adaptable solution for the complex challenges of real-world autonomous navigation.

Robust multiple obstacle tracking method based on depth aware OCSORT for agricultural robots

Multiple Object Tracking (MOT) of dynamic obstacles in field is an important prerequisite for agricultural robots to achieve dynamic obstacle avoidance. The complex and unpredictable road environment in the rural areas will cause severe vibration to the robot, affecting the camera pose and consequently leading to object matching errors. Therefore, we propose an improved method named Depth Aware Observation Centric Simple Online and Realtime Tracking (DA-OCSORT) with two new modules named Inertial Measurement Unit (IMU)-based Camera Motion Compensation (ICMC) and Depth Aware (DA). This method can use IMU information to compensate for camera ego-motion and perform multi-dimensional matching through object depth information, thereby minimizing the influence of camera motion on tracking process. We created an open-source MOT dataset, named MOTorchard. Building upon the pure motion-based method OCSORT, we achieve SOTA on this MOTorchard with HOTA 54.1, MOTA 83.3, IDF1 52.9 and FPS 22.7. Moreover, the proposed two modules prove to have good real-time performance and strong adaptability to other published MOT algorithms.

Robust RGB: D-SLAM in highly dynamic environments based on probability observations and clustering optimization

Visual simultaneous localization and mapping (SLAM) is the underlying support of unmanned systems. Currently, most visual SLAM methods are based on the static environment assumption so that dynamic objects in the camera’s field of view will seriously disrupt its working performance. In view of this, an RGB-D SLAM approach based on probability observations and clustering optimization for highly dynamic environments is proposed, which can effectively eliminate the influence of dynamic objects and accurately estimate the ego-motion of an RGB-D camera. The method contains a dual static map point detection strategy, which is carried out simultaneously in the current and previous frames. First, to enhance tracking robustness in highly dynamic environments, the probabilities of map points being static, calculated by both reprojection deviation and intensity deviation, are used to weight the cost function for pose estimation. Meanwhile, by taking the previous frames as a reference, a static velocity probability based on sparse scene flow is acquired to preliminarily recognize static map points and further improve the tracking accuracy. Then, an improved map point optimization strategy based on K-means clustering is designed, which effectively takes advantage of the clustering algorithm to refine the static map point labels while alleviating its stubborn problem. Finally, the experimental results on the TUM dataset and real scenes compared with the state-of-the-art visual SLAM methods illustrate that the proposed method achieves an extremely robust and accurate performance for estimating camera pose in highly dynamic environments.

Accurate and robust visual SLAM with a novel ray-to-ray line measurement model

Line feature is regarded as a more intuitive and accurate landmark than point feature in visual SLAM for its multiple-pixel comprehensiveness. However, uncertain factors, such as partial occlusion and noise, frequently hinder the mapping accuracy and destabilize the line-assisted SLAM system. Structural regulations and prior hypotheses are often used to tackle the issues, whereas only few people explore the impact of line feature optimization. In this paper, we attempt to improve the accuracy and robustness of visual SLAM system through line feature optimization process. First, a concise ray-to-ray residual model is proposed to replace the prevalent point-to-line model to integrally use line features. Second, the information matrix related to observation uncertainties is calculated to normalize the residual model, which aims to better balance the weights of different lines. Third, we add the line model to ORB-SLAM3 system and design the method of point-and-line based tracking and optimization. Finally, quantitative criteria are proposed to objectively evaluate the line feature map. Both synthetical and real datasets experiments are carried out to demonstrate the advantages of our algorithm in terms of camera ego-motion estimation and mapping. For camera ego-motion estimation experiments, the proposed ray-to-ray residual model produces more accurate results compared to state-of-the-art line-assisted SLAM/VIO algorithms. Furthermore, the model runs faster and obtains more robust results than the prevalent point-to-line reprojection residual model. For mapping experiments, quantitative criteria are proposed, which also open a new perspective to evaluate line-assisted SLAM systems, and give clues to evidence that the proposed method builds a more accurate line feature map.

Dynamic obstacle detection method based on U–V disparity and residual optical flow for autonomous driving

As a core step of obstacle avoidance and path planning, dynamic obstacle detection is critical for autonomous driving. This study aimed to propose a dynamic obstacle detection method based on U–V disparity and residual optical flow for autonomous driving. First, a drivable area of an unmanned vehicle was detected using U–V disparity images. Then, obstacles in the drivable area were detected using U–V disparity images and the geometric relationship between obstacle size and its disparity. Finally, the motion likelihood of each obstacle was estimated by compensating the camera ego-motion. The innovation of the proposed method was that the searching range of the moving obstacles was greatly narrowed by detecting the obstacles in the drivable area, which greatly improved not only the moving obstacle detection efficiency but also the detection accuracy. Datasets from the KITTI benchmark and our self-acquired campus scene data were chosen as testing samples. The experimental results showed that our method could achieve high detection precision, low missed detection rate and less time consumption.

DecoupledPoseNet: Cascade Decoupled Pose Learning for Unsupervised Camera Ego-Motion Estimation

Although many impressive works on learning-based camera ego-motion estimation methods have been proposed recently, most of them promote the accuracy of camera pose estimation by various sequential learning with loop closure optimization, while neglecting the improvement of PoseNet itself. In this paper, we focus on the coupling of rotation and translation in ego-motion estimation, and design a cascade decoupling structure to separately learn the rotation and translation of camera relative motion between adjacent frames. Meanwhile, a rigid-aware unsupervised learning framework with iterative pose refinement scheme is proposed for camera ego-motion estimation. It can disambiguate rigid motion and deformations in dynamic scenarios by jointly learning of optical flow, stereo disparity and camera pose. Validated with evaluation experiments on the public available datasets, our method is superior to the state-of-the-art unsupervised methods, and can achieve comparable results with the supervised ones.

Unsupervised monocular visual odometry via combining instance and RGB information.

Unsupervised deep learning methods have made significant progress in monocular visual odometry (VO) tasks. However, due to the complexity of the real-world scene, learning the camera ego-motion from the RGB information of monocular images in an unsupervised way is still challenging. Existing methods mainly learn motion from the original RGB information, lacking higher-level input from scene understanding. Hence, this paper proposes an unsupervised monocular VO framework that combines the instance and RGB information, named combined information based (CI-VO). The proposed method includes two stages. First is obtaining the instance maps of the monocular images, without finetuning on the VO dataset. Then we obtain the combined information from the two types of information, which is input into the proposed combined information based pose estimation network, named CI-PoseNet, to estimate the relative pose of the camera. To make better use of the two types of information, we propose a fusion feature extraction network to extract the fused features from the combined information. Experiments on the KITTI odometry and KITTI raw dataset show that the proposed method has good performance in the camera pose estimation task, which exceeds the existing mainstream methods.

Rigid-aware self-supervised GAN for camera ego-motion estimation

Learning-based camera ego-motion estimation has attracted increasing attention and has made impressive improvements. However, the accuracy of unsupervised paradigm is still limited, especially in complex dynamic environment. In this paper, we propose a rigid-aware self-supervised generative adversarial network (GAN) for camera ego-motion estimation, which can effectively learn the rigidity of the scene and improve the accuracy of ego-motion estimation by combining the pixel- and the structure-level perception. Specifically, a rigid-aware generator is first designed for joint unsupervised learning of optical flow, stereo depth and camera pose from two consecutive frames. Then, an iterative pose refinement strategy with rigidity learning is presented to reduce the impact of moving objects in scenes. To overcome the limitation of the purely pixel-wised photometric methods, a rigidity mask embedded discriminator is attached to perceive structural distortion artifacts in synthesized fake images, which encourages the generator to learn additional structure-level information to improve the accuracy of pose estimation. Experiments on the benchmark datasets show that our model achieves the state-of-the-art performance in terms of both RPE and ATE compared to recent GAN-based methods.

Unsupervised Learning of Monocular Depth and Ego-Motion with Optical Flow Features and Multiple Constraints.

This paper proposes a novel unsupervised learning framework for depth recovery and camera ego-motion estimation from monocular video. The framework exploits the optical flow (OF) property to jointly train the depth and the ego-motion models. Unlike the existing unsupervised methods, our method extracts the features from the optical flow rather than from the raw RGB images, thereby enhancing unsupervised learning. In addition, we exploit the forward-backward consistency check of the optical flow to generate a mask of the invalid region in the image, and accordingly, eliminate the outlier regions such as occlusion regions and moving objects for the learning. Furthermore, in addition to using view synthesis as a supervised signal, we impose additional loss functions, including optical flow consistency loss and depth consistency loss, as additional supervision signals on the valid image region to further enhance the training of the models. Substantial experiments on multiple benchmark datasets demonstrate that our method outperforms other unsupervised methods.

SAM-Net: Semantic probabilistic and attention mechanisms of dynamic objects for self-supervised depth and camera pose estimation in visual odometry applications

3D scene understanding is an essential research topic in the field of Visual Odometry (VO). VO is usually built under the assumption of a static environment, which does not always hold in real scenarios. Existing works fail to consider the dynamic objects, leading to poor performance. To tackle the aforementioned issues, we propose a self-supervised learning-based VO framework with Semantic probabilistic and Attention Mechanism, SAM-Net, which can jointly learn the single view depth, the ego motion of camera and object detection. For depth estimation, semantic probabilistic fusion mechanism is employed to detect the dynamic objects and generate the semantic probability map as a prior before feeding it to the network to generate a more refined depth map, and attention mechanism is explored to enhance perception ability in spatial and channel view. For pose estimation, we present a novel PoseNet with the atrous separable convolution to expand receptive field. And the photometric consistency loss is employed to alleviate the impact of large rotations. Intensive experiments on the KITTI dataset demonstrate that the proposed approach achieves excellent performance in terms of pose and depth accuracy.

A Robust Pixel-Aware Gyro-Aided KLT Feature Tracker for Large Camera Motions

Tracking fails in the optimization step of conventional KLT (Kanade-Lucas-Tomasi) feature tracker mainly due to the inadequate initial condition that falls out of the convergence region, especially when a camera rotates rapidly or shakes severely. To overcome the problem, we propose a pixel-aware gyro-aided KLT feature tracker that remains accurate and robust under fast camera-ego motion conditions. In particular, we develop a pixel-aware gyro-aided feature prediction algorithm to predict the initial optical flow and obtain the patch deformation matrix of each feature point. It increases the probability of initial estimates to locate in its convergence region. Unlike the existing methods, which assume all the tracked feature pairs were constrained by the same homography prediction matrix, our prediction matrix is adjustable for each feature as it considers the pixel coordinates in the prediction process. A geometric validation based on homography and fundamental check is also adopted to remove outlier tracks. Experimental results on both public datasets and real-world sequences demonstrate that the feature tracking accuracy and robustness can be significantly improved by the proposed method. To facilitate further development, the code is publicly available at https://github.com/weibohuang0314/ pixel aware gyro aided klt feature tracker.

Using Appearance to Predict Pedestrian Trajectories Through Disparity-Guided Attention and Convolutional LSTM

Reasoning about the motions of other objects is important for moving platforms such as intelligent vehicles. In this paper, we propose a pedestrian trajectory prediction approach that utilizes pedestrian appearance as well as historical locations and camera ego-motion to predict future trajectories of pedestrians for vehicles. The proposed model consists of three encoders and a decoder, namely: appearance encoder, location encoder, and ego-motion encoder; and a Long Short-term Memory (LSTM) decoder. The appearance encoder employs an embedding Convolutional Neural Network (CNN) and a Convolutional LSTM (ConvLSTM) to encode pedestrian appearance. A disparity-guided attention mechanism is designed in the appearance encoder to attend to salient dynamic regions in pedestrian appearance. The location encoder and ego-motion encoder employ LSTMs to encode pedestrian historical locations and camera ego-motion. The outputs of all three encoders are concatenated as a representation of input information, which is decoded by the decoder to finally generate the future trajectories of observed pedestrians. We evaluated the proposed model on two public datasets with several baselines and the results showed improvement of about 20% compared with the baselines. We also visualized the predicted trajectories and heat maps generated by disparity-guided attention. Both the visualization and quantitative results validated that our proposed model outperforms the baselines on predicting accuracy and that the model is feasible to capture salient spatiotemporal regions in pedestrian appearance.

Monocular Visual Odometry based on joint unsupervised learning of depth and optical flow with geometric constraints

Inferring camera ego-motion from consecutive images is essential in visual odometry (VO). In this work, we present a jointly unsupervised learning system for monocular VO, consisting of single-view depth, two-view optical flow, and camera-motion estimation module. Our work mitigates the scale drift issue which can further result in a degraded performance in the long-sequence scene. We achieve this by incorporating standard epipolar geometry into the framework. Specifically, we extract correspondences over predicted optical flow and then recover ego-motion. Additionally, we obtain pseudo-ground-truth depth via triangulating 2D-2D pixel matches, which makes the depth scale is closely relevant to the pose. Experimentation on the KITTI driving dataset shows competitive performance compared to established methods.

Unsupervised Learning of Depth and Camera Pose with Feature Map Warping.

Estimating the depth of image and egomotion of agent are important for autonomous and robot in understanding the surrounding environment and avoiding collision. Most existing unsupervised methods estimate depth and camera egomotion by minimizing photometric error between adjacent frames. However, the photometric consistency sometimes does not meet the real situation, such as brightness change, moving objects and occlusion. To reduce the influence of brightness change, we propose a feature pyramid matching loss (FPML) which captures the trainable feature error between a current and the adjacent frames and therefore it is more robust than photometric error. In addition, we propose the occlusion-aware mask (OAM) network which can indicate occlusion according to change of masks to improve estimation accuracy of depth and camera pose. The experimental results verify that the proposed unsupervised approach is highly competitive against the state-of-the-art methods, both qualitatively and quantitatively. Specifically, our method reduces absolute relative error (Abs Rel) by 0.017–0.088.

Ego-Motion Estimation Using Recurrent Convolutional Neural Networks through Optical Flow Learning

Visual odometry (VO) refers to incremental estimation of the motion state of an agent (e.g., vehicle and robot) by using image information, and is a key component of modern localization and navigation systems. Addressing the monocular VO problem, this paper presents a novel end-to-end network for estimation of camera ego-motion. The network learns the latent subspace of optical flow (OF) and models sequential dynamics so that the motion estimation is constrained by the relations between sequential images. We compute the OF field of consecutive images and extract the latent OF representation in a self-encoding manner. A Recurrent Neural Network is then followed to examine the OF changes, i.e., to conduct sequential learning. The extracted sequential OF subspace is used to compute the regression of the 6-dimensional pose vector. We derive three models with different network structures and different training schemes: LS-CNN-VO, LS-AE-VO, and LS-RCNN-VO. Particularly, we separately train the encoder in an unsupervised manner. By this means, we avoid non-convergence during the training of the whole network and allow more generalized and effective feature representation. Substantial experiments have been conducted on KITTI and Malaga datasets, and the results demonstrate that our LS-RCNN-VO outperforms the existing learning-based VO approaches.

Stereo camera visual SLAM with hierarchical masking and motion-state classification at outdoor construction sites containing large dynamic objects

ABSTRACT At modern construction sites, utilizing GNSS (Global Navigation Satellite System) to measure the real-time location and orientation (i.e. pose) of construction machines and navigate them is very common. However, GNSS is not always available. Replacing GNSS with on-board cameras and visual simultaneous localization and mapping (visual SLAM) to navigate the machines is a cost-effective solution. Nevertheless, at construction sites, multiple construction machines will usually work together and side-by-side, causing large dynamic occlusions in the cameras' view. Standard visual SLAM cannot handle large dynamic occlusions well. In this work, we propose a motion segmentation method to efficiently extract static parts from crowded dynamic scenes to enable robust tracking of camera ego-motion. Our method utilizes semantic information combined with object-level geometric constraints to quickly detect the static parts of the scene. Then, we perform a two-step coarse-to-fine ego-motion tracking with reference to the static parts. This leads to a novel dynamic visual SLAM formation. We test our proposals through a real implementation based on ORB-SLAM2, and datasets we collected from real construction sites. The results show that when standard visual SLAM fails, our method can still retain accurate camera ego-motion tracking in real-time. Comparing to state-of-the-art dynamic visual SLAM methods, ours shows outstanding efficiency and competitive result trajectory accuracy.

RDMO-SLAM: Real-Time Visual SLAM for Dynamic Environments Using Semantic Label Prediction With Optical Flow

Visual simultaneous localization and mapping (vSLAM) are considered a fundamental technology for augmented reality and intelligent mobile robots. However, rigid scene assumption is common in vSLAM, which limits the wide usage in populated real-world environments. Recently, with the widespread use of artificial neural networks, many solutions have tried to eliminate the influence of dynamic objects using semantic information provided by object detection or semantic segmentation. Mask R-CNN is popular in many applications, but is usually slow and limits the speed of vSLAM because it waits for the semantic results before camera ego-motion estimation. We had previously introduced a real-time vSLAM, RDS-SLAM, which isolates tracking and semantic segmentation by adding a semantic thread and moving probability estimation. However, Mask R-CNN only supplies a small amount of semantic information because only a few keyframes can be segmented within a short time. Therefore, in this study, we propose a novel vSLAM, RDMO-SLAM, which can leverage more semantic information while ensuring the real-time nature by adding semantic label prediction using dense optical flow. Besides, we also estimate the velocity of each landmark and use them as constraints to reduce the influence of dynamic objects in tracking. Demonstrations are presented, which compare the proposed method to comparable state-of-the-art approaches using dynamic sequences. We improved the real-time performance from 15 Hz (RDS-SLAM) to 30 Hz while keeping robust tracking in dynamic scenes.

Independent Moving Object Detection Based on a Vehicle Mounted Binocular Camera

Accurate detection of independent moving objects is the key to ensure the safety of driverless vehicles during travelling. In this study, a moving object detection method based on a vehicle-mounted binocular camera is proposed. Firstly, a feature matching points selection strategy is designed for high accuracy camera ego-motion estimation. Then, the residual optical flow filed generated only by moving objects is estimated by taking into account the camera ego-motion and global mixed optical flow. A dynamic threshold segmentation method based on disparity is proposed to separate the moving regions from the residual optical flow field. Finally, 3D and 2D information are combined for extracting each single moving object from moving regions. The innovation of the proposed method is that it can detect any type and any size of moving object theoretically by using the dense motion information of the images. By taking the data in KITTI database as the test samples, the detection precision of the moving objects using the proposed algorithm can reach up to 91% without considering tracking strategy.