Ego-motion Estimation Research Articles

Monocular Depth and Ego-motion Estimation with Scale Based on Superpixel and Normal Constraints

3D perception in intelligent VR/AR and autonomous vehicles (AV) applications is critical and attracting significant attention. The self-supervised monocular depth and ego-motion estimation serves as a more intelligent learning approach that provides the required scene depth and location for 3D perception. However, the existing self-supervised learning methods suffer from scale ambiguity, boundary blur and imbalanced depth distribution, limiting the practical applications of VR/AR and AV. In this paper, we propose a new self-supervised learning framework based on superpixel and normal constraints to address these problems. Specifically, we formulate a novel 3D edge structure consistency loss to alleviate the boundary blur of depth estimation. To address the scale ambiguity of estimated depth and ego-motion, we propose a novel surface normal network for efficient camera height estimation. The surface normal network is composed of a deep fusion module and a full-scale hierarchical feature aggregation module. Meanwhile, to realize the global smoothing and boundary discriminability of the predicted normal map, we introduce a novel fusion loss which is based on the consistency constraints of the normal in edge domains and superpixel regions. Experiments are conducted on several benchmarks, and the results illustrate that the proposed approach outperforms the state-of-the-art methods in depth, ego-motion and surface normal estimation.

Fully neuromorphic vision and control for autonomous drone flight.

Biological sensing and processing is asynchronous and sparse, leading to low-latency and energy-efficient perception and action. In robotics, neuromorphic hardware for event-based vision and spiking neural networks promises to exhibit similar characteristics. However, robotic implementations have been limited to basic tasks with low-dimensional sensory inputs and motor actions because of the restricted network size in current embedded neuromorphic processors and the difficulties of training spiking neural networks. Here, we present a fully neuromorphic vision-to-control pipeline for controlling a flying drone. Specifically, we trained a spiking neural network that accepts raw event-based camera data and outputs low-level control actions for performing autonomous vision-based flight. The vision part of the network, consisting of five layers and 28,800 neurons, maps incoming raw events to ego-motion estimates and was trained with self-supervised learning on real event data. The control part consists of a single decoding layer and was learned with an evolutionary algorithm in a drone simulator. Robotic experiments show a successful sim-to-real transfer of the fully learned neuromorphic pipeline. The drone could accurately control its ego-motion, allowing for hovering, landing, and maneuvering sideways-even while yawing at the same time. The neuromorphic pipeline runs on board on Intel's Loihi neuromorphic processor with an execution frequency of 200 hertz, consuming 0.94 watt of idle power and a mere additional 7 to 12 milliwatts when running the network. These results illustrate the potential of neuromorphic sensing and processing for enabling insect-sized intelligent robots.

TEFu-Net: A time-aware late fusion architecture for robust multi-modal ego-motion estimation

Ego-motion estimation plays a critical role in autonomous driving systems by providing accurate and timely information about the vehicle’s position and orientation. To achieve high levels of accuracy and robustness, it is essential to leverage a range of sensor modalities to account for highly dynamic and diverse scenes, and consequent sensor limitations.In this work, we introduce TEFu-Net, a Deep-Learning-based late fusion architecture that combines multiple ego-motion estimates from diverse data modalities, including stereo RGB, LiDAR point clouds and GNSS/IMU measurements. Our approach is non-parametric and scalable, making it adaptable to different sensor set configurations. By leveraging a Long Short-Term Memory (LSTM), TEFu-Net produces reliable and robust spatiotemporal ego-motion estimates. This capability allows it to filter out erroneous input measurements, ensuring the accuracy of the car’s motion calculations over time. Extensive experiments show an average accuracy increase of 63% over TEFu-Net’s input estimators and on par results with the state-of-the-art in real-world driving scenarios. We also demonstrate that our solution can achieve accurate estimates under sensor or input failure. Therefore, TEFu-Net enhances the accuracy and robustness of ego-motion estimation in real-world driving scenarios, particularly in challenging conditions such as cluttered environments, tunnels, dense vegetation, and unstructured scenes. As a result of these enhancements, it bolsters the reliability of autonomous driving functions.

A UWB-Ego-Motion Particle Filter for Indoor Pose Estimation of a Ground Robot Using a Moving Horizon Hypothesis.

Ultra-wideband (UWB) has gained increasing interest for providing real-time positioning to robots in GPS-denied environments. For a robot to act on this information, it also requires its heading. This is, however, not provided by UWB. To overcome this, either multiple tags are used to create a local reference frame connected to the robot or a single tag is combined with ego-motion estimation from odometry or Inertial Measurement Unit (IMU) measurements. Both odometry and the IMU suffer from drift, and it is common to use a magnetometer to correct the drift on the heading; however, magnetometers tend to become unreliable in typical GPS-denied environments. To overcome this, a lightweight particle filter was designed to run in real time. The particle filter corrects the ego-motion heading and location drift using the UWB measurements over a moving horizon time frame. The algorithm was evaluated offline using data sets collected from a ground robot that contains line-of-sight (LOS) and non-line-of-sight conditions. An RMSE of 13 cm and 0.12 (rad) was achieved with four anchors in the LOS condition. It is also shown that it can be used to provide the robot with real-time position and heading information for the robot to act on it in LOS conditions, and it is shown to be robust in both experimental conditions.

ARD-SLAM: Accurate and robust dynamic SLAM using dynamic object identification and improved multi-view geometrical approaches

In the evolving landscape of autonomous navigation, traditional Visual Simultaneous Localization and Mapping (SLAM) systems often encounter challenges in dynamic environments, primarily due to their reliance on assumptions of static surroundings. In response to these limitations, we introduce ARD-SLAM, a groundbreaking approach to dynamic SLAM that innovatively combines global dense optical tracking with sophisticated geometric methodologies. The core innovation of ARD-SLAM lies in its dynamic object identification technique, which harmoniously integrates geometric motion information with prospective motion data. This integration facilitates effective segmentation of moving objects, thereby substantially diminishing their impact on camera ego-motion estimation. ARD-SLAM is further enhanced by an advanced multi-view geometry method that emphasizes the selection of well-matched feature points. This approach is instrumental in efficiently managing dynamic scenarios while also reducing computational load. Rigorous testing on the TUM RGB-D and Bonn RGB-D benchmark datasets has established ARD-SLAM's superiority over established techniques like ORB-SLAM2/3, DynaSLAM, SD-SLAM, DGS-SLAM, and OVD-SLAM. Notably, ARD-SLAM achieves a substantial average reduction in Absolute Trajectory Error (ATE) by 86.1% and in Relative Pose Error (RPE) by 88.0% compared to ORB-SLAM3. The results from the Bonn RGB-D Dataset further underscore ARD-SLAM's effectiveness. Compared to other SLAM methods, ARD-SLAM shows remarkable improvements: 37.8% and 66.4% over DynaSLAM, 41.2% and 73.1% over DGS-SLAM, and 48.9% and 79.7% over OVD-SLAM in ATE and RPE metrics, respectively. This robust performance in dynamically changing environments solidifies ARD-SLAM as a significant advancement in SLAM technology, offering a more precise and adaptable solution for the complex challenges of real-world autonomous navigation.

Semantic geometric fusion multi-object tracking and lidar odometry in dynamic environment

AbstractSimultaneous localization and mapping systems based on rigid scene assumptions cannot achieve reliable positioning and mapping in a complex environment with many moving objects. To solve this problem, this paper proposes a novel dynamic multi-object lidar odometry (MLO) system based on semantic object recognition technology. The proposed system enables the reliable localization of robots and semantic objects and the generation of long-term static maps in complex dynamic scenes. For ego-motion estimation, the proposed system extracts environmental features that take into account both semantic and geometric consistency constraints. Then, the filtered features can be robust to the semantic movable and unknown dynamic objects. In addition, we propose a new least-squares estimator that uses geometric object points and semantic box planes to realize the multi-object tracking (SGF-MOT) task robustly and precisely. In the mapping module, we implement dynamic semantic object detection using the absolute trajectory tracking list. By using static semantic objects and environmental features, the system eliminates accumulated localization errors and produces a purely static map. Experiments on the public KITTI dataset show that the proposed MLO system provides more accurate and robust object tracking performance and better real-time localization accuracy in complex scenes compared to existing technologies.

Correction to “Joint Velocity Ambiguity Resolution and Ego-Motion Estimation Method for mmWave Radar” [Aug 23 4753-4760

Correction to “Joint Velocity Ambiguity Resolution and Ego-Motion Estimation Method for mmWave Radar” [Aug 23 4753-4760

Influence of Neural Network Receptive Field on Monocular Depth and Ego-Motion Estimation

We present an analysis of a self-supervised learning approach for monocular depth and ego-motion estimation. This is an important problem for computer vision systems of robots, autonomous vehicles and other intelligent agents, equipped only with monocular camera sensor. We have explored a number of neural network architectures that perform single-frame depth and multi-frame camera pose predictions to minimize photometric error between consecutive frames on a sequence of camera images. Unlike other existing works, our proposed approach called ERF-SfMLearner examines the influence of the deep neural network receptive field on the performance of depth and ego-motion estimation. To do this, we study the modification of network layers with two convolution operators with extended receptive field: dilated and deformable convolutions. We demonstrate on the KITTI dataset that increasing the receptive field leads to better metrics and lower errors both in terms of depth and ego-motion estimation. Code is publicly available at github.com/linukc/ERF-SfMLearner.

Accurate and robust visual SLAM with a novel ray-to-ray line measurement model

Line feature is regarded as a more intuitive and accurate landmark than point feature in visual SLAM for its multiple-pixel comprehensiveness. However, uncertain factors, such as partial occlusion and noise, frequently hinder the mapping accuracy and destabilize the line-assisted SLAM system. Structural regulations and prior hypotheses are often used to tackle the issues, whereas only few people explore the impact of line feature optimization. In this paper, we attempt to improve the accuracy and robustness of visual SLAM system through line feature optimization process. First, a concise ray-to-ray residual model is proposed to replace the prevalent point-to-line model to integrally use line features. Second, the information matrix related to observation uncertainties is calculated to normalize the residual model, which aims to better balance the weights of different lines. Third, we add the line model to ORB-SLAM3 system and design the method of point-and-line based tracking and optimization. Finally, quantitative criteria are proposed to objectively evaluate the line feature map. Both synthetical and real datasets experiments are carried out to demonstrate the advantages of our algorithm in terms of camera ego-motion estimation and mapping. For camera ego-motion estimation experiments, the proposed ray-to-ray residual model produces more accurate results compared to state-of-the-art line-assisted SLAM/VIO algorithms. Furthermore, the model runs faster and obtains more robust results than the prevalent point-to-line reprojection residual model. For mapping experiments, quantitative criteria are proposed, which also open a new perspective to evaluate line-assisted SLAM systems, and give clues to evidence that the proposed method builds a more accurate line feature map.

Self-supervised deep monocular visual odometry and depth estimation with observation variation

Recent research in deep learning has greatly advanced the development of monocular visual depth estimation and odometry. These learning-based methods predominantly leverage neural networks to extract prevalent high-dimensional features for regression-based estimation. However, the consistency of observations is still regarded as a core challenge that limits the generalization ability. We attribute this challenge to the significant inconsistency in feature levels due to changes in the environment or motion, collectively referred to as observation variation. To this end, we propose a novel monocular depth estimation and a pose estimation network with attention mechanism to recalibrate features for different environment. The depth and ego-motion estimation tasks are formulated by coupling together in an end-to-end reconstruction problem, and attention modules adaptively recalibrate specific feature responses. Furthermore, we propose a simple sliding window optimization without RNN-based architecture, which is introduced to overcome the movement speed variation and error accumulation. Finally, we collect an observation variation dataset with environmental and motion changes. Evaluations of depth and pose estimation demonstrate that our methods effectively overcomes these observation variation challenges and achieves better performance. Our project will be available at https://github.com/alikesierzhao/OV-SfmLearner.

Image Formation Algorithms for Low-Cost Freehand Ultrasound Scanner Based on Ego-Motion Estimation and Unsupervised Clustering

This paper describes the application of unsupervised learning techniques to improve ego-motion estimation for a low-cost freehand ultrasound probe. Echo decorrelation measurements, which are used to estimate the lateral velocity of a scanning probe as it is passed over the skin, are found to be sensitive to varying tissue types and echogenicity in the imaged scene, and this can impact the geometric accuracy of the generated images. Here, we investigate algorithms to cluster the collated 1D echo data into regions of different echogenicity by applying a Gaussian mixture model (GMM), spatial fuzzy c-means (SFCM) or k-means clustering techniques, after which the decorrelation measurements can focus on the regions that yield the most accurate velocity estimates. A specially designed mechanical rig is used to provide the ground truth for the quantitative analysis of probe position estimation on phantom and in vivo data using different clustering techniques. It is concluded that the GMM is the most effective in classifying regions of echo data, leading to the reconstruction of the most geometrically correct 2D B-mode ultrasound image.

Vision-Aided Localization and Mapping in Forested Environments Using Stereo Images.

Forests are traditionally characterized by stand-level descriptors, such as basal area, mean diameter, and stem density. In recent years, there has been a growing interest in enhancing the resolution of forest inventory to examine the spatial structure and patterns of trees across landscapes. The spatial arrangement of individual trees is closely linked to various non-monetary forest aspects, including water quality, wildlife habitat, and aesthetics. Additionally, associating individual tree positions with dendrometric variables like diameter, taper, and species can provide data for highly optimized, site-specific silvicultural prescriptions designed to achieve diverse management objectives. Aerial photogrammetry has proven effective for mapping individual trees; however, its utility is limited due to the inability to directly estimate many dendrometric variables. In contrast, terrestrial mapping methods can directly observe essential individual tree characteristics, such as diameter, but their mapping accuracy is governed by the accuracy of the global satellite navigation system (GNSS) receiver and the density of the canopy obstructions between the receiver and the satellite constellation. In this paper, we introduce an integrated approach that combines a camera-based motion and tree detection system with GNSS positioning, yielding a stem map with twice the accuracy of using a consumer-grade GNSS receiver alone. We demonstrate that large-scale stem maps can be generated in real time, achieving a root mean squared position error of 2.16 m. We offer an in-depth explanation of a visual egomotion estimation algorithm designed to enhance the local consistency of GNSS-based positioning. Additionally, we present a least squares minimization technique for concurrently optimizing the pose track and the positions of individual tree stem[s].

Radar Sensor-Based Ego-Motion Estimation and Indoor Environment Mapping

In this paper, we propose a method for performing the simultaneous localization and mapping (SLAM) in indoor environments using only multiple-input and multiple-output frequency-modulated continuous wave radar. The SLAM is a technology that maps the surrounding environment and position of a platform simultaneously. The following are the steps of the overall the SLAM implementation of the proposed method. First, the ego-velocity of the platform is estimated using the relative velocity with respect to stationary objects. In this case, a random sample consensus algorithm is used in the velocity-angle map generated from the detection results to identify stationary objects. Second, the rotation angle of the platform is estimated using the linear component extracted from the walls in the xy plane. Then, we determine the ego-motion of the platform using the estimated ego-velocity and rotation angle. Finally, we map the position of the platform and the indoor environment simultaneously in the absolute coordinate system. We validate the mapping result generated using our proposed method by comparing it with the ground truth-based mapping result.

Joint Optimization of Depth and Ego-Motion for Intelligent Autonomous Vehicles

The three-dimensional (3D) perception of autonomous vehicles is crucial for localization and analysis of the driving environment, while it involves massive computing resources for deep learning, which can't be provided by vehicle-mounted devices. This requires the use of seamless, reliable, and efficient massive connections provided by the 6G network for computing in the cloud. In this paper, we propose a novel deep learning framework with 6G enabled transport system for joint optimization of depth and ego-motion estimation, which is an important task in 3D perception for autonomous driving. A novel loss based on feature map and quadtree is proposed, which uses feature value loss with quadtree coding instead of photometric loss to merge the feature information at the texture-less region. Besides, we also propose a novel multi-level V-shaped residual network to estimate the depths of the image, which combines the advantages of V-shaped network and residual network, and solves the problem of poor feature extraction results that may be caused by the simple fusion of low-level and high-level features. Lastly, to alleviate the influence of image noise on pose estimation, we propose a number of parallel sub-networks that use RGB image and its feature map as the input of the network. Experimental results show that our method significantly improves the quality of the depth map and the localization accuracy and achieves the state-of-the-art performance.

FADM-SLAM: a fast and accurate dynamic intelligent motion SLAM for autonomous robot exploration involving movable objects

PurposeMany popular simultaneous localization and mapping (SLAM) techniques have low accuracy, especially when localizing environments containing dynamically moving objects since their presence can potentially cause inaccurate data associations. To address this issue, the proposed FADM-SLAM system aims to improve the accuracy of SLAM techniques in environments containing dynamically moving objects. It uses a pipeline of feature-based approaches accompanied by sparse optical flow and multi-view geometry as constraints to achieve this goal.Design/methodology/approachFADM-SLAM, which works with monocular, stereo and RGB-D sensors, combines an instance segmentation network incorporating an intelligent motion detection strategy (iM) with an optical flow technique to improve location accuracy. The proposed AS-SLAM system comprises four principal modules, which are the optical flow mask and iM, the ego motion estimation, dynamic point detection and the feature-based extraction framework.FindingsExperiment results using the publicly available RGBD-Bonn data set indicate that FADM-SLAM outperforms established visual SLAM systems in highly dynamic conditions.Originality/valueIn summary, the first module generates the indication of dynamic objects by using the optical flow and iM with geometric-wise segmentation, which is then used by the second module to compute the starting point of a posture. The third module, meanwhile, first searches for the dynamic feature points in the environment, and second, eliminates them from further processing. An algorithm based on epipolar constraints is implemented to do this. In this way, only the static feature points are retained, which are then fed to the fourth module for extracting important features.

4D iRIOM: 4D Imaging Radar Inertial Odometry and Mapping

Millimeter wave radar can measure distances, directions, and Doppler velocity for objects in harsh conditions such as fog. The 4D imaging radar with both vertical and horizontal data resembling an image can also measure objects' height. Previous studies have used 3D radars for ego-motion estimation. But few methods leveraged the rich data of imaging radars, and they usually omitted the mapping aspect, thus leading to inferior odometry accuracy. This paper presents a real-time imaging radar inertial odometry and mapping method, iRIOM, based on the submap concept. To deal with moving objects and multipath reflections, we use the graduated non-convexity method to robustly and efficiently estimate ego-velocity from a single scan. To measure the agreement between sparse non-repetitive radar scan points and submap points, the distribution-to-multi-distribution distance for matches is adopted. The ego-velocity, scan-to-submap matches are fused with the 6D inertial data by an iterative extended Kalman filter to get the platform's 3D position and orientation. A loop closure module is also developed to curb the odometry module's drift. To our knowledge, iRIOM based on the two modules is the first 4D radar inertial SLAM system. On our and third-party data, we show iRIOM's favorable odometry accuracy and mapping consistency against the FastLIO-SLAM and the EKFRIO. Also, the ablation study reveal the benefit of inertial data versus the constant velocity model, and scan-to-submap matching versus scan-to-scan matching.

RIOT: Recursive Inertial Odometry Transformer for Localisation from Low-Cost IMU Measurements.

Inertial localisation is an important technique as it enables ego-motion estimation in conditions where external observers are unavailable. However, low-cost inertial sensors are inherently corrupted by bias and noise, which lead to unbound errors, making straight integration for position intractable. Traditional mathematical approaches are reliant on prior system knowledge, geometric theories and are constrained by predefined dynamics. Recent advances in deep learning, which benefit from ever-increasing volumes of data and computational power, allow for data-driven solutions that offer more comprehensive understanding. Existing deep inertial odometry solutions rely on estimating the latent states, such as velocity, or are dependent on fixed-sensor positions and periodic motion patterns. In this work, we propose taking the traditional state estimation recursive methodology and applying it in the deep learning domain. Our approach, which incorporates the true position priors in the training process, is trained on inertial measurements and ground truth displacement data, allowing recursion and learning both motion characteristics and systemic error bias and drift. We present two end-to-end frameworks for pose invariant deep inertial odometry that utilises self-attention to capture both spatial features and long-range dependencies in inertial data. We evaluate our approaches against a custom 2-layer Gated Recurrent Unit, trained in the same manner on the same data, and tested each approach on a number of different users, devices and activities. Each network had a sequence length weighted relative trajectory error mean ≤0.4594 m, highlighting the effectiveness of our learning process used in the development of the models.

KISS-ICP: In Defense of Point-to-Point ICP – Simple, Accurate, and Robust Registration If Done the Right Way

Robust and accurate pose estimation of a robotic platform, so-called sensor-based odometry, is an essential part of many robotic applications. While many sensor odometry systems made progress by adding more complexity to the ego-motion estimation process, we move in the opposite direction. By removing a majority of parts and focusing on the core elements, we obtain a surprisingly effective system that is simple to realize and can operate under various environmental conditions using different LiDAR sensors. Our odometry estimation approach relies on point-to-point ICP combined with adaptive thresholding for correspondence matching, a robust kernel, a simple but widely applicable motion compensation approach, and a point cloud subsampling strategy. This yields a system with only a few parameters that in most cases do not even have to be tuned to a specific LiDAR sensor. Our system performs on par with state-of-the-art methods under various operating conditions using different platforms using the same parameters: automotive platforms, UAV-based operation, vehicles like segways, or handheld LiDARs. We do not require integrating IMU data and solely rely on 3D point clouds obtained from a wide range of 3D LiDAR sensors, thus, enabling a broad spectrum of different applications and operating conditions. Our open-source system operates faster than the sensor frame rate in all presented datasets and is designed for real-world scenarios.

DeepEgo: Deep Instantaneous Ego-Motion Estimation Using Automotive Radar

The problem of instantaneous ego-motion estimation with mm-wave automotive radar is studied. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DeepEgo</i> , a deep learning-based method, is proposed for achieving robust and accurate ego-motion estimation. A hybrid approach that uses neural networks to extract complex features from input point clouds and applies weighted least squares (WLS) for motion estimation is utilized in <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DeepEgo</i> . Additionally, a novel loss function, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Doppler loss</i> , is proposed to locate ‘inlier points’ originating from detected stationary objects without human annotation. Finally, a challenging real-world automotive radar dataset is selected for extensive performance evaluation. Compared to other methods selected from the literature, significant improvements in estimation accuracy, long-term stability, and runtime performance of DeepEgo in comparison to other methods are demonstrated.

DecoupledPoseNet: Cascade Decoupled Pose Learning for Unsupervised Camera Ego-Motion Estimation

Although many impressive works on learning-based camera ego-motion estimation methods have been proposed recently, most of them promote the accuracy of camera pose estimation by various sequential learning with loop closure optimization, while neglecting the improvement of PoseNet itself. In this paper, we focus on the coupling of rotation and translation in ego-motion estimation, and design a cascade decoupling structure to separately learn the rotation and translation of camera relative motion between adjacent frames. Meanwhile, a rigid-aware unsupervised learning framework with iterative pose refinement scheme is proposed for camera ego-motion estimation. It can disambiguate rigid motion and deformations in dynamic scenarios by jointly learning of optical flow, stereo disparity and camera pose. Validated with evaluation experiments on the public available datasets, our method is superior to the state-of-the-art unsupervised methods, and can achieve comparable results with the supervised ones.