Multi-sensor Fusion Framework Research Articles

FGO-MFI: factor graph optimization-based multi-sensor fusion and integration for reliable localization

Reliable and precise information pertaining to the position, velocity, and attitude is essential for automated driving. This paper proposes FGO-MFI, a cost-effective and robust multi-sensor fusion and integration localization framework that utilizes factor graph optimization. Firstly, a tightly coupled Global Navigation Satellite Systems (GNSS)/on-board sensor fusion localization framework is established to estimate vehicle states, including position, velocity, and attitude. To address the large drift rate of the Inertial Measurement Unit (IMU), this study introduces a novel IMU/Dynamics pre-integration method based on the vehicle dynamics model. We establish a two-degree-of-freedom vehicle dynamics model utilizing measurements from the wheel speed sensor and steering wheel angle sensor. The IMU/Dynamics factor is devised through a close integration of the model output and IMU pre-integration, enabling the construction of precise odometry with low-cost on-board sensors. Then, to address the issue of the non-Gaussian distribution of GNSS pseudorange error, this paper employs a Gaussian Mixture Model (GMM) to characterize the pseudorange noise, which is then applied to further sensor fusion. Given the time-varying nature of pseudorange noise, the expectation maximization algorithm is utilized to estimate GMM parameters online, leveraging the pseudorange residuals within a sliding window. Comprehensive experiments, inclusive of challenging scenarios such as urban canyons, tunnels, and wooded areas, have been carried out. They affirm the superior performance of the proposed method. Experiments have shown that our method demonstrates reliable localization across different statuses of GNSS signal, exhibiting a 39.4% improvement in the root mean square error of position error when compared to the state-of-the-art. Additionally, this FGO-MFI is a general sensor data fusion framework and is able to incorporate diverse sensor measurements, for example, from cameras and LiDARs to provide more reliable and accurate localization information.

Unified framework for multi-sensor distributed fusion with memory configuration

This paper investigates the multi-sensor distributed fusion with memory configuration for multi-target automatic tracking (MTAT) in clutter. Aimed at fully addressing the problems of both track origin ambiguity and the strong-correlation among system track and its paired local tracks in the framework, this paper derives an Exact Unified Multisensory Tracks Fusion with Memory (EU-MTFwM) method that fuses the system track with enumerated combinatorial tracks to achieve fusing all sensors' local tracks at once. It deploys track-to-track probabilistic association (T2TPA) and target-existence based track automatic management (TAM) to mitigate track origin ambiguity caused by clutter disturbance and the presence of multiple targets. Additionally, it implements the exact distributed kinematic state estimation (exact DKSE) to remove strong correlation stemming from common historical measurements and process noise. However, as the number of sensors to be fused increases, the proposed EU-MTFwM suffers from the combinatorial explosion. An Implementable Unified Multisensory Tracks Fusion with Memory (IU-MTFwM) method is then derived to sequentially fuse the system track with origin-ambiguous local tracks from multiple sensors one-by-one, providing a unified but computationally efficient alternative to the EU-MTFwM. Furthermore, a novel system track initialization technology is proposed to trigger the multisensory tracks fusion recursion. Numerical results show the proposed two unified methods achieve comparable tracking performance to the centralized fusion benchmark but with significantly reduced communication consumption, and demonstrate intensive improvement over the enhanced state-of-the-art that incorporates our proposed system track initialization into its fusion recursion.

A novel multi-sensor hybrid fusion framework

Multi-sensor data fusion has emerged as a powerful approach to enhance the accuracy and robustness of diagnostic systems. However, effectively integrating multiple sensor data remains a challenge. To address this issue, this paper proposes a novel multi-sensor fusion framework. Firstly, a vibration signal weighted fusion rule based on Kullback–Leibler divergence-permutation entropy is introduced, which adaptively determines the weighting coefficients by considering the positional differences of different sensors. Secondly, a lightweight multi-scale convolutional neural network is designed for feature extraction and fusion of multi-sensor data. An ensemble classifier is employed for fault classification, and an improved hard voting strategy is proposed to achieve more reliable decision fusion. Finally, the superiority of the proposed method is validated using modular state detection data from the Kaggle database.

High Precision Vehicle Location Method Based on Multi-sensor Fusion in the Urban Complex Environment

In this paper, a high-precision vehicle positioning method is proposed, which can achieve centimeter-level positioning accuracy in a complex environment. Based on the information of multiple sensors (GNSS, Lidar, and IMU), the multi-sensor fusion framework is used to fuse the positioning measurements from different sources with Kalman filter, which achieves high-precision positioning in the case of GNSS disconnection, the accuracy and robustness of the positioning system are significantly improved, and the effectiveness of the method is verified. The average RMS accuracy of 0.07m is realized, which meets the requirements of high-precision positioning for autonomous vehicles.

A multi-channel hybrid deep learning framework for multi-sensor fusion enabled human activity recognition

Smart and connected health (SCH) accelerates the development and integration of information science and engineering approaches to support the digital transformation of health and medicine in populated societies. Sensor data based human activity recognition (HAR) as an effective means of SCH is promising for healthcare monitoring and ambient assisted living. This paper focuses on multi-position sensor data fusion enabled HAR, and proposes a multi-channel deep learning framework. The main contributions include: (1) A multi-channel hybrid deep learning model (1DCNN-Att-BiLSTM) that merges a one-dimensional convolutional neural network, a bidirectional long short-term memory model, and an attention mechanism is proposed to significantly improve the recognition performance of HAR models by extracting and selecting local and global behavioral features in the spatial and temporal domains; (2) Publicly accessible datasets Shoaib AR, Shoaib SA, and HAPT are collected and processed to build multi-position sensor data pool for model's evaluation; (3) Thoroughly evaluating the classification performance of seven sensor data fusion patterns from different body positions within the parallel network structure of the multi-channel framework; (4) Comparing the recognition performance metrics of traditional machine learning models, deep learning models, and hybrid models with our proposed model. Extensive experiments demonstrate that our approach achieves competitive performance.

A multi-sensor fusion framework with tight coupling for precise positioning and optimization

In the dynamic landscape of artificial intelligence and robotics, the pursuit of accurate positioning in mobile robots has intensified. This research addresses the limitations of single-sensor SLAM (Simultaneous Localization and Mapping) techniques in complex settings by harnessing the collective strengths of LiDAR (Light Detection And Ranging), Camera, IMU (Inertial Measurement Unit), and GNSS (Global Navigation Satellite System) sensors. The proposed multi-sensor tightly-coupled SLAM framework is an integration of point-line feature-based laser–visual–inertial odometry, visual–laser fusion loop closure detection, and factor graph-based back-end optimization. Within the visual–inertial subsystem, an advanced LSD (Line Segment Detector) feature extraction strategy is introduced, incorporating point-line fusion to enhance visual line features. Additionally, the laser point cloud is projected onto the camera coordinate system, establishing depth associations with visual attributes. Strengthening the robustness of the visual–inertial subsystem in low-texture environments, camera poses undergo optimization through a sliding-window bundle adjustment method. In the laser-inertial subsystem, IMU preintegration mitigates laser point cloud distortion. Extracting edge and plane features, coupled with frame-to-local-map matching, enhances matching efficiency while streamlining computational intricacies. This amalgamation forms the basis of the laser–visual–inertial odometry fusion system. To overcome the limitations of standalone visual and laser-based loop closure detection, a dual-loop closure method utilizing visual–laser fusion is proposed. Leveraging the DBoW2 bag-of-words model, complemented by temporal–spatial consistency checks, enhances detection efficiency and accuracy. The integration of GNSS factors imparts global constraints for expansive outdoor scenarios. Employing factor graph-based back-end optimization, the refinement of laser–visual–inertial odometry factors, visual–inertial odometry factors, IMU preintegration factors, loop closure factors, and GNSS factors culminates in precise global pose estimation and high-fidelity point cloud maps. Through rigorous evaluation of the M2DGR dataset and a mobile robot platform, the proposed methodology emerges as an exemplar of performance, showcasing superiority over the state-of-the-art LIO-SAM technique. Achieving a reduction of 2.86m and 3.23m in the root mean square error of absolute pose estimation across divergent environments, this approach exhibits remarkable efficacy in outdoor scenarios, thereby elevating the precision and resilience of SLAM algorithms for mobile robots.

AFTR: A Robustness Multi-Sensor Fusion Model for 3D Object Detection Based on Adaptive Fusion Transformer.

Multi-modal sensors are the key to ensuring the robust and accurate operation of autonomous driving systems, where LiDAR and cameras are important on-board sensors. However, current fusion methods face challenges due to inconsistent multi-sensor data representations and the misalignment of dynamic scenes. Specifically, current fusion methods either explicitly correlate multi-sensor data features by calibrating parameters, ignoring the feature blurring problems caused by misalignment, or find correlated features between multi-sensor data through global attention, causing rapidly escalating computational costs. On this basis, we propose a transformer-based end-to-end multi-sensor fusion framework named the adaptive fusion transformer (AFTR). The proposed AFTR consists of the adaptive spatial cross-attention (ASCA) mechanism and the spatial temporal self-attention (STSA) mechanism. Specifically, ASCA adaptively associates and interacts with multi-sensor data features in 3D space through learnable local attention, alleviating the problem of the misalignment of geometric information and reducing computational costs, and STSA interacts with cross-temporal information using learnable offsets in deformable attention, mitigating displacements due to dynamic scenes. We show through numerous experiments that the AFTR obtains SOTA performance in the nuScenes 3D object detection task (74.9% NDS and 73.2% mAP) and demonstrates strong robustness to misalignment (only a 0.2% NDS drop with slight noise). At the same time, we demonstrate the effectiveness of the AFTR components through ablation studies. In summary, the proposed AFTR is an accurate, efficient, and robust multi-sensor data fusion framework.

Solid-State-LiDAR-Inertial-Visual Odometry and Mapping via Quadratic Motion Model and Reflectivity Information

This paper proposes a solid-state-LiDAR-inertial-visual fusion framework containing two subsystems: the solid-state-LiDAR-inertial odometry (SSLIO) subsystem and the visual-inertial odometry (VIO) subsystem. Our SSLIO subsystem has two novelties that enable it to handle drastic acceleration and angular velocity changes: (1) the quadratic motion model is adopted in the in-frame motion compensation step of the LiDAR feature points, and (2) the system has a weight function for each residual term to ensure consistency in geometry and reflectivity. The VIO subsystem renders the global map in addition to further optimizing the state output by the SSLIO. To save computing resources, we calibrate our VIO subsystem’s extrinsic parameter indirectly in advance, instead of using real-time estimation. We test the SSLIO subsystem using publicly available datasets and a steep ramp experiment, and show that our SSLIO exhibits better performance than the state-of-the-art LiDAR-inertial SLAM algorithm Point-LIO in terms of coping with strong vibrations transmitted to the sensors due to the violent motion of the crawler robot. Furthermore, we present several outdoor field experiments evaluating our framework. The results show that our proposed multi-sensor fusion framework can achieve good robustness, localization and mapping accuracy, as well as strong real-time performance.

Multi-Modal Feature Constraint Based Tightly Coupled Monocular Visual-LiDAR Odometry and Mapping

In this paper, we present a novel multi-sensor fusion framework for tightly coupled monocular visual-LiDAR odometry and mapping. Compared to previous visual-LiDAR fusion frameworks, our proposed framework leverages more constraints among LiDAR features and visual features and integrates that in a tightly coupled approach. Specifically, the framework starts with a preprocess module which contains LiDAR feature extraction, visual feature extraction and tracking, and visual feature depth recover. Then a frame-to-frame odometry module is established by fusing visual feature tracking and LiDAR feature match between frames, aiming to provide a coarse pose estimation for next module. Finally, to refine the pose and build a multi-modal map, we introduce a multi-modal mapping module that tightly couple multi-modal feature constraints by matching or registering multi-modal features to multi-modal map. In addition, the proposed fusion framework also functions well in sensor-degraded environment (texture-less or structure-less), which increases its robustness. The effectiveness and performance of the proposed fusion framework are demonstrated and evaluated on the public KITTI odometry benchmark, and results show that our proposed fusion framework achieves comparable performance compared with the state-of-the-art visual-LiDAR odometry frameworks.

Learning a Novel LiDAR Submap-Based Observation Model for Global Positioning in Long-Term Changing Environments

In complex environments with long-term changes such as light, seasonal, and viewpoint changes, robust, accurate, and high-frequency global positioning based on light detection and ranging (LiDAR) map is still a challenge, which is crucial for autonomous vehicles or robots. To this end, a novel observation model that relies on the siamese multitask convolutional neural networks (CNNs) with multimodule cascaded is creatively presented in this article. In particular, a new pseudoimage representing LiDAR submap is designed to enrich scene texture and enhance rotation invariance. Besides, novel siamese CNNs that are coupled by NeXtVLAD and long short-term memory is designed for the first time, which can reliably predict similarity and quaternion at the same time. Finally, the predicted quaternion observation is integrated into the extended Kalman filter framework for multisensor fusion to achieve robust high-frequency global pose estimation. Extensive evaluations on KITTI, NCLT, and real-world datasets suggest that the proposed method not only obtains the remarkable precision-recall performance, but also effectively enhances and improves the robustness and accuracy of long-term positioning.

Multi-Sensor Data Fusion for Real-Time Multi-Object Tracking

Sensor data fusion is essential for environmental perception within smart traffic applications. By using multiple sensors cooperatively, the accuracy and probability of the perception are increased, which is crucial for critical traffic scenarios or under bad weather conditions. In this paper, a modular real-time capable multi-sensor fusion framework is presented and tested to fuse data on the object list level from distributed automotive sensors (cameras, radar, and LiDAR). The modular multi-sensor fusion architecture receives an object list (untracked objects) from each sensor. The fusion framework combines classical data fusion algorithms, as it contains a coordinate transformation module, an object association module (Hungarian algorithm), an object tracking module (unscented Kalman filter), and a movement compensation module. Due to the modular design, the fusion framework is adaptable and does not rely on the number of sensors or their types. Moreover, the method continues to operate because of this adaptable design in case of an individual sensor failure. This is an essential feature for safety-critical applications. The architecture targets environmental perception in challenging time-critical applications. The developed fusion framework is tested using simulation and public domain experimental data. Using the developed framework, sensor fusion is obtained well below 10 milliseconds of computing time using an AMD Ryzen 7 5800H mobile processor and the Python programming language. Furthermore, the object-level multi-sensor approach enables the detection of changes in the extrinsic calibration of the sensors and potential sensor failures. A concept was developed to use the multi-sensor framework to identify sensor malfunctions. This feature will become extremely important in ensuring the functional safety of the sensors for autonomous driving.

Research on Multi-Sensor Fusion SLAM Algorithm Based on Improved Gmapping

Simultaneous Localization and Mapping (SLAM) is the core technology of the intelligent robot system, and it is also the basis for its autonomous movement. In recent years, it has been found that SLAM using a single sensor has certain limitations, such as Inertial Measurement Unit (IMU) noise and serious drift, and 2D radar can only detect environmental information on the same horizontal plane. In this regard, this paper constructs a multi-sensor back-end fusion SLAM algorithm that combines vision, laser, encoder and IMU information. Experiments have proved that compared with using a single sensor, the application of a multi-sensor fusion system makes the edges of the constructed map clearer and the noise reduced. Aiming at the problem of increased calculation caused by particle degradation and too many particles, this paper improves the Gmappping algorithm, and uses the combination of selective resampling and Kullback-Leibler Distance (KLD) sampling to complete resampling. It has been proved by experiments that compared with the original algorithm of Gmapping, the application of the improved algorithm increases the particle convergence speed by 39.85% in the process of indoor mapping. Aiming at the problems that the traditional loop detection algorithm is easily affected by environmental factors, resulting in low detection accuracy, and the loop detection algorithm based on deep convolutional neural network has a large amount of calculation and takes a long time to detect. The main research of this paper is to apply a deep learning-based loop detection algorithm on the multi-sensor fusion framework, and use the combination of high-dimensional and low-dimensional features of the image for loop detection. This paper uses different algorithms to conduct comparative experiments on the dataset CityCentre. The experimental results show that compared with the traditional algorithms Bag of Words (BoW), AlexNet algorithm, VGG19 algorithm, and ResNet32 algorithm, the accuracy of the algorithm proposed in this paper has increased by 31.26%, 14.21%, 3.05%, and 1.56%, respectively. In addition, the comparison experiment results of SLAM mapping with the original Real-Time Appearance-Based Mapping (RTAB-MAP) algorithm prove that the loop closure detection algorithm based on deep learning proposed in this paper can enable the system to better build a globally consistent map, including more environmental information.

VWR-SLAM: Tightly Coupled SLAM System Based on Visible Light Positioning Landmark, Wheel Odometer, and RGB-D Camera

Visible light positioning (VLP) is a promising technology since it can provide high-accuracy indoor localization based on the existing lighting infrastructure. Most VLP systems require a prior light-emitting diode (LED) location map, termed a VLP-landmark map in this article, for which manual surveys are costly in practical deployment at scale. What is more, the existing approaches also require dense LED deployments. In this work, we proposed a multisensor fusion framework, termed VWR-simultaneous localization and mapping (SLAM), which tightly fused the VLP, wheel odometer, and red green dlue-depth map (RGB-D) camera to achieve SLAM. Our VWR-SLAM can provide accurate and robust robot localization and navigation in LED shortage/outage situations, meanwhile, constructing the 3-D sparse environment map and the 3-D VLP-landmark map without tedious manual measurements. The experimental results show that our proposed scheme can provide an average robot positioning accuracy of 1.81 cm and an LED mapping accuracy of 3.01 cm.

The IMU/UWB/odometer fusion positioning algorithm based on EKF

High-precision indoor positioning is the basis of factory intelligent management. However, the positioning accuracy will decrease because of the complex environment. This study proposes a multi-sensor fusion framework to fuse the data of Ultra Wide Band (UWB), inertial measurement unit (IMU), and odometer. First fuse the data from UWB and IMU by using EKF to obtain attitude, velocity, and position. And then fuse the speed and output with the odometer output using complementary filtering to increase accuracy. At the same time, the algorithm can output higher frequency positioning data. We evaluate the performance of the algorithm on mobile robots. Experimental data is from a 6-axis IMU and 5 UWB radio sensor devices. And the result shows that the position RMSE of our algorithm is 3.29 centimeters and our comprehensive localization algorithm can increase localization accuracy in complex environments compared with only UWB algorithm.

Health, Security and Fire Safety Process Optimisation Using Intelligence at the Edge.

The proliferation of sensors to capture parametric measures or event data over a myriad of networking topologies is growing exponentially to improve our daily lives. Large amounts of data must be shared on constrained network infrastructure, increasing delays and loss of valuable real-time information. Our research presents a solution for the health, security, safety, and fire domains to obtain temporally synchronous, credible and high-resolution data from sensors to maintain the temporal hierarchy of reported events. We developed a multisensor fusion framework with energy conservation via domain-specific "wake up" triggers that turn on low-power model-driven microcontrollers using machine learning (TinyML) models. We investigated optimisation techniques using anomaly detection modes to deliver real-time insights in demanding life-saving situations. Using energy-efficient methods to analyse sensor data at the point of creation, we facilitated a pathway to provide sensor customisation at the "edge", where and when it is most needed. We present the application and generalised results in a real-life health care scenario and explain its application and benefits in other named researched domains.

Rolling bearing fault diagnosis method based on multi-sensor two-stage fusion

Despite the great achievements of deep learning methods based on a single sensor in fault diagnosis, learning useful information from multi-sensor data is still a challenge. In order to make full use of multi-sensor information and improve the performance of rolling bearing fault diagnosis, a novel multi-sensor information fusion framework is proposed in this paper. First, a multi-sensor-based multi-frequency information fusion method is proposed. The multi-frequency information of each sensor is segmented first to enhance the datasets, and then a weighted fusion rule based on fuzzy entropy is constructed to fuse the information of different frequency components for multi-sensors. Second, a multi-kernel attention convolutional neural network is designed to realize multi-frequency feature capture, fusion, and fault classification of multi-sensors. Finally, two different rolling bearing datasets are used to implement fault diagnosis experiments. Experimental results show that the proposed method outperforms the comparative methods in terms of diagnostic performance and robustness.

Vehicle accident sub-classification modeling using stacked generalization: A multisensor fusion approach

Road accidents caused by several factors are responsible for the deaths and injuries of people. The majority of deaths occur within first few hours of an accident. If in case of an accident the victim becomes incapacitated, and there is no onlooker to help, then an accurate automated accident reporting system can significantly reduce the number of such deaths. This paper reports an Android smartphone-based end-to-end Internet of Things (IoT) system that can transmit accident information to emergency services and affected families once a vehicle accident is detected. Along with the detection, the classification of vehicle accidents can be very helpful in identifying suitable life-saving medical-aid and appropriate rescue operation equipment. The main objective of this research work is to develop a machine learning (ML) model that can detect as well as classify vehicle accidents accurately into eight categories. To enhance the classification efficacy of the system, a multi-sensor fusion framework has been proposed that incorporates various sensor-fusion techniques at different levels of its implementation along with several preprocessing methods such as 10-ms moving maximum, complementary filters, and a 2-sec sliding window. The framework uses Logistic Regression (LR) based stacked generalization approach to combine the decisions of three ML classifiers, which are based on Decision Tree (DT), Naïve Bayes (NB), and Random Forest (RF) methods. The LR-based stacking classifier is found to be very accurate with an F1-score of 0.95, whose performance is significantly better than the performance of all individual base-classifiers.

Map Construction Based on LiDAR Vision Inertial Multi-Sensor Fusion

In order to make up for the shortcomings of independent sensors and provide more reliable estimation, a multi-sensor fusion framework for simultaneous localization and mapping is proposed in this paper. Firstly, the light detection and ranging (LiDAR) point cloud is screened in the front-end processing to eliminate abnormal points and improve the positioning and mapping accuracy. Secondly, for the problem of false detection when the LiDAR is surrounded by repeated structures, the intensity value of the laser point cloud is used as the screening condition to screen out robust visual features with high distance confidence, for the purpose of softening. Then, the initial factor, registration factor, inertial measurement units (IMU) factor and loop factor are inserted into the factor graph. A factor graph optimization algorithm based on a Bayesian tree is used for incremental optimization estimation to realize the data fusion. The algorithm was tested in campus and real road environments. The experimental results show that the proposed algorithm can realize state estimation and map construction with high accuracy and strong robustness.

R $^2$ LIVE: A Robust, Real-Time, LiDAR-Inertial-Visual Tightly-Coupled State Estimator and Mapping

In this letter, we propose a robust, real-time tightly-coupled multi-sensor fusion framework, which fuses measurements from LiDAR, inertial sensor, and visual camera to achieve robust and accurate state estimation. Our proposed framework is composed of two parts: the filter-based odometry and factor graph optimization. To guarantee real-time performance, we estimate the state within the framework of error-state iterated Kalman-filter, and further improve the overall precision with our factor graph optimization. Taking advantage of measurements from all individual sensors, our algorithm is robust enough to various visual failure, LiDAR-degenerated scenarios, and is able to run in real time on an on-board computation platform, as shown by extensive experiments conducted in indoor, outdoor, and mixed environments of different scale (see attached video). Moreover, the results show that our proposed framework can improve the accuracy of state-of-the-art LiDAR-inertial or visual-inertial odometry. To share our findings and to make contributions to the community, we open source our codes on our Github.

A generic multi-sensor fusion scheme for localization of autonomous platforms using moving horizon estimation

In this paper, a generic multi-sensor fusion framework is developed for the localization of intelligent vehicles and mobile robots. The localization framework is based on moving horizon estimation (MHE). Unlike the commonly used probabilistic filtering algorithms – for example, extended Kalman filter (EKF) and unscented Kalman filter (UKF) – MHE relies on solving successive least squares optimization problems over the innovation of multiple sensors’ measurements and a specific estimation horizon. In this paper, we present an efficient and generic multi-sensor fusion scheme, based on MHE. The proposed multi-sensor fusion scheme is capable of operating with different sensors’ rates, missing measurements, and outliers. Moreover, the proposed scheme is based on a multi-threading architecture to reduce its computational cost, making it more feasible for practical applications. The MHE fusion method is tested using simulated data as well as real experimental data sequences from an intelligent vehicle and a mobile robot combining measurements from different sensors to get accurate localization results. The performance of MHE is compared against that of UKF, where the MHE estimation results show superior performance.