Multi-sensor Fusion Method Research Articles

PV-LaP: Multi-sensor fusion for 3D Scene Understanding in intelligent transportation systems

Intelligent transportation systems are pivotal in modern urban development, aiming to enhance traffic management efficiency, safety, and sustainability. However, existing 3D Visual Scene Understanding methods often face challenges of robustness and high computational complexity in complex traffic environments. This paper proposes a Multi-Sensor Signal Fusion method based on PV-RCNN and LapDepth (PV-LaP) to improve 3D Visual Scene Understanding. By integrating camera and LiDAR data, the PV-LaP method enhances environmental perception accuracy. Evaluated on the KITTI and WHU-TLS datasets, the PV-LaP framework demonstrated superior performance. On the KITTI dataset, our method achieved an Absolute Relative Error (Abs Rel) of 0.079 and a Root Mean Squared Error (RMSE) of 3.014, outperforming state-of-the-art methods. On the WHU-TLS dataset, the method improved 3D reconstruction precision with a PSNR of 19.15 and an LPIPS of 0.299. Despite its high computational demands, PV-LaP offers significant improvements in accuracy and robustness, providing valuable insights for the future development of intelligent transportation systems.

A Multi-Sensor Fusion Underwater Localization Method Based on Unscented Kalman Filter on Manifolds.

In recent years, the simplified computation of position and velocity changes in nonlinear systems using Lie groups and Lie algebra has been widely used in the study of robot localization systems. The unscented Kalman filter (UKF) can effectively deal with nonlinear systems through the unscented transformation, and in order to more accurately describe the robot localization system, the UKF method based on Lie groups has been studied successively. The computational complexity of the UKF on Lie groups is high, and in order to simplify its computation, the Lie groups are applied to the manifold, which efficiently handles the state and uncertainty and ensures that the system maintains the geometric constraints and computational simplicity during the updating process. In this paper, a multi-sensor fusion localization method based on an unscented Kalman filter on manifolds (UKF-M) is investigated. Firstly, a system model and a multi-sensor model are established based on an Autonomous Underwater Vehicle (AUV), and a corresponding UKF-M is designed for the system. Secondly, the multi-sensor fusion method is designed, and the fusion method is applied to the UKF-M. Finally, the proposed method is validated using an underwater cave dataset. The experiments demonstrate that the proposed method is suitable for underwater environments and can significantly correct the cumulative error in the trajectory estimation to achieve accurate underwater localization.

Multi-sensor signal fusion method for rolling bearing based on the standard relative mean–variance value and random weighting algorithm

In the study, an optimization fusion method based on the standard relative mean–variance value and a random weighting algorithm is proposed to effectively utilize multisensory-acquired signals for realizing accurate monitoring of rolling bearing faults. First, a novel feature-standard relative mean variance value is developed to adjust the balance parameter in the random weighting algorithm to adaptively obtain the best estimate value. Second, upper and lower thresholds of the balance parameter are provided to improve the noise reduction and feature retention abilities of the algorithm in the fusion process. Finally, the weight of each sensor signal is determined based on the overall variance of each sensor. The parameters in the optimization fusion method can be adaptively adjusted based on the collected signal, which has practical value. Simulations and laboratory experiments demonstrate the effectiveness of the method.

End-to-end multi-sensor fusion method based on deep reinforcement learning in UASNs

Underwater acoustic sensor networks (UASNs) have attracted considerable attention and are extensively employed for tracking underwater targets. However, due to the complex dynamics of underwater tracking environments, establishing accurate correlations between sensors remains elusive and challenging to precisely model. Traditional multi-sensor fusion methods for handling unknown correlations often face issues related to dependence on prior knowledge and a lack of flexibility. In response, we propose a novel end-to-end multi-sensor fusion algorithm rooted in deep reinforcement learning (DRL). To address the challenge of unknown correlations among sensors in UASNs, we formulate the multi-sensor fusion strategy as a Markov Decision Process (MDP) within the framework of reinforcement learning. The Proximal Policy Optimization (PPO) method is implemented to tackle this MDP without scalability limitations, aiming to derive an optimal sensor scheduling policy for UASNs. To enhance practical applicability, we adopt a mock data approach for algorithm training, eliminating the necessity for ground truth information on non-cooperative targets. Finally, both simulation and real-world experimental results demonstrate the superior performance of the proposed algorithm. It attains an improvement of at least 15% in tracking accuracy and exhibits a notable enhancement in stability when contrasted with traditional algorithms.

Multi-sensor data fusion reconstruction method for vibration dynamic responses of aerospace structures

The dynamic responses of key locations are important inputs for the life and reliability assessment of spacecraft structures. Due to the limited sensing resources, most critical responses are difficult to measure directly. A structural dynamic response reconstruction method is necessary. The responses of target locations can be reconstructed based on the empirical mode decomposition (EMD) of measured signals and the modal superposition. However, the structural modal information contained in the measured signal of a single sensor is limited, affecting the reconstruction accuracy. In this paper, a response reconstruction method based on multi-sensor data fusion is proposed. It is applied to a main load-bearing structure of a spacecraft and its typical components to verify its strain response reconstruction effect under random vibration loads. The experimental results show that multi-sensor data fusion improves the strain reconstruction accuracy. The maximum reduction in reconstruction error is from 8.7% to 1.3%. The reconstruction accuracy is further improved with the increase in the number of sensors. The optimal weighted fusion strategy for this problem is the weights defined by the Euclidean distance (EUC) or the dynamic time warping distance (DTW). The fusion results show a better performance with the increase in the power of the defined distance. The proposed multi-sensor fusion method improves the reconstruction accuracy via supplementing structural information to each other and eliminating the instability of single measured signals. More accurate dynamic responses via reconstruction reduce the large input uncertainty in life prediction and lay the foundation for building structural digital twins and managing structural health more effectively.

High-Precision Map Construction in Degraded Long Tunnel Environments of Urban Subways

In response to the demand for high-precision point cloud mapping of subway trains in long tunnel degradation scenarios in major urban cities, we propose a map construction method based on LiDAR and inertial measurement sensors. This method comprises a tightly coupled frontend odometry system based on error Kalman filters and backend optimization using factor graphs. In the frontend odometry, inertial calculation results serve as predictions for the filter, and residuals between LiDAR points and local map plane point clouds are used for filter updates. The global pose graph is constructed based on inter-frame odometry and other constraint factors, followed by a smoothing optimization for map building. Multiple experiments in subway tunnel scenarios demonstrate that the proposed method achieves robust trajectory estimation in long tunnel scenes, where classical multi-sensor fusion methods fail due to sensor degradation. The proposed method achieves a trajectory consistency of 0.1 m in tunnel scenes, meeting the accuracy requirements for train arrival, parking, and interval operations. Additionally, in an industrial park scenario, the method is compared with ground truth provided by inertial navigation, showing an accumulated error of less than 0.2%, indicating high precision.

GRVINS: Tightly Coupled GNSS-Range-Visual-Inertial System

Bridge inspection is currently a labor intensive task. Utilizing unmanned aerial vehicles (UAVs) to assist in inspection tasks is a promising direction. However, enabling UAVs for autonomous inspection involves the UAV state estimation problems. Since parts of UAV sensors could be unavailable, how to estimate states via sensor fusion is the key. In this paper, we propose a tightly-coupled nonlinear optimization-based system that integrates four kinds of sensors: camera, IMU, Ultra-wideband (UWB) range measurements, and global navigation satellite system (GNSS). Due to the tightly-coupled multi-sensor fusion method and system design, the system takes the advantage of the four sensors, and can seamlessly respond to indoor and outdoor GNSS and UWB loss or reacquisition. It can effectively reduce the long-term trajectory drift and provide smooth and continuous state estimation. The experimental results show that the proposed method outperforms the state-of-the-art approaches.

An advanced subway train localization system using a vision-based kilometer marker recognition-assisted multi-sensor fusion method

To solve the high costs of the existing transponder/odometer-based wayside subway train localization method and the discontinuity of the GNSS-based subway train localization method in blocked GNSS environments, an advanced and alternative subway train localization system using the INS/Vision/ODO/MM fusion method is first proposed in this research. To evaluate the performance of the proposed INS/Vision/ODO/MM fusion method, a railway test was conducted on the Beijing Changping Line. The results show the proposed localization method can provide available and continuous 1-D positioning results with an accuracy of 10.82 m, which is a 52.71% improvement compared with the existing ATO. It also shows that the proposed vision-aided multi-sensor fusion method can be used as an alternative to the existing wayside-transponder-based localization system to provide navigation solutions for train safe operation without relying on wayside equipment and with low cost and easy maintenance.

FusionPlanner: A multi-task motion planner for mining trucks via multi-sensor fusion

In recent years, significant achievements have been made in motion planning for intelligent vehicles. However, as a typical unstructured environment, open-pit mining attracts limited attention due to its complex operational conditions and adverse environmental factors. A comprehensive paradigm for unmanned transportation in open-pit mines is proposed in this research. Firstly, we propose a multi-task motion planning algorithm, called FusionPlanner, for autonomous mining trucks by the multi-sensor fusion method to adapt both lateral and longitudinal control tasks for unmanned transportation. Then, we develop a novel benchmark called MiningNav, which offers three validation approaches to evaluate the trustworthiness and robustness of well-trained algorithms in transportation roads of open-pit mines. Finally, we introduce the Parallel Mining Simulator (PMS), a new high-fidelity simulator specifically designed for open-pit mining scenarios. PMS enables the users to manage and control open-pit mine transportation from both the single-truck control and multi-truck scheduling perspectives. The performance of FusionPlanner is tested by MiningNav in PMS, and the empirical results demonstrate a significant reduction in the number of collisions and takeovers of our planner. We anticipate our unmanned transportation paradigm will bring mining trucks one step closer to trustworthiness and robustness in continuous round-the-clock unmanned transportation.

NETSI: Nonequidistant time series imaging-based multi-sensor fusion method for smart fruit sorting

NETSI: Nonequidistant time series imaging-based multi-sensor fusion method for smart fruit sorting

Truth-Correction-Based Multisensor Fusion Method for HTS Maglev Train’s Position Detection

Truth-Correction-Based Multisensor Fusion Method for HTS Maglev Train’s Position Detection

Human-Robot Interaction Using Learning from Demonstrations and a Wearable Glove with Multiple Sensors.

Human-robot interaction is of the utmost importance as it enables seamless collaboration and communication between humans and robots, leading to enhanced productivity and efficiency. It involves gathering data from humans, transmitting the data to a robot for execution, and providing feedback to the human. To perform complex tasks, such as robotic grasping and manipulation, which require both human intelligence and robotic capabilities, effective interaction modes are required. To address this issue, we use a wearable glove to collect relevant data from a human demonstrator for improved human-robot interaction. Accelerometer, pressure, and flexi sensors were embedded in the wearable glove to measure motion and force information for handling objects of different sizes, materials, and conditions. A machine learning algorithm is proposed to recognize grasp orientation and position, based on the multi-sensor fusion method.

Chinese sign language recognition based on surface electromyography and motion information

Sign language (SL) has strong structural features. Various gestures and the complex trajectories of hand movements bring challenges to sign language recognition (SLR). Based on the inherent correlation between gesture and trajectory of SL action, SLR is organically divided into gesture-based recognition and gesture-related movement trajectory recognition. One hundred and twenty commonly used Chinese SL words involving 9 gestures and 8 movement trajectories, are selected as research and test objects. The method based on the amplitude state of surface electromyography (sEMG) signal and acceleration signal is used for vocabulary segmentation. The multi-sensor decision fusion method of coupled hidden Markov model is used to complete the recognition of SL vocabulary, and the average recognition rate is 90.41%. Experiments show that the method of sEMG signal and motion information fusion has good practicability in SLR.

Multi-sensor information fusion localization of rare-earth suspended permanent magnet maglev trains based on adaptive Kalman algorithm.

Since the positioning accuracy of sensors degrades due to noise and environmental interference when a single sensor is used to localize a suspended rare-earth permanent magnetically levitated train, a multi-sensor information fusion method using multiple sensors and self-correcting weighting is proposed for permanent magnetic levitated train localization. A decay memory factor is introduced to reduce the weight of the influence of historical measurement data on the fusion estimation, thus enhancing the robustness of the fusion algorithm. The Kalman filtering results suffer from inaccuracy when process noise is present in the system. In this paper, we use a covariance adaptive scheme that replaces the prediction step of the Kalman filter with covariance. It uses the covariance adaptive scheme to search the posterior sequence online and reconstruct the prior error covariance. Since the process noise covariance is not used in the new adaptive scheme, the negative impact of the mismatch noise statistics is greatly reduced. Simulation and experimental results show that the use of multi-sensor information fusion and covariance adaptive Kalman algorithm has significant advantages in terms of adaptability, accuracy and simplicity.

Forward Collision Warning Strategy Based on Millimeter-Wave Radar and Visual Fusion

Forward collision warning (FCW) is a critical technology to improve road safety and reduce traffic accidents. However, the existing multi-sensor fusion methods for FCW suffer from a high false alarm rate and missed alarm rate in complex weather and road environments. For these issues, this paper proposes a decision-level fusion collision warning strategy. The vision algorithm and radar tracking algorithm are improved in order to reduce the false alarm rate and omission rate of forward collision warning. Firstly, this paper proposes an information entropy-based memory index for an adaptive Kalman filter for radar target tracking that can adaptively adjust the noise model in a variety of complex environments. Then, for visual detection, the YOLOv5s model is enhanced in conjunction with the SKBAM (Selective Kernel and Bottleneck Attention Mechanism) designed in this paper to improve the accuracy of vehicle target detection. Finally, a decision-level fusion warning fusion strategy for millimeter-wave radar and vision fusion is proposed. The strategy effectively fuses the detection results of radar and vision and employs a minimum safe distance model to determine the potential danger ahead. Experiments are conducted under various weather and road conditions, and the experimental results show that the proposed algorithm reduces the false alarm rate by 11.619% and the missed alarm rate by 15.672% compared with the traditional algorithm.

Multi-head spatio-temporal attention based parallel GRU architecture: a novel multi-sensor fusion method for mechanical fault diagnosis

The sensor signals with multiple measuring points and data types not only bring sufficient information, but also put forward more stringent requirements for multi-sensor fusion efficiency and fault feature extraction capability. The redundancy and conflicts in the information of multi-sensor signals often hinder the accurate extraction of crucial fault features. To address this problem, our study proposes an intelligent mechanical fault diagnosis method, which is based on a multi-head spatio-temporal attention mechanism and parallel gated recurrent units (GRUs) architecture. This method utilizes multiple attention heads to model the correlation information in spatial and temporal dimensions, and employs a parallel GRU network for targeted feature extraction. Finally, it combines local features from different attention heads to achieve flexible scheduling of various spatio-temporal attention modes. This novel application and fusion approach of multi-head attention enables accurate identification of the spatio-temporal value differences in the collected multi-sensor signals from multiple perspectives. Experimental results on multiple mechanical fault datasets show that the proposed method performs well in multi-sensor signals based mechanical fault diagnosis tasks and can maintain effectiveness under small samples and imbalanced data conditions.

Multi-Sensor Graph Transfer Network for Health Assessment of High-Speed Rail Suspension Systems

Suspension systems are significant for safe and comfort operation of high-speed trains. Health assessment is a useful tool to schedule maintenance plans of suspension systems, and furthermore ensure safety operation of high-speed railway transportation. In real operating condition, two problems, i.e. data imbalance and shortage of labelled data, result in difficult for health assessment of the suspension system using deep learning. In this work, a multi-sensor information fusion method, called as multi-sensor graph transfer network (MSGTN), is proposed in basis of deep transfer learning and graph neural network. In the proposed method, a domain-share multi-sensor graph neural network (MSGNN) is firstly proposed to extract features from vibration signals collected from three different positions in train vehicles. A graph-based fusion layer in MSGNN is proposed to fuse multi-sensor information by combining frequency response curves of the suspension system. The MSGTN mainly includes two parts in source and target domains respectively. In the source domain, a simple physical dynamic model of high-speed rail suspension system is built to generate labelled simulation datasets to pre-train MSGNN. In the target domain, the initial hyper-parameters of MSGNN are that of the pre-train model in the source domain. The labelled data in the target domain is fed to fine-tune MSGNN and then the final model for health assessment can be obtained by minimizing the loss function. The effectiveness of the proposed method was verified using real-work operation data.

Multi sensor fusion methods for state of charge estimation of smart lithium-ion batteries

Multi-dimensional sensing is the key characteristic of next generation smart batteries. But the existing researches on multi-sensor fusion methods haven't focused on algorithm mechanism, the global optimal solution has not been built, the superiority has not been proved theoretically. For the first time, this research built the global optimal structure of multi-sensor fusion state estimation algorithm. Specifically, the state of charge (SoC) estimation problem of lithium iron phosphate (LFP) batteries is studied, cooperating with voltage signal, expansion force (EF) signal is introduced. Firstly, a normalization algorithm is introduced to overcome the drift of EF under different cycles and different pre-tightening force. Secondly, the non-monotonic relationship of LFP battery's EF-SoC curve is addressed with a forced monotone transformation method in the transition areas. Then the global optimal multi-sensor fusion method is built, theoretical reductions are carried out to prove the higher precision of multi-sensor fusion methods compared to single-signal methods. Experiments are conducted to verify the effectiveness of the methods, even under the most serious situations, the fusion methods exhibit powerful correction ability, and the root mean square error can be controlled within 3 %. Moreover, the proposed algorithms shows strong tolerance to error sources.

Multi-sensor signals multi-scale fusion method for fault detection of high-speed and high-power diesel engine under variable operating conditions

Detecting faults in high-speed and high-power diesel engines under complex variable operating conditions is highly challenging. Online vibration monitoring systems have been used in such diesel engines in key fields, in which vibration sensors are installed on each cylinder to enable comprehensive monitoring. In this paper, a fault detection method for diesel engines under variable operating conditions is proposed based on multi-sensor signal multi-scale fusion. Firstly, a preprocessing framework is established for the raw vibration signals collected from each cylinder to eliminate random interference and system noise. Then, the resulting signals are phase-aligned based on the engine firing sequence and analyzed using a signal correlation algorithm to produce a multi-sensor multi-scale similarity matrix (MSMSSM). Finally, a multi-branch residual convolutional neural network (MBRCNN) model is constructed with the MSMSSM as the input to detect abnormal health states of the diesel engine. Fault simulation experiments are conducted on a 12-cylinder V-type high-speed and high-power diesel engine test rig. The comparative test results indicate that the proposed MSMSSM-MBRCNN method shows both the highest accuracy of 95.28% and the lowest standard deviation of 3.57% compared to other typical methods. The multi-sensor signals multi-scale fusion method proposed in this paper fully utilizes the key information that remains basically consistent in the synchronous acquisition signals of multiple sensors under different operating conditions. This can effectively reduce the interference of operating condition changes and improve the accuracy and robustness of fault detection.

A Fuzzy Dempster–Shafer Evidence Theory Method with Belief Divergence for Unmanned Surface Vehicle Multi-Sensor Data Fusion

The safe navigation of unmanned surface vehicles in the marine environment requires multi-sensor collaborative perception, and multi-sensor data fusion technology is a prerequisite for realizing the collaborative perception of different sensors. To address the problem of poor fusion accuracy for existing multi-sensor fusion methods without prior knowledge, a fuzzy evidence theory multi-sensor data fusion method with belief divergence is proposed in this paper. First of all, an adjustable distance for measuring discrepancies between measurements is devised to evaluate the degree of measurement closeness to the true value, which improves the adaptability of the method to different classes of sensor data. Furthermore, an adaptive multi-sensor measurement fusion strategy is designed for the case where the sensor accuracy is known in advance. Secondly, the affiliation function of the fuzzy theory is introduced into the evidence theory approach to assign initial evidence of measurements in terms of defining the degree of fuzzy support between measurements, which improves the fusion accuracy of the method. Finally, the belief Jensen–Shannon divergence and the Rényi divergence are combined for measuring the conflict between the evidence pieces to obtain the credibility degree as the reliability of the evidence, which solves the problem of high conflict between evidence pieces. Three examples of multi-sensor data fusion in different domains are employed to validate the adaptability of the proposed method to different kinds of multi-sensors. The maximum relative error of the proposed method for multiple sensor experiments is greater than or equal to 0.18%, and its error accuracy is much higher than the best result of 0.46% among other comparative methods. The experimental results verify that the proposed data fusion method is more accurate than other existing methods.