Multi-sensor Fusion Scheme Research Articles

A novel residual global context shrinkage network based fault diagnosis method for rotating machinery under noisy conditions

In real industrial environments, vibration signals generated during the operation of rotating machinery are typically accompanied by significant noise. Existing deep learning methods often yield unsatisfactory diagnostic results when dealing with noisy signals. To address this problem, a novel residual global context shrinkage network (RGNet) is proposed in this paper. Firstly, to fully utilize the useful information in the raw vibration signal, a multi-sensor fusion strategy based on dispersion entropy is designed as the input of the deep network. Then, the RGNet is designed, which improves the long-distance modeling capability of the deep network while suppressing noise, optimizes the network gradient and computational performance. Finally, the noise suppression ability and feature extraction ability of the RGNet are intuitively revealed through an interpretability study. The advantages of the proposed method are proved through a series of comparison experiments under noisy backgrounds.

End-to-end multi-sensor fusion method based on deep reinforcement learning in UASNs

Underwater acoustic sensor networks (UASNs) have attracted considerable attention and are extensively employed for tracking underwater targets. However, due to the complex dynamics of underwater tracking environments, establishing accurate correlations between sensors remains elusive and challenging to precisely model. Traditional multi-sensor fusion methods for handling unknown correlations often face issues related to dependence on prior knowledge and a lack of flexibility. In response, we propose a novel end-to-end multi-sensor fusion algorithm rooted in deep reinforcement learning (DRL). To address the challenge of unknown correlations among sensors in UASNs, we formulate the multi-sensor fusion strategy as a Markov Decision Process (MDP) within the framework of reinforcement learning. The Proximal Policy Optimization (PPO) method is implemented to tackle this MDP without scalability limitations, aiming to derive an optimal sensor scheduling policy for UASNs. To enhance practical applicability, we adopt a mock data approach for algorithm training, eliminating the necessity for ground truth information on non-cooperative targets. Finally, both simulation and real-world experimental results demonstrate the superior performance of the proposed algorithm. It attains an improvement of at least 15% in tracking accuracy and exhibits a notable enhancement in stability when contrasted with traditional algorithms.

Research on Multi-Sensor Data Collection and Mapping Methods

In today's information society, the importance of collecting data through multiple channels continues to increase. This article focuses on the use of drones for large-area forest information collection and tree analysis. In complex environments, there are a series of problems in obtaining tree mapping information with a single sensor, such as single use scenarios, low accuracy, limited viewing angles, and high requirements for computing power. To address these challenges, this paper proposes a Bayesian-based multi-sensor fusion strategy to overcome the limitations of a single sensor by integrating depth camera and lidar data. Experimental results show that the proposed method significantly improves the accuracy of mapping surrounding trees and the ability to collect information. The multi-sensor fusion strategy effectively reduces the problem of repeated data processing and improves the comprehensiveness of collected data. It meets the requirements of real-time performance and accurate collection of surrounding scenes.

Monitoring of simultaneous saccharification and fermentation of ethanol by multi-source data deep fusion strategy based on near-infrared spectra and electronic nose signals

Fuel ethanol represents a future energy trajectory, and the simultaneous saccharification and fermentation (SSF) technique emerges as the principal approach for ethanol production. This scholarly inquiry offers an innovative means to monitor the SSF process for ethanol meticulously. Employing a profound fusion strategy that effectively amalgamates diverse data sources. The convolutional neural network and recurrent neural network (RNN) architectures are thoughtfully crafted and designed to enable autonomous feature self-learning from near-infrared spectra and electronic nose data. These intricately devised networks further implement data fusion strategies at the granular level of features. Ultimately, a deep fusion correction model was devised and rigorously validated using two distinct data sources, namely near-infrared spectroscopy and electronic nose data. The obtained results demonstrate a discernible improvement in the overall predictive accuracy of the model when employing the fusion feature strategy, surpassing the model constructed solely on a single technical data source. Regarding the monitoring of ethanol content, the optimal RNN fusion model exhibited remarkable performance metrics, with a root mean square error of prediction (RMSEP) value of 3.2265, a coefficient of determination (R2) value of 0.9880, and a relative percent deviation (RPD) value of 9.2662. In terms of monitoring glucose content, the optimal RNN fusion model also demonstrated commendable performance, with the following respective parameters: RMSEP was 3.2770, R2 was 0.9840, and RPD was 8.0085. The overall results indicate that the multi-sensor data fusion strategy not only improves the performance of the model but also provides valuable insights into the fermentation process.

A Fuzzy Dempster–Shafer Evidence Theory Method with Belief Divergence for Unmanned Surface Vehicle Multi-Sensor Data Fusion

The safe navigation of unmanned surface vehicles in the marine environment requires multi-sensor collaborative perception, and multi-sensor data fusion technology is a prerequisite for realizing the collaborative perception of different sensors. To address the problem of poor fusion accuracy for existing multi-sensor fusion methods without prior knowledge, a fuzzy evidence theory multi-sensor data fusion method with belief divergence is proposed in this paper. First of all, an adjustable distance for measuring discrepancies between measurements is devised to evaluate the degree of measurement closeness to the true value, which improves the adaptability of the method to different classes of sensor data. Furthermore, an adaptive multi-sensor measurement fusion strategy is designed for the case where the sensor accuracy is known in advance. Secondly, the affiliation function of the fuzzy theory is introduced into the evidence theory approach to assign initial evidence of measurements in terms of defining the degree of fuzzy support between measurements, which improves the fusion accuracy of the method. Finally, the belief Jensen–Shannon divergence and the Rényi divergence are combined for measuring the conflict between the evidence pieces to obtain the credibility degree as the reliability of the evidence, which solves the problem of high conflict between evidence pieces. Three examples of multi-sensor data fusion in different domains are employed to validate the adaptability of the proposed method to different kinds of multi-sensors. The maximum relative error of the proposed method for multiple sensor experiments is greater than or equal to 0.18%, and its error accuracy is much higher than the best result of 0.46% among other comparative methods. The experimental results verify that the proposed data fusion method is more accurate than other existing methods.

Research on Mechanical Equipment Fault Diagnosis Method Based on Deep Learning and Information Fusion.

Solving the problem of the transmission of mechanical equipment is complicated, and the interconnection between equipment components in a complex industrial environment can easily lead to faults. A multi-scale-sensor information fusion method is proposed, overcoming the shortcomings of fault diagnosis methods based on the analysis of one signal, in terms of diagnosis accuracy and efficiency. First, different sizes of convolution kernels are applied to extract multi-scale features from original signals using a multi-scale one-dimensional convolutional neural network (1DCNN); this not only improves the learning ability of the features but also enables the fine characterization of the features. Then, using Dempster-Shafer (DS) evidence theory, improved by multi-sensor information fusion strategy, the feature signals extracted by the multi-scale 1DCNN are fused to realize the fault detection and location. Finally, the experimental results of fault detection on a flash furnace show that the accuracy of the proposed method is more than 99.65% and has better fault diagnosis, which proves the feasibility and effectiveness of the proposed method.

V-SLAM ENHANCED INS/GNSS FUSION SCHEME FOR LANE LEVEL VEHICULAR NAVIGATION APPLICATIONS IN DYNAMIC ENVIRONMENT

Abstract. With the development of different sensors, such as global navigation satellite system (GNSS), inertial measurement unit (IMU), LiDAR, radar and camera, more localization information is available for autonomous vehicular applications. However, each sensor has its limitations in different circumstances. For example, visual Simultaneous Localization and Mapping (SLAM) easily loses tracking in an open sky area where accurate GNSS measurements can be obtained. Sensors can complement each other by integrated their information in a multi-sensor fusion scheme. In this study, we proposed a visual-SLAM enhanced INS/GNSS localization fusion scheme for a high dynamic environment. Oriented FAST and rotated BRIEF (ORB) SLAM are used to pre-process image sequences from monocular camera, rescaled and refreshed after applying GNSS measurements, and convert to position and velocity information, which can provide updates to the system. The performance of the fusion system was verified through two field tests at different speed ranges (about 30–60 km/s), using a reliable reference system as ground-truth to assess the accuracy of the proposed localization fusion scheme. The results indicated that the proposed system could improve the navigation accuracy compared to INS/GNSS integration scheme and achieve which-lane level or even where-in-lane level.

Multi-Sensor Fusion Tomato Picking Robot Localization and Mapping Research

To address the current problem of unsatisfactory localization and map building of an intelligent picking robot mobile platform in an unstructured picking environment, a multi-sensor fusion localization and map building method is proposed to fuse the visual odometer position estimation results with IMU and wheel odometer. And the pre-fusion processing of wheel odometer data is proposed to solve the problem of large errors of wheel odometers on uneven ground. The multi-sensor fusion scheme is proposed to fuse vision and laser SLAM methods to build a more information-rich point cloud and raster fusion map for the problem of non-closure caused by sparse branches. The experimental results show that the multi-sensor fusion SLAM method can obtain more accurate, richer, and more robust maps.

Affect and stress detection based on feature fusion of LSTM and 1DCNN

The impact of emotions on health, especially stress, is receiving increasing attention. It is important to provide a non-invasive affect detection system that can be continuously monitored for a long period of time. Multi-sensor fusion strategies can better improve the performance of affect detection models, but there are also problems such as insufficient feature extraction and poor spatiotemporal feature fusion. Therefore, this study proposes a feature-level fusion method based on long short-term memory and one-dimensional convolutional neural network to extract temporal and spatial features of electrocardiogram, electromyogram, electrical activity, temperature, accelerator and response data, respectively, and then fuse them in a summation fashion for affect and stress detection. In particular, we added the tanh activation function before feature fusion, which can improve the model’s performance. We used the wearable affect and stress detection dataset to train the model, which includes three different emotion states (neutral, stress, and amusement) for three-class emotion classification with accuracy and F1-scores of 87.82% and 86.68%, respectively. Due to the importance of stress, we also studied binary classification for stress detection, where neutral and amusement were combined as non-stress, with accuracy and F1-scores of 94.9% and 94.98%, respectively. The performance of the proposed model outperforms other control models and can effectively improve the performance of affect and stress detection, and promote medical care, health care and elderly care.

A Multisensor Fusion Scheme and Its Engineering Application

Sensor fusion is a key technology in ADAS (FCW, PCW, AEB, ACC, etc.) In this paper, the target-level fusion method is used to realize the recognition of key targets through the algorithms such as estimation of the driving area, screening for effective targets, targets matching, target fusion and target tracking. Simulation test and real vehicle test show that the algorithms can identify key targets steadily and screen out radar jamming targets such as guardrails and viaducts.

Characteristic evaluation via multi-sensor information fusion strategy for spherical underwater robots

Currently, most of the existing fusion methods ignore the rich multi-source information of different types of sensor nodes in the underwater unknown environment, which makes it challenging for Autonomous Underwater Vehicles (AUVs) to accurately perceive the external environment and make actionable decisions. Considering the key issues such as attitude estimation, positioning and obstacle avoidance involved in performing AUV tasks, this paper proposed a Multi-Source Information Fusion (MSIF) model for Spherical Underwater Robots (SURs) we developed based on various low-cost sensors. Multi-source information from an Inertial Measurement Unit (IMU), Pressure Sensor Array (PSA), Obstacle Avoidance Sensor Array (OASA), Depth Sensor (DS), Looking-Down Camera (LDC) and Acoustic Communication System (ACS) were fused to enable SUR to obtain high-precision estimated data for attitude estimation, positioning and obstacle avoidance, etc. More precisely, according to the correlation between the sensors, the optimized model was constructed to compensate for angle errors, velocity errors, orientation errors, etc. Subsequently, a machine learning method using Back Propagation Neural Network (BPNN) was proposed to improve the accuracy and effectiveness of the MSIF model through feature selection, data training, and feature estimation, etc. Finally, a series of experiments were performed under different scenarios, such as motion and obstacle avoidance experiments. The theoretical derivation and comprehensive evaluations demonstrated the effectiveness and feasibility of the proposed model, which provided a new reference value for solving issues such as attitude estimation, positioning and obstacle avoidance of AUVs.

A GNSS/LiDAR/IMU Pose Estimation System Based on Collaborative Fusion of Factor Map and Filtering

One of the core issues of mobile measurement is the pose estimation of the carrier. The classic Global Navigation Satellite System/Inertial Measurement Unit (GNSS/IMU) integrated navigation scheme has high positioning accuracy but is vulnerable to Global Navigation Satellite System (GNSS) signal occlusion and multipath effect. Simultaneous Localization and Mapping (SLAM) is not affected by signal occlusion, but there are problems such as error accumulation and scene degradation. The multi-sensor fusion scheme combining the two technologies can effectively expand the scene coverage and improve the positioning accuracy and system robustness. However, the current scheme has some limitations. On the one hand, GNSS plays a less important role in most SLAM systems, only for initialization or as a closed-loop factor and other auxiliary work. On the other hand, in the fusion method, most of the current systems only use filtering or graph optimization, without taking into account the advantages of both. Aiming at pose estimation for mobile carriers, this paper combines the advantages of the global optimization of the factor graph and the local optimization of filtering and proposes a GNSS-IMU-LiDAR Constraint Kalman Filter (abbreviated as GIL-CKF), which has the characteristics of full coverage and effectively improving absolute accuracy and high output frequency. The scheme proposed in this paper consists of three parts. Firstly, an extensible factor map is used to fuse the positioning information from different sources, including GNSS, IMU, LiDAR, and a closed-loop map, to maintain a high-precision SLAM system, and the output is used as Multi-Sensor-Fusion-Odometry (MSFO). Then, the scene is divided according to the GNSS quality factor, and a Scene Optimizer (SO) is designed to filter GNSS pose and MSFO. Finally, the results are input into the Extended Kalman Filter (EKF) together with the original IMU data for further optimization and smoothing. The experimental results show that the integration of high-precision GNSS positioning information with IMU, LiDAR, a closed-loop map, and other information through the factor map can effectively suppress error accumulation and improve the overall accuracy of the SLAM system. The SO based on GNSS indicators can fully retain high-precision GNSS positioning information, effectively play their respective advantages of filtering and graph optimization, and alleviate the conflict between global and local optimization. SO with EKF filtering furthers optimization, can improve the output frequency, and smooth the trajectory. GIL-CKF can effectively improve the accuracy and robustness of pose estimation and has obvious advantages in multi-sensor scene complementarity, partial road section accuracy improvement, and high input frequency.

A Lightweight and Drift-Free Fusion Strategy for Drone Autonomous and Safe Navigation

Self-localization and state estimation are crucial capabilities for agile drone autonomous navigation. This article presents a lightweight and drift-free vision-IMU-GNSS tightly coupled multisensor fusion (LDMF) strategy for drones’ autonomous and safe navigation. The drone is carried out with a front-facing camera to create visual geometric constraints and generate a 3D environmental map. Ulteriorly, a GNSS receiver with multiple constellations support is used to continuously provide pseudo-range, Doppler frequency shift and UTC time pulse signals to the drone navigation system. The proposed multisensor fusion strategy leverages the Kanade–Lucas algorithm to track multiple visual features in each input image. The local graph solution is bounded in a restricted sliding window, which can immensely predigest the computational complexity in factor graph optimization procedures. The drone navigation system can achieve camera-rate performance on a small companion computer. We thoroughly experimented with the LDMF system in both simulated and real-world environments, and the results demonstrate dramatic advantages over the state-of-the-art sensor fusion strategies.

HIGH-DEFINITION POINT CLOUD MAP-BASED 3D LiDAR-IMU CALIBRATION FOR SELF-DRIVING APPLICATIONS

Abstract. The multi-sensor fusion scheme has become more and more popular these days with its great potential to estimate reliable navigation information for the modern development in automated driving system (ADS) and mobile mapping systems (MMS). Since these systems are combined with numerous navigation sensors, thus their geometric relationship should be precisely known. This study focuses on practical aspects when calibrating LiDAR-IMU mounting parameters (lever-arms and bore-sight angles) in land-based MMS. This calibration model is based on expressing the mounting parameters within the direct georeferencing equation for each epoch time and conditioning a set of INS/GNSS and LiDAR navigation solutions to lie on it. There is no need for a required information about the planar features in the calibration field as part of the unknowns. Such conditions are only benefitable in the residential area where the presence of sufficient planes in form of building is abundant. We present an approach for recovery the mounting parameters by conditioning the high-definition (HD) point cloud map-based LiDAR information and INS/GNSS navigation solutions through the least-squares solutions. The presented results and discussion mainly focus on practical examples with data from land-based MMS. Preliminary results indicate that correct calibration parameters are not only capable to improve the performance of point cloud georeferencing but also dramatically provide reliable performance evaluation of navigation estimation. Moreover, these findings show that the studied method is not only applicable in the featureless environment but also in its practicality to the self-driving applications.

Decomposed POMDP Optimization-Based Sensor Management for Multi-Target Tracking in Passive Multi-Sensor Systems

This paper presents an efficient information-theoretic sensor management method to maximize the performance of the passive multi-sensor system for multi-target tracking. We model the multi-target state as a generalized labeled multi-Bernoulli (GLMB) random finite set and formulate the dynamic sensor selection process as a partially observable Markov decision process (POMDP). The optimization objective is to maximum the information gain obtained from the observed data, which is measured by the Cauchy-Schwarz divergence. This is accomplished with two main technical innovations. The first is a tractable decomposed POMDP based sensor selection solution, in which the informative sensors are selected sequentially from the candidates based on the Cauchy-Schwarz divergence. The second is a novel dual-stage multi-sensor fusion strategy based on the iterated-corrector GLMB filter. Since the uncertainty of the passive multi-sensor system is generally large, the performance of the iterated-corrector scheme can be greatly influenced by the order of sensor updates. To fix this problem, the selected sensors are ranked in order of the Cauchy-Schwarz divergence obtained in sensor selection, followed by the iterated-corrector update. For the multi-sensor fusion, the effect of poor performance sensors is weakened and that of better performance sensors is enhanced. Simulation studies demonstrate the effectiveness and efficiency of the proposed method in challenging passive multi-target tracking scenarios.

Crop Type Mapping from Optical and Radar Time Series Using Attention-Based Deep Learning

Crop maps are key inputs for crop inventory production and yield estimation and can inform the implementation of effective farm management practices. Producing these maps at detailed scales requires exhaustive field surveys that can be laborious, time-consuming, and expensive to replicate. With a growing archive of remote sensing data, there are enormous opportunities to exploit dense satellite image time series (SITS), temporal sequences of images over the same area. Generally, crop type mapping relies on single-sensor inputs and is solved with the help of traditional learning algorithms such as random forests or support vector machines. Nowadays, deep learning techniques have brought significant improvements by leveraging information in both spatial and temporal dimensions, which are relevant in crop studies. The concurrent availability of Sentinel-1 (synthetic aperture radar) and Sentinel-2 (optical) data offers a great opportunity to utilize them jointly; however, optimizing their synergy has been understudied with deep learning techniques. In this work, we analyze and compare three fusion strategies (input, layer, and decision levels) to identify the best strategy that optimizes optical-radar classification performance. They are applied to a recent architecture, notably, the pixel-set encoder–temporal attention encoder (PSE-TAE) developed specifically for object-based classification of SITS and based on self-attention mechanisms. Experiments are carried out in Brittany, in the northwest of France, with Sentinel-1 and Sentinel-2 time series. Input and layer-level fusion competitively achieved the best overall F-score surpassing decision-level fusion by 2%. On a per-class basis, decision-level fusion increased the accuracy of dominant classes, whereas layer-level fusion improves up to 13% for minority classes. Against single-sensor baseline, multi-sensor fusion strategies identified crop types more accurately: for example, input-level outperformed Sentinel-2 and Sentinel-1 by 3% and 9% in F-score, respectively. We have also conducted experiments that showed the importance of fusion for early time series classification and under high cloud cover condition.

Modular Multi-Sensor Fusion: A Collaborative State Estimation Perspective

Fusing information from multiple sensors in aprobabilistic framework can significantly improve the estimation of the system's states, allow for redundancy, and for adaptations in dynamic environments. However, the computation complexity increases with the number of sensors and the dimension of the estimated states. In this letter, we show that techniques used in the field of CSE for distributed estimators on decoupled agents can be applied on local estimators to decouple states from different sensors. Bridging the gap between these domains allows us to propose a novel unified modular multi-sensor fusion strategy for recursive filters that (i) achieves constant maintenance complexity in propagation and private update steps, (ii) supports any-sensor-to-any-sensor observations, (iii) isolates state propagation supporting high rates for propagation sensors, (iv) isolates private sensor updates, and (v) isolates joint updates requiring only the participating sensors' estimates all while maintaining probabilistic consistency. We port existing CSE strategies into MMSF formulations and compare them as well as a classical MMSF approach against the proposed one in terms of execution time, accuracy, and credibility on real data.

A generic multi-sensor fusion scheme for localization of autonomous platforms using moving horizon estimation

In this paper, a generic multi-sensor fusion framework is developed for the localization of intelligent vehicles and mobile robots. The localization framework is based on moving horizon estimation (MHE). Unlike the commonly used probabilistic filtering algorithms – for example, extended Kalman filter (EKF) and unscented Kalman filter (UKF) – MHE relies on solving successive least squares optimization problems over the innovation of multiple sensors’ measurements and a specific estimation horizon. In this paper, we present an efficient and generic multi-sensor fusion scheme, based on MHE. The proposed multi-sensor fusion scheme is capable of operating with different sensors’ rates, missing measurements, and outliers. Moreover, the proposed scheme is based on a multi-threading architecture to reduce its computational cost, making it more feasible for practical applications. The MHE fusion method is tested using simulated data as well as real experimental data sequences from an intelligent vehicle and a mobile robot combining measurements from different sensors to get accurate localization results. The performance of MHE is compared against that of UKF, where the MHE estimation results show superior performance.

A ubiquitous wheelchair fall detection system using low-cost embedded inertial sensors and unsupervised one-class SVM

Fall detector systems are one of the highly researched areas of Ambient Assisted Living (AAL) applications ensuring safety and autonomous living for the elderly. Today, despite the diversity of methods proposed for fall detectors, there is still a need to develop accurate and robust architectures, methods, and protocols for the detection of the occurrence of falls for a special-class of wheelchair-bound people. To address this issue, this paper proposes a wheelchair fall detector system based on a low-cost, light-weight inertial sensing method utilizing a hybrid scheme and unsupervised One-Class SVM (OCSVM) for detection of cases leading to a ‘fall’ anomaly during wheelchair maneuver and for the case of unassisted transfers. To make the system robust against noise, a novel hybrid multi-sensor fusion strategy combining Zero Angular Rate Update (ZART) and Complementary Filter (CF) to compensate sensor integral errors is utilized. A heterogeneous dataset is constructed using the publicly available Sis-Fall dataset to include possible fall cases due to unassisted transfers from wheelchairs and secondly, a prototype is developed to emulate the wheelchair system with the embedded inertial sensors to capture trends in the sensor measurements due to wheelchair tips and falls. The OCSVM anomaly detection technique is utilized to overcome the major disadvantage of supervised learning methods requiring huge datasets from activities performed by human subjects needed to guarantee higher accuracy rates from these detectors. In this regard, to capture the best features from the generated accelerometer and gyroscope feature set, the ReliefF algorithm is used. The proposed method is compared with the widely reported approaches in the literature for fall-detectors, i.e., threshold-based methods and other one-class learning approaches. It is demonstrated that the fall-detection accuracy (i.e., the g-mean score) was achieved up to 96% with the proposed method.

Complex Surface Reconstruction Based on Fusion of Surface Normals and Sparse Depth Measurement

Precision measurement and reconstruction of detailed surfaces topography is a challenging task for non-diffuse complex parts. Although coordinate measurement machines (CMM) with the touch-trigger probe are widely used in current industry, the measurement efficiency limits their application in the measurement of complex surfaces. This article proposes a multisensor data fusion strategy by integrating the technical merits of CMM and photometric stereo (PS) to achieve multiscale reconstruction of a complex surface with high efficiency. Considering the complementary measurement characteristics of the two approaches, the sparse points from CMM are used to provide global shape information, and the high-resolution surface normal map from PS is used to provide local detailed structure. A multistage neural network is then proposed to fuse these two kinds of modality information such that the global features from the sparse points and the local features from the surface normal map are fused in a coarse-to-fine multistage process so as to make the training process more stable and the reconstruction more accurate. To enhance the generality of the fusion neural network, a synthetic training data set is also designed to include a large variety of multiscale features enriched surfaces. Experiments are conducted to verify the effectiveness of the proposed multisensor fusion strategy in accurate reconstruction of complex surfaces with high efficiency.