Multi-sensor Data Fusion Model Research Articles

Spatial–Temporal Federated Transfer Learning with multi-sensor data fusion for cooperative positioning

With the development of advanced embedded and communication systems, location information has become a crucial factor in supporting context-aware or location-aware intelligent services. Among these services, modern Intelligent Transportation System (ITS) has the strictest requirements for real-time, accurate, and private location data. In this study, a Spatial-Temporal Federated Transfer Learning (ST-FTL) framework is designed and introduced to achieve more precise cooperative positioning with multi-sensor data fusion while protecting the location data privacy in urban ITS. Specifically, a three-layer FTL architecture is constructed to enhance the prediction accuracy on GPS positioning errors for vehicles in different regions especially when facing missing local data in some specific scenarios (e.g., urban canyons), in which Transfer Learning (TL) is incorporated to optimize the initialization of global model with faster convergence but less communication cost in Federated Learning (FL). A multi-attribute based spatial-temporal clustering algorithm is developed to facilitate the finding of more appropriate source domains similar to the target domain in a density-based scheme, while a convolutional-gated unit is newly designed to further filter out the useless features and add new features based on the convolution operations and gating mechanism, resulting in more effective global model initialization and weight aggregation from cross-region selection. A multi-sensor data fusion model is built in local to improve the prediction accuracy on positioning errors, in which an improved time-aware asymmetric attention mechanism is involved to selectively adjust the weight importance of the incorporated GPS and Inertial Measurement Unit (IMU) data in a more targeted fusion process. Additionally, an improved Siamese network structure, which breaks the traditional dual network structure and only adopts one single network structure, is leveraged to realize the lightweight data augmentation based on the better utilization of historical similar data from local vehicles themselves, when suffering from insufficient training data. Experiments and evaluations based on two different public datasets demonstrate the outstanding performance of our proposed model and method in achieving superior prediction accuracy and faster convergence for cooperative positioning compared with other state-of-the-art positioning methods in urban ITS.

Multi-Sensor Data Fusion Method Based on Self-Attention Mechanism

In 3D reconstruction tasks, single-sensor data fusion based on deep learning is limited by the integrity and accuracy of the data, which reduces the accuracy and reliability of the fusion results. To address this issue, this study proposes a multi-sensor data fusion method based on a self-attention mechanism. A multi-sensor data fusion model for acquiring multi-source and multi-modal data is constructed, with the core component being a convolutional neural network with self-attention (CNN-SA), which employs CNNs to process multi-source and multi-modal data by extracting their features. Additionally, it introduces an SA mechanism to weigh and sum the features of different modalities, adaptively focusing on the importance of different modal data. This enables mutual support, complementarity, and correction among the multi-modal data. Experimental results demonstrate that the accuracy of the CNN-SA network is improved by 72.6%, surpassing the improvements of 29.9% for CNN-CBAM, 23.6% for CNN, and 11.4% for CNN-LSTM, exhibiting enhanced generalization capability, accuracy, and robustness. The proposed approach will contribute to the effectiveness of multi-sensor data fusion processing.

Mathematical modeling of underwater signal anomaly perception based on multi-sensor data fusion

There is a lot of interference in underwater environment, which can affect underwater signal transmission. For this reason, an abnormal perception model of underwater signals based on multi-sensor data fusion is proposed. The underwater frequency hopping communication method is used to select the channel and construct the mathematical model of underwater communication signal. A signal modulation and recognition system for multi-sensor reception is established, and the multi-sensor data fusion model is obtained by calculating the support margin. The energy spectrum distribution is obtained by decomposition of abnormal features with wavelet detector in time domain. The deep learning algorithm is used to reorganize the signal with the minimum detection error, and the mathematical model of underwater signal anomaly perception is obtained. The experimental results show that the average accuracy of the method for locating abnormal underwater communication signals is 95.5%, the misrecognition rate of abnormal underwater signals is less than 2%, and the time-consuming for sensing abnormal signals of 420 MB underwater communication is less than 3 s.

UAV-Based Multi-Sensor Data Fusion for Urban Land Cover Mapping Using a Deep Convolutional Neural Network

Accurate and up-to-date land cover classification information is essential for many applications, such as land-use change detection, global environmental change, and forest management, among others. Unoccupied aerial systems (UASs) provide the advantage of flexible and rapid data acquisition at low cost compared to conventional platforms, such as satellite and airborne systems. UASs are often equipped with high spatial resolution cameras and/or light detection and ranging (LiDAR). However, the high spatial resolution imagery has a high information content, which makes land cover classification quite challenging. Recently, deep convolutional neural networks (DCNNs) have been effectively applied to remote sensing applications, which overcome the drawback of traditional techniques. In this research, a low-cost UAV-based multi-sensor data fusion model was developed for land cover classification based on a DCNN. For the purpose of this research, two datasets were collected at two different urban locations using two different UASs. A DCNN model, based on U-net with Resnet101 as a backbone, was used to train and test the fused image/LiDAR data. The maximum likelihood and support vector machine techniques were used as a reference for classifier comparison. It was shown that the proposed DCNN approach improved the overall accuracy of land cover classification for the first dataset by 15% compared to the reference classifiers. In addition, the overall accuracy of land cover classification improved by 7%, and the precision, recall, and F-measure improved by 18% when the fused image/LiDAR data were used compared to the images only. The trained DCNN model was also tested on the second dataset, and the obtained results were largely similar to those of the first dataset.

Multi-Sensor Data Fusion with a Reconfigurable Module and Its Application to Unmanned Storage Boxes.

We present a multi-sensor data fusion model based on a reconfigurable module (RM) with three fusion layers. In the data layer, raw data are refined with respect to the sensor characteristics and then converted into logical values. In the feature layer, a fusion tree is configured, and the values of the intermediate nodes are calculated by applying predefined logical operations, which are adjustable. In the decision layer, a final decision is made by computing the value of the root according to predetermined equations. In this way, with given threshold values or sensor characteristics for data refinement and logic expressions for feature extraction and decision making, we reconstruct an RM that performs multi-sensor fusion and is adaptable for a dedicated application. We attempted to verify its feasibility by applying the proposed RM to an actual application. Considering the spread of the COVID-19 pandemic, an unmanned storage box was selected as our application target. Four types of sensors were used to determine the state of the door and the status of the existence of an item inside it. We implemented a prototype system that monitored the unmanned storage boxes by configuring the RM according to the proposed method. It was confirmed that a system built with only low-cost sensors can identify the states more reliably through multi-sensor data fusion.

Application of Internet of Things Framework in Physical Education System

<p>In order to improve the effect of physical education, based on the Internet of Things technology, this paper combines sensor technology to construct a physical education system based on the Internet of Things framework. Based on the characteristics of system integrated multi-sensor in the collection of environmental characteristic parameters, this paper designs the system’s multi-sensor data fusion model from the perspective of sensor type classification. Moreover, this paper combines the actual needs of physical education to construct the framework of the Internet of Things physical education system. In addition, this paper verifies the performance of the system through comparative experiments, and statistically researches the results and analyzes the data by mathematical statistical methods. The experimental research results show that the physical education system based on the framework of the Internet of Things can play an important auxiliary role in the physical education of college students.</p> <p> </p>

A Detachable and Expansible Multisensor Data Fusion Model for Perception in Level 3 Autonomous Driving System

Various sensors are adopted by autonomous driving systems to perceive objects and surroundings. Thus, the multi-sensor data fusion techniques become essential to combine different sensors’ advantages for better perception performance. However, the current multi-sensor data fusion techniques suffer from the high cost of computation resources, low expansibility for more diverse sensors, and insufficient systematic consideration for modeling. This paper first constructs a detachable and expansible multi-sensor data fusion model based on three main modules: front fusion, global fusion, and synthesizer, where the methods for flexible association gating and virtual targets have been designed. The model can be disassembled and configured for different trim levels of vehicles and is easily expansible for adding more heterogeneous sensors. Next, the presented multi-sensor data fusion model is compared with the cheap Joint Probabilistic Data Association (C-JPDA) method. The comparison shows the superior accuracy of the designed model on false association and effective narrowing of the variance of object detection. Finally, the presented multi-sensor data fusion model is integrated into an embedded system and experimented on urban roads and highways with the engaged Level 3 autonomous driving function. The experiment results indicate that the proposed model has excellent sensor data fusion performance and provides accurate and timely object information in the Level 3 autonomous driving system.

A New View of Multisensor Data Fusion: Research on Generalized Fusion

Multisensor data generalized fusion algorithm is a kind of symbolic computing model with multiple application objects based on sensor generalized integration. It is the theoretical basis of numerical fusion. This paper aims to comprehensively review the generalized fusion algorithms of multisensor data. Firstly, the development and definition of multisensor data fusion are analyzed and the definition of multisensor data generalized fusion is given. Secondly, the classification of multisensor data fusion is discussed, and the generalized integration structure of multisensor and its data acquisition and representation are given, abandoning the research characteristics of object oriented. Then, the principle and architecture of multisensor data fusion are analyzed, and a generalized multisensor data fusion model is presented based on the JDL model. Finally, according to the multisensor data generalized fusion architecture, some related theories and methods are reviewed, and the tensor-based multisensor heterogeneous data generalized fusion algorithm is proposed, and the future work is prospected.

Multi-Sensor Guided Hand Gesture Recognition for a Teleoperated Robot Using a Recurrent Neural Network

Touch-free guided hand gesture recognition for human-robot interactions plays an increasingly significant role in teleoperated surgical robot systems. Indeed, despite depth cameras provide more practical information for recognition accuracy enhancement, the instability and computational burden of depth data represent a tricky problem. In this letter, we propose a novel multi-sensor guided hand gesture recognition system for surgical robot teleoperation. A multi-sensor data fusion model is designed for performing interference in the presence of occlusions. A multilayer Recurrent Neural Network (RNN) consisting of a Long Short-Term Memory (LSTM) module and a dropout layer (LSTM-RNN) is proposed for multiple hand gestures classification. Detected hand gestures are used to perform a set of human-robot collaboration tasks on a surgical robot platform. Classification performance and prediction time is compared among the LSTM-RNN model and several traditional Machine Learning (ML) algorithms, such as k-Nearest Neighbor (k-NN) and Support Vector Machines (SVM). Results show that the proposed LSTM-RNN classifier is able to achieve a higher recognition rate and faster inference speed. In addition, the present adaptive data fusion system shows a strong anti-interference capability for hand gesture recognition in real-time.

Remaining useful life prediction based on a multi-sensor data fusion model

With the rapid development of Industrial Internet of Things, more and more sensors have been used for condition monitoring and prognostics of industrial systems. Big data collected from sensor networks bring abundant information resources as well as technical challenges for remaining useful life (RUL) prediction. The major technical challenges include how to select informative sensors and fuse multi-sensor data to improve the prediction performance. To deal with the challenges, this paper proposes a RUL prediction method based on a multi-sensor data fusion model. In this method, the inherent degradation process of the system state is expressed using a state transition function following a Wiener process. Multi-sensor signals are explicated as various proxies of the inherent system degradation process using a multivariate measurement function. The system state is estimated by fusing multi-sensor signals using particle filtering. Informative sensors are selected by a prioritized sensor group selection algorithm. This algorithm first prioritizes sensors according to their individual performances in RUL prediction, and then selects an optimal sensor group based on their combined performances. The effectiveness of the proposed method is demonstrated using a simulation study and aircraft engine degradation data from NASA repository.

Experimental Validation of Multi-Sensor Data Fusion Model for Railway Wheel Defect Identification

Wheel defects are detrimental for railway train and track components and should be detected and identified as early as possible. Wheel Impact Load Detector (WILD) is a commercial condition monitoring system used for detecting the defective wheels. This system usually measures the rail strain at different points by multiple sensors. WILD converts the measured strains to the force and uses the peak force, dynamic force, and ratio of the peak force to the static force to estimate the condition of the in-service wheels. These methods are useful for detecting the severe defects contributing to the contact force to the extent that exceed a predetermined threshold. Therefore, in the prior research a fusion method has been developed to reconstruct a new informative pattern from the data collected by the multiple sensors. The reconstructed pattern provides a comprehensive description of the wheel condition. This paper validates the fusion method using a set of lab tests to investigate the applicability of the proposed method. For this purpose, a test rig has been built consisting of a circular rail, a rotating arm, and a wheel. Six strain sensors have been installed under the rail in the symmetric locations over the rail circle with 60 degree intervals. The fusion method used to reconstruct a signal from the bending strain signals measured by the multiple sensors. Different wheel defects including the flat and out-of-round wheels have been tested and the results validated the fusion method by providing informative patterns.

Sensor Data Fusion Framework for Improvement of Temperature Sensor Characteristics

This paper presents a multi-sensor data fusion model for measurement of temperature. The proposed paper puts forward an objective to develop a temperature measuring instrument (a) having improved performance characteristics like sensitivity and linearity (b) that produces accurate measurement even though a sensor is faulty. The technique is designed using the framework of multi-sensor data fusion consisting of sensor like thermistor, thermocouple, and resistance temperature detector. Output from all these sensors are converted to a common representation format using radiometric normalization. Pau’s framework is used for implementing the task of fusion in the proposed paper. Implemented laboratory model is tested for functionality using extensive set of data. Output shows that the proposed technique was able to produce improved linear and sensitive output as compared to system with individual sensor. Root mean square of percentage error obtained for tests conducted is about 0.86%, which is a significant improvement.

Multi-sensor Data Fusion Based on Fuzzy Neural Network and Its Application in Piggery Environmental Control Strategies

In intensive and large-scale farming, piggery environmental monitoring is an important method for improving productivity and preventing disease in pig herds. The piggery environment mainly includes air temperature, air humidity, and ammonia concentration. Piggery environmental monitoring mostly concentrates on a single environmental factor and data processing. Few studies focus on multiple environmental factors and information fusion from a variety of monitoring sensors. Therefore, in this paper, the main environmental factors that affect herd health, such as temperature, humidity, and ammonia concentration, were monitored. According to the analysis of piggery environmental adaptability indicators, a multi-sensor data fusion model with five layers based on the adaptive fuzzy neural network of T-S inference was established. The structure of the model is 3-8-18-18-1. To check the performance of the model, 432 groups of data were selected by monitoring a piggery environment for three consecutive days. A total of 288 sets of data were obtained for training, and 144 sets of data were obtained for checking. The result shows that the model effectively fuses multi-sensor data. After 600 training cycles, a Root Mean Squared Error (RMSE) of 0:14 10 −3 between the actual output of the system and the desired output can be achieved. The system output has high accuracy. The RMSE of the checking data and fuzzy system output is 0:18 10 −2 . Moreover, the operation efficiency of the system is high. Thus, the system is suitable for analyzing data in real time and for the processing of piggery environments. This system provides a good data processing method for monitoring and controlling piggery environments.

Medical decision-making inspired from aerospace multisensor data fusion concepts

In recent years, Internet-delivered treatments have been largely used for pain monitoring, offering healthcare professionals and patients the ability to interact anywhere and at any time. Electronic diaries have been increasingly adopted as the preferred methodology to collect data related to pain intensity and symptoms, replacing traditional pen-and-paper diaries. This article presents a multisensor data fusion methodology based on the capabilities provided by aerospace systems to evaluate the effects of electronic and pen-and-paper diaries on pain. We examined English-language studies of randomized controlled trials that use computerized systems and the Internet to collect data about chronic pain complaints. These studies were obtained from three data sources: BioMed Central, PubMed Central and ScienceDirect from the year 2000 until 30 June 2012. Based on comparisons of the reported pain intensity collected during pre- and post-treatment in both the control and intervention groups, the proposed multisensor data fusion model revealed that the benefits of technology and pen-and-paper are qualitatively equivalent . We conclude that the proposed model is suitable, intelligible, easy to implement, time efficient and resource efficient.