Level Data Fusion Research Articles

A data-level fusion model for unsupervised attribute selection in multi-source homogeneous data

Information fusion refers to derive an overall precise description of data by using certain fusion technique for utilizing the complementary information from multiple sources of data, which can facilitate effective decision-making, prediction and classification, etc. Multi-source homogeneous data, characterizing the data type of variables in the sample in form of one type (i.e., numerical or categorical) in different information sources, which widely exists in many practical applications. This paper concentrates on efficient fusion of multi-source homogeneous data with a data-level fusion model which involves the consolidation of multiple information sources and unsupervised attribute selection of the fused data. A unified description and modeling method of a multi-source homogeneous information system is introduced. The neighborhood rough sets model is used to construct the neighborhood granular structure, which uses the idea of granular computing to build methods of uncertainty measures. Given the uncertainty of fusing multiple information sources, Sup–Inf fusion functions are developed based on the proposed uncertainty measures, which can fuse the multi-source homogeneous information system into a single-source information system. Finally, an unsupervised attribute selection approach is employed to eliminate redundant attribute of the single-source information system. Theoretical analysis and comprehensive experiments on several datasets demonstrate the feasibility and superiority of our method.

Prediction of vinegar processing parameters with chemometric modelling of spectroscopic data

Spectroscopic methods have the advantages of being rapid and environmentally friendly and can be used in measurement and control of processing parameters during food production. It was aimed to predict several quality and chemical parameters of vinegar processing from UV-visible and mid-infrared spectroscopic profiles. Two processing lines of both traditional and submerged vinegar production from 2 separate grape varieties (green and red grapes) were monitored. Some of the important markers of the fermentation processes; pH, brix, total acidity, total flavonoid content, total and individual phenolic contents, organic acid, sugar, ethanol concentrations as well as UV-visible and mid-infrared spectra were obtained during both types of vinegar processing and quality and chemical parameters were predicted from spectroscopic data using chemometric methods. Individual UV-visible and mid-infrared spectral profiles along with low level of data fusion were used in building of chemometric prediction models. Accurate, reliable and robust prediction models (R2cal and R2val >0.9) were obtained for quality parameters mostly with combination of two spectroscopic datasets. Predictive models used for phenolic components were below average except for p-coumaric and syringic acids. Citric and acetic acids were the most accurately estimated ones among organic acids along with ethanol. Close agreements between reference and predicted values were obtained during the monitoring of changes of some quality parameters for vinegar fermentation process through rapid and simultaneous spectroscopic measurements.

GeoAI in terrain analysis: Enabling multi-source deep learning and data fusion for natural feature detection

In this paper we report on a new GeoAI research method which enables deep machine learning from multi-source geospatial data for natural feature detection. In particular, a multi-source, deep learning-based object detection pipeline was developed. This pipeline introduces three new features: First, strategies of both data-level fusion (i.e., channel expansion on convolutional neural networks) and feature-level fusion were integrated into the object detection model to allow simultaneous machine learning from multi-source data, including remote sensing imagery and Digital Elevation Model (DEM) data. Second, a new data fusion strategy was developed to blend DEM data and its derivatives to create a new, fused data source with enriched information content and image features. The model has also enabled deep learning by combining both the proposed data fusion and feature-level fusion strategies to yield a much-improved detection result. Third, two different sets of data augmentation techniques were applied to the multi-source training data to further improve the model performance. A series of experiments were conducted to verify the effectiveness of the proposed strategies in multi-source deep learning.

Obstacle detection and tracking algorithm based on multi‐lidar fusion in urban environment

Adjacent obstacles are difficult to be distinguished, and remote obstacle is easily detected as multiple targets. Besides, occluded obstacles are difficult to track, and tracking velocity of the obstacles in rapid acceleration or deceleration converges slowly in the urban environment. In view of this, an obstacle detection and tracking method based on multi-lidar is proposed. Firstly, based on the vehicle kinematics model, motion compensation is adopted to solve the space-time synchronization problem among lidars after road segmentation, and data level fusion is completed. Next, the obstacles are detected by combining adaptive voxel grid DBSCAN (AVG-DBSCAN) algorithm and Region Growing (RG) algorithm. Then, the Breadth First Search (BFS) algorithm and KD tree are adopted to improve the Evolutionary Hungarian algorithm for the fast association of obstacles. Finally, the occlusion rate and dynamic region selection are used to track obstacles accurately, based on the Kalman Filter of uniform acceleration model. The experimental results on the authors' extracted urban dataset show that the proposed method can effectively fuse multi-lidar data and outperforms other methods in obstacle detection and tracking. Its average accuracy for detection is 97.53% and its average accuracy for tracking is 95.1%. The average duration of the entire process is only 30 ms.

Human Activity Recognition With Accelerometer and Gyroscope: A Data Fusion Approach

This paper compares the three levels of data fusion with the goal of determining the optimal level of data fusion for multi-sensor human activity data. Using the data processing pipeline, gyroscope and accelerometer data was fused at the sensor-level, feature-level and decision-level. For each level of data fusion four different techniques were used with varying levels of success. This analysis was performed on four human activity publicly-available datasets along with four well-known machine learning classifiers to validate the results. The decision-level fusion (Acc = 0.7443±0.0850) outperformed the other two levels of fusion in regards to accuracy, sensor level (Acc = 0.5934 ± 0.1110) and feature level (Acc = 0.6742 ± 0.0053), but, the processing time and computational power required for training and classification were far greater than practical for a HAR system. However, Kalman filter appear to be the more efficient method, since it exhibited both good accuracy (Acc = 0.7536 ± 0.1566) and short processing time (time = 61.71ms ± 63.85); properties that play a large role in real-time applications using wearable devices. The results of this study also serve as baseline information in the HAR literature to compare future methods of data fusion.

Mechatronics Fault Prediction and Diagnosis Based on Multi Sensor Information Fusion

This paper studies the mechatronics fault prediction and diagnosis based on multi-sensor information fusion. According to the method of data fusion, the fault diagnosis system is divided into data level fusion module, feature level fusion module and decision level fusion module. The data level fusion module mainly processes the measured signals of multi-sensor to extract the feature information of fault diagnosis. The feature level fusion module uses three parallel neural networks with the same structure. In this paper, the basic probability assignment method of decision level D-S evidence theory is used as the basic algorithm of data processing. The decision-making level uses the method of D-S evidence theory to fuse the results of feature level local diagnosis to get the final diagnosis result. The experimental data show that the accuracy and efficiency of the proposed Mechatronics fault prediction and diagnosis system based on multi-sensor information fusion can meet the requirements of engineering technology.

Performance degradation assessment methodology of harmonic reducer by using low-frequency time series data and genetic programming

The harmonic reducer, as an important transmission component in the industrial robot, determines the positioning accuracy, carrying capacity, and service life of the industrial robot. However, the performance and reliability of harmonic reducer will gradually decline or even fail under long-time and high-intensity operations. The sudden stop and failure of the industrial robot will directly affect production efficiency. In this paper, a data-driven method for performance degradation assessment (PDA) of the harmonic reducer is studied. An accelerated degradation test rig was set up to collect the run-to-failure datasets of harmonic reducer, including process data, such as torque and speed, and non-process data, such as vibration signals, which were captured under a low sampling frequency. Firstly, a sliding window-based kurtosis and root mean square (RMS) are applied to non-process data, which can detect the incipient failure time (IFT) of harmonic reducer. Subsequently, a data-level fusion of process data is conducted based on genetic programming to construct a monotonic health index for degradation assessment and trend analysis. The experimental study shows that the proposed method is effective to realize the PDA of harmonic reducer.

Multi-Sensor and Decision-Level Fusion-Based Structural Damage Detection Using a One-Dimensional Convolutional Neural Network.

This paper presents a novel approach to substantially improve the detection accuracy of structural damage via a one-dimensional convolutional neural network (1-D CNN) and a decision-level fusion strategy. As structural damage usually induces changes in the dynamic responses of a structure, a CNN can effectively extract structural damage information from the vibration signals and classify them into the corresponding damage categories. However, it is difficult to build a large-scale sensor system in practical engineering; the collected vibration signals are usually non-synchronous and contain incomplete structure information, resulting in some evident errors in the decision stage of the CNN. In this study, the acceleration signals of multiple acquisition points were obtained, and the signals of each acquisition point were used to train a 1-D CNN, and their performances were evaluated by using the corresponding testing samples. Subsequently, the prediction results of all CNNs were fused (decision-level fusion) to obtain the integrated detection results. This method was validated using both numerical and experimental models and compared with a control experiment (data-level fusion) in which all the acceleration signals were used to train a CNN. The results confirmed that: by fusing the prediction results of multiple CNN models, the detection accuracy was significantly improved; for the numerical and experimental models, the detection accuracy was 10% and 16–30%, respectively, higher than that of the control experiment. It was demonstrated that: training a CNN using the acceleration signals of each acquisition point and making its own decision (the CNN output) and then fusing these decisions could effectively improve the accuracy of damage detection of the CNN.

Two-Level Multimodal Fusion for Sentiment Analysis in Public Security

Large amounts of data are widely stored in cyberspace. Not only can they bring much convenience to people’s lives and work, but they can also assist the work in the information security field, such as microexpression recognition and sentiment analysis in the criminal investigation. Thus, it is of great significance to recognize and analyze the sentiment information, which is usually described by different modalities. Due to the correlation among different modalities data, multimodal can provide more comprehensive and robust information than unimodal in data analysis tasks. The complementary information from different modalities can be obtained by multimodal fusion methods. These approaches can process multimodal data through fusion algorithms and ensure the accuracy of the information used for subsequent classification or prediction tasks. In this study, a two-level multimodal fusion (TlMF) method with both data-level and decision-level fusion is proposed to achieve the sentiment analysis task. In the data-level fusion stage, a tensor fusion network is utilized to obtain the text-audio and text-video embeddings by fusing the text with audio and video features, respectively. During the decision-level fusion stage, the soft fusion method is adopted to fuse the classification or prediction results of the upstream classifiers, so that the final classification or prediction results can be as accurate as possible. The proposed method is tested on the CMU-MOSI, CMU-MOSEI, and IEMOCAP datasets, and the empirical results and ablation studies confirm the effectiveness of TlMF in capturing useful information from all the test modalities.

Fisher’s discriminant ratio based health indicator for locating informative frequency bands for machine performance degradation assessment

A general tactic for machine performance degradation assessment (PDA) is to develop a health indicator (HI) to identify different degradation stages. A suitable HI is not only capable of distinguishing different machine degradation stages but also has an inherent characteristic of monotonicity because of irreversible machine degradation. Since informative frequency bands, such as resonance frequency bands, can indicate the occurrence of machine faults and fault levels, it is desirable to locate informative frequency bands and consider integrating informative frequency bands for machine PDA. In this paper, a Fisher’s discriminant ratio-based health indicator is proposed to fully consider the contributions of all spectrum amplitudes in the frequency domain to machine PDA. Firstly, locating informative frequency bands of non-process data, such as vibration and acoustic data, is formulated as a Fisher’s discriminant ratio based optimization problem. Once the optimization problem is solved, informative frequency bands, such as resonance frequency bands, can be automatically determined by observing optimized weights used in a designed HI. The larger an absolute optimized weight, the more informative a frequency is. Secondly, the sum of the multiplication of frequency amplitudes and optimized weights can be directly used as a HI for machine PDA. Unlike most existing feature-level fusion methods, the proposed method can directly realize data-level fusion, namely fusion of frequency amplitudes. Thirdly, the recovery of a Fourier spectrum by considering optimized weights can be further used for squared envelope analysis for machine fault diagnosis. Two case studies showed that the proposed HI can not only distinguish different degradation stages but also can monotonically assess degradation trends. More interestingly, optimized weights can automatically indicate the locations of informative frequency bands. Comparisons with some popular sparse measures, the fast Kurtogram and machine learning algorithms demonstrate the superiority of the proposed method.

Non-destructive characterization and discrimination of vehicle bumpers fragments in forensic science using molecular spectral fusion analysis and chemometrics

A rapid and non-destructive approach for characterizing and discriminating trace evidences is one of the hotspots and focuses in forensic science. The vehicle bumpers fragments, as a trace evidence, are often found and extracted from the clothing of victims, the crashed vehicle, and the road at the scenes. Micro-laser Raman spectroscopy (MLRM) and attenuated total reflection-Fourier transform infrared spectroscopy (ATR-FTIR) has been used with principal component analysis (PCA), Multi-layer perceptron neural network (MLPNN), and Fisher discriminant analysis (FDA) to propose a method for characterizing and discriminating 160 vehicle bumpers fragments from 8 manufacturers. The spectra were pre-treated by automatic baseline correction, multivariate scatter correction, standard normal variate and Savitzky-Golay algorithm. PCA was used to reduce the dimension and extraction of feature variables, while MLPNN, FDA, the several supervised pattern recognition methods, were used as algorithms of constructing classifiers. The results displayed that the discrimination ability of fusion spectra model was stronger than that of single spectra dataset. In MLPNN model, the order of it from high to low was feature-level fusion (93.5%) > data-level fusion (90.3%) > MLRM (74.2%) > ATR-FTIR (64.5%). In FDA model, the order of it from high to low was feature-level fusion (100%) > data-level fusion (93.5%) > MLRM (77.4%) > ATR-FTIR (61.3%). Fisher discriminant analysis model based on feature-level fusion dataset was the more optimal and practical method for discriminating the vehicle bumpers fragments. “Audi”, “Hyundai”, and “Chery” samples, the similar kind from different manufacturers, were also distinguished exactly, which was satisfactory. The designed method represented a potentially rapid and non-destructive approach for differentiation trace evidences in forensic science.

Discrimination of automotive glass by conjoint Raman and laser-induced breakdown spectroscopy and multivariate data analysis

Conjoint laser-induced breakdown spectroscopy (LIBS) and Raman spectroscopy and high level data fusion were used to discriminate sixteen different glass samples by principal component analysis (PCA). LIBS provides information on the elemental composition of a sample and Raman spectroscopy gives information about the molecular composition. The results were compared to single LIBS and Raman measurements as well as to μ-XRF and SEM-EDS which are also frequently used in glass analysis. The direction dependent Mahalanobis distance between the sample point groups in the space of the most significant principle components was evaluated as a criterion for potential discrimination of the different samples. Very good discrimination (up to 99%) was achieved with conjoint LIBS and Raman and high level data fusion, as well as LIBS alone. The discrimination is based primarily on different contents of trace elements (e.g., Fe, Ti, Ba, Sr), which was confirmed by the μ-XRF and SEM-EDS and differences in fluorescence (detected by Raman). Besides the similar discrimination power of LIBS and conjoint LIBS and Raman, the combination and fusion of LIBS and Raman leads to greater distances between the data in the principal component space due to the additional information added by Raman. This should result in a lower misclassification rate of unknown samples. Results show that conjoint LIBS and Raman spectroscopy can be an alternative in the forensic analysis of glass samples.

Remaining Useful Life Prediction of IIoT-Enabled Complex Industrial Systems With Hybrid Fusion of Multiple Information Sources

Industrial Internet of Things has significantly boosted predictive maintenance for complex industrial systems, where the accurate prediction of remaining useful life (RUL) with high-level confidence is challenging. By aggregating multiple informative sources of system degradation, information fusion can be applied to improve the prediction accuracy and reduce the uncertainty. It can be performed on the data-level, feature-level, and decision-level. To fully exploit the available degradation information, this article proposes a hybrid fusion method on both the data level and decision level to predict the RUL. On the data level, genetic programming (GP) is adopted to integrate physical sensor sources into a composite health indicator (HI), resulting in an explicit nonlinear data-level fusion model. Subsequently, the predictions of the RUL based on each physical sensor and the developed composite HI are synthesized in the framework of belief functions theory, as the decision-level fusion method. Moreover, the decision-level method is flexible for incorporating other statistical data-driven methods with explicit estimations of the RUL. The proposed method is verified via a case study on NASA's C-MAPSS data set. Compared to the single-level fusion methods, the results confirm the superiority of the proposed method for higher accuracy and certainty of predicting the RUL.

Structure and Performance Analysis of Signal Acquisition and Doppler Tracking in LEO Augmented GNSS Receiver.

Due to the low signal power, the Global Navigation Satellite System (GNSS) signal is vulnerable to interference and even cannot be captured or tracked in harsh environments. As an alternative, the Low Earth Orbit (LEO) satellite has been widely used in the navigation field due to the advantages of low cost and strong signals. It is becoming a significant component of the new combined navigation system with GNSS. The combination of an LEO Doppler signal and GNSS observables can improve the positioning accuracy and high-precision positioning convergence time of the GNSS receiver. However, the GNSS signal receiving capability cannot be improved from this data fusion level. We propose a novel assisted structure where GNSS signal acquisition and Doppler tracking are assisted by LEO Doppler positioning. The receiver uses the LEO signal to achieve Doppler positioning firstly. Then, the coarse position with the GNSS navigation messages received from LEO, as well as the estimated clock information, is used to assist in the acquisition and tracking of GNSS. In this way, the GNSS receiver’s sensitivity can get the benefit from this integrated system. The paper presents the structure of the assisted receiver and analyzes the assisted GNSS signal acquisition and carrier tracking performance in detail. Simulation experiments of this assisted structure are carried out to verify its superiority of acquisition and tracking sensitivity in comparison with standalone GNSS receivers. Theoretical analysis and experimental results show that the proposed acquisition method can achieve 90% detection probability at a carrier-to-noise ratio (C/N0) of 15 dB-Hz, which is about 8 dB higher than the conventional acquisition method without assistance; the proposed tracking method can track weak signals of 5 dB-Hz, which is about 4 dB higher than the conventional method. Therefore, this novel LEO-assisted receiver has significantly improved weak signal acquisition and tracking sensitivity.

Design and Evaluation Method of Wireless Fire Detection Node Based on Multi-source Sensor Data Fusion

In view of the problem that the traditional fire detection node is not intelligent enough for data processing when building fire occurs, a design and evaluation method of wireless fire detection node based on multi-source sensor data fusion is proposed. Firstly, the temperature and humidity sensor, carbon monoxide sensor and smoke sensor are selected to collect three kinds of fire information eigenvalues at the same time. Secondly, the analysis and processing method of two-level data fusion is established, and the fire information is processed in STC89C52 microcontroller. The first level data fusion uses fuzzy proximity algorithm to get the eigenvectors of environmental parameters, so that the measured values are closer. The second level data fusion uses D-S evidence theory to analyze the data systematically. According to the probability distribution function after fusion, more accurate decision results are obtained. Finally, the fire signal is sent to the superior data management center through ZigBee wireless transmission network, and the alarm signal is triggered in time. The whole node design gives a complete hardware selection and software data fusion processing method. Compared with the traditional fire detection node, it significantly reduces the false alarm rate. At the same time, the node has good application effect in the experiment, and realizes a wireless intelligent fire detection node with low cost, wide application range and high intelligence.

A Multimodal Feature Selection Method for Remote Sensing Data Analysis Based on Double Graph Laplacian Diagonalization

When dealing with multivariate remotely sensed records collected by multiple sensors, an accurate selection of information at the data, feature, or decision level is instrumental in improving the scenes' characterization. This will also enhance the system's efficiency and provide more details on modeling the physical phenomena occurring on the Earth's surface. In this paper, we introduce a flexible and efficient method based on Graph Laplacians for information selection at different levels of data-fusion. The proposed approach combines data structure and information content to address the limitations of existing Graph Laplacians-based methods in dealing with heterogeneous datasets. Moreover, it adapts the selection to each homogenous area of the considered images according to their underlying properties. Experimental tests carried out on several multivariate remote sensing datasets show the consistency of the proposed approach.

Improved Measurement Accuracy in Critical Parameters of Safety-Critical Systems With Multisensor Data Fusion

Data fusion enables the integration of multiple sensor data associated with the same physical process for more comprehensive process representation, thereby improving accuracy in measuring critical parameters (CPs) of safety-critical systems (SCSs). In this article, we adopt the multisensor data fusion (MSDF) method to measure the CPs of SCS. The method divides the problem into three levels: data fusion level, feature level, and decision level. The proposed approach is applied on a case study of nuclear power plant (NPP). The result proves the effectiveness of measurement accuracy using MSDF.

Review of Data Fusion Methods for Real-Time and Multi-Sensor Traffic Flow Analysis

Recently, development in intelligent transportation systems (ITS) requires the input of various kinds of data in real-time and from multiple sources, which imposes additional research and application challenges. Ongoing studies on Data Fusion (DF) have produced significant improvement in ITS and manifested an enormous impact on its growth. This paper reviews the implementation of DF methods in ITS to facilitate traffic flow analysis (TFA) and solutions that entail the prediction of various traffic variables such as driving behavior, travel time, speed, density, incident, and traffic flow. It attempts to identify and discuss real-time and multi-sensor data sources that are used for various traffic domains, including road/highway management, traffic states estimation, and traffic controller optimization. Moreover, it attempts to associate abstractions of data level fusion, feature level fusion, and decision level fusion on DF methods to better understand the role of DF in TFA and ITS. Consequently, the main objective of this paper is to review DF methods used for real-time and multi-sensor (heterogeneous) TFA studies. The review outcomes are (i) a guideline of constructing DF methods which involve preprocessing, filtering, decision, and evaluation as core steps, (ii) a description of the recent DF algorithms or methods that adopt real-time and multi-sensor sources data and the impact of these data sources on the improvement of TFA, (iii) an examination of the testing and evaluation methodologies and the popular datasets and (iv) an identification of several research gaps, some current challenges, and new research trends.

Evaluation of different approaches to the fusion of Sentinel -1 SAR data and Resourcesat 2 LISS III optical data for use in crop classification

ABSTRACT This study evaluates various combinations of data fusion techniques at Pixel, Feature, and Decision level for crop classification using Sentinel-1 Synthetic Aperture Radar (SAR) data and Resourcesat-2 LISS (Linear Imaging Self Scanning) III, optical data for Yadgir District of Karnataka, India. For Pixel level data fusion, techniques such as brovey transformation (BT), principal component analysis (PCA), multiplicative transformation (MLT), and wavelet with IHS (intensity-hue-saturation) were used. Results were compared between different fusion techniques visually, statistically (using universal image quality index), and through image classification (Rule-based and Maximum likelihood) for major crops (Rice, Cotton, and Pigeon pea) in the area. The estimated crop area for all three major crops was compared with the Government statistics. Among the four pixel-level fusion techniques used, the wavelet method performed best in retaining the image quality. However, the study showed that using the feature-level fusion technique, maximum accuracy was obtained for Rice crop. In contrast, the decision-level fusion improved the efficiency for other crops (Cotton and Pigeon pea).

Investigation on Data Fusion of Multisource Spectral Data for Rice Leaf Diseases Identification Using Machine Learning Methods.

Rice diseases are major threats to rice yield and quality. Rapid and accurate detection of rice diseases is of great importance for precise disease prevention and treatment. Various spectroscopic techniques have been used to detect plant diseases. To rapidly and accurately detect three different rice diseases [leaf blight (Xanthomonas oryzae pv. Oryzae), rice blast (Pyricularia oryzae), and rice sheath blight (Rhizoctonia solani)], three spectroscopic techniques were applied, including visible/near-infrared hyperspectral imaging (HSI) spectra, mid-infrared spectroscopy (MIR), and laser-induced breakdown spectroscopy (LIBS). Three different levels of data fusion (raw data fusion, feature fusion, and decision fusion) fusing three different types of spectral features were adopted to categorize the diseases of rice. Principal component analysis (PCA) and autoencoder (AE) were used to extract features. Identification models based on each technique and different fusion levels were built using support vector machine (SVM), logistic regression (LR), and convolution neural network (CNN) models. Models based on HSI performed better than those based on MIR and LIBS, with the accuracy over 93% for the test set based on PCA features of HSI spectra. The performance of rice disease identification varied with different levels of fusion. The results showed that feature fusion and decision fusion could enhance identification performance. The overall results illustrated that the three techniques could be used to identify rice diseases, and data fusion strategies have great potential to be used for rice disease detection.