Data Fusion Scheme Research Articles

Data fusion of LIBS and PIL hyperspectral imaging: Understanding the luminescence phenomenon of a complex mineral sample

Laser-induced breakdown spectroscopy (LIBS) imaging is an innovative technique that associates the valuable atomic, ionic and molecular emission signals of the parent spectroscopy with spatial information. LIBS works using a powerful pulse laser as excitation source, to generate a plasma exhibiting emission lines of atoms, ions and molecules present in the ablated matter. The advantages of LIBS imaging are potential high sensitivity (in the order of ppm), easy sample preparation, fast acquisition rate (up to 1 kHz) and μm scale spatial resolution (weight of the ablated material in the order of ng). Despite these positive aspects, LIBS imaging easily provides datasets consisting of several million spectra, each containing several thousand spectral channels. Under these conditions, the current chemometric analyses of the raw data are still possible, but require too high computing resources. Therefore, the aim of this work is to propose a data compression strategy oriented to keep the most relevant spectral channel and pixel information to facilitate, fast and reliable signal unmixing for an exhaustive exploration of complex samples. This strategy will apply not only to the context of LIBS image analysis, but to the fusion of LIBS with other imaging technologies, a scenario where the data compression step becomes even more mandatory. The data fusion strategy will be applied to the analysis of a heterogeneous kyanite mineral sample containing several trace elements by LIBS imaging associated with plasma induced luminescence (PIL) imaging, these two signals being acquired simultaneously by the same microscope. The association of compression and spectral data fusion will allow extracting the compounds in the mineral sample associated with a fused LIBS/PIL fingerprint. This LIBS/PIL association will be essential to interpret the PIL spectral information, which is nowadays very complex due to the natural overlapped signals provided by this technique.

Fusion of visible-to-near-infrared and mid-infrared spectroscopy to estimate soil organic carbon

Spectral techniques such as visible-to-near-infrared (VIS–NIR) and mid-infrared (MIR) spectroscopies have been regarded as effective alternatives to laboratory-based methods for determining soil organic carbon (SOC). Research to explore the potential of the fusion of VIS–NIR and MIR absorbance for improving SOC prediction is needed, since each individual spectral range may not contain sufficient information to yield reasonable estimation accuracy. Here, we investigated two data fusion strategies that differed in input data, including direct concatenation of full-spectral absorbance and concatenation of selected predictors by optimal band combination (OBC) algorithm. Specifically, continuous wavelet transform (CWT) was adopted to optimize the spectral data before and after data fusion. Prediction models for SOC were developed using partial least squares regression. Results demonstrated that estimations for SOC using MIR absorbance (i.e., validation R2 = 0.45–0.64) generally outperformed those using VIS–NIR (i.e., validation R2 = 0.20–0.44). Compared to the raw absorbance counterparts, CWT decomposing could improve the prediction accuracy for SOC, for both the individual absorbance and the fusion of VIS–NIR and MIR absorbance. Among all the models investigated, the combinational use of VIS–NIR and MIR using OBC fusion at CWT scale of 1 yielded the optimal prediction, providing the highest validation R2 of 0.66. This model with 10 selected spectral parameters as input is of small total data volume, large processing speed and efficiency, confirming the potential of OBC in fusing both types of spectral data. In summary, CWT decomposing and OBC strategy are powerful algorithms in analyzing the spectral data, and allow the VIS–NIR and MIR spectral fusion models to improve the SOC estimation.

Segmentation phase measuring deflectometry for measuring structured specular surfaces

Accurate and fast three-dimensional (3D) measurement for industrial products/components designed to possess 3D structured shapes is a key driver for improved productivity. However, challenges for current techniques are considerable to measure structured specular surfaces. A technique named segmentation phase measuring deflectometry (SPMD) is proposed in this paper, which enables structured specular surfaces to be measured with high accuracy in one setup. Concept of segmentation in topology is introduced into phase measuring deflectometry, which separates a surface with complex structures into continuous segments. Each segment can be reconstructed based on gradient information to achieve good form accuracy, and all reconstructed segments can be fused into a whole 3D strucutred form result based on their absolute spatial positioning data. Here, we propose and discuss the principle of SPMD, a segmentation technique to separate a strucured surface into segments, a spatial positioning technique to obtain absolute position of the segments, and a data fusion strategy to fuse all reconstructed segments. Experimental results show SPMD can achieve nanometer level accuracy for form measurement of continuous segments by comparing with stylus profilometer, which is significantly higher than the accuracy of direct phase measuring deflectometry. Meanwhile, SPMD has micron level spatial positioning accuracy for structures by measuring two specular steps and comparing with coordinate measuring machine, which differentiates this technique from gradient-based phase measuring deflectometry that extends measurement capability from continuous specular surfaces to complex structured specular surfaces. Compared with the existing measurement techniques, SPMD significantly improved the convenience and ability to measure freeform and structured specular surfaces with the advantages of high measurement accuracy, fast measurement, and potential application for embedded measurement.

Predictive Error Compensating Wavelet Neural Network Model for Multivariable Time Series Prediction

Multivariable machine learning (ML) models are increasingly used for time series predictions. However, avoiding the overfitting and underfitting in ML-based time series prediction requires special consideration depending on the size and characteristics of the available training dataset. Predictive error compensating wavelet neural network (PEC-WNN) improves the time series prediction accuracy by enhancing the orthogonal features within a data fusion scheme. In this study, time series prediction performance of the PEC-WNNs have been evaluated on two different problems in comparison to conventional machine learning methods including the long short-term memory (LSTM) network. The results have shown that PECNET provides significantly more accurate predictions. RMSPE error is reduced by more than 60% with respect to other compared ML methods for Lorenz Attractor and wind speed prediction problems.

Geographical identification of Italian extra virgin olive oil by the combination of near infrared and Raman spectroscopy: A feasibility study

The potential of the combination of near infrared (NIR) spectroscopy and Raman spectroscopy to differentiate Italian and Greek extra virgin olive oil (EVOO) by geographical origin was evaluated. Near infrared spectroscopy and Raman fingerprints of both study groups (extra virgin olive oil from the two countries) were pre-processed, merged by low-level and mid-level data fusion strategies and submitted to partial least-squares discriminant analysis. The classification models were cross-validated. After low-level data fusion, the partial least-squares discriminant analysis correctly predicted the geographical origins of extra virgin olive oils in cross-validation with 93.9% accuracy, while sensitivity and specificity were 77.8% and 100%, respectively. After mid-level data fusion, the partial least-squares discriminant analysis correctly predicted the geographical origins of extra virgin olive oils in cross-validation with 97.0% accuracy, while sensitivity and specificity were 88.9% and 100%, respectively. In this preliminary study, improved discrimination of Italian extra virgin olive oils was achieved by the synergism of near infrared spectroscopy and Raman spectroscopy as compared to the discrimination obtained by the separate laboratory techniques. This pilot study shows encouraging results that could open a new avenue for the authentication of Italian extra virgin olive oil.

Multi-modal deep learning of functional and structural neuroimaging and genomic data to predict mental illness.

Neuropsychiatric disorders such as schizophrenia are very heterogeneous in nature and typically diagnosed using self-reported symptoms. This makes it difficult to pose a confident prediction on the cases and does not provide insight into the underlying neural and biological mechanisms of these disorders. Combining neuroimaging and genomic data with a multi-modal 'predictome' paves the way for biologically informed markers and may improve prediction reliability. With that, we develop a multi-modal deep learning framework by fusing data from different modalities to capture the interaction between the latent features and evaluate their complementary information in characterizing schizophrenia. Our deep model uses structural MRI, functional MRI, and genome-wide polymorphism data to perform the classification task. It includes a multi-layer feed-forward network, an encoder, and a long short-term memory (LSTM) unit with attention to learn the latent features and adopt a joint training scheme capturing synergies between the modalities. The hybrid network also uses different regularizers for addressing the inherent overfitting and modality-specific bias in the multi-modal setup. Next, we run the network through a saliency model to analyze the learned features. Integrating modalities enhances the performance of the classifier, and our framework acquired 88% (P < 0.0001) accuracy on a dataset of 437 subjects. The trimodal accuracy is comparable to the state-of-the-art performance on a data collection of this size and outperforms the unimodal and bimodal baselines we compared. Model introspection was used to expose the salient neural features and genes/biological pathways associated with schizophrenia. To our best knowledge, this is the first approach that fuses genomic information with structural and functional MRI biomarkers for predicting schizophrenia. We believe this type of modality blending can better explain the disorder's dynamics by adding cross-modal prospects.Clinical Relevance- This study combinedly learns imaging and genomic features for the classification of schizophrenia. The data fusion scheme extracts modality interactions, and the saliency experiments report multiple functional and structural networks closely connected to the disorder.

FIA–HRMS fingerprinting subjected to chemometrics as a valuable tool to address food classification and authentication: Application to red wine, paprika, and vegetable oil samples

The rise of food fraud practices, affecting a wide variety of goods and their specific characteristics (e.g., quality or geographical origin), demands rapid high-throughput analytical approaches to ensure consumers protection. In this context, this study assesses flow injection analysis coupled to high-resolution mass spectrometry (FIA–HRMS), using a fingerprinting approach and combined with chemometrics, to address four food authentication issues: (i) the geographical origin of three Spanish red wines, (ii) the geographical origin of three European paprikas, (iii) the distinction of olive oil from other vegetable oils and (iv) the assessment of its quality category. In each case, negative and positive ionisation FIA–HRMS fingerprints, and two different data fusion strategies, were evaluated. After external validation, excellent classification accuracies were reached. Moreover, high-resolution mass spectrometry (HRMS) allowed sample matrices characterisation by the putative identification of the most common ions.

Chestnut Cultivar Identification through the Data Fusion of Sensory Quality and FT-NIR Spectral Data.

The world production of chestnuts has significantly grown in recent decades. Consumer attitudes, increasingly turned towards healthy foods, show a greater interest in chestnuts due to their health benefits. Consequently, it is important to develop reliable methods for the selection of high-quality products, both from a qualitative and sensory point of view. In this study, Castanea spp. fruits from Italy, namely Sweet chestnut cultivar and the Marrone cultivar, were evaluated by an official panel, and the responses for sensory attributes were used to verify the correlation to the near-infrared spectra. Data fusion strategies have been applied to take advantage of the synergistic effect of the information obtained from NIR and sensory analysis. Large nuts, easy pellicle removal, chestnut aroma, and aromatic intensity render Marrone cv fruits suitable for both the fresh market and candying, i.e., marron glacé. Whereas, sweet chestnut samples, due to their characteristics, have the potential to be used for secondary food products, such as jam, mash chestnut, and flour. The research lays the foundations for a superior data fusion approach for chestnut identification in terms of classification sensitivity and specificity, in which sensory and spectral approaches compensate each other’s drawbacks, synergistically contributing to an excellent result.

Multisource information fusion strategies of mass spectrometry and Fourier transform infrared spectroscopy data for authenticating the age and parts of Vietnamese ginseng

AbstractAiming at two different classification tasks in the field of quality evaluation of valuable Chinese herbal medicine, the applicability of data fusion strategy based on different complementary analysis techniques was studied. In this study, attenuated total reflection Fourier transform infrared spectroscopy (ATR‐FTIR) and ultra‐performance liquid chromatography quadrupole time‐of‐flight mass spectrometry (UPLC‐QTOF/MS) were used to analyze the different parts (including roots, stems, leaves, and fibrils) of 2‐to 5‐year‐old Vietnamese ginseng. The multivariate classification models (orthogonal partial least squares discrimination analysis, OPLS‐DA; support vector machine, SVM) were established using low‐level and mid‐level data fusion methods to identify different parts and age of Vietnamese ginseng. The OPLS‐DA model have shown that the fusion data matrix with low‐level fusion processing could separate Vietnamese ginseng samples with different growth years to the greatest extent, whereas the single data matrix had partial overlap. But 100% of prediction set classification accuracy could be achieved. The SVM model combined with two parameter optimization algorithms (grid search, GS; genetic algorithm, GA) was used to identify Vietnamese ginseng samples with different ages. The mid‐level fusion strategy based on the recursive feature elimination (RFE) features variable extraction method was more suitable for SVM model. In the model established by combining the two parameter optimization methods, the identification effect can reach 83.33%. The results showed that data fusion, as an effective strategy, could distinguish different ages and parts of Vietnamese ginseng.

Coupling of spectrometric, chromatographic, and chemometric analysis in the investigation of the photodegradation of sulfamethoxazole

A workflow is proposed for the study of the photodegradation process of the sulfamethoxazole (SMX) based on the combination of different experimental techniques, including liquid chromatography, mass spectrometry, UV–Visible spectrophotometry, and the treatment of all the analytical data with advanced chemometric methods. SMX, which is one of the most widely used antibiotics worldwide and has been found at remarkable concentrations in various rivers and effluents over all Europe, was degraded in the laboratory under a controlled source of UV radiation, which simulates the environmental solar radiation (Suntest). Kinetic monitoring of the photodegradation process was performed using UV–Visible spectrophotometric measurements and by further Liquid Chromatography with Diode Array Detector and Mass Spectrometry analysis (LC-DAD-MS). Additionally, the acid-base properties were also investigated to see how the pH can affect the speciation of this substance during the photodegradation process. Based on the Multivariate Curve Resolution–Alternating Least Squares (MCR-ALS) application, the proposed chemometric method coped with the large amounts of data generated by the different analytical techniques used to monitor the evolution of the photodegradation process. Their simultaneous analysis involved applying a data fusion strategy and an advanced MCR-ALS constrained analysis, which allowed and improved the description of the complete degradation process, detecting the different species of the reaction, and identifying the possible transformation products formed. A total number of six species were resolved in the degradation process of SMX. In addition to the initial SMX, a second species corresponded to a conformational isomer, and the other four species represented different photoproducts, which have also been identified. Furthermore, three different acid-base species of SMX were obtained, and their pKa values were estimated.

Distributed State Estimation for Stochastic Linear Hybrid Systems With Finite-Time Fusion

Building on the classical interacting multiple model algorithm, this article develops a novel distributed approach for solving the problem of distributed state estimation for stochastic linear hybrid systems. In our distributed framework, a network of sensors is employed and each of them measures only a portion of the system outputs, thus, the system may not necessarily be observable for any individual sensor. To tackle with this, we develop an effective data fusion scheme that enables the sensor network to collectively measure the output of the hybrid system, by leveraging the communication between neighboring sensors. Consequently, each sensor only needs to process a relatively small set of data and is able to locally and identically observe the states (both the continuous states and discrete modes) of stochastic linear hybrid systems. Stability of the proposed distributed algorithm is proved theoretically, in the sense that the covariance of estimation error is lower and upper bounded. Finally, the performance of the proposed algorithm is demonstrated via illustrative simulations on a maneuvering aircraft tracking problem.

Toward Secure Data Fusion in Industrial IoT Using Transfer Learning

As an emerging technology, the industrial Internet of Things (IIoT) can promote the development of industrial intelligence, improve production efficiency, and reduce manufacturing costs. In IIoT, the improvement and progress of industrial production and applications are inseparable from data fusion, a process that realizes the collection, analysis, and processing of the massive IoT data generated by industrial equipment and applications. IIot demands a real-time, effective, and privacy-preserving data fusion process. However, the existing works need to train different learning models for data analysis, which cannot meet real-time requirements in IIoT. Meanwhile, the lack of defense against internal attacks and the difficulty to balance system performance and privacy protection hinder the effectiveness and privacy protection in the data fusion process. To solve the abovementioned problems, in this article, we propose a new transfer learning-based secure data fusion strategy (TSDF) for IIoT. The proposed TSDF consists of three parts, guidance based deep deterministic policy gradient (GDDPG) algorithm for task classification, transfer learning based GDDPG for grouping of task receivers, and a multiblockchain mechanism for privacy preservation. The experiment results show that TSDF can achieve high system throughput and low latency, providing privacy preservation in data fusion under various IIoT application environments.

재진입 표적추적을 위한 복수 레이더 2단계 정보융합 기법

This paper proposes a two-stage track-to-track fusion algorithm for re-entry target tracking using multiple radars. The existing track-to-track fusion strategy relying only on the kinematic information shows unreliable target tracking performance when the target of interest is adjacent with other objects. To prevent tracking performance degradation due to false track fusion, the track paring hypothesis is evaluated using both the feature data and the kinematic data provided by radars. A recursive Bernoulli filter is desinged to discriminate the target identity by fusing binary decision data which correspond to the most probable track pairing hypothesis. Since our approach exploits the statistical property of the available decision data, it can enhance the target tracking and identification performance. Through the computer simulations for a typical re-entry target tracking scenario, the effectiveness of the suggested data fusion scheme is demonstrated.

Direct authentication and composition quantitation of red wines based on Tri-step infrared spectroscopy and multivariate data fusion

A robust data fusion strategy integrating Tri-step infrared spectroscopy (IR) with electronic nose (E-nose) was established for rapid qualitative authentication and quantitative evaluation of red wines using Cabernet Sauvignon as an example. The chemical fingerprints of four types of wines were thoroughly interpreted by Tri-step IR, and the defined spectral fingerprint region of alcohol and sugar was 1200–950 cm−1. The wine types were authenticated by IR-based principal component analysis (PCA). Furthermore, ten quantitative models by partial least squares (PLS) were built to evaluate alcohol and total sugar contents. In particular, the model based on the fusion datasets of spectral fingerprint region and E-nose was superior to the others, in which RMSEP reduced by 47.95% (alcohol) and 79.90% (total sugar), rp increased by 11.95% and 43.47%, and RPD >3.0. The developed methodology would be applicable for mass screening and rapid multi-chemical-component quantification of wines in a more comprehensive and efficient manner.

Data fusion of Laser Induced Breakdown Spectroscopy and Diffuse Reflectance for improved analysis of mineral species in copper concentrates

Copper concentrates is an intermediate product of the copper production process, which can be sold or further processed by pyrometallurgical and electrolytic processes to obtain copper with a grade of 99.99%. The mineralogical analysis of these concentrates is very important for quality control and monitoring the feed of smelters. The use of spectroscopy provides a means to carry out this analysis by measuring the reflected or emitted spectra when the samples are illuminated by different light sources. Diffuse Reflectance (DRS) and Laser Induced Breakdown Spectroscopy (LIBS) provide key complementary information for identifying and quantifying mineralogical components. Combining the information of these spectra is crucial to perform the measurements at the same spatial domain. This contribution presents the analysis of copper concentrates using a multi-purpose set-up, measuring both spectra at the same spatial position, and the use of data fusion techniques for blending the information. Low and mid-level data fusion strategies are compared in terms of their effectiveness. In this application, the experimental results show that mid-level data fusion provides the best result outperforming the predictions done by the separated information sources. These results indicate that the spectral measurements obtained by LIBS and DRS from the same measuring point with high spatial resolution can be combined for enhancing the mineralogical analysis of copper concentrates.

GeoAI in terrain analysis: Enabling multi-source deep learning and data fusion for natural feature detection

In this paper we report on a new GeoAI research method which enables deep machine learning from multi-source geospatial data for natural feature detection. In particular, a multi-source, deep learning-based object detection pipeline was developed. This pipeline introduces three new features: First, strategies of both data-level fusion (i.e., channel expansion on convolutional neural networks) and feature-level fusion were integrated into the object detection model to allow simultaneous machine learning from multi-source data, including remote sensing imagery and Digital Elevation Model (DEM) data. Second, a new data fusion strategy was developed to blend DEM data and its derivatives to create a new, fused data source with enriched information content and image features. The model has also enabled deep learning by combining both the proposed data fusion and feature-level fusion strategies to yield a much-improved detection result. Third, two different sets of data augmentation techniques were applied to the multi-source training data to further improve the model performance. A series of experiments were conducted to verify the effectiveness of the proposed strategies in multi-source deep learning.

Internet of Things and occupational well-being in industry 4.0: A systematic mapping study and taxonomy

A high estimate of not only workplace fatal injuries but also nonfatal injuries and illnesses via overexertion, and contact with objects, equipment, and machinery workplace has been reported over the years. To address this issue, the Occupational Health and Safety (OHS) program has put an emphasis on policy and regulation for prevention, protection, and improvement of the individuals’ health related to the working conditions and industrial environment. In the last 10 years, the Internet of Things (IoT) has matured to seamlessly enable real-time communications and cooperation among machines, environments, and humans using data analytics. Thus, IoT offers potential technical solutions for the prevention and protection of workplace injuries and illnesses. Moreover, IoT invites opportunities for collaboration with OHS in various industries. This article presents a systematic mapping study of the literature to address the impact of IoT on occupational well-being, analyzing the progress of Industry 4.0 during the last decade. This study systematizes the literature providing a taxonomy of the area through the results of four general, four focused, and two statistical research questions. These questions outline industrial environments and aspects of health concerning workers’ well-being, concentrating on a human-centered approach leveraged by physiological measurements and psychological health. In addition, this paper explores questions regarding IoT’s technical components, such as sensors, devices, and communication technologies, investigating methods of data processing supported by the employment of classification algorithms and data fusion strategies. As a result, the systematic mapping process initially found 7515 articles from six academic databases in the period from 2009 to 2019. After the execution of filtering methods, a complete read of 67 articles allowed to answer quantitatively and qualitatively the research questions. The classification of the answers contributed to systematize the literature through the taxonomy and the relationships among the topics covered by the articles. Accordingly, this research produced theoretical benefits, mainly, a broad view of the state-of-the-art, a taxonomy to guide related researches, and guidelines for future works. Furthermore, this research would benefit management and corporations by shedding light on technologies explored in the literature and elucidating their feasibility in support of the workforce’s safety, psychological, and physical health.

Quantitative analysis of caprolactam in sauce-based food using infrared spectroscopy combined with data fusion strategies

In this study, we aimed to quantify caprolactam content in sauce-based food using near-infrared (NIR) and mid-infrared (MIR) spectroscopy, combined with chemometrics and data fusion strategy. Sauce-based food samples with different caprolactam levels were prepared. Partial least squares (PLS) and support vector machine (SVM) regression models were developed based on NIR and MIR spectral data. The high-level fusion model showed the best performance for quantifying caprolactam for sauces, with square of correlation coefficient of prediction (Rp2) in range of 0.9929−0.9990 and root mean square error of prediction (RMSEP) in range of 1.0794–2.9303 mg kg−1. Finally, a comprehensive model of caprolactam content in sauces was established with Rp2 of 0.9961 and RMSEP of 2.1505. These findings indicated that a combination of NIR and MIR spectroscopy as high-level fusion strategy can achieve rapid, non-destructive, and accurate quantification of caprolactam content in sauces and hence can be used for monitoring the safety of food.

Multi-scale deep coupling convolutional neural network with heterogeneous sensor data for intelligent fault diagnosis

With the innovation and development of detection technology, various types of sensors are installed to monitor the operating status of equipment in modern industry. Compared with the same type of sensors for monitoring, heterogeneous sensors can collect more comprehensive complementary fault information. Due to the large distribution differences and serious noise pollution of heterogeneous sensor data collected in industrial sites, this brings certain challenges to the development of heterogeneous data fusion strategies. In view of the large distribution difference in the feature spatial of heterogeneous data and the difficulty of effective fusion of fault information, this paper presents a multi-scale deep coupling convolutional neural network (MDCN), which is used to map the heterogeneous fault information from different feature spaces to the common spaces for full fusion. Specifically, a multi-scale convolution module (MSC) with multiple filters of different sizes is adopted to extract multi-scale fault features of heterogeneous sensor data. Then, the maximum mean discrepancy (MMD) is applied to measure the distance between different spatial features in the coupling layer, and the common failure information in the heterogeneous data is mined by minimizing MMD to fuse effectively in order to identify the failure state of the device. The validity of this method is verified by the data collected on a first-level parallel gearbox mixed fault experiment platform.

A Measurement Framework for Explicit and Implicit Urban Traffic Sensing

Urban traffic sensing has been investigated extensively by different real-time sensing approaches due to important applications such as navigation and emergency services. Basically, the existing traffic sensing approaches can be classified into two categories by sensing natures, i.e., explicit and implicit sensing. In this article, we design a measurement framework called EXIMIUS for a large-scale data-driven study to investigate the strengths and weaknesses of two sensing approaches by using two particular systems for traffic sensing as concrete examples. In our investigation, we utilize TB-level data from two systems: (i) GPS data from five thousand vehicles, (ii) signaling data from three million cellphone users, from the Chinese city Hefei. Our study adopts a widely used concept called crowdedness level to rigorously explore the impacts of contexts on traffic conditions including population density, region functions, road categories, rush hours, holidays, weather, and so on, based on various context data. We quantify the strengths and weaknesses of these two sensing approaches in different scenarios and then we explore the possibility of unifying two sensing approaches for better performance by using a truth discovery-based data fusion scheme. Our results provide a few valuable insights for urban sensing based on explicit and implicit data from transportation and telecommunication domains.