Multiple Sensor Data Fusion Research Articles

Deep learning in standard least-squares theory of linear models: Perspective, development and vision

Inspired by the attractive features of least-squares theory in many practical applications, this contribution introduces least-squares-based deep learning (LSBDL). Least-squares theory connects explanatory variables to predicted variables, called observations, through a linear(ized) model in which the unknown parameters of this relation are estimated using the principle of least-squares. Conversely, deep learning (DL) methods establish nonlinear relationships for applications where predicted variables are unknown (nonlinear) functions of explanatory variables. This contribution presents the DL formulation based on least-squares theory in linear models. As a data-driven method, a network is trained to construct an appropriate design matrix of which its entries are estimated using two descent optimization methods: steepest descent and Gauss–Newton. In conjunction with interpretable and explainable artificial intelligence, LSBDL leverages the well-established least-squares theory for DL applications through the following three-fold objectives: (i) Quality control measures such as covariance matrix of predicted outcome can directly be determined. (ii) Available least-squares reliability theory and hypothesis testing can be established to identify mis-specification and outlying observations. (iii) Observations’ covariance matrix can be exploited to train a network with inconsistent, heterogeneous and statistically correlated data. Three examples are presented to demonstrate the theory. The first example uses LSBDL to train coordinate basis functions for a surface fitting problem. The second example applies LSBDL to time series forecasting. The third example showcases a real-world application of LSBDL to downscale groundwater storage anomaly data. LSBDL offers opportunities in many fields of geoscience, aviation, time series analysis, data assimilation and data fusion of multiple sensors.

Chip Morphology Prediction in Inconel 718 Milling through Machine Learning to Control Surface Integrity

A nickel-based aerospace superalloy, Inconel 718 presents machining challenges because of its hardness and strength. Monitoring and predicting chip morphology during milling is essential for early defect detection and process optimisation. This study examines the correlation between sensor signals with surface roughness and chip morphology in milling Inconel 718 using machine learning (ML). Due to progressive tool wear and heat generation, the surface roughness varies in addition to the chip exhibiting different morphologies, such as continuous, discontinuous, and oxidised chips. AE signals were analysed in the time and frequency domains to identify chip morphology transitions. An accelerometer captured cutting vibration signals that showed higher instability during discontinuous chip formation. Chip colour due to oxidization varies with milling forces as a result of tool wear. Based on multiple sensor data fusion, a random forest model predicts better chip morphology from different machining parameters. The integrated ML system enables real-time monitoring of chip morphology mechanisms through diverse signals. This permits early diagnosis of surface integrity and chip morphologies indicating imminent tool wear. The approach enhances process stability and tool life when milling difficult-to-machine alloys. It demonstrates the viability of relating sensor signals to fundamental mechanisms through AI for intelligent machining.

Perspectives on AI-driven systems for multiple sensor data fusion

Abstract Artificially intelligent automation has not only impact on sensor technologies, but also on comprehensive multiple sensor systems for assisting situational awareness and decision-making. This is particularly true for integrated Manned-unManned-Teaming (MuM-T), for example. From a systems engineering perspective which does not exclude applications in the defence domain, three tasks need to be fulfilled: (1) Design artificially intelligent automation in a way that human beings are mentally and emotionally able to master each situation. (2) Identify technical design principles to facilitate the responsible use of AI in ethically critical applications. (3) Guarantee that human decision makers always have full superiority of information, decision-making, and options of action. Our discussion of AI-driven systems for multiple sensor data fusion results in recommendations and key results. We are addressing the algorithms needed, the data to be processed, the programming skills required, the computing devices to be used, the inevitable anthropocentric design, the reviewing of R & D efforts necessary, and the integration of different dimensions in a systems-of-systems point of view.

Full-body pose estimation for excavators based on data fusion of multiple onboard sensors

To reduce machine-related accidents on sites, automatically monitoring the full-body poses of operating heavy machines is crucial. Conventional pose estimation systems relying on homogeneous sensors are vulnerable to negative environmental impacts, leading to inaccurate and unstable estimation of machine states. Hence, a full-body pose estimation framework is proposed for excavators, with a data fusion strategy to utilize different types of onboard sensors for enhanced accuracy and robustness. Specifically, a non-invasive onboard visual-inertial sensor system is designed for data fusion. Then, through competitive and complementary data fusion, the keypoints describing the full-body poses of the excavator are tracked in 3D space. Especially, an EKF-based localization algorithm is developed for optimized multi-keypoint tracking, which is verified to improve the accuracy and robustness of pose estimation by a real-world excavator case study. The proposed sensor-fusion method can effectively improve operational safety, by accurately monitoring the motion of heavy machines operating on construction sites.

Multiple Sensor Data Fusion Based Automatic Plant Pot Watering Employing Mapping byLinear Regression

Multiple Sensor Data Fusion Based Automatic Plant Pot Watering Employing Mapping byLinear Regression

Similarity-based probabilistic remaining useful life estimation for an aeroengine under variable operational conditions

System-level remaining useful life (RUL) estimation is difficult due to multiple degrading components, external disturbances, and variable operational conditions. A similarity-based approach does not rely on health assessment and is more suitable for system-level RUL estimation. However, for practical applications, how to capture effective degradation features from raw data, how to fuse multiple nonlinear sensor data, and how to handle multiple source uncertainties need to be considered. To solve the above challenges, this study focuses on RUL estimation for systems under variable operational conditions. A similarity-based probabilistic RUL estimation strategy is proposed and verified using the NASA aeroengine dataset. First, measurement uncertainty can be addressed. Proper degradation features are extracted by three defined indicators. Subsequently, multiple nonlinear sensor data fusion and unsupervised synthesized health index construction can be realized using the proposed deep autoencoder-based polynomial regression approach. Finally, this strategy can handle the modeling and prediction uncertainties, including providing probabilistic RUL estimation results by well-trained residual-based similarity models. The verification results indicate the effectiveness and feasibility of the proposed strategy.

An information fusion approach for increased reliability of condition monitoring With homogeneous and heterogeneous sensor systems

In machinery condition monitoring, it is often vital to consider information from multiple sources due to possible sensor failure or signal distortion, which may result in misclassification of the health status. An issue with multiple sensor data fusion, however, is that the classification can be affected by conflicting results between sensor signals. The proposed method uses a novel three-module approach to information fusion in order to address the problem. Features corresponding to signal integrity are extracted and employed for training a one-class support vector machine to detect unwanted distortions or sensor failures. Different classifiers are trained for the different sensor types available and each signal recorded is used to determine machine health. Decision-level fusion is conducted through a majority voting system using the integrity scores derived from the OCSVMs and the separate classification results. From this, a dynamically weighted fault diagnosis based on sensor signal quality is obtained. Experimental verification using vibration and acoustic emission signals show that the framework is viable and allows for an increased reliability in machinery health diagnosis.

Convolutional Neural Network-Based Travel Mode Recognition Based on Multiple Smartphone Sensors

Nowadays, large-scale human mobility has led to increasingly severe traffic congestion in cities, how to accurately identify people’s travel mode has become particularly important for urban traffic planning and management. However, traditional methods are based on telephone interviews or questionnaires, which makes it difficult to obtain accurate and effective data. Nowadays, numbers of smartphones are equipped with various sensors, including accelerometers, gyroscopes, and GPS, providing a novel social sensing data source to detect people’s travel modes. The fusion of multiple sensor data is a promising way for travel mode detection. However, how to use these sensor data to accurately detect travel mode is still a challenging task. In this paper, we presented a light-weight method for travel mode detection based on four types of smartphone sensor data collected from an accelerometer, gyroscope, magnetometer, and barometer, and a prototype application was developed. Then, a novel convolutional neural network (CNN) was designed to identify five representative travel modes (walk, bicycle, bus, car, and metro). We compared the overall performance of the proposed method via different hyperparameters, and the experimental results show that the F value of the proposed method reaches 97%, which verified the effectiveness of the proposed method for travel mode classification.

Energy-Efficiency Computation Offloading Strategy in UAV Aided V2X Network With Integrated Sensing and Communication

Integrated Sensing and Communications (ISAC) technology can jointly design radio sensing and communication functionalities, which enable 6G Ere to have the ability to “see” the physical world rather than communication-only. Benefitting from ISAC, vehicular-to-everything (V2X) networks may efficiently complete high-precision traffic environment perception. Furthermore, with the assistance of flexibly deployed Unmanned Aerial Vehicles (UAVs), the V2X networks overcome the limited sensing range of sensors equipped on a vehicle and guarantee safe driving. This paper proposes an energy-efficient computation offloading strategy for multiple sensor data fusion in UAV Aided V2X Network supported by Integrated Sensing and Communication. Firstly, a vehicle-UAV cooperative perception architecture is proposed to perceive a wide range of traffic environments. Secondly, we introduce a computation offloading strategy jointly considering offloading decisions and dynamic computing resource allocation. Finally, a successive convex approximation (SCA) algorithm transforms a non-convex formulation problem into a tractable convex approximation problem. The simulation results show that the strategy proposed in this paper reduces the UAV energy consumption and data fusion task processing delay.

Review on the wearable health-care monitoring system with robust motion artifacts reduction techniques

PurposeWearables are gaining prominence in the health-care industry and their use is growing. The elderly and other patients can use these wearables to monitor their vitals at home and have them sent to their doctors for feedback. Many studies are being conducted to improve wearable health-care monitoring systems to obtain clinically relevant diagnoses. The accuracy of this system is limited by several challenges, such as motion artifacts (MA), power line interference, false detection and acquiring vitals using dry electrodes. This paper aims to focus on wearable health-care monitoring systems in the literature and provides the effect of MA on the wearable system. Also presents the problems faced while tracking the vitals of users.Design/methodology/approachMA is a major concern and certainly needs to be suppressed. An analysis of the causes and effects of MA on wearable monitoring systems is conducted. Also, a study from the literature on motion artifact detection and reduction is carried out and presented here. The benefits of a machine learning algorithm in a wearable monitoring system are also presented. Finally, distinct applications of the wearable monitoring system have been explored.FindingsAccording to the study reduction of MA and multiple sensor data fusion increases the accuracy of wearable monitoring systems.Originality/valueThis study also presents the outlines of design modification of dry/non-contact electrodes to minimize the MA. Also, discussed few approaches to design an efficient wearable health-care monitoring system.

Driving assistance system based on data fusion of multisource sensors for autonomous unmanned ground vehicles

Autonomous unmanned ground vehicle is a product of technological development. At present, it has become a trend to provide multi-view real-time environmental state for autonomous unmanned ground vehicles. And more and more object detection algorithms will be applied in autonomous driving. In the recent years, multi-sensor-based driving assistance systems have become a way to solve these problems. In this paper, we introduce a driving assistance system based on data fusion of multiple sensors for autonomous driving. On the one hand, six fisheye cameras and twelve ultrasonic radars are used to collect surround-view environmental state. On the other hand, this system uses a low-light camera and a lidar to provide forward-view environmental state. In the meantime, we design panoramic mosaic algorithm, surround-view data fusion algorithm and forward-view data fusion algorithm. Besides, object detection algorithm is used to provide forward-view detection results. This driving assistance system will advance research into autonomous unmanned ground vehicles.

ANN Assisted Multi Sensor Information Fusion for BLDC Motor Fault Diagnosis

Multiple sensor data fusion is necessary for effective condition monitoring as the electric machines operate in a wide range of diverse operations. This study investigates sensor acquired vibration and current signals to establish a reliable multi-fault diagnosis framework of a brushless DC (BLDC) motor. Faults in stator and rotor were created deliberately by shorting two adjacent windings and creating a hole on the surface, respectively. The threshold for different health states was obtained by the third harmonic analysis of motor current. Later, the key features from sensor acquired current and vibration signals are selected based on monotonicity and reduced using the principal component analysis (PCA). For future predictions, an artificial neural network (ANN) is used to classify different fault features and its performance is evaluated using several metrics. Analysis of motor current harmonics and impulsive vibration response at the same time provides a thorough health estimation of BLDC motor in the presence of both electrical and mechanical faults. Multiple sensor information is fused to obtain a better understanding of the fault characteristics and mitigate the randomness of fault diagnosis. The proposed model was able to detect and classify multiple fault features with higher accuracy compared to other similar methods.

Perceptual Feedback Mechanism Sensor Technology in e‐Commerce IoT Application Research

With the development of sensor technology and the Internet of Things (IoT) technology, the trend of miniaturization of sensors has prompted the inclusion of more sensors in IoT, and the perceptual feedback mechanism among these sensors has become particularly important, thus promoting the development of multiple sensor data fusion technologies. This paper deeply analyzes and summarizes the characteristics of sensory data and the new problems faced by the processing of sensory data under the new trend of IoT, deeply studies the acquisition, storage, and query of sensory data from the sensors of IoT in e‐commerce, and proposes a ubiquitous storage method for massive sensory data by combining the sensory feedback mechanism of sensors, which makes full use of the storage resources of IoT storage network elements and maximally meets the massive. In this paper, we propose a ubiquitous storage method for massive sensing data, which makes full use of the storage resources of IoT storage network elements to maximize the storage requirements of massive sensing data and achieve load‐balanced data storage. In this paper, starting from the overall development of IoT in recent years, the weak link of intelligent information processing is reinforced based on the sensory feedback mechanism of sensor technology.

Multiple sensor data fusion algorithm based on fuzzy sets and statistical theory

Multiple sensor data fusion algorithm based on fuzzy sets and statistical theory

A review on multi sensor data fusion technique in CNC machining of tailor-made nanocomposites

In this study, a multiple sensor data fusion system is anticipated as essential for monitoring of cutting operations, by identifying suitable sensor locations to obtain feedback signals periodically. The sensor signal derives the failure during machining owing to complex cutting tool geometry even in machining of composites. Nano metal matrix composites (NMMC) being extremely upright in mechanical characteristics, consequently machining of these hybrid nano metal matrix composites reinforced with difficult-to-cut nano particles leads to reduced tool life thereby causing rapid flank wear. Therefore, it is a challenge to identify the wear features caused during machining of tailor made NMMC’s, reducing wastage and preventing machine malfunction. This paper presents a comprehensive review on the machining strategies in extreme output conditions which rely on input parameters of speed, feed and depth of cut influencing tool life during CNC machining. This can be achieved only with multiple sensor data fusion technique during CNC machining.

Analytical modeling of tool health monitoring system using multiple sensor data fusion approach in hard machining

In recent years, a lot of work has been carried out on advance materials, especially hard metals and alloys, which is a vital requirement in many manufacturing industries. Nowadays steel with hardness beyond 30 HRC is employed in aircraft, hydraulics, car part manufacturing areas, etc. In the machining process, as the hardness of work material increases, tool health prone to deteriorate. Therefore, for healthier machinability requirements in machining of hard metals, the online tool health monitoring systems (THMS) are being developed using different feedback techniques. This study mainly concentrates on the advancement of online THMS using multiple sensors. Force, tool vibration, and surface roughness signals were recorded while machining EN24 hardened steel using coated carbide insert on CNC lathe. A novel analytical model of sensor data fusion has been presented for better understanding of sensors and their interaction. A comparative assessment of sensor fusion approach has been investigated. Experimental findings have successfully shown that the results obtained using a fusion function (U), validates better confirmability over single sensor-based approach.

Real-time 3D reconstruction method using massive multi-sensor data analysis and fusion

This paper proposes a method to reconstruct three-dimensional (3D) objects using real-time fusion and analysis of multiple sensor data. This paper attempts to create a realistic 3D visualization with which a remote pilot can intuitively control a remote unmanned robot by utilizing the characteristics of massive sensor data. The 3D reconstruction system proposed in this paper is comprised of 3D and two-dimensional (2D) data segmentation method, a 3D reconstruction method applied to each object, and a projective texture mapping method. Specifically, we propose applying both a 2D region extraction method and a 3D mesh modeling method to each object. The proposed schemes are implemented as a real-time application to verify real-time performance. This paper proves that 3D meshes can be modeled in real time by using the proposed method. The proposed method allows the remote control of a robot for real-time 3D rendering of remote scenes, which is essential for various tasks in areas that cannot be easily accessed by humans.

A data-level fusion approach for degradation modeling and prognostic analysis under multiple failure modes

ABSTRACTOperating units, in practice, often suffer from multiple modes of failure, and each failure mode has a distinct influence on the service life cycle path of a unit. The rapid development of sensor and communication technologies has enabled multiple sensors to simultaneously monitor and track the health status of a unit in real time. However, one challenging question that remains to be resolved is how to leverage data from multiple sensors for better degradation modeling and prognostic analysis, especially when there are multiple failure modes. Currently, many of the existing approaches in prognostics either (a) fail to capture the dependency between sensors and instead focus on analyzing each sensor independently or (b) fail to incorporate the failure-mode diagnosis for better degradation modeling and prognostics during condition monitoring. To address the limitations in the existing literature, we propose a data-level fusion methodology to construct a composite failure-mode index, named FM-INDEX, via the fusion of multiple sensor data. Our goal is to utilize the FM-INDEX to better characterize the failure mode of an operating unit in real time, thus leading to better degradation modeling and prognostic analysis. A case study that involves the degradation data set of an aircraft gas turbine engine with two potential failure modes is conducted to numerically evaluate the performance of our proposed method compared to other techniques in the related literature.

Data Fusion-Based Multi-Object Tracking for Unconstrained Visual Sensor Networks

Camera node perception capability is one of the crucial issues for visual sensor networks, which belongs to the field of Internet of Things. Multi-object tracking is an important feature in analyzing object trajectories across multiple cameras, thus allowing synthesis of data and security analysis of images in various scenarios. Despite intensive research in the last decades, it remains challenging for tracking systems to perform in real-world situations. We therefore focus on key issues of multi-object state estimation for unconstrained multi-camera systems, e.g., data fusion of multiple sensors and data association. Unlike previous work that rely on camera network topology inference, we construct a graph from 2-D observations of all camera pairs without any assumption of network configuration. We apply the shortest path algorithm to the graph to find fused 3-D observation groups. Our approach is thus able to reject false positive reconstructions automatically, and also minimize computational complexity to guarantee feasible data fusion. Particle filters are applied as the 3-D tracker to form tracklets that integrate local features. These tracklets are modeled by a graph and linked into full tracks incorporating global spatial-temporal features. Experiments on the real-world PETS2009 S2/L1 sequence show the accuracy of our approach. Analyses of the different components of our approach provide meaningful insights for object tracking using multiple cameras. Evidence is provided for selecting the best view for a visual sensor network.

Unmanned Aerial System (UAS)-based phenotyping of soybean using multi-sensor data fusion and extreme learning machine

Estimating crop biophysical and biochemical parameters with high accuracy at low-cost is imperative for high-throughput phenotyping in precision agriculture. Although fusion of data from multiple sensors is a common application in remote sensing, less is known on the contribution of low-cost RGB, multispectral and thermal sensors to rapid crop phenotyping. This is due to the fact that (1) simultaneous collection of multi-sensor data using satellites are rare and (2) multi-sensor data collected during a single flight have not been accessible until recent developments in Unmanned Aerial Systems (UASs) and UAS-friendly sensors that allow efficient information fusion. The objective of this study was to evaluate the power of high spatial resolution RGB, multispectral and thermal data fusion to estimate soybean (Glycine max) biochemical parameters including chlorophyll content and nitrogen concentration, and biophysical parameters including Leaf Area Index (LAI), above ground fresh and dry biomass. Multiple low-cost sensors integrated on UASs were used to collect RGB, multispectral, and thermal images throughout the growing season at a site established near Columbia, Missouri, USA. From these images, vegetation indices were extracted, a Crop Surface Model (CSM) was advanced, and a model to extract the vegetation fraction was developed. Then, spectral indices/features were combined to model and predict crop biophysical and biochemical parameters using Partial Least Squares Regression (PLSR), Support Vector Regression (SVR), and Extreme Learning Machine based Regression (ELR) techniques. Results showed that: (1) For biochemical variable estimation, multispectral and thermal data fusion provided the best estimate for nitrogen concentration and chlorophyll (Chl) a content (RMSE of 9.9% and 17.1%, respectively) and RGB color information based indices and multispectral data fusion exhibited the largest RMSE 22.6%; the highest accuracy for Chl a+b content estimation was obtained by fusion of information from all three sensors with an RMSE of 11.6%. (2) Among the plant biophysical variables, LAI was best predicted by RGB and thermal data fusion while multispectral and thermal data fusion was found to be best for biomass estimation. (3) For estimation of the above mentioned plant traits of soybean from multi-sensor data fusion, ELR yields promising results compared to PLSR and SVR in this study. This research indicates that fusion of low-cost multiple sensor data within a machine learning framework can provide relatively accurate estimation of plant traits and provide valuable insight for high spatial precision in agriculture and plant stress assessment.