Sensor Fusion Model Research Articles

Comparing Sensor Fusion and Multimodal Chemometric Models for Monitoring U(VI) in Complex Environments Representative of Irradiated Nuclear Fuel.

Optical sensors and chemometric models were leveraged for the quantification of uranium(VI) (0-100 μg mL-1), europium (0-150 μg mL-1), samarium (0-250 μg mL-1), praseodymium (0-350 μg mL-1), neodymium (0-1000 μg mL-1), and HNO3 (2-4 M) with varying corrosion product (iron, nickel, and chromium) levels using laser fluorescence, Raman scattering, and ultraviolet-visible-near-infrared absorption spectra. In this paper, an efficient approach to developing and evaluating tens of thousands of partial least-squares regression (PLSR) models, built from fused optical spectra or multimodal acquisitions, is discussed. Each PLSR model was optimized with unique preprocessing combinations, and features were selected using genetic algorithm filters. The 7-factor D-optimal design training set contained just 55 samples to minimize the number of samples. The performance of PLSR models was evaluated by using an automated latent variable selection script. PLS1 regression models tailored to each species outperformed a global PLS2 model. PLS1 models built using fused spectra data and a multimodal (i.e., analyzed separately) approach yielded similar information, resulting in percent root-mean-square error of prediction values of 0.9-5.7% for the seven factors. The optical techniques and data processing strategies established in this study allow for the direct analysis of numerous species without measuring luminescence lifetimes or relying on a standard addition approach, making it optimal for near-real-time, in situ measurements. Nuclear reactor modeling helped bound training set conditions and identified elemental ratios of lanthanide fission products to characterize the burnup of irradiated nuclear fuel. Leveraging fluorescence, spectrophotometry, experimental design, and chemometrics can enable the remote quantification and characterization of complex systems with numerous species, monitor system performance, help identify the source of materials, and enable rapid high-throughput experiments in a variety of industrial processes and fundamental studies.

Multimodal sensor fusion models for real-time exercise repetition counting with IMU sensors and respiration data

This study introduces an automated exercise repetition counting model for a variety of exercises. Although external devices such as fitness trackers and smart watches are capable of tracking exercise progress, they are still incapable of recording all pertinent information essential for comprehensive exercise tracking, such as the number of repetitions.To address this limitation, we propose a deep learning model that utilizes multimodal sensor and respiration audio data obtained from wearable devices, specifically smart earbuds that record IMU sensors and collect surrounding sounds. A diverse dataset of sensor data and respiratory audio data are collected by leveraging these smart earbuds during 30 different types of exercises, which enables the study to develop a comprehensive and accurate multimodal deep learning model that can be applied to a wide range of physical activities. Advanced techniques such as data augmentation and time-series interpolation are also employed to address the challenges of working with large and complex datasets.The proposed deep learning model, which incorporates multimodal sensor data and advanced data processing techniques, can be utilized to promote more effective and personalized exercise plans, ultimately leading to improved health outcomes for individuals who seek to maintain an active and healthy lifestyle.

Spectral sensor fusion for prediction of Li and Zr in rocks: Neural network and PLS methods

Integrating information from multiple sensors, known as sensor fusion, is particularly challenging for small datasets where selecting from the plethora of available methods poses a significant challenge in chemometric analysis. This study compares several sensor fusion methods (spanning data-level, feature-level, and decision-level fusion) based on partial least squares (PLS) and convolutional neural network (CNN) models. This study is the first to compare simple sensor fusion methods to the latest multiblock PLS models and parallel-input CNNs, to the best of our knowledge. We demonstrate sensor fusion using a small dataset of 177 rock samples on two prediction tasks: predicting lithium (Li) concentrations and zirconium (Zr) concentrations using three types of spectra, namely X-ray fluorescence (XRF), visible to short-wave infrared (Vis-NIR-SWIR), and laser-induced breakdown spectroscopy (LIBS). The best-performing Li model was PLS using XRF (mean RMSEP: 0.078%) and the best sensor fusion model was ROSA (mean RMSEP: 0.087%). The best Zr model was a high-level fusion of PLS models (mean RMSEP: 0.042%).

Convolutional neural network and sensor fusion for obstacle classification in the context of powered prosthetic leg applications

Powered prostheses for lower extremities are an advancing technology, with recent developments reducing their weight while increasing their autonomy and capabilities. One problem that powered prosthesis users face is the change of gait pattern that is associated with navigating structures such as staircases or ramps. While a powered prosthesis may implement more than one mode of locomotion to address those transitions, a recent area of study is their automatic detection. Previous developments have used different sensor combinations, including wearable cameras, to anticipate the change between level-ground walking and stair climbing. We propose combining information from a camera with signals from an inertial measurement unit, thus creating a system that is less susceptible to false classifications. The principle behind using inertial data is to provide useful information to classify an object as an obstacle only when the user is going to pass or is passing through it. To this end, we modify the architectures of different neural networks that were designed to classify images, extending them to handle inertial information as well. Our results show that the introduction of inertial information increases several performance metrics for each network, effectively reducing misclassifications. All chosen network architectures provide real-time performance, even on embedded hardware. Introducing inertial sensor information increases the Matthews correlation coefficient (MCC) of the neural networks, with gains ranging from 0.85% to 3.49%. The proposed sensor fusion model achieves accuracy over 98%, and an MCC of 0.9137.

Sensor fusion and the application of artificial intelligence to identify tool wear in turning operations

This paper aims to develop an effective sensor fusion model for turning processes for the detection of tool wear. Fusion of sensors’ data combined with novelty detection algorithm and learning vector quantisation (LVQ) neural networks is used to detect tool wear and present diagnostic and prognostic information. To reduce the number of sensors required in the monitoring system and support sensor fusion, the ASPS approach (Automated Sensor and Signal Processing Selection System) is used to select the most appropriate sensors and signal processing methods for the design of the condition monitoring system. The experimental results show that the proposed approach has demonstrated its efficacy in the implementation of an effective solution for the monitoring tool wear in turning. The results prove that the fusion of sensitive sensory characteristic features and the use of AI methods have been successful for the detection and prediction of the tool wear in turning processes and show the capability of the proposed approach to reduce the complexity of the design of condition monitoring systems and the development of a sensor fusion system using a self-learning method.

Sensor fusion model for weld quality monitoring in friction stir welding process using machine learning technique

Weld quality monitoring in welding processes is not new and many research works have been carried in this area. Sensor information based methodologies have been developed to an appreciable extent and many found its recognition in many industrial applications. The current study will present an approach of monitoring weld quality in friction stir welding process with sensor fusion technique. Real-time signals acquired using force and tool rotational speed sensors are fused at two levels. First level of fusion is performed at signal level and next level of fusion is performed at feature level. The results obtained from sensor fusion approach are compared with the results obtained with monitoring using individual sensors. Support vector machine based regression models are developed for the joint quality prediction. It is observed that the machine learning model developed with fused signal features results in prediction accuracy of 99.16% and the same obtained with only process parameters results in prediction accuracy of 95.34%. The proposed research work can be used as the first line of model for estimation of joint qualities in an industrial environment.

CNN-LSTM-Based Late Sensor Fusion for Human Activity Recognition in Big Data Networks

The technological advancement in sensor technology and pervasive computing has brought smart devices into our daily life. Due to the continuous connectivity of the internet with our everyday devices, researchers can deploy IoT sensors to health care and other applications, such as human activity recognition. Most of the state-of-the-art sensor-based human activity recognition systems can detect basic activities (such as standing, sitting, and walking), but they cannot accurately distinguish similar activities (ascending stairs or descending stairs). Such systems are not efficient for critical healthcare applications having complex activity sets. This paper proposes two sensor fusion approaches, i.e., position-based early and late sensor fusion using convolutional neural network (CNN) and convolutional long-short-term memory (CNN-LSTM). The performance of our proposed models is evaluated on two publicly available datasets. We also evaluated the effect of different normalization techniques on recognition accuracy. Our results show that the CNN-LSTM-based late sensor fusion model also improves the recognition accuracy of similar activities.

Non-Invasive Human-Machine Interface (HMI) Systems With Hybrid On-Body Sensors for Controlling Upper-Limb Prosthesis: A Review

In this work, we present a systematic review on non-invasive HMIs employing hybrid wearable sensor modalities for recognition of upper limb intentions. Different combinations of the sensors are investigated. As sEMG is dominant in the applications of externally powered prosthetic hands, it is involved in most hybrid sensor combinations. The combined use of sEMG and IMU is most studied in the literature as IMU is easy to be integrated. Though limited, the investigation on other hybrid modalities has been drawing more and more research attention and efforts, especially those with FMG and NIRS. For all the reported hybrid sensors, it is verified that this strategy can enrich the information of user intention and help the pattern recognition and/or intensity regulation of robotic hand/arm prosthesis. Though it is the trend, the development of these hybrid-sensor-based HMIs are still at the preliminary stage. More dedicated sensor fusion models and system architectures as well as new hybrid features and algorithms need to be developed to make the best use of each sensing modality’s strength to achieve robust and stable user intention recognition, which is essential for the progress and user acceptance of upper limb prosthesis.

Identifying energy savings opportunities in vacant commercial buildings using a semi-supervised sensor fusion model

Recent events have forced building managers to examine energy use during vacant periods and revealed miscellaneous electrical loads (MELs) as an opportunity for savings. This paper addresses a key step in unlocking these savings, specifically the reliable identification of when a building is vacant. A Vacancy Inference Engine (VIE), using sensor fusion, was developed to identify vacant periods based on outputs from common sensors, historical building vacancy patterns, and expert knowledge. The VIE calculates the confidence that a building is vacant, allowing building managers to balance the capture of energy savings with the possibility of complaints due to powering down MELs. The VIE has the advantage over logistic regression and other models in that it does not require a full set of ground truth for the training process.The VIE successfully predicted vacancy in an office building using input data streams of instantaneous electricity demand, indoor carbon dioxide concentrations, and the number of active Wi-Fi connections. The VIE’s ability to predict vacancy was compared to that of logistic regression using a metric based on the Complaint Opportunity Rate and found to be nearly identical (0.94 versus 0.95, respectively).

Camera, LiDAR, and Radar Sensor Fusion Based on Bayesian Neural Network (CLR-BNN)

<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Perception</i> in automated vehicles (AV) is the main factor in achieving safe driving. In this perception task, multi-object detection (MOD) in diverse driving situations is the main challenge. Our recent survey [Ravindran <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> (2021)] shows the limitations of deep neural networks (DNN) in predicting the uncertainties of object detection in MOD. This research proposed a camera, LiDAR and RADAR sensor fusion Bayesian neural network (CLR-BNN) to improve detection accuracy and reduce uncertainties in diverse driving situations using these three primary sensing devices. The experiments were performed using the nuScence dataset with incorporation of various noises. The CLR-BNN performed better than its deterministic sensor fusion model (CLR-DNN) in terms of mAP. The CLR-BNN also showed improvement in categorical and bounding box location uncertainty using sensor fusion in diverse driving conditions. The uncertainty predictions of the CLR-BNN were validated using the calibration curve and other performance metrics.

Pedogenic-weathering evolution and soil discrimination by sensor fusion combined with machine-learning-based spectral modeling

Loess deposits are important records of the evolution of the soil environment and pedogenic weathering. Changes in pedogenic weathering conditions through space and time, as well as discrimination/sourcing of loess-derived soils, are important scientific issues in the soil (paleosol) community. Here, 502 soil samples from four loess chronosequences representing different climatic zones of China were investigated using portable X-ray fluorescence (PXRF) and visible to near-infrared reflectance (VNIR) in combination with previously published mineralogic and magnetic datasets. A spectral modeling approach was employed to discriminate loess-derived soils from different regions of China. We developed a three-dimensional model to fingerprint loess-derived soils using principal component analysis (PCA), showing that soils accumulating under varying climatic conditions were effectively discriminated with sensor fusion data. Six key soil mineralogic and magnetic attributes serving as pedogenic-weathering proxies were analyzed and predicted with conventional methods and chemometric models. Predictive models were constructed with machine learning methods, including partial least squares regression (PLSR), random forest (RF), and Cubist algorithms. The results indicate that the Cubist algorithm works better than PLSR and RF in predicting pedogenic-weathering proxies. The cross-validation results indicate that, although models derived from single sensors (i.e., PXRF or VNIR) work well in predicting pedogenic-weathering proxies, the sensor fusion approach is superior with regard to accuracy and robustness of results in most cases. We suggest that the combined elemental and secondary-mineral information provided by the fused PXRF-VNIR datasets can yield high-accuracy models in soil (paleosol) investigations. The sensor fusion models reveal that pedogenic processes in the loess chronosequences are diversified both spatially and temporally in different climate zones of China. Our results suggest that spectral modeling can be an alternative to geochemical, mineralogic and magnetic pedogenic-weathering proxies, and that it has great potential for investigating soil-forming conditions and pedogenic-weathering evolution, especially when large-area and/or high-resolution analysis is required.

Learning When Agents Can Talk to Drivers Using the INAGT Dataset and Multisensor Fusion

This paper examines sensor fusion techniques for modeling opportunities for proactive speech-based in-car interfaces. We leverage the Is Now a Good Time (INAGT) dataset, which consists of automotive, physiological, and visual data collected from drivers who self-annotated responses to the question "Is now a good time?," indicating the opportunity to receive non-driving information during a 50-minute drive. We augment this original driver-annotated data with third-party annotations of perceived safety, in order to explore potential driver overconfidence. We show that fusing automotive, physiological, and visual data allows us to predict driver labels of availability, achieving an 0.874 F1-score by extracting statistically relevant features and training with our proposed deep neural network, PazNet. Using the same data and network, we achieve an 0.891 F1-score for predicting third-party labeled safe moments. We train these models to avoid false positives---determinations that it is a good time to interrupt when it is not---since false positives may cause driver distraction or service deactivation by the driver. Our analyses show that conservative models still leave many moments for interaction and show that most inopportune moments are short. This work lays a foundation for using sensor fusion models to predict when proactive speech systems should engage with drivers.

Camera-LiDAR Multi-Level Sensor Fusion for Target Detection at the Network Edge.

There have been significant advances regarding target detection in the autonomous vehicle context. To develop more robust systems that can overcome weather hazards as well as sensor problems, the sensor fusion approach is taking the lead in this context. Laser Imaging Detection and Ranging (LiDAR) and camera sensors are two of the most used sensors for this task since they can accurately provide important features such as target´s depth and shape. However, most of the current state-of-the-art target detection algorithms for autonomous cars do not take into consideration the hardware limitations of the vehicle such as the reduced computing power in comparison with Cloud servers as well as the reduced latency. In this work, we propose Edge Computing Tensor Processing Unit (TPU) devices as hardware support due to their computing capabilities for machine learning algorithms as well as their reduced power consumption. We developed an accurate and small target detection model for these devices. Our proposed Multi-Level Sensor Fusion model has been optimized for the network edge, specifically for the Google Coral TPU. As a result, high accuracy results are obtained while reducing the memory consumption as well as the latency of the system using the challenging KITTI dataset.

Applying a ToF/IMU-Based Multi-Sensor Fusion Architecture in Pedestrian Indoor Navigation Methods.

The advancement of indoor Inertial Navigation Systems (INS) based on the low-cost Inertial Measurement Units (IMU) has been long reviewed in the field of pedestrian localization. There are various sources of error in these systems which lead to unstable and unreliable positioning results, especially in long term performances. These inaccuracies are usually caused by imperfect system modeling, inappropriate sensor fusion models, heading drift, biases of IMUs, and calibration methods. This article addresses the issues surrounding unreliability of the low-cost Micro-Electro-Mechanical System (MEMS)-based pedestrian INS. We designed a novel multi-sensor fusion method based on a Time of Flight (ToF) distance sensor and dual chest- and foot-mounted IMUs, aided by an online calibration technique. An Extended Kalman Filter (EKF) is accounted for estimating the attitude, position, and velocity errors, as well as estimation of IMU biases. A fusion architecture is derived to provide a consistent velocity measurement by operative contribution of ToF distance sensor and foot mounted IMU. In this method, the measurements of the ToF distance sensor are used for the time-steps in which the Zero Velocity Update (ZUPT) measurements are not active. In parallel, the chest mounted IMU is accounted for attitude estimation of the pedestrian’s chest. As well, by designing a novel corridor detection filter, the heading drift is restricted in each straightway. Compared to the common INS method, developed system proves promising and resilient results in two-dimensional corridor spaces for durations of up to 11 min. Finally, the results of our experiments showed the position RMS error of less than 3 m and final-point error of less than 5 m.

Sensor fusion and random forest modeling for identifying frozen and green wood during lumber manufacturing

The sawmilling industry faces challenges in winter due to cutting frozen logs. This study proposes a methodology using sensor fusion and random forest modeling to differentiate frozen and green wood using sensory features acquired during the wood sawing. The process was monitored with power, sound, vibration and acoustic emission (AE) sensors, and different sensory features were extracted to develop random forest models. The vibration sensor showed superior monitoring performance and combined with sound and AE sensors, maximum accuracy (93%) was achieved. The model could accurately identify frozen wood in lumber manufacturing, which can be used to adjust the cutting process.

3D car-detection based on a Mobile Deep Sensor Fusion Model and real-scene applications.

Unmanned vehicles need to make a comprehensive perception of the surrounding environmental information during driving. Perception of automotive information is of significance. In the field of automotive perception, the sterevision of car-detection plays a vital role and sterevision can calculate the length, width, and height of a car, making the car more specific. However, under the existing technology, it is impossible to obtain accurate detection in a complex environment by relying on a single sensor. Therefore, it is particularly important to study the complex sensing technology based on multi-sensor fusion. Recently, with the development of deep learning in the field of vision, a mobile sensor-fusion method based on deep learning is proposed and applied in this paper——Mobile Deep Sensor Fusion Model (MDSFM). The content of this article is as follows. It does a data processing that projects 3D data to 2D data, which can form a dataset suitable for the model, thereby training data more efficiently. In the modules of LiDAR, it uses a revised squeezeNet structure to lighten the model and reduce parameters. In the modules of cameras, it uses the improved design of detecting module in R-CNN with a Mobile Spatial Attention Module (MSAM). In the fused part, it uses a dual-view deep fusing structure. And then it selects images from the KITTI’s datasets for validation to test this model. Compared with other recognized methods, it shows that our model has a fairly good performance. Finally, it implements a ROS program on the experimental car and our model is in good condition. The result shows that it can improve performance of detecting easy cars significantly through MDSFM. It increases the quality of the detected data and improves the generalized ability of car-detection model. It improves contextual relevance and preserves background information. It remains stable in driverless environments. It is applied in the realistic scenario and proves that the model has a good practical value.

Wi-Fi-Inertial Indoor Pose Estimation for Microaerial Vehicles

This article presents an indoor pose estimation system for microaerial vehicles (MAVs) with a single Wi-Fi access point. Conventional approaches based on computer vision are limited by illumination conditions and environmental texture. Our system is free of visual limitations and instantly deployable, working upon existing Wi-Fi infrastructure without any deployment cost. Our system consists of two coupled modules. First, we propose an angle-of-arrival (AoA) estimation algorithm to estimate MAV attitudes and disentangle the AoA for positioning. Second, we formulate a Wi-Fi-inertial sensor fusion model that fuses the AoA and the odometry measured by inertial sensors to optimize MAV poses. Considering the practicality of MAVs, our system is designed to be real-time and initialization-free for the need of agile flight in unknown environments. The indoor experiments show that our system achieves the accuracy of pose estimation with the position error of 61.7 cm and the attitude error of $0.92^{\circ }$ .

Proximity-Based Sensor Fusion of Depth Cameras and Isotropic Rad-Detectors

Finding and tracking radioactive sources have numerous security applications in civilian energy installations, military facilities, and ports of entry. The price of radiological sensors varies proportionally to size and imaging characteristics such as angular resolution, and the cheapest devices are nearly isotropic–i.e., they integrate radiation from a sphere of directions centered at the sensor. While many radiation sensors have high aspect ratios or odd shapes, the sensors used here are right cylinders, with a near identical directional efficiency such that for analysis purposes, other aspects, such as counting statistics, would make nonisotropy of the sensor negligible. In this article, we propose a simple and robust way to integrate measurements from both isotropic radiological sensors and depth sensors, whose reliability and resolution benefit from recent advances in computer vision and imaging. Our key idea is to convert all sensor measurements into proximity signals based on radial distance variations over time. Based on this sensor fusion model, we show that for moving radiological sources even a simple Kalman filter can trade-off the complementary strength of high-resolution depth sensors and isotropic radiological sensors. We show novel results with a LIDAR sensor and a thermal stereo pair, and demonstrate applications such as tracking and rendering non-line-of-sight imagery behind obstacles and detecting multiple radiological sources in the same scene.

A Survey on Anomalous Behavior Detection for Elderly Care Using Dense-Sensing Networks

Facing the gradual ageing society, elderly people living independently are in need of serious attention. In order to assist them to live in a safer environment, the increasing cost of nursing care and the shortage of health-care workers urges the demand of home-based assisted living in recent times. Therefore, home-based health-care has become an active research domain, particularly the abnormal activities detection involving information and communications technologies. This survey paper highlights this kind of technologies that exist for human anomalous behavior detection. It also reviews and discusses the current research trends, their outcomes and effects in elderly care. Our study is mainly focused on dense sensing network based activities and anomaly detection, which are robust to environment change, non-intrusive, user-friendly in the sense that do not require the occupant to wear any devices. From our study, we observe that employing sensor fusion techniques could significantly increases the efficiency of dense sensing network. In addition, sensor fusion models ensure a high level of robustness and effectiveness compared to the traditional methods.

An ISE-based On-Site Soil Nitrate Nitrogen Detection System.

Soil nitrate–nitrogen (NO3−-N) is one of the primary factors used to control nitrogen topdressing application during the crop growth period. The ion-selective electrode (ISE) is a promising method for rapid lower-cost in-field detection. Due to the simplification of sample preparation, the accuracy and stability of ISE-based in-field detection is doubted. In this paper, a self-designed prototype system for on-site soil NO3−-N detection was developed. The procedure of spinning centrifugation was used to avoid interference from soil slurry suspension. A modified Nernstian prediction model was quantitatively characterized with outputs from both the ISE and the soil moisture sensor. The measurement accuracy of the sensor fusion model was comparable with the laboratory ISE detections with standard sample pretreatment. Compared with the standard spectrometric method, the average absolute error (AE) and root-mean-square error (RMSE) were found to be less than 4.7 and 6.1 mg/L, respectively. The on-site soil testing efficiency was 4–5 min/sample, which reduced the operation time by 60% compared with manual sample preparation. The on-site soil NO3−-N status was dynamically monitored for 42 consecutive days. The declining peak of NO3−-N was observed. In all, the designed ISE-based detection system demonstrated a promising capability for the dynamic on-site monitoring of soil macronutrients.