Sensor Fusion Approach Research Articles

Toward User-Independent Emotion Recognition Using Physiological Signals

Many techniques have been developed to improve the flexibility and the fit of detection models beyond user-dependent models, yet detection tasks continue to be complex and challenging. For emotion, which is known to be highly user-dependent, improvements to the emotion learning algorithm can greatly boost predictive power. Our aim is to improve the accuracy rate of the classifier using peripheral physiological signals. Here, we present a hybrid sensor fusion approach based on a stacking model that allows for data from multiple sensors and emotion models to be jointly embedded within a user-independent model. WMD-DTW, which is a weighted multi-dimensional DTW, and the k-nearest neighbor’s algorithm k-NN are used to classify the emotions as a base model. The ensemble methods were used to learn a high-level classifier on top of the two base models. We applied a meta-learning approach to the data set and showed that the ensemble approach outperforms any individual method. We also compared the results using two data sets. Our proposed system achieved an overall accuracies of 65.6% for all users for the E4-data set and 94.0% and 93.6% for recognizing valence and arousal emotions, respectively, using the MAHNOB data set.

Hierarchical Sensor Fusion for Micro-Gesture Recognition With Pressure Sensor Array and Radar

This paper presents a hierarchical sensor fusion approach for human micro-gesture recognition by combining an Ultra Wide Band (UWB) Doppler radar and wearable pressure sensors. First, the wrist-worn pressure sensor array (PSA) and Doppler radar are used to respectively identify static and dynamic gestures through a Quadratic-kernel SVM (Support Vector Machine) classifier. Then, a robust wrapper method is applied on the features from both sensors to search the optimal combination. Subsequently, two hierarchical approaches where one sensor acts as “enhancer” of the other are explored. In the first case, scores from Doppler radar related to the confidence level of its classifier and the prediction label corresponding to the posterior probabilities are utilized to maximize the static hand gestures classification performance by hierarchical combination with PSA data. In the second case, the PSA acts as an “enhancer” for radar to improve the dynamic gesture recognition. In this regard, different weights of the “enhancer” sensor in the fusion process have been evaluated and compared in terms of classification accuracy. A realistic cross-validation method is chosen to test one unknown participant with the model trained by data from others, demonstrating that this hierarchical fusion approach for static and dynamic gestures yields approximately 15% improvement in classification accuracy in the best cases.

How Might Wet Retroreflective Pavement Markings Enable More Robust Machine Vision?

This study is a preliminary investigation of the effects of levels of wet retroreflectivity of pavement markings on factors that determine robust feature detection in machine vision and light detection and ranging (LiDAR) systems in continuously wet road conditions. Luminance and Weber contrast of a range of pavement markings were characterized as functions of wet retroreflectivity and distance based on calibrated charge-coupled device (CCD) camera measurements. Both were found to trend with wet retroflectivity over the range of distances considered in this study. Artifacts arising from glare sources in wet conditions and their intensities relative to pavement markings of different wet retroreflectivity levels were demonstrated. Image data suggests that markings with high wet retroreflectivity may help to mitigate identification of these artifacts as false positives in lane awareness/lane detection algorithms. As LiDAR presents a viable sensor fusion approach to identifying and avoiding these false positives and artifacts in both nighttime wet and daytime wet road conditions, LiDAR return was characterized on pavement markings comprising both optics designed only for dry retroreflectivity and optics designed to be retroreflective in both dry and wet conditions. Preliminary results suggest that for common pavement marking constructions based on exposed beaded optics that might be completely immersed by a rainstorm or puddling, incorporation of high index (n~2.4) wet retroreflective beaded optics is likely to be advantageous to both visible machine vision systems and LiDAR for detection of those retroreflective markings in both night and day.

Beyond position-awareness—Extending a self-adaptive fall detection system

Ambient Assisted Living using mobile device sensors is an active area of research in pervasive computing. Multiple approaches have shown that wearable sensors perform very well and distinguish falls reliably from Activities of Daily Living. However, these systems are tested in a controlled environment and are optimized for a given set of sensor types, sensor positions, and subjects. We propose a self-adaptive pervasive fall detection approach that is robust to the heterogeneity of real life situations. Using the data of four publicly available datasets, we show that our system is not only robust regarding the different dimensions of heterogeneity, but also adapts autonomously to spontaneous changes in the sensor’s position at runtime. In this paper, we extend our self-adaptive fall detection system with (i) additional algorithms for fall detection, (ii) an approach for cross-positional sensor fusion, (iii) a fall detection approach that relies on outlier detection, and (iv) a smart fall alert. Additionally, we present implementation and evaluation of these extensions.

A multi-rate sensor fusion approach using information filters for estimating aero-engine performance degradation

Gas-path performance estimation plays an important role in aero-engine health management, and Kalman Filter (KF) is a well-known technique to estimate performance degradation. In previous studies, it is assumed that different kinds of sensors are with the same sampling rate, and they are used for state estimation by the KF simultaneously. However, it is hard to achieve state estimation using various kinds of sensor measurements at the same sampling rate due to a complex network and physical characteristic differences between sensors, especially in an advanced multi-sensor architecture. For this purpose, a multi-rate sensor fusion using the information filtering approach is proposed based on the square-root cubature rule, which is called Multi-rate Square-root Cubature Information Filter (MSCIF) to track engine performance degradation. Soft measurement synchronization of the MSCIF is designed to provide a sensor fusion condition for multiple sampling rates of measurement, and a fault sensor is isolated by maximum likelihood validation before state estimation. The contribution of this paper is to supply a novel multi-rate information filter approach for sensor fault tolerant health estimation of an aero-engine in a multi-sensor system. Tests are conducted for aero-engine performance degradation estimation with multiple sampling rates of sensor measurement on both digital simulation and semi-physical experiment. Experimental results illustrate the superiority of the proposed algorithm in terms of degradation estimation accuracy and robustness to sensor failure in a multi-sensor system.

Analytical modeling of tool health monitoring system using multiple sensor data fusion approach in hard machining

In recent years, a lot of work has been carried out on advance materials, especially hard metals and alloys, which is a vital requirement in many manufacturing industries. Nowadays steel with hardness beyond 30 HRC is employed in aircraft, hydraulics, car part manufacturing areas, etc. In the machining process, as the hardness of work material increases, tool health prone to deteriorate. Therefore, for healthier machinability requirements in machining of hard metals, the online tool health monitoring systems (THMS) are being developed using different feedback techniques. This study mainly concentrates on the advancement of online THMS using multiple sensors. Force, tool vibration, and surface roughness signals were recorded while machining EN24 hardened steel using coated carbide insert on CNC lathe. A novel analytical model of sensor data fusion has been presented for better understanding of sensors and their interaction. A comparative assessment of sensor fusion approach has been investigated. Experimental findings have successfully shown that the results obtained using a fusion function (U), validates better confirmability over single sensor-based approach.

Monitoring Chemical Processes Using Judicious Fusion of Multi-Rate Sensor Data.

With the emergence of Industry 4.0, also known as the fourth industrial revolution, an increasing number of hardware and software sensors have been implemented in chemical production processes for monitoring key variables related to product quality and process safety. The accuracy of individual sensors can be easily impaired by a variety of factors. To improve process monitoring accuracy and reliability, a sensor fusion scheme based on Bayesian inference is proposed. The proposed method is capable of combining multi-rate sensor data and eliminating the spurious signals. The efficacy of the method has been verified using a process implemented at the Dow Chemical Company. The sensor fusion approach has improved the process monitoring reliability, quantified by the rates of correctly identified impurity alarms, as compared to the case of using an individual sensor.

Multimodal Sensor Fusion for Robust Obstacle Detection and Classification in the Maritime RobotX Challenge

This paper describes a novel probabilistic sensor fusion framework aimed at improving obstacle detection accuracy and classification of various targets experienced in the Maritime RobotX Challenge. In both the 2014 and 2016 Maritime RobotX Challenges, it was found that detecting obstacles using LIDAR only and classifying obstacles using vision only can be challenged by environmental conditions, such as glare from the sun or by objects such as the spherical black buoys from the obstacle field that disperses LIDAR rays. In this paper, a new multimodal sensor fusion approach is proposed that combines data streams from perception sensors, such as LIDAR, RADAR, and cameras, to improve the robustness of detection and classification performance over a single sensor method. Using data collected from both the 2014 and 2016 Maritime RobotX Challenges, an evaluation of the perception framework is provided. The proposed detection and classification framework is now being transferred to the queensland university of technology (QUT) autonomous surface vehicle to improve overall mapping accuracy and task execution.

A Sensor Fusion Approach to Indoor Human Localization Based on Environmental and Wearable Sensors

The goal of this paper is to localize a resident in indoor environments by using motion information from distributed environmental sensors and body activity information from wearable sensors. The passive infrared sensor nodes distributed in a home provide binary information about human motion in their field of views, while the wearable inertial measurement unit sensor node collects motion data that can be used in body activity recognition, walking velocity, and heading estimation. Basic human activities such as sitting, sleeping, standing, and walking are recognized. We proposed a particle filter-based sensor fusion algorithm that takes advantage of the human location/activity correlation in indoor environments to increase the localization accuracy. Experiments were conducted in a mock apartment testbed. We used the ground truth data obtained from a motion capture system to evaluate the results. Note to Practitioners —This paper is motivated by a goal to realize various home automation applications through human localization. For example, with the knowledge of human location, lighting, and air conditioning systems can be more efficiently managed, or a home service robot can provide better service to the resident. There currently lacks an accurate human localization method in indoor environments. The theoretical framework proposed in this paper aims to combine simple environmental sensors (passive infrared sensors) and a single wearable motion sensor to accurately track human locations. This method reduces the intrusiveness to the human, while keeping the cost of the infrastructure low. The theoretical framework has been tested and evaluated in a mock apartment environment as a proof of concept. For deployment into real homes, we need to conduct more realistic tests and thorough evaluation of the system. It is also necessary to reduce the size and weight of the wearable motion sensor and simplify the installation of the environmental sensors.

Automated Rutting Measurement Using an Inexpensive RGB-D Sensor Fusion Approach

Pavement management systems have become more crucial through the development of road networks. Data collection is at the core of any pavement management system. Rutting is a critical defect on pavements that can lead to hydroplaning and accidents. In this study, a system is proposed to quantify the rut depths autonomously by utilizing an inexpensive red green blue—depth (RGB-D) sensor. To this end, a cart is designed, built, and used to mount an array of RGB-D sensors for data collection purposes. Furthermore, a procedure is proposed for autonomous stitching of three-dimensional (3D) point clouds obtained from multiple RGB-D sensors to generate the pavement profile of a lane. Next, transverse profiles are extracted and the rut depths are estimated based on the simulated straight-edge method. Several field experiments are conducted to evaluate the capabilities as well as the limitations of the proposed system. Results show that the proposed approach is capable of estimating different severity levels of rutting accurately, particularly moderate and high severity levels with relative errors of 15% and 9%, respectively. Additionally, the repeatability of the proposed approach is evaluated and confirmed.

Single-photon 3D imaging with deep sensor fusion

Sensors which capture 3D scene information provide useful data for tasks in vehicle navigation, gesture recognition, human pose estimation, and geometric reconstruction. Active illumination time-of-flight sensors in particular have become widely used to estimate a 3D representation of a scene. However, the maximum range, density of acquired spatial samples, and overall acquisition time of these sensors is fundamentally limited by the minimum signal required to estimate depth reliably. In this paper, we propose a data-driven method for photon-efficient 3D imaging which leverages sensor fusion and computational reconstruction to rapidly and robustly estimate a dense depth map from low photon counts. Our sensor fusion approach uses measurements of single photon arrival times from a low-resolution single-photon detector array and an intensity image from a conventional high-resolution camera. Using a multi-scale deep convolutional network, we jointly process the raw measurements from both sensors and output a high-resolution depth map. To demonstrate the efficacy of our approach, we implement a hardware prototype and show results using captured data. At low signal-to-background levels, our depth reconstruction algorithm with sensor fusion outperforms other methods for depth estimation from noisy measurements of photon arrival times.

A sensor fusion approach for inertial sensors based 3D kinematics and pathological gait assessments: toward an adaptive control of stimulation in post-stroke subjects.

Pathological gait assessment and assistive control based on functional electrical stimulation (FES) in post-stroke individuals, brings out a common need to robustly quantify kinematics facing multiple constraints. This study proposes a novel approach using inertial sensors to compute dorsiflexion angles and spatio-temporal parameters, in order to be later used as inputs for online close-loop control of FES. 26 post-stroke subjects were asked to walk on a pressure mat equipped with inertial measurement units (IMU) and passive reflective markers. A total of 930 strides were individually analyzed and results between IMU-based algorithms and reference systems compared. Mean absolute (MA) errors of dorsiflexion angles were found to be less than 4°, while stride lengths were robustly segmented and estimated with a MA error less than 10 cm. These results open new doors to rehabilitation using adaptive FES closed-loop control strategies in "foot drop" syndrome correction.

Monitoring Sucking Abilities in Newborns: Design and Validation on Adult Of a Wearable System for Non-Invasive Deglutition Detection.

Nutritive Sucking (NS) is one of the earliest motor activity performed by infants, strictly related to both neurological and motor development of newborns. The main components of NS are sucking, respiration and deglutition. Despite its recognized importance, current clinical practice lacks quantitative tools for the assessment of NS. This work aims to identify a non-invasive objective method to assess deglutition. In details, we propose a new sensor fusion approach to merge both inertial and acoustic data in order to estimate deglutition time. The algorithm uses two classification criteria: one is based on signal intensity thresholding and the other on the evaluation of Waveform Dimension Trajectory (WDT). Our preliminary results on 9 healthy adult volunteers show that the sensor fusion of audio and IMU signals provides a high precision (0.93) and a good recall (0.72). Moreover, the algorithm has a good accuracy (0.84) and high specificity (0.95).

Fault diagnostics of wind turbine electric pitch systems using sensor fusion approach

Failures in wind turbine pitch systems can cause significant outages in offshore wind turbines due to the finite weather windows for maintenance. In a complex system such as the pitch system, a fault in the gearbox can contaminate the motor current signals and result in a misdiagnosis. This paper investigates a sensor fusion technique to reliably diagnose faults in pitch motors and multistage planetary gearboxes in pitch drives. A support vector machine classifier is used for fault diagnosis based on features extracted from motor currents and gearbox vibration signals. The approach is validated with three commonly occurring pitch drive faults, namely, the stator turns fault in the pitch motor, input shaft bearing and planet gear fault in the planetary gearbox. The developed diagnostic method is validated with artificially seeded faults in a laboratory setup of a scaled pitch drive.

Infrared–ultrasonic sensor fusion for support vector machine–based fall detection

This article presents an infrared–ultrasonic sensor fusion approach for support vector machine–based fall detection, often required by elderly healthcare. Its detection algorithms and performance evaluation are detailed. The location, size, and temperature profile of the user can be estimated based on a novel sensory fusion algorithm. Different feature sets of the support vector machine–based machine learning algorithm are analyzed and their impact on fall detection accuracy is evaluated and compared empirically. Experiments study three non-fall activities, standing, sitting, and stooping, and two fall actions, forward falling and sideway falling, to simulate daily activities of the elderly. Fall detection accuracy studies are performed based on discretely and continuously (closer to reality) recorded experimental data, respectively. For the discrete data recording, an average accuracy of 92.2% is achieved when the stand-alone Grid-EYE is used and the accuracy is increased to 96.7% when sensor fusion is used. For the continuous data recording (180 training sets, 60 test sets at each distance), an average accuracy less than 70.0% is achieved when the stand-alone Grid-EYE is used and the accuracy is increased to around 90.3% after sensor fusion. New features will be explored in the next step to further increase detection accuracy.

A sensor fusion approach for drowsiness detection in wearable ultra-low-power systems

Drowsiness detection mechanisms have been extensively studied in the last years since they are one of the prevalent causes of accidents within the mining, driving and industrial activities. Many research efforts were done to quantify the drowsiness levels using behavioral analyses based on camera eye tracking systems as well as by analyzing physiological features contained in EEG signals. Detection systems typically use specific drowsiness indicators from only one of these methods, leaving a risk of missed detection since not all the population presents same symptoms of drowsiness. Thus, multi-feature systems are preferable even though most of the current State-of-the-Art (SoA) solutions are based on power-hungry platforms and they have meager chance to be used in embedded wearable applications with long battery lifetime. This work presents a drowsiness detection scheme fusing behavioral information coming from user motion through an IMU sensor and physiological information coming from brain activity through a single EEG electrode. The solution is implemented and tested on a low power programmable platform based on an ARM Cortex-M4 microcontroller, resulting in a wearable device capable to detect 5 different levels of drowsiness with an average accuracy of 95.2% and a battery life of 6 hours, using a 200 mAh battery. We also study the energy optimization achievable by accelerating the sensor fusion-based drowsiness detector on a parallel ultra-low power (PULP) platform. Results show that the use of PULP as efficient processing platform provides an energy improvement of 63x with respect to a solution based on a commercial microcontroller. This may extend the battery life of the complete system up to 46 h with a 7x improvement, paving the way for a completely wearable, always-on system.

Accuracy of angular displacements and velocities from inertial-based inclinometers

The objective of this study was to evaluate the accuracy of various sensor fusion algorithms for measuring upper arm elevation relative to gravity (i.e., angular displacement and velocity summary measures) across different motion speeds. Thirteen participants completed a cyclic, short duration, arm-intensive work task that involved transfering wooden dowels at three work rates (slow, medium, fast). Angular displacement and velocity measurements of upper arm elevation were simultaneously measured using an inertial measurement unit (IMU) and an optical motion capture (OMC) system. Results indicated that IMU-based inclinometer solutions can reduce root-mean-square errors in comparison to accelerometer-based inclination estimates by as much as 87%, depending on the work rate and sensor fusion approach applied. The findings suggest that IMU-based inclinometers can substantially improve inclinometer accuracy in comparison to traditional accelerometer-based inclinometers. Ergonomists may use the non-proprietary sensor fusion algorithms provided here to more accurately estimate upper arm elevation.

MR-based respiratory and cardiac motion correction for PET imaging.

To develop a motion correction for Positron-Emission-Tomography (PET) using simultaneously acquired magnetic-resonance (MR) images within 90 s. A 90 s MR acquisition allows the generation of a cardiac and respiratory motion model of the body trunk. Thereafter, further diagnostic MR sequences can be recorded during the PET examination without any limitation. To provide full PET scan time coverage, a sensor fusion approach maps external motion signals (respiratory belt, ECG-derived respiration signal) to a complete surrogate signal on which the retrospective data binning is performed. A joint Compressed Sensing reconstruction and motion estimation of the subsampled data provides motion-resolved MR images (respiratory + cardiac). A 1-POINT DIXON method is applied to these MR images to derive a motion-resolved attenuation map. The motion model and the attenuation map are fed to the Customizable and Advanced Software for Tomographic Reconstruction (CASToR) PET reconstruction system in which the motion correction is incorporated. All reconstruction steps are performed online on the scanner via Gadgetron to provide a clinically feasible setup for improved general applicability. The method was evaluated on 36 patients with suspected liver or lung metastasis in terms of lesion quantification (SUVmax, SNR, contrast), delineation (FWHM, slope steepness) and diagnostic confidence level (3-point Likert-scale). A motion correction could be conducted for all patients, however, only in 30 patients moving lesions could be observed. For the examined 134 malignant lesions, an average improvement in lesion quantification of 22%, delineation of 64% and diagnostic confidence level of 23% was achieved. The proposed method provides a clinically feasible setup for respiratory and cardiac motion correction of PET data by simultaneous short-term MRI. The acquisition sequence and all reconstruction steps are publicly available to foster multi-center studies and various motion correction scenarios.

Robust Indoor Mobile Localization with a Semantic Augmented Route Network Graph

In recent years, using smartphones to determine pedestrian locations in indoor environments is an extensively promising technique for improving context-aware applications. However, the applicability and accuracy of the conventional approaches are still limited due to infrastructure-dependence, and there is seldom consideration of the semantic information inherently existing in maps. In this paper, a semantically-constrained low-complexity sensor fusion approach is proposed for the estimation of the user trajectory within the framework of the smartphone-based indoor pedestrian localization, which takes into account the semantic information of indoor space and its compatibility with user motions. The user trajectory is established by pedestrian dead reckoning (PDR) from the mobile inertial sensors, in which the proposed semantic augmented route network graph with adaptive edge length is utilized to provide semantic constraint for the trajectory calibration using a particle filter algorithm. The merit of the proposed method is that it not only exploits the knowledge of the indoor space topology, but also exhausts the rich semantic information and the user motion in a specific indoor space for PDR accumulation error elimination, and can extend the applicability for diverse pedestrian step length modes. Two experiments are conducted in the real indoor environment to verify of the proposed approach. The results confirmed that the proposed method can achieve highly acceptable pedestrian localization results using only the accelerometer and gyroscope embedded in the phones, while maintaining an enhanced accuracy of 1.23 m, with the indoor semantic information attached to each pedestrian’s motion.

On the performance of surface electromyography-based onset detection methods with real data in assistive technologies

Assistive Technologies are used to increase the autonomy of people with motor disabilities by enhancing their functional capabilities. Surface Electromyography sensors have been explored in the scope of Assistive Technologies as an input modality, to provide greater control and flexibility. In this case, triggering signals are dependent on the detection of the moment when the user performs a voluntary muscular contraction. In the literature, various methods to determine this onset have been studied, but mainly for the non-disabled population and may not be designed to deal with the low signal-to-noise ratio, motion artifacts and spasms, frequently observed in people with motor disabilities, which may trigger false positives. The main purpose of this article is to perform a comparative analysis of different methods in multiple configurations of their parameters, with the goal of selecting one that can be implemented in an embedded system, targeting real time and wireless operation of a tool for Human-Computer Interaction. Furthermore, in this work we seek to improve the performance of existing onset detection methods, through a proposed sensor fusion approach, combining an Accelerometer with the Surface Electromyography sensor, to integrate motion analysis in the process of validating or rejecting muscle events.