Multimodal Sensor Data Research Articles

A Sensor-enabled Digital Trier Social Stress Test in an Enterprise Context.

The Trier Social Stress Test (TSST) protocol is a widely accepted method of inducing social and/or cognitive stress in participants and studying its effects. Traditionally, this protocol is administered in laboratory or university settings, which are less formal than in offices. In this paper, we report the results of the analysis of multi-modal sensor data collected from employees of an enterprise who underwent the test. We briefly discuss the adaptations that enabled administering it digitally in a semi-automatic mode with minimal researcher/test-administrator intervention. In our setup, noninvasive sensor-signals, including the Galvanic Skin Response and Photoplethysmogram, were collected during and outside the stress-inducing tasks. We analyze the data collected from twenty participants and show that the State Trait Anxiety Inventory (STAI) score is needed in assessing the effect of the digital version of the TSST. A support vector machine classifier yielded an F1 score of 0.723 with the STAI score taken as ground truth. We also introduce the idea of ground truth based on the change in the STAI scores to reduce variation due to subjective interpretation, for which an F1 score of 0.847 was obtained.

Delay and energy-efficient data collection scheme-based matrix filling theory for dynamic traffic IoT

Data collection is the basic functions of the Internet of Things (IoT), in which the sensed data are concentrations from sensor nodes to the sink, with a timely style, so the smart response can be done for emergency. The goal of multi-modal sensor data fusion is to obtain simple and accurate data to enhance system reliability and fault tolerance. Energy efficiency and small delay are the most important indicators which govern the performance of IoT. Convergecast is a low-latency data collection strategy based on effective time division multiple access (TDMA), in which each sensor node generates a packet, and m packets can aggregate to a packet. However, in most practical networks, sensor nodes do not necessarily generate packets during each data collection cycle, but instead generate packets from time to time. In the previous convergecast strategy, each node was fixedly allocated a slot, which increased the delay and wasted energy. A delay and energy-efficient data collection (DEEDC) scheme-based matrix filling theory is proposed to collect data in a randomly generated WSNs with minimum delay and energy consumption. The DEEDC scheme uses a clustering approach. For each cluster, the number of slots required for transmission is calculated by matrix filling theory, not the number of nodes that actually generate data. This ensures that data can be collected in a network with randomly generated data (number of slots ≤ number of nodes), thereby avoiding the allocation of slots for each node and the acquisition of redundant data to lead to the wastage of time and energy. Based on the above, a mixed slot scheduling strategy is proposed to construct energy and delay-efficient, collision-free schedule scheme. After extensive theoretical analysis, by using the DEEDC scheme, the delay is reduced by about 50~80%, and the energy consumed is reduced by about 40~57%.

Discovery of Multimodal Sensor Data Through Webpage Exploration

Technological advances allow perceptual physical objects to be connected to the Internet and share their information over webpages. As a predominant source of public sensor data, automatic discovery of these data is critical and desirable for general Internet of Things search service, which is fundamental to many intelligent applications. However, discovering sensor data from webpages is quite challenging, since there are diverse data presentation modes, and webpage layouts and structures are complicated and heterogeneous. To this end, we explore webpages to discover and collect sensor data under a hierarchical mechanism. In this paper, we first devise novel textual features (TFs) to recognize potential webpages that may contain sensor data; specifically, we construct sensing information corpus to provide keyword reference for the features. Then to get the position of sensor data, we develop granularity adaptive page segmentation (GAPS) algorithm to segment potential webpages into a set of informative blocks; and accordingly, we extract several visual features (VFs) of the blocks so that sensor data can be identified via a block classifier. Based on the novel created sensing information pages dataset, which consists of webpages and manual markings about sensing data, extensive experiments are conducted to evaluate the performance of our exploration methods. Results demonstrate that the TFs achieve supreme performance in sensing data recognition when compared to the state-of-the-art approaches, and GAPS is efficient to locate multimodal sensor data in cooperation with the VFs.

A Review on Automatic Facial Expression Recognition Systems Assisted by Multimodal Sensor Data.

Facial Expression Recognition (FER) can be widely applied to various research areas, such as mental diseases diagnosis and human social/physiological interaction detection. With the emerging advanced technologies in hardware and sensors, FER systems have been developed to support real-world application scenes, instead of laboratory environments. Although the laboratory-controlled FER systems achieve very high accuracy, around 97%, the technical transferring from the laboratory to real-world applications faces a great barrier of very low accuracy, approximately 50%. In this survey, we comprehensively discuss three significant challenges in the unconstrained real-world environments, such as illumination variation, head pose, and subject-dependence, which may not be resolved by only analysing images/videos in the FER system. We focus on those sensors that may provide extra information and help the FER systems to detect emotion in both static images and video sequences. We introduce three categories of sensors that may help improve the accuracy and reliability of an expression recognition system by tackling the challenges mentioned above in pure image/video processing. The first group is detailed-face sensors, which detect a small dynamic change of a face component, such as eye-trackers, which may help differentiate the background noise and the feature of faces. The second is non-visual sensors, such as audio, depth, and EEG sensors, which provide extra information in addition to visual dimension and improve the recognition reliability for example in illumination variation and position shift situation. The last is target-focused sensors, such as infrared thermal sensors, which can facilitate the FER systems to filter useless visual contents and may help resist illumination variation. Also, we discuss the methods of fusing different inputs obtained from multimodal sensors in an emotion system. We comparatively review the most prominent multimodal emotional expression recognition approaches and point out their advantages and limitations. We briefly introduce the benchmark data sets related to FER systems for each category of sensors and extend our survey to the open challenges and issues. Meanwhile, we design a framework of an expression recognition system, which uses multimodal sensor data (provided by the three categories of sensors) to provide complete information about emotions to assist the pure face image/video analysis. We theoretically analyse the feasibility and achievability of our new expression recognition system, especially for the use in the wild environment, and point out the future directions to design an efficient, emotional expression recognition system.

State Estimation via Communication for Monitoring

Monitoring using sensors is ubiquitous in our environment. In this paper, a state estimation model is proposed for continuous activity monitoring from multimodal and heterogenous sensor data. Each sensor is modeled as an independent agent in the predictive coding framework. It can sample its environment, communicate with other agents, and adapt its internal model to its environment in an unsupervised manner. Using controlled experiments, we show that limitations of each sensor, such as inference inaccuracy and delay, can be overcome through communication with the other sensors. The model is tested for human action recognition on the UTD-MHAD and MHEALTH datasets and for gait freeze recognition from wearable acceleration sensors in Parkinson's disease patients. In both cases, the proposed model yields results comparable to the state-of-the-art. To the best of our knowledge, this is the first work taking advantage of opportunistic communication by a predictive coding agent for estimating the state of its environment.

Multi-model Z-compression for high speed data streaming and low-power wireless sensor networks

Wireless Sensor Networks (WSNs) have significant limitations in terms of available bandwidth and energy. The limited bandwidth in WSNs can cause delays in message delivery, which does not suit the real-time sensing applications such as a gas leak. Moreover, in such applications, there are multi-modal sensors whose values like temperature, gas concentration, location, and CO$$_2$$ levels must be transmitted together for correct and timely detection of a gas leak. In this paper, we propose a novel Z-order based data compression scheme, Z-compression, to reduce energy consumption and save bandwidth without increasing message delivery latency. Instead of using the popular Huffman tree style based encoding, the Z-compression uses Z-order encoding to map the multidimensional sensing data into one-dimensional binary stream transmitted using a single packet. Our experimental evaluations using different real-world datasets show that the Z-compression has a much better compression ratio, energy, and bandwidth savings than known schemes like LEC, Adaptive-LEC, FELACS, and TinyPack for multi-modal sensor data. Through the extensive experiments, we show that Z-compression is suitable for real-world sensing applications requiring high-stream rate WSNs and delay-tolerant low-power listening WSNs. In high-stream rate WSNs, the Z-compression can save bandwidth and increases the throughput. In low-power listening WSNs, by concatenating the Z-compressed data at selected reporting nodes, we can reduce the duty cycles of the nodes in WSNs, thus prolong the lifetime of the network, and still maintain the low distortion rate.

A Hybrid Deep Learning Model for Human Activity Recognition Using Multimodal Body Sensing Data

Human activity recognition from multimodal body sensor data has proven to be an effective approach for the care of elderly or physically impaired people in a smart healthcare environment. However, traditional machine learning techniques are mostly focused on a single sensing modality, which is not practical for robust healthcare applications. Therefore, recently increasing attention is being given by the researchers on the development of robust machine learning techniques that can exploit multimodal body sensor data and provide important decision making in Smart healthcare. In this paper, we propose an effective multi-sensors-based framework for human activity recognition using a hybrid deep learning model, which combines the simple recurrent units (SRUs) with the gated recurrent units (GRUs) of neural networks. We use the deep SRUs to process the sequences of multimodal input data by using the capability of their internal memory states. Moreover, we use the deep GRUs to store and learn how much of the past information is passed to the future state for solving fluctuations or instability in accuracy and vanishing gradient problems. The system has been compared against the conventional approaches on a publicly available standard dataset. The experimental results show that the proposed approach outperforms the available state-of-the-art methods.

Incremental Learning Introspective Movement Primitives From Multimodal Unstructured Demonstrations

Learning movement primitive from unstructured demonstrations has become a popular topic in recent years, which provides a natural way to endow human-inspired skills to robots. The main idea of movement primitives is that should suffice to reconstruct a large set of complex manipulation tasks. However, conventional learning methods mostly focus on the kinesthetic variables and ignore those critical introspective capacities in manipulation such as movement generalization and assessment of the sensory signals. In this paper, we investigate the association of generalization, fault detection, fault diagnoses, and task exploration during manipulation task, and call such movement primitives augmented with introspective capacities Introspective Movement Primitives (IMP). With our previous work, this paper mainly addresses how IMPs can be acquired by assessing the quality of multimodal sensory data of unstructured demonstrations and how they can incrementally create manipulation task by reverse execution and human interaction. Experimental evaluation on a human-robot collaborative packaging task with a Rethink Baxter robot, results indicate that our proposed method can effectively increase robustness towards external perturbations and adaptive exploration during robot manipulation task.

Personal Climatization Systems—A Review on Existing and Upcoming Concepts

To accomplish the current climate goals of the federal republic of Germany, energy efficiency within the building and automotive sector must improve considerably. One possible way to reduce the high amount of energy required for heating, ventilation, and air-conditioning (HVAC) is the introduction of personal climatization systems in combination with the extension of the standardized room air temperature range. Personal systems allow improvements of climatic conditions (heating, cooling, and air quality) within sub-areas of the room instead of conditioning an entire room air volume. In this regard, personal systems are perfectly suitable for locations with local air-conditioning focal points, such as open-plan offices and vehicle cabins, where they substantially improve the energy efficiency of the entire system. This work aims to summarize previously conducted research in the area of personal climatization systems. The investigated local thermal actuators comprise fans for the generation of air movement, ventilators for the improvement of the air quality within the respiratory area of persons, water-conditioned panels for the climatization of persons via longwave radiation and conduction, radiant heaters, and combinations of the systems. Personal systems are superior to mixing ventilation regarding the improvement of the perceived air quality and thermal comfort. Furthermore, the introduced overview shows that personal climatization systems are generally more energy-efficient than conventional air-conditioning and facilitates the extension of the indoor air temperature corridor of the HVAC. Table fans and climatized seats are highly effective in connection with the improvement of personal thermal comfort. The performance of the overwhelming majority of applied personal environmental control systems is user-controlled or depends on a predefined load profile, which is generally defined person independent. Single studies reveal that effectively controlled automated systems have a similar thermal impact on a user’s thermal comfort as user-controlled ones. The implementation of an automated control system is feasible by using novel approaches such as the so-called human-centered closed loop control-platform (HCCLC-platform). The latter contains a central data server which allows asynchronous, bi-directional communication between multi-modal sensor data, user feedback systems, thermal actuators and numerical calculation models used to assess the individual thermal comfort of a person. This enables a continuous and holistic reflection of the thermal situation inside a room and the estimation of the corresponding impact on an individual’s thermal comfort. Considering the measured and simulated thermal state of a single person, the described system is capable of determining body-part-specific energy requirements that are needed to keep the overall thermal comfort level of an individual person on a high level.

A model for integrating heterogeneous sensory data in IoT systems

With the development of Internet of Things (IoT), heterogeneous sensory data appears everywhere in our lives. Unlike traditional sensory data, heterogeneous sensory data often involves variety modalities of data in one set, so that it is called as the multi-modal sensory data in this paper. The appearance of such data making it possible to monitor more complicated objects and improve monitoring accuracy. However, due to lack of integration model for multi-modal sensory data, most of the existing sensory data management algorithms only consider single modal sensory data, resulting in insufficient utilization of sensory data. Thus, we propose a model for integrating the heterogeneous sensory data generated in a IoT system based on Hidden Markov Process in the paper. The distributed algorithm for constructing such a model is then presented. The integration model can be applied to many applications, while we take the cooperative event detection as an example for illustration. The extensive theoretical analysis and experimental results show that all the proposed algorithms are efficient and effective .

Robust Fusion of LiDAR and Wide-Angle Camera Data for Autonomous Mobile Robots.

Autonomous robots that assist humans in day to day living tasks are becoming increasingly popular. Autonomous mobile robots operate by sensing and perceiving their surrounding environment to make accurate driving decisions. A combination of several different sensors such as LiDAR, radar, ultrasound sensors and cameras are utilized to sense the surrounding environment of autonomous vehicles. These heterogeneous sensors simultaneously capture various physical attributes of the environment. Such multimodality and redundancy of sensing need to be positively utilized for reliable and consistent perception of the environment through sensor data fusion. However, these multimodal sensor data streams are different from each other in many ways, such as temporal and spatial resolution, data format, and geometric alignment. For the subsequent perception algorithms to utilize the diversity offered by multimodal sensing, the data streams need to be spatially, geometrically and temporally aligned with each other. In this paper, we address the problem of fusing the outputs of a Light Detection and Ranging (LiDAR) scanner and a wide-angle monocular image sensor for free space detection. The outputs of LiDAR scanner and the image sensor are of different spatial resolutions and need to be aligned with each other. A geometrical model is used to spatially align the two sensor outputs, followed by a Gaussian Process (GP) regression-based resolution matching algorithm to interpolate the missing data with quantifiable uncertainty. The results indicate that the proposed sensor data fusion framework significantly aids the subsequent perception steps, as illustrated by the performance improvement of a uncertainty aware free space detection algorithm.

A joint manifold leaning-based framework for heterogeneous upstream data fusion

A joint manifold learning fusion (JMLF) approach is proposed for nonlinear or mixed sensor modalities with large streams of data. The multimodal sensor data are stacked to form joint manifolds, from which the embedded low intrinsic dimensionalities are discovered for moving targets. The intrinsic low dimensionalities are mapped to resolve the target locations. The JMLF framework is tested on digital imaging and remote sensing image generation scenes with mid-wave infrared (WMIR) data augmented with distributed radio-frequency (RF) Doppler data. Eight manifold learning methods are explored to train the system with the neighborhood preserving embedding showing promise for robust target tracking using video–radio-frequency fusion. The JMLF method shows a 93% improved accuracy as compared to a standard target tracking (e.g., Kalman-filter based) approach.

Depth Maps and Deep Learning for Facial Analysis

Gathering and examining progressively multi-modular sensor information of human faces is a critical issue in PC vision, with applications in examinations, entertainment, and security. However, due to the exigent nature of the problem, there is a lack of affordable and easy-to-use systems, with real-time, annotations capability, 3D analysis, replay capability and with a frame speed capable of detecting facial patterns in working behavior environments. In the context of an ongoing effort to develop tools to support the monitoring and evaluation of the human affective state in working environments, the authors investigate the applicability of a facial analysis approach to map and evaluate human facial patterns. The challenge is to interpret this multi-modal sensor data to classify it with deep learning algorithms and fulfill the following requirements: annotations capability, 3D analysis, and replay capability. In addition, the authors want to be able to continuously enhance the output result of the system with a training process in order to improve and evaluate different patterns of the human face.

Data fusion and multiple classifier systems for human activity detection and health monitoring: Review and open research directions

Activity detection and classification using different sensor modalities have emerged as revolutionary technology for real-time and autonomous monitoring in behaviour analysis, ambient assisted living, activity of daily living (ADL), elderly care, rehabilitations, entertainments and surveillance in smart home environments. Wearable devices, smart-phones and ambient environments devices are equipped with variety of sensors such as accelerometers, gyroscopes, magnetometer, heart rate, pressure and wearable camera for activity detection and monitoring. These sensors are pre-processed and different feature sets such as time domain, frequency domain, wavelet transform are extracted and transform using machine learning algorithm for human activity classification and monitoring. Recently, deep learning algorithms for automatic feature representation have also been proposed to lessen the burden of reliance on handcrafted features and to increase performance accuracy. Initially, one set of sensor data, features or classifiers were used for activity recognition applications. However, there are new trends on the implementation of fusion strategies to combine sensors data, features and classifiers to provide diversity, offer higher generalization, and tackle challenging issues. For instances, combination of inertial sensors provide mechanism to differentiate activity of similar patterns and accurate posture identification while other multimodal sensor data are used for energy expenditure estimations, object localizations in smart homes and health status monitoring. Hence, the focus of this review is to provide in-depth and comprehensive analysis of data fusion and multiple classifier systems techniques for human activity recognition with emphasis on mobile and wearable devices. First, data fusion methods and modalities were presented and also feature fusion, including deep learning fusion for human activity recognition were critically analysed, and their applications, strengths and issues were identified. Furthermore, the review presents different multiple classifier system design and fusion methods that were recently proposed in literature. Finally, open research problems that require further research and improvements are identified and discussed.

Deep Coupling Autoencoder for Fault Diagnosis With Multimodal Sensory Data

Effective fault diagnosis of rotating machinery has multifarious benefits, such as improved safety, enhanced reliability, and reduced maintenance cost, for complex engineered systems. With many kinds of installed sensors for conducting fault diagnosis, one of the key tasks is to develop data fusion strategies that can effectively handle multimodal sensory signals. Most traditional methods use hand-crafted statistical features and then combine these multimodal features simply by concatenating them into a long vector to achieve data fusion. The present study proposes a deep coupling autoencoder (DCAE) model that handles the multimodal sensory signals not residing in a commensurate space, such as vibration and acoustic data, and integrates feature extraction of multimodal data seamlessly into data fusion for fault diagnosis. Specifically, a coupling autoencoder (CAE) is constructed to capture the joint information between different multimodal sensory data, and then a DCAE model is devised for learning the joint feature at a higher level. The CAE is developed by coupling hidden representations of two single-modal autoencoders, which can capture the joint information from multimodal data. The performance of the proposed method is evaluated by two experiments, which shows that the DCAE model succeeds in efficiently utilizing multisource sensory data to perform accurate fault diagnosis. Compared with other methods, the proposed method exhibits better performance.

Accurate laser scanner to camera calibration with application to range sensor evaluation

Multi-modal sensory data plays an important role in many computer vision and robotics tasks. One popular multi-modal pair is cameras and laser scanners. To overlay and jointly use the data from both modalities, it is necessary to calibrate the sensors, i.e., to obtain the spatial relation between the sensors.Computing such a calibration is challenging as both sensors provide quite different data: cameras yield color or brightness information, laser scanners yield 3-D points. However, several laser scanners additionally provide reflectances, which turn out to make calibration to a camera well feasible. To this end, we first estimate a rough alignment of the coordinate systems of both modalities. Then, we use the laser scanner reflectances to compute a virtual image of the scene. Stereo calibration on the virtual image and the camera image are then used to compute a refined, high-accuracy calibration.It is encouraging that the accuracies in our experiments are comparable to camera-camera stereo setups and outperform another of other target-based calibration approach. This shows that the proposed algorithm reliably integrates the point cloud with the intensity image. As an example application, we use the calibration results to obtain ground-truth distance images for range cameras. Furthermore, we utilize this data to investigate the accuracy of the Microsoft Kinect V2 time-of-flight and the Intel RealSense R200 structured light camera.

MuSeSe - A multisensor armchair for unobtrusive vital sign estimation and motion artifact analysis.

Unobtrusive vital sign estimation with sensors integrated into objects of everyday living can substantially advance the field of remote monitoring. At the same time, motion artifacts cause severe problems and have to be dealt with. Here, the fusion of multimodal sensor data is a promising approach. In this paper, we present an armchair equipped with capacitively coupled electrocardiogram, two types of ballistocardiographic sensors, photoplethysmographic and two high-frequency impedance sensors. In addition, a video-based sensor for motion analysis is integrated. Using a defined motion protocol, the feasibility of the system is demonstrated in a self-experimentation. Moreover, the influence of different movements on different modalities is analyzed. Finally, robust beat-to-beat interval estimation demonstrates the benefits of multimodal sensor fusion for vital sign estimation in the presence of motion artifacts.

A Temporal Order Modeling Approach to Human Action Recognition from Multimodal Sensor Data

From wearable devices to depth cameras, researchers have exploited various multimodal data to recognize human actions for applications, such as video gaming, education, and healthcare. Although there many successful techniques have been presented in the literature, most current approaches have focused on statistical or local spatiotemporal features and do not explicitly explore the temporal dynamics of the sensor data. However, human action data contain rich temporal structure information that can characterize the unique underlying patterns of different action categories. From this perspective, we propose a novel temporal order modeling approach to human action recognition. Specifically, we explore subspace projections to extract the latent temporal patterns from different human action sequences. The temporal order between these patterns are compared, and the index of the pattern that appears first is used to encode the entire sequence. This process is repeated multiple times and produces a compact feature vector representing the temporal dynamics of the sequence. Human action recognition can then be efficiently solved by the nearest neighbor search based on the Hamming distance between these compact feature vectors. We further introduce a sequential optimization algorithm to learn the optimized projections that preserve the pairwise label similarity of the action sequences. Experimental results on two public human action datasets demonstrate the superior performance of the proposed technique in both accuracy and efficiency.

Driver state estimation by convolutional neural network using multimodal sensor data

A driver state estimation algorithm that uses multimodal vehicular and physiological sensor data is proposed. Deep learning is applied to the fused multimodal data rather than each modality being treated as a different feature. A convolutional neural network model is developed and the driver state estimation algorithm is implemented using Google TensorFlow. The results show that deep learning is a very promising approach for driver state estimation compared with previously studied algorithms, such as dynamic Bayesian networks.

Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices: Design Considerations.

In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intensive task that demands datacenter-scale computational resources recruited for many days. Here we propose a concept of resistive processing unit (RPU) devices that can potentially accelerate DNN training by orders of magnitude while using much less power. The proposed RPU device can store and update the weight values locally thus minimizing data movement during training and allowing to fully exploit the locality and the parallelism of the training algorithm. We evaluate the effect of various RPU device features/non-idealities and system parameters on performance in order to derive the device and system level specifications for implementation of an accelerator chip for DNN training in a realistic CMOS-compatible technology. For large DNNs with about 1 billion weights this massively parallel RPU architecture can achieve acceleration factors of 30, 000 × compared to state-of-the-art microprocessors while providing power efficiency of 84, 000 GigaOps∕s∕W. Problems that currently require days of training on a datacenter-size cluster with thousands of machines can be addressed within hours on a single RPU accelerator. A system consisting of a cluster of RPU accelerators will be able to tackle Big Data problems with trillions of parameters that is impossible to address today like, for example, natural speech recognition and translation between all world languages, real-time analytics on large streams of business and scientific data, integration, and analysis of multimodal sensory data flows from a massive number of IoT (Internet of Things) sensors.