Sensor Data Streams Research Articles

Wearable Sensors for Activity Recognition in Ultimate Frisbee Using Convolutional Neural Networks and Transfer Learning.

In human activity recognition (HAR), activities are automatically recognized and classified from a continuous stream of input sensor data. Although the scientific community has developed multiple approaches for various sports in recent years, marginal sports are rarely considered. These approaches cannot directly be applied to marginal sports, where available data are sparse and costly to acquire. Thus, we recorded and annotated inertial measurement unit (IMU) data containing different types of Ultimate Frisbee throws to investigate whether Convolutional Neural Networks (CNNs) and transfer learning can solve this. The relevant actions were automatically detected and were classified using a CNN. The proposed pipeline reaches an accuracy of 66.6%, distinguishing between nine different fine-grained classes. For the classification of the three basic throwing techniques, we achieve an accuracy of 89.9%. Furthermore, the results were compared to a transfer learning-based approach using a beach volleyball dataset as the source. Even if transfer learning could not improve the classification accuracy, the training time was significantly reduced. Finally, the effect of transfer learning on a reduced dataset, i.e., without data augmentations, is analyzed. While having the same number of training subjects, using the pre-trained weights improves the generalization capabilities of the network, i.e., increasing the accuracy and F1 score. This shows that transfer learning can be beneficial, especially when dealing with small datasets, as in marginal sports, and therefore, can improve the tracking of marginal sports.

Online Activity Recognition Combining Dynamic Segmentation and Emergent Modeling.

Activity recognition is fundamental to many applications envisaged in pervasive computing, especially in smart environments where the resident’s data collected from sensors will be mapped to human activities. Previous research usually focuses on scripted or pre-segmented sequences related to activities, whereas many real-world deployments require information about the ongoing activities in real time. In this paper, we propose an online activity recognition model on streaming sensor data that incorporates the spatio-temporal correlation-based dynamic segmentation method and the stigmergy-based emergent modeling method to recognize activities when new sensor events are recorded. The dynamic segmentation approach integrating sensor correlation and time correlation judges whether two consecutive sensor events belong to the same window or not, avoiding events from very different functional areas or with a long time interval in the same window, thus obtaining the segmented window for every single event. Then, the emergent paradigm with marker-based stigmergy is adopted to build activity features that are explicitly represented as a directed weighted network to define the context for the last sensor event in this window, which does not need sophisticated domain knowledge. We validate the proposed method utilizing the real-world dataset Aruba from the CASAS project and the results show the effectiveness.

STranGAN: Adversarially-learnt Spatial Transformer for scalable human activity recognition

We tackle the problem of domain adaptation for inertial sensing-based human activity recognition (HAR) applications -i.e., in developing mechanisms that allow a classifier trained on sensor samples collected under a certain narrow context to continue to achieve high activity recognition accuracy even when applied to other contexts. This is a problem of high practical importance as the current requirement of labeled training data for adapting such classifiers to every new individual, device, or on-body location is a major roadblock to community-scale adoption of HAR-based applications. We particularly investigate the possibility of ensuring robust classifier operation, without requiring any new labeled training data, under changes to (a) the individual performing the activity, and (b) the on-body position where the sensor-embedded mobile or wearable device is placed. We propose STranGAN, a framework that explicitly decouples the domain adaptation functionality from the classification model by learning and applying a set of optimal spatial affine transformations on the target domain inertial sensor data stream by employing adversarial learning, which only requires collecting raw data samples (but no accompanying activity labels) from both source and target domains. STranGAN’s uniqueness lies in its ability to perform practically useful adaptation (a) without any labeled training data and without requiring paired, synchronized generation of source and target domain samples, and (b) without requiring any changes to a pre-trained HAR classifier. Empirical results using three publicly available benchmark datasets indicate that STranGAN(a) is particularly effective in handling on-body position heterogeneity(achieving a 5% improvement in classification F1 score compared to state-of-the-art baselines), (b) offers competitive performance for handling cross-individual variations, and (c) the affine transformation parameters can be analyzed to gain interpretable insights on the domain heterogeneity.

An Efficient Outlier Detection Approach for Streaming Sensor Data Based on Neighbor Difference and Clustering

In wireless sensor networks (WSNs), the widely distributed sensors make the real-time processing of data face severe challenges, which prompts the use of edge computing. However, some problems that occur during the operation of sensors will cause unreliability of the collected data, which can result in inaccurate results of edge computing-based processing; thus, it is necessary to detect potential abnormal data (also known as outliers) in the sensor data to ensure their quality. Although the clustering-based outlier detection approaches can detect outliers from the static data, the feature of streaming sensor data requires the detection operation in a one-pass fashion; in addition, the clustering-based approaches also do not consider the time correlation among the streaming sensor data, which leads to its low detection accuracy. To solve these problems, we propose an efficient outlier detection approach based on neighbor difference and clustering, namely, ODNDC, which not only quickly and accurately detects outliers but also identifies the source of outliers in the streaming sensor data. Experiments on a synthetic dataset and a real dataset show that the proposed ODNDC approach achieves great performance in detecting outliers and identifying their sources, as well as the low time consumption.

Online Missing Data Imputation Using Virtual Temporal Neighbor in Wireless Sensor Networks

A wireless sensor network (WSN) is one of the most typical applications of the Internet of Things (IoT). Missing values exist in the sensor data streams unavoidably because of the way WSNs work and the environments they are deployed in. In most cases, imputing missing values is the universally adopted approach before making further data processing. There are different ways to implement it, among which the exploitation of correlation information hidden in the sensor data interests many researchers, and lots of results have emerged. Researching in the same way, in this paper, we propose VTN imputation, an online missing data imputation algorithm based on virtual temporal neighbors. Firstly, the virtual temporal neighbor (VTN) in the sensor data stream is defined, and the calculation method is given. Next, the VTN imputation algorithm, which applies VTN to make estimates for missing values by regression is presented. Finally, we make experiments to evaluate the performance of imputing accuracy and computation time for our algorithm on three different real sensor datasets. The experiment results show that the VTN imputation algorithm benefited from the fuller exploitation of the correlation in sensor data and obtained better accuracy of imputation and acceptable processing time in the real applications of WSNs.

Wireless Sensor Networks for Building Information Modeling

Building Information Modeling (BIM) is a critical element for the “digitalization” of the construction industry and can be exploited for energy-driven renovation procedures of existing residences. Advancing beyond a BIM with data-capturing capabilities that are limited to building static information only requires sensor data streams related to indoor/outdoor ambient conditions, as well as to energy-consumption parameters of the residences. The data streams require the deployment of robust Wireless Sensor Networks (WSNs) that are able to capture and transmit real-time data to appropriate cloud-based renovation toolkits. The technology and topology of such networks are addressed herein. The paper sets the lines for similar installations that are required by the construction industry for collecting dynamic data, since it is based on the outcome of real-world WSN installations in pilot sites in three European countries, carried out in the context of a major collaborative BIM research project. An application example of the WSN data is also provided in the context of training occupant behavior models in order to demonstrate the use of the measured data.

Data portability for activities of daily living and fall detection in different environments using radar micro-doppler

The health status of an older or vulnerable person can be determined by looking into the additive effects of aging as well as any associated diseases. This status can lead the person to a situation of ‘unstable incapacity’ for normal aging and is determined by the decrease in response to the environment and to specific pathologies with apparent decrease of independence in activities of daily living (ADL). In this paper, we use micro-Doppler images obtained using a frequency-modulated continuous wave radar (FMCW) operating at 5.8 GHz with 400 MHz bandwidth as the sensor to perform assessment of this health status. The core idea is to develop a generalized system where the data obtained for ADL can be portable across different environments and groups of subjects, and critical events such as falls in mature individuals can be detected. In this context, we have conducted comprehensive experimental campaigns at nine different locations including four laboratory environments and five elderly care homes. A total of 99 subjects participated in the experiments where 1453 micro-Doppler signatures were recorded for six activities. Different machine learning, deep learning algorithms and transfer learning technique were used to classify the ADL. The support vector machine (SVM), K-nearest neighbor (KNN) and convolutional neural network (CNN) provided adequate classification accuracies for particular scenarios; however, the autoencoder neural network outperformed the mentioned classifiers by providing classification accuracy of ~ 88%. The proposed system for fall detection in elderly people can be deployed in care centers and is application for any indoor settings with various age group of people. For future work, we would focus on monitoring multiple older adults, concurrently in indoor settings using continuous radar sensor data stream which is limitation of the present system.

Recovering Missing Data via Top-k Repeated Patterns for Fuzzy-Based Abnormal Node Detection in Sensor Networks

The stream data acquired by heterogeneous Internet of Things (IoT) sensors are seldom perfect. Most of the collected data streams include either missing or abnormal values caused by various factors such as failure, malfunction, or integrity attacks. Such unreliable data affect the real-time monitoring and compromise the quality of data analysis. By simply analyzing the sensor data via anomaly detection, applications may still be unreliable over the incomplete sensor data streams. Therefore, a reliable method for recovering the missing data and detecting the abnormal ones is indispensable in the IoT environment. This paper presents FuzHD++, a new method to recover missing sensor data and detect abnormal nodes jointly rather than independently. Both elements, data recovery and abnormal node detection, rely on the observed temporal and spatial correlation of sensor data to effectively achieve reliable recovery estimation and detection performance. In the data recovery process, the system adopts a matrix profile to extract the top-<i>k</i> repeated patterns from different sensor nodes. Furthermore, it utilizes the <i>k</i>-nearest neighbor estimator to recover the missing data based on the extracted pattern information of multiple neighbor nodes. During the abnormal node detection process, the system adopts a refined fuzzy rule-based detection method. The refined fuzzy rule-based inference system integrates the expert rules and the rules obtained from sensor data analysis to treat the ambiguity in the decision-making process. We validated the performance of FuzHD++ by comparing it with existing methods using two real-world datasets. Our results showed that the proposed missing sensor data recovery method achieves more than 20% improved root mean square error results than most existing methods. Furthermore, FuzHD++ achieved an average accuracy of 92% for analyzing the sensor readings and detecting the abnormal ones. According to the results, the proposed mechanisms based on the observed temporal and spatial correlation analysis improve the robustness of IoT against data loss and integrity attacks.

Managing streamed sensor data for mobile equipment prognostics

Abstract The ability to wirelessly stream data from sensors on heavy mobile equipment provides opportunities to proactively assess asset condition. However, data analysis methods are challenging to apply due to the size and structure of the data, which contain inconsistent and asynchronous entries, and large periods of missing data. Current methods usually require expertise from site engineers to inform variable selection. In this work, we develop a data preparation method to clean and arrange this streaming data for analysis, including a data-driven variable selection. Data are drawn from a mining industry case study, with sensor data from a primary production excavator over a period of 9 months. Variables include 58 numerical sensors and 40 binary indicators captured in 45-million rows of data describing the conditions and status of different subsystems of the machine. A total of 57% of time stamps contain missing values for at least one sensor. The response variable is drawn from fault codes selected by the operator and stored in the fleet management system. Application to the hydraulic system, for 21 failure events identified by the operator, shows that the data-driven selection contains variables consistent with subject matter expert expectations, as well as some sensors on other systems on the excavator that are less easy to explain from an engineering perspective. Our contribution is to demonstrate a compressed data representation using open-high-low-close and variable selection to visualize data and support identification of potential indicators of failure events from multivariate streamed data.

PCMCR: A Novel Conflict Resolution Strategy based on Possibility Theory for Human Activity Recognition

DCR is a Distributed Collaborative Reasoning multi-agent model that aims to recognize human activities in smart homes from distributed, heterogeneous and dynamic sensor data. In this model, distributed agents with diverse classifiers, detect sensor stream data, make local predictions, communicate and collaborate to identify current activities. Conflict resolution strategies are applied to generate one final predicted activity when the local predicted activity of an agent is different from received predicted activities of other agents. In this paper, a possibilistic conflict resolution strategy, PCMCR, is proposed. Possibility theory is particularly efficient in combining multiple agents predictions that can be incomplete, uncertain, and conflicting. The PCMCR strategy deals with uncertainty factor which can be presented in the predictions of poor agents. It can take advantage of the complementary information given by each agent, even the weak ones. We experimentally test this strategy by performing an evaluation study on Aruba dataset. The obtained results indicate an enhancement in terms of accuracy, F-measure and G-mean metrics compared to the existing conflict resolution method max-trust of DCR, to the centralized model and to an existing distributed system.

Pocketable Labs for Everyone: Synchronized Multi-Sensor Data Streaming and Recording on Smartphones with the Lab Streaming Layer.

The streaming and recording of smartphone sensor signals is desirable for mHealth, telemedicine, environmental monitoring and other applications. Time series data gathered in these fields typically benefit from the time-synchronized integration of different sensor signals. However, solutions required for this synchronization are mostly available for stationary setups. We hope to contribute to the important emerging field of portable data acquisition by presenting open-source Android applications both for the synchronized streaming (Send-a) and recording (Record-a) of multiple sensor data streams. We validate the applications in terms of functionality, flexibility and precision in fully mobile setups and in hybrid setups combining mobile and desktop hardware. Our results show that the fully mobile solution is equivalent to well-established desktop versions. With the streaming application Send-a and the recording application Record-a, purely smartphone-based setups for mobile research and personal health settings can be realized on off-the-shelf Android devices.

Compensated methods for networked control system with packet drops based on compressed sensing

Due to unreliable and bandwidth-limited characteristics of communication link in networked control systems, the real-time compensated methods for single-output systems and multioutput systems are proposed in this paper based on the compressed sensing (CS) theory and sliding window technique, by which the estimates of dropping data packets in the feedback channel are obtained and the performance degradation induced by packet drops is reduced. Specifically, in order to reduce the cumulative error caused by the algorithm, the compensated estimates for single-output systems are corrected via the regularization term; considering the process of single-packet transmission, a new sequential CS framework of sensor data streams is introduced to effectively compensate the dropping packet on single-channel of multi-output systems; in presence of the medium access constraints on multi-channel, the communication sequence for scheduling is coupled to the algorithm and the estimates of the multiple sensors for multi-output systems are obtained via the regularization term. Simulation results illustrate that the proposed methods perform well and receive satisfactory performance.

Methods to Improve the Prognostics of Time-to-Failure Models

Autonomous and autonomic systems have started to develop machine learning (ML) methods for prognostics and health management (PHM) directly at the platform level. Remaining-useful-life (RUL) estimation, also known as Time-to-failure (TTF) estimation, using streaming sensor data is critical for PHM as it can help to decide and schedule appropriate courses of action (COAs). This work casts the RUL-estimation problem as a classification problem over a finite-time horizon. Rather than using a winner-take-all method to develop a RUL estimator, we propose a top-K estimator that considers the RUL values corresponding to the K-largest probabilities yielded by the classifier to develop our estimator. The top-K RUL values can be used to drive the execution of conservative or aggressive PHM strategies, or be tracked over time to develop robust RUL estimators that leverage the history of RUL estimates. The performance of the proposed RUL estimators is illustrated on a dataset from NASA’s Prognostics Center of Excellence.

Probabilistic Value-Deviation-Bounded Source-Dependent Bit-Level Channel Adaptation for Approximate Communication

Computing systems that can tolerate effects of errors in their communicated data values can trade this tolerance for improved resource efficiency. Many important applications of computing, such as embedded sensor systems, can tolerate errors that are bounded in their distribution of deviation from correctness (distortion). We present a channel adaptation technique which modulates properties of I/O channels typical in embedded sensor systems, to provide a tradeoff between I/O power dissipation and distortion of communicated data. We provide an efficient-to-compute formulation for the distribution of integer distortion accounting for the distribution of transmitted values. Using this formulation we implement our value-deviation-bounded (VDB) channel adaptation. We experimentally quantify the achieved reduction in power dissipation on a hardware prototype integrated with the required programmable channel modulation circuitry. We augment these experimental measurements with an analysis of the distributions of distortions. We show that our probabilistic VDB channel adaptation can provide up to a 2× reduction in I/O power dissipation. When synthesized for a miniature low-power FPGA intended for use in sensor interfaces, a register transfer level implementation of the channel adaptation control logic requires only 106 flip-flops and 224 4-input LUTs for implementing per-bit channel adaptation on serialized streams of 8-bit sensor data.

Employing Multimodal Machine Learning for Stress Detection.

In the current information age, the human lifestyle has become more knowledge-oriented, leading to sedentary employment. This has given rise to a number of health and mental disorders. Mental wellness is one of the most neglected, however crucial, aspects of today's fast-paced world. Mental health issues can, both directly and indirectly, affect other sections of human physiology and impede an individual's day-to-day activities and performance. However, identifying the stress and finding the stress trend for an individual that may lead to serious mental ailments is challenging and involves multiple factors. Such identification can be achieved accurately by fusing these multiple modalities (due to various factors) arising from a person's behavioral patterns. Specific techniques are identified in the literature for this purpose; however, very few machine learning-based methods are proposed for such multimodal fusion tasks. In this work, a multimodal AI-based framework is proposed to monitor a person's working behavior and stress levels. We propose a methodology for efficiently detecting stress due to workload by concatenating heterogeneous raw sensor data streams (e.g., face expressions, posture, heart rate, and computer interaction). This data can be securely stored and analyzed to understand and discover personalized unique behavioral patterns leading to mental strain and fatigue. The contribution of this work is twofold: firstly, proposing a multimodal AI-based strategy for fusion to detect stress and its level and, secondly, identifying a stress pattern over a period of time. We were able to achieve 96.09% accuracy on the test set in stress detection and classification. Further, we were able to reduce the stress scale prediction model loss to 0.036 using these modalities. This work can prove important for the community at large, specifically those working sedentary jobs, to monitor and identify stress levels, especially in current times of COVID-19.

Adaptive Segmentation of Streaming Sensor Data on Edge Devices.

Sensor data streams often represent signals/trajectories which are twice differentiable (e.g., to give a continuous velocity and acceleration), and this property must be reflected in their segmentation. An adaptive streaming algorithm for this problem is presented. It is based on the greedy look-ahead strategy and is built on the concept of a cubic splinelet. A characteristic feature of the proposed algorithm is the real-time simultaneous segmentation, smoothing, and compression of data streams. The segmentation quality is measured in terms of the signal approximation accuracy and the corresponding compression ratio. The numerical results show the relatively high compression ratios (from 135 to 208, i.e., compressed stream sizes up to 208 times smaller) combined with the approximation errors comparable to those obtained from the state-of-the-art global reference algorithm. The proposed algorithm can be applied to various domains, including online compression and/or smoothing of data streams coming from sensors, real-time IoT analytics, and embedded time-series databases.

Contaminant Source Identification from Finite Sensor Data: Perron–Frobenius Operator and Bayesian Inference

Sensors in the built environment ensure safety and comfort by tracking contaminants in the occupied space. In the event of contaminant release, it is important to use the limited sensor data to rapidly and accurately identify the release location of the contaminant. Identification of the release location will enable subsequent remediation as well as evacuation decision-making. In previous work, we used an operator theoretic approach—based on the Perron–Frobenius (PF) operator—to estimate the contaminant concentration distribution in the domain given a finite amount of streaming sensor data. In the current work, the approach is extended to identify the most probable contaminant release location. The release location identification is framed as a Bayesian inference problem. The Bayesian inference approach requires considering multiple release location scenarios, which is done efficiently using the discrete PF operator. The discrete PF operator provides a fast, effective and accurate model for contaminant transport modeling. The utility of our PF-based Bayesian inference methodology is illustrated using single-point release scenarios in both two and three-dimensional cases. The method provides a fast, accurate, and efficient framework for real-time identification of contaminant source location.

IoTSAS: An Integrated System for Real-Time Semantic Annotation and Interpretation of IoT Sensor Stream Data

Sensors and other Internet of Things (IoT) technologies are increasingly finding application in various fields, such as air quality monitoring, weather alerts monitoring, water quality monitoring, healthcare monitoring, etc. IoT sensors continuously generate large volumes of observed stream data; therefore, processing requires a special approach. Extracting the contextual information essential for situational knowledge from sensor stream data is very difficult, especially when processing and interpretation of these data are required in real time. This paper focuses on processing and interpreting sensor stream data in real time by integrating different semantic annotations. In this context, a system named IoT Semantic Annotations System (IoTSAS) is developed. Furthermore, the performance of the IoTSAS System is presented by testing air quality and weather alerts monitoring IoT domains by extending the Open Geospatial Consortium (OGC) standards and the Sensor Observations Service (SOS) standards, respectively. The developed system provides information in real time to citizens about the health implications from air pollution and weather conditions, e.g., blizzard, flurry, etc.

The Data Sensor Hub (DaSH): A Physical Computing System to Support Middle School Inquiry Science Instruction.

This article describes a sensor-based physical computing system, called the Data Sensor Hub (DaSH), which enables students to process, analyze, and display data streams collected using a variety of sensors. The system is built around the portable and affordable BBC micro:bit microcontroller (expanded with the gator:bit), which students program using a visual, cloud-based programming environment intended for novices. Students connect a variety of sensors (measuring temperature, humidity, carbon dioxide, sound, acceleration, magnetism, etc.) and write programs to analyze and visualize the collected sensor data streams. The article also describes two instructional units intended for middle grade science classes that use this sensor-based system. These inquiry-oriented units engage students in designing the system to collect data from the world around them to investigate scientific phenomena of interest. The units are designed to help students develop the ability to meaningfully integrate computing as they engage in place-based learning activities while using tools that more closely approximate the practices of contemporary scientists as well as other STEM workers. Finally, the article articulates how the DaSH and units have elicited different kinds of teacher practices using student drawn modeling activities, facilitating debugging practices, and developing place-based science practices.

Toward a Mobile Platform for Real-world Digital Measurement of Depression: User-Centered Design, Data Quality, and Behavioral and Clinical Modeling.

BackgroundAlthough effective mental health treatments exist, the ability to match individuals to optimal treatments is poor, and timely assessment of response is difficult. One reason for these challenges is the lack of objective measurement of psychiatric symptoms. Sensors and active tasks recorded by smartphones provide a low-burden, low-cost, and scalable way to capture real-world data from patients that could augment clinical decision-making and move the field of mental health closer to measurement-based care.ObjectiveThis study tests the feasibility of a fully remote study on individuals with self-reported depression using an Android-based smartphone app to collect subjective and objective measures associated with depression severity. The goals of this pilot study are to develop an engaging user interface for high task adherence through user-centered design; test the quality of collected data from passive sensors; start building clinically relevant behavioral measures (features) from passive sensors and active inputs; and preliminarily explore connections between these features and depression severity.MethodsA total of 600 participants were asked to download the study app to join this fully remote, observational 12-week study. The app passively collected 20 sensor data streams (eg, ambient audio level, location, and inertial measurement units), and participants were asked to complete daily survey tasks, weekly voice diaries, and the clinically validated Patient Health Questionnaire (PHQ-9) self-survey. Pairwise correlations between derived behavioral features (eg, weekly minutes spent at home) and PHQ-9 were computed. Using these behavioral features, we also constructed an elastic net penalized multivariate logistic regression model predicting depressed versus nondepressed PHQ-9 scores (ie, dichotomized PHQ-9).ResultsA total of 415 individuals logged into the app. Over the course of the 12-week study, these participants completed 83.35% (4151/4980) of the PHQ-9s. Applying data sufficiency rules for minimally necessary daily and weekly data resulted in 3779 participant-weeks of data across 384 participants. Using a subset of 34 behavioral features, we found that 11 features showed a significant (P<.001 Benjamini-Hochberg adjusted) Spearman correlation with weekly PHQ-9, including voice diary–derived word sentiment and ambient audio levels. Restricting the data to those cases in which all 34 behavioral features were present, we had available 1013 participant-weeks from 186 participants. The logistic regression model predicting depression status resulted in a 10-fold cross-validated mean area under the curve of 0.656 (SD 0.079).ConclusionsThis study finds a strong proof of concept for the use of a smartphone-based assessment of depression outcomes. Behavioral features derived from passive sensors and active tasks show promising correlations with a validated clinical measure of depression (PHQ-9). Future work is needed to increase scale that may permit the construction of more complex (eg, nonlinear) predictive models and better handle data missingness.