Sensor Data Streams Research Articles

Evaluation of Jenks Natural Breaks Clustering Algorithm for Changepoint Identification in Streaming Sensor Data

Evaluation of Jenks Natural Breaks Clustering Algorithm for Changepoint Identification in Streaming Sensor Data

Archive-based Covert Channel in Sensor Streaming Data

Archive-based Covert Channel in Sensor Streaming Data

RADAR-IoT: An Open-Source, Interoperable, and Extensible IoT Gateway Framework for Health Research.

IoT sensors offer a wide range of sensing capabilities, many of which have potential health applications. Existing solutions for IoT in healthcare have notable limitations, such as closed-source, limited I/O protocols, limited cloud platform support, and missing specific functionality for health use cases. Developing an open-source internet of things (IoT) gateway solution that addresses these limitations and provides reliability, broad applicability, and utility is highly desirable. Combining a wide range of sensor data streams from IoT devices with ambulatory mHealth data would open up the potential to provide a detailed 360-degree view of the relationship between patient physiology, behavior, and environment. We have developed RADAR-IoT as an open-source IoT gateway framework, to harness this potential. It aims to connect multiple IoT devices at the edge, perform limited on-device data processing and analysis, and integrate with cloud-based mobile health platforms, such as RADAR-base, enabling real-time data processing. We also present a proof-of-concept data collection from this framework, using prototype hardware in two locations. The RADAR-IoT framework, combined with the RADAR-base mHealth platform, provides a comprehensive view of a user's health and environment by integrating static IoT sensors and wearable devices. Despite its current limitations, it offers a promising open-source solution for health research, with potential applications in managing infection control, monitoring chronic pulmonary disorders, and assisting patients with impaired motor control or cognitive ability.

Using Graphs to Perform Effective Sensor-Based Human Activity Recognition in Smart Homes.

There has been a resurgence of applications focused on human activity recognition (HAR) in smart homes, especially in the field of ambient intelligence and assisted-living technologies. However, such applications present numerous significant challenges to any automated analysis system operating in the real world, such as variability, sparsity, and noise in sensor measurements. Although state-of-the-art HAR systems have made considerable strides in addressing some of these challenges, they suffer from a practical limitation: they require successful pre-segmentation of continuous sensor data streams prior to automated recognition, i.e., they assume that an oracle is present during deployment, and that it is capable of identifying time windows of interest across discrete sensor events. To overcome this limitation, we propose a novel graph-guided neural network approach that performs activity recognition by learning explicit co-firing relationships between sensors. We accomplish this by learning a more expressive graph structure representing the sensor network in a smart home in a data-driven manner. Our approach maps discrete input sensor measurements to a feature space through the application of attention mechanisms and hierarchical pooling of node embeddings. We demonstrate the effectiveness of our proposed approach by conducting several experiments on CASAS datasets, showing that the resulting graph-guided neural network outperforms the state-of-the-art method for HAR in smart homes across multiple datasets and by large margins. These results are promising because they push HAR for smart homes closer to real-world applications.

A Survey of Visual Transformers.

Transformer, an attention-based encoder-decoder model, has already revolutionized the field of natural language processing (NLP). Inspired by such significant achievements, some pioneering works have recently been done on employing Transformer-liked architectures in the computer vision (CV) field, which have demonstrated their effectiveness on three fundamental CV tasks (classification, detection, and segmentation) as well as multiple sensory data stream (images, point clouds, and vision-language data). Because of their competitive modeling capabilities, the visual Transformers have achieved impressive performance improvements over multiple benchmarks as compared with modern convolution neural networks (CNNs). In this survey, we have reviewed over 100 of different visual Transformers comprehensively according to three fundamental CV tasks and different data stream types, where taxonomy is proposed to organize the representative methods according to their motivations, structures, and application scenarios. Because of their differences on training settings and dedicated vision tasks, we have also evaluated and compared all these existing visual Transformers under different configurations. Furthermore, we have revealed a series of essential but unexploited aspects that may empower such visual Transformers to stand out from numerous architectures, e.g., slack high-level semantic embeddings to bridge the gap between the visual Transformers and the sequential ones. Finally, two promising research directions are suggested for future investment. We will continue to update the latest articles and their released source codes at https://github.com/liuyang-ict/awesome-visual-transformers.

Toward Human-Like Representation Learning for Cognitive Architectures

Human-like learning includes an ability to learn concepts from a stream of embodiment sensor data. Echoing previous thoughts such as those from Barsalou that cognition and perception share a common representation system, we suggest an addendum to the common model of cognition. This addendum poses a simultaneous semantic memory and perception learning that bypasses working memory, and that uses parallel processing to learn concepts apart from deliberate reasoning. The goal is to provide a general outline for how to extend a class of cognitive architectures to implement a more human-like interface between cognition and embodiment of an agent, where a critical aspect of that interface is that it is dynamic because of learning.

Exploring energy efficiency of LSTM accelerators: A parameterized architecture design for embedded FPGAs

Long Short-Term Memory Networks (LSTMs) are pivotal in on-device time series analysis for embedded systems, particularly for managing sensor data streams. Yet, their deployment on resource-constrained embedded devices presents notable challenges. In response, we introduce a novel parameterized architecture for LSTM accelerators designed explicitly for embedded Field-Programmable Gate Arrays (FPGAs). Our approach involves strategic design choices, such as employing computationally efficient activation functions and optimizing clock frequency with a pipelined Arithmetic Logic Unit (ALU). These decisions drive our architecture towards enhanced energy efficiency while maintaining adaptability across diverse application scenarios. A key feature of our architecture is its configurable parameters, which allow for tailored optimization through the optional use of Digital Signal Processor Slices for ALUs and the selective implementation of activation functions. Our empirical evaluations conducted on the Spartan-7 XC7S15 FPGA demonstrate the robustness of our methodology, achieving a 2.33× improvement in energy efficiency over previous solutions. Furthermore, our study examines the correlation between memory resource types and energy efficiency across various LSTM model sizes. Impressively, even with a 9× increase in the hidden size of the LSTM cell, our accelerator maintains an energy efficiency of 10.03 GOP/s/W, with only a minor decrease of 14.65%. However, it is critical to note that our current design is not yet optimized for larger FPGA models such as the Spartan-7 XC7S25 and XC7S50. For these models, timing constraints, rather than resource limitations, pose challenges to scaling, highlighting a potential area for future optimization.

Innovative Exploration of a Bio-Inspired Sensor Fusion Algorithm: Enhancing Micro Satellite Functionality through Touretsky's Decentralized Neural Networks

Insect-inspired sensor fusion algorithms have presented a promising avenue in the development of robust and efficient systems, owing to the insects' ability to process numerous streams of noisy sensory data. The ring attractor neural network architecture has been identified as a noteworthy model for the optimal integration of diverse insect sensors. Expanding on this, our research presents an innovative bio-inspired ring attractor neural network architecture designed to augment the performance of microsatellite attitude determination systems through the fusion of data from multiple gyroscopic sensors.Extensive simulations using a nonlinear model of the microsatellite, while incorporating specific navigational disturbances, have been conducted to ascertain the viability and effectiveness of this approach. The results obtained have been superior to those of alternative methodologies, thus highlighting the potential of our proposed bio-inspired fusion technique. The findings indicate that this approach could significantly improve the accuracy and robustness of microsatellite systems across a wide range of applications.

VO2 memristor-based frequency converter with in-situ synthesize and mix for wireless internet-of-things

Wireless internet-of-things (WIoT) with data acquisition sensors are evolving rapidly and the demand for transmission efficiency is growing rapidly. Frequency converter that synthesizes signals at different frequencies and mixes them with sensor datastreams is a key component for efficient wireless transmission. However, existing frequency converters employ separate synthesize and mix circuits with complex digital and analog circuits using complementary metal-oxide semiconductor (CMOS) devices, naturally incurring excessive latency and energy consumption. Here we report a highly uniform and calibratable VO2 memristor oscillator, based on which we build memristor-based frequency converter using 8×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ imes$$\\end{document}8 VO2 array that can realize in-situ frequency synthesize and mix with help of compact periphery circuits. We investigate the self-oscillation based on negative differential resistance of VO2 memristors and the programmability with different driving currents and calibration resistances, demonstrating capabilities of such frequency converter for in-situ frequency synthesize and mix for 2 ~ 8 channels with frequencies up to 48 kHz for low frequency transmission link. When transmitting classical sensor data (acoustic, vision and spatial) in an end-to-end WIoT experimental setup, our VO2-based memristive frequency converter presents up to 1.45× ~ 1.94× power enhancement with only 0.02 ~ 0.21 dB performance degradations compared with conventional CMOS-based frequency converter. This work highlights the potential in solving frequency converter’s speed and energy efficiency problems in WIoT using high crystalline quality epitaxially grown VO2 and calibratable VO2-based oscillator array, revealing a promising direction for next-generation WIoT system design.

PArtNNer: Platform-Agnostic Adaptive Edge-Cloud DNN Partitioning for Minimizing End-to-End Latency

The last decade has seen the emergence of Deep Neural Networks (DNNs) as the de facto algorithm for various computer vision applications. In intelligent edge devices, sensor data streams acquired by the device are processed by a DNN application running on either the edge device itself or in the cloud. However, “edge-only” and “cloud-only” execution of State-of-the-Art DNNs may not meet an application’s latency requirements due to the limited compute, memory, and energy resources in edge devices, dynamically varying bandwidth of edge-cloud connectivity networks, and temporal variations in the computational load of cloud servers. This work investigates distributed (partitioned) inference across edge devices (mobile/end device) and cloud servers to minimize end-to-end DNN inference latency. We study the impact of temporally varying operating conditions and the underlying compute and communication architecture on the decision of whether to run the inference solely on the edge, entirely in the cloud, or by partitioning the DNN model execution among the two. Leveraging the insights gained from this study and the wide variation in the capabilities of various edge platforms that run DNN inference, we propose PArtNNer , a platform-agnostic adaptive DNN partitioning algorithm that finds the optimal partitioning point in DNNs to minimize inference latency. PArtNNer can adapt to dynamic variations in communication bandwidth and cloud server load without requiring pre-characterization of underlying platforms. Experimental results for six image classification and object detection DNNs on a set of five commercial off-the-shelf compute platforms and three communication standards indicate that PArtNNer results in 10.2× and 3.2× (on average) and up to 21.1× and 6.7× improvements in end-to-end inference latency compared to execution of the DNN entirely on the edge device or entirely on a cloud server, respectively. Compared to pre-characterization-based partitioning approaches, PArtNNer converges to the optimal partitioning point 17.6× faster.

HuBar: A Visual Analytics Tool to Explore Human Behavior based on fNIRS in AR Guidance Systems.

The concept of an intelligent augmented reality (AR) assistant has significant, wide-ranging applications, with potential uses in medicine, military, and mechanics domains. Such an assistant must be able to perceive the environment and actions, reason about the environment state in relation to a given task, and seamlessly interact with the task performer. These interactions typically involve an AR headset equipped with sensors which capture video, audio, and haptic feedback. Previous works have sought to facilitate the development of intelligent AR assistants by visualizing these sensor data streams in conjunction with the assistant's perception and reasoning model outputs. However, existing visual analytics systems do not focus on user modeling or include biometric data, and are only capable of visualizing a single task session for a single performer at a time. Moreover, they typically assume a task involves linear progression from one step to the next. We propose a visual analytics system that allows users to compare performance during multiple task sessions, focusing on non-linear tasks where different step sequences can lead to success. In particular, we design visualizations for understanding user behavior through functional near-infrared spectroscopy (fNIRS) data as a proxy for perception, attention, and memory as well as corresponding motion data (acceleration, angular velocity, and gaze). We distill these insights into embedding representations that allow users to easily select groups of sessions with similar behaviors. We provide two case studies that demonstrate how to use these visualizations to gain insights about task performance using data collected during helicopter copilot training tasks. Finally, we evaluate our approach by conducting an in-depth examination of a think-aloud experiment with five domain experts.

Edge-Based Real-Time Sensor Data Processing for Anomaly Detection in Industrial IoT Applications

The Industrial Internet of Things (IIoT), which uses devices with sensors to provide real-time insights into crucial processes, has completely changed how industries function. However, there are many problems associated with the sheer volume and speed of data created in industrial environments, particularly when it comes to anomaly detection. The development of edge-based real-time sensor data processing techniques was required because traditional cloud-based solutions frequently experience latency problems and privacy issues. This study suggests a novel method for IIoT applications that focuses on processing sensor data at the edge, close to the data source, for anomaly identification. We offer real-time analysis of sensor data without the need for continuous data transfer to the cloud by utilising the processing capabilities of edge devices, such as industrial gateways and embedded systems. To find anomalies in streams of real-time sensor data, our methodology integrates data pre-processing, feature engineering, and machine learning algorithms. This strategy not only lessens the strain on the network's bandwidth but also ensures quick reaction to urgent situations, cutting downtime and boosting operational effectiveness. Proposed system has adaptive learning features that enable it to continuously adjust to altering ambient factors and sensor properties, enhancing the precision of anomaly detection over time. We provide experimental findings that show how our edge-based anomaly detection system performs well in diverse industrial situations. The results show that, while protecting data privacy and minimising latency, our methodology outperforms conventional cloud-based methods in terms of anomaly detection performance.

An Analytic Framework for Assessing Artificial Intelligence and Assistive Automation Enabled Command and Control Decision Aids for Mission Effectiveness

The U.S. Army has significant interest in operationalizing Artificial Intelligence and Assistive Automation (AI/AA) technologies on the battlefield to help collate, classify, and clarify multiple streams of situational and sensor data to provide a Commander with a clear, accurate operating picture to enable rapid and appropriate decision-making. This paper offers a methodology integrated with combat simulation output data into an analytic assessment framework. This framework helps assess AI/AA enabled Decision Aids for command and control with respect to mission effectiveness. Our methodology is demonstrated via a real-world operational vignette of an AI/AA-augmented Battalion assigned to clearing a sector of the battlefield. Results indicate that the simulated scenario with an AI/AA advantage modeled led to a higher expected mission effectiveness score.

Physical Unclonable Function-Based Lightweight and Verifiable Data Stream Transmission for Industrial IoT

The deep integration of informatization and industrialization has resulted in an increasingly close connection between supervisory control and data acquisition (SCADA) systems and the Internet, leading to a growing number of new applications in industrial IoT. The boundaries of the SCADA system are monitored by industrial smart sensors, which only have limited security protection and face severe security threats. One of the major threats is that smart sensors are vulnerable to physical attacks because they are often installed in unsafe areas far from plant protection. Under this attack, the data of sensors can be easily tampered with, which may not only mislead the system with corrupted information but also cheat the system to make incorrect decisions. Moreover, since sensors are resource-constrained physical devices, complex and expensive encryption algorithms are not applicable. In this paper, we design a lightweight industrial smart sensor data stream integrity verification scheme based on physical unclonable function (PUF) for industrial IoT, which can protect the physical security of sensors and the integrity of data streams to ensure the secure transmission of industrial smart sensor data streams. We utilize PUF, fuzzy extractor and bit selection algorithm to generate stable PUF responses. A malicious attacker cannot extract the key information through the physical attack. In addition, we design a lightweight integrity verification algorithm with efficient key updating based on lightweight cryptographic primitives, making it suitable for resource-constrained physical devices. We perform the security analysis to demonstrate the security of the scheme to known security vulnerabilities. We implement the proposed scheme and evaluate the performance of our scheme with extensive experiments. The experimental results show that the proposed scheme is efficient in secret key generation and integrity verification and is superior to existing schemes in computational and communication efficiency.

S_IDS: An efficient skyline query algorithm over incomplete data streams

The efficient processing of mass stream data has attracted wide attention in the database field. The skyline query on the sensor data stream can monitor multiple targets in real time, to avoid abnormal events such as fire and explosion, which is very useful in the practical application of sensor data monitoring. However, real-world stream data may often contain incomplete data attributes due to faulty sensing devices or imperfect data collection techniques. Skyline queries over incomplete data streams may lead to a lack of transitivity and loop domination issues. To solve the problem of the skyline query over incomplete data streams, firstly, this paper uses differential dependency rule (DD) to fill the missing attribute values of data in the incomplete data stream. Then, the hierarchical grid index (HGrid) is introduced into the field of skyline query to improve pruning efficiency. In the process of skyline calculation, this paper only keeps as few calculation results as possible for the data that may affect the result to avoid a large number of repeated calculations. Thus, S_IDS (Skyline query algorithm over Incomplete Data Stream) is proposed to query skyline results with high confidence from the incomplete data stream. Finally, by comparing with the most advanced skyline query algorithms over incomplete data streams, the correctness and efficiency of the proposed S_IDS algorithm are proved.

Collaborative Use of Sensor Networks and Cyberinfrastructure to Understand Complex Ecosystem Interactions in a Tropical Rainforest: Challenges and Lessons Learned.

Collaborations between ecosystem ecologists and engineers have led to impressive progress in developing complex models of biogeochemical fluxes in response to global climate change. Ecology and engineering iteratively inform and transform each other in these efforts. Nested data streams from local sources, adjacent networks, and remote sensing sources together magnify the capacity of ecosystem ecologists to observe systems in near real-time and address questions at temporal and spatial scales that were previously unobtainable. We describe our research experiences working in a Costa Rican rainforest ecosystem with the challenges presented by constant high humidity, 4300 mm of annual rainfall, flooding, small invertebrates entering the tiniest openings, stinging insects, and venomous snakes. Over the past two decades, we faced multiple challenges and learned from our mistakes to develop a broad program of ecosystem research at multiple levels of integration. This program involved integrated networks of diverse sensors on a series of canopy towers linked to multiple belowground soil sensor arrays that could transport sensor data streams from the forest directly to an off-site location via a fiber optic cable. In our commentary, we highlight three components of our work: (1) the eddy flux measurements using canopy towers; (2) the soil sensor arrays for measuring the spatial and temporal patterns of CO2 and O2 fluxes at the soil-atmosphere interface; and (3) focused investigations of the ecosystem impact of leaf-cutter ants as "ecosystem engineers" on carbon fluxes.

Missing Value Imputation for Multi-Attribute Sensor Data Streams via Message Propagation

Sensor data streams occur widely in various real-time applications in the context of the Internet of Things (IoT). However, sensor data streams feature missing values due to factors such as sensor failures, communication errors, or depleted batteries. Missing values can compromise the quality of real-time analytics tasks and downstream applications. Existing imputation methods either make strong assumptions about streams or have low efficiency. In this study, we aim to accurately and efficiently impute missing values in data streams that satisfy only general characteristics in order to benefit real-time applications more widely. First, we propose a message propagation imputation network (MPIN) that is able to recover the missing values of data instances in a time window. We give a theoretical analysis of why MPIN is effective. Second, we present a continuous imputation framework that consists of data update and model update mechanisms to enable MPIN to perform continuous imputation both effectively and efficiently. Extensive experiments on multiple real datasets show that MPIN can outperform the existing data imputers by wide margins and that the continuous imputation framework is efficient and accurate.

A novel WGF-LN based edge driven intelligence for wearable devices in human activity recognition

Human activity recognition (HAR) is one of the key applications of health monitoring that requires continuous use of wearable devices to track daily activities. The most efficient supervised machine learning (ML)-based approaches for predicting human activity are based on a continuous stream of sensor data. Sensor data analysis for human activity recognition using conventional algorithms and deep learning (DL) models shows promising results, but evaluating their ambiguity in decision-making is still challenging. In order to solve these issues, the paper proposes a novel Wasserstein gradient flow legonet WGF-LN-based human activity recognition system. At first, the input data is pre-processed. From the pre-processed data, the features are extracted using Haar Wavelet mother- Symlet wavelet coefficient scattering feature extraction (HS-WSFE). After that, the interest features are selected from the extracted features using (Binomial Distribution integrated-Golden Eagle Optimization) BD-GEO. The important features are then post-processed using the scatter plot matrix method. Obtained post-processing features are finally given into the WGF-LN for classifying human activities. From these experiments, the results can be obtained and showed the efficacy of the proposed model.

Occupancy estimation with environmental sensors: The possibilities and limitations

Occupancy detection and estimation in buildings paves the way to improve the utilization of lighting and HVAC systems, induce energy savings and enhance the well-being of the occupants. This paper presents a comparative study of state-of-art machine learning techniques that solve two different occupancy monitoring problems using environmental sensor data. One is the regression problem that estimates the actual count of occupants while the other is the classification problem which estimates the level of occupancy (empty, sparse, full). The results of the best performing machine learning techniques that solve both problems for the open dataset from the University of Southern Denmark, Odense are presented to compare the accuracy of both approaches and the ease of implementation. The impact of CO2, temperature, and humidity features on the occupancy count/levels and detection accuracy (occupied versus unoccupied) are studied. Comprehensive analysis with different combinations of environmental features and other free features such as time-of-day along with different sampling techniques for training and testing are performed to understand how such models can be adapted for actual deployment. Our results indicate detection accuracy between 66% to 82% for different sampling schemes; with day-based sampling showing a better performance while random sampling generically showcasing lower accuracy (66.2%). The occupancy estimation (levels or counts) has accuracy in the range of 69% to 79% for random sampling and 71% to 80% for day-based sampling. Finally, results demonstrate that models based single environmental sensor data streams do not perform as well as the models with sensor fusion.

Collecting Multi-type and Correlation-Constrained Streaming Sensor Data with Local Differential Privacy

Local differential privacy (LDP) is a promising privacy model for distributed data collection. It has been widely deployed in real-world systems (e.g. Chrome, iOS, macOS). In LDP-based mechanisms, an aggregator collects private values perturbed by each user and then analyses these values to estimate their statistics, such as frequency and mean. Most existing works focus on simple scalar value types, such as boolean and categorical values. However, with the emergence of smart sensors and Internet of Things, high-dimensional data are gaining increasing popularity. In many cases where more than one type of sensor data are collected simultaneously, correlations exist between various attributes of such data, e.g. temperature and luminance. To ensure LDP for high-dimensional data, existing solutions either partition the privacy budget ϵ among these correlated attributes or adopt sampling, both of which dilute the density of useful information and thus result in poor data utility. In this paper, we propose a relaxed LDP model, namely, univariate dominance local differential privacy (UDLDP), for high-dimensional data. We quantify the correlations between attributes and present a correlation-bounded perturbation (CBP) mechanism that optimizes the partitioning of privacy budget on each correlated attribute. Furthermore, we extend CBP to support sampling, which is a common bandwidth reduction technique in sensor networks and Internet of Things. We derive the best allocation strategy of sampling probabilities among attributes in terms of data utility, which leads to the correlation-bounded perturbation mechanism with sampling (CBPS). Finally, we discuss how to collect and leverage the correlation from real-time data stream with a by-round algorithm to enhance the utility. The performance of the proposed mechanisms is evaluated and compared with state-of-the-art LDP mechanisms on real-world and synthetic datasets.