Multi-dimensional Data Streams Research Articles

A Contemporary and Comprehensive Survey on Streaming Tensor Decomposition

Tensor decomposition has been demonstrated to be successful in a wide range of applications, from neuroscience and wireless communications to social networks. In an online setting, factorizing tensors derived from multidimensional data streams is however nontrivial due to several inherent problems of real-time stream processing. In recent years, many research efforts have been dedicated to developing online techniques for decomposing such tensors, resulting in significant advances in streaming tensor decomposition or tensor tracking. This topic is emerging and enriches the literature on tensor decomposition, particularly from the data stream analystics perspective. Thus, it is imperative to carry out an overview of tensor tracking to help researchers and practitioners understand its developments and achievements, summarise the current trends and advances, and identify challenging problems. In this article, we provide a contemporary and comprehensive survey on different types of tensor tracking techniques. We particularly categorize the state-of-the-art methods into three main groups: streaming CP decompositions, streaming Tucker decompositions, and streaming decompositions under other tensor formats (i.e., tensor-train, t-SVD, and BTD). In each group, we further divide the existing algorithms into sub-categories based on their main optimization framework and model architectures. Finally, we present several applications, research challenges, open problems, and potential directions of tensor tracking in the future.

Artificial Intelligence‐based Module Type Package‐compatible Smart Sensors in the Process Industry

AbstractImage analysis presents a set of powerful methods to receive additional information about multiphase processes. It enables the development of advanced applications for process monitoring and optimization or, so‐called, soft sensors. However, the integration of advanced smart sensor systems based on image analysis into the process control system presents a complex task. To address this challenge, a modular automation concept offers a standardized interface to integrate modules. This paper presents an integration profile as a service specification that allows a plug‐and‐measure integration of smart visual sensors into modular plants. To verify the concept, we applied it to three different use cases. At the end, we discuss open challenges in the integration of complex analysis systems with multidimensional data streams into modular plants.

Context-Aware Clustering and the Optimized Whale Optimization Algorithm: An Effective Predictive Model for the Smart Grid

For customers to participate in key peak pricing, period-of-use fees, and individualized responsiveness to demand programmes taken from multi-dimensional data flows, energy use projection and analysis must be done well. However, it is a difficult study topic to ascertain the knowledge of use of electricity as recorded in the electricity records' Multi-Dimensional Data Streams (MDDS). Context-Aware Clustering (CAC) and the Optimized Whale Optimization Algorithm were suggested by researchers as a fresh power usage knowledge finding model from the multi-dimensional data streams (MDDS) to resolve issue (OWOA). The proposed CAC-OWOA framework first performs the data cleaning to handle the noisy and null elements. The predictive features are extracted from the novel context-aware group formation algorithm using the statistical context parameters from the pre-processed MDDS electricity logs. To perform the energy consumption prediction, researchers have proposed the novel Artificial Neural Network (ANN) predictive algorithm using the bio-inspired optimization algorithm called OWOA. The OWOA is the modified algorithm of the existing WOA to overcome the problems of slow convergence speed and easily falling into the local optimal solutions. The ANN training method is used in conjunction with the suggested bio-inspired OWOA algorithm to lower error rates and boost overall prediction accuracy. The efficiency of the CAC-OWOA framework is evaluated using the publicly available smart grid electricity consumption logs. The experimental results demonstrate the effectiveness of the CAC-OWOA framework in terms of forecasting accuracy, precision, recall, and duration when compared to underlying approaches.

Panakos: Chasing the Tails for Multidimensional Data Streams

System operators are often interested in extracting different feature streams from multi-dimensional data streams; and reporting their distributions at regular intervals, including the heavy hitters that contribute to the tail portion of the feature distribution. Satisfying these requirements to increase data rates with limited resources is challenging. This paper presents the design and implementation of Panakos that makes the best use of available resources to report a given feature's distribution accurately, its tail contributors, and other stream statistics (e.g., cardinality, entropy, etc.). Our key idea is to leverage the skewness inherent to most feature streams in the real world. We leverage this skewness by disentangling the feature stream into hot, warm, and cold items based on their feature values. We then use different data structures for tracking objects in each category. Panakos provides solid theoretical guarantees and achieves high performance for various tasks. We have implemented Panakos on both software and hardware and compared Panakos to other state-of-the-art sketches using synthetic and real-world datasets. The experimental results demonstrate that Panakos often achieves one order of magnitude better accuracy than the state-of-the-art solutions for a given memory budget.

CAC-WOA: context aware clustering with whale optimization algorithm for knowledge discovery from multidimensional space in electricity application

Energy consumption forecasting is a hot field of research; despite the number of developed models, projecting electric consumption in residential buildings remains problematic owing to the significant unpredictability of occupant energy use behavior. Discovering the electricity consumption knowledge from the multi-dimensional data streams (MDDS) of electricity logs is a challenging research problem. We propose a novel electricity knowledge discovery model proposed from the MDDS using clustering and machine learning. Context-aware clustering with whale optimization algorithm (CAC-WOA) is proposed to discover the predictive features from the electricity MDDS and perform the predictions using WOA. The CAC-WOA consists of two phases context-aware group formation and a WOA-based machine learning predictive model. In the CAC algorithm, group formation using electricity contextual information to estimate the robust predictive features are proposed. Using such predictive features, the predictive model using the WOA-based artificial neural network (ANN) is built. The modified ANN technique using the WOA algorithm is used to reduce the error rates and improve the prediction accuracy. The experimental outcomes using publicly available electricity consumption datasets prove the efficiency of the CAC-WOA model. Overall prediction accuracy is improved by 3.27% and prediction time is reduced by 11.31% using CAC-WOA compared state-of-the-art solutions.

An Overview of Wearable Biosensor Systems for Real-Time Substance Use Detection

Wearable biosensors represent an opportunity to improve treatment and research into a variety of diseases, including substance use disorder. They provide continuous, real-time data about the wearer’s condition in their natural environment in an unobtrusive, increasingly capable, and cost-effective way. However, generating clinically relevant insights from high-velocity, noisy, multidimensional data streams requires new approaches in real-time anomaly machine learning (ML). We present a survey of the existing algorithms for substance use monitoring in wearable biosensor data streams and how the advent of 5G and 6G wireless communications will drive further changes in this field. Our work highlights trends that have emerged among the different efforts published to-date as well as identifying ongoing challenges not adequately addressed by existing ML algorithms.

A Machine Learning Approach to Anomaly Detection Based on Traffic Monitoring for Secure Blockchain Networking

While blockchain technology provides strong cryptographic protection on the ledger and the system operations, the underlying blockchain networking remains vulnerable due to potential threats such as denial of service (DoS), Eclipse, spoofing, and Sybil attacks. Effectively detecting such malicious events should thus be an essential task for securing blockchain networks and services. Due to its importance, several studies investigated anomaly detection in Bitcoin and blockchain networks, but their analyses mainly focused on the blockchain ledger in the application context (e.g., transactions) and targets specific types of attacks (e.g., double-spending, deanonymization, etc). In this study, we present a security mechanism based on the analysis of blockchain network traffic statistics (rather than ledger data) to detect malicious events, through the functions of data collection and anomaly detection. The data collection engine senses the underlying blockchain traffic and generates multi-dimensional data streams in a periodic, real-time manner. The anomaly detection engine then detects anomalies from the created data instances based on semi-supervised learning, which is capable of detecting previously unseen patterns, and we introduce our profiling-based detection engine implemented on top of AutoEncoder (AE). Our experimental results evaluated with real and simulated traffic data support the effectiveness of our security mechanism and design choices based on the AE structure, with the approximate detection performance to the supervised learning methods only through the profiling of normal instances. The measured time complexity is sufficiently cheap to perform real-time analysis, with less than 1.4 msec for per-instance testing on a single core setting.

Enabling efficient and general subpopulation analytics in multidimensional data streams

Today's large-scale services ( e.g. , video streaming platforms, data centers, sensor grids) need diverse real-time summary statistics across multiple subpopulations of multidimensional datasets. However, state-of-the-art frameworks do not offer general and accurate analytics in real time at reasonable costs. The root cause is the combinatorial explosion of data subpopulations and the diversity of summary statistics we need to monitor simultaneously. We present Hydra, an efficient framework for multidimensional analytics that presents a novel combination of using a "sketch of sketches" to avoid the overhead of monitoring exponentially-many subpopulations and universal sketching to ensure accurate estimates for multiple statistics. We build Hydra as an Apache Spark plugin and address practical system challenges to minimize overheads at scale. Across multiple real-world and synthetic multidimensional datasets, we show that Hydra can achieve robust error bounds and is an order of magnitude more efficient in terms of operational cost and memory footprint than existing frameworks (e.g., Spark, Druid) while ensuring interactive estimation times.

Box queries over multi-dimensional streams

Answering statistical queries about streams of online arriving data is becoming increasingly important. Often, such data includes multiple-attributes, so data elements can be viewed as points in a multi-dimensional universe. This paper extends existing works on streaming algorithms by studying the ability to perform box queries on online multi-dimensional data streams. We develop three algorithms C-DARQ, DARQ and MARQ that support such capabilities for a large number of statistical functions including (but not limited to) counting, frequency estimation, heavy-hitters etc. We also apply our algorithms in distributed settings, in which measurements are recorded independently by multiple sites (e.g., multiple routers), and the goal is to obtain a global network analysis. The protocols are analyzed and evaluated over synthetic dataset, Chicago dataset, and a Facebook dataset from Kaggle in multiple dimensions (up to 10). Our algorithms asymptotically improve the space bounds as well as update and query performance of existing works. Unlike known approaches, our algorithms can also be used to solve a larger class of problems beyond counting. We further discuss extending our work to the sliding window model and when the dimensions’ bounds are a-priori unknown.

BBoxDB streams: scalable processing of multi-dimensional data streams

BBoxDB Streams is a distributed stream processing system, which allows the handling of multi-dimensional data. Multi-dimensional streams consist of n-dimensional elements, such as position data (e.g., two-dimensional positions of cars or three-dimensional positions of aircraft). The software is an enhancement of BBoxDB, a distributed key-bounding-box-value store that allows the handling of n-dimensional big data. BBoxDB Streams supports continuous range queries and continuous spatial joins; n-dimensional point and non-point data are supported. Operations in BBoxDB Streams are performed primarily on the bounding boxes of the data. With user-defined filters (UDFs), custom data formats can be decoded, and the bounding box-based operations are refined (e.g., a UDF decodes and performs intersection tests on the real geometries of WKT encoded stream elements). A unique feature of BBoxDB Streams is the ability to perform continuous spatial joins between stream elements and previously stored multi-dimensional big data. For example, the dynamic position of a car can be efficiently joined with the static spatial data of a street network.

Change detection in parametric multivariate dynamic data streams using the ARMAX-GARCH model

Dynamic data detection is one of the main concerns in the statistical process control (SPC) field. Here we focus on monitoring parametric multivariate dynamic data streams using the ARMAX-GARCH model, which reflects both the influence of exogenous variables on the mean vector and the heterogeneity of the covariance matrix. A quasi maximum likelihood estimator is used to estimate the parameter vector of a dynamic process, and a top-r control scheme is proposed to monitor the parameters of multi-dimensional data streams. Finally, a real-data example of monitoring landslide illustrates the superiorities of the proposed scheme.

Safety critical event prediction through unified analysis of driver and vehicle volatilities: Application of deep learning methods

Transportation safety is highly correlated with driving behavior, especially human error playing a key role in a large portion of crashes. Modern instrumentation and computational resources allow for the monitorization of driver, vehicle, and roadway/environment to extract leading indicators of crashes from multi-dimensional data streams. To quantify variations that are beyond normal in driver behavior and vehicle kinematics, the concept of volatility is applied. The study measures driver-vehicle volatilities using the naturalistic driving data. By integrating and fusing multiple real-time streams of data, i.e., driver distraction, vehicular movements and kinematics, and instability in driving, this study aims to predict occurrence of safety critical events and generate appropriate feedback to drivers and surrounding vehicles. The naturalistic driving data is used which contains 7566 normal driving events, and 1315 severe events (i.e., crash and near-crash), vehicle kinematics, and driver behavior collected from more than 3500 drivers. In order to capture the local dependency and volatility in time-series data 1D-Convolutional Neural Network (1D-CNN), Long Short-Term Memory (LSTM), and 1DCNN-LSTM are applied. Vehicle kinematics, driving volatility, and impaired driving (in terms of distraction) are used as the input parameters. The results reveal that the 1DCNN-LSTM model provides the best performance, with 95.45% accuracy and prediction of 73.4% of crashes with a precision of 95.67%. Additional features are extracted with the CNN layers and temporal dependency between observations is addressed, which helps the network learn driving patterns and volatile behavior. The model can be used to monitor driving behavior in real-time and provide warnings and alerts to drivers in low-level automated vehicles, reducing their crash risk.

The Titan Control Center for Industrial DevOps analytics research

<h2>Abstract</h2> The Titan Control Center is a software platform supporting research on industrial big data analytics. Building upon a scalable and extensible architecture, the Titan Control Center analyzes and visualizes data streams from Internet of Things sensors in industrial production. It performs different types of aggregations, correlation, forecasting, and anomaly detection to provide deeper insights into industrial production data for enabling Industrial DevOps. Furthermore, the Titan Control Center is used in research for implementing and evaluating novel approaches on multi-dimensional sensor data stream aggregation, as a reference platform for benchmarking scalability in modular analytics software, and for research on analyzing industrial energy consumption.

An Intelligent Smart Ranked Feature Construction Analysis based on Multi-dimensional Data Streams

An Intelligent Smart Ranked Feature Construction Analysis based on Multi-dimensional Data Streams

Thumbnail Tensor-A Method for Multidimensional Data Streams Clustering with an Efficient Tensor Subspace Model in the Scale-Space.

In this paper an efficient method for signal change detection in multidimensional data streams is proposed. A novel tensor model is suggested for input signal representation and analysis. The model is built from a part of the multidimensional stream by construction of the representing orthogonal tensor subspaces, computed with the higher-order singular value decomposition (HOSVD). Parts of the input data stream from successive time windows are then compared with the model, which is either updated or rebuilt, depending on the result of the proposed statistical inference rule. Due to processing of the input signal tensor in the scale-space, the thumbnail like output is obtained. Because of this, the method is called a thumbnail tensor. The method was experimentally verified on annotated video databases and on real underwater sequences. The results show a significant improvement over other methods both in terms of accuracy as well as in speed of operation time.

Error-space representations for multi-dimensional data streams with temporal dependence

In many application scenarios, data points are not only temporally dependent, but also expected in the form of a fast-moving stream. A broad selection of efficient learning algorithms exists which may be applied to data streams, but they typically do not take into account the temporal nature of the data. We motivate and design a method which creates an efficient representation of a data stream, where temporal information is embedded into each instance via the error space of forecasting models. Unlike many other methods in the literature, our approach can be rapidly initialized and does not require iterations over the full data sequence, thus it is suitable for a streaming scenario. This allows the application of off-the-shelf data-stream methods, depending on the application domain. In this paper, we investigate classification. We compare to a large variety of methods (auto-encoders, HMMs, basis functions, clustering methodologies, and PCA) and find that our proposed methods perform very competitively, and offers much promise for future work.

Smart building creation in large scale HVAC environments through automated fault detection and diagnosis

Modernization and retrofitting of older buildings has created a drive to install Building Energy Management Systems (BEMS) that can assist building managers in paving the way for smarter energy use and indirectly, using appropriate methods, occupant comfort understanding. BEMS may discover problems that can inform managers of building maintenance and energy wastage issues and in-directly, via repetitive data patterns appreciate user comfort requirements. The main focus of this paper is to describe a method to detect faulty Heating, Ventilation and Air-Conditioning (HVAC) Terminal Unit (TU) and diagnose them in an automatic and remote manner. For this purpose, a typical big-data framework has been constructed to process the very large volume of data. A novel feature extraction method encouraged by Proportional Integral Derivative (PID) controller has been proposed to describe events from multidimensional TU data streams. These features are further used to categorize different TU behaviours using unsupervised data-driven strategy and supervised learning is applied to diagnose faults. X-Means clustering has been performed to group diverse TU behaviours which are experimented on daily, weekly, monthly and randomly selected dataset. Subsequently, Multi-Class Support Vector Machine (MC-SVM) has been employed based on categorical information to generate an automated fault detection and diagnosis system towards making the building smarter. The clustering and classification results further compared with well-known and established algorithms and validated through statistical measurements.

Multidimensional surrogate stability to detect data stream concept drift

Abstract Concept drift detection plays a very important role in the context of data streams. It allows to point out data behavior modifications along time, which are intrinsically associated to the phenomena responsible for producing such sequences of observations. By detecting such modifications, one can better understand those phenomena and take better decisions in different application domains, e.g. stock market, climate change, population growth, etc. Besides several proposals, most of the studies lack in formal guarantees to ensure the concept drift detection. More recently, Vallim and Mello proposed 1DFT (Unidimensional Fourier Transform), an algorithm that detects drifts on unidimensional streams while holding a stability property based on surrogate series. Motivated by their work we here propose the multidimensional surrogate stability concept, which extends their approach to multidimensional data streams. In addition, our approach, named MDFT (Multidimensional Fourier Transform), employs a different and more robust measurement to analyze drifts, which is based on the Shannon’s and Von Neumann’s Entropies to quantify variations in data spaces. As final contribution, MDFT allows unidimensional streams to be reconstructed in phase spaces so their data dependencies can also be analyzed to take conclusions on concept drifts along time. Experiments considered seven 120,000-observation synthetic data streams. Synthetic data was taken into account as it allows us to define the exact points of change, using the largest Lyapunov exponent, for which our approach should trigger the concept drift events. Experiments compared MDFT against the main algorithms to detect concept drift in the context of Machine Learning (Page-Hinkley Test – PHT, Adaptive Windowing – ADWIN, and Cumulative Sum Control Chart – CUSUM) and Dynamical Systems (Recurrence Quantification Analysis using different measurements – RQA, and Permutation Entropy – PE). Results confirm MDFT outperforms the other algorithms in terms of an average measurement (using the Euclidean distance) based on: the Missed Detection Rate (MDR), the Mean Time for Detection (MTD) and the Mean Time between False Alarms (MTFA).

Adaptive fuzzy pattern classification for the online detection of driver lane change intention

In this paper we introduce a new fuzzy system using adaptive fuzzy pattern classification (AFPC) for data-based online evolvement. The fuzzy pattern concept represents an efficient tool for handling uncertainty in multi-dimensional data streams and combines powerful performance, flexibility and meaningful interpretability within one consistent framework. We outline AFPC for non-linear, multi-dimensional transition processes, namely, for the identification of lane change intention in car driving. While lane changes are rare, they are highly safety-relevant transition processes, showing high fuzziness and large individual and inter-individual variations (e.g., in lane change duration). The method employs a combined knowledge- and data-based approach, and the underlying fuzzy potential membership function concept models expert knowledge, closely mirroring human cognition. The design of AFPC comprises (I) an initial training phase (off-line and supervised), which generates a meaningful start-classifier, (II) an online application phase, and finally (III) an evolvement phase (online and unsupervised). Here we consider parametric and structural adaptations and discuss prospects and future challenges. Furthermore, we present specific modeling results for such online data from a real driving study. Next-generation advanced driver assistance systems, as well as autonomously driven vehicles need to evolve, in terms of parameters and structure, based on online real-time data. AFPC presents an efficient tool for application in this area and others (e.g., medicine).

Supervised Semi-Automated Data Analysis Software for Gas Chromatography / Differential Mobility Spectrometry (GC/DMS) Metabolomics Applications.

Modern differential mobility spectrometers (DMS) produce complex and multi-dimensional data streams that allow for near-real-time or post-hoc chemical detection for a variety of applications. An active area of interest for this technology is metabolite monitoring for biological applications, and these data sets regularly have unique technical and data analysis end user requirements. While there are initial publications on how investigators have individually processed and analyzed their DMS metabolomic data, there are no user-ready commercial or open source software packages that are easily used for this purpose. We have created custom software uniquely suited to analyze gas chromatograph / differential mobility spectrometry (GC/DMS) data from biological sources. Here we explain the implementation of the software, describe the user features that are available, and provide an example of how this software functions using a previously-published data set. The software is compatible with many commercial or home-made DMS systems. Because the software is versatile, it can also potentially be used for other similarly structured data sets, such as GC/GC and other IMS modalities.