Streaming Data Sources Research Articles

An optimized isolation forest based intrusion detection system for heterogeneous and streaming data in the industrial Internet of Things (IIoT) networks

While conventional Intrusion Detection Systems (IDS) are essential for defending against intruders in the Industrial Internet of Things (IIoT), handling data from heterogeneous and streaming data sources should receive more attention. This work introduces a novel Optimized IForest-based Intrusion Detection System (OIFIDS) which is designed to handle both heterogeneous and streaming data efficiently. The suggested approach employs a collection of optimized binary trees, each of which is trained on a distinctive subset of data, and in which the location of empty leaves determines the anomaly score assigned to a certain data point. Optimizing isolation Forest (iForest) utilizing a modified version of the Harris Hawks Optimization algorithm, which exploits both Exploration factor and Random walk strategies (ERHHO) decreases the dataset's dimension, decreases its learning time, and enhances the detection precision, accuracy, F1-score, FPR, and recall. To demonstrate how effective the proposed approach is, it is evaluated using three datasets: CICIDS-2018, NSL-KDD, and UNSW-NB15. The experimental results prove the ability of the suggested approach in handling both heterogeneous and streaming data efficiently and delivering results that were comparable to the cutting-edge baseline techniques. Moreover, it performs effectively when there are no anomalies in the training sample and when dealing with challenging scenarios with several irrelevant features and high dimensions. Based on the comparison with various state-of-the-art IDSs, the suggested approach is able to detect intrusion with greater accuracies of 95.6%, 94.8%, and 99% than the other approaches on the NSL-KDD, UNSW-NB15, and CICIDS-2018 datasets, respectively. Experiments on heterogenous data reveal that Area Under the ROC Curve (AUC) of OIFIDS beats the baseline approach for UNSW-NB15 dataset and is higher than the second-best method by 8% and 2.4% for NSL-KDD and CICIDS-2018 respectively. Evaluating the proposed system on streaming data illustrates that it can address the concept drift problem well with high AUC value of 0.948, 0.97, and 0.922 on the NSL-KDD, CICIDS-2018, and UNSW-NB15 datasets, respectively.

A mediation system for continuous spatial queries on a unified schema using Apache Spark

ABSTRACT Recent advances in big and streaming data systems have enabled real-time analysis of data generated by Internet of Things (IoT) systems and sensors in various domains. In this context, many applications require integrating data from several heterogeneous sources, either stream or static sources. Frameworks such as Apache Spark are able to integrate and process large datasets from different sources. However, these frameworks are hard to use when the data sources are heterogeneous and numerous. To address this issue, we propose a system based on mediation techniques for integrating stream and static data sources. The integration process of our system consists of three main steps: configuration, query expression and query execution. In the configuration step, an administrator designs a mediated schema and defines mapping between the mediated schema and local data sources. In the query expression step, users express queries using customized SQL grammar on the mediated schema. Finally, our system rewrites the query into an optimized Spark application and submits the application to a Spark cluster. The results are continuously returned to users. Our experiments show that our optimizations can improve query execution time by up to one order of magnitude, making complex streaming and spatial data analysis more accessible.

Integrating Heterogeneous Stream and Historical Data Sources using SQL

Applications capable of integrating data from historical and streaming sources can make the most contextualized and enriched decision-making. However, the complexity of data integration over heterogeneous data sources can be a hard task for querying in this context. Approaches that facilitate data integration, abstracting details and formats of the primary sources can meet these needs. This work presents a framework that allows the integration of streaming and historical data in real-time, abstracting syntactic aspects of queries through the use of SQL as a standard language for querying heterogeneous sources. The framework was evaluated through an experiment using relational datasets and real data produced by sensors. The results point to the feasibility of the approach.

Instance exploitation for learning temporary concepts from sparsely labeled drifting data streams

Continual learning from streaming data sources becomes more and more popular due to the increasing number of online tools and systems. Dealing with dynamic and everlasting problems poses new challenges for which traditional batch-based offline algorithms turn out to be insufficient in terms of computational time and predictive performance. One of the most crucial limitations is that we cannot assume having an access to a finite and complete data set – we always have to be ready for new data that may complement our model. This poses a critical problem of providing labels for potentially unbounded streams. In real world, we are forced to deal with very strict budget limitations, therefore, we will most likely face the scarcity of annotated instances, which are essential in supervised learning. In our work, we emphasize this problem and propose a novel instance exploitation technique. We show that when: (i) data is characterized by temporary non-stationary concepts, and (ii) there are very few labels spanned across a long time horizon, it is actually better to risk overfitting and adapt models more aggressively by exploiting the only labeled instances we have, instead of sticking to a standard learning mode and suffering from severe underfitting. We present different strategies and configurations for our methods, as well as an ensemble algorithm that attempts to maintain a sweet spot between risky and normal adaptation. Finally, we conduct a complex in-depth comparative analysis of our methods, using state-of-the-art streaming algorithms relevant for the given problem.

Resource scheduling and provisioning for processing of dynamic stream workflows under latency constraints

There has been an increasing demand for large-scale processing of stream workflows. Many streaming data sources, such as video and audio, differ in terms of the type and density of information they contain over time, which require dynamic workflow structures where paths change in response to data variations. Additionally, information is often only inferred from examining discrete chunks and hence, cannot be arbitrarily parallelised like other streaming data sources. Stream processes using these data sources often operate as dynamic workflows, where different paths are selected based on prior processing information, relating to differences in the nature of the data itself. To better enable low latency analysis in such scenarios, this work investigates strategies to schedule and provision streaming dynamic workflow scenarios of various complexities and degrees of unpredictability. It is found that, as scenarios become increasingly uncertain and unpredictable, it is better to use simpler strategies which do not make assumptions about workflow structures or completion times.

An Anthropocentric and Enhanced Predictive Approach to Smart City Management

Cities are becoming increasingly complex to manage, as they increase in size and must provide higher living standards for their populations. New technology-based solutions must be developed towards attending this growth and ensuring that it is socially sustainable. This paper puts forward the notion that these solutions must share some properties: they should be anthropocentric, holistic, horizontal, multi-dimensional, multi-modal, and predictive. We propose an architecture in which streaming data sources that characterize the city context are used to feed a real-time graph of the city’s assets and states, as well as to train predictive models that hint into near future states of the city. This allows human decision-makers and automated services to take decisions, both for the present and for the future. To achieve this, multiple data sources about a city were gradually connected to a message broker, that enables increasingly rich decision-support. Results show that it is possible to predict future states of a city, in aspects such as traffic, air pollution, and other ambient variables. The key innovative aspect of this work is that, as opposed to the majority of existing approaches which focus on a real-time view of the city, we also provide insights into the near-future state of the city, thus allowing city services to plan ahead and adapt accordingly. The main goal is to optimize decision-making by anticipating future states of the city and make decisions accordingly.

Time Series Forecasting of Univariate Agrometeorological Data: A Comparative Performance Evaluation via One-Step and Multi-Step Ahead Forecasting Strategies.

High-frequency monitoring of agrometeorological parameters is quintessential in the domain of Precision Agriculture (PA), where timeliness of collected observations and the ability to generate ahead-of-time predictions can substantially impact the crop yield. In this context, state-of-the-art internet-of-things (IoT)-based sensing platforms are often employed to generate, pre-process and assimilate real-time data from heterogeneous sensors and streaming data sources. Simultaneously, Time-Series Forecasting Algorithms (TSFAs) are responsible for generating reliable forecasts with a pre-defined forecast horizon and confidence. These TSFAs often rely on modelling the correlation between endogenous variables, the impact of exogenous variables on latent form and structural properties of data such as autocorrelation, periodicity, trend, pattern, and causality to approximate the model parameters. Traditionally, TSFAs such as the Holt–Winters (HW) and Autoregressive family of models (ARIMA) apply a linear and parametric approach towards model approximation, whilst models like Support Vector Regression (SVRs) and Neural Networks (NNs) adhere to a non-linear, non-parametric approach for modelling the historical data. Recently, Deep-Learning-based TSFAs such as Recurrent Neural Networks (RNNs), and Long-Short-Term-Memory (LSTMS) have gained popularity due to their capability to model long sequences of highly non-linear and stochastic data effectively. However, the evolution of TSFAs for predicting agrometeorological parameters pivots around one-step-ahead forecasting, which often overestimates the performance metrics defined for validating forecast capabilities of potential TSFAs. Hence, this paper attempts to evaluate and compare the performance of different machine learning (ML) and deep learning (DL) based TSFAs under one-step and multi-step-ahead forecast scenarios, thereby estimating the generalization capabilities of TSFA models over unseen data. The data used in this study are collected from an Automatic Weather Station (AWS), sampled at an interval of 15 min, and range over one month. Temperature (T) and Humidity (H) observations from the AWS are further converted into univariate, supervised time-series diurnal data profiles. Finally, walk-forward validation is used to evaluate recursive one-step-ahead forecasts until the desired prediction horizon is achieved. The results show that the Seasonal Auto-Regressive Integrated Moving Average (SARIMA) and SVR models outperform their DL-based counterparts in one-step and multi-step ahead settings with a fixed forecast horizon. This work aims to present a baseline comparison between different TSFAs to assist the process of model selection and facilitate rapid ahead-of-time forecasting for end-user applications.

Correct and stable sorting for overflow streaming data with a limited storage size and a uniprocessor.

Tremendous quantities of numeric data have been generated as streams in various cyber ecosystems. Sorting is one of the most fundamental operations to gain knowledge from data. However, due to size restrictions of data storage which includes storage inside and outside CPU with respect to the massive streaming data sources, data can obviously overflow the storage. Consequently, all classic sorting algorithms of the past are incapable of obtaining a correct sorted sequence because data to be sorted cannot be totally stored in the data storage. This paper proposes a new sorting algorithm called streaming data sort for streaming data on a uniprocessor constrained by a limited storage size and the correctness of the sorted order. Data continuously flow into the storage as consecutive chunks with chunk sizes less than the storage size. A theoretical analysis of the space bound and the time complexity is provided. The sorting time complexity is O (n), where n is the number of incoming data. The space complexity is O (M), where M is the storage size. The experimental results show that streaming data sort can handle a million permuted data by using a storage whose size is set as low as 35% of the data size. This proposed concept can be practically applied to various applications in different fields where the data always overflow the working storage and sorting process is needed.

Using Machine Learning Techniques to Aid Environmental Policy Analysis

For a growing class of prediction problems, big data and machine learning (ML) analyses can greatly enhance our understanding of the effectiveness of public investments and public policy. However, the outputs of many ML models are often abstract and inaccessible to policy communities or the general public. In this article, we describe a hands-on teaching case that is suitable for use in a graduate or advanced undergraduate public policy, public affairs, or environmental studies classroom. Students will engage on the use of increasingly popular ML classification algorithms and cloud-based data visualization tools to support policy and planning on the theme of electric vehicle mobility and connected infrastructure. By using these tools, students will critically evaluate and convert large and complex data sets into human understandable visualization for communication and decision making. The tools also enable user flexibility to engage with streaming data sources in a new creative design with little technical background.

Real Time Anomaly Detection Techniques Using PySpark Frame Work

The identification of anomaly in a network is a process of observing keenly the minute behavioral changes from the usual pattern followed. These are often referred with different names malware, exceptions, and anomaly or as outlier according to the dominion of the application. Though many works have emerged for the detection of the outlier, the identification of the abnormality in the multiple source data stream structure is still under research. To identify the abnormalities in the cloud data center that is encompassed with the multiple-source VMWare, by observing the behavioral changes in the load of the CPU, utilization of the memory etc. consistently the paper has developed a real time identification process. The procedure followed utilizes the PySpark to compute the batches of data and make predictions, with minimized delay. Further a flat-increment based clustering is used to frame the normal attributes in the PySpark Structure. The latencies in computing the tuple while clustering and predicting, was compared for PySpark, Storm and other dispersed structure that were used in processing the batches of data and was experimentally found that the processing time of tuple in a PySpark was much lesser compared to the other methods.

A fast stream transaction system for real-time IoT applications

Due to the recent prevalence of IoT (Internet of Things) technologies, various IoT devices connect to the Internet and continuously send their generated data to remote processing computers such as video data or sensor data. The transaction rate is one of the main factors to improve the performance of some IoT applications. For instance, in surveillance systems, the probability to catch a thief increases as the processing computer analyzes the video with a higher transaction rate. To improve the transaction rate, some methods reduce the transaction time between a processing computer and stream data sources under a static transaction interval. However, the transaction rate can be further improved by changing the transaction interval dynamically depending on the transaction time. In this paper, we propose a method to improve the transaction rate by changing the transaction interval dynamically. In our proposed method, a processing computer sometimes changes the transaction interval to be the same length as the average transaction time. Moreover, our proposed method adopts a progressive quality improvement (PQI) approach to reduce the transaction time. We measured the transaction rate of our proposed method by both a simulator and an implemented system. We confirmed that our proposed method can improve the transaction rate by 4.4 times and the transaction time by 21% at least compared with the conventional method. Moreover, we confirmed that the average frame rate increases 22% compared with a simple method in a real situation.

Review of Big Data and Processing Frameworks for Disaster Response Applications

Natural hazards result in devastating losses in human life, environmental assets and personal, and regional and national economies. The availability of different big data such as satellite imageries, Global Positioning System (GPS) traces, mobile Call Detail Records (CDRs), social media posts, etc., in conjunction with advances in data analytic techniques (e.g., data mining and big data processing, machine learning and deep learning) can facilitate the extraction of geospatial information that is critical for rapid and effective disaster response. However, disaster response systems development usually requires the integration of data from different sources (streaming data sources and data sources at rest) with different characteristics and types, which consequently have different processing needs. Deciding which processing framework to use for a specific big data to perform a given task is usually a challenge for researchers from the disaster management field. Therefore, this paper contributes in four aspects. Firstly, potential big data sources are described and characterized. Secondly, the big data processing frameworks are characterized and grouped based on the sources of data they handle. Then, a short description of each big data processing framework is provided and a comparison of processing frameworks in each group is carried out considering the main aspects such as computing cluster architecture, data flow, data processing model, fault-tolerance, scalability, latency, back-pressure mechanism, programming languages, and support for machine learning libraries, which are related to specific processing needs. Finally, a link between big data and processing frameworks is established, based on the processing provisioning for essential tasks in the response phase of disaster management.

A multimodal and hybrid deep neural network model for Remaining Useful Life estimation

Aging critical infrastructures and valuable machineries together with recent catastrophic incidents such as the collapse of Morandi bridge calls for an urgent quest to design advanced and innovative data-driven solutions and efficiently incorporate multi-sensor streaming data sources for condition-based maintenance. Remaining Useful Life (RUL) is a crucial measure used in this regard within manufacturing and industrial systems, and its accurate estimation enables improved decision-making for operations and maintenance. Capitalizing on the recent success of multiple-model (also referred to as hybrid or mixture of experts) deep learning techniques, the paper proposes a hybrid deep neural network framework for RUL estimation, referred to as the Hybrid Deep Neural Network Model (HDNN). The proposed HDNN framework is the first hybrid deep neural network model designed for RUL estimation that integrates two deep learning models simultaneously and in a parallel fashion. More specifically, in contrary to the majority of existing data-driven prognostic approaches for RUL estimation, which are developed based on a single deep model and can hardly maintain good generalization performance across various prognostic scenarios, the proposed HDNN framework consists of two parallel paths (one LSTM and one CNN) followed by a fully connected multilayer fusion neural network which acts as the fusion centre combining the output of the two paths to form the target RUL. The HDNN uses the LSTM path to extract temporal features while simultaneously the CNN is utilized to extract spatial features. The proposed HDNN framework is tested on the NASA commercial modular aero-propulsion system simulation (C-MAPSS) dataset. Our comprehensive experiments and comparisons with several recently proposed RUL estimation methodologies developed based on the same data-sets show that the proposed HDNN framework significantly outperforms all its counterparts in the complicated prognostic scenarios with increased number of operating conditions and fault modes.

Optimizing Data Stream Representation: An Extensive Survey on Stream Clustering Algorithms

Analyzing data streams has received considerable attention over the past decades due to the widespread usage of sensors, social media and other streaming data sources. A core research area in this field is stream clustering which aims to recognize patterns in an unordered, infinite and evolving stream of observations. Clustering can be a crucial support in decision making, since it aims for an optimized aggregated representation of a continuous data stream over time and allows to identify patterns in large and high-dimensional data. A multitude of algorithms and approaches has been developed that are able to find and maintain clusters over time in the challenging streaming scenario. This survey explores, summarizes and categorizes a total of 51 stream clustering algorithms and identifies core research threads over the past decades. In particular, it identifies categories of algorithms based on distance thresholds, density grids and statistical models as well as algorithms for high dimensional data. Furthermore, it discusses applications scenarios, available software and how to configure stream clustering algorithms. This survey is considerably more extensive than comparable studies, more up-to-date and highlights how concepts are interrelated and have been developed over time.

Machine learning and AI for long-term fault prognosis in complex manufacturing systems

Recent advances in sensors and other streaming data sources of plant floor automation and information systems open an exciting possibility to predict the risks of faults and breakdowns across a manufacturing plant over much longer time horizons than what is conceivable today. This paper introduces a Manufacturing System-wide Balanced Random Survival Forest (MBRSF), a nonparametric machine learning approach that can fuse complex dynamic dependencies underlying these data streams to provide a long-term prognosis of machine breakdowns. Experimental investigations with a 20 machine automotive manufacturing line suggest that MBRSF reduces prediction errors (Brier scores) by over 90% compared to other methods tested.

A local algorithm to approximate the global clustering of streams generated in ubiquitous sensor networks

In ubiquitous streaming data sources, such as sensor networks, clustering nodes by the data they produce gives insights on the phenomenon being monitored. However, centralized algorithms force communication and storage requirements to grow unbounded. This article presents L2GClust, an algorithm to compute local clusterings at each node as an approximation of the global clustering. L2GClust performs local clustering of the sources based on the moving average of each node’s data over time: the moving average is approximated using memory-less statistics; clustering is based on the furthest-point algorithm applied to the centroids computed by the node’s direct neighbors. Evaluation is performed both on synthetic and real sensor data, using a state-of-the-art sensor network simulator and measuring sensitivity to network size, number of clusters, cluster overlapping, and communication incompleteness. A high level of agreement was found between local and global clusterings, with special emphasis on separability agreement, while an overall robustness to incomplete communications emerged. Communication reduction was also theoretically shown, with communication ratios empirically evaluated for large networks. L2GClust is able to keep a good approximation of the global clustering, using less communication than a centralized alternative, supporting the recommendation to use local algorithms for distributed clustering of streaming data sources.

InDM2: Interactive Dynamic Multi-Objective Decision Making Using Evolutionary Algorithms

Abstract Dynamic optimization problems involving two or more conflicting objectives appear in many real-world scenarios, and more cases are expected to appear in the near future with the increasing interest in the analysis of streaming data sources in the context of Big Data applications. However, approaches combining dynamic multi-objective optimization with preference articulation are still scarce. In this paper, we propose a new dynamic multi-objective optimization algorithm called InDM2 that allows the preferences of the decision maker (DM) to be incorporated into the search process. When solving a dynamic multi-objective optimization problem with InDM2, the DM can not only express her/his preferences by means of one or more reference points (which define the desired region of interest), but these points can be also modified interactively. InDM2 is enhanced with methods to graphically display the different approximations of the region of interest obtained during the optimization process. In this way, the DM is able to inspect and change, in optimization time, the desired region of interest according to the information displayed. We describe the main features of InDM2 and detail how it is implemented. Its performance is illustrated using both synthetic and real-world dynamic multi-objective optimization problems.

JMetalSP: A framework for dynamic multi-objective big data optimization

Multi-objective metaheuristics have become popular techniques for dealing with complex optimization problems composed of a number of conflicting functions. Nowadays, we are in the Big Data era, so metaheuristics must be able to solve dynamic problems that may vary over time due to the processing and analysis of several streaming data sources. As this is a new field, there is a need for software platforms to solve dynamic multi-objective Big Data optimization problems. In this paper, we present jMetalSP, which combines the multi-objective optimization features of the jMetal framework with the streaming facilities of the Apache Spark cluster computing system. Thus, existing state-of-the-art multi-objective metaheuristics can be easily adapted to deal with dynamic optimization problems that are fed by multiple streaming data sources. Moreover, these algorithms can take advantage of the parallel computing features of Spark. We describe the architecture of jMetalSP and show how it can be used to solve a dynamic bi-objective instance of the Traveling Salesman Problem (TSP) based on New York City's real-time traffic data. We have also carried out an experimental study to assess the performance of the resultant jMetalSP application in a Hadoop cluster composed of 100 nodes.

Distributed data-driven platform for urgent decision making in cardiological ambulance control

This paper presents ongoing research aimed at developing a data-driven platform for clinical decision support systems (DSSs) that require integration and processing of various data sources within a single solution. Resource management is developed within a framework of an urgent computing approach to address changing requirements defined by the incoming flow of patients with urgent diseases. This work presents DSS for support of ambulance and emergency medical service management for patients with acute coronary syndrome (ACS) as a working example with integration distributed streaming data sources as well as data storages (containing electrocardiography (ECG) data, electronic medical records (EMR), real-time monitoring of medical facilities, and schedules of hospitals within a network). This DSS has been developed in collaboration with the Federal Almazov North-West Medical Research Centre in Saint Petersburg.

The Graph of Things: A Step Towards the Live Knowledge Graph of Connected Things

The Internet of Things (IoT) with billions of connected devices has been generating an enormous amount of data every hour. Connecting every data item generated by IoT to the rest of the digital world to turn this data into meaningful actions will create new capabilities, richer experiences, and unprecedented economic opportunities for businesses, individuals, and countries. However, providing an integrated view for exploring and querying such data at real-time is extremely challenging due to its Big Data natures: big volume, fast real-time update and messy data sources. To address this challenge, we provide a unified integrated and live view for heterogeneous IoT data sources using Linked Data, called the Graph of Things (GoT). GoT is backed by a scalable and elastic software stack to deal with billions of records of historical and static datasets in conjunction with millions of triples being fetched and enriched to connect to GoT per hour in real time. GoT makes approximately a half of million stream data sources queryable via a SPARQL endpoint and a continuous query channel that enable us to create a live explorer of GoT with just HTML and Javascript.