Analysis Of Data Streams Research Articles

PR-sketch

Computing per-key aggregation is indispensable in streaming data analysis formulated as two phases, an update phase and a recovery phase. As the size and speed of data streams rise, accurate per-key information is useful in many applications like anomaly detection, attack prevention, and online diagnosis. Even though many algorithms have been proposed for per-key aggregation in stream processing, their accuracy guarantees only cover a small portion of keys. In this paper, we aim to achieve nearly full accuracy with limited resource usage. We follow the line of sketch-based techniques. We observe that existing methods suffer from high errors for most keys. The reason is that they track keys by complicated mechanism in the update phase and simply calculate per-key aggregation from some specific counter in the recovery phase. Therefore, we present PR-Sketch, a novel sketching design to address the two limitations. PR-Sketch builds linear equations between counter values and per-key aggregations to improve accuracy, and records keys in the recovery phase to reduce resource usage in the update phase. We also provide an extension called fast PR-Sketch to improve processing rate further. We derive space complexity, time complexity, and guaranteed error probability for both PR-Sketch and fast PR-Sketch. We conduct trace-driven experiments under 100K keys and 1M items to compare our algorithms with multiple state-of-the-art methods. Results demonstrate the resource efficiency and nearly full accuracy of our algorithms.

Evaluative Review of Streaming Analytics: Tools and Technologies in Real-Time Data Processing

Nowadays, big data processing systems are evolving to be more stream-oriented; where each data record is processed as it arrives by distributed and low latency computational frameworks [18]. Data streams have been extensively used in several fields of computational analytics such as data mining, business intelligence etc. [17]. In every field, the data stream can be considered as an ordered sequence of data items, as they continuously arrive over the period. Due to this characteristic, streaming data analytics is a challenging area of research [5, 11]. This paper aims to present data stream processing as a growing research field , along with streaming analytics frameworks as a rich focus area. The paper also contributes to evaluate the efficacy of available stream analytics frameworks. One of the Industry 4.0 use case - predictive maintenance rail transportation - has been illustrated here as a case study design mapped with streaming analytics framework.

WindFlow: High-Speed Continuous Stream Processing With Parallel Building Blocks

Nowadays, we are witnessing the diffusion of Stream Processing Systems (SPSs) able to analyze data streams in near realtime. Traditional SPSs like <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Storm</small> and <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Flink</small> target distributed clusters and adopt the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">continuous streaming model</i> , where inputs are processed as soon as they are available while outputs are continuously emitted. Recently, there has been a great focus on SPSs for scale-up machines. Some of them (e.g., <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">BriskStream</small> ) still use the continuous model to achieve low latency. Others optimize throughput with batching approaches that are, however, often inadequate to minimize latency for live-streaming applications. Our contribution is to show a novel software engineering approach to design the runtime system of SPSs targeting multicores, with the aim of providing a uniform solution able to optimize throughput and latency. The approach has a formal nature based on the assembly of components called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">building blocks</i> , whose composition allows optimizations to be easily expressed in a compositional manner. We use this methodology to build a new SPS called <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">WindFlow</small> . Our evaluation showcases the benefits of <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">WindFlow</small> : it provides lower latency than SPSs for continuous streaming, and can be configured to optimize throughput, to perform similarly and even better than batch-based scale-up SPSs.

Heterogeneity-aware elastic scaling of streaming applications on cloud platforms

Rise of Big Data techniques has led to the requirement for low latency analysis of high-velocity continuous data streams in real time. Several solutions, including Stream Processing Systems (SPSs), have been developed to enable real-time distributed stream processing. However, emerging application scenarios such as smart cities and wearable assistance that involve highly variable data rates keep on posing new challenges to the established stream processing engines for maintaining cost-effective executions. To cater to such scenarios, many modern SPSs have been proposed that leverage Cloud environment. The run-time scalability incorporated in these SPSs is in their early adaptations and are based on fixed scale sizes. Moreover, these scaling approaches do not adequately consider both the structure of the hosted streaming applications and the characteristic features of the underlying Cloud environment. Achieving true cost benefits of orchestrating streaming applications on Cloud-based pay-as-you-go model while maintaining the desired QoS, necessitates that both these issues are accounted in making the scaling decisions. This work presents a heterogeneity-aware, efficient auto-scaling strategy StreamScale-H which addresses both these issues. Simulation experiments, on representative stream applications, indicate that the proposed StreamScale-H auto-scaling algorithm exhibits much better performance in comparison with the state-of-the-art algorithms.

Data stream analysis: Foundations, major tasks and tools

AbstractThe significant growth of interconnected Internet‐of‐Things (IoT) devices, the use of social networks, along with the evolution of technology in different domains, lead to a rise in the volume of data generated continuously from multiple systems. Valuable information can be derived from these evolving data streams by applying machine learning. In practice, several critical issues emerge when extracting useful knowledge from these potentially infinite data, mainly because of their evolving nature and high arrival rate which implies an inability to store them entirely. In this work, we provide a comprehensive survey that discusses the research constraints and the current state‐of‐the‐art in this vibrant framework. Moreover, we present an updated overview of the latest contributions proposed in different stream mining tasks, particularly classification, regression, clustering, and frequent patterns.This article is categorized under: Fundamental Concepts of Data and Knowledge &gt; Key Design Issues in Data Mining Fundamental Concepts of Data and Knowledge &gt; Motivation and Emergence of Data Mining

Protected Network Architecture for Ensuring Consistency of Medical Data through Validation of User Behavior and DICOM Archive Integrity

The problem of consistency of medical data in Hospital Data Management Systems is considered in the context of correctness of medical images stored in a PACS (Picture Archiving and Communication System) and legality of actions authorized users perform when accessing MIS (Medical Information System) facilities via web interfaces. The purpose of the study is to develop a SIEM-like (Security Information and Event Management) architecture for offline analysis of DICOM (Digital Imaging and Communications in Medicine) archive integrity and users’ activity. To achieve amenable accuracy when validating DICOM archive integrity, two aspects are taken into account: correctness of periodicity of the incoming data stream and correctness of the image data (time series) itself for the considered modality. Validation of users’ activity assumes application of model-driven approaches using state-of-the-art machine learning methods. This paper proposes a network architecture with guard clusters to protect sensitive components like the DICOM archive and application server of the MIS. New server roles were designed to perform traffic interception, data analysis and alert management without reconfiguration of production software components. The cluster architecture allows the analysis of incoming big data streams with high availability, providing horizontal scalability and fault tolerance. To minimize possible harm from spurious DICOM files the approach should be considered as an addition to other securing techniques like watermarking, encrypting and testing data conformance with a standard.

Improved Real-Time Natural Hazard Monitoring Using Automated DInSAR Time Series

As part of the collaborative GeoSciFramework project, we are establising a monitoring system for the Yellowstone volcanic area that integrates multiple geodetic and seismic data sets into an advanced cyber-infrastructure framework that will enable real-time streaming data analytics and machine learning and allow us to better characterize associated long- and short-term hazards. The goal is to continuously ingest both remote sensing (GNSS, DInSAR) and ground-based (seismic, thermal and gas observations, strainmeter, tiltmeter and gravity measurements) data and query and analyse them in near-real time. In this study, we focus on DInSAR data processing and the effects from using various atmospheric corrections and real-time orbits on the automated processing and results. We find that the atmospheric correction provided by the European Centre for Medium-Range Weather Forecasts (ECMWF) is currently the most optimal for automated DInSAR processing and that the use of real-time orbits is sufficient for the early-warning application in question. We show analysis of atmospheric corrections and using real-time orbits in a test case over the Kilauea volcanic area in Hawaii. Finally, using these findings, we present results of displacement time series in the Yellowstone area between May 2018 and October 2019, which are in good agreement with GNSS data where available. These results will contribute to a baseline model that will be the basis of a future early-warning system that will be continuously updated with new DInSAR data acquisitions.

Molecular function recognition by supervised projection pursuit machine learning

Identifying mechanisms that control molecular function is a significant challenge in pharmaceutical science and molecular engineering. Here, we present a novel projection pursuit recurrent neural network to identify functional mechanisms in the context of iterative supervised machine learning for discovery-based design optimization. Molecular function recognition is achieved by pairing experiments that categorize systems with digital twin molecular dynamics simulations to generate working hypotheses. Feature extraction decomposes emergent properties of a system into a complete set of basis vectors. Feature selection requires signal-to-noise, statistical significance, and clustering quality to concurrently surpass acceptance levels. Formulated as a multivariate description of differences and similarities between systems, the data-driven working hypothesis is refined by analyzing new systems prioritized by a discovery-likelihood. Utility and generality are demonstrated on several benchmarks, including the elucidation of antibiotic resistance in TEM-52 beta-lactamase. The software is freely available, enabling turnkey analysis of massive data streams found in computational biology and material science.

A Frequent Pattern Conjunction Heuristic for Rule Generation in Data Streams

This paper introduces a new and expressive algorithm for inducing descriptive rule-sets from streaming data in real-time in order to describe frequent patterns explicitly encoded in the stream. Data Stream Mining (DSM) is concerned with the automatic analysis of data streams in real-time. Rapid flows of data challenge the state-of-the art processing and communication infrastructure, hence the motivation for research and innovation into real-time algorithms that analyse data streams on-the-fly and can automatically adapt to concept drifts. To date, DSM techniques have largely focused on predictive data mining applications that aim to forecast the value of a particular target feature of unseen data instances, answering questions such as whether a credit card transaction is fraudulent or not. A real-time, expressive and descriptive Data Mining technique for streaming data has not been previously established as part of the DSM toolkit. This has motivated the work reported in this paper, which has resulted in developing and validating a Generalised Rule Induction (GRI) tool, thus producing expressive rules as explanations that can be easily understood by human analysts. The expressiveness of decision models in data streams serves the objectives of transparency, underpinning the vision of ‘explainable AI’ and yet is an area of research that has attracted less attention despite being of high practical importance. The algorithm introduced and described in this paper is termed Fast Generalised Rule Induction (FGRI). FGRI is able to induce descriptive rules incrementally for raw data from both categorical and numerical features. FGRI is able to adapt rule-sets to changes of the pattern encoded in the data stream (concept drift) on the fly as new data arrives and can thus be applied continuously in real-time. The paper also provides a theoretical, qualitative and empirical evaluation of FGRI.

A system for off-line validation of medical data in DICOM archive

The problem of consistency of medical data in Hospital Data Management Systems is considered in the context of correctness of medical images itself to minimize possible harm from spurious DICOM files. The approach should be considered as an addition to other securing techniques like watermarking, encrypting, testing conformance with the standard. To achieve amenable accuracy in practice two aspects are taken into account: correctness of periodicity and correctness of image data (time series) itself for considered modality. This paper proposes an architecture of an information system and network filter integrated to it to provide facilities for analysis and alert management. The architecture is designed to perform the analysis of incoming data streams within components working in a clustered manner, providing horizontal scalability and fault tolerance to be BigData-ready.

Data Streams Oriented Outlier Detection Method: A Fast Minimal Infrequent Pattern Mining

Outlier detection is a common method for analyzing data streams. In the existing outlier detection methods, most of methods compute distance of points to solve certain specific outlier detection problems. However, these methods are computationally expensive and cannot process data streams quickly. The outlier detection method based on pattern mining resolves the aforementioned issues, but the existing methods are inefficient and cannot meet requirements of quickly mining data streams. In order to improve the efficiency of the method, a new outlier detection method is proposed in this paper. First, a fast minimal infrequent pattern mining method is proposed to mine the minimal infrequent pattern from data streams. Second, an efficient outlier detection algorithm based on minimal infrequent pattern is proposed for detecting the outliers in the data streams by mining minimal infrequent pattern. The algorithm proposed in this paper is demonstrated by real telemetry data of a satellite in orbit. The experimental results show that the proposed method not only can be applied to satellite outlier detection, but also is superior to the existing methods.

Impact of Distance Measures on the Performance of AIS Data Clustering

Automatic Identification System (AIS) data stream analysis is based on the AIS data of different vessel’s behaviours, including the vessels’ routes. When the AIS data consists of outliers, noises, or are incomplete, then the analysis of the vessel’s behaviours is not possible or is limited. When the data consists of outliers, it is not possible to automatically assign the AIS data to a particular vessel. In this paper, a clustering method is proposed to support the AIS data analysis, to qualify noises and outliers with respect to their suitability, and finally to aid the reconstruction of the vessel’s trajectory. In this paper, clustering results have been obtained using selected algorithms, including k-means, k-medoids, and fuzzy c-means. Based on the clustering results, it is possible to decide on the qualification of data with outliers and on their usefulness in the reconstruction of the vessel trajectory. The main aim of this paper is to answer how different distance measures during a clustering process can influence AIS data clustering quality. The main core question is whether or not they have an impact on the process of reconstruction of the vessel trajectories when the data are damaged. The research question during the computational experiments asked whether or not distance measure influence AIS data clustering quality. The computational experiments have been carried out using original AIS data. In general, the experiment and the results confirm the usefulness of the cluster-based analysis when the data include outliers that are derived from the natural environment. It is also possible to monitor and to analyse AIS data using clustering when the data include outliers. The computational experiment results confirm that the k-means with Euclidean distance has the best performance.

Two‐level pruning based ensemble with abstained learners for concept drift in data streams

AbstractMining data streams for predictive analysis is one of the most interesting topics in machine learning. With the drifting data distributions, it becomes important to build adaptive systems which are dynamic and accurate. Although ensembles are powerful in improving accuracy of incremental learning, it is crucial to maintain a set of best suitable learners in the ensemble while considering the diversity between them. By adding diversity‐based pruning to the traditional accuracy‐based pruning, this paper proposes a novel concept drift handling approach named Two‐Level Pruning based Ensemble with Abstained Learners (TLP‐EnAbLe). In this approach, deferred similarity based pruning delays the removal of under performing similar learners until it is assured that they are no longer fit for prediction. The proposed scheme retains diverse learners that are well suited for current concept. Two‐level abstaining monitors performance of learners and chooses the best set of competent learners for participating in decision making. This is an enhancement to traditional majority voting system which dynamically chooses high performing learners and abstains the ones which are not suitable for prediction. In our experiments, it has been demonstrated that TLP‐EnAbLe handles concept drift more effectively than other state‐of‐the‐art algorithms on nineteen artificially drifting and ten real‐world datasets. Further, statistical tests conducted on various drift patterns which include gradual, abrupt, recurring and their combinations prove efficiency of the proposed approach.

Mapping the Big Data Landscape: Technologies, Platforms and Paradigms for Real-Time Analytics of Data Streams

The `Big Data' of yesterday is the `data' of today. As technology progresses, new challenges arise and new solutions are developed. Due to the emergence of Internet of Things applications within the last decade, the field of Data Mining has been faced with the challenge of processing and analysing data streams in real-time, and under high data throughput conditions. This is often referred to as the Velocity aspect of Big Data. Whereas there are numerous reviews on Data Stream Mining techniques and applications, there is very little work surveying Data Stream processing paradigms and associated technologies, from data collection through to pre-processing and feature processing, from the perspective of the user, not that of the service provider. In this article, we evaluate a particular type of solution, which focuses on streaming data, and processing pipelines that permit online analysis of data streams that cannot be stored as-is on the computing platform. We review foundational computational concepts such as distributed computation, fault-tolerant computing, and computational paradigms/architectures. We then review the available technological solutions, and applications that pertain to data stream mining as case studies of these theoretical concepts. We conclude with a discussion of the field of data stream processing/analytics, future directions and research challenges.

Streaming Data Analysis: Clustering or Classification?

This article is a position paper about models and algorithms that are generally called “stream clustering.” Semantics and methods used in this field are often co-opted from static clustering, but they do not serve well for streaming data analysis. Most “state-of-the-art” methods, such as sequential k-means, Birch, CluStream, DenStream, etc., acknowledge that the data are seen but once in real streaming analysis (e.g., intrusion detection, voter fraud, etc.). Interpretation of their outputs generally overlooks the fact that when the data cannot be saved, batch clustering ideas, such as preclustering assessment, partitioning, and cluster validity are not relevant. But in the current literature, the data, or some subset of it, are often saved for hindsight evaluation (we call this fake stream clustering). Our position? Useful analysis of real streaming data is in its infancy. We do not argue that current approaches to streaming clustering are wrong: rather, we regard them as transitional methods which will eventually lead to a new and useful paradigm for this type of computation. We think that this class of models and algorithms are actually classifiers, but with a special added component, viz., continuously updated cluster footprints of the instream processing. We need to carefully define the objectives of streaming analysis, and then choose terminology and methods that suit this evolving paradigm.

Fossel: Efficient Latency Reduction in Approximating Streaming Sensor Data

The volume of streaming sensor data from various environmental sensors continues to increase rapidly due to wider deployments of IoT devices at much greater scales than ever before. This, in turn, causes massive increase in the fog, cloud network traffic which leads to heavily delayed network operations. In streaming data analytics, the ability to obtain real time data insight is crucial for computational sustainability for many IoT enabled applications such as environmental monitors, pollution and climate surveillance, traffic control or even E-commerce applications. However, such network delays prevent us from achieving high quality real-time data analytics of environmental information. In order to address this challenge, we propose the Fog Sampling Node Selector (Fossel) technique that can significantly reduce the IoT network and processing delays by algorithmically selecting an optimal subset of fog nodes to perform the sensor data sampling. In addition, our technique performs a simple type of query executions within the fog nodes in order to further reduce the network delays by processing the data near the data producing devices. Our extensive evaluations show that Fossel technique outperforms the state-of-the-art in terms of latency reduction as well as in bandwidth consumption, network usage and energy consumption.

Combined analysis of helicopter-borne magnetic and stream sediment geochemical data around an ancient Tiouit gold mine (Eastern Anti-Atlas, Morocco): Geological and mining interpretations

This study focuses on the Tiouit mining area, which is reputed for its gold mineralization associated with sulfides and hematite hosted in quartz veins. The composition of granitoid hosts and their tectonic structuring, guided by tectonic accidents, NE-SW, and E-W directions related to the late Pan-African phase, are metallotects for gold exploration. Besides, the physical properties of its mineralogical associations and their setting in the mining area of Tiouit and other mining sites in the Jbel Saghro massif are characterized by a strong contrast of magnetization that can be mapped by the magnetic method. This latter is supplemented by a geochemical exploration of 632 samples of stream sediment to emphasize the geochemical anomalies associated with magnetic anomalies and lineaments within which sources may generally not outcrop on the surface.The processing and interpretation of the magnetic data covering the Tiouit area have highlighted several zones with promising prospects from a mining viewpoint. The residual magnetic map reduced to the pole reveals close connections between high amplitude anomalies and Precambrian age formations, represented by diorites, granites, rhyolites, and ignimbrites. Those of low amplitude is linked to Paleozoic coverage marked by conglomerates, sandstones, dolomites, and argillites. The vertical gradient allowed a better understanding of Triassic volcanism, manifested by magmatic injections in the form of microgabbroic dyke with a NE-SW direction.From a mining point of view, a detailed mapping of the main magnetic axes has been carried out, and which are correlated with faults of E-W, NNW-SSE, NNE- SSW, and ENE-WSW directions, corresponding to highly crushed and hematized areas. Moreover, the magnetic contacts obtained by Euler's deconvolution show a dominance of the NE-SW to E-W directions, reflecting the structural and geological context of Tiouit, which is dominated by Hercynian faults characterized by a rooting exceeding 1600 m. The examination of geochemical anomaly maps in Arsenic, Zinc, and Iron shows very high levels in the order of 136 ppm, 392 ppm, and 10.63 %, respectively.The compilation of the various results and their calibrations with geological data makes it possible to highlight 7 main potential mining areas, which could be considered promising. Two of these areas are located in the Northern and Southern peripheries of the Tiouit gold mine.

FOG ANALYTICS - A SURVEY

Fog computing has emerged as an essential alternative to the cloud. Fog computing is the future as it is nearer to the edge where actually the IOT devices and sensors are located. A Fog Server or Fog Node is located near to the IOT devices, connecting directly (wired or wireless) to them. The Fog Server has a functionality of fast accessibility to the data arising out of IOT devices or sensors, as against cloud server which may be located in data centers (near core Network Centers) located far away from the edge resulting in extreme delays in network transmission and latency, especially when the data is large volume as stream (or ‘Big Data’) arising out of IOT devices or sensors including cameras, etc. Real time response after completing the necessary Analytics on the data generated by IOT devices and sensors becomes critically essential for meeting the real time response requirements of critical applications such as in health care and transportation. What are the relevant techniques for Fog Analytics? In this paper we provide a brief survey of Fog Analytics techniques in stream data analytics, machine learning, deep learning techniques and also game theoretical adversarial learning.

Analysis and prediction of big stream data in real-time water quality monitoring system

Large scale real-time water quality monitoring system usually produces vast amounts of high frequency data, and it is difficult for traditional water quality monitoring system to process such large and high frequency data generated by wireless sensor network. A real-time processing and early warning system framework is proposed to solve this problem, Apache Storm is used as the big data processing platform, and Kafka message queue is applied to classify the sample data into several data streams so as to reserve the time series data property of a sensor. In storm platform, Daubechies Wavelet is used to decompose the data series to obtain the trend of the series, then Long Short Term Memory Network (LSTM) model is used to model and predict the trend of the data. This paper provides a detailed description concerning the distribution mechanism of aggregated data in Storm, data storage format in HBase, the process of wavelet decomposition, model training and the application of mode for prediction. The application results in Xin’an River in Yantai City reveal that the prosed system framework has a very good ability to model big data with high prediction accuracy and robust processing capability.

Preprocessed dynamic classifier ensemble selection for highly imbalanced drifted data streams

This work aims to connect two rarely combined research directions, i.e., non-stationary data stream classification and data analysis with skewed class distributions. We propose a novel framework employing stratified bagging for training base classifiers to integrate data preprocessing and dynamic ensemble selection methods for imbalanced data stream classification. The proposed approach has been evaluated based on computer experiments carried out on 135 artificially generated data streams with various imbalance ratios, label noise levels, and types of concept drift as well as on two selected real streams. Four preprocessing techniques and two dynamic selection methods, used on both bagging classifiers and base estimators levels, were considered. Experimentation results showed that, for highly imbalanced data streams, dynamic ensemble selection coupled with data preprocessing could outperform online and chunk-based state-of-art methods.