Analysis Of Data Streams Research Articles

Heterogeneity-aware elastic scaling of streaming applications on cloud platforms

Rise of Big Data techniques has led to the requirement for low latency analysis of high-velocity continuous data streams in real time. Several solutions, including Stream Processing Systems (SPSs), have been developed to enable real-time distributed stream processing. However, emerging application scenarios such as smart cities and wearable assistance that involve highly variable data rates keep on posing new challenges to the established stream processing engines for maintaining cost-effective executions. To cater to such scenarios, many modern SPSs have been proposed that leverage Cloud environment. The run-time scalability incorporated in these SPSs is in their early adaptations and are based on fixed scale sizes. Moreover, these scaling approaches do not adequately consider both the structure of the hosted streaming applications and the characteristic features of the underlying Cloud environment. Achieving true cost benefits of orchestrating streaming applications on Cloud-based pay-as-you-go model while maintaining the desired QoS, necessitates that both these issues are accounted in making the scaling decisions. This work presents a heterogeneity-aware, efficient auto-scaling strategy StreamScale-H which addresses both these issues. Simulation experiments, on representative stream applications, indicate that the proposed StreamScale-H auto-scaling algorithm exhibits much better performance in comparison with the state-of-the-art algorithms.

Data stream analysis: Foundations, major tasks and tools

AbstractThe significant growth of interconnected Internet‐of‐Things (IoT) devices, the use of social networks, along with the evolution of technology in different domains, lead to a rise in the volume of data generated continuously from multiple systems. Valuable information can be derived from these evolving data streams by applying machine learning. In practice, several critical issues emerge when extracting useful knowledge from these potentially infinite data, mainly because of their evolving nature and high arrival rate which implies an inability to store them entirely. In this work, we provide a comprehensive survey that discusses the research constraints and the current state‐of‐the‐art in this vibrant framework. Moreover, we present an updated overview of the latest contributions proposed in different stream mining tasks, particularly classification, regression, clustering, and frequent patterns.This article is categorized under: Fundamental Concepts of Data and Knowledge > Key Design Issues in Data Mining Fundamental Concepts of Data and Knowledge > Motivation and Emergence of Data Mining

Protected Network Architecture for Ensuring Consistency of Medical Data through Validation of User Behavior and DICOM Archive Integrity

The problem of consistency of medical data in Hospital Data Management Systems is considered in the context of correctness of medical images stored in a PACS (Picture Archiving and Communication System) and legality of actions authorized users perform when accessing MIS (Medical Information System) facilities via web interfaces. The purpose of the study is to develop a SIEM-like (Security Information and Event Management) architecture for offline analysis of DICOM (Digital Imaging and Communications in Medicine) archive integrity and users’ activity. To achieve amenable accuracy when validating DICOM archive integrity, two aspects are taken into account: correctness of periodicity of the incoming data stream and correctness of the image data (time series) itself for the considered modality. Validation of users’ activity assumes application of model-driven approaches using state-of-the-art machine learning methods. This paper proposes a network architecture with guard clusters to protect sensitive components like the DICOM archive and application server of the MIS. New server roles were designed to perform traffic interception, data analysis and alert management without reconfiguration of production software components. The cluster architecture allows the analysis of incoming big data streams with high availability, providing horizontal scalability and fault tolerance. To minimize possible harm from spurious DICOM files the approach should be considered as an addition to other securing techniques like watermarking, encrypting and testing data conformance with a standard.

Improved Real-Time Natural Hazard Monitoring Using Automated DInSAR Time Series

As part of the collaborative GeoSciFramework project, we are establising a monitoring system for the Yellowstone volcanic area that integrates multiple geodetic and seismic data sets into an advanced cyber-infrastructure framework that will enable real-time streaming data analytics and machine learning and allow us to better characterize associated long- and short-term hazards. The goal is to continuously ingest both remote sensing (GNSS, DInSAR) and ground-based (seismic, thermal and gas observations, strainmeter, tiltmeter and gravity measurements) data and query and analyse them in near-real time. In this study, we focus on DInSAR data processing and the effects from using various atmospheric corrections and real-time orbits on the automated processing and results. We find that the atmospheric correction provided by the European Centre for Medium-Range Weather Forecasts (ECMWF) is currently the most optimal for automated DInSAR processing and that the use of real-time orbits is sufficient for the early-warning application in question. We show analysis of atmospheric corrections and using real-time orbits in a test case over the Kilauea volcanic area in Hawaii. Finally, using these findings, we present results of displacement time series in the Yellowstone area between May 2018 and October 2019, which are in good agreement with GNSS data where available. These results will contribute to a baseline model that will be the basis of a future early-warning system that will be continuously updated with new DInSAR data acquisitions.

Molecular function recognition by supervised projection pursuit machine learning

Identifying mechanisms that control molecular function is a significant challenge in pharmaceutical science and molecular engineering. Here, we present a novel projection pursuit recurrent neural network to identify functional mechanisms in the context of iterative supervised machine learning for discovery-based design optimization. Molecular function recognition is achieved by pairing experiments that categorize systems with digital twin molecular dynamics simulations to generate working hypotheses. Feature extraction decomposes emergent properties of a system into a complete set of basis vectors. Feature selection requires signal-to-noise, statistical significance, and clustering quality to concurrently surpass acceptance levels. Formulated as a multivariate description of differences and similarities between systems, the data-driven working hypothesis is refined by analyzing new systems prioritized by a discovery-likelihood. Utility and generality are demonstrated on several benchmarks, including the elucidation of antibiotic resistance in TEM-52 beta-lactamase. The software is freely available, enabling turnkey analysis of massive data streams found in computational biology and material science.

A Frequent Pattern Conjunction Heuristic for Rule Generation in Data Streams

This paper introduces a new and expressive algorithm for inducing descriptive rule-sets from streaming data in real-time in order to describe frequent patterns explicitly encoded in the stream. Data Stream Mining (DSM) is concerned with the automatic analysis of data streams in real-time. Rapid flows of data challenge the state-of-the art processing and communication infrastructure, hence the motivation for research and innovation into real-time algorithms that analyse data streams on-the-fly and can automatically adapt to concept drifts. To date, DSM techniques have largely focused on predictive data mining applications that aim to forecast the value of a particular target feature of unseen data instances, answering questions such as whether a credit card transaction is fraudulent or not. A real-time, expressive and descriptive Data Mining technique for streaming data has not been previously established as part of the DSM toolkit. This has motivated the work reported in this paper, which has resulted in developing and validating a Generalised Rule Induction (GRI) tool, thus producing expressive rules as explanations that can be easily understood by human analysts. The expressiveness of decision models in data streams serves the objectives of transparency, underpinning the vision of ‘explainable AI’ and yet is an area of research that has attracted less attention despite being of high practical importance. The algorithm introduced and described in this paper is termed Fast Generalised Rule Induction (FGRI). FGRI is able to induce descriptive rules incrementally for raw data from both categorical and numerical features. FGRI is able to adapt rule-sets to changes of the pattern encoded in the data stream (concept drift) on the fly as new data arrives and can thus be applied continuously in real-time. The paper also provides a theoretical, qualitative and empirical evaluation of FGRI.

A system for off-line validation of medical data in DICOM archive

The problem of consistency of medical data in Hospital Data Management Systems is considered in the context of correctness of medical images itself to minimize possible harm from spurious DICOM files. The approach should be considered as an addition to other securing techniques like watermarking, encrypting, testing conformance with the standard. To achieve amenable accuracy in practice two aspects are taken into account: correctness of periodicity and correctness of image data (time series) itself for considered modality. This paper proposes an architecture of an information system and network filter integrated to it to provide facilities for analysis and alert management. The architecture is designed to perform the analysis of incoming data streams within components working in a clustered manner, providing horizontal scalability and fault tolerance to be BigData-ready.

Data Streams Oriented Outlier Detection Method: A Fast Minimal Infrequent Pattern Mining

Outlier detection is a common method for analyzing data streams. In the existing outlier detection methods, most of methods compute distance of points to solve certain specific outlier detection problems. However, these methods are computationally expensive and cannot process data streams quickly. The outlier detection method based on pattern mining resolves the aforementioned issues, but the existing methods are inefficient and cannot meet requirements of quickly mining data streams. In order to improve the efficiency of the method, a new outlier detection method is proposed in this paper. First, a fast minimal infrequent pattern mining method is proposed to mine the minimal infrequent pattern from data streams. Second, an efficient outlier detection algorithm based on minimal infrequent pattern is proposed for detecting the outliers in the data streams by mining minimal infrequent pattern. The algorithm proposed in this paper is demonstrated by real telemetry data of a satellite in orbit. The experimental results show that the proposed method not only can be applied to satellite outlier detection, but also is superior to the existing methods.

Impact of Distance Measures on the Performance of AIS Data Clustering

Automatic Identification System (AIS) data stream analysis is based on the AIS data of different vessel’s behaviours, including the vessels’ routes. When the AIS data consists of outliers, noises, or are incomplete, then the analysis of the vessel’s behaviours is not possible or is limited. When the data consists of outliers, it is not possible to automatically assign the AIS data to a particular vessel. In this paper, a clustering method is proposed to support the AIS data analysis, to qualify noises and outliers with respect to their suitability, and finally to aid the reconstruction of the vessel’s trajectory. In this paper, clustering results have been obtained using selected algorithms, including k-means, k-medoids, and fuzzy c-means. Based on the clustering results, it is possible to decide on the qualification of data with outliers and on their usefulness in the reconstruction of the vessel trajectory. The main aim of this paper is to answer how different distance measures during a clustering process can influence AIS data clustering quality. The main core question is whether or not they have an impact on the process of reconstruction of the vessel trajectories when the data are damaged. The research question during the computational experiments asked whether or not distance measure influence AIS data clustering quality. The computational experiments have been carried out using original AIS data. In general, the experiment and the results confirm the usefulness of the cluster-based analysis when the data include outliers that are derived from the natural environment. It is also possible to monitor and to analyse AIS data using clustering when the data include outliers. The computational experiment results confirm that the k-means with Euclidean distance has the best performance.

Two‐level pruning based ensemble with abstained learners for concept drift in data streams

AbstractMining data streams for predictive analysis is one of the most interesting topics in machine learning. With the drifting data distributions, it becomes important to build adaptive systems which are dynamic and accurate. Although ensembles are powerful in improving accuracy of incremental learning, it is crucial to maintain a set of best suitable learners in the ensemble while considering the diversity between them. By adding diversity‐based pruning to the traditional accuracy‐based pruning, this paper proposes a novel concept drift handling approach named Two‐Level Pruning based Ensemble with Abstained Learners (TLP‐EnAbLe). In this approach, deferred similarity based pruning delays the removal of under performing similar learners until it is assured that they are no longer fit for prediction. The proposed scheme retains diverse learners that are well suited for current concept. Two‐level abstaining monitors performance of learners and chooses the best set of competent learners for participating in decision making. This is an enhancement to traditional majority voting system which dynamically chooses high performing learners and abstains the ones which are not suitable for prediction. In our experiments, it has been demonstrated that TLP‐EnAbLe handles concept drift more effectively than other state‐of‐the‐art algorithms on nineteen artificially drifting and ten real‐world datasets. Further, statistical tests conducted on various drift patterns which include gradual, abrupt, recurring and their combinations prove efficiency of the proposed approach.

Mapping the Big Data Landscape: Technologies, Platforms and Paradigms for Real-Time Analytics of Data Streams

The `Big Data' of yesterday is the `data' of today. As technology progresses, new challenges arise and new solutions are developed. Due to the emergence of Internet of Things applications within the last decade, the field of Data Mining has been faced with the challenge of processing and analysing data streams in real-time, and under high data throughput conditions. This is often referred to as the Velocity aspect of Big Data. Whereas there are numerous reviews on Data Stream Mining techniques and applications, there is very little work surveying Data Stream processing paradigms and associated technologies, from data collection through to pre-processing and feature processing, from the perspective of the user, not that of the service provider. In this article, we evaluate a particular type of solution, which focuses on streaming data, and processing pipelines that permit online analysis of data streams that cannot be stored as-is on the computing platform. We review foundational computational concepts such as distributed computation, fault-tolerant computing, and computational paradigms/architectures. We then review the available technological solutions, and applications that pertain to data stream mining as case studies of these theoretical concepts. We conclude with a discussion of the field of data stream processing/analytics, future directions and research challenges.

Streaming Data Analysis: Clustering or Classification?

This article is a position paper about models and algorithms that are generally called “stream clustering.” Semantics and methods used in this field are often co-opted from static clustering, but they do not serve well for streaming data analysis. Most “state-of-the-art” methods, such as sequential k-means, Birch, CluStream, DenStream, etc., acknowledge that the data are seen but once in real streaming analysis (e.g., intrusion detection, voter fraud, etc.). Interpretation of their outputs generally overlooks the fact that when the data cannot be saved, batch clustering ideas, such as preclustering assessment, partitioning, and cluster validity are not relevant. But in the current literature, the data, or some subset of it, are often saved for hindsight evaluation (we call this fake stream clustering). Our position? Useful analysis of real streaming data is in its infancy. We do not argue that current approaches to streaming clustering are wrong: rather, we regard them as transitional methods which will eventually lead to a new and useful paradigm for this type of computation. We think that this class of models and algorithms are actually classifiers, but with a special added component, viz., continuously updated cluster footprints of the instream processing. We need to carefully define the objectives of streaming analysis, and then choose terminology and methods that suit this evolving paradigm.

Fossel: Efficient Latency Reduction in Approximating Streaming Sensor Data

The volume of streaming sensor data from various environmental sensors continues to increase rapidly due to wider deployments of IoT devices at much greater scales than ever before. This, in turn, causes massive increase in the fog, cloud network traffic which leads to heavily delayed network operations. In streaming data analytics, the ability to obtain real time data insight is crucial for computational sustainability for many IoT enabled applications such as environmental monitors, pollution and climate surveillance, traffic control or even E-commerce applications. However, such network delays prevent us from achieving high quality real-time data analytics of environmental information. In order to address this challenge, we propose the Fog Sampling Node Selector (Fossel) technique that can significantly reduce the IoT network and processing delays by algorithmically selecting an optimal subset of fog nodes to perform the sensor data sampling. In addition, our technique performs a simple type of query executions within the fog nodes in order to further reduce the network delays by processing the data near the data producing devices. Our extensive evaluations show that Fossel technique outperforms the state-of-the-art in terms of latency reduction as well as in bandwidth consumption, network usage and energy consumption.

Combined analysis of helicopter-borne magnetic and stream sediment geochemical data around an ancient Tiouit gold mine (Eastern Anti-Atlas, Morocco): Geological and mining interpretations

This study focuses on the Tiouit mining area, which is reputed for its gold mineralization associated with sulfides and hematite hosted in quartz veins. The composition of granitoid hosts and their tectonic structuring, guided by tectonic accidents, NE-SW, and E-W directions related to the late Pan-African phase, are metallotects for gold exploration. Besides, the physical properties of its mineralogical associations and their setting in the mining area of Tiouit and other mining sites in the Jbel Saghro massif are characterized by a strong contrast of magnetization that can be mapped by the magnetic method. This latter is supplemented by a geochemical exploration of 632 samples of stream sediment to emphasize the geochemical anomalies associated with magnetic anomalies and lineaments within which sources may generally not outcrop on the surface.The processing and interpretation of the magnetic data covering the Tiouit area have highlighted several zones with promising prospects from a mining viewpoint. The residual magnetic map reduced to the pole reveals close connections between high amplitude anomalies and Precambrian age formations, represented by diorites, granites, rhyolites, and ignimbrites. Those of low amplitude is linked to Paleozoic coverage marked by conglomerates, sandstones, dolomites, and argillites. The vertical gradient allowed a better understanding of Triassic volcanism, manifested by magmatic injections in the form of microgabbroic dyke with a NE-SW direction.From a mining point of view, a detailed mapping of the main magnetic axes has been carried out, and which are correlated with faults of E-W, NNW-SSE, NNE- SSW, and ENE-WSW directions, corresponding to highly crushed and hematized areas. Moreover, the magnetic contacts obtained by Euler's deconvolution show a dominance of the NE-SW to E-W directions, reflecting the structural and geological context of Tiouit, which is dominated by Hercynian faults characterized by a rooting exceeding 1600 m. The examination of geochemical anomaly maps in Arsenic, Zinc, and Iron shows very high levels in the order of 136 ppm, 392 ppm, and 10.63 %, respectively.The compilation of the various results and their calibrations with geological data makes it possible to highlight 7 main potential mining areas, which could be considered promising. Two of these areas are located in the Northern and Southern peripheries of the Tiouit gold mine.

FOG ANALYTICS - A SURVEY

Fog computing has emerged as an essential alternative to the cloud. Fog computing is the future as it is nearer to the edge where actually the IOT devices and sensors are located. A Fog Server or Fog Node is located near to the IOT devices, connecting directly (wired or wireless) to them. The Fog Server has a functionality of fast accessibility to the data arising out of IOT devices or sensors, as against cloud server which may be located in data centers (near core Network Centers) located far away from the edge resulting in extreme delays in network transmission and latency, especially when the data is large volume as stream (or ‘Big Data’) arising out of IOT devices or sensors including cameras, etc. Real time response after completing the necessary Analytics on the data generated by IOT devices and sensors becomes critically essential for meeting the real time response requirements of critical applications such as in health care and transportation. What are the relevant techniques for Fog Analytics? In this paper we provide a brief survey of Fog Analytics techniques in stream data analytics, machine learning, deep learning techniques and also game theoretical adversarial learning.

Analysis and prediction of big stream data in real-time water quality monitoring system

Large scale real-time water quality monitoring system usually produces vast amounts of high frequency data, and it is difficult for traditional water quality monitoring system to process such large and high frequency data generated by wireless sensor network. A real-time processing and early warning system framework is proposed to solve this problem, Apache Storm is used as the big data processing platform, and Kafka message queue is applied to classify the sample data into several data streams so as to reserve the time series data property of a sensor. In storm platform, Daubechies Wavelet is used to decompose the data series to obtain the trend of the series, then Long Short Term Memory Network (LSTM) model is used to model and predict the trend of the data. This paper provides a detailed description concerning the distribution mechanism of aggregated data in Storm, data storage format in HBase, the process of wavelet decomposition, model training and the application of mode for prediction. The application results in Xin’an River in Yantai City reveal that the prosed system framework has a very good ability to model big data with high prediction accuracy and robust processing capability.

Preprocessed dynamic classifier ensemble selection for highly imbalanced drifted data streams

This work aims to connect two rarely combined research directions, i.e., non-stationary data stream classification and data analysis with skewed class distributions. We propose a novel framework employing stratified bagging for training base classifiers to integrate data preprocessing and dynamic ensemble selection methods for imbalanced data stream classification. The proposed approach has been evaluated based on computer experiments carried out on 135 artificially generated data streams with various imbalance ratios, label noise levels, and types of concept drift as well as on two selected real streams. Four preprocessing techniques and two dynamic selection methods, used on both bagging classifiers and base estimators levels, were considered. Experimentation results showed that, for highly imbalanced data streams, dynamic ensemble selection coupled with data preprocessing could outperform online and chunk-based state-of-art methods.

Statistical hierarchical clustering algorithm for outlier detection in evolving data streams

Anomaly detection is a hard data analysis process that requires constant creation and improvement of data analysis algorithms. Using traditional clustering algorithms to analyse data streams is impossible due to processing power and memory issues. To solve this, the traditional clustering algorithm complexity needed to be reduced, which led to the creation of sequential clustering algorithms. The usual approach is two-phase clustering, which uses online phase to relax data details and complexity, and offline phase to cluster concepts created in the online phase. Detecting anomalies in a data stream is usually solved in the online phase, as it requires unreduced data. Contrarily, producing good macro-clustering is done in the offline phase, which is the reason why two-phase clustering algorithms have difficulty being equally good in anomaly detection and macro-clustering. In this paper, we propose a statistical hierarchical clustering algorithm equally suitable for both detecting anomalies and macro-clustering. The proposed algorithm is single-phased and uses statistical inference on the input data stream, resulting in statistical distributions that are constantly updated. This makes the classification adaptable, allowing agglomeration of outliers into clusters, tracking population evolution, and to be used without knowing the expected number of clusters and outliers. The proposed algorithm was tested against typical clustering algorithms, including two-phase algorithms suitable for data stream analysis. A number of typical test cases were selected, to show the universality and qualities of the proposed clustering algorithm.

AI (Artificial Intelligence) and Hypertension Research.

Purpose of ReviewThis review a highlights that to use artificial intelligence (AI) tools effectively for hypertension research, a new foundation to further understand the biology of hypertension needs to occur by leveraging genome and RNA sequencing technology and derived tools on a broad scale in hypertension.Recent FindingsFor the last few years, progress in research and management of essential hypertension has been stagnating while at the same time, the sequencing of the human genome has been generating many new research tools and opportunities to investigate the biology of hypertension. Cancer research has applied modern tools derived from DNA and RNA sequencing on a large scale, enabling the improved understanding of cancer biology and leading to many clinical applications. Compared with cancer, studies in hypertension, using whole genome, exome, or RNA sequencing tools, total less than 2% of the number cancer studies. While true, sequencing the genome of cancer tissue has provided cancer research an advantage, DNA and RNA sequencing derived tools can also be used in hypertension to generate new understanding how complex protein network, in non-cancer tissue, adapts and learns to be effective when for example, somatic mutations or environmental inputs change the gene expression profiles at different network nodes. The amount of data and differences in clinical condition classification at the individual sample level might be of such magnitude to overwhelm and stretch comprehension. Here is the opportunity to use AI tools for the analysis of data streams derived from DNA and RNA sequencing tools combined with clinical data to generate new hypotheses leading to the discovery of mechanisms and potential target molecules from which drugs or treatments can be developed and tested.SummaryBasic and clinical research taking advantage of new gene sequencing-based tools, to uncover mechanisms how complex protein networks regulate blood pressure in health and disease, will be critical to lift hypertension research and management from its stagnation. The use of AI analytic tools will help leverage such insights. However, applying AI tools to vast amounts of data that certainly exist in hypertension, without taking advantage of new gene sequencing-based research tools, will generate questionable results and will miss many new potential molecular targets and possibly treatments. Without such approaches, the vision of precision medicine for hypertension will be hard to accomplish and most likely not occur in the near future.

GeoDenStream: An improved DenStream clustering method for managing entity data within geographical data streams

Data streams have become an integral part of the rapidly evolving modern information landscape in various application domains. Stream clustering, and in particular density-based clustering, has emerged as one of the most commonly used data stream analysis tasks. Several density-based stream clustering methods have been proposed; chief among them is DenStream. Existing DenStream clustering methods usually preserve only the key summary descriptors about each cluster such as the center and radius. Such approach is not suitable for streams that observe discrete entities, since the clustering process does not maintain the entity-level composition of each cluster over time. The primary challenge we explore in this paper is therefore how existing DenStream clustering methods can be enhanced to support entity-based stream mining in geographical space. In view of this consideration, this paper presents GeoDenStream, a spatiotemporal entity-based stream clustering method. Building on DenStream, GeoDenStream is particularly suitable for clustering discrete entities due to its ability to track the relationship between entities and clusters over time and its ability to recover data that has been incorrectly labeled as noise. Memory efficiency in GeoDenStream is achieved by using a combination of data pruning and indexing. The performance of GeoDenStream was evaluated with both synthetic and real-world stream data from a popular social media platform (Twitter). The results of these evaluations show that GeoDenStream is able to efficiently handle memory constraints, overlapping data points, and false noise.