Analysis Of Data Streams Research Articles

On a two-stage progressive clustering algorithm with graph-augmented density peak clustering

Due to the rapidly growing volume and velocity of big data, real-time streaming data analysis has become increasingly important in many applications. To discover knowledge from such data, a wide range of machine learning techniques have been proposed and used in practice. Among them, clustering, which aims at grouping objects into different classes on the basis of their similarity, is the most common form of unsupervised learning. However, most existing clustering algorithms are designed for static data, and hence are not best suited for streaming data. In this paper, we propose PC-DPC, a two-stage progressive clustering algorithm with graph-augmented density peak clustering. PC-DPC first identifies clusters of streaming data using an improved density peak clustering algorithm, and then merges newly arriving data into the existing data pool by measuring inter-cluster structural similarity, which considers the distance between a center and representative points. We illustrate the superiority of PC-DPC over several state-of-the-art clustering algorithms in terms of clustering accuracy and running time on publicly available benchmark datasets.

The Metamorphosis (of RAM3S)

The real-time analysis of Big Data streams is a terrific resource for transforming data into value. For this, Big Data technologies for smart processing of massive data streams are available, but the facilities they offer are often too raw to be effectively exploited by analysts. RAM3S (Real-time Analysis of Massive MultiMedia Streams) is a framework that acts as a middleware software layer between multimedia stream analysis techniques and Big Data streaming platforms, so as to facilitate the implementation of the former on top of the latter. RAM3S has been proven helpful in simplifying the deployment of non-parallel techniques to streaming platforms, such as Apache Storm or Apache Flink. In this paper, we show how RAM3S has been updated to incorporate novel stream processing platforms, such as Apache Samza, and to be able to communicate with different message brokers, such as Apache Kafka. Abstracting from the message broker also provides us with the ability to pipeline several RAM3S instances that can, therefore, perform different processing tasks. This represents a richer model for stream analysis with respect to the one already available in the original RAM3S version. The generality of this new RAM3S version is demonstrated through experiments conducted on three different multimedia applications, proving that RAM3S is a formidable asset for enabling efficient and effective Data Mining and Machine Learning on multimedia data streams.

Multivariate online regression analysis with heterogeneous streaming data

New data collection and storage technologies have given rise to a new field of streaming data analytics, called real‐time statistical methodology for online data analyses. Most existing online learning methods are based on homogeneity assumptions, which require the samples in a sequence to be independent and identically distributed. However, inter‐data batch correlation and dynamically evolving batch‐specific effects are among the key defining features of real‐world streaming data such as electronic health records and mobile health data. This article is built under a state‐space mixed model framework in which the observed data stream is driven by a latent state process that follows a Markov process. In this setting, online maximum likelihood estimation is made challenging by high‐dimensional integrals and complex covariance structures. In this article, we develop a real‐time Kalman‐filter‐based regression analysis method that updates both point estimates and their standard errors for fixed population average effects while adjusting for dynamic hidden effects. Both theoretical justification and numerical experiments demonstrate that our proposed online method has statistical properties similar to those of its offline counterpart and enjoys great computational efficiency. We also apply this method to analyze an electronic health record dataset.

USAGE OF HASH METHODS IN THE CRYPTOGRAPHIC DATA ANALYSIS

The tasks of information security system include identifying potential or actual sources of threat to system’s work and minimizing consequences of unauthorized influence on it. While solving them, arises the need of restoration of the initial state of the information system, especially the data integrity. While performing information message analysis the other task may be finding differences between two data fragments or their instances. This paper offers methods of the complex solution of the information security tasks and the analysis of data streams using the means of cryptography and presents the experience of developing the reliable implementation of these methods. The developed methods allow detecting falsifications in data part of the sent message and restoring the initial message. During the cryptographic analysis, the area of change in a data block is localized using cross hashing which is performed by computing the hash of information message block by block. The result is the program implementation of the offered method of information stream analysis that is based on comparing three frames of input data. The effectiveness of detecting falsifications in a data stream depending on algorithm’s sensitivity was researched with the developed instrument. The dependence of the share of falsifications detected by the system in the information block on the established maximum allowable relative deviation from the median and the properties of the input stream, in particular, the division of the input data into frames, was experimentally revealed. It is expected that the advantages of the method will be higher in the preliminary stage of data flow analysis related to its segmentation before addressing the selected fragments to more accurate and specialized algorithms.

PrivStream: A privacy-preserving inference framework on IoT streaming data at the edge

Edge computing combining with artificial intelligence (AI) has enabled the timely processing and analysis of streaming data produced by IoT intelligent applications. However, it causes privacy risk due to the data exchanges between local devices and untrusted edge servers. The powerful analytical capability of AI further exacerbates the risks because it can even infer private information from insensitive data. In this paper, we propose a privacy-preserving IoT streaming data analytical framework based on edge computing, called PrivStream, to prevent the untrusted edge server from making sensitive inferences from the IoT streaming data. It utilizes a well-designed deep learning model to filter the sensitive information and combines with differential privacy to protect against the untrusted edge server. The noise is also injected into the framework in the training phase to increase the robustness of PrivStream to differential privacy noise. Taking into account the dynamic and real-time characteristics of streaming data, we realize PrivStream with two types of models to process data segment with fixed length and variable length, respectively, and implement it on a distributed streaming platform to achieve real-time streaming data transmission. We theoretically prove that Privstream satisfies ε-differential privacy and experimentally demonstrate that PrivStream has better performance than the state-of-the-art and has acceptable computation and storage overheads.

Concept drift type identification based on multi-sliding windows

Concept drift is a common and important issue in streaming data analysis and mining. Thus far, many concept drift detection methods have been proposed but may not be able to identify the type of concept drift, which will result in some difficulties, such as extracting the wrong key information, inadequate model learning and poor detection efficiency. To solve these problems, a concept drift type identification method is proposed based on multi-sliding windows (CDT_MSW). This method consists of three processes. During the first detection process, the drift position is detected by sliding the basic window forward. Then, in the growth process, the drift length is detected using the growth of the adjoint window, and the drift category is identified according to the drift length. Finally, during tracking process, the drift subcategory can be accurately identified according to the different tracking flow ratio curves generated during window tracking. Experimental results show that the proposed method can effectively identify the type of concept drift, accurately analyze the key information during online learning and improve the efficiency and generalization performance of streaming data analysis and mining.

A fault-tolerant clustering algorithm for processing data from multiple streams

Nowadays often multiple sensors provide real-time data about the same process. However, most data stream analysis algorithms work over a single feature vector. A common approach to overcome this issue is to combine the features extracted from each sensor into a single vector. The more sensors monitor a process, the more likely there is a malfunction or delay in data transmission. Hence the interest in developing algorithms capable of handling missing data or incorporating delayed data when it becomes available. This work presents a dynamic ensemble clustering algorithm based on evidence accumulation that can process multiple data streams. Each algorithm of the ensemble only processes one data stream, and obtaining the final partition does not require that data from all the streams is available. Therefore, final results can be provided even when there are sensor malfunctions or network delays. Furthermore, if delayed data arrives it can be used to extract evidence from the moment of arrival onwards. The algorithm was applied to identify arrhythmias over 71 standard 12-lead electrocardiogram recordings, achieving a 0.314% ± 0.010 error rate. The strategies of combining all the leads in a single feature vector or working with each lead independently were compared, obtaining better results in the second case.

Renewable quantile regression for streaming datasets

Streaming data analysis has drawn much attention, where large amounts of data arrive in streams. Because limited memory can only store a small batch of data, fast analysis without access to the historical data is necessary. Quantile regression has been widely used in many fields because of its robustness and comprehensiveness. However, in the streaming data environment, it is challenging to implement quantile regression by the conventional methods, because they are all based on the assumption that the memory can fit all the data. To fix this issue, this paper proposes a novel online renewable quantile regression strategy, in which the resulting estimator is renewed with current data and summary statistics of historical data. Thus, it is computationally efficient, and not storage-intensive. What is more, the theoretical results also confirm that the proposed estimator is asymptotically equivalent with the oracle estimator calculated using the entire data together. Numerical experiments on both synthetic and real data verify the theoretical results and illustrate the good performance of the new method.

Im Vestibül der Algorithmen

Abstract Online-access to audiovisual content is rapidly changing the system of film distribution and film presentation. The streaming portal acts as a threshold for online access to moving image works. Algorithms and artificial intelligence are increasingly taking over the selection and curation of audiovisual content and direct viewers’ attention via personalised recommendation systems. The article outlines the market development towards online access to audiovisual content and analyses selected aspects such as artwork (thumbnails) and genres to explore the functionalities of accessory (paratexts) to the film for recommendation systems in video streaming portals and data analysis. From the analysis, the article derives preliminary considerations on requirements that result from the use of artificial intelligence technologies for promoting film and visual literacy in cultural heritage institutions.

REAL-TIME STREAMING TECHNOLOGY AND ANALYTICS FOR VALUE CREATION

ABSTRACT Real-time streaming technology and analytics capabilities are growing rapidly, whereas a great number of firms and organizations are considering implementing this technology to meet rising business demands. Traditional computer infrastructures for high performance computing and big data analytics are not able to conduct such tasks. To tackle this obstacle, rapid analysis of streaming data requires significant amounts of computer and data storage capacity, which requires real-time streaming technology and analytics. Real-time streaming has become a crucial component where tremendous volumes of data from thousands of sensors and other information sources are processed so that a company extracting the copious amount of real-time data can react to changing conditions instantaneously. Streaming technology and analytics generate real value from real-time data. This paper presents the current architecture, status, and trend of real-time streaming technology and analytics. It discusses value creation of streaming analytics. The paper describes continuous intelligence and value for streaming analytics and the current architecture and status of streaming technology and analytics; showcases the leading vendors for streaming technology and analytics; discusses various real-world use cases and benefits across various industries; analyzes value creation of streaming analytics; and proposes several research issues, along with challenges and recommendations.

HCache: A Hash-based Hybrid Caching Model for Real-Time Streaming Data Analytics

Up-to-date results in data stream analytics are difficult to obtain because the data are in rapid sequence and can be accessed only once. Due to the exactly-once delivery nature of streaming data, online processing is quite unstable. To guarantee comprehensive and accurate results, aggregating historical data is essential when processing streaming data. In this paper, we propose a hash-based hybrid cache model, namely, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">HCache</i> , for fast data analytics covering real-time streaming data and historical data. The <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">HCache</i> model integrates the online cache and batch cache for hybrid online and batch processing and uses a hash structure to accelerate storage. When executing analytic tasks, the batch cache and online cache are accessed in parallel. Computed streaming data are stored in the online cache, which returns qualified results based on one-time visiting. The most recently visited historical data are stored in the batch cache, and they are also used to correct errors in the online cache. Efficient replacement strategies are used to keep the caches within a relatively stable size. To coordinate the online cache with the batch cache, an LRU-based selection strategy is designed to achieve comprehensive results. Experimental results show that the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">HCache</i> model can quickly and efficiently execute analytic tasks with little additional overhead; moreover <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">HCache</i> is more stable and effective at data storage, access and query with less memory utilization than other models.

Extended Hapicare: A telecare system with probabilistic diagnosis and self-adaptive treatment

The massive growth of the population with chronic diseases calls for a telecare system to enhance their quality of life and reduce their treatment costs. Most of the current solutions depend on reliable data, deterministic rules, or the similarity of patients, while studies have shown otherwise. To this end, in this paper, we have extended our previous work on the Hapicare framework to integrate probabilistic diagnosis and self-adaptive treatment. Our new framework enables sensors’ datastream analysis and online decision-making. Its ontology-based reasoning uses Systematized Nomenclature of Medicine - Clinical Terms (SNOMED-CT) ontology to add contextual information to the collected data. Moreover, probabilistic reasoning is applied for diagnosis and screening to manage the uncertainty and unreliability of data as well as the indeterministic medical rules. The treatment system is designed to be modifiable by the experts and automatically adaptable to patients’ needs. The probabilistic diagnosis performance has been evaluated based on two public datasets regarding symptoms and risk factors of two chronic diseases: chronic kidney disease and dermatologic disease. The results show that our solution outperforms a classical classifier specifically when more than 40% of the data are missing. The proposed framework is also validated using four scenarios. The evaluation results demonstrate the ability of the proposed framework to help patients and doctors diagnose and treat medical conditions and episodes.

Multiple Event Identification and Characterization by Retrospective Analysis of Structured Data Streams

The sensors installed in complex systems generate massive data, which contain rich information about the system operation status. This article proposes a retrospective analysis method for a historical data set, which simultaneously identifies when multiple events occur to the system and characterizes how they affect the multiple sensing signals. The problem formulation is motivated by the dictionary learning method and the solution is obtained by iteratively updating the event signatures and sequences using ADMM algorithms. A simulation study and a case study of the steel rolling process validate our approach. The supplementary materials including the appendices and the reproduction report are available online.

Live Processing of Momentum-Resolved STEM Data for First Moment Imaging and Ptychography

A reformulated implementation of single-sideband ptychography enables analysis and display of live detector data streams in 4D scanning transmission electron microscopy (STEM) using the LiberTEM open-source platform. This is combined with live first moment and further virtual STEM detector analysis. Processing of both real experimental and simulated data shows the characteristics of this method when data are processed progressively, as opposed to the usual offline processing of a complete data set. In particular, the single-sideband method is compared with other techniques such as the enhanced ptychographic engine in order to ascertain its capability for structural imaging at increased specimen thickness. Qualitatively interpretable live results are obtained also if the sample is moved, or magnification is changed during the analysis. This allows live optimization of instrument as well as specimen parameters during the analysis. The methodology is especially expected to improve contrast- and dose-efficient in situ imaging of weakly scattering specimens, where fast live feedback during the experiment is required.

Towards scalable and reusable predictive models for cyber twins in manufacturing systems

Smart factories are intelligent, fully-connected and flexible systems that can continuously monitor and analyse data streams from interconnected systems to make decisions and dynamically adapt to new circumstances. The implementation of smart factories represents a leap forward compared to traditional automation. It is underpinned by the deployment of cyberphysical systems that, through the application of Artificial Intelligence, integrate predictive capabilities and foster rapid decision-making. Deep Learning (DL) is a key enabler for the development of smart factories. However, the implementation of DL in smart factories is hindered by its reliance on large amounts of data and extreme computational demand. To address this challenge, Transfer Learning (TL) has been proposed to promote the efficient training of models by enabling the reuse of previously trained models. In this paper, by means of a specific example in aluminium can manufacturing, an empirical study is presented, which demonstrates the potential of TL to achieve fast deployment of scalable and reusable predictive models for Cyber Manufacturing Systems. Through extensive experiments, the value of TL is demonstrated to achieve better generalisation and model performance, especially with limited datasets. This research provides a pragmatic approach towards predictive model building for cyber twins, paving the way towards the realisation of smart factories.

Stream Processing Tools for Analyzing Objects in Motion Sending High-Volume Location Data

Recently we observe a significant increase in the amount of easily accessible data on transport and mobility. This data is mostly massive streams of high velocity, magnitude, and heterogeneity, which represent a flow of goods, shipments and the movements of fleet. It is therefore necessary to develop a scalable framework and apply tools capable of handling these streams. In the paper we propose an approach for the selection of software for stream processing solutions that may be used in the transportation domain. We provide an overview of potential stream processing technologies, followed by the method for choosing the selected software for real-time analysis of data streams coming from objects in motion. We have selected two solutions: Apache Spark Streaming and Apache Flink, and benchmarked them on a real-world task. We identified the caveats and challenges when it comes to implementation of the solution in practice.

Distinguishing active and legacy source contributions to stream water quality: Comparative quantification for chloride and metals

AbstractHydrochemical constituents in streams may originate from currently active sources at the surface and/or legacy sources from earlier surface inputs, waste deposits and land contamination. Distinction and quantification of these source contributions are needed for improved interpretation of tracer data and effective reduction of waterborne environmental pollutants. This article develops a methodology that recognizes and quantifies some general mechanistic differences in stream concentration and load behavior versus discharge between such source contributions. The methodology is applied to comparative analysis of stream concentration data for chloride (Cl−), copper (Cu), lead (Pb), and zinc (Zn), and corresponding data for water discharge, measured over the period 1990–2018 in multiple hydrological catchments (19 for Cl−, 11 for Cu and Zn, 10 for Pb) around the major Lake Mälaren in Sweden. For Cl−, the average load fraction of active sources is quantified to be 19%, and the average active and legacy concentration contributions as 2.9 and 11 mg/L, respectively. For the metals, the average active load fractions at outlets are 1%–3% over all catchments and 9%–14% in the relatively few catchments with mixed metal sources. Average active and legacy concentration contributions are 0.14 and 3.2 μg/L for Cu, 0.05 and 1.5 μg/L for Pb, and 1.4 and 12 μg/L for Zn, respectively. This multi‐catchment analysis thus indicates a widespread prevalence of legacy sources, with greater legacy than active concentration contributions for both Cl− and the metals, and active contributions playing a greater role for chloride than for the metals. The relatively simple first‐order methodology developed and applied in the study can be used to screen commonly available stream monitoring data for possible distinction of active and legacy contributions of any hydrochemical constituent in and across various hydrological catchment settings.

MAPPING OF 3D EYE-TRACKING IN URBAN OUTDOOR ENVIRONMENTS

Abstract. New geospatial technologies and ubiquitous sensing allow new insights into people’s spatial practices and experiences of public spaces. These tools offer new data streams for analysis and interpretation of social phenomena. Mobile augmented reality tools such as smartphones and wearables merge the experience of entangled online and offline spaces in citizen’s daily life. This paper demonstrates a concept that combines eye-tracking tools with innovative mapping in order to enhance the interpretability of real outdoor environmental experiences. Through videogrammetry, a participants’ head posture can be reconstructed. Subsequently the fixations measured through eye-tracking are projected onto a 3D point cloud of the surrounding environment. The presented methodological approach is implemented in the interdisciplinary project DigitAS – The Digital, Affects and Space – which investigates the perception of public places as spaces of recreation, security or fear. The project’s Mixed Methods approach combined qualitative, mobile, in-situ and reconstructive methods with eye-tracking in an outdoor setting. Potentials of the geospatial mapping concept for social science research is discussed.

A Lightweight Concept Drift Detection and Adaptation Framework for IoT Data Streams

In recent years, with the increasing popularity of "Smart Technology", the number of Internet of Things (IoT) devices and systems have surged significantly. Various IoT services and functionalities are based on the analytics of IoT streaming data. However, IoT data analytics faces concept drift challenges due to the dynamic nature of IoT systems and the ever-changing patterns of IoT data streams. In this article, we propose an adaptive IoT streaming data analytics framework for anomaly detection use cases based on optimized LightGBM and concept drift adaptation. A novel drift adaptation method named Optimized Adaptive and Sliding Windowing (OASW) is proposed to adapt to the pattern changes of online IoT data streams. Experiments on two public datasets show the high accuracy and efficiency of our proposed adaptive LightGBM model compared against other state-of-the-art approaches. The proposed adaptive LightGBM model can perform continuous learning and drift adaptation on IoT data streams without human intervention.

PR-sketch

Computing per-key aggregation is indispensable in streaming data analysis formulated as two phases, an update phase and a recovery phase. As the size and speed of data streams rise, accurate per-key information is useful in many applications like anomaly detection, attack prevention, and online diagnosis. Even though many algorithms have been proposed for per-key aggregation in stream processing, their accuracy guarantees only cover a small portion of keys. In this paper, we aim to achieve nearly full accuracy with limited resource usage. We follow the line of sketch-based techniques. We observe that existing methods suffer from high errors for most keys. The reason is that they track keys by complicated mechanism in the update phase and simply calculate per-key aggregation from some specific counter in the recovery phase. Therefore, we present PR-Sketch, a novel sketching design to address the two limitations. PR-Sketch builds linear equations between counter values and per-key aggregations to improve accuracy, and records keys in the recovery phase to reduce resource usage in the update phase. We also provide an extension called fast PR-Sketch to improve processing rate further. We derive space complexity, time complexity, and guaranteed error probability for both PR-Sketch and fast PR-Sketch. We conduct trace-driven experiments under 100K keys and 1M items to compare our algorithms with multiple state-of-the-art methods. Results demonstrate the resource efficiency and nearly full accuracy of our algorithms.