Real Data Streams Research Articles

Clustering categorical data streams

In this paper, we propose an efficient clustering algorithm for analyzing categorical data streams. It has been proved that the proposed algorithm uses small memory footprints. We provide empirical analysis on the performance of the algorithm in clustering both synthetic and real data streams.

Active complex event processing over event streams

State-of-the-art Complex Event Processing technology (CEP), while effective for pattern query execution, is limited in its capability of reacting to opportunities and risks detected by pattern queries. Especially reactions that affect the query results in turn have not been addressed in the literature. We propose to tackle these unsolved problems by embedding active rule support within the CEP engine, henceforth called Active CEP (ACEP). Active rules in ACEP allow us to specify a pattern query's dynamic condition and real-time actions. The technical challenge is to handle interactions between queries and reactions to queries in the high-volume stream execution. We hence introduce a novel stream-oriented transactional model along with a family of stream transaction scheduling algorithms that ensure the correctness of concurrent stream execution. We demonstrate the power of ACEP technology by applying it to the development of a healthcare system being deployed in UMass Medical School hospital. Through extensive performance experiments using real data streams, we show that our unique Low-Water-Mark stream transaction scheduler, customized for streaming environments, successfully achieves near-real-time system responsiveness and gives orders-of-magnitude better throughput than our alternative schedulers.

EclatDS: An efficient sliding window based frequent pattern mining method for data streams

Mining frequent patterns over data streams is an interesting problem due to its wide application area. The researchers in this field have been facing two key challenges, namely reduction in runtime and memory usage. In this study, a novel method for efficient mining of frequent patterns over data streams is proposed. The method is based on sliding window model which divides the window into a number of panes. This method provides a new sliding window mechanism by utilizing a set of simple short lists. Each list stores related information about an item in the sliding window. The proposed mechanism dynamically adopts itself with the concept change. This method is empirically evaluated against recently proposed pane based sliding window algorithms. Experimental results on synthetically generated and real life data streams show the superiority of the proposed method with multiple orders of magnitude in terms of runtime and memory usage with respect to other pane based sliding window algorithms.

Prediction-based load shedding for burst data streams

Overload management has become very important in telecommunication networks, especially in the case of monitoring network elements that generate time-varying and burst data streams. Efficient overload management improves the quality of provided services, reduces revenue leakage, and minimizes network failure detection time. We argue that in order to improve the quality of bursty data stream processing, data stream management systems (DSMS) should be introduced into network management infrastructures and should monitor and optimize the execution of data stream queries through load-shedding techniques. We introduce a new mechanism for data prioritization and implement a new load-shedding framework, partially based on the popular TelegraphCQ DSMS. Our approach significantly improves the accuracy of data stream query results by using a prediction module to rank the values using preferential ranks. The prediction module tracks and registers all symptoms of overload, and either prevents overload before it occurs, or minimizes its effects. We present empirical evidence showing that our solution significantly improves data stream query accuracy. The effectiveness of the framework has been verified using both artificial and real data streams collected from network elements.

A RANDOM DECISION TREE ENSEMBLE FOR MINING CONCEPT DRIFTS FROM NOISY DATA STREAMS

Detecting concept drifts and reducing the impact from the noise in real applications of data streams are challenging but valuable for inductive learning. It is especially a challenge in a light demand on the overheads of time and space. However, though a great number of inductive learning algorithms based on ensemble classification models have been proposed for handling concept drifting data streams, little attention has been focused on the detection of the diversity of concept drifts and the influence from noise in data streams simultaneously. Motivated by this, we present a new light-weighted inductive algorithm for concept drifting detection in virtue of an ensemble model of random decision trees (named CDRDT) to distinguish various types of concept drifts from noisy data streams in this article. We use variably small data chunks to generate random decision trees incrementally. Meanwhile, we introduce the inequality of Hoeffding bounds and the principle of statistical quality control to detect the different types of concept drifts and noise. Extensive studies on synthetic and real streaming data demonstrate that CDRDT could effectively and efficiently detect concept drifts from the noisy streaming data. Therefore, our algorithm provides a feasible reference framework of classification for concept drifting data streams with noise.

Effectivity and effective continuity of multifunctions

AbstractIf one wants to compute with infinite objects like real numbers or data streams, continuity is a necessary requirement: better and better (finite) approximations of the input are transformed into better and better (finite) approximations of the output. In case the objects are constructively generated, they can be represented by a finite description of the generating procedure. By effectively transforming such descriptions for the generation of the input (respectively, their codes) into (the code of) a description for the generation of the output another type of computable operation is obtained. Such operations are also called effective. The relationship of both classes of operations has always been a question of great interest.In this paper the setting is extended to the case of multifunctions. Various ways of coding (indexing) sets are discussed and their relationship is investigated. Moreover, effective versions of several continuity notions for multifunctions are introduced. For each of these notions an indexing system for sets is exhibited so that the multifunctions that are effective with respect to this indexing system are exactly the multifunction which are effectively continuous with respect to the continuity notion under consideration. Mostly, in addition to being effective the multifunctions need also possess certain witnessing functions. Important special cases are discussed where such witnessing functions always exist.

Auditory Augmentation

With auditory augmentation, the authors describe building blocks supporting the design of data representation tools, which unobtrusively alter the auditory characteristics of structure-borne sounds. The system enriches the structure-borne sound of objects with a sonification of (near) real time data streams. The object’s auditory gestalt is shaped by data-driven parameters, creating a subtle display for ambient data streams. Auditory augmentation can be easily overlaid to existing sounds, and does not change prominent auditory features of the augmented objects like the sound’s timing or its level. In a peripheral monitoring situation, the data stay out of the users’ attention, which thereby remains free to focus on a primary task. However, any characteristic sound change will catch the users’ attention. This article describes the principles of auditory augmentation, gives an introduction to the Reim Software Toolbox, and presents the first observations made in a preliminary long-term user study.

Guest editorial: Thematic issue on ‘Adaptive Soft Computing Techniques and Applications’

In the last 20years, we have witnessed the remarkable progress in computational-intelligence modelling for various applications. However, a majority of these researches make one fundamental assumption that sufficient and representative data are provided in advance for training. Because this assumption often does not hold in many real applications, recent efforts on adaptive soft computing techniques, such as evolving connectionist systems (ECOS), incremental learning and other adaptive learning models aim to relax the “sufficiency” requirement by continuously updating a model which keeps learning from data streams. ECOS addresses the learning from a data stream or chunks of data whose underlying distribution changes over time, by training a neural network continuously and adapting its structure and functionality through repeated interactions with the environment or other learning systems. Similarly, incremental learning develops the ability of a computational model to continuously accumulate knowledge learned over time from noisy and/or incomplete data. In practice, incremental learning is also featured by one-pass property, which enables the algorithm to work with real time data streams presented only once to the learning machine. Other adaptive soft computing

Efficient incremental mining of contrast patterns in changing data

A contrast pattern is a set of items (itemset) whose frequency differs significantly between two classes of data. Such patterns describe distinguishing characteristics between datasets, are meaningful to human experts, have strong discriminating ability and can be used for powerful classifiers. Incrementally mining such patterns is very important for evolving datasets, where transactions can be either inserted or deleted and mining needs to be repeated after changes occur. When the change is small, it is undesirable to carry out mining from scratch. Rather, the set of previously mined contrast patterns should be reused where possible to compute the new patterns. A primary example of evolving data is a data stream, where the data is a sequence of continuously arriving transactions (or itemsets). In this paper, we propose an efficient technique for incrementally mining contrast patterns. Our algorithm particularly aims to avoid redundant computation which might occur due to simultaneous transaction insertion and deletion, as is the case for data streams. In an experimental study using real and synthetic data streams, we show our algorithm can be substantially faster than the previous approach.

A shared execution strategy for multiple pattern mining requests over streaming data

In diverse applications ranging from stock trading to traffic monitoring, popular data streams are typically monitored by multiple analysts for patterns of interest. These analysts may submit similar pattern mining requests, such as cluster detection queries, yet customized with different parameter settings. In this work, we present an efficient shared execution strategy for processing a large number of density-based cluster detection queries with arbitrary parameter settings. Given the high algorithmic complexity of the clustering process and the real-time responsiveness required by streaming applications, serving multiple such queries in a single system is extremely resource intensive. The naive method of detecting and maintaining clusters for different queries independently is often in-feasible in practice, as its demands on system resources increase dramatically with the cardinality of the query workload. To overcome this, we analyze the interrelations between the cluster sets identified by queries with different parameters settings, including both pattern-specific and window-specific parameters. We introduce the notion of the growth property among the cluster sets identified by different queries, and characterize the conditions under which it holds. By exploiting this growth property we propose a uniform solution, called Chandi , which represents identified cluster sets as one single compact structure and performs integrated maintenance on them -- resulting in significant sharing of computational and memory resources. Our comprehensive experimental study, using real data streams from domains of stock trades and moving object monitoring, demonstrates that Chandi is on average four times faster than the best alternative methods, while using 85% less memory space in our test cases. It also shows that Chandi scales in handling large numbers of queries on the order of hundreds or even thousands under high input data rates.

HIDS

In the research of high-speed data streams, large amounts of synthetic data are needed. These days, more and more researchers focus on hierarchical multi-dimensional data streams or data sets, which is beyond the ability of traditional synthetic data generators. In this paper we propose a two-phased method to generate hierarchical multi-dimensional data streams, in which a tree-like structure is built first, and then an unlimited number of items chosen among the tree leaves according to a distribution are inserted into the stream. Our generator, HIDS, integrates all of the functions of existing data generators, and can customize the tree structure according to usersý requirements, producing tree structures such as equal-depth trees, equal-fan-out trees, balanced trees and different-fan-out trees. An experimental study using real data streams shows that HIDS can generate data streams tailored to specific applications.

On the Continuity of Effective Multifunctions

If one wants to compute with infinite objects like real numbers or data streams, continuity is a necessary requirement: better and better (finite) approximations of the input are transformed in better and better (finite) approximations of the output. In case the objects are constructively generated, they can be represented by a finite description of the generating procedure. By effectively transforming such descriptions for the generation of the input (respectively, their codes) in (the code of) a description for the generation of the output another type of computable operation is obtained. Such operations are also called effective. The relationship of both classes of operations has always been a question of great interest and well known theorems such as those of Myhill and Shepherdson, Kreisel, Lacombe and Shoenfield, Ceĭtin, and/or Moschovakis present answers for important special cases. A general, unifying approach has been developed by the present author in [D. Spreen. On effective topological spaces. The Journal of Symbolic Logic, 63 (1998), 185–221. Corrections ibid., 65 (2000), 1917–1918].In this paper the approach is extended to the case of multifunctions. Such functions appear very naturally in applied mathematics, logic and theoretical computer science. Various ways of coding (indexing) sets are discussed and effective versions of several continuity notions for multifunctions are introduced. For each of these notions an indexing system for sets is exhibited so that the multifunctions that are effective with respect to this indexing system and possess certain witness functions are exactly the multifunction which are effectively continuous with respect to the continuity notion under consideration. Important special cases are discussed where such witnessing functions always exist.

Efficient Process of Top-k Range-Sum Queries over Multiple Streams with Minimized Global Error

Due to the resource limitation in the data stream environments, it has been reported that answering user queries according to the wavelet synopsis of a stream is an essential ability of a Data Stream Management System (DSMS). In the literature, recent research has been elaborated upon minimizing the local error metric of an individual stream. However, many emergent applications, such as stock marketing and sensor detection, also call for the need of recording multiple streams in a commercial DSMS. As shown in our thorough analysis and experimental studies, minimizing global error in multiple-stream environments leads to good reliability for DSMS to answer the queries; in contrast, only minimizing local error may lead to significant loss of query accuracy. As such, we first study in this paper the problem of maintaining the wavelet coefficients of multiple streams within collective memory so that the predetermined global error metric is minimized. Moreover, we also examine a promising application in the multistream environment, i.e., the queries for top-k range sum. We resolve the problem of efficient top-k query processing with minimized global error by developing a general framework. For the purposes of maintaining the wavelet coefficients and processing top-k queries, several well-designed algorithms are utilized to optimize the performance of each primary component of this general framework. We also evaluate the proposed algorithms empirically on real and simulated data streams and show that our framework can process top-k queries accurately and efficiently.

Comparing data streams using hamming norms (how to zero in)

Massive data streams are now fundamental to many data processing applications. For example, Internet routers produce large scale diagnostic data streams. Such streams are rarely stored in traditional databases and instead must be processed "on the fly" as they are produced. Similarly, sensor networks produce multiple data streams of observations from their sensors. There is growing focus on manipulating data streams and, hence, there is a need to identify basic operations of interest in managing data streams, and to support them efficiently. We propose computation of the Hamming norm as a basic operation of interest. The Hamming norm formalizes ideas that are used throughout data processing. When applied to a single stream, the Hamming norm gives the number of distinct items that are present in that data stream, which is a statistic of great interest in databases. When applied to a pair of streams, the Hamming norm gives an important measure of (dis)similarity: the number of unequal item counts in the two streams. Hamming norms have many uses in comparing data streams. We present a novel approximation technique for estimating the Hamming norm for massive data streams; this relies on what we call the "l/sub 0/ sketch" and we prove its accuracy. We test our approximation method on a large quantity of synthetic and real stream data, and show that the estimation is accurate to within a few percentage points.

Clustering data streams: theory and practice

The data stream model has recently attracted attention for its applicability to numerous types of data, including telephone records, Web documents, and clickstreams. For analysis of such data, the ability to process the data in a single pass, or a small number of passes, while using little memory, is crucial. We describe such a streaming algorithm that effectively clusters large data streams. We also provide empirical evidence of the algorithm's performance on synthetic and real data streams.

One-pass wavelet decompositions of data streams

We present techniques for computing small space representations of massive data streams. These are inspired by traditional wavelet-based approximations that consist of specific linear projections of the underlying data. We present general "sketch"-based methods for capturing various linear projections and use them to provide pointwise and rangesum estimation of data streams. These methods use small amounts of space and per-item time while streaming through the data and provide accurate representation as our experiments with real data streams show.

Spectrally efficient multiplexing of O-QPSK or VSB signals using wavelet packet-based filter banks

This paper presents fundamental principles for the design of a data transmission scheme that obtains spectrally efficient orthogonal frequency division multiplexing of digital vestigial sideband (VSB) signals. Real binary input data streams are fed into a synthesis filter bank from wavelet packet theory. Mobile satellite and personal communications applications are considered. Desirable properties of this approach are identified and discussed. A new proof of the equivalence between digital vestigial sideband and OQPSK is derived that enables convenient use of real binary signal inputs and descriptions in terms of VSB concepts. Two new closely related digital filters are added at the roots of two identical trees to obtain VSB multiplexer channels that are closely packed in frequency. Frequency and time resolution from multiresolution theory are related to bandwidth concepts. It is shown how bandwidth on demand by factors of 2 can be obtained.

Estimation of bit-level transition activity in data-paths based on word-level statistics and conditional entropy

Accurate models for the calculation of bit-level switching activity from word-level statistics are presented. The dual-bit type (DBT) model is combined with an entropy-based transition activity calculation. Assuming that the input signal of a DSP system is described by a random Gaussian process, the conditional entropy per bit is estimated by means of a novel sigmoidal model. An accurate polynomial approximation to the conditional entropy of the sign bits has been developed permitting the direct calculation of the switching activity. The propagation of signals through common data-path operators is studied. A triple-bit type (TBT) model is introduced to accurately describe the three activity regions of the multiplier output. Moreover, the proposed model is applied for the estimation of the switching activity at the nodes of a multiply–accumulate (MAC) architecture implementing a 24-tap FIR filter, for both synthetic and real data streams. The average error between measured and estimated values of the switching activity is less than 6% in all the cases encountered.

Visualization in biomedical computing

Visualizable objects in biology and medicine extend across a vast range of scale, from individual molecules and cells, to the varieties of tissue and interstitial interfaces, to complete organs, organ systems and body parts, and include functional attributes of these systems, such as biophysical, biomechanical and physiological properties. Medical applications include accurate anatomy and function mapping, enhanced diagnosis, accurate treatment planning and rehearsal, and education/training. However, the greatest potential for revolutionary innovation in the practice of medicine lies in direct, fully immersive, real-time multisensory fusion of real and virtual information data streams into online, real-time visualizations available during an actual clinical procedure. Current high-performance computers and advanced image processing capabilities have facilitated major progress toward realization of this goal. With these advances in hand, there are several important applications possible to be delivered soon that will have a significant impact on the practice of medicine and on biological research.

3-D visualization in biomedical applications.

Visualizable objects in biology and medicine extend across a vast range of scale, from individual molecules and cells through the varieties of tissue and interstitial interfaces to complete organs, organ systems, and body parts. These objects include functional attributes of these systems, such as biophysical, biomechanical, and physiological properties. Visualization in three dimensions of such objects and their functions is now possible with the advent of high-resolution tomographic scanners and imaging systems. Medical applications include accurate anatomy and function mapping, enhanced diagnosis, accurate treatment planning and rehearsal, and education/training. Biologic applications include study and analysis of structure-to-function relationships in individual cells and organelles. The potential for revolutionary innovation in the practice of medicine and in biologic investigations lies in direct, fully immersive, real-time multisensory fusion of real and virtual information data streams into online, real-time visualizations available during actual clinical procedures or biological experiments. Current high-performance computing, advanced image processing, and high-fidelity rendering capabilities have facilitated major progress toward realization of these goals. With these advances in hand, there are several important applications of three-dimensional visualization that will have a significant impact on the practice of medicine and on biological research.