High-speed Data Streams Research Articles

A complex event processing framework for an adaptive language learning system

Ubiquitous learning applications and worldwide educational websites such as MOOC (Massive Open Online Courses) are rapidly producing large volume of user data. Current delayed analysis processing in adaptive language learning systems is difficult to cope with the high-speed and high-volume data streams. To overcome this problem, we introduce a complex event processing (CEP) framework for an Adaptive Language Learning System. The system consists of an event adapter sub-system that can process various inputs such as voice, video, text and other interaction events. The event adapter extracts relevant data to support the operational events module, the learning activity events module and the learner knowledge space events module. These three modules in the event hierarchies provide support to the learner adaptation and learner visual analytics modules. In this study, we conduct three simulations to evaluate the initialization time, delay time and throughput of the proposed system. Each of the experiments simulates 1000 learners and 1000 rules and generates 10 events per second. The results indicate the CEP framework is efficient with a processing delay of less than 1.2μs and throughput of 80,000 events per second. We conclude by discussing the study’s implications and suggest ideas for future research.

A Parameter Space Framework for Online Outlier Detection Over High-Volume Data Streams

In diverse applications ranging from social networks to location-based online services to traffic monitoring, data streams are continuously monitored by multiple outlier analysts customized with different parameter settings. Real-time response to such complex outlier analytics in high-speed streaming data has been recognized as critical for many domains. In this paper, we propose a parameter space framework, called PSOD, for online outlier detection over sliding window streams to support a large variety of query requests in parameter space with both diverse pattern and window parameter settings. First, we design an ingenious neighbor table that records the neighbors for each point in different distance intervals and different slides, which enables us to maximally reuse the already acquired neighbor information across the entire parameter space. In addition, we propose a series of shared strategies in sliding window environment to minimize processing cost by eliminating the redundant query requests. Moreover, the PSOD effectively transforms the query group in 4-D parameter space into a periodic query group in 3-D parameter space to minimize the number of queries. Our experimental study on three real-world steaming data demonstrates that our PSOD successfully drives down the CPU costs by more than 100 folds compared with the state-of-the-art method.

BRPL: Backpressure RPL for High-Throughput and Mobile IoTs

RPL, an IPv6 routing protocol for Low power Lossy Networks (LLNs), is considered to be the de facto routing standard for the Internet of Things (IoT). However, more and more experimental results demonstrate that RPL performs poorly when it comes to throughput and adaptability to network dynamics. This significantly limits the application of RPL in many practical IoT scenarios, such as an LLN with high-speed sensor data streams and mobile sensing devices. To address this issue, we develop BRPL, an extension of RPL, providing a practical approach that allows users to smoothly combine any RPL Object Function (OF) with backpressure routing. BRPL uses two novel algorithms, QuickTheta and QuickBeta, to support time-varying data traffic loads and node mobility respectively. We implement BRPL on Contiki OS, an open-source operating system for the Internet of Things. We conduct an extensive evaluation using both real-world experiments based on the FIT IoT-LAB testbed and large-scale simulations using Cooja over 18 virtual servers on the Cloud. The evaluation results demonstrate that BRPL not only is fully backward compatible with RPL (i.e., devices running RPL and BRPL can work together seamlessly), but also significantly improves network throughput and adaptability to changes in network topologies and data traffic loads. The observed packet loss reduction in mobile networks is, at a minimum, 60 and up to 1,000 percent can be seen in extreme cases.

Real-time secure communication for Smart City in high-speed Big Data environment

The recent development in the technology brings the concept of Smart City that is achieved through real-time city related intelligent decisions by analyzing the data harvested from various smart systems in the city using millions of sensors and devices connected over the Internet, termed as Internet of Things (IoT). These devices generate the overwhelming volume of high-speed streaming data, termed as Big Data. However, the generation of city data at a remote location and then transmitting it to central city servers for analysis purpose raises the concerns of security and privacy. On the other hand, providing security to such Big Data streaming requires a high-speed security system that can work in a real-time environment without providing any delay that may slow down the overall performance of the Smart City System. To overthrown these challenges, in this paper, we proposed an efficient and real-time Smart City security system by providing strong intrusion detection at intelligent city building (ICB) and also a security protocol to protect the communication between the remote smart system(RSS)/User and the city analysis building, i.e., ICB. The proposed communication security protocol consists of various phases, i.e., registration phase, session key exchange phase, session key revocation phase, and data transmission phases from RSS to ICB as well as from User to ICB. Vast security analyses are performed to evaluate the credibility of the system. The proposed system is also evaluated on efficiency in terms of computation cost and throughput of overall functions used in the system. The system’s evaluation and the comparative study with existing system show that the prosed system is secure, more efficient, and able to work in a real-time, high-speed Smart City environment.

Nearest Neighbor Classification for High-Speed Big Data Streams Using Spark

Mining massive and high-speed data streams among the main contemporary challenges in machine learning. This calls for methods displaying a high computational efficacy, with ability to continuously update their structure and handle ever-arriving big number of instances. In this paper, we present a new incremental and distributed classifier based on the popular nearest neighbor algorithm, adapted to such a demanding scenario. This method, implemented in Apache Spark, includes a distributed metric-space ordering to perform faster searches. Additionally, we propose an efficient incremental instance selection method for massive data streams that continuously update and remove outdated examples from the case-base. This alleviates the high computational requirements of the original classifier, thus making it suitable for the considered problem. Experimental study conducted on a set of real-life massive data streams proves the usefulness of the proposed solution and shows that we are able to provide the first efficient nearest neighbor solution for high-speed big and streaming data.

Reliability or Sustainability

As a typical cyber-physical system (CPS), smart water distribution networks require monitoring of underground water pipes with high sample rates for precise data analysis and water network control. Due to poor underground wireless channel quality and long-range communication requirements, high transmission power is typically adopted to communicate high-speed sensor data streams, posing challenges for long-term sustainable monitoring. In this article, we develop the first sustainable water sensing system, exploiting energy harvesting opportunities from water flows. Our system does this by scheduling the transmission of a subset of the data streams, whereas other correlated streams are estimated using autoregressive models based on the sound-velocity propagation of pressure signals inside water networks. To compute the optimal scheduling policy, we formalize a stochastic optimization problem to maximize the estimation reliability while ensuring the system’s sustainable operation under dynamic conditions. We develop data transmission scheduling (DTS), an asymptotically optimal scheme, and FAST-DTS, a lightweight online algorithm that can adapt to arbitrary energy and correlation dynamics. Using more than 170 days of real data from our smart water system deployment and conducting in vitro experiments to our small-scale testbed, our evaluation demonstrates that Fast-DTS significantly outperforms three alternatives, considering data reliability, energy utilization, and sustainable operation.

Improved clustering algorithm based on high-speed network data stream

With the continuous development of network technology, the attack has become the biggest threat to the stable operation of the network. Intrusion detection technology is a proactive safety protection measure which provides real-time monitoring of internal attacks, external attacks, and misuse. Traditional intrusion detection system is short of adaptability due to the complication and scale of the network. The main problem is that the real-time performance of the network is poor and the reliability is not high. This paper designs the intrusion detection mechanism combined with data stream clustering algorithm and intrusion detection system to solve the problem in processing a large amount of high-speed data streams. The performance of processing data streams is improved through the clustering algorithm based on density and the sliding window and the experiments show that the intrusion detection efficiency is higher than DenStream algorithm.

Time-weighted counting for recently frequent pattern mining in data streams

How can we discover interesting patterns from time-evolving high-speed data streams? How to analyze the data streams quickly and accurately, with little space overhead? How to guarantee the found patterns to be self-consistent? High-speed data stream has been receiving increasing attention due to its wide applications such as sensors, network traffic, social networks, etc. The most fundamental task on the data stream is frequent pattern mining; especially, focusing on recentness is important in real applications. In this paper, we develop two algorithms for discovering recently frequent patterns in data streams. First, we propose TwMinSwap to find top-k recently frequent items in data streams, which is a deterministic version of our motivating algorithm TwSample providing theoretical guarantees based on item sampling. TwMinSwap improves TwSample in terms of speed, accuracy, and memory usage. Both require only O(k) memory spaces and do not require any prior knowledge on the stream such as its length and the number of distinct items in the stream. Second, we propose TwMinSwap-Is to find top-k recently frequent itemsets in data streams. We especially focus on keeping self-consistency of the discovered itemsets, which is the most important property for reliable results, while using O(k) memory space with the assumption of a constant itemset size. Through extensive experiments, we demonstrate that TwMinSwap outperforms all competitors in terms of accuracy and memory usage, with fast running time. We also show that TwMinSwap-Is is more accurate than the competitor and discovers recently frequent itemsets with reasonably large sizes (at most 5–7) depending on datasets. Thanks to TwMinSwap and TwMinSwap-Is, we report interesting discoveries in real world data streams, including the difference of trends between the winner and the loser of U.S. presidential candidates, and temporal human contact patterns.

A 28 nm, 475 mW, and 0.4–1.7 GHz Embedded Transceiver Front-End Enabling High-Speed Data Streaming Within Home Cable Networks

A 28 nm CMOS software-defined transceiver (SDTRX) enabling high-speed data (HSD) streaming, including ultra HD TV, within home cable networks is presented. By making efficient use of available cable bandwidth, the SDTRX dynamically handles up to 1024-QAM OFDM-modulated HSD streams. This paper addresses SDTRX system-level design methodology as the key driver in enabling performance optimization for achieving a wide frequency range of operation at lowest power and area consumption. By employing an optimized architecture constructed on available state-of-the-art 28 nm functional building blocks, the monolithic SDTRX consists of a mixer-based harmonic rejection RX with a digital-to-analog converter-based TX and a smart phase-locked loop system. It operates over 0.4–1.7 GHz frequency range while consuming less than 475 mW in half-duplex mode. Moreover, by developing a simple transmitter (TX) to receiver (RX) loopback circuit, the system is enabled to efficiently calibrate TX output power and to remove the need for a dedicated external pin. This low-cost SDTRX is embedded in various 28 nm CMOS multimedia system-on-chip and is, to the authors’ knowledge, the first reported transceiver front-end to enable true HSD streaming within home cable networks.

On Frequency Estimation and Detection of Frequent Items in Time Faded Streams

We deal with the problem of detecting frequent items in a stream under the constraint that items are weighted, and recent items must be weighted more than older ones. This kind of problem naturally arises in a wide class of applications in which recent data is considered more useful and valuable with regard to older, stale data. The weight assigned to an item is, therefore, a function of its arrival timestamp. As a consequence, whilst in traditional frequent item mining applications we need to estimate frequency counts, we are instead required to estimate decayed counts . These applications are said to work in the time fading model. Two sketch-based algorithms for processing time-decayed streams have been recently published independently near the end of 2016. The Filtered Space Saving with Quasi-Heap (FSSQ) algorithm, besides a sketch, also uses an additional data structure called quasi-heap to maintain frequent items. Forward Decay Count-Min Space Saving (FDCMSS), our algorithm, cleverly combines key ideas borrowed from forward decay, the Count-Min sketch and the Space Saving algorithm. Therefore, it makes sense to compare and contrast the two algorithms in order to fully understand their strengths and weaknesses. We show, through extensive experimental results, that FSSQ is better for detecting frequent items than for frequency estimation. The use of the quasi-heap data structure slows down the algorithm owing to the huge number of maintenance operations. Therefore, FSSQ may not be able to cope with high-speed data streams. FDCMSS is better suitable for frequency estimation; moreover, it is extremely fast and can be used in the context of high-speed data streams and for the detection of frequent items as well, since its recall is always greater than 99%, even when using an extremely tiny amount of space. Therefore, FDCMSS proves to be an overall good choice when considering jointly the recall, precision, average relative error and the speed.

Real-time Public Mood Tracking of Chinese Microblog Streams with Complex Event Processing

There are not many real-time public mood tracking frameworks over social media streams at present. Real-time public mood tracking over microblogs becomes necessary for further studies with low-latency requirements. To address this issue, we propose a hierarchical framework for real-time public mood time series tracking over Chinese microblog streams using complex event processing. Complex event processing is able to handle high-speed and high-volume data streams. First, we transform microblogs into emotional microblog events through the text sentiment analysis. Then, we apply an online batch window technique to summarize the public mood in different periods. For the public mood time series, we use smoothing and trend following methods to find the rising or falling trends of the public mood. Finally, we apply the method to 6606 microblogs to verify its feasibility. The result demonstrates that the proposed model is not only feasible but also effective.

Fast Memory Efficient Local Outlier Detection in Data Streams

Outlier detection is an important task in data mining, with applications ranging from intrusion detection to human gait analysis. With the growing need to analyze high speed data streams, the task ...

Fast Memory Efficient Local Outlier Detection in Data Streams

Outlier detection is an important task in data mining, with applications ranging from intrusion detection to human gait analysis. With the growing need to analyze high speed data streams, the task of outlier detection becomes even more challenging as traditional outlier detection techniques can no longer assume that all the data can be stored for processing. While the well-known Local Outlier Factor (LOF) algorithm has an incremental version, it assumes unbounded memory to keep all previous data points. In this paper, we propose a memory efficient incremental local outlier (MiLOF) detection algorithm for data streams, and a more flexible version (MiLOF_F), both have an accuracy close to Incremental LOF but within a fixed memory bound. Our experimental results show that both proposed approaches have better memory and time complexity than Incremental LOF while having comparable accuracy. In addition, we show that MiLOF_F is robust to changes in the number of data points, the number of underlying clusters and the number of dimensions in the data stream. These results show that MiLOF/MiLOF_F are well suited to application environments with limited memory (e.g., wireless sensor networks), and can be applied to high volume data streams.

Mining Frequent Item Sets in Asynchronous Transactional Data Streams over Time Sensitive Sliding Windows Model

EPs (Extracting Frequent Patterns) from the continuous transactional data streams is a challenging and critical task in some of the applications, such as web mining, data analysis and retail market, prediction and network monitoring, or analysis of stock market exchange data. Many algorithms have been developed previously for mining FPs (Frequent Patterns) from a data stream. Such algorithms are currently highly required to develop new solutions and approaches to the precise handling of data streams. New techniques, solutions, or approaches are developed to address unbounded, ordered, and continuous sequences of data and for the generation of data at a rapid speed from data streams. Hence, extracting FPs using fresh or recent data involves the high-level analysis of data streams. We have suggested an efficient technique for the window sliding model; this technique extracts new and fresh FPs from high-speed data streams. In this study, a CPILT (Compacted Tree Compact Pattern Tree) is developed to capture the latest contents in the stream and to efficiently remove outdated contents from the data stream. The main concept introduced in this work on CPILT is the dynamic restructuring of a tree, which is helpful in producing a compacted tree and the frequency descending structure of a tree on runtime. With the help of the mining technique of FP growth, a complete list of new and fresh FPs is obtained from a CPILT using an existing window. The memory usage and time complexity of the latest FPs in high-speed data streams can efficiently be determined through proper experimentation and analysis.

Evaluation of a New Incremental Classification Tree Algorithm for Mining High Speed Data Streams

Evaluation of a New Incremental Classification Tree Algorithm for Mining High Speed Data Streams

Stream Data Mining: Platforms, Algorithms, Performance Evaluators and Research Trends

Streaming data are potentially infinite sequence of incoming data at very high speed and may evolve over the time. This causes several challenges in mining large scale high speed data streams in real time. Hence, this field has gained a lot of attention of researchers in previous years. This paper discusses various challenges associated with mining such data streams. Several available stream data mining algorithms of classification and clustering are specified along with their key features and significance. Also, the significant performance evaluation measures relevant in streaming data classification and clustering are explained and their comparative significance is discussed. The paper illustrates various streaming data computation platforms that are developed and discusses each of them chronologically along with their major capabilities. This paper clearly specifies the potential research directions open in high speed large scale data stream mining from algorithmic, evolving nature and performance evaluation measurement point of view. Finally, Massive Online Analysis (MOA) framework is used as a use case to show the result of key streaming data classification and clustering algorithms on the sample benchmark dataset and their performances are critically compared and analyzed based on the performance evaluation parameters specific to streaming data mining.

A novel online multi-label classifier for high-speed streaming data applications

In this paper, a high-speed online neural network classifier based on extreme learning machines for multi-label classification is proposed. In multi-label classification, each of the input data sample belongs to one or more than one of the target labels. The traditional binary and multi-class classification where each sample belongs to only one target class forms the subset of multi-label classification. Multi-label classification problems are far more complex than binary and multi-class classification problems, as both the number of target labels and each of the target labels corresponding to each of the input samples are to be identified. The proposed work exploits the high-speed nature of the extreme learning machines to achieve real-time multi-label classification of streaming data. A new threshold-based online sequential learning algorithm is proposed for high speed and streaming data classification of multi-label problems. The proposed method is experimented with six different datasets from different application domains such as multimedia, text, and biology. The hamming loss, accuracy, training time and testing time of the proposed technique is compared with nine different state-of-the-art methods. Experimental studies shows that the proposed technique outperforms the existing multi-label classifiers in terms of performance and speed.

Adaptive Model Rules From High-Speed Data Streams

Decision rules are one of the most expressive and interpretable models for machine learning. In this article, we present Adaptive Model Rules (AMRules), the first stream rule learning algorithm for regression problems. In AMRules, the antecedent of a rule is a conjunction of conditions on the attribute values, and the consequent is a linear combination of the attributes. In order to maintain a regression model compatible with the most recent state of the process generating data, each rule uses a Page-Hinkley test to detect changes in this process and react to changes by pruning the rule set. Online learning might be strongly affected by outliers. AMRules is also equipped with outliers detection mechanisms to avoid model adaption using anomalous examples. In the experimental section, we report the results of AMRules on benchmark regression problems, and compare the performance of our system with other streaming regression algorithms.

Review of Data mining (Knowledge discovery) in the Future

Data mining (sometimes called data or knowledge discovery) is the process of analyzing data from different perspectives and summarizing it into useful information - information that can be used to increase revenue, cuts costs, or both. Data mining software is one of a number of analytical tools for analyzing data. It allows users to analyze data from many different dimensions or angles, categorize it, and summarize the relationships identified. Technically, data mining is the process of finding correlations or patterns among dozens of fields in large relational databases. The term data mining is often used to apply to the two separate processes of knowledge discovery and prediction. Knowledge discovery provides explicit information that has a readable form and can be understood by a user (e.g., association rule mining). Forecasting, or predictive modeling provides predictions of future events and may be transparent and readable in some approaches (e.g., rule-based systems) and opaque in others such as neural networks. Moreover, some data-mining systems such as neural networks are inherently geared towards prediction and pattern recognition, rather than knowledge discovery. In Future different Scope of data mining are. 1. Developing a unifying theory of data mining. 2. Scaling up for high dimensional data and high speed data streams. 3. Data mining in a network setting. 4. Data mining for biological and environmental problems. 5. Security, privacy and data integrity. 6. Dealing with non-static, unbalanced and cost-sensitive data

A Novel High Dimensional and High Speed Data Streams Algorithm: HSDStream

This paper presents a novel high speed clustering scheme for high-dimensional data stream. Data stream clustering has gained importance in different applications, for example, network monitoring, intrusion detection, and real-time sensing. High dimensional stream data is inherently more complex when used for clustering because the evolving nature of the stream data and high dimensionality make it non-trivial. In order to tackle this problem, projected subspace within the high dimensions and limited window sized data per unit of time are used for clustering purpose. We propose a High Speed and Dimensions data stream clustering scheme (HSDStream) which employs exponential mov-ing averages to reduce the size of the memory and speed up the processing of projected subspace data stream. It works in three steps: i) initialization, ii) real-time maintenance of core and outlier micro-clusters, and iii) on-demand offline generation of the final clusters. The proposed algorithm is tested against high dimensional density-based projected clustering (HDDStream) for cluster purity, memory usage, and the cluster sensitivity. Experi-mental results are obtained for corrected KDD intrusion detection dataset. These results show that HSDStream outperforms the HDDStream in all performance metrics, especially, the memory usage and the processing speed.