High-speed Data Streams Research Articles

An intrusion detection algorithm based on data streams mining and cognitive computing

With the emergence of large-volume and high-speed streaming data, traditional techniques for mining closed frequent itemsets has become inefficient. Online mining of closed frequent itemsets over streaming data is one of the most important issues in data streams minging. In view of the low efficiency of traditional closed frequent item data mining, a combined data structure based on the principle of cognitive computing is proposed, that is, combining the effective bit first with the extended dictionary frequent item list to form a mixed data structure that can identify the closed frequent information in data streams. At the same time, a variety of pruning strategies based on cognitive computing are proposed to avoid the generation of a large number of intermediate itemsets and to remove the non closed frequent term sets from the Hash Table of Closed Itemsets (CIHT). Closed Frequent Itemset Deletion and Search Strategy (CFIDWSS) is used to effectively add or remove the closed frequent itemsets, so as to greatly reduce the search space and improve the user response speed. The proposed algorithm solves the problem of low efficiency of data streams mining of closed frequent items. On the basis of the above algorithms, this paper proposes a new intrusion detection model. Through the mining of normal or abnormal patterns of data stream information, the corresponding database of network access pattern is established. Then the database is used to detect the intrusion online in real time and improve the detection accuracy of the system. Theoretical and experimental results show that the proposed algorithm and intrusion detection system have good performance.

Lr-Stream: Using latency and resource aware scheduling to improve latency and throughput for streaming applications

Low latency and high throughput are two of the most critical performance requirements for big data stream computing systems. As multi-source high-speed data streams arrive in real time, it is essential to study latency-aware and resource-aware scheduling to reduce latency and increase throughput. In this paper, we propose a latency- and resource-aware scheduling framework (Lr-Stream) targeting stream-oriented big data applications. Our key contributions can be summarized as follows: (1) a stream topology model and resource scheduling model Lr-Stream are proposed, aiming at optimizing latency and throughput; (2) a latency-aware scheduling strategy and a resource-aware scheduling strategy are proposed; (3) Lr-Stream together with monitor, calculator, and deployment function modules are implemented, and integrated into Apache Storm; (4) system metrics are thoroughly evaluated from latency- and resource-aware perspective on a typical distributed stream computing platform. Experimental results demonstrate that the proposed Lr-Stream yields significant performance improvements in terms of reducing system latency and increasing system throughput.

Automated Adaptation and Stabilization of a Tunable WDM Polarization-Independent Receiver on Active Silicon Photonic Platform

We demonstrate automated adaptation, and stabilization of a silicon photonic wavelength-division multiplexing (WDM), polarization-independent receiver. A two-channel, tunable WDM polarization-independent receiver is designed, and used to demonstrate automated WDM polarization control. Using a control algorithm based on Barzilai, and Borwein's two-point step size gradient descent method, we realize automated polarization adaptation, and wavelength stabilization for two arbitrarily polarized input data streams. 10 Gb/s on-off keying, and 20 Gb/s pulse-amplitude modulation 4-level formats are generated as the high-speed input data streams. In addition, we implement a long-duration experiment, in which we measure the bit-error-ratio for continuously varying polarization states, and changing chip temperatures. The experimental results show that, with the automated control, the WDM polarization-independent receiver can adapt, stabilize, and track the arbitrary input polarization states from a standard optical fiber into the transverse electric mode of a silicon waveguide, and simultaneously stabilize the transmitted wavelength channels at various chip temperatures. We also show how the presented WDM polarization-independent receiver scales with N channels, and propose an improved design for large-scale WDM applications.

SAPTM: Towards High-Throughput Per-Flow Traffic Measurement with a Systolic Array-Like Architecture on FPGA

Per-flow traffic measurement has emerged as a critical but challenging task in data centers in recent years in the face of massive network traffic. Many approximate methods have been proposed to resolve the existing resource-accuracy trade-off in per-flow traffic measurement, one of which is the sketch-based method. However, sketches are affected by their high computational cost and low throughput; moreover, their measurement accuracy is hard to guarantee under the conditions of changing network bandwidth or flow size distribution. Recently, FPGAplatforms have been widely deployed in data centers, as they demonstrate a good fit for high-speed network processing. In this work, we aim to address the problem of per-flow traffic measurement from a hardware architecture perspective. We thus design SAPTM, a pipelined systolic array-like architecture for high-throughput per-flow traffic measurement on FPGA. We adopt memory-friendly D-left hashing in the design of SAPTM, which guarantees high space utilization during flow insertion and eviction, successfully addressing the challenge of tracking a high-speed data stream under limited memory resources on FPGA. Evaluations on the Xilinx VCU118 platform with real-world benchmarks demonstrate that SAPTM possesses high space utilization. Comparisons with state-of-the-art sketch-based solutions show that SAPTM outperforms comparison methods in terms of throughput by a factor of 14.1x–70.5x without any accuracy loss.

MOBILE INFOCOMMUNICATION SYSTEMS

Background. An increase in the number of users and services with the support of the required quality of service, an increase in the coverage area of one operator’s radio access network, and most importantly, the emergence of packet-switched technologies and systems, has led to a significant complication of the core mobile communications network and its nodes, an increase in the functionality and intellectualization of the network core. Gradually, it was the core network and the technologies of its functioning that became more significant than the radio access network. Hence, for the construction ofmobile communication systems, more integrated information and telecommunication technologies, especially from the field of network and information technologies, became popular.Objective. The aim of the paper is to present the concept of the architecture of mobile infocommunication systems as systems whose services make a significant, even decisive, contribution to the functioning of the infrastructure of modern society, built on the constant exchange and assimilation of information.Methods. The structural and functional methods of constructing the architectures of information and telecommunication systems are investigated.Results. The signaling network of mobile infocommunications manages the maintenance of both individual wireless user terminals (telephones, smartphones, tablets), and many wireless inter-machine communication devices and the Internet of Things. In addition, the same alarm system supports various telecommunication networks of wireless subscriber access, automobile and other vehicles; creates an interface with wired / cable access networks. Thus, the core network of mobile infocommunications can fully integrate interfaces with all types of telecommunications, which are designed to transmit many low-speed data streams, as well as to transmit high-speed data streams. There is atendency to take over traditional old services, for example, the public telephone network and radio-relay communication lines, and turn them into private services of the mobile infocommunication network, which further enhances their importance. All this creates the prerequisites for the transformation of mobile infocommunication operator systems into the main (determining)players in the telecommunications market. Confirmation of this is the high characteristics that are to be achieved in the networks of the 5th and 6th generations of mobile infocommunications.Conclusions. Mobile infocommunication systems have become a defining component of all telecommunication services provided to users and non-servicing nodes of sensor networks, and the infrastructure of core networks and access networks of mobile infocommunications has begun to determine the general technical infrastructure of telecommunications

Model Compression for IoT Applications in Industry 4.0 via Multiscale Knowledge Transfer

Recently, Industry 4.0 has attracted much attention. It has close relations with the Internet of Things (IoT). On the other hand, convolutional neural networks (CNNs) have shown promising performance in many foundational services of the IoT applications. For the IoT applications with high-speed data streams and the requirement of time-sensitive actions, fast processing is demanded on small-scale platforms or even on IoT devices themselves. Therefore, it is inappropriate to employ cumbersome CNNs in IoT applications, making the study of model compression necessary. In knowledge transfer, it is common to employ a deep, well-trained network, called teacher , to guide a shallow, untrained network, called student , to have better performance. Previous works have made many attempts to transfer single-scale knowledge from teacher to student , leading to degradation of generalization ability. In this article, we introduce multiscale representations to knowledge transfer, which facilitates the generalization ability of student . We divide student and teacher into several stages. Student learns from multiscale knowledge provided by teacher at the end of each stage. Extensive experiments demonstrate the effectiveness of our proposed method both on image classification and on single image super-resolution. The huge performance gap between student and teacher is significantly narrowed down by our proposed method, making student suitable for IoT applications.

A Sketching Approach for Obtaining Real-Time Statistics Over Data Streams in Cloud

Many applications of complex event processing (CEP) in Cloud can tolerate analytical errors to some extent, and it provides us an opportunity to optimize real-time analytics using methods of approximate query processing over big data streams. In this article, we present a novel rules-based sampling technique, which supports to construct sketch over one-pass and high-speed asynchronous data streams and provides accurate answers for different types of analytical queries. Moreover, we propose two methods of distributed sketching implementation, i.e., D-AQP <inline-formula><tex-math notation="LaTeX">$_b$</tex-math></inline-formula> and D-AQP <inline-formula><tex-math notation="LaTeX">$_i$</tex-math></inline-formula> , to make our approach to be compatible with batch processing and interactive processing architectures respectively, and be appropriate for stream processing systems in Cloud. Experimental results with real-world and synthetic datasets indicate that our approach can obtain more accurate estimates and improve two times of system throughput when compared with state-of-the-art Hadoop-based approximate engine BlinkDB. When compared with current batch processing systems Spark and stream processing system Spark-Streaming, our methods of D-AQP <inline-formula><tex-math notation="LaTeX">$_b$</tex-math></inline-formula> and D-AQP <inline-formula><tex-math notation="LaTeX">$_i$</tex-math></inline-formula> can achieve 2 and 4 orders of magnitude improvement on query response time respectively.

Temporally interleaved optical time-stretch imaging

Optical time-stretch imaging has shown potential in diverse fields for its capability of acquiring images at high speed and high resolution. However, its wide application is hindered by the stringent requirement on the instrumentation hardware caused by the high-speed serial data stream. Here we demonstrate temporally interleaved optical time-stretch imaging that lowers the requirement without sacrificing the frame rate or spatial resolution by interleaving the high-speed data stream into multiple channels in the time domain. Its performance is validated with both a United States Air Force (USAF)-1951 resolution chart and a single-crystal diamond film. We achieve a 101 Mfps 1D scanning rate and 3 µm spatial resolution with only a 2.5 GS/s sampling rate by using a two-channel-interleaved system.

An Online Tree-Based Approach for Mining Non-Stationary High-Speed Data Streams

This paper presents a new learning algorithm for inducing decision trees from data streams. In these domains, large amounts of data are constantly arriving over time, possibly at high speed. The proposed algorithm uses a top-down induction method for building trees, splitting leaf nodes recursively, until none of them can be expanded. The new algorithm combines two split methods in the tree induction. The first method is able to guarantee, with statistical significance, that each split chosen would be the same as that chosen using infinite examples. By doing so, it aims at ensuring that the tree induced online is close to the optimal model. However, this split method often needs too many examples to make a decision about the best split, which delays the accuracy improvement of the online predictive learning model. Therefore, the second method is used to split nodes more quickly, speeding up the tree growth. The second split method is based on the observation that larger trees are able to store more information about the training examples and to represent more complex concepts. The first split method is also used to correct splits previously suggested by the second one, when it has sufficient evidence. Finally, an additional procedure rebuilds the tree model according to the suggestions made with an adequate level of statistical significance. The proposed algorithm is empirically compared with several well-known induction algorithms for learning decision trees from data streams. In the tests it is possible to observe that the proposed algorithm is more competitive in terms of accuracy and model size using various synthetic and real world datasets.

Processing streams in a monitoring cloud cluster

The creation of monitoring clusters based on cloud computing technologies is a promising direction for the development of systems for continuous monitoring of objects for various purposes in the web space. Hadoop web-programming environment is the technological basis for the development of algorithmic and software solutions for the synthesis of monitoring clusters, including information security and information counteraction systems. The International Telecommunication Union’ (ITU) recommendations Y. 3510 present the requirements for cloud infrastructure that require monitoring the performance of deployed applications based on the collection of real-world statistics. Often, computing resources of monitoring clusters of cloud data centers are allocated for continuous parallel processing of high-speed streaming data, which imposes new requirements to monitoring technologies, necessitating the creation and research of new models of parallel computing. The need to use service monitoring plays an important role in the cloud computing industry, especially for SLA/QoS assessment, as the application or service may experience problems even if the virtual machines on which the work is taking place appear to be operational. This requires to study the methodological possibilities of organization to study of parallel processing high-speed streaming services with the processing of huge amounts of bit data, and, simultaneously, to estimate the necessary computational resource. In the conditions of high dynamics of changes in the bit rate of information generation from the source, a model of the bit rate of Discretized Stream (DStream) formation is proposed, which has a common application. Based on the poly-burst nature of the bit rate model, a model of group content traffic of any sources of different services processed in the cloud cluster was created. The obtained results made it possible to develop mathematical models of parallel DStreams from sources processed in a cloud cluster via Hadoop technology using the micro-batch architecture of the Spark Streaming module. These models take into account the flow of requests for maintenance from sources of different services, on the one hand, and, on the other hand, the needs of services in bit rate, taking into account the multichannel traffic of sources of various services. At the same time, analytical relations are obtained to calculate the required performance of the Hadoop cluster at a given value of the probability of batch loss.

A Survey of Approximate Quantile Computation on Large-Scale Data

As data volume grows extensively, data profiling helps to extract metadata of large-scale data. However, one kind of metadata, order statistics, is difficult to be computed because they are not mergeable or incremental. Thus, the limitation of time and memory space does not support their computation on large-scale data. In this paper, we focus on an order statistic, quantiles, and present a comprehensive analysis of studies on approximate quantile computation. Both deterministic algorithms and randomized algorithms that compute approximate quantiles over streaming models or distributed models are covered. Then, multiple techniques for improving the efficiency and performance of approximate quantile algorithms in various scenarios, such as skewed data and high-speed data streams, are presented. Finally, we conclude with coverage of existing packages in different languages and with a brief discussion of the future direction in this area.

A Shared Satellite Ground Station Using User-Oriented Virtualization Technology

With the development of space missions, the increasing communication data volume, the diversity of mission-specific interfaces, and the growing security needs, the costs and system complexity of ground stations have increased. The architecture of the existing ground terminal is outdated and the operation and maintenance are labor-intensive, so the terminal components are unable to meet the demand of multi-satellite TT&C in the future. In this paper, a Shared Satellite Ground Station(SSGS) using user-oriented virtualization technology is proposed for coping with complex multi-satellite TT&C in the future. Based on a pool architecture, SSGS utilizes baseband processor virtualization and software-defined networking to implement resource sharing and a network that fuses high-speed and low-speed data streams with customization control. The case analysis verifies that when SSGS has a virtualization rate greater than 1, the execution rate of tasks is much larger than that of traditional ground stations. The case analysis also verifies that the virtualization rate of current commercial equipment is no less than 1.25 through equipment testing.

Quality Improvement through Intensive Care Registries: The Value of Big Data in Clinical Decision Making

Background: Quality of Intensive care has got more attention in case of the high cost of healthcare and the potential for harm. Poor-quality care causes high cost and quality improvement initiatives in the ICU lead to an improvement in outcomes as well as a decrease in costs. One of the crucial tools that allow physicians and nurses to monitor change in a quality improvement effort is the development of an electronic database for data collection and reporting. The objective of Intensive Care Registries is to create a high-quality registry of patients through a collaboration of academic health centers performing uniform data collection with the purpose of improving the quality and accuracy of healthcare decisions and provide a data-driven clinical decision support system for critical care medicine. Methods: This article reviews real-world data sources in healthcare and considers registry as the main tool to address health services and outcomes research questions in critical care, and briefly describes objective, inputs and outputs of intensive care registries. As it can be comprehended from library research, the combination of patient clinical care data, quality parameters, and ICU operating costs, integrated into an electronic database, provides a valuable tool for quality improvement and overall efficiency of offered care. Results: Using Big Data effectively within ICUs for supporting clinical decision making can lead to predict numerous diseases and help to discover new patterns in healthcare. The ability to process multiple high-speed clinical data streams from multiple centers could dramatically improve both healthcare efficiency and patient outcomes. Conclusion: To gain this goal, developing reliable and standardized health analytics platforms as well as quality improvement processes that translate analytical results into new clinical guidelines, is recommended.

Use one‐bit technique to measure the coherence in the time, frequency and space domain in a reverberation chamber

High speed data streams are everywhere in the fifth generation (5G) system measurement. To develop a high efficient and low cost 5G testbed in a reverberation chamber, the authors propose a high efficient measurement method in the time, frequency and space coherence measurement. In this study, the one-bit technique is introduced to measure the time domain, frequency domain and space domain correlations in a reverberation chamber. The coherence time, coherence bandwidth and correlated angle can all be measured using the one-bit correlation; float number calculations are simplified to logical operations which accelerate the data processing greatly. Measurements are performed to verify the results.

Multi-model Z-compression for high speed data streaming and low-power wireless sensor networks

Wireless Sensor Networks (WSNs) have significant limitations in terms of available bandwidth and energy. The limited bandwidth in WSNs can cause delays in message delivery, which does not suit the real-time sensing applications such as a gas leak. Moreover, in such applications, there are multi-modal sensors whose values like temperature, gas concentration, location, and CO$$_2$$ levels must be transmitted together for correct and timely detection of a gas leak. In this paper, we propose a novel Z-order based data compression scheme, Z-compression, to reduce energy consumption and save bandwidth without increasing message delivery latency. Instead of using the popular Huffman tree style based encoding, the Z-compression uses Z-order encoding to map the multidimensional sensing data into one-dimensional binary stream transmitted using a single packet. Our experimental evaluations using different real-world datasets show that the Z-compression has a much better compression ratio, energy, and bandwidth savings than known schemes like LEC, Adaptive-LEC, FELACS, and TinyPack for multi-modal sensor data. Through the extensive experiments, we show that Z-compression is suitable for real-world sensing applications requiring high-stream rate WSNs and delay-tolerant low-power listening WSNs. In high-stream rate WSNs, the Z-compression can save bandwidth and increases the throughput. In low-power listening WSNs, by concatenating the Z-compressed data at selected reporting nodes, we can reduce the duty cycles of the nodes in WSNs, thus prolong the lifetime of the network, and still maintain the low distortion rate.

Nearest Neighbor Classification for High-Speed Big Data Streams using Spark

Nearest Neighbor Classification for High-Speed Big Data Streams using Spark

Continuously Distinct Sampling over Centralized and Distributed High Speed Data Streams

Distinct sampling is fundamental for computing statistics (e.g., the age and gender distribution of distinct users accessing a particular website) depending on the set of distinct keys (e.g., user IDs) in a large and high speed data stream such as a sequence of key-update pairs. However, the major shortcoming of existing methods is their high computational cost incurred by determining whether each incoming key in the data stream is currently in the set of sampled keys and keeping track of sampled keys’ update aggregations. To solve this challenge, we develop a new method random projection and eviction (RPE) that uses a list of buckets to continuously sample distinct keys and their update aggregations. RPE processes each key-update pair with small and nearly constant time complexity $O(1)$ . Besides centralized data streams, we also develop a novel method DRPE to deal with distributed data streams consisting of key-update pairs observed at multiple distributed sites. We conduct extensive experiments on real-world datasets, and the results demonstrate that RPE and DRPE reduce the memory, computational, and message costs of state-of-the-art methods by several times.

An Evaluation of Predictive Accuracy of Common Machine Learning Algorithms on Data Stream

Streamed data are a potentially infinite sequence of incoming data at every high speed and may evolve over real-time. This causes several challenges in large scale high-speed data streams in real-time applications. Hence, this field has gained a lot of attention from researchers in recent years. In recent times, there are several areas of human endeavors where data generated are periodical or constantly growing. For instance, capital market, social networks, metrological data, E-commerce, online gaming and betting platforms. This research implemented the common machine learning algorithm on data streams and random data sets to describe the kind of data with more accurate predictions. Data samples were obtained from social media platforms such as Twitter and Instagram within the periods of 4th to 15th February 2019 with a total of 510,738 data samples collected. This is due to the sheer size of these platforms. Random forest, multi-layer perceptron and the k-nearest neighbor algorithms were used to model the data streams using the WEKA and RapidMiner data mining programs. The result of the research shows that Multilayer Perceptron produced the highest level of accuracy in both programs used when compared to the other algorithms used in the research. The findings of this research will be relevant to other researchers willing to develop machine learning tools to test the accuracy of data streams on social media platforms and related fields.

Evolving rule-based classifiers with genetic programming on GPUs for drifting data streams

Designing efficient algorithms for mining massive high-speed data streams has become one of the contemporary challenges for the machine learning community. Such models must display highest possible accuracy and ability to swiftly adapt to any kind of changes, while at the same time being characterized by low time and memory complexities. However, little attention has been paid to designing learning systems that will allow us to gain a better understanding of incoming data. There are few proposals on how to design interpretable classifiers for drifting data streams, yet most of them are characterized by a significant trade-off between accuracy and interpretability. In this paper, we show that it is possible to have all of these desirable properties in one model. We introduce ERulesD2S: evolving rule-based classifier for drifting data Streams. By using grammar-guided genetic programming, we are able to obtain accurate sets of rules per class that are able to adapt to changes in the stream without a need for an explicit drift detector. Additionally, we augment our learning model with new proposals for rule propagation and data stream sampling, in order to maintain a balance between learning and forgetting of concepts. To improve efficiency of mining massive and non-stationary data, we implement ERulesD2S parallelized on GPUs. A thorough experimental study on 30 datasets proves that ERulesD2S is able to efficiently adapt to any type of concept drift and outperform state-of-the-art rule-based classifiers, while using small number of rules. At the same time ERulesD2S is highly competitive to other single and ensemble learners in terms of accuracy and computational complexity, while offering fully interpretable classification rules. Additionally, we show that ERulesD2S can scale-up efficiently to high-dimensional data streams, while offering very fast update and classification times. Finally, we present the learning capabilities of ERulesD2S for sparsely labeled data streams.

Multi-label classification from high-speed data streams with adaptive model rules and random rules

Multi-label classification is a methodology that tries to solve classification problems where multiple classes are associated with each data example. Data streams pose new challenges to this methodology caused by the massive amounts of structured data production. In fact, most of the existent batch mode methods may not support this condition. Therefore, this paper proposes two multi-label classification methods based on rule and ensembles learning from continuous flow of data. These methods are derived from a multi-target regression algorithm. The main contribution of this work is the rule specialization for subsets of class labels, instead of the usual local (individual models for each output) or a global (one model for all outputs) methods. Prequential evaluation was conducted where global, local and subset operation modes were compared against other online classifiers found in the literature. Six real-world data sets were used. The evaluation demonstrated that the subset specialization presents competitive performance, when compared to local and global approaches and online classifiers found in the literature.