High-speed Data Streams Research Articles

An Ultrawide Bandwidth Digital Backend System Based on PFB Algorithm for QTT

For the planning of the QiTai radio Telescope ultrawide bandwidth low-frequency pulsar receiving system, we designed and implemented a Field-Programmable Gate Array (FPGA)+CPU/GPU hybrid architecture digital backend system based on the Polyphase FilterBank (PFB) channeling algorithm. We used the FPGA signal acquisition and processing platform to implement ultrawide bandwidth signal sampling and designed the PFB algorithm to realize the digital channelization of multiple analog bandwidth signals. We also developed data encapsulation and multichannel parallel distribution firmware algorithms and realized the real-time parallel transmission of high-speed astronomical data streams based on the VLBI Data Interchange Format. We developed the Ultra Wide bandwidth Low-frequency pulsar data process PIPEline, which realized the real-time processing and data packaging of massive pulsar signals. Using the L-band (964–1732 MHz bandwidth) receiving system of the Nanshan 26 m radio telescope, we conducted a systematic test on the designed digital backend system and obtained high-quality observation data. By using the professional pulsar data processing software DSPSR to process the observation data, we obtained high signal-to-noise ratio pulse profiles.

Location-aware job scheduling for IoT systems using cloud and fog

The rapid growth of Internet of Things (IoT) applications generates vast volumes of high-speed data streams from numerous devices. Cloud computing solutions often handle and manage this data; however, for certain applications, the latency introduced by transmitting data from edge devices to the cloud may be unacceptable. This issue is exacerbated by the bandwidth constraints of public networks, which become significant barriers in large-scale IoT implementations. Consequently, effective resource management, service management, data storage, and power management require more robust infrastructure and complex protocols. An "intelligent gateway" based on fog computing can enhance the efficient utilization of cloud and network resources. Resource planning and management in a fog-cloud environment significantly impact system performance, particularly latency. This problem is known to be NP-hard. This paper addresses the challenge of lifetime optimization for scheduling data-intensive tasks in fog and cloud-based IoT systems. Initially, we propose an Integer Linear Programming (ILP) optimization model to frame the problem. Subsequently, we introduce a heuristic method named Data-Locality Aware Job Scheduling in Fog-Cloud (DLSFC), which is derived from the proposed formulation. The effectiveness of DLSFC is evaluated across various system characteristics, with results demonstrating that DLSFC achieves solutions within 85 % of the ideal outcome provided by the LP solver.

Challenges and strategies of software engineering in big data environment

Software engineering in a big data environment involves developing and maintaining large-scale, scalable, data-intensive applications in a rapidly evolving technology domain. As data volumes grow exponentially, so do the requirements for software systems to process and analyze this data. These systems need to be able to efficiently store, retrieve, process and analyze data collected from sources such as social media, IoT devices, enterprises and scientific research. Software engineers must address challenges including data heterogeneity and quality, scalability of data storage and computation, and real-time processing of high-speed data streams. With the evolution of technology, new architecture and database technologies have emerged, such as NoSQL and distributed computing frameworks. These technologies have challenged traditional methods of software engineering practice and also brought new design and implementation strategies.

Enhancing Accuracy for Super Spreader Identification in High-Speed Data Streams

This paper addresses the challenge of identifying super spreaders within large, high-speed data streams. In these streams, data is segmented into flows, with each flow's spread defined as the number of distinct items it contains. A super spreader is characterized as a flow with a notably large spread. Current compact solutions, known as sketches, are designed to fit within the constrained memory of online devices. However, they struggle to accurately track the spread of all flows due to the substantial memory requirement for monitoring a single flow --- a problem exacerbated when numerous flows are involved. To overcome these limitations, this study proposes a more precise sketch-based approach. Our solution introduces an innovative non-duplicate sampler that effectively eliminates duplicates, allowing for accurate post-sampling count of flow spread using only counters. Additionally, it incorporates an exponential-weakening decay technique to highlight large flows, markedly enhancing the accuracy of super spreader identification. We offer a comprehensive theoretical analysis of our method. Trace-driven experiments validate that our approach statistically surpasses existing state-of-the-art solutions in identifying super spreaders. It also demonstrates the lowest time required to restore super spreaders and significantly reduces bandwidth consumption by an order of magnitude when offline restoration is conducted remotely.

Efficient Online Stream Clustering Based on Fast Peeling of Boundary Micro-Cluster.

A growing number of applications generate streaming data, making data stream mining a popular research topic. Classification-based streaming algorithms require pre-training on labeled data. Manually labeling a large number of samples in the data stream is impractical and cost-prohibitive. Stream clustering algorithms rely on unsupervised learning. They have been widely studied for their ability to effectively analyze high-speed data streams without prior knowledge. Stream clustering plays a key role in data stream mining. Currently, most data stream clustering algorithms adopt the online-offline framework. In the online stage, micro-clusters are maintained, and in the offline stage, they are clustered using an algorithm similar to density-based spatial clustering of applications with noise (DBSCAN). When data streams have clusters with varying densities and ambiguous boundaries, traditional data stream clustering algorithms may be less effective. To overcome the above limitations, this article proposes a fully online stream clustering algorithm called fast boundary peeling stream clustering (FBPStream). First, FBPStream defines a decay-based kernel density estimation (KDE). It can discover clusters with varying densities and identify the evolving trend of streams well. Then, FBPStream implements an efficient boundary micro-cluster peeling technique to identify the potential core micro-clusters. Finally, FBPStream employs a parallel clustering strategy to effectively cluster core and boundary micro-clusters. The proposed algorithm is compared with ten popular algorithms on 15 data streams. Experimental results show that FBPStream is competitive with the other ten popular algorithms.

A Dynamic Drilling Sampling Method and Evaluation Model for Big Streaming Data

The big data sampling method for real-time and high-speed streaming data is prone to lose the value and information of a large amount of discrete data, and it is not easy to make an efficient and accurate evaluation of the value characteristics of streaming data. The SDSLA sampling method based on mineral drilling exploration can evaluate the valuable information of streaming data containing many discrete data in real-time, but when the range of discrete data is irregular, it has low sampling accuracy for discrete data. Based on the SDSLA algorithm, we propose a dynamic drilling sampling method SDDS, which takes well as the analysis unit, dynamically changes the size and position of the well, and accurately locates the position and range of discrete data. A new model SDVEM is further proposed for data valuation, which evaluates the sample set from discrete, centralized, and overall dimensions. Experiments show that compared with the SDSLA algorithm, the sample sampled by the SDDS algorithm has higher evaluation accuracy, and the probability distribution of the sample is closer to the original streaming data, with the AOCV indicator being nearly 10% higher. In addition, the SDDS algorithm can achieve over 90% accuracy, recall, and F1 score for training and testing neural networks with small sampling rates, all of which are higher than the SDSLA algorithm. In summary, the SDDS algorithm not only accurately evaluates the value characteristics of streaming data but also facilitates the training of neural network models, which has important research significance in big data estimation.

A generic sketch for estimating super-spreaders and per-flow cardinality distribution in high-speed data streams

For a high-speed network, it is an important task to process the IP packet stream using limited memory and measure its statistical metrics of interest. While many algorithms have been proposed to estimate the cardinality of a single data stream (i.e., the number of distinct elements), it remains a great challenge when a stream contains numerous sub-streams, called flows. In this paper, we focus on a problem of designing a generic data structure to measure multiple types of per-flow statistics in a high-speed stream, including per-flow cardinality, top-K super-spreading flows with the greatest cardinalities, per-flow cardinality moments and per-flow cardinality distribution. Previous solutions for generic measurement mainly focus on the frequency-related statistics measurement, while this paper makes a step forward to support deduplication, i.e., cardinality-related measuring. To address this new problem, we propose a generic sketch named M2D. The challenge is that the per-flow cardinality distribution is often highly skewed with a small proportion of super-spreaders. To tame the skewness, we adopt the adjustable progressive sampling technique, which samples subsets of flows by an exponentially decreasing probability according to their cardinalities. Based on the sampled super-spreaders, we estimate the moments of per-flow cardinalities with different orders. We finally apply the method of moments to reconstruct the per-flow cardinality distribution with no priori knowledge about its formula. Our experiments show M2D’s high memory efficiency (average savings of 38%) and satisfactory distribution estimation accuracy (2% to 98% improvement) than other algorithms.

Design and Implementation of Airborne Hyperspectral Camera Control and Data Storage System

With the great progress of hyperspectral imaging technology, airborne hyperspectral cameras have been widely used in the field of earth detection and remote sensing. However, with the increasing resolution of airborne hyperspectral cameras, the amount of data acquired is also growing explosive, The speed and accuracy in the transmission process are easily affected, which poses challenges to the real-time control and data acquisition and storage of airborne hyperspectral cameras. In this paper, a general airborne hyperspectral image acquisition and storage system is designed, which converts the hyperspectral data from the Cameralink interface into a high-speed serial PCIe data stream and realizes high-speed storage of image data. The image width is 2048 pixels, the number of spectral segments is 120, the pixel size is 12bit, and the maximum frame rate is 300 fps. The maximum transmission rate is 2048×120×12bit×300fps, and the data rate is 844Mbps. Based on this, we perform thermal analysis and vibration analysis of the system platform. Experimental results show that the system can achieve stable data transmission and storage, and the data is intact and reliable.

An Encryption Transmission System for High-speed Private Data Streams in Online Education in the Specialty of “Traffic Engineering”

An Encryption Transmission System for High-speed Private Data Streams in Online Education in the Specialty of “Traffic Engineering”

CIP- Efficient Method for Mining Frequent Itemsets from Data streams using Landmark Window Model

Continuous stream transactions like network monitoring, retail market data analysis and stock market prediction need the “frequent patterns” to be detected recurrently. Literature suggests that several pattern mining solutions are being developed over years. Still lot of challenges need to be addressed due to rapidness in generation of continuous, unbounded and ordered data real time. Hence extraction of frequent patterns from recent data will improve the analysis of stream data. In this article, a new landmark window model CIP (candidate indexing and pruning) is considered for mining the datasets. CIP allows us to mine over entire history of data streams, which improves the accuracy. This article also proposes the candidate indexed sub (CIS)-tree scheme to extract the essential information from each incoming transactions of data streams. Our proposal is compared with the existing “improved data stream mining” (ISDM) for maximal frequent itemsets algorithm. Extensive experimental analyses prove the superiority of the proposed CIP over the popular ISDM in terms of accuracy and time complexity for high-speed data stream. This article also covers up a case study where the proposed approach is applied for an application called “web prefetching”.

Virtual self-adaptive bitmap for online cardinality estimation

Cardinality estimation is the task of obtaining the number of distinct items in a data stream, which plays an important role in many application domains. However, when dealing with high-speed data streams, it remains a significant challenge to estimate cardinality considering record/query overhead and memory efficiency. This paper proposes a virtual self-adaptive bitmap estimator to support online cardinality estimation, which reduces the record overhead to one hash per item for the first time. By logically adding virtual bits, our estimator automatically adapts its sampling probability to different stream sizes. We evaluate the virtual self-adaptive bitmap theoretically and experimentally. The experimental results show that our estimator significantly improves over the existing work in terms of record throughput, query throughput and estimation accuracy.

Persistent Summaries

A persistent data structure , also known as a multiversion data structure in the database literature, is a data structure that preserves all its previous versions as it is updated over time. Every update (inserting, deleting, or changing a data record) to the data structure creates a new version, while all the versions are kept in the data structure so that any previous version can still be queried. Persistent data structures aim at recording all versions accurately, which results in a space requirement that is at least linear to the number of updates. In many of today’s big data applications, in particular, for high-speed streaming data, the volume and velocity of the data are so high that we cannot afford to store everything. Therefore, streaming algorithms have received a lot of attention in the research community, which uses only sublinear space by sacrificing slightly on accuracy. All streaming algorithms work by maintaining a small data structure in memory, which is usually called a sketch , summary , or synopsis . The summary is updated upon the arrival of every element in the stream, thus it is ephemeral , meaning that it can only answer queries about the current status of the stream. In this article, we aim at designing persistent summaries, thereby giving streaming algorithms the ability to answer queries about the stream at any prior time.

EVtracker: An Event-Driven Spatiotemporal Method for Dynamic Object Tracking.

An event camera is a novel bio-inspired sensor that effectively compensates for the shortcomings of current frame cameras, which include high latency, low dynamic range, motion blur, etc. Rather than capturing images at a fixed frame rate, an event camera produces an asynchronous signal by measuring the brightness change of each pixel. Consequently, an appropriate algorithm framework that can handle the unique data types of event-based vision is required. In this paper, we propose a dynamic object tracking framework using an event camera to achieve long-term stable tracking of event objects. One of the key novel features of our approach is to adopt an adaptive strategy that adjusts the spatiotemporal domain of event data. To achieve this, we reconstruct event images from high-speed asynchronous streaming data via online learning. Additionally, we apply the Siamese network to extract features from event data. In contrast to earlier models that only extract hand-crafted features, our method provides powerful feature description and a more flexible reconstruction strategy for event data. We assess our algorithm in three challenging scenarios: 6-DoF (six degrees of freedom), translation, and rotation. Unlike fixed cameras in traditional object tracking tasks, all three tracking scenarios involve the simultaneous violent rotation and shaking of both the camera and objects. Results from extensive experiments suggest that our proposed approach achieves superior accuracy and robustness compared to other state-of-the-art methods. Without reducing time efficiency, our novel method exhibits a increase in accuracy over other recent models. Furthermore, results indicate that event cameras are capable of robust object tracking, which is a task that conventional cameras cannot adequately perform, especially for super-fast motion tracking and challenging lighting situations.

A survey on dendritic neuron model: Mechanisms, algorithms and practical applications

Research on dendrites has been conducted for decades, providing valuable information for the development of dendritic computation. Creating an ideal neuron model is crucial for computer science and may also provide robust guidance for understanding our brain’s underlying mechanisms and principles. This paper aims to review the related studies regarding a newly emerging, non-spiking and biologically inspired model, the dendritic neuron model (DNM). By mimicking the biological phenomena of neurons in vivo, the DNM incorporates a neural pruning scheme to eliminate superfluous synapses and dendrites, simplifying its architecture and forming unique neuron morphology for a specific task. Furthermore, the simplified structure can be transformed into logic circuits consisting of the comparators and logic AND, OR and NOT gates, without sacrificing model accuracy. The rapidity of binary operations in hardware implementation gives the DNM a distinct advantage to handle high-speed data streams. The advent of the big data era has led to an exponential explosion in the amount and variety of available information. The appealing properties of the DNM lead us to believe that it is worthy of more attention and that it might be a promising data mining technique. This article presents an in-depth analysis of the pruning and transformation mechanisms and a comprehensive review of the learning algorithms and real-world applications of the DNM. It also presents an empirical comparison of the optimization performance of different algorithms. Finally, we outline some critical issues and future works of the DNM. All the source code of DNM is available at http://www.dnm.net.cn/.

LPsec: a fast and secure cryptographic system for optical connections

High capacity and low latency of optical connections are ideal for supporting current and future communication services, including 5G and beyond. Although some of those services are already secured at the packet layer using standard stream ciphers, like the Advanced Encryption Standard and ChaCha, secure transmission at the optical layer is still not implemented. To secure the optical layer, cryptographic methods need to be fast enough to support high-speed optical transmission and cannot introduce significant delay. Moreover, methods for key exchange, key generation, and key expansion are required, which can be implemented on standard coherent transponders. In this paper, we propose Light Path SECurity (LPsec), a secure cryptographic solution for optical connections that involves fast data encryption using stream ciphers and key exchange using Diffie–Hellman protocol through the optical channel. To support encryption of high-speed data streams, a fast, general-purpose pseudorandom number generator is used. Moreover, to make the scheme more secure against exhaustive search attacks, an additional substitution cipher is proposed. In contrast to the limited encryption speeds that standard stream ciphers can support, LPsec can support high-speed rates. Numerical simulation for 16 quadrature amplitude modulation (QAM), 32-QAM, and 64-QAM show that LPsec provides a sufficient security level while introducing only negligible delay.

Using IoT devices for fast data processing

IoT systems make a significant contribution to the growth of global traffic. There is a trend to reduce the volume of transmitted and stored data, for which different approaches are used. Most approaches use traditional data processing in the Cloud and at the gate level. The end devices are not involved to the data processing. The article describes the correlation method of data processing on the end device. The results of a research of the maximum performance when implemented on an FPGA, with different number of taps of the matched filter and different bit width of the input data stream are presented. This method allows to work with high-speed data streams and communication channels typical for IoT systems. The boundary conditions of operability for the correlation method are given. On the basis of the presented model of the IoT system, data processing delays for the correlation method and the method using data processing on the gate application of IoT devices for fast data processing are analyzed.

On the Improvement of the Isolation Forest Algorithm for Outlier Detection with Streaming Data

In recent years, detecting anomalies in real-world computer networks has become a more and more challenging task due to the steady increase of high-volume, high-speed and high-dimensional streaming data, for which ground truth information is not available. Efficient detection schemes applied on networked embedded devices need to be fast and memory-constrained, and must be capable of dealing with concept drifts when they occur. Different approaches for unsupervised online outlier detection have been designed to deal with these circumstances in order to reliably detect malicious activity. In this paper, we introduce a novel framework called PCB-iForest, which generalized, is able to incorporate any ensemble-based online OD method to function on streaming data. Carefully engineered requirements are compared to the most popular state-of-the-art online methods with an in-depth focus on variants based on the widely accepted isolation forest algorithm, thereby highlighting the lack of a flexible and efficient solution which is satisfied by PCB-iForest. Therefore, we integrate two variants into PCB-iForest—an isolation forest improvement called extended isolation forest and a classic isolation forest variant equipped with the functionality to score features according to their contributions to a sample’s anomalousness. Extensive experiments were performed on 23 different multi-disciplinary and security-related real-world datasets in order to comprehensively evaluate the performance of our implementation compared with off-the-shelf methods. The discussion of results, including AUC, F1 score and averaged execution time metric, shows that PCB-iForest clearly outperformed the state-of-the-art competitors in 61% of cases and even achieved more promising results in terms of the tradeoff between classification and computational costs.

Towards real-time in-situ monitoring of hot-spot defects in L-PBF: a new classification-based method for fast video-imaging data analysis

The increasing interest towards additive manufacturing (AM) is pushing the industry to provide new solutions to improve process stability. Monitoring is a key tool for this purpose but the typical AM fast process dynamics and the high data flow required to accurately describe the process are pushing the limits of standard statistical process monitoring (SPM) techniques. The adoption of novel smart data extraction and analysis methods are fundamental to monitor the process with the required accuracy while keeping the computational effort to a reasonable level for real-time application. In this work, a new framework for the detection of defects in metal additive manufacturing processes via in-situ high-speed cameras is presented: a new data extraction method is developed to efficiently extract only the relevant information from the regions of interest identified in the high-speed imaging data stream and to reduce the dimensionality of the anomaly detection task performed by three competitor machine learning classification methods. The defect detection performance and computational speed of this approach is carefully evaluated through computer simulations and experimental studies, and directly compared with the performance and computational speed of other existing methods applied on the same reference dataset. The results show that the proposed method is capable of quickly detecting the occurrence of defects while keeping the high computational speed that would be required to implement this new process monitoring approach for real-time defect detection.

Efficient unsupervised drift detector for fast and high-dimensional data streams

Stream mining considers the online arrival of examples at high speed and the possibility of changes in its descriptive features or class definitions compared with past knowledge (i.e., concept drifts). The fast detection of drifts is essential to keep the predictive model updated and stable in changing environments. For many applications, such as those related to smart sensors, the high number of features is an additional challenge in terms of memory and time for stream processing. This paper presents an unsupervised and model-independent concept drift detector suitable for high-speed and high-dimensional data streams. We propose a straightforward two-dimensional data representation that allows the faster processing of datasets with a large number of examples and dimensions. We developed an adaptive drift detector on this visual representation that is efficient for fast streams with thousands of features and is accurate as existing costly methods that perform various statistical tests considering each feature individually. Our method achieves better performance measured by execution time and accuracy in classification problems for different types of drifts. The experimental evaluation considering synthetic and real data demonstrates the method’s versatility in several domains, including entomology, medicine, and transportation systems.

LTC: A Fast Algorithm to Accurately Find Significant Items in Data Streams

Finding top- <inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> frequent items has been a hot issue in databases. Finding top- <inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> persistent items is a new issue, and has attracted increasing attention in recent years. In practice, users often want to know which items are significant, <i>i.e.</i> , not only frequent but also persistent. No prior art can address both of the above two issues at the same time. Also, for high-speed data streams, prior art cannot achieve high accuracy when the memory is tight. In this paper, we define a new issue, named finding significant items, and propose a novel algorithm namely LTC to address this issue. It includes two key techniques, Long-tail Restoring and CLOCK, as well as three optimizations. In addition, LTC is extended to support finding significant items with thresholds. We theoretically derive the correct rate and error bound, and conduct extensive experiments on three real datasets to test the performance of LTC. Our experimental results show that LTC can achieve <inline-formula><tex-math notation="LaTeX">$10^5$</tex-math></inline-formula> times higher accuracy in terms of average relative error than other related algorithms. Lastly, LTC is applied to a DDoS detection task and it shows that finding significant items is more powerful than finding frequent items.