Streaming Algorithm Research Articles

A Framework for Meta-Learning in Dynamic Adaptive Streaming over HTTP

This work presents a framework with a taxonomy on meta-learning used in Dynamic Adaptive Streaming over HTTP (DASH). With the increasing complexity of network conditions and user preferences, there is a need for efficient adaptation mechanisms in DASH to provide optimal quality of experience (QoE) for users. Meta-learning, or learning to learn, has emerged as a promising approach to enhance adaptive streaming algorithms in DASH by leveraging prior knowledge and experiences. The proposed framework provides a systematic and structured approach for applying meta-learning techniques in the context of DASH. It encompasses essential components, including data collection and preprocessing, meta-model architecture, meta-training, meta-testing, fine-tuning, and continuous improvement. The taxonomy within the framework categorizes various aspects of meta-learning in DASH, such as meta-learning approaches, components, objectives, and applications

An improved monarch butterfly spectrum allocation algorithm for multi-source data stream in complex electromagnetic environment

In the era of the Internet of Everything, various wireless devices and sensors use spectrum, which is a precious and non-renewable resource, to communication. Due to the characteristics of massive, heterogeneous, and multi-source, the generated multi-source data stream brings difficulties to spectrum cognition. As a result, unreasonable spectrum allocation strategy leads to low utilization of spectrum resources. Optimizing spectrum allocation strategy can effectively improve spectrum utilization. Aiming at the problem of trapped local optimum solution in the genetic algorithm (GA) and particle swarm optimization algorithm (PSO), an improved monarch butterfly algorithm is proposed. Firstly, this paper employs the simulated annealing algorithm to select the migration rate, which increases the diversity of monarch butterfly population. Secondly, chaos mapping algorithm is utilized to improve the optimization ability and convergence speed. Finally, in the view of the problem that the monarch butterfly algorithm is easy to fall into the local optimal solution, there is no better way to escape from the local optimal solution. The Wolf pack updating operator is selected to improve the diversity of the population to generate new monarch butterflies. This method updates the population by generating new monarch butterfly individuals, so as to increasing the diversity of the population. The experimental results show that the improved monarch butterfly algorithm outperforms the other two algorithms in terms of convergence speed and system revenue.

Towards lowering computational power in IoT systems: Clustering algorithm for high-dimensional data stream using entropy window reduction

In a world of connectivity empowered by the advancement of the Internet of Things (IoT), an infinite number of data streams have emerged. Thus, data stream clustering is crucial for extracting hidden knowledge and data mining. Various data stream clustering methods have lately been introduced. Yet, the majority of such algorithms are affected by the curse of high dimensionality. Lately, a fully online buffer-based clustering algorithm for handling evolving data streams (BOCEDS) was developed. Similarly to other existing density-based clustering methods, BOCEDS is not capable of handling high-dimensional data and has high computational power and high memory utilization. This paper introduces an Entropy Window Reduction (EWR) algorithm, which is an improved version of the BOCEDS technique. EWR is a fully online clustering technique for handling high-dimensional data streams using feature ranking and sorting. This process is accomplished by calculating the entropy of specific features with respect to the time window. The findings of the experiments are compared to the outcomes of BOCEDS, CEDAS, and MuDi-Stream algorithms. The outcomes indicate that the EWR algorithm outperformed the baseline clustering algorithms. The results are demonstrated using the KDDCup’99 dataset in terms of quality and complexity evaluation on the average of F-Measures, Jaccard Index, Fowlkes–Mallows index, Purity, and Rand Index as well as the memory usage and computational power with 88%, 66%, 81%, 100%, and 66%, respectively. The results also show low memory usage and computing power in comparison with the baseline algorithms.

Real-time data processing for ultrafast X-ray computed tomography using modular CUDA based pipelines

In this article, a new version of the Real-time Image Stream Algorithms (RISA) data processing suite is introduced. It now features online detector data acquisition, high-throughput data dumping and enhanced real-time data processing capabilities. The achieved low-latency real-time data processing extends the application of ultrafast electron beam X-ray computed tomography (UFXCT) scanners to real-time scanner control and process control. We implemented high performance data packet reception based on data plane development kit (DPDK) and high-throughput data storing using both hierarchical data format version 5 (HDF5) as well as the adaptable input/output system version 2 (ADIOS2). Furthermore, we extended RISA's underlying pipelining framework to support the fork-join paradigm. This allows for more complex workflows as it is necessary, e.g. for online data processing. Also, the pipeline configuration is moved from compile-time to runtime, i.e. processing stages and their interconnections can now be configured using a configuration file. In several benchmarks, RISA is profiled regarding data acquisition performance, data storage throughput and overall processing latency. We found that using direct IO mode significantly improves data writing performance on the local data storage. We could further prove that RISA is now capable of concurrently receiving, processing and storing data from up to 768 detector channels (3072 MB/s) at 8000 fps on a single-GPU computer in real-time. Program summaryProgram Title: GLADOS/RISACPC Library link to program files:https://doi.org/10.17632/65sx747rvm.2Developer's repository link:https://codebase.helmholtz.cloud/risaLicensing provisions: Apache-2.0Programming language: C++Journal reference of previous version: Comput. Phys. Commun. 219 (2017) 353-360 [1]Does the new version supersede the previous version?: Yes.Reasons for the new version: Extended capabilities for real-time operation with latest UFXCT hardware.Summary of revisions: (i) Add forking and joining of processing pipeline branches(ii) Add runtime (re-)configuration of pipeline stages and connections(iii) Add UDP receiver stage to acquire detector data in real-time(iv) Add high-throughput data dumpingNature of problem: Ultrafast electron beam X-ray computed tomography scanners stream multiple Gigabytes of raw data per second via Ethernet to a control computer. Receiving the data with low latency, real-time image-based control would become possible. For this, data need to be captured from the network, stored on disk, reconstructed and post-processed concurrently. The current total data rate of up to 3072 MB/s requires high-throughput solutions for each of these tasks.Solution method: Using a pipeline scheme, RISA processes incoming raw data in distinct stages (sources, processors, sinks). These are implemented in GPU kernels and are executed concurrently to exploit data parallelism as well as task parallelism. To capture detector data, we implemented a UDP packet capturing stage based on DPDK [2] which acts as a source stage. By allowing the pipeline to fork up into multiple branches, we concurrently acquire, store and process the data. For storing these data, we use the HDF5 format [3]. We achieve the required data rates by writing in direct IO mode onto an SSD array in RAID 0 configuration.Additional comments including restrictions and unusual features: RISA provides a set of general-purpose processing stages which are suitable for generic image stream processing.

Exact PPS sampling with bounded sample size

Probability proportional to size (PPS) sampling schemes with a target sample size aim to produce a sample comprising a specified number n of items while ensuring that each item in the population appears in the sample with a probability proportional to its specified “weight” (also called its “size”). These two objectives, however, cannot always be achieved simultaneously. Existing PPS schemes prioritize control of the sample size, violating the PPS property if necessary. We provide a new PPS scheme, called EB-PPS, that allows a different trade-off: EB-PPS enforces the PPS property at all times while ensuring that the sample size never exceeds the target value n. The sample size is exactly equal to n if possible, and otherwise has maximal expected value and minimal variance. Thus we bound the sample size, thereby avoiding storage overflows and helping to control the time required for analytics over the sample, while allowing the user complete control over the sample contents. In the context of training classifiers at scale under imbalanced loss functions, we show that such control yields superior classifiers. The method is both simple to implement and efficient, being a one-pass streaming algorithm with an amortized processing time of O(1) per item, which makes it computationally preferable even in cases where both EB-PPS and prior algorithms can ensure the PPS property and a target sample size simultaneously.

Fast approximation of the top‐k items in data streams using FPGAs

Two methods are presented for finding the top-k items in data streams using Field Programmable Gate Arrays (FPGAs). These methods deploy two variants of a novel accelerator architecture capable of extracting an approximate list of the topmost frequently occurring items in a single pass over the input stream without the need for random access. The first variant of the accelerator implements the well-known Probabilistic sampling algorithm by mapping its main processing stages to a hardware architecture consisting of two custom systolic arrays. The proposed architecture retains all the properties of this algorithm, which works even if the stream size is unknown at run time. The architecture shows better scalability compared to other architectures that are based on other stream algorithms. In addition, experimental results on both synthetic and real datasets, when implementing the accelerator on an Intel Arria 10 GX 1150 FPGA device, showed very good accuracy and significant throughput gains compared to the existing software and hardware-accelerated solutions. The second variant of the accelerator is specifically tailored for applications requiring higher accuracy, provided that the size of the stream is known at run time. This variant takes advantage of the embedded memory resources in an FPGA to implement a sketch-based filter that precedes the main systolic array in the accelerator's pipeline. This filter enhances the accuracy of the accelerator by pre-processing the stream to remove much of the insignificant items, allowing the accelerator to process a significantly smaller filtered stream.

Online Tensor Low-Rank Representation for Streaming Data Clustering

Tensor low-rank representation (TLRR) has attracted increasing attention in recent years due to its effectiveness in capturing the intrinsic low-rank structure of tensor data. However, it is known that TLRR suffers from high computational complexity for large-scale data because it minimizes the tensor rank of the representation tensor whose size is proportional to the sample size. This paper develops a streaming algorithm—termed online low-rank tensor subspace clustering (OLRTSC)—for low-rank tensor data recovery and clustering based on the tensor singular value decomposition (t-SVD) algebraic framework. The key idea of OLRTSC is the non-convex reformulation of the objective function of TLRR by decomposing the tensor nuclear norm into an explicit tensor product of two low-rank tensors, which can be solved by leveraging the stochastic optimization technique. Compared to batch TLRR, OLRTSC is online by nature, can handle dynamic data, avoids computing t-SVD of the representation tensor, and reduces the storage cost significantly. This paper provides the theoretical guarantee that the sequence of the solutions produced by OLRTSC converges almost surely to a stationary point of the objective function. Moreover, an extension of OLRTSC is also proposed to handle the case when parts of the data are missing. Finally, experimental results on both synthetic and real data demonstrate the efficiency and robustness of the proposed approaches.

Quantum Versus Classical Online Streaming Algorithms with Logarithmic Size of Memory

Quantum Versus Classical Online Streaming Algorithms with Logarithmic Size of Memory

Performance Analysis of Hybrid Cryptographic Algorithms Rabbit Stream and Enhanced Dual RSA

Cryptography is a technique for encoding data by encrypting plaintext into an unreadable (meaningless) form. Cryptographic methods have good and bad performance depending on the type of algorithm we use. Therefore, the purpose of this study is to measure speed by combining the two algorithms used. The Rabbit Stream algorithm is a stream cipher algorithm whose system security depends on the generation of a key bit stream (keystream), which only guarantees 128-bit key security but has the advantage of being fast in the encryption and decryption process, while the Enhanced Dual RSA algorithm is an asymmetric algorithm to increase data protection from the Dual RSA algorithm by utilizing the Pells equation as a substitute for public key exponents. On the other hand, the algorithm in question requires a significant amount of time to encrypt messages with a large capacity when compared to the Rabbit Stream algorithm. Nonetheless, the study's findings suggest that using a hybrid method is comparatively faster for processing substantial amounts of data.

Submodular maximization over data streams with differential privacy noise

In the big data era, data often comes in the form of streams and fast data stream analysis has recently attracted intensive research interest. Submodular optimization naturally appears in many streaming data applications such as social network influence maximization with the property of diminishing returns. However, in a practical setting, streaming data frequently comes with noises that are small but significant enough to impact the optimality of submodular optimization solutions. Following the framework of differential privacy (DP), this paper considers a streaming model with DP noise that is small by construction. Within this noisy streaming model, the paper strives to address the general problem of submodular maximization with a cardinality constraint. The main theoretical result we obtained is a streaming algorithm that is one-pass and has an approximation guarantee of 1(2+(1+1k)2)(1+1k)−δ for any δ>0. Finally, we implement the algorithm and evaluate it against several baseline methods. Numerical results support the practical performance of our algorithm across several real datasets.

A One Pass Streaming Algorithm for Finding Euler Tours

Given an undirected graph G on n nodes and m edges in the form of a data stream we study the problem of finding an Euler tour in G. Our main result is the first one-pass streaming algorithm computing an Euler tour of G in the form of an edge successor function with only mathcal O(nlog (n)) RAM, which is optimal for this setting (e.g. Sun and Woodruff (2015)). Since the output size can be much larger, we use a write-only tape to gradually output the solution. The previously best-known result for finding Euler tours in data streams is implicitly given by the W-stream algorithm of Demetrescu et al. (2010) using mathcal O(m/n) passes under the same RAM limitation. Our approach is to partition the edges into edge-disjoint cycles and to merge the cycles until a single Euler tour is achieved. In the streaming environment such a merging is far from being obvious as the limited RAM allows the processing of only a constant number of cycles at once. This enforces merging of cycles that partially are no longer present in RAM. We solve this problem with a new edge swapping technique, for which storing two certain edges per node is sufficient to merge tours without having all tour edges in RAM. The mathematical key is to model tours and their merging in an algebraic way, where certain equivalence classes represent subtours. This quite general approach might be of interest also in other routing problems.

Streaming Algorithms for Constrained Submodular Maximization

It is of great importance to design streaming algorithms for submodular maximization, as many applications (e.g., crowdsourcing) have large volume of data satisfying the well-known ''diminishing returns'' property, which cannot be handled by offline algorithms requiring full access to the whole dataset. However, streaming submodular maximization has been less studied than the offline algorithms due to the hardness brought by more stringent requirements on memory consumption. In this paper, we consider the fundamental problem of Submodular Maximization under k -System and d -Knapsack constraints (SMSK), which has only been successfully addressed by offline algorithms in previous studies, and we propose the first streaming algorithm for it with provable performance bounds. Our approach adopts a novel algorithmic framework dubbed MultiplexGreedy , making it also perform well under a single k -system constraint. For the special case of SMSK with only d -knapsack constraints, we further propose a streaming algorithm with better performance ratios than the state-of-the-art algorithms. As the SMSK problem generalizes most of the major problems studied in submodular maximization, our algorithms have wide applications in big data processing.

Adversarially Robust Streaming Algorithms via Differential Privacy

A streaming algorithm is said to be adversarially robust if its accuracy guarantees are maintained even when the data stream is chosen maliciously, by an adaptive adversary . We establish a connection between adversarial robustness of streaming algorithms and the notion of differential privacy . This connection allows us to design new adversarially robust streaming algorithms that outperform the current state-of-the-art constructions for many interesting regimes of parameters.

DWDP-Stream: A Dynamic Weight and Density Peaks Clustering Algorithm for Data Stream

Identifying clusters of arbitrary shapes and constantly processing the newly arrived data points are two critical challenges in the study of clustering. This paper proposes a dynamic weight and density peaks clustering algorithm to simultaneously solve these two key issues. An online–offline framework is used, creating and maintaining micro-clusters in the online phase, and treating the micro-clusters as pseudo-points to form the final cluster in the offline phase. In the online phase, when a new data point is merged into the corresponding micro-cluster, a dynamic weight method is proposed to update the weight of the micro-cluster according to the distance between the point and the center of the micro-cluster, so as to more accurately describe the information of the micro-cluster. In the offline phase, the density peak clustering algorithm is improved, natural neighbors are introduced to adaptively obtain the local density of the data point, and the allocation process is improved to reduce the probability of allocation errors. The algorithm is evaluated on different synthetic and real-world datasets using different quality metrics. The experimental results show that the proposed algorithm improves the clustering quality in both static and streaming environments.

Improved Salsa20 Stream Cipher Diffusion Based on Random Chaotic Maps

To enhance stream ciphers, numerous studies have concentrated on the randomness, unpredictable nature, and complexity of keystream. Numerous stream algorithms have been put forth. Most of them require a significant amount of computational power. Salsa20 is a high-performance stream encryption solution that works on computers with fewer resources and uses a secure method that is faster than AES. They suggest Salsa20 for encryption in common cryptographic applications. Users who value speed over certainty should utilize the Salsa20 family of reduced-round ciphers, such as the (8,12) round cipher. It uses a 256-bit key and a hash algorithm. A successful fusion makes use of both the Salsa20 algorithm's and the random maps' advantages to improve the Salsa20 algorithm's shortcomings by increasing its unpredictability. Particularly now that Salsa20/7 has been hacked and Salsa20/12 is no longer as secure as it previously was. As a result, Salsa20 needs to achieve a high level of diffusion to withstand known attacks. Right now, salsa20 and its shortened versions rank among the fastest ciphers. This study presents a novel lightweight approach to construct a strong keystream that is sufficiently random to avoid being predicted by adversaries, achieve good diffusion, and withstand known assaults. A NIST test found that the performance of the (Salsa20-chaotic maps) approach in terms of data integrity and secrecy is nearly 0.3131 higher than that of the Salsa20 .

Streaming algorithms for multitasking scheduling with shared processing

In this paper, we design the first streaming algorithms for the problem of multitasking scheduling on parallel machines with shared processing. In one pass, our streaming approximation schemes can provide an approximate value of the optimal makespan. If the jobs can be read in two passes, the algorithm can find the schedule with the approximate value. This work not only provides an algorithmic big data solution for the studied problem, but also gives an insight into the design of streaming algorithms for other problems in the area of scheduling.

Streaming algorithms for monotone non-submodular function maximization under a knapsack constraint on the integer lattice

The study of non-submodular maximization on the integer lattice is an important extension of submodular optimization. In this paper, streaming algorithms for maximizing non-negative monotone non-submodular functions with knapsack constraint on integer lattice are considered. We first design a two-pass StreamingKnapsack algorithm combining with BinarySearch as a subroutine for this problem. By introducing the DR ratio γ and the weak DR ratio γw of the non-submodular objective function, we obtain that the approximation ratio is min⁡{γ2(1−ε)/2γ+1,1−1/γw2γ−ε}, the total memory complexity is O(Klog⁡K/ε), and the total query complexity for each element is O(log⁡Klog⁡(K/ε2)/ε). Then, we design a one-pass streaming algorithm by dynamically updating the maximal function value among unit vectors along with the currently arriving element. Finally, in order to decrease the memory complexity, we design an improved StreamingKnapsack algorithm and reduce the memory complexity to O(K/ε2).

Automatic Recognition for IoT Supervision Images Based on Modal Decomposition

The automatic recognition for Internet of Things (IoT) supervision images is a prerequisite for the detection of abnormalities in monitoring images. This technology is a developmental trend in video surveillance. Various video image detection and recognition methods have certain weaknesses, such as poor generalization ability and poor anti-interference ability. In response, this paper conducts a study on automatic recognition for IoT supervision images based on modal decomposition. The paper presents an overall framework of the IoT supervision system. For the problems of poor real-time performance and few samples that commonly exist in video stream target recognition, the paper proposes a dynamic modal decomposition-based feature extraction algorithm for IoT supervision video stream to build a suitable platform for IoT supervision image foreground segmentation. The paper selects a dictionary with rich elements and exchanges a high computation time for minimizing the reconstruction error generated by applying the dynamic modal decomposition method. Experimental results validate the effectiveness of the proposed algorithm.

Persistent Summaries

A persistent data structure , also known as a multiversion data structure in the database literature, is a data structure that preserves all its previous versions as it is updated over time. Every update (inserting, deleting, or changing a data record) to the data structure creates a new version, while all the versions are kept in the data structure so that any previous version can still be queried. Persistent data structures aim at recording all versions accurately, which results in a space requirement that is at least linear to the number of updates. In many of today’s big data applications, in particular, for high-speed streaming data, the volume and velocity of the data are so high that we cannot afford to store everything. Therefore, streaming algorithms have received a lot of attention in the research community, which uses only sublinear space by sacrificing slightly on accuracy. All streaming algorithms work by maintaining a small data structure in memory, which is usually called a sketch , summary , or synopsis . The summary is updated upon the arrival of every element in the stream, thus it is ephemeral , meaning that it can only answer queries about the current status of the stream. In this article, we aim at designing persistent summaries, thereby giving streaming algorithms the ability to answer queries about the stream at any prior time.

Adaptive Streaming Algorithms and Network Protocols

Adaptive Streaming Algorithms and Network Protocols