Infinite Data Streams Research Articles

Protecting Infinite Data Streams from Wearable Devices with Local Differential Privacy Techniques

The real-time data collected by wearable devices enables personalized health management and supports public health monitoring. However, sharing these data with third-party organizations introduces significant privacy risks. As a result, protecting and securely sharing wearable device data has become a critical concern. This paper proposes a local differential privacy-preserving algorithm designed for continuous data streams generated by wearable devices. Initially, the data stream is sampled at key points to avoid prematurely exhausting the privacy budget. Then, an adaptive allocation of the privacy budget at these points enhances privacy protection for sensitive data. Additionally, the optimized square wave (SW) mechanism introduces perturbations to the sampled points. Afterward, the Kalman filter algorithm is applied to maintain data flow patterns and reduce prediction errors. Experimental validation using two real datasets demonstrates that, under comparable conditions, this approach provides higher data availability than existing privacy protection methods for continuous data streams.

EdgeSyn: Privacy-preserving Data Publishing on Edge Network over Infinite Multimedia Data Stream

To privately publish sensitive multimedia data in an edge network with fog devices, one of the best privacy-preserving solutions is to use differential privacy (DP) mechanisms. However, existing DP data publication mechanisms for the infinite data stream of edge networks mainly focus on publishing data with specific types of data or a set of predetermined queries. This approach is not suitable for multimedia data with numerous features that require a more flexible data publishing mechanism. In this paper, we propose EdgeSyn, a novel mechanism for accurately publishing multimedia data over infinite data streams in an edge network. It allocates privacy budgets with a sliding window, adopting data synthesis mechanisms to support dynamic publishing without loss of accuracy. In more detail, EdgeSyn addresses the limitations associated with data types in prior data stream publishing approaches and introduces a privacy budget management strategy that optimally allocates budgets for the implementation of data synthesis mechanisms over an infinite data stream. The experimental results show that EdgeSyn performs well under different privacy budgets and various lengths of active windows.

NLOCL: Noise-Labeled Online Continual Learning

Continual learning (CL) from infinite data streams has become a challenge for neural network models in real-world scenarios. Catastrophic forgetting of previous knowledge occurs in this learning setting, and existing supervised CL methods rely excessively on accurately labeled samples. However, the real-world data labels are usually misled by noise, which influences the CL agents and aggravates forgetting. To address this problem, we propose a method named noise-labeled online continual learning (NLOCL), which implements the online CL model with noise-labeled data streams. NLOCL uses an empirical replay strategy to retain crucial examples, separates data streams by small-loss criteria, and includes semi-supervised fine-tuning for labeled and unlabeled samples. Besides, NLOCL combines small loss with class diversity measures and eliminates online memory partitioning. Furthermore, we optimized the experience replay stage to enhance the model performance by retaining significant clean-labeled examples and carefully selecting suitable samples. In the experiment, we designed noise-labeled data streams by injecting noisy labels into multiple datasets and partitioning tasks to simulate infinite data streams realistically. The experimental results demonstrate the superior performance and robust learning capabilities of our proposed method.

Median Filters on FPGAs for Infinite Data and Large, Rectangular Windows

Efficient architectures and implementations of median filters have been well investigated in the past. In this article, we focus on median filters for very big scientific applications with very large windows and an infinite stream of data, inspired by big data needs in the Square Kilometre Array (SKA) pulsar search engine, but transferable to other big data domains. We propose a novel approach for very large rectangular windows on an FPGA accelerator device able to support the processing of infinite streams of data. OpenCL is used for rapid parameter sweeping and design space exploration based on a pipelined model of the system. Evaluation on a host/accelerator system with an Arria 10 device surpassed 64 million values processed per second considered for the SKA real time requirement, achieving 83.4M value/s while reading from/writing to disk. These results are compared with a state-of-the-art software implementation only achieving 41M value/s for over twice the total system energy cost.

PCFed: Privacy-Enhanced and Communication-Efficient Federated Learning for Industrial IoTs

Federated learning (FL) is capable of analyzing tremendous data from smart edge devices in Industrial Internet of Things (IIoTs), empowering numerous industrial applications. However, the increasing privacy concerns and deployment costs of IIoT environment have been posing new challenges for FL. This article proposes PCFed, a novel privacy-enhanced and communication-efficient FL framework to provide higher model accuracy with rigorous privacy guarantees and great communication efficiency. In particular, we develop a sampling-based intermittent communication strategy via a PID (proportional, integral, and derivative) controller on the cloud server to adaptively reduce the communication frequency. In addition, we design a budget allocation mechanism to balance the tradeoff between model accuracy and privacy loss. Then, we develop PCFed+, an enhanced variant for PCFed, with further consideration of infinite data streams on edge servers. Extensive experiments demonstrate that both PCFed and PCFed+ can significantly outperform existing schemes, in terms of communication efficiency, privacy protection, and model accuracy.

Reducing Catastrophic Forgetting in Self Organizing Maps with Internally-Induced Generative Replay (Student Abstract)

A lifelong learning agent is able to continually learn from potentially infinite streams of pattern sensory data. One major historic difficulty in building agents that adapt in this way is that neural systems struggle to retain previously-acquired knowledge when learning from new samples. This problem is known as catastrophic forgetting (interference) and remains an unsolved problem in the domain of machine learning to this day. While forgetting in the context of feedforward networks has been examined extensively over the decades, far less has been done in the context of alternative architectures such as the venerable self-organizing map (SOM), an unsupervised neural model that is often used in tasks such as clustering and dimensionality reduction. Although the competition among its internal neurons might carry the potential to improve memory retention, we observe that a fixed-sized SOM trained on task incremental data, i.e., it receives data points related to specific classes at certain temporal increments, it experiences severe interference. In this study, we propose the c-SOM, a model that is capable of reducing its own forgetting when processing information.

Hierarchical Correlations Replay for Continual Learning

Continual Learning (CL) aims at incrementally learning new knowledge from an infinite stream of data while preserving old knowledge. A perfect CL method is expected to retain the old knowledge greedily to alleviate the catastrophic interference caused by the new knowledge. However, most existing methods only focus on exploring the knowledge carried by the instance itself but neglect the correlations of inter-instance and inter-class, which are also valuable for preserving knowledge. To this end, we propose a novel method dubbed Hierarchical Correlations Replay (HCR) consisting of an Instance-level Correlation Replay (ICR) module and a Class-level Correlation Replay (CCR) module, which retain both the instance-level and the class-level correlations to consolidate old knowledge. Specifically, the ICR module employs a correlation matrix to represent the instance-level correlation, and a random triplet probability is utilized to construct the class-level correlation in the CCR module. Extensive experiments on five benchmark image datasets show that our HCR is competitive with or superior to state-of-the-art methods under diverse continual learning settings.

Coordinating Experience Replay: A Harmonious Experience Retention approach for Continual Learning

Continual Learning (CL) aims at learning new knowledge gradually from an infinite data streams while retaining the old knowledge. A key problem in CL is the stability–plasticity dilemma, i.e., a high stability weakens the model from learning new data, while a high plasticity results in forgetting old classes. To alleviate this dilemma, some work has been reported to replay a finite subset of stored examples when training on new tasks, while the effectiveness heavily depends on the rehearsal mode and the validity of these exemplars. To this end, this paper proposes a Coordinating Experience Replay approach consisting of a Harmonious Experience Retention approach and a Vital Exemplar Reserved Sampling strategy, which constrains the rehearsal process and develops two criteria to select examples for better replaying the beneficial experience. Specifically, for the Harmonious Experience Retention approach, an indirect rehearsal process is supplemented on Dark Experience Replay by Imitative Experience Replay (IER), which imitates the response process via constructing empirical constraints on the saved logits and ground-truth labels directly. For the Vital Exemplar Reserved Sampling strategy, we apply the classification loss value and the class information as the criteria to select examples, which balance the criteria of class integrity and example vitality. Extensive experiments on seven benchmark datasets show the superiority of the proposed approach under diverse continual learning settings.

Online sequential ensembling of predictive fuzzy systems

Evolving fuzzy systems (EFS) have enjoyed a wide attraction in the community to handle learning from data streams in an incremental, single-pass and transparent manner. The main concentration so far lied in the development of approaches for single EFS models, basically used for prediction purposes. Forgetting mechanisms have been used to increase their flexibility, especially for the purpose to adapt quickly to changing situations such as drifting data distributions. These require forgetting factors steering the degree of timely out-weighing older learned concepts, whose adequate setting in advance or in adaptive fashion is not an easy and not a fully resolved task. In this paper, we propose a new concept of learning fuzzy systems from data streams, which we call online sequential ensembling of fuzzy systems (OS-FS). It is able to model the recent dependencies in streams on a chunk-wise basis: for each new incoming chunk, a new fuzzy model is trained from scratch and added to the ensemble (of fuzzy systems trained before). This induces (i) maximal flexibility in terms of being able to apply variable chunk sizes according to the actual system delay in receiving target values and (ii) fast reaction possibilities in the case of arising drifts. The latter are realized with specific prediction techniques on new data chunks based on the sequential ensemble members trained so far over time. We propose four different prediction variants including various weighting concepts in order to put higher weights on the members with higher inference certainty during the amalgamation of predictions of single members to a final prediction. In this sense, older members, which keep in mind knowledge about past states, may get dynamically reactivated in the case of cyclic drifts, which induce dynamic changes in the process behavior which are re-occurring from time to time later. Furthermore, we integrate a concept for properly resolving possible contradictions among members with similar inference certainties. The reaction onto drifts is thus autonomously handled on demand and on the fly during the prediction stage (and not during model adaptation/evolution stage as conventionally done in single EFS models), which yields enormous flexibility. Finally, in order to cope with large-scale and (theoretically) infinite data streams within a reasonable amount of prediction time, we demonstrate two concepts for pruning past ensemble members, one based on atypical high error trends of single members and one based on the non-diversity of ensemble members. The results based on two data streams showed significantly improved performance compared to single EFS models in terms of a better convergence of the accumulated chunk-wise ahead prediction error trends, especially in the case of regular and cyclic drifts. Moreover, the more advanced prediction schemes could significantly outperform standard averaging over all members’ outputs. Furthermore, resolving contradictory outputs among members helped to improve the performance of the sequential ensemble further. Results on a wider range of data streams from different application scenarios showed (i) improved error trend lines over single EFS models, as well as over related AI methods OS-ELM and MLPs neural networks retrained on data chunks, and (ii) slightly worse trend lines than on-line bagged EFS (as specific EFS ensembles), but with around 100 times faster processing times (achieving low processing times way below requiring milli-seconds for single samples updates).

Revisiting Data Protection, Privacy and State Surveillance

We are surrounded by massive, infinite streams of data. With the advent of new technologies which enable the extraction of useful and practical insights, the increasingly vast amount of data in our digital universe promises new forms of value, improvement in processes, and unleashes a wave of new opportunities for businesses and people. However, as data abounds, people are increasingly concerned about the security and protection of their data. It is no wonder, then, that data protection and privacy (DPP) legislation now form an important part of the legal framework of many countries around the world. This paper first examines the evolution of DPP legislation from an international perspective, considering the significant catalysts for the emergence of DPP laws. In considering the catalysts for the emergence of DPP laws, note that the expression “data protection and privacy (DPP)” is adopted loosely to accommodate historical and jurisdictional nuances. The second part of this paper discusses the right to privacy and its current status amidst growing state surveillance activities. This essay argues that the right to privacy still exists, albeit under increasing attack. The essay concludes with recommendations on ways to limit the scope of state surveillance activities.

Few Sample Learning without Data Storage for Lifelong Stream Mining (Student Abstract)

Continuously mining complexity data stream has recently been attracting an increasing amount of attention, due to the rapid growth of real-world vision/signal applications such as self-driving cars and online social media messages. In this paper, we aim to address two significant problems in the lifelong/incremental stream mining scenario: first, how to make the learning algorithms generalize to the unseen classes only from a few labeled samples; second, is it possible to avoid storing instances from previously seen classes to solve the catastrophic forgetting problem? We introduce a novelty stream mining framework to classify the infinite stream of data with different categories that occurred during different times. We apply a few-sample learning strategy to make the model recognize the novel class with limited samples; at the same time, we implement an incremental generative model to maintain old knowledge when learning new coming categories, and also avoid the violation of data privacy and memory restrictions simultaneously. We evaluate our approach in the continual class-incremental setup on the classification tasks and ensure the sufficient model capacity to accommodate for learning the new incoming categories.

On the randomness that generates biased samples: The limited randomness approach

We introduce two new algorithms for creating an exponentially biased sample over a possibly infinite data stream. Such an algorithm exists in the literature and uses O(log n) random bits per stream element, where n is the number of elements in the sample. In this paper we present algorithms that use O(1) random bits per stream element. In essence, what we achieve is to be able to choose an element at random, out of n elements, by sparing O(1) random bits. Although in general this is not possible, the exact problem we are studying makes it possible. The needed randomness for this task is provided through a random walk. To prove the correctness of our algorithms we use a model also introduced in this paper, the limited randomness model. It is based on the fact that survival probabilities are assigned to the stream elements before they start to arrive.

A Theory of Sequence Indexing and Working Memory in Recurrent Neural Networks.

To accommodate structured approaches of neural computation, we propose a class of recurrent neural networks for indexing and storing sequences of symbols or analog data vectors. These networks with randomized input weights and orthogonal recurrent weights implement coding principles previously described in vector symbolic architectures (VSA) and leverage properties of reservoir computing. In general, the storage in reservoir computing is lossy, and crosstalk noise limits the retrieval accuracy and information capacity. A novel theory to optimize memory performance in such networks is presented and compared with simulation experiments. The theory describes linear readout of analog data and readout with winner-take-all error correction of symbolic data as proposed in VSA models. We find that diverse VSA models from the literature have universal performance properties, which are superior to what previous analyses predicted. Further, we propose novel VSA models with the statistically optimal Wiener filter in the readout that exhibit much higher information capacity, in particular for storing analog data. The theory we present also applies to memory buffers, networks with gradual forgetting, which can operate on infinite data streams without memory overflow. Interestingly, we find that different forgetting mechanisms, such as attenuating recurrent weights or neural nonlinearities, produce very similar behavior if the forgetting time constants are matched. Such models exhibit extensive capacity when their forgetting time constant is optimized for given noise conditions and network size. These results enable the design of new types of VSA models for the online processing of data streams.

Gloss

An important class of applications computes on long-running or infinite streams of data, often with known fixed data rates. The latter is referred to as synchronous data flow ~(SDF) streams. These stream applications need to run on clusters or the cloud due to the high performance requirement. Further, they require live reconfiguration and reoptimization for various reasons such as hardware maintenance, elastic computation, or to respond to fluctuations in resources or application workload. However, reconfiguration and reoptimization without downtime while accurately preserving program state in a distributed environment is difficult. In this paper, we introduce Gloss, a suite of compiler and runtime techniques for live reconfiguration of distributed stream programs. Gloss, for the first time, avoids periods of zero throughput during the reconfiguration of both stateless and stateful SDF based stream programs. Furthermore, unlike other systems, Gloss globally reoptimizes and completely recompiles the program during reconfiguration. This permits it to reoptimize the application for entirely new configurations that it may not have encountered before. All these Gloss operations happen in-situ, requiring no extra hardware resources. We show how Gloss allows stream programs to reconfigure and reoptimize with no downtime and minimal overhead, and demonstrate the wider applicability of it via a variety of experiments.

Higher-Fidelity Frugal and Accurate Quantile Estimation Using a Novel Incremental <italic>Discretized</italic> Paradigm

Traditional pattern classification works with the moments of the distributions of the features and involves the estimation of the means and variances. As opposed to this, more recently, research has indicated the power of using the quantiles of the distributions because they are more robust and applicable for non-parametric methods. The estimation of the quantiles is even more pertinent when one is mining data streams. However, the complexity of quantile estimation is much higher than the corresponding estimation of the mean and variance, and this increased complexity is more relevant as the size of the data increases. Clearly, in the context of infinite data streams, a computational and space complexity that is linear in the size of the data is definitely not affordable. In order to alleviate the problem complexity, recently, a very limited number of studies have devised incremental quantile estimators [1] , [2] . Estimators within this class resort to updating the quantile estimates based on the most recent observation(s), and this yields updating schemes with a very small computational footprint–a constant-time (i.e., $O(1)$ ) complexity. In this paper, we pursue this research direction and present an estimator that we refer to as a higher-fidelity frugal [1] quantile estimator. First, it guarantees a substantial advancement of the family of Frugal estimators introduced in [1] . The highlight of the present scheme is that it works in the discretized space, and it is thus a pioneering algorithm within the theory of discretized algorithms. 1 The convergence results that we have proven are based on the theory of stochastic point location [3] , which we advocate as a new tool for solving a large class of online estimation problems. Extensive simulation results show that our estimator outperforms the original Frugal algorithm in terms of both speed and accuracy. 1 The fact that discretized Learning Automata schemes are superior to their continuous counterparts has been clearly demonstrated in the literature. This is the first paper, to our knowledge, that proves the advantages of discretization within the domain of quantile estimation.

Privacy-protected statistics publication over social media user trajectory streams

An increasing amount of user location information is being generated due to the widespread use of social network applications and the ubiquitous adoption of mobile and wearable technologies. This data can be analysed to identify precise trajectories of individuals — where they went and when they were there. This is an obvious privacy issue, yet publication of real-time aggregate over such location streams can provide valuable resources for researchers and government agencies, e.g., in case of pandemics it would be very useful to identify who might have come into contact with an infected individual at a given time. Differential privacy techniques have become popular and widely adopted to address privacy concerns. However, there are three key issues that limit the application of existing differential privacy approaches to user trajectory data: (a) the heterogeneous nature of the trajectories, (b) uniform sliding window mechanisms do not meet individual privacy requirements and (c) limited privacy budgets and impact on data utility when applied to infinite data streams. To tackle these problems, this paper proposes a private real-time trajectory stream statistics publication mechanism utilizingdifferential privacy (DP-PSP). To relieve the heterogeneity issues, anchor point discovery (e.g., fixed locations like museums, parks, etc.) and road segmenting mechanisms are proposed. We provide an adaptive w-step sliding window approach that allows users to specify their own dynamic privacy budget distribution to optimize their own privacy budget. To preserve the data utility, we present multi-timestamp prediction models and private k-nearest neighbour selection and perturbation algorithms to reduce the amount of perturbation distortion induced through the differential privacy mechanism. Comprehensive experiments over real-life location-based social network user trajectories show that DP-PSP provides private data aggregate over infinite trajectory streams and boosts the utility and quality of the perturbed aggregation without compromising individual privacy.

New Splitting Criteria for Decision Trees in Stationary Data Streams.

The most popular tools for stream data mining are based on decision trees. In previous 15 years, all designed methods, headed by the very fast decision tree algorithm, relayed on Hoeffding's inequality and hundreds of researchers followed this scheme. Recently, we have demonstrated that although the Hoeffding decision trees are an effective tool for dealing with stream data, they are a purely heuristic procedure; for example, classical decision trees such as ID3 or CART cannot be adopted to data stream mining using Hoeffding's inequality. Therefore, there is an urgent need to develop new algorithms, which are both mathematically justified and characterized by good performance. In this paper, we address this problem by developing a family of new splitting criteria for classification in stationary data streams and investigating their probabilistic properties. The new criteria, derived using appropriate statistical tools, are based on the misclassification error and the Gini index impurity measures. The general division of splitting criteria into two types is proposed. Attributes chosen based on type- splitting criteria guarantee, with high probability, the highest expected value of split measure. Type- criteria ensure that the chosen attribute is the same, with high probability, as it would be chosen based on the whole infinite data stream. Moreover, in this paper, two hybrid splitting criteria are proposed, which are the combinations of single criteria based on the misclassification error and Gini index.

Distributed analysis and filtering of application event streams

Abstract Large information systems comprise different interconnected hardware and software components, that collectively generate large volumes of data. Furthermore, the run-time analysis of such data involves computationally expensive algorithms, and is pivotal to a number of software engineering activities such as, system understanding, diagnostics, and root cause analysis. In a quest to increase the performance of run-time analysis for large sets of logged data, we present an approach that allows for the real time reduction of one or more event streams by utilizing a set of filtering criteria. More specifically, the approach employs a similarity measure that is based on information theory principles, and is applied between the features of the incoming events, and the features of a set of retrieved or constructed events, that we refer to as beacons. The proposed approach is domain and event schema agnostic, can handle infinite data streams using a caching algorithm, and can be parallelized in order to tractably process high volume, high frequency, and high variability data. Experimental results obtained using the KDD’99 and CTU-13 labeled data sets, indicate that the approach is scalable, and can yield highly reduced sets with high recall values with respect to a use case.

How to adjust an ensemble size in stream data mining?

In this paper we propose a new approach for designing an ensemble applied to stream data classification. Our approach is supported by two theorems showing how to decide whether a new component should be added to the ensemble or not, based on the assumption that such an action should increase the accuracy of the ensemble not only for the current portion of observations but also for the whole (infinite) data stream. The conclusions of these theorems hold with a certain probability (confidence) set by the user. Through computer simulations, among others, we show that decreasing the confidence that decision based on the finite portion of the stream is the same as based on the whole (infinite) data stream only slightly improves the accuracy at the expense of significant memory consumption. Moreover, we will introduce a novel procedure of weighting ensemble components, i.e. decision trees, by assigning a weight to each leaf of the tree. In previous approaches a weight was assigned to the whole ensemble component. The new approach is based on the observation that probability of the correct tree outcome is different in various tree sections.

Online Categorical Subspace Learning for Sketching Big Data with Misses

With the scale of data growing every day, reducing the dimensionality (a.k.a. sketching) of high-dimensional data has emerged as a task of paramount importance. Relevant issues to address in this context include the sheer volume of data that may consist of categorical samples, the typically streaming format of acquisition, and the possibly missing entries. To cope with these challenges, the present paper develops a novel categorical subspace learning approach to unravel the latent structure for three prominent categorical (bilinear) models, namely, Probit, Tobit, and Logit. The deterministic Probit and Tobit models treat data as quantized values of an analog-valued process lying in a low-dimensional subspace, while the probabilistic Logit model relies on low dimensionality of the data log-likelihood ratios. Leveraging the low intrinsic dimensionality of the sought models, a rank regularized maximum-likelihood estimator is devised, which is then solved recursively via alternating majorization-minimization to sketch high-dimensional categorical data `on the fly.' The resultant procedure alternates between sketching the new incomplete datum and refining the latent subspace, leading to lightweight first-order algorithms with highly parallelizable tasks per iteration. As an extra degree of freedom, the quantization thresholds are also learned jointly along with the subspace to enhance the predictive power of the sought models. Performance of the subspace iterates is analyzed for both infinite and finite data streams, where for the former asymptotic convergence to the stationary point set of the batch estimator is established, while for the latter sublinear regret bounds are derived for the empirical cost. Simulated tests with both synthetic and real-world datasets corroborate the merits of the novel schemes for real-time movie recommendation and chess-game classification.