Infinite Streams Research Articles

A simple and general axiomatization of average utility maximization for infinite streams

This paper provides the most general preference axiomatization of average utility (AU) maximization over infinite sequences presently available, reaching almost complete generality. The only restriction is that all periodic sequences should be contained in the domain. Infinite sequences may designate intertemporal outcomes streams where AU models patience, welfare allocations where AU models fairness, or decisions under ambiguity where AU models complete ignorance. As a methodological contribution, this paper shows that infinite-dimensional representations can be simpler, rather than more complex, than finite-dimensional ones. Infinite dimensions provide a richness that may be convenient rather than cumbersome. In particular, (empirically problematic) continuity assumptions are not needed in our axiomatization. Continuity is optional.

Continuous Functions on Final Comodels of Free Algebraic Theories

In 2009, Ghani, Hancock and Pattinson gave a tree-like representation of stream processors $A^{\mathbb{N}} \rightarrow B^{\mathbb{N}}$. In 2021, Garner showed that this representation can be established in terms of algebraic theory and comodels: the set of infinite streams $A^{\mathbb{N}}$ is the final comodel of the algebraic theory of $A$-valued input $\mathbb{T}_A$ and the set of stream processors $\mathit{Top}(A^{\mathbb{N}},B^{\mathbb{N}})$ can be seen as the final $\mathbb{T}_A$-$\mathbb{T}_B$-bimodel. In this paper, we generalize Garner's results to the case of free algebraic theories.

Median Filters on FPGAs for Infinite Data and Large, Rectangular Windows

Efficient architectures and implementations of median filters have been well investigated in the past. In this article, we focus on median filters for very big scientific applications with very large windows and an infinite stream of data, inspired by big data needs in the Square Kilometre Array (SKA) pulsar search engine, but transferable to other big data domains. We propose a novel approach for very large rectangular windows on an FPGA accelerator device able to support the processing of infinite streams of data. OpenCL is used for rapid parameter sweeping and design space exploration based on a pipelined model of the system. Evaluation on a host/accelerator system with an Arria 10 device surpassed 64 million values processed per second considered for the SKA real time requirement, achieving 83.4M value/s while reading from/writing to disk. These results are compared with a state-of-the-art software implementation only achieving 41M value/s for over twice the total system energy cost.

Reducing Catastrophic Forgetting in Self Organizing Maps with Internally-Induced Generative Replay (Student Abstract)

A lifelong learning agent is able to continually learn from potentially infinite streams of pattern sensory data. One major historic difficulty in building agents that adapt in this way is that neural systems struggle to retain previously-acquired knowledge when learning from new samples. This problem is known as catastrophic forgetting (interference) and remains an unsolved problem in the domain of machine learning to this day. While forgetting in the context of feedforward networks has been examined extensively over the decades, far less has been done in the context of alternative architectures such as the venerable self-organizing map (SOM), an unsupervised neural model that is often used in tasks such as clustering and dimensionality reduction. Although the competition among its internal neurons might carry the potential to improve memory retention, we observe that a fixed-sized SOM trained on task incremental data, i.e., it receives data points related to specific classes at certain temporal increments, it experiences severe interference. In this study, we propose the c-SOM, a model that is capable of reducing its own forgetting when processing information.

Anomalous Alliances: Spinoza and Abolition

What effects are produced in an encounter between what Gilles Deleuze calls Spinoza's ‘practical philosophy’ and abolition? Closely following Deleuze's account of Spinoza, this essay moves from the reifying and weakening punitive moralism of carceral state thought towards a joyful materialist abolitionist ethic. It starts with the three theses for which, Deleuze argues, Spinoza was denounced in his own lifetime: materialism (devaluation of consciousness), immoralism (devaluation of all values) and atheism (devaluation of the sad passions). From these three, it derives three parallel abolitionist theses: (1) Spinozan materialism undermines the reifications of carceral state thought; (2) Spinozan ethics undermines the punitivism of the carceral state; and (3) Spinozan joy is inversely proportional to the power of the carceral state. While Spinoza's corpus may not give us an adequate account of the complex dynamics of the carceral state and racial capitalism today, this essay argues that in the infinite streams of the Ethics we nonetheless find some vital strategies through which we might compose an anomalous alliance between this condemned philosopher and abolition.

Positioning resilience science more centrally in affirming LGBTIQA+ persons and communities

Historically, in psychology, there was a tendency towards focusing on the individual and intrapsychic pathologisation of LGBTIQA+ persons. Despite a recent shift to affirmative, systemic, interpersonal, and contextual stances, too much emphasis in South African LGBTIQA+ scholarly work remains on adversity. Adversity derived from historical tensions may have accelerated adaptive problem-solving capabilities and solution-focused behaviours in some LGBTIQA+ populations. Certain solutions have generated creative resilience responses inexorably situated in race, ethnicity, culture, religion, gender, history, political oppression, and social class. Resilience science that has similarly evolved from the individual trait conceptualisation may have an important complementary contribution to make in affirming LGBTIQA+ persons and communities and enhancing the understanding of their resilience. Exploring resilience for wellbeing and survival addresses the infinite streams of human experiences of vulnerability and sustainable adaptive solutions. The multisystemic resilience perspective in this article endeavours to summarise current thinking in resilience science and position its applicability to future South African LGBTIQA+ scholarly work, building on existing systematic and critical reviews relating to the resilience of LGBTIQA+ persons. The aim of this position piece, suggesting the expansion of a ‘resilience’ frame, is to offer an important intervention in the overarching ways in which South African scholars and psychology professionals theorise and conduct research; to inform therapeutic and other psychological services to LGBTIQA+ persons and provide an important counterweight to the more general focus in South African LGBTIQA+ scholarship and psychological practice on adversity.

Matching Regular Expressions on uncertain data

In this paper we study regular expression matching in cases in which the identity of the symbols received is subject to uncertainty. We develop a model of symbol emission and uses a modification of the shortest path algorithm to find optimal matches on the Cartesian Graph of an expression provided that the input is a finite list. In the case of infinite streams, we show that the problem is in general undecidable but, if each symbols is received with probability 0 infinitely often, then with probability 1 the problem is decidable.

Differentially Private Web Browsing Trajectory over Infinite Streams

Nowadays, a lot of data mining applications, such as web traffic analysis and content popularity prediction, leverage users’ web browsing trajectories to improve their performance. However, the disclosure of web browsing trajectory is the most prominent issue. A novel privacy model, named Differential Privacy, is used to rigorously protect user’s privacy. Some works have applied this privacy model to spatial-temporal streams. However, these works either protect the users’ activities in different places separately or protect their activities in all places jointly. The former one cannot protect trajectories that traverse multiple places; while the latter ignores the differences among places and suffers the degradation of data utility (i.e., data accuracy). In this paper, we propose a w , n -differential privacy to protect any spatial-temporal sequence occurring in w successive timestamps and n -range places. To achieve better data utility, we propose two implementation algorithms, named Spatial-Temporal Budget Distribution (STBD) and Spatial-Temporal RescueDP (STR). Theoretical analysis and experimental results show that these two algorithms can achieve a balance between data utility and trajectory privacy guarantee.

Watermarks in stream processing systems

Streaming data processing is an exercise in taming disorder: from oftentimes huge torrents of information, we hope to extract powerful and timely analyses. But when dealing with streaming data, the unbounded and temporally disordered nature of real-world streams introduces a critical challenge: how does one reason about the completeness of a stream that never ends? In this paper, we present a comprehensive definition and analysis of watermarks , a key tool for reasoning about temporal completeness in infinite streams. First, we describe what watermarks are and why they are important, highlighting how they address a suite of stream processing needs that are poorly served by eventually-consistent approaches: • Computing a single correct answer, as in notifications. • Reasoning about a lack of data, as in dip detection. • Performing non-incremental processing over temporal subsets of an infinite stream, as in statistical anomaly detection with cubic spline models. • Safely and punctually garbage collecting obsolete inputs and intermediate state. • Surfacing a reliable signal of overall pipeline health . Second, we describe, evaluate, and compare the semantically equivalent, but starkly different, watermark implementations in two modern stream processing engines: Apache Flink and Google Cloud Dataflow.

Impact of data correlation on privacy budget allocation in continuous publication of location statistics

Continuous publication of statistics collected from various location-based applications may compromise users’ privacy as the statistics could be procured from users’ private data. Differential Privacy (DP) is a new privacy notion that offers a strong privacy guarantee to all users who participate in the statistics. However, the existing DP mechanism for continuous publication of location statistics provides a privacy guarantee with the assumption that the data-points of users stream at consecutive timestamps are independent. In reality, users’ data-points may be temporally correlated, resulting in more privacy leakage due to an inadequate supply of privacy budget to the timestamps where the data-points are correlated. In this paper, we present a reformulated differential privacy definition to quantify the impact of temporal correlation on privacy leakage. Then, we introduce a privacy budget allocation method for allocating an adequate amount of privacy budget to each successive timestamps under the protection of differential privacy. Our solution adopts w-event privacy for continuously releasing statistics over infinite streams. The main idea is to check the dissimilarity between statistics at each timestamp and decide whether to publish current statistics or last release statistics. Finally, we evaluate the data utility of our proposed method by presenting experimental results for real and synthetic data sets.

Revisiting Data Protection, Privacy and State Surveillance

We are surrounded by massive, infinite streams of data. With the advent of new technologies which enable the extraction of useful and practical insights, the increasingly vast amount of data in our digital universe promises new forms of value, improvement in processes, and unleashes a wave of new opportunities for businesses and people. However, as data abounds, people are increasingly concerned about the security and protection of their data. It is no wonder, then, that data protection and privacy (DPP) legislation now form an important part of the legal framework of many countries around the world. This paper first examines the evolution of DPP legislation from an international perspective, considering the significant catalysts for the emergence of DPP laws. In considering the catalysts for the emergence of DPP laws, note that the expression “data protection and privacy (DPP)” is adopted loosely to accommodate historical and jurisdictional nuances. The second part of this paper discusses the right to privacy and its current status amidst growing state surveillance activities. This essay argues that the right to privacy still exists, albeit under increasing attack. The essay concludes with recommendations on ways to limit the scope of state surveillance activities.

A Distributed Stream Processing Middleware Framework for Real-Time Analysis of Heterogeneous Data on Big Data Platform: Case of Environmental Monitoring.

In recent years, the application and wide adoption of Internet of Things (IoT)-based technologies have increased the proliferation of monitoring systems, which has consequently exponentially increased the amounts of heterogeneous data generated. Processing and analysing the massive amount of data produced is cumbersome and gradually moving from classical ‘batch’ processing—extract, transform, load (ETL) technique to real-time processing. For instance, in environmental monitoring and management domain, time-series data and historical dataset are crucial for prediction models. However, the environmental monitoring domain still utilises legacy systems, which complicates the real-time analysis of the essential data, integration with big data platforms and reliance on batch processing. Herein, as a solution, a distributed stream processing middleware framework for real-time analysis of heterogeneous environmental monitoring and management data is presented and tested on a cluster using open source technologies in a big data environment. The system ingests datasets from legacy systems and sensor data from heterogeneous automated weather systems irrespective of the data types to Apache Kafka topics using Kafka Connect APIs for processing by the Kafka streaming processing engine. The stream processing engine executes the predictive numerical models and algorithms represented in event processing (EP) languages for real-time analysis of the data streams. To prove the feasibility of the proposed framework, we implemented the system using a case study scenario of drought prediction and forecasting based on the Effective Drought Index (EDI) model. Firstly, we transform the predictive model into a form that could be executed by the streaming engine for real-time computing. Secondly, the model is applied to the ingested data streams and datasets to predict drought through persistent querying of the infinite streams to detect anomalies. As a conclusion of this study, a performance evaluation of the distributed stream processing middleware infrastructure is calculated to determine the real-time effectiveness of the framework.

Abstract machines, optimal reduction, and streams

AbstractIn this paper, we propose and explore a new approach to abstract machines and optimal reduction via streams, infinite sequences of elements. We first define a sequential abstract machine capable of performing directed virtual reduction (DVR) and then we extend it to its parallel version, whose equivalence is explained through the properties of DVR itself. The result is a formal definition of the λ-calculus interpreter called Parallel Environment for Lambda Calculus Reduction (PELCR), a software for λ-calculus reduction based on the Geometry of Interaction. In particular, we describe PELCR as a stream-processing abstract machine, which in principle can also be applied to infinite streams.

SRS-LM: differentially private publication for infinite streaming data

Recently, differential privacy achieves good trade-offs between data publishing and sensitive information hiding. But in data publishing for infinite streams, along with the increasing release of streaming data, the privacy budget consumption of every data continues grow, leading to a low-level data utility. To remedy this, this paper proposes a Laplace Mechanism based on Simple Random Sampling (SRS-LM) to differentially private publishing infinite streaming data. Specifically, we generate a Laplace series, which has a finite length, as the fundamental noise series and when an update comes, we randomly simple the noise from the Laplace series to add to the update data. Experimental results show that SRS-LM outperforms state-of-the-art differential privacy mechanisms in terms of security and mean absolute error for large quantities of queries.

Gloss

An important class of applications computes on long-running or infinite streams of data, often with known fixed data rates. The latter is referred to as synchronous data flow ~(SDF) streams. These stream applications need to run on clusters or the cloud due to the high performance requirement. Further, they require live reconfiguration and reoptimization for various reasons such as hardware maintenance, elastic computation, or to respond to fluctuations in resources or application workload. However, reconfiguration and reoptimization without downtime while accurately preserving program state in a distributed environment is difficult. In this paper, we introduce Gloss, a suite of compiler and runtime techniques for live reconfiguration of distributed stream programs. Gloss, for the first time, avoids periods of zero throughput during the reconfiguration of both stateless and stateful SDF based stream programs. Furthermore, unlike other systems, Gloss globally reoptimizes and completely recompiles the program during reconfiguration. This permits it to reoptimize the application for entirely new configurations that it may not have encountered before. All these Gloss operations happen in-situ, requiring no extra hardware resources. We show how Gloss allows stream programs to reconfigure and reoptimize with no downtime and minimal overhead, and demonstrate the wider applicability of it via a variety of experiments.

Asymptotic properties of welfare relations

We introduce and discuss notions of efficiency in the aggregation of infinite utility streams. For any utility streams x and y, our efficiency criteria roughly require this: If a utility stream x dominates another utility stream y and if the asymptotic density of the set of coordinates in favor of x is strictly positive, then x is socially preferred to y. As a robustness check of the proposed efficiency axioms we explore the consistency of the axioms with notions of equity. Our main results characterize one period utility domains, i.e., the set of utilities Y attainable by each generation, admitting a social welfare aggregator with the desired properties.

Samza

Distributed stream processing systems need to support stateful processing, recover quickly from failures to resume such processing, and reprocess an entire data stream quickly. We present Apache Samza, a distributed system for stateful and fault-tolerant stream processing. Samza utilizes a partitioned local state along with a low-overhead background changelog mechanism, allowing it to scale to massive state sizes (hundreds of TB) per application. Recovery from failures is sped up by re-scheduling based on Host Affinity. In addition to processing infinite streams of events, Samza supports processing a finite dataset as a stream, from either a streaming source (e.g., Kafka), a database snapshot (e.g., Databus), or a file system (e.g. HDFS), without having to change the application code (unlike the popular Lambda-based architectures which necessitate maintenance of separate code bases for batch and stream path processing). Samza is currently in use at LinkedIn by hundreds of production applications with more than 10, 000 containers. Samza is an open-source Apache project adopted by many top-tier companies (e.g., LinkedIn, Uber, Netflix, TripAdvisor, etc.). Our experiments show that Samza: a) handles state efficiently, improving latency and throughput by more than 100X compared to using a remote storage; b) provides recovery time independent of state size; c) scales performance linearly with number of containers; and d) supports reprocessing of the data stream quickly and with minimal interference on real-time traffic.

Distributed analysis and filtering of application event streams

Abstract Large information systems comprise different interconnected hardware and software components, that collectively generate large volumes of data. Furthermore, the run-time analysis of such data involves computationally expensive algorithms, and is pivotal to a number of software engineering activities such as, system understanding, diagnostics, and root cause analysis. In a quest to increase the performance of run-time analysis for large sets of logged data, we present an approach that allows for the real time reduction of one or more event streams by utilizing a set of filtering criteria. More specifically, the approach employs a similarity measure that is based on information theory principles, and is applied between the features of the incoming events, and the features of a set of retrieved or constructed events, that we refer to as beacons. The proposed approach is domain and event schema agnostic, can handle infinite data streams using a caching algorithm, and can be parallelized in order to tractably process high volume, high frequency, and high variability data. Experimental results obtained using the KDD’99 and CTU-13 labeled data sets, indicate that the approach is scalable, and can yield highly reduced sets with high recall values with respect to a use case.

Stream fusion, to completeness

Stream processing is mainstream (again): Widely-used stream libraries are now available for virtually all modern OO and functional languages, from Java to C# to Scala to OCaml to Haskell. Yet expressivity and performance are still lacking. For instance, the popular, well-optimized Java 8 streams do not support the zip operator and are still an order of magnitude slower than hand-written loops. We present the first approach that represents the full generality of stream processing and eliminates overheads, via the use of staging. It is based on an unusually rich semantic model of stream interaction. We support any combination of zipping, nesting (or flat-mapping), sub-ranging, filtering, mapping—of finite or infinite streams. Our model captures idiosyncrasies that a programmer uses in optimizing stream pipelines, such as rate differences and the choice of a “for” vs. “while” loops. Our approach delivers hand-written–like code, but automatically. It explicitly avoids the reliance on black-box optimizers and sufficiently-smart compilers, offering highest, guaranteed and portable performance. Our approach relies on high-level concepts that are then readily mapped into an implementation. Accordingly, we have two distinct implementations: an OCaml stream library, staged via MetaOCaml, and a Scala library for the JVM, staged via LMS. In both cases, we derive libraries richer and simultaneously many tens of times faster than past work. We greatly exceed in performance the standard stream libraries available in Java, Scala and OCaml, including the well-optimized Java 8 streams.

Clustering Big Data streams: recent challenges and contributions

Abstract Traditional clustering algorithms merely considered static data. Today's various applications and research issues in big data mining have however to deal with continuous, possibly infinite streams of data, arriving at high velocity. Web traffic data, surveillance data, sensor measurements and stock trading are only some examples of these daily-increasing applications. Since the growth of data volumes is accompanied by a similar raise in their dimensionalities, clusters cannot be expected to completely appear when considering all attributes together. Subspace clustering is a general approach that solved that issue by automatically finding the hidden clusters within different subsets of the attributes rather than considering all attributes together. In this article, novel methods for an efficient subspace clustering of high-dimensional big data streams are presented. Approaches that efficiently combine the anytime clustering concept with the stream subspace clustering paradigm are discussed. Additionally, efficient and adaptive density-based clustering algorithms are presented for high-dimensional data streams. Novel open-source assessment framework and evaluation measures are additionally presented for subspace stream clustering.