Sublinear Space Research Articles

Centrifuger: lossless compression of microbial genomes for efficient and accurate metagenomic sequence classification

Centrifuger is an efficient taxonomic classification method that compares sequencing reads against a microbial genome database. In Centrifuger, the Burrows-Wheeler transformed genome sequences are losslessly compressed using a novel scheme called run-block compression. Run-block compression achieves sublinear space complexity and is effective at compressing diverse microbial databases like RefSeq while supporting fast rank queries. Combining this compression method with other strategies for compacting the Ferragina-Manzini (FM) index, Centrifuger reduces the memory footprint by half compared to other FM-index-based approaches. Furthermore, the lossless compression and the unconstrained match length help Centrifuger achieve greater accuracy than competing methods at lower taxonomic levels.

The 2CNF Boolean formula satisfiability problem and the linear space hypothesis

We aim at investigating the solvability/insolvability of nondeterministic logarithmic-space (NL) decision, search, and optimization problems parameterized by natural size parameters using simultaneously polynomial time and sub-linear space. We are particularly focused on 2SAT3—a restricted variant of the 2CNF Boolean (propositional) formula satisfiability problem in which each variable of a given 2CNF formula appears at most 3 times in the form of literals—parameterized by the total number mvbl(ϕ) of variables of each given Boolean formula ϕ. We propose a new, practical working hypothesis, called the linear space hypothesis (LSH), which asserts that (2SAT3,mvbl) cannot be solved in polynomial time using only “sub-linear” space (i.e., mvbl(x)εpolylog(|x|) space for a constant ε∈[0,1)) on all instances x. Immediate consequences of LSH include L≠NL, LOGDCFL≠LOGCFL, and SC≠NSC. For our investigation, we fully utilize a key notion of “short reductions”, under which the class PsubLIN of all parameterized polynomial-time sub-linear-space solvable problems is indeed closed.

Auxo: A Scalable and Efficient Graph Stream Summarization Structure

A graph stream refers to a continuous stream of edges, forming a huge and fast-evolving graph. The vast volume and high update speed of a graph stream bring stringent requirements for the data management structure, including sublinear space cost, computation-efficient operation support, and scalability of the structure. Existing designs summarize a graph stream by leveraging a hash-based compressed matrix and representing an edge using its fingerprint to achieve practical storage for a graph stream with a known upper bound of data volume. However, they fail to support the dynamically extending of graph streams. In this paper, we propose Auxo, a scalable structure to support space/time efficient summarization of dynamic graph streams. Auxo is built on a proposed novel prefix embedded tree (PET) which leverages binary logarithmic search and common binary prefixes embedding to provide an efficient and scalable tree structure. PET reduces the item insert/query time from O (| E |) to O ( log | E |) as well as reducing the total storage cost by a log | E | scale, where | E | is the size of the edge set in a graph stream. To further improve the memory utilization of PET during scaling, we propose a proportional PET structure that extends a higher level in a proportionally incremental style. We conduct comprehensive experiments on large-scale real-world datasets to evaluate the performance of this design. Results show that Auxo significantly reduces the insert and query time by one to two orders of magnitude compared to the state of the arts. Meanwhile, Auxo achieves efficiently and economically structure scaling with an average memory utilization of over 80%.

Persistent Summaries

A persistent data structure , also known as a multiversion data structure in the database literature, is a data structure that preserves all its previous versions as it is updated over time. Every update (inserting, deleting, or changing a data record) to the data structure creates a new version, while all the versions are kept in the data structure so that any previous version can still be queried. Persistent data structures aim at recording all versions accurately, which results in a space requirement that is at least linear to the number of updates. In many of today’s big data applications, in particular, for high-speed streaming data, the volume and velocity of the data are so high that we cannot afford to store everything. Therefore, streaming algorithms have received a lot of attention in the research community, which uses only sublinear space by sacrificing slightly on accuracy. All streaming algorithms work by maintaining a small data structure in memory, which is usually called a sketch , summary , or synopsis . The summary is updated upon the arrival of every element in the stream, thus it is ephemeral , meaning that it can only answer queries about the current status of the stream. In this article, we aim at designing persistent summaries, thereby giving streaming algorithms the ability to answer queries about the stream at any prior time.

Complete integral convergence for weighted sums of widely negative dependent random variables under the sub-linear expectations

In this article, we establish some results on the complete convergence and complete integral convergence for weighted sums of widely negative dependent random variables under the sub-linear expectations, which extend some complete moment convergence theorems from the classical probability space to the situation of sub-linear expectation space.

Complete convergence theorems for arrays of row-wise extended negatively dependent random variables under sub-linear expectations

The goal of this paper is to establish complete convergence theorems for an array of row-wise END random variables under sub-linear expectation space. As applications of the exponential inequalities, we extend some complete convergence theorems from the traditional probability space to the sub-linear expectation space and our results generalize corresponding results obtained by Hu.

Central limit theorem for linear processes generated by m-dependent random variables under the sub-linear expectation

We prove the Rosnethal’s inequality of m-dependent random variables under the sub-linear expectation in this paper. Furthermore, we use this inequality to investigate the central limit theorem for linear processes generated by m-dependent random variables under sub-linear expectations. This article use the basic definitions of sub-linear expectation space, Kronecker lemma, Cr inequality etc. to demonstrate the main conclusion.

Leveraging Active Decremental TTL Measuring for Flexible and Efficient NAT identification

Malicious attacks can be launched by misusing the network address translation technique as a camouflage. To mitigate such threats, network address translation identification is investigated to identify network address translation devices and detect abnormal behaviors. However, existing methods in this field are mainly developed for relatively small-scale networks and work in an offline manner, which cannot adapt to the real-time inference requirements in high-speed network scenarios. In this paper, we propose a flexible and efficient network address translation identification scheme based on actively measuring the distance of a round trip to a target with decremental time-to-live values. The basic intuition is that the incoming and outgoing traffic from a network address translation device usually experiences the different number of hops, which can be discovered by probing with dedicated time-to-live values. We explore a joint effort of parallel transmission, stateless probes, and flexible measuring reuse to accommodate the efficiency of the measuring process. We further accelerate statistical counting with a new sublinear space data structure Bi-sketch. We implement a prototype and conduct real-world deployments with 1000 volunteers in 31 Chinese provinces, which is believed to bring insight for ground truth collection in this field. Experiments on multi-sources datasets show that our proposal can achieve as high precision and recall as 95% with a traffic handling throughput of over 106 pps.

Large deviation principle for linear processes generated by real stationary sequences under the sub-linear expectation

Let be the linear processes, where is an absolutely summable sequence of real numbers and is a sequence of real stationary independent random variables under the sub-linear expectation space In the framework of sub-linear expectation space, the Beveridge–Nelson decomposition of linear processes and the sub-additivity of capacity are used to establish the large deviation principle for linear processes in the sense of upper probability.

Convergence for sums of i.i.d. random variables under sublinear expectations

In this paper, we obtain equivalent conditions of complete moment convergence of the maximum for partial weighted sums of independent identically distributed random variables under sublinear expectations space. The results obtained in the paper are extensions of the equivalent conditions of complete moment convergence of the maximum under classical linear expectation space.

Complete Convergence for Weighted Sums of Widely Acceptable Random Variables under Sublinear Expectations

Using different methods than the probability space, under the condition that the Choquet integral exists, we study the complete convergence theorem for weighted sums of widely acceptable random variables under sublinear expectation space. We proved corresponding theorem which was extended to the sublinear expectations’ space from the probability space, and similar results were obtained.

Graph Sparsification for Derandomizing Massively Parallel Computation with Low Space

The Massively Parallel Computation (MPC) model is an emerging model that distills core aspects of distributed and parallel computation, developed as a tool to solve combinatorial (typically graph) problems in systems of many machines with limited space. Recent work has focused on the regime in which machines have sublinear (in n , the number of nodes in the input graph) space, with randomized algorithms presented for the fundamental problems of Maximal Matching and Maximal Independent Set. However, there have been no prior corresponding deterministic algorithms. A major challenge underlying the sublinear space setting is that the local space of each machine might be too small to store all edges incident to a single node. This poses a considerable obstacle compared to classical models in which each node is assumed to know and have easy access to its incident edges. To overcome this barrier, we introduce a new graph sparsification technique that deterministically computes a low-degree subgraph, with the additional property that solving the problem on this subgraph provides significant progress towards solving the problem for the original input graph. Using this framework to derandomize the well-known algorithm of Luby [SICOMP’86], we obtain O (log Δ + log log n )-round deterministic MPC algorithms for solving the problems of Maximal Matching and Maximal Independent Set with O ( n ɛ ) space on each machine for any constant ɛ > 0. These algorithms also run in O (log Δ) rounds in the closely related model of CONGESTED CLIQUE, improving upon the state-of-the-art bound of O (log 2 Δ) rounds by Censor-Hillel et al. [DISC’17].

Several Different Types of Convergence for ND Random Variables under Sublinear Expectations

The goal of this paper is to build average convergence and almost sure convergence for ND (negatively dependent) sequences of random variables under sublinear expectation space. By using the basic definition of sublinear expectation space, Markov inequality, and C r inequality, we extend average convergence and almost sure convergence theorems for ND sequences of random variables under sublinear expectation space, and we provide a way to learn this subject.

Longest common subsequence in sublinear space

We present the first o(n)-space polynomial-time algorithm for computing the length of a longest common subsequence. Given two strings of length n, the algorithm runs in O(n3) time with O(nlog1.5⁡n2log⁡n) bits of space.

Fast and Accurate Traffic Measurement With Hierarchical Filtering

Sketches have been widely used to record traffic statistics using sub-linear space data structure. Most sketches focus on the traffic estimation of elephant flows (i.e., heavy hitters) due to their importance to many network optimization tasks, e.g., traffic engineering and load balancing. In fact, the information of aggregate mice flows (e.g., all the mice flows with the same source IP) is also crucial to many security-associated tasks, e.g., DDoS detection and network scan detection. However, the previous solutions, e.g., measuring each individual flow or using multiple sketches for independent measurement tasks, will result in worse estimation error or higher computational overhead. To conquer the above disadvantages, we propose an accurate traffic measurement framework with multiple filters, called Sketchtree, to efficiently measure both elephant flows and aggregate mice flows. These filters in Sketchtree are organized in a hierarchical manner, and help to alleviate the hash collision and improve the measurement accuracy, as the number of flows through hierarchical filters in turn will be decreased gradually. We also design some mechanisms to improve the resource utilization efficiency. To validate our proposal, we have implemented Sketchtree and conducted experimental evaluation using real campus traffic traces. The experimental results show that Sketchtree can increase the processing speed by 100 percent, and reduce the measurement error by over 30 percent compared with state-of-the-art sketches.

More Time-Space Tradeoffs for Finding a Shortest Unique Substring

We extend recent results regarding finding shortest unique substrings (SUSs) to obtain new time-space tradeoffs for this problem and the generalization of finding k-mismatch SUSs. Our new results include the first algorithm for finding a k-mismatch SUS in sublinear space, which we obtain by extending an algorithm by Senanayaka (2019) and combining it with a result on sketching by Gawrychowski and Starikovskaya (2019). We first describe how, given a text T of length n and m words of workspace, with high probability we can find an SUS of length L in O(n(L/m)logL) time using random access to T, or in O(n(L/m)log2(L)loglogσ) time using O((L/m)log2L) sequential passes over T. We then describe how, for constant k, with high probability, we can find a k-mismatch SUS in O(n1+ϵL/m) time using O(nϵL/m) sequential passes over T, again using only m words of workspace. Finally, we also describe a deterministic algorithm that takes O(nτlogσlogn) time to find an SUS using O(n/τ) words of workspace, where τ is a parameter.

Complete convergence theorem for negatively dependent random variables under sub-linear expectations

Under the condition that the Choquet integral exists, we study the complete convergence theorem for negatively dependent random variables under sub-linear expectation space. Two general complete convergence theorems under sub-linear expectation space are obtained, where the coefficient of weighted sum is the general function. This paper not only extends the complete convergence theorem in the traditional probability space to the sub-linear expectation space, but also extends the coefficient of weighted sum as a general function.

Biclustering and boolean matrix factorization in data streams

We study clustering of bipartite graphs and Boolean matrix factorization in data streams. We consider a streaming setting in which the vertices from the left side of the graph arrive one by one together with all of their incident edges. We provide an algorithm which after one pass over the stream recovers the set of clusters on the right side of the graph using sublinear space; to the best of our knowledge this is the first algorithm with this property. We also show that after a second pass over the stream the left clusters of the bipartite graph can be recovered and we show how to extend our algorithm to solve the Boolean matrix factorization problem (by exploiting the correspondence of Boolean matrices and bipartite graphs). We evaluate an implementation of the algorithm on synthetic data and on real-world data. On real-world datasets the algorithm is orders of magnitudes faster than a static baseline algorithm while providing quality results within a factor 2 of the baseline algorithm. Our algorithm scales linearly in the number of edges in the graph. Finally, we analyze the algorithm theoretically and provide sufficient conditions under which the algorithm recovers a set of planted clusters under a standard random graph model.

Similarity Join and Similarity Self-Join Size Estimation in a Streaming Environment

We study the problem of similarity self-join and similarity join size estimation in a streaming setting where the goal is to estimate, in one scan of the input and with sublinear space in the input size, the number of record pairs that have a similarity within a given threshold. The problem has many applications in data cleaning and query plan generation, where the cost of a similarity join may be estimated before actually doing the join. On unary input where two records either match or don't match, the problem becomes join and self-join size estimation for which one-pass algorithms are readily available. Our work addresses the problem for d-ary input, for d >= 1, where the degree of similarity can vary from 1 to d. We show that our proposed algorithm gives an accurate estimate and scales well with the input size. We provide error bounds and time and space costs, and conduct an extensive experimental evaluation of our algorithm, comparing its estimation accuracy to a few competitors, including some multi-pass algorithms. Our results show that given the same space, the proposed algorithm has an order of magnitude less error for a large range of similarity thresholds.

The Impact of Big Data Analytics on Risk Management and Decision Making: International conference on Recent Trends in Artificial Intelligence, IOT, Smart Cities & Applications (ICAISC-2020)

An important area related to Risk Management and Decision Making is to undergo systemic risk modelling that estimates the interrelationships among different institutions, with the direction to establish the central tendency and focus on more related subjects. Bigdata Analytics is intricately intertwined with other important concepts which fabricate many new advanced opportunities which were speculative in the past. This makes a growing importance in the field of data driven risk management and decision making. Extracting and utilizing a huge amount of unstructured data is far from easy and eventually these data are linked to real time events. Due to time and space constraint, the data need to be analyzed in a fewer pass preferably in a single pass with sublinear space and time. This help in proper replanning and rescheduling of business activities. Bigdata Analytics would definitely make an impact in valuable knowledge gain for many organizations, which in turn benefited in Business Operations, Risk Management and Decision Making. The aim of the study is to propose a systemic risk model with the help of streaming model and statistical analysis which not only concern about the information contained in organizational or industrial overall activity but also to handle huge unstructured data collected from social media like tweets. This study focuses on both structural and methodological view point.