Linear Sketches Research Articles

Streaming Algorithms for Geometric Steiner Forest

We consider a generalization of the Steiner tree problem, the Steiner forest problem , in the Euclidean plane: the input is a multiset \(X\subseteq{\mathbb{R}}^{2}\) , partitioned into \(k\) color classes \(C_{1},\ldots,C_{k}\subseteq X\) . The goal is to find a minimum-cost Euclidean graph \(G\) such that every color class \(C_{i}\) is connected in \(G\) . We study this Steiner forest problem in the streaming setting, where the stream consists of insertions and deletions of points to \(X\) . Each input point \(x {\in} X\) arrives with its color \(\mathsf{color}(x) {\in} [k]\) , and as usual for dynamic geometric streams, the input is restricted to the discrete grid \(\{1,\ldots,\Delta\}^{2}\) . We design a single-pass streaming algorithm that uses \(\operatorname{poly}(k\cdot\log\Delta)\) space and time, and estimates the cost of an optimal Steiner forest solution within ratio arbitrarily close to the famous Euclidean Steiner ratio \(\alpha_{2}\) (currently \(1.1547\leq\alpha_{2}\leq 1.214\) ). This approximation guarantee matches the state-of-the-art bound for streaming Steiner tree, i.e., when \(k=1\) , and it is a major open question to improve the ratio to \(1+\varepsilon\) even for this special case. Our approach relies on a novel combination of streaming techniques, like sampling and linear sketching, with the classical Arora-style dynamic-programming framework for geometric optimization problems, which usually requires large memory and so far has not been applied in the streaming setting. We complement our streaming algorithm for the Steiner forest problem with simple arguments showing that any finite multiplicative approximation requires \(\Omega(k)\) bits of space.

GraphZeppelin : How to Find Connected Components (Even When Graphs Are Dense, Dynamic, and Massive)

Finding the connected components of a graph is a fundamental problem with uses throughout computer science and engineering. The task of computing connected components becomes more difficult when graphs are very large, or when they are dynamic, meaning the edge set changes over time subject to a stream of edge insertions and deletions. A natural approach to computing the connected components problem on a large, dynamic graph stream is to buy enough RAM to store the entire graph. However, the requirement that the graph fit in RAM is an inherent limitation of this approach and is prohibitive for very large graphs. Thus, there is an unmet need for systems that can process dense dynamic graphs, especially when those graphs are larger than available RAM. We present a new high-performance streaming graph-processing system for computing the connected components of a graph. This system, which we call GraphZeppelin , uses new linear sketching data structures ( CubeSketch ) to solve the streaming connected components problem and as a result requires space asymptotically smaller than the space required for a lossless representation of the graph. GraphZeppelin is optimized for massive dense graphs: GraphZeppelin can process millions of edge updates (both insertions and deletions) per second, even when the underlying graph is far too large to fit in available RAM. As a result GraphZeppelin vastly increases the scale of graphs that can be processed.

Sampling Methods for Inner Product Sketching

Recently, Bessa et al. (PODS 2023) showed that sketches based on coordinated weighted sampling theoretically and empirically outperform popular linear sketching methods like Johnson-Lindentrauss projection and CountSketch for the ubiquitous problem of inner product estimation. We further develop this finding by introducing and analyzing two alternative sampling-based methods. In contrast to the computationally expensive algorithm in Bessa et al., our methods run in linear time (to compute the sketch) and perform better in practice, significantly beating linear sketching on a variety of tasks. For example, they provide state-of-the-art results for estimating the correlation between columns in unjoined tables, a problem that we show how to reduce to inner product estimation in a black-box way. While based on known sampling techniques (threshold and priority sampling) we introduce significant new theoretical analysis to prove approximation guarantees for our methods.

Sketching Approximability of All Finite CSPs

A constraint satisfaction problem (CSP), \(\textsf {Max-CSP}(\mathcal {F})\) , is specified by a finite set of constraints \(\mathcal {F}\subseteq \lbrace [q]^k \rightarrow \lbrace 0,1\rbrace \rbrace\) for positive integers q and k . An instance of the problem on n variables is given by m applications of constraints from \(\mathcal {F}\) to subsequences of the n variables, and the goal is to find an assignment to the variables that satisfies the maximum number of constraints. In the (γ ,β)-approximation version of the problem for parameters 0 ≤ β ≤ γ ≤ 1, the goal is to distinguish instances where at least γ fraction of the constraints can be satisfied from instances where at most β fraction of the constraints can be satisfied. In this work, we consider the approximability of this problem in the context of sketching algorithms and give a dichotomy result. Specifically, for every family \(\mathcal {F}\) and every β < γ, we show that either a linear sketching algorithm solves the problem in polylogarithmic space or the problem is not solvable by any sketching algorithm in \(o(\sqrt {n})\) space. In particular, we give non-trivial approximation algorithms using polylogarithmic space for infinitely many constraint satisfaction problems. We also extend previously known lower bounds for general streaming algorithms to a wide variety of problems, and in particular the case of q = k =2, where we get a dichotomy, and the case when the satisfying assignments of the constraints of \(\mathcal {F}\) support a distribution on \([q]^k\) with uniform marginals. Prior to this work, other than sporadic examples, the only systematic classes of CSPs that were analyzed considered the setting of Boolean variables q = 2, binary constraints k =2, and singleton families \(|\mathcal {F}|=1\) and only considered the setting where constraints are placed on literals rather than variables. Our positive results show wide applicability of bias-based algorithms used previously by [ 47 ] and [ 41 ], which we extend to include richer norm estimation algorithms, by giving a systematic way to discover biases. Our negative results combine the Fourier analytic methods of [ 56 ], which we extend to a wider class of CSPs, with a rich collection of reductions among communication complexity problems that lie at the heart of the negative results. In particular, previous works used Fourier analysis over the Boolean cube to initiate their results and the results seemed particularly tailored to functions on Boolean literals (i.e., with negations). Our techniques surprisingly allow us to get to general q -ary CSPs without negations by appealing to the same Fourier analytic starting point over Boolean hypercubes.

Encoder Blind Combinatorial Compressed Sensing

In its most elementary form, compressed sensing studies the design of decoding algorithms to recover a sufficiently sparse vector or code from a lower dimensional linear measurement vector. Typically it is assumed that the decoder has access to the encoder matrix, which in the combinatorial case is sparse and binary. In this paper we consider the problem of designing a decoder to recover a set of sparse codes from their linear measurements alone, that is without access to encoder matrix. To this end we study the matrix factorisation task of recovering both the encoder and sparse coding matrices from the associated linear measurement matrix. The contribution of this paper is a computationally efficient decoding algorithm, Decoder-Expander Based Factorisation, with strong performance guarantees. Under mild assumptions on the sparse coding matrix and by deploying a novel random encoder matrix, we prove that Decoder-Expander Based Factorisation recovers both the encoder and sparse coding matrix at the optimal measurement rate with high probability and from a near optimal number of measurement vectors. In addition, our experiments demonstrate the efficacy and computational efficiency of our algorithm in practice. Beyond compressed sensing, our results may be of interest for researchers working in areas as diverse as linear sketching, coding theory, matrix compression and dictionary learning.

Two-Step Dimension Reduction Strategy for Real-Aperture Radar Fast Super-Resolution Imaging

For real aperture radar, its azimuth resolution is much coarser than the range resolution after pulse compression, super-resolution algorithms are desired to enhance its azimuth resolution. However, the super-resolution algorithms must require enough azimuth sampling to ensure its performance. When wide scanning scope or dense azimuth sampling, the amount of data will increase significantly, which brings large computational burden to super-resolution processing. To cover this problem, we propose a two-step dimension reduction strategy. Firstly, by using linear sketching technology, the high-dimensional matrices are projected to the low-dimensional space, thus accelerating the matrix-matrix multiplications in super-resolution algorithms. Secondly, exploiting Sherman-Morrison formula, we further realized the acceleration of the matrix inversion in super-resolution algorithms. The proposed two-step acceleration strategy in our work is applicable to the existing deconvolution super-resolution algorithms, including regularization methods, Bayesian methods. It can be verified by simulation and experimental data that the proposed accelerated algorithms have advantages in computing time without losing the quality of super-resolution imaging.

Correlation Clustering in Data Streams

Clustering is a fundamental tool for analyzing large data sets. A rich body of work has been devoted to designing data-stream algorithms for the relevant optimization problems such as k-center, k-median, and k-means. Such algorithms need to be both time and and space efficient. In this paper, we address the problem of correlation clustering in the dynamic data stream model. The stream consists of updates to the edge weights of a graph on n nodes and the goal is to find a node-partition such that the end-points of negative-weight edges are typically in different clusters whereas the end-points of positive-weight edges are typically in the same cluster. We present polynomial-time, O(ncdot {{,mathrm{polylog},}}n)-space approximation algorithms for natural problems that arise. We first develop data structures based on linear sketches that allow the “quality” of a given node-partition to be measured. We then combine these data structures with convex programming and sampling techniques to solve the relevant approximation problem. Unfortunately, the standard LP and SDP formulations are not obviously solvable in O(ncdot {{,mathrm{polylog},}}n)-space. Our work presents space-efficient algorithms for the convex programming required, as well as approaches to reduce the adaptivity of the sampling.

A Fast Sparse Azimuth Super-Resolution Imaging Method of Real Aperture Radar Based on Iterative Reweighted Least Squares With Linear Sketching

It is greatly significant to achieve radar forward-looking region imaging. Due to the limitation of phase ambiguity and small Doppler gradient in forward-looking region, synthetic aperture radar and Doppler beam sharpening cannot work for forward-looking imaging, while real aperture radar (RAR) has arbitrary imaging geometry. Nevertheless, restricted by the antenna aperture, azimuth resolution of RAR is coarse, super-resolution technology is required to improve its azimuth resolution. Exploiting the sparse prior information of the target, the super-resolution problem can be transformed into an $L_1$ norm minimization problem mathematically. Iterative reweighted algorithm can effectively solve the $L_1$ norm minimization problem by replacing $L_1$ norm with reweighted $L_2$ norm and computing the weight in each iteration. However, it suffers from a large computational load due to the repeated multiplications and inversions of large matrices. In this article, a fast azimuth super-resolution imaging method of RAR based on iterative reweighted least squares (IRLS) with linear sketching (LS) was proposed to achieve fast super-resolution imaging of RAR. The LS theory is employed to compress echo matrix and antenna measurement matrix into much smaller matrices via multiplying them by an embedded matrix. Then, the IRLS solver was utilized to address the reconstructed objective function. Much of the expensive computation can then be performed on the smaller matrices, thereby accelerating the algorithm. Simulations and experimental data prove that the proposed algorithm can offer a time complexity reduction without loss of imaging performance.

A Review of and Some Results for Ollivier–Ricci Network Curvature

Characterizing topological properties and anomalous behaviors of higher-dimensional topological spaces via notions of curvatures is by now quite common in mainstream physics and mathematics, and it is therefore natural to try to extend these notions from the non-network domains in a suitable way to the network science domain. In this article we discuss one such extension, namely Ollivier’s discretization of Ricci curvature. We first motivate, define and illustrate the Ollivier–Ricci Curvature. In the next section we provide some “not-previously-published” bounds on the exact and approximate computation of the curvature measure. In the penultimate section we review a method based on the linear sketching technique for efficient approximate computation of the Ollivier–Ricci network curvature. Finally in the last section we provide concluding remarks with pointers for further reading.

Gradient preconditioned mini-batch SGD for ridge regression

Data preconditioning technique, which reduces the condition number of the problem by a linear transformation of the data matrix, is typically used to accelerate the convergence of the first-order optimization methods for regularized loss minimization. One obvious limitation of the technique is exceedingly expensive of computational cost for the large-scale problems, especially an ocean of samples. In this paper, we have a gradient preconditioning trick and combine it with mini-batch SGD. The proposed gradient preconditioned mini-batch SGD algorithm boosts indeed the convergence with lower computational cost than that of the data preconditioning technique for ridge regression. Concretely, we use recent random projection and linear sketching methods to randomly low rank approximate the data matrix, then we can achieve a appropriate preconditioner through numerical linear algebra. Finally, we apply obtained preconditioner to the gradient to reduce computational cost. The experimental results on both synthetic data and real data sets validate the feasibility and effectiveness of our trick and algorithm.

Discrete optimization methods for group model selection in compressed sensing

In this article we study the problem of signal recovery for group models. More precisely for a given set of groups, each containing a small subset of indices, and for given linear sketches of the true signal vector which is known to be group-sparse in the sense that its support is contained in the union of a small number of these groups, we study algorithms which successfully recover the true signal just by the knowledge of its linear sketches. We derive model projection complexity results and algorithms for more general group models than the state-of-the-art. We consider two versions of the classical iterative hard thresholding algorithm (IHT). The classical version iteratively calculates the exact projection of a vector onto the group model, while the approximate version (AM-IHT) uses a head- and a tail-approximation iteratively. We apply both variants to group models and analyse the two cases where the sensing matrix is a Gaussian matrix and a model expander matrix. To solve the exact projection problem on the group model, which is known to be equivalent to the maximum weight coverage problem, we use discrete optimization methods based on dynamic programming and Benders’ decomposition. The head- and tail-approximations are derived by a classical greedy-method and LP-rounding, respectively.

Sketching for sequential change-point detection

We present sequential change-point detection procedures based on linear sketches of high-dimensional signal vectors using generalized likelihood ratio (GLR) statistics. The GLR statistics allow for an unknown post-change mean that represents an anomaly or novelty. We consider both fixed and time-varying projections, derive theoretical approximations to two fundamental performance metrics: the average run length (ARL) and the expected detection delay (EDD); these approximations are shown to be highly accurate by numerical simulations. We further characterize the relative performance measure of the sketching procedure compared to that without sketching and show that there can be little performance loss when the signal strength is sufficiently large, and enough number of sketches are used. Finally, we demonstrate the good performance of sketching procedures using simulation and real-data examples on solar flare detection and failure detection in power networks.

The one-way communication complexity of dynamic time warping distance

We resolve the randomized one-way communication complexity of Dynamic Time Warping (DTW) distance. We show that there is an efficient one-way communication protocol using $\widetilde{O}(n/\alpha)$ bits for the problem of computing an $\alpha$-approximation for DTW between strings $x$ and $y$ of length $n$, and we prove a lower bound of $\Omega(n / \alpha)$ bits for the same problem. Our communication protocol works for strings over an arbitrary metric of polynomial size and aspect ratio, and we optimize the logarithmic factors depending on properties of the underlying metric, such as when the points are low-dimensional integer vectors equipped with various metrics or have bounded doubling dimension. We also consider linear sketches of DTW, showing that such sketches must have size $\Omega(n)$.

Streaming Low-Rank Matrix Approximation with an Application to Scientific Simulation

This paper argues that randomized linear sketching is a natural tool for on-the-fly compression of data matrices that arise from large-scale scientific simulations and data collection. The technica...

Dimensionality reduction of SDPs through sketching

We show how to sketch semidefinite programs (SDPs) using positive maps in order to reduce their dimension. More precisely, we use Johnson–Lindenstrauss transforms to pro- duce a smaller SDP whose solution preserves feasibility or approximates the value of the original problem with high probability. These techniques allow to improve both complexity and storage space requirements. They apply to problems in which the Schatten 1-norm of the matrices specifying the SDP and also of a solution to the problem is constant in the problem size. Furthermore, we provide some results which clarify the limitations of positive, linear sketches in this setting.

On the Construction of Sparse Matrices From Expander Graphs

We revisit the asymptotic analysis of probabilistic construction of adjacency matrices of expander graphs proposed in Bah and Tanner [2013]. With better bounds we derived a new reduced sample complexity for d, the number of nonzeros per column of these matrices (or equivalently the left-degree of the underlying expander graph). Precisely d = O(log(N/s) /log(s)); as opposed to the standard d = O(\log(N/s)), where N is the number of columns of the matrix (also the cardinality of set of left vertices of the expander graph) or the ambient dimension of the signals that can be sensed by such matrices. This gives insights into why using such sensing matrices with small d performed well in numerical compressed sensing experiments. Furthermore, we derive quantitative sampling theorems for our constructions which show our construction outperforming the existing state-of-the-art. We also used our results to compare performance of sparse recovery algorithms where these matrices are used for linear sketching.

Access to Data and Number of Iterations

In this article, we consider graph algorithms in models of computation where the space usage (random accessible storage, in addition to the read-only input) is sublinear in the number of edges m and the access to input is constrained. These questions arise in many natural settings, and in particular in the analysis of streaming algorithms, MapReduce or similar algorithms, or message passing distributed computing that model constrained parallelism with sublinear central processing. We focus on weighted nonbipartite maximum matching in this article. For any constant p > 1, we provide an iterative sampling-based algorithm for computing a (1 − ε)-approximation of the weighted nonbipartite maximum matching that uses O ( p /ε) rounds of sampling, and O ( n 1+1/ p ) space. The results extend to b -Matching with small changes. This article combines adaptive sketching literature and fast primal-dual algorithms based on relaxed Dantzig-Wolfe decision procedures. Each round of sampling is implemented through linear sketches and can be executed in a single round of streaming or two rounds of MapReduce. The article also proves that nonstandard linear relaxations of a problem, in particular penalty-based formulations, are helpful in reducing the adaptive dependence of the iterations.

Rate Distortion for Lossy In-Network Linear Function Computation and Consensus: Distortion Accumulation and Sequential Reverse Water-Filling

We consider the problem of distributed lossy linear function computation in a tree network. We examine two cases: 1) data aggregation (only one sink node computes) and 2) consensus (all nodes compute the same function). By quantifying the accumulation of information loss in distributed computing, we obtain fundamental limits on network computation rate as a function of incremental distortions (and hence incremental loss of information) along the edges of the network. The above characterization, based on quantifying distortion accumulation, offers an improvement over classical cut-set type techniques, which are based on overall distortions instead of incremental distortions. This quantification of information loss qualitatively resembles information dissipation in cascaded channels [2] . Surprisingly, this accumulation effect of distortion happens even at infinite blocklength. Combining this observation with an inequality on the dominance of mean-square quantities over relative-entropy quantities, we obtain outer bounds on the rate distortion function that are tighter than classical cut-set bounds by a difference, which can be arbitrarily large in both data aggregation and consensus. We also obtain inner bounds on the optimal rate using random Gaussian coding, which differ from the outer bounds by $\mathcal {O}(\sqrt {D})$ , where $D$ is the overall distortion. The obtained inner and outer bounds can provide insights on rate (bit) allocations for both the data aggregation problem and the consensus problem. We show that for tree networks, the rate allocation results have a mathematical structure similar to classical reverse water-filling for parallel Gaussian sources. Apart from data aggregation and distributed consensus, the distortion accumulation analysis framework is also applicable in large-scale data summarization through histograms and linear sketching, e.g., word counting tasks for document summarization.

Bias-aware sketches

Linear sketching algorithms have been widely used for processing large-scale distributed and streaming datasets. Their popularity is largely due to the fact that linear sketches can be naturally composed in the distributed model and be efficiently updated in the streaming model. The errors of linear sketches are typically expressed in terms of the sum of coordinates of the input vector excluding those largest ones, or, the mass on the tail of the vector. Thus, the precondition for these algorithms to perform well is that the mass on the tail is small, which is, however, not always the case - in many real-world datasets the coordinates of the input vector have a bias , which will generate a large mass on the tail. In this paper we propose linear sketches that are bias- aware . We rigorously prove that they achieve strictly better error guarantees than the corresponding existing sketches, and demonstrate their practicality and superiority via an extensive experimental evaluation on both real and synthetic datasets.

On deterministic sketching and streaming for sparse recovery and norm estimation

We study classic streaming and sparse recovery problems using deterministic linear sketches, including ℓ1/ℓ1 and ℓ∞/ℓ1 sparse recovery problems (the latter also being known as ℓ1-heavy hitters), norm estimation, and approximate inner product. We focus on devising a fixed matrix A∈Rm×n and a deterministic recovery/estimation procedure which work for all possible input vectors simultaneously. Our results improve upon existing work, the following being our main contributions:• A proof that ℓ∞/ℓ1 sparse recovery and inner product estimation are equivalent, and that incoherent matrices can be used to solve both problems. Our upper bound for the number of measurements is m=O(ε-2min{logn,(logn/log(1/ε))2}), which holds for any 0<ε<1/2. We can also obtain fast sketching and recovery algorithms by making use of the Fast Johnson–Lindenstrauss transform. Both our running times and number of measurements improve upon previous work. We can also obtain better error guarantees than previous work in terms of a smaller tail of the input vector.• A new lower bound for the number of linear measurements required to solve ℓ1/ℓ1 sparse recovery. We show Ω(k/ε2+klog(n/k)/ε) measurements are required to recover an x′ with ‖x-x′‖1⩽(1+ε)‖xtail(k)‖1, where xtail(k) is x projected onto all but its largest k coordinates in magnitude.• A tight bound of m=Θ(ε-2log(ε2n)) on the number of measurements required to solve deterministic norm estimation, i.e., to recover ‖x‖2±ε‖x‖1.For all the problems we study, tight bounds are already known for the randomized complexity from previous work, except in the case of ℓ1/ℓ1 sparse recovery, where a nearly tight bound is known. Our work thus aims to study the deterministic complexities of these problems. We remark that some of the matrices used in our algorithms, although known to exist, currently are not yet explicit in the sense that deterministic polynomial time constructions are not yet known, although in all cases polynomial time Monte Carlo algorithms are known.