Leverage Scores Research Articles

Adaptive sampling with tensor leverage scores for exact low-rank third-order tensor completion

Tensor completion aims at estimating the missing entries from the incomplete observation. Under the tensor singular value decomposition framework, the exact recovery of a low-tubal-rank third-order tensor could be achieved via convex optimization with high probability if the tensor satisfies the tensor incoherence condition. In this work, we show that, when the random selection of entries is made adaptive to a distribution which is dependent on the coherence structure of the tensor, any low-tubal-rank tensor of the size n×n×n with tubal-rank r can be exactly recovered with high probability from as few as O(rn2log2⁡(n)) randomly chosen entries. In practice, tensor leverage scores are not known a priori, and we design a two-phase adaptive sampling strategy to obtain the leverage scores. Numerical experiments on synthetic and real-world third-order tensor data sets are used to validate our theoretical results and illustrate that the tensor recovery performance of the proposed two-phase adaptive sampling scheme is better than that of the other state-of-the-art methods.

Gradient Coding With Iterative Block Leverage Score Sampling

Gradient Coding With Iterative Block Leverage Score Sampling

Leveraged Matrix Completion With Noise.

Completing low-rank matrices from subsampled measurements has received much attention in the past decade. Existing works indicate that O(nrlog2(n)) datums are required to theoretically secure the completion of an n ×n noisy matrix of rank r with high probability, under some quite restrictive assumptions: 1) the underlying matrix must be incoherent and 2) observations follow the uniform distribution. The restrictiveness is partially due to ignoring the roles of the leverage score and the oracle information of each element. In this article, we employ the leverage scores to characterize the importance of each element and significantly relax assumptions to: 1) not any other structure assumptions are imposed on the underlying low-rank matrix and 2) elements being observed are appropriately dependent on their importance via the leverage score. Under these assumptions, instead of uniform sampling, we devise an ununiform/biased sampling procedure that can reveal the "importance" of each observed element. Our proofs are supported by a novel approach that phrases sufficient optimality conditions based on the Golfing scheme, which would be of independent interest to the wider areas. Theoretical findings show that we can provably recover an unknown n×n matrix of rank r from just about O(nrlog2 (n)) entries, even when the observed entries are corrupted with a small amount of noisy information. The empirical results align precisely with our theories.

Randomized greedy magic point selection schemes for nonlinear model reduction

An established way to tackle model nonlinearities in projection-based model reduction is via relying on partial information. This idea is shared by the methods of gappy proper orthogonal decomposition (POD), missing point estimation (MPE), masked projection, hyper reduction, and the (discrete) empirical interpolation method (DEIM). The selected indices of the partial information components are often referred to as “magic points.” The original contribution of the work at hand is a novel randomized greedy magic point selection. It is known that the greedy method is associated with minimizing the norm of an oblique projection operator, which, in turn, is associated with solving a sequence of rank-one SVD update problems. We propose simplification measures so that the resulting greedy point selection has the following main features: (1) The inherent rank-one SVD update problem is tackled in a way, such that its dimension does not grow with the number of selected magic points. (2) The approach is online efficient in the sense that the computational costs are independent from the dimension of the full-scale model. To the best of our knowledge, this is the first greedy magic point selection that features this property. We illustrate the findings by means of numerical examples. We find that the computational cost of the proposed method is orders of magnitude lower than that of its deterministic counterpart. Nevertheless, the prediction accuracy is just as good if not better. When compared to a state-of-the-art randomized method based on leverage scores, the randomized greedy method outperforms its competitor.

Minimum volume covering ellipsoids

We present a new initialisation of an adaptive batch strategy to compute the ε-approximate minimum volume covering ellipsoid (MVCE) for a set of n points. We focus on moderately sized datasets (up to dimension d = 100 and n = 1 000 000). The adaptive batch strategy works in an optimisation-deletion-adaptation cycle: we solve the MVCE problem using a smaller number of points, we delete points from consideration that are guaranteed to not lie on the boundary of the MVCE, and then carefully select a new batch of points. We propose a new initialisation, which involves selecting the points corresponding to some highest leverage scores. We show using numerical examples that this new initialisation tends to improve computation time as well as reduce the total number of cycles, as compared with initialising with a random selection of points.

A Provably Accurate Randomized Sampling Algorithm for Logistic Regression

In statistics and machine learning, logistic regression is a widely-used supervised learning technique primarily employed for binary classification tasks. When the number of observations greatly exceeds the number of predictor variables, we present a simple, randomized sampling-based algorithm for logistic regression problem that guarantees high-quality approximations to both the estimated probabilities and the overall discrepancy of the model. Our analysis builds upon two simple structural conditions that boil down to randomized matrix multiplication, a fundamental and well-understood primitive of randomized numerical linear algebra. We analyze the properties of estimated probabilities of logistic regression when leverage scores are used to sample observations, and prove that accurate approximations can be achieved with a sample whose size is much smaller than the total number of observations. To further validate our theoretical findings, we conduct comprehensive empirical evaluations. Overall, our work sheds light on the potential of using randomized sampling approaches to efficiently approximate the estimated probabilities in logistic regression, offering a practical and computationally efficient solution for large-scale datasets.

Faster free pseudoinverse greedy block Kaczmarz method for image recovery

<abstract> <p>The greedy block Kaczmarz (GBK) method has been successfully applied in areas such as data mining, image reconstruction, and large-scale image restoration. However, the computation of pseudo-inverses in each iterative step of the GBK method not only complicates the computation and slows down the convergence rate, but it is also ill-suited for distributed implementation. The leverage score sampling free pseudo-inverse GBK algorithm proposed in this paper demonstrated significant potential in the field of image reconstruction. By ingeniously transforming the problem framework, the algorithm not only enhanced the efficiency of processing systems of linear equations with multiple solution vectors but also optimized specifically for applications in image reconstruction. A methodology that combined theoretical and experimental approaches has validated the robustness and practicality of the algorithm, providing valuable insights for technical advancements in related disciplines.</p> </abstract>

A Fast and Scalable Computational Framework for Large-Scale High-Dimensional Bayesian Optimal Experimental Design

We develop a fast and scalable computational framework to solve Bayesian optimal experimental design problems governed by partial differential equations (PDEs) with application to optimal sensor placement by maximizing expected information gain (EIG). Such problems are particularly challenging due to the curse of dimensionality for high-dimensional parameters and the expensive solution of large-scale PDEs. To address these challenges, we exploit two fundamental properties: (1) the low-rank structure of the Jacobian of the parameter-to-observable map, to extract the intrinsically low-dimensional data-informed subspace, and (2) a series of approximations of the EIG that reduce the number of PDE solves while retaining high correlation with the true EIG. Based on these properties, we propose an efficient offline-online decomposition for the optimization problem. The offline stage dominates the cost and entails precomputing all components that require PDE solves. The online stage optimizes sensor placement and does not require any PDE solves. For the online stage, we propose a new greedy algorithm that first places an initial set of sensors using leverage scores and then swaps the selected sensors with other candidates until certain convergence criteria are met, which we call a swapping greedy algorithm. We demonstrate the efficiency and scalability of the proposed method by both linear and nonlinear inverse problems. In particular, we show that the number of required PDE solves is small, independent of the parameter dimension, and only weakly dependent on the data dimension for both problems.

Efficient Least-Squares State Estimation Using Uniform Sampling

Formulating state estimation for a large-scale, discrete-time, linear time-invariant system as a least-squares problem can be computationally challenging as the problem dimensions increase with time. Recently, randomized sampling has demonstrated promising results in approximating this problem by using fewer rows, resulting in a polynomial-sized approximate problem. However, these algorithms necessitate calculating the statistical leverage scores of the rows, which can be challenging. In this paper, we propose an alternative approach to approximate leverage scores by uniformly sampling rows. We demonstrate that this method provides a sufficiently weak form of approximation for obtaining an estimate of each leverage score. Our approach delivers a reasonable approximation of the leverage scores, which is suitable for approximating and solving the least-squares estimation problem with proven theoretical guarantees.

Data selection and data-enabled predictive control for a fuel cell system

Data-enabled predictive control is applied for power tracking of a fuel cell system and algorithmic aspects are investigated. For the system realization, the column subset selection algorithms norm sampling, iterative norm sampling, leverage score sampling, and selection based on the strong rank-revealing QR factorization are compared. Regularization using the 1-norm and the squared 2-norm on the column combination vector g is evaluated in terms of runtime and closed-loop results. The results indicate that column selection based on a strong rank-revealing QR (sRRQR) decomposition and leverage score sampling lead to consistently small closed-loop costs even with comparatively few columns in the data matrix resulting in reduced computation time. The closed-loop performance with 1-norm and squared 2-norm is similar, with lower turnaround times for the latter.

Pylspack : Parallel Algorithms and Data Structures for Sketching, Column Subset Selection, Regression, and Leverage Scores

We present parallel algorithms and data structures for three fundamental operations in Numerical Linear Algebra: (i) Gaussian and CountSketch random projections and their combination, (ii) computation of the Gram matrix, and (iii) computation of the squared row norms of the product of two matrices, with a special focus on “tall-and-skinny” matrices, which arise in many applications. We provide a detailed analysis of the ubiquitous CountSketch transform and its combination with Gaussian random projections, accounting for memory requirements, computational complexity and workload balancing. We also demonstrate how these results can be applied to column subset selection, least squares regression and leverage scores computation. These tools have been implemented in pylspack , a publicly available Python package 1 whose core is written in C++ and parallelized with OpenMP and that is compatible with standard matrix data structures of SciPy and NumPy. Extensive numerical experiments indicate that the proposed algorithms scale well and significantly outperform existing libraries for tall-and-skinny matrices.

Practical Leverage-Based Sampling for Low-Rank Tensor Decomposition

The low-rank canonical polyadic tensor decomposition is useful in data analysis and can be computed by solving a sequence of overdetermined least squares subproblems. Motivated by consideration of sparse tensors, we propose sketching each subproblem using leverage scores to select a subset of the rows, with probabilistic guarantees on the solution accuracy. We randomly sample rows proportional to leverage score upper bounds that can be efficiently computed using the special Khatri--Rao subproblem structure inherent in tensor decomposition. Crucially, for a $(d+1)$-way tensor, the number of rows in the sketched system is $O(r^d/\epsilon)$ for a decomposition of rank $r$ and $\epsilon$-accuracy in the least squares solve, independent of both the size and the number of nonzeros in the tensor. Along the way, we provide a practical solution to the generic matrix sketching problem of sampling overabundance for high-leverage-score rows, proposing to include such rows deterministically and combine repeated samples in the sketched system; we conjecture that this can lead to improved theoretical bounds. Numerical results on real-world large-scale tensors show the method is significantly faster than deterministic methods at nearly the same level of accuracy.

Exact minimax risk for linear least squares, and the lower tail of sample covariance matrices

We consider random-design linear prediction and related questions on the lower tail of random matrices. It is known that, under boundedness constraints, the minimax risk is of order d/n in dimension d with n samples. Here, we study the minimax expected excess risk over the full linear class, depending on the distribution of covariates. First, the least squares estimator is exactly minimax optimal in the well-specified case, for every distribution of covariates. We express the minimax risk in terms of the distribution of statistical leverage scores of individual samples, and deduce a minimax lower bound of d/(n−d+1) for any covariate distribution, nearly matching the risk for Gaussian design. We then obtain sharp nonasymptotic upper bounds for covariates that satisfy a “small ball”-type regularity condition in both well-specified and misspecified cases. Our main technical contribution is the study of the lower tail of the smallest singular value of empirical covariance matrices at small values. We establish a lower bound on this lower tail, valid for any distribution in dimension d≥2, together with a matching upper bound under a necessary regularity condition. Our proof relies on the PAC-Bayes technique for controlling empirical processes, and extends an analysis of Oliveira devoted to a different part of the lower tail.

Non-uniform Nyström approximation for sparse kernel regression: Theoretical analysis and experimental evaluation

Solving a kernel regression problem usually suffers from expensive computation and storage costs due to the large kernel size. To tackle this problem, the Nyström method is proposed and widely applied to large-scale kernel methods as an approximate solution. The key idea of this method is to select a subset of columns of the kernel matrix and rebuilds a low-rank approximation to the dense kernel matrix. To reduce computational costs of sparse kernel regression, we take the merits of the Nyström approximation and present two non-uniform Nyström methods with theoretical guarantees for sparse kernel regression in this paper. In detail, we first provide an upper bound to the solution of sparse kernel regression via Nyström approximation. Based on this bound, we prove the upper bounds of the optimal solutions when adopting two notable non-uniform landmark selection strategies, including Determinantal Point Processes (DPPs) and Ridge Leverage Scores (RLS). Compared with the uniform Nyström method, we empirically demonstrate the superior performance of non-uniform Nyström in sparse kernel regression on a synthetic dataset and several real-world datasets.

Passive sampling in reproducing kernel Hilbert spaces using leverage scores

This paper deals with the selection of the training dataset in kernel-based methods for function reconstruction, with a focus on kernel ridge regression. A functional analysis is performed which, in the absence of noise, links the optimal sampling distribution to the one minimizing the difference between the kernel matrix and its low-rank Nyström approximation. From this standpoint, a statistical passive sampling approach is derived which uses the leverage scores of the columns of the kernel matrix to design a sampling distribution that minimizes an upper bound of the risk function. The proposed approach constitutes a passive method, able to select the optimal subset of training samples using only information provided by the input set and the kernel, but without needing to know the values of the function to be approximated. Furthermore, the proposed approach is backed up by numerical tests on real datasets.

Nyström landmark sampling and regularized Christoffel functions

Selecting diverse and important items, called landmarks, from a large set is a problem of interest in machine learning. As a specific example, in order to deal with large training sets, kernel methods often rely on low rank matrix Nyström approximations based on the selection or sampling of landmarks. In this context, we propose a deterministic and a randomized adaptive algorithm for selecting landmark points within a training data set. These landmarks are related to the minima of a sequence of kernelized Christoffel functions. Beyond the known connection between Christoffel functions and leverage scores, a connection of our method with finite determinantal point processes (DPPs) is also explained. Namely, our construction promotes diversity among important landmark points in a way similar to DPPs. Also, we explain how our randomized adaptive algorithm can influence the accuracy of Kernel Ridge Regression.

Detection and Identification of Cyber and Physical Attacks on Distribution Power Grids with PVs: An Online High-Dimensional Data-driven Approach.

Cyber and physical attacks threaten the security of distribution power grids. The emerging renewable energy sources such as photovoltaics (PVs) introduce new potential vulnerabilities. Based on the electric waveform data measured by waveform sensors in the distribution power networks, in this paper, we propose a novel high-dimensional data-driven cyber physical attack detection and identification approach (HCADI). Firstly, we analyze the cyber and physical attack impacts (including cyber attacks on the solar inverter causing unusual harmonics) on electric waveforms in distribution power grids. Then, we construct a high dimensional streaming data feature matrix based on signal analysis of multiple sensors in the network. Next, we propose a novel mechanism including leverage score based attack detection and binary matrix factorization based attack diagnosis. By leveraging the data structure and binary coding, our HCADI approach does not need the training stage for both detection and the root cause diagnosis, which is needed for machine learning/deep learning-based methods. To the best of our knowledge, it is the first attempt to use raw electrical waveform data to detect and identify the power electronics cyber/physical attacks in distribution power grids with PVs.

Fast spectral analysis for approximate nearest neighbor search

In large-scale machine learning, of central interest is the problem of approximate nearest neighbor (ANN) search, where the goal is to query particular points that are close to a given object under certain metric. In this paper, we develop a novel data-driven ANN search algorithm where the data structure is learned by fast spectral technique based on s landmarks selected by approximate ridge leverage scores. We show that with overwhelming probability, our algorithm returns the \((1+\epsilon /4)\)-ANN for any approximation parameter \(\epsilon \in (0,1)\). A remarkable feature of our algorithm is that it is computationally efficient. Specifically, learning k-length hash codes requires \(O((s^3+ns^2)\log n)\) running time and \(O(d^2)\) extra space, and returning the \((1+\epsilon /4)\)-ANN of the query needs \(O(k\log n)\) running time. The experimental results on computer vision and natural language understanding tasks demonstrate the significant advantage of our algorithm compared to state-of-the-art methods.

A Model-free Variable Screening Method Based on Leverage Score

With rapid advances in information technology, massive datasets are collected in all fields of science, such as biology, chemistry, and social science. Useful or meaningful information is extracted from these data often through statistical learning or model fitting. In massive datasets, both sample size and number of predictors can be large, in which case conventional methods face computational challenges. Recently, an innovative and effective sampling scheme based on leverage scores via singular value decompositions has been proposed to select rows of a design matrix as a surrogate of the full data in linear regression. Analogously, variable screening can be viewed as selecting rows of the design matrix. However, effective variable selection along this line of thinking remains elusive. In this article, we bridge this gap to propose a weighted leverage variable screening method by using both the left and right singular vectors of the design matrix. We show theoretically and empirically that the predictors selected using our method can consistently include true predictors not only for linear models but also for complicated general index models. Extensive simulation studies show that the weighted leverage screening method is highly computationally efficient and effective. We also demonstrate its success in identifying carcinoma related genes using spatial transcriptome data.

Integrating Linguistics, Social Structure, and Geography to Model Genetic Diversity within India.

India represents an intricate tapestry of population substructure shaped by geography, language, culture, and social stratification. Although geography closely correlates with genetic structure in other parts of the world, the strict endogamy imposed by the Indian caste system and the large number of spoken languages add further levels of complexity to understand Indian population structure. To date, no study has attempted to model and evaluate how these factors have interacted to shape the patterns of genetic diversity within India. We merged all publicly available data from the Indian subcontinent into a data set of 891 individuals from 90 well-defined groups. Bringing together geography, genetics, and demographic factors, we developed Correlation Optimization of Genetics and Geodemographics to build a model that explains the observed population genetic substructure. We show that shared language along with social structure have been the most powerful forces in creating paths of gene flow in the subcontinent. Furthermore, we discover the ethnic groups that best capture the diverse genetic substructure using a ridge leverage score statistic. Integrating data from India with a data set of additional 1,323 individuals from 50 Eurasian populations, we find that Indo-European and Dravidian speakers of India show shared genetic drift with Europeans, whereas the Tibeto-Burman speaking tribal groups have maximum shared genetic drift with East Asians.