Column Subset Selection Research Articles

On the optimality of voronoi‐based column selection

AbstractWhile there exists a rich array of matrix column subset selection problem (CSSP) algorithms for use with interpolative and CUR‐type decompositions, their use can often become prohibitive as the size of the input matrix increases. In an effort to address these issues, in earlier work we developed a general framework that pairs a column‐partitioning routine with a column‐selection algorithm. Two of the four algorithms presented in that work paired the Centroidal Voronoi Orthogonal Decomposition (CVOD; Centroidal Voronoi Tessellation Based Proper Orthogonal Decomposition Analysis, 2003, 137–150) and an adaptive variant (adaptCVOD) with the Discrete Empirical Interpolation Method (DEIM; SIAM J. Sci. Computer. 38 (2016), no. 3, A1454–A1482). In this work, we extend this framework and pair the CVOD‐type algorithms with any CSSP algorithm that returns linearly independent columns. Our results include detailed error bounds for the solutions provided by these paired algorithms, as well as expressions that explicitly characterize how the quality of the selected column partition affects the resulting CSSP solution. In addition to examples involving matrix approximation, we test several of our partition‐based constructions on tasks commonly encountered in model order reduction (MOR).

Graph Clustering Methods Derived from Column Subset Selection (Student Abstract)

Spectral clustering is a powerful clustering technique. It leverages the spectral properties of graphs to partition data points into meaningful clusters. The most common criterion for evaluating multi-way spectral clustering is NCut. Column Subset Selection is an important optimization technique in the domain of feature selection and dimension reduction which aims to identify a subset of columns of a given data matrix that can be used to approximate the entire matrix. We show that column subset selection can be used to compute spectral clustering and use this to obtain new graph clustering algorithms.

Equivalence between Graph Spectral Clustering and Column Subset Selection (Student Abstract)

The common criteria for evaluating spectral clustering are NCut and RatioCut. The seemingly unrelated column subset selection (CSS) problem aims to compute a column subset that linearly approximates the entire matrix. A common criterion is the approximation error in the Frobenius norm (ApproxErr). We show that any algorithm for CSS can be viewed as a clustering algorithm that minimizes NCut by applying it to a matrix formed from graph edges. Conversely, any clustering algorithm can be seen as identifying a column subset from that matrix. In both cases, ApproxErr and NCut have the same value. Analogous results hold for RatioCut with a slightly different matrix. Therefore, established results for CSS can be mapped to spectral clustering. We use this to obtain new clustering algorithms, including an optimal one that is similar to A*. This is the first nontrivial clustering algorithm with such an optimality guarantee. A variant of the weighted A* runs much faster and provides bounds on the accuracy. Finally, we use the results from spectral clustering to prove the NP-hardness of CSS from sparse matrices.

Interlacing Polynomial Method for the Column Subset Selection Problem

Abstract This paper investigates the spectral norm version of the column subset selection problem. Given a matrix $\textbf{A}\in \mathbb{R}^{n\times d}$ and a positive integer $k\leq \textrm{rank}(\textbf{A})$, the objective is to select exactly $k$ columns of $\textbf{A}$ that minimize the spectral norm of the residual matrix after projecting $\textbf{A}$ onto the space spanned by the selected columns. We use the method of interlacing polynomials introduced by Marcus–Spielman–Srivastava to derive a new upper bound on the minimal approximation error. This new bound is asymptotically sharp when the matrix $\textbf{A}\in \mathbb{R}^{n\times d}$ obeys a spectral power-law decay. The relevant expected characteristic polynomial is a variation of the expected polynomial for the restricted invertibility problem, incorporating two extra variable substitution operators. Finally, we propose a deterministic polynomial-time algorithm that achieves this error bound up to a computational error.

Data selection and data-enabled predictive control for a fuel cell system

Data-enabled predictive control is applied for power tracking of a fuel cell system and algorithmic aspects are investigated. For the system realization, the column subset selection algorithms norm sampling, iterative norm sampling, leverage score sampling, and selection based on the strong rank-revealing QR factorization are compared. Regularization using the 1-norm and the squared 2-norm on the column combination vector g is evaluated in terms of runtime and closed-loop results. The results indicate that column selection based on a strong rank-revealing QR (sRRQR) decomposition and leverage score sampling lead to consistently small closed-loop costs even with comparatively few columns in the data matrix resulting in reduced computation time. The closed-loop performance with 1-norm and squared 2-norm is similar, with lower turnaround times for the latter.

Pylspack : Parallel Algorithms and Data Structures for Sketching, Column Subset Selection, Regression, and Leverage Scores

We present parallel algorithms and data structures for three fundamental operations in Numerical Linear Algebra: (i) Gaussian and CountSketch random projections and their combination, (ii) computation of the Gram matrix, and (iii) computation of the squared row norms of the product of two matrices, with a special focus on “tall-and-skinny” matrices, which arise in many applications. We provide a detailed analysis of the ubiquitous CountSketch transform and its combination with Gaussian random projections, accounting for memory requirements, computational complexity and workload balancing. We also demonstrate how these results can be applied to column subset selection, least squares regression and leverage scores computation. These tools have been implemented in pylspack , a publicly available Python package 1 whose core is written in C++ and parallelized with OpenMP and that is compatible with standard matrix data structures of SciPy and NumPy. Extensive numerical experiments indicate that the proposed algorithms scale well and significantly outperform existing libraries for tall-and-skinny matrices.

Spectrum Pursuit With Residual Descent for Column Subset Selection Problem: Theoretical Guarantees and Applications in Deep Learning

We propose a novel technique for dataset summarization by selecting representatives from a large, unsupervised dataset. The approach is based on the concept of self-rank, defined as the minimum number of samples needed to express all dataset samples with an accuracy proportional to the rank-<i>K</i> approximation. As the exact computation of self-rank requires a computationally expensive combinatorial search, we propose an efficient algorithm that jointly estimates self-rank and selects the most informative samples in a linear order of complexity w.r.t the data size. We derive a new upper bound for the approximation ratio (AR), the ratio of obtained projection error using selected samples to the best rank-<i>K</i> approximation error. The best previously known AR for self-representative low-rank approximation was presented in ICML 2017 [1], which was further improved by the bound &#x221A;1 + <i>K</i> reported in NeurIPS 2019 [2]. Both of these bounds are obtained by brute force search, which is not practical, and these bounds depend solely on <i>K</i>, the number of selected samples. In contrast, we describe an adaptive AR that takes into consideration the spectral properties and spikiness measure of the original dataset, A &#x03F5; R^{<i>N</i>&#x00D7;<i>M</i>}. In particular, our performance bound is proportional to the condition number &#x03BA;(<i>A</i>). Our derived AR is expressed as 1+(&#x03BA;(<i>A</i>)² - 1)/(<i>N</i> - <i>K</i>), which approaches 1 and is optimal in two extreme spectral distribution instances. In the worst case, AR is shown not to exceed 1.25 for the proposed method. Our proposed algorithm enjoys linear complexity w.r.t. size of original dataset, which results in filling a historical gap between practical and theoretical methods in finding representatives. In addition to evaluating the proposed algorithm on a synthetic dataset, we show that it can be utilized in real-world applications such as graph node sampling for optimizing the shortest path criterion, learning a classifier with representative data, and open-set identification.

Relevant and Non-Redundant Feature Selection for Cancer Classification and Subtype Detection.

Simple SummaryHere we introduce a new feature selection algorithm DTA, which selects important, non-redundant, and relevant features from diverse omics data. DTA selects non-redundant features by maximizing the similarity between each patient pair by an approximate k-cover algorithm. We successfully applied this algorithm to three different biological problems: (a) disease to healthy sample classification, (b) multiclass classification of different disease samples, and (c) disease subtypes detection. DTA outperformed other feature selection techniques in the binary classification of healthy and disease samples and multiclass classification of various diseases. It also improved the performance of a subtype detection algorithm by selecting the important features for few cancer types.Biologists seek to identify a small number of significant features that are important, non-redundant, and relevant from diverse omics data. For example, statistical methods such as LIMMA and DEseq distinguish differentially expressed genes between a case and control group from the transcript profile. Researchers also apply various column subset selection algorithms on genomics datasets for a similar purpose. Unfortunately, genes selected by such statistical or machine learning methods are often highly co-regulated, making their performance inconsistent. Here, we introduce a novel feature selection algorithm that selects highly disease-related and non-redundant features from a diverse set of omics datasets. We successfully applied this algorithm to three different biological problems: (a) disease-to-normal sample classification; (b) multiclass classification of different disease samples; and (c) disease subtypes detection. Considering the classification of ROC-AUC, false-positive, and false-negative rates, our algorithm outperformed other gene selection and differential expression (DE) methods for all six types of cancer datasets from TCGA considered here for binary and multiclass classification problems. Moreover, genes picked by our algorithm improved the disease subtyping accuracy for four different cancer types over state-of-the-art methods. Hence, we posit that our proposed feature reduction method can support the community to solve various problems, including the selection of disease-specific biomarkers, precision medicine design, and disease sub-type detection.

Surrogate model\u2013based inverse parameter estimation in deep drawing using automatic knowledge acquisition

In this paper, we propose a new approach for the simulation-based support of tryout operations in deep drawing which can be schematically classified as automatic knowledge acquisition. The central idea is to identify information maximising sensor positions for draw-in as well as local blank holder force sensors by solving the column subset selection problem with respect to the sensor sensitivities. Inverse surrogate models are then trained using the selected sensor signals as predictors and the material and process parameters as targets. The final models are able to observe the drawing process by estimating current material and process parameters, which can then be compared to the target values to identify process corrections. The methodology is examined on an Audi A8L side panel frame using a set of 635 simulations, where 20 out of 21 material and process parameters can be estimated with an R2 value greater than 0.9. The result shows that the observational models are not only capable of estimating all but one process parameters with high accuracy, but also allow the determination of material parameters at the same time. Since no assumptions are made about the type of process, sensors, material or process parameters, the methodology proposed can also be applied to other manufacturing processes and use cases.

A Distributed Integrated Feature Selection Scheme for Column Subset Selection

Most of the existing distributed feature selection schemes neglect how good the subsets are that are mapped to the computational nodes, which causes a waste of time and hardware resources. A distributed integrated feature selection scheme (DIFS) with Subset Quality Evaluation (SQE) is proposed. SQE studies the relevance between the quality of a subset and the number of selected features from this subset, which helps shorten the feature selection time efficiently. We have given the implementation of our scheme for the Column Subset Selection (CSS) problem. We integrate a CSS algorithm in DIFS and information entropy as the SQE metric. We prove that the speedup of DIFS can reach m^3 compared to the centralized algorithm in ideal situations where m is the number of computational nodes, and give a well bounded approximation guarantee of the solution for CSS problem. Extensive experiments on eight data sets are used to verify the performance of scheme. Experiments results demonstrate the effectiveness of SQE and the impressive speedup DIFS can achieve. Although there is a slight increase of the reconstruction error value in some situations. Additional experiments of classification tasks reveal that the performance of DIFS is better than existing state-of-the-art distributed algorithms.

Estimating Leverage Scores via Rank Revealing Methods and Randomization

We study algorithms for estimating the statistical leverage scores of rectangular dense or sparse matrices of arbitrary rank. Our approach is based on combining rank revealing methods with compositions of dense and sparse randomized dimensionality reduction transforms. We first develop a set of fast novel algorithms for rank estimation, column subset selection and least squares preconditioning. We then describe the design and implementation of leverage score estimators based on these primitives. These estimators are also effective for rank deficient input, which is frequently the case in data analytics applications. We provide detailed complexity analyses for all algorithms as well as meaningful approximation bounds and comparisons with the state-of-the-art. We conduct extensive numerical experiments to evaluate our algorithms and to illustrate their properties and performance using synthetic and real world data sets.

Column subset selection is NP-complete

Let M be a real r×c matrix, and let k be a positive integer. In the column subset selection problem (CSSP), we need to minimize the quantity ‖M−SA‖, where A can be an arbitrary k×c matrix, and S runs over all r×k submatrices of M. This problem and its applications in numerical linear algebra are being discussed for several decades, but its algorithmic complexity remained an open issue. We show that CSSP is NP-complete.

Unsupervised Feature Selection by Pareto Optimization

Dimensionality reduction is often employed to deal with the data with a huge number of features, which can be generally divided into two categories: feature transformation and feature selection. Due to the interpretability, the efficiency during inference and the abundance of unlabeled data, unsupervised feature selection has attracted much attention. In this paper, we consider its natural formulation, column subset selection (CSS), which is to minimize the reconstruction error of a data matrix by selecting a subset of features. We propose an anytime randomized iterative approach POCSS, which minimizes the reconstruction error and the number of selected features simultaneously. Its approximation guarantee is well bounded. Empirical results exhibit the superior performance of POCSS over the state-of-the-art algorithms.

On a new method for controlling the entire spectrum in the problem of column subset selection

The problem of extracting a well conditioned submatrix from any rectangular matrix (with normalized columns) has been studied for some time in functional and harmonic analysis. In the seminal work of Bourgain and Tzafriri and many subsequent improvements, methods using random column selection were considered. Constructive approaches have been proposed lately, mainly sparked by the work of Batson, Spielman and Srivastava. The column selection problem we consider in this paper is concerned with extracting a well conditioned submatrix, and more precisely, a matrix with all its singular values being contained in the interval [1−ε,1+ε]. Such results are known to have far reaching connections with many fields in mathematics and engineering.Our main contribution is a new deterministic method that achieves the same order R for the number of selected columns as in Bourgain and Tzafriri’s original Theorem, up to a log(R) multiplicative factor. Our analysis is elementary and shows how a simple eigenvalue perturbation argument can lead to an intuitive and very short proof. We also obtain individual lower and upper bounds for each singular value of the extracted matrix.

Sampling and multilevel coarsening algorithms for fast matrix approximations

SummaryThis paper addresses matrix approximation problems for matrices that are large, sparse, and/or representations of large graphs. To tackle these problems, we consider algorithms that are based primarily on coarsening techniques, possibly combined with random sampling. A multilevel coarsening technique is proposed, which utilizes a hypergraph associated with the data matrix and a graph coarsening strategy based on column matching. We consider a number of standard applications of this technique as well as a few new ones. Among standard applications, we first consider the problem of computing partial singular value decomposition, for which a combination of sampling and coarsening yields significantly improved singular value decomposition results relative to sampling alone. We also consider the column subset selection problem, a popular low‐rank approximation method used in data‐related applications, and show how multilevel coarsening can be adapted for this problem. Similarly, we consider the problem of graph sparsification and show how coarsening techniques can be employed to solve it. We also establish theoretical results that characterize the approximation error obtained and the quality of the dimension reduction achieved by a coarsening step, when a proper column matching strategy is employed. Numerical experiments illustrate the performances of the methods in a few applications.

Regularized greedy column subset selection

The Column Subset Selection Problem is a hard combinatorial optimization problem that provides a natural framework for unsupervised feature selection, and there exist efficient algorithms that provide good approximations. The drawback of the problem formulation is that it incorporates no form of regularization, and is therefore very sensitive to noise when presented with scarce data. In this paper we propose a regularized formulation of this problem, and derive a correct greedy algorithm that is similar in efficiency to existing greedy methods for the unregularized problem. We study its adequacy for feature selection and propose suitable formulations. Additionally, we derive a lower bound for the error of the proposed problems. Through various numerical experiments on real and synthetic data, we demonstrate the significantly increased robustness and stability of our method, as well as the improved conditioning of its output, all while remaining efficient for practical use.

Deterministic column subset selection for single-cell RNA-Seq.

Analysis of single-cell RNA sequencing (scRNA-Seq) data often involves filtering out uninteresting or poorly measured genes and dimensionality reduction to reduce noise and simplify data visualization. However, techniques such as principal components analysis (PCA) fail to preserve non-negativity and sparsity structures present in the original matrices, and the coordinates of projected cells are not easily interpretable. Commonly used thresholding methods to filter genes avoid those pitfalls, but ignore collinearity and covariance in the original matrix. We show that a deterministic column subset selection (DCSS) method possesses many of the favorable properties of common thresholding methods and PCA, while avoiding pitfalls from both. We derive new spectral bounds for DCSS. We apply DCSS to two measures of gene expression from two scRNA-Seq experiments with different clustering workflows, and compare to three thresholding methods. In each case study, the clusters based on the small subset of the complete gene expression profile selected by DCSS are similar to clusters produced from the full set. The resulting clusters are informative for cell type.

Unsupervised Band Selection Method Based on Importance-Assisted Column Subset Selection

Band selection is an important preprocessing technique for hyperspectral images to select a band subset with representative information and low correlation. However, most methods focus on removing redundant components without loss of original information, but not distinguishing the noisy and low-discriminating bands which must be manually removed in advance. To find high-discriminating and high-quality bands from the original hyperspectral cube, we propose an importance-assisted column subset band selection (iCSBS) method. First, an active gradient-reference (AGR) index based on iterative reference gradient map is designed to evaluate the importance of each band. Then, the AGR index is incorporated into a column subset selection method to select high-discriminating bands, via simultaneously minimizing the redundancy and maximizing the quality of the selected band subset. Furthermore, as the high dimensionality decreases the contrast between bands, we use Manhattan distance instead of Euclidean distance. The experimental results on three real-world hyperspectral images demonstrate that the proposed method can achieve higher classification accuracy than other state-of-the-art comparison methods, and is especially superior to the geometry-based methods.

Solving Generalized Column Subset Selection With Heuristic Search

We address the problem of approximating a matrix by the linear combination of a column sparse matrix and a low rank matrix. Two variants of a heuristic search algorithm are described. The first produces an optimal solution but may be slow, as these problems are believed to be NP-hard. The second is much faster, but only guarantees a suboptimal solution. The quality of the approximation and the optimality criterion can be specified in terms of unitarily invariant norms.

A salient dictionary learning framework for activity video summarization via key-frame extraction

Recently, dictionary learning methods for unsupervised video summarization have surpassed traditional video frame clustering approaches. This paper addresses static summarization of videos depicting activities, which possess certain recurrent properties. In this context, a flexible definition of an activity video summary is proposed, as the set of key-frames that can both reconstruct the original, full-length video and simultaneously represent its most salient parts. Both objectives can be jointly optimized across several information modalities. The two criteria are merged into a “salient dictionary” learning task that is proposed as a strict definition of the video summarization problem, encapsulating many existing algorithms. Three specific, novel video summarization methods are derived from this definition: the Numerical, the Greedy and the Genetic Algorithm. In all formulations, the reconstruction term is modeled algebraically as a Column Subset Selection Problem (CSSP), while the saliency term is modeled as an outlier detection problem, a low-rank approximation problem, or a summary dispersion maximization problem. In quantitative evaluation, the Greedy Algorithm seems to provide the best balance between speed and overall performance, with the faster Numerical Algorithm a close second. All the proposed methods outperform a baseline clustering approach and two competing state-of-the-art static video summarization algorithms.