Low-rank Matrices Research Articles

Sketching Sparse Low-Rank Matrices With Near-Optimal Sample- and Time-Complexity Using Message Passing

We consider the problem of recovering an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> × <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> low-rank matrix with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> -sparse singular vectors from a small number of linear measurements (sketch). We propose a sketching scheme and an algorithm that can recover the singular vectors with high probability, with a sample complexity and running time that both depend only on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> and not on the ambient dimensions <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> . Our sketching operator, based on a scheme for compressed sensing by Li et al. [1] and Bakshi et al. [2], uses a combination of a sparse parity check matrix and a partial DFT matrix. Our main contribution is the design and analysis of a two-stage iterative algorithm which recovers the singular vectors by exploiting the simultaneously sparse and low-rank structure of the matrix. We derive a nonasymptotic bound on the probability of exact recovery, which holds for any <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> × <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> sparse, low-rank matrix. We also show how the scheme can be adapted to tackle matrices that are approximately sparse and low-rank. The theoretical results are validated by extensive numerical simulations and comparisons with existing schemes that use convex optimization for recovery.

Low-rank Tensor Restoration for ERP extraction

Event-related potential (ERP) data is essentially multi-dimensional, with correlated data in some spaces. Therefore, matrix and vector analysis results in structural information loss. Tensor decomposition can be used to explore the shared structural information of ERP signal among related conditions. Perceiving the decaying trends of the singular value changes of unfolding matrices indicate that they are low-rank matrices. Based on this assumption, in this work, a low-rank tensor restoration (LTR) method is proposed. An operator splitting method known as the alternating direction method of multipliers (ADMM) is adapted to tackle the proposed optimization problem with orthogonality and sparsity constraints. Accordingly, the problem is solved in a sequential fashion by computing the unconstrained and orthogonality constrained quadratic sub-problems with closed-form solutions. The algorithm is examined under three application areas, namely, noise removal, feature extraction and subject-to-subject transfer learning. The empirical evaluations on real P300-based ERP dataset demonstrate the robustness and effectiveness of the proposed method.

Low-Rank Combinatorial Optimization and Statistical Learning by Spatial Photonic Ising Machine.

The spatial photonic Ising machine (SPIM) [13D. Pierangeli et al., Large-Scale Photonic Ising Machine by Spatial Light Modulation, Phys. Rev. Lett. 122, 213902 (2019).PRLTAO0031-900710.1103/PhysRevLett.122.213902] is a promising optical architecture utilizing spatial light modulation for solving large-scale combinatorial optimization problems efficiently. The primitive version of the SPIM, however, can accommodate Ising problems with only rank-one interaction matrices. In this Letter, we propose a new computing model for the SPIM that can accommodate any Ising problem without changing its optical implementation. The proposed model is particularly efficient for Ising problems with low-rank interaction matrices, such as knapsack problems. Moreover, it acquires the learning ability of Boltzmann machines. We demonstrate that learning, classification, and sampling of the MNIST handwritten digit images are achieved efficiently using the model with low-rank interactions. Thus, the proposed model exhibits higher practical applicability to various problems of combinatorial optimization and statistical learning, without losing the scalability inherent in the SPIM architecture.

Generalized Matrix Local Low Rank Representation by Random Projection and Submatrix Propagation.

Matrix low rank approximation is an effective method to reduce or eliminate the statistical redundancy of its components. Compared with the traditional global low rank methods such as singular value decomposition (SVD), local low rank approximation methods are more advantageous to uncover interpretable data structures when clear duality exists between the rows and columns of the matrix. Local low rank approximation is equivalent to low rank submatrix detection. Unfortunately, existing local low rank approximation methods can detect only submatrices of specific mean structure, which may miss a substantial amount of true and interesting patterns. In this work, we develop a novel matrix computational framework called RPSP (Random Probing based submatrix Propagation) that provides an effective solution for the general matrix local low rank representation problem. RPSP detects local low rank patterns that grow from small submatrices of low rank property, which are determined by a random projection approach. RPSP is supported by theories of random projection. Experiments on synthetic data demonstrate that RPSP outperforms all state-of-the-art methods, with the capacity to robustly and correctly identify the low rank matrices when the pattern has a similar mean as the background, background noise is heteroscedastic and multiple patterns present in the data. On real-world datasets, RPSP also demonstrates its effectiveness in identifying interpretable local low rank matrices.

Structured Matrices and Their Application in Neural Networks: A Survey

Modern neural network architectures are becoming larger and deeper, with increasing computational resources needed for training and inference. One approach toward handling this increased resource consumption is to use structured weight matrices. By exploiting structures in weight matrices, the computational complexity for propagating information through the network can be reduced. However, choosing the right structure is not trivial, especially since there are many different matrix structures and structure classes. In this paper, we give an overview over the four main matrix structure classes, namely semiseparable matrices, matrices of low displacement rank, hierarchical matrices and products of sparse matrices. We recapitulate the definitions of each structure class, present special structure subclasses, and provide references to research papers in which the structures are used in the domain of neural networks. We present two benchmarks comparing the classes. First, we benchmark the error for approximating different test matrices. Second, we compare the prediction performance of neural networks in which the weight matrix of the last layer is replaced by structured matrices. After presenting the benchmark results, we discuss open research questions related to the use of structured matrices in neural networks and highlight future research directions.

Fast hierarchical low-rank view factor matrices for thermal irradiance on planetary surfaces

We present an algorithm for compressing the radiosity view factor model commonly used in radiation heat transfer and computer graphics. We use a format inspired by the hierarchical off-diagonal low rank format, where elements are recursively partitioned using a quadtree or octree and blocks are compressed using a sparse singular value decomposition—the hierarchical matrix is assembled using dynamic programming. The motivating application is time-dependent thermal modeling on vast planetary surfaces, with a focus on permanently shadowed craters which receive energy through indirect irradiance. In this setting, shape models are comprised of a large number of triangular facets which conform to a rough surface. At each time step, a quadratic number of triangle-to-triangle scattered fluxes must be summed; that is, as the sun moves through the sky, we must solve the same view factor system of equations for a potentially unlimited number of time-varying righthand sides. We first conduct numerical experiments with a synthetic spherical cap-shaped crater, where the equilibrium temperature is analytically available. We also test our implementation with triangle meshes of planetary surfaces derived from digital elevation models recovered by orbiting spacecraft. Our results indicate that the compressed view factor matrix can be assembled in quadratic time, which is comparable to the time it takes to assemble the full view matrix itself. Memory requirements during assembly are reduced by a large factor. Finally, for a range of compression tolerances, the size of the compressed view factor matrix and the speed of the resulting matrix vector product both scale linearly (as opposed to quadratically for the full matrix), resulting in orders of magnitude savings in processing time and memory space.

Semi-device-dependent blind quantum tomography

Extracting tomographic information about quantum states is a crucial task in the quest towards devising high-precision quantum devices. Current schemes typically require measurement devices for tomography that are a priori calibrated to high precision. Ironically, the accuracy of the measurement calibration is fundamentally limited by the accuracy of state preparation, establishing a vicious cycle. Here, we prove that this cycle can be broken and the dependence on the measurement device&amp;apos;s calibration significantly relaxed. We show that exploiting the natural low-rank structure of quantum states of interest suffices to arrive at a highly scalable `blind&amp;apos; tomography scheme with a classically efficient post-processing algorithm. We further improve the efficiency of our scheme by making use of the sparse structure of the calibrations. This is achieved by relaxing the blind quantum tomography problem to the de-mixing of a sparse sum of low-rank matrices. We prove that the proposed algorithm recovers a low-rank quantum state and the calibration provided that the measurement model exhibits a restricted isometry property. For generic measurements, we show that it requires a close-to-optimal number of measurement settings. Complementing these conceptual and mathematical insights, we numerically demonstrate that robust blind quantum tomography is possible in a practical setting inspired by an implementation of trapped ions.

Multilinear multitask learning by transformed tensor singular value decomposition

In this paper, we study the problem of multilinear multitask learning (MLMTL), in which all tasks are stacked into a third-order tensor for consideration. In contrast to conventional multitask learning, MLMTL can explore inherent correlations among multiple tasks in a better manner by utilizing multilinear low rank structure. Existing approaches about MLMTL are mainly based on the sum of singular values for approximating low rank matrices obtained by matricizing the third-order tensor. However, these methods are suboptimal in the Tucker rank approximation. In order to elucidate intrinsic correlations among multiple tasks, we present a new approach by the use of transformed tensor nuclear norm (TTNN) constraint in the objective function. The main advantage of the proposed approach is that it can acquire a low transformed multi-rank structure in a transformed tensor by applying suitable unitary transformations which is helpful to determine principal components in grouping multiple tasks for describing their intrinsic correlations more precisely. Furthermore, we establish an excess risk bound of the minimizer of the proposed TTNN approach. Experimental results including synthetic problems and real-world images, show that the mean-square errors of the proposed method is lower than those of the existing methods for different number of tasks and training samples in MLMTL.

Euclidean Representation of Low-Rank Matrices and Its Geometric Properties

Euclidean Representation of Low-Rank Matrices and Its Geometric Properties

Compressed sensing of low-rank plus sparse matrices

Expressing a matrix as the sum of a low-rank matrix plus a sparse matrix is a flexible model capturing global and local features in data. This model is the foundation of robust principle component analysis (Candes et al., 2011) (Chandrasekaran et al., 2009), and popularized by dynamic-foreground/static-background separation (Bouwmans et al., 2016) amongst other applications. Compressed sensing, matrix completion, and their variants (Eldar and Kutyniok, 2012) (Foucart and Rauhut, 2013) have established that data satisfying low complexity models can be efficiently measured and recovered from a number of measurements proportional to the model complexity rather than the ambient dimension. This manuscript develops similar guarantees showing that $m\times n$ matrices that can be expressed as the sum of a rank-$r$ matrix and a $s$-sparse matrix can be recovered by computationally tractable methods from $\mathcal{O}(r(m+n-r)+s)\log(mn/s)$ linear measurements. More specifically, we establish that the restricted isometry constants for the aforementioned matrices remain bounded independent of problem size provided $p/mn$, $s/p$, and $r(m+n-r)/p$ reman fixed. Additionally, we show that semidefinite programming and two hard threshold gradient descent algorithms, NIHT and NAHT, converge to the measured matrix provided the measurement operator's RIC's are sufficiently small. Numerical experiments illustrating these results are shown for synthetic problems, dynamic-foreground/static-background separation, and multispectral imaging.

Low-Rank Tensor Based Proximity Learning for Multi-View Clustering

Graph-oriented multi-view clustering methods have achieved impressive performances by employing relationships and complex structures hidden in multi-view data. However, most of them still suffer from the following two common problems. (1) They target at studying a common representation or pairwise correlations between views, neglecting the comprehensiveness and deeper higher-order correlations among multiple views. (2) The prior knowledge of view-specific representation can not be taken into account to obtain the consensus indicator graph in a unified graph construction and clustering framework. To deal with these problems, we propose a novel Low-rank Tensor Based Proximity Learning (LTBPL) approach for multi-view clustering, where multiple low-rank probability affinity matrices and consensus indicator graph reflecting the final performances are jointly studied in a unified framework. Specifically, multiple affinity representations are stacked in a low-rank constrained tensor to recover their comprehensiveness and higher-order correlations. Meanwhile, view-specific representation carrying different adaptive confidences is jointly linked with the consensus indicator graph. Extensive experiments on nine real-world datasets indicate the superiority of LTBPL compared with the state-of-the-art methods.

Approximately low-rank recovery from noisy and local measurements by convex program

Abstract Low-rank matrix models have been universally useful for numerous applications, from classical system identification to more modern matrix completion in signal processing and statistics. The nuclear norm has been employed as a convex surrogate of the low-rankness since it induces a low-rank solution to inverse problems. While the nuclear norm for low rankness has an excellent analogy with the $\ell _1$ norm for sparsity through the singular value decomposition, other matrix norms also induce low-rankness. Particularly as one interprets a matrix as a linear operator between Banach spaces, various tensor product norms generalize the role of the nuclear norm. We provide a tensor-norm-constrained estimator for the recovery of approximately low-rank matrices from local measurements corrupted with noise. A tensor-norm regularizer is designed to adapt to the local structure. We derive statistical analysis of the estimator over matrix completion and decentralized sketching by applying Maurey’s empirical method to tensor products of Banach spaces. The estimator provides a near-optimal error bound in a minimax sense and admits a polynomial-time algorithm for these applications.

Decentralized Gradient-Quantization Based Matrix Factorization for Fast Privacy-Preserving Point-of-Interest Recommendation

With the rapidly growing of location-based social networks, point-of-interest (POI) recommendation has been attracting tremendous attentions. Previous works for POI recommendation usually use matrix factorization (MF)-based methods, which achieve promising performance. However, existing MF-based methods suffer from two critical limitations: (1) Privacy issues: all users’ sensitive data are collected to the centralized server which may leak on either the server side or during transmission. (2) Poor resource utilization and training efficiency: training on centralized server with potentially huge low-rank matrices is computational inefficient. In this paper, we propose a novel decentralized gradient-quantization based matrix factorization (DGMF) framework to address the above limitations in POI recommendation. Compared with the centralized MF methods which store all sensitive data and low-rank matrices during model training, DGMF treats each user’s device (e.g., phone) as an independent learner and keeps the sensitive data on each user’s end. Furthermore, a privacy-preserving and communication-efficient mechanism with gradient-quantization technique is presented to train the proposed model, which aims to handle the privacy problem and reduces the communication cost in the decentralized setting. Theoretical guarantees of the proposed algorithm and experimental studies on real-world datasets demonstrate the effectiveness of the proposed algorithm.

Spiked singular values and vectors under extreme aspect ratios

The behavior of the leading singular values and vectors of noisy low-rank matrices is fundamental to many statistical and scientific problems. Theoretical understanding currently derives from asymptotic analysis under one of two regimes: classical, with a fixed number of rows, large number of columns or vice versa; and proportional, with large numbers of rows and columns, proportional to one another. This paper is concerned with the disproportional regime, where the matrix is either “tall and narrow” or “short and wide”: we study sequences of matrices of size n×mn with aspect ratio n/mn→0 or n/mn→∞ as n→∞. This regime has important “big data” applications.Theory derived here shows that the displacement of the empirical singular values and vectors from their noise-free counterparts and the associated phase transitions—well-known under proportional growth asymptotics—still occur in the disproportionate setting. They must be quantified, however, on a novel scale of measurement that adjusts with the changing aspect ratio as the matrix size increases. In this setting, the top singular vectors corresponding to the longer of the two matrix dimensions are asymptotically uncorrelated with the noise-free signal.

Robust Matrix Completion Based on Factorization and Truncated-Quadratic Loss Function

Robust matrix completion refers to recovering a low-rank matrix given a subset of the entries corrupted by gross errors, and has various applications since many real-world signals can be modeled as low-rank matrices. Most of the existing methods only perform well for noise-free data or those with zero-mean white Gaussian noise, and their performance will be degraded in the presence of outliers. In this paper, based on the factorization framework, we propose a novel robust matrix completion scheme via using the truncated-quadratic loss function, which is non-convex and non-smooth, and half-quadratic theory is adopted for its optimization. By introducing an auxiliary variable, half-quadratic optimization (HO) can transform the loss function into two tractable forms, that is, additive and multiplicative formulations. Block coordinate descent method is then exploited as their solver. Compared with the additive form, the multiplicative variant has lower computational cost since we attempt to take the observations contaminated by outliers as missing entries. Numerical simulations and experimental results based on image inpainting and hyperspectral image recovery demonstrate that our algorithms are superior to the state-of-the-art methods in terms of restoration accuracy and runtime. MATLAB code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/bestzywang</uri> .

Fast algorithms for singular value decomposition and the inverse of nearly low-rank matrices.

This perspective focuses on the fast algorithm design for singular value decomposition and inverse computation of nearly low-rank matrices that are potentially of big sizes.

Similarity matrix enhanced collaborative filtering for e-government recommendation

Personalized e-government recommendation aims to predict the user’s next action in a near future based on his or her historical behavior data. Similarity measures and distances are widely used for powering personalized recommendations. The classical similarity-based models in the recommender systems are known as K-Nearest Neighbor Collaborative Filtering (KNN-based CF). Most prior work adopts a new similarity formula to obtain the k-nearest neighbors for the user and/or item. This is a good try but falls short of solving the problem of expensive computation and introducing the order of users’ interaction sequences. In this paper, we propose SACF, a similarity matrix enhanced the e-government recommendation model that takes advantage of pairwise learning to introduce the order of uses’ interaction sequences and quickly obtain the similarity matrix using matrix factorization. In other words, the similarity matrix can be presented as the production of two low-rank feature matrices. Therefore, it is possible to effectively estimate the similarity between two users even with few co-rated items. Experimental results on a real-world e-government dataset show that the proposed approach improves the performance significantly compared with other counterparts.

Unsupervised Single-Channel Singing Voice Separation with Weighted Robust Principal Component Analysis Based on Gammatone Auditory Filterbank and Vocal Activity Detection

Singing-voice separation is a separation task that involves a singing voice and musical accompaniment. In this paper, we propose a novel, unsupervised methodology for extracting a singing voice from the background in a musical mixture. This method is a modification of robust principal component analysis (RPCA) that separates a singing voice by using weighting based on gammatone filterbank and vocal activity detection. Although RPCA is a helpful method for separating voices from the music mixture, it fails when one single value, such as drums, is much larger than others (e.g., the accompanying instruments). As a result, the proposed approach takes advantage of varying values between low-rank (background) and sparse matrices (singing voice). Additionally, we propose an expanded RPCA on the cochleagram by utilizing coalescent masking on the gammatone. Finally, we utilize vocal activity detection to enhance the separation outcomes by eliminating the lingering music signal. Evaluation results reveal that the proposed approach provides superior separation outcomes than RPCA on ccMixter and DSD100 datasets.

Optimal Convex Lifted Sparse Phase Retrieval and PCA With an Atomic Matrix Norm Regularizer

We present novel analysis and algorithms for solving sparse phase retrieval and sparse principal component analysis (PCA) with convex lifted matrix formulations. The key innovation is a new mixed atomic matrix norm that, when used as regularization, promotes low-rank matrices with sparse factors. We show that convex programs with this atomic norm as a regularizer provide near-optimal sample complexity and error rate guarantees for sparse phase retrieval and sparse PCA. While we do not know how to solve the convex programs exactly with an efficient algorithm, for the phase retrieval case we carefully analyze the program and its dual and thereby derive a practical heuristic algorithm. We show empirically that this practical algorithm performs similarly to existing state-of-the-art algorithms.

BALLORG: State-of-the-art Image Restoration using Block-augmented Lagrangian and Low-rank Gradients

In this paper, we propose a blind image deblurring algorithm using block-augmented Lagrangian and low-rank priors (BALLORG) as a non-learning method that can give better results without the complexity of learning-based methods. The proposed algorithm achieves faster convergence within 20 iterations than conventional methods. Regularization priors are used in the form of gradients and sparse low-rank matrices, and recursive rank improvements result in better deblurring performance. The steepest descent in minimization is maintained through weight selection for penalty and regularization parameters. The block processing introduces local and global optimization, leading to better visual quality outputs. The proposed method has excellent performance in terms of the PSNR, SSIM, and FSIM matrix, which is on par with or better than that of other state-of-the-art learning and non-learning-based approaches.