Low-rank Matrix Estimation Research Articles

Statistically optimal firstorder algorithms: a proof via orthogonalization

Abstract We consider a class of statistical estimation problems in which we are given a random data matrix ${\boldsymbol{X}}\in{\mathbb{R}}^{n\times d}$ (and possibly some labels ${\boldsymbol{y}}\in{\mathbb{R}}^{n}$) and would like to estimate a coefficient vector ${\boldsymbol{\theta }}\in{\mathbb{R}}^{d}$ (or possibly a constant number of such vectors). Special cases include low-rank matrix estimation and regularized estimation in generalized linear models (e.g. sparse regression). Firstorder methods proceed by iteratively multiplying current estimates by ${\boldsymbol{X}}$ or its transpose. Examples include gradient descent or its accelerated variants. Under the assumption that the data matrix ${\boldsymbol{X}}$ is standard Gaussian, Celentano, Montanari, Wu (2020, Conference on Learning Theory, pp. 1078–1141. PMLR) proved that for any constant number of iterations (matrix vector multiplications), the optimal firstorder algorithm is a specific approximate message passing algorithm (known as ‘Bayes AMP’). The error of this estimator can be characterized in the high-dimensional asymptotics $n,d\to \infty $, $n/d\to \delta $, and provides a lower bound to the estimation error of any firstorder algorithm. Here we present a simpler proof of the same result, and generalize it to broader classes of data distributions and of firstorder algorithms, including algorithms with non-separable nonlinearities. Most importantly, the new proof is simpler, does not require to construct an equivalent tree-structured estimation problem, and is therefore susceptible of a broader range of applications.

Fundamental limits of low-rank matrix estimation with diverging aspect ratios

Fundamental limits of low-rank matrix estimation with diverging aspect ratios

High-Dimensional Multivariate Linear Regression with Weighted Nuclear Norm Regularization

We consider a low-rank matrix estimation problem when the data is assumed to be generated from the multivariate linear regression model. To induce the low-rank coefficient matrix, we employ the weighted nuclear norm (WNN) penalty defined as the weighted sum of the singular values of the matrix. The weights are set in a nondecreasing order, which yields the non-convexity of the WNN objective function in the parameter space. Although the objective function has been widely applied, studies on the estimation properties of its resulting estimator are limited. We propose an efficient algorithm under the framework of the alternative directional method of multipliers (ADMM) to estimate the coefficient matrix. The estimator from the suggested algorithm converges to a stationary point of an augmented Lagrangian function. Under the orthogonal design setting, the effects of the weights for estimating the singular values of the ground-truth coefficient matrix are derived. Under the Gaussian design setting, a minimax convergence rate on the estimation error is derived. We also propose a generalized cross-validation (GCV) criterion for selecting the tuning parameter and an iterative algorithm for updating the weights. Simulations and a real data analysis demonstrate the competitive performance of our new method. Supplementary materials for this article are available online.

Iteratively reweighted accurate sparse low-rank matrix estimation algorithm for bearing fault diagnosis

Bearing fault diagnosis is significant to ensure the safe functioning of mechanical systems. One of the key issues in bearing fault diagnosis is the accurate and effective extraction of fault characteristic features. In order to further explore the construction of penalty function, further alleviate the problem of bias estimation and enhance the capabilities of the sparse low-rank (SLR) method in effectively diagnosing bearing faults, in this paper, an iteratively reweighted accurate sparse low-rank (IRASLR) matrix estimation algorithm is proposed. Taking the sparse and low-rank properties of fault transient signals in time–frequency domain into account, an optimization problem is formulated. Two penalty functions, derived from a core minimax-concave penalty (MCP), induce sparse and low-rank properties, respectively. Specifically, to facilitate the design of weighting schemes, we construct an equivalent variant of MCP (EVMCP). By separately extending EVMCP to matrix elements and singular values, we obtain equivalent weighted MCP and equivalent weighted matrix γ-norm. Finally, based on the proximal operators of the two penalty functions, we deduce an IRASLR algorithm under the alternating direction method of the multipliers (ADMM) framework. The performance of IRASLR is verified by processing simulated and experimental signals, while the superiority of the algorithm is confirmed through comparative experiments.

Graph-based approximate message passing iterations

Abstract Approximate message passing (AMP) algorithms have become an important element of high-dimensional statistical inference, mostly due to their adaptability and concentration properties, the state evolution (SE) equations. This is demonstrated by the growing number of new iterations proposed for increasingly complex problems, ranging from multi-layer inference to low-rank matrix estimation with elaborate priors. In this paper, we address the following questions: is there a structure underlying all AMP iterations that unifies them in a common framework? Can we use such a structure to give a modular proof of state evolution equations, adaptable to new AMP iterations without reproducing each time the full argument? We propose an answer to both questions, showing that AMP instances can be generically indexed by an oriented graph. This enables to give a unified interpretation of these iterations, independent from the problem they solve, and a way of composing them arbitrarily. We then show that all AMP iterations indexed by such a graph verify rigorous SE equations, extending the reach of previous proofs and proving a number of recent heuristic derivations of those equations. Our proof naturally includes non-separable functions and we show how existing refinements, such as spatial coupling or matrix-valued variables, can be combined with our framework.

Seismic Data Reconstruction and Denoising by Enhanced Hankel Low-Rank Matrix Estimation

Seismic data reconstruction and denoising play a fundamental role in most seismic data processing algorithms which are often designed for regularly sampled and reliable data. By using the fact that the (block) Hankel matrix formulated from clean seismic data is low-rank if it corresponds to a few linear events, the low-rank based approach has been successfully used for seismic data reconstruction and denoising. In this paper, we simultaneously exploit the Hankel and low-rank structures within the (block) Hankel matrices of the clear and complete seismic data and formulate the problem of seismic data reconstruction and denoising as a Hankel low-rank reconstruction problem. The additional Hankel structure can further improve the performance for reconstruction and denoising. We then propose an iterative algorithm for solving the corresponding Hankel low-rank reconstruction problem. The algorithm is based on the alternating direction method of multipliers (ADMM) and has closed-form solutions for each update. Experiments on both synthetical and real seismic data demonstrate the superior performance of the proposed algorithm compared with conventional low-rank based methods for simultaneous seismic data reconstruction and denoising.

Sparse Low-Rank Matrix Estimation With Nonconvex Enhancement for Fault Diagnosis of Rolling Bearings

The diagnosis of bearing early fault is significant and fundamental in the machine condition monitoring. The accurate and effective diagnosis is of great importance to avoid further serious accidents. However, existing sparse low-rank (SLR) methods for bearing fault diagnosis suffer from underestimation of amplitude and inaccurate approximation of singular values. Therefore, in this paper, a novel sparse low-rank matrix estimation method with nonconvex enhancement (SLRNE) is proposed, extracting the fault transients from observed noisy signal. Specifically, fault transients have both sparse and low-rank properties in time-frequency domain. Based on this, a SLR optimization model is proposed to simultaneously promote the above two properties via truncated nuclear norm (TNN) and generalized minimax concave (GMC) penalty function. The two nonconvex functions aim to promote low-rank property and sparsity respectively. Then, based on derived convexity conditions of the optimization problem, convex optimization algorithm, alternating direction method of multipliers (ADMM) and forward-and-backward splitting (FBS) algorithm, are applied to obtain global optimal solution. In the iterative algorithm, a weighting strategy is designed for the singular value threshold operator to enhance the effect of fault feature extraction. Simulated and experimental signals verify the effectiveness of SLRNE and contrast experiments verify its superiority.

A Model-Agnostic Method for PMU Data Recovery Using Optimal Singular Value Thresholding

This paper presents a fast model-agnosticmethod for recovering noisy Phasor Measurement Unit (PMU) data streams with missing entries. The measurements are first transformed into a Page matrix, and the original signals are reconstructed using low-rank matrix estimation based on optimal singular value thresholding. Two variations of the recovery algorithm are shown- a) an offline block-processing method for imputing past measurements, and b) an online method for predicting future measurements. Information within a PMU channel (temporal correlation) as well as from different PMU channels in a network (spatial correlation) are utilized to recover degraded data. The proposed method is fast and needs no explicit knowledge of the underlying system model or measurement noise distribution. The performance of the recovery algorithms is illustrated using simulated measurements from the IEEE 39-bus test system as well as real measurements from an anonymized U.S. electric utility. Extensive numeric tests show that the original signals can be accurately recovered in the presence of additive noise, consecutive data drop, as well as simultaneous data erasures across multiple PMU channels.

Novel cascade filter design of improved sparse low-rank matrix estimation and kernel adaptive filtering for ECG denoising and artifacts cancellation

ElectroCardioGram (ECG) signals are highly vulnerable to disturbances caused by noise and artifact sources which can degrade the ECG signal quality and increase the difficulty in obtaining reliable and accurate clinical interpretations for heart conditions. This paper introduces, for the first time, the Improved Sparse Low-Rank (ISLR) algorithm for suppressing white/colored noises, and the Kernel Recursive Least Squares with Approximate Linear Dependency (ALDKRLS) algorithm for eliminating various artifact sources. A novel automated multi-stage filter is introduced for suppressing artifact components in the first stage using ALDKRLS and eliminating noise sources in the subsequent stage using ISLR. The robustness of the suggested multi-stage filter is demonstrated by eliminating noise and artifact components individually and when both present concurrently using real ECG data. Experimental results elucidate the outstanding accuracy of the suggested framework in eliminating interference sources and keeping the essential and important characteristics of the original ECG data. Also, the application of the suggested framework in practical systems is examined by investigating a new efficient ECG multi-class classification system before and after suppressing noise and artifact interferences. Results show that the suggested framework manages not only to eliminate effectively noise and artifact components, but also to achieve very accurate ECG diagnosis results by maintaining the essential characteristics of the ECG signal that differentiate different heart disorders. This elucidates the usefulness of the proposed multi-stage filter as a promising preprocessing tool for obtaining high-resolution ECG data and consequently enhancing the diagnosis performance of several heart diseases.

The basins of attraction of the global minimizers of non-convex inverse problems with low-dimensional models in infinite dimension

Abstract Non-convex methods for linear inverse problems with low-dimensional models have emerged as an alternative to convex techniques. We propose a theoretical framework where both finite dimensional and infinite dimensional linear inverse problems can be studied. We show how the size of the basins of attraction of the minimizers of such problems is linked with the number of available measurements. This framework recovers known results about low-rank matrix estimation and off-the-grid sparse spike estimation, and it provides new results for Gaussian mixture estimation from linear measurements.

Low-rank matrix denoising for count data using unbiased Kullback-Leibler risk estimation

Many statistical studies are concerned with the analysis of observations organized in a matrix form whose elements are count data. When these observations are assumed to follow a Poisson or a multinomial distribution, it is of interest to focus on the estimation of either the intensity matrix (Poisson case) or the compositional matrix (multinomial case) when it is assumed to have a low rank structure. In this setting, it is proposed to construct an estimator minimizing the regularized negative log-likelihood by a nuclear norm penalty. Such an approach easily yields a low-rank matrix-valued estimator with positive entries which belongs to the set of row-stochastic matrices in the multinomial case. Then, as a main contribution, a data-driven procedure is constructed to select the regularization parameter in the construction of such estimators by minimizing (approximately) unbiased estimates of the Kullback-Leibler (KL) risk in such models, which generalize Stein's unbiased risk estimation originally proposed for Gaussian data. The evaluation of these quantities is a delicate problem, and novel methods are introduced to obtain accurate numerical approximation of such unbiased estimates. Simulated data are used to validate this way of selecting regularizing parameters for low-rank matrix estimation from count data. For data following a multinomial distribution, the performances of this approach are also compared to K-fold cross-validation. Examples from a survey study and metagenomics also illustrate the benefits of this methodology for real data analysis.

Minimax perturbation bounds of the low-rank matrix under Ky Fan norm

<abstract><p>This paper considers the minimax perturbation bounds of the low-rank matrix under Ky Fan norm. We first explore the upper bounds via the best rank-$ r $ approximation $ \hat{A}_r $ of the observation matrix $ \hat{A} $. Next, the lower bounds are established by constructing special matrix groups to show the upper bounds are tight on the low-rank matrix estimation error. In addition, we derive the rate-optimal perturbation bounds for the left and right singular subspaces under Ky Fan norm $ \sin\Theta $ distance. Finally, some simulations have been carried out to support our theories.</p></abstract>

On the computational tractability of statistical estimation on amenable graphs

We consider the problem of estimating a vector of discrete variables \(\theta = (\theta _1,\cdots ,\theta _n)\), based on noisy observations \(Y_{uv}\) of the pairs \((\theta _u,\theta _v)\) on the edges of a graph \(G=([n],E)\). This setting comprises a broad family of statistical estimation problems, including group synchronization on graphs, community detection, and low-rank matrix estimation. A large body of theoretical work has established sharp thresholds for weak and exact recovery, and sharp characterizations of the optimal reconstruction accuracy in such models, focusing however on the special case of Erdös–Rényi-type random graphs. An important finding of this line of work is the ubiquity of an information-computation gap. Namely, for many models of interest, a large gap is found between the optimal accuracy achievable by any statistical method, and the optimal accuracy achieved by known polynomial-time algorithms. Moreover, it is expected in many situations that this gap is robust to small amounts of additional side information revealed about the \(\theta _i\)’s. How does the structure of the graph G affect this picture? Is the information-computation gap a general phenomenon or does it only apply to specific families of graphs? We prove that the picture is dramatically different for graph sequences converging to amenable graphs (including, for instance, d-dimensional grids). We consider a model in which an arbitrarily small fraction of the vertex labels is revealed, and show that a linear-time local algorithm can achieve reconstruction accuracy that is arbitrarily close to the information-theoretic optimum. We contrast this to the case of random graphs. Indeed, focusing on group synchronization on random regular graphs, we prove that local algorithms are unable to have non-trivial performance below the so-called Kesten–Stigum threshold, even when a small amount of side information is revealed.

BRIDGING CONVEX AND NONCONVEX OPTIMIZATION IN ROBUST PCA: NOISE, OUTLIERS, AND MISSING DATA.

This paper delivers improved theoretical guarantees for the convex programming approach in low-rank matrix estimation, in the presence of (1) random noise, (2) gross sparse outliers, and (3) missing data. This problem, often dubbed as robust principal component analysis (robust PCA), finds applications in various domains. Despite the wide applicability of convex relaxation, the available statistical support (particularly the stability analysis vis-à-vis random noise) remains highly suboptimal, which we strengthen in this paper. When the unknown matrix is well-conditioned, incoherent, and of constant rank, we demonstrate that a principled convex program achieves near-optimal statistical accuracy, in terms of both the Euclidean loss and the ℓ ∞ loss. All of this happens even when nearly a constant fraction of observations are corrupted by outliers with arbitrary magnitudes. The key analysis idea lies in bridging the convex program in use and an auxiliary nonconvex optimization algorithm, and hence the title of this paper.

Enhanced low-rank matrix estimation for simultaneous denoising and reconstruction of 5D seismic data

Noise and missing traces usually influence the quality of multidimensional seismic data. Therefore, it is necessary to estimate the useful signal from its noisy observation. The damped rank-reduction (DRR) method has emerged as an effective method to reconstruct the useful signal matrix from noisy and incomplete observations. However, the higher the noise level and the larger the ratio of missing traces, the weaker the DRR operator becomes. Consequently, the estimated low-rank (LR) signal matrix includes a significant amount of residual noise that influences the following processing steps. Therefore, we focus on the problem of estimating an LR signal matrix from its noisy observation. To elaborate on the novel algorithm, we formulate an improved proximity function by mixing the moving-average filter and the arctangent penalty function. First, we apply the proximity function to the level-4 block Hankel matrix before singular-value decomposition (SVD) and, then, to singular values, during the damped truncated SVD process. The combination of the novel proximity function and the DRR framework leads to an optimization problem, which results in better recovery performance. Our algorithm aims at producing an enhanced rank-reduction operator to estimate the useful signal matrix with higher quality. Experiments are conducted on synthetic and real 5D seismic data to compare the effectiveness of our approach to the DRR approach. Our approach obtains better performance because the estimated LR signal matrix is cleaner and contains fewer artifacts compared to that reconstructed by the DRR algorithm.

A Schatten-q low-rank matrix perturbation analysis via perturbation projection error bound

This paper studies the Schatten-q error of low-rank matrix estimation by singular value decomposition under perturbation. We specifically establish a perturbation bound on the low-rank matrix estimation via a perturbation projection error bound. Then, we establish lower bounds to justify the tightness of the upper bound on the low-rank matrix estimation error. We further develop a user-friendly sinΘ bound for singular subspace perturbation based on the matrix perturbation projection error bound. Finally, we demonstrate the advantage of our results over the ones in the literature by simulation.

Estimation of low-rank matrices via approximate message passing

Consider the problem of estimating a low-rank matrix when its entries are perturbed by Gaussian noise, a setting that is also known as “spiked model” or “deformed random matrix.” If the empirical distribution of the entries of the spikes is known, optimal estimators that exploit this knowledge can substantially outperform simple spectral approaches. Recent work characterizes the asymptotic accuracy of Bayes-optimal estimators in the high-dimensional limit. In this paper, we present a practical algorithm that can achieve Bayes-optimal accuracy above the spectral threshold. A bold conjecture from statistical physics posits that no polynomial-time algorithm achieves optimal error below the same threshold (unless the best estimator is trivial). Our approach uses Approximate Message Passing (AMP) in conjunction with a spectral initialization. AMP algorithms have proved successful in a variety of statistical estimation tasks, and are amenable to exact asymptotic analysis via state evolution. Unfortunately, state evolution is uninformative when the algorithm is initialized near an unstable fixed point, as often happens in low-rank matrix estimation problems. We develop a new analysis of AMP that allows for spectral initializations, and builds on a decoupling between the outlier eigenvectors and the bulk in the spiked random matrix model. Our main theorem is general and applies beyond matrix estimation. However, we use it to derive detailed predictions for the problem of estimating a rank-one matrix in noise. Special cases of this problem are closely related—via universality arguments—to the network community detection problem for two asymmetric communities. For general rank-one models, we show that AMP can be used to construct confidence intervals and control false discovery rate. We provide illustrations of the general methodology by considering the cases of sparse low-rank matrices and of block-constant low-rank matrices with symmetric blocks (we refer to the latter as to the “Gaussian block model”).

Low-Rank Matrix Estimation from Rank-One Projections by Unlifted Convex Optimization

We study an estimator with a convex formulation for recovery of low-rank matrices from rank-one projections. Using initial estimates of the factors of the target $d_1\times d_2$ matrix of rank-$r$,...

Matrix Completion under Low-Rank Missing Mechanism

Matrix completion is a modern missing data problem where both the missing structure and the underlying parameter are high dimensional. Although missing structure is a key component to any missing data problems, existing matrix completion methods often assume a simple uniform missing mechanism. In this work, we study matrix completion from corrupted data under a novel low-rank missing mechanism. The probability matrix of observation is estimated via a high dimensional low-rank matrix estimation procedure, and further used to complete the target matrix via inverse probabilities weighting. Due to both high dimensional and extreme (i.e., very small) nature of the true probability matrix, the effect of inverse probability weighting requires careful study. We derive optimal asymptotic convergence rates of the proposed estimators for both the observation probabilities and the target matrix.

Beyond Procrustes: Balancing-Free Gradient Descent for Asymmetric Low-Rank Matrix Sensing

Low-rank matrix estimation plays a central role in various applications across science and engineering. Recently, nonconvex formulations based on matrix factorization are provably solved by simple gradient descent algorithms with strong computational and statistical guarantees. However, when the low-rank matrices are asymmetric, existing approaches rely on adding a regularization term to balance the scale of the two matrix factors which in practice can be removed safely without hurting the performance when initialized via the spectral method. In this paper, we provide a theoretical justification to this for the matrix sensing problem, which aims to recover a low-rank matrix from a small number of linear measurements. As long as the measurement ensemble satisfies the restricted isometry property, gradient descent -- in conjunction with spectral initialization -- converges linearly without the need of explicitly promoting balancedness of the factors; in fact, the factors stay balanced automatically throughout the execution of the algorithm. Our analysis is based on analyzing the evolution of a new distance metric that directly accounts for the ambiguity due to invertible transforms, and might be of independent interest.