Robust Principal Component Analysis Problem Research Articles

Bregman Proximal Linearized ADMM for Minimizing Separable Sums Coupled by a Difference of Functions

AbstractIn this paper, we develop a splitting algorithm incorporating Bregman distances to solve a broad class of linearly constrained composite optimization problems, whose objective function is the separable sum of possibly nonconvex nonsmooth functions and a smooth function, coupled by a difference of functions. This structure encapsulates numerous significant nonconvex and nonsmooth optimization problems in the current literature including the linearly constrained difference-of-convex problems. Relying on the successive linearization and alternating direction method of multipliers (ADMM), the proposed algorithm exhibits the global subsequential convergence to a stationary point of the underlying problem. We also establish the convergence of the full sequence generated by our algorithm under the Kurdyka–Łojasiewicz property and some mild assumptions. The efficiency of the proposed algorithm is tested on a robust principal component analysis problem and a nonconvex optimal power flow problem.

Adaptive accelerated proximal gradient algorithm for auto-regressive exogenous models with outliers

This study introduces an enhanced recursive least-squares algorithm that applies the adaptive accelerated proximal gradient method to identify Auto-Regressive Exogenous models with output outliers. First, the outlier problem was converted into a robust principal component analysis problem. The adaptive accelerated proximal gradient method is then introduced to recover the information matrix, and the recursive least squares algorithm is applied to estimate the model parameters. Furthermore, we demonstrated that the parameter estimation is unbiased and that the parameter estimation error converges to zero under persistent excitation. A series of bench tests consistently highlighted the practicality and effectiveness of the proposed algorithm.

Multidimensional Data Processing With Bayesian Inference via Structural Block Decomposition.

How to handle large multidimensional datasets, such as hyperspectral images and video information, efficiently and effectively plays a critical role in big-data processing. The characteristics of low-rank tensor decomposition in recent years demonstrate the essentials in describing the tensor rank, which often leads to promising approaches. However, most current tensor decomposition models consider the rank-1 component simply to be the vector outer product, which may not fully capture the correlated spatial information effectively for large-scale and high-order multidimensional datasets. In this article, we develop a new novel tensor decomposition model by extending it to the matrix outer product or called Bhattacharya-Mesner product, to form an effective dataset decomposition. The fundamental idea is to decompose tensors structurally in a compact manner as much as possible while retaining data spatial characteristics in a tractable way. By incorporating the framework of the Bayesian inference, a new tensor decomposition model on the subtle matrix unfolding outer product is established for both tensor completion and robust principal component analysis problems, including hyperspectral image completion and denoising, traffic data imputation, and video background subtraction. Numerical experiments on real-world datasets demonstrate the highly desirable effectiveness of the proposed approach.

Tensor recovery based on Bivariate Equivalent Minimax-Concave Penalty

In tensor recovery problem, larger singular values often correspond to the primary information of the image, such as contours, sharp edges and smooth areas. The minimum–maximum concave penalty (MCP) function has been effective in preserving larger singular values and achieving good tensor recovery results. However, in the process of solving this problem, singular values may vary during iterations, and the fixed parameters of the MCP function may not sufficiently preserve all the larger singular values, which may hinder the attainment of optimal results in tensor recovery. To overcome this challenge, we propose the Bivariate Equivalent Minimax-Concave Penalty (BEMCP) theorem, which allows the parameters to adapt to the changes in singular values and more comprehensively preserve the larger singular values. For low-rank tensor completion and tensor robust principal component analysis problems, we propose BEMCP-based models. Finally, experiments with various real-world data demonstrate that the proposed methods outperform state-of-the-art methods.

Interpretable Neural Networks for Video Separation: Deep Unfolding RPCA with Foreground Masking.

We present two deep unfolding neural networks for the simultaneous tasks of background subtraction and foreground detection in video. Unlike conventional neural networks based on deep feature extraction, we incorporate domain-knowledge models by considering a masked variation of the robust principal component analysis problem (RPCA). With this approach, we separate video clips into low-rank and sparse components, respectively corresponding to the backgrounds and foreground masks indicating the presence of moving objects. Our models, coined ROMAN-S and ROMAN-R, map the iterations of two alternating direction of multipliers methods (ADMM) to trainable convolutional layers, and the proximal operators are mapped to non-linear activation functions with trainable thresholds. This approach leads to lightweight networks with enhanced interpretability that can be trained on limited data. In ROMAN-S, the correlation in time of successive binary masks is controlled with side-information based on ℓ1-ℓ1 minimization. ROMAN-R enhances the foreground detection by learning a dictionary of atoms to represent the moving foreground in a high-dimensional feature space and by using reweighted-ℓ1-ℓ1 minimization. Experiments are conducted on both synthetic and real video datasets, for which we also include an analysis of the generalization to unseen clips. Comparisons are made with existing deep unfolding RPCA neural networks, which do not use a mask formulation for the foreground, and with a 3D U-Net baseline. Results show that our proposed models outperform other deep unfolding networks, as well as the untrained optimization algorithms. ROMAN-R, in particular, is competitive with the U-Net baseline for foreground detection, with the additional advantage of providing video backgrounds and requiring substantially fewer training parameters and smaller training sets.

Graph Convolution RPCA With Adaptive Graph.

Principal component analysis (PCA) is warmly welcomed in dimensionality reduction and its applications. Due to the high sensitivity of PCA to outliers, a series of PCA methods are proposed to enhance the robustness of PCA. Besides, the representation ability of the existing PCA methods has limitations as well. To enhance the robustness and representation ability of robust PCA, we elaborate a novel Graph Convolution Robust PCA method (GRPCA) to incorporate the manifold structure into PCA. It constructs a sparse graph based on the local connectivity structure of samples. Graph auto-encoder is utilized to solve the robust PCA problem under the low-rank and sparse constraints. With the dual-decoder, GRPCA learns the low-dimensional embeddings that reconstruct the manifold structure and low-rank approximation simultaneously. Furthermore, since the graph suffers from misconnection triggered by occlusions, the local connectivity structure of low-dimensional embeddings is utilized to modify the graph. Our proposed method excels in both the clustering of low-dimensional embeddings and the low-rank recovery. Lastly, extensive experiments conducted on six real-world datasets demonstrated the efficiency and superiority of the proposed GRPCA.

Modified proximal symmetric ADMMs for multi-block separable convex optimization with linear constraints

We consider the linearly constrained separable convex optimization problem whose objective function is separable with respect to m blocks of variables. A bunch of methods have been proposed and extensively studied in the past decade. Specifically, a modified strictly contractive Peaceman–Rachford splitting method (SC-PRCM) [S. H. Jiang and M. Li, A modified strictly contractive Peaceman–Rachford splitting method for multi-block separable convex programming, J. Ind. Manag. Optim. 14(1) (2018) 397-412] has been well studied in the literature for the special case of m = 3. Based on the modified SC-PRCM, we present modified proximal symmetric ADMMs (MPSADMMs) to solve the multi-block problem. In MPSADMMs, all subproblems but the first one are attached with a simple proximal term, and the multipliers are updated twice. At the end of each iteration, the output is corrected via a simple correction step. Without stringent assumptions, we establish the global convergence result and the O(1/t) convergence rate in the ergodic sense for the new algorithms. Preliminary numerical results show that our proposed algorithms are effective for solving the linearly constrained quadratic programming and the robust principal component analysis problems.

Rapid Robust Principal Component Analysis: CUR Accelerated Inexact Low Rank Estimation

Robust principal component analysis (RPCA) is a widely used tool for dimension reduction. In this work, we propose a novel non-convex algorithm, coined Iterated Robust CUR (IRCUR), for solving RPCA problems, which dramatically improves the computational efficiency in comparison with the existing algorithms. IRCUR achieves this acceleration by employing CUR decomposition when updating the low rank component, which allows us to obtain an accurate low rank approximation via only three small submatrices. Consequently, IRCUR is able to process only the small submatrices and avoid expensive computing on the full matrix through the entire algorithm. Numerical experiments establish the computational advantage of IRCUR over the state-of-art algorithms on both synthetic and real-world datasets.

Small Target Detection for Infrared Image Based on Optimal Infrared Patch-Image Model by Solving Modified Adaptive RPCA Problem

Small target detection in infrared (IR) images has been widely applied for both military and civilian purposes. In this study, because IR images contain sparse and low-rank features in most scenarios, we propose an optimal IR patch-image (OIPI) model-based detection method to detect small targets in heavily cluttered IR images. First, the OIPI model was generated based on a conventional IR image model using a novel optimal patch size and sliding step adaptive selection algorithm. Secondly, the sparse and low-rank features of IR images were extracted and fused to generate an adaptive weighted parameter. Thirdly, the adaptive inexact augmented Lagrange multiplier (AIALM) algorithm was applied in the OIPI model to solve the robust principal component analysis (RPCA) optimization problem. Finally, an adaptive threshold method is proposed to segment and calibrate targets. Experimental results indicate that the proposed algorithm is capable of detecting small targets more stably and accurately, compared with state-of-the-art methods.

New Robust Principal Component Analysis for Joint Image Alignment and Recovery via Affine Transformations, Frobenius and L2,1 Norms

This paper proposes an effective and robust method for image alignment and recovery on a set of linearly correlated data via Frobenius and L2,1 norms. The most popular and successful approach is to model the robust PCA problem as a low-rank matrix recovery problem in the presence of sparse corruption. The existing algorithms still lack in dealing with the potential impact of outliers and heavy sparse noises for image alignment and recovery. Thus, the new algorithm tackles the potential impact of outliers and heavy sparse noises via using novel ideas of affine transformations and Frobenius and L2,1 norms. To attain this, affine transformations and Frobenius and L2,1 norms are incorporated in the decomposition process. As such, the new algorithm is more resilient to errors, outliers, and occlusions. To solve the convex optimization involved, an alternating iterative process is also considered to alleviate the complexity. Conducted simulations on the recovery of face images and handwritten digits demonstrate the effectiveness of the new approach compared with the main state-of-the-art works.

Sparse Bayesian Learning for Robust PCA: Algorithms and Analyses

In this paper, we propose a new Bayesian model to solve the Robust PCA problem - recovering the underlying low-rank matrix and sparse matrix from their noisy compositions. We first derive and analyze a new objective function, which is proven to be equivalent to the fundamental objective of minimizing the “rank+sparsity”. To solve this objective, we develop a concise Sparse Bayesian Learning (SBL) method that makes minimal assumptions and effectively deals with the requirements of the problem. The concise modeling allows simple and effective Empirical Bayesian inference via maximum a posteriori expectation-maximization (MAP-EM). We further propose a modified SBL method that additionally utilizes the sparsity pattern information of the outliers in the Robust PCA problem. Simulation studies demonstrate the superiority of the proposed methods over the existing state-of-the-art methods. The efficacy of the proposed methods is further verified through two image processing tasks.

Nonconvex Regularized Robust PCA Using the Proximal Block Coordinate Descent Algorithm

This work addresses the robust principal component analysis (PCA) problem using generalized nonoconvex regularization for low-rank and sparsity promotion. While the popular nuclear and $\ell _1$ -norm penalties have a bias problem, nonconvex regularization can alleviate the bias problem and can be expected to achieve better performance. In this paper, a proximal block coordinate descent (BCD) algorithm is used to efficiently solve the nonconvex regularized robust PCA problem. It is globally convergent under weak conditions. Further, for a popular class of penalties having discontinuous threshoding functions, we establish the convergence to a restricted strictly local minimizer and, also, a local linear convergence rate for the proximal BCD algorithm. Moreover, convergence to a local minimizer has been derived for hard-thresholding. Our result is the first on nonconvex robust PCA with established convergence to strictly local minimizer with local linear convergence rate. Numerical experiments have been provided to demonstrate the performance of the new algorithm.

A Nonconvex Projection Method for Robust PCA

Robust principal component analysis (RPCA) is a well-studied problem whose goal is to decompose a matrix into the sum of low-rank and sparse components. In this paper, we propose a nonconvex feasibility reformulation of RPCA problem and apply an alternating projection method to solve it. To the best of our knowledge, this is the first paper proposing a method that solves RPCA problem without considering any objective function, convex relaxation, or surrogate convex constraints. We demonstrate through extensive numerical experiments on a variety of applications, including shadow removal, background estimation, face detection, and galaxy evolution, that our approach matches and often significantly outperforms current state-of-the-art in various ways.

Detecting moving objects via the low-rank representation

Moving object detection is a fundamental and necessary step in many computer vision algorithms. These algorithms are built in many intelligent devices such as in the smartphones, the tachographs and the personal video recorders. Recently, the methods for performing the moving object detection based on the low-rank representation have been proposed. For these methods, it is assumed that the background is represented by a low-rank matrix. On the other hand, the foreground objects cannot be represented by low-rank matrices. They are seen as the outliers. Hence, detecting the contiguous outliers in the low-rank representation (DECELOR) can be formulated as an extension of the robust principal component analysis problem. This method fully utilizes the spatial continuity of the foreground regions. To achieve a more accurate detection, this paper integrates both the concave penalty function and the priori target rank information into a single optimization problem based on the DECELOR formulation. The optimization problem is efficiently solved by an alternating direction scheme. The computer numerical simulation results on the real-world scenes demonstrate the superiority of our method in terms of the effective handling of a wide range of complex scenarios.

Robust PCA Using Generalized Nonconvex Regularization

Recently, the robustification of principal component analysis (PCA) has attracted much research attention in numerous areas of science and engineering. The most popular and successful approach is to model the robust PCA problem as a low-rank matrix recovery problem in the presence of sparse corruption. With this model, the nuclear norm and $\ell _{1}$ -norm penalties are usually used for low-rank and sparsity promotion. Although the nuclear norm and $\ell _{1}$ -norm are favorable due to their convexity, they have a bias problem. In comparison, nonconvex penalties can be expected to yield better recovery performance. In this paper, we consider a formulation for robust PCA using generalized nonconvex penalties for low-rank and sparsity inducing. This nonconvex formulation is efficiently solved by a multi-block alternative direction method of multipliers (ADMM) algorithm. A sufficient condition for the convergence of this new ADMM algorithm has been derived. Furthermore, to address the important issue of nonconvex penalty selection, we have evaluated the new algorithm via numerical experiments in various low-rank and sparsity conditions. The results indicate that, “exact” recovery of the low-rank principle component can be achieved only by nonconvex regularization. MATLAB code is available at https://github.com/FWen/RPCA.git .

Turbo-Type Message Passing Algorithms for Compressed Robust Principal Component Analysis

Compressed robust principal component analysis (RPCA), in which a low-rank matrix $\boldsymbol {L}$ and a sparse matrix $\boldsymbol {S}$ are recovered from an underdetermined amount of noisy linear measurements of their sum $\boldsymbol {L}+\boldsymbol {S}$ , arises in various applications such as face recognition and video foreground/background separation. This problem can be solved by Bayesian inference based iterative algorithms. However, most existing Bayesian algorithms factorize $\boldsymbol {L}$ into the product of two rank- $r$ matrices, and estimate the two rank- $r$ matrices (rather than $\boldsymbol {L}$ itself) in the iterative process, where $r$ is the rank of $\boldsymbol {L}$ . On one hand, this factorization is not essential to the original problem and so may cause a potential performance loss. On the other hand, the existing Bayesian algorithms assume a certain probability model for the low-rank matrix $\boldsymbol {L}$ and the sparse matrix $\boldsymbol {S}$ , whereas the probability model of $\boldsymbol {L}$ and $\boldsymbol {S}$ is usually difficult to acquire in real applications. In this paper, we develop a Bayesian message passing algorithm, termed turbo-type message passing (TMP), for the compressed RPCA problem. We show that the proposed TMP algorithm significantly outperforms the state-of-the-art compressed RPCA algorithms, and requires a much lower computational complexity. We also show that TMP does not assume any prior probability model for $\boldsymbol {L}$ and $\boldsymbol {S}$ ; TMP even does not require the knowledge of the environmental information, such as the rank of $\boldsymbol {L}$ and the sparsity level of $\boldsymbol {S}$ . Therefore, TMP gives a promising approach for real applications of the compressed RPCA problem.

Online Schatten quasi-norm minimization for robust principal component analysis

The Schatten-p quasi-norm (0 < p < 1) is widely and successfully used as a superior rank surrogate to the well-known nuclear norm. It has been proved that the Schatten-p quasi-norm is admirable for the robust principal component analysis problem both in theory and in practice. However, existing algorithms for solving the Schatten-p quasi-norm minimization problem suffer from large memory requirement and high computational complexity, which is due to (1) they work in batch manner, which requires to load all the data in the memory during process; (2) they relay on singular value decomposition on a dense matrix in every iteration. Hence they are only able to handle problems of certain scale and highly inefficient in large scale cases. In this paper, by reformulating the Schatten-p quasi-norm in variational formulation, we design an online algorithm to solve the Schatten-p quasi-norm minimization problem, in which the memory cost is independent of the data sample size. Moreover, the new algorithm is totally free of singular value decomposition. Numerical study on subspace recovery demonstrates encouraging results of our algorithm compared with the widely used rank surrogate functions for online optimization. Besides, experiments on real video background subtraction indicate that the newly proposed online algorithm is extremely fast, and meets the real-time background subtraction for practical video processing tasks.

Static and Dynamic Robust PCA and Matrix Completion: A Review

Principal Components Analysis (PCA) is one of the most widely used dimension reduction techniques. Robust PCA (RPCA) refers to the problem of PCA when the data may be corrupted by outliers. Recent work by Cand{\`e}s, Wright, Li, and Ma defined RPCA as a problem of decomposing a given data matrix into the sum of a low-rank matrix (true data) and a sparse matrix (outliers). The column space of the low-rank matrix then gives the PCA solution. This simple definition has lead to a large amount of interesting new work on provably correct, fast, and practical solutions to RPCA. More recently, the dynamic (time-varying) version of the RPCA problem has been studied and a series of provably correct, fast, and memory efficient tracking solutions have been proposed. Dynamic RPCA (or robust subspace tracking) is the problem of tracking data lying in a (slowly) changing subspace while being robust to sparse outliers. This article provides an exhaustive review of the last decade of literature on RPCA and its dynamic counterpart (robust subspace tracking), along with describing their theoretical guarantees, discussing the pros and cons of various approaches, and providing empirical comparisons of performance and speed. A brief overview of the (low-rank) matrix completion literature is also provided (the focus is on works not discussed in other recent reviews). This refers to the problem of completing a low-rank matrix when only a subset of its entries are observed. It can be interpreted as a simpler special case of RPCA in which the indices of the outlier corrupted entries are known.

A proximal Peaceman–Rachford splitting method for solving the multi-block separable convex minimization problems

ABSTRACTThe Peaceman–Rachford splitting method (PRSM) is well studied for solving the two-block separable convex minimization problems with linear constraints recently. In this paper, we consider the separable convex minimization problem where its objective function is the sum of more than two functions without coupled variables, when applying the PRSM to this case directly, it is not necessarily convergent. To remedy this difficulty, we propose a proximal Peaceman–Rachford splitting method for solving this multi-block separable convex minimization problems, which updates the Lagrangian multiplier two times at each iteration and solves some subproblems parallelly. Under some mild conditions, we prove global convergence of the new method and analyse the worst-case convergence rate in both ergodic and nonergodic senses. In addition, we apply the new method to solve the robust principal component analysis problem and report some preliminary numerical results to indicate the feasibility and effectiveness of the proposed method.

Small target detection based on reweighted infrared patch‐image model

To further improve the effect of infrared small target detection, a reweighted infrared patch-image model is proposed. First, the authors point out that the nuclear norm in the infrared patch-image model could easily leave some sparse background edges in the target patch-image, leading to an inaccurate background estimation. Then, to overcome this defect, the reweighted nuclear norm is adopted to constrain the background patch-image, which could preserve the background edges better. Considering that some non-target sparse points could not be suppressed by only using l1 norm, the authors introduce the reweighted l1 norm to further enhance the sparsity of target image. Finally, the proposed model is formulated as a reweighted robust principal component analysis problem and solved by the inexact augmented Lagrangian multiplier method. Extensive experiments show that the proposed model outperforms the other six competitive methods in suppressing background clutter and detecting target.