Low-rank Matrix Factorization Problem Research Articles

Boundary Conditions for Linear Exit Time Gradient Trajectories Around Saddle Points: Analysis and Algorithm

Gradient-related first-order methods have become the workhorse of large-scale numerical optimization problems. Many of these problems involve nonconvex objective functions with multiple saddle points, which necessitates an understanding of the behavior of discrete trajectories of first-order methods within the geometrical landscape of these functions. This paper concerns convergence of first-order discrete methods to a local minimum of nonconvex optimization problems that comprise strict-saddle points within the geometrical landscape. To this end, it focuses on analysis of discrete gradient trajectories around saddle neighborhoods, derives sufficient conditions under which these trajectories can escape strict-saddle neighborhoods in linear time, explores the contractive and expansive dynamics of these trajectories in neighborhoods of strict-saddle points that are characterized by gradients of moderate magnitude, characterizes the non-curving nature of these trajectories, and highlights the inability of these trajectories to re-enter the neighborhoods around strict-saddle points after exiting them. Based on these insights and analyses, the paper then proposes a simple variant of the vanilla gradient descent algorithm, termed Curvature Conditioned Regularized Gradient Descent (CCRGD) algorithm, which utilizes a check for an initial boundary condition to ensure its trajectories can escape strict-saddle neighborhoods in linear time. Convergence analysis of the CCRGD algorithm, which includes its rate of convergence to a local minimum, is also presented in the paper. Numerical experiments are then provided on a test function as well as a low-rank matrix factorization problem to evaluate the efficacy of the proposed algorithm.

A Low-Rank Model for Compressive Spectral Image Classification

Compressive sensing enables efficient acquisition of hyperspectral images (HSIs) by assuming high redundancy on natural scenes. Several reconstruction algorithms have been proposed to retrieve the underlying image, and most of them take advantage of the spatial and spectral correlations. However, reconstruction may not be necessary in certain applications such as land cover classification. Instead of knowing the full image, researchers are interested in features that could be extracted directly from the compressed measurements, which provide high inference capabilities. Low-rank (LR) matrix approximation has been widely used in feature extraction (FE), because it reduces the data dimension and computational cost. Therefore, in this paper, compressive hyperspectral imaging and FE are combined in a framework for HSI classification using an LR matrix approximation model. In the proposed framework, the compressed measurements are acquired from a single-pixel spectrometer. Instead of using the traditional high-complexity reconstruction model, an LR matrix factorization problem is formulated. The LR problem maximizes the posterior distribution with respect to the feature space and coefficients, and it is numerically solved based on an alternating optimization strategy. By incorporating spatial information, the numerical procedure minimizes the total variational of the feature coefficients subject to an orthogonality constraint for the feature space. Experiments on real HSIs show that the proposed approach can provide equally competitive classification results when compared to the traditional approach that performs FE and classification on the recovered images.

Symmetry, Saddle Points, and Global Optimization Landscape of Nonconvex Matrix Factorization

We propose a general theory for studying the landscape of nonconvex optimization with underlying symmetric structures for a class of machine learning problems (e.g., low-rank matrix factorization, phase retrieval, and deep linear neural networks). In particular, we characterize the locations of stationary points and the null space of Hessian matrices of the objective function via the lens of invariant groups. As a major motivating example, we apply the proposed general theory to characterize the global landscape of the nonconvex optimization in low-rank matrix factorization problem. We illustrate how the rotational symmetry group gives rise to infinitely many nonisolated strict saddle points and equivalent global minima of the objective function. By explicitly identifying all stationary points, we divide the entire parameter space into three regions: ( $ \mathcal {R}_{1}$ ) the region containing the neighborhoods of all strict saddle points where the objective has negative curvature; ( $ \mathcal {R}_{2}$ ) the region containing neighborhoods of all global minima, where the objective enjoys strong convexity along certain directions; and ( $ \mathcal {R}_{3}$ ) the complement of the above regions, where the gradient has sufficiently large magnitude. We further extend our result to the matrix sensing problem. Such global landscape implies that strong global convergence guarantees for popular iterative algorithms with arbitrary initial solutions.

Identification of structured state-space models

Identification of structured state-space (gray-box) model is popular for modeling physical and network systems. Due to the non-convex nature of the gray-box identification problem, good initial parameter estimates are crucial for successful applications. In this paper, the non-convex gray-box identification problem is reformulated as a structured low-rank matrix factorization problem by exploiting the rank and structured properties of a block Hankel matrix constructed by the system impulse response. To address the low-rank optimization problem, it is first transformed into a difference-of-convex (DC) formulation and then solved using the sequentially convex relaxation method. Compared with the classical gray-box identification methods like the prediction-error method (PEM), the new approach turns out to be more robust against converging to non-global minima, as supported by a simulation study. The developed identification can either be directly used for gray-box identification or provide an initial parameter estimate for the PEM.

Subspace Identification of Local Systems in One-Dimensional Homogeneous Networks

This note considers the identification of large-scale one-dimensional networks consisting of identical LTI dynamical systems. A subspace identification method is developed that only uses local input-output information and does not rely on knowledge about the local state interaction. The proposed identification method estimates the Markov parameters of a locally lifted system, following the state-space realization of a single subsystem. The Markov-parameter estimation is formulated as a rank minimization problem by exploiting the low-rank property and the two-layer Toeplitz structural property in the data equation, whereas the state-space realization of a single subsystem is formulated as a structured low-rank matrix-factorization problem. The effectiveness of the proposed identification method is demonstrated by simulation examples.

Structured Low-Rank Matrix Factorization for Haplotype Assembly

In matrix factorization problems, one seeks to decompose a data matrix into a product of two matrices—frequently, one captures meaningful information contained in the data, and the other specifies how this information is combined to generate the data matrix. In this paper, matrix factorization that arises in haplotype assembly, an important NP-hard problem in genomics, is studied. Haplotypes are sequences of chromosomal variations in an individual’s genome, which are of critical importance for understudying the individual’s susceptibility to various diseases. A novel formulation of haplotype assembly as the partially observed low-rank matrix factorization problem is proposed and efficiently solved via a modified gradient descent method that exploits salient structural properties of sequencing data. In particular, the observed matrix in the problem at hand contains noisy samples of the product of an informative matrix with rows having entries from a finite alphabet and a matrix with rows that are standard unit basis. Convergence of the proposed algorithm is analyzed and its performance tested on both synthetic and experimental data. The results demonstrate superior accuracy and speed of the proposed method as compared to state-of-the-art haplotype assembly techniques.

Robustness Analysis of Structured Matrix Factorization via Self-Dictionary Mixed-Norm Optimization

We are interested in a low-rank matrix factorization problem where one of the matrix factors has a special structure; specifically, its columns live in the unit simplex. This problem finds applications in diverse areas such as hyperspectral unmixing, video summarization, spectrum sensing, and blind speech separation. Prior works showed that such a factorization problem can be formulated as a self-dictionary sparse optimization problem under some assumptions that are considered realistic in many applications, and convex mixed norms were employed as optimization surrogates to realize the factorization in practice. Numerical results have shown that the mixed-norm approach demonstrates promising performance. In this letter, we conduct performance analysis of the mixed-norm approach under noise perturbations. Our result shows that using a convex mixed norm can indeed yield provably good solutions. More importantly, we also show that using nonconvex mixed (quasi) norms is more advantageous in terms of robustness against noise.