Low-rank Matrix Factorization Research Articles

Robust Matrix Completion with Heavy-tailed Noise

This paper studies noisy low-rank matrix completion in the presence of heavy-tailed and possibly asymmetric noise, where we aim to estimate an underlying low-rank matrix given a set of highly incomplete noisy entries. Though the matrix completion problem has attracted much attention in the past decade, there is still lack of theoretical understanding when the observations are contaminated by heavy-tailed noises. Prior theory falls short of explaining the empirical results and is unable to capture the optimal dependence of the estimation error on the noise level. In this paper, we adopt an adaptive Huber loss to accommodate heavy-tailed noise, which is robust against large and possibly asymmetric errors when the parameter in the Huber loss function is carefully designed to balance the Huberization biases and robustness to outliers. Then, we propose an efficient nonconvex algorithm via a balanced low-rank Burer-Monteiro matrix factorization and gradient descent with robust spectral initialization. We prove that under merely a bounded second-moment condition on the error distributions, rather than the sub-Gaussian assumption, the Euclidean errors of the iterates generated by the proposed algorithm decrease geometrically fast until achieving a minimax-optimal statistical estimation error, which has the same order as that in the sub-Gaussian case. The key technique behind this significant advancement is a powerful leave-one-out analysis framework. The theoretical results are corroborated by our numerical studies.

Paper Recommendation Using SPECTER with Low-Rank and Sparse Matrix Factorization

Paper Recommendation Using SPECTER with Low-Rank and Sparse Matrix Factorization

Few-Shot Font Generation With Weakly Supervised Localized Representations.

Automatic few-shot font generation aims to solve a well-defined, real-world problem because manual font designs are expensive and sensitive to the expertise of designers. Existing methods learn to disentangle style and content elements by developing a universal style representation for each font style. However, this approach limits the model in representing diverse local styles because it is unsuitable for complicated letter systems. For example, Chinese characters consist of a varying number of components (often called "radical") with a highly complex structure. In this paper, we propose a novel font generation method that learns localized styles, namely component-wise style representations, instead of universal styles. The proposed style representations enable synthesizing complex local details in text designs. However, learning component-wise styles solely from a few reference glyphs is infeasible when a target script has a large number of components, for example, over 200 for Chinese. To reduce the number of required reference glyphs, we represent component-wise styles by a product of component and style factors inspired by low-rank matrix factorization. Owing to the combination of strong representation and a compact factorization strategy, our method shows remarkably better few-shot font generation results (with only eight reference glyphs) than other state-of-the-art methods. Moreover, strong locality supervision was not utilized, such as the location of each component, skeleton, or strokes. The source code is available at https://github.com/clovaai/lffont.

A new approach for atmospheric turbulence removal using low-rank matrix factorization.

In this article, a novel method for removing atmospheric turbulence from a sequence of turbulent images and restoring a high-quality image is presented. Turbulence is modeled using two factors: the geometric transformation of pixel locations represents the distortion, and the varying pixel brightness represents spatiotemporal varying blur. The main framework of the proposed method involves the utilization of low-rank matrix factorization, which achieves the modeling of both the geometric transformation of pixels and the spatiotemporal varying blur through an iterative process. In the proposed method, the initial step involves the selection of a subset of images using the random sample consensus method. Subsequently, estimation of the mixture of Gaussian noise parameters takes place. Following this, a window is chosen around each pixel based on the entropy of the surrounding region. Within this window, the transformation matrix is locally estimated. Lastly, by considering both the noise and the estimated geometric transformations of the selected images, an estimation of a low-rank matrix is conducted. This estimation process leads to the production of a turbulence-free image. The experimental results were obtained from both real and simulated datasets. These results demonstrated the efficacy of the proposed method in mitigating substantial geometrical distortions. Furthermore, the method showcased the ability to improve spatiotemporal varying blur and effectively restore the details present in the original image.

Tensor Completion Using Bilayer Multimode Low-Rank Prior and Total Variation.

In this article, we propose a novel bilayer low-rankness measure and two models based on it to recover a low-rank (LR) tensor. The global low rankness of underlying tensor is first encoded by LR matrix factorizations (MFs) to the all-mode matricizations, which can exploit multiorientational spectral low rankness. Presumably, the factor matrices of all-mode decomposition are LR, since local low-rankness property exists in within-mode correlation. In the decomposed subspace, to describe the refined local LR structures of factor/subspace, a new low-rankness insight of subspace: a double nuclear norm scheme is designed to explore the so-called second-layer low rankness. By simultaneously representing the bilayer low rankness of the all modes of the underlying tensor, the proposed methods aim to model multiorientational correlations for arbitrary N -way ( N ≥ 3 ) tensors. A block successive upper-bound minimization (BSUM) algorithm is designed to solve the optimization problem. Subsequence convergence of our algorithms can be established, and the iterates generated by our algorithms converge to the coordinatewise minimizers in some mild conditions. Experiments on several types of public datasets show that our algorithm can recover a variety of LR tensors from significantly fewer samples than its counterparts.

Color image inpainting based on low-rank quaternion matrix factorization

Color image inpainting based on low-rank quaternion matrix factorization

A Low-Rank Bayesian Temporal Matrix Factorization for the Transfer Time Prediction Between Metro and Bus Systems

A Low-Rank Bayesian Temporal Matrix Factorization for the Transfer Time Prediction Between Metro and Bus Systems

Efficient Image Classification via Structured Low-Rank Matrix Factorization Regression

Efficient Image Classification via Structured Low-Rank Matrix Factorization Regression

Correntropy-Based Low-Rank Matrix Factorization With Constraint Graph Learning for Image Clustering.

This article proposes a novel low-rank matrix factorization model for semisupervised image clustering. In order to alleviate the negative effect of outliers, the maximum correntropy criterion (MCC) is incorporated as a metric to build the model. To utilize the label information to improve the clustering results, a constraint graph learning framework is proposed to adaptively learn the local structure of the data by considering the label information. Furthermore, an iterative algorithm based on Fenchel conjugate (FC) and block coordinate update (BCU) is proposed to solve the model. The convergence properties of the proposed algorithm are analyzed, which shows that the algorithm exhibits both objective sequential convergence and iterate sequential convergence. Experiments are conducted on six real-world image datasets, and the proposed algorithm is compared with eight state-of-the-art methods. The results show that the proposed method can achieve better performance in most situations in terms of clustering accuracy and mutual information.

Unsupervised learning for medical data: A review of probabilistic factorization methods.

We review popular unsupervised learning methods for the analysis of high-dimensional data encountered in, for example, genomics, medical imaging, cohort studies, and biobanks. We show that four commonly used methods, principal component analysis, K-means clustering, nonnegative matrix factorization, and latent Dirichlet allocation, can be written as probabilistic models underpinned by a low-rank matrix factorization. In addition to highlighting their similarities, this formulation clarifies the various assumptions and restrictions of each approach, which eases identifying the appropriate method for specific applications for applied medical researchers. We also touch upon the most important aspects of inference and model selection for the application of these methods to health data.

Adaptive Rank-One Matrix Completion Using Sum of Outer Products

Matrix completion refers to recovering a matrix from a small subset of its entries. It is an important topic because numerous real-world data can be modeled as low-rank matrices. One popular approach for matrix completion is based on low-rank matrix factorization, but it requires knowing the matrix rank, which is difficult to accurately determine in many practical scenarios. We propose a novel algorithm based on rank-one approximation that a matrix can be decomposed as a sum of outer products. The key idea is to find the basis vectors of the underlying matrix according to the observed entries, and gradually increase the vector number until an appropriate rank estimate is reached. In contrast to the conventional rank-one schemes that employ unchanging rank-one basis matrices, our algorithm performs completion from the vector viewpoint and is able to generate continuously updated rank-one basis matrices. Besides, we theoretically show that the developed method has a linear convergence rate and a smaller recovery error than existing rank-one based algorithms. Experimental results using both synthetic data and real-world images demonstrate that our solution has the best recovery performance among the competing algorithms when the observations are contaminated by Gaussian noise.

Exponential Signal Reconstruction With Deep Hankel Matrix Factorization.

Exponential function is a basic form of temporal signals, and how to fast acquire this signal is one of the fundamental problems and frontiers in signal processing. To achieve this goal, partial data may be acquired but result in severe artifacts in its spectrum, which is the Fourier transform of exponentials. Thus, reliable spectrum reconstruction is highly expected in the fast data acquisition in many applications, such as chemistry, biology, and medical imaging. In this work, we propose a deep learning method whose neural network structure is designed by imitating the iterative process in the model-based state-of-the-art exponentials' reconstruction method with the low-rank Hankel matrix factorization. With the experiments on synthetic data and realistic biological magnetic resonance signals, we demonstrate that the new method yields much lower reconstruction errors and preserves the low-intensity signals much better than compared methods.

Hierarchical nuclear norm penalization for multi-view data integration.

The prevalence of data collected on the same set of samples from multiple sources (i.e., multi-view data) has prompted significant development of data integration methods based on low-rank matrix factorizations. These methods decompose signal matrices from each view into the sum of shared and individual structures, which are further used for dimension reduction, exploratory analyses, and quantifying associations across views. However, existing methods have limitations in modeling partially-shared structures due to either too restrictive models, or restrictive identifiability conditions. To address these challenges, we propose a new formulation for signal structures that include partially-shared signals based on grouping the views into so-called hierarchical levels with identifiable guarantees under suitable conditions. The proposed hierarchy leads us to introduce a new penalty, hierarchical nuclear norm (HNN), for signal estimation. In contrast to existing methods, HNN penalization avoids scores and loadings factorization of the signals and leads to a convex optimization problem, which we solve using a dual forward-backward algorithm. We propose a simple refitting procedure to adjust the penalization bias and develop an adapted version of bi-cross-validation for selecting tuning parameters. Extensive simulation studies and analysis of the genotype-tissue expression data demonstrate the advantages of our method over existingalternatives.

Matrix Factorization Techniques in Machine Learning, Signal Processing, and Statistics

Compressed sensing is an alternative to Shannon/Nyquist sampling for acquiring sparse or compressible signals. Sparse coding represents a signal as a sparse linear combination of atoms, which are elementary signals derived from a predefined dictionary. Compressed sensing, sparse approximation, and dictionary learning are topics similar to sparse coding. Matrix completion is the process of recovering a data matrix from a subset of its entries, and it extends the principles of compressed sensing and sparse approximation. The nonnegative matrix factorization is a low-rank matrix factorization technique for nonnegative data. All of these low-rank matrix factorization techniques are unsupervised learning techniques, and can be used for data analysis tasks, such as dimension reduction, feature extraction, blind source separation, data compression, and knowledge discovery. In this paper, we survey a few emerging matrix factorization techniques that are receiving wide attention in machine learning, signal processing, and statistics. The treated topics are compressed sensing, dictionary learning, sparse representation, matrix completion and matrix recovery, nonnegative matrix factorization, the Nyström method, and CUR matrix decomposition in the machine learning framework. Some related topics, such as matrix factorization using metaheuristics or neurodynamics, are also introduced. A few topics are suggested for future investigation in this article.

Multivariate Transfer Passenger Flow Forecasting with Data Imputation by Joint Deep Learning and Matrix Factorization

Accurate forecasting of the future transfer passenger flow from historical data is essential for helping travelers to adjust their trips, optimal resource allocation and alleviating traffic congestion. However, current studies have mainly emphasized predicting traffic parameters for a single type of transport, while lacking research into transfer passenger flow influenced by multiple factors across different transport modes. Additionally, efficient traffic prediction relies on high-quality traffic data, yet data loss issues are inevitable but often ignored. To fill these gaps, we present for the first time a reliable joint long short-term memory with matrix factorization deep learning model (i.e., Joint-IF) for accurate imputation and forecasting of transfer passenger flow between metro and bus. This hybrid Joint-IF model uses a repair-before-prediction strategy to deliver the final high-quality outputs. In particular, we simulate a variety of missing combinations under the natural conditions and apply a low-rank matrix factorization to infer those lost values. In addition, we investigate the effects of crucial parameters and spatiotemporal features on transfer flow prediction. To validate the effectiveness of Joint-IF, a large series of experiments are carried out for models’ comparison and validation on the real-world transfer passenger flow dataset of the Shenzhen public transport system, and the results show that the proposed Joint-IF performs better for both imputation and forecasting of transfer passenger flow relative to the baseline models in terms of accuracy and stability.

Small Target Detection Method Based on Low-Rank Sparse Matrix Factorization for Side-Scan Sonar Images

Target detection in side-scan sonar images plays a significant role in ocean engineering. However, the target images are usually severely interfered by the complex background and strong environmental noise, which makes it difficult to extract robust features from small targets and makes the target detection task quite challenging. In this paper, a novel small target detection method in sonar images is proposed based on the low-rank sparse matrix factorization. Initially, the side-scan sonar images are preprocessed so as to highlight the individual differences of the target. Then, the problems of target feature extraction and noise removal are characterized as the problem of matrix decomposition. An improved Robust Principal Component Analysis algorithm is used to extract target information, and the fast proximal gradient method is used to optimize the solution. The original sonar image is reconstructed into the low-rank background matrix, the sparse target matrix, and the noise matrix. Eventually, a morphological operation is used to filter out the noise and refine the target edges in the target matrix for improving the accuracy of target detection. Experimental results show that the proposed method not only achieves better detection performance in comparison to the conventional baseline algorithms but also performs robustly in various signal-to-clutter ratio conditions.

Boundary Conditions for Linear Exit Time Gradient Trajectories Around Saddle Points: Analysis and Algorithm

Gradient-related first-order methods have become the workhorse of large-scale numerical optimization problems. Many of these problems involve nonconvex objective functions with multiple saddle points, which necessitates an understanding of the behavior of discrete trajectories of first-order methods within the geometrical landscape of these functions. This paper concerns convergence of first-order discrete methods to a local minimum of nonconvex optimization problems that comprise strict-saddle points within the geometrical landscape. To this end, it focuses on analysis of discrete gradient trajectories around saddle neighborhoods, derives sufficient conditions under which these trajectories can escape strict-saddle neighborhoods in linear time, explores the contractive and expansive dynamics of these trajectories in neighborhoods of strict-saddle points that are characterized by gradients of moderate magnitude, characterizes the non-curving nature of these trajectories, and highlights the inability of these trajectories to re-enter the neighborhoods around strict-saddle points after exiting them. Based on these insights and analyses, the paper then proposes a simple variant of the vanilla gradient descent algorithm, termed Curvature Conditioned Regularized Gradient Descent (CCRGD) algorithm, which utilizes a check for an initial boundary condition to ensure its trajectories can escape strict-saddle neighborhoods in linear time. Convergence analysis of the CCRGD algorithm, which includes its rate of convergence to a local minimum, is also presented in the paper. Numerical experiments are then provided on a test function as well as a low-rank matrix factorization problem to evaluate the efficacy of the proposed algorithm.

Learning and Compressing: Low-Rank Matrix Factorization for Deep Neural Network Compression

Recently, the deep neural network (DNN) has become one of the most advanced and powerful methods used in classification tasks. However, the cost of DNN models is sometimes considerable due to the huge sets of parameters. Therefore, it is necessary to compress these models in order to reduce the parameters in weight matrices and decrease computational consumption, while maintaining the same level of accuracy. In this paper, in order to deal with the compression problem, we first combine the loss function and the compression cost function into a joint function, and optimize it as an optimization framework. Then we combine the CUR decomposition method with this joint optimization framework to obtain the low-rank approximation matrices. Finally, we narrow the gap between the weight matrices and the low-rank approximations to compress the DNN models on the image classification task. In this algorithm, we not only solve the optimal ranks by enumeration, but also obtain the compression result with low-rank characteristics iteratively. Experiments were carried out on three public datasets under classification tasks. Comparisons with baselines and current state-of-the-art results can conclude that our proposed low-rank joint optimization compression algorithm can achieve higher accuracy and compression ratios.

Efficient and Rigorous Modeling of Light Scattering From Complex Plasmonic Nanostructures With Multiple Incident Excitations

Optical responses to incident lights with different incident directional angles or polarization angles are often of concern in the analysis of light scattering from plasmonic nanostructures. This leads to a multiple-excitation problem for the modeling and prediction of light scattering. An efficient and rigorous numerical method is presented to deal with this problem. In this method, the hybrid finite element and boundary integral (FEBI) formula is adopted to model the interior complex plasmonic materials and the exterior scattering space rigorously, and a low-rank compressed matrix factorization algorithm based on the FEBI framework is proposed to deal with the multiple-excitation problem efficiently. Numerical experiments are performed to illustrate the effectiveness of the proposed method.

Factorizer: A scalable interpretable approach to context modeling for medical image segmentation.

Convolutional Neural Networks (CNNs) with U-shaped architectures have dominated medical image segmentation, which is crucial for various clinical purposes. However, the inherent locality of convolution makes CNNs fail to fully exploit global context, essential for better recognition of some structures, e.g., brain lesions. Transformers have recently proven promising performance on vision tasks, including semantic segmentation, mainly due to their capability of modeling long-range dependencies. Nevertheless, the quadratic complexity of attention makes existing Transformer-based models use self-attention layers only after somehow reducing the image resolution, which limits the ability to capture global contexts present at higher resolutions. Therefore, this work introduces a family of models, dubbed Factorizer, which leverages the power of low-rank matrix factorization for constructing an end-to-end segmentation model. Specifically, we propose a linearly scalable approach to context modeling, formulating Nonnegative Matrix Factorization (NMF) as a differentiable layer integrated into a U-shaped architecture. The shifted window technique is also utilized in combination with NMF to effectively aggregate local information. Factorizers compete favorably with CNNs and Transformers in terms of accuracy, scalability, and interpretability, achieving state-of-the-art results on the BraTS dataset for brain tumor segmentation and ISLES'22 dataset for stroke lesion segmentation. Highly meaningful NMF components give an additional interpretability advantage to Factorizers over CNNs and Transformers. Moreover, our ablation studies reveal a distinctive feature of Factorizers that enables a significant speed-up in inference for a trained Factorizer without any extra steps and without sacrificing much accuracy. The code and models are publicly available at https://github.com/pashtari/factorizer.