Sparse Low-rank Approximation Research Articles

Multiclass Classification of Alzheimer's Disease Prodromal Stages using Sequential Feature Embeddings and Regularized Multikernel Support Vector Machine

The detection of patients in the cognitive normal (CN), mild cognitive impairment (MCI), and Alzheimer's disease (AD) stages of neurodegeneration is crucial for early treatment interventions. However, the heterogeneity of MCI data samples poses a challenge for CN vs. MCI vs. AD multiclass classification, as some samples are closer to AD while others are closer to CN in the feature space. Previous attempts to address this challenge produced inaccurate results, leading most frameworks to break the assessment into binary classification tasks such as AD vs. CN, AD vs. MCI, and CN vs. MCI. Other methods proposed sequential binary classifications such as CN vs. others and dividing others into AD vs. MCI. While those approaches may have yielded encouraging results, the sequential binary classification method makes interpretation and comparison with other frameworks challenging and subjective. Those frameworks exhibited varying accuracy scores for different binary tasks, making it unclear how to compare the model performance with other direct multiclass methods. Therefore, we introduce a classification framework comprising unsupervised ensemble manifold regularized sparse low-rank approximation and regularized multikernel support vector machine (SVM). This framework first extracts a joint feature embedding from MRI and PET neuroimaging features, which were then combined with the Apoe4, Adas11, MPACC digits, and Intracranial volume features using a regularized multikernel SVM. Using that framework, we achieved a state-of-the-art (SOTA) result in a CN vs. MCI vs. AD multiclass classification (mean accuracy: 84.87±6.09, F1 score: 84.83±6.12 vs 67.69). The methods generalize well to binary classification tasks, achieving SOTA results in all but the CN vs. MCI category, which was slightly lower than the best score by just 0.2%.

Sparse low-rank approximation of matrix and local preservation for unsupervised image feature selection

Sparse low-rank approximation of matrix and local preservation for unsupervised image feature selection

An Iterative Coordinate Descent Algorithm to Compute Sparse Low-Rank Approximations

In this paper, we describe a new algorithm to build a few sparse principal components from a given data matrix. Our approach does not explicitly create the covariance matrix of the data and can be viewed as an extension of the Kogbetliantz algorithm to build an approximate singular value decomposition for a few principal components. We show the performance of the proposed algorithm to recover sparse principal components on various datasets from the literature and perform dimensionality reduction for classification applications.

Dimension-wise sparse low-rank approximation of a matrix with application to variable selection in high-dimensional integrative analyzes of association

Many research proposals involve collecting multiple sources of information from a set of common samples, with the goal of performing an integrative analysis describing the associations between sources. We propose a method that characterizes the dominant modes of co-variation between the variables in two datasets while simultaneously performing variable selection. Our method relies on a sparse, low rank approximation of a matrix containing pairwise measures of association between the two sets of variables. We show that the proposed method shares a close connection with another group of methods for integrative data analysis – sparse canonical correlation analysis (CCA). Under some assumptions, the proposed method and sparse CCA aim to select the same subsets of variables. We show through simulation that the proposed method can achieve better variable selection accuracies than two state-of-the-art sparse CCA algorithms. Empirically, we demonstrate through the analysis of DNA methylation and gene expression data that the proposed method selects variables that have as high or higher canonical correlation than the variables selected by sparse CCA methods, which is a rather surprising finding given that objective function of the proposed method does not actually maximize the canonical correlation.

Analysis of parametric models

Parametric models in vector spaces are shown to possess an associated linear map, leading directly to reproducing kernel Hilbert spaces and affine/linear representations in terms of tensor products. From this map, analogues of correlation operators can be formed such that the associated linear map factorises the correlation. Its spectral decomposition and the associated Karhunen-Loeve- or proper orthogonal decomposition in a tensor product follow directly, including an extension to continuous spectra. It is shown that all factorisations of a certain class are unitarily equivalent, as well as that every factorisation induces a different representation, and vice versa. No particular assumptions are made on the parameter set, other than that the vector space of real valued functions on this set allows an appropriate inner product on a subspace. A completely equivalent spectral and factorisation analysis can be carried out in kernel space. The relevance of these abstract constructions is shown on a number of mostly familiar examples, thus unifying many such constructions under one theoretical umbrella. From the factorisation, one obtains tensor representations, which may be cascaded, leading to tensors of higher degree. When carried over to a discretised level in the form of a model order reduction, such factorisations allow sparse low-rank approximations which lead to very efficient computations especially in high dimensions.

Coupled multiview autoencoders with locality sensitivity for three-dimensional human pose estimation

Estimating three-dimensional (3D) human poses from a single camera is usually implemented by searching pose candidates with image descriptors. Existing methods usually suppose that the mapping from feature space to pose space is linear, but in fact, their mapping relationship is highly nonlinear, which heavily degrades the performance of 3D pose estimation. We propose a method to recover 3D pose from a silhouette image. It is based on the multiview feature embedding (MFE) and the locality-sensitive autoencoders (LSAEs). On the one hand, we first depict the manifold regularized sparse low-rank approximation for MFE and then the input image is characterized by a fused feature descriptor. On the other hand, both the fused feature and its corresponding 3D pose are separately encoded by LSAEs. A two-layer back-propagation neural network is trained by parameter fine-tuning and then used to map the encoded 2D features to encoded 3D poses. Our LSAE ensures a good preservation of the local topology of data points. Experimental results demonstrate the effectiveness of our proposed method.

Analysis on a Nonnegative Matrix Factorization and Its Applications

In this work we perform some mathematical analysis on a special nonnegative matrix trifactorization (NMF) and apply this NMF to some imaging and inverse problems. We will propose a sparse low-rank approximation of positive data and images in terms of tensor products of positive vectors and investigate its effectiveness in terms of the number of tensor products to be used in the approximation. A new multilevel analysis (MLA) framework is suggested to extract major components in the matrix representing structures of different resolutions but still preserve the positivity of the basis and sparsity of the approximation. We will also propose and formulate a semismooth Newton method based on primal-dual active sets for the nonnegative factorization. Numerical results are given to demonstrate the effectiveness of the proposed method at capturing features in images and structures of inverse problems under no a priori assumption on the underlying structure in the data as well as to provide a sparse low-rank representation of the data.

Ensemble manifold regularized sparse low-rank approximation for multiview feature embedding

In computer vision and pattern recognition researches, the studied objects are often characterized by multiple feature representations with high dimensionality, thus it is essential to encode that multiview feature into a unified and discriminative embedding that is optimal for a given task. To address this challenge, this paper proposes an ensemble manifold regularized sparse low-rank approximation (EMR-SLRA) algorithm for multiview feature embedding. The EMR-SLRA algorithm is based on the framework of least-squares component analysis, in particular, the low dimensional feature representation and the projection matrix are obtained by the low-rank approximation of the concatenated multiview feature matrix. By considering the complementary property among multiple features, EMR-SLRA simultaneously enforces the ensemble manifold regularization on the output feature embedding. In order to further enhance its robustness against the noise, the group sparsity is introduced into the objective formulation to impose direct noise reduction on the input multiview feature matrix. Since there is no closed-form solution for EMR-SLRA, this paper provides an efficient EMR-SLRA optimization procedure to obtain the output feature embedding. Experiments on the pattern recognition applications confirm the effectiveness of the EMR-SLRA algorithm compare with some other multiview feature dimensionality reduction approaches.

A Least-Squares Method for Sparse Low Rank Approximation of Multivariate Functions

In this paper, we propose a low-rank approximation method based on discrete least-squares for the approximation of a multivariate function from random, noisy-free observations. Sparsity inducing regularization techniques are used within classical algorithms for low-rank approximation in order to exploit the possible sparsity of low-rank approximations. Sparse low-rank approximations are constructed with a robust updated greedy algorithm which includes an optimal selection of regularization parameters and approximation ranks using cross validation techniques. Numerical examples demonstrate the capability of approximating functions of many variables even when very few function evaluations are available, thus proving the interest of the proposed algorithm for the propagation of uncertainties through complex computational models.

Low-Rank Approximations with Sparse Factors II: Penalized Methods with Discrete Newton-Like Iterations

In [SIAM J. Matrix Anal. Appl., 23 (2002), pp. 706--727], we developed numerical algorithms for computing sparse low-rank approximations of matrices, and we also provided a detailed error analysis of the proposed algorithms together with some numerical experiments. The low-rank approximations are constructed in a certain factored form with the degree of sparsity of the factors controlled by some user-specified parameters. In this paper, we cast the sparse low-rank approximation problem in the framework of penalized optimization problems. We discuss various approximation schemes for the penalized optimization problem which are more amenable to numerical computations. We also include some analysis to show the relations between the original optimization problem and the reduced one. We then develop a globally convergent discrete Newton-like iterative method for solving the approximate penalized optimization problems. We also compare the reconstruction errors of the sparse low-rank approximations computed by our new methods with those obtained using the methods in the earlier paper and several other existing methods for computing sparse low-rank approximations. Numerical examples show that the penalized methods are more robust and produce approximations with factors which have fewer columns and are sparser.

Linear low-rank approximation and nonlinear dimensionality reduction

We present our recent work on both linear and nonlinear data reduction methods and algorithms: for the linear case we discuss results on structure analysis of SVD of column-partitioned matrices and sparse low-rank approximation; for the nonlinear case we investigate methods for nonlinear dimensionality reduction and manifold learning. The problems we address have attracted great deal of interest in data mining and machine learning.

Low-Rank Approximations with Sparse Factors I: Basic Algorithms and Error Analysis

We consider the problem of computing low-rank approximations of matrices. The novel aspects of our approach are that we require the low-rank approximations to be written in a factorized form with sparse factors, and the degree of sparsity of the factors can be traded off for reduced reconstruction error by certain user-determined parameters. We give a detailed error analysis of our proposed algorithms and compare the computed sparse low-rank approximations with those obtained from singular value decomposition. We present numerical examples arising from some application areas to illustrate the efficiency and accuracy of our algorithms.