Low-rank Projection Research Articles

Compact Model Training by Low-Rank Projection With Energy Transfer.

Low-rankness plays an important role in traditional machine learning but is not so popular in deep learning. Most previous low-rank network compression methods compress networks by approximating pretrained models and retraining. However, the optimal solution in the Euclidean space may be quite different from the one with low-rank constraint. A well-pretrained model is not a good initialization for the model with low-rank constraints. Thus, the performance of a low-rank compressed network degrades significantly. Compared with other network compression methods such as pruning, low-rank methods attract less attention in recent years. In this article, we devise a new training method, low-rank projection with energy transfer (LRPET), that trains low-rank compressed networks from scratch and achieves competitive performance. We propose to alternately perform stochastic gradient descent training and projection of each weight matrix onto the corresponding low-rank manifold. Compared to retraining on the compact model, this enables full utilization of model capacity since solution space is relaxed back to Euclidean space after projection. The matrix energy (the sum of squares of singular values) reduction caused by projection is compensated by energy transfer. We uniformly transfer the energy of the pruned singular values to the remaining ones. We theoretically show that energy transfer eases the trend of gradient vanishing caused by projection. In modern networks, a batch normalization (BN) layer can be merged into the previous convolution layer for inference, thereby influencing the optimal low-rank approximation (LRA) of the previous layer. We propose BN rectification to cut off its effect on the optimal LRA, which further improves the performance. Comprehensive experiments on CIFAR-10 and ImageNet have justified that our method is superior to other low-rank compression methods and also outperforms recent state-of-the-art pruning methods. For object detection and semantic segmentation, our method still achieves good compression results. In addition, we combine LRPET with quantization and hashing methods and achieve even better compression than the original single method. We further apply it in Transformer-based models to demonstrate its transferability. Our code is available at https://github.com/BZQLin/LRPET.

Exploring combinations of dimensionality reduction, transfer learning, and regularization methods for predicting binary phenotypes with transcriptomic data

BackgroundNumerous transcriptomic-based models have been developed to predict or understand the fundamental mechanisms driving biological phenotypes. However, few models have successfully transitioned into clinical practice due to challenges associated with generalizability and interpretability. To address these issues, researchers have turned to dimensionality reduction methods and have begun implementing transfer learning approaches.MethodsIn this study, we aimed to determine the optimal combination of dimensionality reduction and regularization methods for predictive modeling. We applied seven dimensionality reduction methods to various datasets, including two supervised methods (linear optimal low-rank projection and low-rank canonical correlation analysis), two unsupervised methods [principal component analysis and consensus independent component analysis (c-ICA)], and three methods [autoencoder (AE), adversarial variational autoencoder, and c-ICA] within a transfer learning framework, trained on > 140,000 transcriptomic profiles. To assess the performance of the different combinations, we used a cross-validation setup encapsulated within a permutation testing framework, analyzing 30 different transcriptomic datasets with binary phenotypes. Furthermore, we included datasets with small sample sizes and phenotypes of varying degrees of predictability, and we employed independent datasets for validation.ResultsOur findings revealed that regularized models without dimensionality reduction achieved the highest predictive performance, challenging the necessity of dimensionality reduction when the primary goal is to achieve optimal predictive performance. However, models using AE and c-ICA with transfer learning for dimensionality reduction showed comparable performance, with enhanced interpretability and robustness of predictors, compared to models using non-dimensionality-reduced data.ConclusionThese findings offer valuable insights into the optimal combination of strategies for enhancing the predictive performance, interpretability, and generalizability of transcriptomic-based models.

Euclidean-Riemannian Joint Low-Rank Projections for Industrial Process Monitoring

Euclidean-Riemannian Joint Low-Rank Projections for Industrial Process Monitoring

Joint Discriminative Analysis With Low-Rank Projection for Finger Vein Feature Extraction

Joint Discriminative Analysis With Low-Rank Projection for Finger Vein Feature Extraction

Dual Discriminative Low-Rank Projection Learning for Robust Image Classification

Numerous methods have exploited projection learning to extract low-dimensional features for image classification. Some projection learning methods integrate low-rank matrix recovery into classification models to equip the projection subspace with discrimination and robustness against corruption. However, these methods cannot directly recover “clean” components from the new datum in a low-dimensional subspace. Additionally, they are sensitive to the selection of projection dimensions. To overcome these shortcomings, we propose a dual discriminative low-rank projection learning framework for robust image classification. Specifically, the proposed method learns a low-rank projection and a semi-orthogonal projection to recover “clean” components from the original data and simultaneously obtain a low-dimensional subspace. Thereafter, to preserve discriminative information in the low-dimensional subspace, an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2,1</sub> -norm term is constructed by concentrating the projected intra-class samples around their adaptive class centroids. Regression-based terms are appended using the low-dimensional features extracted from the recovered clean data and the class centroids for more accurate classification. Experiments on five public databases with various corruptions demonstrate that the proposed method can robustly classify image data despite a small training sample sizes and gross corruption. The superiority of the proposed method is further verified on the large-scale PubFig83 database, on which it achieves an 87.58% classification accuracy.

Gradient descent for deep matrix factorization: Dynamics and implicit bias towards low rank

In deep learning, it is common to use more network parameters than training points. In such scenario of over-parameterization, there are usually multiple networks that achieve zero training error so that the training algorithm induces an implicit bias on the computed solution. In practice, (stochastic) gradient descent tends to prefer solutions which generalize well, which provides a possible explanation of the success of deep learning. In this paper we analyze the dynamics of gradient descent in the simplified setting of linear networks and of an estimation problem. Although we are not in an overparameterized scenario, our analysis nevertheless provides insights into the phenomenon of implicit bias. In fact, we derive a rigorous analysis of the dynamics of vanilla gradient descent, and characterize the dynamical convergence of the spectrum. We are able to accurately locate time intervals where the effective rank of the iterates is close to the effective rank of a low-rank projection of the ground-truth matrix. In practice, those intervals can be used as criteria for early stopping if a certain regularity is desired. We also provide empirical evidence for implicit bias in more general scenarios, such as matrix sensing and random initialization. This suggests that deep learning prefers trajectories whose complexity (measured in terms of effective rank) is monotonically increasing, which we believe is a fundamental concept for the theoretical understanding of deep learning.

Latent Low-Rank Projection Learning with Graph Regularization for Feature Extraction of Hyperspectral Images

Due to the great benefit of rich spectral information, hyperspectral images (HSIs) have been successfully applied in many fields. However, some problems of concern also limit their further applications, such as high dimension and expensive labeling. To address these issues, an unsupervised latent low-rank projection learning with graph regularization (LatLRPL) method is presented for feature extraction and classification of HSIs in this paper, in which discriminative features can be extracted from the view of latent space by decomposing the latent low-rank matrix into two different matrices, also benefiting from the preservation of intrinsic subspace structures by the graph regularization. Different from the graph embedding-based methods that need two phases to obtain the low-dimensional projections, one step is enough for LatLRPL by constructing the integrated projection learning model, reducing the complexity and simultaneously improving the robustness. To improve the performance, a simple but effective strategy is exploited by conducting the local weighted average on the pixels in a sliding window for HSIs. Experiments on the Indian Pines and Pavia University datasets demonstrate the superiority of the proposed LatLRPL method.

A Unified Federated DNNs Framework for Heterogeneous Mobile Devices

Mobile devices can generate a tremendous amount of unique data, and thus, create countless opportunities for deep learning tasks. Due to the concerns of data privacy, it is often impractical to log all the data to a central server for training a satisfactory model. In federated learning, the participating devices can train a shared global model collaboratively while keeping their data locally. However, it is not a trivial task to train the deep neural networks (DNNs) with millions and billions of parameters on resource-constrained mobile devices in a federated manner. We replace each fully connected (FC) layer with two low-rank projection matrices to compact the DNNs model, and establish a global error function to recover the outputs of the compressed DNNs model. Then, we design a communication-efficient federated optimation Algorithm to reduce communication cost further. Considering that the heterogeneous devices may run different models at the same time, we devise three different training patterns to integrate the heterogeneous devices running different models. We conduct extensive experiments on both independently identically distribution (IID) and non-IID data sets. The experimental results demonstrate that the proposed framework can significantly reduce the number of parameters and communication cost while maintaining performance.

Robust low-rank representation via residual projection for image classification

Noisy interference and high dimensionality are the main challenges in image classification. In this paper, a robust low-rank representation via residual projection is proposed. Different from traditional low-rank representation methods, which regard residues as noisy data or outliers, our proposed method tries to find robust features by learning the structure of residues. That is, we regard residues as a measurement of input data and its low-rank representation. Therefore, robust low-rank projections are then learned from residues to find a good matching structure between input data and its low-rank representation, which leads to accomplishing dimensionality reduction and noise suppression simultaneously. Experimental results on several public image datasets, such as ORL, Yale, YaleB, AR, LFW, COIL20, COIL100, and Caltech101, demonstrate that our proposed method can achieve higher classification accuracy than the state-of-the-art (SOTA) methods in the image classification task. Furthermore, the classification results are much better than those of comparison methods in corrupted datasets, demonstrating that our proposed method can achieve the best robust low-rank representation in noisy datasets.

An Evaluation of Supervised Dimensionality Reduction For Large Scale Data

Experimenters today frequently quantify millions or even billions of characteristics (measurements) each sample to address critical biological issues, in the hopes that machine learning tools would be able to make correct data-driven judgments. An efficient analysis requires a low-dimensional representation that preserves the differentiating features in data whose size and complexity are orders of magnitude apart (e.g., if a certain ailment is present in the person's body). While there are several systems that can handle millions of variables and yet have strong empirical and conceptual guarantees, there are few that can be clearly understood. This research presents an evaluation of supervised dimensionality reduction for large scale data. We provide a methodology for expanding Principal Component Analysis (PCA) by including category moment estimations in low-dimensional projections. Linear Optimum Low-Rank (LOLR) projection, the cheapest variant, includes the class-conditional means. We show that LOLR projections and its extensions enhance representations of data for future classifications while retaining computing flexibility and reliability using both experimental and simulated data benchmark. When it comes to accuracy, LOLR prediction outperforms other modular linear dimension reduction methods that require much longer computation times on conventional computers. LOLR uses more than 150 million attributes in brain image processing datasets, and many genome sequencing datasets have more than half a million attributes.

Orthogonal Low-Rank Projection Learning for Robust Image Feature Extraction

Projecting the original data into a low-dimensional target space for feature extraction is a common method. Recently, presentation-based approaches have been widely concerned and many feature extraction algorithms based on this have been proposed. However, in the process of acquiring real data, the pollution of complex noise cannot always be avoided, which will greatly increase the difficulty of feature extraction and even lead to failed feature extraction results. Thus, a robust image feature extraction model based on Orthogonal Low-Rank Projection Learning (OLRPL) is proposed, in which the introduction of orthogonal matrix can encourage the preservation of the main components of the sample. Particularly, the row sparsity constraint introduced on the projection matrix can encourage the features to be more compact, discriminative and interpretable. In particular, the Weighted Truncated Schatten <i>p</i>-norm (WTSN) is proposed to better solve the optimization problem of low-rank constraints. At the same time, the correntropy is applied in OLRPL to suppress the complex noise in the data and thus improve the robustness of the model. Finally, we specially design a robust classification loss function so that our model can be fitted the supervised scene effectively. Experiments on five general databases have proved that OLRPL has better effectiveness and robustness than existing advanced methods.

Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias Towards Low Rank

In deep learning, it is common to use more network parameters than training points. In such scenarioof over-parameterization, there are usually multiple networks that achieve zero training error so that thetraining algorithm induces an implicit bias on the computed solution. In practice, (stochastic) gradientdescent tends to prefer solutions which generalize well, which provides a possible explanation of thesuccess of deep learning. In this paper we analyze the dynamics of gradient descent in the simplifiedsetting of linear networks and of an estimation problem. Although we are not in an overparameterizedscenario, our analysis nevertheless provides insights into the phenomenon of implicit bias. In fact, wederive a rigorous analysis of the dynamics of vanilla gradient descent, and characterize the dynamicalconvergence of the spectrum. We are able to accurately locate time intervals where the effective rankof the iterates is close to the effective rank of a low-rank projection of the ground-truth matrix. Inpractice, those intervals can be used as criteria for early stopping if a certain regularity is desired. Wealso provide empirical evidence for implicit bias in more general scenarios, such as matrix sensing andrandom initialization. This suggests that deep learning prefers trajectories whose complexity (measuredin terms of effective rank) is monotonically increasing, which we believe is a fundamental concept for thetheoretical understanding of deep learning.

Two-Step Approach to Processing Raw Strain Monitoring Data for Damage Detection of Structures under Operational Conditions.

Strain data of structural health monitoring is a prospective to be made full use of, because it reflects the stress peak and fatigue, especially sensitive to local stress redistribution, which is the probably damage in the vicinity of the sensor. For decoupling structural damage and masking effects caused by operational conditions to eliminate the adverse impacts on strain-based damage detection, small time-scale structural events, i.e., the short-term dynamic strain responses, are analyzed in this paper by employing unsupervised modeling. A two-step approach to successively processing the raw strain monitoring data in the sliding time window is presented, consisting of the wavelet-based initial feature extraction step and the decoupling step to draw damage indicators. The principal component analysis and a low-rank property-based subspace projection method are adopted as two alternative decoupling methodologies. The approach’s feasibility and robustness are substantiated by analyzing the strain monitoring data from a customized truss experiment to successfully remove the masking effects of operating loads and identify local damages even concerning accommodating situations of missing data and limited measuring points. This work also sheds light on the merit of a low-rank property to separate structural damages from masking effects by comparing the performances of the two optional decoupling methods of the distinct rationales.

Image fusion using a multi-level image decomposition and fusion method.

In recent years, image fusion has emerged as an important research field due to its various applications. Images acquired by different sensors have significant differences in feature representation due to the different imaging principles. Taking visible and infrared image fusion as an example, visible images contain abundant texture details with high spatial resolution. In contrast, infrared images can obtain clear target contour information according to the principle of thermal radiation, and work well in all day/night and all weather conditions. Most existing methods employ the same feature extraction algorithm to get the feature information from visible and infrared images, ignoring the differences among these images. Thus, this paper proposes what we believe to be a novel fusion method based on a multi-level image decomposition method and deep learning fusion strategy for multi-type images. In image decomposition, we not only utilize a multi-level extended approximate low-rank projection matrix learning decomposition method to extract salient feature information from both visible and infrared images, but also apply a multi-level guide filter decomposition method to obtain texture information in visible images. In image fusion, a novel fusion strategy based on a pretrained ResNet50 network is presented to fuse multi-level feature information from both visible and infrared images into corresponding multi-level fused feature information, so as to improve the quality of the final fused image. The proposed method is evaluated subjectively and objectively in a large number of experiments. The experimental results demonstrate that the proposed method exhibits better fusion performance than other existing methods.

Latent Discriminative Low-Rank Projection for Visual Dimension Reduction in Green Internet of Things

Internet of Things (IoT) terminals have been widely deployed for data sensing and analysis, and efficient data storage and transmission plays an important role in green IoT due to the explosive data growth. To simultaneously reduce the data dimension and preserves the discriminative intrinsic knowledge of data, this paper develops a novel latent discriminative low-rank projection (LDLRP) method for visual dimension reduction. Specifically, a data self-expressiveness model is established by considering the low-rank and discriminative similarity relations of data. Then, the developed model is efficiently optimized and solved via an augmented Lagrange multiplier (ALM) based-iterative algorithm, and a block-diagonal solution can be found for intraclass and interclass graph construction. Afterwards, a discriminative dimension reduced-subspace is derived by concurrently minimizing the intraclass scatter and maximizing the interclass scatter. The experimental results on benchmark datasets show that the proposed method can learn discriminative lower-dimensional expressions of high-dimensional data, and yield promising classification accuracy compared with several state-of-the-art methods. Hence, the effectiveness and efficiency of proposed method in data dimension reduction and knowledge preservation are verified, which will facilitate efficient data storage, transmission and application in green IoT.

Supervised dimensionality reduction for big data

To solve key biomedical problems, experimentalists now routinely measure millions or billions of features (dimensions) per sample, with the hope that data science techniques will be able to build accurate data-driven inferences. Because sample sizes are typically orders of magnitude smaller than the dimensionality of these data, valid inferences require finding a low-dimensional representation that preserves the discriminating information (e.g., whether the individual suffers from a particular disease). There is a lack of interpretable supervised dimensionality reduction methods that scale to millions of dimensions with strong statistical theoretical guarantees. We introduce an approach to extending principal components analysis by incorporating class-conditional moment estimates into the low-dimensional projection. The simplest version, Linear Optimal Low-rank projection, incorporates the class-conditional means. We prove, and substantiate with both synthetic and real data benchmarks, that Linear Optimal Low-Rank Projection and its generalizations lead to improved data representations for subsequent classification, while maintaining computational efficiency and scalability. Using multiple brain imaging datasets consisting of more than 150 million features, and several genomics datasets with more than 500,000 features, Linear Optimal Low-Rank Projection outperforms other scalable linear dimensionality reduction techniques in terms of accuracy, while only requiring a few minutes on a standard desktop computer.

Clutter Removal in Through-the-Wall Radar Imaging Using Sparse Autoencoder With Low-Rank Projection

Through-the-wall radar imaging is a sensing technology that can be used by first responders to see through obscure barriers during search-and-rescue missions or deployed by law enforcement and military personnel to maintain situational awareness during tactical operations. However, the strong reflections from the front wall and other obstacles render the detection of stationary targets very difficult. In this article, a learning-based approach is proposed to mitigate the effect of the wall and background clutter. A sparse autoencoder with a low-rank projection is developed to mitigate the wall clutter and recover the target signal. The weights of the proposed autoencoder are determined by solving an augmented Lagrange multiplier optimization problem, and the regularization parameters are estimated using the Bayesian optimization technique. Experiments using real data from a stepped-frequency radar were conducted to illustrate its effectiveness for wall clutter removal. The results show that the proposed method achieves superior performance compared with the existing approaches.

On the Convergence of Projected-Gradient Methods with Low-Rank Projections for Smooth Convex Minimization over Trace-Norm Balls and Related Problems

Smooth convex minimization over the unit trace-norm ball is an important optimization problem in machine learning, signal processing, statistics, and other fields that underlies many tasks in which one wishes to recover a low-rank matrix given certain measurements. While first-order methods for convex optimization enjoy optimal convergence rates, they require in the worst-case to compute a full-rank SVD on each iteration, in order to compute the Euclidean projection onto the trace-norm ball. These full-rank SVD computations, however, prohibit the application of such methods to large-scale problems. A simple and natural heuristic to reduce the computational cost of such methods is to approximate the Euclidean projection using only a low-rank SVD. This raises the question if, and under what conditions, this simple heuristic can indeed result in provable convergence to the optimal solution. In this paper we show that any optimal solution is a center of a Euclidean ball inside which the projected-gradient mapping admits a rank that is at most the multiplicity of the largest singular value of the gradient vector at this optimal point. Moreover, the radius of the ball scales with the spectral gap of this gradient vector. We show how this readily implies the local convergence (i.e., from a “warm-start" initialization) of standard first-order methods such as the projected-gradient method and accelerated gradient methods, using only low-rank SVD computations. We also quantify the effect of “over-parameterization," i.e., using SVD computations with higher rank, on the radius of this ball, showing it can increase dramatically with moderately larger rank. We extend our results also to the setting of smooth convex minimization with trace-norm regularization and smooth convex optimization over bounded-trace positive semidefinite matrices. Our theoretical investigation is supported by concrete empirical evidence that demonstrates the correct convergence of first-order methods with low-rank projections for the matrix completion task on real-world datasets.

Statistical Inferences of Linear Forms for Noisy Matrix Completion

AbstractWe introduce a flexible framework for making inferences about general linear forms of a large matrix based on noisy observations of a subset of its entries. In particular, under mild regularity conditions, we develop a universal procedure to construct asymptotically normal estimators of its linear forms through double-sample debiasing and low-rank projection whenever an entry-wise consistent estimator of the matrix is available. These estimators allow us to subsequently construct confidence intervals for and test hypotheses about the linear forms. Our proposal was motivated by a careful perturbation analysis of the empirical singular spaces under the noisy matrix completion model which might be of independent interest. The practical merits of our proposed inference procedure are demonstrated on both simulated and real-world data examples.

Discriminative low-rank projection for robust subspace learning

The robustness to outliers, noises, and corruptions has been paid more attention recently to increase the performance in linear feature extraction and image classification. As one of the most effective subspace learning methods, low-rank representation (LRR) can improve the robustness of an algorithm by exploring the global representative structure information among the samples. However, the traditional LRR cannot project the training samples into low-dimensional subspace with supervised information. Thus, in this paper, we integrate the properties of LRR with supervised dimensionality reduction techniques to obtain optimal low-rank subspace and discriminative projection at the same time. To achieve this goal, we proposed a novel model named Discriminative Low-Rank Projection (DLRP). Furthermore, DLRP can break the limitation of the small class problem which means the number of projections is bound by the number of classes. Our model can be solved by alternatively linearized alternating direction method with adaptive penalty and the singular value decomposition. Besides, the analyses of differences between DLRP and previous related models are shown. Extensive experiments conducted on various contaminated databases have confirmed the superiority of the proposed method.