Simultaneous Dimension Reduction Research Articles

A New Basis for Sparse Principal Component Analysis

Previous versions of sparse principal component analysis (PCA) have presumed that the eigen-basis (a p × k matrix) is approximately sparse. We propose a method that presumes the p × k matrix becomes approximately sparse after a k × k rotation. The simplest version of the algorithm initializes with the leading k principal components. Then, the principal components are rotated with an k × k orthogonal rotation to make them approximately sparse. Finally, soft-thresholding is applied to the rotated principal components. This approach differs from prior approaches because it uses an orthogonal rotation to approximate a sparse basis. One consequence is that a sparse component need not to be a leading eigenvector, but rather a mixture of them. In this way, we propose a new (rotated) basis for sparse PCA. In addition, our approach avoids “deflation” and multiple tuning parameters required for that. Our sparse PCA framework is versatile; for example, it extends naturally to a two-way analysis of a data matrix for simultaneous dimensionality reduction of rows and columns. We provide evidence showing that for the same level of sparsity, the proposed sparse PCA method is more stable and can explain more variance compared to alternative methods. Through three applications—sparse coding of images, analysis of transcriptome sequencing data, and large-scale clustering of social networks, we demonstrate the modern usefulness of sparse PCA in exploring multivariate data. An R package, epca, and the supplementary materials for this article are available online.

Accelerating inverse inference of ensemble Kalman filter via reduced-order model trained using adaptive sparse observations

The implementation of data assimilation often struggles with low computational efficiency due to the complexity of high-fidelity numerical modeling and the large size of the observation vectors. To address this challenge, it is necessary to employ reduced-order strategies for both the full-order dynamical model and observation operator. Proper orthogonal decomposition (POD) based reduced-order model (ROM) and discrete empirical interpolation method (DEIM) based sparse observation identification are often considered as primary solutions. However, their effective combination requires the determination of high-quality POD reduced-order basis vectors. The current research focuses on incorporating the ROM and DEIM into a Kalman filter framework, resulting in a reduced-order Kalman filter with sparse observations. The key aspect is identifying the optimal POD basis, which can be achieved iteratively by introducing a relative root-mean-square-error (RMSE) for a near-optimal exploration of both the desired low-order POD basis and observation locations. The singular value decomposition (SVD) of the snapshots is performed in a low-dimensional subspace, leading to significant computational savings. This improved POD basis not only enhances the reconstruction of the ROM, but also helps build the DEIM observation matrix, resulting in simultaneous dimensionality reduction of the state and observation space. The effectiveness of this algorithm is demonstrated through the retrieval of initial conditions for one-dimensional tsunami wave equations with real data scenarios. Experiments are conducted for both smooth and nonsmoothed initial conditions. The results show that when the observation data is available at 3 and 6 spatial observation locations, our method only requires fewer vectors with the first 45 and 90 dominant modes derived using the RMSE, compared to as many as 210 and 252 derived from the relative energy criterion. Our method enables a low-modal ROM to effectively reconstruct the wave height field, leading to considerable prediction capability for potential long-wave dynamics with comparable accuracy. The CPU time savings achieved through our method is at least one to two orders of magnitude compared to the CPU time required by the full-order model.

RELIEF: A structured multivariate approach for removal of latent inter-scanner effects.

Combining data collected from multiple study sites is becoming common and is advantageous to researchers to increase the generalizability and replicability of scientific discoveries. However, at the same time, unwanted inter-scanner biases are commonly observed across neuroimaging data collected from multiple study sites or scanners, rendering difficulties in integrating such data to obtain reliable findings. While several methods for handling such unwanted variations have been proposed, most of them use univariate approaches that could be too simple to capture all sources of scanner-specific variations. To address these challenges, we propose a novel multivariate harmonization method called RELIEF (REmoval of Latent Inter-scanner Effects through Factorization) for estimating and removing both explicit and latent scanner effects. Our method is the first approach to introduce the simultaneous dimension reduction and factorization of interlinked matrices to a data harmonization context, which provides a new direction in methodological research for correcting inter-scanner biases. Analyzing diffusion tensor imaging (DTI) data from the Social Processes Initiative in Neurobiology of the Schizophrenia (SPINS) study and conducting extensive simulation studies, we show that RELIEF outperforms existing harmonization methods in mitigating inter-scanner biases and retaining biological associations of interest to increase statistical power. RELIEF is publicly available as an R package.

Penalized model-based clustering of complex functional data

High dimensional data, large-scale data, imaging and manifold data are all fostering new frontiers of statistics. These type of data are commonly considered in Functional Data Analysis where they are viewed as infinite-dimensional random vectors in a functional space. The rapid development of new technologies has generated a flow of complex data that have led to the development of new modeling strategies by scientists. In this paper, we basically deal with the problem of clustering a set of complex functional data into homogeneous groups. Working in a mixture model-based framework, we develop a flexible clustering technique achieving dimensionality reduction schemes through an L_1 penalization. The proposed procedure results in an integrated modelling approach where shrinkage techniques are applied to enable sparse solutions in both the means and the covariance matrices of the mixture components, while preserving the underlying clustering structure. This leads to an entirely data-driven methodology suitable for simultaneous dimensionality reduction and clustering. The proposed methodology is evaluated through a Monte Carlo simulation study and an empirical analysis of real-world datasets showing different degrees of complexity.

Constraint principal components for linear discrimination

In many modern data, the number of variables is much higher than the number of observations and the within-group scatter matrix is singular. Then, the Fisher's linear discriminant analysis (LDA) cannot be applied. The work considers a way to circumvent this problem by doing principal component analysis (PCA) enhanced with additional discriminating features. Two approaches are proposed: the original PCs are rotated to maximize the Fisher's LDA criterion, and second, penalized PCs are produced to achieve simultaneous dimension reduction and maximization of the Fisher's LDA criterion. Both approaches are illustrated and compared to other existing methods on several well known data sets.

Development of a metabolite calculator for diagnosis of pancreatic cancer.

Carbohydrate antigen (CA) 19-9 is a known pancreatic cancer (PC) biomarker, but is not commonly used for general screening due to its low sensitivity and specificity. This study aimed to develop a serum metabolites-based diagnostic calculator for detecting PC with high accuracy. A targeted quantitative approach of direct flow injection-tandem mass spectrometry combined with liquid chromatography-tandem mass spectrometry was employed for metabolomic analysis of serum samples using an Absolute IDQ™ p180 kit. Integrated metabolomic analysis was performed on 241 pooled or individual serum samples collected from healthy donors and patients from nine disease groups, including chronic pancreatitis, PC, other cancers, and benign diseases. Orthogonal partial least squares discriminant analysis (OPLS-DA) based on characteristics of 116 serum metabolites distinguished patients with PC from those with other diseases. Sparse partial least squares discriminant analysis (SPLS-DA) was also performed, incorporating simultaneous dimension reduction and variable selection. Predictive performance between discrimination models was compared using a 2-by-2 contingency table of predicted probabilities obtained from the models and actual diagnoses. Predictive values obtained through OPLS-DA for accuracy, sensitivity, specificity, balanced accuracy, and area under the receiver operating characteristic curve (AUC) were 0.9825, 0.9916, 0.9870, 0.9866, and 0.9870, respectively. The number of metabolite candidates was narrowed to 76 for SPLS-DA. The SPLS-DA-obtained predictive values for accuracy, sensitivity, specificity, balanced accuracy, and AUC were 0.9773, 0.9649, 0.9832, 0.9741, and 0.9741, respectively. We successfully developed a 76 metabolome-based diagnostic panel for detecting PC that demonstrated high diagnostic performance in differentiating PC from other diseases.

Gradient-based sparse principal component analysis with extensions to online learning.

Sparse principal component analysis is an important technique for simultaneous dimensionality reduction and variable selection with high-dimensional data. In this work we combine the unique geometric structure of the sparse principal component analysis problem with recent advances in convex optimization to develop novel gradient-based sparse principal component analysis algorithms. These algorithms enjoy the same global convergence guarantee as the original alternating direction method of multipliers, and can be more efficiently implemented with the rich toolbox developed for gradient methods from the deep learning literature. Most notably, these gradient-based algorithms can be combined with stochastic gradient descent methods to produce efficient online sparse principal component analysis algorithms with provable numerical and statistical performance guarantees. The practical performance and usefulness of the new algorithms are demonstrated in various simulation studies. As an application, we show how the scalability and statistical accuracy of our method enable us to find interesting functional gene groups in high-dimensional RNA sequencing data.

Group linear non-Gaussian component analysis with applications to neuroimaging

Independent component analysis (ICA) is an unsupervised learning method popular in functional magnetic resonance imaging (fMRI). Group ICA has been used to search for biomarkers in neurological disorders including autism spectrum disorder and dementia. However, current methods use a principal component analysis (PCA) step that may remove low-variance features. Linear non-Gaussian component analysis (LNGCA) enables simultaneous dimension reduction and feature estimation including low-variance features in single-subject fMRI. A group LNGCA model is proposed to extract group components shared by more than one subject. Unlike group ICA methods, this novel approach also estimates individual (subject-specific) components orthogonal to the group components. To determine the total number of components in each subject, a parametric resampling test is proposed that samples spatially correlated Gaussian noise to match the spatial dependence observed in data. In simulations, estimated group components achieve higher accuracy compared to group ICA. The method is applied to a resting-state fMRI study on autism spectrum disorder in 342 children (252 typically developing, 90 with autism), where the group signals include resting-state networks. The discovered group components appear to exhibit different levels of temporal engagement in autism versus typically developing children, as revealed using group LNGCA. This novel approach to matrix decomposition is a promising direction for feature detection in neuroimaging.

Simultaneous dimensionality reduction and integration for single-cell ATAC-seq data using deep learning

Advances in single-cell technologies enable the routine interrogation of chromatin accessibility for tens of thousands of single cells, elucidating gene regulatory processes at an unprecedented resolution. Meanwhile, size, sparsity and high dimensionality of the resulting data continue to pose challenges for its computational analysis, and specifically the integration of data from different sources. We have developed a dedicated computational approach: a variational auto-encoder using a noise model specifically designed for single-cell ATAC-seq (assay for transposase-accessible chromatin with high-throughput sequencing) data, which facilitates simultaneous dimensionality reduction and batch correction via an adversarial learning strategy. We showcase its benefits for detailed cell-type characterization on individual real and simulated datasets as well as for integrating multiple complex datasets.

Modified Genetic Algorithm for Feature Selection and Hyper Parameter Optimization: Case of XGBoost in Spam Prediction

Recently, spam on online social networks has attracted attention in the research and business world. Twitter has become the preferred medium to spread spam content. Many research efforts attempted to encounter social networks spam. Twitter brought extra challenges represented by the feature space size, and imbalanced data distributions. Usually, the related research works focus on part of these main challenges or produce black-box models. In this paper, we propose a modified genetic algorithm for simultaneous dimensionality reduction and hyper parameter optimization over imbalanced datasets. The algorithm initialized an eXtreme Gradient Boosting classifier and reduced the features space of tweets dataset; to generate a spam prediction model. The model is validated using a 50 times repeated 10-fold stratified cross-validation, and analyzed using nonparametric statistical tests. The resulted prediction model attains on average 82.32% and 92.67% in terms of geometric mean and accuracy respectively, utilizing less than 10% of the total feature space. The empirical results show that the modified genetic algorithm outperforms <i>Chi</i>² and <i>PCA</i> feature selection methods. In addition, eXtreme Gradient Boosting outperforms many machine learning algorithms, including BERT-based deep learning model, in spam prediction. Furthermore, the proposed approach is applied to SMS spam modeling and compared to related works.

Dimension reduction graph‐based sparse subspace clustering for intelligent fault identification of rolling element bearings

AbstractSparse subspace clustering (SSC) is a spectral clustering methodology. Since high‐dimensional data are often dispersed over the union of many low‐dimensional subspaces, their representation in a suitable dictionary is sparse. Therefore, SSC is an effective technology for diagnosing mechanical system faults. Its main purpose is to create a representation model that can reveal the real subspace structure of high‐dimensional data, construct a similarity matrix by using the sparse representation coefficients of high‐dimensional data, and then cluster the obtained representation coefficients and similarity matrix in subspace. However, the design of SSC algorithm is based on global expression in which each data point is represented by all possible cluster data points. This leads to nonzero terms in nondiagonal blocks of similar matrices, which reduces the recognition performance of matrices. To improve the clustering ability of SSC for rolling bearing and the robustness of the algorithm in the presence of a large number of background noise, a simultaneous dimensionality reduction subspace clustering technology is provided in this work. Through the feature extraction of envelope signal, the dimension of the feature matrix is reduced by singular value decomposition, and the Euclidean distance between samples is replaced by correlation distance. A dimension reduction graph‐based SSC technology is established. Simulation and bearing data of Western Reserve University show that the proposed algorithm can improve the accuracy and compactness of clustering.

A new model for simultaneous dimensionality reduction and time-varying functional connectivity estimation.

An important question in neuroscience is whether or not we can interpret spontaneous variations in the pattern of correlation between brain areas, which we refer to as functional connectivity or FC, as an index of dynamic neuronal communication in fMRI. That is, can we measure time-varying FC reliably? And, if so, can FC reflect information transfer between brain regions at relatively fast-time scales? Answering these questions in practice requires dealing with the statistical challenge of having high-dimensional data and a comparatively lower number of time points or volumes. A common strategy is to use PCA to reduce the dimensionality of the data, and then apply some model, such as the hidden Markov model (HMM) or a mixture model of Gaussian distributions, to find a set of distinct FC patterns or states. The distinct spatial properties of these FC states together with the time-resolved switching between them offer a flexible description of time-varying FC. In this work, I show that in this context PCA can suffer from systematic biases and loss of sensitivity for the purposes of finding time-varying FC. To get around these issues, I propose a novel variety of the HMM, named HMM-PCA, where the states are themselves PCA decompositions. Since PCA is based on the data covariance, the state-specific PCA decompositions reflect distinct patterns of FC. I show, theoretically and empirically, that fusing dimensionality reduction and time-varying FC estimation in one single step can avoid these problems and outperform alternative approaches, facilitating the quantification of transient communication in the brain.

Ensemble dimensionality reduction and feature gene extraction for single-cell RNA-seq data

Single-cell RNA sequencing (scRNA-seq) technologies allow researchers to uncover the biological states of a single cell at high resolution. For computational efficiency and easy visualization, dimensionality reduction is necessary to capture gene expression patterns in low-dimensional space. Here we propose an ensemble method for simultaneous dimensionality reduction and feature gene extraction (EDGE) of scRNA-seq data. Different from existing dimensionality reduction techniques, the proposed method implements an ensemble learning scheme that utilizes massive weak learners for an accurate similarity search. Based on the similarity matrix constructed by those weak learners, the low-dimensional embedding of the data is estimated and optimized through spectral embedding and stochastic gradient descent. Comprehensive simulation and empirical studies show that EDGE is well suited for searching for meaningful organization of cells, detecting rare cell types, and identifying essential feature genes associated with certain cell types.

Extending Fuzzy Cognitive Maps with Tensor-Based Distance Metrics

Cognitive maps are high level representations of the key topological attributes of real or abstract spatial environments progressively built by a sequence of noisy observations. Currently such maps play a crucial role in cognitive sciences as it is believed this is how clusters of dedicated neurons at hippocampus construct internal representations. The latter include physical space and, perhaps more interestingly, abstract fields comprising of interconnected notions such as natural languages. In deep learning cognitive graphs are effective tools for simultaneous dimensionality reduction and visualization with applications among others to edge prediction, ontology alignment, and transfer learning. Fuzzy cognitive graphs have been proposed for representing maps with incomplete knowledge or errors caused by noisy or insufficient observations. The primary contribution of this article is the construction of cognitive map for the sixteen Myers-Briggs personality types with a tensor distance metric. The latter combines two categories of natural language attributes extracted from the namesake Kaggle dataset. To the best of our knowledge linguistic attributes are separated in categories. Moreover, a fuzzy variant of this map is also proposed where a certain personality may be assigned to up to two types with equal probability. The two maps were evaluated based on their topological properties, on their clustering quality, and on how well they fared against the dataset ground truth. The results indicate a superior performance of both maps with the fuzzy variant being better. Based on the findings recommendations are given for engineers and practitioners.

AdaCLV for interpretable variable clustering and dimensionality reduction of spectroscopic data

This paper introduces a new method, Adaptive Clustering around Latent Variables (AdaCLV), for simultaneous dimensionality reduction and variable clustering, the partitioning of variables into groups. This unsupervised method is particularly well suited for the exploration of spectroscopic datasets, such as Nuclear Magnetic Resonance (NMR) spectra, and can be used for the identification of potential biomarkers. AdaCLV is inspired by existing multivariate methods from the Clustering around Latent Variables (CLV) family, but it offers several key advantages with respect to these methods. First, AdaCLV estimates cluster membership degrees that are more interpretable and representative of spectroscopic data, where peaks for different molecules (i.e. variable clusters) may overlap and variables within a cluster have different degrees of importance. Second, AdaCLV is less sensitive to its hyperparameters than other competing methods, adapting to the clustering structure present in the data. This paper compares AdaCLV with existing CLV methods and other competitors in experiments involving real and semi-artificial NMR spectra. AdaCLV is shown to be more robust to hyperparameter choice and to have better precision than the other methods, for all cluster numbers, sample sizes and levels of signal tested, while achieving a comparable level of recall.

Derivation of a Homogenized Bending\u2013Torsion Theory for Rods with Micro-Heterogeneous Prestrain

In this paper we investigate rods made of nonlinearly elastic, composite–materials that feature a micro-heterogeneous prestrain that oscillates (locally periodic) on a scale that is small compared to the length of the rod. As a main result we derive a homogenized bending–torsion theory for rods as Gamma -limit from 3D nonlinear elasticity by simultaneous homogenization and dimension reduction under the assumption that the prestrain is of the order of the diameter of the rod. The limit model features a spontaneous curvature–torsion tensor that captures the macroscopic effect of the micro-heterogeneous prestrain. We devise a formula that allows to compute the spontaneous curvature–torsion tensor by means of a weighted average of the given prestrain. The weight in the average depends on the geometry of the composite and invokes correctors that are defined with help of boundary value problems for the system of linear elasticity. The definition of the correctors depends on a relative scaling parameter gamma , which monitors the ratio between the diameter of the rod and the period of the composite’s microstructure. We observe an interesting size-effect: For the same prestrain a transition from flat minimizers to curved minimizers occurs by just changing the value of gamma . Moreover, in the paper we analytically investigate the microstructure-properties relation in the case of isotropic, layered composites, and consider applications to nematic liquid–crystal–elastomer rods and shape programming.

Simultaneous dimension reduction and clustering via the NMF-EM algorithm

Mixture models are among the most popular tools for clustering. However, when the dimension and the number of clusters is large, the estimation of the clusters become challenging, as well as their interpretation. Restriction on the parameters can be used to reduce the dimension. An example is given by mixture of factor analyzers for Gaussian mixtures. The extension of MFA to non-Gaussian mixtures is not straightforward. We propose a new constraint for parameters in non-Gaussian mixture model: the $K$ components parameters are combinations of elements from a small dictionary, say $H$ elements, with $H \ll K$. Including a nonnegative matrix factorization (NMF) in the EM algorithm allows us to simultaneously estimate the dictionary and the parameters of the mixture. We propose the acronym NMF-EM for this algorithm, implemented in the R package {\tt nmfem}. This original approach is motivated by passengers clustering from ticketing data: we apply NMF-EM to data from two Transdev public transport networks. In this case, the words are easily interpreted as typical slots in a timetable.

An Impartial Trimming Approach for Joint Dimension and Sample Reduction

A robust version of reduced and factorial k-means is proposed that is based on the idea of trimming. Reduced and factorial k-means are data reduction techniques well suited for simultaneous dimension and sample reduction through PCA and clustering. The occurrence of data inadequacies can invalidate standard analyses. Actually, contamination in the data at hand can hide the underlying clustered structure of the data. An appealing approach to develop robust counterparts of factorial and reduced k-means is given by impartial trimming. The idea is to discard a fraction of observations that are selected as the most distant from the centroids. The finite sample behavior of the proposed methods has been investigated by some numerical studies and real data examples.

Model-based clustering with envelopes

Clustering analysis is an important unsupervised learning technique in multivariate statistics and machine learning. In this paper, we propose a set of new mixture models called CLEMM (in short for Clustering with Envelope Mixture Models) that is based on the widely used Gaussian mixture model assumptions and the nascent research area of envelope methodology. Formulated mostly for regression models, envelope methodology aims for simultaneous dimension reduction and efficient parameter estimation, and includes a very recent formulation of envelope discriminant subspace for classification and discriminant analysis. Motivated by the envelope discriminant subspace pursuit in classification, we consider parsimonious probabilistic mixture models where the cluster analysis can be improved by projecting the data onto a latent lower-dimensional subspace. The proposed CLEMM framework and the associated envelope-EM algorithms thus provide foundations for envelope methods in unsupervised and semi-supervised learning problems. Numerical studies on simulated data and two benchmark data sets show significant improvement of our propose methods over the classical methods such as Gaussian mixture models, K-means and hierarchical clustering algorithms. An R package is available at https://github.com/kusakehan/CLEMM.

Simultaneous Dimensionality Reduction and Classification via Dual Embedding Regularized Nonnegative Matrix Factorization.

Nonnegative matrix factorization (NMF) is a well-known paradigm for data representation. Traditional NMF-based classification methods first perform NMF or one of its variants on input data samples to obtain their low-dimensional representations, which are successively classified by means of a typical classifier [e.g., k -nearest neighbors (KNN) and support vector machine (SVM)]. Such a stepwise manner may overlook the dependency between the two processes, resulting in the compromise of the classification accuracy. In this paper, we elegantly unify the two processes by formulating a novel constrained optimization model, namely dual embedding regularized NMF (DENMF), which is semi-supervised. Our DENMF solution simultaneously finds the low-dimensional representations and assignment matrix via joint optimization for better classification. Specifically, input data samples are projected onto a couple of low-dimensional spaces (i.e., feature and label spaces), and locally linear embedding is employed to preserve the identical local geometric structure in different spaces. Moreover, we propose an alternating iteration algorithm to solve the resulting DENMF, whose convergence is theoretically proven. Experimental results over five benchmark datasets demonstrate that DENMF can achieve higher classification accuracy than state-of-the-art algorithms.