Linear Dimensionality Reduction Techniques Research Articles

Methods for Measurement and Analysis of Full Hand Angular Kinematics Using Electromagnetic Tracking Sensors

The human hand needs a large number of sensors to measure kinematics owing to its large number of degrees of freedom. Existing devices like data gloves and optical trackers are associated with calibration, line of sight, and accuracy problems. In this paper, we attempt to measure the full hand kinematics using Electromagnetic Tracking Sensors (EMTS) which are accurate, free of line-of-sight problems, and require no calibration. However, EMTS provides output in the form of rotation groups which are defined on a nonlinear manifold. Hence, linear operations required for experimental analysis such as linear dimensionality reduction are not valid. Also, these sensors are expensive, utilize space in terms of cabling, and require a reduced sensor layout. In this paper, we present measurement methods, test the utility of linear and non-linear dimensionality reduction techniques on quaternions and exponential maps. We also performed sensor reduction using a Gini feature selection based on random forest algorithm. The kinematic measurement results show that EMTS yield superior posture reproduction with an error of less than 1 degree. Autoencoder, a nonlinear dimensionality reduction technique, was successfully applied on quaternions which was tuned to perform better than Principal Component Analysis (PCA) in reducing dimensions. The reduced sensor layout with 8 sensors was able to predict full hand kinematics with a Root Mean Square Error (RMSE) of 5.1 degrees.

Supervised dimensionality reduction for big data

To solve key biomedical problems, experimentalists now routinely measure millions or billions of features (dimensions) per sample, with the hope that data science techniques will be able to build accurate data-driven inferences. Because sample sizes are typically orders of magnitude smaller than the dimensionality of these data, valid inferences require finding a low-dimensional representation that preserves the discriminating information (e.g., whether the individual suffers from a particular disease). There is a lack of interpretable supervised dimensionality reduction methods that scale to millions of dimensions with strong statistical theoretical guarantees. We introduce an approach to extending principal components analysis by incorporating class-conditional moment estimates into the low-dimensional projection. The simplest version, Linear Optimal Low-rank projection, incorporates the class-conditional means. We prove, and substantiate with both synthetic and real data benchmarks, that Linear Optimal Low-Rank Projection and its generalizations lead to improved data representations for subsequent classification, while maintaining computational efficiency and scalability. Using multiple brain imaging datasets consisting of more than 150 million features, and several genomics datasets with more than 500,000 features, Linear Optimal Low-Rank Projection outperforms other scalable linear dimensionality reduction techniques in terms of accuracy, while only requiring a few minutes on a standard desktop computer.

Algorithms for Nonnegative Matrix Factorization with the Kullback–Leibler Divergence

Nonnegative matrix factorization (NMF) is a standard linear dimensionality reduction technique for nonnegative data sets. In order to measure the discrepancy between the input data and the low-rank approximation, the Kullback-Leibler (KL) divergence is one of the most widely used objective function for NMF. It corresponds to the maximum likehood estimator when the underlying statistics of the observed data sample follows a Poisson distribution, and KL NMF is particularly meaningful for count data sets, such as documents or images. In this paper, we first collect important properties of the KL objective function that are essential to study the convergence of KL NMF algorithms. Second, together with reviewing existing algorithms for solving KL NMF, we propose three new algorithms that guarantee the non-increasingness of the objective function. We also provide a global convergence guarantee for one of our proposed algorithms. Finally, we conduct extensive numerical experiments to provide a comprehensive picture of the performances of the KL NMF algorithms.

Nonlinear machine learning pattern recognition and bacteria-metabolite multilayer network analysis of perturbed gastric microbiome

The stomach is inhabited by diverse microbial communities, co-existing in a dynamic balance. Long-term use of drugs such as proton pump inhibitors (PPIs), or bacterial infection such as Helicobacter pylori, cause significant microbial alterations. Yet, studies revealing how the commensal bacteria re-organize, due to these perturbations of the gastric environment, are in early phase and rely principally on linear techniques for multivariate analysis. Here we disclose the importance of complementing linear dimensionality reduction techniques with nonlinear ones to unveil hidden patterns that remain unseen by linear embedding. Then, we prove the advantages to complete multivariate pattern analysis with differential network analysis, to reveal mechanisms of bacterial network re-organizations which emerge from perturbations induced by a medical treatment (PPIs) or an infectious state (H. pylori). Finally, we show how to build bacteria-metabolite multilayer networks that can deepen our understanding of the metabolite pathways significantly associated to the perturbed microbial communities.

Transformed central quantile subspace

ABSTRACT Quantile regression (QR) is a well-established method of tail analysis. Application of QR can become very challenging when dealing with high-dimensional data, thus requiring dimension reduction techniques. While the current literature on these techniques focuses on extracting linear combinations of the predictor variables that contain all the information about the conditional quantile, non-linear features can potentially achieve greater dimension reduction. We, therefore, present the first application of transformed dimension reduction for conditional quantiles, which serves as an intermediate step between linear and nonlinear dimension reduction. The idea is to transform the predictors monotonically and then look for low-dimensional projections by applying linear dimension reduction techniques. The performance of the proposed methodology is demonstrated through simulation examples and a real data application.

Evaluation of Dimensionality Reduction Techniques for Load Profiling Application in Smart Grid Environment

The traditional power grid system is evolving toward a smart grid system, which will improve the user experience. However, this system is capable of generating high-dimensional data at a very high sample rate. A common technique for reducing high-dimensional data is dimensionality reduction. With this technique, we are able to reduce the data to a lower dimension, making it suitable for smart grid applications including transmission, storage, and visualization. Linear dimensionality reduction techniques are mostly explored in the context of smart grid applications. Due to the nonlinear nature of data generation in the smart grid, we anticipate that the nonlinear dimensionality reduction techniques can perform better. This work evaluates different nonlinear dimensionality reduction techniques and compares them with principal component analysis, which is a widely used linear dimensionality reduction technique in the smart grid environment. We use the visualization of load profile data and adjusted rank index (ARI) for comparison of dimensionality reduction techniques. Load profiling is an important task to complement the demand-side management and tariff selection. The visual depiction of the load profiles and ARI suggest that the nonlinear dimensionality reduction techniques perform better compared with linear dimensionality reduction techniques.

Wear monitoring in fine blanking processes using feature based analysis of acoustic emission signals

Tool wear during fine blanking impairs the quality of the sheared part, which is assessed in regular samples in an industrial environment. This leads to scrap production and low planning reliability due to low wear predictability. A tool condition monitoring based on acoustic emission (AE) data for the prediction of the remaining useful life of the tool would mitigate those effects. In a production series, AE signals were recorded, and the tool wear observed. The AE signals were then preprocessed using feature engineering and visualized using linear and nonlinear dimensionality reduction techniques. These visualizations preserve information about the data structure even in two dimensions and resemble the temporal dependent observed tool wear during fine blanking.

Distributionally Robust and Multi-Objective Nonnegative Matrix Factorization.

Nonnegative matrix factorization (NMF) is a linear dimensionality reduction technique for analyzing nonnegative data. A key aspect of NMF is the choice of the objective function that depends on the noise model (or statistics of the noise) assumed on the data. In many applications, the noise model is unknown and difficult to estimate. In this paper, we define a multi-objective NMF (MO-NMF) problem, where several objectives are combined within the same NMF model. We propose to use Lagrange duality to judiciously optimize for a set of weights to be used within the framework of the weighted-sum approach, that is, we minimize a single objective function which is a weighted sum of the all objective functions. We design a simple algorithm based on multiplicative updates to minimize this weighted sum. We show how this can be used to find distributionally robust NMF (DR-NMF) solutions, that is, solutions that minimize the largest error among all objectives, using a dual approach solved via a heuristic inspired from the Frank-Wolfe algorithm. We illustrate the effectiveness of this approach on synthetic, document and audio data sets. The results show that DR-NMF is robust to our incognizance of the noise model of the NMF problem.

Conformational Landscapes of Halohydrin Dehalogenases and Their Accessible Active Site Tunnels

Halohydrin dehalogenases (HHDH) are industrially relevant biocatalysts exhibiting a promiscuous epoxide-ring opening reactivity in the presence of small nucleophiles, thus giving access to novel carbon–carbon, carbon–oxygen, carbon–nitrogen, and carbon–sulfur bonds. Recently, the repertoire of HHDH has been expanded, providing access to some novel HHDH subclasses exhibiting a broader epoxide substrate scope. In this work, we develop a computational approach based on the application of linear and non-linear dimensionality reduction techniques to long time-scale Molecular Dynamics (MD) simulations to study the HHDH conformational landscapes. We couple the analysis of the conformational landscapes to CAVER calculations to assess their impact on the active site tunnels and potential ability towards bulky epoxide ring opening reaction. Our study indicates that the analyzed HHDHs subclasses share a common breathing motion of the halide binding pocket, but present large deviations in the loops adjacent to the active site pocket and N-terminal regions. Such conformational differences affect the available tunnels for epoxide binding to the active site. The superior activity of the HHDH G subclass towards bulkier substrates is explained by the additional structural elements delimiting the active site region, its rich conformational heterogeneity, and the substantially wider and frequently observed active site tunnels. This study therefore provides key information for HHDH promiscuity and engineering.

A Deep Learning Approach to Population Structure Inference in Inbred Lines of Maize.

Analysis of population genetic variation and structure is a common practice for genome-wide studies, including association mapping, ecology, and evolution studies in several crop species. In this study, machine learning (ML) clustering methods, K-means (KM), and hierarchical clustering (HC), in combination with non-linear and linear dimensionality reduction techniques, deep autoencoder (DeepAE) and principal component analysis (PCA), were used to infer population structure and individual assignment of maize inbred lines, i.e., dent field corn (n = 97) and popcorn (n = 86). The results revealed that the HC method in combination with DeepAE-based data preprocessing (DeepAE-HC) was the most effective method to assign individuals to clusters (with 96% of correct individual assignments), whereas DeepAE-KM, PCA-HC, and PCA-KM were assigned correctly 92, 89, and 81% of the lines, respectively. These findings were consistent with both Silhouette Coefficient (SC) and Davies–Bouldin validation indexes. Notably, DeepAE-HC also had better accuracy than the Bayesian clustering method implemented in InStruct. The results of this study showed that deep learning (DL)-based dimensional reduction combined with ML clustering methods is a useful tool to determine genetically differentiated groups and to assign individuals into subpopulations in genome-wide studies without having to consider previous genetic assumptions.

An empirical analysis of graph‐based linear dimensionality reduction techniques

SummaryMany emerging applications such as social networks have prompted remarkable attention in graph data analysis. Graph data is typically high‐dimensional in nature, and dimensionality reduction is critical regarding storage, analysis, and querying of such data efficiently. Although there are many dimensionality reduction methods, it is not clear to what extent the performances of the various dimensionality reduction techniques differ. In this article, we review some of the well‐known linear dimensionality reduction methods and perform an empirical analysis of these approaches using large multidimensional graph datasets. Our results show that in linear unsupervised learning methods, the principal component analysis, singular value decomposition, and neighborhood preserving embedding methods achieve better retrieval data performance than other methods of the statistical information category, dictionary methods, and embedding methods, respectively. Regarding supervised learning methods, the experimental results demonstrate that linear discriminant analysis and partial least squares presented almost similar results.

Orthogonal canonical correlation analysis and applications

Canonical correlation analysis (CCA) is a cornerstone of linear dimensionality reduction techniques that jointly maps two datasets to achieve maximal correlation. CCA has been widely used in applications for capturing data features of interest. In this paper, we establish a range constrained orthogonal CCA (OCCA) model and its variant and apply them for three data analysis tasks of datasets in real-life applications, namely unsupervised feature fusion, multi-target regression and multi-label classification. Numerical experiments show that the OCCA and its variant produce superior accuracy compared to the traditional CCA.

Near-Convex Archetypal Analysis

Nonnegative matrix factorization (NMF) is a widely used linear dimensionality reduction technique for nonnegative data. NMF requires that each data point is approximated by a convex combination of basis elements. Archetypal analysis (AA), also referred to as convex NMF, is a well-known NMF variant imposing that the basis elements are themselves convex combinations of the data points. AA has the advantage to be more interpretable than NMF because the basis elements are directly constructed from the data points. However, it usually suffers from a high data fitting error because the basis elements are constrained to be contained in the convex cone of the data points. In this letter, we introduce near-convex archetypal analysis (NCAA) which combines the advantages of both AA and NMF. As for AA, the basis vectors are required to be linear combinations of the data points and hence are easily interpretable. As for NMF, the additional flexibility in choosing the basis elements allows NCAA to have a low data fitting error. We show that NCAA compares favorably with a state-of-the-art minimum-volume NMF method on synthetic datasets and on a real-world hyperspectral image.

Manifold Approximation by Moving Least-Squares Projection (MMLS)

In order to avoid the curse of dimensionality, frequently encountered in big data analysis, there has been vast development in the field of linear and nonlinear dimension reduction techniques in recent years. These techniques (sometimes referred to as manifold learning) assume that the scattered input data is lying on a lower-dimensional manifold; thus the high dimensionality problem can be overcome by learning the lower dimensionality behavior. However, in real-life applications, data is often very noisy. In this work, we propose a method to approximate $$\mathcal {M}$$ a d-dimensional $$C^{m+1}$$ smooth submanifold of $$\mathbb {R}^n$$ ( $$d \ll n$$ ) based upon noisy scattered data points (i.e., a data cloud). We assume that the data points are located “near” the lower-dimensional manifold and suggest a nonlinear moving least-squares projection on an approximating d-dimensional manifold. Under some mild assumptions, the resulting approximant is shown to be infinitely smooth and of high approximation order (i.e., $$\mathcal {O}(h^{m+1})$$ , where h is the fill distance and m is the degree of the local polynomial approximation). The method presented here assumes no analytic knowledge of the approximated manifold and the approximation algorithm is linear in the large dimension n. Furthermore, the approximating manifold can serve as a framework to perform operations directly on the high-dimensional data in a computationally efficient manner. This way, the preparatory step of dimension reduction, which induces distortions to the data, can be avoided altogether.

Diagnosis algorithms for indirect structural health monitoring of a bridge model via dimensionality reduction

Abstract We present a data-driven approach based on physical insights to achieve damage diagnosis of bridges using only vibration signals collected on board the vehicles passing over the bridge. Though data-driven models have been shown to produce promising results in this context, they generally require labeled examples to fit the models (i.e., supervised learning) and make it difficult to interpret the physical mechanisms. We posit that these shortcomings can be alleviated by studying the physical relationship between damage and the distribution of the resulting acceleration signals, and then choosing an appropriate model to invert this process. To help guide the development of appropriate damage diagnosis algorithms, we first make use of the theoretical formulation of the vehicle-bridge interaction system in the frequency domain and conduct a finite element simulation of this system. From the derived numerical solution, we observe that not only is the trend of the acceleration signals of a passing vehicle with different damage severity non-linear, but also that both the low- and high-frequency responses of a passing vehicle contain information about damage severity. Guided by these observations, we use several dimensionality reduction methods to extract representative features from the vehicle’s vibration response. We then propose an unsupervised damage severity comparison model and a semi-supervised damage severity estimation model aiming at indirect monitoring of bridges. We apply the algorithms to diagnose changes that occur in a laboratory bridge model to which a concentrated mass of gradually changing magnitude is attached at mid-span. The experimental results of the damage severity comparison and estimation show that a non-convex and non-linear dimensionality reduction technique (stacked autoencoders) outperforms other linear and/or convex dimensionality reduction techniques. Overall, our results provide evidence for the applicability of indirect structural health monitoring in bridge models and suggest the feasibility of extending this approach to actual structures.

Hybrid projective nonnegative matrix factorization based on α-divergence and the alternating least squares algorithm

Nonnegative Matrix Factorization (NMF) is a linear dimensionality reduction technique for extracting hidden and intrinsic features of high-dimensional data sets. Recently, several Projective NMF (P-NMF) methods have been proposed for the purpose of resolving issues associated with the standard NMF approach. Experimental results show that P-NMF algorithms outperform the standard NMF method in some aspects. But some basic issues still affect the existing NMF and P-NMF methods, these include slow convergence rate, low reconstruction accuracy and dense basis factors. In this article, we propose a new and generalized hybrid algorithm by combining the concept of alternating least squares with the multiplicative update rules of the α-divergence-based P-NMF method. We have conducted extensive numerical experiments on 7 real-world data sets and compared the new algorithm with several state-of-the-art methods. The attractive features and added advantages of the new algorithm include remarkable clustering performances, providing highly “orthogonal” and very sparse basis factors, and extracting distinctive and better localized features of the original data than its counterparts.

Uncertainty-Aware Principal Component Analysis.

We present a technique to perform dimensionality reduction on data that is subject to uncertainty. Our method is a generalization of traditional principal component analysis (PCA) to multivariate probability distributions. In comparison to non-linear methods, linear dimensionality reduction techniques have the advantage that the characteristics of such probability distributions remain intact after projection. We derive a representation of the PCA sample covariance matrix that respects potential uncertainty in each of the inputs, building the mathematical foundation of our new method: uncertainty-aware PCA. In addition to the accuracy and performance gained by our approach over sampling-based strategies, our formulation allows us to perform sensitivity analysis with regard to the uncertainty in the data. For this, we propose factor traces as a novel visualization that enables to better understand the influence of uncertainty on the chosen principal components. We provide multiple examples of our technique using real-world datasets. As a special case, we show how to propagate multivariate normal distributions through PCA in closed form. Furthermore, we discuss extensions and limitations of our approach.

Impact identification using nonlinear dimensionality reduction and supervised learning

Real-time monitoring systems that can automatically locate and identify impacts as they occur have become increasingly attractive for ensuring safety and preventing catastrophic accidents in aerospace structures. In most cases, a set of piezoelectric transducers distributed over the structure captures strain–time data, which are preprocessed to extract relevant features that are fed to a supervised learning algorithm to detect, locate, and quantify impacts. The best results achieved to date in feature extraction for impact identification have been obtained with the use of principal component analysis (PCA). However, this technique cannot handle complex nonlinear data. The primary contribution of this study is the implementation of a novel impact identification algorithm that uses a supervised learning algorithm called linear approximation with maximum entropy (LME) in conjunction with different linear and nonlinear dimensionality reduction techniques, including PCA, kernel PCA, Isomap, local linear embedding (LLE), and multilayer autoencoders. The performance of LME with the different reduction techniques is tested with two experimental applications. The results show that the techniques that do not employ graphs, such as PCA, kernel PCA, and autoencoders, perform better, and the method that provides the best results is LME in conjunction with autoencoders. It is further demonstrated that LME with autoencoders works better than the algorithms available in the literature for similar problems.

Efficient algorithm for sparse symmetric nonnegative matrix factorization

Symmetric Nonnegative Matrix Factorization (symNMF) is a special case of the standard Nonnegative Matrix Factorization (NMF) method which is the most popular linear dimensionality reduction technique for analyzing nonnegative data. Examples of symmetric matrices that arise in real-life applications include covariance matrices in finance, adjacency matrices associated with undirected graphs, and distance matrices arising from video and other media summarization technologies. The advantages of having sparse factors in symNMF include saving a great deal of storage space and enhancing the extraction of localized basis features that better represent the latent features of the original data. In order to obtain sparser basis factors, we formulate a new sparse symNMF model by imposing an l1-norm based sparsity constraint on the symNMF problem. We use the concept of rank-one nonnegative matrix approximation to decouple this non-convex optimization problem into convex nonnegative least squares sub-problems which are easier to solve. We develop a new and efficient coordinate-descent based algorithm and test it on four well-known databases of facial images. The proposed algorithm is shown to have fast convergence, well-suited for solving the sparse symNMF problem, and extract localized basis features of image datasets that facilitate interpretation.

Alternating Diffusion Map Based Fusion of Multimodal Brain Connectivity Networks for IQ Prediction.

To explain individual differences in development, behavior, and cognition, most previous studies focused on projecting resting-state functional MRI (fMRI) based functional connectivity (FC) data into a low-dimensional space via linear dimensionality reduction techniques, followed by executing analysis operations. However, linear dimensionality analysis techniques may fail to capture the nonlinearity of brain neuroactivity. Moreover, besides resting-state FC, the FC based on task fMRI can be expected to provide complementary information. Motivated by these considerations, we nonlinearly fuse resting-state and task-based FC networks (FCNs) to seek a better representation in this paper. We propose a framework based on alternating diffusion map (ADM), which extracts geometry-preserving low-dimensional embeddings that successfully parameterize the intrinsic variables driving the phenomenon of interest. Specifically, we first separately build resting-state and task-based FCNs by symmetric positive definite matrices using sparse inverse covariance estimation for each subject, and then utilize the ADM to fuse them in order to extract significant low-dimensional embeddings, which are used as fingerprints to identify individuals. The proposed framework is validated on the Philadelphia Neurodevelopmental Cohort data, where we conduct extensive experimental study on resting-state and fractal n-back task fMRI for the classification of intelligence quotient (IQ). The fusion of resting-state and n-back task fMRI by the proposed framework achieves better classification accuracy than any single fMRI, and the proposed framework is shown to outperform several other data fusion methods. To our knowledge, this paper is the first to demonstrate a successful extension of the ADM to fuse resting-state and task-based fMRI data for accurate prediction of IQ.