Local Dimensionality Reduction Research Articles

Novel modified weights and cosine similarity based maximum marginal projection and its application in fault diagnosis

Abstract Facing the problem that the data generated in industrial processes have few labeled samples and the local manifold learning dimensionality reduction method ignores the local spatial structure of sample points and the distance relationship in constructing different weights. To solve the above problems, this paper presents a novel modified weights and cosine similarity based maximum marginal projection named MCMMP. In MCMMP, cosine similarity is used to consider the space feature of sample points, which enhances the performance of dimensionality reduction. The new modified weights are applied to measure the between-class and the within-class sample points, which enhance the divisibility of sample points. After MCMMP dimensionality reduction, the classifier is used to classify the dimensionality reduction sample points. Finally, the proposed new method is used in two cases Tennessee Eastman Process (TEP) and Three-phase Flow Facility (TFF) to test the fault diagnosis performance. The results of the simulation process indicated that the new fault diagnosis method based on MCMMP, compared with other related diagnosis methods, has good performance.

Learning adaptive Grassmann neighbors for image-set analysis

Representing image sets as subspaces on Grassmann manifold and leveraging the Riemannian geometry of this space has proven to be highly effective in various visual recognition tasks. However, the computational cost associated with operating on Riemannian manifolds, especially of high-dimensional ones, poses a significant limitation to the applicability of existing techniques. Moreover, the image sets with labels are scarce and valuable in real-world scenarios, where the manual annotation is time-consuming and costly. To tackle these issues, in this paper, we propose an unsupervised Grassmannian dimensionality reduction method for image set classification and clustering, which simultaneously performs local structure learning and dimensionality reduction under the subspace view. Specifically, instead of relying on the pre-defined similarity matrix that obtained from raw data and may contain much noise, we adaptively learn the local structures from the projected low-dimensional Grassmann manifold. The refined structures then provide more discriminative power to learn the accurate manifold-to-manifold mapping. To better preserve the global structure of the data, this mapping is constrained to satisfy the orthogonal condition. In addition, our model can be compatible with F-norm OLPP for Grassmann manifold when the squared F-norm is transformed into projection metric. A re-weighted iterative scheme is carefully developed to optimize the problem. Experimental results on several benchmark datasets for image set classification and clustering tasks clearly demonstrate that our model consistently outperforms baselines, yielding significant improvements.

Local nonlinear dimensionality reduction via preserving the geometric structure of data

Dimensionality reduction has many applications in data visualization and machine learning. Existing methods can be classified into global ones and local ones. The global methods usually learn the linear relationship in data, while the local ones learn the manifold intrinsic geometry structure, which has a significant impact on pattern recognition. However, most of existing local methods obtain an embedding with eigenvalue or singular value decomposition, where the computational complexities are very high in a large amount of high-dimensional data. In this paper, we propose a local nonlinear dimensionality reduction method named Vec2vec, which employs a neural network with only one hidden layer to reduce the computational complexity. We first build a neighborhood similarity graph from the input matrix, and then define the context of data points with the random walk properties in the graph. Finally, we train the neural network with the context of data points to learn the embedding of the matrix. We conduct extensive experiments of data classification and clustering on nine image and text datasets to evaluate the performance of our method. Experimental results show that Vec2vec is better than several state-of-the-art dimensionality reduction methods, except that it is equivalent to UMAP on data clustering tasks in the statistical hypothesis tests, but Vec2vec needs less computational time than UMAP in high-dimensional data. Furthermore, we propose a more lightweight method named Approximate Vec2vec (AVec2vec) with little performance degradation, which employs an approximate method to build the neighborhood similarity graph. AVec2vec is still better than some state-of-the-art local dimensionality reduction methods and competitive with UMAP on data classification and clustering tasks in the statistical hypothesis tests.

A transparent artificial intelligence framework to assess lung disease in pulmonary hypertension

Recent studies have recognized the importance of characterizing the extent of lung disease in pulmonary hypertension patients by using Computed Tomography. The trustworthiness of an artificial intelligence system is linked with the depth of the evaluation in functional, operational, usability, safety and validation dimensions. The safety and validation of an artificial tool is linked to the uncertainty estimation of the model’s prediction. On the other hand, the functionality, operation and usability can be achieved by explainable deep learning approaches which can verify the learning patterns and use of the network from a generalized point of view. We developed an artificial intelligence framework to map the 3D anatomical models of patients with lung disease in pulmonary hypertension. To verify the trustworthiness of the framework we studied the uncertainty estimation of the network’s prediction, and we explained the learning patterns of the network. Therefore, a new generalized technique combining local explainable and interpretable dimensionality reduction approaches (PCA-GradCam, PCA-Shape) was developed. Our open-source software framework was evaluated in unbiased validation datasets achieving accurate, robust and generalized results.

Integrated proteomic and metabolomic modules identified as biomarkers of mortality in the Atherosclerosis Risk in Communities study and the African American Study of Kidney Disease and Hypertension

BackgroundProteins and metabolites are essential for many biological functions and often linked through enzymatic or transport reactions. Individual molecules have been associated with all-cause mortality. Many of these are correlated and might jointly represent pathways or endophenotypes involved in diseases.ResultsWe present an integrated analysis of proteomics and metabolomics via a local dimensionality reduction clustering method. We identified 224 modules of correlated proteins and metabolites in the Atherosclerosis Risk in Communities (ARIC) study, a general population cohort of older adults (N = 4046, mean age 75.7, mean eGFR 65). Many of the modules displayed strong cross-sectional associations with demographic and clinical characteristics. In comprehensively adjusted analyses, including fasting plasma glucose, history of cardiovascular disease, systolic blood pressure and kidney function among others, 60 modules were associated with mortality. We transferred the network structure to the African American Study of Kidney Disease and Hypertension (AASK) (N = 694, mean age 54.5, mean mGFR 46) and identified mortality associated modules relevant in this disease specific cohort. The four mortality modules relevant in both the general population and CKD were all a combination of proteins and metabolites and were related to diabetes / insulin secretion, cardiovascular disease and kidney function. Key components of these modules included N-terminal (NT)-pro hormone BNP (NT-proBNP), Sushi, Von Willebrand Factor Type A, EGF And Pentraxin (SVEP1), and several kallikrein proteases.ConclusionThrough integrated biomarkers of the proteome and metabolome we identified functions of (patho-) physiologic importance related to diabetes, cardiovascular disease and kidney function.

Fault Diagnosis Using Improved Discrimination Locality Preserving Projections Integrated With Sparse Autoencoder

In order to ensure safe operations of industrial processes, it is especially important to find fault types accurately and quickly based on historical data and then deal with faults in time. Unfortunately, due to the tricky characteristics of industrial process data such as massive number, high dimensionality, and nonlinearity, it turns out that timely and accurate diagnosis of faults becomes of great difficulty in industrial processes. To address this problem, novel effective fault diagnosis using an improved global and local dimensionality reduction (DR) method named discrimination locality preserving projections integrated with sparse autoencoder (SAEDLPP) is proposed in this article. In SAEDLPP, the global DR information of data is first obtained by sparse autoencoder (SAE); next, the DR data obtained through SAE are passed through discrimination locality preserving projections (DLPP) to obtain local DR information, preserving not only the global information but the local information of the extracted features. Finally, fault diagnosis is achieved by separating the extracted features by SAEDLPP using an AdaBoost classifier to recognize fault types. Simulations are conducted on the Tennessee Eastman process (TEP) and the results indicate that the provided SAEDLPP-based fault diagnosis methodology can achieve much higher accuracy in fault diagnosis than other associated methods.

An Orthogonal Locality and Globality Dimensionality Reduction Method Based on Twin Eigen Decomposition

Dimensionality reduction is a hot research topic in pattern recognition. Traditional dimensionality reduction methods can be separated into linear dimensionality reduction methods and nonlinear dimensionality reduction methods. Linear dimensionality reduction methods usually utilize Euclidean distances to explore global geometric structures, and nonlinear dimensionality reduction methods can preserve local manifold structures in learned low-dimensional subspaces. However, redundant information and noises of the raw high-dimensional data restrict the classification performance of these methods. To solve the problem, we propose a novel orthogonal dimensionality reduction method based on twin eigen decomposition called orthogonal locality and globality preserving projections (OLGPP). Orthogonality, as a commonly used criterion in pattern recognition, is insensitive to the redundant information and the noises. OLGPP not only combines the advantages of global Euclidean structures and local manifold structures, but also is insensitive to the redundant information and noises of the raw high-dimensional data by embedding the orthogonality criterion. Additionally, the objective function based on twin eigen decomposition can be solved sequentially to obtain analytical solutions. On four well-known datasets, we design extensive experiments to evaluate the performance of OLGPP. From the experimental results, OLGPP has the optimal experimental results, and its average recognition rates are about 10% higher than the classic local dimensionality reduction method (i.e., locality preserving projection), which proves OLGPP is an effective dimensionality reduction method.

Longitudinal assessment of carotid plaque texture in three-dimensional ultrasound images based on semi-supervised graph-based dimensionality reduction and feature selection

With continuous development of therapeutic options for atherosclerosis, image-based biomarkers sensitive to the effect of new interventions are required to be developed for cost-effective clinical evaluation. Although 3D ultrasound measurement of total plaque volume (TPV) showed the efficacy of high-dose statin, more sensitive biomarkers are needed to establish the efficacy of dietary supplements expected to confer a smaller beneficial effect. This study involved 171 subjects who participated in a one-year placebo-controlled trial evaluating the effect of pomegranate. A framework involving a feature selection technique known as discriminative feature selection (DFS) and a semi-supervised graph-based regression (SSGBR) technique was proposed for sensitive detection of plaque textural changes over the trial. 376 textual features of plaques were extracted from 3D ultrasound images acquired at baseline and a follow-up session. A scalar biomarker for each subject were generated by SSGBR based on prominent textural features selected by DFS. The ability of this biomarker for discriminating pomegranate from placebo subjects was quantified by the p-values obtained in Mann–Whitney U test. The discriminative power of SSGBR was compared with global and local dimensionality reduction techniques, including linear discriminant analysis (LDA), maximum margin criterion (MMC) and Laplacian Eigenmap (LE). Only SSGBR (p=4.12×10−6) and normalized LE (p=0.002) detected a difference between the two groups at the 5% significance level. As compared with ΔTPV, SSGBR reduced the sample size required to establish a significant difference by a factor of 60. The application of this framework will substantially reduce the cost incurred in clinical trials.

Computational analysis of viable parameter regions in models of synthetic biological systems

BackgroundGene regulatory networks with different topological and/or dynamical properties might exhibit similar behavior. System that is less perceptive for the perturbations of its internal and external factors should be preferred. Methods for sensitivity and robustness assessment have already been developed and can be roughly divided into local and global approaches. Local methods focus only on the local area around nominal parameter values. This can be problematic when parameters exhibits the desired behavior over a large range of parameter perturbations or when parameter values are unknown. Global methods, on the other hand, investigate the whole space of parameter values and mostly rely on different sampling techniques. This can be computationally inefficient. To address these shortcomings ’glocal’ approaches were developed that apply global and local approaches in an effective and rigorous manner.ResultsHerein, we present a computational approach for ’glocal’ analysis of viable parameter regions in biological models. The methodology is based on the exploration of high-dimensional viable parameter spaces with global and local sampling, clustering and dimensionality reduction techniques. The proposed methodology allows us to efficiently investigate the viable parameter space regions, evaluate the regions which exhibit the largest robustness, and to gather new insights regarding the size and connectivity of the viable parameter regions. We evaluate the proposed methodology on three different synthetic gene regulatory network models, i.e. the repressilator model, the model of the AC-DC circuit and the model of the edge-triggered master-slave D flip-flop.ConclusionsThe proposed methodology provides a rigorous assessment of the shape and size of viable parameter regions based on (1) the mathematical description of the biological system of interest, (2) constraints that define feasible parameter regions and (3) cost function that defines the desired or observed behavior of the system. These insights can be used to assess the robustness of biological systems, even in the case when parameter values are unknown and more importantly, even when there are multiple poorly connected viable parameter regions in the solution space. Moreover, the methodology can be efficiently applied to the analysis of biological systems that exhibit multiple modes of the targeted behavior.

Quantifying and connecting atomic and crystallographic grain boundary structure using local environment representation and dimensionality reduction techniques

In this work, the relationship between grain boundary crystallography and grain boundary atomic structure is examined, using [100] - symmetric tilt grain boundaries in nickel. The structural unit model is used as a benchmark to evaluate the atomic structure description capacities of an emerging structural descriptor, the local environment representation, which itself is a refinement of the Smooth Overlap of Atomic Positions (SOAP) descriptor. We show that the local environment representation encodes both the information of the structural unit model and additional information, such as distortion in the structural units and the arrangement of the structural units at the interface. The use of the local environment representation permits the use of a visualization tool known as SPRING to represent structural similarities between grain boundaries. With the SPRING representation, we produce objective evidence of a relationship between crystallography and atomic structure, at least for [100] - symmetric tilt grain boundaries.

Evaluation of Histograms Local Features and Dimensionality Reduction for 3D Face Verification

Evaluation of Histograms Local Features and Dimensionality Reduction for 3D Face Verification

Sparse Constrained Motion Synthesis Using Local Regression Models

Based on sparse control constraints, one diffi- culty for synthesizing natural human motions is that low- dimensional control information can not be directly used to construct high-dimensional human poses. This paper intro- duces a novel and powerful local dimensionality reduction approach for synthesizing accurate and natural full-body human motions. The approach is to construct a group of online local dynamic regression models from a pre-captured motion database as a prior to support the full-body human action synthesis. By synthesizing a variety of human motions from as possible as few sparse constraints provided by users, the paper verifies the effectiveness of the proposed approach. Compared with previous statistical models, our model can synthesize more accurate results.

Decision Fusion in Kernel-Induced Spaces for Hyperspectral Image Classification

The one-against-one (OAO) strategy is commonly employed with classifiers-such as support vector machines-which inherently provide binary two-class classification in order to handle multiple classes. This OAO strategy is introduced for the classification of hyperspectral imagery using discriminant analysis within kernel-induced feature spaces, producing a pair of algorithms-kernel discriminant analysis and kernel local Fisher discriminant analysis-for dimensionality reduction, which are followed by a quadratic Gaussian maximum-likelihood-estimation classifier. In the proposed approach, a multiclass problem is broken down into all possible binary classifiers, and various decision-fusion rules are considered for merging results from this classifier ensemble. Experimental results using several hyperspectral data sets demonstrate the benefits of the proposed approach-in addition to improved classification performance, the resulting classifier framework requires reduced memory for estimating kernel matrices.

Learning local factor analysis versus mixture of factor analyzers with automatic model selection

Considering Factor Analysis (FA) for each component of Gaussian Mixture Model (GMM), clustering and local dimensionality reduction can be addressed simultaneously by Mixture of Factor Analyzers (MFA) and Local Factor Analysis (LFA), which correspond to two FA parameterizations, respectively. This paper investigates the performance of Variational Bayes (VB) and Bayesian Ying-Yang (BYY) harmony learning on MFA/LFA for the problem of automatically determining the component number and the local hidden dimensionalities (i.e., the number of factors of FA in each component). Similar to the existing VB learning algorithm on MFA, we develop an alternative VB algorithm on LFA with a similar conjugate Dirichlet–Normal–Gamma (DNG) prior on all parameters of LFA. Also, the corresponding BYY algorithms are developed for MFA and LFA. A wide range of synthetic experiments shows that LFA is superior to MFA in model selection under either VB or BYY, while BYY outperforms VB reliably on both MFA and LFA. These empirical findings are consistently observed from real applications on not only face and handwritten digit images clustering, but also unsupervised image segmentation.

Optimal/Near-Optimal Dimensionality Reduction for Distributed Estimation in Homogeneous and Certain Inhomogeneous Scenarios

We consider distributed estimation of a deterministic vector parameter from noisy sensor observations in a wireless sensor network (WSN). The observation noise is assumed uncorrelated across sensors. To meet stringent power and bandwidth budgets inherent in WSNs, local data dimensionality reduction is performed at each sensor to reduce the number of messages sent to a fusion center (FC). The problem of interest is to jointly design the compression matrices associated with those sensors, aiming at minimizing the estimation error at the FC. Such a dimensionality reduction problem is investigated in this paper. Specifically, we study a homogeneous environment where all sensors have identical noise covariance matrices and an inhomogeneous environment where the noise covariance matrices across the sensors have the same correlation structure but with different scaling factors. Given a total number of messages sent to the FC, theoretical lower bounds on the estimation error of any compression strategy are derived for both cases. Compression strategies are developed to approach or even attain the corresponding theoretical lower bounds. Performance analysis and simulations are carried out to illustrate the optimality and effectiveness of the proposed compression strategies.

Electrostatic field framework for supervised and semi-supervised learning from incomplete data

In this paper a classification framework for incomplete data, based on electrostatic field model is proposed. An original approach to exploiting incomplete training data with missing features, involving extensive use of electrostatic charge analogy, has been used. The framework supports a hybrid supervised and unsupervised training scenario, enabling learning simultaneously from both labelled and unlabelled data using the same set of rules and adaptation mechanisms. Classification of incomplete patterns has been facilitated by introducing a local dimensionality reduction technique, which aims at exploiting all available information using the data `as is', rather than trying to estimate the missing values. The performance of all proposed methods has been extensively tested in a wide range of missing data scenarios, using a number of standard benchmark datasets in order to make the results comparable with those available in current and future literature. Several modifications to the original Electrostatic Field Classifier aiming at improving speed and robustness in higher dimensional spaces have also been introduced and discussed.

General Cost Models for Evaluating Dimensionality Reduction in High-Dimensional Spaces

Similarity search usually encounters a serious problem in the high-dimensional space, known as the curse of dimensionality. In order to speed up the retrieval efficiency, most previous approaches reduce the dimensionality of the entire data set to a fixed lower value before building indexes (referred to as global dimensionality reduction (GDR)). More recent works focus on locally reducing the dimensionality of data to different values (called the local dimensionality reduction (LDR)). In addition, random projection is proposed as an approximate dimensionality reduction (ADR) technique to answer the approximate similarity search instead of the exact one. However, so far little work has formally evaluated the effectiveness and efficiency of GDR, LDR, and ADR for the range query. Motivated by this, in this paper, we propose general cost models for evaluating the query performance over the reduced data sets by GDR, LDR, and ADR, in light of which we introduce a novel (A)LDR method, partitioning based on randomized search (PRANS). It can achieve high retrieval efficiency with the guarantee of optimality given by the formal models. Finally, a B+-tree index is constructed over the reduced partitions for fast similarity search. Extensive experiments validate the correctness of our cost models on both real and synthetic data sets and demonstrate the efficiency and effectiveness of the proposed PRANS method.

Factor Analysis Latent Subspace Modeling and Robust Fuzzy Clustering Using $t$-Distributions

Factor analysis is a latent subspace model commonly used for local dimensionality reduction tasks. Fuzzy <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</i> -means (FCM) type fuzzy clustering approaches are closely related to Gaussian mixture models (GMMs), and expectation-maximization (EM) like algorithms have been employed in fuzzy clustering with regularized objective functions. Student's <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</i> -mixture models (SMMs) have been proposed recently as an alternative to GMMs, resolving their outlier vulnerability problems. In this paper, we propose a novel FCM-type fuzzy clustering scheme providing two significant benefits when compared with the existing approaches. First, it provides a well-established observation space dimensionality reduction framework for fuzzy clustering algorithms based on factor analysis, allowing concurrent performance of fuzzy clustering and, within each cluster, local dimensionality reduction. Second, it exploits the outlier tolerance advantages of SMMs to provide a novel, soundly founded, nonheuristic, robust fuzzy clustering framework by introducing the effective means to incorporate the explicit assumption about student's <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</i> -distributed data into the fuzzy clustering procedure. This way, the proposed model yields a significant performance increase for the fuzzy clustering algorithm, as we experimentally demonstrate.

Local Dimensionality Reduction for Non-Parametric Regression

Locally-weighted regression is a computationally-efficient technique for non-linear regression. However, for high-dimensional data, this technique becomes numerically brittle and computationally too expensive if many local models need to be maintained simultaneously. Thus, local linear dimensionality reduction combined with locally-weighted regression seems to be a promising solution. In this context, we review linear dimensionality-reduction methods, compare their performance on non-parametric locally-linear regression, and discuss their ability to extend to incremental learning. The considered methods belong to the following three groups: (1) reducing dimensionality only on the input data, (2) modeling the joint input-output data distribution, and (3) optimizing the correlation between projection directions and output data. Group 1 contains principal component regression (PCR); group 2 contains principal component analysis (PCA) in joint input and output space, factor analysis, and probabilistic PCA; and group 3 contains reduced rank regression (RRR) and partial least squares (PLS) regression. Among the tested methods, only group 3 managed to achieve robust performance even for a non-optimal number of components (factors or projection directions). In contrast, group 1 and 2 failed for fewer components since these methods rely on the correct estimate of the true intrinsic dimensionality. In group 3, PLS is the only method for which a computationally-efficient incremental implementation exists. Thus, PLS appears to be ideally suited as a building block for a locally-weighted regressor in which projection directions are incrementally added on the fly.

Stable local dimensionality reduction approaches

Dimensionality reduction is a big challenge in many areas. A large number of local approaches, stemming from statistics or geometry, have been developed. However, in practice these local approaches are often in lack of robustness, since in contrast to maximum variance unfolding (MVU), which explicitly unfolds the manifold, they merely characterize local geometry structure. Moreover, the eigenproblems that they encounter, are hard to solve. We propose a unified framework that explicitly unfolds the manifold and reformulate local approaches as the semi-definite programs instead of the above-mentioned eigenproblems. Three well-known algorithms, locally linear embedding (LLE), laplacian eigenmaps (LE) and local tangent space alignment (LTSA) are reinterpreted and improved within this framework. Several experiments are presented to demonstrate the potential of our framework and the improvements of these local algorithms.