Low-dimensional Representation Of Data Research Articles

Cascaded maximum median-margin discriminant projection with its application to face recognition

This paper introduces a novel method called cascaded maximum median-margin discriminant projection (CMMDP) for robust face recognition. Initially, we introduce the maximum median-margin discriminant projection (MMDP) method, which aims to maximize the distances between class medians and their inter-class neighbors, while simultaneously minimizing the distances between class medians and their intra-class samples. To extract features at multiple levels, we extend the MMDP method to its deep version, known as CMMDP. The CMMDP algorithm incorporates the MMDP technique to learn multistage filter banks specifically designed for facial images. Afterward, we utilize straightforward binary hashing techniques for efficient indexing, along with block histograms for effective feature pooling. The CMMDP model is simple and easy to train, while taking full account of the global and local structure of the samples by maximizing the distances between samples that are dissimilar and minimizing the distances between samples that are similar after projection. This results in a low-dimensional data representation with better discriminative performance. We evaluate the performance of CMMDP on face datasets including AR, ORL, ExtYaleB and CMU PIE databases. The results indicate that CMMDP achieves superior performance than most advanced methods.

Transferability of Machine Learning Models for Geogenic Contaminated Groundwaters.

Machine learning models show promise in identifying geogenic contaminated groundwaters. Modeling in regions with no or limited samples is challenging due to the need for large training sets. One potential solution is transferring existing models to such regions. This study explores the transferability of high fluoride groundwater models between basins in the Shanxi Rift System, considering six factors, including modeling methods, predictor types, data size, sample/predictor ratio (SPR), predictor range, and data informing. Results show that transferability is achieved only when model predictors are based on hydrochemical parameters rather than surface parameters. Data informing, i.e., adding samples from challenging regions to the training set, further enhances the transferability. Stepwise regression shows that hydrochemical predictors and data informing significantly improve transferability, while data size, SPR, and predictor range have no significant effects. Additionally, despite their stronger nonlinear capabilities, random forests and artificial neural networks do not necessarily surpass logistic regression in transferability. Lastly, we utilize the t-SNE algorithm to generate low-dimensional representations of data from different basins and compare these representations to elucidate the critical role of predictor types in transferability.

Probing the Diversity of Type Ia Supernova Light Curves in the Open Supernova Catalog

The ever-growing sample of observed supernovae (SNe) enhances our capacity for comprehensive SN population studies, providing a richer data set for understanding the diverse characteristics of Type Ia supernovae (SNe Ia) and possibly those of their progenitors. Here, we present a data-driven analysis of observed SN Ia photometric light curves collected in the Open Supernova Catalog. Where available, we add the environmental information from the host galaxy. We focus on identifying subclasses of SNe Ia without imposing the predefined subclasses found in the literature to date. To do so, we employ an implicit rank-minimizing autoencoder neural network for developing low-dimensional data representations, providing a compact representation of the SN light-curve diversity. When we analyze light curves alone, we find that one of our resulting latent variables is strongly correlated with redshift, allowing us to approximately “de-redshift” the other latent variables describing each event. After doing so, we find that three of our latent variables account for ∼95% of the variance in our sample, and provide a natural separation between 91T and 91bg thermonuclear SNe. Of note, the 02cx subclass is not unambiguously delineated from the 91bg sample in our results, nor do either the overluminous 91T or the underluminous 91bg/02cx samples form a clearly distinct population from the broader sample of “other” SN Ia events. We identify the physical characteristics of SN light curves that best distinguish SNe 91T from SNe 91bg and 02cx, and discuss prospects for future refinements and applications to other classes of SNe as well as other transients.

Automated State Estimation for Summarizing the Dynamics of Complex Urban Systems Using Representation Learning

Complex urban systems can be difficult to monitor, diagnose and manage because the complete states of such systems are only partially observable with sensors. State estimation techniques can be used to determine the underlying dynamic behavior of such complex systems with their highly non-linear processes and external time-variant influences. States can be estimated by clustering observed sensor readings. However, clustering performance degrades as the number of sensors and readings (i.e. feature dimension) increases. To address this problem, we propose a framework that learns a feature-centric lower dimensional representation of data for clustering to support analysis of system dynamics. We propose Unsupervised Feature Attention with Compact Representation (UFACR) to rank features contributing to a cluster assignment. These weighted features are then used to learn a reduced-dimension temporal representation of the data with a deep-learning model. The resulting low-dimensional representation can be effectively clustered into states. UFACR is evaluated on real-world and synthetic wastewater treatment plant data sets, and feature ranking outcomes were validated by Wastewater treatment domain experts. Our quantitative and qualitative experimental analyses demonstrate the effectiveness of UFACR for uncovering system dynamics in an automated and unsupervised manner to offer guidance to wastewater engineers to enhance industrial productivity and treatment efficiency.

A New Mechanism for Eliminating Implicit Conflict in Graph Contrastive Learning

Graph contrastive learning (GCL) has attracted considerable attention because it can self-supervisedly extract low-dimensional representation of graph data. InfoNCE-based loss function is widely used in graph contrastive learning, which pulls the representations of positive pairs close to each other and pulls the representations of negative pairs away from each other. Recent works mainly focus on designing new augmentation methods or sampling strategies. However, we argue that the widely used InfoNCE-based methods may contain an implicit conflict which seriously confuses models when learning from negative pairs. This conflict is engendered by the encoder's message-passing mechanism and the InfoNCE loss function. As a result, the learned representations between negative samples cannot be far away from each other, compromising the model performance. To our best knowledge, this is the first time to report and analysis this conflict of GCL. To address this problem, we propose a simple but effective method called Partial ignored Graph Contrastive Learning (PiGCL). Specifically, PiGCL first dynamically captures the conflicts during training by detecting the gradient of representation similarities. It then enables the loss function to ignore the conflict, allowing the encoder to adaptively learn the ignored information without self-supervised samples. Extensive experiments demonstrate the effectiveness of our method.

Comparing storm resolving models and climates via unsupervised machine learning

Global storm-resolving models (GSRMs) have gained widespread interest because of the unprecedented detail with which they resolve the global climate. However, it remains difficult to quantify objective differences in how GSRMs resolve complex atmospheric formations. This lack of comprehensive tools for comparing model similarities is a problem in many disparate fields that involve simulation tools for complex data. To address this challenge we develop methods to estimate distributional distances based on both nonlinear dimensionality reduction and vector quantization. Our approach automatically learns physically meaningful notions of similarity from low-dimensional latent data representations that the different models produce. This enables an intercomparison of nine GSRMs based on their high-dimensional simulation data (2D vertical velocity snapshots) and reveals that only six are similar in their representation of atmospheric dynamics. Furthermore, we uncover signatures of the convective response to global warming in a fully unsupervised way. Our study provides a path toward evaluating future high-resolution simulation data more objectively.

An Entropy Weighted Nonnegative Matrix Factorization Algorithm for Feature Representation.

Nonnegative matrix factorization (NMF) has been widely used to learn low-dimensional representations of data. However, NMF pays the same attention to all attributes of a data point, which inevitably leads to inaccurate representations. For example, in a human-face dataset, if an image contains a hat on a head, the hat should be removed or the importance of its corresponding attributes should be decreased during matrix factorization. This article proposes a new type of NMF called entropy weighted NMF (EWNMF), which uses an optimizable weight for each attribute of each data point to emphasize their importance. This process is achieved by adding an entropy regularizer to the cost function and then using the Lagrange multiplier method to solve the problem. Experimental results with several datasets demonstrate the feasibility and effectiveness of the proposed method. The code developed in this study is available at https://github.com/Poisson-EM/Entropy-weighted-NMF.

Faster TKD: Towards Lightweight Decomposition for Large-Scale Tensors With Randomized Block Sampling

The Tucker Decomposition (TKD) is able to provide the low-dimensional and informative representations of real-world large-scale tensorial data, which are necessary to extract potential features and enhance the original data. However, computing such decomposition directly for a dense tensor is usually computationally elusive, due to the repetitive operations of computing large-scale tensor-matrix product. Instead of direct decomposition, this paper proposes an efficient algorithm for seeking the Faster TKD of the large-scale tensor, which is a lightweight decomposition approach based on the technique of randomized sampling. The proposed algorithm first converts the original large-scale tensor into a small-scale subtensor via full-mode sampling operation, and then the core tensor of TKD can be computed directly based on the subtensor with low complexity. Finally, an approximate TKD of the original large-scale tensor can be obtained after sequentially computing approximate full-mode factor matrices. A theoretical error analysis is provided to show that the approximation error approximates zero with high probability, and the proposed algorithm is verified based on real tensorial data of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text{23821.24}~GB$</tex-math></inline-formula> .

Biologically informed variational autoencoders allow predictive modeling of genetic and drug-induced perturbations.

Variational autoencoders (VAEs) have rapidly increased in popularity in biological applications and have already successfully been used on many omic datasets. Their latent space provides a low-dimensional representation of input data, and VAEs have been applied, e.g. for clustering of single-cell transcriptomic data. However, due to their non-linear nature, the patterns that VAEs learn in the latent space remain obscure. Hence, the lower-dimensional data embedding cannot directly be related to input features. To shed light on the inner workings of VAE and enable direct interpretability of the model through its structure, we designed a novel VAE, OntoVAE (Ontology guided VAE) that can incorporate any ontology in its latent space and decoder part and, thus, provide pathway or phenotype activities for the ontology terms. In this work, we demonstrate that OntoVAE can be applied in the context of predictive modeling and show its ability to predict the effects of genetic or drug-induced perturbations using different ontologies and both, bulk and single-cell transcriptomic datasets. Finally, we provide a flexible framework, which can be easily adapted to any ontology and dataset. OntoVAE is available as a python package under https://github.com/hdsu-bioquant/onto-vae.

Physics-assisted reduced-order modeling for identifying dominant features of transonic buffet

Transonic buffet is a flow instability phenomenon that arises from the interaction between the shock wave and the separated boundary layer. This flow phenomenon is considered to be highly detrimental during flight and poses a significant risk to the structural strength and fatigue life of aircraft. Up to now, there has been a lack of an accurate, efficient, and intuitive metric to predict buffet and impose a feasible constraint on aerodynamic design. In this paper, a Physics-Assisted Variational Autoencoder is proposed to identify dominant features of transonic buffet, which combines unsupervised reduced-order modeling with additional physical information embedded via a buffet classifier. Specifically, four models with various weights adjusting the contribution of the classifier are trained so as to investigate the impact of buffet information on the latent space. Statistical results reveal that the buffet state can be determined exactly with just one latent space when a proper weight of classifier is chosen. The dominant latent space further reveals a strong relevance with the key flow features located in the boundary layers downstream of shock. Based on this identification, the displacement thickness at 80% chordwise location is proposed as a metric for buffet prediction. This metric achieves an accuracy of 98.5% in buffet state classification, which is more reliable than the existing separation metric used in design. The proposed method integrates the benefits of feature extraction, flow reconstruction, and buffet prediction into a unified framework, demonstrating its potential in low-dimensional representations of high-dimensional flow data and interpreting the “black box” neural network.

XTS: A Hybrid Framework to Detect DNS-Over-HTTPS Tunnels Based on XGBoost and Cooperative Game Theory

This paper proposes a hybrid approach called XTS that uses a combination of techniques to analyze highly imbalanced data with minimum features. XTS combines cost-sensitive XGBoost, a game theory-based model explainer called TreeSHAP, and a newly developed algorithm known as Sequential Forward Evaluation algorithm (SFE). The general aim of XTS is to reduce the number of features required to learn a particular dataset. It assumes that low-dimensional representation of data can improve computational efficiency and model interpretability whilst retaining a strong prediction performance. The efficiency of XTS was tested on a public dataset, and the results showed that by reducing the number of features from 33 to less than five, the proposed model achieved over 99.9% prediction efficiency. XTS was also found to outperform other benchmarked models and existing proof-of-concept solutions in the literature. The dataset contained data related to DNS-over-HTTPS (DoH) tunnels. The top predictors for DoH classification and characterization were identified using interactive SHAP plots, which included destination IP, packet length mode, and source IP. XTS offered a promising approach to improve the efficiency of the detection and analysis of DoH tunnels while maintaining accuracy, which can have important implications for behavioral network intrusion detection systems.

Fidelity of hyperbolic space for Bayesian phylogenetic inference.

Bayesian inference for phylogenetics is a gold standard for computing distributions of phylogenies. However, Bayesian phylogenetics faces the challenging computational problem of moving throughout the high-dimensional space of trees. Fortunately, hyperbolic space offers a low dimensional representation of tree-like data. In this paper, we embed genomic sequences as points in hyperbolic space and perform hyperbolic Markov Chain Monte Carlo for Bayesian inference in this space. The posterior probability of an embedding is computed by decoding a neighbour-joining tree from the embedding locations of the sequences. We empirically demonstrate the fidelity of this method on eight data sets. We systematically investigated the effect of embedding dimension and hyperbolic curvature on the performance in these data sets. The sampled posterior distribution recovers the splits and branch lengths to a high degree over a range of curvatures and dimensions. We systematically investigated the effects of the embedding space's curvature and dimension on the Markov Chain's performance, demonstrating the suitability of hyperbolic space for phylogenetic inference.

Deep feature screening: Feature selection for ultra high-dimensional data via deep neural networks

The applications of traditional statistical feature selection methods to high-dimension, low-sample-size data often struggle and encounter challenging problems, such as overfitting, curse of dimensionality, computational infeasibility, and strong model assumptions. In this paper, we propose a novel two-step nonparametric approach called Deep Feature Screening (DeepFS) that can overcome these problems and identify significant features with high precision for ultra high-dimensional, low-sample-size data. This approach first extracts a low-dimensional representation of input data and then applies feature screening on the original input feature space based on multivariate rank distance correlation recently developed by Deb and Sen (2021). This approach combines the strengths of both deep neural networks and feature screening, thereby having the following appealing features in addition to its ability of handling ultra high-dimensional data with small number of samples: (1) it is model free and distribution free; (2) it can be used for both supervised and unsupervised feature selection; and (3) it is capable of recovering the original input data. The superiority of DeepFS is demonstrated via extensive simulation studies and real data analyses.

Fatigue monitoring and maneuver identification for vehicle fleets using a virtual sensing approach

Extensive monitoring comes at a prohibitive cost, limiting Predictive Maintenance strategies for vehicle fleets. This paper presents a measurement-based virtual sensing technique where local strain gauges are only required for few reference vehicles, while the remaining fleet relies exclusively on accelerometers. The scattering transform is used to perform feature extraction, while principal component analysis provides a reduced, low dimensional data representation. This enables direct fatigue damage regression, parameterized from unlabeled usage data. Identification measurements allow for a physical interpretation of the reduced representation. The approach is demonstrated using experimental data from a sensor equipped eBike, which is made publicly available.

Toward Robust Fault Identification of Complex Industrial Processes Using Stacked Sparse-Denoising Autoencoder With Softmax Classifier.

This article proposes a robust end-to-end deep learning-induced fault recognition scheme by stacking multiple sparse-denoising autoencoders with a Softmax classifier, called stacked spare-denoising autoencoder (SSDAE)-Softmax, for the fault identification of complex industrial processes (CIPs). Specifically, sparse denoising autoencoder (SDAE) is established by integrating a sparse AE (SAE) with a denoising AE (DAE) for the low-dimensional but intrinsic feature representation of the CIP monitoring data (CIPMD) with possible noise contamination. SSDAE-Softmax is established by stacking multiple SDAEs with a layerwise pretraining procedure, and a Softmax classifier with a global fine-tuning strategy. Furthermore, SSDAE-Softmax hyperparameters are optimized by a relatively new global optimization algorithm, referred to as the state transition algorithm (STA). Benefiting from the deep learning-based feature representation scheme with the STA-based hyperparameter optimization, the underlying intrinsic characteristics of CIPMD can be learned automatically and adaptively for accurate fault identification. A numeric simulation system, the benchmark Tennessee Eastman process (TEP), and a real industrial process, that is, the continuous casting process (CCP) from a top steel plant of China, are used to validate the performance of the proposed method. Experimental results show that the proposed SSDAE-Softmax model can effectively identify various process faults, and has stronger robustness and adaptability against the noise interference in CIPMD for the process monitoring of CIPs.

Invertible Kernel PCA With Random Fourier Features

Kernel principal component analysis (kPCA) is a widely studied method to construct a low-dimensional data representation after a nonlinear transformation. The prevailing method to reconstruct the original input signal from kPCA—an important task for denoising—requires us to solve a supervised learning problem. In this paper, we present an alternative method where the reconstruction follows naturally from the compression step. We first approximate the kernel with random Fourier features. Then, we exploit the fact that the nonlinear transformation is invertible in a certain subdomain. Hence, the name <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">invertible kernel PCA (ikPCA)</i> . We experiment with different data modalities and show that ikPCA performs similarly to kPCA with supervised reconstruction on denoising tasks, making it a strong alternative.

Multiple Graph Adaptive Regularized Semi-Supervised Nonnegative Matrix Factorization with Sparse Constraint for Data Representation

Multiple graph and semi-supervision techniques have been successfully introduced into the nonnegative matrix factorization (NMF) model for taking full advantage of the manifold structure and priori information of data to capture excellent low-dimensional data representation. However, the existing methods do not consider the sparse constraint, which can enhance the local learning ability and improve the performance in practical applications. To overcome this limitation, a novel NMF-based data representation method, namely, the multiple graph adaptive regularized semi-supervised nonnegative matrix factorization with sparse constraint (MSNMFSC) is developed in this paper for obtaining the sparse and discriminative data representation and increasing the quality of decomposition of NMF. Particularly, based on the standard NMF, the proposed MSNMFSC method combines the multiple graph adaptive regularization, the limited supervised information and the sparse constraint together to learn the more discriminative parts-based data representation. Moreover, the convergence analysis of MSNMFSC is studied. Experiments are conducted on several practical image datasets in clustering tasks, and the clustering results have shown that MSNMFSC achieves better performance than several most related NMF-based methods.

Spatially aware dimension reduction for spatial transcriptomics

Spatial transcriptomics are a collection of genomic technologies that have enabled transcriptomic profiling on tissues with spatial localization information. Analyzing spatial transcriptomic data is computationally challenging, as the data collected from various spatial transcriptomic technologies are often noisy and display substantial spatial correlation across tissue locations. Here, we develop a spatially-aware dimension reduction method, SpatialPCA, that can extract a low dimensional representation of the spatial transcriptomics data with biological signal and preserved spatial correlation structure, thus unlocking many existing computational tools previously developed in single-cell RNAseq studies for tailored analysis of spatial transcriptomics. We illustrate the benefits of SpatialPCA for spatial domain detection and explores its utility for trajectory inference on the tissue and for high-resolution spatial map construction. In the real data applications, SpatialPCA identifies key molecular and immunological signatures in a detected tumor surrounding microenvironment, including a tertiary lymphoid structure that shapes the gradual transcriptomic transition during tumorigenesis and metastasis. In addition, SpatialPCA detects the past neuronal developmental history that underlies the current transcriptomic landscape across tissue locations in the cortex.

Hybrid Kronecker Product Decomposition and Approximation

Discovering underlying low dimensional structure of a high-dimensional matrix is traditionally done through low rank matrix approximations in the form of a sum of rank-one matrices. In this article, we propose a new approach. We assume a high-dimensional matrix can be approximated by a sum of a small number of Kronecker products of matrices with potentially different configurations, named as a hybrid Kronecker outer Product Approximation (hKoPA). It provides an extremely flexible way of dimension reduction compared to the low-rank matrix approximation. Challenges arise in estimating a hKoPA when the configurations of component Kronecker products are different or unknown. We propose an estimation procedure when the set of configurations are given, and a joint configuration determination and component estimation procedure when the configurations are unknown. Specifically, a least squares backfitting algorithm is used when the configurations are given. When the configurations are unknown, an iterative greedy algorithm is developed. Both simulation and real image examples show that the proposed algorithms have promising performances. Some identifiability conditions are also provided. The hybrid Kronecker product approximation may have potentially wider applications in low dimensional representation of high-dimensional data. Supplementary materials for this article are available online.

Analysis of UMAP, the method for reducing the dimensionality of initial data in machine learning for the purpose of failure prediction in a motive power service

Aim.Feature transformation is one of the stages of machine learning applicationthat has a significant effect on the quality of regression models. The paper aims to developcriteria for evaluating the quality of data dimensionality reduction at the stage of featuretransformation and adaptation of the UMAP method to the problem of prediction of the numberof days to failure in the locomotives of JSC RZD.Methods.The data transformation methodsare divided into two groups, those that attempt to preserve the global data structure, andthose that attempt to preserve the distances between points. The paper examines in detail theUMAP no-linear method of dimensionality reduction, whose low-dimensional data presentationis based on a transformation of a nearest neighbour graph retaining the data structure. Thestructure of the initial data manifold is examined using topological data analysis and simplifiedfuzzy set construction methods.Results.The analysis of UMAP theory conducted in theRussian language for the first time enabled a substantiated identification of the three primaryparameters of the method, whose variation significantly affects the type of data obtained asthe result of a transformation. In particular, that pertains to the quality of class separationover a two-dimensional space. Additionally, the characteristics of the input set of parameterswere identified that affect the UMAP results. Practical results of UMAP application weredemonstrated. Intermediate results included a list of nearest neighbours, a weighted graph of nearest neighbours. The fundamental result is a low-dimensional data representation (outof 44 initial measurements) over a two-dimensional space with class separation, which isconfirmed both by calculations, and visually.Conclusions.It was identified that UMAP is anefficient and substantiated method of dimensionality reduction that allows – through parametervariation – transforming data in such a way as to improve the quality of data submitted tomachine learning models by the criterion of “evident class separation”. The transformation is anintermediate stage of data preparation for regression model application, and class separationwas performed for the purpose of eliminating the probability of gross regression errors.