Supervised Dimensionality Reduction Research Articles

Learning active subspaces and discovering important features with Gaussian radial basis functions neural networks

Providing a model that achieves a strong predictive performance and is simultaneously interpretable by humans is one of the most difficult challenges in machine learning research due to the conflicting nature of these two objectives. To address this challenge, we propose a modification of the radial basis function neural network model by equipping its Gaussian kernel with a learnable precision matrix. We show that precious information is contained in the spectrum of the precision matrix that can be extracted once the training of the model is completed. In particular, the eigenvectors explain the directions of maximum sensitivity of the model revealing the active subspace and suggesting potential applications for supervised dimensionality reduction. At the same time, the eigenvectors highlight the relationship in terms of absolute variation between the input and the latent variables, thereby allowing us to extract a ranking of the input variables based on their importance to the prediction task enhancing the model interpretability. We conducted numerical experiments for regression, classification, and feature selection tasks, comparing our model against popular machine learning models, the state-of-the-art deep learning-based embedding feature selection techniques, and a transformer model for tabular data. Our results demonstrate that the proposed model does not only yield an attractive prediction performance compared to the competitors but also provides meaningful and interpretable results that potentially could assist the decision-making process in real-world applications. A PyTorch implementation of the model is available on GitHub at the following link.11https://github.com/dannyzx/Gaussian-RBFNN.

Supervised dimension reduction for functional time series

Supervised dimension reduction for functional time series

Chemical space deconstruction-based dynamic model ensemble architecture for molecular property prediction

Green solvents, catalysts, functional materials, drugs, and other chemical products now have a quick engine for design thanks to machine learning (ML) based prediction of molecular properties. However, the accuracy and stability of ML-based models can be impeded by poor data quality, which is rarely studied in chemical product discovery and design. Inspired by the dynamic ensemble selection (DES), an improved DES based on chemical space deconstruction is proposed in this work to accommodate the prediction task of molecular properties. We innovatively developed a chemical space representation and deconstruction model based on a self-organizing mapping (SOM) neural network, facilitating the rapid implementation of the improved DES on molecular samples. Consequently, a novel dynamic model ensemble architecture (SOM-DES) is proposed as a model enhancement technology to build a more accurate and stable ensemble model, aiming to improve the predictive performance on the chemical subspace within poor-quality data. To achieve the architecture, a supervised dimensionality reduction algorithm has been improved to enhance the deep mining of molecular feature information for DES optimization. Additionally, a novel resampling strategy based on the combination of the geometric synthetic minority oversampling technique (G-SMOTE) algorithm and chemical space deconstruction, as a data augmentation technology, has been proposed for mitigating the disadvantage of unbalanced data during DES training. The prediction task for enthalpy of formation of the ideal gas is applied as a case study to demonstrate the superiority of the proposed SOM-DES. The results indicate that the proposed SOM-DES (R2 = 0.9731, RMSE = 55.4639) outperforms the traditional static ensemble strategy (SES, R2 = 0.9552, RMSE = 71.5045) in terms of global chemical spatial precision. More importantly, for chemical subspaces that are difficult to predict due to low data quality, SOM-DES shows a significant reduction in prediction errors compared to SES.

Toward Robust Discriminative Projections Learning Against Adversarial Patch Attacks.

As one of the most popular supervised dimensionality reduction methods, linear discriminant analysis (LDA) has been widely studied in machine learning community and applied to many scientific applications. Traditional LDA minimizes the ratio of squared l2 norms, which is vulnerable to the adversarial examples. In recent studies, many l1 -norm-based robust dimensionality reduction methods are proposed to improve the robustness of model. However, due to the difficulty of l1 -norm ratio optimization and weakness on defending a large number of adversarial examples, so far, scarce works have been proposed to utilize sparsity-inducing norms for LDA objective. In this article, we propose a novel robust discriminative projections learning (rDPL) method based on the l1,2 -norm trace-ratio minimization optimization algorithm. Minimizing the l1,2 -norm ratio problem directly is a much more challenging problem than the traditional methods, and there is no existing optimization algorithm to solve such nonsmooth terms ratio problem. We derive a new efficient algorithm to solve this challenging problem and provide a theoretical analysis on the convergence of our algorithm. The proposed algorithm is easy to implement and converges fast in practice. Extensive experiments on both synthetic data and several real benchmark datasets show the effectiveness of the proposed method on defending the adversarial patch attack by comparison with many state-of-the-art robust dimensionality reduction methods.

A Revised Formation of Trace Ratio LDA for Small Sample Size Problem.

Linear discriminant analysis (LDA) is a classic tool for supervised dimensionality reduction. Because the projected samples can be classified effectively, LDA has been successfully applied in many applications. Among the variants of LDA, trace ratio LDA (TR-LDA) is a classic form due to its explicit meaning. Unfortunately, when the sample size is much smaller than the data dimension, the algorithm for solving TR-LDA does not converge. The so-called small sample size (SSS) problem severely limits the application of TR-LDA. To solve this problem, we propose a revised formation of TR-LDA, which can be applied to datasets with different sizes in a unified form. Then, we present an optimization algorithm to solve the proposed method, explain why it can avoid the SSS problem, and analyze the convergence and computational complexity of the optimization algorithm. Next, based on the introduced theorems, we quantitatively elaborate on when the SSS problem will occur in TR-LDA. Finally, the experimental results on real-world datasets demonstrate the effectiveness of the proposed method.

BrainPepPass: A Framework Based on Supervised Dimensionality Reduction for Predicting Blood-Brain Barrier-Penetrating Peptides.

Peptides that pass through the blood-brain barrier (BBB) not only are implicated in brain-related pathologies but also are promising therapeutic tools for treating brain diseases, e.g., as shuttles carrying active medicines across the BBB. Computational prediction of BBB-penetrating peptides (B3PPs) has emerged as an interesting approach because of its ability to screen large peptide libraries in a cost-effective manner. In this study, we present BrainPepPass, a machine learning (ML) framework that utilizes supervised manifold dimensionality reduction and extreme gradient boosting (XGB) algorithms to predict natural and chemically modified B3PPs. The results indicate that the proposed tool outperforms other classifiers, with average accuracies exceeding 94% and 98% in 10-fold cross-validation and leave-one-out cross-validation (LOOCV), respectively. In addition, accuracy values ranging from 45% to 97.05% were achieved in the independent tests. The BrainPepPass tool is available in a public repository for academic use (https://github.com/ewerton-cristhian/BrainPepPass).

Supervised dimensionality reduction for multi-dimensional classification

Supervised dimensionality reduction for multi-dimensional classification

Geno-Multiomics Strategy Informs Drug Response Profiles in High-Risk Myeloid Neoplasia

Theclonal diversity of myeloid neoplasia (MN) results from the interplay of genomic entities. While diverse molecular lesions suggest distinct leukemogenic mechanisms, various combinations may lead to similar molecular convergence. Such convergence in the context of clinical heterogeneity may be difficult to discern by classic analytic tools leading to selection based on outcome rather than functional overlaps. We leveraged the state-of-the-art machine learning (ML) as a new approach to define likelihood of pharmacologic sensitivity/resistance in high-risk MDS/secondary AML (HR-MDS/sAML). We applied an integrative dimension reduction and subsequent clustering via neural networks generating an Encoder-Decoder architecture. Multiple omics were included as input/output to increase the resolution of drug response characterization. Further, drug response data were used to supervise the embedding. Specifically, we generated a deep feed-forward Encoder with 3 layers to embed distinct data modalities and 2 layers for combined embedding generating 100-d latent representation. Drug response measures (IC 50 values) extracted from The BEAT AML Master Trial 1 was used to supervise the landscape and effectively to generate a non-linear supervised dimension reduction framework. We previously reported the identification of distinct molecular subclasses in MDS. 2 Unsupervised ML via consensus-clustering approach of an autoencoder architecture coupled with GMM resolved clusters according to molecular features, diagnosis, and disease risk. Here, we focus on HR-MDS/sAML (n=1509) clusters. HR-MDS/sAML were grouped in six clusters with mean frequency of 37%. Such clusters, while sharing molecular features were characterized by diverse combinations of genomic features. Therefore, synergistic combination of computational and experimental therapeutics might inform on treatment indications in this group of patients. We noted that treatment response was heterogeneous across the clusters and, in some cases, there was a limited resolution due to molecular heterogeneity. Considering the invariant combinations of dominant and subclonal mutations, we incorporated clonal hierarchy to refine drug response. By adding a loss function for drug-sensitivity mapping, we generated a multi-omic-phenotype map. Similar joint-dimension reduction (jDR) approaches have been successfully utilized previously as well, specifically in the context of multi-omic single-cell sequencing where majority of the tools are either highly regularized/linear or fully unsupervised. 3 We simulated drug response prediction and show two examples of clusters associated with response profiles using two known drugs ( Fig1A, HMA, upper 3D surface plot; low plot 3D surface plot: lenalidomide). We identified clusters of patients likely refractory to HMA and responding to lenalidomide (red circles), clusters of patients predicted to respond to HMA and less likely to lenalidomide (black circles) or poorly to both drugs (yellow dots). Since multiple drugs could be assessed simultaneously, clusters of patients sensitive or resistant to drug combination could be also identified (overlapping hills/valleys). We then applied cluster assignments based on our previous model 2 augmented with VAF and copy number using The Beat AML cohort and characterized drugs with significant difference (ANOVA test). We found significant differences in sensitivity/resistance across the clusters with lower IC 50 values representing high sensitivity scale against a panel of 120 FDA-approved agents ( Fig1B). As an example, we generated a 2D embedding. Individual axes show the multi-modal latent-features associated linearly with drug response. Color-coding shows sensitive/resistant IC 50 profiles for individual samples from The Beat AML cohort. Four drugs with largest F-statistics are depicted. Interestingly, our prediction show feasibility to simultaneously identify clusters of patients characterized by increased resistance for one drug ( e.g., trametinib) and high sensitivity to others ( e.g., rapamycin, dasatinib, doramapimod). In sum, our study suggests that multi-modal genomic configuration increases the power of resolution in HR-MDS/sAML by reducing molecular heterogeneity, which in turn might help resolving therapeutic vulnerabilities.

A neural signature of social support mitigates negative emotion

Social support can mitigate the impact of distressing events. Such stress buffering elicits activity in many brain regions, but it remains unclear (1) whether this activity constitutes a stable brain signature, and (2) whether brain activity can predict buffering across people. Here, we developed a neural signature that predicted social buffering of negative emotion in response to real life stressors. During neuroimaging, participants (n = 95) responded to stressful autobiographical memories either naturally, or by imagining a conversation with a peer. Using supervised dimensionality reduction and machine learning techniques, we identified a spatio-temporal neural signature that distinguished between these two trials. Activation of this signature was associated with less negative affect across trials, and people who most activated the signature reported more supportive social connections and lower loneliness outside the lab. Together, this work provides a behaviorally relevant neurophysiological marker for social support that underlies stress buffering.

Interpretable many-class decoding for MEG

Multivariate pattern analysis (MVPA) of Magnetoencephalography (MEG) and Electroencephalography (EEG) data is a valuable tool for understanding how the brain represents and discriminates between different stimuli. Identifying the spatial and temporal signatures of stimuli is typically a crucial output of these analyses. Such analyses are mainly performed using linear, pairwise, sliding window decoding models. These allow for relative ease of interpretation, e.g. by estimating a time-course of decoding accuracy, but have limited decoding performance. On the other hand, full epoch multiclass decoding models, commonly used for brain–computer interface (BCI) applications, can provide better decoding performance. However interpretation methods for such models have been designed with a low number of classes in mind. In this paper, we propose an approach that combines a multiclass, full epoch decoding model with supervised dimensionality reduction, while still being able to reveal the contributions of spatiotemporal and spectral features using permutation feature importance. Crucially, we introduce a way of doing supervised dimensionality reduction of input features within a neural network optimised for the classification task, improving performance substantially. We demonstrate the approach on 3 different many-class task-MEG datasets using image presentations. Our results demonstrate that this approach consistently achieves higher accuracy than the peak accuracy of a sliding window decoder while estimating the relevant spatiotemporal features in the MEG signal.

DNA Chromatogram Classification Using Entropy-Based Features and Supervised Dimension Reduction Based on Global and Local Pattern Information

Gene sequence classification can be seen as a challenging task due to the nonstationary, noisy and nonlinear characteristics of sequential data. The primary goal of this research is to develop a general solution approach for supervised DNA chromatogram (DNAC) classification in the absence of sufficient training data. Today, deep learning comes to the fore with its achievements, however this requires a lot of training data. Finding enough training data can be exceedingly challenging, particularly in the medical area and for rare disorders. In this paper, a novel supervised DNAC classification method is proposed, which combines three techniques to classify hepatitis virus DNA trace files as HBV and HCV. The features that are capable of reflecting the complex-structured sequential data are extracted based on both embedding and spectral entropies. After the supervised dimension reduction step, not only global behavior of the entropy features but also local behavior of the entropy features is taken into account for classification purpose. A memory-based learning, which cannot lose any information coming from training data as its nature, is being used as a classifier. Experimental results show that the proposed method achieves good results that although 19% training data is used, a performance of 92% is obtained.

Global subclass discriminant analysis

Linear discriminant analysis (LDA) is a powerful supervised dimensionality reduction method for analysing high-dimensional data. However, LDA cannot use locality information in data, which makes LDA degrade dramatically in performance on multimodal data. A number of LDA variants have been proposed to exploit locality information in data, including subclass-based LDAs. We discover a problem with these variants, which is that subclasses are selected on a within-class basis without considering other classes. This causes the loss of important information at class boundaries. In this paper, we present a novel variant of subclass-based LDA, Global Subclass Discriminant Analysis (GSDA). Unlike other subclass-based LDAs, GSDA selects subclasses from global clusters that may cross class boundaries, thus utilising within-class information and between-class information. More specifically, GSDA applies an effective clustering algorithm to the whole data to construct global clusters. It then utilises the local structure refining strategy on these global clusters to construct subclasses. Finally, GSDA learns a representative data subspace by maximising inter-subclass distance and minimising intra-subclass distance simultaneously. GSDA is extensively evaluated on a wide range of public datasets through comparison with the state-of-the-art LDA algorithms. Experimental results demonstrate its superiority in terms of accuracy and run times.

Two phase cooperative learning for supervised dimensionality reduction

The simultaneous minimization of the reconstruction and classification error is a hard non convex problem, especially when a non-linear mapping is utilized. To overcome this obstacle, motivated by the widespread success of Cooperative Neural Networks, an innovative supervised dimensionality reduction framework is proposed, based on a cooperative two phase optimization strategy. Specifically, the proposed framework that requires minimal parameter adjustment consists of an autoencoder for dimensionality reduction and a separator network for separability assessment of the embedding. This scheme results in meaningful and discriminable codes, which are optimized for the classification task and are exploitable by any trainable classifier. The experimental results showed that the proposed methodology achieved competitive results against the state-of-the-art competing methods, while being much more efficient in terms of parameter count. Finally, it was empirically justified that the proposed methodology introduces advanced behavioural explainability, while enabling applicability for image generation tasks.

Supervised Dimensionality Reduction of Proportional Data Using Exponential Family Distributions

Most well-known supervised dimensionality reduction algorithms suffer from the curse of dimensionality while handling high-dimensional sparse data due to ill-conditioned second-order statistics matrices. They also do not deal with multi-modal data properly since they construct neighborhood graphs that do not discriminate between multi-modal classes of data and single-modal ones. In this paper, a novel method that mitigates the above problems is proposed. In this method, assuming the data is from two classes, they are projected into the low-dimensional space in the first step which removes sparsity from the data and reduces the time complexity of any operation drastically afterwards. These projected data are modeled using a mixture of exponential family distributions for each class, allowing the modeling of multi-modal data. A measure for the similarity between the two projected classes is used as an objective function for constructing an optimization problem, which is then solved using a heuristic search algorithm to find the best separating projection. The conducted experiments show that the proposed method outperforms the rest of the compared algorithms and provides a robust effective solution to the problem of dimensionality reduction even in the presence of multi-modal and sparse data.

Fisher's Linear Discriminant Analysis With Space-Folding Operations.

Fisher's linear discriminant analysis (LDA) is an easy-to-use supervised dimensionality reduction method. However, LDA may be ineffective against complicated class distributions. It is well-known that deep feedforward neural networks with rectified linear units as activation functions can map many input neighborhoods to similar outputs by a succession of space-folding operations. This short paper shows that the space-folding operation can reveal to LDA classification information in the subspace where LDA cannot find any. A composition of LDA with the space-folding operation can find classification information more than LDA can do. End-to-end fine-tuning can improve that composition further. Experimental results on artificial and open data sets have shown the feasibility of the proposed approach.

L₂,₁-Norm Regularized Robust and Sparse Linear Discriminant Analysis via an Alternating Direction Method of Multipliers

Recently, an absolute value inequalities discriminant analysis criterion with robustness and sparseness for supervised dimensionality reduction was studied. However, it obtains discriminant directions one by one through greedy search, which makes the sparseness of multiple discriminant directions unexplainable. In addition, it relaxes the original problem into a series of linear programming problems which makes it time consuming. In this paper, we construct a novel linear discriminant analysis with robustness and sparseness jointly through the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> -norm and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2,1</sub> -norm. The proposed approach obtains all the discriminant directions simultaneously, and rather than solving linear programming problems, it is solved by a more effective alternating direction method of multipliers. The effectiveness of the proposed method is supported by preliminary experimental results on two artificial datasets, some benchmark datasests and two face image datasets.

MR elastography identifies regions of extracellular matrix reorganization associated with shorter survival in glioblastoma patients.

Biomechanical tissue properties of glioblastoma tumors are heterogeneous, but the molecular mechanisms involved and the biological implications are poorly understood. Here, we combine magnetic resonance elastography (MRE) measurement of tissue stiffness with RNA sequencing of tissue biopsies to explore the molecular characteristics of the stiffness signal. MRE was performed preoperatively in 13 patients with glioblastoma. Navigated biopsies were harvested during surgery and classified as "stiff" or "soft" according to MRE stiffness measurements (|G*|norm). Twenty-two biopsies from eight patients were analyzed by RNA sequencing. The mean whole-tumor stiffness was lower than normal-appearing white matter. The surgeon's stiffness evaluation did not correlate with the MRE measurements, which suggests that these measures assess different physiological properties. Pathway analysis of the differentially expressed genes between "stiff" and "soft" biopsies showed that genes involved in extracellular matrix reorganization and cellular adhesion were overexpressed in "stiff" biopsies. Supervised dimensionality reduction identified a gene expression signal separating "stiff" and "soft" biopsies. Using the NIH Genomic Data Portal, 265 glioblastoma patients were divided into those with (n = 63) and without (n = 202) this gene expression signal. The median survival time of patients with tumors expressing the gene signal associated with "stiff" biopsies was 100 days shorter than that of patients not expressing it (360 versus 460 days, hazard ratio: 1.45, P < .05). MRE imaging of glioblastoma can provide noninvasive information on intratumoral heterogeneity. Regions of increased stiffness were associated with extracellular matrix reorganization. An expression signal associated with "stiff" biopsies correlated with shorter survival of glioblastoma patients.

Morphological relationships between vertebrate claw unguals and sheaths and the functional morphology of these structures.

The link between claw morphology and function has been historically difficult to quantify, analyze, and interpret. A confounding factor is the ambiguous morphological relationship between the ungual and the sheath and whether one structure or the other is more useful for inferring function from morphology. In this study, the functional morphology of vertebrate claws is analyzed using sheath and ungual measurements taken from modern claw specimens spanning birds and mammals. Claw measurements were chosen for their potential biomechanical significance and a revised, expanded categorization of claw function is used. When corresponding claw measurements from the ungual and sheath are compared independently, some features are highly correlated whereas others are not. A principal component analysis of the claw measurements reveals that some of the morphological disparity is related to functional differences; however, different functional categories are not clearly separated based solely on morphology. A linear discriminant analysis incorporating a supervised dimensionality reduction method (J-function) successfully classifies 94.52% of the claw specimens to their documented functional categories. When the posterior probabilities of each classification are examined, and the next highest probabilities are considered, the analysis can successfully classify 98.63% of the claw specimens. Sheath measurements perform better than ungual measurements but combining measurements from both structures perform better than considering either structure individually. Both structures contribute valuable morphological information when it comes to inferring claw function from morphology.

Interpretable, predictive spatio-temporal models via enhanced pairwise directions estimation

This article concerns predictive modeling for spatio-temporal data as well as model interpretation using data information in space and time. We develop a novel approach based on supervised dimension reduction for such data in order to capture nonlinear mean structures without requiring a prespecified parametric model. In addition to prediction as a common interest, this approach emphasizes the exploration of geometric information from the data. The method of Pairwise Directions Estimation (PDE) is implemented in our approach as a data-driven function searching for spatial patterns and temporal trends. The benefit of using geometric information from the method of PDE is highlighted, which aids effectively in exploring data structures. We further enhance PDE, referring to it as PDE+, by incorporating kriging to estimate the random effects not explained in the mean functions. Our proposal can not only increase prediction accuracy but also improve the interpretation for modeling. Two simulation examples are conducted and comparisons are made with several existing methods. The results demonstrate that the proposed PDE+ method is very useful for exploring and interpreting the patterns and trends for spatio-temporal data. Illustrative applications to two real datasets are also presented.

SLISEMAP: supervised dimensionality reduction through local explanations

Existing methods for explaining black box learning models often focus on building local explanations of the models’ behaviour for particular data items. It is possible to create global explanations for all data items, but these explanations generally have low fidelity for complex black box models. We propose a new supervised manifold visualisation method, slisemap, that simultaneously finds local explanations for all data items and builds a (typically) two-dimensional global visualisation of the black box model such that data items with similar local explanations are projected nearby. We provide a mathematical derivation of our problem and an open source implementation implemented using the GPU-optimised PyTorch library. We compare slisemap to multiple popular dimensionality reduction methods and find that slisemap is able to utilise labelled data to create embeddings with consistent local white box models. We also compare slisemap to other model-agnostic local explanation methods and show that slisemap provides comparable explanations and that the visualisations can give a broader understanding of black box regression and classification models.