Dimensionality Reduction Framework Research Articles

An encoding generative modeling approach to dimension reduction and covariate adjustment in causal inference with observational studies

In this article, we develop CausalEGM, a deep learning framework for nonlinear dimension reduction and generative modeling of the dependency among covariate features affecting treatment and response. CausalEGM can be used for estimating causal effects in both binary and continuous treatment settings. By learning a bidirectional transformation between the high-dimensional covariate space and a low-dimensional latent space and then modeling the dependencies of different subsets of the latent variables on the treatment and response, CausalEGM can extract the latent covariate features that affect both treatment and response. By conditioning on these features, one can mitigate the confounding effect of the high dimensional covariate on the estimation of the causal relation between treatment and response. In a series of experiments, the proposed method is shown to achieve superior performance over existing methods in both binary and continuous treatment settings. The improvement is substantial when the sample size is large and the covariate is of high dimension. Finally, we established excess risk bounds and consistency results for our method, and discuss how our approach is related to and improves upon other dimension reduction approaches in causal inference.

Regular variation in Hilbert spaces and principal component analysis for functional extremes

Motivated by the increasing availability of data of functional nature, we develop a general probabilistic and statistical framework for extremes of regularly varying random elements X in L2[0,1]. We place ourselves in a Peaks-Over-Threshold framework where a functional extreme is defined as an observation X whose L2-norm ‖X‖ is comparatively large. Our goal is to propose a dimension reduction framework resulting into finite dimensional projections for such extreme observations. Our contribution is double. First, we investigate the notion of Regular Variation for random quantities valued in a general separable Hilbert space, for which we propose a novel concrete characterization involving solely stochastic convergence of real-valued random variables. Second, we propose a notion of functional Principal Component Analysis (PCA) accounting for the principal ‘directions’ of functional extremes. We investigate the statistical properties of the empirical covariance operator of the angular component of extreme functions, by upper-bounding the Hilbert–Schmidt norm of the estimation error for finite sample sizes. Numerical experiments with simulated and real data illustrate this work.

Model-Free Statistical Inference on High-Dimensional Data

This article aims to develop an effective model-free inference procedure for high-dimensional data. We first reformulate the hypothesis testing problem via sufficient dimension reduction framework. With the aid of new reformulation, we propose a new test statistic and show that its asymptotic distribution is χ 2 distribution whose degree of freedom does not depend on the unknown population distribution. We further conduct power analysis under local alternative hypotheses. In addition, we study how to control the false discovery rate of the proposed χ 2 tests, which are correlated, to identify important predictors under a model-free framework. To this end, we propose a multiple testing procedure and establish its theoretical guarantees. Monte Carlo simulation studies are conducted to assess the performance of the proposed tests and an empirical analysis of a real-world dataset is used to illustrate the proposed methodology. Supplementary materials for this article are available online including a standardized description of the materials available for reproducing the work.

Wasserstein Dictionaries of Persistence Diagrams.

This article presents a computational framework for the concise encoding of an ensemble of persistence diagrams, in the form of weighted Wasserstein barycenters Turner et al. (2014), Vidal et al. (2020) of a dictionary of atom diagrams. We introduce a multi-scale gradient descent approach for the efficient resolution of the corresponding minimization problem, which interleaves the optimization of the barycenter weights with the optimization of the atom diagrams. Our approach leverages the analytic expressions for the gradient of both sub-problems to ensure fast iterations and it additionally exploits shared-memory parallelism. Extensive experiments on public ensembles demonstrate the efficiency of our approach, with Wasserstein dictionary computations in the orders of minutes for the largest examples. We show the utility of our contributions in two applications. First, we apply Wassserstein dictionaries to data reduction and reliably compress persistence diagrams by concisely representing them with their weights in the dictionary. Second, we present a dimensionality reduction framework based on a Wasserstein dictionary defined with a small number of atoms (typically three) and encode the dictionary as a low dimensional simplex embedded in a visual space (typically in 2D). In both applications, quantitative experiments assess the relevance of our framework. Finally, we provide a C++ implementation that can be used to reproduce our results.

Research on Stock Price Prediction Model Based on Weighted Sufficient Dimension Reduction and Bagging Framework

Research on Stock Price Prediction Model Based on Weighted Sufficient Dimension Reduction and Bagging Framework

Sliced Inverse Regression in Metric Spaces

In this article, we propose a general nonlinear sufficient dimension reduction (SDR) framework when both the predictor and response lie in some general metric spaces. We construct reproducing kernel Hilbert spaces whose kernels are fully determined by the distance functions of the metric spaces, then leverage the inherent structures of these spaces to define a nonlinear SDR framework. We adapt the classical sliced inverse regression of \citet{Li:1991} within this framework for the metric space data. We build the estimator based on the corresponding linear operators, and show it recovers the regression information unbiasedly. We derive the estimator at both the operator level and under a coordinate system, and also establish its convergence rate. We illustrate the proposed method with both synthetic and real datasets exhibiting non-Euclidean geometry.

ARKA: a framework of dimensionality reduction for machine-learning classification modeling, risk assessment, and data gap-filling of sparse environmental toxicity data.

Due to the lack of experimental toxicity data for environmental chemicals, there arises a need to fill data gaps by in silico approaches. One of the most commonly used in silico approaches for toxicity assessment of small datasets is the Quantitative Structure-Activity Relationship (QSAR), which generates predictive models for the efficient prediction of query compounds. However, the reliability of the predictions from QSARs derived from small datasets is often questionable from a statistical point of view. This is due to the presence of a larger number of descriptors as compared to the number of training compounds, which reduces the degree of freedom of the developed model. To reduce the overall prediction error for a particular QSAR model, we have proposed here the computation of the novel Arithmetic Residuals in K-groups Analysis (ARKA) descriptors. We have reduced the number of modeling descriptors in a supervised manner by partitioning them into K classes (K = 2 here) depending on the higher mean normalized values of the descriptors to a particular response class, thus preventing the loss of chemical information. A scatter plot of the data points using the values of two ARKA descriptors (ARKA_2 vs. ARKA_1) can potentially identify activity cliffs, less confident data points, and less modelable data points. We have used here five representative environmentally relevant endpoints (skin sensitization, earthworm toxicity, milk/plasma partitioning, algal toxicity, and rodent carcinogenicity of hazardous chemicals) with graded responses to which the ARKA framework was applied for classification modeling. On comparing the performance of the models generated using conventional QSAR descriptors and the ARKA descriptors, the prediction quality of the models derived from ARKA descriptors was found, based on multiple graded-data validation metrics-derived decision criteria, much better than the models derived from QSAR descriptors signifying the potential of ARKA descriptors in ecotoxicological classification modeling of small data sets. Additionally, this holds true for the Read-Across approach as well, since the Read-Across predictions using ARKA descriptors supersede the predictions generated from QSAR descriptors. For the ease of users, a Java-based expert system has been developed that computes the ARKA descriptors from the input of QSAR descriptors.

Geno-Multiomics Strategy Informs Drug Response Profiles in High-Risk Myeloid Neoplasia

Theclonal diversity of myeloid neoplasia (MN) results from the interplay of genomic entities. While diverse molecular lesions suggest distinct leukemogenic mechanisms, various combinations may lead to similar molecular convergence. Such convergence in the context of clinical heterogeneity may be difficult to discern by classic analytic tools leading to selection based on outcome rather than functional overlaps. We leveraged the state-of-the-art machine learning (ML) as a new approach to define likelihood of pharmacologic sensitivity/resistance in high-risk MDS/secondary AML (HR-MDS/sAML). We applied an integrative dimension reduction and subsequent clustering via neural networks generating an Encoder-Decoder architecture. Multiple omics were included as input/output to increase the resolution of drug response characterization. Further, drug response data were used to supervise the embedding. Specifically, we generated a deep feed-forward Encoder with 3 layers to embed distinct data modalities and 2 layers for combined embedding generating 100-d latent representation. Drug response measures (IC 50 values) extracted from The BEAT AML Master Trial 1 was used to supervise the landscape and effectively to generate a non-linear supervised dimension reduction framework. We previously reported the identification of distinct molecular subclasses in MDS. 2 Unsupervised ML via consensus-clustering approach of an autoencoder architecture coupled with GMM resolved clusters according to molecular features, diagnosis, and disease risk. Here, we focus on HR-MDS/sAML (n=1509) clusters. HR-MDS/sAML were grouped in six clusters with mean frequency of 37%. Such clusters, while sharing molecular features were characterized by diverse combinations of genomic features. Therefore, synergistic combination of computational and experimental therapeutics might inform on treatment indications in this group of patients. We noted that treatment response was heterogeneous across the clusters and, in some cases, there was a limited resolution due to molecular heterogeneity. Considering the invariant combinations of dominant and subclonal mutations, we incorporated clonal hierarchy to refine drug response. By adding a loss function for drug-sensitivity mapping, we generated a multi-omic-phenotype map. Similar joint-dimension reduction (jDR) approaches have been successfully utilized previously as well, specifically in the context of multi-omic single-cell sequencing where majority of the tools are either highly regularized/linear or fully unsupervised. 3 We simulated drug response prediction and show two examples of clusters associated with response profiles using two known drugs ( Fig1A, HMA, upper 3D surface plot; low plot 3D surface plot: lenalidomide). We identified clusters of patients likely refractory to HMA and responding to lenalidomide (red circles), clusters of patients predicted to respond to HMA and less likely to lenalidomide (black circles) or poorly to both drugs (yellow dots). Since multiple drugs could be assessed simultaneously, clusters of patients sensitive or resistant to drug combination could be also identified (overlapping hills/valleys). We then applied cluster assignments based on our previous model 2 augmented with VAF and copy number using The Beat AML cohort and characterized drugs with significant difference (ANOVA test). We found significant differences in sensitivity/resistance across the clusters with lower IC 50 values representing high sensitivity scale against a panel of 120 FDA-approved agents ( Fig1B). As an example, we generated a 2D embedding. Individual axes show the multi-modal latent-features associated linearly with drug response. Color-coding shows sensitive/resistant IC 50 profiles for individual samples from The Beat AML cohort. Four drugs with largest F-statistics are depicted. Interestingly, our prediction show feasibility to simultaneously identify clusters of patients characterized by increased resistance for one drug ( e.g., trametinib) and high sensitivity to others ( e.g., rapamycin, dasatinib, doramapimod). In sum, our study suggests that multi-modal genomic configuration increases the power of resolution in HR-MDS/sAML by reducing molecular heterogeneity, which in turn might help resolving therapeutic vulnerabilities.

Dimensionality reduction in stochastic complex dynamical networks

Complex systems often exhibit diverse dynamical behaviors in high-dimensional spaces that depend on various factors. Dimensionality reduction is a powerful tool for analyzing and understanding complex systems, aiming to find a low-dimensional representation of the complex system that preserves its essential features and reveals its underlying mechanisms and long-term dynamics. However, most existing methods for dimensionality reduction are limited to deterministic systems and cannot account for the stochastic effects that are ubiquitous in real-world complex networks. Here we develop a general analytical framework for dimensionality reduction of stochastic complex dynamical networks that can capture the essential features and long-term dynamics of the original system in a low-dimensional effective equation. The effective equation is a function of a set of effective parameters that are associated with specific system states and determine the network's dynamical behavior. We show that the standard deviation of the effective equation can be used to analyze the dynamic behavior and possible convergence of the stochastic complex dynamical network. Our framework can be applied to various types of stochastic complex dynamical networks and can reveal the underlying mechanisms and emergent phenomena of these systems.

Two phase cooperative learning for supervised dimensionality reduction

The simultaneous minimization of the reconstruction and classification error is a hard non convex problem, especially when a non-linear mapping is utilized. To overcome this obstacle, motivated by the widespread success of Cooperative Neural Networks, an innovative supervised dimensionality reduction framework is proposed, based on a cooperative two phase optimization strategy. Specifically, the proposed framework that requires minimal parameter adjustment consists of an autoencoder for dimensionality reduction and a separator network for separability assessment of the embedding. This scheme results in meaningful and discriminable codes, which are optimized for the classification task and are exploitable by any trainable classifier. The experimental results showed that the proposed methodology achieved competitive results against the state-of-the-art competing methods, while being much more efficient in terms of parameter count. Finally, it was empirically justified that the proposed methodology introduces advanced behavioural explainability, while enabling applicability for image generation tasks.

Identification of transcriptional programs using dense vector representations defined by mutual information with GeneVector

Deciphering individual cell phenotypes from cell-specific transcriptional processes requires high dimensional single cell RNA sequencing. However, current dimensionality reduction methods aggregate sparse gene information across cells, without directly measuring the relationships that exist between genes. By performing dimensionality reduction with respect to gene co-expression, low-dimensional features can model these gene-specific relationships and leverage shared signal to overcome sparsity. We describe GeneVector, a scalable framework for dimensionality reduction implemented as a vector space model using mutual information between gene expression. Unlike other methods, including principal component analysis and variational autoencoders, GeneVector uses latent space arithmetic in a lower dimensional gene embedding to identify transcriptional programs and classify cell types. In this work, we show in four single cell RNA-seq datasets that GeneVector was able to capture phenotype-specific pathways, perform batch effect correction, interactively annotate cell types, and identify pathway variation with treatment over time.

Discovering epistasis interactions in Alzheimer’s disease using integrated framework of ensemble learning and multifactor dimensionality reduction (MDR)

Alzheimer's disease (AD) is a complex disorder with strong genetic factors. The proposed framework is applied to Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset. We present a novel framework integrating ensemble learning and MDR constructive induction algorithm to discover epistasis interactions associated with AD in a computationally efficient method. Discovering epistasis interactions is a big challenge and significantly impacts personalized medicine (PM). The applied ensemble learning algorithms are random forests (RF) with Gini index and permutation importance, Extreme Gradient Boosting (XGBoost), and classification and regression trees (CART). The classification accuracy of 5-way models varied between (0.8674–0.8758), whereas the accuracy of 2-way, 3-way, and 4-way models varied between (0.6515–0.6649), (0.7071–0.7170), and (0.7811–0.7878) respectively. The promising results of this proposed framework show high-ranked risk genes and up to 5-way epistasis models that contribute to the disease risk efficiently and at higher accuracy.

Semi-supervised t-SNE with multi-scale neighborhood preservation

Unsupervised dimensionality reduction (DR) aims to preserve input data structure in a low-dimensional (LD) space based on neighborhood information. In contrast, supervised DR intends to improve the learning performance, i.e., classification and regression, in an LD representation. Unfortunately, obtaining the complete label outputs of a data set for real-world applications is hard. Here, we introduce a novel DR framework coupling both available class labels and input feature similarities to extend the well-known t-distributed Stochastic Neighbor Embedding (SNE) for semi-supervised scenarios. Our proposal, termed Semi-Supervised t-SNE (SS.t-SNE), properly fixes the widths of Gaussian neighborhoods to reveal the salient local and global data structures in an LD space. Indeed, our approach is presented as a generalization of unsupervised and supervised versions of t-SNE. SS.t-SNE outperforms other semi-supervised DR methods in data visualization and classification tasks in LD embeddings.

Flexible Adaptive Graph Embedding for Semi-supervised Dimension Reduction

Graph-based semi-supervised dimension reduction can use the inherent graph structure of samples to propagate label information, and has become a hot research field in machine learning. However, most current methods have strict linear constraints and cannot handle data sampled from nonlinear manifolds; and rely on predefined graphs, which cannot capture the local structure information of data and cannot handle complex non-Gaussian data. To address these issues, this paper proposes a new locality-preserved flexible dimension reduction framework, called Semi-supervised Flexible Adaptive Graph Embedding (SFAG), which learns the embedding space that can preserve the local neighborhood structure by constructing a k1-nearest neighbor graph over labeled samples. Then, another k2-nearest neighbor graph is constructed on all samples to adaptively construct the optimal graph, clustering labeled and unlabeled embedding sample points with neighborhood relations into the same sub-manifold sharing the same label information. Last but not least, the hard linear projection constraint is relaxed by adding residual terms to obtain not only the nonlinear embedding of the training samples, but also the linear projection matrix applied directly to the out-of-sample. In addition, two different semi-supervised dimension reduction methods for adaptive construction of optimal graphs are proposed based on the SFAG framework. Several evaluation experiments validate the effectiveness of our method in exploring manifold structures and classification tasks.

ParaDime: A Framework for Parametric Dimensionality Reduction.

ParaDime is a framework for parametric dimensionality reduction (DR). In parametric DR, neural networks are trained to embed high-dimensional data items in a low-dimensional space while minimizing an objective function. ParaDime builds on the idea that the objective functions of several modern DR techniques result from transformed inter-item relationships. It provides a common interface for specifying these relations and transformations and for defining how they are used within the losses that govern the training process. Through this interface, ParaDime unifies parametric versions of DR techniques such as metric MDS, t-SNE, and UMAP. It allows users to fully customize all aspects of the DR process. We show how this ease of customization makes ParaDime suitable for experimenting with interesting techniques such as hybrid classification/embedding models and supervised DR. This way, ParaDime opens up new possibilities for visualizing high-dimensional data.

Factor modeling of multivariate time series: A frequency components approach

A frequency domain factor model method for multivariate stationary time series is proposed. The dimension reduction framework aims to find a lower-dimensional multivariate stationary factor series. Frequency components of the observed series are assumed to be linearly generated by the corresponding frequency components of a latent factor series using frequency-specific factor loadings matrices. These loadings matrices are then estimated using an eigendecomposition of symmetric non-negative definite matrices involving the real and imaginary parts of the spectral matrix. The factor dimension is estimated using nonparametric bootstrap tests. Consistency results concerning the estimation of eigenvalues, eigenvectors, factor loadings matrices and the factor dimension are provided. The numerical performance of the proposed method is illustrated through simulation examples. An application to modeling resting-state fMRI time series from autism individuals is demonstrated where a frequency-specific factor analysis helps understand functional connectivity.

Deep learning assisted surrogate modeling of large-scale power grids

The model order reduction (MOR) enterprise has shown unprecedented applications in the area of power systems by allowing tractable and realistic simulations of complex power grids. Nearly all model reduction techniques rely on constructing a linear reduced trial subspace followed by a Galerkin projection to restrict the evolution of the latent dynamics onto the reduced subspace. However, power system models often exhibit a slow decay of singular values and, as such, are often approximated poorly due to the truncation of higher-order modes. Furthermore, employing a linear projection for such models places a fundamental limit on the accuracy of the reduced order models (ROMs). This paper uses techniques from machine learning to develop a dimensionality reduction framework for large-scale power grid models to overcome these limitations effectively. In particular, we use autoencoder (AE) network to learn low-dimensional nonlinear trial manifold to avoid the limitation of a linear trial subspace. This is followed by the use of long short-time memory (LSTM) networks to obtain the evolution of the reduced dynamics in a non-intrusive manner. The resulting framework yields ROMs that are compact and are several orders of magnitude more accurate than the linear projection-based methods. We demonstrate the proposed technique on the standard IEEE 118-Bus system, which has 54 generators, and the European high-voltage system with 260 generators.

A fast dimension reduction framework for large-scale topology optimization of grid-layout offshore wind farm collector systems

The construction of large-scale offshore wind farms faces the difficulty of solving the optimization problem of collector systems. Due to its NP-hard feature, this problem cannot be solved straightforwardly. This paper proposes an efficient and precise solution framework suitable for large-scale collector system optimization and raises the main influencing factors of time complexity. First, according to a heuristic idea, an arc selection algorithm is established to tighten the feasible region. Second, an optimization framework based on the greedy algorithm and mixed integer quadratic programming (MIQP) is established with excellent efficiency and accuracy. The overall problem is decomposed into the master problem of topology optimization and the sub-problem of cable selection. A bi-level model that adopts both greedy algorithm and global optimization is proposed, iterating through the master problem and the sub-problem by transferring parameters and updating constraints. Third, a large number of examples are analyzed, and the advantages of the proposed algorithms and model are comprehensively verified. In the end, time complexity analysis is carried out, and the empirical formula of solution time is obtained by fitting. Based on the proposed method, large-scale problems can be solved much faster than traditional methods and reach a balance of both speed and quality.

Poisson PCA for matrix count data

We develop a dimension reduction framework for data consisting of matrices of counts. Our model is based on the assumption of existence of a small amount of independent normal latent variables that drive the dependency structure of the observed data, and can be seen as the exact discrete analogue of a contaminated low-rank matrix normal model. We derive estimators for the model parameters and establish their limiting normality. An extension of a recent proposal from the literature is used to estimate the latent dimension of the model. The method is shown to outperform both its vectorization-based competitors and matrix methods assuming the continuity of the data distribution in analysing simulated data and real world abundance data.

A Dynamical System-Based Framework for Dimension Reduction

A Dynamical System-Based Framework for Dimension Reduction