Dimensionality Reduction Framework Research Articles

Study on runoff forecasting and error correction driven by atmosphere–ocean-land dataset

Accurate runoff forecasting results can not only provide an important basis for flood control scheduling, but also provide scientific support for water resources optimization, which promotes the maximization of the overall benefits of the basin. To further explore the inherent mechanisms of the atmosphere–ocean-land factors driving runoff changes, this study proposes the factors dimension reduction and interpretation framework based on Pearson, eXtreme Gradient Boosting and SHapley Additive exPlanations (P-XGBoost-SHAP). Base on this, the Gaussian Process Regression (GPR), Long Short-Term Memory neural network (LSTM) and Support Vector Machine (SVM) models are used to construct the atmospheric-ocean-land data-driven runoff prediction model. Meanwhile, for the runoff prediction residuals, this paper proposes an error multi-step correction framework based on ensemble empirical mode decomposition and autoregressive model (EEMD-AR). The case study of Lianghekou hydrological station shows that the factor dimension reduction and interpretation framework can greatly reduce the input dimension of the model, and explain the factors globally and locally by using SHAP Value. Compared with the traditional Random Forest (RF) dimension reduction method, it shows higher prediction accuracy. The Nash-Sutcliffe efficiency coefficient (NSE) can be increased to about 0.93, which is 4.91 % and 1.97 % higher than the series–parallel coupling (AR-Parallel) and empirical mode decomposition-autoregressive (EMD-AR) correction methods, respectively. The accuracy of the runoff forecasting prediction is improved while reducing the input dimension of the model.

Stealthy data integrity attack identification in smart grid networks utilizing deep denoising autoencoder

Smart grids arose as the largest cyber-physical systems with the integration of sophisticated control, computing, and state-of-the-art communications. Like all cyber-physical systems, the smart grids are vulnerable to malicious cyber assaults due to their enormous dependency on communication networks. Various machine learning-based schemes are being investigated in the industry and academia to develop robust defense mechanisms to counter cyber assaults. However, the curse of high dimensionality, which increases with the escalating evolution of an electric power system, infringes upon the efficiency of machine learning models employed to detect such assaults. To this end, this paper proposes a deep denoising autoencoder (DAE)-based framework for dimensionality reduction that learns salient feature representation for high-dimensional, multi-variant smart grid measurement data collected from the smart grids. The latent space apprehended by DAE is then fed to binary support vector machine (SVM) to determine the assaulted data. Various standard IEEE test cases are employed in simulations. The results show that the proposed scheme learns more robust features that reveal the nonlinear properties exhibited in the smart grid measurements, further leading to improved detection accuracy of the classifier as compared to existing approaches.

SMALE: Hyperspectral Image Classification via Superpixels and Manifold Learning

As an extremely efficient preprocessing tool, superpixels have become more and more popular in various computer vision tasks. Nevertheless, there are still several drawbacks in the application of hyperspectral image (HSl) processing. Firstly, it is difficult to directly apply superpixels because of the high dimension of HSl information. Secondly, existing superpixel algorithms cannot accurately classify the HSl objects due to multi-scale feature categorization. For the processing of high-dimensional problems, we use the principle of PCA to extract three principal components from numerous bands to form three-channel images. In this paper, a novel superpixel algorithm called Seed Extend by Entropy Density (SEED) is proposed to alleviate the seed point redundancy caused by the diversified content of HSl. It also focuses on breaking the dilemma of manually setting the number of superpixels to overcome the difficulty of classification imprecision caused by multi-scale targets. Next, a space–spectrum constraint model, termed Hyperspectral Image Classification via superpixels and manifold learning (SMALE), is designed, which integrates the proposed SEED to generate a dimensionality reduction framework. By making full use of spatial context information in the process of unsupervised dimension reduction, it could effectively improve the performance of HSl classification. Experimental results show that the proposed SEED could effectively promote the classification accuracy of HSI. Meanwhile, the integrated SMALE model outperforms existing algorithms on public datasets in terms of several quantitative metrics.

A dimensionality reduction assisted evolutionary algorithm for high-dimensional expensive multi/many-objective optimization

Surrogate-assisted multi/many-objective evolutionary algorithms (SA-MOEAs) have shown significant progress in tackling expensive optimization problems. However, existing research primarily focuses on low-dimensional optimization problems. The main reason lies in the fact that some surrogate techniques used in SA-MOEAs, such as the Kriging model, are not applicable for exploring high-dimensional decision space. This paper introduces a surrogate-assisted multi-objective evolutionary algorithm with dimensionality reduction to address high-dimensional expensive optimization problems. The proposed algorithm includes two key insights. Firstly, we propose a dimensionality reduction framework containing three different feature extraction algorithms and a feature drift strategy to map the high-dimensional decision space into a low-dimensional decision space; this strategy helps to improve the robustness of surrogates. Secondly, we propose a sub-region search strategy to define a series of promising sub-regions in the high-dimensional decision space; this strategy helps to improve the exploration ability of the proposed SA-MOEA. Experimental results demonstrate the effectiveness of our proposed algorithm in comparison to several state-of-the-art algorithms.

Sensitivity analysis and robust optimization to high-dimensional uncertainties of compressors with active subspace method

The performance of compressors often deviates significantly from the design value due to the substantial geometric deviations arising from manufacturing errors and in-service degradation. Robust design optimization (RDO) plays a crucial role in ensuring the high performance and reliability of compressors. The study proposes an active subspace-based dimensionality-reduction RDO method for high-dimensional uncertainties of compressors. By employing the active subspace (AS) method, a dimensionality-reduction artificial neural network (ANN) is raised to solve the "dimension disaster" problem for high-dimensional uncertainty quantification (UQ) analysis. Additionally, to decrease the computation cost of the original AS method, a data-driven method is utilized to calculate the gradient of the quantity of interest (QoI) within the AS dimensionality-reduction framework. Furthermore, Shapley method is applied to study the sensitivity of geometric variables and their variation direction on the mean and variance of the performance, and the robustness improvement method by only modifying key geometric variables based on Shapley results is discussed. The RDO of high dimensional geometrical uncertainties of a transonic compressor stage is studied by using this method. The results demonstrate a reduction from the original 24-dimensional uncertainties to a 4-dimensional space, achieved through this method. Consequently, the required sample size is reduced by 75 % while maintaining nearly unchanged model accuracy. Furthermore, the findings reveal that after RDO, the mean efficiency increases by 1.04 %, with a 42 % reduction in variance. Compared to RDO, which requires simultaneous optimization of all 24 geometric variables, the robustness improvement based on Shapley only necessitates modifying 4 key geometric variables, resulting in a substantial increase in both the mean and variance of efficiency.

A novel robust adaptive subspace learning framework for dimensionality reduction

A novel robust adaptive subspace learning framework for dimensionality reduction

An encoding generative modeling approach to dimension reduction and covariate adjustment in causal inference with observational studies

In this article, we develop CausalEGM, a deep learning framework for nonlinear dimension reduction and generative modeling of the dependency among covariate features affecting treatment and response. CausalEGM can be used for estimating causal effects in both binary and continuous treatment settings. By learning a bidirectional transformation between the high-dimensional covariate space and a low-dimensional latent space and then modeling the dependencies of different subsets of the latent variables on the treatment and response, CausalEGM can extract the latent covariate features that affect both treatment and response. By conditioning on these features, one can mitigate the confounding effect of the high dimensional covariate on the estimation of the causal relation between treatment and response. In a series of experiments, the proposed method is shown to achieve superior performance over existing methods in both binary and continuous treatment settings. The improvement is substantial when the sample size is large and the covariate is of high dimension. Finally, we established excess risk bounds and consistency results for our method, and discuss how our approach is related to and improves upon other dimension reduction approaches in causal inference.

Regular variation in Hilbert spaces and principal component analysis for functional extremes

Motivated by the increasing availability of data of functional nature, we develop a general probabilistic and statistical framework for extremes of regularly varying random elements X in L2[0,1]. We place ourselves in a Peaks-Over-Threshold framework where a functional extreme is defined as an observation X whose L2-norm ‖X‖ is comparatively large. Our goal is to propose a dimension reduction framework resulting into finite dimensional projections for such extreme observations. Our contribution is double. First, we investigate the notion of Regular Variation for random quantities valued in a general separable Hilbert space, for which we propose a novel concrete characterization involving solely stochastic convergence of real-valued random variables. Second, we propose a notion of functional Principal Component Analysis (PCA) accounting for the principal ‘directions’ of functional extremes. We investigate the statistical properties of the empirical covariance operator of the angular component of extreme functions, by upper-bounding the Hilbert–Schmidt norm of the estimation error for finite sample sizes. Numerical experiments with simulated and real data illustrate this work.

Model-Free Statistical Inference on High-Dimensional Data

This article aims to develop an effective model-free inference procedure for high-dimensional data. We first reformulate the hypothesis testing problem via sufficient dimension reduction framework. With the aid of new reformulation, we propose a new test statistic and show that its asymptotic distribution is χ 2 distribution whose degree of freedom does not depend on the unknown population distribution. We further conduct power analysis under local alternative hypotheses. In addition, we study how to control the false discovery rate of the proposed χ 2 tests, which are correlated, to identify important predictors under a model-free framework. To this end, we propose a multiple testing procedure and establish its theoretical guarantees. Monte Carlo simulation studies are conducted to assess the performance of the proposed tests and an empirical analysis of a real-world dataset is used to illustrate the proposed methodology. Supplementary materials for this article are available online including a standardized description of the materials available for reproducing the work.

Wasserstein Dictionaries of Persistence Diagrams.

This article presents a computational framework for the concise encoding of an ensemble of persistence diagrams, in the form of weighted Wasserstein barycenters Turner et al. (2014), Vidal et al. (2020) of a dictionary of atom diagrams. We introduce a multi-scale gradient descent approach for the efficient resolution of the corresponding minimization problem, which interleaves the optimization of the barycenter weights with the optimization of the atom diagrams. Our approach leverages the analytic expressions for the gradient of both sub-problems to ensure fast iterations and it additionally exploits shared-memory parallelism. Extensive experiments on public ensembles demonstrate the efficiency of our approach, with Wasserstein dictionary computations in the orders of minutes for the largest examples. We show the utility of our contributions in two applications. First, we apply Wassserstein dictionaries to data reduction and reliably compress persistence diagrams by concisely representing them with their weights in the dictionary. Second, we present a dimensionality reduction framework based on a Wasserstein dictionary defined with a small number of atoms (typically three) and encode the dictionary as a low dimensional simplex embedded in a visual space (typically in 2D). In both applications, quantitative experiments assess the relevance of our framework. Finally, we provide a C++ implementation that can be used to reproduce our results.

Research on Stock Price Prediction Model Based on Weighted Sufficient Dimension Reduction and Bagging Framework

Research on Stock Price Prediction Model Based on Weighted Sufficient Dimension Reduction and Bagging Framework

Sliced Inverse Regression in Metric Spaces

In this article, we propose a general nonlinear sufficient dimension reduction (SDR) framework when both the predictor and response lie in some general metric spaces. We construct reproducing kernel Hilbert spaces whose kernels are fully determined by the distance functions of the metric spaces, then leverage the inherent structures of these spaces to define a nonlinear SDR framework. We adapt the classical sliced inverse regression of \citet{Li:1991} within this framework for the metric space data. We build the estimator based on the corresponding linear operators, and show it recovers the regression information unbiasedly. We derive the estimator at both the operator level and under a coordinate system, and also establish its convergence rate. We illustrate the proposed method with both synthetic and real datasets exhibiting non-Euclidean geometry.

ARKA: a framework of dimensionality reduction for machine-learning classification modeling, risk assessment, and data gap-filling of sparse environmental toxicity data.

Due to the lack of experimental toxicity data for environmental chemicals, there arises a need to fill data gaps by in silico approaches. One of the most commonly used in silico approaches for toxicity assessment of small datasets is the Quantitative Structure-Activity Relationship (QSAR), which generates predictive models for the efficient prediction of query compounds. However, the reliability of the predictions from QSARs derived from small datasets is often questionable from a statistical point of view. This is due to the presence of a larger number of descriptors as compared to the number of training compounds, which reduces the degree of freedom of the developed model. To reduce the overall prediction error for a particular QSAR model, we have proposed here the computation of the novel Arithmetic Residuals in K-groups Analysis (ARKA) descriptors. We have reduced the number of modeling descriptors in a supervised manner by partitioning them into K classes (K = 2 here) depending on the higher mean normalized values of the descriptors to a particular response class, thus preventing the loss of chemical information. A scatter plot of the data points using the values of two ARKA descriptors (ARKA_2 vs. ARKA_1) can potentially identify activity cliffs, less confident data points, and less modelable data points. We have used here five representative environmentally relevant endpoints (skin sensitization, earthworm toxicity, milk/plasma partitioning, algal toxicity, and rodent carcinogenicity of hazardous chemicals) with graded responses to which the ARKA framework was applied for classification modeling. On comparing the performance of the models generated using conventional QSAR descriptors and the ARKA descriptors, the prediction quality of the models derived from ARKA descriptors was found, based on multiple graded-data validation metrics-derived decision criteria, much better than the models derived from QSAR descriptors signifying the potential of ARKA descriptors in ecotoxicological classification modeling of small data sets. Additionally, this holds true for the Read-Across approach as well, since the Read-Across predictions using ARKA descriptors supersede the predictions generated from QSAR descriptors. For the ease of users, a Java-based expert system has been developed that computes the ARKA descriptors from the input of QSAR descriptors.

Geno-Multiomics Strategy Informs Drug Response Profiles in High-Risk Myeloid Neoplasia

Theclonal diversity of myeloid neoplasia (MN) results from the interplay of genomic entities. While diverse molecular lesions suggest distinct leukemogenic mechanisms, various combinations may lead to similar molecular convergence. Such convergence in the context of clinical heterogeneity may be difficult to discern by classic analytic tools leading to selection based on outcome rather than functional overlaps. We leveraged the state-of-the-art machine learning (ML) as a new approach to define likelihood of pharmacologic sensitivity/resistance in high-risk MDS/secondary AML (HR-MDS/sAML). We applied an integrative dimension reduction and subsequent clustering via neural networks generating an Encoder-Decoder architecture. Multiple omics were included as input/output to increase the resolution of drug response characterization. Further, drug response data were used to supervise the embedding. Specifically, we generated a deep feed-forward Encoder with 3 layers to embed distinct data modalities and 2 layers for combined embedding generating 100-d latent representation. Drug response measures (IC 50 values) extracted from The BEAT AML Master Trial 1 was used to supervise the landscape and effectively to generate a non-linear supervised dimension reduction framework. We previously reported the identification of distinct molecular subclasses in MDS. 2 Unsupervised ML via consensus-clustering approach of an autoencoder architecture coupled with GMM resolved clusters according to molecular features, diagnosis, and disease risk. Here, we focus on HR-MDS/sAML (n=1509) clusters. HR-MDS/sAML were grouped in six clusters with mean frequency of 37%. Such clusters, while sharing molecular features were characterized by diverse combinations of genomic features. Therefore, synergistic combination of computational and experimental therapeutics might inform on treatment indications in this group of patients. We noted that treatment response was heterogeneous across the clusters and, in some cases, there was a limited resolution due to molecular heterogeneity. Considering the invariant combinations of dominant and subclonal mutations, we incorporated clonal hierarchy to refine drug response. By adding a loss function for drug-sensitivity mapping, we generated a multi-omic-phenotype map. Similar joint-dimension reduction (jDR) approaches have been successfully utilized previously as well, specifically in the context of multi-omic single-cell sequencing where majority of the tools are either highly regularized/linear or fully unsupervised. 3 We simulated drug response prediction and show two examples of clusters associated with response profiles using two known drugs ( Fig1A, HMA, upper 3D surface plot; low plot 3D surface plot: lenalidomide). We identified clusters of patients likely refractory to HMA and responding to lenalidomide (red circles), clusters of patients predicted to respond to HMA and less likely to lenalidomide (black circles) or poorly to both drugs (yellow dots). Since multiple drugs could be assessed simultaneously, clusters of patients sensitive or resistant to drug combination could be also identified (overlapping hills/valleys). We then applied cluster assignments based on our previous model 2 augmented with VAF and copy number using The Beat AML cohort and characterized drugs with significant difference (ANOVA test). We found significant differences in sensitivity/resistance across the clusters with lower IC 50 values representing high sensitivity scale against a panel of 120 FDA-approved agents ( Fig1B). As an example, we generated a 2D embedding. Individual axes show the multi-modal latent-features associated linearly with drug response. Color-coding shows sensitive/resistant IC 50 profiles for individual samples from The Beat AML cohort. Four drugs with largest F-statistics are depicted. Interestingly, our prediction show feasibility to simultaneously identify clusters of patients characterized by increased resistance for one drug ( e.g., trametinib) and high sensitivity to others ( e.g., rapamycin, dasatinib, doramapimod). In sum, our study suggests that multi-modal genomic configuration increases the power of resolution in HR-MDS/sAML by reducing molecular heterogeneity, which in turn might help resolving therapeutic vulnerabilities.

Dimensionality reduction in stochastic complex dynamical networks

Complex systems often exhibit diverse dynamical behaviors in high-dimensional spaces that depend on various factors. Dimensionality reduction is a powerful tool for analyzing and understanding complex systems, aiming to find a low-dimensional representation of the complex system that preserves its essential features and reveals its underlying mechanisms and long-term dynamics. However, most existing methods for dimensionality reduction are limited to deterministic systems and cannot account for the stochastic effects that are ubiquitous in real-world complex networks. Here we develop a general analytical framework for dimensionality reduction of stochastic complex dynamical networks that can capture the essential features and long-term dynamics of the original system in a low-dimensional effective equation. The effective equation is a function of a set of effective parameters that are associated with specific system states and determine the network's dynamical behavior. We show that the standard deviation of the effective equation can be used to analyze the dynamic behavior and possible convergence of the stochastic complex dynamical network. Our framework can be applied to various types of stochastic complex dynamical networks and can reveal the underlying mechanisms and emergent phenomena of these systems.

Two phase cooperative learning for supervised dimensionality reduction

The simultaneous minimization of the reconstruction and classification error is a hard non convex problem, especially when a non-linear mapping is utilized. To overcome this obstacle, motivated by the widespread success of Cooperative Neural Networks, an innovative supervised dimensionality reduction framework is proposed, based on a cooperative two phase optimization strategy. Specifically, the proposed framework that requires minimal parameter adjustment consists of an autoencoder for dimensionality reduction and a separator network for separability assessment of the embedding. This scheme results in meaningful and discriminable codes, which are optimized for the classification task and are exploitable by any trainable classifier. The experimental results showed that the proposed methodology achieved competitive results against the state-of-the-art competing methods, while being much more efficient in terms of parameter count. Finally, it was empirically justified that the proposed methodology introduces advanced behavioural explainability, while enabling applicability for image generation tasks.

Identification of transcriptional programs using dense vector representations defined by mutual information with GeneVector

Deciphering individual cell phenotypes from cell-specific transcriptional processes requires high dimensional single cell RNA sequencing. However, current dimensionality reduction methods aggregate sparse gene information across cells, without directly measuring the relationships that exist between genes. By performing dimensionality reduction with respect to gene co-expression, low-dimensional features can model these gene-specific relationships and leverage shared signal to overcome sparsity. We describe GeneVector, a scalable framework for dimensionality reduction implemented as a vector space model using mutual information between gene expression. Unlike other methods, including principal component analysis and variational autoencoders, GeneVector uses latent space arithmetic in a lower dimensional gene embedding to identify transcriptional programs and classify cell types. In this work, we show in four single cell RNA-seq datasets that GeneVector was able to capture phenotype-specific pathways, perform batch effect correction, interactively annotate cell types, and identify pathway variation with treatment over time.

Discovering epistasis interactions in Alzheimer’s disease using integrated framework of ensemble learning and multifactor dimensionality reduction (MDR)

Alzheimer's disease (AD) is a complex disorder with strong genetic factors. The proposed framework is applied to Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset. We present a novel framework integrating ensemble learning and MDR constructive induction algorithm to discover epistasis interactions associated with AD in a computationally efficient method. Discovering epistasis interactions is a big challenge and significantly impacts personalized medicine (PM). The applied ensemble learning algorithms are random forests (RF) with Gini index and permutation importance, Extreme Gradient Boosting (XGBoost), and classification and regression trees (CART). The classification accuracy of 5-way models varied between (0.8674–0.8758), whereas the accuracy of 2-way, 3-way, and 4-way models varied between (0.6515–0.6649), (0.7071–0.7170), and (0.7811–0.7878) respectively. The promising results of this proposed framework show high-ranked risk genes and up to 5-way epistasis models that contribute to the disease risk efficiently and at higher accuracy.

Semi-supervised t-SNE with multi-scale neighborhood preservation

Unsupervised dimensionality reduction (DR) aims to preserve input data structure in a low-dimensional (LD) space based on neighborhood information. In contrast, supervised DR intends to improve the learning performance, i.e., classification and regression, in an LD representation. Unfortunately, obtaining the complete label outputs of a data set for real-world applications is hard. Here, we introduce a novel DR framework coupling both available class labels and input feature similarities to extend the well-known t-distributed Stochastic Neighbor Embedding (SNE) for semi-supervised scenarios. Our proposal, termed Semi-Supervised t-SNE (SS.t-SNE), properly fixes the widths of Gaussian neighborhoods to reveal the salient local and global data structures in an LD space. Indeed, our approach is presented as a generalization of unsupervised and supervised versions of t-SNE. SS.t-SNE outperforms other semi-supervised DR methods in data visualization and classification tasks in LD embeddings.

Flexible Adaptive Graph Embedding for Semi-supervised Dimension Reduction

Graph-based semi-supervised dimension reduction can use the inherent graph structure of samples to propagate label information, and has become a hot research field in machine learning. However, most current methods have strict linear constraints and cannot handle data sampled from nonlinear manifolds; and rely on predefined graphs, which cannot capture the local structure information of data and cannot handle complex non-Gaussian data. To address these issues, this paper proposes a new locality-preserved flexible dimension reduction framework, called Semi-supervised Flexible Adaptive Graph Embedding (SFAG), which learns the embedding space that can preserve the local neighborhood structure by constructing a k1-nearest neighbor graph over labeled samples. Then, another k2-nearest neighbor graph is constructed on all samples to adaptively construct the optimal graph, clustering labeled and unlabeled embedding sample points with neighborhood relations into the same sub-manifold sharing the same label information. Last but not least, the hard linear projection constraint is relaxed by adding residual terms to obtain not only the nonlinear embedding of the training samples, but also the linear projection matrix applied directly to the out-of-sample. In addition, two different semi-supervised dimension reduction methods for adaptive construction of optimal graphs are proposed based on the SFAG framework. Several evaluation experiments validate the effectiveness of our method in exploring manifold structures and classification tasks.