Dimension Reduction Estimation Research Articles

Distributed Estimation of Principal Support Vector Machines for Sufficient Dimension Reduction

The principal support vector machines method (Li et al., 2011) is a powerful tool for sufficient dimension reduction that replaces original predictors with their low-dimensional linear combinations while preserving the information for regression and classification. However, the computational burden of the principal support vector machines method constrains its use for massive data. To address this issue, we propose a naive and a refined distributed estimation algorithms for fast implementation when the sample size is large. Both distributed sufficient dimension reduction estimators exhibit the same statistical efficiency as when all the data is merged together, which provides rigorous statistical guarantees for their application to large-scale datasets, while the refined method requires smaller batch sample sizes and hence is more advantageous when memory limitations exist on distributed machines. The two distributed algorithms are further adapted to principal weighted support vector machines (Shin et al., 2017) for sufficient dimension reduction in binary classification. The statistical accuracy and computational complexity of our proposed methods are examined through comprehensive simulation studies and in a real data application with more than 600000 samples.

Sufficient dimension reduction in the presence of controlling variable and missing multivariate response

In this paper, we focus on partial dimension reduction for conditional mean function in the presence of controlling variable when responses are multivariate and missing at random. A weighted-profile least squares estimate based on inverse probability weighted technique is proposed to estimate the central mean subspace. The profile least squares method does not need any distributional assumptions on the covariates, which is different from existing methods in sufficient dimension reduction. It turns out that under some mild conditions, the estimator of central mean subspace is asymptotically normal and root-n consistent. The proposed test statistic as well as its asymptotic distribution for parametric hypothesis tests are obtained. In addition, a BIC-type criterion is used to determine the structural dimension of the central mean subspace. Its consistency is also established. Both simulated and real data analysis results demonstrate promising performance of the proposed sufficient dimension reduction estimation method.

Correlation Imputation for Single-Cell RNA-seq.

Recent advances in single-cell RNA sequencing (scRNA-seq) technologies have yielded a powerful tool to measure gene expression of individual cells. One major challenge of the scRNA-seq data is that it usually contains a large amount of zero expression values, which often impairs the effectiveness of downstream analyses. Numerous data imputation methods have been proposed to deal with these "dropout" events, but this is a difficult task for such high-dimensional and sparse data. Furthermore, there have been debates on the nature of the sparsity, about whether the zeros are due to technological limitations or represent actual biology. To address these challenges, we propose Single-cell RNA-seq Correlation completion by ENsemble learning and Auxiliary information (SCENA), a novel approach that imputes the correlation matrix of the data of interest instead of the data itself. SCENA obtains a gene-by-gene correlation estimate by ensembling various individual estimates, some of which are based on known auxiliary information about gene expression networks. Our approach is a reliable method that makes no assumptions on the nature of sparsity in scRNA-seq data or the data distribution. By extensive simulation studies and real data applications, we demonstrate that SCENA is not only superior in gene correlation estimation, but also improves the accuracy and reliability of downstream analyses, including cell clustering, dimension reduction, and graphical model estimation to learn the gene expression network.

Group linear non-Gaussian component analysis with applications to neuroimaging

Independent component analysis (ICA) is an unsupervised learning method popular in functional magnetic resonance imaging (fMRI). Group ICA has been used to search for biomarkers in neurological disorders including autism spectrum disorder and dementia. However, current methods use a principal component analysis (PCA) step that may remove low-variance features. Linear non-Gaussian component analysis (LNGCA) enables simultaneous dimension reduction and feature estimation including low-variance features in single-subject fMRI. A group LNGCA model is proposed to extract group components shared by more than one subject. Unlike group ICA methods, this novel approach also estimates individual (subject-specific) components orthogonal to the group components. To determine the total number of components in each subject, a parametric resampling test is proposed that samples spatially correlated Gaussian noise to match the spatial dependence observed in data. In simulations, estimated group components achieve higher accuracy compared to group ICA. The method is applied to a resting-state fMRI study on autism spectrum disorder in 342 children (252 typically developing, 90 with autism), where the group signals include resting-state networks. The discovered group components appear to exhibit different levels of temporal engagement in autism versus typically developing children, as revealed using group LNGCA. This novel approach to matrix decomposition is a promising direction for feature detection in neuroimaging.

Advanced metering infrastructure for low voltage distribution system in smart grid based monitoring applications

Smart grid system is an emerging technology used for solving the issues associated with power and energy. Smart grid system is known to be an automatic network, ensuring power with the transfer of information and electricity between the power plants and household appliances. However, there is a lack in the data flow about the voltage distribution stability, accuracy and applicability. So, the advanced metering infrastructure, personal energy management and Information Technology are overviewed in this model. The features of smart meter enable the advanced metering infrastructure monitoring and investigates low-voltage (LV) network. This is achieved by optimal state, in which the power quality and outage data is transferred by the distribution network from smart meters to the control center. In this model, the smart grid architecture is mapped with optimal states to observe the LV network processing. Recursive Karhunen–Loéve Transform Grid state optimization Model (RK-LTGM) is proposed in this paper for ensuring the usage of new data and narrow down the variations in estimator range. The proposed model is robust against the network loss. Dimension Reduction Estimation Model Probabilistic Loss (DREM-PL), Korhonen Estimation Model (KEM) and Constant Network Loss Rate Estimation Model (CNLREM) are the three state-of-art methods considered for comparing this model. Recursion Least Squares (RLS) and Damped Recursion Least Squares (DRLS) are the index parameters considered in the proposed method. The proposed Recursive Karhunen–Loéve Transform Grid state optimization Model (RK-LTGM) helps in exchanging information in multilevel smart grid, allowing Smart meter to utilize the information to obtain the high quality best technological and economical solution for every consumer. It is found that the proposed RK-LTGM achieves 0.112 % of RMSE, 0.156 % of network loss rate, 56.124 % of success rate under RLS condition, 0.067 % of RMSE, 0.137 % of network loss rate and 88.739 % of success rate under DRLS condition.

Dictionary learning allows model-free pseudotime estimation of transcriptomic data

BackgroundPseudotime estimation from dynamic single-cell transcriptomic data enables characterisation and understanding of the underlying processes, for example developmental processes. Various pseudotime estimation methods have been proposed during the last years. Typically, these methods start with a dimension reduction step because the low-dimensional representation is usually easier to analyse. Approaches such as PCA, ICA or t-SNE belong to the most widely used methods for dimension reduction in pseudotime estimation methods. However, these methods usually make assumptions on the derived dimensions, which can result in important dataset properties being missed. In this paper, we suggest a new dictionary learning based approach, dynDLT, for dimension reduction and pseudotime estimation of dynamic transcriptomic data. Dictionary learning is a matrix factorisation approach that does not restrict the dependence of the derived dimensions. To evaluate the performance, we conduct a large simulation study and analyse 8 real-world datasets.ResultsThe simulation studies reveal that firstly, dynDLT preserves the simulated patterns in low-dimension and the pseudotimes can be derived from the low-dimensional representation. Secondly, the results show that dynDLT is suitable for the detection of genes exhibiting the simulated dynamic patterns, thereby facilitating the interpretation of the compressed representation and thus the dynamic processes. For the real-world data analysis, we select datasets with samples that are taken at different time points throughout an experiment. The pseudotimes found by dynDLT have high correlations with the experimental times. We compare the results to other approaches used in pseudotime estimation, or those that are method-wise closely connected to dictionary learning: ICA, NMF, PCA, t-SNE, and UMAP. DynDLT has the best overall performance for the simulated and real-world datasets.ConclusionsWe introduce dynDLT, a method that is suitable for pseudotime estimation. Its main advantages are: (1) It presents a model-free approach, meaning that it does not restrict the dependence of the derived dimensions; (2) Genes that are relevant in the detected dynamic processes can be identified from the dictionary matrix; (3) By a restriction of the dictionary entries to positive values, the dictionary atoms are highly interpretable.

The ensemble conditional variance estimator for sufficient dimension reduction

Ensemble Conditional Variance Estimation (ECVE) is a novel sufficient dimension reduction (SDR) method in regressions with continuous response and predictors. ECVE applies to general non-additive error regression models and operates under the assumption that the predictors can be replaced by a lower dimensional projection without loss of information. It is a semiparametric forward regression model-based exhaustive sufficient dimension reduction estimation method that is shown to be consistent under mild assumptions. ECVE outperforms central subspace mean average variance estimation (csMAVE), its main competitor, under several simulation settings and in a benchmark data set analysis.

Unsupervised Anomaly Detection Based on Deep Autoencoding and Clustering

The unsupervised anomaly detection task based on high-dimensional or multidimensional data occupies a very important position in the field of machine learning and industrial applications; especially in the aspect of network security, the anomaly detection of network data is particularly important. The key to anomaly detection is density estimation. Although the methods of dimension reduction and density estimation have made great progress in recent years, most dimension reduction methods are difficult to retain the key information of original data or multidimensional data. Recent studies have shown that the deep autoencoder (DAE) can solve this problem well. In order to improve the performance of unsupervised anomaly detection, we propose an anomaly detection scheme based on a deep autoencoder (DAE) and clustering methods. The deep autoencoder is trained to learn the compressed representation of the input data and then feed it to clustering approach. This scheme makes full use of the advantages of the deep autoencoder (DAE) to generate low-dimensional representation and reconstruction errors for the input high-dimensional or multidimensional data and uses them to reconstruct the input samples. The proposed scheme could eliminate redundant information contained in the data, improve performance of clustering methods in identifying abnormal samples, and reduce the amount of calculation. To verify the effectiveness of the proposed scheme, massive comparison experiments have been conducted with traditional dimension reduction algorithms and clustering methods. The results of experiments demonstrate that, in most cases, the proposed scheme outperforms the traditional dimension reduction algorithms with different clustering methods.

Insider threat prediction based on unsupervised anomaly detection scheme for proactive forensic investigation

The complexity, concealment and infrequency of malicious internal actions make it difficult to detect insider threats. In the process of traditional reactive forensic investigation, analysis and interpretation of the digital evidence are performed after a crime has been committed. Even if insiders can be detected, they have already caused huge damage. In this paper, we propose a novel general unsupervised anomaly detection scheme based on cascaded autoencoders (CAEs) and joint optimization network. Our core idea is to utilize CAEs to do data purification among unlabeled digital evidence, then jointly optimize the dimension reduction and density estimation network to avoid sub-optimal problems. Based on this scheme, we design an end-to-end insider threat prediction framework for proactive forensic investigation, through which we can make real time response to prevent the harmful influences of insider threats in advance. We extract the tractable and scalable feature representation automatically through the data driven Bidirectional Long Short-Term Memory (BiLSTM) feature extractor, waiving the time-consuming and customarily expert dependable feature engineering work. Additionally, a hypergraph correction module is applied to solve the commonly existed relatively high false positive rate problem in insider threat detection. We evaluate our scheme and framework on public benchmark datasets. The empirical experiments demonstrate that our models outperform state-of-the-art unsupervised methods.

Automatic sparse principal component analysis

The wide availability of computers enables us to accumulate a huge amount of data, thus effective tools to extract information from the huge volume of data have become critical. Principal component analysis (PCA) is a useful and traditional tool for dimensionality reduction of massive high‐dimensional datasets. Recently, sparse principal component (PC) loading estimation based on L1‐type regularization has drawn a large amount of attention. Although sparse PCA makes interpretation easily and performs dimension reduction without disturbance from noisy features, the existing studies on sparse PCA were based on an arbitrary number of PCs without any statistical justification. We propose a novel method, called as automatic sparse PCA, which can perform PC selection and sparse PC loading estimation, simultaneously. For PC selection, we first develop sparse singular value decomposition (sparse SVD), then incorporate sparsity into PC loading estimation. The proposed method enables us to perform dimension reduction and PC loading estimation, simultaneously. Furthermore, we can perform PCA without disturbance from noisy features. It can be seen through Monte Carlo experiments that the proposed automatic sparse PCA outperforms sparse structure identification and reconstructing data based on low‐dimensional projection. The proposed method is also applied to a number of real datasets and it can be also seen that our method achieves effectiveness for estimation accuracy and interpreting PCA results.

Envelopes in multivariate regression models with nonlinearity and heteroscedasticity

SummaryEnvelopes have been proposed in recent years as a nascent methodology for sufficient dimension reduction and efficient parameter estimation in multivariate linear models. We extend the classical definition of envelopes in Cook et al. (2010) to incorporate a nonlinear conditional mean function and a heteroscedastic error. Given any two random vectors ${X}\in\mathbb{R}^{p}$ and ${Y}\in\mathbb{R}^{r}$, we propose two new model-free envelopes, called the martingale difference divergence envelope and the central mean envelope, and study their relationships to the standard envelope in the context of response reduction in multivariate linear models. The martingale difference divergence envelope effectively captures the nonlinearity in the conditional mean without imposing any parametric structure or requiring any tuning in estimation. Heteroscedasticity, or nonconstant conditional covariance of ${Y}\mid{X}$, is further detected by the central mean envelope based on a slicing scheme for the data. We reveal the nested structure of different envelopes: (i) the central mean envelope contains the martingale difference divergence envelope, with equality when ${Y}\mid{X}$ has a constant conditional covariance; and (ii) the martingale difference divergence envelope contains the standard envelope, with equality when ${Y}\mid{X}$ has a linear conditional mean. We develop an estimation procedure that first obtains the martingale difference divergence envelope and then estimates the additional envelope components in the central mean envelope. We establish consistency in envelope estimation of the martingale difference divergence envelope and central mean envelope without stringent model assumptions. Simulations and real-data analysis demonstrate the advantages of the martingale difference divergence envelope and the central mean envelope over the standard envelope in dimension reduction.

An estimation method of smart meter errors based on DREM and DRLS

Remote estimation of smart meter errors based on measurement data analysis from the user smart meter comes into focus because field calibration has high maintenance cost and difficult to analyze in time. In this paper, a remote estimation method of smart meter errors based on Dimension Reduction Estimation Model (DREM) and Damped Recursion Least Squares (DRLS) is proposed, in which DREM deals with the insolvability of actual model, and DRLS algorithm improves the stability and accuracy of estimation. To verify the effectiveness and practicality the method proposed is applied in both laboratory and actual distribution feeder unit. The results show that the proposed method did not need to calculate the network loss independently in advance, and it can estimate the smart meter errors and network loss rate in real-time. Finally, the important parameters involved in the model and algorithm are discussed, and the advice of parameter selection is proposed.

Some Notes on Error Analysis for Kernel Based Regularized Interpolation

Kernel based regularized interpolation is one of the most important methods for approximating functions. The theory behind the kernel based regularized interpolation is the well-known Representer Theorem, which shows the form of approximation function in the reproducing kernel Hilbert spaces. Because of the advantages of the kernel based regularized interpolation, it is widely used in many mathematical and engineering applications, for example, dimension reduction and dimension estimation. However, the performance of the approximation is not fully understood from the theoretical perspective. In other word, the error analysis for the kernel based regularized interpolation is lacking. In this paper, some error bounds in terms of the reproducing kernel Hilbert space norm and Sobolev space norm are given to understand the behavior of the approximation function.

Model-based clustering with envelopes

Clustering analysis is an important unsupervised learning technique in multivariate statistics and machine learning. In this paper, we propose a set of new mixture models called CLEMM (in short for Clustering with Envelope Mixture Models) that is based on the widely used Gaussian mixture model assumptions and the nascent research area of envelope methodology. Formulated mostly for regression models, envelope methodology aims for simultaneous dimension reduction and efficient parameter estimation, and includes a very recent formulation of envelope discriminant subspace for classification and discriminant analysis. Motivated by the envelope discriminant subspace pursuit in classification, we consider parsimonious probabilistic mixture models where the cluster analysis can be improved by projecting the data onto a latent lower-dimensional subspace. The proposed CLEMM framework and the associated envelope-EM algorithms thus provide foundations for envelope methods in unsupervised and semi-supervised learning problems. Numerical studies on simulated data and two benchmark data sets show significant improvement of our propose methods over the classical methods such as Gaussian mixture models, K-means and hierarchical clustering algorithms. An R package is available at https://github.com/kusakehan/CLEMM.

Dimension reduction estimation for central mean subspace with missing multivariate response

Multivariate response data often arise in practice and they are frequently subject to missingness. Under this circumstance, the standard sufficient dimension reduction (SDR) methods cannot be used directly. To reduce the dimension and estimate the central mean subspace, a profile least squares estimation method is proposed based on an inverse probability weighted technique. The profile least squares method does not need any distributional assumptions on the covariates and hence differs from existing SDR methods. The resulting estimator of the central mean subspace is proved to be asymptotically normal and root n consistent under some mild conditions. The structural dimension is determined by a BIC-type criterion and the consistency of its estimator is established. Comprehensive simulations and a real data analysis show that the proposed method works promisingly.

Common reducing subspace model and network alternation analysis.

Motivated by brain connectivity analysis and many other network data applications, we study the problem of estimating covariance and precision matricesand their differencesacross multiple populations. We propose a common reducing subspace model that leads to substantial dimension reduction and efficient parameter estimation. We explicitly quantify the efficiency gain through an asymptotic analysis. Our method is built upon and further extends a nascent technique, the envelope model, which adopts a generalized sparsity principle. This distinguishes our proposal from most xisting covariance and precision estimation methods that assume element-wise sparsity. Moreover, unlike most existing solutions, our method can naturally handle both covariance and precision matrices in a unified way, and work with matrix-valued data. We demonstrate the efficacy of our method through intensive simulations, and illustrate the method with an autism spectrum disorder data analysis.

Sufficient direction factor model and its application to gene expression quantitative trait loci discovery.

Rapid improvement in technology has made it relatively cheap to collect genetic data, however statistical analysis of existing data is still much cheaper. Thus, secondary analysis of single-nucleotide polymorphism, SNP, data, i.e., reanalysing existing data in an effort to extract more information, is an attractive and cost-effective alternative to collecting new data. We study the relationship between gene expression and SNPs through a combination of factor analysis and dimension reduction estimation. To take advantage of the flexibility in traditional factor models where the latent factors are not required to be normal, we recommend using semiparametric sufficient dimension reduction methods in the joint estimation of the combined model. The resulting estimator is flexible and has superior performance relative to the existing estimator, which relies on additional assumptions on the latent factors. We quantify the asymptotic performance of the proposed parameter estimator and perform inference by assessing the estimation variability and by constructing confidence intervals. The new results enable us to identify, for the first time, statistically significant SNPs concerning gene-SNP relations in lung tissue from genotype-tissue expression data.

Joint Tensor Expectile Regression for Electricity Day-Ahead Price Curves

Forecasting of electricity day-ahead prices has become an important field in recent research due to the tremendous increase in unpredictable renewable power in-feed to the electricity systems. We propose new methodology that is able to capture the intraday structure and the structural brake from working days to weekends incorporated in a functional tensor expectile regression that deploys factorisation for dimension reduction and joint estimation of the coefficient space employing trust region optimisation. Our model captures the common and individual structure among and between the weekdays, respectively. Moreover, in contrast to standard literature, our model does not disrupt the continuity of time as the daily intraday price curves are treated as one observation. We apply our methodology to German electricity prices that show high volatility and negative prices since the strong penetration of renewable power. The EPEXSPOT price data is merged with time series data for electricity load forecasts and renewable forecasts. Further, the out-of-sample forecasting results indicate outperformance of our candidate model over standard practice.

Linear Non-Gaussian Component Analysis Via Maximum Likelihood

ABSTRACTIndependent component analysis (ICA) is popular in many applications, including cognitive neuroscience and signal processing. Due to computational constraints, principal component analysis (PCA) is used for dimension reduction prior to ICA (PCA+ICA), which could remove important information. The problem is that interesting independent components (ICs) could be mixed in several principal components that are discarded and then these ICs cannot be recovered. We formulate a linear non-Gaussian component model with Gaussian noise components. To estimate the model parameters, we propose likelihood component analysis (LCA), in which dimension reduction and latent variable estimation are achieved simultaneously. Our method orders components by their marginal likelihood rather than ordering components by variance as in PCA. We present a parametric LCA using the logistic density and a semiparametric LCA using tilted Gaussians with cubic B-splines. Our algorithm is scalable to datasets common in applications (e.g., hundreds of thousands of observations across hundreds of variables with dozens of latent components). In simulations, latent components are recovered that are discarded by PCA+ICA methods. We apply our method to multivariate data and demonstrate that LCA is a useful data visualization and dimension reduction tool that reveals features not apparent from PCA or PCA+ICA. We also apply our method to a functional magnetic resonance imaging experiment from the Human Connectome Project and identify artifacts missed by PCA+ICA. We present theoretical results on identifiability of the linear non-Gaussian component model and consistency of LCA. Supplementary materials for this article are available online.

Dimension reduction regressions with measurement errors subject to additive distortion

ABSTRACTIn this paper, we propose several dimension reduction methods when the covariates are measured with additive distortion measurement errors. These distortions are modelled by unknown functions of a commonly observable confounding variable. To estimate the central subspace, we propose residuals-based dimension reduction estimation methods and direct estimation methods. The consistency and asymptotic normality of the proposed estimators are investigated. Furthermore, we conduct some simulations to evaluate the performance of our proposed method and compare with existing methods, and a real data set is analysed for illustration.