Presence Of High-dimensional Data Research Articles

Parsimonious ultrametric Gaussian mixture models

Gaussian mixture models represent a conceptually and mathematically elegant class of models for casting the density of a heterogeneous population where the observed data is collected from a population composed of a finite set of G homogeneous subpopulations with a Gaussian distribution. A limitation of these models is that they suffer from the curse of dimensionality, and the number of parameters becomes easily extremely large in the presence of high-dimensional data. In this paper, we propose a class of parsimonious Gaussian mixture models with constrained extended ultrametric covariance structures that are capable of exploring hierarchical relations among variables. The proposal shows to require a reduced number of parameters to be fit and includes constrained covariance structures across and within components that further reduce the number of parameters of the model.

MISNN: Multiple Imputation via Semi-parametric Neural Networks.

Multiple imputation (MI) has been widely applied to missing value problems in biomedical, social and econometric research, in order to avoid improper inference in the downstream data analysis. In the presence of high-dimensional data, imputation models that include feature selection, especially regularized regression (such as Lasso, adaptive Lasso, and Elastic Net), are common choices to prevent the model from underdetermination. However, conducting MI with feature selection is difficult: existing methods are often computationally inefficient and poor in performance. We propose MISNN, a novel and efficient algorithm that incorporates feature selection for MI. Leveraging the approximation power of neural networks, MISNN is a general and flexible framework, compatible with any feature selection method, any neural network architecture, high/low-dimensional data and general missing patterns. Through empirical experiments, MISNN has demonstrated great advantages over state-of-the-art imputation methods (e.g. Bayesian Lasso and matrix completion), in terms of imputation accuracy, statistical consistency and computation speed.

Deep Cellular Recurrent Network for Efficient Analysis of Time-Series Data With Spatial Information.

Efficient processing of large-scale time-series data is an intricate problem in machine learning. Conventional sensor signal processing pipelines with hand-engineered feature extraction often involve huge computational costs with high dimensional data. Deep recurrent neural networks have shown promise in automated feature learning for improved time-series processing. However, generic deep recurrent models grow in scale and depth with the increased complexity of the data. This is particularly challenging in presence of high dimensional data with temporal and spatial characteristics. Consequently, this work proposes a novel deep cellular recurrent neural network (DCRNN) architecture to efficiently process complex multidimensional time-series data with spatial information. The cellular recurrent architecture in the proposed model allows for location-aware synchronous processing of time-series data from spatially distributed sensor signal sources. Extensive trainable parameter sharing due to cellularity in the proposed architecture ensures efficiency in the use of recurrent processing units with high-dimensional inputs. This study also investigates the versatility of the proposed DCRNN model for the classification of multiclass time-series data from different application domains. Consequently, the proposed DCRNN architecture is evaluated using two time-series data sets: a multichannel scalp electroencephalogram (EEG) data set for seizure detection, and a machine fault detection data set obtained in-house. The results suggest that the proposed architecture achieves state-of-the-art performance while utilizing substantially less trainable parameters when compared to comparable methods in the literature.

Real-time and consistent sparse estimation of power system distribution factors using online adaptive elastic-net

Distribution factors (DFs) are vital for power system monitoring and secure operation. However, it is challenging to accurately estimate dominant DFs while promoting the sparsity of DFs in real time, especially in large-scale power systems with high-dimensional data. This paper proposes a novel Online Adaptive Elastic-net (AdaEnet) method to address this challenge, which allows the real-time and consistent sparse estimation of DFs in large-scale power systems. The problem is formulated as a sparse regression problem using data measured by phasor measurement units. The AdaEnet estimator is advocated to promote the sparsity of DFs, and a LARS-AdaEnet algorithm is developed for the real-time estimation of DFs. Thanks to the oracle properties and the grouping effect of the AdaEnet, the method yields unbiased dominant estimates with stability to collinear predictor variables, even for underdetermined systems. Moreover, due to the online updated adaptive weights and the early-stopping property of the LARS-AdaEnet, the method efficiently solves the estimator using only a few latest samples. Test results on the IEEE 300- and the Polish 3375-bus systems demonstrate that the proposed method achieves high estimation consistency and real-time adaptability in the presence of high-dimensional data (i.e., massive collinear predictor variables of small sample size). Hence this method promotes the data-driven, sparsity-aware, and computationally efficient sensitivity analysis of active power line flows in large-scale power systems. It can broadly facilitate real-time situational awareness and decision-making for improving power system security.

Bayesian interaction selection model for multimodal neuroimaging data analysis.

Multimodality or multiconstruct data arise increasingly in functional neuroimaging studies to characterize brain activity under different cognitive states. Relying on those high-resolution imaging collections, it is of great interest to identify predictive imaging markers and intermodality interactions with respect to behavior outcomes. Currently, most of the existing variable selection models do not consider predictive effects from interactions, and the desired higher-order terms can only be included in the predictive mechanism following a two-step procedure, suffering from potential misspecification. In this paper, we propose a unified Bayesian prior model to simultaneously identify main effect features and intermodality interactions within the same inference platform in the presence of high-dimensional data. To accommodate the brain topological information and correlation between modalities, our prior is designed by compiling the intermediate selection status of sequential partitions in light of the data structure and brain anatomical architecture, so that we can improve posterior inference and enhance biological plausibility. Through extensive simulations, we show the superiority of our approach in main and interaction effects selection, and prediction under multimodality data. Applying the method to the Adolescent Brain Cognitive Development (ABCD) study, we characterize the brain functional underpinnings with respect to general cognitive ability under different memory loadconditions.

Deselection of base-learners for statistical boosting-with an application to distributional regression.

We present a new procedure for enhanced variable selection for component-wise gradient boosting. Statistical boosting is a computational approach that emerged from machine learning, which allows to fit regression models in the presence of high-dimensional data. Furthermore, the algorithm can lead to data-driven variable selection. In practice, however, the final models typically tend to include too many variables in some situations. This occurs particularly for low-dimensional data (), where we observe a slow overfitting behavior of boosting. As a result, more variables get included into the final model without altering the prediction accuracy. Many of these false positives are incorporated with a small coefficient and therefore have a small impact, but lead to a larger model. We try to overcome this issue by giving the algorithm the chance to deselect base-learners with minor importance. We analyze the impact of the new approach on variable selection and prediction performance in comparison to alternative methods including boosting with earlier stopping as well as twin boosting. We illustrate our approach with data of an ongoing cohort study for chronic kidney disease patients, where the most influential predictors for the health-related quality of life measure are selected in a distributional regression approach based on beta regression.

Multiple imputation using nearest neighbor methods

Missing values are a major problem in medical research. As the complete case analysis discards useful information, estimation and inference may suffer strongly. Multiple imputation has been shown to be a useful strategy to handle missing data problems and account for the uncertainty of imputation. In the presence of high-dimensional data (p≫n), the missing values raise even more serious problems as the existing software packages tend to fail. We present multiple imputation methods based on nearest neighbors. The distances are computed using the information of correlation among the target and candidate predictors. Thus only the relevant predictors contribute for computing distances. The method successfully imputes missing values also in high-dimensional settings. Using a variety of simulated data with MCAR and MAR missing patterns, the proposed algorithm is compared to existing methods. Various measures are used to compare the performance of methods, including MSE for imputation, MSE of estimated regression coefficients, their standard errors, confidence intervals, and their coverage probabilities. The simulation results, for both cases n<p and n>p, show that the sequential imputation using weighted nearest neighbors can be successfully applied to a wide range of data settings and outperforms or is close to the best when compared to existing methods.

Variational Bayesian partially linear mean shift models for high-dimensional Alzheimer's disease neuroimaging data.

Alzheimer's disease can be diagnosed by analyzing brain images (eg, magnetic resonance imaging, MRI) and neuropsychological tests (eg, mini-mental state examination, MMSE). A partially linear mean shift model (PLMSM) is here proposed to investigate the relationship between MMSE score and high-dimensional regions of interest in MRI, and detect the outliers. In the presence of high-dimensional data, existing Bayesian approaches (eg, Markov chain Monte Carlo) to analyze a PLMSM take intensive computational cost and require huge memory, and have low convergence rate. To address these issues, a variational Bayesian inference is developed to simultaneously estimate parameters and nonparametric functions and identify outliers in a PLMSM. A Bayesian P-splines method is presented to approximate nonparametric functions, a Bayesian adaptive Lasso approach is employed to select predictors, and outliers are detected by the classification variable. Two simulation studies are conducted to assess the finite sample performance of the proposed method. An MRI dataset with elderly cognitive ability is provided to corroborate the proposed method.

Uncertain distance-based outlier detection with arbitrarily shaped data objects

Enabling information systems to face anomalies in the presence of uncertainty is a compelling and challenging task. In this work the problem of unsupervised outlier detection in large collections of data objects modeled by means of arbitrary multidimensional probability density functions is considered. We present a novel definition of uncertain distance-based outlier under the attribute level uncertainty model, according to which an uncertain object is an object that always exists but its actual value is modeled by a multivariate pdf. According to this definition an uncertain object is declared to be an outlier on the basis of the expected number of its neighbors in the dataset. To the best of our knowledge this is the first work that considers the unsupervised outlier detection problem on data objects modeled by means of arbitrarily shaped multidimensional distribution functions. We present the UDBOD algorithm which efficiently detects the outliers in an input uncertain dataset by taking advantages of three optimized phases, that are parameter estimation, candidate selection, and the candidate filtering. An experimental campaign is presented, including a sensitivity analysis, a study of the effectiveness of the technique, a comparison with related algorithms, also in presence of high dimensional data, and a discussion about the behavior of our technique in real case scenarios.

Designing Distributed Fuzzy Rule-Based Models

In the design of fuzzy rule-based models, in the presence of high-dimensional data, we are faced with conceptual and algorithmic challenges. Conceptually, as the dimensionality of data increases, the notion of distance starts to be questionable, resulting in the well-known concentration effect. This has a direct detrimental effect given the fact that the distance is used in fuzzy clustering, using which the condition parts of the rules are being formed. Computationally, with the increase of dimensionality, the computing overhead becomes significant and has to be carefully addressed. In this article, we advocate a construction of distributed fuzzy rule-based models, where instead of a single monolithic (multivariable) rule-based model, we construct a collection of low-dimensional (in particular, 1- or 2-D) rule-based models and aggregate their results through some linear transformations. A suite of experimental studies realized using publicly available data are reported along with a comparative analysis engaging accuracy criteria and computing overhead. Interestingly, building distributed models exhibits some tangible benefits over the construction of the monolithic rule-based models. In terms of accuracy and computing costs, the gains of 1-D rule-based models with optimal linkage matrix (on average) are around 43.46% and 98.85%, respectively.

Optimising Prediction in Overlapping and Non-Overlapping Regions

Researchers working on real world classification data have identified that a combination of class overlap with class imbalance and high dimensional data is a crucial problem and are important factors for degrading performance of the classifier. Hence, it has received significant attention in recent years. Misclassification often occurs in the overlapped region as there is no clear distinction between the class boundaries and the presence of high dimensional data with an imbalanced proportion poses an additional challenge. Only a few studies have ever been attempted to address all these issues simultaneously; therefore; a model is proposed which initially divides the data space into overlapped and non-overlapped regions using a K-means algorithm, then the classifier is allowed to learn from two data space regions separately and finally, the results are combined. The experiment is conducted using the Heart dataset selected from the Keel repository and results prove that the proposed model improves the efficiency of the classifier based on accuracy, kappa, precision, recall, f-measure, FNR, FPR, and time.

Every Bit Counts: Using Deep Learning and Vectorization to Analyze Healthcare Big Data

The rapid digitization of healthcare has generated large volumes of rich and complex data from sources such as claims and electronic health records. Traditional analytic approaches, however, only utilize small subsets of these data and often require deep domain knowledge. New methods are needed to handle both the scale and complexity of big data in healthcare. We employ a recent breakthrough in computer science to develop a new Deep Learning-based Vectorization (DLV) approach for more comprehensive and efficient analysis of healthcare data. This approach automatically converts data elements into standardized numeric vectors, empowering new types of computing and improve performance in traditional data analysis. We demonstrate the potential of DLV to predict 30-day readmission using discharge records that cover all emergency department visits and inpatient hospitalizations in Florida. We find that while traditional approaches struggle even to load large amounts of clinical information (including non-numeric variables), DLV handles this information easily. Furthermore, DLV significantly improves the accuracy of 30-day readmission prediction in the presence of high-dimensional data. While traditional logistic regression yields an AUC of 0.61, DLV gives us 0.79, a 163% improvement in the lift over the baseline of 0.50. In addition, we demonstrate that the vector representations offered by DLV afford easy visualization for better understanding of the clinical data. Overall, the DLV approach thus shows great potential in facilitating the analysis of big healthcare data and can complement traditional methods in high-dimensional environments.

Probabilistic Feature Selection and Classification Vector Machine

Sparse Bayesian learning is a state-of-the-art supervised learning algorithm that can choose a subset of relevant samples from the input data and make reliable probabilistic predictions. However, in the presence of high-dimensional data with irrelevant features, traditional sparse Bayesian classifiers suffer from performance degradation and low efficiency due to the incapability of eliminating irrelevant features. To tackle this problem, we propose a novel sparse Bayesian embedded feature selection algorithm that adopts truncated Gaussian distributions as both sample and feature priors. The proposed algorithm, called probabilistic feature selection and classification vector machine (PFCVM LP ) is able to simultaneously select relevant features and samples for classification tasks. In order to derive the analytical solutions, Laplace approximation is applied to compute approximate posteriors and marginal likelihoods. Finally, parameters and hyperparameters are optimized by the type-II maximum likelihood method. Experiments on three datasets validate the performance of PFCVM LP along two dimensions: classification performance and effectiveness for feature selection. Finally, we analyze the generalization performance and derive a generalization error bound for PFCVM LP . By tightening the bound, the importance of feature selection is demonstrated.

The cluster correlation-network support vector machine for high-dimensional binary classification

ABSTRACTIdentifying homogeneous subsets of predictors in classification can be challenging in the presence of high-dimensional data with highly correlated variables. We propose a new method called cluster correlation-network support vector machine (CCNSVM) that simultaneously estimates clusters of predictors that are relevant for classification and coefficients of penalized SVM. The new CCN penalty is a function of the well-known Topological Overlap Matrix whose entries measure the strength of connectivity between predictors. CCNSVM implements an efficient algorithm that alternates between searching for predictors’ clusters and optimizing a penalized SVM loss function using Majorization–Minimization tricks and a coordinate descent algorithm. This combining of clustering and sparsity into a single procedure provides additional insights into the power of exploring dimension reduction structure in high-dimensional binary classification. Simulation studies are considered to compare the performance of our procedure to its competitors. A practical application of CCNSVM on DNA methylation data illustrates its good behaviour.

A Fuzzy Classifier with Feature Selection Based on the Gravitational Search Algorithm

This paper concerns several important topics of the Symmetry journal, namely, pattern recognition, computer-aided design, diversity and similarity. We also take advantage of the symmetric and asymmetric structure of a transfer function, which is responsible to map a continuous search space to a binary search space. A new method for design of a fuzzy-rule-based classifier using metaheuristics called Gravitational Search Algorithm (GSA) is discussed. The paper identifies three basic stages of the classifier construction: feature selection, creating of a fuzzy rule base and optimization of the antecedent parameters of rules. At the first stage, several feature subsets are obtained by using the wrapper scheme on the basis of the binary GSA. Creating fuzzy rules is a serious challenge in designing the fuzzy-rule-based classifier in the presence of high-dimensional data. The classifier structure is formed by the rule base generation algorithm by using minimum and maximum feature values. The optimal fuzzy-rule-based parameters are extracted from the training data using the continuous GSA. The classifier performance is tested on real-world KEEL (Knowledge Extraction based on Evolutionary Learning) datasets. The results demonstrate that highly accurate classifiers could be constructed with relatively few fuzzy rules and features.

Multiple imputation in the presence of high-dimensional data

Missing data are frequently encountered in biomedical, epidemiologic and social research. It is well known that a naive analysis without adequate handling of missing data may lead to bias and/or loss of efficiency. Partly due to its ease of use, multiple imputation has become increasingly popular in practice for handling missing data. However, it is unclear what is the best strategy to conduct multiple imputation in the presence of high-dimensional data. To answer this question, we investigate several approaches of using regularized regression and Bayesian lasso regression to impute missing values in the presence of high-dimensional data. We compare the performance of these methods through numerical studies, in which we also evaluate the impact of the dimension of the data, the size of the true active set for imputation, and the strength of correlation. Our numerical studies show that in the presence of high-dimensional data the standard multiple imputation approach performs poorly and the imputation approach using Bayesian lasso regression achieves, in most cases, better performance than the other imputation methods including the standard imputation approach using the correctly specified imputation model. Our results suggest that Bayesian lasso regression and its extensions are better suited for multiple imputation in the presence of high-dimensional data than the other regression methods.

Local and Global Preserving Semisupervised Dimensionality Reduction Based on Random Subspace for Cancer Classification

Precise cancer classification is essential to the successful diagnosis and treatment of cancers. Although semisupervised dimensionality reduction approaches perform very well on clean datasets, the topology of the neighborhood constructed with most existing approaches is unstable in the presence of high-dimensional data with noise. In order to solve this problem, a novel local and global preserving semisupervised dimensionality reduction based on random subspace algorithm marked as RSLGSSDR, which utilizes random subspace for semisupervised dimensionality reduction, is proposed. The algorithm first designs multiple diverse graphs on different random subspace of datasets and then fuses these graphs into a mixture graph on which dimensionality reduction is performed. As themixture graph is constructed in lower dimensionality, it can ease the issues on graph construction on highdimensional samples such that it can hold complicated geometric distribution of datasets as the diversity of random subspaces. Experimental results on public gene expression datasets demonstrate that the proposed RSLGSSDR not only has superior recognition performance to competitive methods, but also is robust against a wide range of values of input parameters.