High-dimensional Classification Research Articles

A New Reduced-Rank Linear Discriminant Analysis Method and Its Applications

Many high dimensional classification techniques have been developed recently. However, many works focus on only the binary classification problem and can not be used directly for multi-class data. Most available classification tools for the high dimensional, multi-class data are either based on diagonal covariance structure or computationally complicated. In this paper, following the idea of reduced-rank linear discriminant analysis (LDA), we introduce a new dimension reduction tool with a flavor of supervised principal component analysis (PCA). The proposed method is computationally efficient and can incorporate the correlation structure among the features. Besides the theoretical insights, we show that our method is a competitive classification tool by simulated and real data examples.

Resource optimisation and fault detection algorithms for cloud computing platforms based on SVM and resource reserve strategy

Efficient operation of cloud computing platforms depends on the optimised virtual resources and faster fault diagnosis system of the virtual machines. This paper proposes an algorithm by introducing a virtual machines based on elastic reservation mechanism, which can improve the availability of cloud resources through the demand analysis taking the help of support vector machines which has advantages resolving nonlinear and high dimensional classification problems. Secondly it adopts the anomaly detection algorithm based on support vector machines for failure analysis. In addition, the dimensionality problem can be sorted out by means of principal component analysis (PCA) algorithm and a kernel function used for distance measurement. It establishes the topological structure for the image set of feature space with Delaunay triangulation and analyses the relationship between kernel parameter and regulator, in order to build an effective model.

Hierarchical Flexible Beta Fuzzy Design by a Multi-Objective Evolutionary Hybrid Approach

This paper introduces a novel evolutionary approach for the automated modeling of efficient hierarchical or multilevel fuzzy systems. The proposed approach is formed by two principal learning stages: the first stage consists of using a proposed multi-objective algorithm called the multi-objective extended immune programming algorithm in order to evolve the architecture of the hierarchical fuzzy system. The use of such a step aims to optimize the structure of the hierarchical fuzzy system and to generate in the same time an accurate and an interpretable fuzzy system with a few number of fuzzy rules. In the second stage, the parameters of beta membership function parameters and the consequent parts of rules are tuned by applying the hybrid artificial bee colony algorithm. The proposed hybrid approach interleaves these two learning phases for the architecture learning and the parameter tuning until a near optimum hierarchical fuzzy system is generated. The efficiency of the methodology is evaluated through some well-known benchmark time-series problems, a nonlinear plant identification problem, and some high-dimensional classification data sets. Compared with other existing works, the proposed system proves its superiority in terms of reaching high accuracy, smaller rule-base, and good convergence speed.

A high-dimensional classification approach based on class-dependent feature subspace

Purpose The purpose of this paper is to build a compact and accurate classifier for high-dimensional classification. Design/methodology/approach A classification approach based on class-dependent feature subspace (CFS) is proposed. CFS is a class-dependent integration of a support vector machine (SVM) classifier and associated discriminative features. For each class, our genetic algorithm (GA)-based approach evolves the best subset of discriminative features and SVM classifier simultaneously. To guarantee convergence and efficiency, the authors customize the GA in terms of encoding strategy, fitness evaluation, and genetic operators. Findings Experimental studies demonstrated that the proposed CFS-based approach is superior to other state-of-the-art classification algorithms on UCI data sets in terms of both concise interpretation and predictive power for high-dimensional data. Research limitations/implications UCI data sets rather than real industrial data are used to evaluate the proposed approach. In addition, only single-label classification is addressed in the study. Practical implications The proposed method not only constructs an accurate classification model but also obtains a compact combination of discriminative features. It is helpful for business makers to get a concise understanding of the high-dimensional data. Originality/value The authors propose a compact and effective classification approach for high-dimensional data. Instead of the same feature subset for all the classes, the proposed CFS-based approach obtains the optimal subset of discriminative feature and SVM classifier for each class. The proposed approach enhances both interpretability and predictive power for high-dimensional data.

High dimensional classification with combined adaptive sparse PLS and logistic regression.

The high dimensionality of genomic data calls for the development of specific classification methodologies, especially to prevent over-optimistic predictions. This challenge can be tackled by compression and variable selection, which combined constitute a powerful framework for classification, as well as data visualization and interpretation. However, current proposed combinations lead to unstable and non convergent methods due to inappropriate computational frameworks. We hereby propose a computationally stable and convergent approach for classification in high dimensional based on sparse Partial Least Squares (sparse PLS). We start by proposing a new solution for the sparse PLS problem that is based on proximal operators for the case of univariate responses. Then we develop an adaptive version of the sparse PLS for classification, called logit-SPLS, which combines iterative optimization of logistic regression and sparse PLS to ensure computational convergence and stability. Our results are confirmed on synthetic and experimental data. In particular, we show how crucial convergence and stability can be when cross-validation is involved for calibration purposes. Using gene expression data, we explore the prediction of breast cancer relapse. We also propose a multicategorial version of our method, used to predict cell-types based on single-cell expression data. Our approach is implemented in the plsgenomics R-package. ghislain.durif@inria.fr. Supplementary data are available at Bioinformatics online.

The Role of Gut Microbiome in the Pathogenesis of Prostate Cancer: A Prospective, Pilot Study

To elucidate potential biomarkers or mechanistic principles involved with the gut microbiota and its impact on prostate cancer pathogenesis. We performed a prospective case-control pilot study evaluating the gut microbiome of 20 men with either benign prostatic conditions (n = 8) or intermediate or high risk clinically localized prostate cancer (Gleason ≥4 + 3 cN0M0) (n = 12) undergoing care at tertiary referral center from September 1, 2015 to March 1, 2016. Key exclusion criteria included recent antibiotic use, significant gastrointestinal disorders, hormonal or systemic therapy for prostate cancer. Computational genomics analysis was performed on collected stool samples using MetaPhlAn2 and HUMAnN2 platforms. Linear discriminant analysis effect size method was used to support high-dimensional class comparisons to find biologically relevant features. Kruskal-Wallis sum-rank test was used to detect features with significant differential abundance with respect to class, with biological consistency investigated using a set of pairwise tests among subclasses using the Wilcoxon rank-sum test, both to an α ≤0.05. Higher relative abundance of Bacteriodes massiliensis was seen in prostate cancer cases compared to controls. Faecalibacterium prausnitzii and Eubacterium rectalie had higher relative abundance among controls. Biologically significant differences were also found in relative gene, pathway, and enzyme abundance. Biologically significant differences exist in the gut microbial composition of men with prostate cancer compared to benign controls. These differences may play a role in the pathobiology of prostate cancer, and warrant further exploration.

Machine Learning Approaches on Diagnostic Term Encoding With the ICD for Clinical Documentation.

This work focuses on data mining applied to the clinical documentation domain. Diagnostic terms (DTs) are used as keywords to retrieve valuable information from electronic health records. Indeed, they are encoded manually by experts following the International Classification of Diseases (ICD). The goal of this work is to explore the aid of text mining on DT encoding. From the machine learning (ML) perspective, this is a high-dimensional classification task, as it comprises thousands of codes. This work delves into a robust representation of the instances to improve ML results. The proposed system is able to find the right ICD code among more than 1500 possible ICD codes with 92% precision for the main disease (primary class) and 88% for the main disease together with the nonessential modifiers (fully specified class). The methodology employed is simple and portable. According to the experts from public hospitals, the system is very useful in particular for documentation and pharmacosurveillance services. In fact, they reported an accuracy of 91.2% on a small randomly extracted test. Hence, together with this paper, we made the software publicly available in order to help the clinical and research community.

Random-projection Ensemble Classification

SummaryWe introduce a very general method for high dimensional classification, based on careful combination of the results of applying an arbitrary base classifier to random projections of the feature vectors into a lower dimensional space. In one special case that we study in detail, the random projections are divided into disjoint groups, and within each group we select the projection yielding the smallest estimate of the test error. Our random-projection ensemble classifier then aggregates the results of applying the base classifier on the selected projections, with a data-driven voting threshold to determine the final assignment. Our theoretical results elucidate the effect on performance of increasing the number of projections. Moreover, under a boundary condition that is implied by the sufficient dimension reduction assumption, we show that the test excess risk of the random-projection ensemble classifier can be controlled by terms that do not depend on the original data dimension and a term that becomes negligible as the number of projections increases. The classifier is also compared empirically with several other popular high dimensional classifiers via an extensive simulation study, which reveals its excellent finite sample performance.

High Dimensional Classification – An Overview

Objective: A comprehensive overview of high dimensional data classification techniques is presented for the benefit of researchers, scientists and data engineers in both government and private sectors working on large dimensional data. Methods/Statistical Analysis: A systematic approach was followed by studying and reporting the literature review for the years 1969-2016. Findings: The high dimensional data classification is found to be a challenging task as the data will not fit into main memory as required by conventional classification methods. Many of the features would be irrelevant and as the dimensionality increases with limited number of samples any conventional supervised learning algorithm may over fit to noise. The present study reveals the methods to generate artificial samples to increase the size of training data for better classification performance. It is also noted that reducing dimensionality not only reduces the storage space and computational time but increases the understandability. Applications: Text Classification, Email Classification, Pattern Classification, Information Retrieval, Gene Expression Analysis, Health Care Analysis, Predictive Modelling.

A new adaptive L1-norm for optimal descriptor selection of high-dimensional QSAR classification model for anti-hepatitis C virus activity of thiourea derivatives

A high-dimensional quantitative structure–activity relationship (QSAR) classification model typically contains a large number of irrelevant and redundant descriptors. In this paper, a new design of descriptor selection for the QSAR classification model estimation method is proposed by adding a new weight inside L1-norm. The experimental results of classifying the anti-hepatitis C virus activity of thiourea derivatives demonstrate that the proposed descriptor selection method in the QSAR classification model performs effectively and competitively compared with other existing penalized methods in terms of classification performance on both the training and the testing datasets. Moreover, it is noteworthy that the results obtained in terms of stability test and applicability domain provide a robust QSAR classification model. It is evident from the results that the developed QSAR classification model could conceivably be employed for further high-dimensional QSAR classification studies.

Probing for Sparse and Fast Variable Selection with Model-Based Boosting

We present a new variable selection method based on model-based gradient boosting and randomly permuted variables. Model-based boosting is a tool to fit a statistical model while performing variable selection at the same time. A drawback of the fitting lies in the need of multiple model fits on slightly altered data (e.g., cross-validation or bootstrap) to find the optimal number of boosting iterations and prevent overfitting. In our proposed approach, we augment the data set with randomly permuted versions of the true variables, so-called shadow variables, and stop the stepwise fitting as soon as such a variable would be added to the model. This allows variable selection in a single fit of the model without requiring further parameter tuning. We show that our probing approach can compete with state-of-the-art selection methods like stability selection in a high-dimensional classification benchmark and apply it on three gene expression data sets.

Non-asymptotic Analysis of $\ell_1$-norm Support Vector Machines

Support vector machines (SVMs) with the $\ell _{1}$ -penalty became a standard tool in the analysis of high-dimensional classification problems with sparsity constraints in many applications, including bioinformatics and signal processing. We give non-asymptotic results on the performance of $\ell _{1}$ -SVM in identification of sparse classifiers. We show that an $N$ -dimensional $s$ -sparse classification vector can be (with high probability) well approximated from only $O(s\log (N))$ Gaussian trials. We derive similar estimates also in the presence of misclassifications and for the so-called doubly regularized SVM, which combines the $\ell _{1}$ - and the $\ell _{2}$ -penalty. Similar bounds were obtained earlier in the analysis of LASSO and 1-Bit compressed sensing.

Transformer fault diagnosis based on chemical reaction optimization algorithm and relevance vector machine

Power transformer is one of the most important equipment in power system. In order to predict the potential fault of power transformer and identify the fault types correctly, we proposed a transformer fault intelligent diagnosis model based on chemical reaction optimization (CRO) algorithm and relevance vector machine(RVM). RVM is a powerful machine learning method, which can solve nonlinear, high-dimensional classification problems with a limited number of samples. CRO algorithm has well global optimization and simple calculation, so it is suitable to solve parameter optimization problems. In this paper, firstly, a multi-layer RVM classification model was built by binary tree recognition strategy. Secondly, CRO algorithm was adopted to optimize the kernel function parameters which could enhance the performance of RVM classifiers. Compared with IEC three-ratio method and the RVM model, the CRO-RVM model not only overcomes the coding defect problem of IEC three-ratio method, but also has higher classification accuracy than the RVM model. Finally, the new method was applied to analyze a transformer fault case, Its predicted result accord well with the real situation. The research provides a practical method for transformer fault intelligent diagnosis and prediction.

Inter-platform concordance of gene expression data for the prediction of chemical mode of action.

BackgroundIt is interesting to study the consistency of outcomes arising from two genomic platforms: Microarray and RNAseq, which are established on fundamentally different technologies. This topic has been frequently discussed from the prospect of comparing differentially expressed genes (DEGs). In this study, we explore the inter-platform concordance between microarray and RNASeq in their ability to classify samples based on genomic information. We use a set of 7 standard multi-class classifiers and an adaptive ensemble classifier developed around them to predict Chemical Modes of Actions (MOA) of data profiled by microarray and RNASeq platforms from Rat Liver samples exposed to a variety of chemical compounds. We study the concordance between microarray and RNASeq data in various forms, based on classifier’s performance between two platforms.ResultsUsing an ensemble classifier we observe improved prediction performance compared to a set of standard classifiers. We discover a clear concordance between each individual classifier’s performances in two genomic platforms. Additionally, we identify a set of important genes those specifies MOAs, by focusing on their impact on the classification and later we find that some of these top genes have direct associations with the presence of toxic compounds in the liver.ConclusionOverall there appears to be fair amount of concordance between the two platforms as far as classification is concerned. We observe widely different classification performances among individual classifiers, which reflect the unreliability of restricting to a single classifier in the case of high dimensional classification problems.ReviewersAn extended abstract of this research paper was selected for the Camda Satellite Meeting to Ismb 2015 by the Camda Programme Committee. The full research paper then underwent two rounds of Open Peer Review under a responsible Camda Programme Committee member, Lan Hu, PhD (Bio-Rad Laboratories, Digital Biology Center-Cambridge). Open Peer Review was provided by Yiyi Liu and Partha Dey. The Reviewer Comments section shows the full reviews and author responses.

Rocchio algorithm-based particle initialization mechanism for effective PSO classification of high dimensional data

In recent years, there has been a growing interest in applying Particle Swarm Optimization (PSO) to data classification. Nonetheless, due to the curse of dimensionality, the effectiveness of the PSO applied to high dimensional data classification becomes questionable. This paper proposes a novel specialized PSO initialization mechanism, developed specifically for PSO applications to high dimensional data classification. The proposed initialization mechanism is inspired by the center-based sampling theory, which argues that the center of the search space is a promising region for the initialization step in evolutionary algorithms. Furthermore, the proposed initialization mechanism is based on an information retrieval algorithm called Rocchio Algorithm (RA); that identifies the center region of the search space of data classification. To validate the proposed mechanism, RA-based PSO has been applied to a high dimensional classification task in educational data mining. More specifically, RA-based PSO has been applied to classify a dataset of teachers' classroom questions into Bloom's taxonomy cognitive levels. To do so, a dataset of teachers' classroom questions has been collected and annotated manually with Bloom's taxonomy cognitive levels. Pre-processing steps have been applied to convert questions into a representation suitable for classification. Using this dataset, the standard PSO, PSO with generic initialization mechanisms, and RA-based PSO have been experimented and compared. The results show a poor performance of the standard PSO and the PSO with the generic initialization mechanisms, as well as a significant improvement in the performance of RA-based PSO. These results indicate that a proper task-specific PSO initialization mechanism is crucial for effective PSO performance in high dimensional data classification. Furthermore, a comparison between RA-based PSO and pure RA classification provide a quantitative estimation of the role of initialization mechanism and PSO search for the classification of the dataset. On the other hand, the comparison between RA-based PSO approach and three conventional machine learning approaches, experimented on the same dataset confirms the effectiveness of RA-based PSO for high dimensional data classification. Moreover, the comparison between RA-based PSO approach and machine learning approaches, in terms of computational time efficiency, shows that they are comparable in classification time. However, as the learning of PSO is a time-consuming process, its effectiveness is significantly affected if the learning time is a matter.

Sure feature screening for high-dimensional dichotomous classification

The curse of high-dimensionality has emerged in the statistical fields more and more frequently. Many techniques have been developed to address this challenge for classification problems. We propose a novel feature screening procedure for dichotomous response data. This new method can be implemented as easily as t-test marginal screening approach, and the proposed procedure is free of any subexponential tail probability conditions and moment requirement and not restricted in a specific model structure. We prove that our method possesses the sure screening property and also illustrate the effect of screening by Monte Carlo simulation and apply it to a real data example.

Multiple Bayesian discriminant functions for high-dimensional massive data classification

The presence of complex distributions of samples concealed in high-dimensional, massive sample-size data challenges all of the current classification methods for data mining. Samples within a class usually do not uniformly fill a certain (sub)space but are individually concentrated in certain regions of diverse feature subspaces, revealing the class dispersion. Current classifiers applied to such complex data inherently suffer from either high complexity or weak classification ability, due to the imbalance between flexibility and generalization ability of the discriminant functions used by these classifiers. To address this concern, we propose a novel representation of discriminant functions in Bayesian inference, which allows multiple Bayesian decision boundaries per class, each in its individual subspace. For this purpose, we design a learning algorithm that incorporates the naive Bayes and feature weighting approaches into structural risk minimization to learn multiple Bayesian discriminant functions for each class, thus combining the simplicity and effectiveness of naive Bayes and the benefits of feature weighting in handling high-dimensional data. The proposed learning scheme affords a recursive algorithm for exploring class density distribution for Bayesian estimation, and an automated approach for selecting powerful discriminant functions while keeping the complexity of the classifier low. Experimental results on real-world data characterized by millions of samples and features demonstrate the promising performance of our approach.

Feature selection for high-dimensional classification using a competitive swarm optimizer

When solving many machine learning problems such as classification, there exists a large number of input features. However, not all features are relevant for solving the problem, and sometimes, including irrelevant features may deteriorate the learning performance.Please check the edit made in the article title Therefore, it is essential to select the most relevant features, which is known as feature selection. Many feature selection algorithms have been developed, including evolutionary algorithms or particle swarm optimization (PSO) algorithms, to find a subset of the most important features for accomplishing a particular machine learning task. However, the traditional PSO does not perform well for large-scale optimization problems, which degrades the effectiveness of PSO for feature selection when the number of features dramatically increases. In this paper, we propose to use a very recent PSO variant, known as competitive swarm optimizer (CSO) that was dedicated to large-scale optimization, for solving high-dimensional feature selection problems. In addition, the CSO, which was originally developed for continuous optimization, is adapted to perform feature selection that can be considered as a combinatorial optimization problem. An archive technique is also introduced to reduce computational cost. Experiments on six benchmark datasets demonstrate that compared to the canonical PSO-based and a state-of-the-art PSO variant for feature selection, the proposed CSO-based feature selection algorithm not only selects a much smaller number of features, but result in better classification performance as well.

High-dimensional Canonical Forest

ABSTRACTRecently, a new ensemble classification method named Canonical Forest (CF) has been proposed by Chen et al. [Canonical forest. Comput Stat. 2014;29:849–867]. CF has been proven to give consistently good results in many data sets and comparable to other widely used classification ensemble methods. However, CF requires an adopting feature reduction method before classifying high-dimensional data. Here, we extend CF to a high-dimensional classifier by incorporating a random feature subspace algorithm [Ho TK. The random subspace method for constructing decision forests. IEEE Trans Pattern Anal Mach Intell. 1998;20:832–844]. This extended algorithm is called HDCF (high-dimensional CF) as it is specifically designed for high-dimensional data. We conducted an experiment using three data sets – gene imprinting, oestrogen, and leukaemia – to compare the performance of HDCF with several popular and successful classification methods on high-dimensional data sets, including Random Forest [Breiman L. Random forest. Mach Learn. 2001;45:5–32], CERP [Ahn H, et al. Classification by ensembles from random partitions of high-dimensional data. Comput Stat Data Anal. 2007;51:6166–6179], and support vector machines [Vapnik V. The nature of statistical learning theory. New York: Springer; 1995]. Besides the classification accuracy, we also investigated the balance between sensitivity and specificity for all these four classification methods.

The Kendall and Mallows Kernels for Permutations.

We show that the widely used Kendall tau correlation coefficient, and the related Mallows kernel, are positive definite kernels for permutations. They offer computationally attractive alternatives to more complex kernels on the symmetric group to learn from rankings, or learn to rank. We show how to extend these kernels to partial rankings, multivariate rankings and uncertain rankings. Examples are presented on how to formulate typical problems of learning from rankings such that they can be solved with state-of-the-art kernel algorithms. We demonstrate promising results on clustering heterogeneous rank data and high-dimensional classification problems in biomedical applications.