High-dimensional Classification Research Articles

Probability distribution function-based classification of structural MRI for the detection of Alzheimer’s disease

High-dimensional classification methods have been a major target of machine learning for the automatic classification of patients who suffer from Alzheimer’s disease (AD). One major issue of automatic classification is the feature-selection method from high-dimensional data. In this paper, a novel approach for statistical feature reduction and selection in high-dimensional magnetic resonance imaging (MRI) data based on the probability distribution function (PDF) is introduced. To develop an automatic computer-aided diagnosis (CAD) technique, this research explores the statistical patterns extracted from structural MRI (sMRI) data on four systematic levels. First, global and local differences of gray matter in patients with AD compared to healthy controls (HCs) using the voxel-based morphometric (VBM) technique with 3-Tesla 3D T1-weighted MRI are investigated. Second, feature extraction based on the voxel clusters detected by VBM on sMRI and voxel values as volume of interest (VOI) is used. Third, a novel statistical feature-selection process is employed, utilizing the PDF of the VOI to represent statistical patterns of the respective high-dimensional sMRI sample. Finally, the proposed feature-selection method for early detection of AD with support vector machine (SVM) classifiers compared to other standard feature selection methods, such as partial least squares (PLS) techniques, is assessed. The performance of the proposed technique is evaluated using 130 AD and 130 HC MRI data from the ADNI dataset with 10-fold cross validation11Data used in this article were obtained from the Alzheimer׳s Disease Neuroimaging Initiative (ADNI) database (www.loni.ucla.edu/ADNI). ADNI investigators other than those listed above contributed to study design, implementation or data provision but did not participate in the analyses or writing of this report. The complete listing of ADNI investigators is available at http://www.loni.ucla.edu/ADNI/Data/ADNI_Authorship_List.pdf.. The results show that the PDF-based feature selection approach is a reliable technique that is highly competitive with respect to the state-of-the-art techniques in classifying AD from high-dimensional sMRI samples.

Innovated interaction screening for high-dimensional nonlinear classification

This paper is concerned with the problems of interaction screening and nonlinear classification in a high-dimensional setting. We propose a two-step procedure, IIS-SQDA, where in the first step an innovated interaction screening (IIS) approach based on transforming the original $p$-dimensional feature vector is proposed, and in the second step a sparse quadratic discriminant analysis (SQDA) is proposed for further selecting important interactions and main effects and simultaneously conducting classification. Our IIS approach screens important interactions by examining only $p$ features instead of all two-way interactions of order $O(p^2)$. Our theory shows that the proposed method enjoys sure screening property in interaction selection in the high-dimensional setting of $p$ growing exponentially with the sample size. In the selection and classification step, we establish a sparse inequality on the estimated coefficient vector for QDA and prove that the classification error of our procedure can be upper-bounded by the oracle classification error plus some smaller order term. Extensive simulation studies and real data analysis show that our proposal compares favorably with existing methods in interaction selection and high-dimensional classification.

On random hyper-class random forest for visual classification

Random forest is an effective ensemble classifier. It has shown to be efficient with good generalization performance thanks to the randomness in training. However, for visual classification problems, where high dimensions and large number of classes exist, the impurity measurement in training the tree nodes split function not only neglects the strong conditional dependencies among visual attributes, but also leads to rather weak base classifiers, which may not reflect enough discriminative capability. Addressing this, we develop a random hyper-class random forest (RHC-RF) for visual classification tasks in this paper. During training each tree node, the entire class space is randomly partitioned into two hyper-classes and a weak learner targeting at separating the two hyper-classes is trained, where a linear classifier is learned with multiple feature attributes. This novel training scheme well preserves the randomness by random partition of the label space, and at the same time, the objective of training each node directly targets at discriminating classes in a divide-and-conquer-like manner. The proposed method is expected to benefit from the improved discrimination for visual classification compared with conventional random forests while preserving good generalization. Extensive experiments on various multi-class and high-dimensional visual classification tasks (including scene and object classification, image-based food identification, handwritten digit recognition and face recognition) demonstrate the superior accuracy, as well as its compactness and robustness achieved by the proposed method.

Applying Penalized Binary Logistic Regression with Correlation Based Elastic Net for Variables Selection

Reduction of the high dimensional classification using penalized logistic regression is one of the challenges in applying binary logistic regression. The applied penalized method, correlation based elastic penalty (CBEP), was used to overcome the limitation of LASSO and elastic net in variable selection when there are perfect correlation among explanatory variables. The performance of the CBEP was demonstrated through its application in analyzing two well-known high dimensional binary classification data sets. The CBEP provided superior classification performance and variable selection compared with other existing penalized methods. It is a reliable penalized method in binary logistic regression.

Kernel methods for heterogeneous feature selection

This paper introduces two feature selection methods to deal with heterogeneous data that include continuous and categorical variables. We propose to plug a dedicated kernel that handles both kinds of variables into a Recursive Feature Elimination procedure using either a non-linear SVM or Multiple Kernel Learning. These methods are shown to offer state-of-the-art performances on a variety of high-dimensional classification tasks.

Utility of the CKD273 peptide classifier in predicting chronic kidney disease progression.

Chronic kidney disease (CKD) is a growing public health concern, afflicting approximately one-tenth of adults in developed countries. However, the clinical need for an accurate test, such as a biomarker and/or peptide classifier, for predicting CKD progression and related adverse outcomes remains unaddressed. Recently, a proteomics approach based on capillary electrophoresis-mass spectrometry was employed to develop a urinary peptide-based high-dimensional classifier, namely CKD273, for predicting CKD progression. The study aims to critically appraise the evidence level of the CKD273 classifier's utility in predicting CKD progression, according to the Oxford Evidence-Based Medicine (EBM) and Strength of Recommendation Taxonomy (SORT) guidelines. Eligible studies were identified by a literature search of MEDLINE and Web of Science Expanded Core Collection databases. Limitations were set to prospective cohort studies evaluating the classifier's accuracy in predicting CKD progression. Data extraction was undertaken according to a predefined protocol by two independent reviewers. The EBM and SORT guidelines were applied to appraise the CKD273 classifier's utility for predicting CKD progression. The query search results rendered four prospective cohort studies. The classifier performed independently of age, gender and the type of urine storage containers used. The classifier predicted the development of micro- or macroalbuminuria and rapid decline (i.e. >-5% annual decrease) in the estimated glomerular filtration rate. One study assessed the association of the classifier with end-stage renal disease and death but did not take confounding factors into account. The CKD273 classifier attained high evidence levels according to the EBM (score range 1b), supporting its utility for predicting CKD progression. However, lower scores were attained when the studies were scored according the SORT guidelines (score ranges 1-4). Initial promising evidence supports the CKD273 classifier's utility in predicting CKD progression. The classifier's applicability should be corroborated with additional evidence arising from inception cohort studies assessing patient-oriented outcomes, which demonstrate its added value beyond currently available clinical risk predictors, as well as its cost-effectiveness in clinical practice.

Similarity of feature selection methods: An empirical study across data intensive classification tasks

In the past two decades, the dimensionality of datasets involved in machine learning and data mining applications has increased explosively. Therefore, feature selection has become a necessary step to make the analysis more manageable and to extract useful knowledge about a given domain. A large variety of feature selection techniques are available in literature, and their comparative analysis is a very difficult task. So far, few studies have investigated, from a theoretical and/or experimental point of view, the degree of similarity/dissimilarity among the available techniques, namely the extent to which they tend to produce similar results within specific application contexts. This kind of similarity analysis is of crucial importance when two or more methods are combined in an ensemble fashion: indeed the ensemble paradigm is beneficial only if the involved methods are capable of giving different and complementary representations of the considered domain. This paper gives a contribution in this direction by proposing an empirical approach to evaluate the degree of consistency among the outputs of different selection algorithms in the context of high dimensional classification tasks. Leveraging on a proper similarity index, we systematically compared the feature subsets selected by eight popular selection methods, representatives of different selection approaches, and derived a similarity trend for feature subsets of increasing size. Through an extensive experimentation involving sixteen datasets from three challenging domains (Internet advertisements, text categorization and micro-array data classification), we obtained useful insight into the pattern of agreement of the considered methods. In particular, our results revealed how multivariate selection approaches systematically produce feature subsets that overlap to a small extent with those selected by the other methods.

On two simple and effective procedures for high dimensional classification of general populations

In this paper, we generalize two criteria, the determinant-based and trace-based criteria proposed by Saranadasa (1993), to general populations for high dimensional classification. These two criteria compare some distances between a new observation and several different known groups. The determinant-based criterion performs well for correlated variables by integrating the covariance structure and is competitive to many other existing rules. The criterion however requires the measurement dimension be smaller than the sample size. The trace-based criterion in contrast, is an independence rule and effective in the "large dimension-small sample size" scenario. An appealing property of these two criteria is that their implementation is straightforward and there is no need for preliminary variable selection or use of turning parameters. Their asymptotic misclassification probabilities are derived using the theory of large dimensional random matrices. Their competitive performances are illustrated by intensive Monte Carlo experiments and a real data analysis.

Sparse semiparametric discriminant analysis

In recent years, a considerable amount of work has been devoted to generalizing linear discriminant analysis to overcome its incompetence for high-dimensional classification (Witten and Tibshirani, 2011, Cai and Liu, 2011, Mai et al., 2012 and Fan et al., 2012). In this paper, we develop high-dimensional sparse semiparametric discriminant analysis (SSDA) that generalizes the normal-theory discriminant analysis in two ways: it relaxes the Gaussian assumptions and can handle ultra-high dimensional classification problems. If the underlying Bayes rule is sparse, SSDA can estimate the Bayes rule and select the true features simultaneously with overwhelming probability, as long as the logarithm of dimension grows slower than the cube root of sample size. Simulated and real examples are used to demonstrate the finite sample performance of SSDA. At the core of the theory is a new exponential concentration bound for semiparametric Gaussian copulas, which is of independent interest.

High-Dimensional Data Classification Based on Smooth Support Vector Machines

Abstract Classification on high dimensional data arises in many statistical and data mining studies. Support vector machines (SVM) are one of data mining technique which has been extensively studied and have shown remarkable success in many applications. Many researches developed SVM to increase performance such as smooth support vector machine (SSVM). In this study variants of SSVM (spline SSVM, piecewise polynomial SSVM) are proposed for high-dimensional classification. Theoretical results demonstrate piecewise polynomial SSVM has better classification. And numerical comparison results show that the piecewise polynomial SSVM slightly better performance than spline SSVM.

Flexible high-dimensional classification machines and their asymptotic properties

Classification is an important topic in statistics and machine learning with great potential in many real applications. In this paper, we investigate two popular large-margin classification methods...

Target detection algorithm for airborne thermal hyperspectral data

Abstract. Airborne hyperspectral imaging is constantly being used for classification purpose. But airborne thermal hyperspectral image usually is a challenge for conventional classification approaches. The Telops Hyper-Cam sensor is an interferometer-based imaging system that helps in the spatial and spectral analysis of targets utilizing a single sensor. It is based on the technology of Fourier-transform which yields high spectral resolution and enables high accuracy radiometric calibration. The Hypercam instrument has 84 spectral bands in the 868 cm−1 to 1280 cm−1 region (7.8 μm to 11.5 μm), at a spectral resolution of 6 cm−1 (full-width-half-maximum) for LWIR (long wave infrared) range. Due to the Hughes effect, only a few classifiers are able to handle high dimensional classification task. MNF (Minimum Noise Fraction) rotation is a data dimensionality reducing approach to segregate noise in the data. In this, the component selection of minimum noise fraction (MNF) rotation transformation was analyzed in terms of classification accuracy using constrained energy minimization (CEM) algorithm as a classifier for Airborne thermal hyperspectral image and for the combination of airborne LWIR hyperspectral image and color digital photograph. On comparing the accuracy of all the classified images for airborne LWIR hyperspectral image and combination of Airborne LWIR hyperspectral image with colored digital photograph, it was found that accuracy was highest for MNF component equal to twenty. The accuracy increased by using the combination of airborne LWIR hyperspectral image with colored digital photograph instead of using LWIR data alone.

Sparsifying the Fisher Linear Discriminant by Rotation

Many high dimensional classification techniques have been proposed in the literature based on sparse linear discriminant analysis (LDA). To efficiently use them, sparsity of linear classifiers is a prerequisite. However, this might not be readily available in many applications, and rotations of data are required to create the needed sparsity. In this paper, we propose a family of rotations to create the required sparsity. The basic idea is to use the principal components of the sample covariance matrix of the pooled samples and its variants to rotate the data first and to then apply an existing high dimensional classifier. This rotate-and-solve procedure can be combined with any existing classifiers, and is robust against the sparsity level of the true model. We show that these rotations do create the sparsity needed for high dimensional classifications and provide theoretical understanding why such a rotation works empirically. The effectiveness of the proposed method is demonstrated by a number of simulated and real data examples, and the improvements of our method over some popular high dimensional classification rules are clearly shown.

A Comparison of Machine Learning Methods in a High-Dimensional Classification Problem

Abstract Background: Large-dimensional data modelling often relies on variable reduction methods in the pre-processing and in the post-processing stage. However, such a reduction usually provides less information and yields a lower accuracy of the model. Objectives: The aim of this paper is to assess the high-dimensional classification problem of recognizing entrepreneurial intentions of students by machine learning methods. Methods/Approach: Four methods were tested: artificial neural networks, CART classification trees, support vector machines, and k-nearest neighbour on the same dataset in order to compare their efficiency in the sense of classification accuracy. The performance of each method was compared on ten subsamples in a 10-fold cross-validation procedure in order to assess computing sensitivity and specificity of each model. Results: The artificial neural network model based on multilayer perceptron yielded a higher classification rate than the models produced by other methods. The pairwise t-test showed a statistical significance between the artificial neural network and the k-nearest neighbour model, while the difference among other methods was not statistically significant. Conclusions: Tested machine learning methods are able to learn fast and achieve high classification accuracy. However, further advancement can be assured by testing a few additional methodological refinements in machine learning methods.

Stein’s method in high dimensional classification and applications

In the context of classification, it is a common phenomenon that high-dimensional data such as micro-array data consist of only a few informative components. If one uses standard statistical modeling and estimation procedures with entire information, it tends to overfit the data due to noise information. Therefore, some regularization conditions are required to select important information. A class of regularization methods is proposed through various shrinkage estimators using Stein’s identity. Since hard thresholding does not satisfy the condition of Stein’s identity, the proposed methods consider linear classifiers with soft, firm and SCAD thresholdings incorporating Stein’s identity and show some asymptotic properties. Simulation studies and applications to three different micro array data sets show that the proposed methods work well. Also the proposed methods are compared with some existing methods.

Brain Emotional Learning for Classification Problem

Emotional learning is new tool in the field of machine learning that the inspired from limbic system. The various models of emotional learning (BEL) have been successfully utilized in many learning problems. For example, control applications and prediction problems. In this paper a new architecture based on a brain emotional learning model that can be used in classification problem (BELC). This model is suitable for high dimensional classification applications. To evaluate the proposed method have been compare it with the Multilayer Perceptron (MLP), K-Nearest Neighbor (KNN), Naive Bayes classifier and Brain Emotional Learning-Based Pattern Recognizer (BELPR) methods. The obtained results show the effectiveness and efficiency of the proposed method for classification problems.

Imbalanced Data Classification Using Cost-Sensitive Support Vector Machine Based on Information Entropy

As a supervised classification algorithm, Support Vector Machine (SVM) has an excellent ability in solving small samples, nonlinear and high dimensional classification problems. However, SVM is inefficient for imbalanced data sets classification. Therefore, a cost sensitive SVM (CSSVM) should be designed for imbalanced data sets classification. This paper proposes a method which constructed CSSVM based on information entropy, and in this method the information entropies of different classes of data set are used to determine the values of penalty factor of CSSVM.

Noise-free representation based classification and face recognition experiments

The representation based classification has achieved promising performance in high-dimensional pattern classification problems. As we know, in real-world applications the samples are usually corrupted by noise. However, representation based classification can take only noise in the test sample into account and is not able to deal with noise in the training sample, which causes side-effect on the classification result. In order to make the representation based classification more suitable for real-world applications such as face recognition, we propose a new representation based classification method in this paper. This method can effectively and simultaneously reduce noise in the test and training samples. Moreover, the proposed method can reduce noise in both the original and virtual training samples and then exploits them to determine the label of the test sample. The virtual training sample is generated from the original face image and shows possible variation of the face in scale, facial pose and expression. The experimental results show that the proposed method performs very well in face recognition.

Content based Structural Recognition for Image Classification using PSO Technique and SVM

The issue of SVMs parameter optimization with particle swarm optimization (pso) provide the optimum solution. This new classification approach may be an efficient alternative, in existing paradigms. PSO technique work with high dimensional datasets and mixed attribute data. The structure of the image is recognized through PSO technique which provide optimized parameter for SVM. This approach determines the performance of image classification after structural recognition based on content of image and comparing the obtained results with those reported for various other classification approaches. PSO-SVM technique can be applied mixed-attribute, hyperspectral data, hyperdimension spaces & problem description spaces and it can also be a competitive alternative to well established classification techniques. The optimized process of data reduces the unclassified region of support vector machine and improves the performance of image classification. The feature of region of image is classified by PSO-SVM technique in inside the image. Cassified features are increase recogniztion ratio because the feature of image is optimized. General Terms Pattern Recognition, high dimensional image classification et. al.

An Effective Ensemble-based Classification Algorithm for High-Dimensional Steganalysis

Recently, ensemble learning algorithms are proposed to address the challenges of high dimensional classification for steganalysis caused by the curse of dimensionality and obtain superior performance. In this paper, we extend the state-of-the-art steganalysis tool developed by Kodovsky and Fridrich: the Kodovsky's ensemble classifier and propose a novel method, called CS- RS for high-dimensional steganalysis. Different from the Kodovsky's ensemble classifier which selects features in a completely random way, the proposed CS-RS modifies the generation method of feature subspaces. Firstly, our method employs the chi-square statistic (CS) to measure the weight of each feature in the original feature space and sorts features according to weights. Then the sorted original feature space is partitioned into two parts according to a given dividing point: high correlation part and low correlation part. Finally, the feature subset is formed by selecting features randomly in each part according to the given sampling rate. Experiments with the steganographic algorithms HUGO demonstrate that the proposed CS-RS using the FLD classifier offers training complexity comparable to the Kodovsky's classifier and significantly increases the performance of the Kodovsky's classifier in less than 1000-dimensional feature subspaces, gaining 1.2% on the optimal result. In addition, the proposed algorithm outperforms Bagging and AdaBoost and can offer accuracy comparable to L-SVM.