High-dimensional Classification Problems Research Articles

Improved inverse gravity moment term weighting for text classification

Text classification is one of the popular high dimensional classification problems where providing better feature vector representations explicitly improve classification performances. Thus, assigning appropriate weights to features or terms are crucial for obtaining effective feature vector representations. The methods used for weighting terms in text classification are called term weighting schemes. Although there exist some term weighting schemes for text classification, they are not fully effective and researchers still focus on proposing new term weighting schemes. In this study, two novel term weighting schemes namely SQRT_TF-IGMimp and TF-IGMimp derived from standard inverse gravity moment formula are proposed to improve weighting behaviors of existing TF-IGM scheme especially for some extreme cases. The performances of proposed schemes are compared with two standard IGM based schemes and five other state-of-the-art term weighting methods on both unbalanced (Reuters-21578) and balanced (20 Mini Newsgroups and 20 Newsgroups) datasets with KNN, SVM, and NN classifiers. Micro-F1 and macro-F1 are used as success measures. The experiments are conducted with various different feature sizes to examine the effects of the feature size on the success of weighting. The experimental results showed that the proposed SQRT_TF-IGMimp method generally outperformed all schemes including both standard TF-IGM and SQRT_TF-IGM schemes. However, the proposed TF-IGMimp scheme also showed mostly better performance than standard TF-IGM. To demonstrate validity of the proposed weighting scheme having maximum performance, t-test is also used and it can be stated that the performance gains obtained by the proposed SQRT_TF-IGMimp weighting scheme compared to standard SQRT_TF-IGM are statistically significant.

Fuzzy Forests: Extending Random Forest Feature Selection for Correlated, High-Dimensional Data

In this paper we introduce fuzzy forests, a novel machine learning algorithm for ranking the importance of features in high-dimensional classification and regression problems. Fuzzy forests is specifically designed to provide relatively unbiased rankings of variable importance in the presence of highly correlated features, especially when the number of features, p, is much larger than the sample size, n (p n). We introduce our implementation of fuzzy forests in the R package, fuzzyforest. Fuzzy forests works by taking advantage of the network structure between features. First, the features are partitioned into separate modules such that the correlation within modules is high and the correlation between modules is low. The package fuzzyforest allows for easy use of the package WGCNA (weighted gene coexpression network analysis, alternatively known as weighted correlation network analysis) to form modules of features such that the modules are roughly uncorrelated. Then recursive feature elimination random forests (RFE-RFs) are used on each module, separately. From the surviving features, a final group is selected and ranked using one last round of RFE-RFs. This procedure results in a ranked variable importance list whose size is pre-specified by the user. The selected features can then be used to construct a predictive model.

Gaussian Discriminant Analysis for Optimal Delineation of Mild Cognitive Impairment in Alzheimer's Disease.

Over the past few years, several approaches have been proposed to assist in the early diagnosis of Alzheimer's disease (AD) and its prodromal stage of mild cognitive impairment (MCI). Using multimodal biomarkers for this high-dimensional classification problem, the widely used algorithms include Support Vector Machines (SVM), Sparse Representation-based classification (SRC), Deep Belief Networks (DBN) and Random Forest (RF). These widely used algorithms continue to yield unsatisfactory performance for delineating the MCI participants from the cognitively normal control (CN) group. A novel Gaussian discriminant analysis-based algorithm is thus introduced to achieve a more effective and accurate classification performance than the aforementioned state-of-the-art algorithms. This study makes use of magnetic resonance imaging (MRI) data uniquely as input to two separate high-dimensional decision spaces that reflect the structural measures of the two brain hemispheres. The data used include 190 CN, 305 MCI and 133 AD subjects as part of the AD Big Data DREAM Challenge #1. Using 80% data for a 10-fold cross-validation, the proposed algorithm achieved an average F1 score of 95.89% and an accuracy of 96.54% for discriminating AD from CN; and more importantly, an average F1 score of 92.08% and an accuracy of 90.26% for discriminating MCI from CN. Then, a true test was implemented on the remaining 20% held-out test data. For discriminating MCI from CN, an accuracy of 80.61%, a sensitivity of 81.97% and a specificity of 78.38% were obtained. These results show significant improvement over existing algorithms for discriminating the subtle differences between MCI participants and the CN group.

High-Dimensional Probabilistic Fingerprinting in Wireless Sensor Networks Based on a Multivariate Gaussian Mixture Model.

The extensive deployment of wireless infrastructure provides a low-cost way to track mobile users in indoor environment. This paper demonstrates a prototype model of an accurate and reliable room location awareness system in a real public environment in which three typical problems arise. Firstly, a massive number of access points (APs) can be sensed leading to a high-dimensional classification problem. Secondly, heterogeneous devices record different received signal strength (RSS) levels because of the variations in chip-set and antenna attenuation. Thirdly, APs are not necessarily visible in every scanning cycle leading to missing data issue. This paper presents a probabilistic Wi-Fi fingerprinting method in a hidden Markov model (HMM) framework for mobile user tracking. To account for spatial correlation of the signal strengths from multiple APs, a Multivariate Gaussian Mixture Model (MVGMM) was fitted to model the probability distribution of RSS measurements in each cell. Furthermore, the unseen property of invisible AP was investigated in this research, and demonstrated the efficiency as a beneficial information to differentiate between cells. The proposed system is able to achieve comparable localisation performance. Filed test results achieve a reliable 97% localisation room level accuracy of multiple mobile users in a real university campus Wi-Fi network.

Sparse feature kernel multi-criteria linear programming classifier

Abstract For a classification problem, the nonlinearly separable case can be transformed into a linearly separable one by using the kernel trick, However, introducing the single kernel function to a classifier can cause the difficulty of feature selection and it reduces the model interpretability, which is very important for some practical applications. Moreover, multi-kernel learning used for improving the predictive accuracy can lead to the poor efficiency and interpretability, especially for large-scale high-dimensional classification problems. In this paper, we proposed a novel sparse feature kernel multi-criteria linear programming classifier (SFK-MCLPC), the two-stage classifier model employs new row and column kernel matrices of different feature kernels to iteratively solve two associative linear programming problems so as to identify important features from data in the process of classification. Based on the ten real-world data sets, the experimental results and comparison with multi-criteria linear and quadratic programming classifiers (MCLPC, MCQPC), support vector classifier (SVC), and multiple kernel learning SVC (MKL-SVC) have shown that the proposed SFK-MCLPC can enhance the predictive accuracy of different classes, the efficiency of classification, the interpretability of the pattern analysis, and the generalization ability of predicting the classes of unseen instances by selecting the relevant and important features.

Resource Optimization and Fault Detection Algorithms for Cloud Computing Platforms based on SVM and Resource Reserve Strategy

Efficient operation of cloud computing platforms depends on the optimised virtual resources and faster fault diagnosis system of the virtual machines. This paper proposes an algorithm by introducing a virtual machines based on elastic reservation mechanism, which can improve the availability of cloud resources through the demand analysis taking the help of support vector machines which has advantages resolving nonlinear and high dimensional classification problems. Secondly it adopts the anomaly detection algorithm based on support vector machines for failure analysis. In addition, the dimensionality problem can be sorted out by means of principal component analysis (PCA) algorithm and a kernel function used for distance measurement. It establishes the topological structure for the image set of feature space with Delaunay triangulation and analyses the relationship between kernel parameter and regulator, in order to build an effective model.

Resource optimisation and fault detection algorithms for cloud computing platforms based on SVM and resource reserve strategy

Efficient operation of cloud computing platforms depends on the optimised virtual resources and faster fault diagnosis system of the virtual machines. This paper proposes an algorithm by introducing a virtual machines based on elastic reservation mechanism, which can improve the availability of cloud resources through the demand analysis taking the help of support vector machines which has advantages resolving nonlinear and high dimensional classification problems. Secondly it adopts the anomaly detection algorithm based on support vector machines for failure analysis. In addition, the dimensionality problem can be sorted out by means of principal component analysis (PCA) algorithm and a kernel function used for distance measurement. It establishes the topological structure for the image set of feature space with Delaunay triangulation and analyses the relationship between kernel parameter and regulator, in order to build an effective model.

Transformer fault diagnosis based on chemical reaction optimization algorithm and relevance vector machine

Power transformer is one of the most important equipment in power system. In order to predict the potential fault of power transformer and identify the fault types correctly, we proposed a transformer fault intelligent diagnosis model based on chemical reaction optimization (CRO) algorithm and relevance vector machine(RVM). RVM is a powerful machine learning method, which can solve nonlinear, high-dimensional classification problems with a limited number of samples. CRO algorithm has well global optimization and simple calculation, so it is suitable to solve parameter optimization problems. In this paper, firstly, a multi-layer RVM classification model was built by binary tree recognition strategy. Secondly, CRO algorithm was adopted to optimize the kernel function parameters which could enhance the performance of RVM classifiers. Compared with IEC three-ratio method and the RVM model, the CRO-RVM model not only overcomes the coding defect problem of IEC three-ratio method, but also has higher classification accuracy than the RVM model. Finally, the new method was applied to analyze a transformer fault case, Its predicted result accord well with the real situation. The research provides a practical method for transformer fault intelligent diagnosis and prediction.

Non-asymptotic Analysis of $\ell_1$-norm Support Vector Machines

Support vector machines (SVMs) with the $\ell _{1}$ -penalty became a standard tool in the analysis of high-dimensional classification problems with sparsity constraints in many applications, including bioinformatics and signal processing. We give non-asymptotic results on the performance of $\ell _{1}$ -SVM in identification of sparse classifiers. We show that an $N$ -dimensional $s$ -sparse classification vector can be (with high probability) well approximated from only $O(s\log (N))$ Gaussian trials. We derive similar estimates also in the presence of misclassifications and for the so-called doubly regularized SVM, which combines the $\ell _{1}$ - and the $\ell _{2}$ -penalty. Similar bounds were obtained earlier in the analysis of LASSO and 1-Bit compressed sensing.

The Kendall and Mallows Kernels for Permutations.

We show that the widely used Kendall tau correlation coefficient, and the related Mallows kernel, are positive definite kernels for permutations. They offer computationally attractive alternatives to more complex kernels on the symmetric group to learn from rankings, or learn to rank. We show how to extend these kernels to partial rankings, multivariate rankings and uncertain rankings. Examples are presented on how to formulate typical problems of learning from rankings such that they can be solved with state-of-the-art kernel algorithms. We demonstrate promising results on clustering heterogeneous rank data and high-dimensional classification problems in biomedical applications.

A Novel Active Learning Method Using SVM for Text Classification

Support vector machines (SVMs) are a popular class of supervised learning algorithms, and are particularly applicable to large and high-dimensional classification problems. Like most machine learning methods for data classification and information retrieval, they require manually labeled data samples in the training stage. However, manual labeling is a time consuming and errorprone task. One possible solution to this issue is to exploit the large number of unlabeled samples that are easily accessible via the internet. This paper presents a novel active learning method for text categorization. The main objective of active learning is to reduce the labeling effort, without compromising the accuracy of classification, by intelligently selecting which samples should be labeled. The proposed method selects a batch of informative samples using the posterior probabilities provided by a set of multi-class SVM classifiers, and these samples are then manually labeled by an expert. Experimental results indicate that the proposed active learning method significantly reduces the labeling effort, while simultaneously enhancing the classification accuracy.

A Gaussian-based kernel Fisher discriminant analysis for electronic nose data and applications in spirit and vinegar classification

Fisher discriminant analysis (FDA) is a very useful pattern recognition technique widely used in electronic nose system (e-nose). However, due to its linear characteristic, the classification problems of multi-class and high-dimensional e-nose data cannot be handled effectively. Therefore, a Gaussian-based kernel FDA (KFDA) method is proposed to solve multi-class and high-dimensional classification problems of complex samples such as food classification using e-nose. The key point of the method is how to determine the Gaussian kernel parameter. Firstly, according to distance discriminant analysis viewpoint, a desired kernel matrix adapted to Gaussian kernel function can be given successfully. Secondly, an evaluation function based on Euclidean distance is established for measuring the degree of approximation between actual kernel matrix containing an unknown Gaussian kernel parameter and the desired kernel matrix so as to get an optimal solution of the parameter, and then the actual kernel matrix can be definitely determined. Finally, the principal component analysis (PCA) for the actual kernel matrix is carried out. Meanwhile, FDA for the principal component matrix generated by PCA is also implemented in succession, and the KFDA is completed. Six kinds of Chinese spirit and six kinds of Chinese vinegar samples as two classification applications were respectively carried out accurately with the KFDA method; and the KFDA method is tested to be very simple and effective. The KFDA method may be promising for complex samples classification dataset of e-nose.

Optimization Models for Feature Selection of Decomposed Nearest Neighbor

The traditional ${k}$ -nearest neighbor (KNN) method classifies an object by the majority vote of its neighbors, and only one parameter ${k}$ is to be optimized. We propose a decomposed version of the KKN (DKNN) method, which classifies an unseen object based on the distances to the centroids of the KNNs. DKNN has a training process to learn the local-optima-free distance metric by solving a convex optimization problem. The optimization problem not only learns a metric that minimizes classification errors and maximizes the margin between intraclass and interclass distances. In addition, it also selects important features and removes irrelevant features when L1 regularization is incorporated in the optimization model. The feature selection component is extremely useful for high-dimensional classification problems. Tested on public datasets, the proposed DKNN is competitive and often outperforms traditional machine learning algorithms.

Composite large margin classifiers with latent subclasses for heterogeneous biomedical data.

High dimensional classification problems are prevalent in a wide range of modern scientific applications. Despite a large number of candidate classification techniques available to use, practitioners often face a dilemma of choosing between linear and general nonlinear classifiers. Specifically, simple linear classifiers have good interpretability, but may have limitations in handling data with complex structures. In contrast, general nonlinear classifiers are more flexible, but may lose interpretability and have higher tendency for overfitting. In this paper, we consider data with potential latent subgroups in the classes of interest. We propose a new method, namely the Composite Large Margin Classifier (CLM), to address the issue of classification with latent subclasses. The CLM aims to find three linear functions simultaneously: one linear function to split the data into two parts, with each part being classified by a different linear classifier. Our method has comparable prediction accuracy to a general nonlinear classifier, and it maintains the interpretability of traditional linear classifiers. We demonstrate the competitive performance of the CLM through comparisons with several existing linear and nonlinear classifiers by Monte Carlo experiments. Analysis of the Alzheimer's disease classification problem using CLM not only provides a lower classification error in discriminating cases and controls, but also identifies subclasses in controls that are more likely to develop the disease in the future.

Online Sequential Projection Vector Machine with Adaptive Data Mean Update.

We propose a simple online learning algorithm especial for high-dimensional data. The algorithm is referred to as online sequential projection vector machine (OSPVM) which derives from projection vector machine and can learn from data in one-by-one or chunk-by-chunk mode. In OSPVM, data centering, dimension reduction, and neural network training are integrated seamlessly. In particular, the model parameters including (1) the projection vectors for dimension reduction, (2) the input weights, biases, and output weights, and (3) the number of hidden nodes can be updated simultaneously. Moreover, only one parameter, the number of hidden nodes, needs to be determined manually, and this makes it easy for use in real applications. Performance comparison was made on various high-dimensional classification problems for OSPVM against other fast online algorithms including budgeted stochastic gradient descent (BSGD) approach, adaptive multihyperplane machine (AMM), primal estimated subgradient solver (Pegasos), online sequential extreme learning machine (OSELM), and SVD + OSELM (feature selection based on SVD is performed before OSELM). The results obtained demonstrated the superior generalization performance and efficiency of the OSPVM.

Genetic programming for feature construction and selection in classification on high-dimensional data

Classification on high-dimensional data with thousands to tens of thousands of dimensions is a challenging task due to the high dimensionality and the quality of the feature set. The problem can be addressed by using feature selection to choose only informative features or feature construction to create new high-level features. Genetic programming (GP) using a tree-based representation can be used for both feature construction and implicit feature selection. This work presents a comprehensive study to investigate the use of GP for feature construction and selection on high-dimensional classification problems. Different combinations of the constructed and/or selected features are tested and compared on seven high-dimensional gene expression problems, and different classification algorithms are used to evaluate their performance. The results show that the constructed and/or selected feature sets can significantly reduce the dimensionality and maintain or even increase the classification accuracy in most cases. The cases with overfitting occurred are analysed via the distribution of features. Further analysis is also performed to show why the constructed feature can achieve promising classification performance.

A novel feature selection scheme for high-dimensional data sets: four-Staged Feature Selection

Classification of high-dimensional data set is a big challenge for statistical learning and data mining algorithms. To effectively apply classification methods to high-dimensional data sets, feature selection is an indispensable pre-processing step of learning process. In this study, we consider the problem of constructing an effective feature selection and classification scheme for data set which has a small number of sample size with a large number of features. A novel feature selection approach, named four-Staged Feature Selection, has been proposed to overcome high-dimensional data classification problem by selecting informative features. The proposed method first selects candidate features with number of filtering methods which are based on different metrics, and then it applies semi-wrapper, union and voting stages, respectively, to obtain final feature subsets. Several statistical learning and data mining methods have been carried out to verify the efficiency of the selected features. In order to test the adequacy of the proposed method, 10 different microarray data sets are employed due to their high number of features and small sample size.

A Comparison of Machine Learning Methods in a High-Dimensional Classification Problem

Abstract Background: Large-dimensional data modelling often relies on variable reduction methods in the pre-processing and in the post-processing stage. However, such a reduction usually provides less information and yields a lower accuracy of the model. Objectives: The aim of this paper is to assess the high-dimensional classification problem of recognizing entrepreneurial intentions of students by machine learning methods. Methods/Approach: Four methods were tested: artificial neural networks, CART classification trees, support vector machines, and k-nearest neighbour on the same dataset in order to compare their efficiency in the sense of classification accuracy. The performance of each method was compared on ten subsamples in a 10-fold cross-validation procedure in order to assess computing sensitivity and specificity of each model. Results: The artificial neural network model based on multilayer perceptron yielded a higher classification rate than the models produced by other methods. The pairwise t-test showed a statistical significance between the artificial neural network and the k-nearest neighbour model, while the difference among other methods was not statistically significant. Conclusions: Tested machine learning methods are able to learn fast and achieve high classification accuracy. However, further advancement can be assured by testing a few additional methodological refinements in machine learning methods.

Imbalanced Data Classification Using Cost-Sensitive Support Vector Machine Based on Information Entropy

As a supervised classification algorithm, Support Vector Machine (SVM) has an excellent ability in solving small samples, nonlinear and high dimensional classification problems. However, SVM is inefficient for imbalanced data sets classification. Therefore, a cost sensitive SVM (CSSVM) should be designed for imbalanced data sets classification. This paper proposes a method which constructed CSSVM based on information entropy, and in this method the information entropies of different classes of data set are used to determine the values of penalty factor of CSSVM.

Noise-free representation based classification and face recognition experiments

The representation based classification has achieved promising performance in high-dimensional pattern classification problems. As we know, in real-world applications the samples are usually corrupted by noise. However, representation based classification can take only noise in the test sample into account and is not able to deal with noise in the training sample, which causes side-effect on the classification result. In order to make the representation based classification more suitable for real-world applications such as face recognition, we propose a new representation based classification method in this paper. This method can effectively and simultaneously reduce noise in the test and training samples. Moreover, the proposed method can reduce noise in both the original and virtual training samples and then exploits them to determine the label of the test sample. The virtual training sample is generated from the original face image and shows possible variation of the face in scale, facial pose and expression. The experimental results show that the proposed method performs very well in face recognition.