High-dimensional Classification Research Articles

Supervised non-negative matrix factorization methods for MALDI imaging applications.

MotivationNon-negative matrix factorization (NMF) is a common tool for obtaining low-rank approximations of non-negative data matrices and has been widely used in machine learning, e.g. for supporting feature extraction in high-dimensional classification tasks. In its classical form, NMF is an unsupervised method, i.e. the class labels of the training data are not used when computing the NMF. However, incorporating the classification labels into the NMF algorithms allows to specifically guide them toward the extraction of data patterns relevant for discriminating the respective classes. This approach is particularly suited for the analysis of mass spectrometry imaging (MSI) data in clinical applications, such as tumor typing and classification, which are among the most challenging tasks in pathology. Thus, we investigate algorithms for extracting tumor-specific spectral patterns from MSI data by NMF methods.ResultsIn this article, we incorporate a priori class labels into the NMF cost functional by adding appropriate supervised penalty terms. Numerical experiments on a MALDI imaging dataset confirm that the novel supervised NMF methods lead to significantly better classification accuracy and stability as compared with other standard approaches.Availability and implementaton https://gitlab.informatik.uni-bremen.de/digipath/Supervised_NMF_Methods_for_MALDI.git Supplementary information Supplementary data are available at Bioinformatics online.

Density-sensitive Robust Fuzzy Kernel Principal Component Analysis technique

In order to deal with the sensitivity of traditional kernel principal component analysis (KPCA) to the outliers and high computational complexity of the other existing robust KPCAs, a novel density-sensitive Robust Fuzzy Kernel Principal Component Analysis (DRF-KPCA) is proposed in this paper. First, the initial membership degree of the sample is determined by introducing the relative density. Second, the membership degree is formulated based on reconstruction error and updated by the optimal gradient descent approach, which can effectively solve the problem of the principal component skewing caused by the sensitivity of KPCA to the outliers. Ultimately, the computational complexity and running time of DRF-KPCA are significantly reduced by simplifying the calculation formula of the reconstruction error. In the experiments, as compared with KPCA and other modified approaches on both the datasets with outliers and the datasets without outliers, DRF-KPCA is evaluated to effectively eliminate the impacts of the outliers on the performance with low computational complexity. In addition, the influence of parameters on the performance of DRF-KPCA is also analyzed in detail and the suggestions of determination on the optimal coefficients are given. Eventually, the comparison results with the-state-of-art techniques on UCI benchmark datasets and other high-dimensional classification datasets demonstrate that the performance of DRF-KPCA is significantly improved.

Hyperspectral Classification via Superpixel Kernel Learning-Based Low Rank Representation

High dimensional image classification is a fundamental technique for information retrieval from hyperspectral remote sensing data. However, data quality is readily affected by the atmosphere and noise in the imaging process, which makes it difficult to achieve good classification performance. In this paper, multiple kernel learning-based low rank representation at superpixel level (Sp_MKL_LRR) is proposed to improve the classification accuracy for hyperspectral images. Superpixels are generated first from the hyperspectral image to reduce noise effect and form homogeneous regions. An optimal superpixel kernel parameter is then selected by the kernel matrix using a multiple kernel learning framework. Finally, a kernel low rank representation is applied to classify the hyperspectral image. The proposed method offers two advantages. (1) The global correlation constraint is exploited by the low rank representation, while the local neighborhood information is extracted as the superpixel kernel adaptively learns the high-dimensional manifold features of the samples in each class; (2) It can meet the challenges of multiscale feature learning and adaptive parameter determination in the conventional kernel methods. Experimental results on several hyperspectral image datasets demonstrate that the proposed method outperforms several state-of-the-art classifiers tested in terms of overall accuracy, average accuracy, and kappa statistic.

Sparse quadratic classification rules via linear dimension reduction

We consider the problem of high-dimensional classification between two groups with unequal covariance matrices. Rather than estimating the full quadratic discriminant rule, we propose to perform simultaneous variable selection and linear dimension reduction on the original data, with the subsequent application of quadratic discriminant analysis on the reduced space. In contrast to quadratic discriminant analysis, the proposed framework does not require the estimation of precision matrices; it scales linearly with the number of measurements, making it especially attractive for the use on high-dimensional datasets. We support the methodology with theoretical guarantees on variable selection consistency, and empirical comparisons with competing approaches. We apply the method to gene expression data of breast cancer patients, and confirm the crucial importance of the ESR1 gene in differentiating estrogen receptor status.

Finite population Bayesian bootstrapping in high-dimensional classification via logistic regression

Finite population Bayesian bootstrapping in high-dimensional classification via logistic regression

Real Time Symptomatic Analysis for Efficient Disease Prediction and Recommendation Generation Using Multi Level Symptom Similarity Measure

The problem of high dimensional clustering and classification has been well studied in previous articles. Also, the recommendation generation towards the treatment based on input symptoms has been considered in this research part. Number of approaches has been discussed earlier in literature towards disease prediction and recommendation generation. Still, the efficient of such recommendation systems are not up to noticeable rate. To improve the performance, an efficient multi level symptom similarity based disease prediction and recommendation generation has been presented. The method reads the input data set, performs preprocessing to remove the noisy records. In the second stage, the method performs Class Level Feature Similarity Clustering. The classification of input symptom set has been performed using MLSS (Multi Level Symptom Similarity) measure estimated between different class of samples. According to the selected class, the method selects higher frequent medicine set as recommendation using drug success rate and frequency measures. The proposed method improves the performance of clustering, disease prediction with higher efficient medicine recommendation.

Enriching the random subspace method with margin theory – a solution for the high-dimensional classification task

ABSTRACTThe random subspace method (RSM) has proved its excellence in numbers of pattern recognition tasks. However, the standard RSM is limited owing to the randomness in its feature selection procedure that is likely to lead to feature subset having poor class separability. In this paper, a proposal for a margin-based criterion has been presented for the evaluation of the true significance of the features, together with the true classification ability of base classifiers, so that both the training phase and integration phase of standard RSM could be enhanced. In the training phase, the random feature selection procedure is enhanced using a weighted random feature selection procedure, in order to improve the classification ability of the base classifier. In the integration phase, the simple majority voting strategy is enhanced using a weighted majority voting strategy for the purpose of assigning the base classifiers with poor classification ability to the lower voting weights. Experimental results on 30 benchmark datasets, together with 6 high-dimensional datasets prove that the recommended approach is capable of better providing classification ability to the usual classification task, in addition to the high-dimensional classification task.

Gaussian Discriminant Analysis for Optimal Delineation of Mild Cognitive Impairment in Alzheimer's Disease.

Over the past few years, several approaches have been proposed to assist in the early diagnosis of Alzheimer's disease (AD) and its prodromal stage of mild cognitive impairment (MCI). Using multimodal biomarkers for this high-dimensional classification problem, the widely used algorithms include Support Vector Machines (SVM), Sparse Representation-based classification (SRC), Deep Belief Networks (DBN) and Random Forest (RF). These widely used algorithms continue to yield unsatisfactory performance for delineating the MCI participants from the cognitively normal control (CN) group. A novel Gaussian discriminant analysis-based algorithm is thus introduced to achieve a more effective and accurate classification performance than the aforementioned state-of-the-art algorithms. This study makes use of magnetic resonance imaging (MRI) data uniquely as input to two separate high-dimensional decision spaces that reflect the structural measures of the two brain hemispheres. The data used include 190 CN, 305 MCI and 133 AD subjects as part of the AD Big Data DREAM Challenge #1. Using 80% data for a 10-fold cross-validation, the proposed algorithm achieved an average F1 score of 95.89% and an accuracy of 96.54% for discriminating AD from CN; and more importantly, an average F1 score of 92.08% and an accuracy of 90.26% for discriminating MCI from CN. Then, a true test was implemented on the remaining 20% held-out test data. For discriminating MCI from CN, an accuracy of 80.61%, a sensitivity of 81.97% and a specificity of 78.38% were obtained. These results show significant improvement over existing algorithms for discriminating the subtle differences between MCI participants and the CN group.

Centroid particle swarm optimisation for high-dimensional data classification

ABSTRACTThis paper proposes a new variant of Particle Swarm Optimization (PSO), dubbed CentroidPSO, to tackle data classification problem in high dimensional domains. It is inspired by the center-based sampling theory, which states that the center region of a search space contains points with higher probability to be closer to the optimal solution. The experimental results show striking performance of the CentroidPSO as compared to the standard PSO, four closely related PSO variants, and three recent evolutionary computation approaches. Moreover, a comparison with three machine learning approaches indicate that the CentroidPSO is a very competitive and promising classifier.

Using high-dimensional machine learning methods to estimate an anatomical risk factor for Alzheimer's disease across imaging databases

IntroductionThe main goal of this work is to investigate the feasibility of estimating an anatomical index that can be used as an Alzheimer's disease (AD) risk factor in the Women's Health Initiative Magnetic Resonance Imaging Study (WHIMS-MRI) using MRI data from the Alzheimer's Disease Neuroimaging Initiative (ADNI), a well-characterized imaging database of AD patients and cognitively normal subjects. We called this index AD Pattern Similarity (AD-PS) scores. To demonstrate the construct validity of the scores, we investigated their associations with several AD risk factors. The ADNI and WHIMS imaging databases were collected with different goals, populations and data acquisition protocols: it is important to demonstrate that the approach to estimating AD-PS scores can bridge these differences. MethodsMRI data from both studies were processed using high-dimensional warping methods. High-dimensional classifiers were then estimated using the ADNI MRI data. Next, the classifiers were applied to baseline and follow-up WHIMS-MRI GM data to generate the GM AD-PS scores. To study the validity of the scores we investigated associations between GM AD-PS scores at baseline (Scan 1) and their longitudinal changes (Scan 2 –Scan 1) with: 1) age, cognitive scores, white matter small vessel ischemic disease (WM SVID) volume at baseline and 2) age, cognitive scores, WM SVID volume longitudinal changes respectively. In addition, we investigated their associations with time until classification of independently adjudicated status in WHIMS-MRI. ResultsHigher GM AD-PS scores from WHIMS-MRI baseline data were associated with older age, lower cognitive scores, and higher WM SVID volume. Longitudinal changes in GM AD-PS scores (Scan 2 – Scan 1) were also associated with age and changes in WM SVID volumes and cognitive test scores. Increases in the GM AD-PS scores predicted decreases in cognitive scores and increases in WM SVID volume. GM AD-PS scores and their longitudinal changes also were associated with time until classification of cognitive impairment. Finally, receiver operating characteristic curves showed that baseline GM AD-PS scores of cognitively normal participants carried information about future cognitive status determined during follow-up. DiscussionWe applied a high-dimensional machine learning approach to estimate a novel AD risk factor for WHIMS-MRI study participants using ADNI data. The GM AD-PS scores showed strong associations with incident cognitive impairment and cross-sectional and longitudinal associations with age, cognitive function, cognitive status and WM SVID volume lending support to the ongoing validation of the GM AD-PS score.

Interpoint Distance Classification of High Dimensional Discrete Observations

SummaryClassification is a multivariate technique that is concerned with allocating new observations to two or more groups. We use interpoint distances to measure the closeness of the samples and construct new rules for high dimensional classification of discrete observations. Applicable to high dimensional data, the new method is non‐parametric and uses test‐based classification with permutation testing. We propose a modification of a test‐based rule to use relative values with respect to the training samples baseline. We compare the proposed rule with parametric methods, such as likelihood ratio rule and modified linear discriminate function, and non‐parametric techniques such as support vector machine, nearest neighbour and depth‐based classification, under multivariate Bernoulli, multinomial and multivariate Poisson distributions.

High-Dimensional Probabilistic Fingerprinting in Wireless Sensor Networks Based on a Multivariate Gaussian Mixture Model.

The extensive deployment of wireless infrastructure provides a low-cost way to track mobile users in indoor environment. This paper demonstrates a prototype model of an accurate and reliable room location awareness system in a real public environment in which three typical problems arise. Firstly, a massive number of access points (APs) can be sensed leading to a high-dimensional classification problem. Secondly, heterogeneous devices record different received signal strength (RSS) levels because of the variations in chip-set and antenna attenuation. Thirdly, APs are not necessarily visible in every scanning cycle leading to missing data issue. This paper presents a probabilistic Wi-Fi fingerprinting method in a hidden Markov model (HMM) framework for mobile user tracking. To account for spatial correlation of the signal strengths from multiple APs, a Multivariate Gaussian Mixture Model (MVGMM) was fitted to model the probability distribution of RSS measurements in each cell. Furthermore, the unseen property of invisible AP was investigated in this research, and demonstrated the efficiency as a beneficial information to differentiate between cells. The proposed system is able to achieve comparable localisation performance. Filed test results achieve a reliable 97% localisation room level accuracy of multiple mobile users in a real university campus Wi-Fi network.

Learning to classify from impure samples with high-dimensional data

A persistent challenge in practical classification tasks is that labeled training sets are not always available. In particle physics, this challenge is surmounted by the use of simulations. These simulations accurately reproduce most features of data, but cannot be trusted to capture all of the complex correlations exploitable by modern machine learning methods. Recent work in weakly supervised learning has shown that simple, low-dimensional classifiers can be trained using only the impure mixtures present in data. Here, we demonstrate that complex, high-dimensional classifiers can also be trained on impure mixtures using weak supervision techniques, with performance comparable to what could be achieved with pure samples. Using weak supervision will therefore allow us to avoid relying exclusively on simulations for high-dimensional classification. This work opens the door to a new regime whereby complex models are trained directly on data, providing direct access to probe the underlying physics.

Sparse subspace linear discriminant analysis

ABSTRACTWe study high dimensional multigroup classification from a sparse subspace estimation perspective, unifying the linear discriminant analysis (LDA) with other recent developments in high dimensional multivariate analysis using similar tools, such as penalization method. We develop two two-stage sparse LDA models, where in the first stage, convex relaxation is used to convert two classical formulations of LDA to semidefinite programs, and furthermore subspace perspective allows for straightforward regularization and estimation. After the initial convex relaxation, we use a refinement stage to improve the accuracy. For the first model, a penalized quadratic program with group lasso penalty is used for refinement, whereas a sparse version of the power method is used for the second model. We carefully examine the theoretical properties of both methods, alongside with simulations and real data analysis.

A new two-layer mixture of factor analyzers with joint factor loading model for the classification of small dataset problems

Abstract Dimensionality Reduction (DR) is a fundamental topic of pattern classification and machine learning. For classification tasks, DR is typically employed as a pre-processing step, succeeded by an independent classifier training stage. However, such independent operation of the two stages often limits the final classification performance notably, as the generated subspace may not be maximally beneficial or appropriate to the learning task at hand. This problem is further accentuated for high-dimensional data classification in situations of the limited number of samples. To address this problem, we develop a novel joint learning model for classification, referred to as two-layer mixture of factor analyzers with joint factor loading (2L-MJFA). Specifically, the model adopts a special two-layer mixture or a mixture of mixtures structure, where each component represents each specific class as a mixture of factor analyzers (MFA). Importantly, all the involved factor analyzers are intentionally designed so that they share the same loading matrix. This, apart from operating as the DR matrix, largely reduces the parameters and makes the proposed algorithm very suitable to small dataset situations. Additionally, we propose a modified expectation maximization algorithm to train the proposed model. A series of simulation experiments demonstrate that what we propose significantly outperforms other state-of-the-art algorithms on various benchmark datasets. Finally, since factor analyzers are closely linked with Auto-encoder networks, the proposed idea could be of particular utility to the community of neural networks.

Sparse feature kernel multi-criteria linear programming classifier

Abstract For a classification problem, the nonlinearly separable case can be transformed into a linearly separable one by using the kernel trick, However, introducing the single kernel function to a classifier can cause the difficulty of feature selection and it reduces the model interpretability, which is very important for some practical applications. Moreover, multi-kernel learning used for improving the predictive accuracy can lead to the poor efficiency and interpretability, especially for large-scale high-dimensional classification problems. In this paper, we proposed a novel sparse feature kernel multi-criteria linear programming classifier (SFK-MCLPC), the two-stage classifier model employs new row and column kernel matrices of different feature kernels to iteratively solve two associative linear programming problems so as to identify important features from data in the process of classification. Based on the ten real-world data sets, the experimental results and comparison with multi-criteria linear and quadratic programming classifiers (MCLPC, MCQPC), support vector classifier (SVC), and multiple kernel learning SVC (MKL-SVC) have shown that the proposed SFK-MCLPC can enhance the predictive accuracy of different classes, the efficiency of classification, the interpretability of the pattern analysis, and the generalization ability of predicting the classes of unseen instances by selecting the relevant and important features.

Booster in High Dimensional Data Classification

Booster in High Dimensional Data Classification

Surrogate-Assisted Multi-objective Genetic Algorithms for Fuzzy Rule-Based Classification

In this paper, we present a surrogate-assisted multi-objective genetic algorithm to mine a small number of linguistically interpretable fuzzy rules for high-dimensional classification task in the realm of data mining. However, the difficulties like (1) handling of high-dimensional problems by fuzzy rule-based systems (i.e., the exponential increase in the number of fuzzy rules with the number of input variables), (2) the deterioration in the comprehensibility of fuzzy rules when they involve many antecedent conditions, and (3) the optimization of multiple objectives in fuzzy rule-based system may stand as pertinent issues. Hence, to combat with the aforesaid issues, we design the problem as a combinatorial optimization problem with three objectives: to maximize the number of correctly classified training patterns, minimize the number of fuzzy rules, and minimize the total number of antecedent conditions. We optimize these objectives of the fuzzy rule-based system by using a multi-objective genetic algorithm. Further to reduce the number of required evaluations to approximate the Pareto front of computationally expensive multi-objective genetic algorithm for fuzzy rule-based system, a radial basis neural network surrogate model is adapted. This approach searches for non-dominated rule sets with respect to these three objectives. The performance of the surrogate-assisted model is evaluated through a few benchmark datasets obtained from knowledge extraction based on evolutionary learning data repository. The experimental outcome confirm that this model is competitive compared to the non-surrogate-assisted model. However, the performance of the model has drawn a clear edge over rule mining approaches like Decision Table, JRip, OneR, PART, and ZeroR.

A method for selecting the relevant dimensions for high-dimensional classification in singular vector spaces

In this paper, we give a new feature selection algorithm for the binary class classification problem in sparse high-dimensional spaces. Singular value decomposition (SVD) is a popular dimension reduction method in higher-dimensional classification. The traditional SVD method begins by ranking the Singular Dimensions (SDs) from largest singular value to the smallest. However, when the number of signals is fewer than the number of noise, the first few ranked SDs are not necessarily the best for classification. We demonstrate, theoretically and empirically, that our method efficiently selects the SDs most appropriate for classification and significantly reduces the misclassification error. We also apply our method to real data text mining applications.

Resource Optimization and Fault Detection Algorithms for Cloud Computing Platforms based on SVM and Resource Reserve Strategy

Efficient operation of cloud computing platforms depends on the optimised virtual resources and faster fault diagnosis system of the virtual machines. This paper proposes an algorithm by introducing a virtual machines based on elastic reservation mechanism, which can improve the availability of cloud resources through the demand analysis taking the help of support vector machines which has advantages resolving nonlinear and high dimensional classification problems. Secondly it adopts the anomaly detection algorithm based on support vector machines for failure analysis. In addition, the dimensionality problem can be sorted out by means of principal component analysis (PCA) algorithm and a kernel function used for distance measurement. It establishes the topological structure for the image set of feature space with Delaunay triangulation and analyses the relationship between kernel parameter and regulator, in order to build an effective model.