Random Subspace Classifier Research Articles

Semi-supervised classifier ensemble model for high-dimensional data

To complete the challenging task of high-dimensional data classification with limited labeled samples, we propose two semi-supervised learning models, namely the random subspace classifier ensemble model (SSRS) and its adaptive version (ASSRS). Considering the unique characteristics of high-dimensional data, SSRS selects subspaces of sample and feature dimensions and then reduces the dimensions of each subspace. To improve SSRS performance further, we designed a sample-labeling auxiliary algorithm, adaptive sample subspace algorithm, and adaptive weight voting rule for ASSRS to increase the proportion of labeled samples, obtain a suitable sample subspace for each feature subspace, and acquire a relative optimal weight for each base classifier. Experiments revealed that the performances of SSRS and ASSRS were better than those of other competitive algorithms and that the performance of ASSRS was stronger than that of SSRS. Additionally, we can accurately label samples in datasets where the proportion of labeled samples is relatively low by using SSRS and ASSRS. Because analysts are facing large numbers of high-dimensional datasets with limited labels, it is important to make accurate predictions based on a limited proportion of labeled data.

Meta-learning algorithm development to predict outcomes in patients with hepatitis C virus-related hepatocellular carcinoma

Background and study aimsPrediction of prognosis and treatment outcomes for patients with hepatocellular carcinoma (HCC) is complex for most patients. Machine learning predictive analysis can be used to explore the rich information in electronic health records to discover hidden patterns and relationships. We aimed to develop a noninvasive algorithm for predicting outcome treatment options for patients with HCC. Patients and methodsThis cross-sectional study included 1298 patients with Hepatitis C virus-related HCC attending an HCC multidisciplinary clinic, Kasr Al-Aini Hospital, Cairo University, between 2009 and 2016. Using machine learning analysis, we constructed Reduced Error Pruning (REP) decision tree algorithms and applied Auto-WEKA to select the best classifier out of 39 algorithms. ResultsThe REP-tree algorithm predicted HCC management outcomes with a recall (sensitivity) of 0.658 and a precision (specificity) of 0.653 using only routine data. 854 (65.8%) instances were correctly identified, and 444 (34.2%) instances were incorrectly classified. Out of 31 attributes, liver decompensation was selected by REP-tree as the best predictor of HCC outcome (root node). With Auto-WEKA, the random subspace classifier was chosen as the best predictive algorithm with a recall (sensitivity) of 0.750 and a precision (specificity) of 0.75. There were 974 (75%) correctly classified instances and 324 (25%) incorrectly classified instances, which was better than REP-tree. ConclusionMachine learning analysis explores data to discover hidden patterns and trends and enables the development of models to predict HCC treatment outcomes utilizing simple laboratory data. The random subspace classifier predicted the outcome more accurately than REP-tree.

A novel approach for cardiovascular disease prediction using machine learning algorithms

AbstractFor the past few decades, cardiovascular disease has shown a binding impact on the country's mortality rate. The prediction of cardiovascular disease is more challenging during the process of clinical data analysis. The emergence of Machine Learning approaches paved the way to predict the disease and determining the consequences of the disease in the earlier stage to help the physicians during complex decision‐making. This work adopts k‐Nearest Neighbor as baseline classifier and ensemble X‐boost, Adaboost, and Random subspace classifier model to predict heart disease and predict the features of cardiovascular disease using Linear Support Vector Feature Measure (). This model considers the diverse combination of features to make the better classification process. The model shows superior performance with precision via Clinical Decision Support System. The factors that influence the cardiovascular disease need to predict, and better decision is taken during the critical condition. Here, the online available University of California Irvine (UCI) Machine Learning dataset is used for training and testing where 80% data is considered for training and 20% considered for testing purpose. The simulation is done in MATLAB 2020b simulation environment, and the outcomes are compared with various existing approaches. Here, performance metrics like accuracy, precision, F‐measure, stability rate, region of curve, and recall is measured to show the model efficiency. The prediction accuracy of the proposed model is 96% which is higher than existing approaches. The overall performance of proposed ensemble model is 96% accuracy, 97% precision, 95% sensitivity, 95% F‐measure, 93% Matthew's correlation coefficients, 4.53% False Positive Rate, 3.10% False Negative Rate, and 96% True Positive Rate, respectively.

Recognition of Bean Plants in Weeds Using Neural Networks

The implementation of a random subspace classifier (RSC) neural network for the recognition of bean plants in weeds was proposed. The RSC neural classifier is based on the multilayer perceptron with a single layer of training connections, allowing a high speed training. The input of this classifier can be considered in various modes, for example, histograms of brightness, contrast, and orientation of micro contours. The RSC neural classifier has been developed for recognition and is applied to different tasks such as micromechanics, tissue recognition, and recognition of metallic textures. The RSC application can help automatize the industrial processes in agriculture. For this purpose, the computer vision based on neural networks can be used.

Binaural Acoustic Scene Classification Using Wavelet Scattering, Parallel Ensemble Classifiers and Nonlinear Fusion.

The analysis of ambient sounds can be very useful when developing sound base intelligent systems. Acoustic scene classification (ASC) is defined as identifying the area of a recorded sound or clip among some predefined scenes. ASC has huge potential to be used in urban sound event classification systems. This research presents a hybrid method that includes a novel mathematical fusion step which aims to tackle the challenges of ASC accuracy and adaptability of current state-of-the-art models. The proposed method uses a stereo signal, two ensemble classifiers (random subspace), and a novel mathematical fusion step. In the proposed method, a stable, invariant signal representation of the stereo signal is built using Wavelet Scattering Transform (WST). For each mono, i.e., left and right, channel, a different random subspace classifier is trained using WST. A novel mathematical formula for fusion step was developed, its parameters being found using a Genetic algorithm. The results on the DCASE 2017 dataset showed that the proposed method has higher classification accuracy (about 95%), pushing the boundaries of existing methods.

A KNN based random subspace ensemble classifier for detection and discrimination of high impedance fault in PV integrated power network

This paper proposes an ensemble Random Subspace (RS) classifier for discrimination of High Impedance Fault (HIF) in photovoltaic connected power network. The design and simulation of power network is considered in MATLAB/Simulink environment to analyze various faults (HIF, Symmetrical, and unsymmetrical), and non-fault events. In pre-process stage of classification, the features from current signals of different events are extracted by using discrete wavelet transform technique. Then, features are used to learn the RS ensemble and base classifiers (K-nearest neighbor, Logistic regression, and Random tree) to get predictions in classification phase. The classification analysis is carried out under with and without real-time varying solar irradiance, and addition of noise data over the input of classifiers. The proposed RS ensemble classifier, discriminates HIF with higher accuracy and success rate than base classifiers. Further, the effectiveness was verified with evaluation of performance indices which shows the proffered ensemble classifier outperforms base classifiers.

Does Physical Activity Predict Obesity-A Machine Learning and Statistical Method-Based Analysis.

Background: Obesity prevalence has become one of the most prominent issues in global public health. Physical activity has been recognized as a key player in the obesity epidemic. Objectives: The objectives of this study are to (1) examine the relationship between physical activity and weight status and (2) assess the performance and predictive power of a set of popular machine learning and traditional statistical methods. Methods: National Health and Nutrition Examination Survey (NHANES, 2003 to 2006) data were used. A total of 7162 participants met our inclusion criteria (3682 males and 3480 females), with average age ranging from 48.6 (normal weight) to 52.1 years old (overweight). Eleven classifying algorithms—including logistic regression, naïve Bayes, Radial Basis Function (RBF), local k-nearest neighbors (k-NN), classification via regression (CVR), random subspace, decision table, multiobjective evolutionary fuzzy classifier, random tree, J48, and multilayer perceptron—were implemented and evaluated, and they were compared with traditional logistic regression model estimates. Results: With physical activity and basic demographic status, of all methods analyzed, the random subspace classifier algorithm achieved the highest overall accuracy and area under the receiver operating characteristic (ROC) curve (AUC). The duration of vigorous-intensity activity in one week and the duration of moderate-intensity activity in one week were important attributes. In general, most algorithms showed similar performance. Logistic regression was middle-ranking in terms of overall accuracy, sensitivity, specificity, and AUC among all methods. Conclusions: Physical activity was an important factor in predicting weight status, with gender, age, and race/ethnicity being less but still essential factors associated with weight outcomes. Tailored intervention policies and programs should target the differences rooted in these demographic factors to curb the increase in the prevalence of obesity and reduce disparities among sub-demographic populations.

A novel acoustic scene classification model using the late fusion of convolutional neural networks and different ensemble classifiers

Recent evidence suggests that convolutional neural networks (CNNs) can model acoustic scene classification (ASC) with high accuracy. Ensemble classifiers have also shown high accuracy in different machine learning areas. However, little is known about fusion models between CNNs and different ensemble classifiers for ASC. This study presents an enhanced CNN classification model using the late fusion between CNNs and ensemble classifiers to predict different classes of acoustic scenes. A CNN model was first built to classify fifteen acoustic scene environments. Different ensemble classifier models were then used for this classification problem. Late fusion of CNN and ensemble classifier models was then applied. The results showed that late fusion models have higher classification accuracy, as compared to individual CNN or ensemble classifier models. The best model was obtained by fusion of the CNN and discriminant random subspace classifier with an increase in the average accuracy of 10% as compared to the average accuracy of the CNN model. When compared with previous research on ASC, the late fusion model between CNN and ensemble classifiers showed higher accuracy. Therefore, this method has robust applicability for future ASC problems.

Features extraction and analysis for device-free human activity recognition based on channel statement information in b5G wireless communications

Features extraction and analysis for human activity recognition (HAR) have been studied for decades in the 5th generation (5G) and beyond the 5th generation (B5G) era. Nowadays, with the extensive use of unmanned aerial vehicles (UAVs) in the civil field, integrating wireless signal receivers on UAVs could be a better choice to receive hearable signals more conveniently. In recent years, the HAR system based on CSI based on WiFi radar has received widespread attention due to its low cost and privacy protection property. However, in the existing CSI-based HAR system, there are two disadvantages: (1) The detection threshold is manually set, which limits its adaptability and immediacy in different wireless environments. (2) A sole classifier is used to complete the recognition, resulting in poor robustness and relatively low recognition accuracy. In this paper, we propose a CSI-based device-free HAR (CDHAR) system with WiFi-sensing radar integrated on UAVs to recognize everyday human activities. Firstly, by using machine learning, CDHAR applies kernel density estimation (KDE) to obtain adaptive detection thresholds to complete the extraction of activity duration. Second, we proposed a random subspace classifier ensemble method for classification, which applies the frequency domain feature instead of the time domain feature, and we choose each kind of feature in the same amount. Finally, we prototype CDHAR on commercial WiFi devices and evaluate its performance in both indoor environment and outdoor environments. The experiment results tell that even if experimental scenario varies, the accuracy of activity durations extraction can reach 98% and 99.60% whether in outdoor or indoor environments. According to the extracted data, the recognition accuracy in outdoor and indoor environments can reach 91.2% and 90.2%, respectively. CDHAR ensures high recognition accuracy while improving the adaptability and instantaneity.

Development of Rheumatoid Arthritis Classification from Electronic Image Sensor Using Ensemble Method.

Rheumatoid arthritis (RA) is an autoimmune illness that impacts the musculoskeletal system by causing chronic, inflammatory, and systemic effects. The disease often becomes progressive and reduces physical function, causes suffering, fatigue, and articular damage. Over a long period of time, RA causes harm to the bone and cartilage of the joints, weakens the joints’ muscles and tendons, eventually causing joint destruction. Sensors such as accelerometer, wearable sensors, and thermal infrared camera sensor are widely used to gather data for RA. In this paper, the classification of medical disorders based on RA and orthopaedics datasets using Ensemble methods are discussed. The RA dataset was gathered from the analysis of white blood cell classification using features extracted from the image of lymphocytes acquired from a digital microscope with an electronic image sensor. The orthopaedic dataset is a benchmark dataset for this study, as it posed a similar classification problem with several numerical features. Three ensemble algorithms such as bagging, Adaboost, and random subspace were used in the study. These ensemble classifiers use k-NN (K-nearest neighbours) and Random forest (RF) as the base learners of the ensemble classifiers. The data classification is accessed using holdout and 10-fold cross-validation evaluation methods. The assessment was based on set of performance measures such as precision, recall, F-measure, and receiver operating characteristic (ROC) curve. The performance was also measured based on the comparison of the overall classification accuracy rate between different ensembles classifiers and the base learners. Overall, it was found that for Dataset 1, random subspace classifier with k-NN shows the best results in terms of overall accuracy rate of 97.50% and for Dataset 2, bagging-RF shows the highest overall accuracy rate of 94.84% over different ensemble classifiers. The findings indicate that the efficiency of the base classifiers with ensemble classifier have substantially improved.

Automatic pest detection on bean and potato crops by applying neural classifiers

Instrumentation and Artificial Intelligence (AI) recognition techniques were developed for automatic pest detection. This system is based on pest detection and monitoring, and it improves the efficiency of vegetable and fruit farming and food production. This paper presents an automatic pest detection system that applies artificial neural networks. The system automatically detects two defoliating pests on potato and bean crops: Mexican Bean Beetle (MBB) and Colorado Potato Beetle (CPB) in the adult stage. The neural classifiers utilized for the beetle detection are RSC (Random Subspace Classifier) and LIRA (Limited Receptive Area). The MBB images that were employed as inputs to the classifiers were obtained on Mexican plantations. The CPB images were collected from various Internet sources. We compared the results obtained with both classifiers on image databases. The RSC classifier demonstrates the better result for recognition, which is 89%, while LIRA presents a recognition rate of 88%. These results are good for pest detection and can be used for the diagnosis of pest locations in crops. The purpose was to contribute to the development of automatic detection applications based on images of potato and bean plantations. In Mexico and other countries, it is of great importance to solve pest problems in agriculture. We chose insect recognition due to the importance of potato and bean crop production and consumption. Pest detection in the adult phase is of high priority because of the high rate of crop defoliation and destruction. Our automatic pest-detection system can be employed in pest recognition in monitoring activities.

Recognizing driving postures by combined features of contourlet transform and edge orientation histogram, and random subspace classifier ensembles

In order to develop Human-centered Driver Assistance Systems (HDAS), an efficient Combined Feature (CF) extraction approach from Contourlet Transform (CT) and Edge Orientation Histogram (EOH) is pr...

Recognizing driving postures by combined features of contourlet transform and edge orientation histogram, and random subspace classifier ensembles

In order to develop Human-centered Driver Assistance Systems (HDAS), an efficient Combined Feature (CF) extraction approach from Contourlet Transform (CT) and Edge Orientation Histogram (EOH) is proposed for vehicle driving posture descriptions. A Random Subspace Ensemble (RSE) of Intersection Kernel Support Vector Machines (IKSVMs) is then exploited as the base classifier. Four testing driving postures are grasping the steering wheel, operating the shift lever, eating a cake, and talking on a cellar phone. On a dedicated Southeast University Driving Posture (SEU-DP) Database, the holdout and cross-validation experiments were conducted. The experimental results show that the proposed CF-RSE approach outperforms single Contourlet-IKSVM, EOH-IKSVM recognition strategies. With CF-RSE, the average classification accuracies of four driving posture classes are over 90%. Among the four classes of driving postures, the class of grasping the steering wheel is the most difficult to recognize and the proposed approach achieved over 85% accuracy in both experiments. These encouraging results show that the proposed CF-RSE approach is effective and hence has great promises in developing a successful HDAS.

Breast cancer diagnosis from biopsy images with highly reliable random subspace classifier ensembles

Accurate and reliable classification of microscopic biopsy images is an important issue in computer assisted breast cancer diagnosis. In this paper, a new cascade Random Subspace ensembles scheme with reject options is proposed for microscopic biopsy image classification. The classification system is built as a serial fusion of two different Random Subspace classifier ensembles with rejection options to enhance the classification reliability. The first ensemble consists of a set of Support Vector Machine classifiers that converts the original $$K$$ -class classification problem into a number of $$K$$ 2-class problems. The second ensemble consists of a Multi-Layer Perceptron ensemble, that focuses on the rejected samples from the first ensemble. For both of the ensembles, the reject option is implemented by relating the consensus degree from majority voting to a confidence measure, and abstaining to classify ambiguous samples if the consensus degree is lower than some threshold. We also investigated the effectiveness of a feature description approach by combining Local Binary Pattern (LBP) texture analysis, statistics derived using the Gray Level Co-occurrence Matrix (GLCM) and the Curvelet Transform. While the LBP analysis efficiently describes local texture properties and the GLCM reflects global texture statistics, the Curvelet Transform is particularly appropriate for the representation of piece-wise smooth images with rich edge information. The combined feature description thus provides a comprehensive biopsy image characterization by taking advantages of their complementary strengths. Using a benchmark microscopic biopsy image dataset, obtained from the Israel Institute of Technology, a high classification accuracy of $$99.25 \%$$ was obtained (with a rejection rate of $$1.94 \%$$ ) using the proposed system.

A classifier ensemble approach for the missing feature problem

Many classification problems must deal with data that contains missing values. In such cases data imputation is critical. This paper evaluates the performance of several statistical and machine learning imputation methods, including our novel multiple imputation ensemble approach, using different datasets. Several state-of-the-art approaches are compared using different datasets. Some state-of-the-art classifiers (including support vector machines and input decimated ensembles) are tested with several imputation methods. The novel approach proposed in this work is a multiple imputation method based on random subspace, where each missing value is calculated considering a different cluster of the data. We have used a fuzzy clustering approach for the clustering algorithm. Our experiments have shown that the proposed multiple imputation approach based on clustering and a random subspace classifier outperforms several other state-of-the-art approaches. Using the Wilcoxon signed-rank test (reject the null hypothesis, level of significance 0.05) we have shown that the proposed best approach is outperformed by the classifier trained using the original data (i.e., without missing values) only when >20% of the data are missed. Moreover, we have shown that coupling an imputation method with our cluster based imputation we outperform the base method (level of significance ∼0.05). Starting from the assumptions that the feature set must be partially redundant and that the redundancy is distributed randomly over the feature set, we have proposed a method that works quite well even when a large percentage of the features is missing (≥30%). Our best approach is available (MATLAB code) at bias.csr.unibo.it/nanni/MI.rar.

Phenotype Recognition with Combined Features and Random Subspace Classifier Ensemble

BackgroundAutomated, image based high-content screening is a fundamental tool for discovery in biological science. Modern robotic fluorescence microscopes are able to capture thousands of images from massively parallel experiments such as RNA interference (RNAi) or small-molecule screens. As such, efficient computational methods are required for automatic cellular phenotype identification capable of dealing with large image data sets. In this paper we investigated an efficient method for the extraction of quantitative features from images by combining second order statistics, or Haralick features, with curvelet transform. A random subspace based classifier ensemble with multiple layer perceptron (MLP) as the base classifier was then exploited for classification. Haralick features estimate image properties related to second-order statistics based on the grey level co-occurrence matrix (GLCM), which has been extensively used for various image processing applications. The curvelet transform has a more sparse representation of the image than wavelet, thus offering a description with higher time frequency resolution and high degree of directionality and anisotropy, which is particularly appropriate for many images rich with edges and curves. A combined feature description from Haralick feature and curvelet transform can further increase the accuracy of classification by taking their complementary information. We then investigate the applicability of the random subspace (RS) ensemble method for phenotype classification based on microscopy images. A base classifier is trained with a RS sampled subset of the original feature set and the ensemble assigns a class label by majority voting.ResultsExperimental results on the phenotype recognition from three benchmarking image sets including HeLa, CHO and RNAi show the effectiveness of the proposed approach. The combined feature is better than any individual one in the classification accuracy. The ensemble model produces better classification performance compared to the component neural networks trained. For the three images sets HeLa, CHO and RNAi, the Random Subspace Ensembles offers the classification rates 91.20%, 98.86% and 91.03% respectively, which compares sharply with the published result 84%, 93% and 82% from a multi-purpose image classifier WND-CHARM which applied wavelet transforms and other feature extraction methods. We investigated the problem of estimation of ensemble parameters and found that satisfactory performance improvement could be brought by a relative medium dimensionality of feature subsets and small ensemble size.ConclusionsThe characteristics of curvelet transform of being multiscale and multidirectional suit the description of microscopy images very well. It is empirically demonstrated that the curvelet-based feature is clearly preferred to wavelet-based feature for bioimage descriptions. The random subspace ensemble of MLPs is much better than a number of commonly applied multi-class classifiers in the investigated application of phenotype recognition.

Performance Analysis of k-NN on High Dimensional Datasets

Research on classifying high dimensional datasets is an open direction in the pattern recognition yet. High dimensional feature spaces cause scalability problems for machine learning algorithms because the complexity of a high dimensional space increases exponentially with the number of features. Recently a number of ensemble techniques using different classifiers have proposed for classifying the high dimensional datasets. The task of these techniques is to detect and exploit relevant patterns in data for classification. The k-nearest neighbor (kNN) algorithm is amongst the simplest of all machine learning algorithms. This paper discusses various ensemble k-NN techniques on high dimensional datasets. The techniques mainly include: Random Subspace Classifier (RSM), Divide & Conquer Classification and Optimization using GA (DCC-GA), Random Subsample ensemble (RSE), Improving Fusion of dimensionality reduction (IF-DR). All these approaches generates relevant subset of features from original set and the results is obtain from combined decision of ensemble classifiers. This paper presents an effective study of improvements on ensemble k-NN for the classification of high dimensional datasets. The experimental result shows that these approaches improve the classification accuracy of the k-NN classifier.

Evolved Feature Weighting for Random Subspace Classifier

The problem addressed in this letter concerns the multiclassifier generation by a random subspace method (RSM). In the RSM, the classifiers are constructed in random subspaces of the data feature space. In this letter, we propose an evolved feature weighting approach: in each subspace, the features are multiplied by a weight factor for minimizing the error rate in the training set. An efficient method based on particle swarm optimization (PSO) is here proposed for finding a set of weights for each feature in each subspace. The performance improvement with respect to the state-of-the-art approaches is validated through experiments with several benchmark data sets.

Analysis of separating surfaces formed by a random subspace classifier

The discriminating capabilities of a random subspace classifier are considered. As a result of analysis of the probability density distribution of threshold values, an estimate is obtained for the minimum distinguishable distance. Real examples of separating surfaces for classical two-imensional problems are given. An algorithm is proposed for local averaging of a synapse matrix to improve the classifier performance in solving problems with overlapping probability distributions. The random subspace classifier is proved to be universal.