Performance Of Machine Learning Classifiers Research Articles

Machine Learning Assisted Citation Screening for Systematic Reviews.

Evidence-based practice is highly dependent upon up-to-date systematic reviews (SR) for decision making. However, conducting and updating systematic reviews, especially the citation screening for identification of relevant studies, requires much human work and is therefore expensive. Automating citation screening using machine learning (ML) based approaches can reduce cost and labor. Machine learning has been applied to automate citation screening but not for the SRs with very narrow research questions. This paper reports the results and observations for an ongoing research that aims to automate citation screening for SRs with narrow research questions using machine learning. The research also sheds light on the problem of class imbalance and class overlap on the performance of ML classifiers when applied to SRs with narrow research questions.

Evaluating the Impact of a Two-Stage Multivariate Data Cleansing Approach to Improve to the Performance of Machine Learning Classifiers: A Case Study in Human Activity Recognition.

Human activity recognition (HAR) is a popular field of study. The outcomes of the projects in this area have the potential to impact on the quality of life of people with conditions such as dementia. HAR is focused primarily on applying machine learning classifiers on data from low level sensors such as accelerometers. The performance of these classifiers can be improved through an adequate training process. In order to improve the training process, multivariate outlier detection was used in order to improve the quality of data in the training set and, subsequently, performance of the classifier. The impact of the technique was evaluated with KNN and random forest (RF) classifiers. In the case of KNN, the performance of the classifier was improved from 55.9% to 63.59%.

Pre-harvest classification of crop types using a Sentinel-2 time-series and machine learning

Timely crop type information (preferably before harvest) is useful for predicting food surpluses or shortages. This study assesses the performance of several machine learning classifiers, namely SVM (support vector machine), DT (decision tree), k-NN (k-nearest neighbour), RF (random forest) and ML (maximum likelihood) for crop type mapping based on a series of Sentinel-2 images. Four experiments with different combinations of image sets were carried out. The first three experiments were undertaken with 1) single-date (uni-temporal) images; 2) combinations of five images selected from the best performing single-date images; and 3) five images selected manually based on crop development stages. The fourth experiment involved the chronologic addition of images to assess the performance of the classifiers when only pre-harvest images are used, with the purpose of investigating how early in the season reasonable accuracies can be achieved. The experiments were carried out in two different sites in the Western Cape Province of South Africa to provide a good representation of the grain-producing areas in the region which has a Mediterranean climate. The significance of image selection on classification accuracies as well as the performance of machine learning classifiers when only pre-harvest images are used were evaluated. The classification results were analysed by comparing overall accuracies and kappa coefficients, while McNemar′s test and ANOVA (analysis of variance) were used to assess the statistical significance of the differences in accuracies among experiments. The results show that by selecting images based on individual performance, a viable alternative to selecting images based on crop developmental stages is offered, and that the classification of crops with an entire time series can be just as accurate as when they are classified with a subset of hand-selected images. We also found that good classification accuracies (77.2%) can be obtained with the use of SVM and RF as early as eight weeks before harvest. This result shows that pre-harvest images have the potential to identify crops accurately, which holds much potential for operational within-season crop type mapping.

Machine learning classifiers-based prediction of normal-tension glaucoma progression in young myopic patients.

To assess the performance of machine learning classifiers for prediction of progression of normal-tension glaucoma (NTG) in young myopic patients. Cross-sectional study. One hundred and fifty-five eyes of 155 myopic NTG patients (axial length [AL] ≥ 24.00mm and refractive error ≤ - 3.0D) between the ages of 20 and 40 were enrolled and divided into training (110) and test (45) sets. Sixty-five eyes showed glaucoma progression as defined by standard automated perimetry, while 91 eyes (nonprogressors) had been stable over the course of a follow-up period of at least 3years. Two machine learning classifiers were built using the random forest and extremely randomized trees (extra-trees) models. Baseline clinical measurements obtained only at the initial visit were used as input features. The area under the receiver operating characteristic curve (AUC) was calculated to evaluate the accuracy of prediction. Mean age and AL did not significantly differ between the 2 groups on either the training or the test set. The extra-trees model achieved an AUC of 0.881 [95% CI 0.814-0.945], higher than that of the random forest model (0.811 [0.731-0.888]; P = 0.010). The extra-trees model also outperformed all the clinical measurements for prediction of NTG progression, including average macular ganglion cell-inner plexiform layer thickness (0.735 [0.639-0.831]) and average circumpapillary retinal nerve fiber layer thickness (0.691 [0.590-0.792]; both P < 0.001). In young myopic patients, the machine learning classifier with the extra-trees model can predict glaucomatous progression more effectively than clinical diagnostic parameters.

Tweets Classification on the Base of Sentiments for US Airline Companies

The use of data from social networks such as Twitter has been increased during the last few years to improve political campaigns, quality of products and services, sentiment analysis, etc. Tweets classification based on user sentiments is a collaborative and important task for many organizations. This paper proposes a voting classifier (VC) to help sentiment analysis for such organizations. The VC is based on logistic regression (LR) and stochastic gradient descent classifier (SGDC) and uses a soft voting mechanism to make the final prediction. Tweets were classified into positive, negative and neutral classes based on the sentiments they contain. In addition, a variety of machine learning classifiers were evaluated using accuracy, precision, recall and F1 score as the performance metrics. The impact of feature extraction techniques, including term frequency (TF), term frequency-inverse document frequency (TF-IDF), and word2vec, on classification accuracy was investigated as well. Moreover, the performance of a deep long short-term memory (LSTM) network was analyzed on the selected dataset. The results show that the proposed VC performs better than that of other classifiers. The VC is able to achieve an accuracy of 0.789, and 0.791 with TF and TF-IDF feature extraction, respectively. The results demonstrate that ensemble classifiers achieve higher accuracy than non-ensemble classifiers. Experiments further proved that the performance of machine learning classifiers is better when TF-IDF is used as the feature extraction method. Word2vec feature extraction performs worse than TF and TF-IDF feature extraction. The LSTM achieves a lower accuracy than machine learning classifiers.

Impact of ecological redundancy on the performance of machine learning classifiers in vegetation mapping.

Vegetation maps are models of the real vegetation patterns and are considered important tools in conservation and management planning. Maps created through traditional methods can be expensive and time‐consuming, thus, new more efficient approaches are needed. The prediction of vegetation patterns using machine learning shows promise, but many factors may impact on its performance. One important factor is the nature of the vegetation–environment relationship assessed and ecological redundancy. We used two datasets with known ecological redundancy levels (strength of the vegetation–environment relationship) to evaluate the performance of four machine learning (ML) classifiers (classification trees, random forests, support vector machines, and nearest neighbor). These models used climatic and soil variables as environmental predictors with pretreatment of the datasets (principal component analysis and feature selection) and involved three spatial scales. We show that the ML classifiers produced more reliable results in regions where the vegetation–environment relationship is stronger as opposed to regions characterized by redundant vegetation patterns. The pretreatment of datasets and reduction in prediction scale had a substantial influence on the predictive performance of the classifiers. The use of ML classifiers to create potential vegetation maps shows promise as a more efficient way of vegetation modeling. The difference in performance between areas with poorly versus well‐structured vegetation–environment relationships shows that some level of understanding of the ecology of the target region is required prior to their application. Even in areas with poorly structured vegetation–environment relationships, it is possible to improve classifier performance by either pretreating the dataset or reducing the spatial scale of the predictions.

Assessment of Supervised Classifiers for Land Cover Categorization Based on Integration of ALOS PALSAR and Landsat Data

Many supervised classification algorithms have been proposed, however, they are rarely evaluated for specific application. This research examines the performance of machine learning classifiers support vector machine (SVM), neural network (NN), Random Forest (RF) against maximum classifier (MLC) (traditional supervised classifier) in forest resources and land cover categorization, based on combination of Advanced Land Observing Satellite (ALOS) Phased Array type L-band Synthetic Aperture Radar (PALSAR) and Landsat Thematic Mapper (TM) data, in Northern Tanzania. Various data categories based on Landsat TM surface reflectance, ALOS PALSAR backscattering and their derivatives were generated for various classification scenarios. Then a separate and joint processing of Landsat and ALOS PALSAR data were executed using SVM, NN, RF and ML classifiers. The overall classification accuracy (OA), kappa coefficient (KC) and F1 score index values were computed. The result proves the robustness of SVM and RF in classification of forest resource and land cover using mere Landsat data and integration of Landsat and PALSAR (average OA = 92% and F1 = 0.7 to 1). A two sample t-statistics was utilized to evaluate the performance of the classifiers using different data categories. SVM and RF indicate there is no significance difference at 5% significance level. SVM and RF show a significant difference when compared to NN and ML. Generally, the study suggests that parametric classifiers indicate better performance compared to parametric classifier.

Classification of Content based Medical Image Retrieval Using Texture and Shape feature with Neural Network

Medical image classification and retrieval systems have been finding extensive use in the areas of image classification according to imaging modalities, body part and diseases. One of the major challenges in the medical classification is the large size images leading to a large number of extracted features which is a burden for the classification algorithm and the resources. In this paper, a novel approach for automatic classification of fundus images is proposed. The method uses image and data pre-processing techniques to improve the performance of machine learning classifiers.&lt;em&gt; &lt;/em&gt;Some predominant image mining algorithms such as Classification, Regression Tree (CART), Neural Network, Naive Bayes (NB), Decision Tree (DT) K-Nearest Neighbor. The performance of MCBIR systems using texture and shape features efficient. . The possible outcomes of a two class prediction be represented as True positive (TP), True negative (TN), False Positive (FP) and False Negative (FN).

The impact of training class proportions on binary cropland classification

ABSTRACTThe ground truth data sets required to train supervised classifiers are usually collected as to maximize the number of samples under time, budget and accessibility constraints. Yet, the performance of machine learning classifiers is, among other factors, sensitive to the class proportions of the training set. In this letter, the joint effect of the number of calibration samples and the class proportions on the accuracy was systematically quantified using two state-of-the-art machine learning classifiers (random forests and support vector machines). The analysis was applied in the context of binary cropland classification and focused on two contrasted agricultural landscapes. Results showed that the classifiers were more sensitive to class proportions than to sample size, though sample size had to reach 2,000 pixels before its effect leveled off. Optimal accuracies were obtained when the training class proportions were close to those actually observed on the ground. Then, synthetic minority over-sampling technique (SMOTE) was implemented to artificially regenerate the native class proportions in the training set. This resampling method led to an increase of the accuracy of up to 30%. These results have direct implications for (i) informing data collection strategies and (ii) optimizing classification accuracy. Though derived for cropland mapping, the recommendations are generic to the problem of binary classification.

Friedman and Wilcoxon Evaluations Comparing SVM, Bagging, Boosting, K-NN and Decision Tree Classifiers

Abstract This paper describes a number of experiments to compare and validate the performance of machine learning classifiers. Creating machine learning models for data with wide varieties has huge applications in predictive modelling across multiple domain of science. This work reviews state of the art techniques in machine learning classifiers methods with several extent of magnitude in statistics and key findings that will be helpful in establishing best methodological practices for class predictions. Comprehensive comparative review analysis with statistical validations for various machine learning algorithm for SVM, Bagging, Boosting, Decision Trees and Nearest Neighborhood algorithm on multiple data sets is carried out. Focus on the statistical analysis of the results using Friedman-Test and Wilcoxon Test as well as other interpretative metrics like classification rate, ROC, F-measure are evaluated to benchmark results.

Performance of Machine Learning Classifiers for Indoor Person Localization With Capacitive Sensors

Accurate tagless indoor person localization is important for several applications, such as assisted living and health monitoring. Machine learning (ML) classifiers can effectively mitigate sensor data variability and noise due to deployment-specific environmental conditions. In this paper, we use experimental data from a capacitive sensor-based indoor human localization system in a 3m $\times3\text{m}$ room to comparatively analyze the performance of Weka collection ML classifiers. We compare the localization performance of the algorithms, its variation with the training set size, and the algorithm resource requirements for both training and inferring. The results show a large variance between algorithms, with the best accuracy, precision, and recall exceeding 93% and 0.05m average localization error.

Machine Learning Classifiers: Evaluation of the Performance in Online Reviews

Objectives: This paper aims to evaluate the performance of the machine learning classifiers and identify the most suitable classifier for classifying sentiment value. The term “sentiment value” in this study is referring to the polarity (positive, negative or neutral) of the text. Methods/Analysis: This work applies machine learning classifiers from WEKA (Waikato Environment for Knowledge Analysis) toolkit in order to perform their evaluation. WEKA toolkit is a great set of tools for data mining and classification. The performance of the machine learning classifiers was evaluated by looking at overall accuracy, recall, precision, kappa statistic and few visualization techniques. Finally, the analysis is applied to find the most suitable classifier for classifying sentiment value. Findings: Results show that two classifiers from Rules and Trees categories of classifiers perform equally best comparing to the other classifiers from categories, such as Bayes, Functions, Lazy and Meta. Novelty/Improvement: This paper explores the performance of machine learning classifiers in sentiment value classification of the online reviews. Data used is never been used before to explore the performance of machine learning classifiers.

Probing the Hypothesis of SAR Continuity Restoration by the Removal of Activity Cliffs Generators in QSAR.

In this work we report the first attempt to study the effect of activity cliffs over the generalization ability of machine learning (ML) based QSAR classifiers, using as study case a previously reported diverse and noisy dataset focused on drug induced liver injury (DILI) and more than 40 ML classification algorithms. Here, the hypothesis of structure-activity relationship (SAR) continuity restoration by activity cliffs removal is tested as a potential solution to overcome such limitation. Previously, a parallelism was established between activity cliffs generators (ACGs) and instances that should be misclassified (ISMs), a related concept from the field of machine learning. Based on this concept we comparatively studied the classification performance of multiple machine learning classifiers as well as the consensus classifier derived from predictive classifiers obtained from training sets including or excluding ACGs. The influence of the removal of ACGs from the training set over the virtual screening performance was also studied for the respective consensus classifiers algorithms. In general terms, the removal of the ACGs from the training process slightly decreased the overall accuracy of the ML classifiers and multi-classifiers, improving their sensitivity (the weakest feature of ML classifiers trained with ACGs) but decreasing their specificity. Although these results do not support a positive effect of the removal of ACGs over the classification performance of ML classifiers, the "balancing effect" of ACG removal demonstrated to positively influence the virtual screening performance of multi-classifiers based on valid base ML classifiers. Specially, the early recognition ability was significantly favored after ACGs removal. The results presented and discussed in this work represent the first step towards the application of a remedial solution to the activity cliffs problem in QSAR studies.

Improved automatic English proficiency rating of unconstrained speech with multiple corpora

The performance of machine learning classifiers in automatically scoring the English proficiency of unconstrained speech has been explored. Suprasegmental measures were computed by software, which identifies the basic elements of Brazil's model in human discourse. This paper explores machine learning training with multiple corpora to improve two of those algorithms: prominent syllable detection and tone choice classification. The results show that machine learning training with the Boston University Radio News Corpus can improve automatic English proficiency scoring of unconstrained speech from a Pearson's correlation of 0.677---0.718. This correlation is higher than any other existing computer programs for automatically scoring the proficiency of unconstrained speech and is approaching that of human raters in terms of inter-rater reliability.

Survey and Comparison of Classification Algorithm for Medical Image

Medical image classification and retrieval systems have been finding extensive use in the areas of image classification according to imaging modalities, body part and diseases. One of the major challenges in the medical classification is the large size images leading to a large number of extracted features which is a burden for the classification algorithm and the resources. In this paper, a novel approach for automatic classification of fundus images is proposed. The method uses image and data pre-processing techniques to improve the performance of machine learning classifiers. Some predominant image mining algorithms such as Classification, Regression Tree (CART), Neural Network, Naive Bayes (NB), Decision Tree (DT) K-Nearest Neighbor Index Terms—classification, medical imaging, neural networks, texture and shape features.

Graph-Based Approaches for Over-Sampling in the Context of Ordinal Regression

The classification of patterns into naturally ordered labels is referred to as ordinal regression or ordinal classification. Usually, this classification setting is by nature highly imbalanced, because there are classes in the problem that are a priori more probable than others. Although standard over-sampling methods can improve the classification of minority classes in ordinal classification, they tend to introduce severe errors in terms of the ordinal label scale, given that they do not take the ordering into account. A specific ordinal over-sampling method is developed in this paper for the first time in order to improve the performance of machine learning classifiers. The method proposed includes ordinal information by approaching over-sampling from a graph-based perspective. The results presented in this paper show the good synergy of a popular ordinal regression method (a reformulation of support vector machines) with the graph-based proposed algorithms, and the possibility of improving both the classification and the ordering of minority classes. A cost-sensitive version of the ordinal regression method is also introduced and compared with the over-sampling proposals, showing in general lower performance for minority classes.

Polarity classification using structure-based vector representations of text

The exploitation of structural aspects of content is becoming increasingly popular in rule-based polarity classification systems. Such systems typically weight the sentiment conveyed by text segments in accordance with these segments' roles in the structure of a text, as identified by deep linguistic processing. Conversely, state-of-the-art machine learning polarity classifiers typically aim to exploit patterns in vector representations of texts, mostly covering the occurrence of words or word groups in these texts. However, since structural aspects of content have been shown to contain valuable information as well, we propose to use structure-based features in vector representations of text. We evaluate the usefulness of our novel features on collections of English reviews in various domains. Our experimental results suggest that, even though word-based features are indispensable to good polarity classifiers, structure-based sentiment information provides valuable additional guidance that can help significantly improve the polarity classification performance of machine learning classifiers. The most informative features capture the sentiment conveyed by specific rhetorical elements that constitute a text's core or provide crucial contextual information.

An improved method of early diagnosis of smoking-induced respiratory changes using machine learning algorithms

The purpose of this study was to develop an automatic classifier to increase the accuracy of the forced oscillation technique (FOT) for diagnosing early respiratory abnormalities in smoking patients. The data consisted of FOT parameters obtained from 56 volunteers, 28 healthy and 28 smokers with low tobacco consumption. Many supervised learning techniques were investigated, including logistic linear classifiers, k nearest neighbor (KNN), neural networks and support vector machines (SVM). To evaluate performance, the ROC curve of the most accurate parameter was established as baseline. To determine the best input features and classifier parameters, we used genetic algorithms and a 10-fold cross-validation using the average area under the ROC curve (AUC). In the first experiment, the original FOT parameters were used as input. We observed a significant improvement in accuracy (KNN=0.89 and SVM=0.87) compared with the baseline (0.77). The second experiment performed a feature selection on the original FOT parameters. This selection did not cause any significant improvement in accuracy, but it was useful in identifying more adequate FOT parameters. In the third experiment, we performed a feature selection on the cross products of the FOT parameters. This selection resulted in a further increase in AUC (KNN=SVM=0.91), which allows for high diagnostic accuracy. In conclusion, machine learning classifiers can help identify early smoking-induced respiratory alterations. The use of FOT cross products and the search for the best features and classifier parameters can markedly improve the performance of machine learning classifiers.

Machine learning classifiers for glaucoma diagnosis based on classification of retinal nerve fibre layer thickness parameters measured by Stratus OCT

To compare the performance of two machine learning classifiers (MLCs), artificial neural networks (ANNs) and support vector machines (SVMs), with input based on retinal nerve fibre layer thickness (RNFLT) measurements by optical coherence tomography (OCT), on the diagnosis of glaucoma, and to assess the effects of different input parameters. We analysed Stratus OCT data from 90 healthy persons and 62 glaucoma patients. Performance of MLCs was compared using conventional OCT RNFLT parameters plus novel parameters such as minimum RNFLT values, 10th and 90th percentiles of measured RNFLT, and transformations of A-scan measurements. For each input parameter and MLC, the area under the receiver operating characteristic curve (AROC) was calculated. There were no statistically significant differences between ANNs and SVMs. The best AROCs for both ANN (0.982, 95%CI: 0.966-0.999) and SVM (0.989, 95% CI: 0.979-1.0) were based on input of transformed A-scan measurements. Our SVM trained on this input performed better than ANNs or SVMs trained on any of the single RNFLT parameters (p < or = 0.038). The performance of ANNs and SVMs trained on minimum thickness values and the 10th and 90th percentiles were at least as good as ANNs and SVMs with input based on the conventional RNFLT parameters. No differences between ANN and SVM were observed in this study. Both MLCs performed very well, with similar diagnostic performance. Input parameters have a larger impact on diagnostic performance than the type of machine classifier. Our results suggest that parameters based on transformed A-scan thickness measurements of the RNFL processed by machine classifiers can improve OCT-based glaucoma diagnosis.