Classification Accuracy Metrics Research Articles

Investigation of Combining Logitboost(M5P) under Active Learning Classification Tasks

Active learning is the category of partially supervised algorithms that is differentiated by its strategy to combine both the predictive ability of a base learner and the human knowledge so as to exploit adequately the existence of unlabeled data. Its ambition is to compose powerful learning algorithms which otherwise would be based only on insufficient labelled samples. Since the latter kind of information could raise important monetization costs and time obstacles, the human contribution should be seriously restricted compared with the former. For this reason, we investigate the use of the Logitboost wrapper classifier, a popular variant of ensemble algorithms which adopts the technique of boosting along with a regression base learner based on Model trees into 3 different active learning query strategies. We study its efficiency against 10 separate learners under a well-described active learning framework over 91 datasets which have been split to binary and multi-class problems. We also included one typical Logitboost variant with a separate internal regressor for discriminating the benefits of adopting a more accurate regression tree than one-node trees, while we examined the efficacy of one hyperparameter of the proposed algorithm. Since the application of the boosting technique may provide overall less biased predictions, we assume that the proposed algorithm, named as Logitboost(M5P), could provide both accurate and robust decisions under active learning scenarios that would be beneficial on real-life weakly supervised classification tasks. Its smoother weighting stage over the misclassified cases during training as well as the accurate behavior of M5P are the main factors that lead towards this performance. Proper statistical comparisons over the metric of classification accuracy verify our assumptions, while adoption of M5P instead of weak decision trees was proven to be more competitive for the majority of the examined problems. We present our results through appropriate summarization approaches and explanatory visualizations, commenting our results per case.

Random forest machine learning algorithm predicts virologic outcomes among HIV infected adults in Lausanne, Switzerland using electronically monitored combined antiretroviral treatment adherence

ABSTRACT Machine Learning (ML) can improve the analysis of complex and interrelated factors that place adherent people at risk of viral rebound. Our aim was to build ML model to predict RNA viral rebound from medication adherence and clinical data. Patients were followed up at the Swiss interprofessional medication adherence program (IMAP). Sociodemographic and clinical variables were retrieved from the Swiss HIV Cohort Study (SHCS). Daily electronic medication adherence between 2008–2016 were analyzed retrospectively. Predictor variables included: RNA viral load (VL), CD4 count, duration of ART, and adherence. Random Forest, was used with 10 fold cross validation to predict the RNA class for each data observation. Classification accuracy metrics were calculated for each of the 10-fold cross validation holdout datasets. The values for each range from 0 to 1 (better accuracy). 383 HIV+ patients, 56% male, 52% white, median (Q1, Q3): age 43 (36, 50), duration of electronic monitoring of adherence 564 (200, 1333) days, CD4 count 406 (209, 533) cells/mm3, time since HIV diagnosis was 8.4 (4, 13.5) years, were included. Average model classification accuracy metrics (AUC and F1) for RNA VL were 0.6465 and 0.7772, respectively. In conclusion, combining adherence with other clinical predictors improve predictions of RNA.

Pragmatic Language and Social Communication Tests for Students Aged 8–18 Years: A Review of Test Accuracy

Purpose The accuracy of commercially available pragmatic language and social communication tests was reviewed. Method Online sources were searched to identify tests providing stand-alone pragmatic or social communication scores. Test manuals were examined to extract information about clinical and typically developing group comparisons, including sample size, inclusionary/exclusionary criteria, and mean group differences. Classification accuracy metrics, including sensitivity, specificity, optimal cut scores, and likelihood ratios, were reviewed. Results Six domain-specific pragmatic language and social communication tests and four omnibus language tests were identified. Test content varied and included video and audio recordings and question-and-answer formats to address hypothetical social vignettes, retell and comprehend narratives, interpret nonliteral language and multiple-meaning words, identify and display emotions, and infer another's perspective. Sample size and inclusionary/exclusionary criteria for comparisons of students with typical development (TD), developmental language disorder (DLD), autism spectrum disorder (ASD), and social (pragmatic) communication disorder (SPCD) varied widely. Mean group differences between TD and ASD or SPCD groups were generally larger (i.e., > 0.5 SD to > 2.0 SD s) than TD and DLD comparisons. Test authors interpreted these results as support for the accuracy of tests to identify deficits associated with ASD and SPCD and not DLD. Classification accuracy was reported for five domain-specific tests. Optimal cut scores ranged from 85 to 92, with adequate sensitivity and specificity (i.e., ≥ 80%); however, positive likelihood ratios and associated confidence intervals revealed limited confidence in reported levels. Conclusions Inadequate sample description for comparison and classification studies limit a clinician's ability to compare a student's profile with those in the samples. Although none of the tests meet all standards for test accuracy reviewed in this report, a checklist to guide test selection is provided to assist in identifying the best available test for a student's profile, if a test is needed.

Performance Analysis of Efficient Pre-trained Networks based on Transfer Learning for Tomato Leaf Diseases Classification

Early diagnosis and accurate identification to tomato leaf diseases contribute on controlling the diffusion of infection and guarantee healthy to the plant which in role result in increasing the crop harvest. Nine common types of tomato leaf diseases have a great effect on the quality and quantity of tomato crop yield. The tradition approaches of features extraction and image classification cannot ensure a high accuracy rate for leaf diseases identification. This paper suggests an automatic detection approach for tomato leaf diseases based on the fine tuning and transfer learning to the pre-trained of deep Convolutional Neural Networks. Three pre-trained deep networks based on transfer learning: AlexNet, VGG-16 Net and SqueezeNet are suggested for their performances analysis in tomato leaf diseases classification. The proposed networks are carried out on two different dataset, one of them is a small dataset using only four different diseases while the other is a large dataset of leaves accompanied with symptoms of nine diseases and healthy leaves. The performance of the suggested networks is evaluated in terms of classification accuracy and the elapsed time during their training. The performance of the suggested networks using the small dataset are also compared with that of the-state-of-the-art technique in literature. The experimental results with the small dataset demonstrate that the accuracy of classification of the suggested networks outperform by 8.1% and 15% over the classification accuracy of the technique in literature. On other side when using the large dataset, the proposed pre-trained AlexNet achieves high classification accuracy by 97.4% and the consuming time during its training is lower than those of the other pre-trained networks. Generally, it can be concluded that AlexNet has outstanding performance for diagnosing the tomato leaf diseases in terms of accuracy and execution time compared to the other networks. On contrary, the performance of VGG-16 Net in metric of classification accuracy is the best yet the largest consuming time among other networks.

A new neighbourhood formation approach for solving cold-start user problem in collaborative filtering

Collaborative filtering (CF) is the most widely accepted recommendation technique. Despite its popularity, this approach faces some major challenges like that of a cold-start user problem where a user has rated a handful of items. Due to very few ratings available for the cold-start users, their similarities with rest of the users has been questioned in the past, none have focused on their approach for neighbour identification. Whilst the traditional CF approaches select only those similar users as neighbours who have rated the item under consideration, the neighbourhood comprises of weak neighbours of the cold-start users. To address this shortcoming, our proposed approach selects neighbours with highest similarity irrespective of their availability of ratings for that item. Moreover, for the selected similar neighbours with missing ratings, an item based regression is performed to partially populate the matrix. The efficacy of the proposed neighbourhood formation approach addressing cold-start user problem is validated on two publicly available MovieLens datasets. Our approach provides superior quality of recommendations evaluated on a range of prediction and classification accuracy metrics. The results are encouraging particularly for systems having higher percentage of cold-start users which indicates the effectiveness of our approach in practical settings of new internet portals.

A new neighbourhood formation approach for solving cold-start user problem in collaborative filtering

Collaborative filtering (CF) is the most widely accepted recommendation technique. Despite its popularity, this approach faces some major challenges like that of a cold-start user problem where a user has rated a handful of items. Due to very few ratings available for the cold-start users, their similarities with rest of the users has been questioned in the past, none have focused on their approach for neighbour identification. Whilst the traditional CF approaches select only those similar users as neighbours who have rated the item under consideration, the neighbourhood comprises of weak neighbours of the cold-start users. To address this shortcoming, our proposed approach selects neighbours with highest similarity irrespective of their availability of ratings for that item. Moreover, for the selected similar neighbours with missing ratings, an item based regression is performed to partially populate the matrix. The efficacy of the proposed neighbourhood formation approach addressing cold-start user problem is validated on two publicly available MovieLens datasets. Our approach provides superior quality of recommendations evaluated on a range of prediction and classification accuracy metrics. The results are encouraging particularly for systems having higher percentage of cold-start users which indicates the effectiveness of our approach in practical settings of new internet portals.

A machine learning approach to detect crude oil contamination in a real scenario using hyperspectral remote sensing

One of the most ubiquitous and detrimental types of environmental contamination in the world is crude oil pollution. When released into either the aquatic or terrestrial environments, this pollution can negatively impact flora and fauna, as well as human health. Hence, a rapid and affordable spatial assessment of the pollution is favored to limit the spill’s effects. Using airborne hyperspectral remote sensing (HRS) for crude oil detection in terrestrial areas has been investigated in previous studies, which mainly relied on heavily oiled artificial samples. These studies and others based their methodologies on the premise that the spectral features of petroleum hydrocarbon (PHC) are clearly observable, which might not be true in all cases. In this study, we aimed at assessing the true potential of using HRS for terrestrial oil spill mapping in a real disaster site in southern Israel, where laboratory and controlled conditions do not apply. Using the AISA SPECIM Fenix1 K sensor, we collected airborne image of the study site and analyzed the data with advanced data mining techniques. Various challenges and limitations arose from the airborne HRS image being taken two and a half years after the crude oil had been released into the environment and exposed to the surface. Here, no spectral features of PHC were detectable in the spectrum, preventing the use of PHC indices and spectral methods developed by others. Nevertheless, by using standardization techniques, vicarious band selection, dimension reduction, multivariate calibration, and supervised machine-learning, we were able to successfully distinguish between contaminated pixels from non-contaminated ones. Classification accuracy metrics of overall accuracy, sensitivity, specificity, and Kappa yielded good results of 0.95, 0.95, 0.95 and 0.9, respectively, for cross-validation, and 0.93, 0.91, 0.94 and 0.85, for the validation dataset. Classified image and test scenes also showed strong agreement with an orthophoto image taken several days after the disaster, when the pollution was clearly visible. Thus, we conclude that HRS technology can detect PHC traces in an oil spill site, even under the most challenging conditions.

Use of visible spectrum sUAS photography for land cover classification at nest sites of a declining bird species (Falco sparverius)

AbstractPhotography with small unmanned aircraft systems (sUAS) offers opportunities for researchers to better understand habitat selection in wildlife, especially for species that select habitat from an aerial perspective (e.g., many bird species). The growing number of commercial sUAS being flown by recreational users represents a potentially valuable source of data for documenting and studying wildlife habitat. We used a commercially available quadcopter sUAS with a visible spectrum camera to classify habitat for American Kestrels (Falco sparverius; Aves), as well as to evaluate aspects of image processing and postprocessing relevant to a simple habitat analysis using citizen science photography. We investigated inter–observer repeatability of habitat classification, effectiveness of cross‐image classification and Gaussian filtering, and sensitivity to classification resolution. We photographed vegetation around nests from both 25 m and 50 m above takeoff elevation, and analyzed images via maximum likelihood supervised classification. Our results indicate that commercial off‐the‐shelf sUAS photography can distinguish between grass, herbaceous, woody, bare ground, and human‐modified cover classes with good (kappa > 0.6) or strong (kappa > 0.8) accuracy using a 0.25 m2 minimum patch size for aggregation. There was inter‐subject variability in designating training samples, but high repeatability of supervised classification accuracy. Gaussian filtering reduced classification accuracy, while coarser classification resolution out‐performed finer resolution due to “speckling noise.” Image self‐classification significantly outperformed cross‐image classification. Mean classification accuracy metrics (kappa values) across different photo heights differed little, but, importantly, the rank order of images differed noticeably.

Accurate Classification of EEG Signals Using Neural Networks Trained by Hybrid Population-physic-based Algorithm

A brain-computer interface (BCI) system is one of the most effective ways that translates brain signals into output commands. Different imagery activities can be classified based on the changes in μ and β rhythms and their spatial distributions. Multi-layer perceptron neural networks (MLP-NNs) are commonly used for classification. Training such MLP-NNs has great importance in a way that has attracted many researchers to this field recently. Conventional methods for training NNs, such as gradient descent and recursive methods, have some disadvantages including low accuracy, slow convergence speed and trapping in local minimums. In this paper, in order to overcome these issues, the MLP-NN trained by a hybrid population-physics-based algorithm, the combination of particle swarm optimization and gravitational search algorithm (PSOGSA), is proposed for our classification problem. To show the advantages of using PSOGSA that trains NNs, this algorithm is compared with other meta-heuristic algorithms such as particle swarm optimization (PSO), gravitational search algorithm (GSA) and new versions of PSO. The metrics that are discussed in this paper are the speed of convergence and classification accuracy metrics. The results show that the proposed algorithm in most subjects of encephalography (EEG) dataset has very better or acceptable performance compared to others.

Comparative analysis of selected classifiers in posterior cruciate ligaments computer aided diagnosis

Abstract A study on computer aided diagnosis of posterior cruciate ligaments is presented in this paper. The diagnosis relies on T1-weighted magnetic resonance imaging. During the image analysis stage, the ligament region is automatically detected, localized, and extracted using fuzzy segmentation methods. Eight geometric features are defined for the ligament object. With a clinical reference database containing 107 cases of both healthy and pathological cases, a Fisher linear discriminant is used to select 4 most distinctive features. At the classification stage we employ five different soft computing classifiers to evaluate the feature vector suitability for the computerized ligament diagnosis. Among the classifiers we introduce and specify the particle swarm optimization based Sugeno-type fuzzy inference system and compare its performance to other established classification systems. The classification accuracy metrics: sensitivity, specificity, and Dice index all exceed 90% for each classifier under consideration, indicating high level of the proposed feature vector relevance in the computer aided ligaments diagnosis.

Detection of Moving Objects Using Fuzzy Color Difference Histogram Based Background Subtraction

Detection of moving objects in the presence of complex scenes such as dynamic background (e.g, swaying vegetation, ripples in water, spouting fountain), illumination variation, and camouflage is a very challenging task. In this context, we propose a robust background subtraction technique with three contributions. First, we present the use of color difference histogram (CDH) in the background subtraction algorithm. This is done by measuring the color difference between a pixel and its neighbors in a small local neighborhood. The use of CDH reduces the number of false errors due to the non-stationary background, illumination variation and camouflage. Secondly, the color difference is fuzzified with a Gaussian membership function. Finally, a novel fuzzy color difference histogram (FCDH) is proposed by using fuzzy c-means (FCM) clustering and exploiting the CDH. The use of FCM clustering algorithm in CDH reduces the large dimensionality of the histogram bins in the computation and also lessens the effect of intensity variation generated due to the fake motion or change in illumination of the background. The proposed algorithm is tested with various complex scenes of some benchmark publicly available video sequences. It exhibits better performance over the state-of-the-art background subtraction techniques available in the literature in terms of classification accuracy metrics like $MCC$ and $PCC$ .

Fault diagnosis in DSL networks using support vector machines

The adequate operation for a number of service distribution networks relies on the effective maintenance and fault management of their underlay DSL infrastructure. Thus, new tools are required in order to adequately monitor and further diagnose anomalies that other segments of the DSL network cannot identify due to the pragmatic issues raised by hardware or software misconfigurations. In this work we present a fundamentally new approach for classifying known DSL-level anomalies by exploiting the properties of novelty detection via the employment of one-class Support Vector Machines (SVMs). By virtue of the imbalance residing in the training samples that consequently lead to problematic prediction outcomes when used within two-class formulations, we adopt the properties of one-class classification and construct models for independently identifying and classifying a single type of a DSL-level anomaly. Given the fact that the greater number of the installed Digital Subscriber Line Access Multiplexers (DSLAMs) within the DSL network of a large European ISP were misconfigured, thus unable to accurately flag anomalous events, we utilize as inference solutions the models derived by the one-class SVM formulations built by the known labels as flagged by the much smaller number of correctly configured DSLAMs in the same network in order to aid the classification aspect against the monitored unlabeled events. By reaching an average over 95% on a number of classification accuracy metrics such as precision, recall and F-score we show that one-class SVM classifiers overcome the biased classification outcomes achieved by the traditional two-class formulations and that they may constitute as viable and promising components within the design of future network fault management strategies. In addition, we demonstrate their superiority over commonly used two-class machine learning approaches such as Decision Trees and Bayesian Networks that has been used in the same context within past solutions.

An Evaluation System for Building Footprint Extraction From Remotely Sensed Data

This study proposes a multi-criteria and hierarchical evaluation system for building extraction from remotely sensed data. Most of current evaluation methods are focused on classification accuracy, while the other dimensions of extraction accuracy are usually ignored. The proposed evaluation system consists of three components: 1) the matched rate, including evaluation metrics for the traditional classification accuracy (e.g., completeness, correctness, and quality); 2) the shape similarity that describes the resemblance between reference and extracted buildings, including image-based and polygon-based metrics; and 3) the positional accuracy which is measured by distances at feature points such as building's centroid. The system also hierarchically evaluates extracted buildings at per-building, per-scene, and overall levels. To reduce the redundancy among different metrics, principal component analysis and correlation analysis are employed for metrics selection and aggregation. Four different building extraction methods, using high-resolution optical imagery and/or LiDAR data, are implemented to test the proposed system. The experiment demonstrates that the proposed system is more consistent with human vision compared to traditional classification accuracy metrics. This system can highlight perceptible differences between the extracted building footprints and the reference data, even if this difference is insignificant measured by the traditional metrics.