Classification Performance Metrics Research Articles

Real-Time Heart Arrhythmia Detection Using Apache Spark Structured Streaming.

One of the major causes of death in the world is cardiac arrhythmias. In the field of healthcare, physicians use the patient's electrocardiogram (ECG) records to detect arrhythmias, which indicate the electrical activity of the patient's heart. The problem is that the symptoms do not always appear and the physician may be mistaken in the diagnosis. Therefore, patients need continuous monitoring through real-time ECG analysis to detect arrhythmias in a timely manner and prevent an eventual incident that threatens the patient's life. In this research, we used the Structured Streaming module built top on the open-source Apache Spark platform for the first time to implement a machine learning pipeline for real-time cardiac arrhythmias detection and evaluate the impact of using this new module on classification performance metrics and the rate of delay in arrhythmia detection. The ECG data collected from the MIT/BIH database for the detection of three class labels: normal beats, RBBB, and atrial fibrillation arrhythmias. We also developed three decision trees, random forest, and logistic regression multiclass classifiers for data classification where the random forest classifier showed better performance in classification than the other two classifiers. The results show previous results in performance metrics of the classification model and a significant decrease in pipeline runtime by using more class labels compared to previous studies.

Towards Predicting Student’s Dropout in University Courses Using Different Machine Learning Techniques

Early and precisely predicting the students’ dropout based on available educational data belongs to the widespread research topic of the learning analytics research field. Despite the amount of already realized research, the progress is not significant and persists on all educational data levels. Even though various features have already been researched, there is still an open question, which features can be considered appropriate for different machine learning classifiers applied to the typical scarce set of educational data at the e-learning course level. Therefore, the main goal of the research is to emphasize the importance of the data understanding, data gathering phase, stress the limitations of the available datasets of educational data, compare the performance of several machine learning classifiers, and show that also a limited set of features, which are available for teachers in the e-learning course, can predict student’s dropout with sufficient accuracy if the performance metrics are thoroughly considered. The data collected from four academic years were analyzed. The features selected in this study proved to be applicable in predicting course completers and non-completers. The prediction accuracy varied between 77 and 93% on unseen data from the next academic year. In addition to the frequently used performance metrics, the comparison of machine learning classifiers homogeneity was analyzed to overcome the impact of the limited size of the dataset on obtained high values of performance metrics. The results showed that several machine learning algorithms could be successfully applied to a scarce dataset of educational data. Simultaneously, classification performance metrics should be thoroughly considered before deciding to deploy the best performance classification model to predict potential dropout cases and design beneficial intervention mechanisms.

딥러닝과 PDF 객체분석을 이용한 문서형 악성코드 탐지

Document-type malware is mainly used for APT(Advanced Persistent Threats) attacks using document files, and malicious code threats targeting PDF documents have been rapidly increasing recently by phishing mail related to Covid-19. Recently, document type malware is easy to bypass existing security programs, so we propose detecting malware using static analysis and deep learning. In this paper, we construct a malicious PDF detection model into deep learning by extracting the information and frequency of keywords that exist between objects in normal, malicious PDF files. Evaluation of the classification performance metrics for the proposed method showed 98.75% accuracy for the Random Forest model and 98.33% accuracy for the Support Vector Machine model. The keywords of PDFs used as feature information in this study are insufficient to change, can extract information even when compressed or obfuscated and can respond effectively to variant malware because deep learning is used.

The impact of using biased classification performance metrics on software defect prediction research

The impact of using biased classification performance metrics on software defect prediction research

Classifier uncertainty: evidence, potential impact, and probabilistic treatment.

Classifiers are often tested on relatively small data sets, which should lead to uncertain performance metrics. Nevertheless, these metrics are usually taken at face value. We present an approach to quantify the uncertainty of classification performance metrics, based on a probability model of the confusion matrix. Application of our approach to classifiers from the scientific literature and a classification competition shows that uncertainties can be surprisingly large and limit performance evaluation. In fact, some published classifiers may be misleading. The application of our approach is simple and requires only the confusion matrix. It is agnostic of the underlying classifier. Our method can also be used for the estimation of sample sizes that achieve a desired precision of a performance metric.

Interpretable Machine Learning Models for Three-Way Classification of Cognitive Workload Levels for Eye-Tracking Features.

The paper is focussed on the assessment of cognitive workload level using selected machine learning models. In the study, eye-tracking data were gathered from 29 healthy volunteers during examination with three versions of the computerised version of the digit symbol substitution test (DSST). Understanding cognitive workload is of great importance in analysing human mental fatigue and the performance of intellectual tasks. It is also essential in the context of explanation of the brain cognitive process. Eight three-class classification machine learning models were constructed and analysed. Furthermore, the technique of interpretable machine learning model was applied to obtain the measures of feature importance and its contribution to the brain cognitive functions. The measures allowed improving the quality of classification, simultaneously lowering the number of applied features to six or eight, depending on the model. Moreover, the applied method of explainable machine learning provided valuable insights into understanding the process accompanying various levels of cognitive workload. The main classification performance metrics, such as F1, recall, precision, accuracy, and the area under the Receiver operating characteristic curve (ROC AUC) were used in order to assess the quality of classification quantitatively. The best result obtained on the complete feature set was as high as 0.95 (F1); however, feature importance interpretation allowed increasing the result up to 0.97 with only seven of 20 features applied.

Naive Bayesian fusion based deep learning networks for multisegmented classification of fishes in aquaculture industries

Fish classification is an essential requirement for biomass estimation, disease identification, and quality analysis. In aquaculture industries, fish classification is carried out in the processing unit. The fishes are out of water, subjecting them to structural deformation and orientation misalignments, makes classification challenging. A multisegmented fish classification technique using deep learning networks with naive Bayesian type fusion is proposed in this work to address these challenges. Fish images are acquired using an overhead camera. The fish head is identified by observing a minimal convexity deficiency region to facilitate segmentation. A multi-stage exhaustive enumerative optimization method is used to adjust the orientation, which can minimize unwanted background region in the image segment. Fish head, scales, and body are segmented from the fish image. For each fish segment, AlexNet is trained by using the transfer learning approach. A naive Bayesian fusion layer is introduced to fuse these trained deep learning networks and enhance classification accuracy. Experimental results illustrate a classification accuracy of 98.64% for ‘Fish-Pak’ image dataset with six different fish species and 98.94% for BYU fish dataset with four species. Comparative analysis with standard networks and ablation study demonstrates the accuracy and robustness of the proposed fusion architecture, respectively. Various fusion layers have also been analyzed, and observations illustrate the accuracy of the proposed NBC layer. Significant improvements in other classification performance metrics were also observed.

Development of a Complication- and Treatment-Aware Prediction Model for Favorable Functional Outcome in Aneurysmal Subarachnoid Hemorrhage Based on Machine Learning.

Current prognostic tools in aneurysmal subarachnoid hemorrhage (aSAH) are constrained by being primarily based on patient and disease characteristics on admission. To develop and validate a complication- and treatment-aware outcome prediction tool in aSAH. This cohort study included data from an ongoing prospective nationwide multicenter registry on all aSAH patients in Switzerland (Swiss SOS [Swiss Study on aSAH]; 2009-2015). We trained supervised machine learning algorithms to predict a binary outcome at discharge (modified Rankin scale [mRS]≤3: favorable; mRS 4-6: unfavorable). Clinical and radiological variables on admission ("Early" Model) as well as additional variables regarding secondary complications and disease management ("Late" Model) were used. Performance of both models was assessed by classification performance metrics on an out-of-sample test dataset. Favorable functional outcome at discharge was observed in 1156 (62.0%) of 1866 patients. Both models scored a high accuracy of 75% to 76% on the test set. The "Late" outcome model outperformed the "Early" model with an area under the receiver operator characteristics curve (AUC) of 0.85vs 0.79, corresponding to a specificity of 0.81vs 0.70 and a sensitivity of 0.71vs 0.79, respectively. Both machine learning models show good discrimination and calibration confirmed on application to an internal test dataset of patients with a wide range of disease severity treated in different institutions within a nationwide registry. Our study indicates that the inclusion of variables reflecting the clinical course of the patient may lead to outcome predictions with superior predictive power compared to a model based on admission data only.

Minimum Relevant Features to Obtain Explainable Systems for Predicting Cardiovascular Disease Using the Statlog Data Set

Learning systems have been focused on creating models capable of obtaining the best results in error metrics. Recently, the focus has shifted to improvement in the interpretation and explanation of the results. The need for interpretation is greater when these models are used to support decision making. In some areas, this becomes an indispensable requirement, such as in medicine. The goal of this study was to define a simple process to construct a system that could be easily interpreted based on two principles: (1) reduction of attributes without degrading the performance of the prediction systems and (2) selecting a technique to interpret the final prediction system. To describe this process, we selected a problem, predicting cardiovascular disease, by analyzing the well-known Statlog (Heart) data set from the University of California’s Automated Learning Repository. We analyzed the cost of making predictions easier to interpret by reducing the number of features that explain the classification of health status versus the cost in accuracy. We performed an analysis on a large set of classification techniques and performance metrics, demonstrating that it is possible to construct explainable and reliable models that provide high quality predictive performance.

Machine Learning Versus Usual Care for Diagnostic and Prognostic Prediction in the Emergency Department: A Systematic Review.

Having shown promise in other medical fields, we sought to determine whether machine learning (ML) models perform better than usual care in diagnostic and prognostic prediction for emergency department (ED) patients. In this systematic review, we searched MEDLINE, Embase, Central, and CINAHL from inception to October 17, 2019. We included studies comparing diagnostic and prognostic prediction of ED patients by ML models to usual care methods (triage-based scores, clinical prediction tools, clinician judgment) using predictor variables readily available to ED clinicians. We extracted commonly reported performance metrics of model discrimination and classification. We used the PROBAST tool for risk of bias assessment (PROSPERO registration: CRD42020158129). The search yielded 1,656 unique records, of which 23 studies involving 16,274,647 patients were included. In all seven diagnostic studies, ML models outperformed usual care in all performance metrics. In six studies assessing in-hospital mortality, the best-performing ML models had better discrimination (area under the receiver operating characteristic curve [AUROC]=0.74-0.94) than any clinical decision tool (AUROC=0.68-0.81). In four studies assessing hospitalization, ML models had better discrimination (AUROC=0.80-0.83) than triage-based scores (AUROC=0.68-0.82). Clinical heterogeneity precluded meta-analysis. Most studies had high risk of bias due to lack of external validation, low event rates, and insufficient reporting of calibration. Our review suggests that ML may have better prediction performance than usual care for ED patients with a variety of clinical presentations and outcomes. However, prediction model reporting guidelines should be followed to provide clinically applicable data. Interventional trials are needed to assess the impact of ML models on patient-centered outcomes.

Analysis and classification of kidney ultrasound images using SIFT features with neural network classifier

A unique method to analyse ultrasound scan images of kidney to classify renal abnormalities using SIFT features and artificial neural network is presented in this paper. The ultrasound kidney images are classified into four classes normal, cyst, calculi, and tumour. Preprocessing and denoising techniques are applied for the removal of speckle noise by applying median and Wiener filter. Segmentation to obtain region of interest (ROI) is carried out by applying fuzzy C-means clustering technique. Region of interest specifies to the objects such as kidney, calculi/stones, cysts and tumour inside the kidney present in the ultrasound scan images. First order statistical features and GLCM features are computed. To overcome the operator dependency of ultrasound scanning procedure rotational variance SIFT algorithm is applied. SIFT descriptors as features are obtained. Abnormalities are classified using supervised learning neural network algorithm (ANN). Classifier performance metrics are higher when hidden neurons are fixed as 101.

Development of a Stope Stability Prediction Model Using Ensemble Learning Techniques - A Case Study

The consequences of collapsed stopes can be dire in the mining industry. This can lead to the revocation of a mining license in most jurisdictions, especially when the harm costs lives. Therefore, as a mine planning and technical services engineer, it is imperative to estimate the stability status of stopes. This study has attempted to produce a stope stability prediction model adopted from stability graph using ensemble learning techniques. This study was conducted using 472 case histories from 120 stopes of AngloGold Ashanti Ghana, Obuasi Mine. Random Forest, Gradient Boosting, Bootstrap Aggregating and Adaptive Boosting classification algorithms were used to produce the models. A comparative analysis was done using six classification performance metrics namely Accuracy, Precision, Sensitivity, F1-score, Specificity and Mathews Correlation Coefficient (MCC) to determine which ensemble learning technique performed best in predicting the stability of a stope. The Bootstrap Aggregating model obtained the highest MCC score of 96.84% while the Adaptive Boosting model obtained the lowest score. The Specificity scores in decreasing order of performance were 98.95%, 97.89%, 96.32% and 95.26% for Bootstrap Aggregating, Gradient Boosting, Random Forest and Adaptive Boosting respectively. The results showed equal Accuracy, Precision, F1-score and Sensitivity score of 97.89% for the Bootstrap Aggregating model while the same observation was made for Adaptive Boosting, Gradient Boosting and Random Forest with 90.53%, 92.63% and 95.79% scores respectively. At a 95% confidence interval using Wilson Score Interval, the results showed that the Bootstrap Aggregating model produced the minimal error and hence was selected as the alternative stope design tool for predicting the stability status of stopes.
 
 Keywords: Stope Stability, Ensemble Learning Techniques, Stability Graph, Machine Learning

Classification of Prostate Cancer and Determination of Related Factors with Different Artificial Neural Network

Objective: In this study, it is aimed to classify prostate cancer, compare the predictions of these two models and determine the factors associated with the disease by applying Multilayer Perceptron Neural Network (MLPNN) and Radial-Based Function Neural Network (RBFNN) methods on the open access Prostate cancer dataset.Methods: In this study, the dataset named "Prostate Cancer Data Set" was used by obtaining from https://www.kaggle.com/sajidsaifi/prostate-cancer address. To classify prostate cancer, MLPNN and RBFNN methods, which are artificial neural network models, is used. The classification performance of the models was evaluated with the sensitivity, specificity, accuracy, negative predictive value and positive predictive value, which are among the classification performance metrics. Prostate cancer related factors were estimated by using MLPNN and RBFNN models.Results: With the applied MLPNN model, performance metric values were obtained as AUC 0.937, Sensitivity 100%, accuracy 92.5%, Selectivity 84.6%, Positive predictive value 87.5% and Negative predictive value 100%. With the RBFNN model, the performance metric values were obtained as AUC 0.921, Sensitivity 83.3%, accuracy 86.6%, Selectivity 91.6%, Positive predictive value 93.7% and Negative predictive value 78.5%. When the effects of variables in the dataset in this study on prostate cancer are examined; The three most important variables for the MLPNN model were obtained as perimeter, area and compactness, respectively. For the RBFNN model, the three most important variables were obtained as perimeter, area and compactness, respectively.Conclusion: It was seen that MLPNN and RBFNN models used in this study gave successful predictions in the classification of prostate cancer. In addition, estimating the significance values of factors associated with the disease with these classification models made it different from similar studies with the same dataset.

A novel diagnostic map for computer‐aided diagnosis of skin cancer

In teledermoscopy, images are transmitted through a communication channel to the medical facility for medical consultation. This yields to bandwidth congestion and consumption of large storage size which impairs the transmission of the high-resolution dermoscopy image. This study proposes a novel technique by generating a diagnostic feature map, named diagonal compressive sensing (CS) features map. This proposed map is generated using the significant diagnostic features by aligning the feature vector in a diagonal map, which is then compressed using CS technique. Eventually, the recovered feature map at the receiver side is applied to the classification system for decision-making. In addition, physicians at the receiver side who were trained to read the feature maps can verify the classification decision, and then provide feedback to the patient. The results demonstrated that the proposed diagnostic map achieved less transmission time due to the small size of the feature map along with the compression process. Furthermore, the feature map has drastically improved the classification performance metrics, including the accuracy, which increased, for example, from 88.6% to 98% at 80% compression ratio compared to the traditional method on the compressed whole image.

An Efficient Deep-Learning-Based Detection and Classification System for Cyber-Attacks in IoT Communication Networks

With the rapid expansion of intelligent resource-constrained devices and high-speed communication technologies, the Internet of Things (IoT) has earned wide recognition as the primary standard for low-power lossy networks (LLNs). Nevertheless, IoT infrastructures are vulnerable to cyber-attacks due to the constraints in computation, storage, and communication capacity of the endpoint devices. From one side, the majority of newly developed cyber-attacks are formed by slightly mutating formerly established cyber-attacks to produce a new attack that tends to be treated as normal traffic through the IoT network. From the other side, the influence of coupling the deep learning techniques with the cybersecurity field has become a recent inclination of many security applications due to their impressive performance. In this paper, we provide the comprehensive development of a new intelligent and autonomous deep-learning-based detection and classification system for cyber-attacks in IoT communication networks that leverage the power of convolutional neural networks, abbreviated as IoT-IDCS-CNN (IoT based Intrusion Detection and Classification System using Convolutional Neural Network). The proposed IoT-IDCS-CNN makes use of high-performance computing that employs the robust Compute Unified Device Architectures (CUDA) based Nvidia GPUs (Graphical Processing Units) and parallel processing that employs high-speed I9-core-based Intel CPUs. In particular, the proposed system is composed of three subsystems: a feature engineering subsystem, a feature learning subsystem, and a traffic classification subsystem. All subsystems were developed, verified, integrated, and validated in this research. To evaluate the developed system, we employed the Network Security Laboratory-Knowledge Discovery Databases (NSL-KDD) dataset, which includes all the key attacks in IoT computing. The simulation results demonstrated a greater than 99.3% and 98.2% cyber-attack classification accuracy for the binary-class classifier (normal vs. anomaly) and the multiclass classifier (five categories), respectively. The proposed system was validated using a K-fold cross-validation method and was evaluated using the confusion matrix parameters (i.e., true negative (TN), true positive (TP), false negative (FN), false positive (FP)), along with other classification performance metrics, including precision, recall, F1-score, and false alarm rate. The test and evaluation results of the IoT-IDCS-CNN system outperformed many recent machine-learning-based IDCS systems in the same area of study.

Machine learning predictivity applied to consumer creditworthiness

Credit risk evaluation has a relevant role to financial institutions, since lending may result in real and immediate losses. In particular, default prediction is one of the most challenging activities for managing credit risk. This study analyzes the adequacy of borrower’s classification models using a Brazilian bank’s loan database, and exploring machine learning techniques. We develop Support Vector Machine, Decision Trees, Bagging, AdaBoost and Random Forest models, and compare their predictive accuracy with a benchmark based on a Logistic Regression model. Comparisons are analyzed based on usual classification performance metrics. Our results show that Random Forest and Adaboost perform better when compared to other models. Moreover, Support Vector Machine models show poor performance using both linear and nonlinear kernels. Our findings suggest that there are value creating opportunities for banks to improve default prediction models by exploring machine learning techniques.

Evaluation of European Society of Cardiology (ESC) 0/1-hour algorithm in the diagnosis of 90-day major adverse cardiovascular events: a multicenter United States cohort study

Abstract Background The ESC 0/1-hour algorithm using high sensitivity troponin (ESC 0/1-h) is a rapid triage protocol for diagnosing acute coronary syndrome, however the classification performance of the algorithm in the US population is uncertain. Further, evidence for the use of ESC 0/1-h in the long-term diagnosis of major adverse cardiovascular events (MACE) remains limited. Purpose To evaluate the performance of the ESC 0/1-h algorithm in diagnosing 90-day MACE in a diverse US cohort. Methods In this prospective, multicenter, observational cohort study, adult emergency department patients who were evaluated for suspected ACS were enrolled at eight sites in the US. Serial 1-hour blood samples were collected and high sensitivity troponin T (hs-cTnT) concentrations were measured in a central laboratory using the hs-cTnT assay. Primary outcome included major adverse cardiovascular events (MACE) within 90 (+30) days of enrollment. MACE defined as myocardial infarction (MI), cardiovascular or uncertain death, and coronary revascularization. Presence of MI and cardiovascular death were adjudicated by independent reviewers blinded to hs-cTnT results. Each participant was stratified to one of three risk groups as determined by ESC 0/1-h algorithm. Diagnostic classification performance metrics with exact confidence intervals (i.e. negative predictive value [NPV], positive predicted value [PPV], sensitivity, and specificity) were evaluated for each risk group where appropriate. Results Among 1430 eligible participants, 45.8% (655/1430) were women and 36.6% (524/1430) were African American with a mean age of 57.6±12.8 years. MACE at 90-days occurred in 15.5% (221/1430). ESC 0/1-h stratified 59.5% (851/1430) subjects in Rule-Out range and 13.0% (186/1430) subjects in Rule-In range. The Rule-Out criteria had an NPV and sensitivity for 90-day MACE of 96.8% (95% CI: 95.4–97.9%) and 87.8% (95% CI: 82.7–91.8%), respectively. For 90-day cardiovascular death or MI, Rule-Out criteria had an NPV of 98.2% (95% CI: 97.1–99.0%) and sensitivity was 92.4% (95% CI: 87.8–95.7%). The Rule-In criteria had a PPV of 60.8% (95% CI: 53.3–67.8%) for both outcomes. Rule-In criteria had a specificity for 90-day MACE and 90-day cardiovascular death or MI of 94.0% (95% CI: 92.5–95.2%) and 94.1% (95% CI: 92.6–95.3%), respectively. Among the 27.5% (393/1430) participants classified in neither risk groups, the prevalence of 90-day MACE was 20.6% (81/393) and the prevalence of 90-day cardiovascular death or MI was 17.8% (70/393) Conclusion In a prospective, multicenter, US cohort, the ESC 0/1-h algorithm was unable to achieve a sufficiently high NPV to safely exclude the diagnosis of MACE within 90 days after emergency department presentation. New hs-cTn algorithms specific to the US population may be warranted. ESC 0/1-h 90 Day Outcomes Funding Acknowledgement Type of funding source: Private company. Main funding source(s): Roche Diagnostics

EFS-LSTM (Ensemble-Based Feature Selection With LSTM) Classifier for Intrusion Detection System

In this article, an EFS-LSTM, a deep recurrent learning model, is proposed for network intrusion detection systems. The EFS-LSTM model uses ensemble-based feature selection (EFS) and LSTM (Long Short Term Memory) for the classification of network intrusions. The EFS combines five feature selection mechanisms namely, information gain, gain ratio, chi-square, correlation-based feature selection, and symmetric uncertainty-based feature selection. The experiments were conducted using the benchmark NSL-KDD dataset and implemented using Tensor flow and python. The EFS-LSTM classifier is evaluated using the classification performance metrics and also compared with all the 41 features without any feature selection as well as with each individual feature selection techniques and classified using LSTM. The performance study showed that the EFS-LSTM model outperforms better with 99.8% accuracy with a higher detection and less false alarm rates.

An efficient and robust deep learning based network anomaly detection against distributed denial of service attacks

The number of devices connected to the Internet is increasing day by day. This increase causes cyber-attacks to be larger and more complex. It is important to sdetect the anomalies rapidly when there is a cyber-attack. In detecting anomalies, high false positive rate is obtained by using feature extraction based on statistical calculations and machine learning algorithms. In proposed approach, the measured values obtained from the network are normalized between 0 and 1. These values applied to autoencoder model trained with optimum hyper parameters. This model contributes to feature learning and dimensional reduction. Support vector machines effectively differentiate between normal and DDOS attack traffic by using these features. The CICIDS dataset and virtually generated DDOS traffic are used to validate the proposed approach and measure its performance. The results show that the proposed approach speeds up training and testing times and performs better classification performance metrics than most previous approaches. The novelty of the study is that AE-SVM trained with CICIDS successfully captures virtually generated DDOS traffic data. Despite the unbalanced data set, 99.1% test success was achieved in detection of DDOS traffic which is produced with Kali Linux. This success contributed to the solution of the high false-positive problem compared to other models.

Multi‐objectives optimisation of features selection for the classification of thyroid nodules in ultrasound images

Ultrasound (US) imaging is the leading diagnostic method for assessing the early-stage thyroid nodule. However, the visual evaluation of nodules can be influenced by the subjectivity of radiologists' interpretations. Computer-aided Diagnostic (CAD) systems can be useful in classifying these nodules according to their benign or malignant nature. The extraction of the characteristics, which relate in the author's case to the US of thyroid nodules, is essential in the differentiation of these nodules. The complex nature of images, however, generates a significant number of features, many of which are either redundant or irrelevant. This study presents a new CAD system that has been developed to categorise thyroid nodules. In this survey, 447 US images of thyroid nodules were retained. These images were used to extract features using statistical features extraction methods. A feature selection method based on the multi objective particle swarm optimisation algorithm was used to choose the most relevant and non-redundant ones. Then, support vector machine (SVM) and random forests (RFs) were applied to classify these nodules. 10-fold cross-validation was used to assess the classification performance metrics. Their proposed CAD has reached a maximum accuracy of 94.28% for SVM; and 96.13% for RF using the contour-based ROI.