Abstract
Problem statement: Typically, the accuracy metric is often applied for optimizing the heuristic or stochastic classification models. However, the use of accuracy metric might lead the searching process to the sub-optimal solutions due to its less discriminating values and it is also not robust to the changes of class distribution. Approach: To solve these detrimental effects, we propose a novel performance metric which combines the beneficial properties of accuracy metric with the extended recall and precision metrics. We call this new performance metric as Optimized Accuracy with Recall-Precision (OARP). Results: In this study, we demonstrate that the OARP metric is theoretically better than the accuracy metric using four generated examples. We also demonstrate empirically that a naive stochastic classification algorithm, which is Monte Carlo Sampling (MCS) algorithm trained with the OARP metric, is able to obtain better predictive results than the one trained with the conventional accuracy metric. Additionally, the t-test analysis also shows a clear advantage of the MCS model trained with the OARP metric over the accuracy metric alone for all binary data sets. Conclusion: The experiments have proved that the OARP metric leads stochastic classifiers such as the MCS towards a better training model, which in turn will improve the predictive results of any heuristic or stochastic classification models.
Highlights
To date, many efforts have been carried out to design more advanced algorithms to solve classification problems
This study introduces a new hybridized performance metric that is derived from the combination of accuracy metric with the extended precision and recall metrics
Experimental setup: We have theoretically showed that the new performance metric, Optimized Accuracy with Recall-Precision (OARP) was better than the accuracy metric in selecting and discriminating better solutions using four examples
Summary
Many efforts have been carried out to design more advanced algorithms to solve classification problems. The development of appropriate performance metrics to evaluate the classification performance are at least as importance as algorithm. The use of performance metric during the training stage is to optimize the classifier (Ferri et al, 2002; Ranawana and Palade, 2006). In this particular stage, the performance metric is used to discriminate and to select the optimal solution which can produce a more accurate prediction of future performance. In the testing stage, the performance metric is usually employed for comparing and evaluating the classification models (Bradley, 1997; Caruana and Niculescu-Mizil, 2004; Kononenko and Bratko, 1991; Provost and Domingos, 2003; Seliya et al, 2009)
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
