Vardeman, S. B. and Morris, M. D. (2013), “Majority Voting by Independent Classifiers can Increase Error Rates,” The American Statistician, 67, 94-96: Comment by Baker, Xu, Hu, and Huang and Reply

Abstract

Vardeman and Morris (VM) found a counterexample to the assertion that a majority voting classifier always performs better than its independent component classifiers. VM's counterexample applies to independent classifiers, but biostatisticians are often more interested in conditionally independent classifiers. In biomedical studies, where class is disease status, classifiers are inherently dependent simply because positivity of any reasonable classifier depends on the presence or absence of disease. Conditional independence of classifiers, given disease status, could arise if the classifiers are detecting different biological phenomenon, such as tissue abnormalities versus protein markers. To explore how majority voting affects classification performance with conditionally independent classifiers, we investigated many examples (Figure 1). Much as we expected, we found that it generally works quite well. However, we also found that conditional independence is not a sufficient condition to ensure that majority voting always leads to better classification performance than the individual classifiers. Figure 1 Comparison of ROC curves for majority voting classifier and conditionally independent component classifiers. The 45-degree line is included for reference. As with VM, we considered two classes and component classifiers with identical classification performances. To measure classification performance we used receiver operating characteristic (ROC) curves. ROC curves play a central role in the evaluation of diagnostic and screening tests (Baker 2003; Pepe 2003). In accordance with a decision theory view of ROC curves (Baker, Van Calster, and Steyerberg 2012), we restricted our investigation to ROC curves that are concave, namely with monotonically decreasing slopes from left to right. For a given cutpoint x of a score, let fpr(x) and tpr(x) denote the false positive and true positive rates of the component classifier. The ROC curve for the component classifier plots tpr(x) versus fpr(x). At a given cutpoint, the true positive rate for the majority voting classifier is the probability of three or exactly two true positives among the component classifiers, namely tprM(x) = tpr(x)3 + 3 tpr(x)2 {1−tpr(x)}. Similarly the false positive rate for the majority voting classifier is fprM(x) = fpr(x)3 + 3 fpr(x)2 {1−fpr(x)}. The ROC curve for the majority voting classifier plots tprM(x) versus fprM(x). We considered the following six cases.