Learning with mitigating random consistency from the accuracy measure

Jieting Wang,Feijiang Li,Yuhua Qian

doi:10.1007/s10994-020-05914-3

Abstract

Human beings may make random guesses in decision-making. Occasionally, their guesses may generate consistency with the real situation. This kind of consistency is termed random consistency. In the area of machine leaning, the randomness is unavoidable and ubiquitous in learning algorithms. However, the accuracy (A), which is a fundamental performance measure for machine learning, does not recognize the random consistency. This causes that the classifiers learnt by A contain the random consistency. The random consistency may cause an unreliable evaluation and harm the generalization performance. To solve this problem, the pure accuracy (PA) is defined to eliminate the random consistency from the A. In this paper, we mainly study the necessity, learning consistency and leaning method of the PA. We show that the PA is insensitive to the class distribution of classifier and is more fair to the majority and the minority than A. Subsequently, some novel generalization bounds on the PA and A are given. Furthermore, we show that the PA is Bayes-risk consistent in finite and infinite hypothesis space. We design a plug-in rule that maximizes the PA, and the experiments on twenty benchmark data sets demonstrate that the proposed method performs statistically better than the kernel logistic regression in terms of PA and comparable performance in terms of A. Compared with the other plug-in rules, the proposed method obtains much better performance.

Highlights

In the process of decision-making, human beings may make random guesses without logical reasoning when they lack sufficient evidence or detailed knowledge
The random consistency produces dishonest feedback, misleads the decision direction and harms the improvement of the generalization ability, especially when the tendency of random guesses coincides with the class distribution of the real situation
By the benchmark data sets, we show that learning by pure accuracy (PA) is more fair in majority accuracy and minority accuracy than A and compare the -interval search method with some other plug-in rules to show its effectiveness

Summary

Introduction

In the process of decision-making, human beings may make random guesses without logical reasoning when they lack sufficient evidence or detailed knowledge. Intern doctors are likely to diagnose patients with colds during flu season, and students are likely to choose a lucky option when faced with a difficult multiple-choices question. Sometimes, these random guesses may generate consistency with the real situation. These random guesses may generate consistency with the real situation We term this consistency the random consistency. The prediction results of the learning models may contain the random consistency. The random consistency produces dishonest feedback, misleads the decision direction and harms the improvement of the generalization ability, especially when the tendency of random guesses coincides with the class distribution of the real situation

Objectives

Methods

Findings

Conclusion