Active Learning and Machine Teaching for Online Learning: A Study of Attention and Labelling Cost

Abstract

Interactive Machine Learning (ML) has the potential to lower the manual labelling effort needed, as well as increase classification performance by incorporating a human-in-the loop component. However, the assumptions made regarding the interactive behaviour of the human in experiments are often not realistic. Active learning typically treats the human as a passive, but always correct, participant. Machine teaching provides a more proactive role for the human, but generally assumes that the human is constantly monitoring the learning process. In this paper, we present an interactive online framework and perform experiments to compare active learning, machine teaching and combined approaches. We study not only the classification performance, but also the effort (to label samples) and attention (to monitor the ML system) required of the human. Results from experiments show that a combined approach generally performs better with less effort compared to active learning and machine teaching. With regards to attention, the best performing strategy varied depending on the problem setup.