Trustability-Based Dynamic Active Learning for Crowdsourced Labelling of Emotional Audio Data

Abstract

The process of collecting annotated data is expensive and time-consuming. Making use of crowdsourcing instead of experts in a laboratory setting is a viable alternative to reduce these costs. However, without adequate quality control the obtained labels may be less reliable. Whereas crowdsourcing reduces only the costs per annotation, another technique, active learning, aims at reducing the overall annotation costs by selecting the most important instances of the dataset and only asking for manual annotations for these selected samples. Herein, we investigate the advantages of combining crowdsourcing and different iterative active learning paradigms for audio data annotation. Further, we incorporate an annotator trustability score to further reduce the labelling effort needed and, at the same time, to achieve better classification results. In this context, we introduce a novel active learning algorithm, called Trustability-based dynamic active learning, which accumulates manual annotations in each step until a trustability-weighted agreement level of annotators is reached. Furthermore, we bring this approach into the real world and integrate it in our gamified intelligent crowdsourcing platform iHEARu-PLAY. Key experimental results on an emotion recognition task indicate that a considerable relative annotation cost reduction of up to 90.57 % can be achieved when compared with a non-intelligent annotation approach. Moreover, our proposed method reaches an unweighted average recall value of 73.71 %, while a conventional passive learning algorithm peaks at 60.03 %. Therefore, our novel approach not only efficiently reduces the manual annotation work load but also improves the classification performance.