Exploiting predicted answer in label aggregation to make better use of the crowd wisdom

Abstract

Nowadays, crowdsourcing is a widespread and effective method to gather the crowd wisdom. At the same time, label aggregation is used to aggregate the noisy and biased data generated by the crowd. In the real-world crowdsourcing tasks, most workers only answer a small fraction of questions, which makes the collected answer sparse. However, the existing label aggregation approaches often build upon some probabilistic modeling procedures which is sensitive to the data sparsity. In this paper, we exploit the predicted answers to improve the performance of label aggregation and propose PLA (Prediction-based Label Aggregation) to intelligently aggregate the crowd wisdom. With PLA, we firstly learn representations to capture the characteristics of the workers and questions. Then we deploy a neural network model to predict the answer given by different workers. After that we add the most valuable predicted answers to the answer set. Finally, we use the augmented answer set to enhance representative label aggregation algorithms. To validate our proposed PLA, we compare it with other 6 existing methods on 8 real-world datasets. Our results show that PLA can enhance the performance of different aggregation algorithms in crowdsourcing tasks and achieves up to 16% performance improvement.