Tagging human activities in video by crowdsourcing

Abstract

Activity annotation in videos is necessary to create a training dataset for most of activity recognition systems. This is a very time consuming and repetitive task. Crowdsourcing gains popularity to distribute annotation tasks to a large pool of taggers. We present for the first time an approach to achieve good quality for activity annotation in videos through crowdsourcing on the AmazonMechanical Turk platform (AMT). Taggers must annotate the start, end boundaries and the label of all occurrences of activities in videos. Two strategies to detect non-serious taggers according to temporal annotated results are presented. Individual filtering checks the consistence in the answers of each tagger with the characteristic of dataset to identify and remove nonserious taggers. Collaborative filtering checks the agreement in annotations among taggers. The filtering techniques detect and remove non-serious taggers and finally, the majority voting applied to AMT temporal tags to generate one final AMT activity annotation set. We conduct the experiments to get activity annotation from AMT on a subset of two rich datasets frequently used in activity recognition. The results show that our proposed filtering strategies can increase the accuracy by up to 40%. The final annotation set is of comparable quality of the annotation of experts with high accuracy (76% to 92%).