Abstract
Crowd sourcing has been ubiquitously used for annotating enormous collections of data. However, the major obstacles of using crowd-sourced labels are noise and errors from non-expert annotations. In this work, two approaches dealing with the noise and errors in crowd-sourced labels are proposed. The first approach uses Sharpness-Aware Minimization (SAM), an optimization technique robust to noisy labels. The other approach leverages a neural network layer called crowd layer specifically designed to learn from crowd-sourced annotations. According to the results, the proposed approaches can improve the performance of Wide Residual Network model and Multi-layer Perception model applied on two crowd-sourced datasets in image processing domain.
Highlights
There has been some major advancement in the use of deep learning for solving artificial intelligence problems in different domains such as sentiment analysis, image classification, natural language inference, speech recognition object detection
3.5 Bidirectional Encoder Representations from Transformers (BERT) with softmax-Crowdlayer for Gimpel-part of speech (POS) and PDIS Datasets In Gimpel-POS dataset, each sample consisted of a tweeted text, a specific word/token appears in a tweeted text and a crowd label which is a list of multiple labels from different annotators
We propose to fine-tune the pretrained BERT model for both Gimpel-POS task and PDIS task based on crowd labels
Summary
There has been some major advancement in the use of deep learning for solving artificial intelligence problems in different domains such as sentiment analysis, image classification, natural language inference, speech recognition object detection They have been used in many other numerous cases where human disagreements are encountered such as speech recognition, visual object recognition, object detection and machine translation (Rodrigues and Pereira, 2018). Crowd-sourcing has been used in the annotation of large collections of data and has proven to be an efficient and cost-effective means of obtaining labeled data as compared to expert labelling (Snow et al, 2008) It has been utilised in the generation of image annotations to train computer vision systems (Raykar et al, 2010), to provide the linguistic annotations used for Natural Language Processing (NLP) tasks (Snow et al, 2008), and has been used to collect the relevant judgments needed to optimize search engines (Alonso, 2013). The aim is to investigate the use of a unified testing framework to learn from disagreements using crowd source labels collected from different annotators
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
