Abstract
In this paper, we propose a novel relevance evaluation method using labels collected from crowdsourcing. The proposed method not only predicts the relevance between query texts and responses in information retrieval systems but also performs the label aggregation tasks simultaneously. It first merges two kinds of heterogeneous data (i.e., image and query text) and constructs a CNN-like deep neural network. Then, on the top of its softmax layer, an additional layer was built to model the crowd workers. Finally, classification models for relevance prediction and aggregated labels for training examples can be simultaneously learned from noisy labels. Experimental results show that the proposed method significantly outperforms other state-of-the-art methods on a real-world dataset.
Highlights
1 Introduction Relevance evaluation is a significant component in the domain of information retrieval [1,2,3] to develop and maintain Information retrieval (IR) systems, where relevance between queries and responses is an important indicators to reflect whether an IR system is good or not
To do the comparison experiments, we use Majority voting (MV), GTIC, Opt-D&S, DS, and GLAD respectively to infer the integrated labels of training data, we use these labels to train a base deep learning model without crowdsourcing layer
5 Discussion In this paper, a novel relevance evaluation method was proposed with a deep learning network using crowdsourcing data to improve the accuracy compared with traditional methods and furtherly reduce the cost and time
Summary
Relevance evaluation is a significant component in the domain of information retrieval [1,2,3] to develop and maintain IR systems, where relevance between queries and responses is an important indicators to reflect whether an IR system is good or not. The accuracy and relevance of IR systems could be improved furtherly using the feedback of relevance evaluation. In early years of the IR field, relevance evaluation tasks are usually performed by professional assessors or domain experts, but it has some limitations in practice. It is rather difficult for assessors to read a large number of documents and judge their relevance to corresponding query texts. The process of evaluation is slow and expensive to insure the accuracy of judgments [4,5,6]. In 2006, the term crowdsourcing was first coined by Jeff Howe in the Wired magazine [7], and Merriam-Webster defines crowdsourcing as the process of obtaining needed services, ideas, or content by
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: EURASIP Journal on Wireless Communications and Networking
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
