Crowdsourcing in Systematic Reviews: A Systematic Mapping and Survey

Abstract

Context: Systematic reviews (SRs) have been adopted in the Software Engineering (SE) field for more than a decade to provide synthesis of evidence on various topics. However, the process in conducting an SR remains laborious-intensive and expensive, specially in terms of hours that SR researchers dedicate. It is worth exploring approaches to conduct SRs at lower costs (quicker, using less resources – time of researchers). One such approach is crowdsourcing, since conducting SRs activities among a large number of researchers is a promising alternative to reduce costs associated to SR conduction. Goal: The main goal of this study is to identify and summarize the body of knowledge on crowdsourcing to support the conduction of SRs in SE. Method: Two empirical research methods were used. Initially, we conducted a Systematic Mapping to identify the available and relevant studies on crowdsourcing in SRs in SE. Secondly, a survey was performed with 39 SE researchers aiming to identify their perception related to the value of performing SRs collaboratively. Results: Our results show that how to speed up the SR process; reduce bias through broad participation; and expand team expertise were most potential benefits linked to the use of crowdsourcing in SR. The main challenges were associated with quality control to ensure the quality of results. Conclusions: In spite of the challenges, we believe that crowdsourcing could be successfully employed in SR context. More empirical research is needed on how to use crowdsourcing to support SR conduction in SE and how to minimize the identified challenges.