Multivocal literature reviews in software engineering: Preliminary findings from a tertiary study

Abstract

Background: In recent years, studies involving Grey Literature (GL) have been growing and attracting the attention of researchers in software engineering (SE). One of the sources of GL refers to content produced by professionals based on their practical experiences? Recent researches in the SE states that GL can complement areas of research that are not yet clearly defined in the scientific literature. In this context, the Multivocal Literature Review (MLR), a form of Systematic Literature Review (SLR) with the inclusion of GL, emerges. Goal: Provide preliminary work about the current research involving MLR studies? First, we investigate the motivation of the researchers to include GL in review studies; and second, we examine how GL was included in the studies. Method: A tertiary study was conducted to search MLR studies published between 2009 to April of 2019. Results: The main motivations for including GL in review studies are: lack of academic research on the topic, emerging research on this topic, and complementary evidence in the GL? Internet articles and white papers were the main sources of GL data used. Conclusions: The conducting of MLR studies is still in its early stages; we have identified only 12 secondary studies. The MLR studies were conducted using guidelines for performing SLRs. What we consider to be a threat to the validity of these studies, since guidelines to conduct SLR studies do not provide recommendations for quality analysis and synthesis of primary studies, including GL.