Processes, challenges and recommendations of Gray Literature Review: An experience report

Abstract

Systematic Literature Review (SLR), as a tool of Evidence-Based Software Engineering (EBSE), has been widely used in Software Engineering (SE). However, for certain topics in SE, especially those that are trendy or industry driven, academic literature is generally scarce and consequently Gray Literature (GL) becomes a major source of evidence. In recent years, the adoption of Gray Literature Review (GLR) or Multivocal Literature Review (MLR) is rising steadily to provide the state-of-the-practice of a specific topic where SLR is not a viable option. Although some SLR guidelines recommend the use of GL and several MLR guidelines have already been proposed in SE, researchers still have conflicting views on the value of GL and commonly accepted GLR or MLR studies are generally lacking in terms of publication. This experience report aims to shed some light on GLR through a case study that uses SLR and MLR guidelines to conduct a GLR on an emerging topic in SE to specifically answer the questions related to the reasons of using GL, the processes of conducting GL, and the impacts of GL on review results. We retrospect the review process of conducting a GLR on the topic of DevSecOps with reference to Kitchenham’s SLR and Garousi’s MLR guidelines. We specifically reflect on the processes we had to adapt in order to tackle the challenges we faced. We also compare and contrast our GLR with existing MLRs or GLRs in SE to contextualize our reflections. We distill ten challenges in nine activities of a GLR process. We provide reasons for these challenges and further suggest ways to tackle them during a GLR process. We also discuss the decision process of selecting a suitable review methodology among SLR, MLR and GLR and elaborate the impacts of GL on our review results. Although our experience on GLR is mainly derived from a specific case study on DevSecOps, we conjecture that it is relevant and would be beneficial to other GLR or MLR studies. We also expect our experience would contribute to future GLR or MLR guidelines, in a way similar to how SLR guidelines learned from the SLR experience report a dozen years ago. In addition, other researchers may find our decision making process useful before they conduct their own reviews.