Multivocal Literature Review Research Articles

Blockchain technology applications in the health domain: a multivocal literature review.

Blockchain technology has been changing the nature of several businesses, from supply chain management to electronic record management systems and copyright management to healthcare applications. It provides a resilient and secure platform for modifications due to its distributed and shared nature and cryptographic functions. Each new technology, however, comes with its challenges alongside its opportunities. Previously, we performed a systematic literature review (SLR) to explore how blockchain technology potentially benefits health domain applications. The previous SLR included 27 formal literature papers from 2016 to 2020. Noticing that blockchain technology is rapidly growing, we extended the previous SLR with a multivocal literature review (MLR) approach to present the state of the art in this study. We focused on understanding to what degree blockchain could answer the challenges inherited in the health domain and whether blockchain technology may bring new challenges to health applications. The MLR consists of 78 sources of formal literature and 23 sources of gray literature from 2016 to 2021. As a result of this study, we specified 17 health domain challenges that can be categorized into four groups: (i) meeting regulatory requirements and public health surveillance, (ii) ensuring security and privacy, (iii) ensuring interoperability, and (iv) preventing waste of resources. The analysis shows that blockchain makes significant contributions to the solutions of these challenges. However, 10 new pitfalls come with adopting the technology in the health domain: the inability to delete sensitive data once it is added to a chain, limited ability to keep large-scale data in a blockchain, and performance issues. The data we extracted during the MLR is available in a publicly accessible online repository.

Multivocal Literature Review on the Security of DevSecOp

DevOps revolutionize the software development lifecycle by providing agile and fast-paced solutions. DevOps ignores security prospects since it focuses only on increasing development and speed. DevSecOp is a notion of implanting Security into DevOps operation without distressing its agile nature by discovering contemporary security practices. This research aims to reveal a comprehensive overview of DevSecOp. Here we presented a brief overview of the research methodology. Afterward, it presented the method of gathering the required information. This research paper is distributed in the following section. Section II presented the research methodology, while section III provided results from this study. In the end, we conclude our research
 In this study, we discover essential DevSecOp concepts, leverages of DevSecOp, and potential research challenges in implementing it. We used a Multivocal literature review to explore the aforementioned subjects. For this Multivocal literature review, we searched grey data and, after processed that data, found answers to our research questions. This review concluded that DevSecOp, although challenging to implement, can be a constructive addition to the DevOps paradigm.
 DevSecOp is a relatively new concept that is not even fully concise in its name and definition. The key idea of DevSecOp is to implant security into DevOps procedures to make them more secure. We presented MLR on DevSecOp, keeping in mind pre-designed research questions. Since DevSecOp is not as popular and does not contain enough academic literature, we had to include grey data for our literature review. This MLR concluded that DevSecOp is mainly defined as integrating Security into DevOps

Toward successful DevSecOps in software development organizations: A decision-making framework

Development and Operations (DevOps) is a methodology that aims to establish collaboration between programmers and operators to automate the continuous delivery of new software to reduce the development life cycle and produce quality software. Development, Security, and Operations (DevSecOps) is developing the DevOps concept, which integrates security methods into a DevOps process. DevSecOps is a software development process where security is built in to ensure application confidentiality, integrity, and availability. This paper aims to identify and prioritize the challenges associated with implementing the DevSecOps process. We performed a multivocal literature review (MLR) and conducted a questionnaire-based survey to identify challenges associated with DevSecOps-based projects. Moreover, interpretive structure modeling (ISM) was applied to study the relationships among the core categories of the challenges. Finally, we used the fuzzy technique for order preference by similarity to an ideal solution (TOPSIS) to prioritize the identified challenges associated with DevSecOps projects. We identified 18 challenges for the DevSecOps process and mapped them to 10 core categories. The ISM results indicate that the “standards” category has the most decisive influence on the other nine core categories of the identified challenges. Moreover, the fuzzy TOPSIS indicates that “lack of secure coding standards,” “lack of automated testing tools for security in DevOps,” and “ignorance in static testing for security due to lack of knowledge” are the highest priority challenges for the DevSecOps paradigm. Organizations using DevOps should consider the identified challenges in developing secure software.

Smells and refactorings for microservices security: A multivocal literature review

Context:Securing microservices is crucial, as many IT companies are delivering their businesses through microservices. If security “smells” affect microservice-based applications, they can possibly suffer from security leaks and need to be refactored to mitigate the effects of security smells therein. Objective:As the available knowledge on securing microservices is scattered across different pieces of white and grey literature, our objective here is to distill well-known smells for securing microservices, together with the refactorings enabling to mitigate their effects. Method:To capture the state of the art and practice in securing microservices, we conducted a multivocal review of the existing white and grey literature on the topic. We systematically analysed 58 primary studies, selected among those published from 2011 until the end of 2020. Results:Ten bad smells for securing microservices are identified, which we organized in a taxonomy, associating each smell with the security properties it may violate and the refactorings enabling to mitigate its effects. Conclusions:The security smells and the corresponding refactorings have pragmatic value for practitioners, who can exploit them in their daily work on securing microservices. They also serve as a starting point for researchers wishing to establish new research directions on securing microservices.

A multivocal literature review on record management potential components in CRUD operation for web application development

In recent years, web application frameworks have been widely practised by many developers to increase programming productivity as the frameworks are more flexible, rapidly built using CRUD operation, MVC-based, secure and most of them are published under an open-source license which will reduce the final cost of development. Although the CRUD automation in the web application framework boosts the development process, there are many important aspects of a web application absent from the CRUD output. Therefore, this multivocal literature review investigates the record management aspects that are required in modern WA and the perceived benefit of integrating the record management aspect into CRUD operation. The study extracted 284 publications from respectable scientific resources and the grey resources literature created by WA development practitioners outside academic mediums. After a detailed review process, only 14 scientific primary studies and 13 gray studies were considered for this review based on defined inclusion and exclusion criteria. The review shows that the most important aspects required in WA are search, role-based access control, retention, appraisal, search, audit trail, digital archiving, sharing, reporting, inactive files management and several other features. These important aspects have been analyzed and characterized according to its function and features. The method and procedure for integrating the specified aspect into CRUD operation are identified and discussed. Integrating and implementing the specified record management features into CRUD operation will boost the WA development productivity by producing more features as a standard output with integrated record management functions.

Software-as-a-Service Security Challenges and Best Practices: A Multivocal Literature Review

Cloud computing (CC) is the delivery of computing services on demand and is charged using a “pay per you use” policy. Of the multiple services offered by CC, SaaS is the most popular and widely adapted service platform and is used by billions of organizations due to its wide range of benefits. However, security is a key challenge and obstacle in cloud adoption and therefore needs to be addressed. Researchers and practitioners (R&P) have discussed various security challenges for SaaS along with possible solutions. However, no research study exists that systematically accumulates and analyzes the security challenges and solutions. To fill this gap and provide the state-of-the-art (SOTA) picture of SaaS security, this study provides a comprehensive multivocal literature review (MVLR), including SaaS security issues/challenges and best practices for mitigating these security issues. We identified SaaS security issues/challenges and best practices from the formal literature (FL) as well as the grey literature (GL) to evaluate whether R&P is on the same page or if controversies exist. A total of 93 primary studies were identified, of which 58 are from the FL and 35 belong to the GL. The studies are from the last ten years, from 2010 to 2021. The selected studies were evaluated and analyzed to identify the key security issues faced by SaaS computing and to be aware of the best practices suggested by R&P to improve SaaS security. This MVLR will assist SaaS users to identify the many areas in which additional research and development in SaaS security is required. According to our study findings, data breaches/leakage, identity and access management, governance and regulatory compliance/SLA compliance, and malicious insiders are the key security challenges with the maximum frequency of occurrence in both FL and GL. On the other hand, R&P agree that up-to-date security controls/standards, the use of strong encryption techniques, regulatory compliance/SLA compliance, and multifactor authentication are the most important solutions.

The maturity of open government data maturity: a multivocal literature review

PurposeThe aim of this study is to provide administrators in government institutions a roadmap to achieve benefits of open government data (OGD) by reviewing and classifying studies with assessment or maturity models (MMs) in the OGD domain with a Multivocal Literature Review (MLR).Design/methodology/approachTo achieve this goal, the authors conducted an MLR that includes data from not only the formal literature but also the grey literature (e.g. white papers and online documents).FindingsOut of 3,569 sources, 81 studies were selected by following the elimination scheme and assessing sources by relevance and methodology. As a result of the quality assessment of the identified MMs based on predefined criteria, unambiguity, comparability, repeatability, completeness, clearness and objectivity, it was observed that there are a limited number of MMs in this domain and none of them fully satisfies the requirements.Originality/valueThis study is likely the first MLR on OGD domain. This MLR serves as a first step for future research on OGD assessment and MMs by presenting the need to establish a holistic approach covering all OGD dimensions, creation of an objective assessment method, prescriptive properties, and empirical evaluation demonstrating the applicability and usefulness at different scope levels.

Eliciting Best Practices for Collaboration with Computational Notebooks

Despite the widespread adoption of computational notebooks, little is known about best practices for their usage in collaborative contexts. In this paper, we fill this gap by eliciting a catalog of best practices for collaborative data science with computational notebooks. With this aim, we first look for best practices through a multivocal literature review. Then, we conduct interviews with professional data scientists to assess their awareness of these best practices. Finally, we assess the adoption of best practices through the analysis of 1,380 Jupyter notebooks retrieved from the Kaggle platform. Findings reveal that experts are mostly aware of the best practices and tend to adopt them in their daily work. Nonetheless, they do not consistently follow all the recommendations as, depending on specific contexts, some are deemed unfeasible or counterproductive due to the lack of proper tool support. As such, we envision the design of notebook solutions that allow data scientists not to have to prioritize exploration and rapid prototyping over writing code of quality.

Accessibility Challenges in OER and MOOC: MLR Analysis Considering the Pandemic Years

The review of state of the art on creating and managing learning resources and accessible Open Educational Resources (OER) and Massive Open Online Courses (MOOC) is a topic that cannot only consider formal literature. The evidence and lack of a measurement consensus require the inclusion of contextual information, corroborating scientific results with practical experiences. For this reason, this article presents a review of accessibility models, OER and MOOC, considering the gray literature to capture experiences and trying to establish a shared understanding of the terminology commonly used in research on virtual accessibility and its impact on higher education. The bibliographic review relies on analyzing articles and scientific publications related to the topic following the Multivocal Literature Review (MLR) format. The results of this review establish that it is possible to apply accessibility review methodologies with transversal actions in the creation and management of learning resources and MOOCs. The research is related to one of the seventeen sustainable development goals defined by the United Nations to ensure inclusive and equitable quality education and promote lifelong learning opportunities for all.

Quality Assurance in E-Learning: A Proposal from Accessibility to Sustainability

Given the importance of developing and offering accessible education for all, indispensable aspects of education for sustainable development (ESD) are needed. This study addresses that need by proposing a quality self-assessment for virtual education from an accessibility perspective. This proposal is based on previous literature about quality assurance in e-learning that considered accessibility and its application in the field of higher education. The bibliographic review was conducted by following Multivocal Literature Review (MLR) guidelines. The initial search returned 999 items from 5 academic databases and 32,200 professional sources from Google. After reviewing the sources, 37 of them were included. Then, the accessibility criteria were identified and integrated into an evaluation model. Such a model is divided into four dimensions: (1) organization, (2) student body, (3) teaching, and (4) infrastructure. The model also includes a set of standards (16), requirements (48), and evidence (63) that apply to each dimension. Moreover, self-assessment guidelines for accessible virtual education were proposed. They included a conceptual and theoretical framework, a self-assessment model, and a methodology for applying the model. The methodology included five phases: planning, model tuning or refinement of the model, evaluation, results, and continuous improvement. As future work, the implementation and validation of the guidelines will be carried out.

Software-Engineering Design Patterns for Machine Learning Applications

In this study, a multivocal literature review identified 15 software-engineering design patterns for machine learning applications. Findings suggest that there are opportunities to increase the patterns’ adoption in practice by raising awareness of such patterns within the community.

Guidelines to Establish an Office of Student Accessibility Services in Higher Education Institutions

The objective of this paper is to propose a set of guidelines to establish an office of Student Accessibility Services (SAS) in Higher Education Institutions (HEIs). The proposed guidelines help to integrate disjointed knowledge to facilitate its interpretation and implementation during deployment of basic support services in favor of students with disability. These guidelines can help to mitigate complexity in providing SAS for the first time in HEIs. These guidelines cover both the design and implementation of an office of SAS and its management. Knowledge was found through a multivocal literature review (MLR), which allowed to capture not only academic approaches but also vantage points and experiences from practice. Key concepts and aspects were organized into eight components (five related to the design and implementation, and three associated with the management context). An expert appraisal method was used as a proof of concept, which complemented a previously performed preliminary implementation example. Obtained results demonstrated the pertinence of the conceptual proposal and confirmed guidelines capability for full implementation in a real-world scenario.

Improving test automation maturity: A multivocal literature review

AbstractMature test automation is key for achieving software quality at speed. In this paper, we present a multivocal literature review with the objective to survey and synthesize the guidelines given in the literature for improving test automation maturity. We selected and reviewed 81 primary studies, consisting of 26 academic literature and 55 grey literature sources. From primary studies, we extracted 26 test automation best practices (e.g., Define an effective test automation strategy, Set up good test environments, and Develop high‐quality test scripts) and collected many pieces of advice (e.g., in forms of implementation/improvement approaches, technical techniques, concepts, and experience‐based heuristics) on how to conduct these best practices. We made main observations: (1) There are only six best practices whose positive effect on maturity improvement have been evaluated by academic studies using formal empirical methods; (2) several technical related best practices in this MLR were not presented in test maturity models; (3) some best practices can be linked to success factors and maturity impediments proposed by other scholars; (4) most pieces of advice on how to conduct proposed best practices were identified from experience studies and their effectiveness need to be further evaluated with cross‐site empirical evidence using formal empirical methods; (5) in the literature, some advice on how to conduct certain best practices are conflicting, and some advice on how to conduct certain best practices still need further qualitative analysis.

Dairy Farm Management Information Systems

Nowadays, different types of farm management information systems (FMIS) are being used in practice in several sectors of farming, such as dairy, arable, fruits, vegetables, and meat farming. The goal of this research is to identify, evaluate, and synthesize existing FMISs in the Dutch dairy sector and present the state–of–the–art. We performed a multivocal literature review (MLR) to find sources both in scientific and grey literature. A grey literature search was adopted because most of the FMISs were not reported in the scientific literature. To support and improve the effectiveness of the MLR process, an online survey was first sent to Dutch dairy farmers to identify the FMISs that are being used in practice. With the help of the MLR process, we identified 50 FMISs used by Dutch dairy farmers. We identified 33 features of these FMISs and listed the advantages and disadvantages of the FMISs.

Improving Software Robot Maintenance in Large-Scale Environments–is Center of Excellence a Solution?

Robotic Process Automation (RPA) performs high-volume tasks such as checking invoices. However, the governance and maintenance of large-scale software robot environments can be challenging when robot servers perform automatized tasks simultaneously for customer organizations with complex programming rules, dedicated parameters, and dependencies on timetables. A multivocal literature review (MLR) was conducted to explore whether there are 1) mechanisms to improve software robot maintenance in large-scale robot environments, 2) or software robot maintenance practices for scalable RPA in organizations providing shared services, 3) or governance models for optimizing the performance of software robot maintenance, and 4) is the Center of Excellence (CoE) one of the success factors concerning large-scale robot environments. By doing this, 5) we found eleven functional requirements for the monitoring tool to support maintenance in a large-scale environment. In addition, we adapted them to the RPA monitoring tool abilities for the Finnish Government Shared Services Centre for Finance and HR (Palkeet). As a result, the eleven functional requirements and the monitoring tool abilities are adaptable for other large-scale environments to improve software robot maintenance. However, commercial monitoring tools for RPA maintenance do not fulfil functional requirements, and organizations in large-scale environments must develop their monitoring tools. Based on MLR, either in-house or outsourced CoE seems to be one of the success factors in RPA maintenance in large-scale environments.

NoOps – A Multivocal literature review

Traditionally, an organization had to have in-house servers and hardware to build a web application. This evolved into Cloud computing where the possibility for cost reduction and scalable data storage became a reality. With the introduction of cloud computing came a concept known as NoOps, or No Operations. This paper aims to take a closer look into what NoOps is and the benefits and challenges of NoOps. The authors identified three RQs that could help to give more insight into NoOps. Further we discussed the findings and RQs and lay out the way forward for future studies into NoOps. We also looked at artificial intelligence (AI) and how AI seems to be heavily linked with a true NoOps environment. With the lack of scientific studies into NoOps, a Multivocal literature review was selected as the method used to investigate the concept and its implications. We try to show voices both for and against NoOps. Further, we try to look at a misconception of what NoOps really is, what true NoOps could be. Finally we look at what requirements there are for companies wanting to go NoOps, and discuss the possibility that many companies unknowingly are moving towards a NoOps environment.

Exploration of DevOps testing process capabilities: An ISM and fuzzy TOPSIS analysis

DevOps is an emerging paradigm that refer to a collaborative culture of development and operation teams aiming to develop the high quality software product. Software organizations are adopting DevOps culture for software development and easy maintenance instead of using traditional SDLC mechanism. To enter the production stage, in DevOps process, the software product have to pass through quality gates were the software are tested during development phase to meet the established targeted criteria. This indicates that the mechanism of testing in DevOps process is not straightforward, and to establish strong DevOps testing platform there is a need to explore more automated testing practices. Thus, using multivocal literature review approach, we have selected 39 studies and identify the 20 testing capabilities. Finally, the interpretive structure modeling (ISM) and fuzzy technique for order preference by similarity to ideal solution (fuzzy TOPSIS) were applied. The results shows that (C2, CCi=0.808; C6, CCi=0.720; and C3, CCi=0.705) are top ranked testing capabilities. Using analysis results, we develop a holistic structure of testing capabilities to show their inter-relationship with each other and their priorities to select the best testing capabilities for DevOps process.

Aligning the Production Planning and Control Process with Prognostics and Health Management

Production planning and control (PPC) is the heart of any manufacturing company and entails tasks such as resource planning, sequencing, or capacity control. While an increasing complexity within production makes it difficult to determine the best production plan, the advances in PHM and the emergence of predictive maintenance also offer new opportunities to optimize PPC. While there is much research on PHM and PPC, little has been done to align both disciplines. Through post-prognostics decision-making, different PPC decisions, such as continuing the production, shutting down a machine, or reducing its workload, can be elevated by a remaining useful life (RUL) estimation. However, it is unclear how exactly this prognostics information can be exploited and how processes, organization, and technology must be aligned to attain a more efficient and flexible production. Further, PHM has long been implemented beyond research, but it is unknown whether and how practitioners intertwine it with their PPC. This work aims to analyze how processual, organizational, and technological changes through PHM can lead to advanced PPC. This goal is attained by means of a multivocal literature review (MLR) in which scientific PPC and PHM literature and standards are analyzed, and an aligned PPC process proposed. The findings are juxtaposed with grey literature, revealing fits and gaps between research and practice, and a research agenda is presented.

Quality Assessment Methods for Textual Conversational Interfaces: A Multivocal Literature Review

The evaluation and assessment of conversational interfaces is a complex task since such software products are challenging to validate through traditional testing approaches. We conducted a systematic Multivocal Literature Review (MLR), on five different literature sources, to provide a view on quality attributes, evaluation frameworks, and evaluation datasets proposed to provide aid to the researchers and practitioners of the field. We came up with a final pool of 118 contributions, including grey (35) and white literature (83). We categorized 123 different quality attributes and metrics under ten different categories and four macro-categories: Relational, Conversational, User-Centered and Quantitative attributes. While Relational and Conversational attributes are most commonly explored by the scientific literature, we testified a predominance of User-Centered Attributes in industrial literature. We also identified five different academic frameworks/tools to automatically compute sets of metrics, and 28 datasets (subdivided into seven different categories based on the type of data contained) that can produce conversations for the evaluation of conversational interfaces. Our analysis of literature highlights that a high number of qualitative and quantitative attributes are available in the literature to evaluate the performance of conversational interfaces. Our categorization can serve as a valid entry point for researchers and practitioners to select the proper functional and non-functional aspects to be evaluated for their products.

Barriers of managing cloud outsource software development projects: a multivocal study

Management of COSD projects is a challenging task due to number of distant development locations in different time zones, client and vendor organizations, different cloud deployment models and range of different service level agreements. The objective of this study is to identify the barriers associated with managing COSD projects. We implemented a Multivocal Literature Review to identify barriers that influence management of COSD projects. We identified 21 COSD management barriers from 165 primary studies. The comparison between the barriers identified from formal and grey literature indicate that there are similarities between the barriers investigated from both types of literature. Moreover, client-vendor analysis shows that there is no significant difference between COSD management barriers associated with both types of organizations. We believe that the study findings will assist both research and industry community to better understand and manage COSD projects.