Multivocal Literature Research Articles

Smells in system user interactive tests

Test smells are known as bad development practices that reflect poor design and implementation choices in software tests. Over the last decade, there are few attempts to study test smells in the context of system tests that interact with the System Under Test through a Graphical User Interface. To fill the gap, we conduct an exploratory analysis of test smells occurring in System User Interactive Tests (SUIT). We thus, compose a catalog of 35 SUIT-specific smells, identified through a multi-vocal literature review, and show how they differ from smells encountered in unit tests. We also conduct an empirical analysis to assess the diffuseness and removal of these smells in 48 industrial repositories and 12 open-source projects. Our results show that the same type of smells tends to appear in both industrial and open-source projects, but they are not addressed in the same way. We also find that smells originating from a combination of multiple code locations appear more often than those that are localized on a single line. This happens because of the difficulty to observe non-local smells without tool support. Furthermore, we find that smell-removing actions are not frequent with less than 50% of the affected tests ever undergoing a smell removal. Interestingly, while smell-removing actions are rare, some smells disappear while discarding tests, i.e., these smells do not appear in follow-up tests that replace the discarded ones.

Beyond technology in smart cities: a multivocal literature review on data governance

PurposeSmart cities require data governance to articulate data sharing and use among relevant stakeholders. Given the lack of a comprehensive examination of this research topic, this study aims to review data governance publications to detect and categorize endeavors backing up data sharing in smart cities.Design/methodology/approachA systematic literature review was conducted, and 568 academic and professional sources were identified, but finally, only 10 relevant papers were selected.FindingsResults reveal that data governance must be based on well-defined mechanisms, procedures and roles to achieve accountability and responsibility in a multi-actor environment. Moreover, data governance should be adapted to address power imbalances among all interested parties.Research limitations/implicationsThe main limitation is the list of sources considered for the literature review. However, this study provides a holistic overview for researchers and professionals willing to know more about smart city data sharing.Originality/valueThis review identifies the data governance approaches supporting data sharing in smart cities, analyzes their data dimension, enhances the state-of-the-art literature on this topic and suggests possible areas for future research.

Blockchain technology applications in the health domain: a multivocal literature review.

Blockchain technology has been changing the nature of several businesses, from supply chain management to electronic record management systems and copyright management to healthcare applications. It provides a resilient and secure platform for modifications due to its distributed and shared nature and cryptographic functions. Each new technology, however, comes with its challenges alongside its opportunities. Previously, we performed a systematic literature review (SLR) to explore how blockchain technology potentially benefits health domain applications. The previous SLR included 27 formal literature papers from 2016 to 2020. Noticing that blockchain technology is rapidly growing, we extended the previous SLR with a multivocal literature review (MLR) approach to present the state of the art in this study. We focused on understanding to what degree blockchain could answer the challenges inherited in the health domain and whether blockchain technology may bring new challenges to health applications. The MLR consists of 78 sources of formal literature and 23 sources of gray literature from 2016 to 2021. As a result of this study, we specified 17 health domain challenges that can be categorized into four groups: (i) meeting regulatory requirements and public health surveillance, (ii) ensuring security and privacy, (iii) ensuring interoperability, and (iv) preventing waste of resources. The analysis shows that blockchain makes significant contributions to the solutions of these challenges. However, 10 new pitfalls come with adopting the technology in the health domain: the inability to delete sensitive data once it is added to a chain, limited ability to keep large-scale data in a blockchain, and performance issues. The data we extracted during the MLR is available in a publicly accessible online repository.

Security in Wireless Body Sensor Network: A Multivocal Literature Study

The wireless body sensor network (WBSN) is a wireless communication that might enable 24/7 patient monitoring and health findings through the online platform. Although BSN design is becoming simpler, building a secure BSN seems to be more challenging than designing conventional solutions, and the recent study provides little guidance to designers and developers. The proposed study summarizes the multivocal literature study of security mechanisms for BSN. The investigation found 10,871 academic publications and 697 grey content; duplicates were removed, and selection criteria were employed, resulting in 73 academic papers and 30 grey publications. Various conventional security techniques, scope, and security contexts were used to classify the stated security solutions within each publication. It was crucial to inquire about the frequency of publications, research methods, security mechanisms, and contexts to answer the proposed questions. Our survey concludes that security methods and assessments are categorized into 15 categories, with the most frequently referenced being authentication and authorization; the majority of strategies concentrate on preventing and mitigating security breaches, with a limited number of works focusing on detection and recovery; and the techniques used to conduct the survey vary between the two types of publications. This evaluation might be the first step toward making the BSN platform more consistent by giving professionals and researchers a complete set of security strategies and methods. Experts will apply these solutions to fix security issues while establishing a trustworthy BSN after they have been identified through the process of discovering the most commonly utilized security solutions.

Multivocal Literature Review on the Security of DevSecOp

DevOps revolutionize the software development lifecycle by providing agile and fast-paced solutions. DevOps ignores security prospects since it focuses only on increasing development and speed. DevSecOp is a notion of implanting Security into DevOps operation without distressing its agile nature by discovering contemporary security practices. This research aims to reveal a comprehensive overview of DevSecOp. Here we presented a brief overview of the research methodology. Afterward, it presented the method of gathering the required information. This research paper is distributed in the following section. Section II presented the research methodology, while section III provided results from this study. In the end, we conclude our research
 In this study, we discover essential DevSecOp concepts, leverages of DevSecOp, and potential research challenges in implementing it. We used a Multivocal literature review to explore the aforementioned subjects. For this Multivocal literature review, we searched grey data and, after processed that data, found answers to our research questions. This review concluded that DevSecOp, although challenging to implement, can be a constructive addition to the DevOps paradigm.
 DevSecOp is a relatively new concept that is not even fully concise in its name and definition. The key idea of DevSecOp is to implant security into DevOps procedures to make them more secure. We presented MLR on DevSecOp, keeping in mind pre-designed research questions. Since DevSecOp is not as popular and does not contain enough academic literature, we had to include grey data for our literature review. This MLR concluded that DevSecOp is mainly defined as integrating Security into DevOps

Toward successful DevSecOps in software development organizations: A decision-making framework

Development and Operations (DevOps) is a methodology that aims to establish collaboration between programmers and operators to automate the continuous delivery of new software to reduce the development life cycle and produce quality software. Development, Security, and Operations (DevSecOps) is developing the DevOps concept, which integrates security methods into a DevOps process. DevSecOps is a software development process where security is built in to ensure application confidentiality, integrity, and availability. This paper aims to identify and prioritize the challenges associated with implementing the DevSecOps process. We performed a multivocal literature review (MLR) and conducted a questionnaire-based survey to identify challenges associated with DevSecOps-based projects. Moreover, interpretive structure modeling (ISM) was applied to study the relationships among the core categories of the challenges. Finally, we used the fuzzy technique for order preference by similarity to an ideal solution (TOPSIS) to prioritize the identified challenges associated with DevSecOps projects. We identified 18 challenges for the DevSecOps process and mapped them to 10 core categories. The ISM results indicate that the “standards” category has the most decisive influence on the other nine core categories of the identified challenges. Moreover, the fuzzy TOPSIS indicates that “lack of secure coding standards,” “lack of automated testing tools for security in DevOps,” and “ignorance in static testing for security due to lack of knowledge” are the highest priority challenges for the DevSecOps paradigm. Organizations using DevOps should consider the identified challenges in developing secure software.

Smells and refactorings for microservices security: A multivocal literature review

Context:Securing microservices is crucial, as many IT companies are delivering their businesses through microservices. If security “smells” affect microservice-based applications, they can possibly suffer from security leaks and need to be refactored to mitigate the effects of security smells therein. Objective:As the available knowledge on securing microservices is scattered across different pieces of white and grey literature, our objective here is to distill well-known smells for securing microservices, together with the refactorings enabling to mitigate their effects. Method:To capture the state of the art and practice in securing microservices, we conducted a multivocal review of the existing white and grey literature on the topic. We systematically analysed 58 primary studies, selected among those published from 2011 until the end of 2020. Results:Ten bad smells for securing microservices are identified, which we organized in a taxonomy, associating each smell with the security properties it may violate and the refactorings enabling to mitigate its effects. Conclusions:The security smells and the corresponding refactorings have pragmatic value for practitioners, who can exploit them in their daily work on securing microservices. They also serve as a starting point for researchers wishing to establish new research directions on securing microservices.

A multivocal literature review on record management potential components in CRUD operation for web application development

In recent years, web application frameworks have been widely practised by many developers to increase programming productivity as the frameworks are more flexible, rapidly built using CRUD operation, MVC-based, secure and most of them are published under an open-source license which will reduce the final cost of development. Although the CRUD automation in the web application framework boosts the development process, there are many important aspects of a web application absent from the CRUD output. Therefore, this multivocal literature review investigates the record management aspects that are required in modern WA and the perceived benefit of integrating the record management aspect into CRUD operation. The study extracted 284 publications from respectable scientific resources and the grey resources literature created by WA development practitioners outside academic mediums. After a detailed review process, only 14 scientific primary studies and 13 gray studies were considered for this review based on defined inclusion and exclusion criteria. The review shows that the most important aspects required in WA are search, role-based access control, retention, appraisal, search, audit trail, digital archiving, sharing, reporting, inactive files management and several other features. These important aspects have been analyzed and characterized according to its function and features. The method and procedure for integrating the specified aspect into CRUD operation are identified and discussed. Integrating and implementing the specified record management features into CRUD operation will boost the WA development productivity by producing more features as a standard output with integrated record management functions.

Software-as-a-Service Security Challenges and Best Practices: A Multivocal Literature Review

Cloud computing (CC) is the delivery of computing services on demand and is charged using a “pay per you use” policy. Of the multiple services offered by CC, SaaS is the most popular and widely adapted service platform and is used by billions of organizations due to its wide range of benefits. However, security is a key challenge and obstacle in cloud adoption and therefore needs to be addressed. Researchers and practitioners (R&P) have discussed various security challenges for SaaS along with possible solutions. However, no research study exists that systematically accumulates and analyzes the security challenges and solutions. To fill this gap and provide the state-of-the-art (SOTA) picture of SaaS security, this study provides a comprehensive multivocal literature review (MVLR), including SaaS security issues/challenges and best practices for mitigating these security issues. We identified SaaS security issues/challenges and best practices from the formal literature (FL) as well as the grey literature (GL) to evaluate whether R&P is on the same page or if controversies exist. A total of 93 primary studies were identified, of which 58 are from the FL and 35 belong to the GL. The studies are from the last ten years, from 2010 to 2021. The selected studies were evaluated and analyzed to identify the key security issues faced by SaaS computing and to be aware of the best practices suggested by R&P to improve SaaS security. This MVLR will assist SaaS users to identify the many areas in which additional research and development in SaaS security is required. According to our study findings, data breaches/leakage, identity and access management, governance and regulatory compliance/SLA compliance, and malicious insiders are the key security challenges with the maximum frequency of occurrence in both FL and GL. On the other hand, R&P agree that up-to-date security controls/standards, the use of strong encryption techniques, regulatory compliance/SLA compliance, and multifactor authentication are the most important solutions.

The maturity of open government data maturity: a multivocal literature review

PurposeThe aim of this study is to provide administrators in government institutions a roadmap to achieve benefits of open government data (OGD) by reviewing and classifying studies with assessment or maturity models (MMs) in the OGD domain with a Multivocal Literature Review (MLR).Design/methodology/approachTo achieve this goal, the authors conducted an MLR that includes data from not only the formal literature but also the grey literature (e.g. white papers and online documents).FindingsOut of 3,569 sources, 81 studies were selected by following the elimination scheme and assessing sources by relevance and methodology. As a result of the quality assessment of the identified MMs based on predefined criteria, unambiguity, comparability, repeatability, completeness, clearness and objectivity, it was observed that there are a limited number of MMs in this domain and none of them fully satisfies the requirements.Originality/valueThis study is likely the first MLR on OGD domain. This MLR serves as a first step for future research on OGD assessment and MMs by presenting the need to establish a holistic approach covering all OGD dimensions, creation of an objective assessment method, prescriptive properties, and empirical evaluation demonstrating the applicability and usefulness at different scope levels.

Eliciting Best Practices for Collaboration with Computational Notebooks

Despite the widespread adoption of computational notebooks, little is known about best practices for their usage in collaborative contexts. In this paper, we fill this gap by eliciting a catalog of best practices for collaborative data science with computational notebooks. With this aim, we first look for best practices through a multivocal literature review. Then, we conduct interviews with professional data scientists to assess their awareness of these best practices. Finally, we assess the adoption of best practices through the analysis of 1,380 Jupyter notebooks retrieved from the Kaggle platform. Findings reveal that experts are mostly aware of the best practices and tend to adopt them in their daily work. Nonetheless, they do not consistently follow all the recommendations as, depending on specific contexts, some are deemed unfeasible or counterproductive due to the lack of proper tool support. As such, we envision the design of notebook solutions that allow data scientists not to have to prioritize exploration and rapid prototyping over writing code of quality.

Accessibility Challenges in OER and MOOC: MLR Analysis Considering the Pandemic Years

The review of state of the art on creating and managing learning resources and accessible Open Educational Resources (OER) and Massive Open Online Courses (MOOC) is a topic that cannot only consider formal literature. The evidence and lack of a measurement consensus require the inclusion of contextual information, corroborating scientific results with practical experiences. For this reason, this article presents a review of accessibility models, OER and MOOC, considering the gray literature to capture experiences and trying to establish a shared understanding of the terminology commonly used in research on virtual accessibility and its impact on higher education. The bibliographic review relies on analyzing articles and scientific publications related to the topic following the Multivocal Literature Review (MLR) format. The results of this review establish that it is possible to apply accessibility review methodologies with transversal actions in the creation and management of learning resources and MOOCs. The research is related to one of the seventeen sustainable development goals defined by the United Nations to ensure inclusive and equitable quality education and promote lifelong learning opportunities for all.

Quality Assurance in E-Learning: A Proposal from Accessibility to Sustainability

Given the importance of developing and offering accessible education for all, indispensable aspects of education for sustainable development (ESD) are needed. This study addresses that need by proposing a quality self-assessment for virtual education from an accessibility perspective. This proposal is based on previous literature about quality assurance in e-learning that considered accessibility and its application in the field of higher education. The bibliographic review was conducted by following Multivocal Literature Review (MLR) guidelines. The initial search returned 999 items from 5 academic databases and 32,200 professional sources from Google. After reviewing the sources, 37 of them were included. Then, the accessibility criteria were identified and integrated into an evaluation model. Such a model is divided into four dimensions: (1) organization, (2) student body, (3) teaching, and (4) infrastructure. The model also includes a set of standards (16), requirements (48), and evidence (63) that apply to each dimension. Moreover, self-assessment guidelines for accessible virtual education were proposed. They included a conceptual and theoretical framework, a self-assessment model, and a methodology for applying the model. The methodology included five phases: planning, model tuning or refinement of the model, evaluation, results, and continuous improvement. As future work, the implementation and validation of the guidelines will be carried out.

A teamwork effectiveness model for agile software development

Teamwork is crucial in software development, particularly in agile development teams which are cross-functional and where team members work intensively together to develop a cohesive software solution. Effective teamwork is not easy; prior studies indicate challenges with communication, learning, prioritization, and leadership. Nevertheless, there is much advice available for teams, from agile methods, practitioner literature, and general studies on teamwork to a growing body of empirical studies on teamwork in the specific context of agile software development. This article presents the agile teamwork effectiveness model (ATEM) for colocated agile development teams. The model is based on evidence from focus groups, case studies, and multi-vocal literature and is a revision of a general team effectiveness model. Our model of agile teamwork effectiveness is composed of shared leadership, team orientation, redundancy, adaptability, and peer feedback. Coordinating mechanisms are needed to facilitate these components. The coordinating mechanisms are shared mental models, communication, and mutual trust. We critically examine the model and discuss extensions for very small, multi-team, distributed, and safety-critical development contexts. The model is intended for researchers, team members, coaches, and leaders in the agile community.

Software-Engineering Design Patterns for Machine Learning Applications

In this study, a multivocal literature review identified 15 software-engineering design patterns for machine learning applications. Findings suggest that there are opportunities to increase the patterns’ adoption in practice by raising awareness of such patterns within the community.

Guidelines to Establish an Office of Student Accessibility Services in Higher Education Institutions

The objective of this paper is to propose a set of guidelines to establish an office of Student Accessibility Services (SAS) in Higher Education Institutions (HEIs). The proposed guidelines help to integrate disjointed knowledge to facilitate its interpretation and implementation during deployment of basic support services in favor of students with disability. These guidelines can help to mitigate complexity in providing SAS for the first time in HEIs. These guidelines cover both the design and implementation of an office of SAS and its management. Knowledge was found through a multivocal literature review (MLR), which allowed to capture not only academic approaches but also vantage points and experiences from practice. Key concepts and aspects were organized into eight components (five related to the design and implementation, and three associated with the management context). An expert appraisal method was used as a proof of concept, which complemented a previously performed preliminary implementation example. Obtained results demonstrated the pertinence of the conceptual proposal and confirmed guidelines capability for full implementation in a real-world scenario.

Improving test automation maturity: A multivocal literature review

AbstractMature test automation is key for achieving software quality at speed. In this paper, we present a multivocal literature review with the objective to survey and synthesize the guidelines given in the literature for improving test automation maturity. We selected and reviewed 81 primary studies, consisting of 26 academic literature and 55 grey literature sources. From primary studies, we extracted 26 test automation best practices (e.g., Define an effective test automation strategy, Set up good test environments, and Develop high‐quality test scripts) and collected many pieces of advice (e.g., in forms of implementation/improvement approaches, technical techniques, concepts, and experience‐based heuristics) on how to conduct these best practices. We made main observations: (1) There are only six best practices whose positive effect on maturity improvement have been evaluated by academic studies using formal empirical methods; (2) several technical related best practices in this MLR were not presented in test maturity models; (3) some best practices can be linked to success factors and maturity impediments proposed by other scholars; (4) most pieces of advice on how to conduct proposed best practices were identified from experience studies and their effectiveness need to be further evaluated with cross‐site empirical evidence using formal empirical methods; (5) in the literature, some advice on how to conduct certain best practices are conflicting, and some advice on how to conduct certain best practices still need further qualitative analysis.

Dairy Farm Management Information Systems

Nowadays, different types of farm management information systems (FMIS) are being used in practice in several sectors of farming, such as dairy, arable, fruits, vegetables, and meat farming. The goal of this research is to identify, evaluate, and synthesize existing FMISs in the Dutch dairy sector and present the state–of–the–art. We performed a multivocal literature review (MLR) to find sources both in scientific and grey literature. A grey literature search was adopted because most of the FMISs were not reported in the scientific literature. To support and improve the effectiveness of the MLR process, an online survey was first sent to Dutch dairy farmers to identify the FMISs that are being used in practice. With the help of the MLR process, we identified 50 FMISs used by Dutch dairy farmers. We identified 33 features of these FMISs and listed the advantages and disadvantages of the FMISs.

NoOps – A Multivocal literature review

Traditionally, an organization had to have in-house servers and hardware to build a web application. This evolved into Cloud computing where the possibility for cost reduction and scalable data storage became a reality. With the introduction of cloud computing came a concept known as NoOps, or No Operations. This paper aims to take a closer look into what NoOps is and the benefits and challenges of NoOps. The authors identified three RQs that could help to give more insight into NoOps. Further we discussed the findings and RQs and lay out the way forward for future studies into NoOps. We also looked at artificial intelligence (AI) and how AI seems to be heavily linked with a true NoOps environment. With the lack of scientific studies into NoOps, a Multivocal literature review was selected as the method used to investigate the concept and its implications. We try to show voices both for and against NoOps. Further, we try to look at a misconception of what NoOps really is, what true NoOps could be. Finally we look at what requirements there are for companies wanting to go NoOps, and discuss the possibility that many companies unknowingly are moving towards a NoOps environment.

Introduction to the Special Issue on: Grey Literature and Multivocal Literature Reviews (MLRs) in software engineering

In parallel to academic (peer-reviewed) literature (e.g., journal and conference papers), an enormous extent of grey literature (GL) has accumulated since the inception of software engineering (SE). GL is often defined as “literature that is not formally published in sources such as books or journal articles”, e.g., in the form of trade magazines, online blog-posts, technical reports, and online videos such as tutorial and presentation videos. GL is typically produced by SE practitioners. We have observed that researchers are increasingly using and benefitting from the knowledge available within GL. Related to the notion of GL is the notion of Multivocal Literature Reviews (MLRs) in SE, i.e., a MLR is a form of a Systematic Literature Review (SLR) which includes knowledge and/or evidence from the GL in addition to the peer-reviewed literature. MLRs are useful for both researchers and practitioners because they provide summaries of both the state-of-the-art and -practice in a given area. MLRs are popular in other fields and have started to appear in SE community. It is timely then for a Special Issue (SI) focusing on GL and MLRs in SE. From the pool of 13 submitted papers, and after following a rigorous peer review process, seven papers were accepted for this SI. In this introduction we provide a brief overview of GL and MLRs in SE, and then a brief summary of the seven papers published in this SI.