Multivocal Literature Review Research Articles

DevSecOps practices and tools

Nowadays, software development happens at a fast pace. At the same time, Information Technology organizations face higher demands and competition while struggling with external threats such as cyberattacks. Therefore, many organizations adopt DevOps as a working culture to improve their Software Development Lifecycle (SDL). However, the success of DevOps adoption remains inconsistent, and recently, IEEE introduced a DevOps standard that might help improve DevOps adoption. The standard mentions DevSecOps as the security aspect of DevOps, adding security practices to the SDL from inception, but what are these practices or capabilities? Which tools can be used to implement these practices? Therefore, a Multivocal Literature Review was performed to identify DevSecOps practices and their definitions, and which tools can be used to implement them.

An extensive multivocal literature review of blockchain technology: Evolution, challenges, platforms, security, and interoperability

AbstractBlockchain technology has gained enormous interest from industry and academia recently. Technology enthusiasts are exploring its use case beyond cryptocurrencies and claim that blockchain technology can overcome the inefficiencies of centralized systems. In this study, we continue the work of previous authors, aiming to provide a more comprehensive understanding of the technical aspects of blockchain. This study is the first of its kind to review and analyze the current status of different technical aspects of blockchain technology influencing its adoption. We performed an extensive multivocal review to (i) demonstrate the progress of blockchain, (ii) discuss the challenges related to the wide‐scale adoption of the technology, (iii) present a detailed analysis of blockchain platforms, (iv) highlight security and interoperability issues followed by the solutions proposed in the literature. We have considered 259 peer‐reviewed research papers and the gray literature related to 40 blockchain platforms to provide an in‐depth analysis of blockchain technology. In conclusion, this comprehensive survey provides a holistic view of blockchain technology's progress. It identifies challenges, trends, and future research directions, serving as a valuable resource for researchers and practitioners seeking to navigate the dynamic blockchain landscape.

A multivocal literature review of digital twins, architectures, and elements in civil engineering

Recent structural health monitoring (SHM) strategies in civil engineering increasingly leverage digital twins, which digitally represent the structures being monitored as well as the SHM systems installed on the structures. Despite the widespread adoption of digital twin applications in recent years in SHM, there is a lack of agreement on a common definition of digital twins. Furthermore, there is no consensus on digital twin architectures and on the internal elements that constitute digital twins. A common digital twin definition would advance digital twin implementations and operability, and insights into digital twin architectures and digital twin elements would be vital to enhance the reliability and performance of digital-twin-based SHM systems. A significant number of digital twin definitions have been proposed and a plethora of reviews have been published; however, little emphasis has been given to digital twin architectures and internal elements. This paper presents a multivocal review of digital twins in civil engineering, aiming to provide a panorama of the digital twin landscape in civil engineering with explicit insights into the architectures and internal elements used in digital twin applications. From a methodological standpoint, the review follows a twofold approach that encompasses (i) peer-reviewed, indexed literature (“white literature”) as well as (ii) non-indexed sources (“gray literature”) that include industrial digital twin applications. Besides the multivocal review, a generic digital twin reference architecture is drawn from the review results and a digital twin definition is formulated both in an informal and formal (i.e., mathematical) manner. It is expected that the generic reference architecture and the definitions proposed in this study may serve as a blueprint for digital-twin-based SHM applications, with significant implications for researchers, practitioners, and policymakers in structural health monitoring.

Multi-central bank digital currencies arrangements: a multivocal literature review

PurposeThe present study aims to systematically synthesize the academic and industrial literature on multi-central bank digital currencies (m-CBDCs) arrangements.Design/methodology/approachThe study adopted a unique multivocal literature review methodology that considers both white and grey literature. For white literature searches, the study relied on Scopus, Web of Science (WOS), and Google Scholar bibliometric databases; for grey literature searches, the study used the Google search engine.FindingsThe findings of the study illustrated that M-CBDC arrangements, through various design options, have the potential to revolutionize the contemporary international payment system. M-CBDC arrangements will lead to more integrated financial systems and promote economic growth. However, m-CBDC arrangements will also have serious macroeconomic implications, such as contagion and currency substitution risks.Research limitations/implicationsThe present review is one of the earliest reviews of m-CBDC arrangements. In addition, the findings of the study offer valuable insights for both academicians and policymakers.Originality/valueThe study is also one of the pioneer studies in management studies that apply a multivocal literature review methodology.

Quantum computing challenges and solutions in software industry—A multivocal literature review

AbstractQuantum computing (QC) hinged upon the bedrock principles of quantum theory and holds promise for reforming a large number of industries. The researcher in this area aims to deliver a comprehensive understanding of the current state of the art and future trajectories of QC. The authors have discovered that most academic studies have concentrated upon dissecting specific aspects of QC. This discernment underscores the exigency of identifying challenges that might impede the seamless integration of QC within the software industry. Moreover, it becomes crucial to ascertain the panoply of solutions/practices required to overcome these barriers. A comprehensive multi‐vocal literature review was performed and culled a total of 49 academic papers for data extraction. A total of 13 challenges encountered by organisations were identified during the adoption of QC. Subsequently, these challenges were examined deeply and determined that five of them are the most critical, these are ‘Lack of quantum specific algorithms, dev and testing methodologies’, ‘Difficult compilation and debugging’, ‘Lack of development tools and technology’, ‘Lack of development guidelines & Quality Assurance Standards’ and ‘Lack of professional expert’, together founding over 30% of occurrences. These challenges from various perspectives were evaluated, including time frame, methodology, geographical region and publication platform. To address these barriers and implement the QC in software industry effectively, a total of 53 practices/solutions. This research aims to share valuable knowledge to simplify and amplify quantum application development.

DevOps Metrics and KPIs: A Multivocal Literature Review

Context: Information Technology organizations are aiming to implement DevOps capabilities to fulfill market, customer, and internal needs. While many are successful with DevOps implementation, others still have difficulty measuring DevOps success in their organization. As a result, the effectiveness of assessing DevOps remains erratic. This emphasizes the need to withstand management in measuring the implementation process with suitable DevOps Metrics. But what are these metrics? Objective: This research seeks to provide relevant DevOps Metrics to facilitate the efficiency of DevOps adoption and better analyze DevOps performance in enterprises. Method: A Multivocal Literature Review (MLR) is conducted, with 139 documents gathered and thoroughly examined from throughout the community, including books, scientific articles, white papers, conferences, among others. Results: This article conducts an extensive and rigorous MLR, contributing with a definition of DevOps Metrics, 22 main metrics, their definitions, importance, and categorization in sets of Key Performance Indicators, as well as exposing clear indicators on how to improve them. It is also discussed how metrics could be put into practice and what constitutes a change in the context of DevOps Metrics. The study’s outcomes will assist researchers and practitioners understand DevOps Metrics and how to better implement them.

Identifying the primary dimensions of DevSecOps: A multi-vocal literature review

Context:Security as a key non-functional requirement of software development is often ignored and devalued in DevOps programs, with security seen as an inhibitor to high velocity required in DevOps implementation. Hence, the DevSecOps approach as a security-orientated expansion to DevOps, has aimed to integrate security into DevOps implementation by promoting collaboration among development, operation and security teams. DevSecOps is a topical concept and rapidly emerging area of practice in both academic and industrial settings. Objective:We reviewed both the white and grey literature to identify recent researches and practical trends of DevSecOps, aiming to: (a) review, document and analyze the current state of DevSecOps in the existing literature; (b) investigate the application of DevSecOps in Global Software Engineering (GSE) contexts. Method:A Multi-vocal Literature Review on DevSecOps and its global application was conducted, by executing a dual-track strategy including white (104 studies) and grey (43 studies) literature from 2012 to 2021. A Thematic Analysis was performed to identify, synthesize and analyze the themes within data for reporting the MLR results. Results:Through the Multi-vocal Literature Review and Thematic Analysis, this paper identifies five major aspects of DevSecOps (Definitions, Challenges, Practices, Tools/Technologies, and Metrics/Measurement); collects related themes of each aspect; and generates a Challenge-Practice-Tool-Metric (CPTM) model by integrating the themes of the latter four aspects within a lifecycle model. Moreover, an unexplored area relating to the global application of DevSecOps has been identified. Conclusion:Based on MLR results, a CPTM (Challenge-Practice-Tool-Metric) model is built to reveal the current status of DevSecOps. The model provides a breakdown and a broad landscape of DevSecOps, from which researchers and practitioners may select an area of focus to improve their knowledge or practice. With DevSecOps spanning the many stages of the lifecycle, we believe the model will enable emphases and absences such as global aspects to be investigated.Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

From Microservice to Monolith: A Multivocal Literature Review

Recently, the phenomenon of switching back from microservice to monolith has increased in frequency, leading to intense debate in the industry. In this paper, we conduct a multivocal literature review to investigate reasoning and key aspects to pay attention to when switching back and analyze other practitioners’ opinions. The results show four cases of switching back from microservice to monolith: Istio control plane, Amazon Prime Video monitoring service, Segment, and InVision. The five main reasons that led to switching back are cost, complexity, scalability, performance, and organization. During the switching back process, six key aspects need to be addressed: (1) stopping the development of more services, (2) consolidating and testing paths, (3) unifying data storage, (4) implementing the message bus principle, (5) giving up diverse techniques, and (6) learning to use modular design principles. As to the practitioners’ opinions, they had mixed views about the switching back phenomenon. However, most thought that switching back required consideration of the actual system situation and principles. These results pave the way for further research and guide researchers and companies through the process of switching back from microservice to monolith.

Responsible AI Pattern Catalogue: A Collection of Best Practices for AI Governance and Engineering

Responsible Artificial Intelligence (RAI) is widely considered as one of the greatest scientific challenges of our time and is key to increase the adoption of Artificial Intelligence (AI). Recently, a number of AI ethics principles frameworks have been published. However, without further guidance on best practices, practitioners are left with nothing much beyond truisms. In addition, significant efforts have been placed at algorithm level rather than system level, mainly focusing on a subset of mathematics-amenable ethical principles, such as fairness. Nevertheless, ethical issues can arise at any step of the development lifecycle, cutting across many AI and non-AI components of systems beyond AI algorithms and models. To operationalize RAI from a system perspective, in this article, we present an RAI Pattern Catalogue based on the results of a multivocal literature review. Rather than staying at the principle or algorithm level, we focus on patterns that AI system stakeholders can undertake in practice to ensure that the developed AI systems are responsible throughout the entire governance and engineering lifecycle. The RAI Pattern Catalogue classifies the patterns into three groups: multi-level governance patterns, trustworthy process patterns, and RAI-by-design product patterns. These patterns provide systematic and actionable guidance for stakeholders to implement RAI.

Taxonomy of inline code comment smells

Code comments play a vital role in source code comprehension and software maintainability. It is common for developers to write comments to explain a code snippet, and commenting code is generally considered a good practice in software engineering. However, low-quality comments can have a detrimental effect on software quality or be ineffective for code understanding. This study aims to create a taxonomy of inline code comment smells and determine how frequently each smell type occurs in software projects. We conducted a multivocal literature review to define the initial taxonomy of inline comment smells. Afterward, we manually labeled 2447 inline comments from eight open-source projects where half of them were Java, and another half were Python projects. We created a taxonomy of 11 inline code comment smell types and found out that the smells exist in both Java and Python projects with varying degrees. Moreover, we conducted an online survey with 41 software practitioners to learn their opinions on these smells and their impact on code comprehension and software maintainability. The survey respondents generally agreed with the taxonomy; however, they reported that some smell types might have a positive effect on code comprehension in certain scenarios. We also opened pull requests and issues fixing the comment smells in the sampled projects, where we got a 27% acceptance rate. We share our manually labeled dataset online and provide implications for software engineering practitioners, researchers, and educators.

Organizations' readiness for insider attacks: A process‐oriented approach

AbstractContextOrganizations constantly strive to protect their assets from outsider attacks by implementing various security controls, such as data encryption algorithms, intrusion detection software, firewalls, and antivirus programs. Unfortunately, attackers strike not only from outside the organization but also from within. Such internal attacks are called insider attacks or threats, and the people responsible for them are insider attackers or insider threat agents. Insider attacks pose more significant risks and can result in greater organizational losses than outsider attacks. Thus, every organization should be vigilant regarding such attackers to protect its valuable resources from harm. Finding solutions to protect organizations from such attacks is critical. Despite the importance of this topic, little research has been conducted on providing solutions to mitigate insider attacks.ObjectiveThis study aims to develop an organizational readiness model to assess an organization's readiness for insider attacks.MethodWe conducted a multivocal literature review to identify practices that can be used to assess organizations' readiness against insider attacks. These practices were grouped into different knowledge areas of insider attacks for organizations. The insider attack readiness model was developed using identified best practices and knowledge areas: compliance, top management, human resources, and technical.ResultsThis model was evaluated at two levels—academic and real‐world environments. The evaluation results show that the proposed model can identify organizations' readiness against insider attacks.ConclusionThe proposed model can guide organizations through a secure environment against insider attacks.

Text readability in augmented reality: a multivocal literature review

Augmented reality (AR) is making its way into many sectors. Its rapid evolution in recent years has led to the development of prototypes demonstrating its effectiveness. However, to be able to push these prototypes to the scale of fully usable applications, it is important to ensure the readability of the texts they include. To this end, we conducted a multivocal literature review (MLR) to determine the text parameters a designer can tune, as well as the contextual constraints they need to pay attention to, in relation to Optical See-Through (OST) and Video See-Through (VST) displays. We also included guidelines from device manufacturing and game engines sites to compare the current state of research in the academic and industrial worlds. The results show that parameters pertaining more to letter legibility have been extensively studied (e.g., color and size), while those pertaining to the whole text still require further research (e.g., alignment or space between lines). The former group of parameters, and their associated constraints, were assembled in the form of two decision trees to facilitate implementation of AR applications. Finally, we also concluded that there was a lack of alignment between academic and industrial recommendations.

Cross-Chain Smart Contract Invocations: A Systematic Multi-Vocal Literature Review

The introduction of smart contracts has expanded the applicability of blockchains to many domains beyond finance and cryptocurrencies. Moreover, different blockchain technologies have evolved that target special requirements. As a result, in practice, often a combination of different blockchain systems is required to achieve an overall goal. However, due to the heterogeneity of blockchain protocols, the execution of distributed business transactions that span several blockchains leads to multiple interoperability and integration challenges. Therefore, in this article, we examine the domain of Cross-Chain Smart Contract Invocations (CCSCIs), which are distributed transactions that involve the invocation of smart contracts hosted on two or more blockchain systems. We conduct a systematic multi-vocal literature review to get an overview of the available CCSCI approaches. We select 20 formal literature studies and 13 high-quality gray literature studies, extract data from them, and analyze it to derive the CCSCI Classification Framework. With the help of the framework, we group the approaches into two categories and eight subcategories. The approaches differ in multiple characteristics, e.g., the mechanisms they follow, and the capabilities and transaction processing semantics they offer. Our analysis indicates that all approaches suffer from obstacles that complicate real-world adoption, such as the low support for handling heterogeneity and the need for trusted third parties.

A Multivocal Literature Review on Non-Technical Debt in Software Development: {A}n Insight into Process, Social, People, Organizational, and Culture Debt

Software development encompasses various factors beyond technical considerations. Neglecting non-technical elements like individuals, processes, culture, and social and organizational aspects can lead to debt-like characteristics that demand attention. Therefore, we introduce the non-technical debt (NTD) concept to encompass and explore these aspects. This indicates the applicability of the debt analogy to non-technical facets of software development. Technical debt (TD) and NTD share similarities and often arise from risky decision-making processes, impacting both software development professionals and software quality. Overlooking either type of debt can lead to significant implications for software development success. The current study conducts a comprehensive multivocal literature review (MLR) to explore the most recent research on NTD, its causes, and potential mitigation strategies. For analysis, we carefully selected 40 primary studies among 110 records published until October 1, 2022. The study investigates the factors contributing to the accumulation of NTD in software development and proposes strategies to alleviate the adverse effects associated with it. This MLR offers a contemporary overview and identifies prospects for further investigation, making a valuable contribution to the field. The findings of this research highlight that NTD’s impacts extend beyond monetary aspects, setting it apart from TD. Furthermore, the findings reveal that rectifying NTD is more challenging than addressing TD, and its consequences contribute to the accumulation of TD. To avert software project failures, a comprehensive approach that addresses NTD and TD concurrently is crucial. Effective communication and coordination play a vital role in mitigating NTD, and the study proposes utilizing the 3C model as a recommended framework to tackle NTD concerns.

CULTIVATING DATA OBSERVABILITY AS THE NEXT FRONTIER OF DATA ENGINEERING: A PATH TO ENHANCED DATA QUALITY, TRANSPARENCY, AND DATA GOVERNANCE IN THE DIGITAL AGE

In the age of increasing process automation and data-driven decision-making, ensuring the reliability, transparency and usability of data is of paramount importance. In this context, the concept of "data observability" has aroused the interest of practitioners and there is also a lot of grey content on it. On the other hand, there is a lack of academic effort to define and build on the concept. This conference paper will therefore examine the importance of "data observability" in modern data ecosystems. The focus is on the definition and characterisation of the concept, the differentiation from other concepts (e.g. data quality, data monitoring, data discovery, data operations) and why this concept appears to be so important in an increasingly data-driven world. In addition, the concept of "data observability" is categorised in the dynamically developing research field of data governance. For this purpose, a multivocal literature review (MLR) was conducted, a form of systematic literature review (SLR) which, in addition to the published (formal) literature (e.g. journal and conference papers), also includes and brings together the grey literature (e.g. blog posts, videos and white papers). The results show that the concept of "data observability" has the potential to revolutionise the way companies manage, analyse and derive insights from their data, ultimately leading to more informed and confident decision-making. Nevertheless, there is still plenty of room for further research into the specific contribution to better data and therefore better business processes and decisions.

Creating Serverless Web Applications with AWS

Abstract: Serverless computing has emerged as a groundbreaking innovation, allowing organizations to alleviate the complexities associated with provisioning, scaling, and managing infrastructure. The surge in serverless adoption poses challenges in choosing optimal solutions amid a myriad of proposed designs by experts. This paper categorizes serverless patterns using a multivocal literature review, highlighting benefits and challenges in coordination, aggregation, event management, availability, communication, and validation. To illustrate the practical application of serverless computing, we present a case study involving the use of AWS Lambda, Amazon DynamoDB, and other services to process real-time data streams for a fictional ride-sharing company. The study showcases the advantages of serverless computing, emphasizing its ability to free developers from server-related concerns, enabling them to focus on building scalable and reliable applications. This paper contributes to the understanding of serverless computing's role in the broader context of cloud technologies and its application in real-world scenarios

Defect Categorization in Compilers: A Multi-vocal Literature Review

Context: Compilers are the fundamental tools for software development. Thus, compiler defects can disrupt development productivity and propagate errors into developer-written software source code. Categorizing defects in compilers can inform practitioners and researchers about the existing defects in compilers and techniques that can be used to identify defects systematically. Objective: The goal of this paper is to help researchers understand the nature of defects in compilers by conducting a review of Internet artifacts and peer-reviewed publications that study defect characteristics of compilers. Methodology: We conduct a multi-vocal literature review (MLR) with 26 publications and 32 Internet artifacts to characterize compiler defects. Results: From our MLR, we identify 13 categories of defects, amongst which optimization defects have been the most reported defects in our artifacts publications. We observed 15 defect identification techniques tailored for compilers and no single technique identifying all observed defect categories. Conclusion: Our MLR lays the groundwork for practitioners and researchers to identify defects in compilers systematically.

A Multi-vocal Literature Review on challenges and critical success factors of phishing education, training and awareness

Background:Phishing is a malicious attempt by cyber attackers to steal personal information through deception. Phishing attacks are often aided by carefully crafted phishing emails, which can go undetected by automated anti-phishing tools due to their limited accuracy. Studies found that user education, training, and awareness can thwart phishing attacks. Understanding diverse interconnected challenges and critical success factors of phishing education, training, and awareness (PETA) approaches can help improve organizations’ defense against phishing. Objective:This study presents a comprehensive, structured view of the challenges and critical success factors of the design, implementation, and evaluation stages of PETA. Method:We have conducted a Multi-vocal Literature Review (MLR) by systematically collecting 53 academic studies and 16 grey studies from popular databases by following a well-known MLR guideline. Results:We identified 20 challenges and 23 critical success factors, some of which involve human-centric and socio-technical factors in PETA. Our findings point out the need for designing explainable anti-phishing systems and developing automated tools and platforms to conduct real-world phishing studies. Conclusion:Our systematic analysis of 69 studies has enabled us to highlight the need for addressing human-centric issues, incorporating users’ knowledge gaps, and adopting personalized approaches in PETA.

Data pipeline quality: Influencing factors, root causes of data-related issues, and processing problem areas for developers

Data pipelines are an integral part of various modern data-driven systems. However, despite their importance, they are often unreliable and deliver poor-quality data. A critical step toward improving this situation is a solid understanding of the aspects contributing to the quality of data pipelines. Therefore, this article first introduces a taxonomy of 41 factors that influence the ability of data pipelines to provide quality data. The taxonomy is based on a multivocal literature review and validated by eight interviews with experts from the data engineering domain. Data, infrastructure, life cycle management, development & deployment, and processing were found to be the main influencing themes. Second, we investigate the root causes of data-related issues, their location in data pipelines, and the main topics of data pipeline processing issues for developers by mining GitHub projects and Stack Overflow posts. We found data-related issues to be primarily caused by incorrect data types (33%), mainly occurring in the data cleaning stage of pipelines (35%). Data integration and ingestion tasks were found to be the most asked topics of developers, accounting for nearly half (47%) of all questions. Compatibility issues were found to be a separate problem area in addition to issues corresponding to the usual data pipeline processing areas (i.e., data loading, ingestion, integration, cleaning, and transformation). These findings suggest that future research efforts should focus on analyzing compatibility and data type issues in more depth and assisting developers in data integration and ingestion tasks. The proposed taxonomy is valuable to practitioners in the context of quality assurance activities and fosters future research into data pipeline quality.

Retrieving arXiv, SocArXiv, and SSRN metadata for initial review screening

Researchers around the globe invest a lot of time searching the literature for performing reviews (Systematic Literature Review (SLR), Multivocal Literature Review (MLR)). The steps to performing the review includes inclusion of the grey literature, preprints, and quality assessed non-peer reviewed literature (the purpose is to minimize the publication bias). The initial screening of the papers takes time and bibliographic information is only available online for the researcher(s). Objective of our study is to propose, design, and develop a method that will help the research community to download the basic information of the papers (title, abstract, author) for the searched query from arxiv, SSRN, and SocArxiv (Social Science ArXiv). We used Web scraping to extract data from the servers and save it in excel file. To retrieve the desired query from the databases, a Python code is used. Two methods have been discussed in the study to download the metadata of the searched query. We have used different queries (such as “grey literature”, “testing software”, and “python” etc.) to see the results of our proposed method. Furthermore, we cross-verified the results with the online search results of the databases. Initial results from the preliminary pilot evaluations show that it is a viable method to search, download, and shortlist the research articles information (title, abstract etc.) from arXiv,1 SSRN,2 and SocArXiv.3 For external validity more evaluations are needed.