Challenges Of Software Testing Research Articles

Challenges in Software Testing

Challenges in Software Testing

Exploring the challenges in software testing of the 5G system at Nokia: A survey

The ever-growing size and complexity of industrial software products pose significant quality assurance challenges to engineering researchers and practitioners, despite the constant effort to increase knowledge and improve the processes. 5G technology developed by Nokia is one example of such a grand and highly complex system with improvement potential. The following paper provides an overview of the current quality assurance processes used by Nokia to develop the 5G technology and provides insight into the most prominent challenges by an evaluation of perceived importance, urgency, and difficulty to understand the future opportunities. Nokia mode of operation, briefly introduced in this paper, has been subjected to extensive analysis by a selected group of experienced test-oriented professionals to define the most critical areas of concern. Secondly, the identified problems were evaluated by Nokia gNB system-level test professionals in a dedicated survey. The questionnaire was completed by 312 out of 2935 (10.63%) possible respondents. The challenges are seen as the most important and urgent: customer scenario testing, performance testing, and competence ramp-up. Challenges seen as the most difficult to solve are low occurrence failures, hidden feature dependencies, and hardware configuration-specific problems. Our research identified several improvement areas in the quality assurance processes used to develop the 5G technology by determining the most important and urgent problems that at the same time have a low perceived difficulty. Such initiatives are attractive from a business perspective. On the other hand, challenges seen as the most impactful yet difficult may be of interest to the academic research community. • Our survey, designed to elicit understanding of the most significant challenges in testing large and complex 5G gNB software system, received 312 out of 2935 (10.63%) system-level test practitioners in Nokia. • Challenges seen as having the highest importance are: customer scenario testing, performance testing, and competence ramp-up. • Challenges seen as having the highest urgency are: customer scenario testing, performance testing, and competence ramp-up. • Challenges seen as being the most difficult are: low occurrence failures, hidden feature dependencies, and hardware configuration-specific problems. • Obtained results imply which future improvement opportunities can have the most significant impact on the quality and cost of the software testing processes in the company. • Interestingly, respondents still seem to be not well familiar with software defect prediction models.

Exploring the challenges in software testing of the 5G system at Nokia: A Survey

Exploring the challenges in software testing of the 5G system at Nokia: A Survey

An Empirical Investigation to Understand the Issues of Distributed Software Testing amid COVID-19 Pandemic

Generally, software developers make errors during the distributed software development process; therefore, software testing delay is a significant concern. Some of the software mistakes are minor, but others may be costly or harmful. Since things can still go wrong—individuals encounter mistakes from time to time—there is a need to double-check any software we develop in a distributed environment. The current global pandemic, COVID-19, has exacerbated and generated new challenges for IT organizations. Many issues exist for distributed software testing that prevent the achievement of successful and timely risk reduction when several of the mechanisms on which testing is based are disrupted. The environment surrounding COVID-19 is quickly evolving on a daily basis. Moreover, the pandemic has exposed or helped to develop flaws in production systems, which obstruct software test completion. Although some of these issues were urgent and needed to be evaluated early during the distributed software development process, this paper attempts to capture the details that represent the current pandemic reality in the software testing process. We used a Fuzzy TOPSIS-based multiple-criteria decision-making approach to evaluate the distributed software testing challenges. The statistical findings show that data insecurity is the biggest challenge for successful distributed software testing.

Simulator Coupled with Distributed Co-Simulation Protocol for Automated Driving Tests

To meet the challenges in software testing for automated vehicles, such as increasing system complexity and an infinite number of operating scenarios, new simulation methods must be developed. Closed-loop simulations for automated driving (AD) require highly complex simulation models for multiple controlled vehicles with their perception systems as well as their surrounding context. For the realization of such models, different simulation domains must be coupled with co-simulation. However, widely supported model integration standards such as functional mock-up interface (FMI) lack native support for distributed platforms, which is a key feature for AD due to the computational intensity and platform exclusivity of certain models. The newer FMI companion standard distributed co-simulation protocol (DCP) introduces platform coupling but must still be used in conjunction with AD co-simulations. As part of an assessment framework for AD, this paper presents a DCP compliant implementation of an interoperable interface between a 3D environment and vehicle simulator and a co-simulation platform. A universal Python wrapper is implemented and connected to the simulator to allow its control as a DCP slave. A C-code-based interface enables the co-simulation platform to act as a DCP master and to realize cross-platform data exchange and time synchronization of the environment simulation with other integrated models. A model-in-the-loop use case is performed with the traffic simulator CARLA running on a Linux machine connected to the co-simulation master xMOD on a Windows computer via DCP. Several virtual vehicles are successfully controlled by cooperative adaptive cruise controllers executed outside of CARLA. The standard compliance of the implementation is verified by exemplary connection to prototypic DCP solutions from 3rd party vendors. This exemplary application demonstrates the benefits of DCP compliant tool coupling for AD simulation with increased tool interoperability, reuse potential, and performance.

Embedding and classifying test execution traces using neural networks

Classifying test executions automatically as pass or fail remains a key challenge in software testing and is referred to as the test oracle problem. It is being attempted to solve this problem with supervised learning over test execution traces. A programme is instrumented to gather execution traces as sequences of method invocations. A small fraction of the programme's execution traces is labelled with pass or fail verdicts. Execution traces are then embedded as fixed length vectors and a neural network (NN) component that uses the line-by-line information to classify traces as pass or fail is designed. The classification accuracy of this approach is evaluated using subject programs from different application domains—1. Module from Ethereum Blockchain, 2. Module from PyTorch deep learning framework, 3. Microsoft SEAL encryption library components, 4. Sed stream editor, 5. Nine network protocols from Linux packet identifier, L7-Filter and 6. Utilities library, commons-lang for Java. For all subject programs, it was found that test execution classification had high precision, recall and specificity, averaging to 93%, 94% and 96%, respectively, while only training with an average 14% of the total traces. Experiments show that the proposed NN-based approach is promising in classifying test executions from different application domains.

AI Testing: Ensuring a Good Data Split Between Data Sets (Training and Test) using K-means Clustering and Decision Tree Analysis

Artificial Intelligence and Machine Learning have been around for a long time. In recent years, there has been a surge in popularity for applications integrating AI and ML technology. As with traditional development, software testing is a critical component of a successful AI/ML application. The development methodology used in AI/ML contrasts significantly from traditional development. In light of these distinctions, various software testing challenges arise. The emphasis of this paper is on the challenge of effectively splitting the data into training and testing data sets. By applying a k-Means clustering strategy to the data set followed by a decision tree, we can significantly increase the likelihood of the training data set to represent the domain of the full dataset and thus avoid training a model that is likely to fail because it has only learned a subset of the full data domain.

Software Testing: Issues and Challenges of Artificial Intelligence & Machine Learning

The history of Artificial Intelligence and Machine Learning dates back to 1950’s. In recent years, there has been an increase in popularity for applications that implement AI and ML technology. As with traditional development, software testing is a critical component of an efficient AI/ML application. However, the approach to development methodology used in AI/ML varies significantly from traditional development. Owing to these variations, numerous software testing challenges occur. This paper aims to recognize and to explain some of the biggest challenges that software testers face in dealing with AI/ML applications. For future research, this study has key implications. Each of the challenges outlined in this paper is ideal for further investigation and has great potential to shed light on the way to more productive software testing strategies and methodologies that can be applied to AI/ML applications.

Software Testing for Machine Learning

Machine learning has become prevalent across a wide variety of applications. Unfortunately, machine learning has also shown to be susceptible to deception, leading to errors, and even fatal failures. This circumstance calls into question the widespread use of machine learning, especially in safety-critical applications, unless we are able to assure its correctness and trustworthiness properties. Software verification and testing are established technique for assuring such properties, for example by detecting errors. However, software testing challenges for machine learning are vast and profuse - yet critical to address. This summary talk discusses the current state-of-the-art of software testing for machine learning. More specifically, it discusses six key challenge areas for software testing of machine learning systems, examines current approaches to these challenges and highlights their limitations. The paper provides a research agenda with elaborated directions for making progress toward advancing the state-of-the-art on testing of machine learning.

Exploring the industry's challenges in software testing: An empirical study

AbstractContextSoftware testing is an important and costly software engineering activity in the industry. Despite the efforts of the software testing research community in the last several decades, various studies show that still many practitioners in the industry report challenges in their software testing tasks.ObjectiveTo shed light on industry's challenges in software testing, we characterize and synthesize the challenges reported by practitioners. Such concrete challenges can then be used for a variety of purposes, eg, research collaborations between industry and academia.MethodOur empirical research method is opinion survey. By designing an online survey, we solicited practitioners' opinions about their challenges in different testing activities. Our dataset includes data from 72 practitioners from eight different countries.ResultsOur results show that test management and test automation are considered the most challenging among all testing activities by practitioners. Our results also include a set of 104 concrete challenges in software testing that may need further investigations by the research community.ConclusionWe conclude that the focal points of industrial work and academic research in software testing differ. Furthermore, the paper at hand provides valuable insights concerning practitioners' “pain” points and, thus, provides researchers with a source of important research topics of high practical relevance.

Toward reducing the operational risk of emerging technologies adoption in central counterparties through end-to-end testing

Emerging technologies, such as artificial intelligence (AI) and distributed ledger technology, are increasingly being adopted by financial institutions, promising functional efficiency and cost reduction to stakeholders and users. However, the structural and functional changes associated with the technological transformation of software platforms pose significant operational risks. While some aspects of these risks are well known and studied (such as AI trustworthiness, data privacy and consistency, platform availability and information security), others are underestimated. The extreme complexity and non-deterministic nature of existing technology infrastructures still need to be addressed, as they will soon be inherited by the platforms built with new technologies. The only way to mitigate these risks is extensive end-to-end professional testing. This paper discusses the software-testing challenges of traditional central counter-parties as well as the risks, biases and problems related to new technologies. It also outlines a set of requirements for an end-to-end validation and verification solution aimed at the new generation of clearing platforms. Focusing on one of the most common use cases in the capital markets industry, this paper considers the challenges posed by the introduction of blockchain and AI into the post-trade area.

Testing Complexity in Component Based Software Engineering

The Component Based software development which reduces the time, budget and efforts compare to the traditional software development. The components are reusable, developed by third parties and available at Commercial-Of-The Self (COTS). Most recent technologies such embedded systems, mobile phones, control systems and other paradigms are developed with this concept. Software researchers should pay more attention on testing of Component Based technology. The software testing is the most commonly used technique for validating the product quality. It is something different from the traditional software development, due to the invisible source code and design make it very difficult to trace the faults. Software testing strategy is road map integrates test case design methods into well planned series of steps for successful construction of software. The selection right strategy at the right time will make the software product as robust. This paper highlights the various software testing challenges, difficulties, strategies and techniques in perspective of Component Based Software Development

Generation of complete test suites from mealy input/output transition systems

AbstractModel-based testing overcomes challenges in software testing by generating automated test cases from behavior models, e.g. finite state machine (FSM) and input/output transition system (IOTS). Despite the existence of methods for IOTSs, the problem of selection of test cases is an important and difficult topic. The current methods from IOTSs do not provide the same support offered by the existing theory for FSMs, as complete fault coverage. In this paper, we propose a test generation method for IOTSs based on the W method developed for FSMs. The basic idea is to generate a transition cover set and a characterization set and concatenate them to generate complete test suites for IOTSs in a bounded number of steps. The method generates test suites with complete fault coverage for a given fault domain and is targeted at a class of IOTSs, called mealy IOTS, which accepts inputs only in stable states. Results from a case study show the proposed algorithm can achieve better results than a classical method for IOTSs.

Research on Applied-Information Technology with Inter-Partition Integration Testing Strategy under IMA Architecture

IMA (Integrated Modular Avionics) system has been widely adopted by a new generation of aircraft. It with high complexity and integration will bring a new challenge in software testing. In this paper, it states what IMA is and how IMA system works. Because of isolation between partitions, the inter-partition communication is required to exchange information between the partitions. It also presents an integrated testing strategy based on UML sequence diagram according to the inter-partition integration of IMA. The method of test sequences generation is discussed emphatically. Finally, a case study is presented to testify the feasibility of our approach.

Uncertainty Analysis in Software Reliability Modeling by Bayesian Analysis with Maximum-Entropy Principle

In software reliability modeling, the parameters of the model are typically estimated from the test data of the corresponding component. However, the widely used point estimators are subject to random variations in the data, resulting in uncertainties in these estimated parameters. Ignoring the parameter uncertainty can result in grossly underestimating the uncertainty in the total system reliability. This paper attempts to study and quantify the uncertainties in the software reliability modeling of a single component with correlated parameters and in a large system with numerous components. Another characteristic challenge in software testing and reliability is the lack of available failure data from a single test, which often makes modeling difficult. This lack of data poses a bigger challenge in the uncertainty analysis of the software reliability modeling. To overcome this challenge, this paper proposes utilizing experts' opinions and historical data from previous projects to complement the small number of observations to quantify the uncertainties. This is done by combining the maximum-entropy principle (MEP) into the Bayesian approach. This paper further considers the uncertainty analysis at the system level, which contains multiple components, each with its respective model/parameter/ uncertainty, by using a Monte Carlo approach. Some examples with different modeling approaches (NHPP, Markov, Graph theory) are illustrated to show the generality and effectiveness of the proposed approach. Furthermore, we illustrate how the proposed approach for considering the uncertainties in various components improves a large-scale system reliability model.

Surviving the top ten challenges of software testing: a people-oriented approach. William E. Perry and Randall W. Rice. Published by Dorset House, New York, U.S.A., 1997. ISBN: 0-932633-38-2, 202 pages. Price: U.S.A. $33.45, Soft Cover

Software Testing, Verification and ReliabilityVolume 9, Issue 2 p. 136-137 Book ReviewFree Access Surviving the top ten challenges of software testing: a people-oriented approach. William E. Perry and Randall W. Rice. Published by Dorset House, New York, U.S.A., 1997. ISBN: 0-932633-38-2, 202 pages. Price: U.S.A. $33.45, Soft Cover Dorothy Graham, Dorothy Graham Grove Consultants, Grove House, 40 Ryles Park Road, Macclesfield, Cheshire SK11 8AH, U.KSearch for more papers by this author Dorothy Graham, Dorothy Graham Grove Consultants, Grove House, 40 Ryles Park Road, Macclesfield, Cheshire SK11 8AH, U.KSearch for more papers by this author First published: 21 June 1999 https://doi.org/10.1002/(SICI)1099-1689(199906)9:2<136::AID-STVR182>3.0.CO;2-0AboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat No abstract is available for this article. Volume9, Issue2June 1999Pages 136-137 RelatedInformation