Quality Of Test Cases Research Articles

SFIDMT-ART: A metamorphic group generation method based on Adaptive Random Testing applied to source and follow-up input domains

Context:The performance of metamorphic testing relates strongly to the quality of test cases. However, most related research has only focused on source test cases, ignoring follow-up test cases to some extent. In this paper, we identify a potential problem that may be encountered with existing metamorphic group generation algorithms. We then propose a possible solution to address this problem. Based on this solution, we design a new algorithm for generating effective source and follow-up test cases. Objective:To improve the performance (test effectiveness and efficiency) of metamorphic testing. Methods:We introduce the concept of the input-domain difference problem, which is likely to affect the performance of metamorphic group generation algorithms. We propose a new test-case distribution criterion for metamorphic testing to address this problem. Based on our proposed criterion, we further present a new metamorphic group generation algorithm, from a black-box perspective, with new distance metrics to facilitate this algorithm. Results:Our algorithm performs significantly better than existing algorithms, in terms of test effectiveness, efficiency and test-case diversity. Conclusions:Through experiments, we find that the input-domain difference problem is likely to affect the performance of metamorphic group generation algorithms. The experimental results demonstrate that our algorithm can achieve good test efficiency, effectiveness, and test-case diversity.

Implementation Genetic Algorithm for Optimization of Kotlin Software Unit Test Case Generator

Unit testing has a significant role in software development and its impacts depend on the quality of test cases and test data used. To reduce time and effort, unit test generator systems can help automatically generate test cases and test data. However, there is currently no unit test generator for Kotlin programming language even though this language is popularly used for android application developments. In this study, we propose and develop a test generator system that utilizes genetic algorithm (GA) and ANTLR4 parser. GA is used to obtain the most optimal test cases and data for a given Kotlin code. ANTLR4 parser is used to optimize the mutation process in GA so that the mutation process is not totally random. Our model results showed that the average value of code coverage in generated unit tests against instruction coverage is 95.64%, with branch coverage of 76.19% and line coverage of 96.87%. In addition, only two out of eight generated classes produced duplicate test cases with a maximum of one duplication in each class. Therefore, it can be concluded that our optimization with GA on the unit test generator is able to produce unit tests with high code coverage and low duplication.

Advanced testing and debugging support for reactive executable DSLs

Executable Domain-Specific Languages (xDSLs) allow the definition and the execution of behavioral models. Some behavioral models are reactive, meaning that during their execution, they accept external events and react by exposing events to the external environment. Since complex interaction may occur between the reactive model and the external environment, they should be tested as early as possible to ensure the correctness of their behavior. In this paper, we propose a set of generic testing facilities for reactive xDSLs using the standardized Test Description Language (TDL). Given a reactive xDSL, we generate a TDL library enabling the domain experts to write and run event-driven TDL test cases for conforming reactive models. To further support the domain expert, the approach integrates interactive debugging to help in localizing defects, and mutation analysis to measure the quality of test cases. We evaluate the level of genericity of the approach by successfully writing, executing, and analyzing 247 event-driven TDL test cases for 70 models conforming to two different reactive xDSLs.

Toward Improving the Quality of Mutation Operator and Test Case Effectiveness in Higher-Order Mutation Testing

Currently, there are many research studies that apply and improve mutation testing techniques including traditional mutation testing or first-order mutation testing, and higher-order mutation testing (HOMT) for evaluating the quality of the set of test data in particular, and the quality of test suites in general. The results of those studies have proven the effectiveness of mutation testing in the field of software testing. Mutation testing allows the quality of test cases to be automatically evaluated, thereby helping the testers to improve the quality in the design and execution of the software testing. Besides, these studies have also pointed out the main barriers in applying mutation testing techniques in practice. However, we are the first to introduce a method that can reduce the cost, but keep the quality of testing activity based on evaluating the quality of the mutation operator as well as the quality of the test cases. In this paper, we concentrate on two problems regarding higher-order mutation testing: Evaluating the quality of mutation operators as well as generated mutants and prioritizing test cases based upon its capability of killing mutants. This may help developers allocate suitably their resources during testing phase. The study of this paper is an extended version of our previous study titled “Evaluating Mutation Operator and Test Case Effectiveness by Means of Mutation Testing”, which is published in the proceedings of the 13th Asian Conference on Intelligent Information and Database Systems, ACIIDS 2021 (V. N. Do, Q. V. Nguyen and T. B. Nguyen, Evaluating Mutation Operator and Test Case Effectiveness by Means of Mutation Testing, in Intelligent Information and Database Systems. ACIIDS 2021, eds. N. T. Nguyen, S. Chittayasothorn, D. Niyato and B. Trawiński. Lecture Notes in Computer Science, Vol. 12672 (Springer, Cham), https://doi.org/10.1007/978-3-030-73280-6_66 ) to confirm the usefulness of our proposed method.

An initial investigation of the effect of quality factors on Agile test case quality through experts’ review

An initial investigation of the effect of quality factors on Agile test case quality through experts’ review

Modified Harris Hawks Optimization Based Test Case Prioritization for Software Testing

Generally, software testing is considered as a proficient technique to achieve improvement in quality and reliability of the software. But, the quality of test cases has a considerable influence on fault revealing capability of software testing activity. Test Case Prioritization (TCP) remains a challenging issue since prioritizing test cases is unsatisfactory in terms of Average Percentage of Faults Detected (APFD) and time spent upon execution results. TCP is mainly intended to design a collection of test cases that can accomplish early optimization using preferred characteristics. The studies conducted earlier focused on prioritizing the available test cases in accelerating fault detection rate during software testing. In this aspect, the current study designs a Modified Harris Hawks Optimization based TCP (MHHO-TCP) technique for software testing. The aim of the proposed MHHO-TCP technique is to maximize APFD and minimize the overall execution time. In addition, MHHO algorithm is designed to boost the exploration and exploitation abilities of conventional HHO algorithm. In order to validate the enhanced efficiency of MHHO-TCP technique, a wide range of simulations was conducted on different benchmark programs and the results were examined under several aspects. The experimental outcomes highlight the improved efficiency of MHHO-TCP technique over recent approaches under different measures.

A Quantitative Assessment of the Impact of Homogeneity in Personality Traits on Software Quality and Team Productivity

Software industry is increasingly focusing on the improvement of software quality and team performance to survive in a competitive and rapidly growing environment. Previous studies reveal the importance of social and human factors in software engineering. Despite the importance of these personality factors, only a handful of empirical studies have quantitatively evaluated the impact of personality traits on software quality and effectiveness of software engineering teams. This study quantifies the abstract notion of trait-wise team homogeneity based on the Five-Factor Model (FFM). Additionally, this study also aims to measure its impact on team productivity and software quality. Trait-wise team homogeneity measures the Team Homogeneity Index (THI) for each of the five personality traits of FFM. Therefore, five new metrics i.e. Openness THI (O-THI), Conscientiousness THI (C-THI), Extraversion THI (E-THI), Agreeableness THI (A-THI), and Neuroticism THI (N-THI) are proposed. The utility of these five metrics is evaluated by conducting experiments in academic and industrial environments for three different phases of the Software Development Life Cycle (SDLC) i.e. analysis and design, implementation, and testing. During the analysis and design phase, it was observed that teams (composed of either students or professionals) with greater values of C-THI and A-THI produced better quality design models and teams with greater values of C-THI were more productive. In the implementation phase, teams with higher values of O-THI, C-THI, and E-THI were noticeably more productive and produced better quality code. For the testing phase, teams with greater values of C-THI and A-THI produced better quality test cases and were more productive. Results obtained so far indicate that the newly proposed five metrics – O-THI, C-THI, E-THI, A-THI, and N-THI – appear to have the potential to aid managers and academics alike in building productive teams which can produce better quality software.

Test Case Quality Factors

The guarantee of software quality is very important. Thus, before a software is released to the end users, the flaws in the software should be detected by using high quality test cases. Currently, gauging the quality of test cases is carried out without any particular model and the criteria for good test cases is still unclear. Therefore, this study studies the literature using Systematic Literature Review (SLR) technique to identify the criteria of good test cases. The SLR considered papers between 2010 and 2018 in IEEE Xplore, ACM Digital Library, and Science Direct databases. Through the searching, it was found 310 papers are related. After filtering using exclusion and insertion criteria, 14 papers were considered for analysis. As a result, the test managed to identify 30 quality factors from the selected articles. These quality factors were additionally inspected, arranged and finished to be incorporated as the quality factors of test cases evaluation metrics.

Test Case Quality Factors: Content Analysis of Software Testing Websites

Software testing is anessentialprocess for ensuring thequality and reliability of software products. The efficiency of testing activities depends largely on the test case quality, which is considered as one of the major concerns of software testing. Unfortunately, at the moment there is no clear guideline that can be referred by software testers in producing good quality test cases. Hence, producing guideline is certainly required. To construct a pragmatic guideline, it is crucial to identify the factors that lead todesigninggood quality test cases. The existing test case quality factors are not comprehensive and need further investigation and improvement. Therefore,a content analysis was conducted to identify the test case qualityfactors from software testing experts point of view available in the software testing websites. The software testing websites provide explicit information about the quality of test cases in order to avoid the poor design of test cases. Thus, this study presents the outcomes of content analysis from 22 software testing websites which comprise of static content websites and blogs.Consequently, eight (8)factors and their corresponding 30 sub-factors were identified. Among the factors are documentation, manageability, maintainability, reusability, requirement quality, efficiency, tester knowledge, and effectiveness of test cases. These factors are useful to be referred by the practitioners in assuring the quality of the design test cases which implicitly can ensure the quality of the software products.

Quality assessment of crowdsourced test cases

Various software-engineering problems have been solved by crowdsourcing. In many projects, the software outsourcing process is streamlined on cloud-based platforms. Among software engineering tasks, test-case development is particularly suitable for crowdsourcing, because a large number of test cases can be generated at little monetary cost. However, the numerous test cases harvested from crowdsourcing can be high- or low-quality. Owing to the large volume, distinguishing the high-quality tests by traditional techniques is computationally expensive. Therefore, crowdsourced testing would benefit from an efficient mechanism distinguishes the qualities of the test cases. This paper introduces an automated approach — TCQA — to evaluate the quality of test cases based on the onsite coding history. Quality assessment by TCQA proceeds through three steps: (1) modeling the code history as a time series, (2) extracting the multiple relevant features from the time series, and (3) building a model that classifies the test cases based on their qualities. Step (3) is accomplished by feature-based machine-learning techniques. By leveraging the onsite coding history, TCQA can assess the test-case quality without performing expensive source-code analysis or executing the test cases. Using the data of nine test-development tasks involving more than 400 participants, we evaluated TCQA from multiple perspectives. The TCQA approach assessed the quality of the test cases with higher precision, faster speed, and lower overhead than conventional test-case quality-assessment techniques. Moreover, TCQA provided yield real-time insights on test-case quality before the assessment was finished.

Automated Test Case Generation for Effective Spectrum-based Fault Localization

Software testing is the key to ensuring software quality. Test case generation and software fault localization are two important research objects of testing. Spectrum-based fault localization (SBFL) method is a mainstream dynamic fault localization method, which inputs test cases into the program for execution and collects statement coverage information and execution results. By analysing this information, a fault report can be generated. The performance of SBFL is affected by test case quality. Therefore, this paper proposes a test case generation method based on improved genetic algorithm to assist SBFL. This method uses a small number of initial test cases to execute the program, then uses the XGBoost model to pre-process the spectrum information and generates a list of potential fault statements to guide the iterative process of the genetic algorithm. The experimental results show that the test cases generated using the improved genetic algorithm have certain advantages over the test cases generated by general genetic algorithm and the baseline test cases.

Web service composition on IoT reliability test based on cross entropy

AbstractWeb service has developed the managed IoT application to let connected devices easily and securely interact with cloud applications and other devices. As an important factor for web service, the reliability of web services refers to the probability of web service running success. For modeling web service composition, we should abstract the process of web service composition. Due to the diversity and complexity of web service composition, it is unlikely to do exhaustive testing. In order to improve the quality of web service composition test cases and find out which path leads to the greatest probability of service combination failure, heuristic test case generation method is adopted to obtain the optimal test path. First, the web service composition test is abstracted into the MDP model. The QoS of the web service composition is taken as the software test optimization goal, and the cross‐entropy strategy is used to optimize the test case. The experimental results show that the test profile given by the cross‐strategy is better than the random test strategy. Detect and exclude the same number of software defects. Cross‐entropy strategy can significantly reduce the number of test cases, reduce test costs, and improve defect detection efficiency.

GAPSO: Optimal Test Set Generator for Pairwise Testing

Exhaustive testing is impossible for all sorts of software systems, owing to the cost and time consumption. Combinatorial testing is the solution to this issue and aims at picking the necessary set of parameters which can ensure high degree of interaction between the parameters. This paper presents a new approach for generating unique test cases by exploiting Genetic and Particle Swarm Optimization (GAPSO) algorithm for achieving pairwise testing. The generated test cases are refined, so as to arrive at the optimal test set. The outcome of the proposed algorithm is the minimal count of high quality test cases.

Birds of a Feather Gel Together: Impact of Team Homogeneity on Software Quality and Team Productivity

Even though a lot of projects fail due to social issues or personality conflicts, only a small number of empirical studies have been conducted to quantitatively assess the impact of individual personality attributes on the software being developed and the team developing that software. The goal of this paper is to quantify the abstract notion of team homogeneity and to measure its impact on software quality and team productivity. A metric called team homogeneity index (THI) is proposed for this purpose. The six-step process of calculating the THI of a software development team is described and illustrated with the help of an example. Finally, the utility of THI is assessed by conducting a controlled experiment in two different phases of the software development life cycle (SDLC), i.e., implementation and testing. The results reveal that, during the implementation phase, teams with greater THI values were noticeably more productive and produced better quality code. Similarly, during the testing phase, teams with higher values of THI tested more features and wrote better quality test cases. Therefore, the evidence obtained so far suggests that the newly proposed metric, THI, appears to be useful in predicting the quality of software and the productivity of software development teams. Future work includes determining the weights of the five traits using input from the software industry and replication of this empirical study on different phases of SDLC with software practitioners to validate our findings.

Uml based test cases generation for the centralised control system of high speed maglev

Background: The high speed maglev centralized control subsystem (CCS), which realizes the display, automatic train operation (ATO) and diagnose. The CCS is an important security assurance for train operation. System testing (ST) can detect design defects early and performed effective repair to improve the efficiency of the system’ on-site coordinated operation. Therefore, the quality of test cases directly determines the achievements and efficiency of CCS, and it is necessary to be tested before put it into use.
 Aim: The generation of test case of centralized control subsystem is proposed.
 Methods: In order to meet the requirements of the system, the first step is operating the extraction of functional features. Then, the unified modeling language is using to develop the test model in this paper. Then the corresponding unified modeling language models, use case diagram, state diagram, activity diagram and sequence diagram, are developed. The state diagram of operation process is using to describe the state transition during the period from initial state of the OTS to the completion of the operation. The activity diagram of train operation process focuses on the control relationship during the period from one activity to another, which can describe the interaction exactly between OTS, DCS and VCS. The sequence diagram, behavior of use case, focuses on the sequence of information sent by objectives, in which a group of objectives and information transfer are presented. And according to these models, the test cases of the specified case are generated.
 Results: The generated test cases are all executed in the developed lab-test system. The results show that the generated test cases can fully simulate the common situations of the operation scene, and effectively improve the test efficiency and test quality. We designed the experiments as followings: 85 test cases for terminal system design of operator, 68 test cases for train automatic operation design, 31 test cases for central diagnostic system design. All of the designed test cases are examined through the test platform, covering all main scenes in operation process. The errors or detects found in tests are solved by finding the reasons and modifying the code, etc. Finally, the pass rate of the method proposed in this paper is 100 %. 
 Conclusion: The UML based method of test case generation implements the generation process and achieves the test cases and verification for CCS. Through the test practice, test cases designed can fully simulate all kinds of common situations in the operation site. What’s more, the test cases also realize early detection of errors and defects in the system and repair them. It is useful to improve the efficiency on-site testing process, to reduce the cost of time and test quality. The method can provide theoretical basis and reference for further testing of high speed maglev CCS.

Mutation operators for feature‐oriented software product lines

SummaryMutation testing is an approach to assess the quality of test cases. Mutants are modified versions of a system that ideally compose faulty behaviour. Test cases for a system are effective if they kill these mutants. For software product lines, several works have addressed mutation testing to inject variability faults, which may only exist in some variants. These works focus on variability models or specific implementation techniques. In contrast, feature‐oriented programming has been rarely investigated, wherefore, we (1) derive corresponding mutation operators, (2) investigate the feasibility of our proposed and conventional operators on 4 software product lines, and (3) discuss open challenges in mutation testing of software product lines. The results show that our proposed operators are suitable to cause variability faults and extend the capabilities of conventional operators. Nonetheless, mutation testing of software product lines is comparably expensive, due to a high number of variants and mutants—resulting in equivalence and redundancy.

Test Suite Minimization in Regression Testing Using Hybrid Approach of ACO and GA

This article describes about the application of search-based techniques in regression testing and compares the performance of various search-based techniques for software testing. Test cases tend to increase exponentially as the software is modified. It is essential to remove redundant test cases from the existing test suite. Regression testing is very costly and must be performed in restricted ways to ensure the validity of the existing software. There exist different methods to improve the quality of test cases in terms of the number of faults covered, opposed to the number of statements covered in a minimum time. Different methods exist for this purpose, such as minimization, test case selection, and test case prioritization. In this article, search-based methods are applied to improve the quality of the test suite in order to select a minimum set of test cases which covers all the statements in a minimum time. The whole approach is named search based regression testing. In this paper, the performance of different metaheuristics for test suite minimization problem is also compared with a hybrid approach of ant colony optimization algorithm and genetic algorithm.

Test Case Quality Management Procedure for Enhancing the Efficiency of IID Continuous Testing

Test Case Quality Management Procedure for Enhancing the Efficiency of IID Continuous Testing

Utilization of Dependence and Weight to Improve Fault Localization Method of Regression Test Cases

Among the fault localization methods of the coverage-based regression test cases, the ranking of suspiciousness degree is seriously impacted due to the existence of similar test cases, the existence of isolated treatment of each program entity and the coincidental correctness test cases as well as the poor quality of regression test cases, which reduces the efficiency of fault localization. This paper proposes a method that combines the control dependence and data dependence, test cases covering table and test result with vector angular cosine weight to solve the aforesaid problems. Experimental results show that the efficiency of the proposed fault localization of regression test cases is better than that of Tarantula, Jaccard, Ochiai, PPDG and CP methods.

Goal-based testing of semantic web services

Context: Recent years have witnessed growing interests in semantic web and its related technologies. While various frameworks have been proposed for designing semantic web services (SWS), few of them aim at testing.Objective: This paper investigates into the technologies for automatically deriving test cases from semantic web service descriptions based on the Web Service Modeling Ontology (WSMO) framework.Method: WSMO goal specifications were translated into B abstract machines. Test cases were generated via model checking with calculated trap properties from coverage criteria. Furthermore, we employed mutation analysis to evaluate the test suite. In this approach, the model-based test case generation and code-based evaluation techniques are independent of each other, which provides much more accurate measures of the testing results.Results: We applied our approach to a real-world case study of the Amazon E-Commerce Service (ECS). The experimental results have validated the effectiveness of the proposed solution.Conclusion: It is concluded that our approach is capable of automatically generating an effective set of test cases from the WSMO goal descriptions for SWS testing. The quality of test cases was measured in terms of their abilities to discover the injected faults at the code level. We implemented a tool to automate the steps for the mutation-based evaluation.