Structural Testing Criteria Research Articles

Structural testing for CUDA programming model

SummaryGraphic processors offer an accessible solution for high‐performance computing, addressing challenges across various fields. The Compute Unified Device Architecture (CUDA) programming model has emerged to enhance the performance of general‐purpose applications on graphic processors. However, developing CUDA programs is far from straightforward, and developers' lack of experience in parallel programming has led to numerous issues. This article presents a structural testing model and criteria to improve the quality of CUDA programs. These criteria facilitate the selection of test cases and aid in identifying faults. The ValiCUDA tool was developed to implement and validate this testing model and criteria. This tool instruments and analyzes programs, generating the necessary elements for each testing criterion. It also facilitates program execution and evaluation of criterion coverage. A statistical validation experiment assessed these criteria' effectiveness, cost, and strength metrics. The results demonstrate that the criteria can identify nontrivial faults in CUDA programs and assist testers in their testing endeavors for such applications.

Test Data Generation and Selection Using Levy Flight-Based Firefly Algorithm

This article discusses the application of an improved version of the firefly algorithm for the test suite optimization problem. Software test optimization refers to optimizing test data generation and selection for structural testing criteria for white box testing. This will subsequently reduce the two most costly activities performed during testing: time and cost. Recently, various search-based approaches proved very interesting results for the software test optimization problem. Also, due to no free lunch theorem, scientists are continuously searching for more efficient and convergent methods for the optimization problem. In this paper, firefly algorithm is modified in a way that local search ability is improved. Levy flights are incorporated into the firefly algorithm. This modified algorithm is applied to the software test optimization problem. This is the first application of Levy-based firefly algorithm for software test optimization. Results are shown and compared with some existing metaheuristic approaches.

Structural testing for communication events into loops of message‐passing parallel programs

AbstractThere is a growing demand for correct parallel programs, mainly due to nowadays availability of parallel architectures. Structural testing allows identifying defects by analyzing the internal structures of a program. However, communication and synchronization in parallel programs bring new challenges to the testing activity, such as nondeterminism. Message‐passing parallel programs require structural testing criteria to support test models and tools capable of covering synchronization events with dynamic behaviors. Testing such primitives inside loops in message‐passing programs is still challenging with nontrivial solutions for criteria, models, and tools. This article proposes new structural testing criteria to guide the selection of test cases, improving the quality of message‐passing programs by revealing nondeterminism‐related defects present in loops. We present a new test model to support our criteria for structural testing of MPI‐applications and implement the proposed criteria in the tool ValiMPI. The analysis of nondeterminism‐related defects, paths inside loops, loop iterations, and nested loops allow us to establish a structured testing model. We validate the testing criteria through experimental studies using ValiMPI. Our results show that our criteria reveal unknown defects from communication and synchronization events in different loop iterations, increasing the quality of message‐passing parallel programs.

Orderly Generation of Test Data via Sorting Mutant Branches Based on Their Dominance Degrees for Weak Mutation Testing

Compared with traditional structural test criteria, test data generated based on mutation testing are proved more effective at detecting faults. However, not all test data have the same potence in detecting software faults. If test data are prioritized while generating for mutation testing, the defect detectability of the test suite can be further strengthened. In view of this, we propose a method of test data generation for weak mutation testing via sorting mutant branches based on their dominance degrees. First, the problem of weak mutation testing is transformed into that of covering mutant branches for a transformed program. Then, the dominance relation of mutant branches in the transformed program is analyzed to obtain the non-dominated mutant branches and their dominance degrees. Following that, we prioritize all non-dominated mutant branches in descending order by virtue of their dominance degrees. Finally, the test data are generated in an orderly manner by selecting the mutant branches sequentially. The experimental results on 15 programs show that compared with other methods, the proposed test data generation method can not only improve the error detectability of the test suite, but also has higher efficiency.

An automated functional testing approach for virtual reality applications

SummarySoftware testing is regarded as an important method for fault revealing. Despite this advantage, it has been poorly used within the scope of virtual reality (VR) applications because they are highly complex and have peculiar features. Most testing performed of this VR applications are usability, which is conducted manually and only at final of the development process. Although some works try to propose criteria for this domain, there are no approaches that automatize the generation of test data from requirements specification in the VR domain. This paper proposes an approach called virtual reality—requirements specification and testing (VR‐ReST) to assist the requirements specification through a semiformal language and uses structural test criteria to generate test requirements and test data automatically for VR applications using scene graph concepts. The paper also examines the empirical results concerning the cost‐effectiveness of the approach for three different VR applications through two experiments. Mutation testing was used to evaluate effectiveness. We found that the approach achieved a high mutation score outperforming random testing, by 20%, on average. Our results also demonstrate that the approach is promising since it assists in writing and validating the requirements, as well as in reducing the risks of requirement specification by adopting a semiformal language.

Empirical evaluation of a new composite approach to the coverage criteria and reachability testing of concurrent programs

SummaryTesting is a key activity to assure the quality of concurrent applications. In recent years, a variety of different mechanisms have been proposed to test concurrent software. However, a persistent problem is the high testing cost because of the large number of different synchronization sequences that must be tested. When structural testing criteria are adopted, a large number of infeasible synchronization sequences is generated, increasing the testing cost. Although the use of reachability testing reduces the number of infeasible combination (because only feasible synchronization sequences are generated), many synchronization combinations are also generated, and this again results in a testing cost with exponential behavior. This paper presents a new composite approach that uses reachability testing to guide the selection of the synchronization sequences tests according to a specific structural testing criterion. This new composite approach is empirically evaluated in the context of message‐passing concurrent programs developed with MPI. The experimental study evaluates both the cost and effectiveness of proposed composite approach in comparison with traditional reachability testing and structural testing. The results confirm that the use of the new composite approach has advantages for testing of concurrent applications. Copyright © 2015 John Wiley & Sons, Ltd.

New Performance Estimation Formula for Evolutionary Testing of Switch-Case Constructs

Evolutionary structural testing is a technique that uses specific approaches based on guided searches algorithms. It involves evaluating fitness functions to determine whether test data satisfy or not various structural testing criteria. For testing multi-way decision constructs the nested \textit{If-Then-Else} structure and Alternative Critical Branches (ACBs) approaches are generally used. In this paper a new evolutionary structural approach based on Compact and Minimized Control Flow Graph (CMCFG) which uses two different formulas for evaluating the performance of test data, is presented. The CMCFG approach is derived from the concept of Control Flow Graph (CFG). Experiments on different \textit{Switch-Case} constructs with different nesting levels have demonstrated that CMCFG yields significantly better results in finding test data which cover a particular target branch in comparison with the previous approaches.

An Empirical Evaluation of the Cost and Effectiveness of Structural Testing Criteria for Concurrent Programs

Concurrent program testing is not a trivial task. Features like nondeterminism, communication and synchronization impose new challenges that must be considered during the testing activity. Some initiatives have proposed testing approaches for concurrent programs, in which different paradigms and programming languages are considered. However, in general, these contributions do not present a well-formed experimental study to validate their ideas. The problem is that the data used and generated during the validation is not always available, hampering the replication of studies in the context of other testing approaches. This paper presents an experimental study, taking into account the concepts of the Experimental Software En- gineering to evaluate the cost, effectiveness and strength of the structural testing criteria for message-passing programs. The evaluation was conducted considering a benchmark composed of eight MPI programs. A set of eight structural testing criteria defined for message-passing programs was evaluated with the ValiMPI testing tool, which provides the support required to apply the investigated testing criteria. The results indicate the complementary aspect of the criteria and the information about cost and effectiveness has contributed to the establishment of an incremental testing strategy to apply the criteria. All material generated during the experimental study is available for further comparisons.

Structural testing for message‐passing concurrent programs: an extended test model

SUMMARYDeveloping high‐quality, error‐free message‐passing concurrent programs is not trivial. Although a number of different primitives with associated semantics are available to assist such development, they often increase the complexity of the testing process. In this paper, we extend our previous test model for message‐passing programs and present new structural testing criteria, taking into account additional features used in this paradigm, such as collective communication, non‐blocking sends, distinct semantics for non‐blocking receives, and persistent operations. Our new model also recognizes that sender primitives cannot always be matched with every receive primitive. This improvement allows us to remove statically a significant number of infeasible synchronization edges that would otherwise have to be analyzed later by the tester. In this paper, the test model is presented using the Message‐Passing Interface standard; however, our new model has been designed to be flexible, and it can be configured to support a range of different message‐passing environments or languages. We have carried out case studies showing the applicability of the new test model to represent message‐passing programs and also to reveal errors, mainly those errors related to inter‐process communication. In addition to increasing the number of features supported by the test model, we have also reduced the overall cost of testing significantly. Our case studies suggest that the number of synchronization edges can be reduced by up to 93%, mainly by eliminating infeasible edges between unmatchable communication primitives. The main contribution of the paper is to present a more flexible test model that provides improved coverage for message‐passing programs and at the same time reduces the cost of testing significantly. Copyright © 2012 John Wiley & Sons, Ltd.

An Experimental Study to Evaluate the Impact of the Programming Paradigm in the Testing Activity


 
 
 Several techniques and criteria are available to help conducting testing activity. The choice for one of them depends on different aspects, such as the time restrictions, ef- fectiveness of the testing criteria or the features of the program under test. In this context, the programming paradigm might influence in the testing activity cost. This paper presents the results of an experimental study to characterize and evaluate the cost and strength of structural and functional testing criteria, comparing object-oriented and procedural programming paradigms. A set of 32 programs from the data structure do- main was considered in this study. The main goals in the execution of this research were: i) to obtain initial results about the investigated questions; ii) to generate artifacts which can be used as basis to define and conduct further experimental studies; iii) to support training and teaching of software testing activity.
 
 

Mutation based test case generation via a path selection strategy

ContextGenerally, mutation analysis has been identified as a powerful testing method. Researchers have shown that its use as a testing criterion exercises quite thoroughly the system under test while it achieves to reveal more faults than standard structural testing criteria. Despite its potential, mutation fails to be adopted in a widespread practical use and its popularity falls significantly short when compared with other structural methods. This can be attributed to the lack of thorough studies dealing with the practical problems introduced by mutation and the assessment of the effort needed when applying it. Such an incident, masks the real cost involved preventing the development of easy and effective to use strategies to circumvent this problem. ObjectiveIn this paper, a path selection strategy for selecting test cases able to effectively kill mutants when performing weak mutation testing is presented and analysed. MethodThe testing effort is highly correlated with the number of attempts the tester makes in order to generate adequate test cases. Therefore, a significant influence on the efficiency associated with a test case generation strategy greatly depends on the number of candidate paths selected in order to achieve a predefined coverage goal. The effort can thus be related to the number of infeasible paths encountered during the test case generation process. ResultsAn experiment, investigating well over 55 million of program paths is conducted based on a strategy that alleviates the effects of infeasible paths. Strategy details, along with a prototype implementation are reported and analysed through the experimental results obtained by its employment to a set of program units. ConclusionThe results obtained suggest that the strategy used can play an important role in making the mutation testing method more appealing and practical.

Structural testing criteria for message‐passing parallel programs

AbstractParallel programs present some features such as concurrency, communication and synchronization that make the test a challenging activity. Because of these characteristics, the direct application of traditional testing is not always possible and adequate testing criteria and tools are necessary. In this paper we investigate the challenges of validating message‐passing parallel programs and present a set of specific testing criteria. We introduce a family of structural testing criteria based on a test model. The model captures control and data flow of the message‐passing programs, by considering their sequential and parallel aspects. The criteria provide a coverage measure that can be used for evaluating the progress of the testing activity and also provide guidelines for the generation of test data. We also describe a tool, called ValiPar, which supports the application of the proposed testing criteria. Currently, ValiPar is configured for parallel virtual machine (PVM) and message‐passing interface (MPI). Results of the application of the proposed criteria to MPI programs are also presented and analyzed. Copyright © 2008 John Wiley & Sons, Ltd.

A binomial software reliability model based on coverage of structural testing criteria

A new approach to software reliability modeling is discussed where variables indirectly related with software reliability are used to provide additional information for the modeling process. Previous studies, empirical and theoretical evidences, and results from experiments indicate that there is a strong relationship between software reliability and coverage of program elements required to be exercised by structural testing criteria. This paper develops a binomial type coverage-based software reliability model through the definition of a coverage-based failure rate function. The Binomial software reliability Model Based on Coverage—BMBC—is proposed and discussed. In the BMBC test data between failures is used instead of time as independent variable; the model was assessed with test data from a real application, making use of the following structural testing criteria: all-nodes, all-edges, and potential-uses—a data-flow based family of testing criteria. The results from our experiments have shown that our modeling approach has some advantages over some traditional reliability models and points to a very promising research direction in software reliability.

Control and data flow structural testing criteria for aspect-oriented programs

Although it is claimed that, among other features, aspect-oriented programming (AOP) increases understandability and eases the maintenance burden, this technology cannot provide correctness by itself, and thus it also requires the use of systematic verification, validation and testing (VV&T) approaches. With the purpose of producing high quality software, many approaches to apply structural testing criteria for the unit testing of procedural and object-oriented (OO) programs have been proposed. Nevertheless, until now, few works have addressed the application of such criteria to test aspect-oriented programs. In this paper we define a family of control flow and data flow based testing criteria for aspect-oriented programs inspired by the implementation strategy adopted by AspectJ – an aspect-oriented extension of the Java language – and extending a previous work proposed for Java programs. We propose the derivation of a control and data flow model for aspect-oriented programs based upon the static analysis of the object code (the Java bytecode) resulted from the compilation/weaving process. Using this model, called aspect-oriented def-use graph ( AODU ), traditional and also aspect-oriented testing criteria are defined (called Control and Data Flow Structural Testing Criteria for Aspect-Oriented Programs – CDSTC-AOP). The main idea is that composition of aspect-oriented programs leads to new crosscutting interfaces in several modules of the system, which must be considered for coverage during structural testing. The implementation of a prototype tool – the JaBUTi/AJ tool – to support the proposed criteria and model is presented along with an example. Also, theoretical and practical questions regarding the CDSTC-AOP criteria are discussed.

Constraint based structural testing criteria

Structural criteria generally divide the input domain of the program under test and require the execution of at least one point from each derived sub-domain without addressing the most relevant question: “Which points from each sub-domain should be selected?”. This question is related to data-sensitive faults which lead to one of the drawbacks of the testing activity. The constraints and conditions used by fault-based data generation techniques describe faults related to the boundaries of these sub-domains. Our conjecture is that we would improve the efficacy of the adequate test case sets by associating those constraints and conditions to the elements required by a criterion. With this goal, this work presents Constraint Based Criteria (CBC) that associate a constraint C to an element E, required by a structural criterion. CBC allow to combine the fundamentals of different testing generation techniques with structural testing, increasing the probability of revealing faults described by C. We also discuss complexity, inclusion relation and automation aspects of CBC. Results from three experiments of CBC evaluation using the factors cost, efficacy and strength provide evidence that our objectives have been achieved. We also present some results from the evaluation of random test sets.

Evolutionary test environment for automatic structural testing

Testing is the most significant analytic quality assurance measure for software. Systematic design of test cases is crucial for the test quality. Structure-oriented test methods, which define test cases on the basis of the internal program structures, are widely used. A promising approach for the automation of structural test case design is evolutionary testing. Evolutionary testing searches test data that fulfil a given structural test criteria by means of evolutionary computation. In this work, an evolutionary test environment has been developed that performs fully automatic test data generation for most structural test methods. The introduction of an approximation level for fitness evaluation of generated test data and the definition of an efficient test strategy for processing test goals, increases the performance of evolutionary testing considerably.

Automatically detecting equivalent mutants and infeasible paths

Mutation testing is a technique for testing software units that has great potential for improving the quality of testing, and thereby increasing the ability to assure the high reliability of critical software. It will be shown that recent advances in mutation research have brought a practical mutation testing system closer to reality. One recent advance is a partial solution to the problem of automatically detecting equivalent mutant programs. Equivalent mutants are currently detected by hand, which makes it very expensive and time-consuming. The problem of detecting equivalent mutants is a specific instance of a more general problem, commonly called the feasible path problem, which says that for certain structural testing criteria some of the test requirements are infeasible in the sense that the semantics of the program imply that no test case satisfies the test requirements. Equivalent mutants, unreachable statements in path testing techniques, and infeasible DU-pairs in data flow testing are all instances of the feasible path problem. This paper presents a technique that uses mathematical constraints, originally developed for test data generation, to detect some equivalent mutants and infeasible paths automatically. © 1997 John Wiley & Sons, Ltd.

Structural testing of concurrent programs

Although structural testing techniques are among the weakest available with regard to developing confidence in sequential programs, they are not without merit. The authors extend the notion of structural testing criteria to concurrent programs and propose a hierarchy of supporting structural testing techniques. Coverage criteria described include concurrency state coverage, state transition coverage and synchronization coverage. Requisite support tools include a static concurrency analyzer and either a program transformation system or a powerful run-time monitor. Also helpful is a controllable run-time scheduler. The techniques proposed are suitable for Ada or CSP-like languages. Best results are obtained for programs having only static naming of tasking objects. >