Program Testability Research Articles

The Impact of Language Syntax on the Complexity of Programs: A Case Study of Java and Python

Programming is the cornerstone of computer science, yet it is difficult to learn and program. The syntax of a programming language is particularly challenging to comprehend, which makes learning arduous and affects the program's testability. There is currently no literature that definitively gives quantitative evidence about the effect of programming language complex syntax. The main purpose of this article was to examine the effects of programming syntax on the complexity of their source programs. During the study, 298 algorithms were selected and their implementations in Java and Python were analyzed with the cyclomatic complexity matrix. The results of the study show that Python's syntax is less complex than Java's, and thus coding in Python is more comprehensive and less difficult than Java coding. The Mann-Whitney U test was performed on the results of a statistical analysis that showed a significant difference between Java and Python, indicating that the syntax of a programming language has a major impact on program complexity. The novelty of this article lies in the formulation of new knowledge and study patterns that can be used primarily to compare and analyze other programming languages.

An empirical study on the effects of code visibility on program testability

Software testability represents the degree of ease with which a software artifact supports testing. When it is easy to detect defects in a program through testing, the program has high testability; otherwise, the testability of the program is low. As an abstract property of programs, testability can be measured by various metrics, which are affected by different factors of design and implementation. In object-oriented software development, code visibility is important to support design principles, such as information hiding. It is widely believed that code visibility has some effects on testability. However, little empirical evidence has been shown to clarify whether and how software testability is influenced by code visibility. We have performed a comprehensive empirical study to shed light on this problem. We first use code coverage as a concrete proxy for testability. We select 27 real-world software programs as subjects and ran two state-of-the-art automated testing tools, Randoop and EvoSuite, on these programs to analyze their code coverage, in comparison with that of developer-written tests. The results show that code visibility does not necessarily have effects on code coverage, but can significantly affect automated tools. Developer-written tests achieve similar coverage on code areas with different visibility, while low code visibility often leads to low code coverage for automated tools. In addition, we have developed two enhanced variants of Randoop that implement multiple strategies to handle code visibility. The results on Randoop variants show that different treatments on code visibility can result in significant differences in code coverage for automated tools. In the second part, our study uses fault detection rate as another concrete measurement of testability. We apply the automated testing tools on 357 real faults. The result of our in-depth analysis is consistent with that of the first part, demonstrating the significant effects of code visibility on program testability.

Testing of aspect-oriented programs: difficulties and lessons learned based on theoretical and practical experience

Since the first discussions of new challenges posed by aspect-oriented programming (AOP) to software testing, the real difficulties of testing aspect-oriented (AO) programs have not been properly analysed. Firstly, despite the customisation of traditional testing techniques to the AOP context, the literature lacks discussions on how hard it is to apply them to (even ordinary) AO programs based on practical experience. Secondly, and equally important, due to the cautious AOP adoption focused on concern refactoring, test reuse is another relevant issue that has been overlooked so far. This paper deals with these two issues. It discusses the difficulties of testing AO programs from three perspectives: (i) structural-based testing, (ii) fault-based testing and (iii) test set reuse across paradigms. Perspectives (i) and (ii) are addressed by means of a retrospective of research done by the authors’ group. We analyse the impact of using AOP mechanisms on the testability of programs in terms of the underlying test models, the derived test requirements and the coverage of such requirements. The discussion is based on our experience on developing and applying testing approaches and tools to AspectJ programs at both unit and integration levels. Perspective (iii), on the other hand, consists of recent exploratory studies that analyse the effort to adapt test sets for refactored systems and the quality of such test sets in terms of structural coverage. Building test models for AO programs imposes higher complexity when compared to the OO paradigm. Besides this, adapting test suites for OO programs to AO equivalent programs tends to require less effort than doing the other way around, and resulting suites achieve similar quality levels for small-sized aplications. The conclusion is that building test models for AO programs, as well as deriving and covering paradigm-specific test requirements, is not straightforward as it has been for procedural and object-oriented (OO) programs at some extent. Once you have test suites in conformance with programs implemented in both paradigms, the quality of such suited in termos of code coverage may vary depending on the size and characteristics of the applications under testing.

Comparative performance evaluation of applying extended PIE technique to accelerate software testability analysis

The rapid development of technology provides high performance and reliability for the hardware system; based on this, software engineers can focus their developed software on more convenience and ultra-high reliability. To reach this goal, the testing stage of software development life cycle usually takes more time and effort due to the growing complexity of the software. How to build software that can be tested efficiently has become an important topic in addition to enhancing and developing new testing methods. Thus, research on software testability has been conducted and various methods have been developed. In the past, a dynamic technique for estimating program testability was proposed and called propagation, infection and execution (PIE) analysis. Previous research studies have shown that PIE analysis can complement software testing. However, this method requires a lot of computational overhead in estimating the testability of software components. In this article, we propose an extended PIE (EPIE) method to accelerate the conventional PIE analysis, based on generating group testability as a substitute for statement testability. Our proposed method can be systematically separated into three steps: breaking a program into blocks, dividing the blocks into groups and marking target statements. Experiments and evaluations with the Siemens suite, together with cost-effectiveness analysis, clearly show that the number of analysed statements can be effectively decreased, and the calculated values of testability are still acceptable.

Automatic Generation and Execution of Mutants

To obtain the high quality software, there is a use of Mutation testing to measure the quality of our test suite. Fault insertion based techniques have been used for measuring test adequacy and testability of programs. Mutants are generated by introducing the faults in the original program. Tests Cases are adequate if they detect all the mutants. This paper describes a survey study to investigate the generation and execution of mutants. General Terms Software Testing

Critical Analysis of the PIE Testability Technique

The work of Voas and colleagues has introduced, refined and applied the propagation, infection and execution (PIE) analysis technique for measuring testability of programs. The purpose of this pape...

On the use of testability measures for dependability assessment

Program "testability" is informally, the probability that a program will fail under test if it contains at least one fault. When a dependability assessment has to be derived from the observation of a series of failure free test executions (a common need for software subject to "ultra high reliability" requirements), measures of testability can-in theory-be used to draw inferences on program correctness. We rigorously investigate the concept of testability and its use in dependability assessment, criticizing, and improving on, previously published results. We give a general descriptive model of program execution and testing, on which the different measures of interest can be defined. We propose a more precise definition of program testability than that given by other authors, and discuss how to increase testing effectiveness without impairing program reliability in operation. We then study the mathematics of using testability to estimate, from test results: the probability of program correctness and the probability of failures. To derive the probability of program correctness, we use a Bayesian inference procedure and argue that this is more useful than deriving a classical "confidence level". We also show that a high testability is not an unconditionally desirable property for a program. In particular, for programs complex enough that they are unlikely to be completely fault free, increasing testability may produce a program which will be less trustworthy, even after successful testing.