Test Data Selection Research Articles

Application of Program Graphs and Complexity Analysis to Software Development and Testing

Several research studies have shown a strong relationship between program complexity, as measured by the structural properties of a program, and its error properties, as measured by number and types of errors and error detection and correction times. This research applies to: a) the setting of threshold values of complexity in software production in order to avoid undue difficulty with program debugging; b) the use of complexity as an index for allocating resources during the test phase of software development; c) the use of complexity for developing test strategies and the selection of test data. Application #c uses the directed graph representation of a program and its complexity measures to decompose the program into its basic constructs. The identification of the constructs serves to identify a) the components of the program which must be tested, and b) the selection of test data which are needed to exercise these components. Directed-graph properties which apply to program development and testing are defined; examples of the application of graph properties for program development and testing are given; the results of program complexity and error measurements are presented; and a procedure for complexity measurement and its use in programming and testing is summarized.

An evaluation of the effectiveness of symbolic testing

AbstractThe effectiveness in discovering errors of symbolic evaluation and of testing sad static program analysis are studied. The three techniques are applied to a diverse collection of programs and the results compared. Symbolic evaluation is used to carry out symbolic testing and to generate symbolic systems of path predicates. The use of the predicates for automated test data selection is analysed. Several conventional types of program testing strategies are evaluated. The strategies include branch testing, structured testing and testing on input values having special properties. The static source analysis techniques that are studied include anomaly analysis and interface analysis.Examples are included which describe typical situations in which one technique is reliable but another unreliable. The effectiveness of symbolic testing is compared with testing on actual data and with the use of an integrated methodology that includes both testing and static source analysis. Situations in which symbolic testing is difficult to apply or not effective are discussed. Different ways in which symbolic evaluation can be used for generating test data are described. Those ways for which it is most effective are isolated. The paper concludes with a discussion of the most effective uses to which symbolic evaluation can he put in an integrated system which contains all three of the validation techniques that are studied.

Hints on Test Data Selection: Help for the Practicing Programmer

In many cases tests of a program that uncover simple errors are also effective in uncovering much more complex errors. This so-called coupling effect can be used to save work during the testing process.

An application of graph theory to software test data selection

Graph theory is playing an increasingly important role in the design, analysis, and testing of computer programs. It's importance is derived from the fact that flow of control and flow of data for any program can be expressed in terms of directed graphs. From the graph representing the flow of control, called the program graph, many others can be derived that either partially or completely preserve the program control structure. One derived graph known as a cyclomatic tree is of particular value in program testing. It is so named because the number of leaves of the tree is equal to the cyclomatic number of the program graph. A thorough treatment of cyclomatic numbers is provided in [3]. A program called the Complexity/Path Analyzer (CPA) has been developed that builds and utilizes a program cyclomatic tree to provide test planning information, automatically place software counters called probes as discussed in [9] and [10] in a program, and provide selected parameters such as program length and program graph cyclomatic number. The paper discusses the features and derivation of cyclomatic trees as well as their value and application to testing and test data generation. A cyclomatic tree provides a test planner with information useful for planning program tests. In particular, it furnishes test data selection criteria for developing tests that are minimally thorough as defined by Huang in [9]. A test data selection criterion will be defined as minimally thorough if any complete test with respect to the criterion is at least minimally thorough. The term complete is used as defined by Goodenhough and Gerhart in [13]. A test is defined to be a non empty sequence of test cases. Each test case consists of an element selected from the input domain of the program being tested. The paper discusses the merits of one particular technique selected to achieve a minimally thorough test data selection criteria. Part of the technique is automated by the CPA program.

An application of graph theory to software test data selection

Graph theory is playing an increasingly important role in the design, analysis, and testing of computer programs. It's importance is derived from the fact that flow of control and flow of data for any program can be expressed in terms of directed graphs. From the graph representing the flow of control, called the program graph, many others can be derived that either partially or completely preserve the program control structure. One derived graph known as a cyclomatic tree is of particular value in program testing. It is so named because the number of leaves of the tree is equal to the cyclomatic number of the program graph. A thorough treatment of cyclomatic numbers is provided in [3]. A program called the Complexity/Path Analyzer (CPA) has been developed that builds and utilizes a program cyclomatic tree to provide test planning information, automatically place software counters called probes as discussed in [9] and [10] in a program, and provide selected parameters such as program length and program graph cyclomatic number. The paper discusses the features and derivation of cyclomatic trees as well as their value and application to testing and test data generation. A cyclomatic tree provides a test planner with information useful for planning program tests. In particular, it furnishes test data selection criteria for developing tests that are minimally thorough as defined by Huang in [9]. A test data selection criterion will be defined as minimally thorough if any complete test with respect to the criterion is at least minimally thorough. The term complete is used as defined by Goodenhough and Gerhart in [13]. A test is defined to be a non empty sequence of test cases. Each test case consists of an element selected from the input domain of the program being tested. The paper discusses the merits of one particular technique selected to achieve a minimally thorough test data selection criteria. Part of the technique is automated by the CPA program.

Measuring computer software reliability

Under certain conditions, traditional hypothesis-testing techniques may be used as a management tool by software developers or software purchasers who wish to insure that their packages have some specified reliability level. These conditions are: (1) the existence of independent collections of test data, (2) a way of determining the correctness of processing of these collections, and (3) a way of randomly selecting test data. Two basic approaches have been described. In a fixed sample size test, the user decides on the reliability desired. He can then determine the number of test cases which must be examined and the acceptance/rejection criteria. In a sequential test, the desired reliability level is again pre-determined, but samples are tested one at a time until an accept/reject decision can be made. Experiments with a large amount of error data derived from six separate systems indicate that reliability results derived from these models are consistent with actual reliability figures. Most current acceptance procedures are based on a naive assumption that a large program can be exhaustively tested and delivered in an error-free condition. Because these expectations cannot be fulfilled, the manager of a software development project or the purchaser of a software product is provided with no quantitative information on which to base an acceptance decision and is thus forced to make these decisions based mostly on intuition and his own experience in similar situations. These models allow one to replace these intuition-based decisions with quantitatively-based decisions and thus constitute an important contribution to the science of management of software development efforts.

Correction to "toward a theory of test data selection"

In the above paper' the program in Fig. 3 should read then begin on line 7, bufpos := 0 end on line 20, and the semicolon on line 23 should be omitted. The program as given in Table I already reflects these corrections, so no other changes to the paper are necessary. These corrections were pointed out by H. C. Lauer.

Toward a theory of test data selection

Examines the theoretical and practical role of testing in software development. The authors prove a fundamental theorem showing that properly structured tests are capable of demonstrating the absence of errors in a program. The theorem's proof hinges on our definition of test reliability and validity, but its practical utility hinges on being able to show when a test is actually reliable. The authors explain what makes tests unreliable (for example, they show by example why testing all program statements, predicates, or paths is not usually sufficient to insure test reliability), and they outline a possible approach to developing reliable tests. They also show how the analysis required to define reliable tests can help in checking a program's design and specifications as well as in preventing and detecting implementation errors.

SELECT—a formal system for testing and debugging programs by symbolic execution

SELECT is an experimental system for assisting in the formal systematic debugging of programs. It is intended to be a compromise between an automated program proving system and the current ad hoc debugging practice, and is similar to a system being developed by King et al. of IBM. SELECT systematically handles the paths of programs written in a LISP subset that includes arrays. For each execution path SELECT returns simplified conditions on input variables that cause the path to be executed, and simplified symbolic values for program variables at the path output. For conditions which form a system of linear equalities and inequalities SELECT will return input variable values that can serve as sample test data. The user can insert constraint conditions, at any point in the program including the output, in the form of symbolically executable assertions. These conditions can induce the system to select test data in user-specified regions. SELECT can also determine if the path is correct with respect to an output assertion. We present four examples demonstrating the various modes of system operation and their effectiveness in finding bugs. In some examples, SELECT was successful in automatically finding useful test data. In others, user interaction was required in the form of output assertions. SELECT appears to be a useful tool for rapidly revealing program errors, but for the future there is a need to expand its expressive and deductive power.

Toward a theory of test data selection

This paper examines the theoretical and practical role of testing in software development. We prove a fundamental theorem showing that properly structured tests are capable of demonstrating the absence of errors in a program. The theorem's proof hinges on our definition of test reliability and validity, but its practical utility hinges on being able to show when a test is actually reliable. We explain what makes tests unreliable (for example, we show by example why testing all program statements, predicates, or paths is not usually sufficient to insure test reliability), and we outline a possible approach to developing reliable tests. We also show how the analysis required to define reliable tests can help in checking a program's design and specifications as well as in preventing and detecting implementation errors.

On Data Flow Testing Strategy

For the studies on the data flow oriented selection of test data [4], [5], [6], [8], the selection strategies are all based on the intuition that an incorrect definition could never be uncovered if its use is not tested. In this paper, we are proposing another data flow oriented selection strategy based on the intuition that errors tend to be clustered together and are more likely to be associated with the improper or bad programming practice. It has been found that our strategy is complementary to [4], [5], [6] and [8], and can be incorporated at nominal overheads with Laski and Korel's strategy [5].