Effective Test Suite Research Articles

ASSESSING TEST SUITES FOR BUFFER OVERFLOW VULNERABILITIES

Over the last few years, numerous vulnerabilities have been reported in software, and successful exploitations of these vulnerabilities have resulted in severe consequences such as denial of services and application state corruptions. Researches have shown that effective quality assurance methods can prevent such consequences when applied during software (or applications) development processes. Software security testing is a popular assurance method in this direction. However, effective testing involves obtaining an effective test suite (or collection of test cases) that can reveal specific faults. Over the last few years, different testing approaches have been applied for revealing vulnerabilities in software. However, only few works have assessed the effectiveness of test suites for revealing vulnerabilities. We believe that bringing the idea of mutation-based assessment of test adequacy for vulnerabilities can help in detecting and removing vulnerabilities proactively. In this work, we apply mutation-based adequate testing for one of the worst vulnerabilities namely buffer overflow (BOF). We propose 16 mutation operators to force the generation of adequate test suites for BOF vulnerabilities. A prototype tool is developed to automatically generate mutants and perform mutation analysis with input test cases. The effectiveness of the operators is evaluated by using several benchmark programs having BOF vulnerabilities, and the results indicate that the proposed operators are effective for testing BOF vulnerabilities. Moreover, we present an analysis to find selective mutation operators for reducing the cost of mutation-based testing of BOF vulnerabilities.

High-frequency, at-speed scan testing

The authors describe new strategies where at-speed scan tests can be applied with internal PLL. They present techniques for optimizing ATPG across multiple clock domains and methodologies to combine both stuck-at-fault and delay-test vectors into an effective test suite.

Using production grammars in software testing

Extensible typesafe systems, such as Java, rely critically on a large and complex software base for their overall protection and integrity, and are therefore difficult to test and verify. Traditional testing techniques, such as manual test generation and formal verification, are too time consuming, expensive, and imprecise, or work only on abstract models of the implementation and are too simplistic. Consequently, commercial virtual machines deployed so far have exhibited numerous bugs and security holes. In this paper, we discuss our experience with using production grammars in testing large, complex and safety-critical software systems. Specifically, we describe lava, a domain specific language we have developed for specifying production grammars, and relate our experience with using lava to generate effective test suites for the Java virtual machine. We demonstrate the effectiveness of production grammars in generating complex test cases that can, when combined with comparative and variant testing techniques, achieve high code and value coverage. We also describe an extension to production grammars that enables concurrent generation of certificates for test cases. A certificate is a behavioral description that specifies the intended outcome of the generated test case, and therefore acts as an oracle by which the correctness of the tested system can be evaluated in isolation. We report the results of applying these testing techniques to commercial Java implementations. We conclude that the use of production grammars in combination with other automated testing techniques is a powerful and effective method for testing software systems, and is enabled by a special purpose language for specifying extended production grammars.