High-quality Test Cases Research Articles

СВОЙСТВА КАЧЕСТВЕННЫХ ТЕСТ-КЕЙСОВ

The properties of high-quality test cases in the formation of technical documentation are presented

KVFL: Key-Value-Based Persistent Fuzzing for IoT Web Servers

Abstract As the number of Internet of Thing (IoT) devices increases, attacks against their vulnerabilities have become a serious threat. The web servers (WSs) in IoT devices provide management services for end-users, which are currently the major attack surface. Several fuzzing solutions for identifying vulnerabilities in IoT devices have been proposed, but there is currently no grey-box fuzzer specifically designed for the unique features of WSs in IoT to effectively detect memory corruption vulnerabilities. We design and implement KVFL, an efficient grey-box fuzzer, to address the issues of low throughput and slow exploration of deep code when fuzzing for IoT WSs. Firstly, KVFL employs a delicate hooking technology that heuristically hijacks and emulates hardware-dependent functions, ensuring WSs can be accurately and efficiently emulated in user-mode. On this basis, KVFL fully utilizes the loop parsing HTTP requests feature of WSs through a redesigned fork-server, to minimize nonessential rebooting losses of the target, thereby significantly improving fuzzing throughput. Secondly, KVFL leverages code coverage feedback to automatically infer a set of valid Keys and derive a Key-Value mutation. This enables the generation of high-quality test cases that can facilitate deeper code exploration of WSs. The evaluation results show that compared to the state-of-the-art IoT grey-box fuzzer FIRM-AFL, KVFL improves the throughput by over 2× and explores 4.5× more edges. Additionally, it identifies all 1-day vulnerabilities with over 7× faster speed than the baseline and detects three previously unknown 0-day vulnerabilities. These all indicate that KVFL is effective and efficient at fuzzing IoT WSs.

Security Analysis of Zigbee Protocol Implementation via Device-agnostic Fuzzing

Zigbee is widely adopted as a resource-efficient wireless protocol in the IoT network. IoT devices from manufacturers have recently been affected due to major vulnerabilities in Zigbee protocol implementations. Security testing of Zigbee protocol implementations is becoming increasingly important. However, applying existing vulnerability detection techniques such as fuzzing to the Zigbee protocol is not a simple task. Dealing with low-level hardware events still remains a big challenge. For the Zigbee protocol, which communicates over a radio channel, many existing protocol fuzzing tools lack a sufficient execution environment. To narrow the gap, we designed Z-Fuzzer , a device-agnostic fuzzing tool for detecting security flaws in Zigbee protocol implementations. To simulate Zigbee protocol execution, Z-Fuzzer leverages a commercial embedded device simulator with pre-defined peripherals and hardware interrupt setups to interact with the fuzzing engine. Z-Fuzzer generates more high-quality test cases with code-coverage heuristics. We compare Z-Fuzzer with advanced protocol fuzzing tools, BooFuzz and Peach fuzzer, on top of Z-Fuzzer’s simulation platform. Our findings suggest that Z-Fuzzer can achieve greater code coverage in Z-Stack, a widely used Zigbee protocol implementation. Compared to BooFuzz and Peach, Z-Fuzzer found more vulnerabilities with fewer test cases. Three of them have been assigned CVE IDs with high CVSS scores (7.5~8.2).

Automated Software Test Data Generation With Generative Adversarial Networks

With the rapid increase of software scale and complexity, the cost of traditional software testing methods will increase faster than the scale of software. In order to improve test efficiency, it is particularly important to automatically generate high-quality test cases. This paper introduces a framework for automatic test data generation based on the generative adversarial network (GAN). GAN is employed to train a generative model over execution path information to learn the behavior of the software. Then we can use the trained generative model to produce new test data, and select the test data that can improve the branch coverage according to our proposed selection strategy. Compared to prior work, our proposed method is able to handle programs under test with large-scale branches without analyzing branch expressions. In the experiment, we exhibit the performance of our method by using two modules in GNU Scientific Library. In particular, we consider the application of our method in two testing scenarios; unit testing and integration testing, and conduct a series of experiments to compare the performance of three types of GAN models. Results indicate that the WGAN-GP shows the best performance in our framework. Compared with the random testing method, the WGAN-GP based framework improves the test coverage of five functions out of the seven in the unit testing.

Test Case Quality Factors

The guarantee of software quality is very important. Thus, before a software is released to the end users, the flaws in the software should be detected by using high quality test cases. Currently, gauging the quality of test cases is carried out without any particular model and the criteria for good test cases is still unclear. Therefore, this study studies the literature using Systematic Literature Review (SLR) technique to identify the criteria of good test cases. The SLR considered papers between 2010 and 2018 in IEEE Xplore, ACM Digital Library, and Science Direct databases. Through the searching, it was found 310 papers are related. After filtering using exclusion and insertion criteria, 14 papers were considered for analysis. As a result, the test managed to identify 30 quality factors from the selected articles. These quality factors were additionally inspected, arranged and finished to be incorporated as the quality factors of test cases evaluation metrics.

CPA/Tiger-MGP: test-goal set partitioning for efficient multi-goal test-suite generation

Software model checkers can be used to generate high-quality test cases from counterexamples of a reachability analysis. However, naïvely invoking a software model checker for each test goal in isolation does not scale to large programs as a repeated construction of an abstract program model is expensive. In contrast, invoking a software model checker for reaching all test goals in a single run leads to few abstraction possibilities and thus to low scalability. Therefore, our approach pursues a test-suite generation technique that incorporates configurable multi-goal set partitioning (MGP) including configurable partitioning strategies and simultaneous processing of multiple test goals in one reachability analysis. Our approach employs recent techniques from multi-property verification in order to control the computational overhead for tracking multi-goal reachability information. Our tool, called CPA/Tiger-MGP, uses predicate-abstraction-based program analysis in the model-checking framework CPAchecker.

GAPSO: Optimal Test Set Generator for Pairwise Testing

Exhaustive testing is impossible for all sorts of software systems, owing to the cost and time consumption. Combinatorial testing is the solution to this issue and aims at picking the necessary set of parameters which can ensure high degree of interaction between the parameters. This paper presents a new approach for generating unique test cases by exploiting Genetic and Particle Swarm Optimization (GAPSO) algorithm for achieving pairwise testing. The generated test cases are refined, so as to arrive at the optimal test set. The outcome of the proposed algorithm is the minimal count of high quality test cases.

Controller-Centric Combinatorial Wrap-Around Interaction Testing to Evaluate a Stateful PCE-Based Transport Network Architecture

The objective of this paper is to develop a controller-centric combinatorial wrap-around interaction testing methodology for a stateful path computation element (PCE)-based transport network architecture. By exploiting the internal contexts of the controller-centric network in conjunction with combinatorial, wrap-around, and interaction testing methodologies, the proposed testing methodology helps test engineers build a highly configurable testing environment, select high-quality test cases to obtain the best possible combination of interactions, and prune fault cases or useless cases from all possible test cases throughout the entire development process as gray-box testing. The experimental results verify that the combined testing methodology effectively evaluates the controller-centric network for all testing processes in a completely controlled environment.

Tester interactivity makes a difference in search-based software testing: A controlled experiment

Context: Search-based software testing promises to provide users with the ability to generate high quality test cases, and hence increase product quality, with a minimal increase in the time and effort required.The development of the Interactive Search-Based Software Testing (ISBST) system was motivated by a previous study to investigate the application of search-based software testing (SBST) in an industrial setting. ISBST allows users to interact with the underlying SBST system, guiding the search and assessing the results. An industrial evaluation indicated that the ISBST system could find test cases that are not created by testers employing manual techniques. The validity of the evaluation was threatened, however, by the low number of participants.Objective: This paper presents a follow-up study, to provide a more rigorous evaluation of the ISBST system.Method: To assess the ISBST system a two-way crossover controlled experiment was conducted with 58 students taking a Verification and Validation course. The NASA Task Load Index (NASA-TLX) is used to assess the workload experienced by the participants in the experiment.Results:The experimental results validated the hypothesis that the ISBST system generates test cases that are not found by the same participants employing manual testing techniques. A follow-up laboratory experiment also investigates the importance of interaction in obtaining the results.In addition to this main result, the subjective workload was assessed for each participant by means of the NASA-TLX tool. The evaluation showed that, while the ISBST system required more effort from the participants, they achieved the same performance.Conclusions: The paper provides evidence that the ISBST system develops test cases that are not found by manual techniques, and that interaction plays an important role in achieving that result.

Application of Orthogonal Experimental Design for the Automatic Software Testing

According to inputting different combination of conditions so as to produce different impacts, software testing designs a large number of test cases. If the implementation of an overall test, due to the limit of the combination of conditions, it is difficult to carry out. In order to generate high quality test cases as early as possible to improve the efficiency of software testing, it is designed a generation tool of the automatic software testing case on orthogonal experimental design. For the test data, the use of that tool design test cases. The practice shows that a small number of test cases are generated, the error detection ability is strong, and it greatly improves the efficiency of software testing.

Experiences with a workstation prototype for softcopy reading within the bavarian mammography recertification program 1: workstations and education

Rationale and objectives In January 2002, the Bavarian Statutory Health Care Administration (“Kassenärztliche Vereinigung Bayerns”, KVB) started a recertification program for quality assurance and quality improvement in mammography reading. Materials and methods All accredited radiologists and gynecologists are asked to prove their qualification every 1–2 years. The recertification program requires the physicians to read 50 cases randomly selected from a larger collection of high-quality test cases. The portion of malignant and benign cases corresponds to the requirements of the German National Association of Statutory Health Insurance Physicians (“Kassenärztliche Bundesvereinigung”, KBV). In order to perform the recertification in a manageable manner a decentralized approach (test at different locations in parallel) was preferred over a centralized one. Therefore, the X-ray films were digitized and converted to DICOM Digital Mammography format to be read on a softcopy device. To verify the applicability of digitized mammograms for recertification purposes, a comparative study with 32 trained radiologists and gynecologists was performed. Results A system of two high-resolution/high-contrast monitors (2048 × 2560 pixels, ≥350 cd/m 2) in combination with a 5 mega-pixel dual-head graphics adapter with calibrated output was chosen for the mammography workstation. The software was implemented according to the particular requirements of this program. As a result, the comparative study showed that there was no significant difference in the error rate of the reported findings between conventional film and softcopy reading. Conclusion The first intermediate results of this quality initiative are promising. As of 2003, the test is mandatory for all mammography-reading physicians in Bavaria.

Experiences with a workstation prototype for softcopy reading within the Bavarian mammography recertification programme

In January 2002, the Bavarian Statutory Health Care Administration (“Kassenärztliche Vereinigung Bayerns”, KVB) started a recertification programme for quality assurance and quality improvement in mammography reading. All accredited radiologists and gynecologists are asked to prove their qualification every 1–2 years. The recertification programme requires the physicians to read 50 cases randomly selected from a larger collection of high-quality test cases. The portion of malignant and benign cases corresponds to the requirements of the German National Association of Statutory Health Insurance Physicians (“Kassenärztliche Bundesvereinigung”, KBV). In order to read the mammograms on a softcopy device the images are digitised with a high-quality scanner and converted to DICOM Digital Mammography format. The workstation software has been implemented according to the particular requirements of this programme. To verify the applicability of digitised mammograms for recertification purposes, a comparative study with 32 trained radiologists and gynecologists has been performed. As a result, the study showed that there was no significant difference in the error rate of the reported findings between conventional film and softcopy reading. The first intermediate results of this quality initiative are promising. The introduction of a corresponding federal German recertification programme is intended.