Code Coverage Research Articles

Design of an Online Programming Platform and a Study on Learners’ Testing Ability

Online judge (OJ) systems are essential in programming education as they efficiently evaluate learners’ programming skills and reduce instructor workload. However, these systems often overlook the importance of software-testing concepts. To address this gap, we developed a system called Pytutor that integrates software-testing concepts to assess learners’ programming abilities and proficiency by exploring students’ test cases and learning behaviors. Drawing on software engineering theory and practical techniques, test capabilities are evaluated by analyzing the code coverage and mutation testing of Defining Test Cases. Since our experiment is conducted in an online environment, we can collect students’ learning behaviors and further analyze the relationship between software engineering abilities and learning behaviors. We also analyzed the differences in programming and testing abilities between computer science majors and non-computer-science majors. Our findings suggest that better testing abilities may contribute to the improvement in programming abilities, whereas in the current Taiwanese education context, computer science majors do not necessarily have better testing abilities. This result provides suggestions for us to strengthen software-testing education no matter which type of students it is targeted at.

Functional verification of QSPI module based on UVM implementation

This article presents a novel functional verification methodology based on the Universal Verification Methodology (UVM) to validate the functionality of the Quick Serial Peripheral Interface (QSPI) module. QSPI serves as a prevalent serial communication protocol widely employed for high-speed data exchange with external flash devices. By using the code coverage and functional coverage reports provided by UVM, we were able to assess how well the test cases covered the design code, identifying the parts of the design that had been adequately tested and those that needed more test coverage. This verification method based on UVM provides powerful tools and technical support for the design and development of the QSPI module and effectively ensures its reliability and stability. This study is of great significance for further improving the design quality and verification efficiency of the QSPI module.

Uncovering Bugs in Code Coverage Profilers via Control Flow Constraint Solving

Uncovering Bugs in Code Coverage Profilers via Control Flow Constraint Solving

Demystify the Fuzzing Methods: A Comprehensive Survey

Massive software applications possess complex data structures or parse complex data structures; in such cases, vulnerabilities in the software become inevitable. The vulnerabilities are the source of cyber-security threats, and discovering this before the software deployment is challenging. Fuzzing is a vulnerability discovery solution that resonates with random-mutation, feedback-driven, coverage-guided, constraint-guided, seed-scheduling, and target-oriented strategies. Each technique is wrapped beneath the black-, white-, and grey-box fuzzers to uncover diverse vulnerabilities. It consists of methods such as identifying structural information about the test cases to detect security vulnerabilities, symbolic and concrete program states to explore the unexplored locations, and full semantics of code coverage to create new test cases. We methodically examine each kind of fuzzers and contemporary fuzzers with a profound observation that addresses various research questions and systematically reviews and analyze the gaps and their solutions. Our survey comprised the recent related works on fuzzing techniques to demystify the fuzzing methods concerning the application domains and the target that, in turn, achieves higher code coverage and sound vulnerability detection.

Hypermedia-based software architecture enables Test-Driven Development.

Using agile software development practices, develop and evaluate an architecture and implementation for reliable and user-friendly self-service management of bioinformatic data stored in the cloud. Comprehensive Oncology Research Environment (CORE) Browser is a new open-source web application for cancer researchers to manage sequencing data organized in a flexible format in Amazon Simple Storage Service (S3) buckets. It has a microservices- and hypermedia-based architecture, which we integrated with Test-Driven Development (TDD), the iterative writing of computable specifications for how software should work prior to development. Relying on repeating patterns found in hypermedia-based architectures, we hypothesized that hypermedia would permit developing test "templates" that can be parameterized and executed for each microservice, maximizing code coverage while minimizing effort. After one-and-a-half years of development, the CORE Browser backend had 121 test templates and 875 custom tests that were parameterized and executed 3031 times, providing 78% code coverage. Architecting to permit test reuse through a hypermedia approach was a key success factor for our testing efforts. CORE Browser's application of hypermedia and TDD illustrates one way to integrate software engineering methods into data-intensive networked applications. Separating bioinformatic data management from analysis distinguishes this platform from others in bioinformatics and may provide stable data management while permitting analysis methods to advance more rapidly. Software engineering practices are underutilized in informatics. Similar informatics projects will more likely succeed through application of good architecture and automated testing. Our approach is broadly applicable to data management tools involving cloud data storage.

Test Case Prioritization Using Swarm Intelligence Algorithm to Improve Fault Detection and Time for Web Application

Prioritizing test cases based on several parameters where important ones are executed first is known as test case prioritization (TCP). Code coverage, functionality, and features are all possible factors of TCP for detecting bugs in software as early as possible. This research was carried out to test and compare the effectiveness Swarm Intelligence algorithms, where Artificial Bee Colony (ABC) and Ant Colony Optimization (ACO) algorithms were implemented to find the fault detected and execution time as these are the curial aspects in software testing to ensure good quality products are produced within the timeline. As web applications are commonly used by a board population, this research was carried out on an Online Shopping application represented as Case Study One and Education Administrative application known as Case Study Two. In recent years, TCP has been implemented widely, but none has implemented on web application which was conducted to fill the gaps and produce a new contribution in this area. The outcome was compared using Average Percentage Fault Detected (APFD) and execution time. For Case Study One, the APFD value was 0.80 and 0.71 while the execution time was 8.64 seconds and 0.69 seconds respectively for ABC and ACO. For Case Study Two, the APFD values were 0.81 and 0.64 while the execution time was 8.83 seconds and 1.22 seconds for ABC and ACO. It was seen that both algorithms performed well in their respective ways. ABC had shown to give a higher value for APFD while ACO had converged faster for execution time.

NSFuzz: Towards Efficient and State-Aware Network Service Fuzzing

As an essential component responsible for communication, network services are security critical, thus, it is vital to find their vulnerabilities. Fuzzing is currently one of the most popular software vulnerability discovery techniques, widely adopted due to its high efficiency and low false positives. However, existing coverage-guided fuzzers mainly aim at stateless local applications, leaving stateful network services underexplored. Recently, some fuzzers targeting network services have been proposed but have certain limitations, for example, insufficient or inaccurate state representation and low testing efficiency. In this article, we propose a new fuzzing solution NSFuzz for stateful network services. We studied typical implementations of network service programs to determine how they represent states and interact with clients. Accordingly, we propose (1) a program variable–based state representation scheme and (2) an efficient interaction synchronization mechanism to improve fuzzing efficiency. We implemented a prototype of NSFuzz, which uses static analysis and annotation application programming interfaces (APIs) to identify synchronization points and state variables within the services. It then achieves fast I/O synchronization and accurate service state tracing to carry out efficient state-aware fuzzing via lightweight compile-time instrumentation. The evaluation results show that compared with other network service fuzzers, including AFL net and S tate AFL, our solution NSFuzz could infer a more accurate state model during fuzzing and improve fuzzing throughput by up to 200×. In addition, NSFuzz could improve code coverage by up to 25% and trigger more crashes in less time. We also performed a fuzzing campaign to find new bugs in the latest version of the target services; 8 zero-day vulnerabilities have been found by NSFuzz.

JavaScript SBST Heuristics to Enable Effective Fuzzing of NodeJS Web APIs

JavaScript is one of the most popular programming languages. However, its dynamic nature poses several challenges to automated testing techniques. In this paper, we propose an approach and open-source tool support to enable white-box testing of JavaScript applications using Search-Based Software Testing (SBST) techniques. We provide an automated approach to collect search-based heuristics like the common Branch Distance and to enable Testability Transformations . To empirically evaluate our results, we integrated our technique into the EvoMaster test generation tool, and carried out analyses on the automated system testing of RESTful and GraphQL APIs. Experiments on eight Web APIs running on NodeJS show that our technique leads to significantly better results than existing black-box and grey-box testing tools, in terms of code coverage and fault detection.

On transforming model‐based tests into code: A systematic literature review

SummaryModel‐based test design is increasingly being applied in practice and studied in research. Model‐based testing (MBT) exploits abstract models of the software behaviour to generate abstract tests, which are then transformed into concrete tests ready to run on the code. Given that abstract tests are designed to cover models but are run on code (after transformation), the effectiveness of MBT is dependent on whether model coverage also ensures coverage of key functional code. In this article, we investigate how MBT approaches generate tests from model specifications and how the coverage of tests designed strictly based on the model translates to code coverage. We used snowballing to conduct a systematic literature review. We started with three primary studies, which we refer to as the initial seeds. At the end of our search iterations, we analysed 30 studies that helped answer our research questions. More specifically, this article characterizes how test sets generated at the model level are mapped and applied to the source code level, discusses how tests are generated from the model specifications, analyses how the test coverage of models relates to the test coverage of the code when the same test set is executed and identifies the technologies and software development tasks that are on focus in the selected studies. Finally, we identify common characteristics and limitations that impact the research and practice of MBT: (i) some studies did not fully describe how tools transform abstract tests into concrete tests, (ii) some studies overlooked the computational cost of model‐based approaches and (iii) some studies found evidence that bears out a robust correlation between decision coverage at the model level and branch coverage at the code level. We also noted that most primary studies omitted essential details about the experiments.

.NET/C# instrumentation for search-based software testing

AbstractC# is one of the most widely used programming languages. However, to the best of our knowledge, there has been no work in the literature aimed at enabling search-based software testing techniques for applications running on the .NET platform, like the ones written in C#. In this paper, we propose a search-based approach and an open source tool to enable white-box testing for C# applications. The approach is integrated with a .NET bytecode instrumentation, in order to collect code coverage at runtime during the search. In addition, by taking advantage of Branch Distance, we define heuristics to better guide the search, e.g., how heuristically close it is to cover a branch in the source code. To empirically evaluate our technique, we integrated our tool into the EvoMaster test generation tool and conducted experiments on three .NET RESTful APIs as case studies. Results show that our technique significantly outperforms gray-box testing tools in terms of code coverage.

RegFuzz: A Linear Regression-Based Approach for Seed Scheduling in Directed Fuzzing

Directed fuzzing aims to focus on fuzzing specific locations within a target program to enhance the efficiency of vulnerability discovery. However, directed fuzzing may yield fewer vulnerabilities and obtain lower code coverage when the specified locations have little to no vulnerabilities. Additionally, the existing directed fuzzing approaches often overlook the differences in variable values when calculating distances between seeds and specific locations. In order to address these issues, this paper introduces RegFuzz, a method that improves seed scheduling in directed fuzzing. RegFuzz utilizes a linear regression model to predict the effectiveness of a seed and allocates more fuzzing opportunities to efficient seeds. Specifically, first, RegFuzz defines several labels with the corresponding trainable weights for each seed. These labels encompass seed coverage, crash efficiency, seed distance, and more. In the calculation of seed distance, RegFuzz takes into account not only the basic block distance but also the variable distance contained within those basic blocks. Second, the linear regression model continually optimizes the label weights during fuzzing, and these optimized weights are employed to predict the effectiveness of seeds. In comparison with AFLGo, AFL, and AFL++, RegFuzz demonstrates higher code coverage and a more efficient bug-finding capability across seven real-world open-source programs.

State-Sensitive Black-Box Web Application Scanning for Cross-Site Scripting Vulnerability Detection

Black-box web application scanning has been a popular technique to detect Cross-Site Scripting (XSS) vulnerabilities without prior knowledge of the application. However, several limitations lead to low efficiency of current black-box scanners, including (1) the scanners waste time by repetitively visiting similar states, such as similar HTML forms of two different products, and (2) using a First-In-First-Out (FIFO) fuzzing order for the collected forms has led to low efficiency in detecting XSS vulnerabilities, as different forms have different potential possibilities of XSS vulnerability. In this paper, we present a state-sensitive black-box web application scanning method, including a filtering method for excluding similar states and a heuristic ranking method for optimizing the fuzzing order of forms. The filtering method excludes similar states by comparing readily available characteristic information that does not require visiting the states. The ranking method sorts forms based on the number of injection points since it is commonly observed that forms with a greater number of injection points have a higher probability of containing XSS vulnerabilities. To demonstrate the effectiveness of our scanning method, we implement it in our black-box web scanner and conduct experimental evaluations on eight real-world web applications within a limited scanning time. Experimental results demonstrate that the filtering method improves the code coverage about 17% on average and the ranking method helps detect 53 more XSS vulnerabilities. The combination of the filtering and ranking methods helps detect 81 more XSS vulnerabilities.

Time From Authorization by the US Food and Drug Administration to Medicare Coverage for Novel Technologies

A wide variety of novel medical diagnostics and devices are determined safe and effective by the US Food and Drug Administration (FDA) each year, but to our knowledge the literature lacks evidence documenting how long it takes to establish new Medicare coverage for these technologies. To measure time from FDA authorization to at least nominal Medicare coverage for technologies requiring a new reimbursement pathway. In this cross-sectional study, public databases were used to associate each technology to billing codes, determine the effective date of each code and Medicare coverage decisions, and stratify by the maturity of the Medicare coverage. At least nominal coverage was defined as achievement of explicit coverage milestones through a national coverage determination, local coverage determinations by Medicare administrative contractors, or by implicit coverage aligned to a new billing code. Characterization by product type (acute treatment, chronic or ongoing treatment, diagnostic assay, and diagnostic device), manufacturer size, and evidence level were assessed for association with coverage achievement. The study included new product applications authorized by the FDA through the premarket approval pathway, the de novo pathway, or with breakthrough designation in the 510(k) pathway from January 1, 2016, to December 31, 2019. Data analysis took place between May 1, 2022, and December 31, 2022. Time from FDA authorization to the first coverage milestone. Among 281 identified technologies in the total sample, 64 (23%) were deemed novel technologies based on the absence of coverage determinations and/or the use of temporary or miscellaneous billing codes. Twenty-eight of 64 technologies (44%) successfully achieved explicit or implicit coverage following FDA authorization. The median time to at least nominal coverage for the analysis cohort was 5.7 years (90% CI, 4.4-NA years). Analysis of time-to-coverage data highlighted company size (log-rank P<.001) and product type (log-rank P = .01) as significant covariates associated with coverage achievement. No association was observed for technologies with level 1 evidence at FDA authorization and subsequent coverage milestone achievement (log-rank P = .40). In this cross-sectional study of 64 novel technologies, only 28 (44%) achieved coverage milestones over the study timeline. The several-year period observed to establish at least nominal coverage suggests existing coverage processes may affect timely reimbursement of new technologies.

White-Box Fuzzing RPC-Based APIs with EvoMaster: An Industrial Case Study

Remote Procedure Call (RPC) is a communication protocol to support client-server interactions among services over a network. RPC is widely applied in industry for building large-scale distributed systems, such as Microservices. Modern RPC frameworks include, for example, Thrift, gRPC, SOFARPC, and Dubbo. Testing such systems using RPC communications is very challenging, due to the complexity of distributed systems and various RPC frameworks the system could employ. To the best of our knowledge, there does not exist any tool or solution that could enable automated testing of modern RPC-based services. To fill this gap, in this article we propose the first approach in the literature, together with an open source tool, for fuzzing modern RPC-based APIs. The approach is in the context of white-box testing with search-based techniques. To tackle schema extraction of various RPC frameworks, we formulate a RPC schema specification along with a parser that allows the extraction from source code of any JVM RPC-based APIs. Then, with the extracted schema we employ a search to produce tests by maximizing white-box heuristics and newly defined heuristics specific to the RPC domain. We built our approach as an extension to an open source fuzzer (i.e., EvoMaster ), and the approach has been integrated into a real industrial pipeline that could be applied to a real industrial development process for fuzzing RPC-based APIs. To assess our novel approach, we conducted an empirical study with two artificial and four industrial web services selected by our industrial partner. In addition, to further demonstrate its effectiveness and application in industrial settings, we report results of employing our tool for fuzzing another 50 industrial APIs autonomously conducted by our industrial partner in their testing processes. Results show that our novel approach is capable of enabling automated test case generation for industrial RPC-based APIs (i.e., 2 artificial and 54 industrial). We also compared with a simple gray-box technique and existing manually written tests. Our white-box solution achieves significant improvements on code coverage. Regarding fault detection, by conducting a careful review with our industrial partner of the tests generated by our novel approach in the selected four industrial APIs, a total of 41 real faults were identified, which have now been fixed. Another 8,377 detected faults are currently under investigation.

FCEVAL: An effective and quantitative platform for evaluating fuzzer combinations fairly and easily

Multiple base fuzzers collaborate as a fuzzer combination. Fuzzer combinations have been proven to perform more robustly and efficiently when fuzzing complicated real-world programs. The efficiency of finding bugs with limited computational resources would greatly benefit from the fuzzer combinations chosen by an effective and quantitative performance evaluation. However, evaluating fuzzer combinations remains challenging due to the lack of infrastructure for collaborative fuzzing, high enough collaboration efficiency of base fuzzers, unified benchmarks, comprehensive metrics, and unified analysis methods of coverage and bugs. This prevents us from selecting efficient fuzzer combinations and thus impairs vulnerability mining on real-world targets. In this paper, we design and implement FCEVAL, the first open-source platform for evaluating fuzzer combinations. In detail, we propose a new test case-sharing policy for increasing fuzzing potential so that we can provide a more efficient running environment for fuzzer combinations and thus improve evaluation effectiveness. Also, we select a unified set of diverse benchmarks and comprehensive metrics while adopting unified independent methods of real-time coverage statistics and multiple-sanitizers-based bug analysis for evaluation fairness and quantification. In addition, we design tools and guidelines covering the whole evaluation process for usability. With the above methodologies, we first construct an infrastructure special for collaborative fuzzing as the base of FCEVAL. After comparing two test case-sharing policies on the infrastructure and choosing the promising one as a substantial part of FCEVAL, we leverage FCEVAL to evaluate fuzzer combinations for more than 40,000 CPU hours and come up with five important conclusions, including (a) an efficient test case-sharing policy improving fuzzing potential and thus evaluation effectiveness, (b) comprehensive metrics being essential, (c) 24-hour duration and 20 repetitions for evaluation being substantial, (d) independent analysis methods of code coverage and bugs deserving of extensive adoption, and (e) FCEVAL being able to evaluate fuzzer combinations effectively, fairly, comprehensively, and easily. Meanwhile, we suggest how to improve collaborative fuzzing. In addition, source codes and test data are publicly available.

Specification-Based Symbolic Execution for Stateful Network Protocol Implementations in IoT

IoT devices offer insufficient protections against exploitation of critical programming errors (such as buffer overflows) it is therefore paramount to sufficiently test IoT software before deployment. A central source of these errors are implementations of stateful network protocols used in the IoT (e.g. MQTT-SN). Unfortunately, comprehensive automated testing of such protocol implementations is challenging due to their large state space. Furthermore, prior work on automated testing of protocol implementations focuses on conventional desktop systems and does thus not address challenges specific to the IoT domain (e.g. handling of peripheral interactions). In this publication, we leverage symbolic execution (an automated software testing technique) for testing stateful network protocol implementation in the IoT domain. To mitigate issues regarding state space explosion, we propose a novel symbolic execution approach which is based on created protocol state and message format specifications and specifically tailored to the IoT domain. We employ our specification-based symbolic execution approach to test protocol implementations provided by two popular IoT operating systems. Our experiments indicate that a significant increase in code coverage can be achieved using our automated software testing approach. We were further able to discover three critical and previously unknown bugs in network protocol implementations provided by the RIOT operating system.

PosFuzz: augmenting greybox fuzzing with effective position distribution

Mutation-based greybox fuzzing has been one of the most prevalent techniques for security vulnerability discovery and a great deal of research work has been proposed to improve both its efficiency and effectiveness. Mutation-based greybox fuzzing generates input cases by mutating the input seed, i.e., applying a sequence of mutation operators to randomly selected mutation positions of the seed. However, existing fruitful research work focuses on scheduling mutation operators, leaving the schedule of mutation positions as an overlooked aspect of fuzzing efficiency. This paper proposes a novel greybox fuzzing method, PosFuzz, that statistically schedules mutation positions based on their historical performance. PosFuzz makes use of a concept of effective position distribution to represent the semantics of the input and to guide the mutations. PosFuzz first utilizes Good-Turing frequency estimation to calculate an effective position distribution for each mutation operator. It then leverages two sampling methods in different mutating stages to select the positions from the distribution. We have implemented PosFuzz on top of AFL, AFLFast and MOPT, called Pos-AFL, -AFLFast and -MOPT respectively, and evaluated them on the UNIFUZZ benchmark (20 widely used open source programs) and LAVA-M dataset. The result shows that, under the same testing time budget, the Pos-AFL, -AFLFast and -MOPT outperform their counterparts in code coverage and vulnerability discovery ability. Compared with AFL, AFLFast, and MOPT, PosFuzz gets 21% more edge coverage and finds 133% more paths on average. It also triggers 275% more unique bugs on average.

GUI Component Detection-Based Automated Software Crash Diagnosis

This study presents an automated software crash-diagnosis technique using a state transition graph (STG) based on GUI-component detection. An STG is a graph representation of the state changes in an application that are caused by actions that are executed in the GUI, which avoids redundant test cases and generates bug-reproduction scenarios. The proposed technique configures the software application STG using computer vision and artificial intelligence technologies and performs automated GUI testing without human intervention. Four experiments were conducted to evaluate the performance of the proposed technique: a detection-performance analysis of the GUI-component detection model, code-coverage measurement, crash-detection-performance analysis, and crash-detection-performance analysis in a self-configured multi-crash environment. The GUI-component detection model obtained a macro F1-score of 0.843, even with a small training dataset for the deep-learning model in the detection-performance analysis. Furthermore, the proposed technique achieved better performance results than the baseline Monkey in terms of code coverage, crash detection, and multi-crash detection.

Black-Box Testing of Deep Neural Networks through Test Case Diversity

Deep Neural Networks (DNNs) have been extensively used in many areas including image processing, medical diagnostics, and autonomous driving. However, DNNs can exhibit erroneous behaviours that may lead to critical errors, especially when used in safety-critical systems. Inspired by testing techniques for traditional software systems, researchers have proposed neuron coverage criteria, as an analogy to source code coverage, to guide the testing of DNN models. Despite very active research on DNN coverage, several recent studies have questioned the usefulness of such criteria in guiding DNN testing. Further, from a practical standpoint, these criteria are white-box as they require access to the internals or training data of DNN models, which is in many contexts not feasible or convenient. In this paper, we investigate black-box input diversity metrics as an alternative to white-box coverage criteria. To this end, we first select and adapt three diversity metrics and study, in a controlled manner, their capacity to measure actual diversity in input sets. We then analyse their statistical association with fault detection using four datasets and five DNN models. We further compare diversity with state-of-the-art white-box coverage criteria. Our experiments show that relying on the diversity of image features embedded in test input sets is a more reliable indicator than coverage criteria to effectively guide the testing of DNNs. Indeed, we found that one of our selected black-box diversity metrics far outperforms existing coverage criteria in terms of fault-revealing capability and computational time. Results also confirm the suspicions that state-of-the-art coverage metrics are not adequate to guide the construction of test input sets to detect as many faults as possible with natural inputs.

Finding Compiler Bugs Duplicates by Generating Witness Programs

Programming language compilers are complex software projects, the quality of which directly affects the quality of programs created by them. Therefore, compilers are subject to increased requirements for such software quality characteristics as functional suitability, reliability, performance level, safety, etc. To ensure quality, various methods are used: user and manual testing, tools for automatic error detection. And due to many methods of searching for bugs often situations arise when the same error, especially if its cause is trivial, is detected repeatedly, and the test cases that lead to it can be completely different from each other. Such test cases are called duplicates, and their determination is an urgent and acute problem, since their manual search requires a large amount of human resources. Algorithms for automatically finding duplicates would greatly simplify the process of developing and maintaining a compiler. The main idea of the approach for finding duplicates presented in the article is that the causes of the same errors are located in the same place in the compiler source code. To search for this place, the method of generating witness programs is used: for each test program containing an error, similar programs are generated that do not contain it. After that, metrics are calculated based on the source code coverage of the compiler and a list of source code files that are potentially causing the compiler to fail is formed. If these lists for two test programs are similar in terms of the proximity metric, then the test programs are considered duplicates. The proposed approach was developed and implemented for the compiler of the Kotlin programming language. Testing has shown the applicability of the proposed approach for solving the problem of finding duplicate errors of compilers of programming languages