Software Testing Techniques Research Articles

Fuzzing trusted execution environments with Rust

Fuzzing, a software testing technique, aims to uncover bugs by subjecting the target program to random inputs, thus discerning abnormal program behaviors such as crashes. In this paper, we present the design and implementation of a fuzzing framework designed to test TEEs (Trusted Execution Environment). Our framework leverages established software tools in a novel way: (1) We employ the Rust programming language in a two-way code generator: to translate fuzzer output to a sequence of system calls and in a “reverse translation” process, where sample code snippets are used to seed the fuzzer – a single API specification suffices for both endeavors; (2) Our fuzzer exhibits the ability to iteratively traverse the API's specification, scrutinize object dependencies, and judiciously reuse objects. These features significantly amplify its bug-finding prowess. (3) A versatile Rust proc macro mechanism is used to process the API specification. The fuzzer's code is built with the Rust compiler sans the necessity for additional specialized tools. (4) To enable the efficient stateful execution of TEEs, we have tailored the QEMU system emulator accordingly. To verify the usability and performance of our fuzzer, and to test various configuration options we conducted a series of tests with a popular open-source OP-TEE trusted operating system.

FunFuzz : Greybox Fuzzing with Function Significance

Greybox fuzzing is dedicated to revealing software bugs by maximizing code coverage. Concentrating on code coverage, greybox fuzzing effectively exposes bugs in real-world programs by continuously executing the program under test (PUT) with the test inputs generated from initial seeds, making it a popular software testing technique. Although powerful, the effectiveness of greybox fuzzing can be restricted in some cases. Ignoring the significant degrees of executed functions, traditional greybox fuzzing usually fails to identify significant seeds that execute more significant functions, and thus may assign similar energy to significant and trivial seeds when conducting power scheduling. As a result, the effectiveness of greybox fuzzing can be degraded due to wasting too much energy on trivial seeds. In this paper, we introduce function significance (FS) to measure the significant degrees of functions. Our key insight is that the influential functions that connect to many other functions are significant to greybox fuzzing as they provide more probabilities to reach previously unexplored code regions. To quantify FS, we conduct influence analysis upon the call graphs extracted from the PUTs to obtain the centrality values of function nodes. With FS as the significance measurement, we further propose FunFuzz , an FS-aware greybox fuzzing technique, to optimize significant seeds and tackle the aforementioned restriction. To this end, FunFuzz dynamically tracks the functions executed by a seed during fuzzing, and computes the significance score for the seed by accumulating the FS values of the functions executed by it. Based on the computed FS values, FunFuzz then takes an estimation-based power scheduling to assign more (or less) energy to seeds that achieve over-estimated (or under-estimated) significance scores. Specifically, the seed energy is adjusted by multiplying with a scale factor computed regarding the ratio of the actual significance score achieved by executing the seed and the estimated significance score predicted by a linear model constructed on-the-fly. To evaluate FunFuzz , we prototype it on top of AFL++ and conduct experiments with 15 programs, of which 10 are from common real-world projects and five are from Magma, and compare it to seven popular fuzzers. The experimental results obtained through fuzzing exceeding 40,800 CPU hours show that: (1) In terms of covering code, FunFuzz outperforms AFL++ by achieving 0.1% \(\sim\) 18.4% more region coverage on 13 out of 15 targets. (2) In terms of finding bugs, FunFuzz unveils 114 unique crashes and 25 Magma bugs (which are derived from CVEs) in 20 trials of 24-hour fuzzing, which are the most compared to the competitor fuzzers and include 32 crashes and 1 Magma bug that the other fuzzers fail to discover. Besides the experiments focusing on code coverage and bug finding, we evaluate the key components of FunFuzz , namely the FS-centered estimation-based power scheduling and the lazy FS computation mechanism. The extensive evaluation not only suggests FunFuzz ’s superiority in code coverage and bug finding, but also demonstrates the effectiveness of the two components.

My Fuzzers Won’t Build: An Empirical Study of Fuzzing Build Failures

Fuzzing is an automated software testing technique used to find software vulnerabilities that works by sending large amounts of inputs to a software system to trigger bad behaviors. In recent years, the open-source software ecosystem has seen a significant increase in the adoption of fuzzing to avoid spreading vulnerabilities throughout the ecosystem. While fuzzing can uncover vulnerabilities, there is currently a lack of knowledge regarding the challenges of conducting fuzzing activities over time. Specifically, fuzzers are very complex tools to set up and build before they can be used. We set out to empirically find out how challenging is build maintenance in the context of fuzzing. We mine over 1.2 million build logs from Google's OSS-Fuzz service to investigate fuzzing build failures. We first conduct a quantitative analysis to quantify the prevalence of fuzzing build failures. We then manually investigate 677 failing fuzzing builds logs and establish a taxonomy of 25 root causes of build failures. We finally train a machine learning model to recognize common failure patterns in failing build logs. Our taxonomy can serve as a reference for practitioners conducting fuzzing build maintenance. Our modeling experiment shows the potential of using automation to simplify the process of fuzzing.

An empirical study of testing machine learning in the wild

Background: Recently, machine and deep learning (ML/DL) algorithms have been increasingly adopted in many software systems. Due to their inductive nature, ensuring the quality of these systems remains a significant challenge for the research community. Traditionally, software systems were constructed deductively, by writing explicit rules that govern the behavior of the system as program code. However, ML/DL systems infer rules from training data i.e., they are generated inductively). Recent research in ML/DL quality assurance has adapted concepts from traditional software testing, such as mutation testing, to improve reliability. However, it is unclear if these proposed testing techniques are adopted in practice, or if new testing strategies have emerged from real-world ML deployments. There is little empirical evidence about the testing strategies. Aims: To fill this gap, we perform the first fine-grained empirical study on ML testing in the wild to identify the ML properties being tested, the testing strategies, and their implementation throughout the ML workflow. Method: We conducted a mixed-methods study to understand ML software testing practices. We analyzed test files and cases from 11 open-source ML/DL projects on GitHub. Using open coding, we manually examined the testing strategies, tested ML properties, and implemented testing methods to understand their practical application in building and releasing ML/DL software systems. Results: Our findings reveal several key insights: 1.) The most common testing strategies, accounting for less than 40%, are Grey-box and White-box methods, such as Negative Testing , Oracle Approximation , and Statistical Testing . 2.) A wide range of \(17\) ML properties are tested, out of which only 20% to 30% are frequently tested, including Consistency , Correctness , and Efficiency . 3.) Bias and Fairness is more tested in Recommendation (6%) and CV (3.9%) systems, while Security & Privacy is tested in CV (2%), Application Platforms (0.9%), and NLP (0.5%). 4.) We identified 13 types of testing methods, such as Unit Testing , Input Testing , and Model Testing . Conclusions: This study sheds light on the current adoption of software testing techniques and highlights gaps and limitations in existing ML testing practices.

Program Segment Testing for Human–Machine Pair Programming

Human–Machine Pair Programming (HMPP) is a promising technique in the software development process, which means that software construction can be done in the manner that humans are responsible for developing the program while computer is responsible for monitoring the program in real-time and reporting errors. The Java runtime exceptions in the current version of the software under construction can only be effectively detected by means of its execution. Traditional software testing techniques are suitable for testing completed programs but face a challenge in building a suitable testing environment for testing the partial programs produced during HMPP. In this paper, we put forward a novel technique, called Program Segment Testing (PST) for automatically identifying errors caused by runtime exceptions to support HMPP. We first introduce the relevant involved in this technique to detect index out of bounds exceptions, a representative of runtime exceptions. Then we discuss the methodology of this technique in detail and illustrate its workflow with a simple case study. Finally, we carry out an experiment to evaluate this technique and compare it with three existing fault detection techniques using several programs to demonstrate its effectiveness.

On the Impact of Lower Recall and Precision in Defect Prediction for Guiding Search-based Software Testing

Defect predictors, static bug detectors, and humans inspecting the code can propose locations in the program that are more likely to be buggy before they are discovered through testing. Automated test generators such as search-based software testing (SBST) techniques can use this information to direct their search for test cases to likely buggy code, thus speeding up the process of detecting existing bugs in those locations. Often the predictions given by these tools or humans are imprecise, which can misguide the SBST technique and may deteriorate its performance. In this article, we study the impact of imprecision in defect prediction on the bug detection effectiveness of SBST. Our study finds that the recall of the defect predictor, i.e., the proportion of correctly identified buggy code, has a significant impact on bug detection effectiveness of SBST with a large effect size. More precisely, the SBST technique detects 7.5 fewer bugs on average (out of 420 bugs) for every 5% decrements of the recall. However, the effect of precision, a measure for false alarms, is not of meaningful practical significance, as indicated by a very small effect size. In the context of combining defect prediction and SBST, our recommendation is to increase the recall of defect predictors as a primary objective and precision as a secondary objective. In our experiments, we find that 75% precision is as good as 100% precision. To account for the imprecision of defect predictors, in particular low recall values, SBST techniques should be designed to search for test cases that also cover the predicted non-buggy parts of the program, while prioritising the parts that have been predicted as buggy.

Set evolution based test data generation for killing stubborn mutants

Mutation testing is a fault-based and powerful software testing technique, but the large number of mutations can result in extremely high costs. To reduce the cost of mutation testing, researchers attempt to identify stubborn mutants and generate test data to kill them, in order to achieve the same testing effect. However, existing methods suffer from inaccurate identification of stubborn mutants and low productiveness in generating test data, which will seriously affect the effectiveness and efficiency of mutation testing. Therefore, we propose a new method of generating test data for killing stubborn mutants based on set evolution, namely TDGMSE. We first propose an integrated indicator to identify stubborn mutants. Then, we establish a constrained multi-objective model for generating test data of killing stubborn mutants. Finally, we develop a new genetic algorithm based on set evolution to solve the mathematical model. The results on 14 programs depict that the average false positive (or negative) rate of TDGMSE is decreased about 81.87% (or 32.34%); the success rate of TDGMSE is 99.22%; and the average number of iterations of TDGMSE is 16132.23, which is lowest of all methods. The research highlights several potential research directions for mutation testing.

Testing concolic execution through consistency checks

Symbolic execution is a well-known software testing technique that evaluates how a program runs when considering a symbolic input, i.e., an input that can initially assume any concrete value admissible for its data type. The dynamic twist of this technique is dubbed concolic execution and has been demonstrated to be a practical technique for testing even complex real-world programs. Unfortunately, developing concolic engines is hard. Indeed, an engine has to correctly instrument the program to build accurate symbolic expressions, which represent the program computation. Furthermore, to reason over such expressions, it has to interact with an SMT solver. Hence, several implementation bugs may emerge within the different layers of an engine.In this article, we consider the problem of testing concolic engines. In particular, we propose several testing strategies whose main intuition is to exploit the concrete state kept by the executor to identify inconsistencies within the symbolic state. We integrated our strategies into three state-of-the-art concolic executors (SymCC, SymQEMU, and Fuzzolic, respectively) and then performed several experiments to show that our ideas can find bugs in these frameworks. Overall, our approach was able to discover more than 12 bugs across these engines.

Optimal test case generation for boundary value analysis

Boundary value analysis (BVA) is a common technique in software testing that uses input values that lie at the boundaries where significant changes in behavior are expected. This approach is widely recognized and used as a natural and effective strategy for testing software. Test coverage is one of the criteria to measure how much the software execution paths are covered by the set of test cases. This paper focuses on evaluating test coverage with respect to BVA by defining a metric called boundary coverage distance (BCD). The BCD metric measures the extent to which a test set covers the boundaries. In addition, based on BCD, we consider the optimal test input generation to minimize BCD under the random testing scheme. We propose three algorithms, each representing a different test input generation strategy, and evaluate their fault detection capabilities through experimental validation. The results indicate that the BCD-based approach has the potential to generate boundary values and improve the effectiveness of software testing.

ENHANCED VERSION OF SEEDING AND CONSTRAINT SUPPORT IN IPOG STRATEGY FOR VARIABLE STRENGTH INTERACTION T-WAY TESTING

Interaction strength testing (known as t-way testing) is one of the effective software testing techniques used to find problems with how system components interact with one another. The emergence of variable strength interaction occurs because of the covering array's identified limitation, which prevents it from handling varying parameter testing strengths. Several works were formed in t-way testing to generate a small test suite size using either One-Parameter-at-a-Time (OPAT) or One-Test-at-a-Time (OTAT) approach and base on interaction strength that include uniform, variable and/or input-output relation interaction, since t-way testing is highly complicated (NP-hard). However, no single OPAT t-way strategy claims to support seeding and constraint in variable strength interaction. Motivated by these challenges, the paper presents an enhanced version of IPOG strategy for t-way testing with seeding, constraint, and variable strength interaction supports, named SCIPOG-VS. To assess its efficiency, experiments were performed and compared to other strategies supporting variable strength interaction. Experimental results demonstrated that SCIPOG-VS is competitive in terms of test suite size, and execution time against other existing strategies significantly. We statistically tested our findings with a non-parametric Wilcoxon Signed Rank test, despite this, the outcome shows a significant difference in the size of the test suite between the various strategies.

Contemporary Software Testing Techniques: A Review

Contemporary Software Testing Techniques: A Review

CMBMeTest: Generation of Test Suites Using Model-Based Testing Plus Constraint Programming and Metamorphic Testing

Various software testing techniques have been shown to be successful in producing high-quality test suites for software where the code is not accessible (black-box approach). Nevertheless, no method has been found to guide combining some of these in a general way. In this study, a test suite generation method for black-box software called CMBMeTest was created to respond to these challenges. It employs several coupled software testing techniques, namely, model-based testing (MBT), constraint programming (CP), and metamorphic testing (MT). CMBMeTest provides step-by-step instructions for using the information available (such as program specifications, inputs and outputs) to create an initial test suite that covers the model obtained, using a combination of MBT and CP (referred to as MBT+CP). Furthermore, using the metamorphic relations (MRs) of MT, a better test suite was produced from that initial test suite. The method allows particular stages to be iterated to improve the results by building new models and new MRs. A comprehensive case study was conducted, employing CMBMeTest to produce encouraging results. Mutation testing was used to evaluate the test suite, and the first round produced a high mutation score. A more detailed model was used to repeat the process, with similar outcomes.

Design and Implement an Accurate Automated Static Analysis Checker to Detect Insecure Use of SecurityManager

Static analysis is a software testing technique that analyzes the code without executing it. It is widely used to detect vulnerabilities, errors, and other issues during software development. Many tools are available for static analysis of Java code, including SpotBugs. Methods that perform a security check must be declared private or final; otherwise, they can be compromised when a malicious subclass overrides the methods and omits the checks. In Java, security checks can be performed using the SecurityManager class. This paper addresses the aforementioned problem by building a new automated checker that raises an issue when this rule is violated. The checker is built under the SpotBugs static analysis tool. We evaluated our approach on both custom test cases and real-world software, and the results revealed that the checker successfully detected related bugs in both with optimal metrics values.

The Human Side of Fuzzing: Challenges Faced by Developers during Fuzzing Activities

Fuzz testing, also known as fuzzing, is a software testing technique aimed at identifying software vulnerabilities. In recent decades, fuzzing has gained increasing popularity in the research community. However, existing studies led by fuzzing experts mainly focus on improving the coverage and performance of fuzzing techniques. That is, there is still a gap in empirical knowledge regarding fuzzing, especially about the challenges developers face when they adopt fuzzing. Understanding these challenges can provide valuable insights to both practitioners and researchers on how to further improve fuzzing processes and techniques. We conducted a study to understand the challenges encountered by developers during fuzzing. More specifically, we first manually analyzed 829 randomly sampled fuzzing-related GitHub issues and constructed a taxonomy consisting of 39 types of challenges (22 related to the fuzzing process itself, 17 related to using external fuzzing providers). We then surveyed 106 fuzzing practitioners to verify the validity of our taxonomy and collected feedback on how the fuzzing process can be improved. Our taxonomy, accompanied with representative examples and highlighted implications, can serve as a reference point on how to better adopt fuzzing techniques for practitioners, and indicates potential directions researchers can work on toward better fuzzing approaches and practices.

JavaScript SBST Heuristics to Enable Effective Fuzzing of NodeJS Web APIs

JavaScript is one of the most popular programming languages. However, its dynamic nature poses several challenges to automated testing techniques. In this paper, we propose an approach and open-source tool support to enable white-box testing of JavaScript applications using Search-Based Software Testing (SBST) techniques. We provide an automated approach to collect search-based heuristics like the common Branch Distance and to enable Testability Transformations . To empirically evaluate our results, we integrated our technique into the EvoMaster test generation tool, and carried out analyses on the automated system testing of RESTful and GraphQL APIs. Experiments on eight Web APIs running on NodeJS show that our technique leads to significantly better results than existing black-box and grey-box testing tools, in terms of code coverage and fault detection.

.NET/C# instrumentation for search-based software testing

AbstractC# is one of the most widely used programming languages. However, to the best of our knowledge, there has been no work in the literature aimed at enabling search-based software testing techniques for applications running on the .NET platform, like the ones written in C#. In this paper, we propose a search-based approach and an open source tool to enable white-box testing for C# applications. The approach is integrated with a .NET bytecode instrumentation, in order to collect code coverage at runtime during the search. In addition, by taking advantage of Branch Distance, we define heuristics to better guide the search, e.g., how heuristically close it is to cover a branch in the source code. To empirically evaluate our technique, we integrated our tool into the EvoMaster test generation tool and conducted experiments on three .NET RESTful APIs as case studies. Results show that our technique significantly outperforms gray-box testing tools in terms of code coverage.

A Study on the Benefits and Limitations of Software Testing Principles and Techniques: Software Quality Engineering

A Study on the Benefits and Limitations of Software Testing Principles and Techniques: Software Quality Engineering

Various Software Testing Techniques, their Implementation and Significance on Banking Applications: In Context of Bangladesh

Abstract: Software testing is the process of identifying bugs or errors in developed software. Consequently, this process offers interested individuals with precise details on the product's quality. Software testing is done not only to ensure that the program behaves correctly or not, but to check whether it satisfies the user’s requirements and expectations as well. The testing of banking software is very important since they handle millions of transactions and are related to customers directly. Proper testing of the banking applications ensures that all operations are carried out accurately and remain safe and secure. The main objective of this paper is to provide an overview of various software testing techniques, their implementation and significance on the banking applications of Bangladesh.

Optimizing test case prioritization using machine learning algorithms

<p>Software testing is an important aspect of software development to ensure the quality and reliability of the software. With the increasing complexity of software systems, the number of test cases has also increased significantly, making it challenging to execute all the test cases in a limited amount of time. Test case prioritization techniques have been proposed to tackle this problem by identifying and executing the most important test cases first. In this research paper, we propose the use of machine learning algorithms for prioritization of test cases. We explore different machine learning algorithms, including decision trees, random forests, and neural networks, and compare their performance with traditional prioritization techniques such as code coverage-based and risk-based prioritization. We evaluate the effectiveness of these algorithms on various datasets and metrics such as the number of test cases executed, the fault detection rate, and the execution time. Our experimental results demonstrate that machine learning algorithms can effectively prioritize test cases and outperform traditional techniques in terms of reducing the number of test cases executed while maintaining high fault detection rates. Furthermore, we discuss the potential limitations and future research directions of using machine learning algorithms for test case prioritization. Our research findings contribute to the development of more efficient and effective software testing techniques that can improve the quality and reliability of software systems.</p>

Software Testing Techniques and Tools: A Review

Software Testing Techniques and Tools: A Review