Software Testing Techniques Research Articles

Benchmarking the Capability of Symbolic Execution Tools with Logic Bombs

Symbolic execution has become an indispensable technique for software testing and program analysis. However, since several symbolic execution tools are presently available off-the-shelf, there is a need for a practical benchmarking approach. This paper introduces a fresh approach that can help benchmark symbolic execution tools in a fine-grained and efficient manner. The approach evaluates the performance of such tools against known challenges faced by general symbolic execution techniques, e.g., floating-point numbers and symbolic memories. We first survey related papers and systematize the challenges of symbolic execution. We extract 12 distinct challenges from the literature and categorize them into two categories: symbolic-reasoning challenges and path-explosion challenges . Next, we develop a dataset of logic bombs and a framework for benchmarking symbolic execution tools automatically. For each challenge, our dataset contains several logic bombs, each addressing a specific challenging problem. Triggering one or more logic bombs confirms that the symbolic execution tool in question is able to handle the corresponding problem. Real-world experiments with three popular symbolic execution tools, namely, KLEE, angr, and Triton have shown that our approach can reveal the capabilities and limitations of the tools in handling specific issues accurately and efficiently. The benchmarking process generally takes only a few dozens of minutes to evaluate a tool. We have released our dataset on GitHub as open source, with an aim to better facilitate the community to conduct future work on benchmarking symbolic execution tools.

SHFuzz: Selective Hybrid Fuzzing with Branch Scheduling Based on Binary Instrumentation

Hybrid fuzzing is a popular software testing technique that combines random fuzzing with concolic execution. It is widely used in the security domain known for its ability to find deeply hidden vulnerabilities and reach high code coverage. Hybrid fuzzing is based on negating branches in the execution path of a specific input to generate new test cases. However, due to numerous inputs and related branches, it does not show the best of its effectiveness without input and branch selection methods. In this paper, we systematically analyze the branch scheduling problem in the internal attributes of hybrid fuzzing, focusing on the synchronization mechanism. To solve the problems, we propose the Selective Hybrid Fuzzing (SHF) approach with branch scheduling based on binary instrumentation. There are two major parts to the SHF approach: (1) we propose a critical branch selection algorithm to select critical branches by three metrics: hit accuracy, solvability, and complexity; (2) we propose a priority score calculation algorithm to select inputs by the number of critical branches. With the SHF approach, we choose only the branches that can be negated to generate new coverage, instead of repeatedly executing the same branches and generating duplicates of inputs. We implement a hybrid fuzzer called SHFuzz with our SHF approach and compare it with the state-of-the-art hybrid fuzzer QSYM. In the evaluation, SHFuzz outperforms QSYM in 20 real-world applications from the Google Fuzzer Test Suite and other program suites in a 12 h test. On average, SHFuzz achieves 8.40% more code coverage and 100 more unique crashes in each application. Our work also finds existing vulnerabilities 7.85× faster than QSYM. We also find new bugs by SHFuzz, which QSYM fails to find. Our evaluation shows that the selective hybrid fuzzing approach can reduce the number of branches executed in concolic execution, enhancing hybrid fuzzing on code coverage and bug finding capabilities.

An Empirical Investigation of Software Testing Methods and Techniques in the Province of Vojvodina

An Empirical Investigation of Software Testing Methods and Techniques in the Province of Vojvodina

A study of learning likely data structure properties using machine learning models

Data structure properties are important for many testing and analysis tasks. For example, model checkers use these properties to find program faults. These properties are often written manually which can be error prone and lead to false alarms. This paper presents the results of controlled experiments performed using existing machine learning (ML) models on various data structures. These data structures are dynamic and reside on the program heap. We use ten data structure subjects and ten ML models to evaluate the learnability of data structure properties. The study reveals five key findings. One, most of the ML models perform well in learning data structure properties, but some of the ML models such as quadratic discriminant analysis and Gaussian naive Bayes are not suitable for learning data structure properties. Two, most of the ML models have high performance even when trained on just 1% of data samples. Three, certain data structure properties such as binary heap and red black tree are more learnable than others. Four, there are no significant differences between the learnability of varied-size (i.e., up to a certain size) and fixed-size data structures. Five, there can be significant differences in performance based on the encoding used. These findings show that using machine learning models to learn data structure properties is very promising. We believe that these properties, once learned, can be used to provide a run-time check to see whether a program state at a particular point satisfies the learned property. Learned properties can also be employed in the future to automate static and dynamic analysis, which would enhance software testing and verification techniques.

Fast Track Technique for Software Testing and Quality Assurance Practice in Project Development Life Cycle

Fast Track Technique for Software Testing and Quality Assurance Practice in Project Development Life Cycle

Optimization of Software Testing Technique using Novel Genetic Algorithm

Optimization of Software Testing Technique using Novel Genetic Algorithm

Pengujian Perangkat Lunak Sistem Informasi Peminjaman PlayStation dengan Teknik Boundary Value Analysis Menggunakan Metode Black Box Testing

Testing on an application aims to check whether a program has run properly or there are still errors that must be corrected so that the program created will be a program that has good quality. In this study the software that will be reviewed using Black Box Testing is a PlayStation Loan Information Application System where this application aims to facilitate the admin (PlayStation rental) in data collection on the number of borrowers, as well as the return date of PlayStation borrowed by members, where there are 2 forms to be filled in. Member added data form, game added form during the loan. lending data information applications will be tested using Black Box Testing where this test only aims to see the program whether in accordance with the expected function or not without knowing the program code used. Black Box Testing Method consists of several techniques, including Equivalence Partitions, Boundary Value Analysis, Sample Testing, and so on. Among the many testing techniques, in this study the testing technique was chosen using Boundary Value Analysis. Boundary Value Analysis Techniques, are software testing techniques in which tests are designed to include representatives of boundary values in a range.

A Tool Development for Test Case Based Code Optimization in Java

Java has been a popular programming language since its first stable release in 1996 because of its platform independence. Along with its popularity Java has been a focus of performance studies since its debut. Developments in hardware has unbelievably advanced the performance of the devices that run Java and thus software performance has lost its popularity until the release of Android OS and rapid increase in mobile device ownership Java language usage has increased once again. Mobile devices having far less system resources compared to personal computers had re-brought software performance studies into the spotlight. However mobile devices have gone into a fast-paced development like all other information technologies and this brought down the need for software performance studies, again. Also, worth mentioning that development of new Java Virtual Machine (JVM) versions has made the specialized compiler studies, which may threaten the platform independency, obsolete except for specific situations. Today it is not enough to consider code optimization solely in terms of performance improvement. Much broader vision is needed like software development processes such as Maintainability, code readability, improving cooperation in multi-programmer projects, software quality assurance.In this study, white box testing approach is adopted as the software testing technique and static code analysis method is selected to ensure line coverage. A new software (JPA) has been developed based on a currently used testing tool (PMD) to improve the user experience.

Fuzzing

Reviewing software testing techniques for finding security vulnerabilities.

PENGUJIAN SISTEM PENDUKUNG KEPUTUSAN MENGGUNAKAN BLACK BOX TESTING STUDI KASUS E-WISUDAWAN DI INSTITUT SAINS DAN TEKNOLOGI AL-KAMAL

System development has several stages including the analysis, planning, design and implementation stages. At the implementation stage, it is divided into two, namely testing and maintaining the system. Testing is the process of checking whether a system can be run in accordance with specified standards. In general, system testing can be classified into two, namely software testing techniques and software testing strategies. Software testing techniques are classified into two groups namely white box testing and black box testing, while software testing strategies are classified into three, namely unit testing, integration testing, and validation testing. In this research software testing techniques are applied using black box testing techniques, while software testing strategies are implemented using validation testing. Black box testing technique focuses on system behavior based on the functional requirements of the system, while validation testing is carried out to ensure the compatibility of the system with the design that has been applied. In validation testing, there are two tests conducted, namely alpha testing and beta testing. E-Wisuda Institut Sains dan Teknologi Al-Kamal is a web based decision support system created to assist members of the senate in determining the best graduates, as well as being the best graduates at the graduation procession. E-Graduation decision support system test results state that the system has no syntax errors and functionally produces a system that is in accordance with user needs and the system can assist users in determining the best graduates and provide accurate results. Keyword : black box testing, software testing technique, software testing strategies

Multi-Task Optimization-Based Test Data Generation for Mutation Testing via Relevance of Mutant Branch and Input Variable

Mutation testing is a powerful software testing technique. However, it is difficult to obtain test data for killing a large number of mutants, especially for hard-to-kill mutants. Mutant branch is formed by an original statement of a program under test and its mutated statement. The true branch of a mutant branch is covered by a test datum, suggesting the corresponding mutant is killed under the criterion of weak mutation testing. This article focuses on efficiently generating test data for a large number of mutant branches. When generating test data using a search-based method, the size of the search domain is a determining factor affecting the search performance. The key observation is that only partial input variables affect whether a mutant will be killed, so the search domain can be reduced by deleting irrelevant variables. Along this line, we first group the mutant branches based on their relevant input variables, followed by formulating a multi-task optimization model of test data generation for the grouped mutant branches, in which the relevant input variables are taken as the decision variables. Finally, a multi-population genetic algorithm with individual sharing is employed to generate test data by multi-tasking. The experiments based on eight programs of various sizes show that removing related variable helps reduce the search domain, and the efficiency of test data generation by grouping and multitasking is improved.

Investigation onto the Software Testing Techniques and Tools: An Evaluation and Comparative Analysis

Investigation onto the Software Testing Techniques and Tools: An Evaluation and Comparative Analysis

Efficiently Generating Test Data to Kill Stubborn Mutants by Dynamically Reducing the Search Domain

Mutation testing is a fault-oriented software testing technique, and a test suite generated based on the criterion of mutation testing generally has a high capability in detecting faults. A mutant that is hard killed is called a stubborn one. The traditional methods of test data generation often fail to generate test data that kill stubborn mutants. To improve the efficiency of killing stubborn mutants, in this article, we propose a method of generating test data by dynamically reducing the search domain under the criterion of strong mutation testing. To fulfill this task, we first present a method of measuring the stubbornness of a mutant based on the reachability condition of a mutated statement. Then, we formulate the problem of generating test data to kill the mutant as an optimization one with a unique constraint. Finally, we generate test data using a coevolutionary genetic algorithm. Given the fact that the domain of test data that kills a stubborn mutant is generally small, we adopt a method of dynamically reducing the search domain to improve the efficiency of the algorithm. We apply the proposed method to test eight benchmark and industrial programs. The experimental results demonstrate that the proposed method has capabilities in seeking stubborn mutants and efficiently generating test data to kill stubborn mutants.

Importance of an Effective Test Suite Minimization Technique in Software Testing

Software testing is an important activity for developing a quality end product. But it is expensive as it requires both time and cost to perform it during development phase and after its delivery. Therefore, it is required to develop some techniques in this direction in order to make software testing a desirable as well as an affordable activity. Hence, in this study we have proposed an effective method that minimizes the size and redundancy from an automated random generated test suite using data clustering technique. The proposed technique compared to other existing techniques achieves a substantial amount of reduction in the size of the test suite without affecting the fault detection capability of the test suite.

Evolutionary Cost-cognizant Test Case Selection and Prioritization for Object-Oriented Programs

Test case prioritization (TCP) is a software testing technique that finds an ideal ordering of test cases for regression testing, so that testers can obtain the maximum benefit of their test suite, even if the testing process is stop at some arbitrary point. The recent trend of software development uses OO paradigm. This paper proposed a cost-cognizant TCP approach for object-oriented software that uses path-based integration testing. Path-based integration testing will identify the possible execution path and extract these paths from the Java System Dependence Graph (JSDG) model of the source code using forward slicing technique. Afterward evolutionary algorithm (EA) was employed to prioritize test cases based on the severity detection per unit cost for each of the dependent faults. The proposed technique was known as Evolutionary Cost-Cognizant Regression Test Case Prioritization (ECRTP) and being implemented as regression testing approach for experiment.

DEMINER: test generation for high test coverage through mutant exploration

SummaryMost software testing techniques test a target program as it is and fail to utilize valuable information of diverse test executions on many variants/mutants of the original program in test generation.This paper proposes a new test generation technique DEMINER, which utilizes mutant executions to guide test generation on the original program for high test coverage.DEMINER first generates various mutants of an original target program and then extracts runtime information of mutant executions, which covered unreached branches by the mutation effects. Using the obtained runtime information, DEMINER inserts guideposts, artificial branches to replay the observed mutation effects, to the original target programs. Finally, DEMINER runs automated test generation on the original program with guideposts and achieves higher test coverage.We implemented DEMINER for C programs through software mutation and guided test generation such as concolic testing and fuzzing. We have shown the effectiveness of DEMINER on six real‐world target programs: Busybox‐ls, Busybox‐printf, Coreutils‐sort, GNU‐find, GNU‐grep and GNU‐sed. The experiment results show that DEMINER improved branch coverage by 63.4% and 19.6% compared with those of the conventional concolic testing techniques and the conventional fuzzing techniques on average, respectively.

Aspect-Oriented Approach for Testing Software Applications and Automatic Aspect Creation

The existing techniques for software testing can be used to perform only a particular type of testing, and moreover proficiency is required to write the automation test scripts using these techniques. This paper proposes a novel software testing approach using Aspect-Oriented Programming (AOP) that alone suffices for carrying out most of the types of software testing and thus obliterates the need of using distinctive tools for different types of testing. Nevertheless, AOP is a new programming paradigm and not all testers have the proficiency of working with it. Hence, a domain-specific language named Testing Aspect Generator Language (TAGL) was developed which has got a very low learning curve. Using TAGL, testers can write the testing code in the form of natural language-like statements. Further, the lexical analyzer and parser, written using lex and yacc, convert the TAGL statements into actual testing code in the form of AOP. The proposed approach was applied for the testing of widely used open source projects and remarkable bugs were detected into them. A detailed comparison as to how our approach is effective than the conventional testing techniques is provided.

Binary Fuzz Testing Method Based on LSTM

Fuzzing is an effective software testing technique to find bugs. In the binary fuzzing, the attempt of generating test data mainly focuses on the improvement of the mutation algorithm, which lacks further screening of the test data. This paper proposes a binary fuzz testing method based on long short-term memory. This method record the executed path information by QEMU. Then we assign values to different code branches according to the frequency of executed path. The path weights can be calculated. The LSTM model will be trained by test data and the path weights. Finally, we can take the test data as input and filter the data by referring to the path weights. This method can improve the time efficiency of fuzz testing. Experiments on a variety of different types of binary programs show that compared to the state-of-the-art fuzzer American fuzzy lop, this method achieves higher code coverage and time efficiency in the same time.

Testing Complexity in Component Based Software Engineering

The Component Based software development which reduces the time, budget and efforts compare to the traditional software development. The components are reusable, developed by third parties and available at Commercial-Of-The Self (COTS). Most recent technologies such embedded systems, mobile phones, control systems and other paradigms are developed with this concept. Software researchers should pay more attention on testing of Component Based technology. The software testing is the most commonly used technique for validating the product quality. It is something different from the traditional software development, due to the invisible source code and design make it very difficult to trace the faults. Software testing strategy is road map integrates test case design methods into well planned series of steps for successful construction of software. The selection right strategy at the right time will make the software product as robust. This paper highlights the various software testing challenges, difficulties, strategies and techniques in perspective of Component Based Software Development

Effective State Encoding for Breadth-First Generation of Complex Structures

Generation of complex heap structures is required for many software testing and verification techniques, which are used to enhance software reliability. It enables them to work for programs dependent on such structures. Generating all the valid structures within a size bound is time-consuming, and it is hard to predict an input size completely generatable within a time bound. This is due to the depth-first traversal of the candidate space. However, an efficient breadth-first traversal requires a compact state encoding to store candidates for exploration in the next level. In this paper, we present present a novel state encoding technique that stores an incomplete representation of state that can be recovered during search. We build upon the Korat algorithm—demonstrated to effectively generate tests and find bugs for many software programs—and present iKorat , an incremental algorithm for breadth-first exploration of the search space. Our encoding enables a more efficient implementation of iterative deepening that avoids redundant work. Standard iterative deepening algorithms allow a breadth-first search when the underlying algorithm is depth-first by repeating the work of earlier iterations. iKorat, however, uses information from smaller sizes to avoid redundant work for larger sizes. It also enables a new technique for parallelizing Korat by communicating candidates using our encoding. piKorat generates structures of larger sizes in parallel as soon as incremental information is available from smaller sizes. Our evaluation shows that iKorat-based iterative deepening is more efficient than Korat and that piKorat naturally extends the technique for parallel generation of complex structures.