Search-based Techniques Research Articles

White-Box Fuzzing RPC-Based APIs with EvoMaster: An Industrial Case Study

Remote Procedure Call (RPC) is a communication protocol to support client-server interactions among services over a network. RPC is widely applied in industry for building large-scale distributed systems, such as Microservices. Modern RPC frameworks include, for example, Thrift, gRPC, SOFARPC, and Dubbo. Testing such systems using RPC communications is very challenging, due to the complexity of distributed systems and various RPC frameworks the system could employ. To the best of our knowledge, there does not exist any tool or solution that could enable automated testing of modern RPC-based services. To fill this gap, in this article we propose the first approach in the literature, together with an open source tool, for fuzzing modern RPC-based APIs. The approach is in the context of white-box testing with search-based techniques. To tackle schema extraction of various RPC frameworks, we formulate a RPC schema specification along with a parser that allows the extraction from source code of any JVM RPC-based APIs. Then, with the extracted schema we employ a search to produce tests by maximizing white-box heuristics and newly defined heuristics specific to the RPC domain. We built our approach as an extension to an open source fuzzer (i.e., EvoMaster ), and the approach has been integrated into a real industrial pipeline that could be applied to a real industrial development process for fuzzing RPC-based APIs. To assess our novel approach, we conducted an empirical study with two artificial and four industrial web services selected by our industrial partner. In addition, to further demonstrate its effectiveness and application in industrial settings, we report results of employing our tool for fuzzing another 50 industrial APIs autonomously conducted by our industrial partner in their testing processes. Results show that our novel approach is capable of enabling automated test case generation for industrial RPC-based APIs (i.e., 2 artificial and 54 industrial). We also compared with a simple gray-box technique and existing manually written tests. Our white-box solution achieves significant improvements on code coverage. Regarding fault detection, by conducting a careful review with our industrial partner of the tests generated by our novel approach in the selected four industrial APIs, a total of 41 real faults were identified, which have now been fixed. Another 8,377 detected faults are currently under investigation.

Improve Model Testing by Integrating Bounded Model Checking and Coverage Guided Fuzzing

Eectromechanical systems built by Simulink or Ptolemy have been widely used in industry fields, such as autonomous systems and robotics. It is an urgent need to ensure the safety and security of those systems. Test case generation technologies are widely used to ensure the safety and security. State-of-the-art testing tools employ model-checking techniques or search-based methods to generate test cases. Traditional search-based techniques based on Simulink simulation are plagued by problems such as low speed and high overhead. Traditional model-checking techniques such as symbolic execution have limited performance when dealing with nonlinear elements and complex loops. Recently, coverage guided fuzzing technologies are known to be effective for test case generation, due to their high efficiency and impressive effects over complex branches of loops. In this paper, we apply fuzzing methods to improve model testing and demonstrate the effectiveness. The fuzzing methods aim to cover more program branches by mutating valuable seeds. Inspired by this feature, we propose a novel integration technology SPsCGF, which leverages bounded model checking for symbolic execution to generate test cases as initial seeds and then conduct fuzzing based upon these worthy seeds. Over the evaluated benchmarks which consist of industrial cases, SPsCGF could achieve 8% to 38% higher model coverage and 3x-10x time efficiency compared with the state-of-the-art works.

We’re Not Gonna Break It! Consistency-Preserving Operators for Efficient Product Line Configuration

When configuring a software product line, finding a good trade-off between multiple orthogonal quality concerns is a challenging multi-objective optimisation problem. State-of-the-art solutions based on search-based techniques create invalid configurations in intermediate steps, requiring additional repair actions that reduce the efficiency of the search. In this work, we introduce <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">consistency-preserving configuration operators</i> (CPCOs)—genetic operators that maintain valid configurations throughout the entire search. CPCOs bundle coherent sets of changes: the activation or deactivation of a particular feature together with other (de)activations that are needed to preserve validity. In our evaluation, our instantiation of the IBEA algorithm with CPCOs outperforms two state-of-the-art tools for optimal product line configuration in terms of both speed and solution quality. The improvements are especially pronounced in large product lines with thousands of features.

An optimized case-based software project effort estimation using genetic algorithm

Software development companies have long suffered from inaccurate estimation of their software projects. This in turn led to huge losses, especially in the financial resources available for the project as well as the time required to complete it. As a result of this, the research community has developed different methods for estimating effort in software projects in the hope of achieving high levels of accuracy and efficiency in the use of available resources. Among those methods that have proven to be accurate in estimating the effort of software projects is the use of machine learning (ML) techniques, especially the case-based reasoning technique (CBR). This technique is based on adapting previously successful solutions for similar software projects. However, the CBR technique suffers from a problem which is its multiple parameters that are difficult to be tuned. This justifies the importance of the adaptation and adjustment process as an essential part of CBR to produce accurate and efficient results with least absolute estimation error. In this paper, one of the most efficient multi-objective evolutionary techniques, the Genetic Algorithm (GA), are used to help find the best set of classical CBR parameters (feature selection, feature weighting, similarity measures, and k number of nearest neighbors) to produce the most accurate effort estimates for software projects. The proposed CBR-GA model showed the effectiveness of using the GA algorithm to search for the best combination of CBR parameters and thus improve its accuracy. This in turn is beneficial for project managers in the early financial planning phase for effort estimation and thus project cost control. To validate the proposed CBR-GA model, we used a set of public benchmark datasets available on PROMISE data repository, in addition we used a set of reliable evaluation metrics. The obtained results are promising in terms of accuracy and significance tests. This implies the importance of search-based techniques for tuning effort estimation methods.

Basic block coverage for search-based unit testing and crash reproduction

Search-based techniques have been widely used for white-box test generation. Many of these approaches rely on the approach level and branch distance heuristics to guide the search process and generate test cases with high line and branch coverage. Despite the positive results achieved by these two heuristics, they only use the information related to the coverage of explicit branches (e.g., indicated by conditional and loop statements), but ignore potential implicit branchings within basic blocks of code. If such implicit branching happens at runtime (e.g., if an exception is thrown in a branchless-method), the existing fitness functions cannot guide the search process. To address this issue, we introduce a new secondary objective, called Basic Block Coverage (BBC), which takes into account the coverage level of relevant basic blocks in the control flow graph. We evaluated the impact of BBC on search-based unit test generation (using the DynaMOSA algorithm) and search-based crash reproduction (using the STDistance and WeightedSum fitness functions). Our results show that for unit test generation, BBC improves the branch coverage of the generated tests. Although small (sim 1.5%), this improvement in the branch coverage is systematic and leads to an increase of the output domain coverage and implicit runtime exception coverage, and of the diversity of runtime states. In terms of crash reproduction, in the combination of STDistance and WeightedSum, BBC helps in reproducing 3 new crashes for each fitness function. BBC significantly decreases the time required to reproduce 43.5% and 45.1% of the crashes using STDistance and WeightedSum, respectively. For these crashes, BBC reduces the consumed time by 71.7% (for STDistance) and 68.7% (for WeightedSum) on average.

X-SBR: On the Use of the History of Refactorings for Explainable Search-Based Refactoring and Intelligent Change Operators

Refactoring is widely adopted nowadays in industry to restructure the code and meet high quality while preserving the external behavior. Many of the existing refactoring tools and research are based on search-based techniques to find relevant recommendations by finding trade-offs between different quality attributes. While these techniques show promising results on open-source and industry projects, they lack explanations of the recommended changes which can impact their trustworthiness when adopted in practice by developers. Furthermore, most of the adopted search-based techniques are based on random population generation and random change operators (e.g., crossover and mutation). However, it is critical to understand which good refactoring patterns may exist when applying change operators to either keep them or exchange with other solutions rather than destroying them with random changes. In this paper, we propose knowledge-informed change operators and an improved seeding mechanism that we integrated in a multi-objective genetic algorithm. We also provide explanations for refactoring solutions. First, we generate association rules using the Apriori algorithm to find relationships between applied refactorings in previous commits, their locations, and their rationale (quality improvements). Then, we use these rules to 1) initialize the population, 2) improve the change operators and seeding mechanisms of the multi-objective search in order to preserve and exchange good patterns in the refactoring solutions, and 3) explain how a sequence of refactorings collaborate in order to improve the quality of the system (e.g., fitness functions). The validation on large open-source systems shows that X-SBR provides refactoring solutions of a better quality than those given by the state-of-the-art techniques in terms of reducing the invalid refactorings, improving the quality, and increasing trustworthiness of the developers in the suggested refactorings via the provided explanations.

Search-based fairness testing for regression-based machine learning systems

ContextMachine learning (ML) software systems are permeating many aspects of our life, such as healthcare, transportation, banking, and recruitment. These systems are trained with data that is often biased, resulting in biased behaviour. To address this issue, fairness testing approaches have been proposed to test ML systems for fairness, which predominantly focus on assessing classification-based ML systems. These methods are not applicable to regression-based systems, for example, they do not quantify the magnitude of the disparity in predicted outcomes, which we identify as important in the context of regression-based ML systems.Method:We conduct this study as design science research. We identify the problem instance in the context of emergency department (ED) wait-time prediction. In this paper, we develop an effective and efficient fairness testing approach to evaluate the fairness of regression-based ML systems. We propose fairness degree, which is a new fairness measure for regression-based ML systems, and a novel search-based fairness testing (SBFT) approach for testing regression-based machine learning systems. We apply the proposed solutions to ED wait-time prediction software.Results:We experimentally evaluate the effectiveness and efficiency of the proposed approach with ML systems trained on real observational data from the healthcare domain. We demonstrate that SBFT significantly outperforms existing fairness testing approaches, with up to 111% and 190% increase in effectiveness and efficiency of SBFT compared to the best performing existing approaches.Conclusion:These findings indicate that our novel fairness measure and the new approach for fairness testing of regression-based ML systems can identify the degree of fairness in predictions, which can help software teams to make data-informed decisions about whether such software systems are ready to deploy. The scientific knowledge gained from our work can be phrased as a technological rule; to measure the fairness of the regression-based ML systems in the context of emergency department wait-time prediction use fairness degree and search-based techniques to approximate it.

A Systematic Mapping Study of Search-Based Software Engineering for Enterprise Application Integration

Search-Based Software Engineering (SBSE) is widely used in different fields of Software Engineering, notoriously, in Enterprise Application Integrations (EAIs). EAI encompasses methodologies, techniques and tools that a software engineer can use to create integration solutions. SBSE is currently an active research topic of increasing interest. The number and diversity of publications produced yearly are large to the extent that it is hard to identify the active research groups, their locations, techniques used and research topics that have not received enough attention. To answer these questions categorically, we have conducted systematic mapping study of the literature. In this paper, we report our methodology and findings. In our study, we used systematic search strategies that resulted in the retrieval of 560 articles, of which we first selected 25. Second, on the basis of the authors’ experience, we included eight additional articles. Finally, we used a snowballing sample technique to include another 12 articles. The results demonstrate that during the last two decades (1999–2020) EAI has benefited from the use of Search-Based Software Engineering techniques.

Improving Search-Based Automatic Program Repair With Neural Machine Translation

The challenge of automatically repairing bugs in programs to reduce debugging expenses and increase program quality is known as automated program repair. To overcome this issue, test-suite-based repair techniques use a specified test suite as an oracle and alter the input faulty program to pass the full test suite. GenProg is a well-known example of this kind of repair, in which genetic programming is used to reorder the statements already present in the faulty program. However, recent practical experiments suggest that GenProg’s performance, notably for Java, is not sufficient. Improved program dependability necessitates the use of automatic program repair techniques. Template-based program repair techniques have recently been combined with search-based techniques to solve program issues automatically. Although intriguing, it has two fundamental drawbacks: Its search space often lacks the correct solution, and the technique disregards program expertise, such as precise code language. Compared with the template-based program repair approach, existing neural-machine-translation-based approaches are not limited by these constraints due to their ability to learn and generate new solutions. We propose an approach that combines a search-based automatic program repair technique with a neural-machine-translation-based approach. More specifically, we use both redundancy assumption and sequence-to-sequence learning of correct patches as the source for potential fix statements that feed into a multiobjective evolutionary search algorithm to find test-suite-adequate patches. In this work, a novel framework called ARJANMT is introduced for automatically repairing Java programs. Two sets of controlled experiments are conducted on 410 bugs from two benchmarks to investigate the repairability and correctness of our proposed framework. A comparison between state-of-the-art automatic program repair frameworks is made. The experimental results indicate that combining those two types of repair techniques (search-based and neural-machine-translation-based) produces better results or fixes bugs that they previously were unable to fix individually.

Search-Based Optimal Solvers for the Multi-Agent Pathfinding Problem: Summary and Challenges

Multi-agent pathfinding (MAPF) is an area of expanding research interest. At the core of this research area, numerous diverse search-based techniques were developed in the past 6 years for optimally solving MAPF under the sum-of-costs objective function. In this paper we survey these techniques, while placing them into the wider context of the MAPF field of research. Finally, we provide analytical and experimental comparisons that show that no algorithm dominates all others in all circumstances. We conclude by listing important future research directions.

Performance of Enhanced Multiple-Searching Genetic Algorithm for Test Case Generation in Software Testing

Test case generation is an important process in software testing. However, manual generation of test cases is a time-consuming process. Automation can considerably reduce the time required to create adequate test cases for software testing. Genetic algorithms (GAs) are considered to be effective in this regard. The multiple-searching genetic algorithm (MSGA) uses a modified version of the GA to solve the multicast routing problem in network systems. MSGA can be improved to make it suitable for generating test cases. In this paper, a new algorithm called the enhanced multiple-searching genetic algorithm (EMSGA), which involves a few additional processes for selecting the best chromosomes in the GA process, is proposed. The performance of EMSGA was evaluated through comparison with seven different search-based techniques, including random search. All algorithms were implemented in EvoSuite, which is a tool for automatic generation of test cases. The experimental results showed that EMSGA increased the efficiency of testing when compared with conventional algorithms and could detect more faults. Because of its superior performance compared with that of existing algorithms, EMSGA can enable seamless automation of software testing, thereby facilitating the development of different software packages.

Comparison of search strategies for feature location in software models

Search-based model-driven engineering is the application of search-based techniques to specific problems that are related to software engineering that is driven using software models. In this work, we make use of measures from the literature to report feature location problems in models (size and volume of the model and density, multiplicity, and dispersion of the feature being located) and a set of search strategies (random search, iterated local search, hill climbing, an evolutionary algorithm, and a hybrid between an evolutionary algorithm and hill climbing). The goal is to analyze of the impact of different values that are used to describe the feature location problems and the performance obtained by the different search strategies. We apply the search strategies to 1895 feature location problems that are obtained from 40 industrial software product lines. This work shows that: 1) the best results overall are obtained by a hybrid between evolutionary algorithm and hill climbing; 2) the size of the search space has the greatest impact on the results obtained by the search strategies; and 3) the impact of each of the measures is not the same in the five search strategies. This work highlights the use of the search strategy that produces the best results. In addition, we provide recommendations on when to use each search strategy.

A Watershed Moment for Search-Based Software Engineering

Software engineering is about understanding and making the right tradeoffs. Many software engineering tasks-such as test case generation, design, sprint planning, and refactoring-boil down to understanding tradeoffs including concretely expressing the attributes to optimize and elements and decisions that will maximize their intended outcomes. Consequently, many software engineering problems can be formulated as search problems. Driven by these observations, Harman and Jones in 2001 emphasized the importance of concentrated research on the application of search-based techniques in software engineering and coined the subfield of search-based software engineering (SBSE) in software engineering research.1 During the two decades that followed, SBSE has seen a significant amount of increased research where metaheuristic algorithms are used to create recommendations for software engineering tasks.

An Adaptive Search Budget Allocation Approach for Search-Based Test Case Generation

Search-based techniques have been successfully used to automate test case generation. Such approaches allocate a fixed search budget to generate test cases aiming at maximizing code coverage. The search budget plays a crucial role; due to the hugeness of the search space, the higher the assigned budget, the higher the expected coverage. Code components have different structural properties that may affect the ability of search-based techniques to achieve a high coverage level. Thus, allocating a fixed search budget for all the components is not recommended and a component-specific search budget should be preferred. However, deciding the budget to assign to a given component is not a trivial task. In this article, we introduce Budget Optimization for Testing (BOT), an approach to adaptively allocate the search budget to the classes under test. BOT requires information about the branch coverage that will be achieved on each class with a given search budget. Therefore, we also introduce BRANCHOS, an approach that predicts coverage in a budget-aware way. The results of our experiments show that (i) BRANCHOS can approximate the branch coverage in time with a low error, and (ii) BOT can significantly increase the coverage achieved by a test generation tool and the effectiveness of generated tests.

A Review on Different Techniques for Mining Frequent Patterns in Unordered Trees

Numerous fields, such as bioinformatics, web mining, and social network analysis, now require frequent pattern mining in unordered trees. Finding repeating patterns and substructures inside a collection of unordered trees is the key to unlocking this technique's potential to provide important information about the underlying data. This paper gives a thorough overview of various methods for mining frequent patterns in unordered trees, highlighting their advantages, disadvantages, and practical uses.The review starts out by defining the basic terms and concepts related to frequent pattern mining in unordered trees. The discussion then moves on to a number of widely used algorithms in this setting, such as graph-based strategies, bottom-up tree traversal techniques, and depth-first search-based techniques. Each approach is thoroughly explained, including its underlying concepts, computational complexity, and applicability to different kinds of tree datasets.The paper also examines recent developments in the subject, including distributed frameworks and scalable parallel algorithms for mining common patterns in big unordered tree collections. In order to improve the mining process and the calibre of patterns found, it also looks at the incorporation of extra constraints and measurements like weighted support and tree edit distance.The paper also talks about recent developments in the subject, namely scalable parallel algorithms and distributed frameworks for finding common patterns in enormous unordered tree collections. In order to strengthen the mining process and raise the calibre of patterns found, it also looks at the incorporation of extra restrictions and metrics like tree edit distance and weighted support.

Predicting Software Defects for Object-Oriented Software Using Search-based Techniques

Development without any defect is unsubstantial. Timely detection of software defects favors the proper resource utilization saving time, effort and money. With the increasing size and complexity of software, demand for accurate and efficient prediction models is increasing. Recently, search-based techniques (SBTs) have fascinated many researchers for Software Defect Prediction (SDP). The goal of this study is to conduct an empirical evaluation to assess the applicability of SBTs for predicting software defects in object-oriented (OO) softwares. In this study, 16 SBTs are exploited to build defect prediction models for 13 OO software projects. Stable performance measures — GMean, Balance and Receiver Operating Characteristic-Area Under Curve (ROC-AUC) are employed to probe into the predictive capability of developed models, taking into consideration the imbalanced nature of software datasets. Proper measures are taken to handle the stochastic behavior of SBTs. The significance of results is statistically validated using the Friedman test complied with Wilcoxon post hoc analysis. The results confirm that software defects can be detected in the early phases of software development with help of SBTs. This paper identifies the effective subset of SBTs that will aid software practitioners to timely detect the probable software defects, therefore, saving resources and bringing up good quality softwares. Eight SBTs — sUpervised Classification System (UCS), Bioinformatics-oriented hierarchical evolutionary learning (BIOHEL), CHC, Genetic Algorithm-based Classifier System with Adaptive Discretization Intervals (GA_ADI), Genetic Algorithm-based Classifier System with Intervalar Rule (GA_INT), Memetic Pittsburgh Learning Classifier System (MPLCS), Population-Based Incremental Learning (PBIL) and Steady-State Genetic Algorithm for Instance Selection (SGA) are found to be statistically good defect predictors.

Rapid and Robust Adaptive Jaya (Ajaya) Based Maximum Power Point Tracking of a PV-Based Generation System

When subjected to partial shading (PS), photovoltaic (PV) arrays suffer from the significantly reduced output. Although the incorporation of bypass diodes at the output alleviates the effect of PS, such modification results in multiple peaks of output power. Conventional algorithms—such as perturb and observe (P&O) and hill-climbing (HC)—are not suitable to be employed to track the optimal peak due to their convergence to local maxima. To address this issue, various artificial intelligence (AI) based algorithms—such as an artificial neural network (ANN) and fuzzy logic control (FLC)—have been employed to track the maximum power point (MPP). Although these algorithms provide satisfactory results under PS conditions, a very large amount of data is required for their training process, thereby imposing an excessive burden on processor memory. Consequently, this paper proposes a novel optimization algorithm based on stochastic search (random exploration of search space), known as the adaptive jaya (Ajaya) algorithm in which two adaptive coefficients are incorporated for maximum power point tracking (MPPT) with a rapid convergence rate, fewer power fluctuations and high stability. The algorithm successfully eliminates the issues associated with existing conventional and AI-based algorithms. Moreover, the proposed algorithm outperforms other state-of-the-art stochastic search-based techniques in terms of fewer fluctuations, robustness, simplicity, and faster convergence to the optima. Extensive analysis of results obtained from MATLAB® is done to prove the above performance parameters under static insolation conditions (using a three, four and a five-module series-connected PV system), under dynamically varying insolation (using a four-module series connected system), by changing the PV module rating (using a four-module series connected system) and using an IEC standard.

The Symposium on Search-Based Software Engineering: Past, Present and Future

Abstract Context Search-Based Software Engineering (SBSE) is the research field where Software Engineering (SE) problems are modelled as search problems to be solved by search-based techniques. The Symposium on Search Based Software Engineering (SSBSE) is the premier event on SBSE, which had its 11th edition in 2019. Objective In order to better understand the characteristics and evolution of papers published at SSBSE, this work reports results from a mapping study targeting the proceedings of all SSBSE editions. Despite the existing mapping studies on SBSE, our contribution in this work is to provide information to researchers and practitioners willing to enter the SBSE field, being a source of information to strengthen the symposium, guide new studies, and motivate new collaboration among research groups. Method A systematic mapping study was conducted with a set of four research questions, in which 134 studies published in all editions of SSBSE, dated from 2009 to 2019, were evaluated. In a fifth question, 32 papers published in the challenge track were summarised. Results Throughout the years, 290 authors from 25 countries have contributed to the main track of the symposium, with the collaboration of at least two institutions in 46.3% of the papers. SSBSE papers have got substantial external visibility, as most citations are from different venues. The SE tasks addressed by SSBSE are mostly related to software testing, software debugging, software design, and maintenance. Evolutionary algorithms are present in 75% of the papers, being the most common search technique. The evaluation of the SBSE approaches usually includes industrial systems. Conclusions SSBSE has helped increase the popularity of SBSE in the SE research community and has played an important role on making SBSE mature. There are still problems and challenges to be addressed in the SBSE field, which can be tackled by SSBSE authors in further studies.

Using search-based techniques for testing executable software models specified through graph transformations

Design by contract is a software development methodology that uses contracts for defining interfaces among interacting components of a software system. Graph transformation system is used to specify the behavioral aspects of software components by defining the pre- and post-conditions of methods as contracts. In this paper, we focus on testing executable software models specified by a graph transformation system. A set of model-specific coverage criteria and a cost-aware search-based test generation approach are introduced. To evaluate the effectiveness of the proposed coverage criteria and the test generation approach, a type of mutation analysis is presented at the model level. Furthermore, a couple of fault-detection methods are used to assess the quality of the generated tests in the model-level mutation analysis. The proposed approach is implemented in GROOVE, a toolset for model checking graph transformation systems. The empirical results based on some well-known case studies demonstrate the efficiency and scalability of each proposed coverage criterion and testing approach. The comparison of the proposed test generation approach with state-of-the-art techniques indicates a significant improvement in terms of fault-detection capability and testing costs.

Phishing Attack Detection using a Search Engine and Heuristics-based Technique

The language used in the textual content of the webpage is the barrier in most of the existing anti-phishing methods. Most of the existing anti-phishing methods can identify the fake webpages written in the English language only. Therefore, we present a search engine-based method in this article, which identifies phishing webpages accurately regardless of the textual language used within the webpage. The proposed search engine-based method uses a lightweight, consistent and language independent search query to detect the legality of the suspicious URL. We have also integrated five heuristics with the search engine-based mechanism to improve the detection accuracy, as some newly created legitimate sites may not appear in the search engine. The proposed method can also correctly classify the newly created legitimate sites that are not classified by available search engine-based methods. Evaluation results show that our method outperforms the available search-based techniques and achieves 98.15% TPR of and only 0.05% FPR.