SRTEF: Automatic Test Function Recommendation with Scenarios for Implementing Stepwise Test Case

Abstract

Implementing test cases to automate test execution is a popular testing practice currently. A stepwise test case consists of several sequential test steps. Given a test function library, the typical way to implement a test case is calling the existing test functions in the library to reduce test cost. How to find the appropriate test function(s) to implement a test step in a given test case thus becomes an important problem. However, in current testing practices, test engineers usually select the appropriate test function manually by experience. It is time-consuming and could lead to invalid test results by selecting inappropriate or wrong test functions to call. In this paper, we propose an automatic test function recommendation approach with scenario named SRTEF (Scenario-based Recommendation of TEst Function). Given a test step, SRTEF uses two levels of similarities to recommend test functions, description similarity and scenario similarity. The description similarity measures the semantic relatedness between the test step and test function by their literal descriptions. To calculate the scenario similarity, SRTEF at first retrieves a set of historical test cases that contains test step(s) semantically similar to the given test step; then the scenario similarity between test step and test function is calculated according to the calling relation between retrieved test case and test function, and the co-occurrence relation among test functions. SRTEF has been successfully applied in Huawei. We evaluate SRTEF by using the dataset from Huawei and comparing with BIKER, reported as the best recommendation approach so far. The results show that SRTEF outperforms the BIKER approach by at least 49% in Mean Average Precision, 33% in Mean Reciprocal Rank, and 25% in Mean Recall.