An Industrial Case Study on Speeding Up User Acceptance Testing by Mining Execution Logs

Abstract

Software reliability is defined as the probability of failure-free operation for a period of time, under certain conditions. To determine whether the reliability of an application satisfies the reliability requirements, User Acceptance Testing is performed at deployment sites. To support the wide variation in configurations and usage patterns, User Acceptance Testing has become a crucial step in large deployments of mission-critical applications. However, verifying the long-term reliability of an application requires lengthy on-site engagements and dedicated use of costly lab setups. In this paper, we propose a technique to reduce the time and cost needed for User Acceptance Testing. We use a repository of execution logs from related deployments and prior tests of the application to mine reliability estimates. We then customize these estimates by mining logs generated from a limited-time User Acceptance Test (i.e., one day of testing) instead of from traditionally longer tests (e.g., one week of testing). Deployers of applications can use such customized estimates to determine whether an application satisfies their reliability requirements. Through a case study on a large-scale enterprise application, we show that our reliability estimate lies within 4% of the reliability estimate derived from the longer User Acceptance Tests.