Applying Statistical Sampling for Fast and Efficient Simulation of Commercial Workloads

Abstract

Commercial workloads form an important class of applications and have performance characteristics that are distinct from scientific and technical benchmarks such as the SPEC CPU. However, due to the prohibitive simulation time of commercial workloads, it is extremely difficult to use them in computer architecture research. In this paper, we study the efficacy of using statistical sampling-based simulation methodology for two classes of commercial workloads: a Java server benchmark SPECjbb2000 and an online transaction processing (OLTP) benchmark DBT-2. Our results show that, although SPECjbb2000 shows distinct garbage collection phases, there are no large-scale phases in the OLTP benchmark. We take advantage of this stationary behavior in the steady phase and propose a statistical sampling-based simulation technique DynaSim with two dynamic stopping rules. In this approach, the simulation terminates once the target accuracy has been met. We apply DynaSim to simulate commercial workloads and show that, with the simulation of only a few million total instructions, the error can be within 3 percent, at a confidence level of 99 percent. DynaSim compares favorably with random sampling and representative sampling in terms of the total number of instructions simulated (time cost) and with representative sampling in terms of the number of checkpoints (storage cost). DynaSim increases the usability of a sampling-based simulation approach for commercial workloads and will encourage the use of commercial workloads in computer architecture research.