Determining output uncertainty of computer system models

Abstract

Computer system models provide detailed answers to system performance. For a given system configuration, a system model estimates the cycles per instruction that the system would incur while running a given workload. In addition, it estimates the proportion of time that is spent in different parts of the system and other related metrics such as bus utilizations. Consider those inputs to a system model that are estimated with uncertainty. Examples include cache miss rates that are obtained via trace-driven cache simulation and also sometimes by extrapolating beyond the simulation domain. Errors incurred during the measurement and fitting processes are propagated to the system model outputs. On the other hand, other inputs such as hardware latencies are known precisely. In this paper we propose several measures of uncertainty of system model outputs when it stems from uncertainty in the inputs. Some of these measures are based on sensitivity of an output to the inputs. We propose ways of defining and determining these sensitivities and turning them into uncertainty measures. Other measures are based on sampling schemes. Additionally, we determine uncertainty measures for the system model outputs over a wide range of inputs covering large system design spaces. This is done by first selecting a set of input configurations based on an experimental design methodology where the uncertainty measures are determined. Then these data are used to interpolate the uncertainty measure function over the rest of the input space. We quantitatively characterize each input’s contribution to the output uncertainty over the input’s entire range. We also propose ways that call attention to high output uncertainty regions in the input space. The methodology is illustrated on system models developed at Sun Microsystems Laboratories. The particular goal of the performance analysis is a design of level two caches.