The effect of compiler-flag tuning on SPEC benchmark performance

Abstract

The SPEC CINT92 and CFP92 benchmark suites are application-based system benchmarks primarily intended for workstation-class system performance measurements. The SPEC CPU benchmark results are widely disseminated by system vendors and as such have become the de-facto standard for comparing system performance. Recently, many observers have expressed concerns about the suitability of published SPEC benchmark results in representing application performance on typical systems. The most outspoken concern is that there is too much freedom permitted in the manipulation of compiler flags. This has resulted in revisions to the SPEC reporting procedure.This paper presents and discusses many of the issues concerning the tuning of benchmarks through manipulation of compiler flags. We attempt to quantify the impact of these procedures through controlled experiments. Baseline performance results, using a set of uniform, common optimizations are compared to published data. Further experiments measure the performance of the SPEC benchmarks in the other common usage scenarios. These are a centralized file storage configuration and a system using common binaries among several implementations of the same architecture. Despite the great concern over the use of compiler flags in the SPEC community, our experiments show only a modest impact on performance. The more significant performance differential shown in the other experiments draws into question the utility of current SPEC data to many users.