Computer Architecture Research Research Articles

Analysis of redundancy and application balance in the SPEC CPU2006 benchmark suite

The recently released SPEC CPU2006 benchmark suite is expected to be used by computer designers and computer architecture researchers for pre-silicon early design analysis. Partial use of benchmark suites by researchers, due to simulation time constraints, compiler difficulties, or library or system call issues is likely to happen; but a random subset can lead to misleading results. This paper analyzes the SPEC CPU2006 benchmarks using performance counter based experimentation from several state of the art systems, and uses statistical techniques such as principal component analysis and clustering to draw inferences on the similarity of the benchmarks and the redundancy in the suite and arrive at meaningful subsets. The SPEC CPU2006 benchmark suite contains several programs from areas such as artificial intelligence and includes none from the electronic design automation (EDA) application area. Hence there is a concern on the application balance in the suite. An analysis from the perspective of fundamental program characteristics shows that the included programs offer characteristics broader than the EDA programs' space. A subset of 6 integer programs and 8 floating point programs can yield most of the information from the entire suite.

Editor in Chief's Message: Truly "hot" chips--Do we still care?

Despite the move away from very high-frequency, high-ILP cores to multiple, more modest cores ("multicore"), power is still a huge, unsolved problem for the microprocessor industry. The emphasis is no longer power-aware processor microarchitecture but power-aware systems architecture. The "system" extends from the multicore system-on-chip to the external memory, disks, indeed to the entire enterprise. The data center has arisen as a major target of power-related computer architecture research. The greater question is, in our attempts to make the world's information available to all in the blink of an eye, what is the environmental cost, and how can we as a research community address this problem?

Subsetting the SPEC CPU2006 benchmark suite

On August 24, 2006, the Standard Performance Evaluation Corporation (SPEC) announced CPU2006 -- the next generation of industry-standardized CPU-intensive benchmark suite. The SPEC CPU benchmark suite has become the most frequently used suite for simulation-based computer architecture research. Detailed processor simulators take days to weeks to simulate each of the SPEC CPU programs. In order to reduce simulation to a tractable time, architects and researchers often use only a subset of benchmarks from the SPEC CPU suite to evaluate the potential of their ideas. Prior research has demonstrated that statistical techniques are most effective to find a representative subset of benchmark programs from a benchmark suite. The objective of this paper is to apply multivariate statistical data analysis techniques for selecting a representative subset of programs from the SPEC CPU2006 benchmark suite. We measure a set of performance counter based characteristics for the SPEC CPU2006 programs across a large number of architectures and apply multivariate statistical analysis techniques to find a representative subset of benchmarks and representative input sets wherever multiple input sets are provided. The results from this paper will help architects and researchers to find a smaller but representative set of programs from the SPEC CPU2006 benchmark suite, when time or resource constraints prohibit experimentation with the entire benchmark suite.

The future of simulation: a field of dreams

Due to the enormous complexity of computer systems, researchers use simulators to model system behavior and generate quantitative estimates of expected performance. Researchers also use simulators to model and assess the efficacy of future enhancements and novel systems. Arguably the most important tools available to computer architecture researchers, simulators offer a balance of cost, timeliness, and flexibility. Improving the infrastructure, benchmarking, and methodology of simulation - the dominant computer performance evaluation method - results in higher efficiency and let architects gain more insight into processor behavior. For these reasons, architecture researchers have increasingly relied on simulators

BASS

As software vulnerabilities continue to be exposed on a daily basis and the motivation of cunning adversaries to compromise valuable computer assets grows, novel methods must be developed to ensure security. Recently there has been a growing interest within the computer architecture research community in designing architectural and hardware mechanisms to improve security. Unfortunately, there is currently not a representative set of benchmarks for evaluating the security features of proposed hardware modifications. The frequent result is that great effort is often spent searching for vulnerable programs, and/or evaluations suffer from a lack of diversity. To address this problem, we developed BASS, a benchmark suite to evaluate the security features of proposed architectural solutions under various malicious attack scenarios. BASS v 1.0 currently consists of seven benchmarks chosen to cover a diverse range of architectural attack characteristics. To facilitate the use of these benchmarks in architectural security research, we have developed both vulnerable programs and scripts to automatically generate exploits targeting those vulnerable programs across both 32-bit x86 and 64-bit Alpha Linux platforms. The entire BASS framework including documentation, source code, input data sets, and precompiled binaries for the M5 full system simulator is released under the Gnu GPL and can be freely downloaded at http://www.ideal.ece.ufl.edu/bass.

Computer Architecture Research and Future Microprocessors

Computer Architecture Research and Future Microprocessors

Simulation of computer architectures: simulators, benchmarks, methodologies, and recommendations

Simulators have become an integral part of the computer architecture research and design process. Since they have the advantages of cost, time, and flexibility, architects use them to guide design space exploration and to quantify the efficacy of an enhancement. However, long simulation times and poor accuracy limit their effectiveness. To reduce the simulation time, architects have proposed several techniques that increase the simulation speed or throughput. To increase the accuracy, architects try to minimize the amount of error in their simulators and have proposed adding statistical rigor to their simulation methodology. Since a wide range of approaches exist and since many of them overlap, this paper describes, classifies, and compares them to aid the computer architect in selecting the most appropriate one.

An Efficient, Practical Parallelization Methodology for Multicore Architecture Simulation

Multiple core designs have become commonplace in the processor market, and are hence a major focus in modern computer architecture research. Thus, for both product development and research, multiple core processor simulation environments are necessary. A well-known positive feedback property of computer design is that we use today's computers to design tomorrow's. Thus, with the emergence of chip multiprocessors, it is natural to re-examine simulation environments written to exploit parallelism. In this paper we present a programming methodology for directly converting existing uniprocessor simulators into parallelized multiple-core simulators. Our method not only takes significantly less development effort compared to some prior used programming techniques, but also possesses advantages by retaining a modular and comprehensible programming structure. We demonstrate our case with actual developed products after applying this method to two different simulators, one developed from IBM Ibrandot and the other from the SimpleScalar tool set. Our SimpleScalar-based framework achieves a parallel speedup of 2.2times on a dual-CPU dual-core (4-way) Opteron server

Improving Computer Architecture Simulation Methodology by Adding Statistical Rigor

Due to cost, time, and flexibility constraints, computer architects use simulators to explore the design space when developing new processors and to evaluate the performance of potential enhancements. However, despite this dependence on simulators, statistically rigorous simulation methodologies are typically not used in computer architecture research. A formal methodology can provide a sound basis for drawing conclusions gathered from simulation results by adding statistical rigor and, consequently, can increase the architect's confidence in the simulation results. This paper demonstrates the application of a rigorous statistical technique to the setup and analysis phases of the simulation process. Specifically, we apply a Plackett and Burman design to: 1) identify key processor parameters, 2) classify benchmarks based on how they affect the processor, and 3) analyze the effect of processor enhancements. Our results showed that, out of the 41 user-configurable parameters in SimpleScalar, only 10 had a significant effect on the execution time. Of those 10, the number of reorder buffer entries and the L2 cache latency were the two most significant ones, by far. Our results also showed that instruction precomputation - a value reuse-like microarchitectural technique - primarily improves the processor's performance by relieving integer ALU contention.

BLRL: Accurate and Efficient Warmup for Sampled Processor Simulation

Current computer architecture research relies heavily on architectural simulation to obtain insight into the cycle-level behavior of modern microarchitectures. Unfortunately, such architectural simulations are extremely time-consuming. Sampling is an often-used technique to reduce the total simulation time. This is achieved by selecting a limited number of samples from a complete benchmark execution. One important issue with sampling, however, is the unknown hardware state at the beginning of each sample. Several approaches have been proposed to address this problem by warming up the hardware state before each sample. This paper presents the boundary line reuse latency (BLRL) which is an accurate and efficient warmup strategy. BLRL considers reuse latencies (between memory references to the same memory location) that cross the boundary line between the pre-sample and the sample to compute the warmup that is required for each sample. This guarantees a nearly perfect warmup state at the beginning of a sample. Our experimental results obtained using detailed processor simulation of SPEC CPU2000 benchmarks show that BLRL significantly outperforms the previously proposed memory reference reuse latency (MRRL) warmup strategy. BLRL achieves a warmup that is only half the warmup for MRRL on average for the same level of accuracy.

Evaluating the impact of the simulation environment on experimentation results

Simulation has become the tool of choice for advanced computer architecture research. Simulation model abstraction as well as compiler specifics can have significant impact on the observed performance trends and may lead to incorrect conclusions. This study presents a methodology that compares the predicted performance impact of several architectural techniques measured on a variety of simulator and compiler configurations. A significant difference in speedup between experiments indicates that a technique is sensitive to this particular variation. The results demonstrate a strong but non-uniform relationship between modeling abstraction and measurement error, as well as a significant sensitivity to the compiler.

Computer architecture research: Shifting priorities and newer challenges

Computer architecture research: Shifting priorities and newer challenges

Locality-based online trace compression

Trace-driven simulation is one of the most important techniques used by computer architecture researchers to study the behavior of complex systems and to evaluate new microarchitecture enhancements. However, modern benchmarks, which largely resemble real-world applications, result in long and unmanageable traces. Compression techniques can be employed to reduce storage requirement of traces. Special trace compression schemes such as Mache and PDATS/PDI take advantage of spatial locality to compress memory reference addresses. In this paper, we propose the locality-based trace compression (LBTC) method, which employs both spatial locality and temporal locality of program memory references. It efficiently compresses not only the address but also other attributes associated with each memory reference. In addition, LBTC is designed to be simple and on-the-fly. If traces with addresses and other attributes are compressed by LBTC, the compression ratio is better by a factor of 2 over compression by PDI.

Tools for computer architecture research

Simulators are critical tools for computer architecture research and education. They are invaluable when evaluating hardware designs and enhancements, that would otherwise be prohibitively expensive to prototype in practice. Simulators can be useful vehicles for verifying the validity of initial designs, understanding their cost-benefit trade-offs, whether or not a more expensive and time-consuming hardware prototyping effort is undertaken. In addition to being the sole vehicle for conducting an investigation in different research organizations, simulators are extensively used in industry for validating new ideas before justifying further investments on those ideas. Further, simulators can also serve as excellent platforms for teaching difficult concepts in hardware and compilers, by allowing students hands-on access to hardware and software internals that may not be accessible otherwise.

Recent extensions to the SimpleScalar tool suite

Over the past eight years, the SimpleScalar Tool suite has become the most widely used set of simulation tools in the computer architecture research community. The authors have recently completed an NSF-funded project to extend and improve the SimpleScalar tools. In this paper, we describe the extensions and improvements to the tools, which include the capability to simulate more instruction sets, graphical support for performance viewing, and more simulators that model different types of machines, including embedded systems, ISA-specific systems, systems with operating system, and multiprocessing systems.

How to use SimPoint to pick simulation points

Understanding the cycle level behavior of a processor running an application is crucial to modern computer architecture research. To gain this understanding, detailed cycle level simulators are typically employed. Unfortunately, this level of detail comes at the cost of speed, and simulating the full execution of an industry standard benchmark on even the fastest simulator can take weeks to months to complete. This fact has not gone unnoticed, and several techniques have been developed aimed at reducing simulation time.

Temperature-aware computer systems: opportunities and challenges

Temperature-aware design techniques have an important role to play in addition to traditional techniques like power-aware design and package- and board-level thermal engineering. The authors define the role of architecture techniques and describe hotspot, an accurate yet fast thermal model suitable for computer architecture research.

Designing computer architecture research workloads

Although architectural simulators model microarchitectures at a high abstraction level, the increasing complexity of both the microarchitectures themselves and the applications that run on them make simulator use extremely time-consuming. Simulators must execute huge numbers of instructions to create a workload representative of real applications, creating an unreasonably long simulation time and stretching the time to market. Using reduced input sets instead of reference input sets helps to solve this problem. The authors have developed a methodology that reliably quantifies program behavior similarity to verify if reduced input sets result in program behavior similar to the reference inputs.

Embedded instruction memory in automotive engine controllers

Embedded systems currently account for all but 2% of the microprocessor market. Yet often embedded processor cores are simply streamlined versions of microprocessors that were designed for desktop computers. In computer architecture research, the Standard Performance Evaluation Corporation (SPEC) benchmark programs are often used to evaluate the performance of computer systems. The characteristics of the SPEC programs do not match typical embedded applications , however, and therefore it is not clear to what extent performance studies conducted using the SPEC benchmarks are applicable to embedded systems. In this paper, the authors focus on a specific segment of the embedded market: automotive engine controllers. Using data gathered by tracing a state-of-the-art automotive controller, they present simulation results for a memory system that consists of a main memory containing compressed programs, a bus used to fetch compressed instruction blocks, a decompression subsystem, and an uncompressed instruction cache. They look at a number of system parameters, including the bus load, the average memory access time, and the performance ratio of the compressed versus uncompressed system. They also present an analysis of the static and dynamic (run-time) system behavior.

Automatically characterizing large scale program behavior

Understanding program behavior is at the foundation of computer architecture and program optimization. Many programs have wildly different behavior on even the very largest of scales (over the complete execution of the program). This realization has ramifications for many architectural and compiler techniques, from thread scheduling, to feedback directed optimizations, to the way programs are simulated. However, in order to take advantage of time-varying behavior, we must first develop the analytical tools necessary to automatically and efficiently analyze program behavior over large sections of execution.Our goal is to develop automatic techniques that are capable of finding and exploiting theLarge Scale Behaviorof programs (behavior seen over billions of instructions). The first step towards this goal is the development of a hardware independent metric that can concisely summarize the behavior of an arbitrary section of execution in a program. To this end we examine the use ofBasic Block Vectors.We quantify the effectiveness of Basic Block Vectors in capturing program behavior across several different architectural metrics, explore the large scale behavior of several programs, and develop a set of algorithms based on clustering capable of analyzing this behavior. We then demonstrate an application of this technology to automatically determine where to simulate for a program to help guide computer architecture research.