Speculative Execution Research Articles

An evaluation of speculative instruction execution on simultaneous multithreaded processors

Modern superscalar processors rely heavily on speculative execution for performance. For example, our measurements show that on a 6-issue superscalar, 93% of committed instructions for SPECINT95 are speculative. Without speculation, processor resources on such machines would be largely idle. In contrast to superscalars, simultaneous multithreaded (SMT) processors achieve high resource utilization by issuing instructions from multiple threads every cycle. An SMT processor thus has two means of hiding latency: speculation and multithreaded execution. However, these two techniques may conflict; on an SMT processor, wrong-path speculative instructions from one thread may compete with and displace useful instructions from another thread. For this reason, it is important to understand the trade-offs between these two latency-hiding techniques, and to ask whether multithreaded processors should speculate differently than conventional superscalars.This paper evaluates the behavior of instruction speculation on SMT processors using both multiprogrammed (SPECINT and SPECFP) and multithreaded (the Apache Web server) workloads. We measure and analyze the impact of speculation and demonstrate how speculation on an 8-context SMT differs from superscalar speculation. We also examine the effect of speculation-aware fetch and branch prediction policies in the processor. Our results quantify the extent to which (1) speculation is critical to performance on a multithreaded processor because it ensures an ample supply of parallelism to feed the functional units, and (2) SMT actually enhances the effectiveness of speculative execution, compared to a superscalar processor by reducing the impact of branch misprediction. Finally, we quantify the impact of both hardware configuration and workload characteristics on speculation's usefulness and demonstrate that, in nearly all cases, speculation is beneficial to SMT performance.

A compiler framework for speculative analysis and optimizations

Speculative execution, such as control speculation and data speculation, is an effective way to improve program performance. Using edge/path profile information or simple heuristic rules, existing compiler frameworks can adequately incorporate and exploit control speculation. However, very little has been done so far to allow existing compiler frameworks to incorporate and exploit data speculation effectively in various program transformations beyond instruction scheduling. This paper proposes a speculative SSA form to incorporate information from alias profiling and/or heuristic rules for data speculation, thus allowing existing program analysis frameworks to be easily extended to support both control and data speculation. Such a general framework is very useful for EPIC architectures that provide checking (such as advanced load address table (ALAT) [10]) on data speculation to guarantee the correctness of program execution. We use SSAPRE [21] as one example to illustrate how to incorporate data speculation in those important compiler optimizations such as partial redundancy elimination (PRE), register promotion, strength reduction and linear function test replacement. Our extended framework allows both control and data speculation to be performed on top of SSAPRE and, thus, enables more aggressive speculative optimizations. The proposed framework has been implemented on Intel's Open Research Compiler (ORC). We present experimental data on some SPEC2000 benchmark programs to demonstrate the usefulness of this framework and how data speculation benefits partial redundancy elimination.

Realizing high IPC through a scalable memory-latency tolerant multipath microarchitecture

A microarchitecture is described that achieves high performance on conventional single-threaded program codes without compiler assistance. To obtain high instructions per clock (IPC) for inherently sequential (e.g., SpecInt-2000 programs), a large number of instructions must be in flight simultaneously. However, several problems are associated with such microarchitectures, including scalability, issues related to control flow, and memory latency.Our design investigates how to utilize a large mesh of processing elements in order to execute a singlethreaded program. We present a basic overview of our microarchitecture and discuss how it addresses scalability as we attempt to execute many instructions in parallel. The microarchitecture makes use of control and value speculative execution, multipath execution, and a high degree of out-of-order execution to help extract instruction level parallelism. Execution-time predication and time-tags for operands are used for maintaining program order. We provide simulation results for several geometries of our microarchitecture illustrating a range of design tradeoffs. Results are also presented that show the small performance impact over a range of memory system latencies.

Dynamic schemes for speculative execution of code

Speculative execution of code is becoming a key technique for enhancing the performance of pipeline processors. In this work we study schemes that predict the execution path of a program based on the history of branch executions. Building on previous work, we present a model for analyzing the effective speedup from pipelining, when speculative execution is employed. We follow this with stochastic analyses of several schemes for speculative execution. A main result of our study is that if we can predict branch resolution with high probability (as in the Pentium Pro processor, e.g.) the Single Path scheme commonly used on modern processors is within factor of 2 from the optimal. We conclude with simulations covering several of the settings that we study.

Formal Verification of a Complex Pipelined Processor

This paper addresses the problem of formally verifying the correctness of a complex pipelined microprocessor at the micro-architectural level of abstraction. The design verified is an example out-of-order execution processor with a reorder buffer, a store buffer, branch prediction, speculative execution and exceptions. We propose a systematic approach called the Completion Functions Approach to decompose and incrementally build its proof of correctness. The central idea is to construct the abstraction function using completion functions, one per unfinished instruction, each of which specifies the effect on the programmer visible state components of completing the instruction. This construction of the abstraction function leads to a very natural decomposition of the proof into proving a series of verification conditions. The approach prescribes a systematic way to generate these verification conditions which can then be discharged with a high degree of automation using techniques based on decision procedures and rewriting. The verification was completed in 34 person days, which we believe, is a modest investment in return for the significant benefits of formal verification.

On the performance potential of speculative execution based on branch and value prediction

Fluid Stochastic Petri Nets are used to capture the dynamic behavior of an ILP processor, and discrete-event simulation is applied to assess the performance potential of predictions and speculative execution in boosting the performance of ILP processors that fetch, issue, execute and commit a large number of instructions per cycle.

Modeling And Performance Evaluation Of Branch And Value Prediction In Ilp Processors

Speculative execution is one of the key issues to boost the performance of future generation microprocessors. In this paper, we introduce a novel approach to evaluate the effects of branch and value prediction, which allow the processor to execute instructions beyond the limits of control and true data dependences. Until now, almost all the estimations of their performance potential under different scenarios have been obtained using trace-driven or execution-driven simulation. Occasionally, some simple deterministic models have been used. We employ an analytical model based on recently introduced Fluid Stochastic Petri Nets (FSPNs) in order to capture the dynamic behavior of an ILP processor with aggressive use of prediction techniques and speculative execution. Here we define the FSPN model, derive the state equations for the underlying stochastic process and present performance evaluation results to illustrate its usage in deriving measures of interest. Our implementation-independent stochastic modeling framework reveals considerable potential for further research in this area using numerical solution of systems of partial differential equations and/or discrete-event simulation of FSPN models.

Speculative Computation and Action Execution in Multi-Agent Systems

In some multi-agent systems, when an agent cannot retrieve information from another agent, the agent makes an assumption and tentatively performs the computation. When the agent comes across a mistake in the preliminary assumption, the computation is modified. This kind of speculative computation is effective when the assumption is correct. However, once the agent executes an action, it is impossible to modify the computation in these systems. This paper shows how to integrate speculative computation and action execution through logic programming.

On augmenting trace cache for high-bandwidth value prediction

Value prediction is a technique that breaks true data dependences by predicting the outcome of an instruction and speculatively executes its data-dependent instructions based on the predicted outcome. As the instruction fetch rate and issue rate of processors increase, the potential data dependences among instructions issued in the same cycle also increase. Value prediction and speculative execution become critical to keep the issue rate high. Unfortunately, most of the proposed value prediction schemes focused only on the accuracy of the prediction. They have yet to consider the bandwidth required to access the value prediction tables. In this paper, we focus on the bandwidth issues of the value prediction. We propose augmenting the trace cache (which was proposed to provide the required fetch bandwidth for wide-issue ILP processors) with a copy of the predicted values and moving the generation of those predicted values (which require accessing the value prediction tables) from the instruction fetch stage to a later stage, e.g., the writeback stage. Such a change will allow "selective value prediction," i.e., only those instructions which require value prediction will access the value prediction tables. It can significantly reduce the bandwidth requirement of value prediction tables. We also use a dynamic classification scheme to steer predictor updates to behavior-specific tables (such as last-value, stride, two-level, etc.). A relatively even split among such table accesses further moderates the bandwidth requirement of those tables.

Efficient scheduling of conditional behaviors for high-level synthesis

As hardware designs get increasingly complex and time-to-market constraints get tighter there is strong motivation for high-level synthesis (HLS). HLS must efficiently handle both dataflow-dominated and controlflow-dominated designs as well as designs of a mixed nature. In the past efficient tools for the former type have been developed but so far HLS of conditional behaviors lags behind. To bridge this gap an efficient scheduling heuristic for conditional behaviors is presented. Our heuristic and the techniques it utilizes are based on a unifying design representation appropriate for both types of behavioral descriptions, enabling the proposed heuristic to exploit under the same framework several well-established techniques (chaining, multicycling) as well as conditional resource sharing and speculative execution which are essential in efficiently scheduling conditional behaviors. Preliminary experiments confirm the effectiveness of our approach and prompted the development of the CODESIS HLS tool for further experimentation.

Special issue on specialized computer architecture simulators that see the present and may hold the future

Special issue on <i>specialized</i> computer architecture simulators that see the present and may hold the future

The KScalar simulator

Modern processors increase their performance with complex microarchitectural mechanisms, which makes them more and more difficult to understand and evaluate. KScalar is a graphical simulation tool that facilitates the study of such processors. It allows students to analyze the performance behavior of a wide range of processor microarchitectures: from a very simple in-order, scalar pipeline, to a detailed out-of-order, superscalar pipeline with non-blocking caches, speculative execution, and complex branch prediction. The simulator interprets executables for the Alpha AXP instruction set: from very short program fragments to large applications. The object's program execution may be simulated in varying levels of detail: either cycle-by-cycle, observing all the pipeline events that determine processor performance, or million cycles at once, taking statistics of the main performance issues.Instructors may use KScalar in several ways. First, it may be used to provide demonstrations in lectures or online learning environments. Second, it allows students to investigate the characteristics of specific processor microarchitectures as practical short assignments associated to a lecture course. Third, students may undertake major projects involving the optimization of real programs at the software-hardware interface, or involving the optimization of a processor microarchitecture for a given application workload.A preliminary version of KScalar has been successfully used in several lecture courses during the last two years in the University Autónoma of Barcelona. It runs on a x86/Linux/KDE system. The graphical interface has been developed using the KDE and QT libraries. The simulator engine running behind the graphical interface is a heavily-modified version of SimpleScalar. KScalar code is available under the terms of the GNU and SimpleScalar General Public License

Evaluation of a data preload mechanism for a linked list structure

AbstractThe authors have been developing a data preload/prefetch scheme for a linked list structure. In this paper, results of evaluating this preload scheme with benchmark programs requiring frequent pointer operations are reported. This scheme is characterized by the fact that the pointer value of a linked list is preloaded and stored, and a mechanism of providing the pointer value without delay for subsequent instructions is installed inside the processor. While dramatic performance improvement is expected by preloading/prefetching of a pointer value in this scheme, it is feared that a speculative execution using a preloaded pointer value before an update may fail if the linked list structure is modified, which may influence the performance negatively. However, it has been confirmed that in actual programs, such a failure of speculative execution does not occur, so that no performance problem arises. The evaluation results have confirmed a maximum performance improvement of 24.3% and the efficacy of the scheme. © 2002 Scripta Technica, Syst Comp Jpn, 33(3): 21–30, 2002

Functional verification of instruction processing units through control flow modeling

The design verification of state-of-the-art high-performance microprocessors has become a significant challenge for test engineers. Deep pipelines, multiple execution units, out-of-order and speculative execution techniques, typically found in such microprocessors, contribute much to this complexity. Conventional methods, which treat the processor as a logic state machine or apply architectural level tests, fail to provide coverage of all possible corner cases in the design. This paper presents a functional verification method for modern microprocessors, which is based on innovative models of the microprocessor architecture, intended to cover the testing of all corner cases. In order to test the models presented in this work, an architecture independent coverage measurement system has been developed. The models were tested with both random code and real world applications in order to determine which of the two achieves higher coverage.

A Comparison of Asymptotically Scalable Superscalar Processors

The poor scalability of existing superscalar processors has been of great concern to the computer engineering community. In particular, the critical-path lengths of many components in existing implementations grow as Θ(n 2 ) where n is the fetch width, the issue width, or the window size. This paper describes two scalable processor architectures, Ultrascalar I and Ultrascalar II, and compares their VLSI complexities (gate delays, wire-length delays, and area.) Both processors are implemented by a large collection of ALUs with controllers (together called execution stations ) connected together by a network of parallel-prefix tree circuits. A fat-tree network connects an interleaved cache to the execution stations. These networks provide the full functionality of superscalar processors including renaming, out-of-order execution, and speculative execution. The difference between the processors is in the mechanism used to transmit register values from one execution station to another. Both architectures use a parallel-prefix tree to communicate the register values between the execution stations. Ultrascalar I transmits an entire copy of the register file to each station, and the station chooses which register values it needs based on the instruction. Ultrascalar I uses an H-tree layout. Ultrascalar II uses a mesh-of-trees and carefully sends only the register values that will actually be needed by each subtree to reduce the number of wires required on the chip. The complexity results are as follows: The complexity is described for a processor which has an instruction-set architecture containing L logical registers and can execute n instructions in parallel. The chip provides enough memory bandwidth to execute up to M(n) memory operations per cycle. (M is assumed to have a certain regularity property.) In all the processors, the VLSI area is the square of the wire delay. Ultrascalar I has gate delay O(log n) and wire delay \tauwires = \Theta(\sqrt{n}L) if $M(n)$ is $O(n^{1/2-\varepsilon})$, \tauwires = \Theta(\sqrt{n}(L+\log n)) if $M(n)$ is $\Theta(n^{1/2})$, \tauwires = \Theta(\sqrt{n}L+M(n)) if $M(n)$ is $\Omega(n^{1/2+\varepsilon})$ for ɛ>0 . Ultrascalar II has gate delay Θ(log L+log n) . The wire delay is Θ(n) , which is optimal for n=O(L) . Thus, Ultrascalar II dominates Ultrascalar I for n=O(L 2 ) , otherwise Ultrascalar I dominates Ultrascalar II. We introduce a hybrid ultrascalar that uses a two-level layout scheme: Clusters of execution stations are layed out using the Ultrascalar II mesh-of-trees layout, and then the clusters are connected together using the H-tree layout of Ultrascalar I. For the hybrid (in which n≥ L ), the wire delay is Θ(\sqrt nL+M(n)) , which is optimal. For n≥ L , the hybrid dominates both Ultrascalar I and Ultrascalar II. We also present an empirical comparison of Ultrascalar I and the hybrid, both layed out using the Magic VLSI editor. For a processor that has 32 32-bit registers and a simple integer ALU, the hybrid requires about 11 times less area.

Request Reordering to Enhance the Performance of Strict Consistency Models

Advances in ILP techniques enable strict consistencymodels to relax memory order through speculative execution ofmemory operations. However, ordering constraints still hinderthe performance because speculatively executed operationscannot be committed out of program order for the possibility ofmis-speculation. In this paper, we propose a new technique whichallows memory operations to be non-speculatively committed outof order without violating consistency constraints.

Verification of FM9801: An Out-of-Order Microprocessor Model with Speculative Execution, Exceptions, and Program-Modifying Capability

We have verified the FM9801, a microprocessor design whose features include speculative execution, out-of-order issue and completion of instructions using Tomasulo's algorithm, and precise exceptions and interrupts. As a correctness criterion, we used a commutative diagram that compares the result of the pipelined execution from a flushed state to another flushed state with that of the sequential execution. Like many pipelined microprocessors, the FM9801 may not operate correctly if the executed program modifies itself. We discuss the condition under which the processor is guaranteed to operate correctly. In order to show that the correctness criterion is satisfied, we introduce an intermediate abstraction that records the history of executed instructions. Using this abstraction, we define a number of invariant properties that must hold during the operation of the FM9801. We verify these invariant properties, and then derive the proof of the commutative diagram from them. The proof has been mechanically checked by the ACL2 theorem prover.

HIP

The standard way to notify the processor of a network event, such as the arrival or transmission of a packet, is through interrupts. Interrupts are more effective than polling, in terms of the per packet send/receive latency. Interrupts, however, incur a high overhead both during and after the interrupt handling, because modern superscalar processors use long pipelines, out-of-order and speculative execution, and multi-level memory systems, all of which tend to increase the interrupt overhead in terms of clock cycles. In this paper, we attempt to reduce the network interface overhead by introducing a hybrid scheme (HIP) that uses interrupts under low network load conditions and polling otherwise. Even though such hybrid schemes have been proposed in the past, the polling period in HIP is adjusted dynamically based on the rate of the arriving packet stream. In this way, the increase in the per packet latency, which occurs with polling, is quite low. This is quantified with trace-driven simulations, which also show that the per packet overhead with HIP is significantly reduced compared to the conventional interrupt-based mechanism. HIP would be beneficial for high bandwidth network interfacing in servers with a heavy WWW or streaming media workload.

Coarse-grained thread pipelining: a speculative parallel execution model for shared-memory multiprocessors

This paper presents a new parallelization model, called coarse-grained thread pipelining, for exploiting speculative coarse-grained parallelism from general-purpose application programs in shared-memory multiprocessor systems. This parallelization model, which is based on the fine-grained thread pipelining model proposed for the superthreaded architecture, allows concurrent execution of loop iterations in a pipelined fashion with runtime data-dependence checking and control speculation. The speculative execution combined with the runtime dependence checking allows the parallelization of a variety of program constructs that cannot be parallelized with existing runtime parallelization algorithms. The pipelined execution of loop iterations in this new technique results in lower parallelization overhead than in other existing techniques. We evaluated the performance of this new model using some real applications and a synthetic benchmark. These experiments show that programs with a sufficiently large grain size compared to the parallelization overhead obtain significant speedup using this model. The results from the synthetic benchmark provide a means for estimating the performance that can be obtained from application programs that will be parallelized with this model. The library routines developed for this thread pipelining model are also useful for evaluating the correctness of the codes generated by the superthreaded compiler and in debugging and verifying the simulator for the superthreaded processor.

Reference idempotency analysis

Recent proposals for multithreaded architectures allow threads with unknown dependences to execute speculatively in parallel. These architectures use hardware speculative storage to buffer uncertain data, track data dependences and roll back incorrect executions. Because all memory references access the speculative storage, current proposals implement this storage using small memory structures for fast access. The limited capacity of the speculative storage causes considerable performance loss due to speculative storage overflow whenever a thread's speculative state exceeds the storage capacity. Larger threads exacerbate the overflow problem but are preferable to smaller threads, as larger threads uncover more parallelism. In this paper, we discover a new program property called memory reference idempotency . Idempotent references need not be tracked in the speculative storage, and instead can directly access non-speculative storage (i.e., the conventional memory hierarchy). Thus, we reduce the demand for speculative storage space. We define a formal framework for reference idempotency and present a novel compiler-assisted speculative execution model. We prove the necessary and sufficient conditions for reference idempotency using our model. We present a compiler algorithm to label idempotent memory references for the hardware. Experimental results show that for our benchmarks, over 60% of the references in non-parallelizable program sections are idempotent.