Software-controlled Prefetching Research Articles

Tolerating memory latency through software-controlled pre-execution in simultaneous multithreading processors

Hardly predictable data addresses in many irregular applications have rendered prefetching ineffective. In many cases, the only accurate way to predict these addresses is to directly execute the code that generates them. As multithreaded architectures become increasingly popular, one attractive approach is to use idle threads on these machines to perform pre-execution— essentially a combined act of speculative address generation and prefetching—to accelerate the main thread. In this paper, we propose such a pre-execution technique for simultaneous multithreading (SMT) processors. By using software to control pre-execution, we are able to handle some of the most important access patterns that are typically difficult to prefetch. Compared with existing work on pre-execution, our technique is significantly simpler to implement (e.g., no integration of pre-execution results, no need of shortening programs for pre-execution, and no need of special hardware to copy register values upon thread spawns). Consequently, only minimal extensions to SMT machines are required to support our technique. Despite its simplicity, our technique offers an average speedup of 24% in a set of irregular applications, which is a 19% speedup over state-of-the-art software-controlled prefetching.

Compiler Controlled Prefetching for Multiprocessors Using Low-Overhead Traps and Prefetch Engines

In this paper we propose and evaluate a new data-prefetching technique for cache coherent multiprocessors. Prefetches are issued by a functional unit called a prefetch engine which is controlled by the compiler. We let second-level cache misses generate cache miss traps and start the prefetch engine in a trap handler. The trap handler is fast (40–50 cycles) and does not normally delay the program beyond the memory latency of the miss. Once started, the prefetch engine executes on its own and causes no instruction overhead. The only instruction overhead in our approach is when a trap handler completes after data arrives. The advantages of this technique are (1) it exploits static compiler analysis to determine what to prefetch, which is hard to do in hardware, (2) it uses prefetching with very little instruction overhead, which is a limitation for traditional software-controlled prefetching, and (3) it is accurate in the sense that it generates very little useless traffic while maintaining a high prefetching coverage. We also study whether one could emulate the prefetch engine in software, which would not require any additional hardware beyond support for generating cache miss traps and ordinary prefetch instructions. In this paper we present the functionality of the prefetch engine and a compiler algorithm to control it. We evaluate our technique on six parallel scientific and engineering applications using an optimizing compiler with our algorithm and a simulated multiprocessor. We find that the prefetch engine removes up to 67% of the memory access stall time at an instruction overhead less than 0.42%. The emulated prefetch engine removes in general less stall time at a higher instruction overhead.

Automatic compiler-inserted prefetching for pointer-based applications

As the disparity between processor and memory speeds continues to grow, memory latency is becoming an increasingly important performance bottleneck. While software-controlled prefetching is an attractive technique for tolerating this latency, its success has been limited thus far to array-based numeric codes. In this paper, we expand the scope of automatic compiler-inserted prefetching to also include the recursive data structures commonly found in pointer-based applications. We propose three compiler-based prefetching schemes, and automate the most widely applicable scheme (greedy prefetching) in an optimizing research compiler. Our experimental results demonstrate that compiler-inserted prefetching can offer significant performance gains on both uniprocessors and large-scale shared-memory multiprocessors.

The limits and effectiveness of data prefetching on scalable multiprocessors

Abstract Prefetching is a promising technique for hiding and tolerating the large memory latencies expected in scalable multiprocessors. In this paper we present and validate an analytical performance model for software-controlled data prefetching. The model incorporates all the important aspects affecting the performance of prefetching such as: program behavior, network topology, cache coherency protocols, memory consistency models, etc. We use execution-driven simulation to validate the predictions of the model with respect to overall speedup, average memory latency, and cache pollution. We show that the model provides accurate predictions for programs that do not saturate the bandwidth of the network. The model could be used by compilers and/or programmers to determine when to issue prefetch instructions in order to maximize the speedup that can be obtained from software-controlled prefetching.

The limits and effectiveness of data prefetching on scalable multiprocessors

Prefetching is a promising technique for hiding and tolerating the large memory latencies expected in scalable multiprocessors. In this paper we present and validate an analytical performance model for software-controlled data prefetching. The model incorporates all the important aspects affecting the performance of prefetching such as: program behavior, network topology, cache coherency protocols, memory consistency models, etc. We use execution-driven simulation to validate the predictions of the model with respect to overall speedup, average memory latency, and cache pollution. We show that the model provides accurate predictions for programs that do not saturate the bandwidth of the network. The model could be used by compilers and/or programmers to determine when to issue prefetch instructions in order to maximize the speedup that can be obtained from software-controlled prefetching.

A software-controlled prefetching mechanism for software-managed TLBs

The TLB (Translation Lookaside Buffer) miss services have been concealed from operating systems, but some new RISC architectures manage the TLB in software. Since software-managed TLBs provide flexibility to an operating system in page translation, they are considered an important factor in the design of microprocessors for open system environments. However, software-managed TLBs suffer from larger miss penalty than hardware-managed TLBs, since they require more extra context switching overhead than hardware-managed TLBs. This paper introduces a new technique for reducing the miss penalty of software-managed TLBs by prefetching necessary TLB entries before being used. This technique is not inherently limited to specific applications. The key of this scheme is to perform the prefetch operations to update the TLB entries before first accesses so that TLB misses can be avoided. Using trace-driven simulation and a quantitative analysis, the proposed scheme is evaluated in terms of the miss rate and the total miss penalty. Our results show that the proposed scheme reduces the TLB miss rate by a factor of 6% to 77% due to TLB characteristics and page sizes. In addition, it is found that reducing the miss rate by the prefetching scheme reduces the total miss penalty and bus traffics in software-managed TLBs.

Tolerating latency through software-controlled prefetching in shared-memory multiprocessors

The large latency of memory accesses is a major obstacle in obtaining high processor utilization in large-scale shared-memory multiprocessors. Although the provision of coherent caches in many recent machines has alleviated the problem somewhat, cache misses still occur frequently enough that they significantly lower performance. In this paper we evaluate the effectiveness of nonbinding software-controlled prefetching, as proposed in the Stanford DASH multiprocessor, to address this problem. The prefetches are nonbinding in the sense that the prefetched data is brought to a cache close to the processor, but is still available to the cache-coherence protocol to keep it consistent. Prefetching is software-controlled since the program must explicitly issue prefetch instructions. The paper presents results from detailed simulation studies done in the context of the Stanford DASH multiprocessor. Our results show that for applications with regular data access patterns—we evaluate a particle-based simulator used in aeronautics and an LU-decomposition application—prefetching can be very effective. It was easy to augment the applications to do prefetching and their performance was increased by 100–150% when we prefetched directly into the processor's cache. However, for applications with complex data usage patterns, prefetching was less successful. After much effort, the performance of a distributed-time logic simulation application that made extensive use of pointers and linked lists could be increased by only 30%. The paper also evaluates the effects of various hardware optimizations such as separate prefetch issue buffers, prefetching with exclusive ownership, lockup-free caches, and weaker memory consistency models on the performance of prefetching.