Chapter 6 - Efficiently Using GPU Memory

Abstract

This chapter discusses techniques and examples to efficiently use GPU memory. The importance of efficiently using GPU memory cannot be overstated. With roughly three-orders-of-magnitude difference in speed between the fastest on-chip register memory and mapped host memory that must traverse the PCIe bus, literate CUDA developers must understand the most efficient ways to use memory. Latency hiding through ILP or TLP is essential to application performance. Pre-fetching can keep more memory transactions in flight to move data to fast memory and speed even memory bandwidth- limited reduction operations. Irregular data structures are a challenge with current GPU technology, but some techniques can preserve performance even with random memory accesses. The three-orders-of-magnitude performance difference between the slowest and fastest GPU memory systems means that GPU programmers have the opportunity to capitalize on the extreme performance that GPU hardware offers. What makes CUDA so special is that it exposes the features of the underlying hardware so that the full potential of the hardware can be realized. It is possible to delve down into the lowest levels of the hardware execution model to attain high performance. Generic programming lets CUDA programmers create simple, generic methods that fully exploit the capability of the GPU. Finding more and better ways to utilize GPU memory is an area of active research as new libraries become available that support irregular data structures such as graphs and sparse matrices.