Virtual Cache Line: A New Technique to Improve Cache Exploitation for Recursive Data Structures

Abstract

Recursive data structures (lists, trees, graphs, etc.) are used throughout scientific and commercial software. The common approach is to allocate storage to the individual nodes of such structures dynamically, maintaining the logical connection between them via pointers. Once such a data structure goes through a sequence of updates (inserts and deletes), it may get scattered all over memory yielding poor spatial locality, which in turn introduces many cache misses. In this paper we present the new concept of Virtual Cache Lines (VCLs). Basically, the mechanism keeps groups of consecutive nodes in close proximity, forming virtual cache lines, while allowing the groups to be stored arbitrarily far away from each other. Virtual cache lines increase the spatial locality of the given data structure resulting in better locality of references. Furthermore, since the spatial locality is improved, software prefetching becomes much more attractive. Indeed, we also present a software prefetching algorithm that can be used when dealing with VCLs resulting in even higher data cache performance. Our results show that the average performance of linked list operations, like scan, insert, and delete can be improved by more than 200% even in architectures that do not support prefetching, like the Intel Pentium. Moreover, when using prefetching one can gain additional 100% improvement. We believe that given a program that manipulates certain recursive data structures, compilers will be able to generate VCL-based code. Also, until this vision becomes true, VCLs can be used to build more efficient user libraries, operating-systems and applications programs.