Cache Design Research Articles

POEM: Performance Optimization and Endurance Management for Non-volatile Caches

Non-volatile memories (NVMs) with their high storage density and ultra-low leakage power offer promising potential for redesigning the memory hierarchy in next-generation Multi-Processor Systems-on-Chip (MPSoCs). However, the adoption of NVMs in cache designs introduces challenges such as NVM write overheads and limited NVM endurance. The shared NVM cache in an MPSoC experiences requests from different processor cores and responses from the off-chip memory when the requested data is not present in the cache. Besides, upon evictions of dirty data from higher-level caches, the shared NVM cache experiences another source of write operations, known as writebacks . These sources of write operations: writebacks and responses, further exacerbate the contention for the shared bandwidth of the NVM cache, and create significant performance bottlenecks. Uncontrolled write operations can also affect the endurance of the NVM cache, posing a threat to cache lifetime and system reliability. Existing strategies often address either performance or cache endurance individually, leaving a gap for a holistic solution. This study introduces the Performance Optimization and Endurance Management (POEM) methodology, a novel approach that aggressively bypasses cache writebacks and responses to alleviate the NVM cache contention. Contrary to the existing bypass policies which do not pay adequate attention to the shared NVM cache contention, and focus too much on cache data reuse, POEM’s aggressive bypass significantly improves the overall system performance, even at the expense of data reuse. POEM also employs effective wear leveling to enhance the NVM cache endurance by careful redistribution of write operations across different cache lines. Across diverse workloads, POEM yields an average speedup of \(34\% \) over a naïve baseline and \(28.8\% \) over a state-of-the-art NVM cache bypass technique, while enhancing the cache endurance by \(15\% \) over the baseline. POEM also explores diverse design choices by exploiting a key policy parameter that assigns varying priorities to the two system-level objectives.

Enhancing GPU performance and energy efficiency: Innovative strategies for sustainable computing

With the rapid advancement of science and information technology, Graphics Processing Units (GPUs) have become indispensable tools in contemporary scientific research and industrial production. This paper delves into optimizing GPU performance and energy efficiency, with a specific focus on five aspects: core count and layout, memory hierarchy and cache design, clock speed, specialized hardware units for specific tasks, and interconnections among GPU components. Increasing core count generally enhances GPU performance but also elevates energy consumption. Optimizing core count and layout strikes a balance between performance and energy use. Memory hierarchy and cache design are crucial for handling the inherent parallelism of GPU architecture; optimizing these aspects boosts GPU efficiency and sustainability. Clock speed is a vital performance indicator, with the right speed achieving an optimal balance between performance and energy use. Tailoring hardware units for specific tasks enhances computational efficiency and lowers energy consumption. Optimizing interconnections between GPU components enhances data transfer efficiency, yielding higher performance. Through comprehensive research and optimization in these five areas, this paper introduces innovative strategies and techniques to boost GPU performance and reduce energy consumption, laying the foundation for sustainable computing development.

Joint Service Caching and Computing Resource Allocation for Edge Computing-Enabled Networks

In this paper, we consider the service caching and the computing resource allocation in edge computing (EC) enabled networks. We introduce a random service caching design considering multiple types of latency sensitive services and the base stations (BSs)’ service caching storage. We then derive a successful service probability (SSP). We also formulate a SSP maximization problem subject to the service caching distribution and the computing resource allocation. Then, we show that the optimization problem is nonconvex and develop a novel algorithm to obtain the stationary point of the SSP maximization problem by adopting the parallel successive convex approximation (SCA). Moreover, to further reduce the computational complexity, we also provide a low complex algorithm that can obtain the near-optimal solution of the SSP maximization problem in high computing capability region. Finally, from numerical simulations, we show that proposed solutions achieve higher SSP than baseline schemes. Moreover, we show that the near-optimal solution achieves reliable performance in the high computing capability region. We also explore the impacts of target delays, a BSs’ service cache size, and an EC servers’ computing capability on the SSP.

Random Caching Design for Multi-User Multi-Antenna HetNets With Interference Nulling

The strong interference suffered by users can be a severe problem in cache-enabled networks (CENs) due to the content-centric user association mechanism. To tackle this issue, multi-antenna technology may be employed for interference management. In this paper, we consider a user-centric interference nulling (IN) scheme in two-tier multi-user multi-antenna CEN, with a hybrid most-popular and random caching policy at macro base stations (MBSs) and small base stations (SBSs) to provide file diversity. All the interfering SBSs within the IN range of a user are requested to suppress the interference at this user using zero-forcing beamforming. Using stochastic geometry analysis techniques, we derive a tractable expression for the area spectral efficiency (ASE). A lower bound on the ASE is also obtained, with which we then consider ASE maximization, by optimizing the caching policy and IN coefficient. To solve the resultant mixed integer programming problem, we design an alternating optimization algorithm to minimize the lower bound of the ASE. Our numerical results demonstrate that the proposed caching policy yields performance that is close to the optimum, and it outperforms several existing baselines.

Dynamic resource management in integrated NOMA terrestrial–satellite networks using multi-agent reinforcement learning

The integration of terrestrial and satellite wireless communication networks offers a practical solution to enhance network coverage, connectivity, and cost-effectiveness. Moreover, in today’s interconnected world, connectivity’s reliable and widespread availability is increasingly important across various domains. This is especially more crucial for applications like the Internet of Things (IoT), remote sensing, disaster management, and bridging the digital divide. However, allocating the limited network resources efficiently and ensuring seamless handover between satellite and terrestrial networks present significant challenges. Therefore, this study introduces a resource allocation framework for integrated satellite–terrestrial networks to address these challenges. The framework leverages local cache pool deployments and non-orthogonal multiple access (NOMA) to reduce time delays and improve energy efficiency. Our proposed approach utilizes a multi-agent enabled deep deterministic policy gradient algorithm (MADDPG) to optimize user association, cache design, and transmission power control, resulting in enhanced energy efficiency. The approach comprises two phases: User Association and Power Control, where users are treated as agents, and Cache Optimization, where the satellite (Bs) is considered the agent. Through extensive simulations, we demonstrate that our approach surpasses conventional single-agent deep reinforcement learning algorithms in addressing cache design and resource allocation challenges in integrated terrestrial–satellite networks. Specifically, our proposed approach achieves significantly higher energy efficiency and reduced time delays compared to existing methods. This research highlights the importance and addresses the need for efficient resource allocation and cache design in integrated terrestrial–satellite networks, paving the way for enhanced connectivity and improved network performance in various applications.

Multi-Agent DRL for Resource Allocation and Cache Design in Terrestrial-Satellite Networks

Multi-Agent DRL for Resource Allocation and Cache Design in Terrestrial-Satellite Networks

TriCache: A User-Transparent Block Cache Enabling High-Performance Out-of-Core Processing with In-Memory Programs

Out-of-core systems rely on high-performance cache sub-systems to reduce the number of I/O operations. Although the page cache in modern operating systems enables transparent access to memory and storage devices, it suffers from efficiency and scalability issues on cache misses, forcing out-of-core systems to design and implement their own cache components, which is a non-trivial task. This study proposes TriCache, a cache mechanism that enables in-memory programs to efficiently process out-of-core datasets without requiring any code rewrite. It provides a virtual memory interface on top of the conventional block interface to simultaneously achieve user transparency and sufficient out-of-core performance. A multi-level block cache design is proposed to address the challenge of per-access address translations required by a memory interface. It can exploit spatial and temporal localities in memory or storage accesses to render storage-to-memory address translation and page-level concurrency control adequately efficient for the virtual memory interface. Our evaluation shows that in-memory systems operating on top of TriCache can outperform Linux OS page cache by more than one order of magnitude, and can deliver performance comparable to or even better than that of corresponding counterparts designed specifically for out-of-core scenarios.

Preventing Coherence State Side Channel Leaks Using TimeCache

Cache side channel attacks in the presence of shared memory have been used to extract cryptographic keys and enclave data, and are used by Spectre variants for leaking speculatively loaded data. Timing side channels exist in shared caches due to the difference in response latency of cached and uncached data. In prior work, we presented TimeCache, a cache design that prevents side channel exploits from reuse of shared memory. In this work, we extend TimeCache to also defend against attacks that exploit coherence states. TimeCache allows all running applications to use the entire cache, avoiding the need for partitioning in order to effect timing isolation. A per-process caching context prevents cache hits on data filled by another process. A novel bit-serial timestamp-parallel comparison logic allows low-overhead update of stale caching contexts. The defense is suited to all caches levels, and defends against an attacker running on any core. We evaluate TimeCache using the gem5 simulator to show that it is capable of preventing both reuse attacks and an attack based on coherence state leak. The average performance overhead for SPEC2006 is 1.13%, and for PARSEC and SPLASH is 0.46%.

Enterprise-Class Multilevel Cache Design: Low Latency, Huge Capacity, and High Reliability

The IBM Z computing platform is optimized for processing vast amounts of data and transactions with low latency in a highly virtualized and secured environment with sustained processor utilization of over 90%. The platform and its microprocessor chip are designed to deliver consistent system performance, throughput, and response times under all conditions. The innovative cache architecture of the IBM Telum Processor provides low latency, large capacity, and reliable L2 caches. Based on a novel horizontal cache persistence algorithm, these L2 caches also serves as system wide L3 and L4 caches delivering optimal enterprise application performance. When built into an IBM z16 system, these architectural features deliver 11% per-core performance improvement over the prior z15 hardware, running real-world enterprise applications.

Development and Procurement of a National Emergency Medical Team (EMT) Cache for Pacific Island Countries

Background/Introduction:Pacific Island Countries and Areas (PICs) represent some of the most logistically challenging locations, covering vast ocean territory and remote islands. Light, mobile clinical response capability is critical in the disaster-prone Pacific. Beginning in January 2021, WHO researched, tailored, and procured EMT cache “kits” specifically for Pacific Island contexts, based on the core standards of the global EMT initiative.Objectives:To research, tailor, and procure cache “kits” to ensure self-sufficiency and high-quality out-patient mobile medical care for national EMTs in PICs.Method/Description:WHO facilitated the development of national cache kits for 10 PICs EMTs. A need for specialized equipment and supplies or “cache” for team self-reliance is critical. Through a consultative process, including Pacific EMT leadership and team members, EMT mentors, and regional partners, WHO curated and procured cache kits for 10 PICs EMTs.Results/Outcomes:The Pacific EMT cache kit is designed for four-to-six-person teams with the capacity to deploy for a minimum of three days, with full self-sufficiency. Because of the complex and remote access to many Pacific Islands, EMT cache must be practical for transport on small aircrafts and maritime vessels. A consultative process resulted in a curated cache list for Pacific national EMTs of over 125 items, estimated to weigh approximately 440 kilograms per kit. By the end of 2022, a total 31 kits will be delivered to EMTs in ten countries.Conclusion:The design, development, and procurement of Pacific EMT cache for national response operations will allow for increased speed and agility for response to disasters and public health emergencies.

A State Transition Based Cache Side Channel Attack Model

Over the last decades, the Cache Side Channel Attack (CSCA) seriously threatens the security of user information, which repeatedly highlights the security of cache. Even though the secure cache design can mitigate or defend against this attack, the high design period and tap-out cost reduce the development of the specific designs. Therefore, many researchers focus on proposing a secure cache model to analyse the CSCA and simplify the secure design. Nowadays, the cache model can be divided into 1) Indicator analysis model; 2) Cache verify model; 3) Cache abstract model. However, most of these models focus on the establishment phase of CSCA, regardless of the usage phase of CSCA. On the other hand, the granularity of the cache model is either the entire cache or a single cache line, which can’t describe the cache behaviour in CSCA clearly. To overcome these limitations, we proposed a novel cache model based on state transition and gave a formal specification of CSCA by this model in this work. Firstly, we summarize 11 basic states of the cache set and point to the target state in the process of CSCA. Then we describe the CSCA from three aspects: pre-attack cost, state transition cost, and detection accuracy, which gives a more comprehensive analysis of CSCA.

Rethinking the Interactivity of OS and Device Layers in Memory Management

In the big data era, a huge number of services has placed a fast-growing demand on the capacity of DRAM-based main memory. However, due to the high hardware cost and serious leakage power/energy consumption, the growth rate of DRAM capacity cannot meet the increased rate of the required main memory space when the energy or hardware cost is a critical concern. To tackle this issue, hybrid main-memory devices/modules have been proposed to replace the pure DRAM main memory with a hybrid main memory module that provides a large main memory space by integrating a small-sized DRAM and a large-sized non-volatile memory (NVM) into the same memory module. Although NVMs have high-density and low-cost features, they suffer from the low read/write performance and low endurance issue, compared to DRAM. Thus, inside the hybrid main-memory module, it also includes a memory management design to use DRAM as the cache of NVMs to enhance its performance and lifetime. However, it also introduces new design challenges in both the OS and the memory module. In this work, we rethink the interactivity of OS and hybrid main-memory module, and propose a cross-layer cache design that (1) utilizes the information from the operating system to optimize the hit ratio of the DRAM cache inside the memory module, and (2) takes advantage of the bulk-size (or block-based) read/write feature of NVM to minimize the time overhead on the data movement between DRAM and NVM. At the same time, this cross-layer cache design is very lightweight and only introduces limited runtime management overheads. A series of experiments was conducted to evaluate the effectiveness of the proposed cross-layer cache design. The results show that the proposed design could improve access performance for up to 88%, compared to the investigated well-known page replacement algorithms.

POCache: Toward robust and configurable straggler tolerance with parity-only caching

Stragglers (i.e., nodes with slow performance) are prevalent and incur performance instability in large-scale storage systems, yet it is challenging to detect stragglers in practice. We make a case by showing how erasure-coded caching provides robust straggler tolerance without relying on timely and accurate straggler detection, while incurring limited redundancy overhead in caching. We first analytically motivate that caching only parity blocks can achieve effective straggler tolerance. To this end, we present POCache, a parity-only caching design that provides robust straggler tolerance. To limit the erasure coding overhead, POCache slices blocks into smaller subblocks and parallelizes the coding operations at the subblock level. It further adopts a configurable straggler-aware cache algorithm (CSAC) that takes into account both file access popularity and straggler estimation to decide which parity blocks should be cached. CSAC enables POCache to configure various cache admission and eviction algorithms with straggler awareness and supports cache prefetching. We implement a POCache prototype atop Hadoop 3.1 HDFS, while preserving the performance and functionalities of normal HDFS operations. Extensive experiments on both local and Amazon EC2 clusters show that in the presence of stragglers, POCache can reduce the read latency by up to 87.9% compared to vanilla HDFS.

A Quantitative Study of Locality in GPU Caches for Memory-Divergent Workloads

GPUs are capable of delivering peak performance in TFLOPs, however, peak performance is often difficult to achieve due to several performance bottlenecks. Memory divergence is one such performance bottleneck that makes it harder to exploit locality, cause cache thrashing, and high miss rate, therefore, impeding GPU performance. As data locality is crucial for performance, there have been several efforts to exploit data locality in GPUs. However, there is a lack of quantitative analysis of data locality, which could pave the way for optimizations. In this paper, we quantitatively study the data locality and its limits in GPUs at different granularities. We show that, in contrast to previous studies, there is a significantly higher inter-warp locality at the L1 data cache for memory-divergent workloads. We further show that about 50% of the cache capacity and other scarce resources such as NoC bandwidth are wasted due to data over-fetch caused by memory divergence. While the low spatial utilization of cache lines justifies the sectored-cache design to only fetch those sectors of a cache line that are needed during a request, our limit study reveals the lost spatial locality for which additional memory requests are needed to fetch the other sectors of the same cache line. The lost spatial locality presents opportunities for further optimizing the cache design.

Taming Process Variations in CNFET for Efficient Last-Level Cache Design

Carbon nanotube field-effect transistors (CNFET) emerge as a promising alternative to CMOS transistors for the much higher speed and energy efficiency, which makes the technology particularly suitable for building the energy-hungry last level cache (LLC). However, the process variations (PVs) in CNFET caused by the imperfect fabrication lead to large timing variation and the worst-case timing dramatically limits the LLC operation speed. Particularly, we observe that the CNFET-based cache latency distribution is closely related to the LLC layouts. For the two typical LLC layouts that have the CNT growth direction aligned to the cache way direction and cache set direction respectively, we proposed variation-aware set aligned (VASA) cache and variation-aware way aligned (VAWA) cache in combination with corresponding cache optimizations such as data shuffling and page mapping to enable low-latency cache for frequently used data. According to our experiments, the optimized LLC reduces the average access latency by 32% and 45% compared to the baseline designs on the two different CNFET layouts respectively while it improves the overall performance by 6\% and 9\% and reduces the energy consumption by 4% and 8% respectively. In addition, with both the architecture induced latency variation and PV incurred latency variation considered in a unified model, we extended the VAWA and VASA cache design for the CNFET-based NUCA and the proposed NUCA achieves both significant performance improvement and energy saving compared to the straightforward variation-aware NUCA.

Rubik: A Hierarchical Architecture for Efficient Graph Neural Network Training

The graph convolutional network (GCN) emerges as a promising direction to learn the inductive representation in graph data commonly used in widespread applications, such as E-commerce, social networks, and knowledge graphs. However, learning from graphs is nontrivial because of its mixed computation model involving both graph analytics and neural network computing. To this end, we decompose the GCN learning into two hierarchical paradigms: 1) graph-level and 2) node-level computing. Such a hierarchical paradigm facilitates the software and hardware accelerations for GCN learning. We propose a lightweight graph reordering methodology, incorporated with a GCN accelerator architecture that equips a customized cache design to fully utilize the graph-level data reuse. We also propose a mapping methodology aware of data reuse and task-level parallelism to handle various graphs inputs effectively. The results show that Rubik accelerator design improves energy efficiency by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$26.3\times $ </tex-math></inline-formula> – <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1375.2\times $ </tex-math></inline-formula> than GPU platforms across different datasets and GCN models.

CARL: Compiler Assigned Reference Leasing

Data movement is a common performance bottleneck, and its chief remedy is caching. Traditional cache management is transparent to the workload: data that should be kept in cache are determined by the recency information only, while the program information, i.e., future data reuses, is not communicated to the cache. This has changed in a new cache design named Lease Cache . The program control is passed to the lease cache by a compiler technique called Compiler Assigned Reference Lease (CARL). This technique collects the reuse interval distribution for each reference and uses it to compute and assign the lease value to each reference. In this article, we prove that CARL is optimal under certain statistical assumptions. Based on this optimality, we prove miss curve convexity, which is useful for optimizing shared cache, and sub-partitioning monotonicity, which simplifies lease compilation. We evaluate the potential using scientific kernels from PolyBench and show that compiler insertions of up to 34 leases in program code achieve similar or better cache utilization (in variable size cache) than the optimal fixed-size caching policy, which has been unattainable with automatic caching but now within the potential of cache programming for all tested programs and most cache sizes.

Cache Design and Optimization Techniques

Cache Design and Optimization Techniques

Beyond Edge Caching: Freshness and Popularity Aware IoT Data Caching via NDN at Internet-Scale

In-network caching is one of the main pillars of the Named Data Networking (NDN) paradigm, where every Internet router, in the path between data sources and consumers, can cache incoming content packets. Multiple strategies have been designed for caching Internet of Things (IoT) data streamed by resource-constrained devices in edge domains and wireless sensor networks, while the benefits of IoT data caching at Internet-scale, including both edge and core network segments, have not been fully disclosed. In this work, we propose and analyse a novel probabilistic Internet-scale caching design for IoT data, which jointly accounts for the content popularity and lifetime. In the considered scenario, IoT contents are requested by remote consumers and delivered by crossing multiple edge and core network segments of the NDN-based future Internet. The proposal is composed of two distinct reactive caching strategies, a coordinated and an autonomous one, to be implemented in the edge and core domain, respectively. Achieved results show that the proposal outperforms state-of-the-art solutions by providing, among others, the highest cache hit ratio and the shortest number of hops. Such performances testify a lower pressure on energy-constrained devices and on the network infrastructure, overall contributing to the sustainability of the IoT ecosystem.

Thwarting Cache Side Channel Attacks with Encryption Address-Based Set Balance Cache

The deterministic memory-to-cache mapping used by cache side channel attack causes the leakage of sensitive information such as secret keys, which seriously threatens user security and highlights the need to defend against this kind of attack. Recent table-based secure cache design requires more space to store the entry while a purely encryption-based design needs complex encryption units to ensure robustness. What is more, the newer attack algorithm enabling faster eviction set discovery may still break such defenses. Even though increasing the association of cache can be a potential solution, it introduces too much redundancy cache access and storage overhead. In this paper, we eliminate this problem. We present Encryption address-based Set Balance Cache (ESBC), a novel cache design to mitigate cache-based side channel attack. ESBC encrypts an address into two-level sets and displaces the data from a primary set into the secondary set when the primary set is full. The two-level mapping structure increases the complexity for attackers to build eviction sets, which is a vital step for the conflict-based attack. Furthermore, we adopt two different optimized variants from a temporal perspective, ESBC with remapping (ESBCR), which serves dynamic-remapping, and spatial perspective, ESBC with multi-mapping (ESBCM), which performs multi-mapping to improve the robustness. Our security analysis reveals that these designs can confuse the exploitation of conflicting addresses. Simulation-based evaluation on SPEC2017 shows 0.24% instruction per cycle (IPC) degradation for ESBCR with 1% remap rate. While the reduction of ESBCM that has five potential secondary sets is 0.69%. What is exciting is that both miss per kilo instructions (MPKI) and miss rates are even reduced because of the efficient cache usage. The storage overhead of these two variants is only 0.87%. By comparing the two schemes, we can observe that though ESBCR brings less performance overhead, the ESBCM has better scalability.