Last-level Cache Research Articles

COPE

Prefetching helps in reducing the memory access latency in multi-banked NUCA architecture, where the Last Level Cache (LLC) is shared. In such systems, an application running on core generates significant traffic on the shared resources, the underlying network and LLC. While prefetching helps to increase application performance, but an inaccurate prefetcher can cause harm by generating unwanted traffic that additionally increases network and LLC contention. Increased network contention results in untimely prefetching of cache blocks, thereby reducing the effectiveness of a prefetcher. Prefetch accuracy is extensively used to reduce unwanted prefetches that can mitigate the prefetcher caused contention. However, the conventional prefetch accuracy parameter has major limitations in NUCA architectures. The article exposes that prefetch accuracy can create two major false-positive cases of prefetching, Under-estimation and Over-estimation problems, and false feedback loop that can mislead a prefetcher in generating more unwanted traffic. We propose a novel technique, Coordinated Prefetching for Efficient (COPE), which addresses these issues by redefining prefetch accuracy for such architectures and identifies additional parameters that can avoid generating unwanted prefetch requests. Experiment conducted using PARSEC benchmark on a 64-core system shows that COPE achieve 3% reduction in L1 cache miss rate, 12.64% improvement in IPC, 23.2% reduction in average packet latency and 18.56% reduction in dynamic power consumption of the underlying network.

Performance-Energy Trade-off in Modern CMPs

Chip multiprocessors (CMPs) are ubiquitous in all computing systems ranging from high-end servers to mobile devices. In these systems, energy consumption is a critical design constraint as it constitutes the most significant operating cost for computing clouds. Analogous to this, longer battery life continues to be an essential user concern in mobile devices. To optimize on power consumption, modern processors are designed with Dynamic Voltage and Frequency Scaling (DVFS) support at the individual core as well as the uncore level. This allows fine-grained control of performance and energy. For an n core processor with m core and uncore frequency choices, the total DVFS configuration space is now m (n+1) (with the uncore accounting for the + 1). In addition to that, in CMPs, the performance-energy trade-off due to core/uncore frequency scaling concerning a single application cannot be determined independently as cores share critical resources like the last level cache (LLC) and the memory. Thus, unlike the uni-processor environment, the energy consumption of an application running on a CMP depends not only on its characteristics but also on those of its co-runners (applications running on other cores). The key objective of our work is to select a suitable core and uncore frequency that minimizes power consumption while limiting application performance degradation within certain pre-defined limits (can be termed as QoS requirements). The key contribution of our work is a learning-based model that is able to capture the interference due to shared cache, bus bandwidth, and memory bandwidth between applications running on multiple cores and predict near-optimal frequencies for core and uncore.

Polymorphic Memory: A Hybrid Approach for Utilizing On-Chip Memory in Manycore Systems

The key challenges of manycore systems are the large amount of memory and high bandwidth required to run many applications. Three-dimesnional integrated on-chip memory is a promising candidate for addressing these challenges. The advent of on-chip memory has provided new opportunities to rethink traditional memory hierarchies and their management. In this study, we propose a polymorphic memory as a hybrid approach when using on-chip memory. In contrast to previous studies, we use the on-chip memory as both a main memory (called M1 memory) and a Dynamic Random Access Memory (DRAM) cache (called M2 cache). The main memory consists of M1 memory and a conventional DRAM memory called M2 memory. To achieve high performance when running many applications on this memory architecture, we propose management techniques for the main memory with M1 and M2 memories and for polymorphic memory with dynamic memory allocations for many applications in a manycore system. The first technique is to move frequently accessed pages to M1 memory via hardware monitoring in a memory controller. The second is M1 memory partitioning to mitigate contention problems among many processes. Finally, we propose a method to use M2 cache between a conventional last-level cache and M2 memory, and we determine the best cache size for improving the performance with polymorphic memory. The proposed schemes are evaluated with the SPEC CPU2006 benchmark, and the experimental results show that the proposed approaches can improve the performance under various workloads of the benchmark. The performance evaluation confirms that the average performance improvement of polymorphic memory is 21.7%, with 0.026 standard deviation for the normalized results, compared to the previous method of using on-chip memory as a last-level cache.

DTaPO: Dynamic Thermal-Aware Performance Optimization for Dark Silicon Many-Core Systems

Future many-core systems need to handle high power density and chip temperature effectively. Some cores in many-core systems need to be turned off or ‘dark’ to manage chip power and thermal density. This phenomenon is also known as the dark silicon problem. This problem prevents many-core systems from utilizing and gaining improved performance from a large number of processing cores. This paper presents a dynamic thermal-aware performance optimization of dark silicon many-core systems (DTaPO) technique for optimizing dark silicon a many-core system performance under temperature constraint. The proposed technique utilizes both task migration and dynamic voltage frequency scaling (DVFS) for optimizing the performance of a many-core system while keeping system temperature in a safe operating limit. Task migration puts hot cores in low-power states and moves tasks to cooler dark cores to aggressively reduce chip temperature while maintaining high overall system performance. To reduce task migration overhead due to cold start, the source core (i.e., active core) keeps its L2 cache content during the initial migration phase. The destination core (i.e., dark core) can access it to reduce the impact of cold start misses. Moreover, the proposed technique limits tasks migration among cores that share the last level cache (LLC). In the case of major thermal violation and no cooler cores being available, DVFS is used to reduce the hot cores temperature gradually by reducing their frequency. Experimental results for different threshold temperatures show that DTaPO can keep the average system temperature below the thermal limit. Affirmatively, the execution time penalty is reduced by up to 18% compared with using only DVFS for all thermal thresholds. Moreover, the average peak temperature is reduced by up to 10.8°C. In addition, the experimental results show that DTaPO improves the system’s performance by up to 80% compared to optimal sprinting patterns (OSP) and reduces the temperature by up to 13.6°C.

MemSZ

This article describes Memory Squeeze (MemSZ), a new approach for lossy general-purpose memory compression. MemSZ introduces a low latency, parallel design of the Squeeze (SZ) algorithm offering aggressive compression ratios, up to 16:1 in our implementation. Our compressor is placed between the memory controller and the cache hierarchy of a processor to reduce the memory traffic of applications that tolerate approximations in parts of their data. Thereby, the available off-chip bandwidth is utilized more efficiently improving system performance and energy efficiency. Two alternative multi-core variants of the MemSZ system are described. The first variant has a shared last-level cache (LLC) on the processor-die, which is modified to store both compressed and uncompressed data. The second has a 3D-stacked DRAM cache with larger cache lines that match the granularity of the compressed memory blocks and stores only uncompressed data. For applications that tolerate aggressive approximation in large fractions of their data, MemSZ reduces baseline memory traffic by up to 81%, execution time by up to 62%, and energy costs by up to 25% introducing up to 1.8% error to the application output. Compared to the current state-of-the-art lossy memory compression design, MemSZ improves the execution time, energy, and memory traffic by up to 15%, 9%, and 64%, respectively.

Phase-Aware Cache Partitioning to Target Both Turnaround Time and System Performance

The Last Level Cache (LLC) plays a key role in the system performance of current multi-cores by reducing the number of long latency main memory accesses. The inter-application interference at this shared resource, however, can lead the system to undesired situations regarding performance and fairness. Recent approaches have successfully addressed fairness and turnaround time (TT) in commercial processors. Nevertheless, these approaches must face sustaining system performance, which is challenging. This work makes two main contributions. LLC behaviors regarding cache performance, data reuse and cache occupancy, that adversely impact on the final performance are identified. Second, based on these behaviors, we propose the Critical-Phase Aware Partitioning Approach (CPA), which reduces TT while sustaining (and even improving) IPC by making an effective use of the LLC space. Experimental results show that CPA outperforms CA, Dunn and KPart state-of-the-art approaches, and improves TT (over 40 percent in some workloads) over Linux default behavior while sustaining or even improving IPC by more than 3 percent in several mixes.

OPTIMUS: A Security-Centric Dynamic Hardware Partitioning Scheme for Processors that Prevent Microarchitecture State Attacks

Hardware virtualization allows multiple security-critical and ordinary (insecure) processes to co-execute on a processor. These processes temporally share hardware resources and endure numerous security threats on the microarchitecture state. State-of-the-art secure processor architectures, such as MI6 and IRONHIDE enable capabilities to execute security-critical processes in hardware isolated enclaves utilizing the strong isolation security primitive. The MI6 processor purges small state resources on each enclave entry/exit and statically partitions the last-level cache and DRAM regions to ensure strong isolation. IRONHIDE takes a spatial approach and creates two isolated clusters of cores in a multicore processor to ensure strong isolation for processes executing in the enclave cluster. Both architectures observe performance degradation due to static partitioning of shared hardware resources. OPTIMUS proposes a security-centric dynamic hardware resource partitioning scheme that operates entirely at runtime and ensures strong isolation. It enables deterministic resource allocations at the application level granularity, and limits the number of hardware reconfigurations to ensure bounded information leakage via the timing and termination channels. The dynamic hardware resource partitioning capability of OPTIMUS is shown to co-optimize performance and security for the MI6 and IRONHIDE architectures.

Improving power-performance via hybrid cache for chip many cores based on neural network prediction technique

Recently, the increasing need to run applications for significant data analytics, and the augmented demand of useful tools for big data computing systems has resulted in a cumulative necessity for efficient platforms with high performance and realizable power consumption, for example, chip multiprocessors (CMPs). Correspondingly, due to the demand for features like shrinkable sizes, and the concurrent need to pack increasing numbers of transistors into a single chip, has led to serious design challenges, consuming a significant of power within high area densities. We present a reconfigurable hybrid cache system for last level cache (LLC) by the integration of emerging designs, such as STT-RAM with SRAM memories. This approach consists of two phases: off- time and on-time. In off time, training NN is implemented while in the on-time phase, a reconfiguration cache uses a neural network (NN) learning approach to predict demanded latency of the running application. Experimental results of a three-dimensional chip with 64 cores show that the suggested design under PARSEC benchmarks provides a speedup in terms of the performance at 25% and improves energy consumption by 78.4% in comparison to non-reconfigurable pure SRAM cache architectures.

Optimization of Intercache Traffic Entanglement in Tagless Caches With Tiling Opportunities

So-called “tagless” caches have become common as a means to deal with the vast L4 last-level caches (LLCs) enabled by increasing device density, emerging memory technologies, and advanced integration capabilities (e.g., 3-D). Tagless schemes often result in intercache entanglement between tagless cache (L4) and the cache (L3) stewarding its metadata. We explore new cache organization policies that mitigate overheads stemming from the intercache-level replacement entanglement. We incorporate support for explicit tiling shapes that can better match software access patterns to improve the spatial and temporal locality of large block allocations in many essential computational kernels. To address entanglement overheads and pathologies, we propose new replacement policies and energy-friendly mechanisms for tagless LLCs, such as restricted block caching (RBC) and victim tag buffer caching (VBC) to incorporate L4 eviction costs into L3 replacement decisions efficiently. We evaluate our schemes on a range of linear algebra kernels that are software tiled. RBC and VBC demonstrate a reduction in memory traffic of 83/4.4/67% and 69/35.5/76% for 8/32/64 MB L4s, respectively. Besides, RBC and VBC provide speedups of 16/0.3/0.6% and 15.7/1.8/0.8%, respectively, for systems with 8/32/64 MB L4, over a tagless cache with an LRU policy in the L3. We also show that matching the shape of the hardware allocation for each tagless region superblocks to the access order of the software tile improves latency by 13.4% over the baseline tagless cache with reductions in memory traffic of 51% over linear superblocks.

Exploring Time-Predictable and High-Performance Last-Level Caches for Hard Real-Time Integrated CPU-GPU Processors

Time predictability is crucial for hard real-time and safety-critical systems. In an integrated CPU-GPU (graphic processing units) architecture, the shared last-level cache (LLC) can cause a large number of interferences between CPU and GPU LLC accesses with diverse patterns and characteristics, which can significantly impact the performance and time predictability of both CPUs and GPUs. In this paper, we explore cache partitioning, locking, and a combination of them to make the LLC time-predictable for integrated CPU-GPUs while achieving high performance. By evaluating these LLC management approaches, we can provide real-time system developers recommendations on the most effective time-predictable LLC designs for heterogeneous CPU-GPU multicore processors.

SDRP: Safe, Efficient, and SLO-Aware Workload Consolidation Through Secure and Dynamic Resource Partitioning

Workload consolidation is a widely-used technique to improve the resource utilization of services computing systems by consolidating latency-critical (LC) and batch workloads on the same physical server. The resource manager for workload consolidation dynamically allocates hardware resources (e.g., cores, caches) to the workloads to maximize the resource utilization while satisfying the service-level objective (SLO) of the LC workloads. Since security-critical hardware resources are dynamically allocated across consolidated workloads, information leakages can be created among workloads through microarchitectural side-channel (SC) attacks. Despite extensive prior works, it is yet to investigate efficient system software support for achieving high resource utilization without compromising the SLO and security of consolidated workloads. To bridge this gap, we propose SDRP, secure and dynamic resource partitioning for safe, efficient, and SLO-aware workload consolidation. As with the state-of-the-art techniques, SDRP dynamically allocates hardware resources to enhance the resource utilization and provide the SLO guarantees. In contrast to the state-of-the-art techniques, SDRP dynamically sanitizes security-critical hardware resources to robustly defeat microarchitectural SC attacks. Our quantitative evaluation demonstrates that SDRP achieves high resource sanitization quality, introduces low performance overheads, delivers high resource utilization with the SLO and security guarantees, and defeats the last-level cache (LLC)-based SC attack.

Efficient Cache Resizing policy for DRAM-based LLCs in ChipMultiprocessors

In today’s ChipMultiprocessors (CMPs), multiple cores share the common Last Level Cache (LLC), divided into multiple banks. As the data requirement is increasing the demand for larger LLC sizes is also increasing. The traditional SRAM technology is not area efficient to design such larger LLCs as demanded by the modern CMPs. From the last few years, DRAM technologies have been used to propose LLC. DRAM technology has almost 8 times density over the SRAM and hence larger cache size can be designed. Though DRAM is already considered as an alternative to design low cost, area-efficient larger size LLC, it must be used efficiently to get the benefits. Due to its overheads like access latency and refresh operations efficient techniques must be used to get better performance from DRAM LLC. In the existing works, it has been observed that though the larger LLC is required for the current as well as future data-intensive applications, the entire LLC may not be required while executing other applications. In such situations, some banks can be almost idle during a particular period of execution. These idle banks can be powered-off and restart later whenever required. The mechanism is called Cache Resizing as it resizes the cache (LLC) according to the current requirements. Cache resizing techniques are already proposed for SRAM based LLCs but due to the larger size of DRAM LLC, the same mechanisms cannot be used for DRAM LLCs. In this paper, we have proposed an efficient cache resizing policy for large sized LLC, especially for DRAM-based LLCs. We call our proposed cache resizing technique as Efficient Cache Resizing (ECR) which is implemented on top of a 3D Tiled CMP. Experimental analysis shows that ECR can reduce up to 44% more energy consumption as compared to the existing technique.

J SWof 5.5 MA/cm2 and RA of 5.2-Ω · μm2 STT-MRAM Technology for LLC Application

Due to the complexity of device processing, the trade-off between yield and area has resulted in diminishing rate of scaling for the high-density static random access memory (SRAM) cell at advanced CMOS nodes. An introduction of extreme ultraviolet (EUV) and multipatterning has added additional cost to technology in order to realize 3-D device structure and ultrascaled metal routing. In this era, spin-transfer torque (STT)-MRAM technology can provide an alternative to high-density SRAM and for the last level cache (LLC) applications. In this article, we discuss the memory design and technology tradeoff to enable the STT-MRAM as a viable option. We have realized the technology over 300-mm wafer, measuring 1 million samples to build a SPICE model for circuit simulation. Occupying up to 83.3% of an area that of SRAM macro has been designed and simulated for the scaled 5-nm CMOS node. The simulated MRAM macro shows the best read and write access time of 3.1 and 6.2 ns, respectively. Magnetic tunneling junction (MTJ) pillar of 38-nm diameter is realized at 90-nm pitch, measuring resistance area (RA) of 5.2- $\Omega \cdot \mu \text{m}^{2}$ , ${J}_{sw}$ of 5.5 MA/cm2 with improved $\Delta $ avg of 70, and breakdown voltage of 0.99 V. The energy comparison shows increasing gains versus SRAM for the increasing cache sizes crossing over at of 0.3 and 4 MB for the single-cycle read and write operations, respectively.

TSE: Two-Step Elimination for MLC STT-RAM Last-Level Cache

Spin-transfer torque RAM (STT-RAM) is an emerging non-volatile memory that has been recognized as the potential candidate to replace SRAM. Compared with SRAM, STT-RAM has advantages of non-volatility, zero leakage power, and higher density. To further improve data density, multi-level cell (MLC) STT-RAM that can store two bits per cell has been proposed. However, writing hard bit of a cell would write its soft bit to the same value as well, which complicates the write operation of MLC STT-RAM. Although two-step transition (TT) is usually adopted to ensure the data correctness during a write operation, it incurs overhead of additional energy consumption and performance degradation. In this article, we propose the two-step elimination (TSE) scheme to eliminate TTs while ensure data integrity. By flipping hard bits of the cells that suffered from TTs, the TSE scheme could reduce TTs to soft transitions (STs) or zero transitions (ZTs), which incur much less overhead than TTs. To keep track of the flipped cells effectively, 6-bit TSE tag is introduced. We exploit tag reversing and advanced mode of the TSE scheme to further improve the performance. The experimental results showed that our scheme could reduce 61 percent TTs and achieve significant lifetime improvement, compared with conventional MLC STT-RAM (CMLC) scheme.

Mitigating Process Variability for Non-Volatile Cache Resilience and Yield

While inclusion of emerging technology-based Non-Volatile Memory (NVM) devices in on-chip memory subsystems offers excellent potential for energy savings and scalability, their sensing vulnerability creates Process Variation (PV) challenges. This paper presents a circuit-architecture cross-layer solution to realize a radically-different approach to leveraging as-built variations via specific Sense Amplifier (SA) design and use. This novel approach, referred to as a Self-Organized Sub-bank (SOS) design, assigns the preferred SA to each Sub-Bank (SB) based on a PV assessment, resulting in energy consumption reduction and increased read access reliability. To improve the PV immunity of SAs, two reliable and power efficient SAs, called the Merged SA (MSA) and the Adaptive SA (ASA) are introduced herein for use in the SOS scheme. Furthermore, we propose a dynamic PV and energy-aware cache block migration policy that utilizes mixed SRAM and STT-MRAM banks in Last Level Cache (LLC) to maximize the SOS bandwidth. Our experimental results indicate that SOS can alleviate the sensing vulnerability by 89 percent on average, which significantly reduces the risk of application contamination by fault propagation. Furthermore, in the light of the proposed block migration policy, write performance is improved by 12.4 percent on average compared to the STT-MRAM-only design.

Detecting Hardware-Assisted Virtualization With Inconspicuous Features

Recent years have witnessed the proliferation of the deployment of virtualization techniques. Virtualization is designed to be transparent, that is, unprivileged users should not be able to detect whether a system is virtualized. Such detection can result in serious security threats such as evading virtual machine (VM)-based malware dynamic analysis and exploiting vulnerabilities for cross-VM attacks. The traditional software-based virtualization leaves numerous artifacts/fingerprints, which can be exploited without much effort to detect the virtualization. In contrast, current mainstream hardware-assisted virtualization significantly enhances the virtualization transparency, making itself more transparent and difficult to be detected. Nonetheless, we showcase three new identified low-level inconspicuous features, which can be leveraged by an unprivileged adversary to effectively and stealthily detect the hardware-assisted virtualization. All three features come from the chipset fingerprints, rather than the traces of software-based virtualization implementations (e.g., Xen or KVM). The identified features include i) Translation-Lookaside Buffer (TLB) stores an extra layer of address translations; ii) Last-Level Cache (LLC) caches one more layer of page-table entries; and iii) Level-1 Data (L1D) Cache is unstable. Based on the above features, we develop three corresponding virtualization detection techniques, which are then comprehensively evaluated on three native environments and three popular cloud providers: i) Amazon Elastic Compute Cloud, ii) Google Compute Engine and iii) Microsoft Azure. Experimental results validate that these three adversarial detection techniques are effective (with no false positive) and stealthy (without triggering suspicious system events, e.g., VM-exit ) in detecting the above commodity virtualized environments.

Addressing variability in reuse prediction for last-level caches

Last-Level Cache (LLC) represents the bulk of a modern CPU processor's transistor budget and is essential for application performance as LLC enables fast access to data in contrast to much slower main memory. However, applications with large working set size often exhibit streaming and/or thrashing access patterns at LLC. As a result, a large fraction of the LLC capacity is occupied by dead blocks that will not be referenced again, leading to inefficient utilization of the LLC capacity. To improve cache efficiency, the state-of-the-art cache management techniques employ prediction mechanisms that learn from the past access patterns with an aim to accurately identify as many dead blocks as possible. Once identified, dead blocks are evicted from LLC to make space for potentially high reuse cache blocks. In this thesis, we identify variability in the reuse behavior of cache blocks as the key limiting factor in maximizing cache efficiency for state-of-the-art predictive techniques. Variability in reuse prediction is inevitable due to numerous factors that are outside the control of LLC. The sources of variability include control-flow variation, speculative execution and contention from cores sharing the cache, among others. Variability in reuse prediction challenges existing techniques in reliably identifying the end of a block's useful lifetime, thus causing lower prediction accuracy, coverage, or both. To address this challenge, this thesis aims to design robust cache management mechanisms and policies for LLC in the face of variability in reuse prediction to minimize cache misses, while keeping the cost and complexity of the hardware implementation low. To that end, we propose two cache management techniques, one domain-agnostic and one domain-specialized, to improve cache efficiency by addressing variability in reuse prediction.

Coordinated management of DVFS and cache partitioning under QoS constraints to save energy in multi-core systems

Reducing the energy expended to carry out a computational task is important. In this work, we explore the prospects of meeting Quality-of-Service requirements of tasks on a multi-core system while adjusting resources to expend a minimum of energy. This paper considers, for the first time, a QoS-driven coordinated resource management algorithm (RMA) that dynamically adjusts the size of the per-core last-level cache partitions and the per-core voltage–frequency settings to save energy while respecting QoS requirements of every application in multi-programmed workloads run on multi-core systems. It does so by doing configuration-space exploration across the spectrum of LLC partition sizes and Dynamic Voltage–Frequency Scaling (DVFS) settings at runtime at negligible overhead. We show that the energy of 4-core and 8-core systems can be reduced by up to 18% and 14%, respectively, compared to a baseline with even distribution of cache resources and a fixed mid-range core voltage–frequency setting. The energy savings can potentially reach 29% if the QoS targets are relaxed to 40% longer execution time.

Page Reusability-Based Cache Partitioning for Multi-Core Systems

Most modern multi-core processors provide a shared last level cache (LLC) where data from all cores are placed to improve performance. However, this opens a new challenge for cache management, owing to cache pollution. With cache pollution, data with weak temporal locality can evict other data with strong temporal locality when both are mapped into the same cache set. In this article, we propose page reusability-based cache partitioning (PRCP) for multi-core systems to maximize cache utilization by minimizing cache pollution. To achieve this, PRCP divides pages into two groups: (1) highly-reused pages and (2) lowly-reused pages. The reusability of each page is collected online via periodic page table scans. PRCP then dynamically partitions the shared cache into two corresponding areas using page coloring technique. We have implemented PRCP in Linux kernel and evaluated it using SPEC CPU2006 benchmarks. The results show that our scheme can achieve comparable performance to the optimal offline MRC-guided process-based cache partitioning scheme without a priori knowledge of workloads.

Write Back Energy Optimization for STT-MRAM-based Last-level Cache with Data Pattern Characterization

Traditional memory technologies face severe challenges in meeting the ever-increasing power and memory bandwidth requirements for high-performance computing and big-data analyses. Several emerging memory technologies are promising as the replacements of SRAM or DRAM. Among them, STT-MRAM can be used to replace SRAM as the last-level cache (LLC). However, it suffers from high write energy and latency. In this article, we investigate data patterns written from SRAM-based upper-level cache to STT-MRAM-based LLC to explore the write energy reduction potential. Depending on the data layout within a cache line, redundant bits can be identified and eliminated from write back operations to save STT-MRAM write energy. We also propose a dynamic profiling method to accommodate different application characteristics. The extensive simulation results show that write energy can be saved by 37.05% ∼ 38.89% for static profiling and 19.76% ∼ 34.29% for dynamic profiling.