Cache Architecture Research Articles

Parloom: A New Low-Power Set-Associative Instruction Cache Architecture Utilizing Enhanced Counting Bloom Filter and Partial Tags

The cache system dissipates a significant amount of energy compared to the other memory components. This will be intensified if a cache is designed with a set-associative structure to improve the system performance because the parallel accesses to the entries of a set for tag comparisons lead to even more energy consumption. In this paper, a novel method is proposed as a combination of a counting Bloom filter and partial tags to mitigate the energy consumption of set-associative caches. This new hybrid method noticeably decreases the cache energy consumption especially in highly-associative instruction caches. In fact, it uses an enhanced counting Bloom filter to predict cache misses with a high accuracy as well as partial tags to decrease the overall cache size. This way, unnecessary tag comparisons can be prevented and therefore, the cache energy consumption is considerably reduced. Based on the simulation results, the proposed method provides the energy reduction from 22% to 31% for 4-way–32-way set-associative L1 caches bigger than 16[Formula: see text]kB running the MiBench programs. The improvements are attained with a negligible system performance degradation compared to the traditional cache system.

Improving storage performance of high-performance computing systems by using the non-volatile buffer cache

ABSTRACT With the emerging applications of the 4th industrial revolution, such as IoT (Internet-of-Things), big data, and machine learning, the role of high-performance computing systems is becoming increasingly important. One of the significant performance hurdles in these systems is storage devices as the access time of disk drives is limited to tens of milliseconds. Although buffer cache alleviates the speed gap between memory and storage, data in buffer cache may be lost if the system crashes before updated data is reflected to storage. Thus, pdflush is used to periodically flush the updated data to storage. However, pdflush degrades the buffer cache performance significantly as most writes are directly transferred to storage. We show that pdflush accounts for 64–78% of total write traffic to storage. To resolve this problem, we present a new buffer cache architecture that adopts non-volatile memory to maintain updated data. Experimental results show that the proposed buffer cache reduces storage write traffic by 40%, thereby eliminating most of pdflush overhead.

Improving Cache Performance for Large-Scale Photo Stores via Heuristic Prefetching Scheme

Photo service providers are facing critical challenges of dealing with the huge amount of photo storage, typically in a magnitude of billions of photos, while ensuring national-wide or world-wide satisfactory user experiences. Distributed photo caching architecture is widely deployed to meet high performance expectations, where efficient still mysterious caching policies play essential roles. In this work, we present a comprehensive study on internet-scale photo caching algorithms in the case of QQPhoto from Tencent Inc., the largest social network service company in China. We unveil that even advanced cache algorithms can only perform at a similar level as simple baseline algorithms and there still exists a large performance gap between these cache algorithms and the theoretically optimal algorithm due to the complicated access behaviors in such a large multi-tenant environment. We then expound the reasons behind this phenomenon via extensively investigating the characteristics of QQPhoto workloads. Finally, in order to realistically further improve QQPhoto cache efficiency, we propose to incorporate a prefetcher in the cache stack based on the observed immediacy feature that is unique to the QQPhoto workload. The prefetcher proactively prefetches selected photos into cache before they are requested for the first time to eliminate compulsory misses and promote hit ratios. Our extensive evaluation results show that with appropriate prefetching we improve the cache hit ratio by up to 7.4 percent, while reducing the average access latency by 6.9 percent at a marginal cost of 4.14 percent backend network traffic compared to the original system that performs no prefetching.

A game theoretic approach for hierarchical caching resource sharing in 5G networks with virtualization

Caching and virtualization have been considered as the promising techniques in 5G Networks. In 5G networks with virtualization, the caching resources deployed by infrastructure providers (InPs) can be abstracted into isolated slices and transparently shared by mobile virtual network operators (MVNOs). In this case, one of the most important issues is how the MVNOs to share the caching resource. To solve this issue, different from previous works, a hierarchical caching architecture that core network and radio access network (RAN) have the caching capability in 5G networks with virtualization is first considered in this paper. Then, we study the problem of hierarchical caching resource sharing for MVNOs, and a competitive game to maximize their expectation revenue based on the oligopoly market model is formulated. As it is a hard problem to find the optimal solution in the hierarchical caching resource sharing problem, we decompose the optimization problem into two independent caching resource sharing problems in RAN and core network, respectively. Then the local optimal solutions are solved and the global Nash equilibrium solution is achieved. Finally, simulation results are illustrated to demonstrate the performance of the proposed scheme.

UniBuffer: Optimizing Journaling Overhead With Unified DRAM and NVM Hybrid Buffer Cache

Journaling techniques play an important role in addressing the reliability issue of filesystems caused by the volatile dynamic random access memory (DRAM)-based buffer cache. However, journaling techniques introduce a large number of extra storage writes, which greatly degrades system performance, especially for mobile devices. Emerging nonvolatile memory (NVM) technologies bring a new perspective of solving the write amplification issue caused by journaling. By adopting NVM as the buffer cache, the committed data can be maintained in NVM before being written back to the storage, thus, eliminating the journaling overhead. However, simply utilizing NVM as the buffer cache suffers from the limited lifetime and the relatively longer write latency of NVM. In this paper, we present a hybrid buffer cache architecture, named UniBuffer, by combing NVM with dynamic random access memory (DRAM) to reduce the journaling overhead and overcome the inherent constraints of NVM. In UniBuffer, we first propose a journaling-aware page management (JAPM) policy to smartly allocate data to DRAM and NVM pages. JAPM puts infrequently updated data in NVM to reduce the journaling overhead and frequently updated data in DRAM to improve the write performance and lifetime of the hybrid buffer cache. In addition, since data in one journaling transaction may be dispersed in NVM and DRAM simultaneously, different committing policies are required for different storage media, respectively. In order to guarantee the atomicity of the transaction execution in the hybrid cache architecture, a partial in-place commit (PIPC) journaling scheme is proposed to coordinate different committing patterns for NVM and DRAM. We have implemented the proposed techniques on Linux 3.14.52 and measured the performance with representative I/O-intensive benchmarks. The experimental results show that our scheme effectively improves the I/O performance compared with the ext4 filesystem and prolongs the lifetime of the hybrid buffer cache compared with the union of buffer cache and journaling (UBJ) scheme.

A game theoretic approach for hierarchical caching resource sharing in 5G networks with virtualization

Caching and virtualization have been considered as the promising techniques in 5G Networks. In 5G networks with virtualization, the caching resources deployed by infrastructure providers (InPs) can be abstracted into isolated slices and transparently shared by mobile virtual network operators (MVNOs). In this case, one of the most important issues is how the MVNOs to share the caching resource. To solve this issue, different from previous works, a hierarchical caching architecture that core network and radio access network (RAN) have the caching capability in 5G networks with virtualization is first considered in this paper. Then, we study the problem of hierarchical caching resource sharing for MVNOs, and a competitive game to maximize their expectation revenue based on the oligopoly market model is formulated. As it is a hard problem to find the optimal solution in the hierarchical caching resource sharing problem, we decompose the optimization problem into two independent caching resource sharing problems in RAN and core network, respectively. Then the local optimal solutions are solved and the global Nash equilibrium solution is achieved. Finally, simulation results are illustrated to demonstrate the performance of the proposed scheme.

An Efficient GPU Cache Architecture for Applications with Irregular Memory Access Patterns

GPUs provide high-bandwidth/low-latency on-chip shared memory and L1 cache to efficiently service a large number of concurrent memory requests. Specifically, concurrent memory requests accessing contiguous memory space are coalesced into warp-wide accesses. To support such large accesses to L1 cache with low latency, the size of L1 cache line is no smaller than that of warp-wide accesses. However, such L1 cache architecture cannot always be efficiently utilized when applications generate many memory requests with irregular access patterns especially due to branch and memory divergences that make requests uncoalesced and small. Furthermore, unlike L1 cache, the shared memory of GPUs is not often used in many applications, which essentially depends on programmers. In this article, we propose Elastic-Cache, which can efficiently support both fine- and coarse-grained L1 cache line management for applications with both regular and irregular memory access patterns to improve the L1 cache efficiency. Specifically, it can store 32- or 64-byte words in non-contiguous memory space to a single 128-byte cache line. Furthermore, it neither requires an extra memory structure nor reduces the capacity of L1 cache for tag storage, since it stores auxiliary tags for fine-grained L1 cache line managements in the shared memory space that is not fully used in many applications. To improve the bandwidth utilization of L1 cache with Elastic-Cache for fine-grained accesses, we further propose Elastic-Plus to issue 32-byte memory requests in parallel, which can reduce the processing latency of memory instructions and improve the throughput of GPUs. Our experiment result shows that Elastic-Cache improves the geometric-mean performance of applications with irregular memory access patterns by 104% without degrading the performance of applications with regular memory access patterns. Elastic-Plus outperforms Elastic-Cache and improves the performance of applications with irregular memory access patterns by 131%.

CLEAR: Cache Lines Error Accumulation Reduction by exploiting invisible accesses

SRAM caches are the most vulnerable processor component to radiation-induced soft errors. Error-Correcting Codes (ECCs) are the conventional scheme to protect caches against soft errors. These errors in the shape of Single-Event Upset (SEU), in which single bit or multiple adjacent bits are affected, can be correctable by ECCs. However, conventional ECCs are unable to correct temporal MBUs (Multiple-Bit Upset), in which two or more SEUs are accumulated in a data word over time. This paper proposes CLEAR (Cache Lines Error Accumulation Reduction) scheme to reduce the occurrence probability of temporal MBUs. By exploiting inherently available externally invisible accesses to all cache ways in a read request, CLEAR conducts more frequent ECC checking for each data word. Therefore, the ECC checking intervals are shortened, which leads to a significant reduction in occurrence probability of temporal MBUs. The evaluations show that CLEAR increases the Mean Time To Failure (MTTF) of the cache by more than 4× compared to the conventional cache architecture with negligible overheads.

A Cache Fill and Migration Policy for STT-RAM-Based Multi-Level Hybrid Cache in 3D CMPs

Recently, in 3D Chip-Multiprocessors (CMPs), a hybrid cache architecture of SRAM and Non-Volatile Memory (NVM) is generally used to exploit high density and low leakage power of NVM and a low write overhead of SRAM. The conventional access policy does not consider the hybrid cache and cannot make good use of the characteristics of both NVM and SRAM technology. This paper proposes a Cache Fill and Migration policy (CFM) for multi-level hybrid cache. In CFM, data access was optimized in three aspects: Cache fill, cache eviction, and dirty data migration. The CFM reduces unnecessary cache fill, write operations to NVM, and optimizes the victim cache line selection in cache eviction. The results of experiments show that the CFM can improve performance by 24.1% and reduce power consumption by 18% when compared to conventional writeback access policy.

An Efficient Hybrid I/O Caching Architecture Using Heterogeneous SSDs

SSDs are emerging storage devices which unlike HDDs, do not have mechanical parts and therefore, have superior performance compared to HDDs. Due to the high cost of SSDs, entirely replacing HDDs with SSDs is not economically justified. Additionally, SSDs can endure a limited number of writes before failing. To mitigate the shortcomings of SSDs while taking advantage of their high performance, SSD caching is practiced in both academia and industry. Previously proposed caching architectures have only focused on either performance or endurance and neglected to address both parameters in suggested architectures. Moreover, the cost, reliability, and power consumption of such architectures is not evaluated. This paper proposes a hybrid I/O caching architecture that while offers higher performance than previous studies, it also improves power consumption with a similar budget. The proposed architecture uses DRAM, Read-Optimized SSD, and Write-Optimized SSD in a three-level cache hierarchy and tries to efficiently redirect read requests to either DRAM or RO-SSD while sending writes to WO-SSD. To provide high reliability, dirty pages are written to at least two devices which removes any single point of failure. The power consumption is also managed by reducing the number of accesses issued to SSDs. The proposed architecture reconfigures itself between performance- and endurance-optimized policies based on the workload characteristics to maintain an effective tradeoff between performance and endurance. We have implemented the proposed architecture on a server equipped with industrial SSDs and HDDs. The experimental results show that as compared to state-of-the-art studies, the proposed architecture improves performance and power consumption by an average of 8% and 28%, respectively, and reduces the cost by 5% while increasing the endurance cost by 4.7% and negligible reliability penalty.

Time-sensitivity-aware shared cache architecture for multi-core embedded systems

In embedded systems such as automotive systems, multi-core processors are expected to improve performance and reduce manufacturing cost by integrating multiple functions on a single chip. However, inter-core interference in shared last-level cache (LLC) results in increased and unpredictable execution times for time-sensitive tasks (TSTs), which have (soft) timing constraints, thereby increasing the deadline miss rates of such systems. In this paper, we propose a time-sensitivity-aware dead block-based shared LLC architecture to mitigate these problems. First, a time-sensitivity indication bit is added to each cache block, which allows the proposed LLC architecture to be aware of instructions/data belonging to TSTs. Second, portions of the LLC space are allocated to general tasks without interfering with TSTs by developing a time-sensitivity-aware dead block-based cache partitioning technique. Third, to reduce the deadline miss rate of TSTs further, we propose a task matching in shared caches and a cache partitioning scheme that considers the memory access characteristics and the time-sensitivity of tasks (TATS). The TATS is combined with our proposed dead block-based scheme. Our evaluation shows that the proposed schemes reduce deadline miss rates of TSTs compared to conventional shared caches. On a dual-core system, compared to a baseline, equal partitioning, and state-of-the-art quality-of-service-aware cache partitioning, our proposed dead block-based cache partitioning provides 9.3%, 30.5%, and 2.6% lower average deadline miss rates, respectively. On a quad-core system, compared to the baseline, equal partitioning, and state-of-the-art quality-of-service-aware cache partitioning, the combination of our proposed schemes provides 21.2%, 17.7%, and 4.1% lower average deadline miss rates, respectively.

Neural Cache: Bit-Serial In-Cache Acceleration of Deep Neural Networks

This article presents Neural Cache architecture, which repurposes cache structures to transform them into massively parallel compute units capable of running inferences for deep neural networks. Techniques to do in situ arithmetic in SRAM arrays create efficient data mapping, and reducing data movement is proposed. Neural Cache architecture is capable of fully executing convolutional, fully connected, and pooling layers in cache. Our experimental results show that the proposed architecture can improve efficiency over a GPU by $\mathbf {128\times}$128× while requiring a minimal area overhead of 2%.

EMC: Energy-Aware Morphable Cache Design for Non-Volatile Processors

Wearable, implantable and Internet of Things devices are attracting increasing attention from both research and industry fields. Energy harvesting is a promising alternative of battery to power these embedded systems. However, the intrinsic instability of energy harvesting systems leads to potential frequent power interruptions. In traditional volatile processor, all the status will be lost at power failures and the program needs to re-start after power resumes. In order to survive the power failures and enable accumulative execution, non-volatile processor (NVP) is proposed to back up volatile information before power depletion and recover the system status after power resumes. Non-volatile memory (NVM) is typically attached for cache and main memory backup. There are researches working on optimization of the backup. However, little of them involve multiple level cell (MLC) NVM. In this work, we first discuss the benefit of applying MLC NVM for cache backup and the architecture of morphable hybrid cache, and then propose a three-stage energy-aware cache management strategy to improve the system performance and energy utilization while guaranteeing successful backups. Backup-aware cache replacement policies are also developed for backup optimization. Evaluation shows that the proposed EMC scheme can achieve 10.6 percent performance improvement and simultaneous 25.2 percent energy reduction when compared with the single level cell (SLC) based hybrid cache.

Distributed Caching Algorithms in the Realm of Layered Video Streaming

Distributed caching architectures have been proposed for bringing content close to requesters, and the key problem is to design caching algorithms for reducing content delivery delay, which determines to an extent the user Quality of Experience (QoE). This problem obtains an interesting new twist with the advent of advanced layered-video encoding techniques such as Scalable Video Coding. In this paper, we show that the problem of finding the caching configuration of video encoding layers that minimizes delivery delay for a network operator is NP-Hard, and we establish a pseudopolynomial-time optimal solution by using a connection with the multiple-choice knapsack problem. Next, we design caching algorithms for multiple network operators that cooperate by pooling together their co-located caches, in an effort to aid each other, so as to avoid large delays due to fetching content from distant servers. We derive an approximate solution to this cooperative caching problem by using a technique that partitions the cache capacity into amounts dedicated to own and other operators’ caching needs. Trace-driven evaluations demonstrate up to 25 percent reduction in delay over existing caching schemes. As a side benefit, our algorithms achieve smoother playback for video streaming applications, with fewer playback stalls and higher decoded quality.

An Energy-Efficient Reliable Heterogeneous Uncore Architecture for Future 3D Chip-Multiprocessors

Uncore components such as cache hierarchy and on-chip interconnects consume a significant portion of overall energy consumption in emerging embedded processors. In Nanoscale technologies, static power consumption due to leakage current has become a serious issue in the design of SRAM-based on-chip cache memories and interconnections. To address this issue, non-volatile memory technologies such as STT-RAMs have been proposed as a replacement for SRAM cells due to their near-zero static power and high memory density. Nonetheless, STT-RAMs suffer from some failures such as read-disturb and limited endurance as well as high switching energy. One effective way to decrease the STTRAMs’ switching energy is to reduce their retention time; however, reducing the retention time has a negative impact on the reliability of STT-RAM cells. In this paper, we propose a heterogeneous last level cache (LLC) architecture for 3D embedded chip-multiprocessors (3D eCMPs) which employs two types of STT-RAM memory banks with retention time of 1s and 10ms to provide a beneficial trade-off between reliability, energy consumption, and performance. To this end, we also propose a convex optimization model to find the optimal configurations for these two kinds of memory banks. In parallel with hybrid memory architecting, optimizing the number and placement of through silicon vias (TSVs) as a main component of on-chip interconnection for building 3D CMPs is another important target of the proposed optimization approach. Experimental results show that the proposed method improves the energy-delay products and throughput by about 69% and 34.5% on average compared with SRAM configurations.

In-DRAM Cache Management for Low Latency and Low Power 3D-Stacked DRAMs.

Recently, 3D-stacked dynamic random access memory (DRAM) has become a promising solution for ultra-high capacity and high-bandwidth memory implementations. However, it also suffers from memory wall problems due to long latency, such as with typical 2D-DRAMs. Although there are various cache management techniques and latency hiding schemes to reduce DRAM access time, in a high-performance system using high-capacity 3D-stacked DRAM, it is ultimately essential to reduce the latency of the DRAM itself. To solve this problem, various asymmetric in-DRAM cache structures have recently been proposed, which are more attractive for high-capacity DRAMs because they can be implemented at a lower cost in 3D-stacked DRAMs. However, most research mainly focuses on the architecture of the in-DRAM cache itself and does not pay much attention to proper management methods. In this paper, we propose two new management algorithms for the in-DRAM caches to achieve a low-latency and low-power 3D-stacked DRAM device. Through the computing system simulation, we demonstrate the improvement of energy delay product up to 67%.

User Preference Learning-Based Edge Caching for Fog Radio Access Network

In this paper, the edge caching problem in fog radio access network (F-RAN) is investigated. By maximizing the overall cache hit rate, the edge caching optimization problem is formulated to find the optimal policy. Content popularity in terms of time and space is considered from the perspective of regional users. We propose an online content popularity prediction algorithm by leveraging the content features and user preferences, and an offline user preference learning algorithm by using the online gradient descent (OGD) method and the follow the (proximally) regularized leader (FTRL-Proximal) method. Our proposed edge caching policy not only can promptly predict the future content popularity in an online fashion with low complexity, but also can track the content popularity with spatial and temporal popularity dynamic in time without delay. Furthermore, we design two learning-based edge caching architectures. Moreover, we theoretically derive the upper bound of the popularity prediction error, the lower bound of the cache hit rate, and the regret bound of the overall cache hit rate of our proposed edge caching policy. Simulation results show that the overall cache hit rate of our proposed policy is superior to those of the traditional policies and asymptotically approaches the optimal performance.

C3C: A New Static Content-Based Three-Level Web Cache

One way of improving the performance of a search engine is increasing the hit ratio of the search engine cache. A common and widely used approach for increasing the hit ratio is a combination of the result cache, the posting list cache, and the intersection cache, yielding a three-level cache architecture. However, the existing multi-level cache architectures do not consider the dependencies among the content cached in different parts. Thus, the same content might be stored multiple times in the architectures, resulting in duplicate hits. In other words, a large amount of space in the cache is wasted. In this paper, we propose a new static content-based three-level cache architecture that adopts a new content-based three-level Web cache filling algorithm that takes into account the dependencies among the content cached in different parts. In the proposed cache architecture, duplicate hits are reduced and the hit ratio increases. Extensive experiments were conducted on a real data set. The results have shown a significant improvement on the hit ratios compared with two existing approaches.

Lowering the Hit Latencies of Low Voltage Caches Based on the Cross-Sensing Timing Speculation SRAM

The probability of timing failure in SRAM accessing becomes unacceptably high at low voltages, which makes the SRAM become the bottleneck of the system performance. Recently proposed timing speculation SRAM (SSRAM) can access bit cells much earlier than the conservative time margin and detect the potential timing failures. Unfortunately, the performance benefit brought by this speculative scheme might be nullified by the extra cycles used to correct the errors. Therefore, in a cache architecture, cachelines containing bit cells with timing failure bring significant performance degradation in a low-voltage scenario. In this paper, we propose the set-associative L1 and L2 Remapping and Reuse-aware timing Speculation caches (RRS caches) based on the cross-sensing SSRAM to reduce the weak cacheline access penalties under low supply voltages. RRS caches improve the proportion of the error-free cachelines (or strong cachelines) by clustering as more as possible bit cells with timing failures into weak cachelines. In addition, RRS caches swap the frequently reused data to the error-free cachelines to further reduce the average access latency and the energy consumption by an optimized filling/replacement policy. We compared the performance, energy consumption and area overhead of RRS caches with those of state-of-the-art approaches. According to our simulation results, RRS caches improve the system performance close to that of the Perfect caches in a four-core processor with only 0.12% extra energy consumption, 4.12% extra area overhead in the L1 cache and 2.69% extra energy consumption, 5.66% extra area overhead in the L2 cache compared to the cache designs with the raw SSRAM.

FPGA as a Hardware Accelerator for Computation Intensive Maximum Likelihood Expectation Maximization Medical Image Reconstruction Algorithm

The major requirements of a good tomographic reconstruction algorithm are a reduction in radiation dosage, accurate reconstruction, detail enhancement, and rapid reconstruction time. Some of these factors are covered by many algorithms but are not collectively addressed in one. While the Maximum Likelihood Expectation Maximization (MLEM) algorithm fares well on many of these factors, it is difficult to apply this algorithm in real-time due to its long execution time. Our predetermined goal is to reduce the execution time to a large extent so that the MLEM’s advantages can be leveraged by using hardware accelerators such as Field Programmable Gate Arrays (FPGA). The FPGAs are becoming especially popular as hardware accelerators and are well known for their programmability, configurability, and massive parallelism through a large number of Configurable Logic Blocks (CLBs). Although FPGAs are extremely versatile, they require complex languages like Verilog or VHDL to program them, incorporating changes in the design level at a later stage in FPGAs demands increased effort. Here, in this paper, for the first time, we present a parallel structure for hardware acceleration of the MLEM on the mammoth Virtex 7 $VC$ 709 FPGA. Using available tools, we also present a programming flow to design the algorithmic acceleration hardware architecture. The proposed flow does not require prior knowledge of the traditionally cumbersome Hardware Description Languages (HDLs) and this makes post design changes very easy to incorporate and validate. The parallel architecture is implemented on an FPGA operating at 220 MHz and we have achieved a $288\times $ performance compared to an optimized software execution on an Intel Xeon workstation with 12-cores 3.1 GHz 32 GB RAM and 12 MB Cache architecture.