Cache Access Research Articles

IBOM: An Integrated and Balanced On-Chip Memory for High Performance GPGPUs

GPGPU accelerated computing has revolutionized a broad range of applications. To serve between the ever-growing computing capability and external memory, the on-chip memory is becoming increasingly important to GPGPU performance for general-purpose computing. Inherited from the traditional CPUs, however, the contemporary GPGPU on-chip memory design is suboptimal to the SIMT (single instruction, multiple threads) execution. In particular, the on-chip first-level data (L1D) cache thrashing, resulting from insufficient capacity and imbalanced usage, leads to a low hit rate and limits the overall performance. In this study, we reform the contemporary on-chip memory design and propose an integrated and balanced on-chip memory (IBOM) architecture for high-performance GPGPUs. It first virtually enlarges the L1D cache size by an integrated architecture that exploits the under-utilized register file (RF) with lightweight ISA, compiler and microarchitecture supports. Then with sufficient capacity, it is able to improve the cache usage by a set balancing technique that exploits the under-utilized set resources. In our proposed IBOM design, the register and cache accesses are amenable to normal pipeline operations with simple changes. It adequately exploits the size inversion in GPGPU on-chip memory, and enables optimized utilization of the precious resources for higher performance and energy efficiency with even smaller on-chip memory size. The experiment results demonstrate that the proposed IBOM design can offer an average of 29.6 percent increase in L1D hit rate and in turn 3X performance improvement for the cache-sensitive applications.

Fast media caching for geo-distributed data centers

Recent years have witnessed a phenomenal increase in video traffic. Virtual content delivery networks (vCDNs) coordinate video content delivery through the use of computing and storage resources from the cloud and distributes content to edge nodes near consumers to reduce network traffic and improve service experience. An important objective of vCDNs is operation cost minimization. Since cloud data centers are geo-distributed, content transfer costs vary significantly with different data centers, i.e., the cost is high for retrieval from distant data centers and lower for nearby retrievals. Many popular caching algorithms in use today, such as Lru, do not consider cost when making caching decisions, and as a result, suffer from high data transfer costs and increased network congestion. On the other hand, cost-aware caching algorithms such as LandLord [1] are computationally inefficient, with time complexity scaling linearly to the amount of content in the vCDN. Such algorithms are unable to keep pace with the exponential growth in video content over time. In this paper, we propose Fmc(fast media caching), a cost-aware and highly efficient caching algorithm for vCDN delivery over geo-distributed data centers. The load cost of each content item is determined by both the item’s size and distance from the data center it is loaded from. We first prove that Fmc is kk−h+1 competitive under the resource augmentation paradigm, where Fmc and the optimal offline adversary have k and h amount of cache, resp., and k ≥ h. Also, we show our algorithm is straightforward and efficient, requiring only O(log m) time per cache access, where m is the number of data centers and is a small constant in practice. We conduct experimental studies on Fmc using both synthetic and YouTube traces. Our results show that Fmc has on average 50% and up to 66.7% lower cost than Lru. Besides, we show Fmc is much faster than LandLord, and the speedup scales linearly with cache size.

SwSpTRSV

Sparse triangular solve (SpTRSV) is one of the most important kernels in many real-world applications. Currently, much research on parallel SpTRSV focuses on level-set construction for reducing the number of inter-level synchronizations. However, the out-of-control data reuse and high cost for global memory or shared cache access in inter-level synchronization have been largely neglected in existing work. In this paper, we propose a novel data layout called Sparse Level Tile to make all data reuse under control, and design a Producer-Consumer pairing method to make any inter-level synchronization only happen in very fast register communication. We implement our data layout and algorithms on an SW26010 many-core processor, which is the main building-block of the current world fastest supercomputer Sunway Taihulight. The experimental results of testing all 2057 square matrices from the Florida Matrix Collection show that our method achieves an average speedup of 6.9 and the best speedup of 38.5 over parallel level-set method. Our method also outperforms the latest methods on a KNC many-core processor in 1856 matrices and the latest methods on a K80 GPU in 1672 matrices, respectively.

A Request‐Based Handover Strategy Using NDN for 5G

With the developing of 5G, it is widely accepted that 5G will use a system architecture that supports the ultradense networks (UDN) deployments. In this architecture, a user will be covered by a large amount of small cell base stations (SBS) in 5G. However, selecting an SBS for handover is a great challenge. To address the challenge, the emerging content‐oriented Named Data Networking (NDN) has attractive advantages, such as providing name‐based routing. In this paper, a request‐based handover strategy (RBHS) is presented to improve the user experience in performance and obtain the optimal allocation of resources, and a caching mechanism based on the users’ requests is introduced for it. The proposed caching mechanism and access network selection mechanism were validated utilizing ndnSIM. Simulation results demonstrate that our proposed strategy achieves around 30% higher cache hit rate and 20% more traffic reduction, compared with the access network selection based on SINR.

A Hybrid Architecture With Low Latency Interfaces Enabling Dynamic Cache Management

The main focus of the dominant technologies in the high performance computation (HPC) market, such as GPU and multicore systems, is put on processing power, while much less attention has been paid to communication delays inside hybrid architectures. To fill this gap, this paper presents an experimental study on Intel’s Broadwell Xeon multicore processor with integrated Arria 10 FPGA capabilities to characterize the communication delays between CPUs and the FPGA, using both the low latency cache coherent interface and the two PCIe links offered by this platform. The obtained results show that an FPGA cache access latency can be as low as 25 cycles at 400 MHz and that the platform is capable of reaching a bandwidth over 20 GB/s using an aggregate of the three available links. Furthermore, an FPGA-based cache management mechanism is proposed and implemented in this paper. A case study on a Merkle tree hash function shows that a hardware accelerator can achieve a fivefold data access acceleration in the worst case scenario. This scheme takes advantage of the QPI cache coherency and queuing theory to achieve a low latency and efficient memory management. In addition, design recommendations regarding the use of the CPU-FPGA platform for the implementation of fine-grained memory management schemes are suggested.

Caching AP Selection and Channel Allocation in Wireless Caching Networks: A Binary Concurrent Interference Minimizing Game Solution

Caching in wireless networks can alleviate redundant data transmissions and can assist cell base stations in providing service. In this paper, we focus on the problem of joint caching access point (AP) selection and channel allocation for mobile users (MUs) in wireless caching networks. Considering MUs’ dynamic probabilistic content demands, an asymmetric weighted interference graph is proposed to analyze the asymmetric interference among MU and caching AP pairs. We formulate the joint caching AP selection and channel allocation as a nonlinear optimization problem and then present a binary concurrent interference minimizing game theoretical scheme, where MUs and caching APs are two kinds of players with different kinds of actions and utilities. Unlike existing hierarchical game approaches, MUs and caching APs act concurrently without hierarchical constraints. Then, a novel pure strategy binary Nash equilibrium (BNE) and a concurrent binary better reply algorithm are proposed for the above game. It is proved that at least one BNE exists and the reply algorithm converges to a BNE. Moreover, simulation results show that the proposed approach can mitigate interference effectively and achieve a lower global interference level than the traditional better reply algorithm.

CAIRO

Three-dimensional (3D)-stacking technology and the memory-wall problem have popularized processing-in-memory (PIM) concepts again, which offers the benefits of bandwidth and energy savings by offloading computations to functional units inside the memory. Several memory vendors have also started to integrate computation logics into the memory, such as Hybrid Memory Cube (HMC), the latest version of which supports up to 18 in-memory atomic instructions. Although industry prototypes have motivated studies for investigating efficient methods and architectures for PIM, researchers have not proposed a systematic way for identifying the benefits of instruction-level PIM offloading . As a result, compiler support for recognizing offloading candidates and utilizing instruction-level PIM offloading is unavailable. In this article, we analyze the advantages of instruction-level PIM offloading in the context of HMC-atomic instructions for graph-computing applications and propose CAIRO , a compiler-assisted technique and decision model for enabling instruction-level offloading of PIM without any burden on programmers. To develop CAIRO, we analyzed how instruction offloading enables performance gain in both CPU and GPU workloads. Our studies show that performance gain from bandwidth savings, the ratio of number of cache misses to total cache accesses, and the overhead of host atomic instructions are the key factors in selecting an offloading candidate. Based on our analytical models, we characterize the properties of beneficial and nonbeneficial candidates for offloading. We evaluate CAIRO with 27 multithreaded CPU and 36 GPU benchmarks. In our evaluation, CAIRO not only doubles the speedup for a set of PIM-beneficial workloads by exploiting HMC-atomic instructions but also prevents slowdown caused by incorrect offloading decisions for other workloads.

Tile/line access cache memory based on a multi‐level Z‐order tiling data layout

SummaryIneffective column‐directional cache memory access has become a bottleneck for efficient two‐dimensional (2‐D) data processing utilizing extended single instruction multiple data (SIMD) instructions. To solve this problem, we propose a cache memory with tile (column and row directions) and line (row direction) accessibility for efficient 2‐D data processing. 2‐D data access to the proposed cache memory is enabled via a hardware‐based multi‐mode address translation unit that eliminates the overhead of software‐based address calculation. To reduce the hardware overhead of the proposed cache, we propose a tag memory reduction method that replaces multiple tiles with an aligned tile set (RATS) in the cache. To verify the feasibility of the proposed cache, an LSI layout of a SIMD‐based general purpose‐oriented datapath embedding the proposed cache is designed in a 2.5×5 mm2 area using 0.18‐μm CMOS technology. Under a 3.9‐ns clock period (250 MHz), the read latency is limited to 3 clock cycles, which is the same as that for the conventional cache memory. Using the RATS method, the entire hardware overhead of the proposed cache is reduced to only 7% of that required for a conventional cache. In addition, simulation results for the proposed cache indicate a considerable reduction of L1 and L2 cache confliction misses compared with a conventional cache in power‐of‐two matrix size due to the column‐directional address stride being sufficiently smaller than page size. Therefore, the proposed cache provides efficient column‐directional parallel access as same as row‐directional parallel access so that it enables efficient SIMD operation requiring no transposition in matrix multiplication (MM). For LU decomposition (LUD), the proposed cache can provide almost the same performance to the column‐major–based LUD program as that to the row‐major–based LUD program. These results show that the proposed cache does not restrict our freedom in selecting either row‐ or column‐major order coding.

Enhance the Performance of Associative Memory by Using New Methods

Data or instructions that are regularly used are saved in cache so that it is very easy to retrieve for the purpose of increase the cache performance. Evaluating the execution of multi-core systems the part of the cache memory is very important. A multicore processor is shared circuit in which two or more processors are joined to enhance the performance and perform multiple tasks. This paper describes the performance of cache memory based on cache access time, miss rate and miss penalty. Cache mapping methods are defined to increase the performance of cache but it face many difficulties. Some methods and algorithms are used to decrease these difficulties. In this paper describes the study of recent competing processors to evaluate the cache memory performance.

Cost-Effective Enhancement on Both Yield and Reliability for Cache Designs Based on Performance Degradation Tolerance

Guaranteeing functional correctness of cache memories is crucial for computer designs. In the literature, there have been several works addressing this issue. However, fault tolerability of these methods may be limited. In this paper, we present a new cache architecture that has flexible tolerability. Moreover, by using the proposed architecture, both yield and reliability of the cache can be enhanced simultaneously. In our cache, a particular type of cache blocks called tolerable block is further identified among the faulty ones. Such blocks can still be used during cache access in our architecture, while accessing to intolerable blocks will result in additional cache misses, and therefore performance degradation. The number of tolerable cache blocks is thus critical for the achievable yield and reliability enhancement, as well as the incurred cost on performance. In this work, error correcting code (ECC) methods are employed to increase the number of tolerable blocks. In particular, we propose to embed the required check bits in one of the cache ways. Analysis results show that this embedding method only incurs minor performance degradation, while the incurred area overhead due to ECC can thus be significantly reduced from 5.92% to only 0.92%. General applicability of the embedding method to ordinary ECC methods is also investigated. Experimental results show that the performance degradation can be reduced from 16% to only 1.53% by using the proposed cache. This leads to great tolerability improvement, and thus the yield and reliability are enhanced very significantly when compared with the previous work.

Word- and Partition-Level Write Variation Reduction for Improving Non-Volatile Cache Lifetime

Non-volatile memory technologies are among the most promising technologies for implementing the main memories and caches in future microprocessors and replacing the traditional DRAM and SRAM technologies. However, one of the most challenging design issues of the non-volatile memory technologies is the limited write. In this article, we first propose to exploit the narrow-width values to improve the lifetime of non-volatile last-level caches with word-level write variation reduction. Leading zeros masking scheme is proposed to reduce the write stress to the upper half of the narrow-width data. To balance the write variations between the upper half and the lower half of the narrow-width data, two swapping schemes, the swap on write (SW) and swap on replacement (SRepl), are proposed. Two existing optimization schemes, the multiple dirty bit (MDB) and read before write (RBW), are adopted with our word-level swapping design. To further reduce the write variation on the partition level, we propose to exploit the cache partitioning design to improve the lifetime. Based on the observation that different applications demonstrate different cache access (write) behaviors, we propose to partition the last-level cache for different applications and balance the write variations by partition swapping. Both software-based and hardware-based partitioning and swapping schemes are proposed and evaluated for different situations. Our experimental results show that both our word- and partition-level designs can improve the lifetime of the non-volatile caches effectively with low performance and energy overheads.

Low-level implementation of the SISC protocol for thread-level speculation on a multi-core architecture

Abstract Chip Multiprocessors (CMP) have emerged during last decades as a very attractive solution in using the ever-increasing on-chip transistor count. However, classical parallelization techniques failed to fully exploit parallelization from existing sequential applications due to false data dependencies. This paper focuses on the Thread-level Speculation (TLS) technique, an alternative way to exploit the transistor budget in a CMP. With TLS, even possibly data dependent threads can run in parallel as long as the semantics of the sequential execution is preserved. A special hardware support monitors the actual data dependencies between threads at run time and, if they are violated, misspeculation effects are undone usually through replay. This kind of system is known as speculative CMP. However, the TLS mechanism requires complex protocols that integrate cache coherence and speculation to maintain program order among multiple versions of data. Current TLS protocol evaluations are usually inadequate because they are not done low-level enough. A realistic evaluation of speculative CMPs requires either to be performed on a real hardware or very detailed cycle-accurate simulator models. In this paper we are particularly focused on a low-level evaluation of the write-invalidate TLS protocol Speculation Integrated with Snoopy Coherence (SISC) protocol proposed in [1] . This evaluation relies on cycle-level simulation environment with detailed cycle-level cache memories, cache controller and system bus. On top of this, a speculative four core architecture is simulated and three new modules (Scheduler, Squash Arbiter and Supplier Arbiter) are provided to support low-level implementation of the SISC protocol. The overall cost of the SISC protocol is evaluated by means of CACTI tool for the three different domains: the access latency cost, the area cost, and the power cost. The evaluation goal was to keep the cache access time to remain below cycle latency as well as the area and power overheads below an acceptable budget overhead. The SISC protocol has been compared against regular MESI-based architecture in both 32-bit and 64-bit versions. We kept the cache access time below the cycle latency, and we managed to keep both data cache area and static power overheads respectively below 32% and 35%.

A Study on Performance and Power Efficiency of Dense Non-Volatile Caches in Multi-Core Systems

This paper presents a novel cache design based on Multi-Level Cell Spin-Transfer Torque RAM (MLC STT-RAM).Our design exploits the asymmetric nature of the MLC STT-RAM to build cache lines featuring heterogeneous performances, that is, half of the cache lines are read-friendly,while the other half are write-friendly--this asymmetry in read/write latencies are then used by a migration policy in order to overcome the high latency of the baseline MLC cache. Furthermore, in order to enhance the device lifetime, we propose to dynamically deactivate ways of a set in underutilized sets to convert MLC to Single-Level Cell (SLC)mode.Our experiments show that our design gives an average improvement of 12% in system performance and 26% in last-level cache(L3) access energy for various workloads.

Addressing isolation challenges of non-blocking caches for multicore real-time systems

In multicore real-time systems, cache partitioning is commonly used to achieve isolation among different cores. We show, however, that space isolation achieved by cache partitioning does not necessarily guarantee predictable cache access timing in modern COTS multicore platforms, which use non-blocking caches. We find that special hardware registers in non-blocking caches, known as miss status holding registers, which track the status of outstanding cache-misses, can be a significant source of contention that is not addressed by conventional cache partitioning. We propose a hardware and system software (OS) collaborative approach to efficiently eliminate MSHR contention for multicore real-time systems. Our approach includes a low-cost hardware extension that enables dynamic control of per-core memory-level parallelism (MLP) by the OS. Using the hardware extension, the OS scheduler then globally controls each core’s MLP in such a way that eliminates MSHR contention and maximizes overall throughput of the system. We implement the hardware extension in a cycle-accurate full-system simulator and the scheduler modification in Linux 3.14 kernel. Extensive experimental results demonstrate the significance of the MSHR contention problem and the effectiveness of the proposed solution.

Optical Overlay NUCA

In this article, we propose using optical networks-on-chip (NoCs) to design cache access protocols for large shared L2 caches. We observe that the problem is unique because optical networks have very low latency, and in principle all of the cache banks are very close to each other. A naive approach is to broadcast a request to a set of banks that might possibly contain the copy of a block. However, this approach is wasteful in terms of energy and bandwidth. Hence, we propose a set of novel schemes that create a set of virtual networks ( overlays ) of cache banks over a physical optical NoC. We search for a block inside each overlay using a combination of multicast and unicast messages. We first propose two simple protocols: TSI and Broadcast . The former uses unicast messages, and the latter uses multicast messages. We subsequently propose an improved scheme, OP_BCAST , that combines the best of TSI and Broadcast , and mainly uses restricted multicast messages. Then we propose a set of novel hardware structures for creating and managing overlays, for efficiently locating blocks in the overlay, and for implementing dynamically changing overlays with OP_BCAST . The performance of the TSI scheme is within 2% to 3% of a broadcast scheme, and it is faster than traditional schemes with electrical networks by 26%. Compared to the broadcast scheme, it reduces the number of accesses, and consequently the dynamic energy of the caches by 6% to 8%. OP_BCAST is 34% faster than the best solutions with copper-based NoCs; moreover, it reduces the dynamic energy for cache access by 33% compared to the TSI scheme.

Segment access-aware dynamic semantic cache in cloud computing environment

In recent years, researches focus on addressing the query bottleneck issue using semantic cache. However, the challenges of this method are how to increase cache hit ratio, decrease the query processing time, and address cache consistency issue. In this paper, we construct segment access-aware dynamic semantic cache for relational databases. Some definitions of semantic segment, probe query, and remainder query are proposed to describe the semantic cache. Then, estimation of the query result is proposed. Next, cache access algorithm of our proposed segment access-aware dynamic semantic cache is presented in case of cache exact hit, cache extended hit, cache partial hit and cache miss. Cache item with effective lifecycle tag is proposed to address cache consistency issue. Finally, experimental results show that this approach performs better than regular semantic cache and decisional semantic cache.

Towards "Full Containerization" in Containerized Network Function Virtualization

With exploding traffic stuffing existing network infra-structure, today's telecommunication and cloud service providers resort to Network Function Virtualization (NFV) for greater agility and economics. Pioneer service provider such as AT&T proposes to adopt container in NFV to achieve shorter Virtualized Network Function (VNF) provisioning time and better runtime performance. However, we characterize typical NFV work-loads on the containers and find that the performance is unsatisfactory. We observe that the shared host OS net-work stack is the main bottleneck, where the traffic flow processing involves a large amount of intermediate memory buffers and results in significant last level cache pollution. Existing OS memory allocation policies fail to exploit the locality and data sharing information among buffers. In this paper, we propose NetContainer, a software framework that achieves fine-grained hardware resource management for containerized NFV platform. NetContainer employs a cache access overheads guided page coloring scheme to coordinately address the inter-flow cache access overheads and intra-flow cache access overheads. It maps the memory buffer pages that manifest low cache access overheads (across a flow or among the flows) to the same last level cache partition. NetContainer exploits a footprint theory based method to estimate the cache access overheads and a Min-Cost Max-Flow model to guide the memory buffer mappings. We implement the NetContainer in Linux kernel and extensively evaluate it with real NFV workloads. Exper-imental results show that NetContainer outperforms conventional page coloring-based memory allocator by 48% in terms of successful call rate.

Towards "Full Containerization" in Containerized Network Function Virtualization

With exploding traffic stuffing existing network infra-structure, today's telecommunication and cloud service providers resort to Network Function Virtualization (NFV) for greater agility and economics. Pioneer service provider such as AT&T proposes to adopt container in NFV to achieve shorter Virtualized Network Function (VNF) provisioning time and better runtime performance. However, we characterize typical NFV work-loads on the containers and find that the performance is unsatisfactory. We observe that the shared host OS net-work stack is the main bottleneck, where the traffic flow processing involves a large amount of intermediate memory buffers and results in significant last level cache pollution. Existing OS memory allocation policies fail to exploit the locality and data sharing information among buffers. In this paper, we propose NetContainer, a software framework that achieves fine-grained hardware resource management for containerized NFV platform. NetContainer employs a cache access overheads guided page coloring scheme to coordinately address the inter-flow cache access overheads and intra-flow cache access overheads. It maps the memory buffer pages that manifest low cache access overheads (across a flow or among the flows) to the same last level cache partition. NetContainer exploits a footprint theory based method to estimate the cache access overheads and a Min-Cost Max-Flow model to guide the memory buffer mappings. We implement the NetContainer in Linux kernel and extensively evaluate it with real NFV workloads. Exper-imental results show that NetContainer outperforms conventional page coloring-based memory allocator by 48% in terms of successful call rate.

Near-Threshold RISC-V Core With DSP Extensions for Scalable IoT Endpoint Devices

Endpoint devices for Internet-of-Things not only need to work under extremely tight power envelope of a few milliwatts, but also need to be flexible in their computing capabilities, from a few kOPS to GOPS. Near-threshold (NT) operation can achieve higher energy efficiency, and the performance scalability can be gained through parallelism. In this paper, we describe the design of an open-source RISC-V processor core specifically designed for NT operation in tightly coupled multicore clusters. We introduce instruction extensions and microarchitectural optimizations to increase the computational density and to minimize the pressure toward the shared-memory hierarchy. For typical data-intensive sensor processing workloads, the proposed core is, on average, $3.5\times $ faster and $3.2\times $ more energy efficient, thanks to a smart L0 buffer to reduce cache access contentions and support for compressed instructions. Single Instruction Multiple Data extensions, such as dot products, and a built-in L0 storage further reduce the shared-memory accesses by $8\times $ reducing contentions by $3.2\times $ . With four NT-optimized cores, the cluster is operational from 0.6 to 1.2 V, achieving a peak efficiency of 67 MOPS/mW in a low-cost 65-nm bulk CMOS technology. In a low-power 28-nm FD-SOI process, a peak efficiency of 193 MOPS/mW (40 MHz and 1 mW) can be achieved.

Shaping data for application performance and energy optimization in dynamic data view framework

Memory access bottlenecks are often due to the result of mismatch between the processor hardware's view of data and the algorithmic/logical view of data. This variation in data views is especially more pronounced in applications involving large datasets, leading to significantly increased latency and user response times. Previous attempts to tackle this problem were primarily targeted at execution time optimization. We present a dynamic technique piggybacked on the classical dynamic binary optimization (DBO) to shape the data view for each program phase differently resulting in program execution time reduction along with reductions in access energy. Our implementation rearranges non-adjacent data into a contiguous dataview. It uses wrappers to replace irregular data access patterns with spatially local dataview. DDV (Dynamic data view), a runtime dynamic binary translation and optimization framework has been used to perform runtime instrumentation and dynamic data optimization to achieve this goal. This scheme not only ensures a reduced program execution time, but also results in lower cache access energy. Some of the commonly used benchmarks from the SPEC 2006 and SPLASH-2 suite were profiled to determine irregular data accesses from procedures which contributed heavily to the overall execution time. Wrappers built to replace these accesses with spatially adjacent data led to a significant improvement in the total execution time. On average, 20% reduction in time was achieved along with a 5% reduction in energy for SPEC 2006 and 11% reduction in time was achieved along with a 6% reduction in energy for SPLASH-2.