Near-data Processing Research Articles

NDPGNN: A Near-Data Processing Architecture for GNN Training and Inference Acceleration

NDPGNN: A Near-Data Processing Architecture for GNN Training and Inference Acceleration

Nearest data processing in GPU

Memory wall is known as one of the most critical bottlenecks in processors, rooted in the long memory access delay. With the advent of emerging memory-intensive applications such as image processing, the memory wall problem has become even more critical. Near data processing (NDP) has been introduced as an astonishing solution where instead of moving data from the main memory, instructions are offloaded to the cores integrated with the main memory level. However, in NDP, instructions that are to be offloaded, are statically selected at the compilation time prior to run-time. In addition, NDP ignores the benefit of offloading instructions into the intermediate memory hierarchy levels. We propose Nearest Data Processing (NSDP) which introduces a hierarchical processing approach in GPU. In NSDP, each memory hierarchy level is equipped with processing cores capable of executing instructions. By analyzing the instruction status at run-time, NSDP dynamically decides whether an instruction should be offloaded to the next level of memory hierarchy or be processed at the current level. Depending on the decision, either data is moved upward to the processing core or the instruction is moved downward to the data storage unit. With this approach, the data movement rate has been reduced, on average, by 47 % over the baseline. Consequently, NSDP has been able to improve the system performance, on average, by 37 % and reduce the power consumption, on average, by 18 %.

DANSEN: Database Acceleration on Native Computational Storage by Exploiting NDP

This paper introduces DANSEN , the hardware accelerator component for neoDBMS, a full-stack computational storage system designed to manage on-device execution of database queries/transactions as a Near-Data Processing (NDP)-operation. The proposed system enables Database Management Systems (DBMS) to offload NDP-operations to the storage while maintaining control over data through a native storage interface . DANSEN provides an NDP-engine that enables DBMS to perform both low-level database tasks, such as performing database administration, as well as high-level tasks like executing SQL, on the smart storage device while observing the DBMS concurrency control. Furthermore, DANSEN enables the incorporation of custom accelerators as an NDP-operation, e.g., to perform hardware-accelerated ML inference directly on the stored data. We built the DANSEN storage prototype and interface on an Ultrascale+HBM FPGA and fully integrated it with PostgreSQL 12. Experimental results demonstrate that the proposed NDP approach outperforms software-only PostgreSQL using a fast off-the-shelf NVMe drive, and significantly improves the end-to-end execution time of an aggregation operation (similar to Q6 from CH-benCHmark, 150 million records) by ≈ 10.6 ×. The versatility of the proposed approach is also validated by integrating a compute-intensive data analytics application with multi-row results, outperforming PostgreSQL by ≈ 1.5 ×.

Abakus: Accelerating k -mer Counting with Storage Technology

This work seeks to leverage Processing-with-storage-technology (PWST) to accelerate a key bioinformatics kernel called k -mer counting, which involves processing large files of sequence data on the disk to build a histogram of fixed-size genome sequence substrings and thereby entails prohibitively high I/O overhead. In particular, this work proposes a set of accelerator designs called Abakus that offer varying degrees of tradeoffs in terms of performance, efficiency, and hardware implementation complexity. The key to these designs is a set of domain-specific hardware extensions to accelerate the key operations for k -mer counting at various levels of the SSD hierarchy, with the goal of enhancing the limited computing capabilities of conventional SSDs, while exploiting the parallelism of the multi-channel, multi-way SSDs. Our evaluation suggests that Abakus can achieve 8.42×, 6.91×, and 2.32× speedup over the CPU-, GPU-, and near-data processing solutions.

Non-Invasive, Memory Access-Triggered Near-Data Processing for DNN Training Acceleration on GPUs

Non-Invasive, Memory Access-Triggered Near-Data Processing for DNN Training Acceleration on GPUs

TH-iSSD: Design and Implementation of a Generic and Reconfigurable Near-Data Processing Framework

We present the design and implementation of TH-iSSD, a near-data processing framework to address the data movement problem. TH-iSSD does not pose any restriction to the hardware selection and is highly reconfigurable—its core components, such as the on-device compute unit (e.g., FPGA, embedded CPUs) and data collectors (e.g., camera, sensors), can be easily replaced to adapt to different use cases. TH-iSSD achieves this goal by incorporating highly flexible computation and data paths. In the data path, TH-iSSD adopts an efficient device-level data switch that exchanges data with both host CPUs and peripheral sensors; it also enables direct accesses between the sensing, computation, and storage hardware components, which completely eliminates the redundant data movement overhead, and thus delivers both high performance and energy efficiency. In the computation path, TH-iSSD provides an abstraction offilestreamfor developers, which abstracts a collection of data along with the related computation task as a file. Since existing applications are familiar with POSIX-like interfaces, they can be ported on top of our platform with minimal code modification. Moreover, TH-iSSD also introduces mechanisms includingpipelined near-data processingandpriority-aware I/O schedulingto make TH-iSSD perform more effectively. We deploy TH-iSSD to accelerate two types of applications: the content-based information retrieval system and the edge zero-streaming system. Our experimental results show that TH-iSSD achieves up to 1.6× higher throughput and 36% lower latency thancompute-centricdesigns.

An Asynchronous Compaction Acceleration Scheme for Near-Data Processing-enabled LSM-Tree-based KV Stores

LSM-tree-based key-value stores (KV stores) convert random-write requests to sequence-write ones to achieve high I/O performance. Meanwhile, compaction operations in KV stores update SSTables in forms of reorganizing low-level data components to high-level ones, thereby guaranteeing an orderly data layout in each component. Repeated writes caused by compaction ( a.k.a, write amplification) impacts I/O bandwidth and overall system performance. Near-data processing (NDP) is one of effective approaches to addressing this write-amplification issue. Most NDP-based techniques adopt synchronous parallel schemes to perform a compaction task on both the host and its NDP-enabled device. In synchronous parallel compaction schemes, the execution time of compaction is determined by a subsystem that has lower compaction performance coupled by under-utilized computing resources in a NDP framework. To solve this problem, we propose an asynchronous parallel scheme named PStore to improve the compaction performance in KV stores. In PStore, we designed a multi-tasks queue and three priority-based scheduling methods. PStore elects proper compaction tasks to be offloaded in host- and device-side compaction modules. Our proposed cross-leveled compaction mechanism mitigates write amplification induced by asynchronous compaction. PStore featured with the asynchronous compaction mechanism fully utilizes computing resources in both host and device-side subsystems. Compared with the two popular synchronous compaction modes based on KV stores (TStore and LevelDB), our PStore immensely improves the throughput by up to a factor of 14 and 10.52 with an average of a factor of 2.09 and 1.73, respectively.

EMS-i : An Efficient Memory System Design with Specialized Caching Mechanism for Recommendation Inference

Recommendation systems have been widely embedded into many Internet services. For example, Meta’s deep learning recommendation model (DLRM) shows high prefictive accuracy of click-through rate in processing large-scale embedding tables. The SparseLengthSum (SLS) kernel of the DLRM dominates the inference time of the DLRM due to intensive irregular memory accesses to the embedding vectors. Some prior works directly adopt near data processing (NDP) solutions to obtain higher memory bandwidth to accelerate SLS. However, their inferior memory hierarchy induces low performance-cost ratio and fails to fully exploit the data locality. Although some software-managed cache policies were proposed to improve the cache hit rate, the incurred cache miss penalty is unacceptable considering the high overheads of executing the corresponding programs and the communication between the host and the accelerator. To address the issues aforementioned, we propose EMS-i , an efficient memory system design that integrates Solide State Drive (SSD) into the memory hierarchy using Compute Express Link (CXL) for recommendation system inference. We specialize the caching mechanism according to the characteristics of various DLRM workloads and propose a novel prefetching mechanism to further improve the performance. In addition, we delicately design the inference kernel and develop a customized mapping scheme for SLS operation, considering the multi-level parallelism in SLS and the data locality within a batch of queries. Compared to the state-of-the-art NDP solutions, EMS-i achieves up to 10.9× speedup over RecSSD and the performance comparable to RecNMP with 72% energy savings. EMS-i also saves up to 8.7× and 6.6 × memory cost w.r.t. RecSSD and RecNMP, respectively.

FSR: A host-storage collaborative mechanism for data path optimization of NDP operations

Emerging computational SSDs are becoming attractive solutions to boost the energy efficiency and performance of data-intensive applications. Such computational SSDs take advantage of the in-storage Near-Data Processing (NDP) architecture that offloads computing tasks to the storage. However, existing NDP-based applications still depend on the host to search the requested file of NDP tasks, which incurs large overhead for moving data between the host and the storage. In this paper, we propose to optimize the data path of NDP-based applications by a host-storage collaborative mechanism called File Semantics Retriever (FSR). The key idea of FSR is to realize the file system layout and the metadata structures in the storage with the collaboration of a library in the host and a handler in the firmware of the computational SSD. FSR can directly locate and fetch the requested file data in the computational SSD. Moreover, we construct a performance model to understand the suitable scenarios of FSR and the appropriate timing for using FSR. We implement and evaluate FSR on a real computational SSD platform. Extensive experimental results show that FSR can reduce the execution time of real-world NDP operations by 54.0% over existing computational SSDs relying on the host-based data path.

A review on computational storage devices and near memory computing for high performance applications

The von Neumann bottleneck is imposed due to the explosion of data transfers and emerging data-intensive applications in heterogeneous system architectures. The conventional computation approach of transferring data to CPU is no longer suitable especially with the cost it imposes. Given the increasing storage capacities, moving extensive data volumes between storage and computation cannot scale up. Hence, high-performance data processing mechanisms are needed, which may be achieved by bringing computation closer to data. Gathering insights where data is stored helps deal with energy efficiency, low latency, as well as security. Storage bus bandwidth is also saved when only computation results are delivered to the host memory. Various applications, including database acceleration, machine learning, Artificial Intelligence (AI), offloading (compression/encryption/encoding) and others can perform better and become more scalable if the “move process to data” paradigm is applied. Embedding processing engines inside Solid-State Drives (SSDs), transforming them to Computational Storage Devices (CSDs), provides the needed data processing solution. In this paper, we review the prior art on Near Data Processing (NDP) with focus on In-Storage Computing (ISC), identifying main challenges and potential gaps for future research directions.

ALP: Alleviating CPU-Memory Data Movement Overheads in Memory-Centric Systems

Recent advances in memory technology have enabled near-data processing (NDP) to tackle main memory bottlenecks in modern systems. Prior works partition applications into segments (e.g., instructions, loops, functions) and execute memory-bound segments of the applications on NDP computation units, while mapping the cache-friendly application segments to host CPU cores that access a deeper cache hierarchy. Partitioning applications between NDP and host cores causes inter-segment data movement overhead, which is the overhead from moving data generated from one segment and used in the consecutive segments. This overhead can be large if the segments map to cores in different parts of the system (i.e., host and NDP). Prior works take two approaches to the inter-segment data movement overhead when partitioning applications between NDP and host cores. The first class of works maps segments to NDP or host cores based on the properties of each segment, neglecting the performance impact of the inter-segment data movement. Such partitioning techniques suffer from inter-segment data movement overhead. The second class of works maps segments to host or NDP cores based on the overall memory bandwidth savings of each segment (which depends on the memory bandwidth savings within each segment and the inter-segment data movement overhead between other segments). These works do not offload each segment to the best-fitting core if they incur high inter-segment data movement overhead. Therefore these works miss some of the potential NDP performance benefits. We show that mapping each segment (here basic block) to its best-fitting core based on the properties of each segment, assuming no inter-segment data movement, can provide substantial performance benefits. However, we show that the inter-segment data movement reduces this benefit significantly. To this end, we introduce ALP, a new programmer-transparent technique to leverage the performance benefits of NDP by <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alleviating</i> the performance impact of inter-segment data movement between host and memory and enabling efficient partitioning of applications between host and NDP cores. ALP alleviates the inter-segment data movement overhead by <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">proactively and accurately</i> transferring the required data between the segments mapped on host and NDP cores. This is based on the key observation that the instructions that generate the inter-segment data stay the same across different executions of a program on different input sets. ALP uses a compiler pass to identify these instructions and uses specialized hardware support to transfer data between the host and NDP cores at runtime. Using both the compiler and runtime information, ALP efficiently maps application segments to either host or NDP cores considering 1) the properties of each segment, 2) the inter-segment data movement overhead between different segments, and 3) whether this inter-segment data movement overhead can be alleviated proactively and in a timely manner. We evaluate ALP across a wide range of workloads and show on average 54.3% and 45.4% speedup compared to executing the application only on the host CPU or only the NDP cores, respectively.

Computational CXL-Memory Solution for Accelerating Memory-Intensive Applications

CXL interface is the up-to-date technology that enables effective memory expansion by providing a memory-sharing protocol in configuring heterogeneous devices. However, its limited physical bandwidth can be a significant bottleneck for emerging data-intensive applications. In this work, we propose a novel CXL-based memory disaggregation architecture with a real-world prototype demonstration, which overcomes the bandwidth limitation of the CXL interface using near-data processing. The experimental results demonstrate that our design achieves up to 1.9× better performance/power efficiency than the existing CPU system.

XLA-NDP: Efficient Scheduling and Code Generation for Deep Learning Model Training on Near-Data Processing Memory

Deep learning (DL) model training must address the memory bottleneck to continue scaling. Processing-in-memory approaches can be a viable solution as they move computations near or into the memory, reducing substantial data movement. However, to deploy applications on such hardware, end-to-end software support is crucial for efficient computation mapping and scheduling as well as extensible code generation, but no consideration has been made for DL training workloads. In this paper, we propose XLA-NDP, a compiler and runtime solution for NDPX, a near-data processing (NDP) architecture integrated with an existing DL training framework. XLA-NDP offloads NDPX kernels and schedules them to overlap with GPU kernels to maximize parallelism based on GPU and NDPX costs, while providing a template-based code generator with low-level optimizations. The experiments showed that XLA-NDP provides up to 1.41x speedup (1.24x on average) over the GPU baseline for four DL model training.

Horae: A Hybrid I/O Request Scheduling Technique for Near-Data Processing-Based SSD

Near-data processing (NDP) architecture is promised to break the bottleneck of data movement in many scenarios (e.g., databases and recommendation systems), which limits the efficiency of data processing. Different from traditional SSD, NDP-based SSD not only needs to handle normal I/Os (e.g., read and write), but also needs to handle NDP requests that contain data processing operations. NDP and normal I/O requests share some function units of NDP-based SSD, such as flash chips and embedded processors. However, existing works ignore the resource competition between normal I/Os and NDP requests, which drastically degrades the performance. In this article, we propose a novel scheduling technique called Horae, which can efficiently schedule hybrid NDP-normal I/O requests in NDP-based SSD to improve performance. Horae exploits the critical paths on critical resources to maximize the parallelism of multiple stages of requests. The experimental results on typical workloads show that Horae can significantly improve the performance of hybrid NDP-normal I/O requests over the state-of-the-art scheduling algorithms of NDP-based SSDs.

INVMFS: An Efficient File System for NVRAM-Based Intermittent Computing Devices

Developing toward battery-less, energy-harvesting designs is a promising direction for sensor-scale devices. The recent introduction of byte-addressable, nonvolatile memory (NVRAM) enables intermittent computing, which preserves as much program progress across power interruptions as possible using fine-grained checkpoints. Sensing applications strongly demand local storage for near-data processing, and there is no exception for intermittent computing devices. While checkpointing on program progress is being studied recently, there is little work regarding how sensor file systems support intermittent computing. On power recovery, although the program progress is reverted to the latest checkpoint, the storage state stays at the instant of power loss. To remedy this problem, we present a lightweight file system for efficient operations on files and checkpoints. Our design exploits the byte addressability of NVRAM and uses as much in-place byte writing as possible while guaranteeing the safety of checkpoint operations. We evaluated our design against two prior checkpoint-enabled file systems, and results show that on average, our design reduced the total time overhead and energy consumption by 77% and 78%, respectively.

GCIM: Toward Efficient Processing of Graph Convolutional Networks in 3D-Stacked Memory

Graph convolutional networks (GCNs) have become a powerful deep learning approach for graph-structured data. Different from traditional neural networks such as convolutional neural networks, GCNs handle irregular input graph data, and GCNs are both computation-bound and memory-bound. How to efficiently utilize the underlying computation and memory resource becomes a critical issue. The emerging 3D-stacked computation-in-memory (CIM) architecture can reduce the data movement between computing logic and memory, thereby presenting a promising solution for the processing of GCNs. An unsolved key challenge is how to allocate GCNs to take advantage of fast near-data processing of the 3D-stacked CIM architecture. This article presents GCIM, a software–hardware co-design approach to exploit the efficient processing of GCNs on the CIM architecture. At the level of hardware design, GCIM integrates lightweight computing units near memory banks to fully exploit bank-level bandwidth and parallelism. At the level of software design, a locality-aware data mapping algorithm is proposed to partition the input graph and achieve workload balancing. GCIM is evaluated through a set of representative GCN models and standard graph datasets. The experimental results show that GCIM can significantly reduce the processing latency and data movement overhead compared with representative schemes.

NDP-RANK: Prediction and ranking of NDP systems performance using machine learning

The near-data processing (NDP) paradigm has recently gained popularity as a promising method for mitigating the memory wall challenges of future computing systems. Modern 3D-stacked DRAM memories can be leveraged to prevent unnecessary data movement between the main memory and the CPU. However, identifying which type of NDP system, in terms of processing elements and the type of high performance 3D memory used, is suitable for a given application is not a trivial task. Understanding the interactions between modern workloads and the memory subsystem requires a lot of effort. Each memory type presents its unique advantages and shortcomings. In addition, memory access patterns vary greatly across applications. Therefore, the performance of a given application on a given memory type is difficult to intuitively predict. There is no specific NDP system or memory type that can provide high performance for every application.We propose NDP-RANK, a machine learning framework that can efficiently decide which NDP system is suitable for an application. NDP-RANK’s performance prediction is based on an input set of application and microarchitecture characteristics. We build machine learning models that can be used to conduct design-space exploration of NDP systems and accurately predict performance of previously unseen applications on several NDP architectures. Our models are orders of magnitude faster than architectural simulation. They can accurately predict performance with coefficients of determination ranging between 0.89 and 0.96, and root mean square errors ranging between 0.04 and 0.16.

A storage computing architecture with multiple NDP devices for accelerating compaction performance in LSM-tree based KV stores

In the era of big data, log-structured merge-tree (LSM-tree) based key–value stores (KV stores) have become an effective alternative to the traditional relational database owing to their high performance and scalability. KV stores are widely used to manage big unstructured data for data-intensive applications. A compaction operation is triggered to sort data and compact space, but this can degrade write performance, causing write amplification to increase. The compaction process results in extensive data movement and computation. Recently, near-data processing (NDP) models, which place computation close to data sources, aiming to reduce the cost of data movement between the device and the host, have been widely applied to optimize the performance of KV stores. The existing researches on NDP-based KV stores mainly focus on a single NDP-enabled storage device (NDP device for short). In this paper, we present a storage computing architecture with multiple NDP devices named MStore for accelerating compaction performance in LSM-tree based KV stores. MStore can achieve high performance owing to its storage and computing convergence. It employs a multi-column LSM-tree data layout based on key ranges. Compaction tasks on multiple NDP devices are performed in parallel, improving compaction performance. MStore increases the width of the LSM-tree level to reduce the depth, which reduces the execution time and write amplification of compaction. Following the data organization, we design load balancing mechanisms, including storage load balancing by key range adjustment and computation load balancing across compaction tasks. In addition, we develop a batch read interface and a greedy scheduling scheme for user I/Os. MStore aims to fully utilize the computing and I/O resources to boost the compaction performance in KV stores. Compared with single-device NDP systems (such as TStore), MStore with four NDP devices achieves 3.88× speedup under random-write DB_Bench. When running YCSB-C with a random-write workload, MStore is 4.32× faster than TStore. The performance of read and scan on MStore is improved by 2.23× and 2.18×, respectively, compared with TStore. These results have strong implications for the implementations of MStore, and their use as replacements for single NDP based KV stores.

GraNDe: Near-Data Processing Architecture With Adaptive Matrix Mapping for Graph Convolutional Networks

Graph Convolutional Network (GCN) models have attracted attention given their high accuracy in interpreting graph data. One of the primary building blocks of a GCN model is aggregation, which gathers and averages the feature vectors corresponding to the vertices adjacent to each individual vertex. Aggregation works by multiplying the adjacency and feature matrices. The size of both matrices exceeds the on-chip cache capacity, and the adjacency matrix is highly sparse. These lead to little data reuse and cause numerous main-memory accesses during the aggregation process. Thus, aggregation exhibits memory-intensive characteristics. We propose GraNDe, an NDP architecture that accelerates memory-intensive aggregation operations by locating processing elements near the DRAM datapath to exploit rank-level parallelism. By exploring the data mapping of the operand matrices to DRAM ranks, we discover that the optimal mapping differs depending on the configuration of a specific GCN layer. With our optimal layer-by-layer mapping scheme, GraNDe shows a speedup up to 4.3× compared to the baseline system on open-graph benchmark datasets.

Near-data processing in database systems on native computational storage under HTAP workloads

Today's Hybrid Transactional and Analytical Processing (HTAP) systems, tackle the ever-growing data in combination with a mixture of transactional and analytical workloads. While optimizing for aspects such as data freshness and performance isolation, they build on the traditional data-to-code principle and may trigger massive cold data transfers that impair the overall performance and scalability. Firstly, in this paper we show that Near-Data Processing (NDP) naturally fits in the HTAP design space. Secondly, we propose an NDP database architecture, allowing transactionally consistent in-situ executions of analytical operations in HTAP settings. We evaluate the proposed architecture in state-of-the-art key/value-stores and multi-versioned DBMS. In contrast to traditional setups, our approach yields robust, resource- and cost-efficient performance.