Memory Scheduling Research Articles

Parallel implementation for real-time visual SLAM systems based on heterogeneous computing

Simultaneous localization and mapping has become rapidly developed and plays an indispensable role in intelligent vehicles. However, many state-of-the-art visual simultaneous localization and mapping (VSLAM) frameworks are very time-consuming both in front-end and back-end, especially for large-scale scenes. Nowadays, the increasingly popular use of graphics processors for general-purpose computing, and the progressively mature high-performance programming theory based on compute unified device architecture (CUDA) have given the possibility for large-scale VSLAM to solve the conflict between limited computing power and excessive computing tasks. The paper proposes a full-flow optimal parallelization scheme based on heterogeneous computing to speed up the time-consuming modules in VSLAM. Firstly, a parallel strategy for feature extraction and matching is designed to reduce the time consumption arising from multiple data transfers between devices. Secondly, a bundle adjustment method based solely on CUDA is developed. By fully optimizing memory scheduling and task allocation, a large increase in speed is achieved while maintaining accuracy. Besides, CUDA heterogeneous acceleration is fully utilized for tasks such as error computation and linear system construction in the VSLAM back-end to enhance the operation speed. Our proposed method is tested on numerous public datasets on both computer and embedded sides, respectively. A number of qualitative and quantitative experiments are performed to verify its superiority in terms of speed compared to other states-of-the-art.

FPGA-based tiling scheme and on-chip memory scheduling scheme for multi-branch semantic segmentation neural network accelerator

Abstract According to the characteristics of a multi-branch semantic segmentation neural network with a large amount of intermediate data during the inference process, we reduce the requirement of off-chip memory access and the redundant calculation due to tiling through better blocking scheme, better on-chip memory partitioning scheme, and on-chip memory scheduling scheme, improving the energy efficiency to 77.4 GOPS/W.

PatternS: An intelligent hybrid memory scheduler driven by page pattern recognition

Hybrid memory systems integrate a variety of memory technologies, effectively expanding the main memory capacity to meet the demands of emerging big data applications. Hybrid memory systems exhibit disparities in their heterogeneous memory components’ access speeds. Dynamic page scheduling to ensure memory access predominantly occurs in the faster memory components is essential for optimizing the performance of hybrid memory systems. Traditional history schedulers are unable to predict irregular memory accesses. Therefore, recent works attempt to optimize page scheduling by predicting their hotness using neural network models. However, they face two crucial challenges: one is the page explosion problem caused by the massive number of pages and the other is the new pages problem due to shifting memory access regions over time. To address these two challenges, we propose PatternS, an intelligent hybrid memory scheduler driven by page pattern recognition. Based on the insight into the similarities between memory access patterns, we proposed a Page Pattern Recognizer to identify pages with similar patterns and manage them as groups. PatternS is also capable of categorizing new pages into pre-identified patterns using short-term access information, enabling them to be predicted by the trained model. Experimental results demonstrate that our approach outperforms state-of-the-art intelligent schedulers regarding effectiveness and cost.

Design and Implementation of a Highly Efficient Quasi-Cyclic Low-Density Parity-Check Transceiving System Using an Overlapping Decoder.

The traditional LDPC encoding and decoding system is characterized by low throughput and high resource consumption, making it unsuitable for use in cost-efficient, energy-saving sensor networks. Aiming to optimize coding complexity and throughput, this paper proposes a combined design of a novel LDPC code structure and the corresponding overlapping decoding strategies. With regard to structure of LDPC code, a CCSDS-like quasi-cyclic parity check matrix (PCM) with uniform distribution of submatrices is constructed to maximize overlap depth and adapt the parallel decoding. In terms of reception decoding strategies, we use a modified 2-bit Min-Sum algorithm (MSA) that achieves a coding gain of 5 dB at a bit error rate of 10-6 compared to an uncoded BPSK, further mitigating resource consumption, and which only incurs a slight loss compared to the standard MSA. Moreover, a shift-register-based memory scheduling strategy is presented to fully utilize the quasi-cyclic characteristic and shorten the read/write latency. With proper overlap scheduling, the time consumption can be reduced by one third per iteration compared to the non-overlap algorithm. Simulation and implementation results demonstrate that our decoder can achieve a throughput up to 7.76 Gbps at a frequency of 156.25 MHz operating eight iterations, with a two-thirds resource consumption saving.

New YARN sharing GPU based on graphics memory granularity scheduling

As one of the most widely used cluster scheduling frameworks, Hadoop YARN only supported CPU and memory scheduling in the past. Furthermore, due to the widespread use of AI, the demand for GPU is also increasing. So Hadoop YARN V3.0 adds GPU scheduling, but the granularity is on the whole card yet, rather than finer-grained graphics memory scheduling. However, during daily training, although the graphics memory required by tasks may be much smaller than the whole GPU card, they will occupy the whole card, which results in wasted resources. To address this issue, Tensorflow provides the API for graphics memory control. Therefore, we propose to introduce this feature into Hadoop YARN so that it can support the heterogeneous scheduling: CPU, memory and graphics memory. Then we take HadoopV2.7 source code as the underlying architecture and design a new scheduler GSHARE. Compared with previous scheduling strategies, with 3 nodes, 3 GPU cards per node, and 12G graphics memory per card, GSHARE improves efficiency by up to 74% for Tensorflow tasks with 2G of graphics memory. Meanwhile, it minimizes the problem of wasted graphics memory caused by the inability to control graphics memory proportionally by the API of Tensorflow for multiple-card.

A perceptual and predictive batch-processing memory scheduling strategy for a CPU-GPU heterogeneous system

A perceptual and predictive batch-processing memory scheduling strategy for a CPU-GPU heterogeneous system

An Application-oblivious Memory Scheduling System for DNN Accelerators

Deep Neural Networks (DNNs) tend to go deeper and wider, which poses a significant challenge to the training of DNNs, due to the limited memory capacity of DNN accelerators. Existing solutions for memory-efficient DNN training are densely coupled with the application features of DNN workloads, e.g., layer structures or computational graphs of DNNs are necessary for these solutions. This would result in weak versatility for DNNs with sophisticated layer structures or complicated computation graphs. These schemes usually need to be re-implemented or re-adapted due to the new layer structures or the unusual operators in the computational graphs introduced by these DNNs. In this article, we review the memory pressure issues of DNN training from the perspective of runtime systems and model the memory access behaviors of DNN workloads. We identify the iterative, regularity , and extremalization properties of memory access patterns for DNN workloads. Based on these observations, we propose AppObMem, an application-oblivious memory scheduling system. AppObMem automatically traces the memory behaviors of DNN workloads and schedules the memory swapping to reduce the memory pressure of the device accelerators without the perception of high-level information of layer structures or computation graphs. Evaluations on a variety of DNN models show that, AppObMem obtains 40–60% memory savings with acceptable performance loss. AppObMem is also competitive with other open sourced SOTA schemes.

Software-Level Memory Regulation to Reduce Execution Time Variation on Multicore Real-Time Systems

Modern real-time embedded systems are equipped with multi-core processors to execute computationally intensive tasks. In multi-core architecture, last-level cache memory is shared by cores. The shared cache becomes a non-deterministic resource, which affects the independent execution of real-time tasks. We propose a solution to remedy a variation in execution time when interference happens in a shared cache. Current solutions have relied on memory scheduling approaches that avoid concurrent memory access to guarantee deterministic execution time. However, these methods required complex analysis to accurately estimate the worst-case execution time and to schedule tasks in an overly conservative manner. Unlike existing works, the proposed method prevents simultaneous memory access using the side effect of memory barriers rather than the complicated analysis. A memory barrier is inserted based on a simple code analysis that is performed in units of basic blocks using the LLVM compiler. The proposed method not only does not require the modification of the operating system or task execution flow but also relatively shows fast analysis time. Experimental results show that the proposed basic block-based memory barrier insertion method can reduce the variation in execution time by up to 80% when interference occurs.

A Distributed Edge-Based Scheduling Technique with Low-Latency and High-Bandwidth for Existing Driver Profiling Algorithms

The gradual increase in latency-sensitive, real-time applications for embedded systems encourages users to share sensor data simultaneously. Streamed sensor data have deficient performance. In this paper, we propose a new edge-based scheduling method with high-bandwidth for decreasing driver-profiling latency. The proposed multi-level memory scheduling method places data in a key-value storage, flushes sensor data when the edge memory is full, and reduces the number of I/O operations, network latency, and the number of REST API calls in the edge cloud. As a result, the proposed method provides significant read/write performance enhancement for real-time embedded systems. In fact, the proposed application improves the number of requests per second by 3.5, 5, and 4 times, respectively, compared with existing light-weight FCN-LSTM, FCN-LSTM, and DeepConvRNN Attention solutions. The proposed application also improves the bandwidth by 5.89, 5.58, and 4.16 times respectively, compared with existing light-weight FCN-LSTM, FCN-LSTM, and DeepConvRNN Attention solutions.

Research and Design of Open Convolutional Neural Network Based on FPGA

In this paper, using the fluidity and parallelism of FPGA, we design a set of memory scheduling mechanism to implement an open convolutional neural network. The self-built vehicle database and the cropped and zoomed Daimler Pedestrian Detection Benchmark dataset are each trained under different network structures, and their recognition rates reached 97.8% and 98.6%. An open feedforward network is implemented on the FPGA platform, and experiments show that for small convolutional networks with different structures, the FPGA platform can increase its recognition speed when resources permit.

Energy Reduction Through Memory Aware Real-Time Scheduling on Virtual Machine in Multi-Cores Server

Not only weighty energy usage pose issues for the environment, but it also raises server maintenance costs in data centers. The massive task with the various power control functions in computer components was made to minimize energy consumption. Increasing consumption of energy in data server environments means that data centers will have high maintenance costs. Various geo-distributed data centers are starting to grow in an age of data proliferation and information growth. Energy management for servers is now demanded for technological, environmental, and economic reasons. In this environment, the main memory is a major energy consumer, not less than the processor. At the same time, an energy-efficient task scheduling strategy is a viable way to meet these goals. Unfortunately, mapping Virtual Machine (VM) resources to the Main Memory (MM) demands to achieve good performance by minimizing the energy consumption within a certain limit is a huge challenge. This paper simulates energy-efficient task scheduling algorithms in a heterogeneous virtualized environment using real-time virtual machine scheduling to resolve the issue of energy consumption. Using a simulator Real-Time system SIMulator (RTSIM), several hardware-based scheduling algorithms are implemented to observe VM memory scheduling efficiency to save memory energy. The simulation results show that, compared to current energy-efficient scheduling methods Rate Monotonic (RM), Earliest-Deadline-First (EDF), and Least-Laxity-First (LLF), helps to reduce energy consumption and improve performance. It is also observed that memory-aware energy management architecture reduces energy and memory consumption efficiently by using EDF scheduling algorithms. In particular, EDF saves approximately 58.3 percent of memory energy than conventional systems that cannot benefit from memory-aware energy management algorithms. The energy efficiency of the algorithms continues to improve as the level of server consolidation rises. We also implemented the EDF scheduling algorithm in Xen’s Credit scheduler to see if the simulation outcomes can be simulated on physical systems. Results of simulation and deployment are equated, and comparable outcomes are achieved. We also identified that shared memory between virtual machines deliberately affects memory’s energy consumption based on the implementation.

Approximate NoC and Memory Controller Architectures for GPGPU Accelerators

High interconnect bandwidth is crucial for achieving better performance in many-core GPGPU architectures that execute highly data parallel applications. The parallel warps of threads running on shader cores generate a high volume of read requests to the main memory due to the limited size of data caches at the shader cores. This leads to a scenarios with rapid arrival of an even larger volume of reply data from the DRAM, which creates a bottleneck at memory controllers (MCs) that send reply packets back to the requesting cores over the network-on-chip (NoC). Coping with such high volumes of data requires intelligent memory scheduling and innovative NoC architectures. To mitigate memory bottlenecks in GPGPUs, we first propose a novel approximate memory controller architecture ( AMC ) that reduces the DRAM latency by opportunistically exploiting row buffer locality and bank level parallelism in memory request scheduling, and leverages approximability of the reply data from DRAM, to reduce the number of reply packets injected into the NoC. To further realize high throughput and low energy communication in GPGPUs, we propose a low power, approximate NoC architecture ( Dapper ) that increases the utilization of the available network bandwidth by using single cycle overlay circuits for the reply traffic between MCs and shader cores. Experimental results show that Dapper and AMC together increase NoC throughput by up to 21 percent; and reduce NoC latency by up to 45.5 percent and energy consumed by the NoC and MC by up to 38.3 percent, with minimal impact on output accuracy, compared to state-of-the-art approximate NoC/MC architectures.

Design and Implementation of N-Point FFT Processor for MIMO-OFDM Systems using Radix N

This paper presents Single-path Delay Feedback (SDF) architecture for implementing Fast Fourier Transform (FFT) for Multiple-Input Multiple-Output Orthogonal Frequency Division Frequency Multiplexing (MIMO-OFDM). The architecture of Single-path Delay Feedback and memory scheduling are the basic concepts used to implement the FFT processor with variable length. Depending on the SDF architecture, we implement the FFT processor-based design which is proposed in this paper. In this paper, we use MIMOOFDM high data rates, high efficiency and high throughput. In this paper, we use radix-4 algorithm to implement the sequence because the speed of the operation is high. The functionality verification and the synthesis are carried out by using XLINIX.14.2.

Improving Memory Efficiency in Heterogeneous MPSoCs through Row-Buffer Locality-aware Forwarding

In heterogeneous multicore systems, the memory subsystem plays a critical role, since most core-to-core communications are conducted through the main memory. Memory efficiency has a substantial impact on system performance. Although memory traffic from multimedia cores generally manifests high row-buffer locality, which is beneficial to memory efficiency, the locality is often lost as memory streams are forwarded through networks-on-chip (NoC). Previous studies have discussed the techniques that improve memory visibility to reveal scattered row-buffer hit opportunities to the memory scheduler. However, extending local memory visibility introduces little benefit after the locality has been severely diluted. As the alternative approach, preserving row-buffer locality in the NoC has not been well explored. What is worse, it remains to be studied how to perform network traffic scheduling with the awareness of both memory efficiency and quality-of-service (QoS). In this article, we propose a router design with embedded row-index caches to enable locality-aware packet forwarding. The proposed design requires minor modifications to existing router microarchitecture and can be easily implemented with priority arbiters to integrate QoS support. Extensive evaluations show that the proposed design achieves higher memory efficiency than prior memory-aware routers, in addition to providing QoS support. On basis of extant QoS-aware routers, locality-aware forwarding helps to increase row-buffer hits by 58.32% and reduce memory latency by 14.45% on average. It also introduces a net reduction in DRAM and NoC energy cost by 27.82%.

A memory scheduling strategy for eliminating memory access interference in heterogeneous system

Multiple CPUs and GPUs are integrated on the same chip to share memory, and access requests between cores are interfering with each other. Memory requests from the GPU seriously interfere with the CPU memory access performance. Requests between multiple CPUs are intertwined when accessing memory, and its performance is greatly affected. The difference in access latency between GPU cores increases the average latency of memory accesses. In order to solve the problems encountered in the shared memory of heterogeneous multi-core systems, we propose a step-by-step memory scheduling strategy, which improve the system performance. The step-by-step memory scheduling strategy first creates a new memory request queue based on the request source and isolates the CPU requests from the GPU requests when the memory controller receives the memory request, thereby preventing the GPU request from interfering with the CPU request. Then, for the CPU request queue, a dynamic bank partitioning strategy is implemented, which dynamically maps it to different bank sets according to different memory characteristics of the application, and eliminates memory request interference of multiple CPU applications without affecting bank-level parallelism. Finally, for the GPU request queue, the criticality is introduced to measure the difference of the memory access latency between the cores. Based on the first ready-first come first served strategy, we implemented criticality-aware memory scheduling to balance the locality and criticality of application access.

An Efficient Streaming Accelerator for Low Bit-Width Convolutional Neural Networks

Convolutional Neural Networks (CNNs) have been widely applied in various fields, such as image recognition, speech processing, as well as in many big-data analysis tasks. However, their large size and intensive computation hinder their deployment in hardware, especially on the embedded systems with stringent latency, power, and area requirements. To address this issue, low bit-width CNNs are proposed as a highly competitive candidate. In this paper, we propose an efficient, scalable accelerator for low bit-width CNNs based on a parallel streaming architecture. With a novel coarse grain task partitioning (CGTP) strategy, the proposed accelerator with heterogeneous computing units, supporting multi-pattern dataflows, can nearly double the throughput for various CNN models on average. Besides, a hardware-friendly algorithm is proposed to simplify the activation and quantification process, which can reduce the power dissipation and area overhead. Based on the optimized algorithm, an efficient reconfigurable three-stage activation-quantification-pooling (AQP) unit with the low power staged blocking strategy is developed, which can process activation, quantification, and max-pooling operations simultaneously. Moreover, an interleaving memory scheduling scheme is proposed to well support the streaming architecture. The accelerator is implemented with TSMC 40 nm technology with a core size of 0.17 mm 2 . It can achieve 7.03 TOPS/W energy efficiency and 4.14 TOPS/mm 2 area efficiency at 100.1 mW, which makes it a promising design for the embedded devices.

Harvesting Row-Buffer Hits via Orchestrated Last-Level Cache and DRAM Scheduling for Heterogeneous Multicore Systems

In heterogeneous multicore systems, the memory subsystem, including the last-level cache and DRAM, is widely shared among the CPU, the GPU, and the real-time cores. Due to their distinct memory traffic patterns, heterogeneous cores result in more frequent cache misses at the last-level cache. As cache misses travel through the memory subsystem, two schedulers are involved for the last-level cache and DRAM, respectively. Prior studies treated the scheduling of the last-level cache and DRAM as independent stages. However, with no orchestration and limited visibility of memory traffic, neither scheduling stage is able to ensure optimal scheduling decisions for memory efficiency. Unnecessary precharges and row activations happen in DRAM when the memory scheduler is ignorant of incoming cache misses, and DRAM row-buffer states are invisible to the last-level cache. In this article, we propose a unified memory controller for the the last-level cache and DRAM with orchestrated schedulers. The memory scheduler harvests row-buffer hit opportunities in cache request buffers during spare time without inducing significant implementation cost. We further introduce a dynamic orchestrated scheduling policy to improve memory efficiency while achieving target CPU IPC. Extensive evaluations show that the proposed controller improves the total memory bandwidth of DRAM by 16.8% on average and saves DRAM energy by up to 29.7% while achieving comparable CPU IPCs. With the dynamic scheduling policy, the unified controller achieves the same IPC as the conventional design and increases DRAM bandwidth by 9.2%. In addition, we explore the potential of the proposed memory controller to attain improvements on both memory bandwidth and CPU IPC.

Adaptive TB‐LMI: An efficient memory controller and scheduler design

SummaryIn the modern multi‐core systems, concurrently executing applications share common resource such as main memory. Memory scheduling algorithms are developed to resolve memory contention among competing applications so that throughput is high and fairness of the overall multi‐core system is guaranteed. In this paper, we present Adaptive Time‐based Least Memory Intensive (Adaptive TB‐LMI) scheduling, a new memory scheduling algorithm that addresses both fairness and system performance. Adaptive TB‐LMI is based on TB‐LMI which prioritizes applications according to their memory contention every pre‐defined CPU cycle. Adaptive TB‐LMI dynamically prioritizes applications according to their memory contention. Considering the previous algorithms with the best performance, for 16‐core system, TB‐LMI improves system throughput on average by 2.25X and 36% comparing to FCFS and TCM respectively. Adaptive TB‐LMI is 6% better than the TB‐LMI with static threshold. In terms of fairness and slowdown metrics, TB‐LMI show improvements of 30% and 3X, respectively, compared to FCFS and, 18% and 8%, respectively, compared to TCM. Adaptive TB‐LMI and TB‐LMI are 15% more efficient in energy‐delay product, although they are within 5% in terms of area overhead. Moreover, TCM has an area overhead of about 45% more than Adaptive TB‐LMI. In terms of Energy‐Delay Product, Adaptive TB‐LMI is 10% and 24% better than TB‐LMI and TCM, respectively. This is due to the dynamic capabilities of the adaptive algorithm to change the rate of the SQ, hence reducing the energy consumption.

Shared Last-Level Cache Management and Memory Scheduling for GPGPUs with Hybrid Main Memory

Memory intensive workloads become increasingly popular on general purpose graphics processing units (GPGPUs), and impose great challenges on the GPGPU memory subsystem design. On the other hand, with the recent development of non-volatile memory (NVM) technologies, hybrid memory combining both DRAM and NVM achieves high performance, low power, and high density simultaneously, which provides a promising main memory design for GPGPUs. In this article, we explore the shared last-level cache management for GPGPUs with consideration of the underlying hybrid main memory. To improve the overall memory subsystem performance, we exploit the characteristics of both the asymmetric read/write latency of the hybrid main memory architecture, as well as the memory coalescing feature of GPGPUs. In particular, to reduce the average cost of L2 cache misses, we prioritize cache blocks from DRAM or NVM based on observations that operations to NVM part of main memory have a large impact on the system performance. Furthermore, the cache management scheme also integrates the GPU memory coalescing and cache bypassing techniques to improve the overall system performance. To minimize the impact of memory divergence behaviors among simultaneously executed groups of threads, we propose a hybrid main memory and warp aware memory scheduling mechanism for GPGPUs. Experimental results show that in the context of a hybrid main memory system, our proposed L2 cache management policy and memory scheduling mechanism improve performance by 15.69% on average for memory intensive benchmarks, whereas the maximum gain can be up to 29% and achieve an average memory subsystem energy reduction of 21.27%.

MASK

Graphics Processing Units (GPUs) exploit large amounts of threadlevel parallelism to provide high instruction throughput and to efficiently hide long-latency stalls. The resulting high throughput, along with continued programmability improvements, have made GPUs an essential computational resource in many domains. Applications from different domains can have vastly different compute and memory demands on the GPU. In a large-scale computing environment, to efficiently accommodate such wide-ranging demands without leaving GPU resources underutilized, multiple applications can share a single GPU, akin to how multiple applications execute concurrently on a CPU. Multi-application concurrency requires several support mechanisms in both hardware and software. One such key mechanism is virtual memory, which manages and protects the address space of each application. However, modern GPUs lack the extensive support for multi-application concurrency available in CPUs, and as a result suffer from high performance overheads when shared by multiple applications, as we demonstrate. We perform a detailed analysis of which multi-application concurrency support limitations hurt GPU performance the most. We find that the poor performance is largely a result of the virtual memory mechanisms employed in modern GPUs. In particular, poor address translation performance is a key obstacle to efficient GPU sharing. State-of-the-art address translation mechanisms, which were designed for single-application execution, experience significant inter-application interference when multiple applications spatially share the GPU. This contention leads to frequent misses in the shared translation lookaside buffer (TLB), where a single miss can induce long-latency stalls for hundreds of threads. As a result, the GPU often cannot schedule enough threads to successfully hide the stalls, which diminishes system throughput and becomes a first-order performance concern. Based on our analysis, we propose MASK, a new GPU framework that provides low-overhead virtual memory support for the concurrent execution of multiple applications. MASK consists of three novel address-translation-aware cache and memory management mechanisms that work together to largely reduce the overhead of address translation: (1) a token-based technique to reduce TLB contention, (2) a bypassing mechanism to improve the effectiveness of cached address translations, and (3) an application-aware memory scheduling scheme to reduce the interference between address translation and data requests. Our evaluations show that MASK restores much of the throughput lost to TLB contention. Relative to a state-of-the-art GPU TLB, MASK improves system throughput by 57.8%, improves IPC throughput by 43.4%, and reduces applicationlevel unfairness by 22.4%. MASK's system throughput is within 23.2% of an ideal GPU system with no address translation overhead.