Scheduling Overhead Research Articles

Gem5-NVDLA: A Simulation Framework for Compiling, Scheduling and Architecture Evaluation on AI System-on-Chips

Recent years have seen an increasing trend in designing AI accelerators together with the rest of the system, including CPUs and memory hierarchy. This trend calls for high-quality simulators or analytical models that enable such kind of co-exploration. Currently, the majority of such exploration is supported by AI accelerator analytical models. But such models usually overlook the non-trivial impact of congestion of shared resources, non-ideal hardware utilization and non-zero CPU scheduler overhead, which could only be modeled by cycle-level simulators. However, most simulators with full-stack toolchains are proprietary to corporations, and the few open-source simulators are suffering from either weak compilers or limited space of modeling. This framework resolves these issues by proposing a compilation and simulation flow to run arbitrary Caffe neural network models on the NVIDIA Deep Learning Accelerator (NVDLA) with gem5, a cycle-level simulator, and by adding more building blocks including scratchpad allocation, multi-accelerator scheduling, tensor-level prefetching mechanisms and a DMA-aided embedded buffer to map workload to multiple NVDLAs. The proposed framework has been tested and verified on a set of convolution neural networks, showcasing the capability of modeling complex buffer management strategies, scheduling policies and hardware architectures. As a case study of this framework, we demonstrate the importance of adopting different buffering strategies for activation and weight tensors in AI accelerators to acquire remarkable speedup.

Low-cost and high-performance channel access strategies for Internet of Nano-Things applications

Nanodevices, which are only a few nanometers (nm) in size, are interconnected to form the Internet of Nano-Things (IoNT) that performs complex operations. One of the key challenges is ensuring efficient channel access control for nanodevices, especially when dealing with large network sizes. Medium access control (MAC) protocols serve this purpose, but traditional approaches are not practical due to the inherent constraints of nanodevices. In this paper, we propose two novel MAC protocols for IoNT applications. The first protocol, Slot Assignment-Based (SAB) MAC, is a contention-free method relying on scheduling. In contrast to its counterparts, it enables simultaneous packet transmission through Time Spread On-Off Keying (TS-OOK), effectively minimizing the collision probability and end-to-end delay. The second protocol, Receiver-Initiated and Directed (RID) MAC, adopts a contention-based approach to reduce unnecessary transmissions caused by flooding. It achieves this by limiting the number of active nanodevices within a time interval using directional antennas without incurring scheduling overhead. We evaluated the performance of these protocols through comprehensive simulations, comparing them with counterparts in terms of packet transmission success, energy consumption, end-to-end delay and setup overhead. In dense topologies, SAB-MAC outperforms Transparent (TRN) MAC by approximately twice the packet transmission success reaching up to 95.73%. It accomplishes this with 1000 times lower end-to-end delay and reduced setup overhead than Time Division Multiple Access (TDMA). Conversely, RID-MAC achieves twice the packet transmission success of TRN-MAC and ten times that of unicast-based methods, all with lower end-to-end delay and nearly equivalent energy consumption. Consequently, due to its superior performance SAB-MAC is the optimal choice for communication between nanorouters (NRs). However, RID-MAC is more suitable for communication between nanosensors (NSs), as it incurs no setup overhead.

Efficient end-to-end long-read sequence mapping using minimap2-fpga integrated with hardware accelerated chaining

minimap2 is the gold-standard software for reference-based sequence mapping in third-generation long-read sequencing. While minimap2 is relatively fast, further speedup is desirable, especially when processing a multitude of large datasets. In this work, we present minimap2-fpga, a hardware-accelerated version of minimap2 that speeds up the mapping process by integrating an FPGA kernel optimised for chaining. Integrating the FPGA kernel into minimap2 posed significant challenges that we solved by accurately predicting the processing time on hardware while considering data transfer overheads, mitigating hardware scheduling overheads in a multi-threaded environment, and optimizing memory management for processing large realistic datasets. We demonstrate speed-ups in end-to-end run-time for data from both Oxford Nanopore Technologies (ONT) and Pacific Biosciences (PacBio). minimap2-fpga is up to 79% and 53% faster than minimap2 for sim 30times ONT and sim 50times PacBio datasets respectively, when mapping without base-level alignment. When mapping with base-level alignment, minimap2-fpga is up to 62% and 10% faster than minimap2 for sim 30times ONT and sim 50times PacBio datasets respectively. The accuracy is near-identical to that of original minimap2 for both ONT and PacBio data, when mapping both with and without base-level alignment. minimap2-fpga is supported on Intel FPGA-based systems (evaluations performed on an on-premise system) and Xilinx FPGA-based systems (evaluations performed on a cloud system). We also provide a well-documented library for the FPGA-accelerated chaining kernel to be used by future researchers developing sequence alignment software with limited hardware background.

A novel hierarchical distributed vehicular edge computing framework for supporting intelligent driving

Recently, various infrastructure-assisted or onboard driving assistant applications have been proposed as a component of intelligent transportation systems (ITS) to improve the transportation system’s efficiency and release public concern about road safety. However, such AI-assisted intelligent applications are mainly data-driven and put great demands on the computing power of the ITS systems. Therefore, in the highly dynamic Internet-of-Vehicles environment in ITS, how to effectively coordinate the limited computing power of the various components of the system and realize reliable support for such resource-consuming applications through efficient resource allocation methods is the focus of our research. Accordingly, a novel joint computing and communication resource scheduling method is proposed to fulfill those ITS applications’ inherent heterogeneous quality of service (QoS) requirements. By fully exploiting the computing resources provided by the onboard computing device, the edge computing device located in the vehicle’s proximity and remote data center, we designed a hierarchical three-layer Vehicular Edge Computing (VEC) framework. Briefly, an onboard joint computation offloading and transmission scheduling policy is designed to assign corresponding offloading decisions to the locally generated computing tasks by considering the vehicle’s computing resources and real-time network link status. Additionally, a new distributed resource allocation policy is developed for the edge devices, in which we derive a server selection policy and allocate communication time based on our proposed metric. To evaluate the performance and validate the effectiveness of our proposed method, we conduct extensive simulation tests and ablation experiments, respectively. The results show that our approach can achieve stable performance in various experimental settings. Also, compared to the state-of-the-art algorithms, our joint resource allocation policy significantly reduces the scheduling overhead, improves the utilization of system resources, and minimizes the data transmission delay caused by vehicle motion.

A New Federated Scheduling Algorithm for Arbitrary-Deadline DAG Tasks

A parallel task can always be modelled as a directed acyclic graph (DAG), where sequential instruction blocks are modelled as vertices and data dependencies or resource constraints are modelled as edges. We propose a new federated scheduling algorithm for arbitrary-deadline sporadic DAG tasks, assuming that the exact structures of DAG tasks are unknown before runtime. Federated scheduling algorithms are a class of algorithms that can efficiently schedule DAG tasks by assigning several processors exclusively to each task. Existing studies have shown the advantages of federated scheduling, which include increasing the analytical schedulability and minimising the scheduling overhead. We are particularly focused on the scheduling of any task with a deadline longer than its release period; in this case, multiple jobs generated by the task could run concurrently. For such tasks, our algorithm is different from most federated scheduling algorithms in that it assigns dedicated processors to each job instead of letting jobs released by the same task share processors. The main idea is to increase the analytical schedulability by avoiding interference between jobs. The simulation results show that our algorithm outperforms existing algorithms when the exact structures of tasks are unknown before runtime.

Parallel Strong Connectivity Based on Faster Reachability

Computing strongly connected components (SCC) is among the most fundamental problems in graph analytics. Given the large size of today's real-world graphs, parallel SCC implementation is increasingly important. SCC is challenging in the parallel setting and is particularly hard on large-diameter graphs. Many existing parallel SCC implementations can be even slower than Tarjan's sequential algorithm on large-diameter graphs. To tackle this challenge, we propose an efficient parallel SCC implementation using a new parallel reachability approach. Our solution is based on a novel idea referred to as vertical granularity control (VGC). It breaks the synchronization barriers to increase parallelism and hide scheduling overhead. To use VGC in our SCC algorithm, we also design an efficient data structure called the parallel hash bag. It uses parallel dynamic resizing to avoid redundant work in maintaining frontiers (vertices processed in a round). We implement the parallel SCC algorithm by Blelloch et al. (J. ACM, 2020) using our new parallel reachability approach. We compare our implementation to the state-of-the-art systems, including GBBS, iSpan, Multi-step, and our highly optimized Tarjan's (sequential) algorithm, on 18 graphs, including social, web, k-NN, and lattice graphs. On a machine with 96 cores, our implementation is the fastest on 16 out of 18 graphs. On average (geometric means) over all graphs, our SCC is 6.0× faster than the best previous parallel code (GBBS), 12.8× faster than Tarjan's sequential algorithms, and 2.7× faster than the best existing implementation on each graph. We believe that our techniques are of independent interest. We also apply our parallel hash bag and VGC scheme to other graph problems, including connectivity and least-element lists (LE-lists). Our implementations improve the performance of the state-of-the-art parallel implementations for these two problems.

Point cloud segmentation of overhead contact systems with deep learning in high-speed rails

High-speed rails have strategic significance for political and economic development. As a critical part of the power supply system, overhead catenary systems (OCSs) power high-speed rails. OCSs are easily damaged due to artificial and natural factors. To ensure that OCSs work properly, researchers often use light detection and ranging (LiDAR) to recognize and inspect OCS components. However, recent OCS recognition methods relying on LiDAR need to enhance the ability to extract local features, integrate contextual features, and enhance crucial features while maintaining less latency and model complexity. To accomplish this objective, we proposed an accelerated point cloud segmentation algorithm for OCS recognition. The algorithm extracted local point cloud features based on machine learning methods. Then, it integrated the contextual features of point clouds using dot and accumulation operations. Additionally, we captured vital features and neglected unimportant features by constructing a feature enhancement algorithm. We accelerated the proposed algorithms by eliminating the run-time overhead of GPU scheduling. Experiments showed that our method had high precision with low model complexity and latency. For example, our precision was at least 0.64% better than comparison studies in recognizing steady arms. Our parameters were at least 43.59% fewer than others. Our optimized algorithm achieved a 2.47, 2.81, 3.24, 1.54, 7.12, and 7× speedup on the Nano, Tesla T4, RTX 2080 Ti, TX2, Tesla V100S GPU, and TITAN V, respectively. Our visualization effect also outperformed other methods in OCS recognition of high-speed rail scenarios.

Soft-core processor integration based on different instruction set architectures and field programmable gate array custom datapath implementation.

One of the fundamental requirements of a real-time system (RTS) is the need to guarantee re-al-time determinism for critical tasks. Task execution rates, operating system (OS) overhead, and task context switching times are just a few of the parameters that can cause jitter and missed deadlines in RTS with soft schedulers. Control systems that are susceptible to jitter can be used in the control of HARD RTS as long as the cumulative value of periodicity deviation and worst-case response time is less than the response time required by that application. This artcle presents field-programmable gate array (FPGA) soft-core processors integration based on different instruction set architectures (ISA), custom central processing unit (CPU) datapath, dedicated hardware thread context, and hardware real-time operating system (RTOS) implementations. Based on existing work problems, one parameter that can negatively influence the performance of an RTS is the additional costs due to the operating system. The scheduling and thread context switching operations can significantly degrade the programming limit for RTS, where the task switching frequency is high. In parallel with the improvement of software scheduling algorithms, their implementation in hardware has been proposed and validated to relieve the processor of scheduling overhead and reduce RTOS-specific overhead.

An enhanced ordinal optimization with lower scheduling overhead based novel approach for task scheduling in cloud computing environment

Efficient utilization of available computing resources in Cloud computing is one of the most challenging problems for cloud providers. This requires the design of an efficient and optimal task-scheduling strategy that can play a vital role in the functioning and overall performance of the cloud computing system. Optimal Schedules are specifically needed for scheduling virtual machines in fluctuating & unpredictable dynamic cloud scenario. Although there exist numerous approaches for enhancing task scheduling in the cloud environment, it is still an open issue. The paper focuses on an improved & enhanced ordinal optimization technique to reduce the large search space for optimal scheduling in the minimum time to achieve the goal of minimum makespan. To meet the current requirement of optimal schedule for minimum makespan, ordinal optimization that uses horse race conditions for selection rules is applied in an enhanced reiterative manner to achieve low overhead by smartly allocating the load to the most promising schedule. This proposed ordinal optimization technique and linear regression generate optimal schedules that help achieve minimum makespan. Furthermore, the proposed mathematical equation, derived using linear regression, predicts any future dynamic workload for a minimum makespan period target.

Dynamic Virtual Machine Consolidation in the Cloud: A Cuckoo Search Approach

The practice of consolidating virtual machines onto a smaller number of physical servers is known as Virtual Machine (VM) consolidation. This allows for a reduction in the total number of physical servers that are required to host virtual machines with minimum consumption of energy. There are lots of mechanism already developed by the various researchers in this filed to optimize the energy consumption rate. Therefore, in this research work, Cuckoo Search (CS) based VMs consolidation approach for Cloud Computing (CC). It is impossible to effectively optimize energy efficiency without compromising some form of system performance cloud service quality. The majority of current VM consolidation approaches are based on the concept of load and threshold, which affects the system performance in terms of Quality of Service (QoS) metrics such as energy consumption, Service Level Agreement (SLA) violation, and task scheduling overhead. Because of this, optimizing energy efficiency is currently impossible. Therefore, proposed a CS-based VMs consolidation approach to achieve this goal of research. When CS-based method is compared against literature in terms of energy consumption and SLA violation, the model achieves a considerable reduction in energy consumption while concurrently achieving a reduction in the quantity of service violations.

Optimizing Iterative Data-flow Scientific Applications using Directed Cyclic Graphs

Data-flow programming models have become a popular choice for writing parallel applications as an alternative to traditional work-sharing parallelism. They are better suited to write applications with irregular parallelism that can present load imbalance. However, these programming models suffer from overheads related to task creation, scheduling and dependency management, limiting performance and scalability when tasks become too small. At the same time, many HPC applications implement iterative methods or multi-step simulations that create the same directed acyclic graphs of tasks on each iteration. By giving application programmers a way to express that a specific loop is creating the same task pattern on each iteration, we can create a single task DAG once and transform it into a cyclic graph. This cyclic graph is then reused for successive iterations, minimizing task creation and dependency management overhead. This paper presents the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">taskiter</i> , a new construct we propose for the OmpSs-2 and OpenMP programming models, allowing the use of directed cyclic task graphs (DCTG) to minimize runtime overheads. Moreover, we present a simple <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">immediate successor</i> locality-aware heuristic that minimizes task scheduling overhead by bypassing the runtime task scheduler. We evaluate the implementation of the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">taskiter</i> and the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">immediate successor</i> heuristic in 8 iterative benchmarks. Using small task granularities, we obtain a geometric mean speedup of 2.56x over the reference OmpSs-2 implementation, and a 3.77x and 5.2x speedup over the LLVM and GCC OpenMP runtimes, respectively.

Efficient schedulability analysis of hierarchical EDF scheduling with resource sharing

This paper presents an efficient, exact and sustainable schedulability analysis for hierarchical scheduling when a hybrid scheduling strategy is applied to a uniprocessor real-time system. We concentrate on the situation in which each application is to run on a server with an earliest deadline first (EDF) scheduler, and the tasks of each application are required to meet their given hard real-time constraints. We show how blocking can be considered in a hierarchical system to let the applications and the tasks share non-preemptable resources at global and local levels. This hierarchical system model is easy to implement, and the proposed analysis does not require making any online scheduling decisions, hence no extra scheduling overhead is required. Through extensive experiments, we show that our schedulability analysis has a very high acceptance ratio for the application tasks and that the proposed analysis is also highly efficient; in nearly all cases, less than 12 times the calculations of the response time on average is required to complete an analysis for a set of tasks within an application. Therefore, the proposed schedulability analysis significantly reduces the required computations and increases the acceptance ratio. Furthermore, the proposed approach is quite general and there are no restrictions on the task parameters, each task could be periodic or sporadic, and they can have arbitrary relative deadlines.

Tenant-Grained Request Scheduling in Software-Defined Cloud Computing

Cloud providers host various services for tenants’ requests (e.g., software-as-a-service) and seek to serve as many requests as possible for revenue maximization. Considering a large number of requests, the previous works on fine-grained request scheduling may lead to poor system scalability (or high schedule overhead) and break tenant isolation. In this article, we design a tenant-grained request scheduling framework to conquer the above two disadvantages. We formulate the tenant-grained request scheduling problem as an integer linear programming and prove its NP-hardness. We consider two complementary cases: the offline case (where we know all request demands in advance), and the online case (where we have to make immediate scheduling decisions for requests arriving online). A normalization-based algorithm with an approximation factor of <inline-formula><tex-math notation="LaTeX">$ {O}(1)$</tex-math></inline-formula> is proposed to solve the offline problem and a primal-dual-based algorithm with a competitive ratio of <inline-formula><tex-math notation="LaTeX">$[(1-\epsilon), {O}(\log 3\cdot n+\log (1/\epsilon))]$</tex-math></inline-formula> is designed for the online scenario, where <inline-formula><tex-math notation="LaTeX">$\epsilon \in (0,1)$</tex-math></inline-formula> and <inline-formula><tex-math notation="LaTeX">$n$</tex-math></inline-formula> is the number of racks in the cloud. We also discuss how to integrate our proposed algorithms with the previous (fine-grained) request scheduling mechanism. Extensive simulation and experiment results show that our algorithms can obtain significant performance gains, e.g., the online algorithm reduces the scheduler's overhead more than <inline-formula><tex-math notation="LaTeX">$90\%$</tex-math></inline-formula> and achieves tenant isolation, while obtaining similar network performance (e.g., throughput) compared with the fine-grained request scheduling methods.

Task Scheduling for Probabilistic In -Band Network Telemetry

In-band Network Telemetry (INT) is a novel framework for monitoring network health in real-time, and its recent variant, Probabilistic INT (PINT), reduces its bandwidth consumption with a probabilistic approach. However, as we show in this paper, a PINT task can be successfully accomplished only when it is allocated a sufficient number of packets, and if there are many tasks executed in parallel, packets become a scarce resource. Meanwhile, today’s production network generally executes multiple measurement tasks for tracing different network states simultaneously. Therefore, in such a context, scheduling parallel PINT tasks on one single INT flow that has a limited number of packets becomes a critical problem. In this paper, we address this problem for the first time. We propose an algorithm that efficiently schedules multiple parallel PINT tasks on a flow by allocating the flow’s packets to the tasks and showing that the allocation is optimal. We realize the algorithm with a packet processing pipeline and implement it on software and hardware-programmable switches. Comprehensive evaluation on a FatTree testbed shows that at a low scheduling overhead, our algorithm can conduct parallel PINT tasks to detect various network faults in a timely and accurate manner. Additionally, the algorithm accomplishes more PINT tasks with higher quality than the alternative solutions.

Lightweight Monocular Depth Estimation on Edge Devices

Given monocular images as inputs, monocular depth estimation (MDE) infers pixel-level depth. MDE is always a critical stage in scene sensing on edge devices. Existing MDE studies frequently employ deep neural networks (DNNs) for MDE, but they still face some problems, such as sacrificing computational complexity and efficiency in return for great precision, or losing more precision in exchange for increased efficiency. To alleviate these issues; 1) we propose an encoder–decoder network (EdgeNet) for precise and fast MDE on different edge devices. When recovering depth in the decoder, we design upsampling modules to aggregate global depth information with low computational complexity, improving the accuracy of the decoder by extracting its different ranges of depth information; 2) we develop a two-stage channel pruning method to, respectively, prune the encoder and decoder based on their characteristics. Our pruning method further reduces latency and model/computational complexity of EdgeNet, while losing little accuracy; and 3) we optimize the pruned EdgeNet to decrease graphics processing unit (GPU) scheduling overhead. The optimization accelerates MDE inference by an order of magnitude on the TX2 GPU device, when the input resolution is 224 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times $ </tex-math></inline-formula> 224. Extensive experiments show that our strategies are effective on different edge GPU devices, when input resolutions differ in outdoor or indoor scenes. For example, compared with the state of the art, the optimized EdgeNet, respectively, reduces the GPU latency by 76.3% and 89.2% on Nano and TX2 GPU devices with 2.6% lower root mean square error when the input resolution is 128 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times $ </tex-math></inline-formula> 416.

EALI: Energy-aware layer-level scheduling for convolutional neural network inference services on GPUs

The success of convolutional neural networks (CNNs) has made low-latency inference services on Graphic Processing Units (GPUs) a hot research topic. However, GPUs are hardware processors with high power consumption. To have the least energy consumption while meeting latency Service-Level-Objective (SLO), batching strategy and dynamic voltage frequency scaling (DVFS) are two important solutions. However, existing studies do not coordinate them and regard CNN as a black box, which makes inference services less energy-efficient. In this paper, we propose EALI, an energy-aware layer-level adaptive scheduling framework that is comprised of a power prediction model, a layer combination strategy, and an energy-aware layer-level scheduler. The power prediction model uses classic machine learning techniques to predict fine-grained layer-level power consumption. The layer combination strategy combines multiple layers into optimization units to lower scheduling overhead and complexity. The energy-aware layer-level scheduler adaptively coordinates batching strategy and layer-level DVFS according to workloads to minimize the energy consumption while meeting SLO. Our experimental results on NVIDIA Tesla M40 and V100 GPUs show that, compared to the state-of-the-art approaches, EALI decreases energy consumption by up to 36.24% while meeting SLO.

Designing a Custom CPU Architecture Based on Hardware RTOS and Dynamic Preemptive Scheduler

The current trend in real-time operating systems involves executing many tasks using a limited hardware platform. Thus, a single processor system has to execute multiple tasks with different priorities in different real-time system (RTS) work modes. Hardware schedulers can greatly reduce event trigger latency and successfully remove most of the scheduling overhead, providing more computing cycles for applications. In this paper, we present a hardware-accelerated RTOS based on the replication of resources such as program counters, general purpose registers (GPRs) and pipeline registers. The implementation of this new concept, based on real-time event handling implemented in hardware, is intended to meet the current rigorous requirements imposed by critical real-time systems. The most important attribute of this FPGA implementation is the time required for task context switching, which is only one clock cycle or three clock cycles when working with the atomic instructions used in the case of inter-task synchronization and communication mechanisms. The main contribution of this article is its focus on mutexes and the speed of response associated with related events. Thus, fast switching between threads is also validated, considering the handling of events in the hardware using HW_nMPRA_RTOS (HW-RTOS). The proposed architecture implements inter-task synchronization and communication mechanisms with high performance, improving the overall response time when the mutex or message is expected to relate to a higher-priority task.

On the choice of the best chunk size for the speculative execution of loops.

Loops are a rich source of parallelism. Unfortunately, many loops cannot be safely parallelized at compile time because the compiler is not able to guarantee that there will be no dependence violations. Thread-Level Speculation (TLS) techniques, either hardware or software-based, allow the parallel execution of non-analyzable loops, issuing the execution of blocks of consecutive iterations (called chunks) while a hardware or software monitor ensures that no dependence violations arise. If such a dependence violation occurs, the chunk that was fed with incorrect values is discarded and re-started, in order to consume the correct information. In the speculative execution of non-analyzable loops, it is very important to correctly choose the chunk size, because this choice dramatically affects the performance of the parallel execution. Bigger chunks imply less scheduling overheads, but smaller chunks allow fewer calculations to be discarded in the event of a dependence violation. To find a good chunk size is not a simple task, because loops may present dependencies that cannot be detected at compile time. In this paper, we present a comprehensive evaluation of different scheduling methods to estimate the optimal chunk size in the speculative execution of non-analyzable loops. This evaluation ranges from the simple, classical methods originally devised to achieve load balancing in loops with no dependencies, to methods that make some assumptions on the distribution pattern of dependencies, such as Meseta and Just-in-Time scheduling. We also propose and evaluate a general, more complex method called Moody Scheduling, that does not require a-priori assumptions to achieve the highest performance.

Spatial- and time- division multiplexing in CNN accelerator

With the widespread use of real-time data analysis by artificial intelligence (AI), the integration of accelerators is attracting attention from the perspectives of their low power consumption and low latency. The objective of this research is to increase accelerator resource efficiency and further reduce power consumption by sharing accelerators among multiple users while maintaining real-time performance. To achieve the accelerator-sharing system, we define three requirements: high device utilization, fair device utilization among users, and real-time performance. Targeting the AI inference use case, this paper proposes a system that shares a field-programmable gate array (FPGA) among multiple users by switching the convolutional neural network (CNN) models stored in the device memory on the FPGA, while satisfying the three requirements. The proposed system uses different behavioral models for workloads with predictable and unpredictable data arrival timing. For the workloads with predictable data arrival timing, the system uses spatial-division multiplexing of the FPGA device memory to achieve real-time performance and high device utilization. Specifically, the FPGA device memory controller of the system transparently preloads and caches the CNN models into the FPGA device memory before the data arrival. For workloads with unpredictable data arrival timing, the system transfers CNN models to the FPGA device memory upon data arrival using time-division multiplexing of FPGA device memory. In the latter case of unpredictable workloads, the switch cost between CNN models is non-negligible to achieve real-time performance and high device utilization, so the system integrates a new scheduling algorithm that considers the switch time of the CNN models. For both predictable and unpredictable workloads, user fairness is achieved by using an ageing technique in the scheduling algorithm that increases the priority of jobs in accordance with the job waiting time. The evaluation results show that the scheduling overhead of the proposed system is negligible for both predictable and unpredictable workloads providing practical real-time performance. For unpredictable workloads, the new scheduling algorithm improves fairness by 24%–94% and resource efficiency by 31%–33% compared to traditional algorithms using first-come first-served or round-robin. For predictable workloads, the system improves fairness by 50.5 % compared to first-come first-served and achieves 99.5 % resource efficiency.

Early scheduling on steroids: Boosting parallel state machine replication

State machine replication (SMR) is a standard approach to fault tolerance in which replicas execute requests deterministically and often serially. For performance, some techniques allow concurrent execution of requests in SMR while keeping determinism. Such techniques exploit the fact that independent requests can execute concurrently. A promising category of early scheduling solutions trades scheduling freedom for simplicity, allowing to expedite decisions during scheduling. This paper generalizes early scheduling and proposes a general method to schedule requests to threads, restricting scheduling overhead. Moreover, it explores improvements to the original early scheduling mechanism, namely the use of busy-wait synchronization and work-stealing techniques. We integrate early scheduling and its proposed improvements to a popular SMR framework. Performance results of the basic mechanism and its improvements are presented and compared to more classic approaches, where it is shown that early scheduling with our proposed enhancements can outperform the original early scheduling and other systems by a large margin in many scenarios.