Heterogeneous Processing Units Research Articles

Corrigendum to “PartitionTuner: An operator scheduler for deep‐learning compilers supporting multiple heterogeneous processing units”

Corrigendum to “PartitionTuner: An operator scheduler for deep‐learning compilers supporting multiple heterogeneous processing units”

PartitionTuner: An operator scheduler for deep‐learning compilers supporting multiple heterogeneous processing units

AbstractRecently, embedded systems, such as mobile platforms, have multiple processing units that can operate in parallel, such as centralized processing units (CPUs) and neural processing units (NPUs). We can use deep‐learning compilers to generate machine code optimized for these embedded systems from a deep neural network (DNN). However, the deep‐learning compilers proposed so far generate codes that sequentially execute DNN operators on a single processing unit or parallel codes for graphic processing units (GPUs). In this study, we propose PartitionTuner, an operator scheduler for deep‐learning compilers that supports multiple heterogeneous PUs including CPUs and NPUs. PartitionTuner can generate an operator‐scheduling plan that uses all available PUs simultaneously to minimize overall DNN inference time. Operator scheduling is based on the analysis of DNN architecture and the performance profiles of individual and group operators measured on heterogeneous processing units. By the experiments for seven DNNs, PartitionTuner generates scheduling plans that perform 5.03% better than a static type‐based operator‐scheduling technique for SqueezeNet. In addition, PartitionTuner outperforms recent profiling‐based operator‐scheduling techniques for ResNet50, ResNet18, and SqueezeNet by 7.18%, 5.36%, and 2.73%, respectively.

Scheduling of multiple network packet processing applications using Pythia

Modern commodity computing systems are composed by a number of different heterogeneous processing units, each of which has its own unique performance and energy characteristics. However, the majority of current network packet processing frameworks targets only a specific processing unit (either the CPU or accelerator), leaving the remaining computational resources under-utilized or even idle.In this paper, we propose an adaptive scheduling approach for network packet processing applications, that supports any heterogeneous and asymmetric architectures that can be found in a commodity high-end hardware setup. Our scheduler not only distributes the workloads to the appropriate devices in the system to achieve the desired performance results, but also enables the multiplexing of diverse network packet processing applications that execute concurrently, eliminating the interference effects introduced at runtime. The evaluation results show that our scheduler is able to tackle interferences in the shared hardware resources as well to respond quickly to dynamic fluctuations (e.g., application overloads, traffic bursts, infrastructural changes, etc.) that may occur at real time.

Energy-Efficient Acceleration of Deep Neural Networks on Realtime-Constrained Embedded Edge Devices

This paper presents a hardware management technique that enables energy-efficient acceleration of deep neural networks (DNNs) on realtime-constrained embedded edge devices. It becomes increasingly common for edge devices to incorporate dedicated hardware accelerators for neural processing. The execution of neural accelerators in general follows a host-device model, where CPUs offload neural computations (e.g., matrix and vector calculations) to the accelerators for datapath-optimized executions. Such a serialized execution is simple to implement and manage, but it is wasteful for the resource-limited edge devices to exercise only a single type of processing unit in a discrete execution phase. This paper presents a hardware management technique named NeuroPipe that utilizes heterogeneous processing units in an embedded edge device to accelerate DNNs in energy-efficient manner. In particular, NeuroPipe splits a neural network into groups of consecutive layers and pipelines their executions using different types of processing units. The proposed technique offers several advantages to accelerate DNN inference in the embedded edge device. It enables the embedded processor to operate at lower voltage and frequency to enhance energy efficiency while delivering the same performance as uncontrolled baseline executions, or inversely it can dispatch faster inferences at the same energy consumption. Our measurement-driven experiments based on NVIDIA Jetson AGX Xavier with 64 tensor cores and eight-core ARM CPU demonstrate that NeuroPipe reduces energy consumption by 11.4% on average without performance degradation, or it can achieve 30.5% greater performance for the same energy consumption.

The HPC-DAG Task Model for Heterogeneous Real-Time Systems

Recent commercial hardware platforms for embedded real-time systems feature heterogeneous processing units and computing accelerators on the same System-on-Chip. When designing complex real-time applications for such architectures, the designer is exposed to a number of difficult choices, like deciding on which compute engine to execute a certain task, or what degree of parallelism to adopt for a given function. To help the designer exploring the wide space of design choices and tune the scheduling parameters, we propose a novel real-time application model, called HPC-DAG ( Heterogeneous Parallel Condition Directed Acyclic Graph Model ), specifically conceived for heterogeneous platforms. An HPC-DAG allows the system designer to specify alternative implementations of a software component for different processing engines, as well as conditional branches to model if-then-else statements. We also propose a schedulability analysis for the HPC-DAG model and a set of heuristic allocation algorithms aimed at improving schedulability for latency sensitive applications. Our analysis takes into account the cost of preempting a task, which can be non-negligible on certain processors. We show the use of our approach on a realistic case study, and we demonstrate its effectiveness by comparing it with state-of-the-art algorithms previously proposed in literature.

Heterogeneous Computational Resource Allocation for C-RAN: A Contract-Theoretic Approach

In this work, we develop a contract theory framework to tackle the allocations of heterogeneous baseband processing units (BBUs) in cloud radio access network. We first model a monopoly market by viewing the BBUs as a kind of resource. The infrastructure provider (InP), as the monopolist, owns all the heterogeneous BBUs of different processing abilities and maintaining costs, and leases them to multiple mobile network operators (MNOs) to gain profit. The MNOs intend to rent reasonable amount of BBUs to provide services to their mobile clients. Then we propose a contract theory framework, in which contract items are optimized to maximize the InP's utility. We design the optimal contracts with complete and asymmetric information on the MNOs. Our contract design achieves the near optimum solution to heterogeneous computational resource allocation even under the information asymmetric case. Our derivations indicate that the optimal contracts with asymmetric information achieve a lower utility for the InP than the ones with complete information and the utility reduction is higher when the BBUs are heterogeneous rather than homogeneous. Numerical results demonstrate the benefit of having heterogeneous BBUs relative to having homogeneous BBUs in terms of achieving a higher utility for the InP.

Improving the performance and energy of Non-Dominated Sorting for evolutionary multiobjective optimization on GPU/CPU platforms

Non-Dominated Sorting (NDS) is the most time-consuming procedure used in the majority of evolutionary multiobjective optimization algorithms that are based on Pareto dominance ranking without regard to the computation time of the objective functions. It can be accelerated by the exploitation of its parallelism on High Performance Computing systems, that provide heterogeneous processing units, such as multicore processors and GPUs. The optimization of energy efficiency of such systems is a challenge in scientific computation since it depends on the kind of processing which is performed. Our interest is to solve NDS in an efficient way concerning both runtime and energy consumption. In literature, performance improvement has been extensively studied. Recently, a sequential Best Order Sort (BOS) algorithm for NDS has been introduced as one of the most efficient one in terms of practical performance. This work is focused on the acceleration of the NDS on modern architectures. Two efficient parallel NDS algorithms based on Best Order Sort, are introduced, MC-BOS and GPU-BOS. Both algorithms start with the fast sorting of population by objectives. MC-BOS computes in parallel the analysis of the population by objectives on the multicore processors. GPU-BOS is based on the principles of Best Order Sort, with a new scheme designed to harness the massive parallelism provided by GPUs. A wide experimental study of both algorithms on several kinds of CPU and GPU platforms has been carried out. Runtime and energy consumption are analysed to identify the best platform/algorithm of the parallel NDS for every particular population size. The analysis of obtained results defines criteria to help the user when selecting the optimal parallel version/platform for particular dimensions of NDS. The experimental results show that the new parallel NDS algorithms overcome the sequential Best Order Sort in terms of the performance and energy efficiency in relevant factors.

Data-Driven Approach for Feature Drift Detection in Embedded Electronic Devices

This paper is a part of a project aiming to develop supervisor and monitoring devices for embedded systems in airplanes and vehicles. It focuses on the reliability of these systems and establishes a monitoring framework to detect drifts and faults in the behavior of the heterogeneous central processing units (CPU) and graphics processing units (GPU) chips powering them. In this work, we use a previously developed incremental model of these chips and associate it with a fault detection algorithm. Estimations from the model constitute inputs to the diagnosis module. The latter generates alarms in the presence of faults or drifts in the characteristics and features of the System-on-Chip (SoC). The obtained results validate the proposed monitoring algorithm and demonstrate the effectiveness of the fault detection algorithm.

Unite and conquer approach for high scale numerical computing

The ability to exploit emerging exascale computational systems will require a careful review and redesign of core numerical algorithms and their implementations to fully exploit multiple levels of concurrency, hierarchical memory structures and heterogeneous processing units that will become available in these computational platforms. This paper presents the “unite and conquer” approach to solve linear systems of equations and eigenvalue problems for extreme scale computing. Indeed, there are two ways to optimize the execution of a restarted method on a large-scale distributed system. The first one is to optimize the number of floating point operations per restart cycle through maximizing the concurrency inside a restart cycle while minimizing latencies. The second way is to accelerate/improve the rate of convergence for a given computational scheme. The unite and conquer restarted approach focuses on decreasing the number of restart cycles by coupling either synchronously or asynchronously several restarted methods called also co-methods. In the end of a restart cycle, each co-method locally gathers available results of all collaborating co-methods and selects the best one in order to create its restarting information. Consequently this permits the global reduction of the number of cycles to convergence. The unite and conquer restarted methods are heterogeneous, fault tolerant, support asynchronous communications and present a big potential of load balancing. Due to these properties, they are well adapted to large-scale multi-level parallel architectures. We show the relevant programming paradigms that allow multi-level parallel expression of these methods and how the software engineering technology can contribute significantly in achieving high scalability. We present some experiments validating the approach for unite and conquer restarted Krylov methods on several parallel and distributed platforms.

Execution Trace--Driven Energy-Reliability Optimization for Multimedia MPSoCs

Multiprocessor systems-on-chip (MPSoCs) are becoming a popular design choice in current and future technology nodes to accommodate the heterogeneous computing demand of a multitude of applications enabled on these platform. Streaming multimedia and other communication-centric applications constitute a significant fraction of the application space of these devices. The mapping of an application on an MPSoC is an NP-hard problem. This has attracted researchers to solve this problem both as stand-alone (best-effort) and in conjunction with other optimization objectives, such as energy and reliability. Most existing studies on energy-reliability joint optimization are static—that is, design time based. These techniques fail to capture runtime variability such as resource unavailability and dynamism associated with application behaviors, which are typical of multimedia applications. The few studies that consider dynamic mapping of applications do not consider throughput degradation, which directly impacts user satisfaction. This article proposes a runtime technique to analyze the execution trace of an application modeled as Synchronous Data Flow Graphs (SDFGs) to determine its mapping on a multiprocessor system with heterogeneous processing units for different fault scenarios. Further, communication energy is minimized for each of these mappings while satisfying the throughput constraint. Experiments conducted with synthetic and real SDFGs demonstrate that the proposed technique achieves significant improvement with respect to the state-of-the-art approaches in terms of throughput and storage overhead with less than 20% energy overhead.

Development of a SoC for Digital Television Set-Top Box: Architecture and System Integration Issues

This work presents the integration of several IPs to generate a system-on-chip (SoC) for digital television set-top box compliant to the SBTVD standard. Embedded consumer electronics for multimedia applications like video processing systems require large storage capacity and high bandwidth memory. Also, those systems are built from heterogeneous processing units, designed to perform specific tasks in order to maximize the overall system efficiency. A single off-chip memory is generally shared between the processing units to reduce power and save costs. The external memory access is one bottleneck when decoding high-definition video sequences in real time. In this work, a four-level memory hierarchy was designed to manage the decoded video in macroblock granularity with low latency. The use of the memory hierarchy in the system design is challenging because it impacts the system integration process and IP reuse in a collaborative design team. Practical strategies used to solve integration problems are discussed in this text. The SoC architecture was validated and is being progressively prototyped using a Xilinx Virtex-5 FPGA board.

Platform synthesis and partitioning of real-time tasks for energy efficiency

Energy-efficient and power-aware designs have played important roles in modern computing systems to reduce the power bills for server systems or prolong the lifetime of embedded devices. Moreover, systems with multiple heterogeneous processing units have been widely adopted to enhance the computing capability or reduce the power consumption. This work explores how to synthesize a heterogeneous multiprocessor platform or select processing units with the partitioning of real-time tasks so that the energy consumption is minimized. Given a set of processing unit types, characterized by the power consumption for maintaining activeness and executing jobs, this paper proposes an efficient and effective algorithm to allocate processing units with energy-efficient task partitioning. When the constraint on the numbers of allocated processing units is not specified, we show that the algorithm is with a (1+lnn)-approximation factor, in worst cases, for processing unit types with a variety of power consumption models, where n is the number of tasks. The approximation factor is asymptotically optimal for polynomial-time approximation algorithms unless P=NP. Moreover, we also present how to deal with specified maximal numbers of allocated processing units to minimize the energy consumption. Simulation results show that the proposed algorithm is effective for energy consumption minimization.

Performance of the PN superscalar processor as estimated by simulation

The authors have already proposed the multiple instruction-stream execution system by a function-partitioned superscalar processor, for extracting the fine-grain parallelism. In the proposed system, the processor is divided into a number of heterogeneous processing units and the instruction stream is given separately to the processing units. The processing units communicate the tokens (control pulses) at a higher speed to control the execution of instruction according to the precedence relations among the instructions. This paper considers the prototype model based on three kinds of machine instructions and evaluates the fundamental performances of the proposed execution scheme by simulation. Especially, the following items are examined: (1) load distribution by three kinds of instructions, (2) extracted parallelism and execution time of the program, (3) effect of token communication time and the effect of overlapped execution of token transmission/reception instructions, and (4) effect of access time of memory unit.