Chip Multiprocessor Architecture Research Articles

CAP: Communication-Aware Automated Parallelization for Deep Learning Inference on CMP Architectures

Real-time inference of deep learning models on embedded and energy-efficient devices becomes increasingly desirable with the rapid growth of artificial intelligence on edge. Specifically, to achieve superb energy-efficiency and scalability, efficient parallelization of single-pass deep neural network (DNN) inference on chip multiprocessor (CMP) architectures is urgently required by many time-sensitive applications. However, as the number of processing cores scales up and the performance of cores has grown much fast, the on-chip inter-core data movement is prone to be a performance bottleneck for computation. To remedy this problem and further improve the performance of network inference, in this work, we introduce a communication-aware DNN parallelization technique called CAP, by exploiting the elasticity and noise-tolerance of deep learning algorithms on CMP. Moreover, in the hope that the conducted studies can provide new design values for real-time neural network inference on embedded chips, we also have evaluated the proposed approach on both multi-core Neural Network Accelerators (NNA) chips and general-purpose chip-multiprocessors. Our experimental results show that the proposed CAP can achieve 1.12×-1.65× system speedups and 1.14×-2.70× energy efficiency for different neural networks while maintaining the inference accuracy, compared to baseline approaches.

Online frequency-based performance and power estimation for clustered multi-processor systems

Modern Chip Multi-Processors (CMPs) are required to be increasingly power efficient while also offering higher performance and lower costs. A combination of Dynamic Voltage–Frequency Scaling (DVFS) and sophisticated resource-aware scheduling is needed to address the underlying problem of maximizing performance-per-Watt of CMP architectures. In this paper, we propose a methodology to predict the power consumption and performance for groups of concurrently executing applications at all available frequencies of a CMP. The methodology uses a combination of hardware-based application profiling, contention-aware scheduling, and artificial neural networks. Experimental results on an Odroid-XU3 board demonstrate an increase in average performance per Watt of 30.5% (A15 cluster) and 11.4% (A7 cluster) over Linux’s Completely Fair Scheduler (CFS) and power governors. In addition, our methodology outperforms three state-of-the-art resource managers, yielding the highest performance per Watt in all evaluated use cases.

Hardware-Level Thread Migration to Reduce On-Chip Data Movement Via Reinforcement Learning

As the number of processing cores and associated threads in chip multiprocessors (CMPs) continues to scale out, on-chip memory access latency dominates application execution time due to increased data movement. Although tiled CMP architectures with distributed shared caches provide a scalable design, increased physical distance between requesting and responding cores has led to both increased on-chip memory access latency and excess energy consumption. Near data processing is a promising approach that can migrate threads closer to data, however prior hand-engineered rules for fine-grained hardware-level thread migration are either too slow to react to changes in data access patterns, or unable to exploit the large variety of data access patterns. In this article, we propose to use reinforcement learning (RL) to learn relatively complex data access patterns to improve on hardware-level thread migration techniques. By utilizing the recent history of memory access locations as input, each thread learns to recognize the relationship between prior access patterns and future memory access locations. This leads to the unique ability of the proposed technique to make fewer, more effective migrations to intermediate cores that minimize the distance to multiple distinct memory access locations. By allowing a low-overhead RL agent to learn a policy from real interaction with parallel programming benchmarks in a parallel simulator, we show that a migration policy which recognizes more complex data access patterns can be learned. The proposed approach reduces on-chip data movement and energy consumption by an average of 41%, while reducing execution time by 43% when compared to a simple baseline with no thread migration; furthermore, energy consumption and execution time are reduced by an additional 10% when compared to a hand-engineered fine-grained migration policy.

A Table-Free Approximate Q-Learning-Based Thermal-Aware Adaptive Routing for Optical NoCs

Optical networks-on-chips (NoCs) based on silicon photonics have been proposed as an emerging communication architecture for many-core chip multiprocessors. However, the thermal sensitivity of silicon photonics is one of the major challenges. $Q$ -learning-based adaptive routing has been proposed in related work to mitigate the thermal issue. However, table overhead of the traditional table-based $Q$ -routing would scale up quickly with the increase of network size. In this article, we propose a table-free approximate $Q$ -learning-based thermal-aware adaptive routing to find optimal low-loss paths in the presence of on-chip temperature variations. The simulation results show that the proposed table-free approximate $Q$ -learning-based adaptive routing can converge faster and it can achieve similar optimization effect as compared to the best optimization effect of the traditional table-based $Q$ -routing. The performance gap between the proposed approximation method and the traditional table-based $Q$ -routing expands when the network size increases.

Novel fairness-aware co-scheduling for shared cache contention game on chip multiprocessors

Threads running on different cores of chip multiprocessors (CMP) can cause thread performance degradation due to contention for shared resources such as shared L2 cache. Some studies have shown that thread co-scheduling can effectively reduce contention for shared resources. However, in a multi-core system with shared caches, mutual interference between threads is unpredictable. As the number of cores increases, we are unlikely to exhaust all possible co-scheduling schemes. In this paper, a novel fairness-aware thread co-scheduling algorithm base on non-cooperative game is proposed to reduce L2 cache misses. We tried to improve the overall performance of the system by scheduling threads fairly. The originality of this work is to model thread scheduling using a non-cooperative game. The execution time of a thread varies depending on which threads are running on other cores of the same chip, because different thread combinations result in different levels of cache contention. Given the interdependence and competition between threads on the CMP architecture, non-cooperative game is used to solve the problem of thread co-scheduling where each thread is considered as a participant in the game. An iterative algorithm (IA) is proposed to solve the Nash equilibrium of the non-cooperative game in this paper. Subsequently, it is theoretically proved that IA has a potential game process and finally proves that IA can converge to Nash equilibrium in N iterations, where N is the number of threads. The co-scheduling scheme of all threads is obtained by solving the Nash equilibrium of the IA. Finally, the convergence and effectiveness of IA proposed in this paper is verified by experiments. In addition, we use the cache partition to improve the performance of IA. Experimental results show that the number of total cache misses of IA is less than that of the default scheduling algorithm, IA combined with cache partitioning can further reduce the total cache misses.

OS Level Power Optimization in LCMP

With the increasing levels of transistor count and clock rate of microprocessors there is a significant increase in power dissipation. Reducing power consumption in both high power consumption and high performance has developed into one of the main target in designing a system for various devices. As the chip multiprocessor (CMP) are integrating more cores on the die, it will leads to the extent of Large scale CMP (LCMP) architectures with potentially hundreds of thread on the die and thousands of cores. Therefore, we proposed an approach of OS level power optimization in LCMP to optimize the heat dissipation rate and increase computing power under some considerations. To satisfy the main goal of our work, the heat dissipation should be optimizing with increase in computing power. The approach of optimizing the heat dissipation is done at the synthesis level. There are three approaches for modifying the synthetic benchmark: Singly Synthesis, Hierarchical Synthesis and Group Synthesis. The result is that the power dissipation of Group synthesis is equally distributed without giving more loads to only one processor as compared to Hierarchical Synthesis and Singly Synthesis. Therefore, from our result we can conclude that in Group Synthesis power is equally distributed hence heat dissipation is optimized. The future work will be to further optimize the result of the Synthesis level using thread migration. Thread Migration can increase the system throughput; it relies on multiple cores that vary in performance capabilities.

A thermal-sensitive design of a 3D torus-based optical NoC architecture

In order to overcome limitations of traditional electronic interconnects in terms of power efficiency and bandwidth density, optical networks-on-chip (NoC) based on silicon photonics have been proposed as an emerging on-chip communication architecture for chip multiprocessors (CMPs) with large core counts. However, due to thermo-optic effects, wavelength-selective silicon photonic devices such as microresonators suffer from temperature-dependent wavelength shift. In this work, we propose a thermal-sensitive design of a 3D torus-based optical NoC architecture. For the 3D torus-based optical NoC architecture, we propose a hybrid optical-electronic router architecture with a fully-connected 7 × 7 optical switching fabric. Besides, a thermal-sensitive routing algorithm is proposed to optimize the optical power loss in the presence of on-chip temperature variations. Simulation results show that for a set of synthetic traffic patterns, the proposed 3D torus-based optical NoC with the thermal-sensitive routing reduces the average thermal-induced optical power loss by 14.3% and 17% respectively as compared to the matched 3D torus-based optical NoC and the 3D mesh-based optical NoC with the traditional XYZ routing. For a set of real applications, the proposed 3D torus-based optical NoC with the thermal-sensitive routing reduces the worst-case optical power loss by 7.9% and 14.6% respectively as compared to the matched 3D torus-based optical NoC and the 3D mesh-based optical NoC with the traditional XYZ routing.

Loop Parallelism Maximization for Multimedia Data Processing in Mobile Vehicular Clouds

Mobile vehicular cloud has become popular with the rapid development of cloud computing and mobile computing. Nested loops are usually the most critical part in multimedia and high performance Digital Signal Processing (DSP) systems which are widely used in vehicular applications and systems. In order to further explore the parallelism in nested loops, we study how to maximize the system performance with considering the energy reduction for applications on Chip Multiprocessor (CMP) architectures. We propose an algorithm Energy-Aware Loop Parallelism Maximization (EALPM) to maximize the system performance with the consideration of energy reduction for applications with multidimensional nested loops. Our experiment shows that using the EALPM algorithm significantly improves both performance and energy consumption on average in comparision to other algorithms.

Analysis and approximation of optimal co-scheduling on cmp

In recent years, the increasing design complexity and the problems of power and heat dissipation have caused a shift in processor technology to favor Chip Multiprocessors. In Chip Multiprocessors (CMP) architecture, it is common that multiple cores share some on-chip cache. The sharing may cause cache thrashing and contention among co-running jobs. Job co-scheduling is an approach to tackling the problem by assigning jobs to cores appropriately so that the contention and consequent performance degradations are minimized. This dissertation aims to tackle two of the most prominent challenges in job co-scheduling. The first challenge is in the computational complexity for determining optimal job co-schedules. This dissertation presents one of the first systematic analyses on the complexity of job co-scheduling. Besides proving the NP completeness of job co-scheduling, it introduces a set of algorithms, based on graph theory and Integer/Linear Programming, for computing optimal co-schedules or their lower bounds in scenarios with or without job migrations. For complex cases, it empirically demonstrates the feasibility for approximating the optimal schedules effectively by proposing several heuristics-based algorithms. These discoveries facilitate the assessment of job co-schedulers by providing necessary baselines, and shed insights to the development of practical co-scheduling systems. The second challenge resides in the prediction of the performance of processes co-running on a shared cache. This dissertation explores the influence on co-run performance prediction imposed by co-runners, program inputs, and cache configurations. Through a sequence of formal analysis, we derive an analytical co-run locality model, uncovering the inherent statistical connections between the data references of programs single-runs and their co-run locality. The model offers theoretical insights on co-run locality analysis and leads to a lightweight approach for fast prediction of shared cache performance. We demonstrate the effectiveness of the model in enabling proactive job co-scheduling. Together, the two-dimensional findings open up many new opportunities for cache management on modern CMP by laying the foundation for job co-scheduling, and enhancing the understanding to data locality and cache sharing significantly.

Low-power heterogeneous uncore architecture for future 3D chip-multiprocessors

Low-power heterogeneous uncore architecture for future 3D chip-multiprocessors

Dynamic Lifetime Reliability Management for Chip Multiprocessors

We introduce an algorithm for dynamic lifetime reliability optimization of chip multiprocessors (CMPs). The proposed dynamic reliability management (DRM) algorithm combines thread migration and dynamic voltage and frequency scaling (DVFS) as the two primary techniques to change the CMP operation. The goal is to increase the lifetime reliability of the overall system to the desired target with minimal performance degradation. We test the proposed algorithm with a variety of benchmarks on 16 and 64 core network-on-chip (NoC) based CMP architectures. Full-system based simulations using a customized GEM5 simulator demonstrate that lifetime reliability can be improved by 100 percent for an average performance penalty of 7.7 and 8.7 percent for the two CMP architectures.

A novel power model for future heterogeneous 3D chip-multiprocessors in the dark silicon age

Dark silicon has recently emerged as a new problem in VLSI technology. Maximizing performance of chip-multiprocessors (CMPs) under power and thermal constraints is very challenging in the dark silicon era. Providing next-generation analytical models for future CMPs which consider the impact of power consumption of core and uncore components such as cache hierarchy and on-chip interconnect that consume significant portion of the on-chip power consumption is largely unexplored. In this article, we propose a detailed power model which is useful for future CMP power modeling. In the proposed architecture for future CMPs, we exploit emerging technologies such as non-volatile memories (NVMs) and 3D techniques to combat dark silicon. Results extracted from the simulations are compared with those obtained from the analytical model. Comparisons show that the proposed model accurately estimates the power consumption of CMPs running both multi-threaded and multi-programed workloads.

Dynamic energy management for chip multi-processors under performance constraints

We introduce a novel algorithm for dynamic energy management (DEM) under performance constraints in chip multi-processors (CMPs). Using the novel concept of delayed instructions count, performance loss estimations are calculated at the end of each control period for each core. In addition, a Kalman filtering based approach is employed to predict workload in the next control period for which voltage-frequency pairs must be selected. This selection is done with a novel dynamic voltage and frequency scaling (DVFS) algorithm whose objective is to reduce energy consumption but without degrading performance beyond the user set threshold. Using our customized Sniper based CMP system simulation framework, we demonstrate the effectiveness of the proposed algorithm for a variety of benchmarks for 16 core and 64 core network-on-chip based CMP architectures. Simulation results show consistent energy savings across the board. We present our work as an investigation of the tradeoff between the achievable energy reduction via DVFS when predictions are done using the effective Kalman filter for different performance penalty thresholds.

Optimization-based power and thermal management for dark silicon aware 3D chip multiprocessors using heterogeneous cache hierarchy

Management of a problem recently known as “dark silicon” is a new challenge in multicore designs. Prior innovative studies have addressed the dark silicon problem in the fields of power-efficient core design. However, addressing dark silicon challenges in uncore component designs such as cache hierarchy, on-chip interconnect etc. that consume significant portion of the on-chip power consumption is largely unexplored. In this paper, for the first time, we propose an integrated approach which considers the impact of power consumption of core and uncore components simultaneously to improve multi/many-core performance in the dark silicon era. The proposed approach dynamically (1) predicts the changing program behavior on each core; (2) re-determines frequency/voltage, cache capacity and technology in each level of the cache hierarchy based on the program's scalability in order to satisfy the power and temperature constraints. In the proposed architecture, for future chip-multiprocessors (CMPs), we exploit emerging technologies such as non-volatile memories (NVMs) and 3D techniques to combat dark silicon. Also, for the first time, we propose a detailed power model which is useful for future dark silicon CMPs power modeling. Experimental results on SPEC 2000/2006 benchmarks show that the proposed method improves throughput by about 54.3% and energy-delay product by about 61% on average, respectively, in comparison with the conventional CMP architecture with homogenous cache system.(A preliminary short version of this work was presented in the 18th Euromicro Conference on Digital System Design (DSD), 2015.)

An OpenCL software compilation framework targeting an SoC-FPGA VLIW chip multiprocessor

Modern systems-on-chip augment their baseline CPU with coprocessors and accelerators to increase overall computational capability and power efficiency, and thus have evolved into heterogeneous multi-core systems. Several languages have been developed to enable this paradigm shift, including CUDA and OpenCL. This paper discusses a unified compilation environment to enable heterogeneous system design through the use of OpenCL and a highly configurable VLIW Chip Multiprocessor architecture known as the LE1. An LLVM compilation framework was researched and a prototype developed to enable the execution of OpenCL applications on a number of hardware configurations of the LE1 CMP. The presented OpenCL framework fully automates the compilation flow and supports work-item coalescing which better maps onto the ILP processor cores of the LE1 architecture. This paper discusses in detail both the software stack and target hardware architecture and evaluates the scalability of the proposed framework by running 12 industry-standard OpenCL benchmarks drawn from the AMD SDK and the Rodinia suites. The benchmarks are executed on 40 LE1 configurations with 10 implemented on an SoC-FPGA and the remaining on a cycle-accurate simulator. Across 12 OpenCL benchmarks results demonstrate near-linear wall-clock performance improvement of 1.8 × (using 2 dual-issue cores), up to 5.2 × (using 8 dual-issue cores) and on one case, super-linear improvement of 8.4 × (FixOffset kernel, 8 dual-issue cores). The number of OpenCL benchmarks evaluated makes this study one of the most complete in the literature.

An optically-enabled chip–multiprocessor architecture using a single-level shared optical cache memory

We present an optical bus-based chip–multiprocessor architecture where the processing cores share an optical single-level cache implemented in a separate chip next to the Central-Processing-Unit (CPU) die. The interconnection system is realized through Wavelength-Division-Multiplexed optical interfaces connecting the shared cache with the cores and the Main-Memory via spatial-multiplexed waveguides. Evaluating the proposed approach, we realize system-level simulations of a wide-range parallel workloads using Gem5. Optical cache architecture is compared against the conventional one that uses dedicated on-chip Level-1 electronic caches and a shared Level-2 cache. Results show significant Level-1 miss rate reduction of up to 96% for certain cases; on average, a performance speed-up of 19.4% or cache capacity requirements reduction of ~63% is attained. Combined with high-bandwidth CPU-Dynamic Random Access Memory (DRAM) bus solutions based on optical interconnects, the proposed design is a promising architecture bridging the gap between high-speed optically connected CPU-DRAM schemes and high-speed optical memory technologies.

A Holistic Approach Towards Intelligent Hotspot Prevention in Network-on-Chip-Based Multicores

Traffic hotspots, a severe form of network congestion, can be caused unexpectedly in a network-on-chip (NoC) due to the immanent spatio-temporal unevenness of application traffic. Hotspots reduce the NoC's effective throughput, where in the worst-case scenario, network traffic flows can be frozen indefinitely. To alleviate this problematic phenomenon several adaptive routing algorithms employ online load-balancing schemes, aiming to reduce the possibility of hotspots arising. Since most are not explicitly hotspotagnostic, they cannot completely prevent hotspot formation(s) as their reactive capability to hotspots is merely passive. This paper presents a pro-active Hotspot-Preventive Routing Algorithm (HPRA) which uses the advance knowledge gained from network embedded artificial neural network-based (ANN) hotspot predictors to guide packet routing in mitigating any unforeseen near-future hotspot occurrences. First, these ANN-based predictors are trained offline and during multicore operation they gather online statistical data to predict about-to-be-formed hotspots, promptly informing HPRA to take appropriate hotspot-preventive action(s). Next, in a holistic approach, additional ANN training is performed with data acquired after HPRA interferes, so as to further improve hotspot prediction accuracy; hence, the ANN mechanism does not only predict hotspots, but is also aware of changes that HPRA imposes upon the interconnect infrastructure. Evaluation results, including utilizing real application traffic traces gathered from parallelized workload executions onto a chip multiprocessor architecture, show that HPRA can improve network throughput up to 81 percent when compared with prior-art. Hardware synthesis results affirm the HPRA mechanism's moderate overhead requisites.

Reliability aware throughput management of chip multi-processor architecture via thread migration

Integrating the large number of transistor in a single chip leads to significant improvement on the performance of processors. More performance is achieved by putting multiple CPU cores on a single chip which is named as chip multiprocessor (CMP) architecture. On the other hand, miniaturization and integration of the large number of transistors in new silicons such as CMPs increase susceptibility to soft errors and degrade the reliability. Previous researches have exploited traditional redundancy techniques such as dual and triple cores redundancy to tolerate fault in CMP architecture while these methods impose significant performance and energy overheads. In this paper, we present a performance efficient soft error protection scheme for CMP architecture which is based on simultaneous multithreading. Fortunately, some of soft errors are masked at architectural level and don't cause visible output error. Soft error masking effect can be used to decrease a lot of overheads in reliability enhancement techniques against soft errors. Recently, architectural vulnerability factor (AVF) is widely used for estimating the portion of soft errors which are masked. In this article, we propose a reliability aware CMP architecture which use online AVF estimation to specify level of protection. To meet system reliability demands, the estimated AVF is used to exploit partial redundancy against soft errors which leads to significant performance improvement. Also, we introduce a dynamic scheduling method for mapping threads on the cores to enhance total throughput of CMP architecture. Our dynamic scheduling applies thread migration among cores by simultaneous considering to the total vulnerability and throughput of cores. Thread migration between cores balances loads between cores and improves performance. Our experimental results on SPEC CPU2006 show up to 38 % improvement in core throughput in different phases of thread migration compared to static mapping of threads on the cores.

Post-Silicon Validation of Multiprocessor Memory Consistency

Shared-memory chip-multiprocessor (CMP) architectures define memory consistency models that establish the ordering rules for memory operations from multiple threads. Validating the correctness of a CMP's implementation of its memory consistency model requires extensive monitoring and analysis of memory accesses while multiple threads are executing on the CMP. In this paper, we present a low overhead solution for observing, recording and analyzing shared-memory interactions for use in an emulation and/or post-silicon validation environment. Our approach leverages portions of the CMP's own data caches, augmented only by a small amount of hardware logic, to log information relevant to memory accesses. After transferring this information to a central memory location, we deploy our own analysis algorithm to detect any possible memory consistency violations. We build on the property that a violation corresponds to a cycle in an appropriately defined graph representing memory interactions. The solution we propose allows a designer to choose where to run the analysis algorithm: 1) on the CMP itself; 2) on a separate processor residing on the validation platform; or 3) off-line on a separate host machine. Our experimental results show an 83% bug detection rate, in our testbed CMP, over three distinct memory consistency models, namely: relaxed-memory order, total-store order, and sequential consistency. Finally, note that our solution can be disabled in the final product, leading to zero performance overhead and a per-core area overhead that is smaller than the size of a physical integer register file in a modern processor.

Maximizing Yield per Area of Highly Parallel CMPs Using Hardware Redundancy

The manufacturing yield of chip multiprocessors (CMPs) has become a significant problem as more transistors are integrated onto a single die and the defect rate keeps increasing for “end-of-Moore” nano-scale CMOS technologies. Since such CMP designs usually have significant structural symmetry, adding spare copies to these should be an effective method for increasing yield per area, as is the case for memories. However, a systematic approach to add spare copies to optimize CMP yield per area has never been developed, primarily due to the lack of: 1) a general model of CMP architectures and 2) a practically-useable model for computing areas of chip versions with different configurations of spare copies. This paper develops such models and, in conjunction with a systematic approach for enumerating a wide range of spare configurations, uses these to compute the area overhead and yield for each configuration. In particular, this paper proposes a general spare cores sharing technique to maximize yield per area of any CMP by efficiently traversing the design space for adding spare cores. Experimental results show that the advantage of the proposed approach over traditional approaches increases with continued technology scaling. Specifically, the proposed approach achieves \(2\times \) yields per area over previous approaches for 32 nm and 22 nm technologies. Also, the obtained yield per area values provided by our approach are around 70% of that obtained for the ideal scenario where defect density is zero and no redundancy is added.