Roofline Model Research Articles

GPU Parallelization of a Hybrid Pseudospectral Geophysical Turbulence Framework Using CUDA

An existing hybrid MPI-OpenMP scheme is augmented with a CUDA-based fine grain parallelization approach for multidimensional distributed Fourier transforms, in a well-characterized pseudospectral fluid turbulence code. Basics of the hybrid scheme are reviewed, and heuristics provided to show a potential benefit of the CUDA implementation. The method draws heavily on the CUDA runtime library to handle memory management and on the cuFFT library for computing local FFTs. The manner in which the interfaces to these libraries are constructed, and ISO bindings utilized to facilitate platform portability, are discussed. CUDA streams are implemented to overlap data transfer with cuFFT computation. Testing with a baseline solver demonstrated significant aggregate speed-up over the hybrid MPI-OpenMP solver by offloading to GPUs on an NVLink-based test system. While the batch streamed approach provided little benefit with NVLink, we saw a performance gain of 30 % when tuned for the optimal number of streams on a PCIe-based system. It was found that strong GPU scaling is nearly ideal, in all cases. Profiling of the CUDA kernels shows that the transform computation achieves 15% of the attainable peak FlOp-rate based on a roofline model for the system. In addition to speed-up measurements for the fiducial solver, we also considered several other solvers with different numbers of transform operations and found that aggregate speed-ups are nearly constant for all solvers.

Application-driven Cache-Aware Roofline Model

In the coming exascale era, the complexity of modern applications and hardware resources imposes significant challenges for boosting the efficiency via execution fine-tuning. To abstract this complexity in an intuitive way, recent application analysis tools rely on insightful modeling, e.g., Intel® Advisor with Cache-aware Roofline Model. However, these approaches mainly consider the maximum architecture capabilities, which may limit their usability when characterizing real-world applications. To address this issue, a novel Cache-Aware Roofline Model for more accurate performance modeling of multi-cores is proposed, which realistically resembles application requirements. The proposed fine-grain modeling relies on micro-benchmarking to decouple the attainable performance of the micro-architecture for different utilization scenarios and for a diverse set of functional units and memory levels. Memory sub-system traffic simulation, dynamic and static analyses are also used to derive the requirements of the applications. Experimental results for a real multi-core system with an Intel server processor and for a set of 13 kernels from exascale proxy applications, show that the proposed models provide more accurate application characterization, optimization hints and bottleneck detection in comparison to the state-of-the-art models.

Computer Medical Image Segmentation Based on Neural Network

Image segmentation in medical imaging has long been a problem in radiological image processing. Most of the image segmentation methods in traditional vision algorithms are difficult to achieve high-resolution image segmentation due to the complexity of the algorithm. This article proposes an image segmentation method based on an optimized cellular neural network. This method introduces a non-linear template and data quantization on the basis of a basic network model, which greatly reduces the computational complexity while maintaining the accuracy of image segmentation. We then applied the method to a computer-aided system to classify tumor lesions in mammograms. Finally, we propose an FPGA-based multilevel optimization architecture for energy-efficient cellular neural networks. The optimization scheme includes three levels: system level, module level, and design space. This solution improves computing performance by increasing system parallelism, using data reuse technology to fully utilize loading bandwidth, and using data quantization to reduce computational redundancy. It also introduces pipeline and dual cache structures to optimize memory access, and analyzes the limited resources through the Roofline model. System for best performance. The experimental results show that the FPGA accelerator in this article can improve unit performance by 34% compared with other existing research work. The nonlinear quantified cellular neural network proposed in this article can reduce LUT resource consumption by 74% and energy of 48.2%. Compared with the original network, in the two projection position segmentation results of the mammogram, only 1.5% and 0.6% of the accuracy loss, respectively.

The Sky Is Not the Limit: A Visual Performance Model for Cyber-Physical Co-Design in Autonomous Machines

We introduce the “Formula-1” (F-1) roofline model to understand the role of computing in aerial autonomous machines. The model provides insights by exploiting the fundamental relationships between various components in an aerial robot, such as sensor framerate, compute performance, and body dynamics (physics). F-1 serves as a tool that can aid computer and cyber-physical system architects to understand the optimal design (or selection) of various components in the development of autonomous machines.

Performance analysis of deep learning workloads using roofline trajectories

Over the last decade, technologies derived from convolutional neural networks (CNNs) called Deep Learning applications, have revolutionized fields as diverse as cancer detection, self-driving cars, virtual assistants, etc. However, many users of such applications are not experts in Machine Learning itself. Consequently, there is limited knowledge among the community to run such applications in an optimized manner. The performance question for Deep Learning applications has typically been addressed by employing bespoke hardware (e.g., GPUs) better suited for such compute-intensive operations. However, such a degree of performance is only accessibly at increasingly high financial costs leaving only big corporations and governments with resources sufficient enough to employ them at a large scale. As a result, an average user is only left with access to commodity clusters with, in many cases, only CPUs as the sole processing element. For such users to make effective use of resources at their disposal, concerted efforts are necessary to figure out optimal hardware and software configurations. This study is one such step in this direction as we use the Roofline model to perform a systematic analysis of representative CNN models and identify opportunities for black box and application-aware optimizations. Using the findings from our study, we are able to obtain up to 3.5$$\times$$ speedup compared to vanilla TensorFlow with default configurations.

Caffeine: Toward Uniformed Representation and Acceleration for Deep Convolutional Neural Networks

With the recent advancement of multilayer convolutional neural networks (CNNs) and fully connected networks (FCNs), deep learning has achieved amazing success in many areas, especially in visual content understanding and classification. To improve the performance and energy efficiency of the computation-demanding CNN, the FPGA-based acceleration emerges as one of the most attractive alternatives. In this paper, we design and implement Caffeine, a hardware/software co-designed library to efficiently accelerate the entire CNN and FCN on FPGAs. First, we propose a uniformed convolutional matrix-multiplication representation for both computation-bound convolutional layers and communication-bound FCN layers. Based on this representation, we optimize the accelerator micro-architecture and maximize the underlying FPGA computing and bandwidth resource utilization based on a revised roofline model. Moreover, we design an automation flow to directly compile high-level network definitions to the final FPGA accelerator. As a case study, we integrate Caffeine into the industry-standard software deep learning framework Caffe. We evaluate Caffeine and its integration with Caffe by implementing VGG16 and AlexNet networks on multiple FPGA platforms. Caffeine achieves a peak performance of 1460 giga fixed point operations per second on a medium-sized Xilinx KU060 FPGA board; to our knowledge, this is the best published result. It achieves more than $100\boldsymbol {\times }$ speed-up on FCN layers over prior FPGA accelerators. An end-to-end evaluation with Caffe integration shows up to $29\boldsymbol {\times }$ and $150\boldsymbol {\times }$ performance and energy gains over Caffe on a 12-core Xeon server, and $5.7\boldsymbol {\times }$ better energy efficiency over the GPU implementation. Performance projections for a system with a high-end FPGA (Virtex7 690t) show even higher gains.

Performance Limits Study of Stencil Codes on Modern GPGPUs

We study the performance limits of different algorithmic approaches to the implementation of a sample problem of wave equation solution with a cross stencil scheme. With this, we aim to find the highest limit of the achievable performance efficiency for stencil computing. To estimate the limits, we use a quantitative Roofline model to make a thorough analysis of the performance bottlenecks and develop the model further to account for the latency of different levels of GPU memory. These estimates provide an incentive to use spatial and temporal blocking algorithms. Thus, we study stepwise, domain decomposition, and domain decomposition with halo algorithms in that order. The knowledge of the limit incites the motivation to optimize the implementation. This led to the analysis of the block synchronization methods in CUDA, which is also provided in the text. After all optimizations, we have achieved 90% of the peak performance, which amounts to more than 1 trillion cell updates per second on one consumer level GPU device.

An FPGA-Based CNN Accelerator Integrating Depthwise Separable Convolution

The Convolutional Neural Network (CNN) has been used in many fields and has achieved remarkable results, such as image classification, face detection, and speech recognition. Compared to GPU (graphics processing unit) and ASIC, a FPGA (field programmable gate array)-based CNN accelerator has great advantages due to its low power consumption and reconfigurable property. However, FPGA’s extremely limited resources and CNN’s huge amount of parameters and computational complexity pose great challenges to the design. Based on the ZYNQ heterogeneous platform and the coordination of resource and bandwidth issues with the roofline model, the CNN accelerator we designed can accelerate both standard convolution and depthwise separable convolution with a high hardware resource rate. The accelerator can handle network layers of different scales through parameter configuration and maximizes bandwidth and achieves full pipelined by using a data stream interface and ping-pong on-chip cache. The experimental results show that the accelerator designed in this paper can achieve 17.11GOPS for 32bit floating point when it can also accelerate depthwise separable convolution, which has obvious advantages compared with other designs.

A Resources-Efficient Configurable Accelerator for Deep Convolutional Neural Networks

Deep convolutional neural networks (DCNNs) have become one of the most popular approaches to many visual processing tasks. The majority of existing works on the accelerating DCNNs focus on high performance while neglecting the hardware resource utilization, like on-chip memory and DSP. In this paper, we propose a resources-efficient and configurable DCNN accelerator. A four-level processing-element (PE)-array-based structure is presented to realize high parallelism calculation of convolutional operation, and a new storage pattern named hybrid stationary (HS) is proposed to take full advantage of the used on-chip memory footprint and limited off-chip memory bandwidth. Moreover, roofline model is adopted to explore the design space of the given hardware resources. The proposed architecture achieves 113 G-ops/s at 100 MHz and consumes 784 DSP48 modules and 211.5 Block RAM modules on ZYNQ-7 ZC706 evaluation board. To the best of our knowledge, the proposed accelerator is the only implemented system on FGPA platform that can achieve multiple advantages: high-performanced, configurable, and efficient in power and resources utilization. It shows significant utilization improvement compared with the other available architectures.

Accelerating Sequence Alignments Based on FM-Index Using the Intel KNL Processor.

FM-index is a compact data structure suitable for fast matches of short reads to large reference genomes. The matching algorithm using this index exhibits irregular memory access patterns that cause frequent cache misses, resulting in a memory bound problem. This paper analyzes different FM-index versions presented in the literature, focusing on those computing aspects related to the data access. As a result of the analysis, we propose a new organization of FM-index that minimizes the demand for memory bandwidth, allowing a great improvement of performance on processors with high-bandwidth memory, such as the second-generation Intel Xeon Phi (Knights Landing, or KNL), integrating ultra high-bandwidth stacked memory technology. As the roofline model shows, our implementation reaches 95 percent of the peak random access bandwidth limit when executed on the KNL and almost all of the available bandwidth when executed on other Intel Xeon architectures with conventional DDR memory. In addition, the obtained throughput in KNL is much higher than the results reported for GPUs in the literature.

Optimized Hierarchical Cascaded Processing

Recently, there has been an increasing demand for advanced classification capabilities embedded on wearable battery constrained devices, such as smartphones or watches. Achieving such functionality with a tight power and energy budget has proven a real challenge, specifically for large-scale neural network-based applications. Previously, cascaded systems have been proposed to minimize energy consumption for such applications, either through using a single wake-up stage, or by using a linear- or tree based cascade of consecutive classifiers that allow early termination. In this paper, we expand upon these concepts by generalizing cascades to hierarchical cascaded processing, where a hierarchy of increasingly complex classifiers, each designed and trained for a specific subtask is used. This hierarchical approach significantly outperforms the wake-up based approach by up to 2 orders of magnitude in energy consumption at iso-accuracy, specifically in systems with sparse input data such as speech recognition and visual object detection. This paper presents a general design framework for such systems and illustrates how to optimize them toward minimum energy consumption. The text further proposes a roofline model for cascaded systems, derives system level trade-offs and proves the approaches validity through a visual classification case-study.

Modeling Non-Uniform Memory Access on Large Compute Nodes with the Cache-Aware Roofline Model

NUMA platforms, emerging memory architectures with on-package high bandwidth memories bring new opportunities and challenges to bridge the gap between computing power and memory performance. Heterogeneous memory machines feature several performance trade-offs, depending on the kind of memory used, when writing or reading it. Finding memory performance upper-bounds subject to such trade-offs aligns with the numerous interests of measuring computing system performance. In particular, representing applications performance with respect to the platform performance bounds has been addressed in the state-of-the-art Cache-Aware Roofline Model (CARM) to troubleshoot performance issues. In this paper, we present a Locality-Aware extension (LARM) of the CARM to model NUMA platforms bottlenecks, such as contention and remote access. On top of this, the new contribution of this paper is the design and validation of a novel hybrid memory bandwidth model. This new hybrid model quantifies the achievable bandwidth upper-bound under above-described trade-offs with less than 3 percent error. Hence, when comparing applications performance with the maximum attainable performance, software designers can now rely on more accurate information.

MALOC: A Fully Pipelined FPGA Accelerator for Convolutional Neural Networks With All Layers Mapped on Chip

Recently, field-programmable gate arrays (FPGAs) have been widely used in the implementations of hardware accelerator for convolutional neural networks (CNNs). However, most of these existing accelerators are designed in the same idea as their ASIC counterparts, in which all operations from different layers are mapped to the same hardware units and working in a multiplexed way. This manner does not take full advantage of reconfigurability and customizability of FPGAs, resulting in a certain degree of computational efficiency degradation. In this paper, we propose a new architecture for FPGA-based CNN accelerator that maps all the layers to their own on-chip units and working concurrently as a pipeline. A comprehensive mapping and optimizing methodology based on establishing roofline model oriented optimization model is proposed, which can achieve maximum resource utilization as well as optimal computational efficiency. Besides, to ease the programming burden, we propose a design framework which can provide a one-stop function for developers to generate the accelerator with our optimizing methodology. We evaluate our proposal by implementing different modern CNN models on Xilinx Zynq-7020 and Virtex-7 690t FPGA platforms. Experimental results show that our implementations can achieve a peak performance of 910.2 GOPS on Virtex-7 690t, and 36.36 GOP/s/W energy efficiency on Zynq-7020, which are superior to the previous approaches.

Evaluating the SW26010 many-core processor with a micro-benchmark suite for performance optimizations

Abstract The inadequate public information of China’s SW26010 processor’s micro-architecture prevents global researchers from improving application performances on the TaihuLight supercomputer. This study aims to illuminate the uncharted area of SW26010 in order to provide important information for performance optimizations and modeling. First, we developed a micro-benchmark suite, swCandle, to evaluate the key micro-architectural features. The benchmark results revealed some unanticipated findings beyond the publicly available data. For instance, the broadcast mode of register communications has the same latency as the peer-to-peer mode. Second, we applied the roofline model, with the key parameters obtained with swCandle, to identify the key programming challenge of SW26010. Third, based on the micro-benchmark results and the roofline model analysis, we proposed a systematic guideline for performance optimizations on SW26010 and instantiated the guideline with two cases. The methodology we developed in this study, that infers a processor’s micro-architecture design from micro-benchmark results, can also be applied on other processors lacking of public information.

GPU-accelerated 3-D Finite Volume Particle Method

In previous works (Jahanbakhsh et al., CMAME 298 (2016): 80–107, Jahanbakhsh et al., CMAME 317 (2017): 102–127; see [1] and [2]), the authors introduced SPHEROS, a 3-D particle-based solver based on the Finite Volume Particle Method (FVPM) featuring a spherical top-hat kernel. In the present research, the authors present algorithms and optimization procedures that allowed to significantly accelerate computations by taking advantage of the computational power of Graphics Processing Units (GPUs). The new accelerated solver, GPU-SPHEROS, has been developed in CUDA and runs entirely on GPU, are presented. All the parallel algorithms and data structures have been designed specifically for the GPU many-core architecture. A roofline model has been utilized to assess the performance of the kernels and apply appropriate optimization strategies. In particular, the neighbor search algorithm, accounting for almost a third of the overall compute time, features an efficient Space-Filling Curve (SFC) as well as an optimized octree construction procedure. The memory-bound interaction vector computation, accounting for almost two thirds of the overall compute time, features fixed-size memory pre-allocation and an efficient data ordering to reduce memory transactions and cost of dynamic memory operations i.e. allocation and deallocation. As a case study, the numerical simulation results of water jet deviation by the rotating buckets in a Pelton turbine is presented and compared to available experimental data. For that case, a speedup by a factor of almost six times has been achieved on a single NVIDIA® Tesla™ P100-SXM2-16 GB GPU with GP100 Pascal architecture compared to a dual CPU node equipped with two Broadwell Intel® Xeon® E5-2690 v4 CPUs with 28 total physical cores.

Optimizing FPGA-based CNN accelerator for energy efficiency with an extended Roofline model

In recent years, the convolutional neural network (CNN) has found wide acceptance in solving practical computer vision and image recognition problems. Also recently, due to its exibility, faster development time, and energy efficiency, the field-programmable gate array (FPGA) has become an attractive solution to exploit the inherent parallelism in the feedforward process of the CNN. However, to meet the demands for high accuracy of today's practical recognition applications that typically have massive datasets, the sizes of CNNs have to be larger and deeper. Enlargement of the CNN aggravates the problem of off-chip memory bottleneck in the FPGA platform since there is not enough space to save large datasets on-chip. In this work, we propose a memory system architecture that best matches the off-chip memory traffic with the optimum throughput of the computation engine, while it operates at the maximum allowable frequency. With the help of an extended version of the Rooine model proposed in this work, we can estimate memory bandwidth utilization of the system at different operating frequencies since the proposed model considers operating frequency in addition to bandwidth utilization and throughput. In order to find the optimal solution that has the best energy efficiency, we make a trade-off between energy efficiency and computational throughput. This solution saves 18% of energy utilization with the trade-off having less than 2% reduction in throughput performance. We also propose to use a race-to-halt strategy to further improve the energy efficiency of the designed CNN accelerator. Experimental results show that our CNN accelerator can achieve a peak performance of 52.11 GFLOPS and energy efficiency of 10.02 GFLOPS/W on a ZYNQ ZC706 FPGA board running at 250 MHz, which outperforms most previous approaches.

FPGA design space exploration for scientific HPC applications using a fast and accurate cost model based on roofline analysis

High-performance computing on heterogeneous platforms in general and those with FPGAs in particular presents a significant programming challenge. We contend that compiler technology has to evolve to automatically optimize applications by transforming a given original program. We are developing a novel methodology based on type transformations on a functional description of a given scientific kernel, for generating correct-by-construction design variants. An associated lightweight costing mechanism for evaluating these variants is a cornerstone of our methodology, and the focus of this paper. We discuss our use of the roofline model to work with our optimizing compiler to enable us to quickly derive accurate estimates of performance from the design’s representation in our custom intermediate language. We show results confirming the accuracy of our cost model by validating it on different scientific kernels. A case study is presented to demonstrate that a solution created from our optimizing framework outperforms commercial high-level synthesis tools both in terms of throughput and power efficiency.

Performance prediction of finite-difference solvers for different computer architectures

The life-cycle of a partial differential equation (PDE) solver is often characterized by three development phases: the development of a stable numerical discretization; development of a correct (verified) implementation; and the optimization of the implementation for different computer architectures. Often it is only after significant time and effort has been invested that the performance bottlenecks of a PDE solver are fully understood, and the precise details varies between different computer architectures. One way to mitigate this issue is to establish a reliable performance model that allows a numerical analyst to make reliable predictions of how well a numerical method would perform on a given computer architecture, before embarking upon potentially long and expensive implementation and optimization phases. The availability of a reliable performance model also saves developer effort as it both informs the developer on what kind of optimisations are beneficial, and when the maximum expected performance has been reached and optimisation work should stop. We show how discretization of a wave-equation can be theoretically studied to understand the performance limitations of the method on modern computer architectures. We focus on the roofline model, now broadly used in the high-performance computing community, which considers the achievable performance in terms of the peak memory bandwidth and peak floating point performance of a computer with respect to algorithmic choices. A first principles analysis of operational intensity for key time-stepping finite-difference algorithms is presented. With this information available at the time of algorithm design, the expected performance on target computer systems can be used as a driver for algorithm design.

A quantitative roofline model for GPU kernel performance estimation using micro-benchmarks and hardware metric profiling

Typically, the execution time of a kernel on a GPU is a difficult to predict measure as it depends on a wide range of factors. Performance can be limited by either memory transfer, compute throughput or other latencies. In this paper, we improve on the roofline model following a quantitative approach and present a completely automated GPU performance prediction technique. In this respect this model utilizes micro-benchmarking and profiling in a “black box” fashion as no inspection of source/binary code is required. The proposed model combines parameters in order to characterize the performance limiting factor and to estimate execution time. In addition, we propose the quadrant-split visual representation, which captures the characteristics of multiple processors in relation to a particular kernel. We performed experiments on stencil computation (red/black SOR), SGEMM and a total of 28 kernels of the Rodinia benchmark suite, using six CUDA GPUs and we showed an absolute error in predictions of 27.66% in the average case. Furthermore, the performance model was also examined on an AMD GPU through the HIP programming environment. Prediction errors were comparable despite the significant architectural differences between different vendor GPUs.

Dynamic load balancing for CAFE multiscale modelling methods for heterogeneous hardware infrastructure

The paper presents new approach to Static and Dynamic Load Balancing (DBL) for loosely coupled multiscale modelling methods executed in heterogeneous hardware infrastructure. The most popular configurations of computing nodes composed of modern multicore CPUs, GPUs and co-processors, are used. The proposed load balancing approach takes into account previously determined computational character of methods applied in particular scales, which depends on a size of input data, operational intensity and limitations of hardware architecture. Such limitations are defined by the Roofline model and then used in the algorithm as boundary conditions, allowing to determine maximum performance of an algorithm for particular device. Multiscale calculations based on upscaling approach are analysed by using Cellular Automata Finite Element (CAFE) method. The qualitative results, obtained after application of proposed load balancing procedure, are discussed in the paper in details.