Finding Performance Bottlenecks Research Articles

FuzzFactory: domain-specific fuzzing with waypoints

Coverage-guided fuzz testing has gained prominence as a highly effective method of finding security vulnerabilities such as buffer overflows in programs that parse binary data. Recently, researchers have introduced various specializations to the coverage-guided fuzzing algorithm for different domain-specific testing goals, such as finding performance bottlenecks, generating valid inputs, handling magic-byte comparisons, etc. Each such solution can require non-trivial implementation effort and produces a distinct variant of a fuzzing tool. We observe that many of these domain-specific solutions follow a common solution pattern. In this paper, we present FuzzFactory, a framework for developing domain-specific fuzzing applications without requiring changes to mutation and search heuristics. FuzzFactory allows users to specify the collection of dynamic domain-specific feedback during test execution, as well as how such feedback should be aggregated. FuzzFactory uses this information to selectively save intermediate inputs, called waypoints, to augment coverage-guided fuzzing. Such waypoints always make progress towards domain-specific multi-dimensional objectives. We instantiate six domain-specific fuzzing applications using FuzzFactory: three re-implementations of prior work and three novel solutions, and evaluate their effectiveness on benchmarks from Google's fuzzer test suite. We also show how multiple domains can be composed to perform better than the sum of their parts. For example, we combine domain-specific feedback about strict equality comparisons and dynamic memory allocations, to enable the automatic generation of LZ4 bombs and PNG bombs.

FOREPOST: finding performance problems automatically with feedback-directed learning software testing

A goal of performance testing is to find situations when applications unexpectedly exhibit worsened characteristics for certain combinations of input values. A fundamental question of performance testing is how to select a manageable subset of the input data faster in order to automatically find performance bottlenecks in applications. We propose FOREPOST, a novel solution, for automatically finding performance bottlenecks in applications using black-box software testing. Our solution is an adaptive, feedback-directed learning testing system that learns rules from execution traces of applications. Theses rules are then used to automatically select test input data for performance testing. We hypothesize that FOREPOST can find more performance bottlenecks as compared to random testing. We have implemented our solution and applied it to a medium-size industrial application at a major insurance company and to two open-source applications. Performance bottlenecks were found automatically and confirmed by experienced testers and developers. We also thoroughly studied the factors (or independent variables) that impact the results of FOREPOST.

A statistical performance analyzer framework for OpenCL kernels on Nvidia GPUs

Understanding performance bottlenecks of applications in high-performance computing can lead to dramatic improvements in their performances. For example, a key problem in GPU programming is finding performance bottlenecks and solving them to reach the best possible performance. These bottlenecks in GPU architectures include a number of factors such as memory access latency, branch divergence, utilization, and the amount of existing parallelism. In addition, a simple profiling cannot demonstrate the relations between these bottlenecks. In this paper, we propose a statistical performance analyzer framework that not only helps us find bottlenecks, but also indicates the relations between them, which is not possible using a profiler. Recently, OpenCL has been proposed to be used in a variety of platforms, e.g., CPUs and GPUs, enabling a program written in one platform to be imported to other platforms with minimal effort. Therefore, we selected OpenCL to design our performance model for Nvidia GPUs. To construct the model, the values of GPU performance counters for the selected benchmarks are measured. Then, well-known statistical techniques such as regression and principal component analysis use these results to find the most significant parameters and to construct a performance model with up to 99 % accuracy. Finally, this method can be leveraged to characterize unknown applications based on their performance similarities with an existing database of benchmarks to predict their likely performance bottlenecks.

Special Issue: Scalable Tools for High‐end Computing

Current high-end parallel systems consist of hundreds of thousands of compute cores arranged in a complex hierarchical structure; future systems will have millions of cores. Systems, such as the Altix 4700, Blue Gene, Roadrunner, and Cray XT5, deploy multiple compute cores (homogeneous or heterogeneous) with multiple levels of shared and private caches within a processor, clustered into SMP nodes and coupled via a communication network to large-scale distributed systems. The development of efficient programs is extremely complex since the architectural details are exposed to the programmer. Productive use of such machines requires highly scalable programming tools for debugging, performance analysis, and fault tolerance. In addition, new programming models might significantly facilitate the task of the programmer. This special issue of Concurrency and Computation: Practice and Experience is devoted to programming tools that facilitate the development of efficient programs for such large-scale architectures. It is a collection of the best papers submitted to the international workshop on Scalable Tools for High-end Computing (STHEC 2008) that was held in conjunction with the International Conference on Supercomputing on June 7th on the Greek Island Kos. The papers present state-of-the-art tools for performance analysis and checkpointing on those machines. Performance analysis tools use measurements gathered during the execution of the application to detect portions of the code that can be further improved. Thus, they have to be able to cope with the large number of processors. Tools for checkpointing provide the possibility to restart an application in the case of a system failure; they have to be able to handle large number of cores as well. The selected papers present different techniques for building tools that will scale to thousands of cores. HPCToolkit 1 is a profiling-based performance analysis environment presenting the data in close relation to the source code without requiring an instrumentation of the source code. Scalasca 2 performs a parallel replay of the execution on the application's processors to find performance bottlenecks automatically. The combination of TAU and MRNet 3 provides a scalable infrastructure to offload performance data. Establishing the overlay network requires no added support from the job manager or application. Periscope 4 is based on a network of analysis agents that performs an online analysis of the application's performance behavior. When the application is started, additional processors can be allocated for the analysis agents to scale the analysis. CPPC 5 is a tool for portable checkpointing of message-passing applications. It consists of a runtime library and a compiler that assists the user by performing time-consuming tasks, such as data flow and communications analyses as well as code instrumentation. We would like to thank the authors for their excellent contributions to this special issue. We hope that it inspires future research in tools that support programmers of high-end systems in the development of efficient programs.

Scalability analysis of three monitoring and information systems: MDS2, R-GMA, and Hawkeye

Monitoring and information system (MIS) implementations provide data about available resources and services within a distributed system, or Grid. A comprehensive performance evaluation of an MIS can aid in detecting potential bottlenecks, advise in deployment, and help improve future system development. In this paper, we analyze and compare the performance of three implementations in a quantitative manner: the Globus Toolkit ® Monitoring and Discovery Service (MDS2), the European DataGrid Relational Grid Monitoring Architecture (R-GMA), and the Condor project's Hawkeye. We use the NetLogger toolkit to instrument the main service components of each MIS and conduct four sets of experiments to benchmark their scalability with respect to the number of users, the number of resources, and the amount of data collected. Our study provides quantitative measurements comparable across all systems. We also find performance bottlenecks and identify how they relate to the design goals, underlying architectures, and implementation technologies of the corresponding MIS, and we present guidelines for deploying MISs in practice.

Resource consumption-aware QoS in cluster-based VOD servers

For Video-On-Demand (VOD) systems, it is important to provide Quality of Service (QoS) to more clients under limited resources. In this paper, the performance scalability in cluster-based VOD servers is studied with several grouping configurations of cluster nodes. To find performance bottlenecks, the monitoring functions are employed and the maximum QoS streams are measured under the various requests including VCR functions. To support more user friendly interface, an embedded set-top model is suggested for the QoS of TV clients. From our detailed experiment results, a new admission control method is proposed that is based on available system resources and the actual amount of resource consumed for QoS streams. The proposed method provides not only more scalable QoS in cluster-based VOD servers but also the enhancement of resource utilization by guaranteeing the maximum number of QoS streams.

Shared-memory performance profiling

This paper describes a new approach to finding performance bottlenecks in shared-memory parallel programs and its embodiment in the Paradyn Parallel Performance Tools running with the Blizzard fine-grain distributed shared memory system. This approach exploits the underlying system's cache coherence protocol to detect data sharing patterns that indicate potential performance bottlenecks and presents performance measurements in a data-centric manner. As a demonstration, Parodyn helped us improve the performance of a new shared-memory application program by a factor of four.