Workload Specification Research Articles

Optimization of block-scaled integer GeMMs for efficient DNN deployment on scalable in-order vector processors

A continuing rise in DNN usage in distributed and embedded use cases has demanded more efficient hardware execution in the field. Low-precision GeMMs with optimized data formats have played a key role in more memory and computationally-efficient networks. Recently trending formats are block-scaled representations stemming from tight HW-SW co-optimization, that compress network size by sharing exponents per data block. Prior work mostly focuses on deploying such block-scaled GeMM operations on domain-specific accelerators for optimum efficiency at the cost of flexibility and ease of deployment. In this work, we exploit and optimize the deployment of block-scaled GeMMs on fully-programmable in-order vector processors using ARM SVE. We define a systematic methodology for performing design space exploration to optimally match the workload specifications with processor vector-lengths, different microkernels, block sizes and shapes. We introduce efficient intrinsics-based microkernels with effective loop unrollings, and data-transfer efficient fused requantization strategies to maximize kernel performance, while also ensuring several deployment configurations. We enable generalized block-scaled kernel deployments through tunable block sizes and shapes, which helps in accommodating different accuracy-speed trade-off requirements. Utilizing 2D activation blocks instead of conventional 1D blocks, the static and dynamic BS-INT8 configurations yielded on average 3.8x and 2.9x faster speedups over FP32 models respectively, at no accuracy loss for CNN classification tasks on CIFAR10/100 datasets.

Multi-Model Workload Specifications and their application to Cyber-Physical Systems

Abstract To address the question of how to deliver time-sensitive software for Cyber-Physical Systems (CPS) requires a range of modelling and analysis techniques to be developed and integrated. A number of these required techniques are unique to time-sensitive software where timeliness is a correctness property rather than a performance attribute. This paper focuses on how to obtain worst-case estimates of the software’s execution time; in particular, it considers how workload models are derived from assumptions about the system’s run-time behaviour. The specific contribution of this paper is the exploration of the notion that a system can be subject to more than one workload model. Examples illustrate how such multi-models can lead to improved schedulability, and hence more efficient CPS. An important property of the approach is that the derived analysis exhibits model-bounded behaviour. This ensures that the maximum load on the system is never higher than that implied by the individual models.

LWS: A framework for log-based workload simulation in session-based SUT

Artificial intelligence for IT Operations (AIOps) plays a critical role in operating and managing cloud-native systems and microservice-based applications but is limited by the lack of high-quality datasets with diverse scenarios. Realistic workloads are the premise and basis of generating such AIOps datasets, with the session-based workload being one of the most typical examples. Due to privacy concerns, complexity, variety, and requirements for reasonable intervention, it is difficult to copy or generate such workloads directly, showing the importance of effective and intervenable workload simulation. In this paper, we formulate the task of workload simulation and propose a framework for Log-based Workload Simulation (LWS) in session-based systems. LWS extracts the workload specification including the user behavior abstraction based on agglomerative clustering as well as relational models and the intervenable workload intensity from session logs. Then LWS combines the user behavior abstraction with the workload intensity to generate simulated workloads. The experimental evaluation is performed on an open-source cloud-native application with both well-designed and public real-world workloads, showing that the simulated workload generated by LWS is effective and intervenable, which provides the foundation of generating high-quality AIOps datasets.

Design and Performance Characterization of RADICAL-Pilot on Leadership-Class Platforms

Many extreme scale scientific applications have workloads comprised of a large number of individual high-performance tasks. The Pilot abstraction decouples workload specification, resource management, and task execution via job placeholders and late-binding. As such, suitable implementations of the Pilot abstraction can support the collective execution of large number of tasks on supercomputers. We introduce RADICAL-Pilot (RP) as a portable, modular and extensible pilot-enabled runtime system. We describe RP's design, architecture and implementation. We characterize its performance and show its ability to scalably execute workloads comprised of tens of thousands heterogeneous tasks on DOE and NSF leadership-class HPC platforms. Specifically, we investigate RP's weak/strong scaling with CPU/GPU, single/multi core, (non)MPI tasks and Python functions when using most of ORNL Summit and TACC Frontera. RADICAL-Pilot can be used stand-alone, as well as the runtime for third-party workflow systems.

User learning can affect software system's response time

A session-based software system provides various services to its end users through graphical user interfaces. A novice user of a service's user interface takes more think time—the average time to comprehend the content and its layout on the interface—in comparison to expert users. The think time gradually decreases as she repeatedly comprehends the same interface over time. This decrease in think time is the user learning phenomenon. Owing to this learning behavior, the proportion of users—at various learning levels for different services—changes dynamically leading to a difference in the workload. Traditionally though, workload specifications (required for system performance evaluation) never accounted for user learning behavior. They generally assumed a global mean think time, instead. In this work, we first report an experimental study that demonstrates the impact of user learning of the graphical user interfaces of session-based systems. We measure the performance data of a real system while it is getting used by synthetic users (who are) at various learning levels. Statistical tests on our measured data reveal that different human learning curves, combined with different amounts of users arriving the system with various learning levels for different services, produce different system performance. Next, we explain a queueing network (QN) performance model called CogQN that accounts for user learning. It is a multi-class QN model where each service and its learning level constitute a class of users for the service. We present the concept of learning-level dependent, class-switching probabilities to model the probabilistic transition of learning levels of users for different services. This novel concept enables us to avoid the exponential explosion of classes that would otherwise be suffered by traditional class-switching theory, had it been used in this work. We solve the model using discrete-event simulation and validate it against the empirical data. Our CogQN model captures the effect of user learning on overall mean response times across different learning conditions within 10% absolute error in comparison to empirical data.

FESTIval: A versatile framework for conducting experimental evaluations of spatial indices

The use of a spatial index is a common strategy to improve the performance of spatial queries in spatial database systems and Geographic Information Systems. Choosing the right spatial index to be employed in a given context requires a quantitative method to analyze the performance of spatial indices. This is done through extensive experimental evaluations. However, conducting these evaluations is an expensive, error-prone, and challenging task because (i) spatial objects are complex data to manage, (ii) spatial indices can apply different parameter values and thus assume distinct configurations, and (iii) there are indices specifically developed for different storage systems, such as disks and flash memories. In this article, we propose FESTIval, a versatile framework for conducting experimental evaluations of spatial indices. FESTIval has the following main advantages:•the support for different types of disk-based and flash-aware spatial indices;•the specification and execution of user-defined workloads;•the use of a data schema that stores index configurations and statistical data of executed workloads.Because of its characteristics, FESTIval allows users to reproduce executed experiments. Further, FESTIval provides an extensible environment, where any spatial dataset can be handled by spatial indices. FESTIval has been used to validate new proposals of flash-aware spatial indices, such as eFIND-based indices.

WESSBAS: extraction of probabilistic workload specifications for load testing and performance prediction\u2014a model-driven approach for session-based application systems

The specification of workloads is required in order to evaluate performance characteristics of application systems using load testing and model-based performance prediction. Defining workload specifications that represent the real workload as accurately as possible is one of the biggest challenges in both areas. To overcome this challenge, this paper presents an approach that aims to automate the extraction and transformation of workload specifications for load testing and model-based performance prediction of session-based application systems. The approach (WESSBAS) comprises three main components. First, a system- and tool-agnostic domain-specific language (DSL) allows the layered modeling of workload specifications of session-based systems. Second, instances of this DSL are automatically extracted from recorded session logs of production systems. Third, these instances are transformed into executable workload specifications of load generation tools and model-based performance evaluation tools. We present transformations to the common load testing tool Apache JMeter and to the Palladio Component Model. Our approach is evaluated using the industry-standard benchmark SPECjEnterprise2010 and the World Cup 1998 access logs. Workload-specific characteristics (e.g., session lengths and arrival rates) and performance characteristics (e.g., response times and CPU utilizations) show that the extracted workloads match the measured workloads with high accuracy.

Robot ensembles for grafting herbaceous crops

Legislation and regulation is increasingly limiting the use of pesticides or chemical fumigants to counteract soil-borne pathogens. Therefore the use of disease resistant grafted plants is increasing. However, the grafting of herbaceous crops is a labour-intensive technique, with consequent costs. These aspects have encouraged the development of automated machines able to increase productivity and rooting success rate while reducing costs. This paper presents an innovative solution for automatic grafting of vegetable crops, suitable for small to medium sized farms. The concept is to use a group of cooperative robots, adaptable in number, to meet workload specifications. The machine consists of one or more grafting units, able to cut and join scions and rootstocks, and a supplying and sorting system based on an ensemble of single-axis, identical and independent robotic modules, displacing on a unique rail, which coordinates the movement and the selection of scions, rootstocks and grafted plants. Overall productivity is given by the number of grafting units, the number of robotic modules implemented in the supply system, and the efficiency of the control and task allocation strategy. Together with the description of the innovative aspects of the machine mechanics, designed to face intrinsic variability in vegetable objects, the objective of this paper was to present the ensemble synthesis of the control policies, based on heuristic scheduling priorities, to allocate and coordinate the team of robots to a set of spatially distributed tasks.

Demonstration of Hadoop-GIS: A Spatial Data Warehousing System Over MapReduce.

The proliferation of GPS-enabled devices, and the rapid improvement of scientific instruments have resulted in massive amounts of spatial data in the last decade. Support of high performance spatial queries on large volumes data has become increasingly important in numerous fields, which requires a scalable and efficient spatial data warehousing solution as existing approaches exhibit scalability limitations and efficiency bottlenecks for large scale spatial applications. In this demonstration, we present Hadoop-GIS - a scalable and high performance spatial query system over MapReduce. Hadoop-GIS provides an efficient spatial query engine to process spatial queries, data and space based partitioning, and query pipelines that parallelize queries implicitly on MapReduce. Hadoop-GIS also provides an expressive, SQL-like spatial query language for workload specification. We will demonstrate how spatial queries are expressed in spatially extended SQL queries, and submitted through a command line/web interface for execution. Parallel to our system demonstration, we explain the system architecture and details on how queries are translated to MapReduce operators, optimized, and executed on Hadoop. In addition, we will showcase how the system can be used to support two representative real world use cases: large scale pathology analytical imaging, and geo-spatial data warehousing.

A generic construct based workload model for business intelligence benchmark

Benchmarks are vital tools in the performance measurement and evaluation of computer hardware and software systems. Standard benchmarks such as the TREC, TPC, SPEC, SAP, Oracle, Microsoft, IBM, Wisconsin, AS 3AP, OO1, OO7, XOO7 benchmarks have been used to assess the system performance. These benchmarks are domain-specific in that they model typical applications and tie to a problem domain. Test results from these benchmarks are estimates of possible system performance for certain pre-determined problem types. When the user domain differs from the standard problem domain or when the application workload is divergent from the standard workload, they do not provide an accurate way to measure the system performance of the user problem domain. System performance of the actual problem domain in terms of data and transactions may vary significantly from the standard benchmarks. In this research, we address the issue of domain boundness and workload boundness which results in the ir-representative and ir-reproducible performance readings. We tackle the issue by proposing a domain-independent and workload-independent benchmark method which is developed from the perspective of the user requirements. We present a user-driven workload model to develop a benchmark in a process of workload requirements representation, transformation, and generation. We aim to create a more generalized and precise evaluation method which derives test suites from the actual user domain and application. The benchmark method comprises three main components. They are a high-level workload specification scheme, a translator of the scheme, and a set of generators to generate the test database and the test suite. The specification scheme is used to formalize the workload requirements. The translator is used to transform the specification. The generator is used to produce the test database and the test workload. In web search, the generic constructs are main common carriers we adopt to capture and compose the workload requirements. We determine the requirements via the analysis of literature study. In this study, we have conducted ten baseline experiments to validate the feasibility and validity of the benchmark method. An experimental prototype is built to execute these experiments. Experimental results demonstrate that the method is capable of modeling the standard benchmarks as well as more general benchmark requirements.

Synthetic Workload Generation for Cloud Computing Applications

We present techniques for characterization, modeling and generation of workloads for cloud computing applications. Methods for capturing the workloads of cloud computing applications in two different models - benchmark application and workload models are described. We give the design and implementation of a synthetic workload generator that accepts the benchmark and workload model specifications generated by the characterization and modeling of workloads of cloud computing applications. We propose the Georgia Tech Cloud Workload Specification Language (GT-CWSL) that provides a structured way for specification of application workloads. The GT-CWSL combines the specifications of benchmark and workload models to create workload specifications that are used by a synthetic workload generator to generate synthetic workloads for performance evaluation of cloud computing applications.

Benchmarking intelligent information integration – A generic construct-based model

Benchmarks are vital tools in the performance measurement and evaluation of computer hardware and software systems. Standard benchmarks such as the TREC, TPC, SPEC, SAP, Oracle, Microsoft, IBM, Wisconsin, AS 3AP, OO1, OO7, XOO7 benchmarks have been used to assess the system performance. These benchmarks are domain-specific in that they model typical applications and tie to a problem domain. Test results from these benchmarks are estimates of possible system performance for certain pre-determined problem types. When the user domain differs from the standard problem domain or when the application workload is divergent from the standard workload, they do not provide an accurate way to measure the system performance of the user problem domain. System performance of the actual problem domain in terms of data and transactions may vary significantly from the standard benchmarks. In this research, we address the issue of domain boundness and workload boundness which results in the ir-representative and ir-reproducible performance reading. We tackle the issue by proposing a domain-independent and workload-independent benchmark method which is developed from the perspective of the user requirements. We present a user-driven workload model to develop a benchmark in a process of workload requirements representation, transformation, and generation. We aim to create a more generalized and precise evaluation method which derives test suites from the actual user domain and application. The benchmark method comprises three main components. They are a high-level workload specification scheme, a translator of the scheme, and a set of generators to generate the test database and the test suite. The specification scheme is used to formalize the workload requirements. The translator is used to transform the specification. The generator is used to produce the test database and the test workload. In web search, the generic constructs are main common carriers we adopt to capture and compose the workload requirements. We determine the requirements via the analysis of literature study. In this study, we have conducted ten baseline experiments to validate the feasibility and validity of the benchmark method. An experimental prototype is built to execute these experiments. Experimental results demonstrate that the method is capable of modeling the standard benchmarks as well as more general benchmark requirements.

A generic construct based workload model for web search

Benchmarks are vital tools in the performance measurement, evaluation, and comparison of computer hardware and software systems. Standard benchmarks such as the TREC, TPC, SPEC, SAP, Oracle, Microsoft, IBM, Wisconsin, AS 3AP, OO1, OO7, XOO7 benchmarks have been used to assess the system performance. These benchmarks are domain-specific and domain-dependent in that they model typical applications and tie to a problem domain. Test results from these benchmarks are estimates of possible system performance for certain pre-determined problem types. When the user domain differs from the standard problem domain or when the application workload is divergent from the standard workload, they do not provide an accurate way to measure the system performance of the user problem domain. System performance of the actual problem domain in terms of data and transactions may vary significantly from the standard benchmarks. In this research, we address the issue of generalization and precision of benchmark workload model for web search technology. The current performance measurement and evaluation method suffers from the rough estimate of system performance which varies widely when the problem domain changes. The performance results provided by the vendors cannot be reproduced nor reused in the real users’ environment. Hence, in this research, we tackle the issue of domain boundness and workload boundness which represents the root of the problem of imprecise, ir-representative, and ir-reproducible performance results. We address the issue by presenting a domain-independent and workload-independent workload model benchmark method which is developed from the perspective of the user requirements and generic constructs. We present a user-driven workload model to develop a benchmark in a process of workload requirements representation, transformation, and generation via the common carrier of generic constructs. We aim to create a more generalized and precise evaluation method which derives test suites from the actual user domain and application setting. The workload model benchmark method comprises three main components. They are a high-level workload specification scheme, a translator of the scheme, and a set of generators to generate the test database and the test suite. They are based on the generic constructs. The specification scheme is used to formalize the workload requirements. The translator is used to transform the specification. The generator is used to produce the test database and the test workload. We determine the generic constructs via the analysis of search methods. The generic constructs form a page model, a query model, and a control model in the workload model development. The page model describes the web page structure. The query model defines the logics to query the web. The control model defines the control variables to set up the experiments. In this study, we have conducted ten baseline research experiments to validate the feasibility and validity of the benchmark method. An experimental prototype is built to execute these experiments. Experimental results demonstrate that the method based on generic constructs and driven by the perspective of user requirements is capable of modeling the standard benchmarks as well as more general benchmark requirements.

Performance analysis of LAS-based scheduling disciplines in a packet switched network

The Least Attained Service (LAS) scheduling policy, when used for scheduling packets over the bottleneck link of an Internet path, can greatly reduce the average flow time for short flows while not significantly increasing the average flow time for the long flows that share the same bottleneck. No modification of the packet headers is required to implement the simple LAS policy. However, previous work has also shown that a drawback of the LAS scheduler is that, when link utilization is greater than 70%, long flows experience large jitter in their packet transfer times as compared to the conventional First-Come-First-Serve (FCFS) link scheduling. This paper proposes and evaluates new differentiated LAS scheduling policies that reduce the jitter for long flows that are identified as "priority" flows.To evaluate the new policies, we develop analytic models to estimate average flow transfer time as a function of flow size, and average packet transmission time as a function of position in the flow, for the single-bottleneck "dumbbell topology" used in many ns simulation studies. Models are developed for FCFS scheduling, LAS scheduling, and each of the new differentiated LAS scheduling policies at the bottleneck link. Over a wide range of configu-rations, the analytic estimates agree very closely with the ns estimates. Thus, the analytic models can be used instead of simulation for comparing the policies with respect to mean flow transfer time (as a function of flow size) and mean packet transfer time. Furthermore, an initial discrepancy between the analytic and simulation estimates revealed errors in the parameter values that are often specified in the widely used ns Web workload generator. We develop an improved Web workload specification, which is used to estimate the packet jitter for long flows (more accurately than with previous simulation workloads).Results for the scheduling policies show that a particular policy, LAS-log, greatly improves the mean flow transfer time for priority long flows while providing performance similar to LAS for the ordinary flows. Simulations show that the LAS-log policy also greatly reduces the jitter in packet delivery times for the priority flows.

A performance and scalability analysis framework for parallel discrete event simulators

The development of efficient parallel discrete event simulators is hampered by the large number of interrelated factors affecting performance. This problem is made more difficult by the lack of scalable representative models that can be used to analyze optimizations and isolate bottlenecks. This paper proposes a performance and scalability analysis framework (PSAF) for parallel discrete event simulators. PSAF is built on a platform-independent Workload Specification Language (WSL). WSL is a language that represents simulation models using a set of fundamental performance-critical parameters. For each simulator under study, a WSL translator generates synthetic platform-specific simulation models that conform to the performance and scalability characteristics specified by the WSL description. Moreover, sets of portable simulation models that explore the effects of the different parameters, individually or collectively, on the execution performance can easily be constructed using the Synthetic Workload Generator (SWG). The SWG automatically generates simulation workloads with different performance properties. In addition, PSAF supports the seamless integration of real simulation models into the workload specification. Thus, a benchmark with both real and synthetically generated models can be built allowing for realistic and thorough exploration of the performance space. The utility of PSAF in determining the boundaries of performance and scalability of simulation environments and models is demonstrated.

A synthetic workload for a distributed real-time system

In this paper, we describe the design and implementation of a synthetic workload (SW) for a distributed real-time system. A SW is a set of parameterized synthetic or artificial programs which serve as the workload for a system under study. The parameterized nature of the programs allows the user to change their behavior to create different resource demands on the system. The SW is easy to use, flexible, and can be representative of a real-time workload. The SW consists of a driver and a set of synthetic tasks. The synthetic tasks are generated by a synthetic workload generator (SWG) from the user's specification written in SWSL, a synthetic workload specification language. We describe the design goals of our SW and discuss its software structure and how it meets these goals.

SWSL: a synthetic workload specification language for real-time systems

We discuss the issues that must be addressed in the specification and generation of synthetic workloads for distributed real-time systems. We describe a synthetic workload specification language (SWSL) that defines a workload in a form that can be compiled by a synthetic workload generator (SWG) to produce an executable synthetic workload. The synthetic workload is then downloaded to the target machine and executed while performance and dependability measurements are made. SWSL defines the workload at the task level using a data flow graph, and at the operation level using control constructs and synthetic operations taken from a library. It is intended to be easy to use, flexible, and capable of creating synthetic workloads that are representative of real-time workloads. It provides a compact, parameterized notation. It supports automatic replication of objects to facilitate the specification of large workloads for distributed real-time systems. It also provides extensive support for the experimentation process.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Educational tool for design of parallel processing systems

A software package which can be used as an educational tool for the design of a subset of parallel processing architecture (PPA) systems is presented. A user-friendly interface guides the user through the different steps involved in specifying and emulating PPA systems using program-driven simulation. The user can choose any of the following PPA systems categorised according to Flynn's taxonomy: MISD (multiple instruction single data), SIMD (single instruction multiple data) and MIMD (multiple instruction multiple data). Workload and scheduler specifications can be defined by the user. The evaluation model is based on the queuing theory for the modelling and simulation of PPA systems. The simulation program is based on a linked list structure to represent events. The package includes HELP and ERROR LOG utilities, and provides graphical illustrations of the results and performance indices.

Guaranteed task deadlines for fault-tolerant workloads with conditional branches

This work examines scheduling for a real-time multiprocessor (MAFT) in which both hard deadlines and fault-tolerance are necessary system components. A workload for this system consists of a set of concurrent dependent tasks, each with some execution frequency; tasks are also fully ordered by priority. Fault tolerance mechanisms include hardware-supported voting on computation results as well as on task starts, task completions, and branch conditions. The distributed agreement mechanism used on system-level decisions adds a variable threading delay to the run time of each copy of a task. These delays make current schedule verification techniques inapplicable. In the most general execution profile, each processor in the system runs a subset of the tasks, with different tasks possibly having different frequencies. In this work, however, we restrict attention to a special class of workloads, termed uni-schedule, in which each processor executes the entire task set, using the multiple processors to implement full redundancy. In addition, all tasks are assumed to have the same periodicity. Given these restrictions, we produce stable schedules consistent with the initial workload specifications. Algorithms are first given for uni-schedule workloads with no run-time branches, and then for uni-schedule workloads with branches.

Resource contention in shared-memory multiprocessors: A parameterized performance degradation model

A standard metric conventionally employed to compare the performance of different multiprocessor systems is speedup. Although providing a measure of the improvement in execution speed achievable on a system, this metric does not yield any insight into the factors responsible for limiting the potential improvement in speed. This paper studies the performance degradation in shared-memory multiprocessors as a result of contention for shared-memory resources. A replicate workload framework with a flexible mechanism for workload specification is proposed for measuring performance. Two normalized performance metrics— efficiency and overhead factor—are introduced to quantify the factors limiting performance and facilitate comparison across architectures. Finally, the proposed model is employed to measure and compare the performance of three contemporary shared-memory systems, with special emphasis on the newly released BBN Butterfly-II (TC2000), currently undergoing Beta test.