Performance Prediction Of Applications Research Articles

Adaptive Feature Selection for Predicting Application Performance Degradation in Edge Cloud Environments

Adaptive Feature Selection for Predicting Application Performance Degradation in Edge Cloud Environments

Performance prediction of parallel applications using artificial neuronal network and graph representation

SummaryModeling and predicting the performance of HPC applications is exceedingly important for several purposes. These incorporate scheduling and managing tasks, understanding application behavior, and estimating resource requirements. Many issues impact the performance of parallel applications, including data decomposition, load balancing, and inter processors communication. Predicting the performance of parallelism accurately is a challenging task, especially since it depends on several hardware and software parameters that are difficult to identify. The graph representation is an efficient way to model a parallel application. Based on this representation, this study purports to predict the optimal number of partitions (processors) to run efficiently a parallel application on multi‐core architectures using an artificial neural network. The numerical simulation applications are modeled as a graph of tasks and their associated data. Using the characteristics of the graphs as inputs of the ANN model, we obtained satisfactory results with an accuracy of 91% for the optimal number of processors and 93% for the maximum speedup.

A Novel Reinforcement Learning Approach for Spark Configuration Parameter Optimization.

Apache Spark is a popular open-source distributed data processing framework that can efficiently process massive amounts of data. It provides more than 180 configuration parameters for users to manually select the appropriate parameter values according to their own experience. However, due to the large number of parameters and the inherent correlation between them, manual tuning is very tedious. To solve the problem of tuning through personal experience, we designed and implemented a reinforcement-learning-based Spark configuration parameter optimizer. First, we trained a Spark application performance prediction model with deep neural networks, and verified the accuracy and effectiveness of the model from multiple perspectives. Second, in order to improve the search efficiency of better configuration parameters, we improved the Q-learning algorithm, and automatically set start and end states in each iteration of training, which effectively improves the agent’s poor performance in exploring better configuration parameters. Lastly, comparing our proposed configuration with the default configuration as the baseline, experimental results show that the optimized configuration gained an average performance improvement of 47%, 43%, 31%, and 45% for four different types of Spark applications, which indicates that our Spark configuration parameter optimizer could efficiently find the better configuration parameters and improve the performance of various Spark applications.

Framework for Vertical Performance Assessment in Very Large-Scale Cellular Networks

As 4G networks evolve toward full 5G deployment and beyond 5G (B5G) research begins, new vertical opportunities are being introduced in the context of intelligent transportation, power systems, smart-city urban operation, security services, etc. The diversity of the vertical requirements as well as their particular implementation in citywide locations will make it a real challenge to be able to assess or predict application performance for some of those verticals. In this article, we were motivated by the fact that because of the very nature of verticals and their interaction with a very large-scale telecommunications infrastructure, neither measurements nor small-scale simulations will be sufficient for application performance monitoring and prediction. To enable the assessment and prediction of verticals, so-called Quality of Experience (QoE), we propose a comprehensive framework for very large-scale simulation. The framework is based on the random generation of realistic key performance indicators (KPIs), which vary according to changes produced by the dynamics of urban systems. The probability distributions used for context-dependent random generation are built from samples taken from either “traditional” small-scale simulations or real local measurements. Finally, we present the framework’s key features and current roadblocks for an end-to-end widespread implementation.

Modeling Speedup in Multi-OS Environments

For workloads that place strenuous demands on system software, novel operating system designs like unikernels, library OSes, and hybrid runtimes offer a promising path forward. However, while these systems can outperform general-purpose OSes, they have limited ability to support legacy applications. Multi-OS environments, where the application’s execution is split between a control plane and a data plane operating system, can address this challenge, but reasoning about the performance of applications that run in such a split execution environment is currently guided only by expert intuition and empirical analysis. As the level of specialization in system software and hardware continues to increase, there is both a pressing need and ripe opportunity for investigating analytical models that can predict application performance and guide programmers’ intuition when considering multi-OS environments. In this paper we present such a model to place bounds on application speedup, beginning with a simple, intuitive formulation, and progressing to a more refined model. We present an analysis of the model for a diverse set of benchmarks, as well as a prototype tool to project multi-OS speedups for applications on existing systems. Finally, we validate our model on state-of-the-art multi-OS systems, demonstrating that it reliably predicts speedup with 96% average accuracy.

A Hybrid Machine Learning Approach for Performance Modeling of Cloud-Based Big Data Applications

Abstract Nowadays, Apache Hadoop and Apache Spark are two of the most prominent distributed solutions for processing big data applications on the market. Since in many cases these frameworks are adopted to support business critical activities, it is often important to predict with fair confidence the execution time of submitted applications, for instance when service-level agreements are established with end-users. In this work, we propose and validate a hybrid approach for the performance prediction of big data applications running on clouds, which exploits both analytical modeling and machine learning (ML) techniques and it is able to achieve a good accuracy without too many time consuming and costly experiments on a real setup. The experimental results show how the proposed approach attains improvement in accuracy, number of experiments to be run on the operational system and cost over applying ML techniques without any support from analytical models. Moreover, we compare our approach with Ernest, an ML-based technique proposed in the literature by the Spark inventors. Experiments show that Ernest can accurately estimate the performance in interpolating scenarios while it fails to predict the performance when configurations with increasing number of cores are considered. Finally, a comparison with a similar hybrid approach proposed in the literature demonstrates how our approach significantly reduce prediction errors especially when few experiments on the real system are performed.

A Measurement-Based Message-Level Timing Prediction Approach for Data-Dependent SDFGs on Tile-Based Heterogeneous MPSoCs

Fast yet accurate performance and timing prediction of complex parallel data flow applications on multi-processor systems remains a very difficult discipline. The reason for it comes from the complexity of the data flow applications w.r.t. data dependent execution paths and the hardware platform with shared resources, like buses and memories. This combination may lead to complex timing interferences that are difficult to express in pure analytical or classical simulation-based approaches. In this work, we propose the combination of timing measurement and statistical simulation models for probabilistic timing and performance prediction of Synchronous Data Flow (SDF) applications on MPSoCs with shared memories. We exploit the separation of computation and communication in our SDF model of computation to set-up simulation-based performance prediction models following different abstraction approaches. We especially propose a message-level communication model driven by a data-dependent probabilistic execution phase timing model. We compare our work against measurement on two case-studies from the computer vision domain: a Sobel filter and a JPEG decoder. We show that the accuracy and execution time of our modeling and evaluation framework outperforms existing approaches and is suitable for a fast yet accurate design space exploration.

Online Performance Modeling and Prediction for Single-VM Applications in Multi-Tenant Clouds

Clouds have been adopted widely by many organizations for their supports of flexible resource demands and low cost, which is normally achieved through sharing the underlying hardware among multiple cloud tenants. However, such sharing with the changes in resource contentions in virtual machines (VMs) can result in large variations for the performance of cloud applications, which makes it difficult for ordinary cloud users to estimate the run-time performance of their applications. In this article, we propose online learning methodologies for performance modeling and prediction of applications that run repetitively on multi-tenant clouds (such as on-line data analytic tasks). Here, a few micro-benchmarks are utilized to probe the in-situ perceivable performance of CPU, memory and I/O components of the target VM. Then, based on such profiling information and in-place measured application’s performance, the predictive models can be derived with either Regression or Neural-Network techniques. In particular, to address the changes in the intensity of resource contentions of a VM over time and its effects on the target application, we proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">periodic model retraining</i> where the sliding-window technique was exploited to control the frequency and historical data used for model retraining. Moreover, a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">progressive modeling</i> approach has been devised where the Regression and Neural-Network models are gradually updated for better adaptation to recent changes in resource contention. With 17 representative applications from PARSEC, NAS Parallel and CloudSuite benchmarks being considered, we have extensively evaluated the proposed online schemes for the prediction accuracy of the resulting models and associated overheads on both a private and public clouds. The evaluation results show that, even on the private cloud with high and radically changed resource contention, the average prediction errors of the considered models can be less than 20 percent with periodic retraining. The prediction errors generally decrease with higher retraining frequencies and more historical data points but incurring higher run-time overheads. Furthermore, with the neural-network progressive models, the average prediction errors can be reduced by about 7 percent with much reduced run-time overheads (up to 265 <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> ) on the private cloud. For public clouds with less resource contentions, the average prediction errors can be less than 4 percent for the considered models with our proposed online schemes.

A Case for Pricing Bandwidth: Sharing Datacenter Networks With Cost Dominant Fairness

Unlike other resources such as CPU or memory in a virtual machine, inter-virtual-machine (inter-VM) bandwidth has not been explicitly priced in datacenter networks. In this article, we argue that tenants of an IaaS cloud computing platform should be given the flexibility to pay more for explicitly priced datacenter bandwidth beyond traditional virtual machines, in order to achieve better (or more predictable) application performance. We show that a much simpler design principle can be followed to allocate bandwidth fairly, and desirable properties related to fairness can be more easily achieved, compared with state-of-the-art proposals. We call such a design principle cost dominant fairness, which stipulates that bandwidth should be allocated based on the total cost that a tenant incurs for running its applications in the cloud. Guided by the principle of cost dominant fairness, we explore the design space of pricing inter-VM bandwidth, as well as achieving fair bandwidth sharing among multiple tenants. Through our study, we believe that it is best to assign per-VM-pair weights based on individualized prices. We present a distributed bandwidth allocation algorithm that is theoretically supported by a network utility maximization formulation, and practically implemented as a shim layer at each virtual machine. We are also concerned with practical issues of billing, where discounts are needed to ensure that a tenant only pays for the bandwidth share that it is allocated. Finally, we have evaluated our pricing framework and per-VM-pair weighted fair bandwidth allocation in the Mininet emulation testbed and simulations.

Performance prediction of parallel applications: a systematic literature review

Different techniques for estimating the execution time of parallel applications have been studied for the last 25 years. These approaches have proposed different methods for predicting the performance behaviour of applications. Most of these methods rely on analysing one or more of the following aspects: system workload, application structure, platform system, and the computing resources that the application needs to perform its operations. These elements are used and applied by different methods such as analytic and non-analytic methods. However, no wide-ranging survey of these approaches exists at the time of writing. This paper presents a systematic review of performance prediction methods for parallel applications, which were published in the open literature during the period 2005–2020. We define a classification framework to categorise the reviewed approaches. In addition, we identify some directions and trends in performance prediction as well as some unsolved issues.

Scalable Application- and User-aware Resource Allocation in Enterprise Networks Using End-Host Pacing

Providing scalable user- and application-aware resource allocation for heterogeneous applications sharing an enterprise network is still an unresolved problem. The main challenges are as follows: (i) How do we define user- and application-aware shares of resources? (ii) How do we determine an allocation of shares of network resources to applications? (iii) How do we allocate the shares per application in heterogeneous networks at scale? In this article, we propose solutions to the three challenges and introduce a system design for enterprise deployment. Defining the necessary resource shares per application is hard, as the intended use case, the user’s environment, e.g., big or small display, and the user’s preferences influence the resource demand. We tackle the challenge by associating application flows with utility functions from subjective user experience models, selected Key Performance Indicators, and measurements. The specific utility functions then enable a mapping of network resources in terms of throughput and latency budget to a common user-level utility scale. A sensible distribution of the resources is determined by formulating a multi-objective mixed integer linear program to solve the throughput- and delay-aware embedding of each utility function in the network for a max-min fairness criteria. The allocation of resources in traditional networks with policing and scheduling cannot distinguish large numbers of classes and interacts badly with congestion control algorithms. We propose a resource allocation system design for enterprise networks based on Software-Defined Networking principles to achieve delay-constrained routing in the network and application pacing at the end-hosts. The system design is evaluated against best effort networks in a proof-of-concept set-up for scenarios with increasing number of parallel applications competing for the throughput of a constrained link. The competing applications belong to the five application classes web browsing, file download, remote terminal work, video streaming, and Voice-over-IP. The results show that the proposed methodology improves the minimum and total utility, minimizes packet loss and queuing delay at bottlenecks, establishes fairness in terms of utility between applications, and achieves predictable application performance at high link utilization.

Predicting the performance of big data applications on the cloud

Data science applications have become widespread as a means to extract knowledge from large datasets. Such applications are often characterized by highly heterogeneous and irregular data access patterns, thus often being referred to as big data applications. Such characteristics make the application execution quite challenging for existing software and hardware infrastructures to meet their resource demands. The cloud computing paradigm, in turn, offers a natural hosting solution to such applications since its on-demand pricing model allows allocating effectively computing resources according to application’s needs. However, these properties impose extra challenge to the accurate performance prediction of cloud-based applications, which is a key step to adequate capacity planning and managing of the hosting infrastructure. In this article, we tackle this challenge by exploring three modeling approaches for predicting the performance of big data applications running on the cloud. We evaluate two queuing-based analytical models and dagSim, a fast ad-hoc simulator, in various scenarios based on different applications and infrastructure setups. The considered approaches are compared in terms of prediction accuracy and execution time. Our results indicate that our two best approaches, one analytical model and dagSim, can predict average application execution times with only up to a $$7\%$$ relative error, on average. Moreover, a comparison with the widely used event-based simulator available with the Java Modeling Tool (JMT) suite demonstrates that both the analytical model and dagSim run very fast, requiring at least two orders of magnitude lower execution time than JMT while providing slightly better accuracy, being thus practical for online prediction.

The high-speed networks of the Summit and Sierra supercomputers

Oak Ridge National Laboratory's Summit supercomputer and Lawrence Livermore National Laboratory's Sierra supercomputer utilize InfiniBand interconnect in a Fat-tree network topology, interconnecting all compute nodes, storage nodes, administration, and management nodes into one linearly scalable network. These networks are based on Mellanox 100-Gb/s EDR InfiniBand ConnectX-5 adapters and Switch-IB2 switches, with compute-rack packaging and cooling contributions from IBM. These devices support in-network computing acceleration engines such as Mellanox Scalable Hierarchical Aggregation and Reduction Protocol, graphics processor unit (GPU) Direct RDMA, advanced adaptive routing, Quality of Service, and other network and application acceleration. The overall IBM Spectrum Message Passing Interface (MPI) messaging software stack implements Open MPI, and was a collaboration between IBM, Mellanox, and NVIDIA to optimize direct communication between endpoints, whether compute nodes (with IBM POWER CPUs, NVIDIA GPUs, and flash memory devices), or POWER-hosted storage nodes. The Fat-tree network can isolate traffic among the compute partitions and to/from the storage subsystem, providing more predictable application performance. In addition, the high level of redundancy of this network and its reconfiguration capability ensures reliable high performance even after network component failures. This article details the hardware and software architecture and performance of the networks and describes a number of the high-performance computing (HPC) enhancements engineered into this generation of InfiniBand.

Application Performance Prediction in the Datacenter to Accelerate Cloud Servers

Nowadays, cloud services become more and more important while datacenters are growing in scale. But the low effectiveness on cloud servers becomes an increasing issue as time goes. In this research, we focus on predicting application performance on cloud servers for acceleration. Linear regression and random forest are used in this study. Random foreset proves to be a precise, stable and effective tool to do performance counter prediction.

Modeling The Framework for Evaluating the Non-Functional Requirements Affecting Application Efficacy in Cloud Computing Domain

Predicting application performance concerning the parameters such as processor, network, memory, and disk needs to be analyzed and the model is to be built up for various non-functional requirements in cloud computing. The NFR include availability, portability, security, reliability, etc. Cloud computing provides a way for provisioning of resources in one of the three ways. The first could be cloud infrastructure, the second could be provider infrastructure or the third could be a combination of both. Specifying NFR and ascertaining the required resources requires a lot of knowledge and therefore the various models are built up in this context. To exploit the cloud computing domain software engineer's needs to quantify various requirements of the application and based on that framework needs to be designed and developed for controlling cloud services at the multiple levels of abstraction. In this paper, we will analyze models evaluating various NFR affecting the efficacy of application in a cloud computing domain. The model could be both working for static as well as dynamic implementation.

M3 - A hybrid measurement-modeling approach for CPU-bound applications on cross-platform architectures

Predicting performance of CPU intensive applications for a target platform is of significant importance to IT industries. However, the target hardware platform for which we want to predict the performance is often different from the testbed on which the application performance measurements are done, and may not be unavailable for deployment for various practical reasons. This paper presents M3, a Measure-Measure-Model method, which uses a pipeline of three steps to address this problem. The methodology starts with measuring CPU service demands of the application on the testbed. Then, it builds clones that mimic the application code in terms of the type of operations, the number and size of network calls with external servers and API calls made at different layers of the technology stack. The clones are simple and easy to deploy, yet demonstrate the same speedup factor between the source and the target as the original application. The clones are then deployed on the testbed and on the target to estimate the application CPU service demand under light load generation. In the final step, this estimated CPU service demand is fed into specific performance modeling tools that are capable of predicting application performance on the target under arbitrary higher workload. Predictions made using this approach are validated against direct measurements made on five applications in Java and PHP, and on a number of combinations of testbed and target platforms (Intel and AMD servers) and the prediction error is always less than 20%.

Fast classification of MPI applications using Lamport’s logical clocks

We present a novel trace-based analysis tool that rapidly classifies an MPI application as bandwidth-bound, latency-bound, load-imbalance-bound, or computation-bound for different interconnection networks. The tool uses an extension of Lamport’s logical clock to track application progress in the trace replay. It has two unique features. First, it can predict application performance for many latency and bandwidth parameters from a single replay of the trace. Second, it infers the performance characteristics of an application and classifies the application using the predicted performance trend for a range of network configurations instead of using the predicted performance for a particular network configuration. We describe the techniques used in the tool and its design and implementation, and report our performance study of the tool and our experience with classifying twelve applications and mini-apps from the DOE DesignForward project as well as the NAS Parallel Benchmarks.

An Approach to Automatic Performance Prediction for Cloud-Enhanced Mobile Applications with Sparse Data

In mobile cloud computing (MCC), offloading compute-intensive parts of a mobile application onto the cloud is an attractive method to enhance application performance. To make good offloading decisions, history-based machinelearning techniques are proposed to predict application performance under various offloading schemes. However, the data sparsity problem is common in a realistic MCC scenario but is rarely the concern of existing work. In this paper, we employ a two-phase hybrid framework to predict performance for cloud-enhanced mobile applications, which is designed to be robust to the data sparsity. By training several multi-layer neural networks with historical execution records, the first phase automatically predicts some intermediate parameters for each execution of an application. The models learned by these neural networks can be shared among different applications, thus alleviating the data sparsity. Based on these predicted intermediate parameters and the application topology, the second phase deterministically calculates the estimated values of the performance metrics. The deterministic algorithm can partially guarantee the prediction accuracy of newly published applications even with no execution records. We evaluate our approach with a cloud-enhanced object recognition application and show that our approach can precisely predict the application performance and is robust to data sparsity.

New Bandwidth Sharing and Pricing Policies to Achieve a Win-Win Situation for Cloud Provider and Tenants

For predictable application performance or fairness in network sharing in clouds, many bandwidth allocation policies have been proposed. However, with these policies, tenants are not incentivized to use idle bandwidth or prevent link congestion, and may even take advantage of the policies to gain unfair bandwidth allocation. Increasing network utilization while avoiding congestion not only benefits cloud provider but also the tenants by improving application performance. In this paper, we propose a new pricing model that sets different unit prices for reserved bandwidth, the bandwidth on congested links and on uncongested links, and makes the unit price for congested links proportional to their congestion degrees. We use game theory model to analyze tenants’ behaviors in our model and the current pricing models, which shows the effectiveness of our model in providing the incentives. With the pricing model, we propose a network sharing policy to achieve both min-guarantee and proportionality, while prevent tenants from earning unfair bandwidth. We further propose methods for each virtual machine to arrange its traffic to reduce its unsatisfied demand and maximize its utility, while increase network utilization. As a result, our solution creates a win-win situation, where tenants strive to increase their benefits in bandwidth sharing, which also concurrently increases the utilities of cloud provider and other tenants. Our simulation and trace-driven experimental results show the effectiveness of our solutions in creating the win-win situation.

Application performance prediction method based on cross-core performance interference on multi-core processor

Due to the contention for shared resource, applications deployed on different cores would suffer from the performance interference. Therefore, how to predict applications performance reasonably has become the hotspot in current studies. A challenges of the existing application performance prediction methods are hard to determine the pressure indicators and assess the pressure of the multi-interference application forcing, this paper proposes an application performance prediction method which is based on the cross-core performance interference on multi-core processors. In particular, we firstly analyze the relationship between the cross-core shared resource usage and the performance degradation of applications. Then, we- select the appropriate indicators to represent the pressure on shared resources using the correlation analysis, and establish the correlation model between the degree of the performance degradation and the pressure level by using the stepwise regression analysis method. Meanwhile, we consider the mutual of the interference and employ a K-means clustering algorithm to decrease the prediction cost. Experimental results show that the pressure indicators could measure the competition to the resource and the performance model could predict the performance degradation accurately.