Benchmarking Private Cloud Performance with User-Centric Metrics

Abstract

Cloud computing is a new paradigm for the delivery of IT services. It has enabled many promising opportunities for features that cannot be easily implemented in traditional IT environments, such as elastic scalability, self-service deployment, resiliency and recovery, and so forth. Benchmarking the cloud requires a well-defined set of cloud performance metrics that should be able to sensitively distinguish the capabilities of cloud systems that enable those features. One way of defining benchmark metrics is based on observations of the internal mechanisms in a cloud. For example, an elasticity evaluation may be based on measuring a resource provisioning interval in the cloud. However, a more meaningful evaluation should be based on user-centric metrics. In this article, we will introduce a set of performance metrics that can be directly measured, calculated and compared by the cloud users, including workload consumers and the users who deploy and manage the workload life cycles. We will also discuss ways to organize the user-centric metrics, with different emphasis, into a benchmark that represents different use cases.