An Analysis of Holistic Tail Latency Behaviors of Java Microservices

Abstract

Long-tail latency values significantly affect the user experience and hence they are a major concern in modern systems. However, tail latency characteristics of applications developed using microservices architecture are still unknown. In this paper, we focus on analyzing and characterizing the behaviour of tail latency values of microservices workloads under service's peak sustainable throughput with a closed system model. Our particular focus is to investigate the impact of heap size, garbage collection, concurrency and service demand on the tail latency. Our study has two parts. First, we develop and experiment on four microservices micro benchmarks representing CPU bound, network I/O bound, memory bound and database I/O bound to find the tail characteristics under different concurrency, heap, service demand and garbage collector. Then, in the second part, we explore the tail latency characteristics identified on micro benchmarks by testing those conditions using standard microservices benchmark application, Socks Shop. We observe first that microservices latency distributions exhibit the power law behavior. Second, we systematically prove that microservices workload distributions do not exhibit the heavy tailed behavior. We show that increasing the level of concurrency and service demand increases the tail latency. We use queuing simulation to prove our claims. Finally, we show that tail latency values of memory bound and database I/O bound microservices workloads are correlated with heap size and garbage collection.