Pricing Cloud Resource Based on Multi-Agent Reinforcement Learning in the Competing Environment

Abstract

Multiple cloud providers compete against each other in order to attract cloud users and make profits in the cloud market. In doing so, each provider needs to charge fees to users in a proper way. In this paper, we analyze how a cloud provider sets price effectively when competing against other cloud providers. The price set by the cloud provider is affected by its opponent's price, and as well as the prices set in the last round. Specifically, we model this problem as a Markov game by considering two cloud providers competing against each other. We then adopt two different solution concepts in game theory, minimax and Nash equilibrium, to solve this problem. Specifically, we use two different multi-agent reinforcement learning algorithms, minimax-Q and Nash-Q, which correspond to those two solution concepts respectively, to design the pricing policies. Furthermore, we improve the Nash-Q learning algorithm by taking into account the probability of each Nash equilibrium happening. Based on this, we run extensive experiments to analyze the effectiveness of minimax-Q and Nash-Q based pricing policies in terms of making long-term profits. We find that the pricing policy based on Nash-Q learning algorithm with selecting Nash equilibrium according to the probability can beat other Nash-Q based pricing polices with selecting Nash equilibrium according to the maximal payoff. However, in the further experimental analysis, we find that minimax-Q based pricing policies can beat all Nash-Q based pricing policies. This is because the minimax solution concept is more suitable in this competing environment. Our experimental results provide useful insights on designing practical pricing policies for competing cloud providers.