Analysis about Efficiency of Indirect Media Communication on Multi-agent Cooperation Learning

Abstract

Reinforcement learning (RL) is an efficient learning method for Markov decision processes (MDPs); ant colony system (ACS) is an efficient method for solving combinatorial optimization problems. Based on the update policy of reinforcement values in RL and the cooperating method of the indirect media communication in ACS, this paper proposes the Q-ACS multi-agent cooperating learning method for the learning agents to share episodes beneficial to the exploitation of the accumulated knowledge and to utilize the learned reinforcement values efficiently. Further, taking the visited times into account, this paper proposes the T-ACS multi-agent learning method that the learning agents share better policies beneficial to the exploration during agent's learning processes. Meanwhile, in the light of the indirect media communication among heterogeneous multi-agents, this paper presents a heterogeneous multi-agent RL method, the D-ACS. The agents in our methods are given a simply cooperating way exchanging information in the form of reinforcement values updated in the common model of all agents. Owning the advantages of exploring the unknown environment actively and exploiting learned knowledge effectively, the proposed methods are able to solve both MDPs and combinatorial optimization problems effectively. To results of simulations on the hunter game and the traveling salesman problem, this paper discusses the role of the indirect media communication on the multi-agent cooperation learning system and analyzes its efficiency. The results of experiments also demonstrate that our methods perform competitively with representative methods on each domain respectively.