Cooperative Multi-agent Reinforcement Learning Research Articles

Multiobjective Optimization for Stiffness and Position Control in a Soft Robot Arm Module

The central concept of this letter is to develop an assistive manipulator that can automate the bathing task for elderly citizens. We propose to exploit principles of soft robotic technologies to design and control a compliant system to ensure safe human–robot interaction, a primary requirement for the task. The overall system is intended to be modular with a proximal segment that provides structural integrity to overcome gravitational challenges and a distal segment to perform the main bathing activities. The focus of this letter is on the design and control of the latter module. The design comprises of alternating tendons and pneumatics in a radial arrangement, which enables elongation, contraction, and omnidirectional bending. Additionally, a synergetic coactivation of cables and tendons in a given configuration allows for stiffness modulation, which is necessary to facilitate washing and scrubbing. The novelty of the work is twofold: 1) Three base cases of antagonistic actuation are identified that enable stiffness variation. Each category is then experimentally characterized by the application of an external force that imposes a linear displacement at the tip in both axial and lateral directions. 2) The development of a novel algorithm based on cooperative multiagent reinforcement learning that simultaneously optimizes stiffness and position. The results highlight the effectiveness of the design and control to contribute toward the development of the assistive device.

Reward shaping for knowledge-based multi-objective multi-agent reinforcement learning

AbstractThe majority of multi-agent reinforcement learning (MARL) implementations aim to optimize systems with respect to a single objective, despite the fact that many real-world problems are inherently multi-objective in nature. Research into multi-objective MARL is still in its infancy, and few studies to date have dealt with the issue of credit assignment. Reward shaping has been proposed as a means to address the credit assignment problem in single-objective MARL, however it has been shown to alter the intended goals of a domain if misused, leading to unintended behaviour. Two popular shaping methods are potential-based reward shaping and difference rewards, and both have been repeatedly shown to improve learning speed and the quality of joint policies learned by agents in single-objective MARL domains. This work discusses the theoretical implications of applying these shaping approaches to cooperative multi-objective MARL problems, and evaluates their efficacy using two benchmark domains. Our results constitute the first empirical evidence that agents using these shaping methodologies can sample true Pareto optimal solutions in cooperative multi-objective stochastic games.

MOO-MDP: An Object-Oriented Representation for Cooperative Multiagent Reinforcement Learning.

Reinforcement learning (RL) is a widely known technique to enable autonomous learning. Even though RL methods achieved successes in increasingly large and complex problems, scaling solutions remains a challenge. One way to simplify (and consequently accelerate) learning is to exploit regularities in a domain, which allows generalization and reduction of the learning space. While object-oriented Markov decision processes (OO-MDPs) provide such generalization opportunities, we argue that the learning process may be further simplified by dividing the workload of tasks amongst multiple agents, solving problems as multiagent systems (MAS). In this paper, we propose a novel combination of OO-MDP and MAS, called multiagent OO-MDP (MOO-MDP). Our proposal accrues the benefits of both OO-MDP and MAS, better addressing scalability issues. We formalize the general model MOO-MDP and present an algorithm to solve deterministic cooperative MOO-MDPs. We show that our algorithm learns optimal policies while reducing the learning space by exploiting state abstractions. We experimentally compare our results with earlier approaches in three domains and evaluate the advantages of our approach in sample efficiency and memory requirements.

Multiagent Cooperative Reinforcement Learning by Expert Agents (MCRLEA)

The paper gives novel approach Multiagent Cooperative Reinforcement Learning by Expert Agents (MCRLEA) for dynamic decision making in the retail application. Furthermore, it put up different cooperation schemes for multiagent cooperative reinforcement learning i.e. EQ learning, EGroup, EDynamic, EGoal driven and Expert agents scheme. Implementation outcome includes a demonstration of recommended cooperation schemes that are competent enough to speedup the collection of agents that achieve excellent action policies. Accordingly this approach presents three retailer stores in the retail market place. Retailers can help to each other and can obtain profit from cooperation knowledge through learning their own strategies that just stand for their aims and benefit. The vendors are the knowledgeable agents in the hypothesis to employ cooperative learning to train in the circumstances. Assuming significant hypothesis on the vendor’s stock policy, restock period, arrival process of the consumers, the approach is formed as Markov decision process model that makes it possible to design learning algorithms. The proposed algorithms noticeably learn dynamic consumer performance. Moreover, the paper illustrates results of Cooperative Reinforcement Learning Algorithms of three shop agents for the period of one year sale duration and then demonstrated the results using proposed approach for three shop agents for the period of one year sale duration. The results obtained by the proposed expert agent based cooperation approach show that such methods can put into a quick convergence of agents in the dynamic environment.

Erratum to: A multi-agent cooperative reinforcement learning model using a hierarchy of consultants, tutors and workers

Erratum to: A multi-agent cooperative reinforcement learning model using a hierarchy of consultants, tutors and workers

A multi-agent cooperative reinforcement learning model using a hierarchy of consultants, tutors and workers

The hierarchical organisation of distributed systems can provide an efficient decomposition for machine learning. This paper proposes an algorithm for cooperative policy construction for independent learners, named Q-learning with aggregation (QA-learning). The algorithm is based on a distributed hierarchical learning model and utilises three specialisations of agents: workers, tutors and consultants. The consultant agent incorporates the entire system in its problem space, which it decomposes into sub-problems that are assigned to the tutor and worker agents. The QA-learning algorithm aggregates the Q-tables of worker agents into a central repository managed by their tutor agent. Each tutor’s Q-table is then incorporated into the consultant’s Q-table, resulting in a Q-table for the entire problem. The algorithm was tested using a distributed hunter prey problem, and experimental results show that QA-learning converges to a solution faster than single agent Q-learning and some famous cooperative Q-learning algorithms.

Clique-based cooperative multiagent reinforcement learning using factor graphs

In this paper, we propose a clique-based sparse reinforcement learning (RL) algorithm for solving cooperative tasks. The aim is to accelerate the learning speed of the original sparse RL algorithm and to make it applicable for tasks decomposed in a more general manner. First, a transition function is estimated and used to update the Q-value function, which greatly reduces the learning time. Second, it is more reasonable to divide agents into cliques, each of which is only responsible for a specific subtask. In this way, the global Q-value function is decomposed into the sum of several simpler local Q-value functions. Such decomposition is expressed by a factor graph and exploited by the general maxplus algorithm to obtain the greedy joint action. Experimental results show that the proposed approach outperforms others with better performance.

Reinforcement Learning for Multiple Access Control in Wireless Sensor Networks: Review, Model, and Open Issues

Wireless sensor networking is a viable communication technology among low-cost and energy-limited sensor nodes deployed in an environment. Due to high operational features, the application area of this technology is extended significantly but with some energy related challenges. One main cause of the nodes energy wasting in these networks is idle listening characterized with no communication activity. This drawback can be mitigated by the means of energy-efficient multiple access control schemes so as to minimize idle listening. In this paper, we discuss the applicability of distributed learning algorithms namely reinforcement learning towards multiple access control (MAC) in wireless sensor networks. We perform a comparative review of relevant work in the literature and then present a cooperative multi agent reinforcement learning framework for MAC design in wireless sensor networks. Accordingly, the paper concludes with some major challenges and open issues of distributed MAC design using reinforcement learning.

Cooperative Multi-Agent Reinforcement Learning for Multi-Component Robotic Systems: guidelines for future research

AbstractReinforcement Learning (RL) as a paradigm aims to develop algorithms that allow to train an agent to optimally achieve a goal with minimal feedback information about the desired behavior, which is not precisely specified. Scalar rewards are returned to the agent as response to its actions endorsing or opposing them. RL algorithms have been successfully applied to robot control design. The extension of the RL paradigm to cope with the design of control systems for Multi-Component Robotic Systems (MCRS) poses new challenges, mainly related to coping with scaling up of complexity due to the exponential state space growth, coordination issues, and the propagation of rewards among agents. In this paper, we identify the main issues which offer opportunities to develop innovative solutions towards fully-scalable cooperative multi-agent systems.

Contraction Mappings in the Theory Underlying Dynamic Programming

Contraction Mappings in the Theory Underlying Dynamic Programming