Multiple Markov Decision Processes Research Articles

Survey of imitation learning: tradition and new advances

模仿学习是强化学习与监督学习的结合，目标是通过观察专家演示，学习专家策略，从而加速强化学习。通过引入任务相关的额外信息，模仿学习相较于强化学习，可以更快地实现策略优化，为缓解低样本效率问题提供了解决方案。模仿学习已成为解决强化学习问题的一种流行框架，涌现出多种提高学习性能的算法和技术。通过与图形图像学的最新研究成果相结合，模仿学习已经在游戏人工智能（artificial intelligence，AI）、机器人控制和自动驾驶等领域发挥了重要作用。本文围绕模仿学习的年度发展，从行为克隆、逆强化学习、对抗式模仿学习、基于观察量的模仿学习和跨领域模仿学习等多个角度进行深入探讨，介绍了模仿学习在实际应用上的最新情况，比较了国内外研究现状，并展望了该领域未来的发展方向。旨在为研究人员和从业人员提供模仿学习的最新进展，从而为开展工作提供参考与便利。;Imitation learning(IL) is focused on the integration of reinforcement learning and supervised learning through observing demonstrations and learning expert strategies. The additional information related imitation learning can be used to optimize and implement its strategy, which can provide the possibility to alleviate low efficiency of sample problem. In recent years, imitation learning has become a popular framework for solving reinforcement learning problems, and a variety of algorithms and techniques have emerged to improve the performance of learning procedure. Combined with the latest research in the field of image processing, imitation learning has played an important role in such domains like game artificial intelligence(AI), robot control, autonomous driving. Traditional imitation learning methods are mainly composed of behavioral cloning(BC), inverse reinforcement learning(IRL), and adversarial imitation learning(AIL). Thanks to the computing ability and upstream graphics and image tasks(such as object recognition and scene understanding), imitation learning methods can be used to integrate a variety of technologies-emerged for complex tasks. We summarize and analyze imitation learning further, which is composed of imitation learning from observation(ILfO) and cross-domain imitation learning(CDIL). The ILfO can be used to optimize the requirements for expert demonstration, and information-observable can be learnt only without specific action information from experts. This setting makes imitation learning algorithms more practical, and it can be applied to real-life scenes. To alter the environment transition dynamics modeling, ILfO algorithms can be divided into two categories:model-based and model-free. For model-based methods, due to path-constructed of the model in the process of interaction between the agent and the environment, it can be assorted into forward dynamic model and inverse dynamic model further. The other related model-free methods are mainly composed of adversarial-based and function-rewarded engineering. Cross-domain imitation learning are mainly focused on the status of different domains for agents and experts, such as multiple Markov decision processes. Current CDIL research are mainly focused on the domain differences of three aspects of discrepancy in relevant to:transition dynamics, morphological, and view point. The technical solutions to CDIL problems can be mainly divided into such methods like:direct, mapping, adversarial, and optimal transport. The application of imitation learning is mainly on such aspects like game AI, robot control, and automatic driving. The recognition and perception capabilities of intelligent agents are optimized further in image processing, such as object detection, video understanding, video classification, and video recognition. Our critical analysis can be focused on the annual development of imitation learning from the five aspects:behavioral cloning, inverse reinforcement learning, adversarial imitation learning, imitation learning from observation, and cross-domain imitation learning.

Cache-Assisted Collaborative Task Offloading and Resource Allocation Strategy: A Metareinforcement Learning Approach

Multiaccess edge computing (MEC) provides users with better Quality of Experience (QoE) via offloading tasks to the nearby edge. However, the emergence of new Internet of Things applications with multiple tasks and repeated requests brings redundant computation and transmission to the edge. Meanwhile, the current offloading method based on deep reinforcement learning (DRL) has low sampling efficiency and slow convergence issues for training in a changing environment. Therefore, improving QoE of computation offloading services is still the ultimate challenge. In this article, we devise a collaboration of computing and cache resources among multiple edge nodes, which could reduce redundant computation and transmission. Specifically, we formulate a cache-assisted computation offloading process as a QoE-aware utility maximization problem based on multidimensional indicators. Then, we propose a cache-assisted collaborative task offloading and resource allocation strategy to solve it. This strategy is decomposed into two subproblems. First, to determine and obtain task cache state, we propose a collaborative task caching algorithm, which can improve the hit rate of tasks while balancing network overhead. Second, to acquire offloading and resource allocation decisions efficiently, we propose a metareinforcement learning-based cache-assisted computation offloading method (MCCOM), which can achieve rapid offloading decisions with a few gradient updates and samples. The optimization problem was transformed into multiple Markov decision processes (multiple MDPs). The improved learning process includes metapolicy learning that adapts to multiple Markov decision processes (MDPs) and policy learning for a specific MDP based on metapolicy. Simulation results show that our proposed method outperforms baselines in terms of QoE indicators while achieving rapid convergence and decisions.

A multi-action deep reinforcement learning framework for flexible Job-shop scheduling problem

This paper presents an end-to-end deep reinforcement framework to automatically learn a policy for solving a flexible Job-shop scheduling problem (FJSP) using a graph neural network. In the FJSP environment, the reinforcement agent needs to schedule an operation belonging to a job on an eligible machine among a set of compatible machines at each timestep. This means that an agent needs to control multiple actions simultaneously. Such a problem with multi-actions is formulated as a multiple Markov decision process (MMDP). For solving the MMDPs, we propose a multi-pointer graph networks (MPGN) architecture and a training algorithm called multi-Proximal Policy Optimization (multi-PPO) to learn two sub-policies, including a job operation action policy and a machine action policy to assign a job operation to a machine. The MPGN architecture consists of two encoder-decoder components, which define the job operation action policy and the machine action policy for predicting probability distributions over different operations and machines, respectively. We introduce a disjunctive graph representation of FJSP and use a graph neural network to embed the local state encountered during scheduling. The computational experiment results show that the agent can learn a high-quality dispatching policy and outperforms handcrafted heuristic dispatching rules in solution quality and meta-heuristic algorithm in running time. Moreover, the results achieved on random and benchmark instances demonstrate that the learned policies have a good generalization performance on real-world instances and significantly larger scale instances with up to 2000 operations.

Method for Constructing Artificial Intelligence Player With Abstractions to Markov Decision Processes in Multiplayer Game ofMahjong

In this article, we propose a method for constructing artificial intelligence (AI) player of Mahjong, which is a multiplayer imperfect information game. Since the size of the game tree is huge, constructing an expert-level AI player of Mahjong is challenging. We define multiple Markov decision processes (MDPs) as abstractions of Mahjong to construct effective search trees. We also introduce two methods of inferring state values of the Mahjong using these MDPs. We evaluated the effectiveness of our method using gameplays vis-à-vis the current strongest AI player.