Decentralized Partially Observable Markov Decision Processes Research Articles

Hypergraph convolution mix DDPG for multi-aerial base station deployment

Aerial base stations (AeBS), as crucial components of air-ground integrated networks, can serve as the edge nodes to provide flexible services to ground users. Optimizing the deployment of multiple AeBSs to maximize system energy efficiency is currently a prominent and actively researched topic in the AeBS-assisted edge-cloud computing network. In this paper, we deploy AeBSs using multi-agent deep reinforcement learning (MADRL). We describe the multi-AeBS deployment challenge as a decentralized partially observable Markov decision process (Dec-POMDP), taking into consideration the constrained observation range of AeBSs. The hypergraph convolution mix deep deterministic policy gradient (HCMIX-DDPG) algorithm is designed to maximize the system energy efficiency. The proposed algorithm uses the value decomposition framework to solve the lazy agent problem, and hypergraph convolutional (HGCN) network is introduced to strengthen the cooperative relationship between agents. Simulation results show that the suggested HCMIX-DDPG algorithm outperforms alternative baseline algorithms in the multi-AeBS deployment scenario.

An integrated control strategy for simultaneous robot assignment, tool change and preventive maintenance scheduling using Heterogeneous Graph Neural Network

There has been a leap in the field of smart manufacturing with the advancement of automation systems, robotic technology, big data analytics, and state-of-the-art Artificial Intelligence (AI) and Machine Learning (ML) algorithms. Three very important aspects of smart manufacturing systems are system productivity, product quality, and maintenance of machines and equipment. These three issues are strongly interrelated and collectively determine the performance of a smart manufacturing system. Although there has been significant studies in production control, quality control and maintenance scheduling to address each of these aspects individually, there has been a lack of sufficient studies taking all of them into consideration in one control scheme. In this paper, a mobile multi-skilled robot operated Flexible Manufacturing System (FMS) is considered and a model that integrates robots, individual workstation processes and product quality is developed using a Heterogeneous Graph Structure. Heterogeneous Graph Neural Network (HGNN) is used to aggregate local information from different nodes of the graph model to create node embeddings that represent global information. A control problem is then formulated in the Decentralized Partially Observable Markov Decision Process (Dec-POMDP) framework to simultaneously consider robot assignment, product quality and maintenance scheduling. The problem is solved using Multi-Agent Reinforcement Learning (MARL). A case study is presented to demonstrate the effectiveness of the HGNN-MARL control strategy by comparing it to three baselines and the naive MARL policy without HGNN.

Counterfactual-attention multi-agent reinforcement learning for joint condition-based maintenance and production scheduling

Maintenance and production scheduling are interactive activities that should be considered simultaneously to maintain production systems’ reliability and high production delivery rate. This study investigates a problem of joint maintenance and production scheduling in a multi-stage hybrid flow shop experiencing machine deterioration processes. To address the structural dependency among production stages and machines, a decentralized partially observable Markov decision process (Dec-POMDP) is considered. However, the large state and action space pose a challenge for existing reinforcement learning methods to provide satisfactory solutions. To overcome this problem, the study proposes a counterfactual attention multi-agent reinforcement learning (CAMARL) solution framework that comprises three functional modules: The attention mechanism module, which adaptively shrinks the dimension of the state space; the action abstraction representation module that deals with the high-dimension of the action space; and the coordination control unit, which accelerates the exploration for the optimal production policy. Numerical experiments demonstrate the effectiveness of the proposed method by comparing it with seven benchmarks under different production scenarios.

Privacy-Preserving Joint Edge Association and Power Optimization for the Internet of Vehicles via Federated Multi-Agent Reinforcement Learning

Proactive edge association is capable of improving wireless connectivity at the cost of increased handover (HO) frequency and energy consumption, while relying on a large amount of private information sharing required for decision making. In order to improve the connectivity-cost trade-off without privacy leakage, we investigate the privacy-preserving joint edge association and power allocation (JEAPA) problem in the face of the environmental uncertainty and the infeasibility of individual learning. Upon modelling the problem by a decentralized partially observable Markov Decision Process (Dec-POMDP), it is solved by federated multi-agent reinforcement learning (FMARL) through only sharing encrypted training data for federatively learning the policy sought. Our simulation results show that the proposed solution strikes a compelling trade-off, while preserving a higher privacy level than the state-of-the-art solutions.

Multi-Agent Deep Reinforcement Learning-Based Interdependent Computing for Mobile Edge Computing-Assisted Robot Teams

A group of robots can be assigned with different roles to collaboratively conduct interdependent tasks. The robots form a multi-robot system (MRS), where one robot's decision or action relies on the others'. This paper addresses the sequential decision problem of user association and resource allocation in a mobile edge computing (MEC)-enabled, wirelessly-connected MRS to maximize the time-averaged completion rate of interdependent computing tasks. The problem is challenging due to the partial observability of the network environment, and the delicate delay requirements of interdependent computing tasks. A new decentralized partially observable Markov decision process (Dec-POMDP) problem is reformulated, where edge servers act as intelligent agents and can make decentralized decisions about user association and resource management with their local information of the network state. By leveraging the multi-agent deep deterministic policy gradient (MADDPG) theory, a new cooperative multi-agent deep reinforcement learning (MADRL) model is developed to enable interdependent computing. Simulations show the merits of our approach in terms of task completion rate compared to existing techniques.

Cooperative Data Collection With Multiple UAVs for Information Freshness in the Internet of Things

Maintaining the freshness of information in the Internet of Things (IoT) is a critical yet challenging problem. In this paper, we study cooperative data collection using multiple Unmanned Aerial Vehicles (UAVs) with the objective of minimizing the total average Age of Information (AoI). We consider various constraints of the UAVs, including kinematic, energy, trajectory, and collision avoidance, in order to optimize the data collection process. Specifically, each UAV, which has limited on-board energy, takes off from its initial location and flies over sensor nodes to collect update packets in cooperation with the other UAVs. The UAVs must land at their final destinations with non-negative residual energy after the specified time duration to ensure they have enough energy to complete their missions. It is crucial to design the trajectories of the UAVs and the transmission scheduling of the sensor nodes to enhance information freshness. We model the multi-UAV data collection problem as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP), as each UAV is unaware of the dynamics of the environment and can only observe a part of the sensors. To address the challenges of this problem, we propose a multi-agent Deep Reinforcement Learning (DRL)-based algorithm with centralized learning and decentralized execution. In addition to the reward shaping, we use action masks to filter out invalid actions and ensure that the constraints are met. Simulation results demonstrate that the proposed algorithms can significantly reduce the total average AoI compared to the baseline algorithms, and the use of the action mask method can improve the convergence speed of the proposed algorithm.

Hierarchical multi-agent reinforcement learning for repair crews dispatch control towards multi-energy microgrid resilience

Extreme events are greatly impacting the normal operations of microgrids, which can lead to severe outages and affect the continuous supply of energy to customers, incurring substantial restoration costs. Repair crews (RCs) are regarded as crucial resources to provide system resilience owing to their mobility and flexibility characteristics in handling both transportation and energy systems. Nevertheless, effectively coordinating the dispatch of RCs towards system resilience is a complex decision-making problem, especially in the context of a multi-energy microgrid (MEMG) with enormous dynamics and uncertainties. To this end, this paper formulates the dispatch problem of RCs in a coupled transportation and power-gas network as a decentralized partially observable Markov decision process (Dec-POMDP). To solve this Dec-POMDP, a hierarchical multi-agent reinforcement learning (MARL) algorithm is proposed by featuring a two-level framework, where the high-level action is used for switching decision-making between transportation and power-gas networks, and the lower-level action constructed via the multi-agent proximal policy optimization (MAPPO) algorithm is used to compute the routing and repairing decisions of RCs in the transportation and power-gas networks, respectively. The proposed algorithm also introduces an abstracted critic network by integrating the load restoration status, which captures the system dynamics and stabilizes the training performance with privacy protection. Extensive case studies are evaluated on a coupled 6-bus power and 6-bus gas network integrated with a 9-node 12-edge transportation network. The proposed algorithm outperforms the conventional MARL algorithms in terms of policy quality, learning stability, and computational performance. Furthermore, the dispatch strategies of RCs are analyzed and their corresponding benefits for load restoration are also evaluated. Finally, the scalability of the proposed method is also investigated for a larger 33-bus power and 15-bus gas network integrated with an 18-node 27-edge transportation network.

Decentralized Learning of Finite-Memory Policies in Dec-POMDPs

Multi-agent reinforcement learning (MARL) under partial observability is notoriously challenging as the agents only have asymmetric partial observations of the system. In this paper, we study MARL in decentralized partially observable Markov decision processes (Dec-POMDPs) with partial history sharing. In search of decentralized and tractable MARL solutions, we identify the appropriate conditions under which we can adopt the common information approach to naturally extend existing single-agent policy learners to Dec-POMDPs. In particular, under the conditions of bounded local memories and an efficient representation of the common information, we present a MARL algorithm that learns a near-optimal finite-memory policy in Dec-POMDPs. We establish the iteration complexity of the algorithm, which depends only linearly on the number of agents. Simulations on classic Dec-POMDP tasks show that our approach significantly outperforms existing decentralized solutions, and nearly matches the centralized ones that require stronger informational assumptions.

Decentralized Federated Reinforcement Learning for User-Centric Dynamic TFDD Control

The explosive growth of dynamic and heterogeneous data traffic brings great challenges for 5G and beyond mobile networks. To enhance the network capacity and reliability, we propose a learning-based dynamic time-frequency division duplexing (D-TFDD) scheme that adaptively allocates the uplink and downlink time-frequency resources of base stations (BSs) to meet the asymmetric and heterogeneous traffic demands while alleviating the inter-cell interference. We formulate the problem as a decentralized partially observable Markov decision process (Dec-POMDP) that maximizes the long-term expected sum rate under the users' packet dropping ratio constraints. In order to jointly optimize the global resources in a decentralized manner, we propose a federated reinforcement learning (RL) algorithm named federated Wolpertinger deep deterministic policy gradient (FWDDPG) algorithm. The BSs decide their local time-frequency configurations through RL algorithms and achieve global training via exchanging local RL models with their neighbors under a decentralized federated learning framework. Specifically, to deal with the large-scale discrete action space of each BS, we adopt a DDPG-based algorithm to generate actions in a continuous space, and then utilize Wolpertinger policy to reduce the mapping errors from continuous action space back to discrete action space. Simulation results demonstrate the superiority of our proposed algorithm to benchmark algorithms with respect to system sum rate.

Real-Time Operation Management for Battery Swapping-Charging System via Multi-Agent Deep Reinforcement Learning

Battery swapping-charging systems (BSCSs) can provide better battery swapping services for electric vehicles (EVs) in large cities. In BSCSs, EV batteries can be centrally charged at battery charging stations (BCSs) and then dispatched via delivery trucks to battery swapping stations (BSSs) to support local EVs. This paper considers the real-time optimization scheduling problem in BSCS, including battery charging, swapping and truck routing. We model this real-time scheduling problem as a decentralized partially observable Markov decision process (Dec-POMDP) and solve it using multi-agent deep reinforcement learning (MADRL) algorithms. The joint scheduling process of trucks and BCSs has many dynamic hard constraints between them that cannot be solved using the existing MADRL algorithms. To this end, we combine MADRL with binary integer programming (BLP) and propose the Value Decomposition Network (VDN)-BLP algorithm to solve the problem with constraints. We also combine actor-critic architecture and local search with VDN-BLP to substantially improve computational efficiency with little performance loss. Simulation results based on historical battery swapping data in Sanya City verify the effectiveness of the proposed method.

Dual-attention assisted deep reinforcement learning algorithm for energy-efficient resource allocation in Industrial Internet of Things

This paper investigates the distributed resource allocation problem for energy-efficient data forwarding in resource-constrained Industrial Internet of Things (IIoT). We formulate this problem as a decentralized partially observable Markov decision process (Dec-POMDP) and propose a novel energy-efficient resource allocation algorithm based on the dual-attention assisted deep reinforcement learning (DRL) model, namely DADR, which adopts the centralized training and distributed executed (CTDE) framework to provide the intelligent decision-making ability for resource-constrained nodes. In DADR, we design a multi-scale convolutional attention module (CAM) in actor network that can extract the local state’s feature information from different dimensions; meanwhile, we propose a novel critic network based on dual-attention module and experience reconstruction module which provides a more objective and correct state evaluation from a global perspective. Moreover, the proposed critic network can solve the partially observable and non-stationary problems in multi-agent systems and can flexibly scale without changing the model structure even in a dynamic environment. Furthermore, the cooperation between CAM and multi-head self-attention (MHSA) in DADR improves the representation learning ability of DRL and offers better optimization direction for the DRL model to maximize the energy-efficient and data transmission reliability. Simulation results demonstrate that the proposed DADR algorithm outperforms the existing resource allocation algorithms and MARL models in terms of network lifetime, transmission reliability, and network stability, respectively.

Deep reinforcement learning based topology-aware voltage regulation of distribution networks with distributed energy storage

Both the high penetration of clean energy with strong fluctuation and the complicated variable operation condition bring great challenges to the voltage regulation of the distribution network. To deal with the problem, a topology-aware voltage regulation multi-agent deep reinforcement learning (MADRL) algorithm is proposed. The distributed energy storages (DESs) are modeled as agents to regulate voltage autonomously in real-time, which could fast adapt to dynamic topological scenarios. Firstly, taking the minimization of voltage fluctuation and maximization of reserve capacity as the target, the optimal voltage regulation model is established. Secondly, a topology extraction method considering voltage sensitivity is proposed for dynamic topology clustering, and the obtained typical topology is added to the observation set of agents. Then, the optimal voltage regulation model is formulated to the decentralized partially observable Markov decision process (Dec-POMDP) framework, in which only local information is required for the agent during the test process to decision-making to realize the hierarchical and partitioned control of voltage. Finally, the multi-agent deep deterministic policy gradient (MADDPG) algorithm is used to solve the Dec-POMDP model. The feasibility and superiority of the proposed algorithm are verified and analyzed in the simulation under different scenarios.

Cooperative multi-target hunting by unmanned surface vehicles based on multi-agent reinforcement learning

To solve the problem of multi-target hunting by an unmanned surface vehicle (USV) fleet, a hunting algorithm based on multi-agent reinforcement learning is proposed. Firstly, the hunting environment and kinematic model without boundary constraints are built, and the criteria for successful target capture are given. Then, the cooperative hunting problem of a USV fleet is modeled as a decentralized partially observable Markov decision process (Dec-POMDP), and a distributed partially observable multi-target hunting Proximal Policy Optimization (DPOMH-PPO) algorithm applicable to USVs is proposed. In addition, an observation model, a reward function and the action space applicable to multi-target hunting tasks are designed. To deal with the dynamic change of observational feature dimension input by partially observable systems, a feature embedding block is proposed. By combining the two feature compression methods of column-wise max pooling (CMP) and column-wise average-pooling (CAP), observational feature encoding is established. Finally, the centralized training and decentralized execution framework is adopted to complete the training of hunting strategy. Each USV in the fleet shares the same policy and perform actions independently. Simulation experiments have verified the effectiveness of the DPOMH-PPO algorithm in the test scenarios with different numbers of USVs. Moreover, the advantages of the proposed model are comprehensively analyzed from the aspects of algorithm performance, migration effect in task scenarios and self-organization capability after being damaged, the potential deployment and application of DPOMH-PPO in the real environment is verified.

Cooperative Multi-Agent Reinforcement-Learning-Based Distributed Dynamic Spectrum Access in Cognitive Radio Networks

With the development of wireless communication and Internet of Things (IoT), there are massive wireless devices that need to share the limited spectrum resources. Dynamic spectrum access (DSA) is a promising paradigm to remedy the problem of inefficient spectrum utilization brought upon by the historical command-and-control approach to spectrum allocation. In this article, we investigate the distributed DSA problem for multiusers in a typical multichannel cognitive radio network. The problem is formulated as a decentralized partially observable Markov decision process (Dec-POMDP), and we propose a centralized off-line training and distributed online execution framework based on cooperative multi-agent reinforcement learning (MARL). We employ the deep recurrent <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -network (DRQN) to address the partial observability of the state for each cognitive user. The ultimate goal is to learn a cooperative strategy which maximizes the sum throughput of a cognitive radio network in a distributed fashion without information exchange between cognitive users. Finally, we validate the proposed algorithm in various settings through extensive experiments. The experimental results show that the proposed CoMARL-DSA algorithm outperforms the state-of-the-art deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -learning for spectrum access (DQSA) in terms of successful access rate and collision rate by at least 14% and 12%, respectively.

Resource Allocation and Trajectory Design in UAV-Aided Cellular Networks Based on Multiagent Reinforcement Learning

In this article, we focus on a downlink cellular network, where multiple unmanned aerial vehicles (UAVs) serve as aerial base stations for ground users through frequency-division multiple access (FDMA). With user locations and channel parameters inaccessible, the UAVs coordinate to make a decision on resource allocation and trajectory design in a decentralized way. Aiming at optimizing both overall and fairness throughput, we model resource allocation and trajectory design as a decentralized partially observable Markov decision process (Dec-POMDP) and propose multiagent reinforcement learning (RL) as a solution. Specifically, we use parameterized deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -network (P-DQN) for the action space comprising both discrete and continuous actions and the QMIX framework is leveraged to aggregate each UAV’s local critics. For fairness throughput optimization, we introduce an entropy-like fairness indicator to the reward to make the total return decomposable. In addition, we further propose a novel distributed learning framework for overall throughput optimization such that each UAV can contribute its local gradient, and model training can be implemented in parallel without need of observation data sharing among the UAVs. Simulation results show that the proposed multiagent RL approach as well as the distributed learning framework are efficient in model training and present acceptable performance close to that achieved by deterministic optimization, which relies on convention optimization techniques with user locations and channel parameters explicitly known beforehand. For fairness throughput optimization, we also show that ground users achieve individual throughputs close to each other, which verifies the effectiveness of the proposed fairness indicator as the reward definition in the RL framework.

Fast-DRD: Fast decentralized reinforcement distillation for deadline-aware edge computing

Edge computing has recently gained momentum as it provides computing services for mobile devices through high-speed networks. In edge computing system optimization, deep reinforcement learning(DRL) enhances the quality of services(QoS) and shorts the age of information(AoI). However, loosely coupled edge servers saturate a noisy data space for DRL exploration, and learning a reasonable solution is enormously costly. Most existing works assume that the edge is an exact observation system and harvests well-labeled data for the pretraining of DRL neural networks. However, this assumption stands in opposition to the motivation of driving DRL to explore unknown information and increases the scheduling and computing costs in large-scale dynamic systems. This article leverages DRL with a distillation module to drive learning efficiency for edge computing with partial observation. We formulate the deadline-aware offloading problem as a decentralized partially observable Markov decision process (Dec-POMDP) with distillation, called fast decentralized reinforcement distillation(Fast-DRD). Each edge server decides makes offloading decisions in accordance with its own observations and learning strategies in a decentralized manner. By defining trajectory observation history(TOH) distillation and trust distillation to avoid overfitting, Fast-DRD learns a suitable offloading model in a noisy partially observed edge system and reduces the cost for communication among servers. Finally, experimental simulations are presented to evaluate and compare the effectiveness and complexity of Fast-DRD.

Graph-Embedded Multi-Agent Learning for Smart Reconfigurable THz MIMO-NOMA Networks

With the accelerated development of immersive applications and the explosive increment of internet-of-things (IoT) terminals, 6G would introduce terahertz (THz) massive multiple-input multiple-output non-orthogonal multiple access (MIMO-NOMA) technologies to meet the ultra-high-speed data rate and massive connectivity requirements. Nevertheless, the unreliability of THz transmissions and the extreme heterogeneity of device requirements pose critical challenges for practical applications. To address these challenges, we propose a novel smart reconfigurable THz MIMO-NOMA framework, which can realize customizable and intelligent communications by flexibly and coordinately reconfiguring hybrid beams through the cooperation between access points (APs) and reconfigurable intelligent surfaces (RISs). The optimization problem is formulated as a decentralized partially-observable Markov decision process (Dec-POMDP) to maximize the network energy efficiency, while guaranteeing the diversified users’ performance, via a joint RIS element selection, coordinated discrete phase-shift control, and power allocation strategy. To solve the above non-convex, strongly coupled, and highly complex mixed integer nonlinear programming (MINLP) problem, we propose a novel multi-agent deep reinforcement learning (MADRL) algorithm, namely <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">graph-embedded value-decomposition actor-critic (GE-VDAC)</i> , that embeds the interaction information of agents, and learns a locally optimal solution through a distributed policy. Numerical results demonstrate that the proposed algorithm achieves highly customized communications and outperforms traditional MADRL algorithms.

Impact of Heterogeneity and Risk Aversion on Task Allocation in Multi-Agent Teams

Cooperative multi-agent decision-making is a ubiquitous problem with many real-world applications. In many practical applications, it is desirable to design a multi-agent team with a heterogeneous composition where the agents can have different capabilities and levels of risk tolerance to address diverse requirements. While heterogeneity in multi-agent teams offers benefits, new challenges arise including how to find optimal heterogeneous team compositions and how to dynamically distribute tasks among agents in complex operations. In this work, we develop an artificial intelligence framework for multi-agent heterogeneous teams to dynamically learn task distributions among agents through reinforcement learning. The framework extends Decentralized Partially Observable Markov Decision Processes (Dec-POMDP) to be compatible to model various types of heterogeneity. We demonstrate our approach with a benchmark problem on a disaster relief scenario. The effect of heterogeneity and risk aversion in agent capabilities and decision-making strategies on the performance of multi-agent teams in uncertain environments is analyzed. Results show that a well-designed heterogeneous team outperforms its homogeneous counterpart and possesses higher adaptivity in uncertain environments.

Informed Initial Policies for Learning in Dec-POMDPs

Decentralized partially observable Markov decision processes (Dec-POMDPs) offer a formal model for planning in cooperative multiagent systems where agents operate with noisy sensors and actuators, and local information. Prevalent Dec-POMDP solution techniques have mostly been centralized and have assumed knowledge of the model. In real world scenarios, however, solving centrally may not be an option and model parameters maybe unknown. To address this, we propose a distributed, model-free algorithm for learning Dec-POMDP policies, in which agents take turns learning, with each agent not currently learning following a static policy. For agents that have not yet learned a policy, this static policy must be initialized. We propose a principled method for learning such initial policies through interaction with the environment. We show that by using such informed initial policies, our alternate learning algorithm can find near-optimal policies for two benchmark problems.

Sample Bounded Distributed Reinforcement Learning for Decentralized POMDPs

Decentralized partially observable Markov decision processes (Dec-POMDPs) offer a powerful modeling technique for realistic multi-agent coordination problems under uncertainty. Prevalent solution techniques are centralized and assume prior knowledge of the model. We propose a distributed reinforcement learning approach, where agents take turns to learn best responses to each other’s policies. This promotes decentralization of the policy computation problem, and relaxes reliance on the full knowledge of the problem parameters. We derive the relation between the sample complexity of best response learning and error tolerance. Our key contribution is to show that sample complexity could grow exponentially with the problem horizon. We show empirically that even if the sample requirement is set lower than what theory demands, our learning approach can produce (near) optimal policies in some benchmark Dec-POMDP problems.