TRIAG: Tri-reinforced infused generative agents for financial risk compliance

Reinforcement learning is of vital significance in machine learning and is also a promising approach for traffic signal control in urban road networks with assistance of deep neural networks. However, in a large scale urban network, the centralized reinforcement learning approach is beset with difficulties due to the extremely high dimension of joint action space. The multi-agent reinforcement learning (MARL) approach overcomes the high dimension problem by employing distributed local agents whose action space is much smaller. Even though, MARL approach introduces another issue that multiple agents interact with environment simultaneously causing its instability so that training each agent independently may not converge. This paper presents an actor-critic based decentralized MARL approach to control traffic signal which overcomes the shortcomings of both centralized RL approach and independent MARL approach. In particular, a distributed critic network is designed which overcomes the difficulty to train a large-scale neural network in centralized RL approach. Moreover, a difference reward method is proposed to evaluate the contribution of each agent, which accelerates the convergence of algorithm and makes agents optimize policy in a more accurate direction. The proposed MARL approach is compared against the fully independent approach and the centralized learning approach in a grid network. Simulation results demonstrate its effectiveness in terms of average travel speed, travel delay and queue length over other MARL algorithms.

In this paper, we investigate the use of hierarchical reinforcement learning (HRL) to speed up the acquisition of cooperative multi-agent tasks. We introduce a hierarchical multi-agent reinforcement learning (RL) framework, and propose a hierarchical multi-agent RL algorithm called Cooperative HRL. In this framework, agents are cooperative and homogeneous (use the same task decomposition). Learning is decentralized, with each agent learning three interrelated skills: how to perform each individual subtask, the order in which to carry them out, and how to coordinate with other agents. We define cooperative subtasks to be those subtasks in which coordination among agents significantly improves the performance of the overall task. Those levels of the hierarchy which include cooperative subtasks are called cooperation levels. A fundamental property of the proposed approach is that it allows agents to learn coordination faster by sharing information at the level of cooperative subtasks, rather than attempting to learn coordination at the level of primitive actions. We study the empirical performance of the Cooperative HRL algorithm using two testbeds: a simulated two-robot trash collection task, and a larger four-agent automated guided vehicle (AGV) scheduling problem. We compare the performance and speed of Cooperative HRL with other learning algorithms, as well as several well-known industrial AGV heuristics. We also address the issue of rational communication behavior among autonomous agents in this paper. The goal is for agents to learn both action and communication policies that together optimize the task given a communication cost. We extend the multi-agent HRL framework to include communication decisions and propose a cooperative multi-agent HRL algorithm called COM-Cooperative HRL. In this algorithm, we add a communication level to the hierarchical decomposition of the problem below each cooperation level. Before an agent makes a decision at a cooperative subtask, it decides if it is worthwhile to perform a communication action. A communication action has a certain cost and provides the agent with the actions selected by the other agents at a cooperation level. We demonstrate the efficiency of the COM-Cooperative HRL algorithm as well as the relation between the communication cost and the learned communication policy using a multi-agent taxi problem.

Collaborative optimization of multi-energy multi-microgrid system: A hierarchical trust-region multi-agent reinforcement learning approach

Same-day delivery (SDD) services have become increasingly popular in recent years. These have been usually modeled by previous studies as a certain class of dynamic vehicle routing problem (DVRP) where goods must be delivered from a depot to a set of customers in the same day that the orders were placed. Adaptive exact solution methods for DVRPs can become intractable even for small problem instances. In this paper, the same-day delivery problem (SDDP) is formulated as a Markov decision process (MDP) and it is solved using a parameter-sharing Deep Q-Network, which corresponds to a decentralised multi-agent reinforcement learning (MARL) approach. For this, a multi-agent grid-based SDD environment is created, consisting of multiple vehicles, a central depot, and dynamic order generation. In addition, zone-specific order generation and reward probabilities are introduced. The performance of the proposed MARL approach is compared against a mixed-integer programming (MIP) solution. Results show that the proposed MARL framework performs on par with MIP-based policy when the number of orders is relatively low. For problem instances with higher order arrival rates, computational results show that the MARL approach underperforms MIP by up to 30%. The performance gap between both methods becomes smaller when zone-specific parameters are employed. The gap is reduced from 30% to 3% for a 5 × 5 grid scenario with 30 orders. Execution time results indicate that the MARL approach is, on average, 65 times faster than the MIP-based policy, and therefore may be more advantageous for real-time control, at least for small-sized instances.

Today, air traffic controllers communicate with pilots via radio to direct the aircraft. The increasing demand in drone delivery and air mobility will increase air density, which requires highly automated air traffic control systems. To meet the growing demand in air transportation, a high standard autonomous support system is needed. In particular, we need an autonomous air traffic controller to keep the airspace safe and efficient. In this study, we propose a novel Multi-Agent Reinforcement Learning (MARL) approach to handle high-density UAM operations by providing effective guidance to electric vertical takeoff and landing (eVTOL) vehicles to avoid traffic congestion and reduce travel time. The goal of our MARL approach is to reduce the time at the environed urban air intersections by providing the speed advisories to each approaching vehicle for safe separation at the intersection. The proposed model is trained and evaluated in BlueSky, an open-source air traffic control simulation environment. The results of our simulations with real-world data from thousands of aircraft show that using MARL for the separation problem at the intersection is very promising for solving the problem of en-route air traffic control.

Multi-access edge computing (MEC) is a novel computing paradigm that leverages nearby MEC servers to augment the computational capabilities of users with limited computational resources. In this paper, we investigate the computational offloading problem in multi-user multi-server MEC systems with energy harvesting, aiming to minimize both system latency and energy consumption by optimizing task offload location selection and task offload ratio.We propose a hierarchical computational offloading strategy based on multi-agent reinforcement learning (MARL). The proposed strategy decomposes the computational offloading problem into two sub-problems: a high-level task offloading location selection problem and a low-level task offloading ratio problem. The complexity of the problem is reduced by decoupling. To address these sub-problems, we propose a computational offloading framework based on multi-agent proximal policy optimization (MAPPO), where each agent generates actions based on its observed private state to avoid the problem of action space explosion due to the increasing number of user devices. Simulation results show that the proposed HDMAPPO strategy outperforms other baseline algorithms in terms of average task latency, energy consumption, and discard rate.

The connected autonomous vehicle is considered an effective way to improve transport safety and efficiency. To overcome the limited sensing and computing capabilities of individual vehicles, we design a digital twin assisted decision-making framework for Internet of Vehicles, by leveraging the integration of communication, sensing and computing. In this framework, the digital twin entities residing on edge can effectively communicate and cooperate with each other to plan sub-targets for their respective vehicles, while the vehicles only need to achieve the sub-targets by generating a sequence of atomic actions. Furthermore, we propose a hierarchical multiagent reinforcement learning approach to implement the framework, which can be trained in an end-to-end way. In the proposed approach, the communication interval of digital twin entities could adapt to time-varying environment. Extensive experiments on driving decision-making have been performed in traffic junction scenarios of different difficulties. The experimental results show that the proposed approach can largely improve collaboration efficiency while reducing communication overhead.

The traffic assignment problem (TAP) consists of assigning routes to road users in order to minimize traffic congestion. Traditional methods for solving the TAP assume the existence of a central authority who computes and dictates routes to road users. Multi-agent reinforcement learning (MARL) approaches are more realistic in solving this kind of problem because they consider that road users (agents) have complete autonomy for choosing routes. However, MARL approaches usually require a long training period in order to compute the optimal routes, which could be a major limitation in more realistic traffic scenarios. In this paper, we tackle this problem by evaluating the performance of three conceptually different reward functions, namely: expert-designed rewards, difference rewards, and intrinsically motivated rewards. In particular, our focus lies on providing a deeper understanding of the impact of these reward functions on the agents’ performance, thus contributing towards reducing congestion levels. To this end, we perform an extensive experimental evaluation on different road networks, including up to 360,600 concurrently learning agents. Our results show that, although the adopted reward functions were not able to speed up the learning process, the correct reward function choice plays an important role in the quality of the learned solution.

: In this paper, we investigate the use of hierarchical reinforcement learning (HRL) to speed up the acquisition of cooperative multiagent tasks. We introduce a hierarchical multiagent reinforcement learning (RL) framework and propose a hierarchical multiagent RL algorithm called Cooperative HRL. In our approach, agents are cooperative and homogeneous (use the same task decomposition). Learning is decentralized, with each agent learning three interrelated skills: how to perform subtasks, which order to do them in, and how to coordinate with other agents. We define cooperative subtasks to be those subtasks in which coordination among agents significantly improves the performance of the overall task. Those levels of the hierarchy which include cooperative subtasks are called cooperation levels. Since coordination at high levels allows for increased cooperation skills as agents do not get confused by low-level details, we usually define cooperative subtasks at the high levels of the hierarchy.

Hierarchical task network-enhanced multi-agent reinforcement learning: Toward efficient cooperative strategies.

Residential microgrid is widely considered as a new paradigm of the home energy management system. The complexity of Microgrid Energy Scheduling (MES) is increasing with the integration of Electric Vehicles (EVs) and Renewable Generations (RGs). Moreover, it is challenging to determine optimal scheduling strategies to guarantee the efficiency of the microgrid market and to balance all market participants’ benefits. In this paper, a Multi-Agent Reinforcement Learning (MARL) approach for residential MES is proposed to promote the autonomy and fairness of microgrid market operation. First, a multi-agent based residential microgrid model including Vehicle-to-Grid (V2G) and RGs is constructed and an auction-based microgrid market is built. Then, distinguish from Single-Agent Reinforcement Learning (SARL), MARL can achieve distributed autonomous learning for each agent and realize the equilibrium of all agents’ benefits, therefore, we formulate an equilibrium-based MARL framework according to each participant’ market orientation. Finally, to guarantee the fairness and privacy of the MARL process, we proposed an improved optimal Equilibrium Selection-MARL (ES-MARL) algorithm based on two mechanisms, private negotiation and maximum average reward. Simulation results demonstrate the overall performance and efficiency of proposed MARL are superior to that of SARL. Besides, it is verified that the improved ES-MARL can get higher average profit to balance all agents.

HELICON is a novel hierarchical Reinforcement Learning (RL) approach for orchestrating the dynamic placement of Virtual Network Functions (VNFs) in Cloud and Edge 5G environments. It proves capable of addressing an NP-Hard decision-making problem with adopted RL while augmenting the current state of the art in orchestrators with a previously unexplored lightweight distributed and hierarchical RL approach. HELICON can run as a fully autonomous solution or complement orchestrators, thus bridging a significant gap in existing orchestrators, which generally lack intelligent and dynamic adaptation capabilities. Finally, our performance evaluation results over an actual 5G city testbed and use case validate that HELICON outperforms traditional policy-based Open Source MANO and other heuristic policies concerning single or multi-objective optimisation goals. What is more, HELICON’s performance meets with that of node-specific custom supervised learning models, whereas it clearly outperforms supervised learning under dynamic conditions.

Reinforcement learning addresses the problem of learning to select actions in order to maximize an agent's performance in unknown environments. To scale reinforcement learning to complex real-world tasks, agent must be able to discover hierarchical structures within their learning and control systems. In this paper, the use of hierarchical reinforcement learning (HRL) to speed up the acquisition of cooperative multi-agent tasks is investigated, and a hierarchical multi-agent reinforcement learning (RL) framework and a hierarchical multi-agent RL algorithm called cooperative HRL are proposed. A fundamental property of the proposed approach is that it allows agents to learn coordination faster by sharing information at the level of cooperative subtasks, rather than attempting to learn coordination at the level of primitive actions. This approach can significantly speed up learning and make it more scalable with the number of agents.

While fixed topology formation control with a cen-tralized controller has been studied for multi-agent systems, it remains challenging to develop robust distributed control policies that can achieve a flexible formation without a global coordinate system. In this paper, we design a fully decentralized displacement-based formation control policy for multi-agent sys-tems, which can achieve any formation after one-time training. In particular, we use a model-free multi-agent reinforcement learning (MARL) approach to obtain such a policy in the centralized training process. The Hausdorff distance is adopted in the reward function for measuring the distance between the current and target topology. The feasibility of our method is verified by both simulation and implementation on omni-directional vehicles.

The charging price influences electric vehicle (EV) users’ charging decisions. A reasonable pricing strategy not only reduces the operating cost of the power distribution network (PDN), but also enhances the profitability of charging station operators (CSOs). EV charging stations (EVCSs) engage in both competition and cooperation to maximize their own interests or the CSOs. The dynamic states and strategic interactions among heterogeneous EVCSs pose significant challenges to pricing decisions. This paper proposes a multi-agent reinforcement learning (MARL) approach for optimal pricing model of EVCSs exhibiting mix competitive and cooperative behaviors in coupled power and transportation networks (CPTNs). First, a mixed game pricing model is developed, incorporating the economic dispatch of the PDN and user charging decision responses. The pricing problem is formulated as a general-sum Markov game, and a MARL algorithm based on a hierarchical attention mechanism is designed to approximate the Nash equilibrium of pricing games. The algorithm captures interaction dependencies among heterogeneous agents through a hierarchical graph and leverages an attention mechanism to integrate local and global features from neighboring agents for policy generation and value estimation, thereby ensuring stability in multi-agent environments. Finally, the effectiveness of the algorithm is validated on a CPTN system in Xi’an, China.