Multi-agent Approach Research Articles

Graph Convolutional Reinforcement Learning for Collaborative Queuing Agents

This paper explores the use of multi-agent deep learning as well as learning to cooperate principles to meet strict service level agreements, in terms of throughput and end-to-end delay, for a set of classified network flows. We consider agents built on top of a weighted fair queuing algorithm that continuously set weights for three flow groups: gold, silver, and bronze. We rely on a novel graph-convolution based, multi-agent reinforcement learning approach known as DGN. As benchmarks, we propose centralized and distributed deep Q-network algorithms and evaluate their performances in different network, traffic, and routing scenarios, highlighting both the effectiveness of our proposals and the importance of agent cooperation. We show that our DGN-based approach meets stringent throughput and delay requirements across different scenarios, decreasing silver and bronze flow median waiting delays by more than 50 % and reducing the SLA violations of the latter by nearly 60 %, with respect to a classic priority queuing approach.

Delay-Aware Edge-Terminal Collaboration in Green Internet of Vehicles: A Multiagent Soft Actor-Critic Approach

The high performance requirement of task processing and low energy consumption requirement of network operation result in a contradiction for the Internet of Vehicles (IoV). This paper addresses the contradiction and proposes an efficient edge-terminal collaboration scheme to sustain the diverse task requirements for a green IoV network. Specifically, two types of tasks are first modeled in terms of data size and delay requirements and then an energy minimization problem was formulated under the diverse task requirements. Thereafter, a multi-agent reinforcement learning framework is constructed, based on which the time-varying environment of the IoV can be modeled as a partially observable Markov decision process. Furthermore, a multi-agent soft actor-critic (MASAC) scheme is proposed to train the edge-terminal collaboration policy in terms of task offloading, spectrum sharing as well as vehicle power control. Finally, the effectiveness of the proposed scheme is validated by comparing with other baseline schemes through extensive simulation experiments.

The capacitated multi-AGV scheduling problem with conflicting products: Model and a decentralized multi-agent approach

The capacitated multi-AGV scheduling problem with conflicting products: Model and a decentralized multi-agent approach

Resource scheduling and performance analysis of hybrid renewable energy systems with carbon neutrality consideration: A scenario-based multi-agent approach

Given the increasing global consensus on carbon neutrality for sustainable development, a key related challenge is to build a hybrid renewable energy system that simultaneously ensures stable power supply and carbon neutrality. This paper proposes a complete framework for designing, scheduling optimization, evaluation and analysis of hybrid renewable energy systems at the village scale. The framework integrates zero-carbon system design based on agent-based modeling, comprehensive evaluation of scheduling schemes under a multi-index system, and analysis of the impact of policy parameters, implemented through programming on the AnyLogic platform. The proposed framework was applied to a simulated region in Zhoushan, China. The results show that among the seven scheduling schemes, the photovoltaic/wind turbine/battery/diesel system is the best resource scheduling scheme in the region, with renewable energy power supply accounting for 98.5% of the total supply, the net present cost of $42,108.174, the cost of energy of $0.072/kWh, and the annual carbon dioxide emission of 5,064.346 kg. Installing carbon capture, utilization, and storage equipment can achieve carbon neutrality without significantly affecting system costs, with only an additional net present cost of $1,200. Among all meteorological factors, wind speed has the greatest impact on system performance and should be given priority in wind power generation. In terms of micro-behavioral management, discrete gear setting for diesel engines is better than continuous gear setting and more helpful in reducing system economic costs.

Microgrid energy scheduling under uncertain extreme weather: Adaptation from parallelized reinforcement learning agents

Microgrids are useful solutions for integrating renewable energy resources and providing seamless green electricity to minimize carbon footprint. In recent years, extreme weather events happened often worldwide and caused significant economic and societal losses. Such events bring uncertainties to the microgrid energy scheduling problems and increase the challenges of microgrid operation. Traditional optimization approaches suffer from the inaccuracy of the uncertain microgrid model and the unseen events. Existing reinforcement learning (RL) - based approaches are also hampered by the limited generalization and the increasing computational burden when stochastic formulations are required to accommodate the uncertainties. This paper proposes a new parallelized reinforcement learning (PRL) method based on the probabilistic events to handle the microgrid energy uncertainties. Specifically, several local learning agents are employed to interact with pertinent microgrid environments in a distributed manner and report outcomes to the global agent, which will optimize microgrid energy resources online during extreme events. The stochastic microgrid energy optimization problem is reformulated to include all possible scenarios with probabilities. The advantage estimate functions of learning agents are designed with a backward sweep to transfer the outcomes to the value function updating process. Two simulation studies, stochastic optimization and online testing, are performed to compare with several existing RL approaches. Results substantiate that the proposed PRL method can achieve up to 20% optimization performance improvement with 4 and 28 times less computation cost than Q-learning with experience replay and multi-agent Q-learning approaches, respectively.

A Multi-Agent Reinforcement Learning Approach to the Dynamic Job Shop Scheduling Problem

In a production environment, scheduling decides job and machine allocations and the operation sequence. In a job shop production system, the wide variety of jobs, complex routes, and real-life events becomes challenging for scheduling activities. New, unexpected events disrupt the production schedule and require dynamic scheduling updates to the production schedule on an event-based basis. To solve the dynamic scheduling problem, we propose a multi-agent system with reinforcement learning aimed at the minimization of tardiness and flow time to improve the dynamic scheduling techniques. The performance of the proposed multi-agent system is compared with the first-in–first-out, shortest processing time, and earliest due date dispatching rules in terms of the minimization of tardy jobs, mean tardiness, maximum tardiness, mean earliness, maximum earliness, mean flow time, maximum flow time, work in process, and makespan. Five scenarios are generated with different arrival intervals of the jobs to the job shop production system. The results of the experiments, performed for the 3 × 3, 5 × 5, and 10 × 10 problem sizes, show that our multi-agent system overperforms compared to the dispatching rules as the workload of the job shop increases. Under a heavy workload, the proposed multi-agent system gives the best results for five performance criteria, which are the proportion of tardy jobs, mean tardiness, maximum tardiness, mean flow time, and maximum flow time.

Ultimately Bounded TSK Fuzzy Management in Urban Traffic Stream Mechanism: Multi-Agent Modeling Approach

In this paper, a novel control approach based on the selection of the type of traffic light and the period of the green phase to accomplish an optimum balance at intersections is proposed. This balance should be flexible to the static behavior of time, and randomness in a traffic situation, which the goal of the proposed method is to reduce traffic volume in transportation, the average delay for each vehicle, and control over the crash of cars. The proposed method was specifically investigated at the intersection through an appropriate timing of traffic lights by sampling a multi-agent system. It consists of a large numbers of intersections, each of which is considered as an independent agent that exchanges information with each other, and the stability of each agent is provided separately. The robustness against uncertainties, scalability, and stability of the closed-loop overall system are the main merits of the proposed methodology. The simulation results show that the fuzzy intelligent controller in this multi-factor system which is a TS fuzzy is more useful than scheduling in the fixed-time method and it reduces the lengths of vehicles queuing.

Deep Reinforcement Learning for Joint Trajectory Planning, Transmission Scheduling, and Access Control in UAV-Assisted Wireless Sensor Networks.

Unmanned aerial vehicles (UAVs) can be used to relay sensing information and computational workloads from ground users (GUs) to a remote base station (RBS) for further processing. In this paper, we employ multiple UAVs to assist with the collection of sensing information in a terrestrial wireless sensor network. All of the information collected by the UAVs can be forwarded to the RBS. We aim to improve the energy efficiency for sensing-data collection and transmission by optimizing UAV trajectory, scheduling, and access-control strategies. Considering a time-slotted frame structure, UAV flight, sensing, and information-forwarding sub-slots are confined to each time slot. This motivates the trade-off study between UAV access-control and trajectory planning. More sensing data in one time slot will take up more UAV buffer space and require a longer transmission time for information forwarding. We solve this problem by a multi-agent deep reinforcement learning approach that takes into consideration a dynamic network environment with uncertain information about the GU spatial distribution and traffic demands. We further devise a hierarchical learning framework with reduced action and state spaces to improve the learning efficiency by exploiting the distributed structure of the UAV-assisted wireless sensor network. Simulation results show that UAV trajectory planning with access control can significantly improve UAV energy efficiency. The hierarchical learning method is more stable in learning and can also achieve higher sensing performance.

Learning to Routing in UAV Swarm Network: A Multi-Agent Reinforcement Learning Approach

The past few years have witnessed an exponential growth of compelling UAV swarm applications ranging from agricultural production, and intelligent transport, to disaster rescue. The high-speed mobility of the UAV swarm belongs to a new clan of networks, termed flying ad-hoc networks (FANETs). How to design an effective routing mechanism in such a dynamic network is challenging. Traditional flooding searching algorithms (e.g., OLSR, AODV) lead to huge communication overheads, while greedy searching algorithms (e.g., Geolocation-Based Routing protocol) pose low routing efficiency. In this paper, we propose an adaptive communication-based UAV swarm routing algorithm. In our algorithm, we design the Multilayer Perceptron algorithm to learn when routing flooding is required among different UAVs, and design the Gate Recurrent Unit algorithm to greatly compress the volume of communication data. Besides, we further adopt the multi-agent actor-critic algorithm to learn how to integrate shared information for cooperative routing decision-making. Extensive simulation results validate that our algorithms achieve efficient and effective routing under a partially observable distributed environment for large-scale UAV swarm cooperation.

A multi-agent coalition-based approach for order fulfilment in e-commerce

The rapid growth of e-commerce in the recent pandemic is driven by the wide adoption of digitization, extensive use of smartphones, affordable internet services, and the convenience of shopping from anywhere and anytime. Some key indexes for measuring the performance of the e-commerce organization include affordable deals, competitive prices, and efficiency in fulfilling orders while ensuring customer satisfaction. Customer review ratings are one of the significant metrics to measure customer satisfaction, influencing other customers to make purchase decisions. Therefore, favourable customer reviews and ratings of various products presented on e-commerce platforms provide competitive advantages to sellers. Usually, customers placing an order on an e-commerce platform are notified about the estimated delivery date. Consequently, after receiving the product delivery, customers give feedback in the form of review comments and ratings to communicate their satisfaction or dissatisfaction towards the seller. In case of delays in order fulfilment, the customers mostly give negative feedback along with poor ratings, which can affect the seller’s reputation, thus impacting future sales. Therefore, the central idea of this research is to adopt a resilient order fulfilment approach by identifying the optimal set of sellers based on fulfilment cost, lead time for fulfilment and review rating score obtained from previous orders. The research work has been divided into three different phases. Firstly, significant constraints associated with an e-commerce fulfilment system have been identified from existing literature. The relationship between the lead time in order fulfilment and review rating score is studied. Finally, a multi-agent coalition-based framework is proposed for generating the Pareto-optimal set of reliable sellers to fulfil an order. Although sellers in an e-commerce marketplace are usually considered competitors of each other, a temporary coalition among the sellers for aggregated order fulfilment can minimize fulfilment costs. The proposed multi-agent coalition-based framework is implemented on real-time data to confirm the findings.

Immune microenvironment of cholangiocarcinoma: Biological concepts and treatment strategies.

Cholangiocarcinoma is characterized by a poor prognosis with limited treatment and management options. Chemotherapy using gemcitabine with cisplatin is the only available first-line therapy for patients with advanced cholangiocarcinoma, although it offers only palliation and yields a median survival of < 1 year. Recently there has been a resurgence of immunotherapy studies focusing on the ability of immunotherapy to inhibit cancer growth by impacting the tumor microenvironment. Based on the TOPAZ-1 trial, the US Food and Drug Administration has approved the combination of durvalumab and gemcitabine with cisplatin as the first-line treatment of cholangiocarcinoma. However, immunotherapy, like immune checkpoint blockade, is less effective in cholangiocarcinoma than in other types of cancer. Although several factors such as the exuberant desmoplastic reaction are responsible for cholangiocarcinoma treatment resistance, existing literature on cholangiocarcinoma cites the inflammatory and immunosuppressive environment as the most common factor. However, mechanisms activating the immunosuppressive tumor microenvironment contributing to cholangiocarcinoma drug resistance are complicated. Therefore, gaining insight into the interplay between immune cells and cholangiocarcinoma cells, as well as the natural development and evolution of the immune tumor microenvironment, would provide targets for therapeutic intervention and improve therapeutic efficacy by developing multimodal and multiagent immunotherapeutic approaches of cholangiocarcinoma to overcome the immunosuppressive tumor microenvironment. In this review, we discuss the role of the inflammatory microenvironment-cholangiocarcinoma crosstalk and reinforce the importance of inflammatory cells in the tumor microenvironment, thereby highlighting the explanatory and therapeutic shortcomings of immunotherapy monotherapy and proposing potentially promising combinational immunotherapeutic strategies.

Quality of Monitoring Optimization in Underwater Sensor Networks through a Multiagent Diversity-Based Gradient Approach.

Due to the complex underwater environment, conventional measurement and sensing methods used for land are difficult to apply directly in the underwater environment. Especially for seabed topography, it is impossible to perform long-distance and accurate detection by electromagnetic waves. Therefore, various types of acoustic and even optical sensing devices for underwater applications have been used. Equipped with submersibles, these underwater sensors can detect a wide underwater range accurately. In addition, the development of sensor technology will be modified and optimized according to the needs of ocean exploitation. In this paper, we propose a multiagent approach for optimizing the quality of monitoring (QoM) in underwater sensor networks. Our framework aspires to optimize the QoM by resorting to the machine learning concept of diversity. We devise a multiagent optimization procedure which is able to both reduce the redundancy among the sensor readings and maximize the diversity in a distributed and adaptive manner. The mobile sensor positions are adjusted iteratively using a gradient type of updates. The overall framework is tested through simulations based on realistic environment conditions. The proposed approach is compared to other placement approaches and is found to achieve a higher QoM with a smaller number of sensors.

Joint Data Transmission and Energy Harvesting for MISO Downlink Transmission Coordination in Wireless IoT Networks.

The advent of simultaneous wireless information and power (SWIPT) has been regarded as a promising technique to provide power supplies for an energy sustainable Internet of Things (IoT), which is of paramount importance due to the proliferation of high data communication demands of low-power network devices. In such networks, a multi-antenna base station (BS) in each cell can be utilized to concurrently transmit messages and energies to its intended IoT user equipment (IoT-UE) with a single antenna under a common broadcast frequency band, resulting in a multi-cell multi-input single-output (MISO) interference channel (IC). In this work, we aim to find the trade-off between the spectrum efficiency (SE) and energy harvesting (EH) in SWIPT-enabled networks with MISO ICs. For this, we derive a multi-objective optimization (MOO) formulation to obtain the optimal beamforming pattern (BP) and power splitting ratio (PR), and we propose a fractional programming (FP) model to find the solution. To tackle the nonconvexity of FP, an evolutionary algorithm (EA)-aided quadratic transform technique is proposed, which recasts the nonconvex problem as a sequence of convex problems to be solved iteratively. To further reduce the communication overhead and computational complexity, a distributed multi-agent learning-based approach is proposed that requires only partial observations of the channel state information (CSI). In this approach, each BS is equipped with a double deep Q network (DDQN) to determine the BP and PR for its UE with lower computational complexity based on the observations through a limited information exchange process. Finally, with the simulation experiments, we verify the trade-off between SE and EH, and we demonstrate that, apart from the FP algorithm introduced to provide superior solutions, the proposed DDQN algorithm also shows its performance gain in terms of utility to be up to 1.23-, 1.87-, and 3.45-times larger than the Advantage Actor Critic (A2C), greedy, and random algorithms, respectively, in comparison in the simulated environment.

Optimizing energy efficiency of LoRaWAN-based wireless underground sensor networks: A multi-agent reinforcement learning approach

Extended battery lifetime is always desirable for wireless underground sensor networks (WUSNs). The recent integration of LoRaWAN-grade massive machine-type communications (MTC) technology and WUSNs, namely LoRaWAN-based WUSNs, provides a promisingly energy-efficient solution for underground monitoring. However, due to the limited battery energy, massive data collisions and dynamic underground environment, the energy efficiency of LoRaWAN-based WUSNs still possesses a significant challenge. To propose a liable solution, we advocate reinforcement learning (RL) for managing the transmission configuration of underground sensors. In this paper, we firstly develop the multi-agent RL (MARL) algorithm to improve the network energy efficiency, which considers the link quality, energy consumption and packet collisions between packets. Secondly, a reward mechanism is proposed, which is used to define independent state and action for every node to improve the adaptability of the proposed algorithm to dynamic underground environment. Furthermore, through the simulations in different underground environment and at different network scales, our results highlight that the proposed MARL algorithm can quickly optimize the network energy efficiency and far exceed the traditional adaptive data rate (ADR) mechanism. Finally, our proposed algorithm is successfully demonstrated to be able to efficiently adapt to the dynamically changing underground environment. This work provides insights into the energy efficiency optimization, and will lay the foundation for future realistic deployments of LoRaWAN-based WUSNs.

Decentralized multi-agent approach based on A* algorithm for on-demand transport problem

The on-demand transport (ODT) systems have developed worldwide as they have significant social, environmental, and economic benefits. Even with those benefits, it’s still important to gain popular acceptance. The acceptance key is the reactivity of the system in providing fast and reliable solutions whilst respecting vehicles’ and clients’ constraints. This paper presents a decentralized multi-agent approach to model and solve the ODT problem in a static road network. The agents interact with each other using the A* algorithm to find an optimal solution for each transport demand. The optimal solution is expressed by the fastest trajectory taken by the cheapest vehicles. We utilize factual data from a Lebanese city to do experiments evaluating the proposed approach.

Hierarchical Deep Reinforcement Learning With Experience Sharing for Metaverse in Education

Metaverse has gained increasing interest in education, with much of literature focusing on its great potential to enhance both individual and social aspects of learning. However, little work has been done to address the systems and technologies behind providing meaningful Metaverse learning. This article proposes a technical framework to address this research gap, where a hierarchical multiagent reinforcement learning approach with experience sharing is developed to augment the intelligence of nonplayer characters in Metaverse learning for personalization. The utility and benefits of the proposed framework and methodologies are demonstrated in Gridlock, a Metaverse learning game, as well as through extensive simulations.

Fleet Rebalancing for Expanding Shared e-Mobility Systems: A Multi-Agent Deep Reinforcement Learning Approach

The electrification of shared mobility has become popular across the globe. Many cities have their new shared e-mobility systems deployed, with continuously expanding coverage from central areas to the city edges. A key challenge in the operation of these systems is fleet rebalancing, i.e., how EVs should be repositioned to better satisfy future demand. This is particularly challenging in the context of expanding systems, because i) the range of the EVs is limited while charging time is typically long, which constrain the viable rebalancing operations; and ii) the EV stations in the system are dynamically changing, i.e., the legitimate targets for rebalancing operations can vary over time. We tackle these challenges by first investigating rich sets of data collected from a real-world shared e-mobility system for one year, analyzing the operation model, usage patterns and expansion dynamics of this new mobility mode. With the learned knowledge we design a high-fidelity simulator, which is able to abstract key operation details of EV sharing at fine granularity. Then we model the rebalancing task for shared e-mobility systems under continuous expansion as a Multi-Agent Reinforcement Learning (MARL) problem, which directly takes the range and charging properties of the EVs into account. We further propose a novel policy optimization approach with action cascading, which is able to cope with the expansion dynamics and solve the formulated MARL. We evaluate the proposed approach extensively, and experimental results show that our approach outperforms the state-of-the-art, offering significant performance gain in both satisfied demand and net revenue.

Heterogeneous Crowd Simulation using Parametric Reinforcement Learning.

Agent-based synthetic crowd simulation affords the cost-effective large-scale simulation and animation of interacting digital humans. Model-based approaches have successfully generated a plethora of simulators with a variety of foundations. However, prior approaches have been based on statically defined models predicated on simplifying assumptions, limited video-based datasets, or homogeneous policies. Recent works have applied reinforcement learning to learn policies for navigation. However, these approaches may learn static homogeneous rules, are typically limited in their generalization to trained scenarios, and limited in their usability in synthetic crowd domains. In this paper, we present a multi-agent reinforcement learning-based approach that learns a parametric predictive collision avoidance and steering policy. We show that training over a parameter space produces a flexible model across crowd configurations. That is, our goal-conditioned approach learns a parametric policy that affords heterogeneous synthetic crowds. We propose a model-free approach without centralization of internal agent information, control signals, or agent communication. The model is extensively evaluated. The results show policy generalization across unseen scenarios, agent parameters, and out-of-distribution parameterizations. The learned model has comparable computational performance to traditional methods. Qualitatively the model produces both expected (laminar flow, shuffling, bottleneck) and unexpected (side-stepping) emergent qualitative behaviours, and quantitatively the approach is performant across measures of movement quality.

A Delay-Optimal Task Scheduling Strategy for Vehicle Edge Computing Based on the Multi-Agent Deep Reinforcement Learning Approach

Cloudlet-based vehicular networks are a promising paradigm to enhance computation services through a distributed computation method, where the vehicle edge computing (VEC) cloudlet are deployed in the vicinity of the vehicle. In order to further improve the computing efficiency and reduce the task processing delay, we present a parallel task scheduling strategy based on the multi-agent deep reinforcement learning (DRL) approach for delay-optimal VEC in vehicular networks, where multiple computation tasks select the target threads in a VEC server to execute the computing tasks. We model the target thread decision of computation tasks as a multi-agent reinforcement learning problem, which is further solved by using a task scheduling algorithm based on multi-agent DRL that is implemented in a distributed manner. The computation tasks, with each selection acting on the target thread acting as an agent, collectively interact with the VEC environment and receive observations with respect to a common reward and learn to reduce the task processing delay by updating the multi-agent deep Q network (MADQN) using the obtained experiences. The experimental results show that the proposed DRL-based scheduling algorithm can achieve significant performance improvement, reducing the task processing delay by 40% and increasing the processing probability of success for computation tasks by more than 30% compared with the traditional task scheduling algorithms.

Novel Data-Driven decentralized coordination model for electric vehicle aggregator and energy hub entities in multi-energy system using an improved multi-agent DRL approach

Energy hub (EH) is an independent entity that benefits to the efficiency, flexibility, and reliability of integrated energy systems (IESs). On the other hand, the rapid emerging of electric vehicles (EVs) drives the EV aggregator (EVAGG) as another independent entity to facilitate the electricity exchange with the grid. However, due to privacy consideration for different owners, it is challenging to investigate the optimal coordinated strategies for such interconnected entities only by exchanging the information of electrical energy. Besides, the existence of parameter uncertainties (load demands, EVs’ charging behaviors, wind power and photovoltaic generation), continuous decision space, dynamic energy flows, and non-convex multi-objective function is difficult to solve. To this end, this paper proposes a novel model-free multi-agent deep reinforcement learning (MADRL) -based decentralized coordination model to minimize the energy costs of EH entities and maximize profits of EVAGGs. First, a long short-term memory (LSTM) module is used to capture the future trend of uncertainties. Then, the coordination problem is formulated as Markov games and solved by the attention enabled MADRL algorithm, where the EH or EVAGG entity is modeled as an adaptive agent. An attention mechanism makes each agent only focus on state information related to the reward. The proposed MADRL adopts the forms of offline centralized training to learn the optimal coordinated control strategy, and decentralized execution to enable agents’ online decisions to only require local measurements. A safety network is employed to cope with equality constraints (demand–supply balance). Simulation results illustrate that the proposed method achieves similar results compared to the traditional model-based method with perfect knowledge of system models, and the computation performance is at least two orders of magnitudes shorter than the traditional method. The testing results of the proposed method are better than those of the Concurrent and other MADRL method, with 10.79%/3.06% lower energy cost and 17.11%/6.82% higher profits of aggregator. Besides, the electric equality constraint of the proposed method is only 0.25 MW averaged per day, which is a small and acceptable violation.