Multi-agent Technique Research Articles

Multi-Agent Deep Reinforcement Learning for Cooperative Computing Offloading and Route Optimization in Multi Cloud-Edge Networks

Edge computing is a new paradigm to provide computing capability at the edge servers close to end devices. A significant research challenge in edge computing is finding efficient task offloading to edge and cloud servers considering various task characteristics and limited network and server resources. Several reinforcement learning (RL)-based task-offloading methods have been developed, because RL can immediately output efficient offloading by pre-learning. However, these methods do not take into account clouds or focus only on a single cloud. They also do not take into account the bandwidth and topology of the backbone network. Such shortcomings strongly limit the range of applicable networks and degrade task-offloading performance. Therefore, we formulate a task-offloading problem for multi-cloud and multi-edge networks considering network topology and bandwidth constraints. We also propose a task-offloading method that is based on cooperative multi-agent deep RL (Coop-MADRL). This method introduces a cooperative multi-agent technique through centralized training and decentralized execution, improving task-offloading efficiency. Simulations revealed that the proposed method can minimize network utilization and task latency while minimizing constraint violations in less than one millisecond in various network topologies. It also shows that cooperative learning improves the efficiency of task offloading. We demonstrated that the proposed method has generalization performance for various task types by pre-training with many resource-consuming tasks.

Research on path planning techniques based on deep reinforcement learning

Path planning problem stands for a sort of problem that find a secure path from the departure position to the destination in the unique application area without conflicts with other objects under the premise of minimum time or distance cost. Path planning has many applications in real life. This paper summarizes and classifies the current main research results on path planning, which are classified according to conventional techniques, single-agent path planning techniques and multi-agent path planning techniques. Conventional techniques are classified into three different algorithms: Traditional techniques, Graphics techniques and Intelligent bionics techniques. For the single-agent path planning techniques, it is primarily broken down into two categories: value-based and strategy-based. For multi-agent techniques, according to the improvement techniques, it is split in three categories: expert demonstration type, improved communication type and task decomposition type. Based on above classification, the mechanisms, advantages, and disadvantages of the existing algorithms are summarized and compared, and the future work in this field is prospected.

Simulation-Based Schedule Optimization for Virtual Coupling-Enabled Rail Transit Services with Multiagent Technique

Virtual coupling (VC) is a train-centric next generation signalling system, which can enable multiple trains to operate in a formation just like one train or decouple separately on-the-run or at station flexibly or as planned. With the aim of optimizing the interdeparture train headway time, providing the variable capacity for diverse passenger demand, maximizing the passenger riding comfort degree, and minimizing passenger travel cost and train operation cost, the dynamic schedule for VC-enabled rail transit services is investigated with the multiagent simulation technique on NetLogo platform. Our contribution is mainly fourfold. First, VC-enabled rail transit entity for simulation is represented with the multiagent technique, including representation of train unit, train convoy, passenger attributes and behavior, and mathematical formula for calculation of the train operation cost and passenger travel cost, as well as passengers riding comfort degree are proposed. Second, the operational principles for flexible and self-organisingVC-enabled trains are defined. Third, the VC-enabled train-centric, passenger demand-driven, and agent-based simulation flow and algorithms are developed innovatively, which adopt the ergodic strategy for simulation by traversing each O-D pair demand along each route section over the rail transit network. Finally, we test and discuss the proposed methodology on the designed computational experiment on the NetLogo platform, and the simulation results series validated the effectiveness of the proposed methodology. The provided research can effectively support the VC-enabled platoon operation-oriented train service schedule for future study.

Resilient Distributed Constraint Reasoning to Autonomously Configure and Adapt IoT Environments

In this article, we investigate multi-agent techniques to install autonomy and adaptation in IoT-based smart environment settings, like smart home scenarios. We particularly make use of the smart environment configuration problem (SECP) framework, and map it to a distributed optimization problem (DCOP). This consists in enabling smart objects to coordinate and self-configure as to meet both user-defined requirements and energy efficiency, by operating a distributed constraint reasoning process over a computation graph. As to cope with the dynamics of the environment and infrastructure (e.g., by adding or removing devices), we also specify the k -resilient distribution of graph-structured computations supporting agent decisions, over dynamic and physical multi-agent systems. We implement a self-organizing distributed repair method, based on a distributed constraint optimization algorithm to adapt the distribution as to ensure the system still performs collective decisions and remains resilient to upcoming changes. We provide a full stack of mechanisms to install resilience in operating stateless DCOP solution methods, which results in a robust approach using a fast DCOP algorithm to repair any stateless DCOP solution methods at runtime. We experimentally evaluate the performances of these techniques when operating stateless DCOP algorithms to solve SECP instances.

Resource Allocation With Multiagent Trading Over the Edge Services

A number of studies have recently emerged to address the issue of resource allocation in edge computing environments. However, there are few works considering how to optimize resource allocation while satisfying market's requirements in multiagent technique for distributed allocation of Edge resources in distributed control. This study use trading-based multiagent resource allocation model as an allocation mechanism to optimal allocate resources through genetic algorithm in an Edge computing environment. The proposed model supports the optimal process between Edge computing cases to apply and allows Edge buyers and Edge providers both to derive their own pricing strategies and to analyze the respective impact to their welfare. The k-pricing schemes are adjustly to meet the Edge users/providers requirement and constraints set by composed services.

BCAS: A Blockchain Model for Collision Avoidance to Prevent Overtaking Accidents on Roads

Overtaking at high speeds, especially on non-divided roadways, is a leading cause of traffic accidents. During overtaking maneuvers, humans are more likely to make mistakes due to factors that cannot be predicted. For overtaking operations in autonomous vehicles, prior research focused on image processing and distant sensing of the driving environment, which didn't consider the speed of the surrounding traffic, the size of the approaching vehicles, or the fact that they could not see beyond impediments in the road. The past researches didn't focus on the speed of the surrounding traffic or the size of the approaching vehicles. Moreover, most of the techniques were based on single agent systems where one agent manages the source vehicle's (autonomous) mobility within its surroundings. This research conducts a feasibility study on a remote Vehicle-to-Vehicle (V2V) communication framework based on Dedicated Short-Range Communication (DSRC) to improve overtaking safety. This work also tries to improve safety by introducing a blockchain-based safety model called BCAS (Blockchain-based Collision Avoidance System). The proposed multi-agent technique strengthens the ability of real-time, high-speed vehicles to make decisions by allocating the total computation of processing responsibilities to each agent. From the experimental results, it is concluded that the proposed approach performs better than existing techniques and efficiently covers the limitations of existing studies.

Cooperative Multi-Agent Deep Reinforcement Learning for Dynamic Virtual Network Allocation With Traffic Fluctuations

Network traffic and computing demand have been changing dramatically due to the growth of various types of network services, e.g., high-quality video delivery and operating system (OS) updates. To maximize the utilization efficiency of limited network resources, network resource control technology is required for smooth and quick operation when network demands change. Therefore, we propose a dynamic virtual network (VN) allocation method based on cooperative multi-agent deep reinforcement learning (Coop-MADRL). This method can quickly optimize network resources even while network demands are drastically changing by learning the relationship between network demand patterns and optimal allocation by using deep reinforcement learning (DRL) in advance. The key idea is to use a multi-agent technique for a reinforcement learning (RL) based dynamic VN allocation method, which can reduce the number of candidate actions per agent and can improve the performance for VN allocation. Moreover, a cooperation technique improves the efficiency of VN allocation. From results of a simulation evaluation, Coop-MADRL can calculate effective allocation within 1 s, which reduces the maximum server and link utilization and drastically reduces the constraint violations compared with that of the static VN allocation method. Furthermore, we revealed that the learning with various mixed traffic models could achieve a high generalization performance for all traffic patterns.

Multiagent Deep Reinforcement Learning for Cost- and Delay-Sensitive Virtual Network Function Placement and Routing

This paper proposes an effective and novel multi-agent deep reinforcement learning (MADRL)-based method for solving the joint virtual network function (VNF) placement and routing (P&R), where multiple service requests with differentiated demands are delivered at the same time. The differentiated demands of the service requests are reflected by their delay- and cost-sensitive factors. We first construct a VNF P&R problem to jointly minimize a weighted sum of service delay and resource consumption cost, which is NP-complete. Then, the joint VNF P&R problem is decoupled into two iterative subtasks: placement subtask and routing subtask. Each subtask consists of multiple concurrent parallel sequential decision processes. By invoking the deep deterministic policy gradient method and multi-agent technique, an MADRL-P&R framework is designed to perform the two subtasks. The new <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">joint reward and internal rewards</i> mechanism is proposed to match the goals and constraints of the placement and routing subtasks. We also propose the parameter migration-based model-retraining method to deal with changing network topologies. Corroborated by experiments, the proposed MADRL-P&R framework is superior to its alternatives in terms of service cost and delay, and offers higher flexibility for personalized service demands. The parameter migration-based model-retraining method can efficiently accelerate convergence under moderate network topology changes.

RNA World Modeling: A Comparison of Two Complementary Approaches.

Simple SummaryDespite years of dedicated research, scientists are still not sure what the first ”living” cell would have looked like. One of the most well-known hypotheses is the RNA world hypothesis, which assumes that, in the beginning, life relied on RNA molecules instead of DNA as information carriers and primitive enzymes. The population of such RNAs is made up of self-replicating molecules (replicases) that could make copies of themselves and parasite molecules that could only be copied by replicases. In this study, we further investigated the interplay between these hypothetical prebiotic RNA species, since it plays a crucial role in generating diversity and complexity in prebiotic molecular evolution. We compared two approaches that are commonly used to investigate such simple prebiotic systems, representing different modeling and observation scales—namely, microscopic and macroscopic. In both cases, we were able to obtain consistent results.The origin of life remains one of the major scientific questions in modern biology. Among many hypotheses aiming to explain how life on Earth started, RNA world is probably the most extensively studied. It assumes that, in the very beginning, RNA molecules served as both enzymes and as genetic information carriers. However, even if this is true, there are many questions that still need to be answered—for example, whether the population of such molecules could achieve stability and retain genetic information for many generations, which is necessary in order for evolution to start. In this paper, we try to answer this question based on the parasite–replicase model (RP model), which divides RNA molecules into enzymes (RNA replicases) capable of catalyzing replication and parasites that do not possess replicase activity but can be replicated by RNA replicases. We describe the aforementioned system using partial differential equations and, based on the analysis of the simulation, surmise general rules governing its evolution. We also compare this approach with one where the RP system is modeled and implemented using a multi-agent modeling technique. We show that approaching the description and analysis of the RP system from different perspectives (microscopic represented by MAS and macroscopic depicted by PDE) provides consistent results. Therefore, applying MAS does not lead to erroneous results and allows us to study more complex situations where many cases are concerned, which would not be possible through the PDE model.

Ontology concept semantic similarity matching based on Ant Colony Optimization algorithm

Semantic closeness between ontology concepts is of great importance in various applications like information retrieval, text summarization, sentiment analysis, etc. These applications relying on domain ontologies required a computable measure of relationships among ontology concepts. A variety of techniques like semantic distance, information content, characteristics or features of terms, etc., are available to measure semantic similarity among concepts. In this research work, a method based on distance is proposed using Ant Colony Optimization to computing the shortest path, which will use layers to calculate semantic similarity. ACO is a probabilistic multi-agent technique based on the behavior of real ants. Evaluation and a comparison of proposed methods against related processes in a common framework deploy higher semantic similarity among concepts.

Pragmatic Distance in IoT Devices

Stand-alone devices perform their functions without human intervention. These types of devices would benefit from capabilities that allow collaboration among the devices in order to maximize service availability, as well as dynamic adaptation of the devices to user requirements in order to minimize resource usage. For example, when detecting a malfunction in one of the devices, the system should find devices with similar functionality to recover the overall service by functionality substitution. Additionally, for example, in the context of a user with hearing loss, the system should utilize devices with visual feedback functionality and disable audio feedback functionality to conserve resources. In this paper, we describe the implementation of a method to detect similar functionalities between heterogeneous devices, denoted Lightweight Machine to Machine Resource Pragmatic Distance (LwRPD). We define that devices are pragmatically close if they can be substituted with each other in order to perform a specific function in a particular context. We implement the method for enabling the aforementioned capabilities in specific use cases, using resources from real-life devices, to demonstrate the benefits. We also compare LwRPD with a multi-agent pragmatic similarity metric, Fuzzy logic and Levenshtein distance techniques, demonstrating LwRPD superiority over those techniques.

The Charging Characteristics of Large-Scale Electric Vehicle Group Considering Characteristics of Traffic Network

Electric vehicles, as a new generation of road transport, are primarily used for transportation. For this reason, the topological characteristics of traffic network may make great influence on the macroscopic characteristics of electric vehicle group. However, little attention is drawn to the study in this field. Therefore, typical approach to studying the impact of traffic network topological characteristics on the charging characteristics of large-scale electric vehicle group is presented by adopting the charging power distribution as analysis indicator. In this paper, a model of complex adaptive system is constructed including the electric vehicle group, traffic network and charging stations, where the tempo-spatial distribution of charging power can be obtained via the simulation with the multi-agent technique. The charging power of regional electric vehicles is found obedient to logarithmic normal distribution after analysis, while the mathematical expectation of probability density shows obvious cyclicity. Finally, the traffic networks of various cities in comparison testify to that improving the connectivity of traffic network and increasing the average clustering coefficient can effectively reduce the effect on power system brought by the charging load of large-scale electric vehicle group.

Strategic Information Sharing in a Dynamic Supply Chain with a Carrier under Complex Uncertainty

Whether to use an information sharing mechanism is investigated in a dynamic supply chain, where one manufacturer, one carrier, and one retailer are faced with uncertain yield, demand, and lead time during multiple periods. Each member is modeled as an adaptive agent based on multiagent technique, and their decisions can be adjusted timely to adapt to external environment. There are two choices for the whole supply chain to deal with uncertain risks: information sharing (IS) or no information sharing (NS). Under strategy IS, the information about market demand and the retailer’s inventory can be shared within the supply chain. For each strategy, the effects of yield, demand, and lead time uncertainties on costs of the supply chain and channel members are studied. It is found that (i) it is rewarding for the upstream manufacturer to use a retailer’s shared information under uncertain yield or demand; (ii) however, information sharing (IS) strategy sometimes should be abandoned for other members and the whole supply chain; (iii) counterintuitively, the increase of transportation time uncertainty benefits the retailer.

Method of agents’ state estimation in multiresolution multiagent simulation

The paper proposes the multiagent techniques for approximation of agent’s state in the multiresolution multiagent simulation. The key methods we have used for state aggregation and disaggregation are: consensus algorithm and formation control. The idea of the coordination of multiple agents has emerged from both observation and simulation of a collective behavior of biological entities. The consensus algorithms are commonly used for the cooperative control problems in the multiagent systems, whilst the formation control is the most popular and fundamental motion coordination problem in the multiagent systems, where agents converge to predefined geometric shapes. The presented approach shows that multiagent methods seem to be very promising in multiresolution simulation. Consensus and formation control algorithms remove necessity to specify the much more complex algorithms for the aggregation and disaggregation needs.

A state space distribution approach based on system behaviour

In this paper, we propose a novel approach to deal with the state space explosion problem occurring in model checking. We propose an offline algorithm for distributed state space construction. That is carried out by reviewing the behaviour of the constructed system and redistributing the state space according to the accumulated information about the optimal considered behaviour. Therefore, the distribution will be guided by the system's behaviour. The proposed policy maintains the spatial-time balance. The simulation and implementation of our system are based on a multi-agent technique which fits very well the development of distributed systems. The experimental measures performed on a cluster of machines have shown very promising results for both workload balance and communication overhead.

A brief review on computer system control using multi-agent technique

Multi-agent systems consist of several agents that exist at the same time, share common resources and communicate with each other. They are self-controlled and operate in a network environment. Mostly, they are used by control engineers to control specific communication and computing channels using a number of strategies. At large scale, multi-agent systems are being used in smart grids and sensor-controlling applications. Since the nature of multi-agent systems are autonomous, they require a distributed operation environment. Due to this feature of multi-agents, they are different from all other systems, because when network size increases, the control objectives can be achieved accordingly without any complexity of sensing and communication controlling. For this, a communication network is essential among all neighbouring agents. This paper reviews previous studies on controlling of multi-agents in a different framework. It will help us to remove gaps between previous research and our latest understanding of multi-agent systems technology.

A brief review on computer system control using multi-agent technique

A brief review on computer system control using multi-agent technique

Agent-based Control System: A Review and Platform for Reconfigurable Bending Press Machine

Abstract Agent paradigm is a technology recognized as potent tool for the 21st century manufacturing system. Researches are ongoing in the applications of agent technology to manufacturing enterprise, production process planning and scheduling, shop floor control, and to reconfigurable manufacturing system as an implementation case study. The article presents the results of a review of published articles on agents’ usage at manufacturing enterprise level for reconfigurable manufacturing system between year 2008 and 2018 to provide up to date pictures of the technological function of agents and multi agents within the reconfigurable conceptualized manufacturing circles. The paper is view from the angle of wide coverage, broad scope of multi-agent techniques and a focus on real world applications. The outcomes include behavioral pattern and trends of agents and multi-agent’s usage in manufacturing enterprise resource planning and supply chain managements, the gaps yet to be fill for consolidating future on industry 4.0 in reconfigurable manufacturing system, and development of mobile apps interacting platform for IoT real time database communication and coordination for RBPM controller agents.

Scalable multi-agent microgrid negotiations for a transactive energy market

Distributed energy resources are becoming increasingly common and forcing change in conventional energy markets with growing attention given to transactive energy networks that allow power trading between neighboring microgrids or distributed energy resources customers to supplement transactions with an electric utility. This study develops and evaluates a generalizable method for managing energy trading between microgrids in a grid-connected network through multi-agent techniques. The approach is demonstrated for a 3-node network and a 9-node network for a simulated year with hourly load and solar data for each unique microgrid agent. Results are compared against baseline networks without trading enabled to quantify a 3.6% and 5.4% reduction in the levelized cost of energy, respectively, with trading enabled for the 3-node and 9-node cases. Local energy storage capacities are varied to examine impact on the levelized cost of energy and trading behaviors. Results indicate that trading between microgrids reduces the levelized cost of energy for each individual node and the whole network, and that certain trends emerge between agents that allow some microgrids to operate at a lower cost than others.

An Affordable Set of Control System Laboratories Using A Low-Cost Robotic Platform

The development of an inexpensive robotic platform is presented. Our aim is to provide a complete low-cost hardware tool to illustrate control system methods. The capabilities of this platform allow both, researchers and students, to implement from basic techniques to complex ones (e.g., from model-based control techniques to multiagent dynamical systems). The hardware that has been designed and developed performs experiments for single or multiple robot control employing capabilities, such as locomotion, communication, perception, and autonomous decision making. We illustrate different control and system dynamics techniques using this platform. After presenting the model and navigation strategies, single-agent techniques that ensure safe and predictable navigation are shown (PID and fuzzy logic controllers). Then, two multiagent techniques are described in order to prove the versatility of the platform. The first strategy illustrates common problems that arise in nature and power systems (phase synchronization models based on pulse-coupled biological oscillators and the Kuramoto oscillator). Second, a game theoretical approach for multirobot consensus is shown. In all these cases, experimental results show the performance of the platform.