Hierarchical Multi-agent System Research Articles

Architecture for Multiagent Systems with UNS Middleware and mediator agent using object-centric process mining

The increasing complexity of industrial processes demands innovative solutions for the integration and management of multi-agent systems. This work proposes a hierarchical multi-agent system architecture to unify these systems, aiming for efficiency and autonomy in complex and connected production environments. The proposal focuses on the integration of multiple tasks and layers, promoting operational freedom without compromising security and agility. To optimize the use and understanding of data flows, the system uses Object-Centric Process Mining techniques at runtime, an innovative approach for processes with autonomous agents, given its ability to manage event- and object-oriented data. The architecture adopts the OCEL 2.0 log standard, which provides a structured and interoperable basis for the administration of large volumes of production data, facilitating detailed analyses and quick, accurate decisions. The proposed structure is aligned with the requirements of Industry 4.0, especially the RAMI 4.0 model, and relies on the mediation of a unified system core (UNS), which integrates subsystems such as SCADA, intelligent devices, and Process Mediators. Performance tests have demonstrated that the architecture is robust, scalable, and agile, with the OCPM Mediator accelerating secure responses to agents and promoting a resilient, autonomous, and data-driven decision-making process. This work highlights that the proposed architecture offers an effective solution for multi-agent systems in advanced manufacturing, enabling the industry to address the challenges of digital transformation.

Heuristic Guided Hierarchical Reinforcement Learning Approach For The Economic Improvement Of Production Lines

The increasing variety of products poses a challenge for efficient manufacturing on production lines due to resulting small batch sizes and frequent product changes, which lower the average overall effectiveness. Especially for industries such as the Fast Moving Consumer Goods (FMCG) industry that manufacture at high speed on production lines, it is mandatory to increase the performance of production lines in an economical way. Due to the complexity of such production lines, identifying efficient actions and combining them into economic improvement trajectories is challenging. There are numerous approaches based on different concepts, such as simulation-based heuristics, to solve these challenges. In other application areas, reinforcement learning has shown remarkable success in recent years, and first reinforcement learning-based approaches to this specific problem can be found. However, these approaches mostly focus on details instead of providing a holistic view of the possibilities to improve a production line or are limited in their practical application due to lack of integration of existing expert knowledge or limited quality of results. For this reason, this paper proposes a hierarchical reinforcement learning approach that combines discrete event simulation with a heuristically driven multi-agent system. Thus, the selection of the improvement strategy of the production line is performed by one agent, and the dedicated improvement of specific parameters is performed by specialized subordinate other agents. Through this hierarchical multi-agent system, on the one hand, the learning rate can be increased. On the other hand, by guiding the agents through a heuristic based on expert knowledge, the learning quality is increased.

A Cooperative Hierarchical Multi-Agent System for EV Charging Scheduling in Presence of Multiple Charging Stations

The increasing penetration of plug-in electric vehicles (EVs) to the electrical grid raises concerns over secure and economic operation of the system. A coordination mechanism between system operator and EV aggregators is necessary to ensure that the system is operated within the security limits, and to reduce the charging costs while satisfying EV users’ energy needs. In this work, we present a cooperative hierarchical multi-agent system and propose an EV charging scheduling strategy in order to minimize the demand and energy charges while meeting the EV users’ energy requirements and satisfying the system security constraints. Within the designed framework, the higher-level agents calculate a set of proposed control signals by solving the designated optimization problems, and send them to the lower-level agents to facilitate an optimal scheduling in line with the aforementioned objectives. Through this hierarchically distributed approach, it is possible to effectively coordinate multiple EV charging stations without the need of direct communication or any prior information related to EV arrivals. The computational complexity of the problem is reduced by distributing the work among agents, and the privacy of sensitive data, such as system topology, load profiles, and EV parameters, is preserved. Moreover, unlike the traditional distributed solution methods that converge iteratively, the proposed approach calculates the optimal charging schedule after a single round of communication. The efficacy of the proposed methodology is demonstrated by a series of case studies on 33-bus and 118-bus distribution test feeders.

Integrated multi-project planning and scheduling - a multiagent approach

Today’s project organisations are conducting their daily activities for multi-projects where a group of scarce resources are shared amongst several parallel projects. In addition, due to the implementation of lean and agile production, they have consumed more and more resources from external environment (i.e., non-regular resources) when performing projects. Therefore, how to effectively schedule the shared resources across projects and meanwhile optimally plan the required capacities of non-regular resources for an entire multi-project becomes a critical problem for project organisations. This paper proposes a multi-agent approach to address the resource planning and scheduling decisions of the problem in an integrated manner. In this work, a multi-project is represented as a hierarchical multi-agent system consisting of a multi-project portfolio agent (MPA), a resource agent (RA) and a set of project agents (PAs). Decisions are made by these agents through a hierarchical coordination mechanism. Basically, the decision of resource planning for a multi-project is obtained from the resource requirements proposed by projects in their scheduling decisions which are coordinated by MPA through controlling the allocation of shared resources to projects. The scheduling decision of each project is made by its PA using the model of multi-mode resource constrained project scheduling problem. A genetic algorithm and an ant colony optimisation algorithm are proposed for decision optimisations of resource planning and project scheduling, respectively. A numerical study demonstrated that the proposed approach can effectively solve the multi-project problem by producing near-optimal solutions.

МОДЕЛІ ІЄРАРХІЧНОЇ БАГАТОАГЕНТНОЇ СИСТЕМИ ДЛЯ ВИКОНАННЯ БІЗНЕС-ПРОЦЕСІВ

Purpose. Developing a set of models which formally describe the operation environment of the organizational and technical system and the interaction of software agents of different roles in performing business operations. The pres-ence of such a set of models allows you to create information technology for monitoring and routing business processes and reduce costs for business operations. Methodology. The research methods are based on systems analysis methods. Findings. A set of models of hierarchical multi-agent system for business processes has been developed. The complex comprises a static description of the operation environment of the organizational and technical system, where there are models of business process, business operation and its components. Originality. Organization of total monitoring of the operation environment of the organizational and technical system, i.e. the current state of the processes of processing applications, queues and workstations requires the development of static and dynamic business process models. Based on the static description, a model of the dynamics of business processes promoting throughout many workstations has been developed. This makes possible to monitor the status of workstations, queues and applications for business opera-tions. In particular, a formal description of the software agent, its competencies and a model of interaction of three-level agents which perform the functions of business operation executors, monitors and dispatchers has been developed. Having such tools, the second important aspect is the development of a universal hierarchical structure of the multi-agent system, in which different agents perform the roles of performers, monitors and dispatchers. Such a structure should include the regulation of the agent functions, models of agents interaction at all three levels, ways of agents-people communication. Practical value. Tests of the monitoring and scheduling system in different conditions (electronic document management, manufacturing company, human resources management department) showed a decrease in time of business operations, losses on waiting and increase in rhythm of business processes. The results of the work has enabled the development of a queue management strategy, which has showed a reduction in time of operations and a more balanced workload.

A hierarchical multi-agent architecture based on virtual identities to explain black-box personalization policies

Hyper-personalization policies entail a considerable improvement regarding previous personalization approaches. However, they present several issues that need to be addressed, such as minimal explainability and privacy invasion. A hierarchical Multi-Agent System (MAS) is presented in this work to provide a solution to these concerns. The system is formulated as a hybrid approach, where some of the agents work autonomously, while the user input triggers the remaining. At the autonomous level, a set of Virtual Identities (VIs) representing different user profiles interact with Black-Box Hyper-Personalization Online Systems (BBHOS), gathering a set of targeted responses. Associative patterns and profile aggregations can then be inferred from the analysis of these responses. In the user-triggered level, the real user is virtualized as an identity that represents their features. The virtual identity serves as an intermediary between the personalization system and the real user. This virtualization hinders the personalization service from extracting sensitive contextual information about the real user, protecting their privacy. The results obtained by the user identity on its interaction with the personalization service are then analyzed, adjusting the content of the response to fit the user’s requests instead of their features. A use case on the functioning of the analysis of search engines is presented to illustrate the complete behavior of the proposed architecture.

Integrated energy management system for Microgrids of building prosumers

In this paper, an easy to apply and computationally efficient energy management system (EMS) for microgrids of building prosumers is proposed. It is based on a hierarchical multi-agent system (MAS), aiming to minimize the operation cost of a microgrid comprising a group of buildings, renewable energy sources (RES), energy storage systems (ESS) and plug-in electric vehicles (PEVs). The optimization problem is suitably set up to ensure low computation requirements and it is solved with particle swarm optimization (PSO). Optimal operation scheduling for grid-connected and autonomous operation of microgrids of building prosumers, considering their thermal and electrical systems, integrated RES, conventional and dynamic ESS is proposed. The models used are not computationally demanding, allowing the application to complex systems and they are not being limited to one building scale which is the case for the majority of relative research. Another innovative feature of the proposed algorithm, among others, is that it ensures autonomous operation at certain time periods without the use of any emergency generator sets. Detailed simulation results of different operation scenarios, showed that cost savings in range of 11% can be achieved while a large number of operation and technical constraints are satisfactorily met in grid-connected and autonomous operation.

Improving data collection and processing efficiency by using hierarchies of intelligent agents in multi-agent systems

The paper considers the challenge of improving data collection and processing efficiency by using various information and measurement systems. A structural scheme of a hierarchical multi-agent control system has been developed, incorporating horizontal (single-level) and vertical (multi-level) connections between intelligent agents. An algorithm is proposed for controlling a heterogeneous group of unmanned aerial vehicles participating in search and rescue operations in an emergency zone based on the hierarchical interaction of intelligent agents, cloning and coalition formation. KPI values have been calculated and the effectiveness of the proposed solution have been proved.

Hierarchical traffic signal optimization using reinforcement learning and traffic prediction with long-short term memory

Multi-agent systems can be used for modelling large-scale distributed systems in real world applications. In intelligent transportation system (ITS), many interacting entities influence the performance of the system. As part of ITS, traffic signal control can be modelled using a multi-agent system. In this paper, a hierarchical multi-agent system including two levels is employed to control traffic signals. Each traffic signal is controlled by an agent that sits in the physical level, i.e., in the first level. For the other levels, the traffic network is divided into a number of regions, each controlled by a region controller agent. The first level agents utilize reinforcement learning to find the best policy, while they send their local information to the above level agents. The local information is used to train a long short-term memory (LSTM) neural network for traffic status prediction. The agents in the above level can control the traffic signals by finding the best joint policy using the predicted traffic information. Experimental results show the effectiveness of the proposed method in a traffic network including 16 intersections.

Energy Management in Multiple Micro-Grids Considering Uncertainties of Load Using Hierarchical Multi-Agent System

Energy Management in Multiple Micro-Grids Considering Uncertainties of Load Using Hierarchical Multi-Agent System

Optimal Scheduling and Real-Time Operation of Distribution Networks With High Penetration of Plug-In Electric Vehicles

In this article, a method for real-time control and optimal operation scheduling of distribution networks with high penetration of plug-in electric vehicles (PEVs) is proposed. The main targets of the method are to minimize PEV charging cost at aggregator level, satisfy all distribution network and PEV constraints using accurate models of low computational requirements. An innovative definition of the flexibility of a PEV to change its active power is used to achieve efficient optimal active power dispatch to each PEV of a cluster. Real-time optimal dispatch of the reactive power to each PEV is also achieved. A hierarchical multiagent system is designed in order to deal with the increased complexity of the controlled system and collect the necessary information from the network. It is shown that the proposed method requires very small computation and communication times that do not depend on the number of PEVs. The convergence of the method is ensured with only one round of agent negotiations. The efficiency of the method is proved by detailed simulation results of a realistic distribution network and based on processed real data sets of the activity of the drivers.

A new real-time multi-agent system for under frequency load shedding in a smart grid context

Automatic under frequency load shedding schemes need to be carefully designed in order to reduce the risk of widespread system collapse. This paper proposes a centralized hierarchical multi-agent system that coordinates various stages of monitoring and decision making processes. The main contribution is to improve traditional contingency response algorithms such as load shedding schemes, taking advantage of the future smart grid infrastructure. The multi-agent system seeks for a minimum amount of load disconnection in a short period of time, causing the least possible disturbance in the system frequency. A hardware-in-the-loop simulation of a full electric power system using a real time digital simulator was utilized. The solution was embedded in a real time system, consisting of hardware and software, to test and validate the proposed methodology. In addition, the studied methodology was compared with two other load shedding philosophies through a load shedding metric score. Shedding was carried out in a single step and the amount of disconnected load was close to the dynamic power unbalance. The results show that it is possible to improve the traditional load shedding philosophy schemes and use advanced communication infrastructure, monitoring and embedded processing capabilities to provide better stability and reduce unnecessary load disconnections from the system.

Coordinated Traffic Light Control in Cooperative Green Vehicle Routing for Pheromone-based Multi-Agent Systems

Green transportation has been increasingly gaining attention in recent years. Existing pheromone-based traffic management frameworks were developed to reduce urban congestion by fusing traffic lights control strategies and vehicle routing schemes. Despite a significant reduction in traffic congestions, the greener aspects of transportation were not well investigated. In view of this, a Pheromone-based Green Transportation System (PGTS) is proposed to reduce Greenhouse Gas emissions and urban congestion in a three-step approach. First, traffic congestions are predicted based on the transport pheromone intensity of the target and adjacent upstream roads through an online epsilon-Support Vector Regression model. Second, a Coordinated Traffic Light Control (CTLC) strategy generates green wave scenario, dispersing heavy traffic on congested roads to the coordinated downstream paths. Third, a Cooperative Green Vehicle Routing (CGVR) takes a further leap by probabilistically rerouting upstream vehicles from entering the congested road, preventing the accumulation of vehicles that can lead to upstream congestion. Intuitively, the integration of CTLC and CGVR increases the chances that vehicles traversing multiple intersections with fewer frequencies of acceleration, effectively marking down fuel consumption. The proposed PGTS can be realized through a Pheromone-based Hierarchical Multi-Agent System (PHMAS). Based on Singapore traffic data, experimental results from a microscopic simulation SUMO show that the proposed PGTS outperforms other six approaches in reducing carbon dioxide emissions by 37.7%, fuel consumption by 37.6%, mean travel time by 47.5%, mean waiting time by 57.3%, and increasing number of arrived vehicles at designated destinations by 62.6%.

A Bilateral Decision Support Platform for Public Charging of Connected Electric Vehicles

As electric vehicles (EVs) proliferate and dominate the roads, public charging will be required extensively. Wide range coverage of energy information is required on fingertips for taking decisions at different levels and for all stakeholders. For this reason, a need arises for a correlated and all-inclusive framework that trades the unilateral discriminatory benefits with the combined benefit of all. At the heart of such frameworks lies communication technologies to support and manage energy transfer between vehicles and the power grid, known as the Internet of EVs. This paper adds two main contributions to the work presented in the literature on the subject matter. First, the communication requirements, standards, and protocols for applications involving this mobile prosumer management in the pre-connection stage to the electric grid are outlined. Second, and based on the aforementioned assessment, this paper proposes a decentralized hierarchical multiagent system that serves as a framework for real-time bilateral decision support for EVs and the power system. The effectiveness of the proposed framework and the credibility of using the communication middleware and means are experimentally verified on a group of connected vehicles communicating with a higher end server. Also, the seamless integration between the on-board and the higher end decision support systems is demonstrated in case studies.

Proactive eco-friendly pheromone-based green vehicle routing for multi-agent systems

The significant increase in vehicle numbers leads to adverse traffic congestion. Existing pheromone-based vehicle routing randomly assigns an alternative route to vehicles based on global distance and local pheromone intensity. Although urban congestion is reduced as vehicles are assigned paths with lower pheromone intensity, frequent stops due to numerous red signalized intersections within the paths can lead to high fuel consumption. To address this issue, a proactive Eco-friendly Pheromone-based Green Vehicle Routing (E-PGVR) strategy is proposed to prioritize the greener aspects of routing by reducing fuel consumption in addition to traffic congestion. First, paths with multiple green signalized intersections are assigned to increase the chance for vehicles to traverse multiple intersections with fewer stops, marking down fuel consumption. Second, the application of local dynamic information namely traffic light time, mean road speed, and predicted pheromone intensity encourages the implementation of a decentralized Hierarchical Multi-Agent Pheromone-based System (HMAPS). Third, a modified dynamic k-shortest path algorithm is proposed in the path assignment to reduce computational effort. Forth, Singapore traffic data that incorporates heterogeneous vehicle type is employed in the microscopic simulator (SUMO) to model real-world traffic scenario. Experiment results show that the proposed E-PGVR outperforms other approaches in reducing fuel consumption and CO2 by 21.2%, mean waiting time by 30.0%, number of congested roads by 56.9%, mean travel time by 26.3%, and increasing number of arrived vehicles by 45.5%.

Makespan reduction for dynamic workloads in cluster-based data grids using reinforcement-learning based scheduling

Scheduling is one of the important problems within the scope of control and management in grid and cloud-based systems. Data grid still as a primary solution to process data-intensive tasks, deals with managing large amounts of distributed data in multiple nodes. In this paper, a two-phase learning-based scheduling algorithm is proposed for data-intensive tasks scheduling in cluster-based data grids. In the proposed scheduling algorithm, a hierarchical multi agent system, consisting of one global broker agent and several local agents, is applied to scheduling procedure in the cluster-based data grids. At the first step of the proposed scheduling algorithm, the global broker agent selects the cluster with the minimum data cost based on the data communication cost measure, then an adaptive policy based on Q-learning is used by the local agent of the selected cluster to schedule the task to the proper node of the cluster. The impacts of three action selection strategies have been investigated in the proposed scheduling algorithm, and the performance of different versions of the scheduling algorithm regarding different action selection strategies, has been evaluated under three types of workloads with heterogeneous tasks. Experimental results show that for dynamic workloads with varying task submission patterns, the proposed learning-based scheduling algorithm gives better performance compared to four common scheduling algorithm, Queue Length (Shortest Queue), Access Cost, Queue Access Cost (QAC) and HCS, which use regular combinations of primary parameters such as, data communication cost and queue length. Applying a learning-based strategy provides the scheduling algorithm with more adaptability to the changing conditions in the environment.

Modeling the behavior of a hierarchy of command agents with context-based reasoning

Context-based reasoning is a paradigm for modeling agent behavior that is based on the idea that humans only use a small portion of their knowledge at any given time. It was specially designed to represent human tactical behavior and has been successfully implemented in systems with single agents or two agents working together. In this paper, we apply this idea in a hierarchical multi-agent system of command agents, where the agents’ actions are to command and coordinate subordinates, send reports to their superiors, and communicate with other agents at the same level. We focus on how contexts and actions can be defined for these higher level command agents and how the contexts and actions for the different command agents are related. The proposed methodology is implemented and tested for a hierarchy of command agents that are interpreting and planning an operational order at a battalion level and carrying it out in a computer generated forces environment.

Distributed cooperative voltage control based on curve-fitting in active distribution networks

This study investigates a hybrid hierarchical multi-agent system for distributed cooperative voltage control in active distribution networks. The hybrid hierarchical multi-agent system adopts on-load tap-changing (OLTC) agents for the distribution transformers and feeder control section (FCS) agents for the distributed generators (DGs). The objective is to minimize the voltage deviations over the network. The FCS agents also have the objective of minimizing reductions in DG power output. A least squares method is used for curve fitting to achieve the two objectives. The OLTC agent receives voltage information from the FCS agents to evaluate the state of the voltage in each feeder and the distribution network and cooperates with the FCS agents to control the voltage of the network. The FCS agents exchange the fitted curve parameters and basic information on the DGs with other agents to achieve the objectives. The effectiveness of the proposed distributed cooperative voltage control scheme is verified through simulations. Depending on the network voltages obtained by the OLTC agent, different operations are executed to prevent voltage limit violations and to minimize the voltage deviations and reductions in the DG power outputs.

MAS-Based Management and Control Strategies for Integrated Hybrid Energy System

Since a microgrid consists of various distributed energy resources and local loads, integration of a distributed grid and multiple microgrids leads to an integrated hybrid energy system. This paper focuses on improving profit, economy, security, and dynamic performance of the integrated hybrid energy system by means of developing different areas of management and control strategies upon four-level hierarchical multiagent system. The level 1 agent implements optimal price bidding strategies for high profit of the overall system. The level 2 agent optimizes the energy management strategies for economic operation of each microgrid. The level 3 agent executes coordinated switching control for maintaining security of each microgrid. The level 4 agent facilitates local hybrid control of each distributed energy resource for guaranteeing dynamic performance. Finally, validity of the proposed scheme is tested by means of simulation study.

BioASF: a framework for automatically generating executable pathway models specified in BioPAX.

Motivation: Biological pathways play a key role in most cellular functions. To better understand these functions, diverse computational and cell biology researchers use biological pathway data for various analysis and modeling purposes. For specifying these biological pathways, a community of researchers has defined BioPAX and provided various tools for creating, validating and visualizing BioPAX models. However, a generic software framework for simulating BioPAX models is missing. Here, we attempt to fill this gap by introducing a generic simulation framework for BioPAX. The framework explicitly separates the execution model from the model structure as provided by BioPAX, with the advantage that the modelling process becomes more reproducible and intrinsically more modular; this ensures natural biological constraints are satisfied upon execution. The framework is based on the principles of discrete event systems and multi-agent systems, and is capable of automatically generating a hierarchical multi-agent system for a given BioPAX model.Results: To demonstrate the applicability of the framework, we simulated two types of biological network models: a gene regulatory network modeling the haematopoietic stem cell regulators and a signal transduction network modeling the Wnt/β-catenin signaling pathway. We observed that the results of the simulations performed using our framework were entirely consistent with the simulation results reported by the researchers who developed the original models in a proprietary language.Availability and Implementation: The framework, implemented in Java, is open source and its source code, documentation and tutorial are available at http://www.ibi.vu.nl/programs/BioASF.Contact: j.heringa@vu.nl