Agent Plans Research Articles

Dyna-PINN: Physics-informed deep dyna-q reinforcement learning for intelligent control of building heating system in low-diversity training data regimes

This paper introduces Dyna-PINN, a novel physics-informed Deep Dyna-Q (DDQ) reinforcement learning (RL) approach, designed to address the data-intensive training requirements and model-agnostic nature of the conventional model-free RL methods. The DDQ approach blends model-based and model-free elements to enhance both learning and decision-making processes. By utilizing a physics-informed neural network (PINN) based model, our method enriches the learning process with physical information, enhancing the agent's planning capabilities and leading to faster learning compared to conventional model-free RL methods like Deep Q-Network (DQN) in scenarios with low-diversity training data availability. Our results demonstrate that Dyna-PINN has 50% greater sample efficiency than DQN and outperforms rule-based control in terms of thermal discomfort. Due to physics incorporation, the Dyna-PINN implements a more logical and interpretable control policy. It shows consistently good performance compared to all control variants across low-diversity data scenarios, i.e., 6 weeks of building data, and in higher-diversity data regimes, i.e., 6 months of building energy data, demonstrating the value of physics incorporation into the RL training. Additionally, we present two other DDQ-based techniques, RC-DDQ and NN-DDQ, exploring the synergy between neural networks and physical data in intelligent control designs for building energy systems. Rigorous controller testing is performed using the Building Optimization and Testing Framework (BOPTEST), a high-fidelity simulator that closely represents a real building's operation. Through comprehensive comparisons and realistic simulations, our study underscores the effectiveness of incorporating physics-informed approaches into RL-based control strategies, paving the way for more efficient and robust building energy management systems.

Online Reinforcement Learning-Based Pedagogical Planning for Narrative-Centered Learning Environments

Pedagogical planners can provide adaptive support to students in narrative-centered learning environments by dynamically scaffolding student learning and tailoring problem scenarios. Reinforcement learning (RL) is frequently used for pedagogical planning in narrative-centered learning environments. However, RL-based pedagogical planning raises significant challenges due to the scarcity of data for training RL policies. Most prior work has relied on limited-size datasets and offline RL techniques for policy learning. Unfortunately, offline RL techniques do not support on-demand exploration and evaluation, which can adversely impact the quality of induced policies. To address the limitation of data scarcity and offline RL, we propose INSIGHT, an online RL framework for training data-driven pedagogical policies that optimize student learning in narrative-centered learning environments. The INSIGHT framework consists of three components: a narrative-centered learning environment simulator, a simulated student agent, and an RL-based pedagogical planner agent, which uses a reward metric that is associated with effective student learning processes. The framework enables the generation of synthetic data for on-demand exploration and evaluation of RL-based pedagogical planning. We have implemented INSIGHT with OpenAI Gym for a narrative-centered learning environment testbed with rule-based simulated student agents and a deep Q-learning-based pedagogical planner. Our results show that online deep RL algorithms can induce near-optimal pedagogical policies in the INSIGHT framework, while offline deep RL algorithms only find suboptimal policies even with large amounts of data.

ПІДТВЕРДЖУВАЛЬНЕ ВИКРИВЛЕННЯ В ПРОЦЕСАХ АРГУМЕНТАЦІЇ

B a c k g r o u n d . The article is devoted to the study of confirmatory distortion as a cognitive bias within the framework of the modern theory of argumentation. In the context of this study, the effectiveness of the critical questioning technique as an argumentation strategy aimed at reducing the negative impact of confirmatory bias is considered. M e t h o d s . To achieve the goals of the research, the method of critical questions is used, which is based on the traditional principles of logic, dialectics and rhetoric. Specific lists of critical questions are developed for each argumentative scheme. R e s u l t s . The article demonstrates that standard critical questions for schemes of practical reasoning, developed by eminent specialists, are able to minimize the negative impact of confirmation bias. This is achieved by taking into account information that may conflict with the agent's action plan. C o n c l u s i o n s . The study highlights the importance of using critical questions as a tool to minimize confirmation bias. Such an approach not only contributes to the substantiation of arguments, but also stimulates self-control and the search for information that may cause doubts about one's own views. The final part of the article examines propaganda as the deliberate use of confirmatory distortion and points to the need for attention to language patterns where it occurs rather unintentionally. In connection with the specificity of the studied phenomenon, the term "confirmation bias" is proposed as a translation of the English "confirmation bias".

Understanding Bird’s-Eye View of Road Semantics Using an Onboard Camera

Autonomous navigation requires scene understanding of the action-space to move or anticipate events. For planner agents moving on the ground plane, such as autonomous vehicles, this translates to scene understanding in the bird’s-eye view (BEV). However, the onboard cameras of autonomous cars are customarily mounted horizontally for a better view of the surrounding. In this work, we study scene understanding in the form of online estimation of semantic BEV maps using the video input from a single onboard camera. We study three key aspects of this task, image-level understanding, BEV level understanding, and the aggregation of temporal information. Based on these three pillars we propose a novel architecture that combines these three aspects. In our extensive experiments, we demonstrate that the considered aspects are complementary to each other for BEV understanding. Furthermore, the proposed architecture significantly surpasses the current state-of-the-art. Code: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ybarancan/BEV_feat_stitch</uri> .

Plans or Outcomes: How Do We Attribute Intelligence to Others?

Humans routinely make inferences about both the contents and the workings of other minds based on observed actions. People consider what others want or know, but also how intelligent, rational, or attentive they might be. Here, we introduce a new methodology for quantitatively studying the mechanisms people use to attribute intelligence to others based on their behavior. We focus on two key judgments previously proposed in the literature: judgments based on observed outcomes (you're smart if you won the game) and judgments based on evaluating the quality of an agent's planning that led to their outcomes (you're smart if you made the right choice, even if you didn't succeed). We present a novel task, the maze search task (MST), in which participants rate the intelligence of agents searching a maze for a hidden goal. We model outcome-based attributions based on the observed utility of the agent upon achieving a goal, with higher utilities indicating higher intelligence, and model planning-based attributions by measuring the proximity of the observed actions to an ideal planner, such that agents who produce closer approximations of optimal plans are seen as more intelligent. We examine human attributions of intelligence in three experiments that use MST and find that participants used both outcome and planning as indicators of intelligence. However, observing the outcome was not necessary, and participants still made planning-based attributions of intelligence when the outcome was not observed. We also found that the weights individuals placed on plans and on outcome correlated with an individual's ability to engage in cognitive reflection. Our results suggest that people attribute intelligence based on plans given sufficient context and cognitive resources and rely on the outcome when computational resources or context arelimited.

A novel approach for multi-stakeholder agricultural land reallocation using agent-based modeling: A case study in Iran

Improving the agrarian structure is an overall objective of land consolidation schemes. Unfavorably shaped parcels are optimally rearranged in the land reallocation process and redistributed to pre-determined blocks. In conventional reallocation, farmers' preferences are used to determine the prospective location of the new parcels. The most common conflict that can arise during this process is that demand may be high for some blocks. Resolving these disputes, which deal directly with landowners' rights, can affect the success or failure of land consolidation projects. In this study, a novel approach is presented for land reallocation using agent-based modeling. In this method, landowner agents compete to select blocks that have the most utility for the reallocation of their parcels. The utility of a block for each landowners' parcel is determined based on criteria such as the priority of that block from the landowner's view, the rank of the parcel in that block, the presence of the landowner's fixed facilities in the block, and the neighborhood structure of that block. To improve the model's ability to achieve the optimal result, a planner agent has been defined as responsible for controlling and guiding the process towards optimal land reallocation. The proposed model was applied to an Iranian case study. The results show that the reallocation plan made by the proposed method performs better than the conventional method in terms of the percentage of owners' compliance with their priorities, landowners’ satisfaction, reduction of land fragmentation, and balance of land value.

Experimental evaluation of tasking and teaming design patterns for human delegation of unmanned vehicles

This work discusses different approaches for the cooperation between humans as a supervisor and multiple unmanned vehicles (UVs). We evaluated the most promising approach experimentally with expert pilots of the German Air Force. The co-agency of humans and highly automated unmanned systems (i.e., human autonomy teaming, HAT) is described by the use of a design and description language for HAT design patterns. This design language is used to differentiate control modes for tasking, teaming, and swarming of UVs. The different control modes are then combined in a planner agent (PA) design pattern that further enables the UV guidance on scalable delegation levels from a single individual up to a team. The desired system behavior and interaction concept of the PA for these scalable delegation levels is then transferred to the domain of manned-unmanned teaming in fighter aircraft missions. To demonstrate the applicability of the system, we implemented the concept into our fast-jet simulator of the Institute of Flight Systems (IFS) and conducted an experimental campaign with expert pilots. The results of the experiment showed (1) task delegation with the PA design pattern is faster and reduces the error potential; (2) scalable delegation levels enable a pilot and situation-specific task delegation; (3) the delegation of teams is faster and reduces the error potential; however, in some situations, deeper access through the scalable delegation levels is needed; (4) the concept is intuitive and the transparency and trust in UVs and swarms were very high; and (5) the pilots could imagine operating such systems in the future. Overall speaking the presented PA design pattern is suited for the guidance of UVs and the scalable delegation levels are beneficial.

Situated Temporal Planning Using Deadline-aware Metareasoning

We address the problem of situated temporal planning, in which an agent's plan can depend on scheduled exogenous events, and thus it becomes important to take the passage of time into account during the planning process. Previous work on situated temporal planning has proposed simple pruning strategies, as well as complex schemes for a simplified version of the associated metareasoning problem. Although even the simplified version of the metareasoning problem is NP-hard, we provide a pseudo-polynomial time optimal solution to the case with known deadlines. We leverage intuitions emerging from this case to provide a fast greedy scheme that significantly improves upon previous schemes even for the case of unknown deadlines. Finally, we show how this new method can be applied inside a practical situated temporal planner. An empirical evaluation suggests that the new planner provides state-of-the-art results on problems where external deadlines play a significant role.

A Bi-Level Programming Model for Protecting an Important Node in a Network

Protecting important nodes in a network against natural disasters, security threats, attacks and so on is one of the main goals of network planners. In this paper, a new model is presented for protecting an important node (NMPN) in a typical network based on defensive location problem where the threatening agent (t-agent) has the ability to reinforce its power at some nodes. The NMPN is a bi-level programming problem. At the upper level, the planner agent (p-agent) try to find the best locations for protecting resources in order to protect the important node. The lower level problem is represented as the shortest path problem in the network in which the edges are weighted with positive values and sometimes negative values. Thus, Bellman-Ford algorithm is applied to solve the lower level problem. The NMPN is an NP-hard problem. In this work, the genetic, ant colony optimization, binary artificial bee colony with differential evolution, artificial bee colony algorithms and a modified tabu search (MTS) algorithm are used to solve the problem. A test problem is randomly generated to investigate the performance of the used metaheuristic algorithms in this paper. Parameters of the metaheuristic algorithms are tuned by the Taguchi method for solving the test problem. Also, the ANOVA and Tukey tests are used to compare the performance of metaheuristic algorithms. The best results are obtained by the MTS algorithm

Planning and Acting with Non-Deterministic Events: Navigating between Safe States

Automated Planning addresses the problem of finding a sequence of actions, a plan, transforming the environment from its initial state to some goal state. In real-world environments, exogenous events might occur and might modify the environment without agent's consent. Besides disrupting agent's plan, events might hinder agent's pursuit towards its goals and even cause damage (e.g. destroying the robot).In this paper, we leverage the notion of Safe States in dynamic environments under presence of non-deterministic exogenous events that might eventually cause dead-ends (e.g. “damage” the agent) if the agent is not careful while executing its plan. We introduce a technique for generating plans that constrains the number of consecutive “unsafe” actions in a plan and a technique for generating “robust” plans that effectively evade event effects. Combination of both approaches plans and executes robust plans between safe states. We empirically show that such an approach effectively navigates the agent towards its goals in spite of presence of dead-ends.

Dealing with Incompatibilities among Procedural Goals under Uncertainty

By considering rational agents, we focus on the problem of selecting goals out of a set of incompatible ones. We consider three forms of incompatibility introduced by Castelfranchi and Paglieri, namely the terminal, the instrumental (or based on resources), and the superfluity. We represent the agent's plans by means of structured arguments whose premises are pervaded with uncertainty. We measure the strength of these arguments in order to determine the set of compatible goals. We propose two novel ways for calculating the strength of these arguments, depending on the kind of incompatibility thatexists between them. The first one is the logical strength value, it is denoted by a three-dimensional vector, which is calculated from a probabilistic interval associated with each argument. The vector represents the precision of the interval, the location of it, and the combination of precision and location. This type of representation and treatment of the strength of a structured argument has not been defined before by the state of the art. The second way for calculating the strength of the argument is based on the cost of the plans (regarding the necessary resources) and the preference of the goals associated with the plans. Considering our novel approach for measuring the strength of structured arguments, we propose a semantics for the selection of plans and goals that is based on Dung's abstract argumentation theory. Finally, we make a theoretical evaluation of our proposal.

A New Multi-Layer Distributed Approach for a Multi-objective Planning Problem

This paper introduces a new distributed approach to solve multi-objective planning problem applied to multimodal transport network planning (MTNP) problem. In this problem, the commodities should be transported on the international network by at least two different transport modes. The main goal is to find the best multimodal transportation modes and itineraries. The aim of the new approach has assured us that a distributed optimization. We split the MTNP problem into two sub-problems. These sub-problems are the assignment and the planning problems. Each sub-problem is solved at a corresponding layer. Each layer is executed by an agent. These agents interact, collaborate and communicate together to solve the MTNP problem. In this paper, we contribute by introducing a multi-layer distributed approach to solve real case’s problems. Firstly, we define the MTNP problem as a distributed constraint satisfaction multi-criteria optimization problem (DCSMOP). Secondly, we show that the split of the main problem reduces the computational complexity and the communication between the planner agent and the modes agents lead to faster convergence. The experimental results are proof of this work efficiently. This method proves their efficiency, according to the complexity of the problem and the exchange of information, the computational time and the solution quality.

Sequential plan recognition: An iterative approach to disambiguating between hypotheses

Plan recognition algorithms output hypotheses about an agent's plans from its observed actions. Due to imperfect knowledge about the agent's behavior and the environment, it is often the case that there are multiple hypotheses about an agent's plans that are consistent with the observations, though only one of these hypotheses is correct. This paper addresses the problem of how to disambiguate between hypotheses during the recognition process, by querying the acting agent about whether a given plan is part of the correct hypothesis. The main contribution is a sound and complete process for reducing the set of possible hypotheses called Sequential Plan Recognition (SPR). SPR iteratively queries the user and revises the set of possible hypotheses according to the outcome of the query. Several policies are provided for choosing which plans to query the agent. These policies address the problem of how to reduce the number of hypotheses during the recognition process using a minimal number of queries. The proposed policies include policies that use maximum likelihood and information gain measures.The paper provides a complexity analysis of the SPR process and the proposed query policies. It demonstrate its efficiency on two known domains from the literature, describing how performance and runtime are affected by features in the domain. Our results can inform the design of future plan recognition systems that interleave the recognition process with intelligent interventions of their users.

Decision-Theoretic Planning Under Anonymity in Agent Populations

We study the problem of self-interested planning under uncertainty in settings shared with more than a thousand other agents, each of which plans at its own individual level. We refer to such large numbers of agents as an agent population. The decision-theoretic formalism of interactive partially observable Markov decision process (I-POMDP) is used to model the agent's self-interested planning. The first contribution of this article is a method for drastically scaling the finitely-nested I-POMDP to certain agent populations for the first time. Our method exploits two types of structure that is often exhibited by agent populations -- anonymity and context-specific independence. We present a variant called the many-agent I-POMDP that models both these types of structure to plan efficiently under uncertainty in multiagent settings. In particular, the complexity of the belief update and solution in the many-agent I-POMDP is polynomial in the number of agents compared with the exponential growth that challenges the original framework. While exploiting structure helps mitigate the curse of many agents, the well-known curse of history that afflicts I-POMDPs continues to challenge scalability in terms of the planning horizon. The second contribution of this article is an application of the branch-and-bound scheme to reduce the exponential growth of the search tree for look ahead. For this, we introduce new fast-computing upper and lower bounds for the exact value function of the many-agent I-POMDP. This speeds up the look-ahead computations without trading off optimality, and reduces both memory and run time complexity. The third contribution is a comprehensive empirical evaluation of the methods on three new problems domains -- policing large protests, controlling traffic congestion at a busy intersection, and improving the AI for the popular Clash of Clans multiplayer game. We demonstrate the feasibility of exact self-interested planning in these large problems, and that our methods for speeding up the planning are effective. Altogether, these contributions represent a principled and significant advance toward moving self-interested planning under uncertainty to real-world applications.

Plan Recognition Design

Goal Recognition Design (GRD) is the problem of designing a domain in a way that will allow easy identification of agents' goals. This work extends the original GRD problem to the Plan Recognition Design (PRD) problem which is the task of designing a domain using plan libraries in order to facilitate fast identification of an agent's plan. While GRD can help to explain faster which goal the agent is trying to achieve, PRD can help in faster understanding of how the agent is going to achieve its goal. We define a new measure that quantifies the worst-case distinctiveness of a given planning domain, propose a method to reduce it in a given domain and show the reduction of this new measure in three domains from the literature.

A Multi-Agent Based Simulation Model for Rail–Rail Transshipment: An Engineering Approach for Gantry Crane Scheduling

Le Havre Port Authority is putting into service a multimodal hub terminal with massified hinterland links (trains and barges) in order to restrict the intensive use of roads, to achieve a more attractive massification share of hinterland transportation and to provide a river connection to its maritime terminals that do not currently have one. This paper focuses on the rail–rail transshipment yard of this new terminal. In the current organizational policy, this yard is divided into two equal operating areas, and, in each one, a crane is placed, and it is equipped with reach stackers to enable container moves across both operating areas. However, this policy causes poor scheduling of crane moves, because it gives rise to many crane interference situations. For the sake of minimizing the occurrence of these undesirable situations, this paper proposes a multi-agent simulation model including an improved strategy for crane scheduling. This strategy is inspired by the ant colony approach and it is governed by a new configuration for the rail yard’s working area that eliminates the use of reach stackers. The proposed simulation model is based on two planner agents, to each of which a time-horizon planning is assigned. The simulation results show that the model developed here is very successful in significantly reducing unproductive times and moves (undesirable situations), and it outperforms other existing simulation models based on the current organizational policy.

Resilience through Learning in Multi-Agent Cyber-Physical Systems

The paper contributes to the design of secure and resilient supervisory Cyber-Physical Systems (CPS) through learning. The reported approach involves the inclusion of learning modules in each of the supervised agents, and considers a scenario where the system's coordinator privately transmits to individual agents their action plans in the form of symbolic strings. Each agent's plans belong in some particular class of (subregular) languages which is identifiable in the limit from positive data. With knowledge of the class of languages their plans belong to, agents can observe their coordinator's instructions and utilize the Grammatical Inference modules to identify the behavior specified for them. From that, they can then work collectively to infer the executive function of their supervisor. The paper proves that in cases where the coordinator fails, or communication to subordinates is disrupted, agents are able not only to maintain functional capacity but also to recover normalcy of operation by reconstructing their coordinator. Guaranteeing normalcy recovery in supervisory CPSs is critical in cases of a catastrophic failure or malicious attack, and is important for the design of next-generation Cyber-Physical Systems.

Land-use pattern scenario analysis using planner agents

Land-use pattern is one of the key issues in the compilation of urban master plans. In China, government, planners, and residents, all with various requirements and preferences, are the main agents participating in this process. Among them, planners play a role in negotiating with related agents and then establishing land-use patterns. In this paper we propose a planner agent framework to support land-use pattern scenario analysis (LUPSA), based on existing planning support system (PSS) research. Planner agents are divided into three types: nonspatial planner agent (NPA), spatial planner agent (SPA), and chief planner agent (CPA). The NPA is responsible for formulating special plans (such as transport, municipal public facilities, or nature reserve plans) that correspond to available data (such as road network, public facilities, and nature reserve patterns) from LUPSA. The SPA is responsible for establishing land-use patterns. The SPA considers constraints of local development conditions and communicates and coordinates with the NPAs to confirm formulated special plans that can support the implementation of the established land-use pattern. The CPA is responsible for negotiating with the government agent to ensure the reasonability of comprehensive constraints, establishing the final land-use pattern based on an evaluation of established scenarios by several SPAs, then determining it after a public participation process involving local residents. We initially tested this framework in a hypothetical city, then did an experiment in Beijing. Results show that the proposed planner agent framework is suitable for LUPSA.

A Strategy for PMU Placement Considering the Resiliency of Measurement System

This paper aims to find strategic locations for additional Phasor Measurement Units (PMUs) installation while considering resiliency of existing PMU measurement system. A virtual attack agent is modeled based on an optimization framework. The virtual attack agent targets to minimize observability of power system by coordinated attack on a subset of critical PMUs. A planner agent is then introduced which analyzes the attack pattern of virtual attack agent. The goal of the planner agent is to mitigate the vulnerability posed by the virtual attack agent by placing additional PMUs at strategic locations. The ensuing problem is formulated as an optimization problem. The proposed framework is applied on 14, 30, 57 and 118 bus test systems, including a large 2383 node western polish test system to demonstrate the feasibility of proposed approach for large systems.

Incorporating Bayesian learning in agent-based simulation of stakeholders’ negotiation

This paper describes the incorporation of a Bayesian learning algorithm into an agent-based model designed to simulate stakeholders’ negotiation when evaluating scenarios of land development. The objective is to facilitate reaching an agreement at an earlier stage in the negotiation by providing the opportunity to the proposer agent to learn his opponents’ preferences. The modeling approach is tested in the Elbow River watershed, in southern Alberta, Canada, that is under considerable pressure for land development due to the proximity of the fast growing city of Calgary. Five agents are included in the model respectively referred to as the Developer agent, the Planner agent, the Citizen agent, the AgricultureConcerned agent, and the WaterConcerned agent. Two types of land development scenarios are evaluated; in the first case, only the geographical location is considered while in the second case, the internal land-use composition is also varied. The Developer agent that is equipped with the Bayesian learning capability attempts to approximate its opponents’ fuzzy evaluation functions based on the responses he receives from them at each round of the negotiation. The results indicate that using this approach, an agreement can be reached in fewer number of negotiation rounds than in the case where the Developer agent selects the subsequent offers based merely on its own utility. The model also indicates how the satisfaction of each agent evolves during the negotiation. This information is very useful for decision makers who wish to consider stakeholders’ perspectives when dealing with multiple objectives in a spatial context.