Autonomous Agents Research Articles

Formal contracts mitigate social dilemmas in multi-agent reinforcement learning

Multi-agent Reinforcement Learning (MARL) is a powerful tool for training autonomous agents acting independently in a common environment. However, it can lead to sub-optimal behavior when individual incentives and group incentives diverge. Humans are remarkably capable at solving these social dilemmas. It is an open problem in MARL to replicate such cooperative behaviors in selfish agents. In this work, we draw upon the idea of formal contracting from economics to overcome diverging incentives between agents in MARL. We propose an augmentation to a Markov game where agents voluntarily agree to binding transfers of reward, under pre-specified conditions. Our contributions are theoretical and empirical. First, we show that this augmentation makes all subgame-perfect equilibria of all Fully Observable Markov Games exhibit socially optimal behavior, given a sufficiently rich space of contracts. Next, we show that for general contract spaces, and even under partial observability, richer contract spaces lead to higher welfare. Hence, contract space design solves an exploration-exploitation tradeoff, sidestepping incentive issues. We complement our theoretical analysis with experiments. Issues of exploration in the contracting augmentation are mitigated using a training methodology inspired by multi-objective reinforcement learning: Multi-Objective Contract Augmentation Learning. We test our methodology in static, single-move games, as well as dynamic domains that simulate traffic, pollution management, and common pool resource management.

Automation of time series analysis of Google trends data in R studio using autonomous AI agents

Automation of time series analysis of Google trends data in R studio using autonomous AI agents

Merging Paradigms: The Synergy of Symbolic and Connectionist AI in LLM-Powered Autonomous Agents

Merging Paradigms: The Synergy of Symbolic and Connectionist AI in LLM-Powered Autonomous Agents

Adaptive Multi-Agent Reinforcement Learning for Optimizing Dynamic Electric Vehicle Charging Networks in Thailand

The rapid growth of electric vehicles (EVs) necessitates efficient management of dynamic EV charging networks to optimize resource utilization and enhance service reliability. This paper explores the application of adaptive multi-agent reinforcement learning (MARL) to address the complexities of EV charging infrastructure in Thailand. By employing MARL, multiple autonomous agents learn to optimize charging strategies based on real-time data by adapting to fluctuating demand and varying electricity prices. Building upon previous research that applied MARL to static network configurations, this study extends the application to dynamic and real-world scenarios, integrating real-time data to refine agent learning processes and also evaluating the effectiveness of adaptive MARL in maximizing rewards and improving operational efficiency compared to traditional methods. Experimental results indicate that MARL-based strategies increased efficiency by 20% and reduced energy costs by 15% relative to conventional algorithms. Key findings demonstrate the potential of extending MARL in transforming EV charging network management, highlighting its benefits for stakeholders, including EV owners, operators, and utility providers. This research contributes insights into advancing electric mobility and energy management in Thailand through innovative AI-driven approaches. The implications of this study include significant improvements in the reliability and cost-effectiveness of EV charging networks, fostering greater adoption of electric vehicles and supporting sustainable energy initiatives. Future research directions include enhancing MARL adaptability and scalability as well as integrating predictive analytics for proactive network optimization and sustainability. These advancements promise to further refine the efficacy of EV charging networks, ensuring that they meet the growing demands of Thailand’s evolving electric mobility landscape.

PenGym: Realistic training environment for reinforcement learning pentesting agents

Penetration testing, or pentesting, refers to assessing network system security by trying to identify and exploit any existing vulnerabilities. Reinforcement Learning (RL) has recently become an effective method for creating autonomous pentesting agents. However, RL agents are typically trained in a simulated network environment. This can be challenging when deploying them in a real network infrastructure due to the lack of realism of the simulation-trained agents.In this paper, we present PenGym, a framework for training pentesting RL agents in realistic network environments. The most significant features of PenGym are its support for real pentesting actions, full automation of the network environment creation, and good execution performance. The results of our experiments demonstrated the advantages and effectiveness of using PenGym as a realistic training environment in comparison with a simulation approach (NASim). For the largest scenario, agents trained in the original NASim environment behaved poorly when tested in a real environment, having a high failure rate. In contrast, agents trained in PenGym successfully reached the pentesting goal in all our trials. Even after fixing logical modeling issues in simulation to create the revised version NASim(rev.), experiment results with the largest scenario indicated that agents trained in PenGym slightly outperformed, and were more stable, than those trained in NASim(rev.). Thus, the average number of steps required to reach the pentesting goal was 1.4 to 8 steps better for PenGym. Consequently, PenGym provides a reliable and realistic training environment for pentesting RL agents, eliminating the need to model agent actions via simulation.

No Laughing Matter: How Autonomous Agents Can Improve Well-Being and Teamwork

Autonomous agents are a critical factor for maintaining the well-being and health of astronaut teams in long duration spaceflight. As autonomous agents become more independent and interdependent the need to understand how they can interact effectively with humans not only becomes more necessary but will become the paradigm for future team research. In our paper, we describe a conceptual model that explains how autonomous agents and human team members can learn to implicitly coordinate through humorous social interaction and effective task interdependence. We argue that to build a shared mental model between autonomous agents and human team members, HATs need to develop trust with one another, albeit through different mechanisms, to work together effectively.

Specification, validation and verification of social, legal, ethical, empathetic and cultural requirements for autonomous agents

Autonomous agents are increasingly being proposed for use in healthcare, assistive care, education, and other applications governed by complex human-centric norms. To ensure compliance with these norms, the rules they induce need to be unambiguously defined, checked for consistency, and used to verify the agent. In this paper, we introduce a framework for formal specification, validation and verification of social, legal, ethical, empathetic and cultural (SLEEC) rules for autonomous agents. Our framework comprises: (i) a language for specifying SLEEC rules and rule defeaters (that is, circumstances in which a rule does not apply or an alternative form of the rule is required); (ii) a formal semantics (defined in the process algebra tock-CSP) for the language; and (iii) methods for detecting conflicts and redundancy within a set of rules, and for verifying the compliance of an autonomous agent with such rules. We show the applicability of our framework for two autonomous agents from different domains: a firefighter UAV, and an assistive-dressing robot.

Joint antenna selection and resource allocation for mm‐wave directional D2D communications using distributed deep reinforcement learning

AbstractIn this paper, with the promising assumption of using adaptive directional microstrip antenna on user equipment, the problem of joint antenna selection, spectrum assignment, and transmit power allocation for mm‐wave device‐to‐device communications underlying cellular networks is tackled, with the goal of enhancing system throughput and energy efficiency. To address the complexity of this problem, a method based on multi‐agent distributed deep reinforcement learning in which an autonomous intelligent agent is deployed for each user equipment is proposed. The performance evaluation demonstrates its superiority over existing strategies, resulting in improved system performance, reduced outage probability, and enhanced energy efficiency.

ChatEDA: A Large Language Model Powered Autonomous Agent for EDA

ChatEDA: A Large Language Model Powered Autonomous Agent for EDA

Enhancing Autonomous Agents Through Narrative Memory: A Comprehensive Framework for Improved Performance and Interaction

The process of organizing information and experiences into a narrative style is known as narrative memory, and it presents a cutting-edge method for improving the performance of autonomous agents. This essay describes the idea of narrative memory, as well as its elements, construction of narrative structures, and uses. The four main elements of narrative memory are timelines, characters, events, and context. Events are important things that happen, such interactions or acts, and characters are people, other agents, and the autonomous agent. The context gives the underlying knowledge required to comprehend these events and individuals, while the chronology sets the order of occurrences, creating a cause-and-effect link. The process of creating narrative frameworks includes gathering data, identifying events, crafting stories, and refreshing memories. Three methods are used to collect data: external sources, user input, and sensors. This data is used to identify significant events, which are subsequently arranged into logical tales. New situations and experiences are constantly added to the narrative memory. Improved predictive analysis, easier learning and adaptability, and improved human-robot interaction are just a few of the uses of narrative memory in autonomous agents. Robots can interact with people more successfully if they can comprehend human narratives and reply with responses that are contextually relevant. Agents may learn and adapt by identifying patterns in their prior experiences, and they can make better decisions by forecasting future occurrences with the help of historical narratives. Managing the complexity of real-world events, addressing ambiguity and uncertainty, and guaranteeing the ethical use of narrative data are challenges in the implementation of narrative memory. In order to improve narrative memory frameworks, future research options include creating sophisticated story generating tools, integrating narrative memory with other memory systems, and working across disciplines. The goal of this roadmap is to enhance autonomous systems' comprehension, communication, and decision-making skills by utilizing the power of tales.

A Fault-Tolerant Multi-Agent Reinforcement Learning Framework for Unmanned Aerial Vehicles–Unmanned Ground Vehicle Coverage Path Planning

In this paper, we introduce a fault-tolerant multi-agent reinforcement learning framework called SERT-DQN to optimize the operations of UAVs with UGV central control in coverage path planning missions. Our approach leverages dual learning systems that combine individual agent autonomy with centralized strategic planning, thus enhancing the efficiency of cooperative path planning missions. This framework is designed for high performance in environments with fault uncertainty detected and operational challenges such as interruptions in connectivity and compromised sensor reliability. With the integration of an innovative communication system between agents, our system appropriately handles both static and dynamic environments. Also, we introduce similarity-based shared experience replay to attain faster convergence and sample efficiency in the multi-agent system. The architecture is specially designed to respond adaptively to such irregularities by effectively showing enhanced resilience in scenarios where data integrity is impaired due to faults or the UAV faces disruptions. Simulation results indicate that our fault tolerance algorithms are very resilient and do indeed improve mission outcomes, especially under dynamic and highly uncertain operating conditions. This approach becomes critical for the most recent sensor-based research in autonomous systems.

Utilizing ChatGPT for On-the-Fly Decision-Making in Autonomous Systems

In the realm of autonomous systems, rapid and efficient decision-making is critical to ensure optimal performance and adaptability in dynamic environments. This paper explores the application of ChatGPT, a large-scale language model, in on-the-fly decision-making within autonomous systems. By leveraging ChatGPT's capabilities in natural language processing, contextual understanding, and real-time response generation, the study demonstrates how it can support autonomous agents in interpreting situations, generating actionable insights, and making informed decisions. The integration of ChatGPT into autonomous systems provides a flexible and scalable approach to improving decision accuracy and response times in unpredictable scenarios. This research investigates the benefits, challenges, and potential use cases, highlighting the model's contributions to enhancing autonomy in various fields such as robotics, self-driving vehicles, and intelligent drones.

How optimal is the minimum-time manoeuvre of an artificial race driver?

Minimum-lap-time optimal control problems (MLT-OCPs) are a popular tool to assess the best lap time of a vehicle on a racetrack. However, MLT-OCPs with high-fidelity dynamic vehicle models are computationally expensive, which limits them to offline use. When using autonomous agents in place of an MLT-OCP for online trajectory planning and control, the question arises of how far the resulting manoeuvre is from the maximum performance. In this paper, we improve a recently proposed artificial race driver (ARD) for online trajectory planning and control, and we compare it with a benchmark MLT-OCP. The novel challenge of our study is that ARD controls the same high-fidelity vehicle model used by the benchmark MLT-OCP, which enables a direct comparison of ARD and MLT-OCP. Leveraging its physics-driven structure and a new formulation of the g-g-v performance constraint, ARD achieves lap times comparable to the offline MLT-OCP (few milliseconds difference). We analyse the different trajectories resulting from the ARD and MLT-OCP solutions, to understand how ARD minimises the effect of local execution errors in search of the minimum-lap-time. Finally, we show that ARD consistently maintains its performance when tested on unseen circuits, even with unmodelled changes in the vehicle's mass.

Scaling Learning based Policy Optimization for Temporal Logic Tasks by Controller Network Dropout

This paper introduces a model-based approach for training feedback controllers for an autonomous agent operating in a highly nonlinear (albeit deterministic) environment. We desire the trained policy to ensure that the agent satisfies specific task objectives and safety constraints, both expressed in Discrete-Time Signal Temporal Logic (DT-STL). One advantage for reformulation of a task via formal frameworks, like DT-STL, is that it permits quantitative satisfaction semantics. In other words, given a trajectory and a DT-STL formula, we can compute the robustness , which can be interpreted as an approximate signed distance between the trajectory and the set of trajectories satisfying the formula. We utilize feedback control, and we assume a feed forward neural network for learning the feedback controller. We show how this learning problem is similar to training recurrent neural networks (RNNs), where the number of recurrent units is proportional to the temporal horizon of the agent's task objectives. This poses a challenge: RNNs are susceptible to vanishing and exploding gradients, and naïve gradient descent-based strategies to solve long-horizon task objectives thus suffer from the same problems. To tackle this challenge, we introduce a novel gradient approximation algorithm based on the idea of dropout or gradient sampling. One of the main contributions is the notion of controller network dropout , where we approximate the NN controller in several time-steps in the task horizon by the control input obtained using the controller in a previous training step. We show that our control synthesis methodology, can be quite helpful for stochastic gradient descent to converge with less numerical issues, enabling scalable backpropagation over long time horizons and trajectories over high dimensional state spaces. We demonstrate the efficacy of our approach on various motion planning applications requiring complex spatio-temporal and sequential tasks ranging over thousands of time-steps.

Sim-to-real transfer of adaptive control parameters for AUV stabilisation under current disturbance

Learning-based adaptive control methods hold the potential to empower autonomous agents in mitigating the impact of process variations with minimal human intervention. However, their application to autonomous underwater vehicles (AUVs) has been constrained by two main challenges: (1) the presence of unknown dynamics in the form of sea current disturbances, which cannot be modelled or measured due to limited sensor capability, particularly on smaller low-cost AUVs, and (2) the nonlinearity of AUV tasks, where the controller response at certain operating points must be excessively conservative to meet specifications at other points. Deep Reinforcement Learning (DRL) offers a solution to these challenges by training versatile neural network policies. Nevertheless, the application of DRL algorithms to AUVs has been predominantly limited to simulated environments due to their inherent high sample complexity and the distribution shift problem. This paper introduces a novel approach by combining the Maximum Entropy Deep Reinforcement Learning framework with a classic model-based control architecture to formulate an adaptive controller. In this framework, we propose a Sim-to-Real transfer strategy, incorporating a bio-inspired experience replay mechanism, an enhanced domain randomisation technique, and an evaluation protocol executed on a physical platform. Our experimental assessments demonstrate the effectiveness of this method in learning proficient policies from suboptimal simulated models of the AUV. When transferred to a real-world vehicle, the approach exhibits a control performance three times higher compared to its model-based nonadaptive but optimal counterpart.

AI‐induced dehumanization

AbstractRecent technological advancements have empowered nonhuman entities, such as virtual assistants and humanoid robots, to simulate human intelligence and behavior. This paper investigates how autonomous agents influence individuals' perceptions and behaviors toward others, particularly human employees. Our research reveals that the socio‐emotional capabilities of autonomous agents lead individuals to attribute a humanlike mind to these nonhuman entities. Perceiving a high level of humanlike mind in the nonhuman, autonomous agents affects perceptions of actual people through an assimilation process. Consequently, we observe “assimilation‐induced dehumanization”: the humanness judgment of actual people is assimilated toward the lower humanness judgment of autonomous agents, leading to various forms of mistreatment. We demonstrate that assimilation‐induced dehumanization is mitigated when autonomous agents possess capabilities incompatible with humans, leading to a contrast effect (Study 2), and when autonomous agents are perceived as having a high level of cognitive capability only, resulting in a lower level of mind perception of these agents (Study 3). Our findings hold across various types of autonomous agents (embodied: Studies 1–2 and disembodied: Studies 3–5), as well as in real and hypothetical consumer choices.

Behave Yourself! Behavioral Indicators of Trust in Human-Agent Teams

As advancements in artificial intelligence accelerate, there is a rise in the complexity and number of autonomous agents placed in human-agent teams (HATs). With this expansion, it is important to understand how trust in agent teammates evolves and is influenced by contextual events. In support of this, significant research has focused on the factors that influence human trust in an agent and elements that negatively impact this trust. In this research, human trust in agent teammates is typically measured via self-report surveys. Although a reliable format, surveys are not without limitations, as they can be disruptive and lack the temporal resolution needed to capture nuanced and dynamic changes in trust. As a result, researchers are adopting alternative measurement approaches that aim to unobtrusively capture measures that correlate with trust using behavioral and physiological sensors that capture measures such as voice pitch and eye tracking. However, most of these studies utilize only one unobtrusive measure, which can be indicative of various states (e.g., heart rate varies with psychological and physical responses). Given the lack of selectivity in such measures, it may be necessary to utilize multiple unobtrusive measures at once, such as real-time recorded behavioral indicators, to effectively capture changes in these indicators that are due to changes in trust over time. This paper investigated the following research questions: (1) do multiple unobtrusive indicators provide a more holistic picture of trust compared to a single behavioral indicator, and (2) are certain behavioral indicators more indicative of self-reported trust at different times during a mission (e.g., before/after a trust violation event)? Results indicated that following a significant event, changes in self-reported trust are associated with specific behaviors related to improving or mitigating perceived deficiencies in their agent teammates. These insights highlight the opportunity to integrate dynamic behavioral measures in trust assessment frameworks, specifically in scenarios where a significant event compromises trust.

Development of Deep-Learning-Based Autonomous Agents for Low-Speed Maneuvering in Unity

Development of Deep-Learning-Based Autonomous Agents for Low-Speed Maneuvering in Unity

Integration of memory systems supporting non-symbolic representations in an architecture for lifelong development of artificial agents

Compared to autonomous agent learning, lifelong agent learning tackles the additional challenge of accumulating skills in a way favourable to long term development. What an agent learns at a given moment can be an element for the future creation of behaviours of greater complexity, whose purpose cannot be anticipated.Beyond its initial low-level sensorimotor development phase, the agent is expected to acquire, in the same manner as skills, values and goals which support the development of complex behaviours beyond the reactive level. To do so, it must have a way to represent and memorize such information.In this article, we identify the properties suitable for a representation system supporting the lifelong development of agents through a review of a wide range of memory systems and related literature. Following this analysis, our second contribution is the proposition and implementation of such a representation system in MIND, a modular architecture for lifelong development. The new variable module acts as a simple memory system which is strongly integrated to the hierarchies of skill modules of MIND, and allows for the progressive structuration of behaviour around persistent non-symbolic representations. Variable modules have many applications for the development and structuration of complex behaviours, but also offer designers and operators explicit models of values and goals facilitating human interaction, control and explainability.We show through experiments two possible uses of variable modules. In the first experiment, skills exchange information by using a variable representing the concept of “target”, which allows the generalization of navigation behaviours. In the second experiment, we show how a non-symbolic representation can be learned and memorized to develop beyond simple reactive behaviour, and keep track of the steps of a process whose state cannot be inferred by observing the environment.

The Goal after Tomorrow: Offline Goal Reasoning with Norms

Recent studies have focused on autonomous agents that select their own goals and then select actions to achieve these goals, using online Goal Reasoning (GR). GR agents can revise goals and plans at execution time if unexpected outcomes occur. However, for ethical or legal agent design, even the partial execution of an online plan may result in foreseeable norm violations. To prevent these violations, it is crucial to incorporate GR already at the planning phase. To this end, we design an offline GR system that can harbour normative systems or deontic logics for goal generation. Our main results include a characterization and comparison of the completeness classes for a variety of offline GR planners, and a discussion of the irreducibility of offline GR to pure planning methods.