Autonomous Agents Research Articles

Turing-like Experiment in a Cyber Defense Game

During the past decade, researchers of behavioral cyber security have created cognitive agents that are able to learn and make decisions in dynamic environments in ways that assimilate human decision processes. However, many of these efforts have been limited to simple detection tasks and represent basic cognitive functions rather than a whole set of cognitive capabilities required in dynamic cyber defense scenarios. Our current work aims at advancing the development of cognitive agents that learn and make defense-dynamic decisions during cyber attacks by intelligent attack agents. We also aim to evaluate the capability of these cognitive models in ``Turing-like'' experiments, comparing the decisions and performance of these agents against human cyber defenders. In this paper, we present an initial demonstration of a cognitive model of the defender that relies on a cognitive theory of dynamic decision-making, Instance-Based Learning Theory (IBLT); we also demonstrate the execution of the same defense task by human defenders. We rely on OpenAI Gym and CybORG and adapt an existing CAGE scenario to generate a simulation experiment using an IBL defender. We also offer a new Interactive Defense Game (IDG), where \textit{human} defenders can perform the same CAGE scenario simulated with the IBL model. Our results suggest that the IBL model makes decisions against two intelligent attack agents that are similar to those observed in a subsequent human experiment. We conclude with a description of the cognitive foundations required to build autonomous intelligent cyber defense agents that can collaborate with humans in autonomous cyber defense teams.

Assessing Human Reactions in a Virtual Crowd Based on Crowd Disposition, Perceived Agency, and User Traits

Immersive virtual environments populated by real and virtual humans provide valuable insights into human decision-making processes under controlled conditions. Existing literature indicates elevated comfort, higher presence, and a more positive user experience when virtual humans exhibit rich behaviors. Based on this knowledge, we conducted a web-based, interactive study, in which participants were embodied within a virtual crowd with complex behaviors driven by an underlying psychological model. While participants interacted with a group of autonomous humanoid agents in a shopping scenario similar to Black Friday, the platform recorded their non-verbal behaviors. In this independent-subjects study, we investigated behavioral and emotional variances across participants with diverse backgrounds focusing on two conditions: perceived agency and the crowd’s emotional disposition. For perceived agency, one group of participants was told that the other crowd members were avatars controlled by humans, whereas another group was told that they were artificial agents. For emotional disposition, the crowd behaved either in a docile or hostile manner. The results suggest that the crowd’s disposition and specific participant traits significantly affected certain emotions and behaviors. For instance, participants collected fewer items and reported a higher increase of negative emotions when placed in a hostile crowd. However, perceived agency did not yield any statistically significant effects.

Learning Fast and Slow: A Redux of Levels of Learning in General Autonomous Intelligent Agents

Autonomous intelligent agents, including humans, operate in a complex, dynamic environment that necessitates continuous learning. We revisit our thesis that proposes that learning in human-like agents can be categorized into two levels: Level 1 (L1) involving innate and automatic learning mechanisms, while Level 2 (L2) comprises deliberate strategies controlled by the agent. Our thesis draws from our experiences in building artificial agents with complex learning behaviors, such as interactive task learning and open-world learning.

FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design

Recent advancements in Large Language Models (LLMs) have exhibited notable efficacy in question-answering (QA) tasks across diverse domains. Their prowess in integrating extensive web knowledge has fueled interest in developing LLM-based autonomous agents. While LLMs are efficient in decoding human instructions and deriving solutions by holistically processing historical inputs, transitioning to purpose-driven agents requires a supplementary rational architecture to process multi-source information, establish reasoning chains, and prioritize critical tasks. Addressing this, we introduce FinMem, a novel LLM-based agent framework devised for financial decision-making. It encompasses three core modules: Profiling, to customize the agent's characteristics; Memory, with layered message processing, to aid the agent in assimilating hierarchical financial data; and Decision-making, to convert insights gained from memories into investment decisions. Notably, FinMem's memory module aligns closely with the cognitive structure of human traders, offering robust interpretability and real-time tuning. Its adjustable cognitive span allows for the retention of critical information beyond human perceptual limits, thereby enhancing trading outcomes. This framework enables the agent to self-evolve its professional knowledge, react agilely to new investment cues, and continuously refine trading decisions in the volatile financial environment. We first compare FinMem with various algorithmic agents on a scalable real-world financial dataset, underscoring its leading trading performance in stocks. We then fine-tuned the agent's perceptual span and character setting to achieve a significantly enhanced trading performance. Collectively, FinMem presents a cutting-edge LLM agent framework for automated trading, boosting cumulative investment returns.

Shared autonomous vehicles and agent based models: a review of methods and impacts

Shared autonomous vehicles and agent based models: a review of methods and impacts

Probabilistic Higher-Order Velocity Obstacle for Dynamic Collision Avoidance in Uncertain Environments

Autonomous systems like unmanned aircraft systems are touted as front-runners in terms of efficiency and agility in challenging operations like emergency response. The ability of autonomous agents in such systems to avoid collisions with dynamic obstacles while navigating through an obstacle-cluttered environment is critical for success in any mission. In addition to that, further challenges are posed by environments with uncertainties in obstacles’ motion, sensing, and occluded regions. To this end, a higher-order velocity obstacle–based novel motion planner is presented in this paper in a probabilistic setup for smooth, collision-free navigation of the agent with acceleration constraints in uncertain, unstructured environments with an element of anticipation for the future environment. The effectiveness of the developed algorithm for safe, collision-free, and smooth navigation of the agent is investigated in simulation studies in two different kinds of environments, one with known trajectories of obstacles and the other with unknown maneuvering trajectories of obstacles. Extensive simulation studies are performed in the presence of dynamic obstacles to elucidate the performance of the proposed algorithm on four crucial parameters—mission time, computational time, minimum obstacle distance, and an overall control effort under varied obstacle densities. With satisfactory performance in all these aspects, the developed algorithm possesses strong potential for real-time implementation.

Multi-agent system-assisted fault section location scheme for distribution networks with DFIG-based wind DGs

This paper proposes a multi-agent system (MAS)-assisted relaying scheme to identify the fault section in a distribution network (DN) that incorporates doubly-fed induction generators (DFIG)-based wind distributed generations (DGs). Firstly, to mitigate the adverse effect of the uncertain fault behavior of DFIGs on conventional directional elements, the fault direction identifiers (FDIs) are assigned at the I-type and II-type buses in the DN using the high-frequency component principle. Furthermore, a MAS-assisted relaying structure is devised to collect and integrate these identifiers and then accomplish the fault section location. On the one hand, since the setting of FDIs only requires local current information and the logical signals are solely exchanged within each predetermined associated region, the proposed fault section location scheme can be implemented in an immature DN with a limited number of voltage transformers or under weak synchronization conditions. On the other hand, the autonomy and cooperation of intelligent Agents also enable the proposed scheme to determine the fault section in a decentralized decision-making (DDM) mode, eliminating the dependence on strong central control and reducing redundant data transmission in a multi-bus distribution system. Simulation results under representative faulty scenarios validate the expected performance of the protection scheme.

Initial Goal Allocation for Multi-agent Systems

In the multi-agent environment, a human expert engages inthe allocation of objectives among individual agents. How-ever, autonomous agents need to determine and allocate ob-jectives without external intervention from humans. There-fore, in this research, we attempt to solve initial goal distri-bution challenges in multi-agent settings by developing twogoal allocation algorithms. The primary objectives are to findcost-effective goal solution sets and distribute them evenlyamong available agents. We introduce two algorithms whenthe goals are structured in a hierarchical goal tree structure,and then test their efficiency across a variety of baseline al-location methods. Both algorithms were able to increase theperformance of agents in multi-agent settings by finding themost optimal distributions of goals and allowing agents to actindependently from human intervention.

Minimizing Negative Side Effects in Cooperative Multi-Agent Systems using Distributed Coordination

Autonomous agents operating in real-world environments frequently encounter undesirable outcomes or negative side effects (NSEs) when working collaboratively alongside other agents. Even when agents can execute their primary task optimally when operating in isolation, their training may not account for potential negative interactions that arise in the presence of other agents. We frame the challenge of minimizing NSEs as a lexicographic decentralized Markov decision process in which we assume independence of rewards and transitions with respect to the primary assigned tasks, but recognize that addressing negative side effects creates a form of dependence among the agents. We present a lexicographic Q-learning approach to mitigate the NSEs using human feedback models while maintaining near-optimality with respect to the assigned tasks---up to some given slack. Our empirical evaluation across two domains demonstrates that our collaborative approach effectively mitigates NSEs, outperforming non-collaborative methods.

Higher Education in the Age of Artificial Intelligence

The breakneck expansion of artificial intelligence (AI) technologies in recent years has attracted the attention of higher education researchers. However, their curiosity often comes down to using specific AI tools, such as text and image generators, translators, and personal assistants in the educational process. This paper considers a broader question: what fundamental problems and challenges does the penetration of AI technologies into human lives originate for higher education? The authors offer a working definition of what it means to give/receive higher education in the age of AI. This definition identifies five critical characteristics of higher education: 1) the professor and the student continue to be in a subject-subject relationship and learn from each other; 2) higher education should prepare for life in conditions of “human-machine” interdependence; 3) these conditions imply choice in situations of uncertainty; 4) the spread of AI technologies brings enormous opportunities and 5) almost unpredictable dangers, risks and threats to humans. The authors consider general principles and specific problems associated with entering AI tools into society and higher education. They discuss the inevitability of the evolvement of “machine-machine interdependence” related to the development of autonomous agents. In conclusion, several theses and counter-theses summarize the article’s argumentation.

Using LLMs for Augmenting Hierarchical Agents with Common Sense Priors

Solving long-horizon, temporally-extended tasks using Reinforcement Learning (RL) is challenging, compounded by the common practice of learning without prior knowledge (or tabula rasa learning). Humans can generate and execute plans with temporally-extended actions and quickly learn to perform new tasks because we almost never solve problems from scratch. We want autonomous agents to have this same ability. Recently, LLMs have been shown to encode a tremendous amount of knowledge about the world and to perform impressive in-context learning and reasoning. However, using LLMs to solve real world problems is hard because they are not grounded in the current task. In this paper we exploit the planning capabilities of LLMs while using RL to provide learning from the environment, resulting in a hierarchical agent that uses LLMs to solve long-horizon tasks. Instead of completely relying on LLMs, they guide a high-level policy, making learning significantly more sample efficient. This approach is evaluated in simulation environments such as MiniGrid, SkillHack, and Crafter, and on a real robot arm in block manipulation tasks. We show that agents trained using our approach outperform other baselines methods and, once trained, don't need access to LLMs during deployment.

Which Actions Are Always Necessary in Fully-Observable Non-Deterministic Planning?

Explaining the sequential action choices of an autonomous agent is essential for understanding the agent and learning from the agent. Given a goal-reaching action sequence in a non-deterministic environment, one important aspect of explaining action choices is deciding if the action choice is always necessary for the action sequence, or if it may be omitted under certain circumstances. Most works in explainable planning (XAIP) have been focused on explanation of action choices in deterministic environments as assumed by classical planning, and one widely used notion is causal link. In this work, we will apply the notion of causal link to justifying actions choices of a trace in fully-observable non-determinisitic (FOND) settings. We will also introduce a new notion -- the always-necessary actions of a trace -- and propose an approach that derives the always-necessary actions of a trace from the causal link justifications of action choices.

I Act Therefore I Live? Autopoiesis, Sensorimotor Autonomy, and Extended Agency

This paper aims to determine whether extended human-machine cognitive systems and group systems can be regarded as autonomous agents. For this purpose, I compare two notions of agency: one developed within analytical philosophy of action and based on the concept of intention, and the other introduced by enactivists via the concepts of autopoiesis and sensorimotor autonomy. I argue that only the latter approach can be used to demonstrate autonomous agency in respect of systems that are not humans as such, though they contain humans as their elements. After introducing Maturana and Varela’s conception of minimal autonomy as a kind of generalization of autopoiesis, I present the three conditions of agency put forward by Barandiaran, Di Paolo and Rohde, noting that they do not invoke the property of being alive as necessary in that respect. I argue that both extended and group systems can satisfy these conditions of agency, even though they are not alive as such. The fulfillment of these conditions, however, is ensured by the autopoietic nature of the living components of these systems. That being said, an autonomous system itself does not need to be alive in the biological sense. Sensorimotor, adaptive agency could emerge out of other processes than those responsible for biological life. The article concludes with a suggestion that this is exactly what will happen if an autonomous system is ever artificially created. It would be functionally indistinguishable from a living organism, though not alive in a biological sense.

Autonomy in AI and other Artificial Agents

Autonomy in AI and other Artificial Agents

Emergence of integrated behaviors through direct optimization for homeostasis

Homeostasis is a self-regulatory process, wherein an organism maintains a specific internal physiological state. Homeostatic reinforcement learning (RL) is a framework recently proposed in computational neuroscience to explain animal behavior. Homeostatic RL organizes the behaviors of autonomous embodied agents according to the demands of the internal dynamics of their bodies, coupled with the external environment. Thus, it provides a basis for real-world autonomous agents, such as robots, to continually acquire and learn integrated behaviors for survival. However, prior studies have generally explored problems pertaining to limited size, as the agent must handle observations of such coupled dynamics. To overcome this restriction, we developed an advanced method to realize scaled-up homeostatic RL using deep RL. Furthermore, several rewards for homeostasis have been proposed in the literature. We identified that the reward definition that uses the difference in drive function yields the best results. We created two benchmark environments for homeostasis and performed a behavioral analysis. The analysis showed that the trained agents in each environment changed their behavior based on their internal physiological states. Finally, we extended our method to address vision using deep convolutional neural networks. The analysis of a trained agent revealed that it has visual saliency rooted in the survival environment and internal representations resulting from multimodal input.

Committing to the wrong artificial delegate in a collective-risk dilemma is better than directly committing mistakes

While autonomous artificial agents are assumed to perfectly execute the strategies they are programmed with, humans who design them may make mistakes. These mistakes may lead to a misalignment between the humans’ intended goals and their agents’ observed behavior, a problem of value alignment. Such an alignment problem may have particularly strong consequences when these autonomous systems are used in social contexts that involve some form of collective risk. By means of an evolutionary game theoretical model, we investigate whether errors in the configuration of artificial agents change the outcome of a collective-risk dilemma, in comparison to a scenario with no delegation. Delegation is here distinguished from no-delegation simply by the moment at which a mistake occurs: either when programming/choosing the agent (in case of delegation) or when executing the actions at each round of the game (in case of no-delegation). We find that, while errors decrease success rate, it is better to delegate and commit to a somewhat flawed strategy, perfectly executed by an autonomous agent, than to commit execution errors directly. Our model also shows that in the long-term, delegation strategies should be favored over no-delegation, if given the choice.

An adversarial environment reinforcement learning-driven intrusion detection algorithm for Internet of Things

The increasing prevalence of Internet of Things (IoT) systems has made them attractive targets for malicious actors. To address the evolving threats and the growing complexity of detection, there is a critical need to search for and develop new algorithms that are fast and robust in detecting and classifying dangerous network traffic. In this context, deep reinforcement learning (DRL) is gaining recognition as a prospective solution in numerous fields as it enables autonomous agents to cooperate with their environment for decision-making without relying on human experts. This article presents an innovative approach to intrusion detection in IoT systems using an adversarial reinforcement learning (RL) algorithm known for its exceptional predictive capabilities. The predictive process relies on a classifier, implemented as a streamlined and highly efficient neural network. Embedded within this classifier is a policy function meticulously trained using an innovative RL model. Importantly, this model ensures that the environment’s behavior is dynamically fine-tuned simultaneously with the learning process, improving the overall effectiveness of the intrusion detection approach. The efficiency of our proposal was assessed using the Bot-IoT database, consisting of a mixture of legitimate IoT network traffic and simulated attack scenarios. Our scheme shows superior performance compared to existing ones. Therefore, our approach to IoT intrusion detection can be considered a valuable alternative to existing methods, capable of significantly improving the IoT systems’ security.

Interdisciplinary research approach based on a mixed-methods design to explore patient altruism at the end of life: a study protocol

IntroductionIn the end of life context, patients are often seen as somewhat passive recipients of care provided by health professionals and relatives, with little opportunity to be perceived as autonomous...

Distributed Strategies for Constrained Nonsmooth Resource Allocation Problems With Autonomous Second-Order Agents

Distributed Strategies for Constrained Nonsmooth Resource Allocation Problems With Autonomous Second-Order Agents

Modeling Rationality: Toward Better Performance Against Unknown Agents in Sequential Games.

Opponent modeling is necessary for autonomous agents to capture the intents of others during strategic interactions. Most previous works assume that they can access enough interaction history to build the model. However, it may not be realistic. To solve this problem, we present a novel rationality-consistent opponent modeling (ROM) method for games with imperfect information. In our approach, a game-theoretical concept of consistence about rationality is proposed to take advantage of the characteristic of imperfect information sequential games that rational behavior at disjoint information sets is correlated through anticipated opponent's behavior. With the correlation between different information sets, agents could infer the opponents' strategies at information sets correlated to observed behavior. To exploit the correlation, ROM attempts to conduct reasoning from the opponent's perspective and rationalize its past behavior. In this way, ROM acquires the ability to better adapt to different opponents and achieves a more accurate opponent model with insufficient observation history, which is verified by experiments in different settings. A heuristic adaptation approach is also applied in ROM, which updates the opponent model in an online manner and significantly reduces the computation cost. We evaluate ROM in both a grid world game and a poker game. Compared with other opponent modeling methods, ROM shows better performance and has more accurate predictions in both games against different types of opponents with limited action interactions. Experimental results also show that ROM's time cost is significantly reduced through heuristic adaptation.