Autonomous Agents Research Articles

Reinforcement Learning-Based Formation Pinning and Shape Transformation for Swarms

Swarm models hold significant importance as they provide the collective behavior of self-organized systems. Boids model is a fundamental framework for studying emergent behavior in swarms systems. It addresses problems related to simulating the emergent behavior of autonomous agents, such as alignment, cohesion, and repulsion, to imitate natural flocking movements. However, traditional models of Boids often lack pinning and the adaptability to quickly adapt to the dynamic environment. To address this limitation, we introduce reinforcement learning into the framework of Boids to solve the problem of disorder and the lack of pinning. The aim of this approach is to enable drone swarms to quickly and effectively adapt to dynamic external environments. We propose a method based on the Q-learning network to improve the cohesion and repulsion parameters in the Boids model to achieve continuous obstacle avoidance and maximize spatial coverage in the simulation scenario. Additionally, we introduce a virtual leader to provide pinning and coordination stability, reflecting the leadership and coordination seen in drone swarms. To validate the effectiveness of this method, we demonstrate the model’s capabilities through empirical experiments with drone swarms, and show the practicality of the RL-Boids framework.

Autonomous Systems and Technology Resistance: New Tools for Monitoring Acceptance, Trust, and Tolerance

We introduce the notion of Tolerance for autonomous artificial agents (and its antithetical concept, Intolerance), motivating its theoretical adoption in the fields of social robotics and human—agent interaction, where it can effectively complement two contiguous, but essentially distinct, constructs—Acceptance and Trust—that are broadly used by researchers. We offer a comprehensive conceptual model of Tolerance, construed as a user’s insusceptibility or resilience to Autonomy Estrangement (i.e., the uncanny sense of isolation and displacement experienced by the humans who believe, for right or wrong reasons, that robots can subvert and/or control their lives). We also refer to Intolerance to indicate the opposite property, that is the user’s susceptibility or proneness to Autonomy Estrangement. Thus, Tolerance and Intolerance are inverse representations of the same phenomenological continuum, with Intolerance increasing when Tolerance decreases and vice versa. While Acceptance and Trust measure how the user’s interaction with a particular robot is satisfying and efficacious, the dyad Tolerance/Intolerance reflects how the user’s attitude is affected by deeply held normative beliefs about robots in general. So defined, a low Tolerance (that is a high Intolerance) is expected to correlate to antagonistic responses toward the prospect of adoption: specifically, Intolerant attitudes predict the kind of anxious and hostile behaviours toward Agents that originate from the concerns that autonomous systems could deeply disrupt the lives of humans (affecting their work cultures, ways of living, systems of values, etc.) or dominate them (making humans redundant, undermining their authority, threatening their uniqueness, etc.). Thus, Negative beliefs and worldviews about Agents are the cause of the Intolerant attitude toward Agents, which predicts Autonomy Estrangement, which in turn correlates to low Adoption Propensity and avoidance and rejection behaviours.

Language support for verifying reconfigurable interacting systems

Reconfigurable interacting systems consist of a set of autonomous agents, with integrated interaction capabilities that feature opportunistic interaction. Agents seemingly reconfigure their interaction interfaces by forming collectives and interact based on mutual interests. Finding ways to design and analyse the behaviour of these systems is a vigorously pursued research goal. In this article, we provide a modelling and analysis environment for the design of such system. Our tool offers simulation and verification to facilitate native reasoning about the domain concepts of such systems. We present our tool named R-CHECK (please find the associated toolkit repository here: https://github.com/dsynma/recipe). R-CHECK supports a high-level input language with matching enumerative and symbolic semantics and provides modelling convenience for features such as reconfiguration, coalition formation, and self-organisation. For analysis, users can simulate the designed system and explore arising traces. Our included model checker permits reasoning about interaction protocols and joint missions.

Learning inverse kinematics using neural computational primitives on neuromorphic hardware

Current low-latency neuromorphic processing systems hold great potential for developing autonomous artificial agents. However, the variable nature and low precision of the underlying hardware substrate pose severe challenges for robust and reliable performance. To address these challenges, we adopt hardware-friendly processing strategies based on brain-inspired computational primitives, such as triplet spike-timing dependent plasticity, basal ganglia-inspired disinhibition, and cooperative-competitive networks and apply them to motor control. We demonstrate this approach by presenting an example of robust online motor control using a hardware spiking neural network implemented on a mixed-signal neuromorphic processor, trained to learn the inverse kinematics of a two-joint robotic arm. The final system is able to perform low-latency control robustly and reliably using noisy silicon neurons. The spiking neural network, trained to control two joints of the iCub robot arm simulator, performs a continuous target-reaching task with 97.93% accuracy, 33.96 ms network latency, 102.1 ms system latency, and with an estimated power consumption of 26.92 μW during inference (control). This work provides insights into how specific computational primitives used by real neural systems can be applied to neuromorphic computing for solving real-world engineering tasks. It represents a milestone in the design of end-to-end spiking robotic control systems, relying on event-driven sensory encoding, neuromorphic processing, and spiking motor control.

Task-Based Visual Attention for Continually Improving the Performance of Autonomous Game Agents

Deep Reinforcement Learning (DRL) has been effectively performed in various complex environments, such as playing video games. In many game environments, DeepMind’s baseline Deep Q-Network (DQN) game agents performed at a level comparable to that of humans. However, these DRL models require many experience samples to learn and lack the adaptability to changes in the environment and handling complexity. In this study, we propose Attention-Augmented Deep Q-Network (AADQN) by incorporating a combined top-down and bottom-up attention mechanism into the DQN game agent to highlight task-relevant features of input. Our AADQN model uses a particle-filter -based top-down attention that dynamically teaches an agent how to play a game by focusing on the most task-related information. In the evaluation of our agent’s performance across eight games in the Atari 2600 domain, which vary in complexity, we demonstrate that our model surpasses the baseline DQN agent. Notably, our model can achieve greater flexibility and higher scores at a reduced number of time steps.Across eight game environments, AADQN achieved an average relative improvement of 134.93%. Pong and Breakout games both experienced improvements of 9.32% and 56.06%, respectively. Meanwhile, SpaceInvaders and Seaquest, which are more intricate games, demonstrated even higher percentage improvements, with 130.84% and 149.95%, respectively. This study reveals that AADQN is productive for complex environments and produces slightly better results in elementary contexts.

"A place without walls, only opportunities": personal accounts of attending Recovery Colleges in Norway.

Recovery colleges (RCs) are learning environments, first established in the UK, based on principles that support positive life changes and reduce stigma related to challenges with mental health and substance use problems. RCs offer courses based on co-production processes and are designed and delivered jointly by individuals with lived experience and professional experts. The courses are open to anyone, attracting people with a variety of life experiences. RCs are non-clinical environments that provide individuals with the identities of students and/or trainers as autonomous and independent agents. In this paper, we investigate experiences of being a part of a RC in Norway, either as a student and/or as a course trainer with lived experiences of mental health or substance use challenges. We ask the following research question: What kinds of personal and social processes are enabled by being part of a recovery college from the perspective of persons with experience-based competence? The study is qualitative and explorative based on 11 individual (N = 11) and two focus group interviews (N = 8). Participants were recruited from two of the first RCs in Norway between August 2021 and January 2022. Study participants describe how their involvement in a RC provided them with opportunities to assume new positions in their recovery process, both related to former institutional identities given in the course of treatment and related to the way they see themselves as people struggling with mental health and substances use challenges. Attending a RC represented significant transitions (1) from an institutional position as "sick" or as "what's on the paper" into a position as "a whole person"; (2) from being in in a position as a recipient of care to the position as actively responsible for life changes; (3) from seeing themselves as worthless to seeing themselves as persons with resources; (4) from being alone to being part of a fellowship. Participants describe being part of a RC as an invaluable addition to other kind of support or help. It is important to provide alternative arenas like RC for facilitating work with life changes, as an invaluable addition to regular services.

Reinforcement Learning as a Path to Autonomous Intelligent Cyber-Defense Agents in Vehicle Platforms

Technological advancement of vehicle platforms exposes opportunities for new attack paths and vulnerabilities. Static cyber defenses can help mitigate certain attacks, but those attacks must generally be known ahead of time, and the cyber defenses must be hand-crafted by experts. This research explores reinforcement learning (RL) as a path to achieve autonomous, intelligent cyber defense of vehicle control networks—namely, the controller area network (CAN) bus. We train an RL agent for the CAN bus using Toyota’s Portable Automotive Security Testbed with Adaptability (PASTA). We then apply the U.S. Army Combat Capabilities Development Command (DEVCOM) Army Research Laboratory’s methodology for quantitative measurement of cyber resilience to assess the agent’s effect on the vehicle testbed in a contested cyberspace environment. Despite all defenses having similar traditional performance measures, our RL agent averaged a 90% cyber resilience measurement during drive cycles executed on hardware versus 41% for a naïve static timing defense and 98% for the bespoke timing-based defense. Our results also show that an RL-based agent can detect and block injection attacks on a vehicle CAN bus in a laboratory environment with greater cyber resilience than prior learning approaches (1% for convolutional networks and 0% for recurrent networks). With further research, we believe there is potential for using RL in the autonomous intelligent cyber defense agent concept.

Synthesizing Sentience: Integrating Large Language Models and Autonomous Agents for Emulating Human Cognitive Complexity

Synthesizing Sentience: Integrating Large Language Models and Autonomous Agents for Emulating Human Cognitive Complexity

Facilitating cooperation in human-agent hybrid populations through autonomous agents

Cooperative AI has shown its effectiveness in solving the conundrum of cooperation. Understanding how cooperation emerges in human-agent hybrid populations is a topic of significant interest, particularly in the realm of evolutionary game theory. In this article, we scrutinize how cooperative and defective Autonomous Agents (AAs) influence human cooperation in social dilemma games with a one-shot setting. Focusing on well-mixed populations, we find that cooperative AAs have a limited impact in the prisoner's dilemma games but facilitate cooperation in the stag hunt games. Surprisingly, defective AAs can promote complete dominance of cooperation in the snowdrift games. As the proportion of AAs increases, both cooperative and defective AAs have the potential to cause human cooperation to disappear. We then extend our investigation to consider the pairwise comparison rule and complex networks, elucidating that imitation strength and population structure are critical for the emergence of human cooperation in human-agent hybrid populations.

The cell as secret agent—autonomy and intelligence of the living cell: driving force of development

The cell as secret agent—autonomy and intelligence of the living cell: driving force of development

An agent-based simulation and logistics optimization model for managing uncertain demand in forest supply chains

This paper aims to model and minimize transportation costs in collecting tree logs from several regions and delivering them to the nearest collection point. This paper presents agent-based modeling (ABM) that comprehensively encompasses the key elements of the pickup and delivery supply chain model and presents the units as autonomous agents communicating. The modeling combines components such as geographic information systems (GIS) routing, potential facility locations, random tree log pickup locations, fleet sizing, trip distance, and truck and train transportation. ABM models the entire pickup and delivery operation, and modeling outcomes are presented by time series charts such as the number of trucks in use, facilities inventory, and travel distance. In addition, various simulation scenarios are used to investigate potential facility locations and truck numbers and determine the optimal facility location and fleet size.

Virtual Reality Interface Evaluation for Earthwork Teleoperation

Automation and robotics are destined to play a critical role in the Industry 4.0 revolution, as illustrated by the emergence of autonomous machinery in earthwork operations. Despite rapid progress, autonomous agents will always require human supervision to instruct their mission and to guaranty safety when unexpected problems arise. Traditional human supervision requires an operator to physically enter each machine at risk and manually take control. This approach is time-consuming and requires highly qualified personnel capable of operating various machines. This process can be hastened and simplified by means of teleoperated supervision, which itself requires the appropriate interface. In this paper we evaluate a virtual reality (VR)-based interface using hybrid interactions and an immersive digital-twin compared to a real-life control. We compare these interfaces through control tasks performed by expert and non-expert operators, analyzing time and precision, as well as user feedback. The preliminary results show that the VR interface brings equivalent and satisfactory performances for experts and improves the efficiency of apprentices. Therefore, not only does everyone performs well in the virtual environment, but also the training time can be shortened significantly as non-experts can perform similarly under the same conditions.

Navigation in Adversarial Environments Guided by PRA* and a Local RL Planner

Real-time strategy games require players to respond to short-term challenges (micromanagement) and long-term objectives (macromanagement) simultaneously to win. However, many players excel at one of these skills but not both. This research is motivated by the question of whether the burden of micromanagement can be reduced on human players through delegation of responsibility to autonomous agents. In particular, this research proposes an adversarial navigation architecture that enables units to autonomously navigate through places densely populated with enemies by learning to micromanage itself. Our approach models the adversarial pathfinding problem as a Markov Decision Process (MDP) and trains an agent with reinforcement learning on this MDP. We observed that our approach resulted in the agent taking less damage from adversaries while travelling shorter paths, compared to previous approaches for adversarial navigation. Our approach is also efficient in memory use and computation time. Interestingly, the agent using the proposed approach also outperformed baseline approaches while navigating through environments that are significantly different from the training environments. Furthermore, when the game design is modified, the agent discovers effective alternate strategies considering the updated design without any changes in the learning framework. This property is particularly useful because in game development the design of a game is often updated iteratively.

Model-Based Navigation Control for Communication-Aware Motion Using Harmonic Potential Fields

This paper suggests an optimal and provably-correct method for inducing communication-aware, context-sensitive mobility in an autonomous agent. The method can fuse crisp representations of arbitrarily shaped hard-obstacles with soft representations of the wireless signal strength field that forms within the confines of the hard-obstacles. The procedure can produce a navigation control signal able to induce in a wide class of realistic agents optimal, communication-aware, dynamically-friendly trajectories to a specified target point in the environment. The approach is developed with the aid of the potential-field method for motion planning. Its immediate goal is to tackle the model-based case where the navigator assumes full knowledge of the obstacles and Wireless Communication Link (WCL). The treatment in this part provides the theoretical foundations and the tools necessary to extend the method to the sensor-based case where the agent does not a-priori know the obstacles or the WCL which have to be sensed online. Theoretical proofs of the communication-aware navigator’s performance are provided and its behavior is verified using intensive realistic simulation. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —In this paper we target applications like search and rescue, where mobile agents may be utilized for information gathering and mapping purposes in regions that are hazardous to humans. Our proposed approach can simultaneously combine obstacle information and signal strength information to generate guidance signals. Trajectories manifested using these guidance signals can avoid regions with very low signal strength. Typically, guidance signals from a planner need to be smoothed before generating actuation signals. Our proposed planner provides guidance signal that can be directly used to generate actuation signals. Also the proposed approach results in trajectories that are very smooth (infinitely differentiable), therefore, energy consumption due to motion is reduced. Simulations show the effectiveness of our approach. In future work, we intend to use real-time signal strength and obstacle information data in out approach.

SWARM INTELLIGENCE EMBEDDED DATA MINING FOR PRECISION AGRICULTURE ADVANCEMENTS

The present study investigates the potential of Swarm Intelligence (SI) in driving breakthroughs in Precision Agriculture (PA). It focuses on the research of mining techniques to uncover novel insights and developments in the field of PA. Social informatics (SI) is an academic discipline that focuses on the examination of collective behaviour within both herbal and synthetic structures. In order to gather, analyse, and synthesise information, SI utilises self-sufficient mobile devices known as Autonomous Mobile Agents (AMAs). These entities refer to robotic and computational frameworks that engage in mutual interaction, facilitating the examination of collective intelligence. This essay examines the potential impact of utilising the System of International Units (SI) on enhancing the accuracy and precision of commodity production and control in the field of production agriculture (PA). It also highlights the existing advancements that have been achieved in this regard. This analysis examines possible uses of Swarm Intelligence in the Public Administration (PA) industry, as well as the challenges that need to be solved in order to enhance the efficiency and accuracy of PA operations.

Automated and Efficient Test-Generation for Grid-Based Multiagent Systems

Automatic generation of random test inputs is an approach that can alleviate the challenges of manual test case design. However, random test cases may be ineffective in fault detection and increase testing cost, especially in systems where test execution is resource- and time-consuming. To remedy this, the domain knowledge of test engineers can be exploited to select potentially effective test cases. To this end, test selection constraints suggested by domain experts can be utilized either for filtering randomly generated test inputs or for direct generation of inputs using constraint solvers. In this paper, we propose a domain specific language (DSL) for formalizing locality-based test selection constraints of autonomous agents and discuss the impact of test selection filters, specified in our DSL, on randomly generated test cases. We study and compare the performance of filtering and constraint solving approaches in generating selective test cases for different test scenario parameters and discuss the role of these parameters in test generation performance. Through our study, we provide criteria for suitability of the random data filtering approach versus the constraint solving one under the varying size and complexity of our testing problem. We formulate the corresponding research questions and answer them by designing and conducting experiments using QuickCheck for random test data generation with filtering and Z3 for constraint solving. Our observations and statistical analysis indicate that applying filters can significantly improve test efficiency of randomly generated test cases. Furthermore, we observe that test scenario parameters affect the performance of the filtering and constraint solving approaches differently. In particular, our results indicate that the two approaches have complementary strengths: random generation and filtering works best for large agent numbers and long paths, while its performance degrades in the larger grid sizes and more strict constraints. On the contrary, constraint solving has a robust performance for large grid sizes and strict constraints, while its performance degrades with more agents and long paths.

On convergence results for nonlinear cyclic pursuit strategies

The paper focuses on the generation of a circular formation centred at a target point using a homogeneous group of autonomous agents in a cyclic pursuit framework. The agents are assumed to have a severely under-actuated form of the unicycle kinematics, with the linear speeds being constant and the angular speeds being their only control inputs. The target is assumed to be stationary and can represent a landmark, beacon, etc. In order to achieve the desired formation, the controller design is carried out in two stages. In the first stage, a bearing based attractive control law is proposed for every agent. Under the proposed law, the analysis of the relative motions reveals interesting behaviours like gathering and scattering of the agents. Building upon the attractive control law, in the second stage, the appropriate values of the controller parameters are determined which lead to a target centric circular formation. These parameters can also be suitably adjusted to achieve a pre-specified value of the radius of the circle and a desired distribution of the agents along the circle. The highlight of the work is that the convergence to the desired formation is guaranteed from almost all initial conditions of the agents by controlling only their angular speeds. Simulation results are also presented to illustrate the proposed theoretical results.

An Advisor-Based Architecture for a Sample-Efficient Training of Autonomous Navigation Agents with Reinforcement Learning

Recent advancements in artificial intelligence have enabled reinforcement learning (RL) agents to exceed human-level performance in various gaming tasks. However, despite the state-of-the-art performance demonstrated by model-free RL algorithms, they suffer from high sample complexity. Hence, it is uncommon to find their applications in robotics, autonomous navigation, and self-driving, as gathering many samples is impractical in real-world hardware systems. Therefore, developing sample-efficient learning algorithms for RL agents is crucial in deploying them in real-world tasks without sacrificing performance. This paper presents an advisor-based learning algorithm, incorporating prior knowledge into the training by modifying the deep deterministic policy gradient algorithm to reduce the sample complexity. Also, we propose an effective method of employing an advisor in data collection to train autonomous navigation agents to maneuver physical platforms, minimizing the risk of collision. We analyze the performance of our methods with the support of simulation and physical experimental setups. Experiments reveal that incorporating an advisor into the training phase significantly reduces the sample complexity without compromising the agent’s performance compared to various benchmark approaches. Also, they show that the advisor’s constant involvement in the data collection process diminishes the agent’s performance, while the limited involvement makes training more effective.

The MAS4AI framework for human-centered agile and smart manufacturing.

Volatility and uncertainty of today's value chains along with the market's demands for low-batch customized products mandate production systems to become smarter and more resilient, dynamically and even autonomously adapting to both external and internal disturbances. Such resilient behavior can be partially enabled by highly interconnected Cyber-Physical Production Systems (CPPS) incorporating advanced Artificial Intelligence (AI) technologies. Multi-agent solutions can provide better planning and control, improving flexibility and responsiveness in production systems. Small modular parts can autonomously take intelligent decisions and react to local events. The main goal of decentralization and interconnectivity is to enable autonomous and cooperative decision-making. Nevertheless, a more efficient orchestration of various AI components and deeper human integration are required. In addition, global behaviors of coalitions of autonomous agents are not easily comprehensible by workers. Furthermore, it is challenging to implement an Industry 4.0 paradigm where a human should be in charge of decision-making and execution. This paper discusses a Multi-Agent System (MAS) where several software agents cooperate with smart workers to enable a dynamic and reconfigurable production paradigm. Asset Administration Shell (AAS) submodels hold smart workers' descriptions in machine-readable format, serving as an integration layer between various system's components. The self-description capability of the AAS supports the system's adaptability and self-configuration. The proposed concept supports the plug-and-produce functionality of the production modules and improves human-machine integration in the shared assembly tasks.

Three-Dimensional Multi-Agent Foraging Strategy Based on Local Interaction.

This paper considers a multi-agent foraging problem, where multiple autonomous agents find resources (called pucks) in a bounded workspace and carry the found resources to a designated location, called the base. This article considers the case where autonomous agents move in unknown 3-D workspace with many obstacles. This article describes 3-D multi-agent foraging based on local interaction, which does not rely on global localization of an agent. This paper proposes a 3-D foraging strategy which has the following two steps. The first step is to detect all pucks inside the 3-D cluttered unknown workspace, such that every puck in the workspace is detected in a provably complete manner. The next step is to generate a path from the base to every puck, followed by collecting every puck to the base. Since an agent cannot use global localization, each agent depends on local interaction to bring every puck to the base. In this article, every agent on a path to a puck is used for guiding an agent to reach the puck and to bring the puck to the base. To the best of our knowledge, this article is novel in letting multiple agents perform foraging and puck carrying in 3-D cluttered unknown workspace, while not relying on global localization of an agent. In addition, the proposed search strategy is provably complete in detecting all pucks in the 3-D cluttered bounded workspace. MATLAB simulations demonstrate the outperformance of the proposed multi-agent foraging strategy in 3-D cluttered workspace.