Decisions Of Agents Research Articles

Adaptive urban traffic signal control based on enhanced deep reinforcement learning

One of the focal points in the field of intelligent transportation is the intelligent control of traffic signals (TS), aimed at enhancing the efficiency of urban road networks through specific algorithms. Deep Reinforcement Learning (DRL) algorithms have become mainstream, yet they suffer from inefficient training sample selection, leading to slow convergence. Additionally, enhancing model robustness is crucial for adapting to diverse traffic conditions. Hence, this paper proposes an enhanced method for traffic signal control (TSC) based on DRL. This approach utilizes dueling network and double q-learning to alleviate the overestimation issue of DRL. Additionally, it introduces a priority sampling mechanism to enhance the utilization efficiency of samples in memory. Moreover, noise parameters are integrated into the neural network model during training to bolster its robustness. By representing high-dimensional real-time traffic information as matrices, and employing a phase-cycled action space to guide the decision-making of intelligent agents. Additionally, utilizing a reward function that closely mirrors real-world scenarios to guide model training. Experimental results demonstrate faster convergence and optimal performance in metrics such as queue length and waiting time. Testing experiments further validate the method's robustness across different traffic flow scenarios.

Offloading Strategy Based on Graph Neural Reinforcement Learning in Mobile Edge Computing

In the mobile edge computing (MEC) architecture, base stations with computational capabilities are subject to service coverage limitations, and the mobility of devices leads to dynamic changes in their connections, directly impacting the offloading decisions of agents. The connections between base stations and mobile devices, as well as the connections between base stations themselves, are abstracted into an MEC structural diagram due to the difficulty of deep reinforcement learning (DRL) in capturing the complex relationships between nodes and their multi-order neighboring nodes in the graph; decisions solely generated by DRL have limitations. To address this issue, this study proposes a hierarchical mechanism strategy based on Graph Neural Reinforcement Learning (M-GNRL) under multiple constraints. Specifically, the MEC structural graph constructed with the current device as an observation point aggregates to learn node features, thus comprehensively considering the contextual information of nodes, and the learned graph information serves as the environment for deep reinforcement learning, effectively integrating a graph neural network (GNN) with DRL. In the M-GNRL strategy, edge features from GNN are introduced into the architecture of the DRL network to enhance the accuracy of agents’ decision-making. Additionally, this study proposes an updated algorithm to obtain graph data that change with observation points. Comparative experiments demonstrate that the M-GNRL algorithm outperforms other baseline algorithms in terms of system cost and convergence performance.

Long-short-view aware multi-agent reinforcement learning for signal snippet distillation in delirium movement detection

Automatic movement analysis utilizing surveillance video is believed to be an important and convenient way for timely delirium detection in an Intensive Care Unit (ICU). However, video-based delirium movement detection (DMD) faces inherent challenges: 1) Irregular movements with large differences in the four limbs of a patient; 2) similar movements in delirium and normal situations, and large movement variations between patients experiencing delirium. To address the challenges 1, this paper proposes a Long-Short-View Aware Multi-Agent Reinforcement Learning (LS-MARL) method to identify the most representative movement snippet for DMD, considering that the global state provided by the long-view is important for guiding the agent's decision-making, but ignored in existing MARL methods. The proposed LS-MARL has two novel designs. First, a novel Teacher Auxiliary Policy (TAP) is developed for direction preperception of the representative movement snippet. Second, a new reward mechanism, Team Intrinsic Reward (TIR), is introduced to quantify the contribution of each agent. Experiments demonstrate that the proposed LS-MARL method outperforms state-of-the-art methods. Furthermore, to handle the second challenge, a new Self-Adjusting Ensemble Learning (SAEL) strategy is built to adaptively integrate an optimal classifier combination from multidomain features, which further improves the performance of classification tasks in the proposed LS-MARL method.

Nonconvex distributed feedback optimization for aggregative cooperative robotics

Distributed aggregative optimization is a recently emerged framework in which the agents of a network want to minimize the sum of local objective functions, each one depending on the agent decision variable (e.g., the local position of a team of robots) and an aggregation of all the agents’ variables (e.g., the team barycenter). In this paper, we address a distributed feedback optimization framework in which agents implement a local (distributed) policy to reach a steady-state minimizing an aggregative cost function. We propose AGGREGATIVE TRACKING FEEDBACK, i.e., a novel distributed feedback optimization law in which each agent combines a closed-loop gradient flow with a consensus-based dynamic compensator reconstructing the missing global information. By using tools from system theory, we prove that AGGREGATIVE TRACKING FEEDBACK steers the network to a stationary point of an aggregative optimization problem with (possibly) nonconvex objective function. The effectiveness of the proposed method is validated through numerical simulations on a multi-robot surveillance scenario.

Enhanced image steganalysis through reinforcement learning and generative adversarial networks

With the surging prominence of digital communication platforms, there has been an intensified emphasis on ensuring robust security and privacy measures. Against this backdrop, image steganalysis has emerged as a critical discipline, employing advanced methods to detect clandestine data within image files. At the core of our research is an innovative exploration into image steganalysis using an amalgamation of enhanced reinforcement learning techniques and online data augmentation. This methodology ensures the meticulous identification of concealed data within images. Our design integrates triple parallel dilated convolutions, enabling concurrent extraction of feature vectors from the input images. Once extracted, these vectors are synthesized, paving the way for subsequent classification tasks. To substantiate the efficacy of our approach, we conducted tests on a comprehensive dataset sourced from BossBase 1.01. Furthermore, to discern the influence of transfer learning on our proposed model, the BOWS dataset was employed. Notably, these datasets present a challenge due to its inherent imbalance. To counteract this, we incorporated an advanced Reinforcement Learning (RL) framework. Herein, the dataset samples are envisioned as states in a sequence of interrelated decisions, with the neural network playing the role of the decision-making agent. This agent is then incentivized or reprimanded based on its accuracy in discerning between the minority and majority classes. To bolster our classification capabilities, we innovatively employed data augmentation using images generated by a Generative Adversarial Network (GAN). Concurrently, a regularization mechanism was instituted to alleviate prevalent GAN-related challenges, such as mode collapse and unstable training dynamics. Our experimental outcomes underscore the potency of our methodology. The results highlight a remarkable capability to discern between pristine and steganographic images, registering an average accuracy rate of 85%.

Dynamic vaccination strategies in dual-strain epidemics: A multi-agent-based game-theoretic approach on scale-free hybrid networks

This paper proposes a multi-agent-based game-theoretic framework to explore the dynamics of decision-making among agents in a situation involving a dual-strain epidemic, where the emergence of the second strain is directly related to the first strain. It considers the impact of imperfect vaccination on dual strains within the SIRV framework, in a hybrid network that combines scale-free and grid-based structures. This research presents three models based on the level of decision-making of agents: Individual Decision Making (IDM), Collective Decision Making (CDM), and Stochastic Decision Making (SDM) model. In the IDM model, agents can continuously update their strategies based on the infection and vaccination statuses of their immediate neighbors. The CDM model offers a macro-level approach where agents’ willingness to vaccinate varies uniformly, while the SDM model maintains a constant level of willingness for vaccination across agents. This study demonstrates that agents using the IDM model tend to initiate vaccination earlier in scenarios of high infection rates, effectively mitigating the spread of the disease in subsequent epidemic seasons. However, when both infection and vaccination rates heavily influence decision-making, there is a noticeable decrease in vaccination uptake, thereby aggravating the spread of infection. Additionally, the authors found that in scenarios of high infection, an over reliance on vaccination without adequate initial uptake can jeopardize disease control efforts. Another important finding of this study is that despite the CDM and SDM models showing a quicker emergence of the second strain and reaching equilibrium points faster than the IDM model, the overall infection rate remains lower in the IDM model. This research underscores the complexity of managing dual-strain epidemics through vaccination strategies and highlights the significance of micro-level decision-making in epidemic control.

TWOSOME: An Efficient Online Framework to Align LLMs with Embodied Environments via Reinforcement Learning

Despite the impressive performance across numerous tasks, Large Language Models (LLMs) often fail in solving simple decision-making tasks due to the misalignment of the knowledge in LLMs with environments. On the contrary, Reinforcement Learning (RL) agents learn policies from scratch, which makes them always align with environments but difficult to incorporate prior knowledge for efficient explorations. To narrow the gap, we propose TWOSOME, a novel general online framework that deploys LLMs as decision-making agents to efficiently interact and align with embodied environments via RL without requiring any prepared datasets or prior knowledge of the environments. First, we query the joint probabilities of each valid action with LLMs to form behavior policies. Then, to enhance the stability and robustness of the policies, we propose two normalization methods and summarize four prompt design principles. Finally, we design a novel parameter-efficient training architecture where the actor and critic share one frozen LLM equipped with LOw-Rank Adapters (LoRA) updated by PPO. We conduct extensive experiments to evaluate TWOSOME. (i) TWOSOME exhibits significantly better sample efficiency and performance compared to the conventional RL method, PPO, and prompt tuning method, SayCan, in both classical decision-making environment, Overcooked, and simulated household environment, VirtualHome. (ii) Benefiting from LLMs’ open-vocabulary feature, TWOSOME shows superior generalization ability to unseen tasks. (iii) Under our framework, there is no significant loss of the LLMs’ original ability during online PPO finetuning.a

SeismoNet: A proximal policy optimization-based earthquake early warning system using dilated convolution layers and online data augmentation

In seismic safety, Earthquake Early Warning (EEW) systems are indispensable for mitigating earthquake hazards. These systems strive to quickly evaluate earthquake magnitudes to determine which events warrant immediate alerts. Despite their importance, conventional approaches to rapid earthquake classification face significant hurdles, particularly with the skewed data distribution where instances requiring alerts are far less common. This study proposes an EEW method, called SeismoNet, that leverages a 7-second snapshot of three-dimensional seismic waveforms from the China Earthquake Network Center (CENC). SeismoNet incorporates a trio of dilated convolution layers to distill essential features for effective classification, addressing the critical challenge of data imbalance. Our novel integration of the Off-policy Proximal Policy Optimization (Off-policy PPO) algorithm enriches the model by rewarding accurate classifications, refining its sensitivity to less frequent yet critical seismic events. The training approach is conceptualized as a sequence of decision-making steps, with each seismic sample treated as a unique scenario. The model, acting as a decision-making agent, is rewarded or penalized based on its ability to differentiate between less and more common events. To further enhance classification accuracy, we employ a generative adversarial network (GAN) for dynamic data augmentation and a regularization technique to prevent common pitfalls like mode collapse and training instability. The efficacy of SeismoNet is underscored by its superior performance metrics, including an F-measure of 88.14% and a geometric mean of 90.66%, outperforming existing deep-learning solutions, traditional algorithms, and transfer learning methods. Our exhaustive evaluations, including ablation studies that remove key components like dilated convolutions and PPO, affirm the vital contribution of these elements to SeismoNet’s exceptional capability in EEW. SeismoNet can successfully issue timely warnings for significant seismic events, allowing for essential preventative measures and significantly reducing the potential impact on these communities.

Declarative Logic-Based Pareto-Optimal Agent Decision Making.

There are many applications where an autonomous agent can perform many sets of actions. It must choose one set of actions based on some behavioral constraints on the agent. Past work has used deontic logic to declaratively express such constraints in logic, and developed the concept of a feasible status set (FSS), a set of actions that satisfy these constraints. However, multiple FSSs may exist and an agent needs to choose one in order to act. As there may be many different objective functions to evaluate status sets, we propose the novel concept of Pareto-optimal FSSs or . We show that checking if a status set is a is co-NP-hard. We develop an algorithm to find a and in special cases when the objective functions are monotonic (or anti-monotonic), we further develop more efficient algorithms. Finally, we conduct experiments to show the efficacy of our approach and we discuss possible ways to handle multiple Pareto-optimal Status Sets.

SWOAM: Swarm optimized agents for energy management in grid-interactive connected buildings

With increasing complexity of smart city energy systems and rising energy demand, effective energy management solutions are crucial. Buildings now incorporate renewable energy sources and battery storage for efficient energy utilization, making optimal control strategies important. Compared to rule-based controllers and model-based methods, swarm and evolutionary algorithms have the advantages of providing cost-effective, stable, and scalable alternatives. However, their potential in data-rich environments and multi-building energy systems remains underexplored. This research bridges this gap by demonstrating the cooperative capabilities of population-based optimization agents for efficient energy management. Specifically, novel algorithm and framework, named Swarm Optimized Agents for Sequential Decision Making (SWOAM) is proposed. It combines an online k-means classifier and a swarm optimizer for optimal control of multi-building energy systems. By designing an online K-means learning strategy and an agent-based sequential episodic sampling approach, it is feasible to train a swarm of agents find an optimal energy management policy for buildings in a district. For training and evaluation, we use the standardized CityLearn building energy management environment. The agents are evaluated on three key metrics: electricity cost, carbon dioxide emissions and grid ramping. SWOAM delivers state-of-the-art performance and outperforms modern reinforcement learning and rule-based controllers.

Crowd evacuation simulation based on hierarchical agent model and physics‐based character control

AbstractCrowd evacuation has gained increasing attention in recent years. The agent‐based method has shown a superior capability to simulate complex behaviors during crowd evacuation simulation. For agent modeling, most existing methods only consider the decision process but ignore the detailed physical motion. In this article, we propose a hierarchical framework for crowd evacuation simulation, which combines the agent decision model with the agent motion model. In the decision model, we integrate emotional contagion and scene information to determine global path planning and local collision avoidance. In the motion model, we introduce a physics‐based character control method and control agent motion using deep reinforcement learning. Based on the decision strategy, the decision model can use a signal to control the agent motion in the motion model. Compared with existing methods, our framework can simulate physical interactions between agents and the environment. The results of the crowd evacuation simulation demonstrate that our framework can simulate crowd evacuation with physical fidelity.

Children’s interests and early childhood curriculum: A critical analysis of the relationship between research, policy, and practice

Policy makers have a powerful influence on educational practice. When such bodies are vague about the evidence-base for their policies they may uncritically rely on outdated theories, beliefs, and selective research evidence. A tension may then exist where practitioners become undermined as agents in curricular decision-making. Practitioners may aim to provide curriculum and pedagogy aligned with contemporary knowledge, but are also bound to the policy bodies who hold persuasive power. In England and Aotearoa New Zealand, two particular organisations in each country have most influence on early childhood education. Focused on the notion of children’s interests, this article questions the basis for the key curricular policy, accompanying advice and guidance, and evaluation standards of these organisations. We do so having discussed children’s interests from historical and contemporary research perspectives. We then trace and critique ways children’s interests present in significant policy documents. We suggest that both policy and practice adopt contemporary perspectives of children’s interests and move towards a middle space between curriculum-as-plan and curriculum-as-lived (Aoki, 2005). Such a space provides a way forward for ongoing curriculum conversations about children’s interests.

Religiosity predicts the delegation of decisions between moral and self-serving immoral outcomes

Studies support an association between religious belief and prosocial behavior. Such religious prosociality has been attributed to fear of supernatural punishment and enhanced concern for a prosocial reputation and self-image. Hence, religious individuals may be more prone to pursue their self-interest indirectly, thereby averting personal responsibility. We conducted 12 studies (Ntotal = 4468) to examine whether religiosity predicts delegation in incentivized deception, dictator, and die-roll cheating games and in realistic scenarios. Participants could choose between an immoral (e.g., lying) and a moral, prosocial (e.g., honest/fair) option or leave this decision to another individual (the agent) who equally benefited from the immoral option. Across all studies, religiosity positively predicted delegation, even though participants could directly implement prosocial outcomes. Employing experimental manipulations of participants' interests, we found that the predictive effect of religiosity on delegation only emerged when participants could expect to benefit from the agent's decision, but not when they were not affected by it or could be harmed by it. At the same time, religiosity predicted prosocial decisions among non-delegating participants. Moreover, delegating participants felt less bad and responsible about their decisions and victims' outcomes. Taken together, these findings suggest that delegation is strategically employed by individuals who would otherwise act prosocially to pursue selfish interests while avoiding responsibility and blame. They further support the notion of religious prosociality as a multi-faceted, context-dependent phenomenon.

An autonomous decision-making agent for offshore wind turbine blades under leading edge erosion

The increasing pressure of offshore wind developments is leading to projects being located in areas with more difficult access and greater weather barriers. As these constraints increase, O&M costs also grow in importance. Therefore, the current scenario requires a careful planning to avoid unnecessary costly maintenance decisions or unexpected failures. To overcome the problem of increasing O&M costs and difficult access, this manuscript presents an autonomous decision-making Reinforcement Learning (RL) agent to improve O&M planning for the Leading Edge Erosion (LEE) problem. The method developed in this work makes use of a linear degradation model to account for the damage progression dynamics and site-specific weather models. The RL-based agent proposed in this manuscript is able to reduce expected O&M costs in the range of 12%–21% when compared with condition-based policies.

Bayesian Reinforcement Learning With Limited Cognitive Load.

All biological and artificial agents must act given limits on their ability to acquire and process information. As such, a general theory of adaptive behavior should be able to account for the complex interactions between an agent's learning history, decisions, and capacity constraints. Recent work in computer science has begun to clarify the principles that shape these dynamics by bridging ideas from reinforcement learning, Bayesian decision-making, and rate-distortion theory. This body of work provides an account of capacity-limited Bayesian reinforcement learning, a unifying normative framework for modeling the effect of processing constraints on learning and action selection. Here, we provide an accessible review of recent algorithms and theoretical results in this setting, paying special attention to how these ideas can be applied to studying questions in the cognitive and behavioral sciences.

Adaptive mean field multi-agent reinforcement learning

Large-scale Multi-Agent Reinforcement Learning (MARL) is fundamentally a challenge due to the curse of dimensionality. In a homogeneous multi-agent setting, mean field theory gives an effective way of scalable MARL by abstracting other agents to a virtual mean agent, assuming that the influence between agents is equal and infinitesimal. However, in some real scenarios, only several neighboring agents, rather than all agents, affect the decision-making of an agent, and different neighboring agents may have varying degrees of influence on the agent's decision-making. In this paper, not restricted to a homogeneous setting, we propose adaptive mean field MARL, which is based on the attention mechanism and can be used to deal with many-agent scenarios where there may be different influence relationships among agents. Specifically, we first derive the mean field approximation with adaptive weight and give the error bound of the approximation. Then, we propose adaptive mean field Q-Learning and describe how to obtain the adaptive weight. In addition, we discuss the differences between the proposed approach and existing mean-field MARL methods. Finally, we conduct experiments on simulation platforms, and the results show that the performance of the proposed approach outperforms that of the state-of-the-art method.

Generalizable Policy Improvement via Reinforcement Sampling (Student Abstract)

Current policy gradient techniques excel in refining policies over sampled states but falter when generalizing to unseen states. To address this, we introduce Reinforcement Sampling (RS), a novel method leveraging a generalizable action value function to sample improved decisions. RS is able to improve the decision quality whenever the action value estimation is accurate. It works by improving the agent's decision on the fly on the states the agent is visiting. Compared with the historically experienced states in which conventional policy gradient methods improve the policy, the currently visited states are more relevant to the agent. Our method sufficiently exploits the generalizability of the value function on unseen states and sheds new light on the future development of generalizable reinforcement learning.

Generalising Planning Environment Redesign

In Environment Design, one interested party seeks to affect another agent's decisions by applying changes to the environment. Most research on planning environment (re)design assumes the interested party's objective is to facilitate the recognition of goals and plans, and search over the space of environment modifications to find the minimal set of changes that simplify those tasks and optimise a particular metric. This search space is usually intractable, so existing approaches devise metric-dependent pruning techniques for performing search more efficiently. This results in approaches that are not able to generalise across different objectives and/or metrics. In this paper, we argue that the interested party could have objectives and metrics that are not necessarily related to recognising agents' goals or plans. Thus, to generalise the task of Planning Environment Redesign, we develop a general environment redesign approach that is metric-agnostic and leverages recent research on top-quality planning to efficiently redesign planning environments according to any interested party's objective and metric. Experiments over a set of environment redesign benchmarks show that our general approach outperforms existing approaches when using well-known metrics, such as facilitating the recognition of goals, as well as its effectiveness when solving environment redesign tasks that optimise a novel set of different metrics.

Autonomous Policy Explanations for Effective Human-Machine Teaming

Policy explanation, a process for describing the behavior of an autonomous system, plays a crucial role in effectively conveying an agent's decision-making rationale to human collaborators and is essential for safe real-world deployments. It becomes even more critical in effective human-robot teaming, where good communication allows teams to adapt and improvise successfully during uncertain situations by enabling value alignment within the teams. This thesis proposal focuses on improving human-machine teaming by developing novel human-centered explainable AI (xAI) techniques that empower autonomous agents to communicate their capabilities and limitations via multiple modalities, teach and influence human teammates' behavior as decision-support systems, and effectively build and manage trust in HRI systems.

Memory Asymmetry Creates Heteroclinic Orbits to Nash Equilibrium in Learning in Zero-Sum Games

Learning in games considers how multiple agents maximize their own rewards through repeated games. Memory, an ability that an agent changes his/her action depending on the history of actions in previous games, is often introduced into learning to explore more clever strategies and discuss the decision-making of real agents like humans. However, such games with memory are hard to analyze because they exhibit complex phenomena like chaotic dynamics or divergence from Nash equilibrium. In particular, how asymmetry in memory capacities between agents affects learning in games is still unclear. In response, this study formulates a gradient ascent algorithm in games with asymmetry memory capacities. To obtain theoretical insights into learning dynamics, we first consider a simple case of zero-sum games. We observe complex behavior, where learning dynamics draw a heteroclinic connection from unstable fixed points to stable ones. Despite this complexity, we analyze learning dynamics and prove local convergence to these stable fixed points, i.e., the Nash equilibria. We identify the mechanism driving this convergence: an agent with a longer memory learns to exploit the other, which in turn endows the other's utility function with strict concavity. We further numerically observe such convergence in various initial strategies, action numbers, and memory lengths. This study reveals a novel phenomenon due to memory asymmetry, providing fundamental strides in learning in games and new insights into computing equilibria.