Agent Framework Research Articles

Skill matters: Dynamic skill learning for multi-agent cooperative reinforcement learning

With the popularization of intelligence, the necessity of cooperation between intelligent machines makes the research of collaborative multi-agent reinforcement learning (MARL) more extensive. Existing approaches typically address this challenge through task decomposition of the environment or role classification of agents. However, these studies may rely on the sharing of parameters between agents, resulting in the homogeneity of agent behavior, which is not effective for complex tasks. Or training that relies on external rewards is difficult to adapt to scenarios with sparse rewards. Based on the above challenges, in this paper we propose a novel dynamic skill learning (DSL) framework for agents to learn more diverse abilities motivated by internal rewards. Specifically, the DSL has two components: (i) Dynamic skill discovery, which encourages the production of meaningful skills by exploring the environment in an unsupervised manner, using the inner product between a skill vector and a trajectory representation to generate intrinsic rewards. Meanwhile, the Lipschitz constraint of the state representation function is used to ensure the proper trajectory of the learned skills. (ii) Dynamic skill assignment, which utilizes a policy controller to assign skills to each agent based on its different trajectory latent variables. In addition, in order to avoid training instability caused by frequent changes in skill selection, we introduce a regularization term to limit skill switching between adjacent time steps. We thoroughly tested the DSL approach on two challenging benchmarks, StarCraft II and Google Research Football. Experimental results show that compared with strong benchmarks such as QMIX and RODE, DSL effectively improves performance and is more adaptable to difficult collaborative scenarios.

A generic self-learning emotional framework for machines

In nature, intelligent living beings have developed emotions to modulate their behavior as a fundamental evolutionary advantage. However, researchers seeking to endow machines with this advantage lack a clear theory from cognitive neuroscience describing emotional elicitation from first principles, namely, from raw observations to specific affects. As a result, they often rely on case-specific solutions and arbitrary or hard-coded models that fail to generalize well to other agents and tasks. Here we propose that emotions correspond to distinct temporal patterns perceived in crucial values for living beings in their environment (like recent rewards, expected future rewards or anticipated world states) and introduce a fully self-learning emotional framework for Artificial Intelligence agents convincingly associating them with documented natural emotions. Applied in a case study, an artificial neural network trained on unlabeled agent’s experiences successfully learned and identified eight basic emotional patterns that are situationally coherent and reproduce natural emotional dynamics. Validation through an emotional attribution survey, where human observers rated their pleasure-arousal-dominance dimensions, showed high statistical agreement, distinguishability, and strong alignment with experimental psychology accounts. We believe that the framework’s generality and cross-disciplinary language defined, grounded on first principles from Reinforcement Learning, may lay the foundations for further research and applications, leading us toward emotional machines that think and act more like us.

PenGym: Realistic training environment for reinforcement learning pentesting agents

Penetration testing, or pentesting, refers to assessing network system security by trying to identify and exploit any existing vulnerabilities. Reinforcement Learning (RL) has recently become an effective method for creating autonomous pentesting agents. However, RL agents are typically trained in a simulated network environment. This can be challenging when deploying them in a real network infrastructure due to the lack of realism of the simulation-trained agents.In this paper, we present PenGym, a framework for training pentesting RL agents in realistic network environments. The most significant features of PenGym are its support for real pentesting actions, full automation of the network environment creation, and good execution performance. The results of our experiments demonstrated the advantages and effectiveness of using PenGym as a realistic training environment in comparison with a simulation approach (NASim). For the largest scenario, agents trained in the original NASim environment behaved poorly when tested in a real environment, having a high failure rate. In contrast, agents trained in PenGym successfully reached the pentesting goal in all our trials. Even after fixing logical modeling issues in simulation to create the revised version NASim(rev.), experiment results with the largest scenario indicated that agents trained in PenGym slightly outperformed, and were more stable, than those trained in NASim(rev.). Thus, the average number of steps required to reach the pentesting goal was 1.4 to 8 steps better for PenGym. Consequently, PenGym provides a reliable and realistic training environment for pentesting RL agents, eliminating the need to model agent actions via simulation.

Stabilizing financial networks via mergers and acquisitions

A bi-level model is proposed to explore efficient policies for supporting negotiations on financial crisis resolution. In a principal–agent framework, this model minimizes the welfare loss function of a central authority (social planner, SP) through the simultaneous choice of subsidy levels and potential pairs of banks to merge. The SP’s choice of mergers needs to be incentive-compatible with the autonomous choices of banks, and the evaluation of the financial network must adhere to standard accounting principles. Incentive compatibility is enforced by two options for conditions, based on stable matching or competitive bidding. For the evaluation of the financial network, we employ an extended Eisenberg-Noe clearing payment equilibrium by considering bankruptcy costs and the seniority levels of liabilities. Additionally, liabilities are not cleared among solvent banks, and corporate bonds may be used for clearing payments. The bi-level model specifies conditions for the clearing equilibrium. For demonstration, we use major European banks, and a scenario linked to the adverse economic conditions used in the 2016 EU-wide stress testing.

Epistemic Logics for Relevant Reasoners

We present a neighbourhood-style semantic framework for modal epistemic logic modelling agents who process information using relevant logic. The distinguishing feature of the framework in comparison to relevant modal logic is that the environment the agent is situated in is assumed to be a classical possible world. This framework generates two-layered logics combining classical logic on the propositional level with relevant logic in the scope of modal operators. Our main technical result is a general soundness and completeness theorem.

Blockchain-Enabled Provenance and Supply Chain Governance for Indigenous Foods and Botanicals: A Design Approach Study

Ensuring Indigenous producers realize and capture provenance value from the native foods and botanicals supply chain is a key part of achieving economic, community, and ecological sustainability for Indigenous communities. Utilizing blockchain technology to support validated provenance claims throughout supply chain processes is an important intervention toward achieving this objective. This paper presents the preliminary results of an ongoing project in which blockchain technology underpins a “whole of supply chain” approach to addressing issues of provenance value claims and how these are validated within a digitalized environment. The paper focuses on the overarching objectives of achieving provenance value-based growth, and sustainability within a collaborative governance framework that reflects Indigenous community practice. We discuss how technology design and application developments have been undertaken in the context of a cooperative governance model, with the long-term view of enabling ecosystem participants to share responsibility for system development, operations, and benefits. The paper presents a provenance claims approach anchored in a Resources, Events, and Agents (REA) framework. It showcases the first version of a digital application that was developed by engaging a user community. How the application may be applied to other sectors is also briefly explored.

Hybrid platforms with free entry: demand-enhancing activities

AbstractWe study the decision of a platform as to the quantity and quality of the products to sell directly on its own marketplace, where also third-party sellers decide how much to invest in the quality of their products. Using a representative agent framework that is based on a quasi-linear quality-augmented indirect utility function, we show that, under free entry, the quality investments of sellers do not change with platform entry, while the number of joining sellers does. Moreover, contrarily to what is found in the received literature, the platform may go hybrid even in the case it does not enjoy a competitive advantage vis-á-vis third-party sellers. We then study the welfare implications of a platform’s entry decision and show that promoting sellers’ investments and/or contrasting platform entry may lead to a larger as well as a lower consumer welfare. This depends on the platform’s response in equilibrium, both in terms of changes of its quality-enhancing investments and the fee charged on the revenues of third-party sellers.

Cyber–physical collaborative control for DC microgrid clusters under joint cyber-attacks

DC microgrid (DCMG) clusters, as deeply integrated cyber–physical systems (CPSs), are vulnerable to cyber-attacks like false data-injection attacks (FDIAs) and denial-of-service (DoS) attacks. This article proposes a cyber–physical collaborative control method, against the joint attacks mentioned above. The main contribution of this work is as follows: firstly, a collaborative control architecture based on multiagent systems (MASs) is constructed for the CPS; secondly, in the physical layer, a hierarchical control framework for DCMG agents is adopted, including primary, secondary, and tertiary control layers. The secondary and tertiary control layers can adjust the scheme of power flow management and frequency/voltage control, through the interaction information between agents; thirdly, in the cyber layer, an active defense strategy based on layered event-triggered protocol has been proposed: (1) according to the tampering of transmission data/information caused by FDIA, a support vector machine (SVM) based prediction learning compensation method is employed, to ensure the effectiveness of energy management and voltage/frequency control. (2) according to the transmission path interruption caused by DoS attack, the optimal path selection method and path reconstruction method are proposed to maintain the transmission of power flow management information and control information. Simulation results are presented to illustrate the effectiveness of real-time collaborative control for the proposed control method.

Voting with the wallet: a principal–agent framework for the analysis of sustainable supply chains

Voting with the wallet: a principal–agent framework for the analysis of sustainable supply chains

FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design

Recent advancements in Large Language Models (LLMs) have exhibited notable efficacy in question-answering (QA) tasks across diverse domains. Their prowess in integrating extensive web knowledge has fueled interest in developing LLM-based autonomous agents. While LLMs are efficient in decoding human instructions and deriving solutions by holistically processing historical inputs, transitioning to purpose-driven agents requires a supplementary rational architecture to process multi-source information, establish reasoning chains, and prioritize critical tasks. Addressing this, we introduce FinMem, a novel LLM-based agent framework devised for financial decision-making. It encompasses three core modules: Profiling, to customize the agent's characteristics; Memory, with layered message processing, to aid the agent in assimilating hierarchical financial data; and Decision-making, to convert insights gained from memories into investment decisions. Notably, FinMem's memory module aligns closely with the cognitive structure of human traders, offering robust interpretability and real-time tuning. Its adjustable cognitive span allows for the retention of critical information beyond human perceptual limits, thereby enhancing trading outcomes. This framework enables the agent to self-evolve its professional knowledge, react agilely to new investment cues, and continuously refine trading decisions in the volatile financial environment. We first compare FinMem with various algorithmic agents on a scalable real-world financial dataset, underscoring its leading trading performance in stocks. We then fine-tuned the agent's perceptual span and character setting to achieve a significantly enhanced trading performance. Collectively, FinMem presents a cutting-edge LLM agent framework for automated trading, boosting cumulative investment returns.

Designing mascot character based on believable agent framework

Designing mascot character based on believable agent framework

Validation, adaptation and application of a multi‐criteria decision analysis‐based framework for chemotherapeutic agents in Egypt

AbstractHealth technology assessment is a systematic, transparent, robust, and unbiased assessment of healthcare technology that links evidence and decisions. Healthcare decisions usually require trading‐off, and using a structured approach, such as multi‐criteria decision analysis (MCDA), could improve decision quality. This study aimed to adapt an MCDA‐based framework for oncological drug evaluation in an Egyptian clinical setting to explore the perspectives of different stakeholders. The 10th edition of the Evidence and Value: Impact on decision making framework was translated, validated, adapted, and applied to explore key differences between the priorities of different stakeholders for coverage decisions in the oncology field in Egypt. A direct rating scale or hierarchical point allocation was used to elicit criteria weights. Normalized weights were calculated and compared using the preset cut‐off value of 0.008. Understandability, willingness to use the adopted weighting techniques, and cognitive burden were assessed and compared. Ninety participants completed the criteria‐weighting interview for the adapted tool. The elicited criteria weights differed between the two adopted weighting techniques when comparing stakeholders' priorities using the preset cut‐off (0.008). However, there was no significant difference in the weighting technique understandability, future use willingness, and cognitive burden parameters on categorization by either the weighting technique or stakeholder group. This study provides a specially tailored, straightforward procedure for Egyptian stakeholders to evaluate oncologic drugs in a standardized manner by combining various inputs. It also highlights the main variations across Egyptian stakeholder preferences.

From coordination devices to coordination failures: on the changing epistemology of sunspots since the 1970s

The widespread adoption of the rational expectations hypothesis in macroeconomic literature from the 1970s onwards has often overlooked the issue of coordinating individual decisions, particularly regarding expectations. How is common knowledge, essential for rational expectations equilibrium, established? When common knowledge supports multiple equilibria, how do expectations converge to a specific one? These pivotal questions have always been at the core of sunspot literature, which emerged at the turn of the 1980s, albeit sometimes implicitly. While unspots were initially regarded by pioneers David Cass and Karl Shell as essential coordination devices, their epistemological status evolved with the introduction of competing models in ndogenous fluctuations (chaos, bifurcations, deterministic cycles) and the transition from the overlapping generations to the infinitely lived agent framework. Ultimately identified as anticipation shocks, sunspots were portrayed as instability triggers and markers of coordination failures. However, recent advancements in sunspot theory, especially in the areas of financial crises and experimental economics, have started to rehabilitate the original interpretation of sunspots as coordination mechanisms. This article maps out the historical development, shedding light on the evolving concepts surrounding sunspots.

Evaluating Machine Learning Models for Predicting Football Players' Market Values: A Review

In the realm of football management, the accurate assessment of player market values stands as a critical managerial task. Traditional methods reliant on expert opinions often lack transparency and can be prone to inaccuracies. This study proposes a data-driven approach employing machine learning algorithms to determine football players' market values objectively. Utilizing FIFA 20 data from sofifa.com, four regression models - linear regression, multiple linear regression, decision trees, and random forests - were evaluated for their efficacy in estimating market values. This research offers a robust framework for football clubs and player agents to enhance negotiation strategies and make well- informed decisions in the dynamic football transfer market. Key Words: Linear regression, multiple linear regression, decision trees, random forests, machine learning , FIFA video game data.

A conversational agent framework for mental health screening: design, implementation, and usability

ABSTRACT While chatbots show promise for large-scale mental health screening, few offer interactive, free-text conversations, limiting their appeal for self-administered screening and impeding the timely detection of mental health issues. This study introduces an AI-based chatbot that allows users to respond to validated screening surveys for mental disorders (PHQ-9, GAD-7, and PCL-5) in a natural, free-text conversation manner with real-time feedback. The study's objectives include evaluating the chatbot's usability and reducing the frequency of response clarifications while accurately interpreting users’ responses. The system was assessed running in hybrid NLU mode (Phase 2; N = 587; Mage = 21.56, SD = 5.56, 67.8% women) after being trained on data collected while running in rule-based mode (Phase 1; N = 274; Mage = 21.86, SD = 5.50). During user-chatbot interactions, the chatbot required clarification only 4.64% of the time. Using the AI NLU model, the chatbot could understand user responses in 85.65% of cases and interpret free-text similarly to human annotators. In terms of usability, the chatbot in hybrid NLU mode was perceived as more engaging, friendly, and easier to use than in the rule-based NLU mode, which may be indirectly attributed to the enhanced autonomy provided by the AI NLU model.

A computational framework for modeling socio-technical agents in the life-cycle sustainability assessment of supply networks

Supply networks are now more complex and intertwined than ever, and it is necessary to have the capacity of proposing effective policies that consider aspects of complexity and human behavior. To address this complexity, practitioners commonly rely on Agent-Based Modeling (ABM) as modeling paradigm since it permits to analyze unpredictable system properties that arise from the interaction of agents. Nevertheless, current ABM implementations in life-cycle sustainability studies lack of modularity, meaning that practitioners repeat efforts when modeling socio-technical agents. To deal with this issue, a common framework for modeling production and consumption entities is necessary, for which we propose an algebraic framework for representing computational agents. Our framework provides notions and instructions to set an operational configuration that describes socio-technological agents in a systematic manner to then be used in an ABM model. The framework is rooted on principles of Life Cycle Assessment, and it is a bottom-up generalization of the Stochastic Technology-of-Choice Model. The framework's components are conceived to depict the technical and non-technical relationships of an agent's decision space, while also being suited to handle the logic of different decision problems, such as the sourcing problem. To demonstrate the capabilities of our framework, we provide a proof of concept in which we study the effects of introducing agents with sustainable behaviors (Agents of Change) in a system dominated by profit-driven agents. We evaluated two criteria to introduce Agents of Change: strategic and random. The results indicate that a strategic introduction of Agents of Change can be beneficial from a financial and environmental perspective. We show that for a particular strategy, Agents of Change contribute to the reduction of the financial risk of other Agents of Change, while also decreasing the impact of the system. Finally, this proof of concept shows that our framework can be used to address fundamental sustainability inquiries while still being compatible with current life-oriented methods.

The benefits of shared leadership on minister-junior minister delegation and accountability

The relationships between ministers and their junior ministers have hardly been addressed in the literature, yet they are crucial to understanding government departmental dynamics. Using a principal agent framework, this article analyses 144 minister-junior minister relationships in three Portuguese governments (2005–2015), uncovering serious divergences (agency losses). The results were obtained through a qualitative approach, using many sources, including 111 interviews with former Portuguese prime ministers, ministers and junior ministers, content analysis, and one Minister's direct observation. It concludes that sharing the decisional power (shared leadership) improves accountability (promoting preference alignment and reducing asymmetric information), preventing and dealing effectively with divergences.

Semantic agent framework for automated flood assessment using dynamic knowledge graphs

Abstract This article proposes a framework of linked software agents that continuously interact with an underlying knowledge graph to automatically assess the impacts of potential flooding events. It builds on the idea of connected digital twins based on the World Avatar dynamic knowledge graph to create a semantically rich asset of data, knowledge, and computational capabilities accessible to humans, applications, and artificial intelligence. We develop three new ontologies to describe and link environmental measurements and their respective reporting stations, flood events, and their potential impact on population and built infrastructure as well as the built environment of a city itself. These coupled ontologies are deployed to dynamically instantiate near real-time data from multiple fragmented sources into the World Avatar. Sequences of autonomous agents connected via the derived information framework automatically assess consequences of newly instantiated data, such as newly raised flood warnings, and cascade respective updates through the graph to ensure up-to-date insights into the number of people and building stock value at risk. Although we showcase the strength of this technology in the context of flooding, our findings suggest that this system-of-systems approach is a promising solution to build holistic digital twins for various other contexts and use cases to support truly interoperable and smart cities.

The Rational Agent Benchmark for Data Visualization.

Understanding how helpful a visualization is from experimental results is difficult because the observed performance is confounded with aspects of the study design, such as how useful the information that is visualized is for the task. We develop a rational agent framework for designing and interpreting visualization experiments. Our framework conceives two experiments with the same setup: one with behavioral agents (human subjects), and the other one with a hypothetical rational agent. A visualization is evaluated by comparing the expected performance of behavioral agents to that of a rational agent under different assumptions. Using recent visualization decision studies from the literature, we demonstrate how the framework can be used to pre-experimentally evaluate the experiment design by bounding the expected improvement in performance from having access to visualizations, and post-experimentally to deconfound errors of information extraction from errors of optimization, among other analyses.

Social network learning efficiency in the principal–agent relationship

Under the bounded rationality assumption, a principal rarely provides an optimal contract to an agent. Learning from others is one way to improve such a contract. This paper studies the efficiency of social network learning (SNL) in the principal–agent framework. We first introduce the Cobb-Douglas production function into the classic Holmstrom and Milgrom (1987) model with a constant relative risk-averse agent and work out the theoretically optimal contract. Algorithms are then designed to model the SNL process based on profit gaps between contracts in a network of principals. Considering the uncertainty of the agent's labor output, we find that the principals can reach a consensus that tends to result in overcompensation compared to the optimal contract. Then, this study examines how network attributes and model parameters impact learning efficiency and posits several summative hypotheses. The simulation results validate these hypotheses, and we discuss the relevant economic implications of the observed changes in SNL efficiency.