Subordinate Agents Research Articles

Exploring the limits of hierarchical world models in reinforcement learning.

Hierarchical model-based reinforcement learning (HMBRL) aims to combine the sample efficiency of model-based reinforcement learning with the abstraction capability of hierarchical reinforcement learning. While HMBRL has great potential, the structural and conceptual complexities of current approaches make it challenging to extract general principles, hindering understanding and adaptation to new use cases, and thereby impeding the overall progress of the field. In this work we describe a novel HMBRL framework and evaluate it thoroughly. We construct hierarchical world models that simulate the environment at various levels of temporal abstraction. These models are used to train a stack of agents that communicate top-down by proposing goals to their subordinate agents. A significant focus of this study is the exploration of a static and environment agnostic temporal abstraction, which allows concurrent training of models and agents throughout the hierarchy. Unlike most goal-conditioned H(MB)RL approaches, it also leads to comparatively low dimensional abstract actions. Although our HMBRL approach did not outperform traditional methods in terms of final episode returns, it successfully facilitated decision-making across two levels of abstraction. A central challenge in enhancing our method's performance, as uncovered through comprehensive experimentation, is model exploitation on the abstract level of our world model stack. We provide an in depth examination of this issue, discussing its implications and suggesting directions for future research to overcome this challenge. By sharing these findings, we aim to contribute to the broader discourse on refining HMBRL methodologies.

Leveraging Organizational Hierarchy to Simplify Reward Design in Cooperative Multi-agent Reinforcement Learning

The effectiveness of multi-agent reinforcement learning (MARL) hinges largely on the meticulous arrangement of objectives. Yet, conventional MARL methods might not completely harness the inherent structures present in environmental states and agent relationships for goal organization. This study is conducted within the domain of military training simulations, which are typically characterized by complexity, heterogeneity, non-stationary and doctrine-driven environments with a clear organizational hierarchy and a top-down chain of command. This research investigates the approximation and integration of the organizational hierarchy into MARL for cooperative training scenarios, with the goal of streamlining the processes of reward engineering and enhancing team coordination. In the preliminary experiments, we employed two-tiered commander-subordinate feudal hierarchical (CSFH) networks to separate the prioritized team goal and individual goals. The empirical results demonstrate that the proposed framework enhances learning efficiency. It guarantees the learning of a prioritized policy for the commander agent and encourages subordinate agents to explore areas of interest more frequently, guided by appropriate soft constraints imposed by the commander.

Enhancing hierarchical remote preparation of an arbitrary n-qudit equatorial state by weak measurement and measurement reversal

Based on the demand for safer and more efficient quantum network communication, we propose two deterministic protocols to achieve the remote preparation of an arbitrary n-qudit equatorial states among multiple agents by selecting appropriate entanglement resources. The level of the agents varies depending on their authorities to restore the secret state. The superior agent requires the cooperation of the residual superior agents and any one of the subordinate agents, while the subordinate agent needs the help of all the other agents. To be emphasized, the recovery operations are deduced by general formulas that distinctly elucidate their relevance with the measurement outcomes. Additionally, we discuss the impact of amplitude damping noise and utilize weak measurement and measurement reversal to suppress noise.

The nonmeek inherit the earth: Children generalize dominance, but not submissiveness.

Children and even infants have clear intuitions about power early in development; they can infer who is dominant and subordinate from observing a single interaction. However, it is unclear what children infer about each individual's status from these interactions-do they think dominants and subordinates will maintain their status when interacting with novel partners? In three experiments, we investigate this question. Children (4- to 10-year-olds, N = 365) heard stories about a dominant and subordinate agent and predicted the dominant or subordinate agent's behavior with a novel agent. In all studies, we found that 7- to 10-year-olds generalized dominance, thinking the dominant would again be dominant or "in charge," both for social power (e.g., granting permission) in Study 1 and physical dominance (e.g., a fistfight) in Studies 2 and 3. Furthermore, although they believed dominant agents would win dominance contests (fistfights), they did not believe they would win contests unrelated to dominance (math contests). Younger children did not generalize social power (Study 1) but did generalize physical dominance (Studies 2 and 3). However, even for physical dominance, their generalizations were less selective (i.e., they believed the dominant would win fistfights and math contests). Notably, neither age group generalized an agent's submissiveness in any of the studies-they did not believe a subordinate agent would again be subordinate when paired with a novel partner. We discuss how these results extend past work on children's developing intuitions about dominance and prompt deeper questions about the inferences children draw from dominance interactions. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Метод оптимизации организационной структуры централизованных многоагентных систем в автоматическом режиме

The organizational structure of a multi-agent system (MAS) is a set of roles and relationships of agents, com-ponents that control their behavior, as well as rules governing the interaction of system elements. The effectiveness of the MAS largely depends on the characteristics of the organizational structure used. Existing solutions in this area have a significant disadvantage, namely low adaptability to changes in environmental parameters or adjustment of the conditions of the task, which consists in the need to restart the procedure for synthesizing the organizational structure of the MAS. The problem of automatic optimization of the organizational structure of centralized MAS in the conditions of changed environmental parameters or the task is of particular relevance for centralized MAS with a strict hierarchical structure, whose agents can be divided into two classes – agents-managers, agents-subordinates. The subject of the study is the methods of synthesis of the organizational structure of the MAS. The aim of the work is to develop a method for optimizing the organizational structure of centralized MAS in automatic mode. To achieve this goal, the concepts of primary and secondary agent resources were introduced, based on a biogeographic algorithm, a method was developed that regulates the movement of subordinate agents between groups depending on the degree of attractiveness of the agent manager (depends on the sum of estimates of the effectiveness of subordinate agents managed by him, as well as on the distance from this subordinate agent). The developed method can find its practical application in the implementation of the following tasks: inspection or patrolling (protection) of infrastructure facilities by mobile robots, the implementation of artificial intelligence in computer games.

Present Pasts of Colonial Modernity: Embroideries by Lucie Kamuswekera

Present Pasts of Colonial Modernity: Embroideries by Lucie Kamuswekera

AN ALGORITHM OF THE COLLECTIVE INTERACTION OF INTELLIGENT AGENTS IN CENTRALIZED MULTI-AGENT SYSTEMS

A centralized multiagent system based on the methods of feudal reinforcement learning, including agents-managers and agents-subordinates, is considered. A brief review of the author’s previous works on this topic is given. The standard algorithm for the functioning of systems of this type is considered, including the translation of the decision maker to agents-managers, the division of tasks by agents-managers into a set of subtasks, the choice by the agent-manager of the strategy used, the formation of reward functions by agents-managers, the assignment of tasks to agents-subordinates, the execution by agents-subordinates assigned tasks. The main problems of this algorithm are presented, changes are made to ensure the possibility of automatically assigning agent-managers and forming groups of subordinate agents around them, reproducing and exchanging experience. More attention is paid to the problem of experience exchange, the main ways of experience exchange are given. The principles of operation of a machine vision system that implements an upgraded algorithm are described. An assessment of the effectiveness of the obtained algorithm for the collective interaction of intelligent agents using a software model developed in Microsoft Unity is given. A comparison is made between the standard algorithm for multiagent interaction and the proposed algorithm for the collective interaction of intelligent agents in centralized multi-agent systems based on the approach of reinforcement learning and visualization of three-dimensional scenes. The conclusion is made about the expediency of using the developed algorithmt.

The boss is not always right: Norwegian preschoolers do not selectively endorse the testimony of a novel dominant agent

Theories of cultural evolution posit that cues of competence-based prestige, rather than formidability-based dominance, should guide culturally transmitted learning, but recent work suggested that French and Kaqchikel Guatamalan preschoolers place their epistemic trust in dominant others. In contrast, this study shows that 249 three- to six-year-olds (116 girls, tested between 2016 and 2018 across metropolitan locations with varying ethnic composition and socioeconomic status) randomly endorsed the word-labels of dominant and subordinate agents in the egalitarian culture of Norway, using stimuli which solicit dominance inferences among infants and manipulating anonymity across studies to control for egalitarian desirability bias. A meta-analysis estimated that 48% endorsed the dominant's testimony. This demonstrates that the tendency to endorse the epistemic claims of dominant individuals does not emerge reliably in early childhood.

A HYBRID ALGORITHM FOR FORMING THE SHORTEST TRAJECTORY BASED ON THE APPLICATION OF MULTI-AGENT LEARNING WITH REINFORCEMENT, THE SEARCH ALGORITHM A* AND EXCHANGE OF EXPERIENCE

The article presents a hybrid algorithm for the formation of the shortest trajectory for intelligent agents of a multi-agent system, based on the synthesis of methods of the reinforcement learning paradigm, the heuristic search algorithm A*, which has the functions of exchange of experience, as well as the automatic formation of subgroups of agents based on their visibility areas. The experimental evaluation of the developed algorithm was carried out by simulating the task of finding the target state in the maze in the Microsoft Unity environment. The results of the experiment showed that the use of the developed hybrid algorithm made it possible to reduce the time for solving the problem by an average of 12.7 % in comparison with analogs. The differences between the proposed new “hybrid algorithm for the formation of the shortest trajectory based on the use of multi-agent reinforcement learning, search algorithm A* and exchange of experience” from analogs are as follows: – application of the algorithm for the formation of subgroups of subordinate agents based on the “scope” of the leader agent for the implementation of a multi-level hierarchical system for managing a group of agents; – combining the principles of reinforcement learning and the search algorithm A*.

Semantic-relations taxonomy for knowledge representation

Semantic relations in knowledge representation characterizes the association between the concepts of a domain. Concepts are units of knowledge and the relationships that link these units, which gives meaning to the knowledge as represented. In this way, semantic relations allow users to assimilate the purpose of the association between concepts in the presentational context, avoiding misinterpretation of the information, mainly that presented in instruments of knowledge representation.The objective of this paper is to propose a taxonomy of semantic relations that compiles the different approaches on the subject. The methodology applied bibliographic research for theoretical foundation and literature review based on the main bibliographic references on semantic relations. As result, some classifications of semantic relations were found to have been raised by different authors, each offering a singular point of view, which has resulted in a range of discordant terms. To answer this need, a taxonomy was arrived at of sixty-three semantic relations, including a new relation discovered by this study, dubbed here ‘subordinate agent’.

Presidential directives in a resistant bureaucracy

AbstractPresidential directives are often assumed to be checked only by external actors, like Congress and the courts. But the internal constraints facing presidents can also be substantial. I study a model where a president can induce compliance with a directive by removing some subordinate agents (the appointees) but not others (the careerists), and where the relative contribution of each agent to the directive’s success is unobservable. The model suggests that theformalauthority presidents have to issue directives and remove subordinates can advance presidential goals, affording presidentsrealauthority. But real authority is not guaranteed, and the resulting uncertainty can shape presidential decision-making: when to issue a directive, how ambitious to make it, and which agencies to target. To illustrate, I analyse two prominent directives in Clinton’s regulatory planning order, E.O. 12866, showing why they targeted different agencies, despite belonging to the same order, and why compliance has been uneven.

Energy-Efficient Monitoring of Fire Scenes for Intelligent Networks

In the current surveillance networks, a vast amount of data is generated from different sources, resulting in big data. Such data need intelligent technologies and big data analytics for its realtime analysis to provide different services. This article describes an efficient artificial intelligence and big data analytics-assisted system for real-time fire scene analysis in surveillance networks. The proposed system is based on intelligent independent and subordinate agents, where each agent has a different task to report to the fire brigade and disaster management instantly. A deep yet efficient CNN is utilized in the system for feature extraction, classification, localization, and detection of fire in video frames. When the fire is detected in the frames, a fire alert is instantly sent to the emergency department and all agents immediately start their processing for checking the severity and growth rate of the fire, recognizing the scene and all objects on fire, and evacuation monitoring. Each agent of the system instantly sends information to disaster management to stop the loss of precious human lives and minimizes other economic and ecological loss. Experimental validation shows the promising results compared to existing systems. It is believed that using such a system is the demand of the time to save humanity from massive fire disasters and can make the current surveillance networks more intelligent.

Command Agent Belief Architecture to Support Commander Decision Making in Military Simulation

<p class="p1">In the war, military conflicts have many aspects that are consistent with complexity theory e.g., the higher commander’s decision is directed at animate entity that react under hierarchical and self-organised structure in decentralised command and control for the collectivist dynamism of decomposed elements due to nonlinear complexity of warfare on the battlefield. Agent technology have been found to be suitable for modelling tactical behaviour of entities at multiple level of resolution under hierarchical command and control (C2) structure and provide a powerful abstraction mechanism required for designing simulations of complex and dynamic battlefield situations. Intelligent agents can potentially reduce the overhead on such experiments and studies. Command agents, plan how to carry out the operation and assign tasks to subordinate agents. They receive information from battlefield environment and use such information to build situation awareness and also to respond to unforeseen situations. In the paper, we have proposed a mechanism for modelling tactical behaviour of an intelligent agent by which higher command level entities should be able to synthesize their beliefs derived from the lower level sub ordinates entities. This paper presents a role-based belief, desire and intention mechanism to facilitate in the representation of military hierarchy, modelling of tactical behaviour based on agent current belief, teammate’s belief propagation, and coordination issues. Higher commander can view the battlefield information at different levels of abstraction based on concept of aggregation and disaggregation and take appropriate reactive response to any unforeseen circumstances happening in battlefield.</p>

Cooperative Resource Allocation in Open Systems of Systems

Resource allocation is a common problem in many technical systems. In multi-agent systems, the decentralized or regionalized solution of this problem usually requires the agents to cooperate due to their limited resources and knowledge. At the same time, if these systems are of large scale, scalability issues can be addressed by a self-organizing hierarchical system structure that enables problem decomposition and compartmentalization. In open systems, various uncertainties—introduced by the environment as well as the agents’ possibly self-interested or even malicious behavior—have to be taken into account to be able to allocate the resources according to the actual demand. In this article, we present a trust- and cooperation-based algorithm that solves a dynamic resource allocation problem in open systems of systems. To measure and deal with uncertainties imposed by the environment and the agents at runtime, the algorithm uses the social concept of trust. In a hierarchical setting, we additionally show how agents create constraint models by learning the capabilities of subordinate agents if these are not able or willing to disclose this information. Throughout the article, the creation of power plant schedules in decentralized autonomous power management systems serves as a running example.

Usages, pactes et « passes du droit »

À partir d’une enquête empirique dans une cité de logements sociaux à Buenos Aires, l’article explore le travail de légitimité interne des habitants sans titre de logement pour accéder à un statut juridique plus solide. Se concentrer sur les relations entretenues entre les habitants et l’Institut du logement, c’est découvrir la fabrique d’un ordre consenti, négocié, qui tend vers une régularisation locative. Dès lors, la mise en œuvre de règles n’est pas simplement le produit d’une action venant du sommet de l’administration, ni des stratégies unilatérales des agents subalternes. Elle est tout autant le fait des habitants qui mobilisent les règles légales et les « passe-droits » qu’elles recèlent (Lascoumes, Le Bourhis, 1996). Plus qu’un espace de déviance et d’« informalité », la cité se présente à l’observateur comme le terrain d’un montage de légalité centré sur des pratiques d’occupation qui tendent à être normalisées.

Modeling of Network Cooperative Design of a Hydraulic System Based on Multi-Agent

To shorten the design period of a hydraulic system and achieve distributed cooperative design by experts, this paper proposes a model of the network collaborative design of a hydraulic system based on multi-agent technology. This model consists of three layers: the management agent layer, the middle agent layer and the resource agent layer. Its agent of the same layer adopts distributed structure, and each autonomous agent is constituted of a group of lower subordinate agents. This structure meets the need of distributed control of the cooperative design. By dynamic data exchange of agents, it carries out their communication, coordination and conflict resolution, and finally gets the global target. From the developed example, a multi-agent system can easily integrate existing engineering design tools with analysis tools. Its concurrent design process can improve design efficiency.

Production control model of integrated reverse engineering system based on MAS

To improve the intelligent and cooperative performance of the integrated reverse engineering system, a control model based on the architecture of Multi-Agent System (MAS) was constructed. This model consisted of three layers: the management Agent layer, the cell Agent layer and the execution Agent layer. The management Agent layer was in charge of coordinating, scheduling and managing all Agents. Each Agent in the cell Agent layer finished the task by cooperation and executed the first step of distribution. The execution Agent layer was responsible for the hardware and software in RE system. According to the local resources information and current status, the CAs in this layer accepted the tasks assigned by the cell Agents and resolved the tasks. Agents in the same layer were in distributed structure and equal cooperation. The superior and subordinate Agents adopted the master-menial cooperation.

Desmarest’s “Determination of some epochs of nature through volcanic products” (1775/1779)

In 1775, Nicolas Desmarest (1725-1815) presented an interpretation of geological evidence he had been examining for over a decade, among the extinct volcanoes of Auvergne in south-central France. He interpreted the volcanic remnants in that region in terms of three successive epochs, distinguished from one another by their degree of erosive alteration and the different topographic positions they held following extensive denudation. He insisted on the utility, for the establishment of sequential stages in the region's history, of what he called his 'analytical method' of working backward in time , from recognition of the most recent and therefore least altered products, to reconstruct in the imagination those more thoroughly disturbed and confusing remnants of earlier periods. In addition to being a landmark effort in historical geology based on local observation, Desmarest's paper was also notable as an expression of what would come later to be called uniformitarian thinking: he argued that the erosive processes responsible for wearing down the lavas must have operated gradually and constantly . While Desmarest seems never to have wavered in his view of volcanic action as a subordinate agent in the overall dynamics and history of the globe, he showed awareness that his analysis of the Auvergne lavas pointed toward a considerable role for volcanism during long portions of the province's geological history. The nearest thing to a contemporary publication of Desmarest's 1775 presentation came more than three years later, in the form of an 'excerpt' in the February 1779 issue of the journal Observations sur la physique. An English translation of that excerpt is presented in the present paper for the first time.

The Familial World of the Company’s kacceri in Early Colonial Madras

The Familial World of the Company’s kacceri in Early Colonial Madras Bhavani Raman The English East India Company’s offices or kacceris (Anglo-Indian: Cutcherry, Kahcherri) are widely acknowledged as the nerve centers of the British Empire in India in the late eighteenth and early nineteenth century but their micro-practices have been remarkably understudied. Studies of colonial administration stress the continuity of pre-colonial administrative practices in the Company kacceri and present it as a place where lower-level Indian officials and servants severely subverted and compromised the intentions of colonial policy and reform.1 Robert Frykenberg’s pioneering study on South India’s Guntur district, for example, conceived of imperial integration through colonial revenue and judicial policy. According to Frykenberg, imperial integration was restricted to the elevated realm of juridical law because corrupt and venal rural magnates and local elite continued to dominate the everyday practice of statecraft by continuing earlier practices.2 Since then, scholarship on colonial knowledge has underscored the role of the objectifying practices of documentation in the exercise of colonial power while vigorously debating the contributions of subordinate agents, especially scribal specialists, language teachers, and translators (and their important pre-colonial antecedents) to the production of the forms of knowledge that underwrote technologies of dominance. These debates on colonial knowledge have yielded a rich harvest of studies on service specialists, but they have also often overlooked the locus of much of that knowledge production: the colonial bureau, the “office” or kacceri.3 In a way, then, studies of colonial knowledge forms and colonial administration have underscored the role of elite mediators in the exercise of colonial power but neglected to investigate the social form of the kacceri of the early colonial state in South India. The Company kacceri was a site of everyday struggles over sovereignty in the late eighteenth and early nineteenth century. The exigencies of colonial conquest created an urgent and self-perpetuating crisis of trust between Company administrators and their local subordinates while enabling select subordinate officers to accrue a great deal of influence over the agrarian order of Madras Presidency. The crises of trust greatly determined kacceri culture because it often resulted in officials describing their subordinates as venal, nepotistic and hence, corrupt. Scholarly accounts tend to take these descriptions of nepotism to represent evidence of Indian exceptionalism or primordial casteism. 4 In contrast, kin and family ties—“the networks of nepotism”—that bound the kacceri cadre into factional clusters contain some clues that can help frame the patrimonial character of work in the kacceri. This paper suggests that the idiom of family, social and kinship networks, in the constitution of the kacceri is integral to any understanding of the exercise of colonial power. Kacceri history, then, is not just intertwined with the histories of the social groups who compose the administration, in itself a sparsely researched topic, but this history can illustrate how the Company kacceri reworked rather than merely continued to foster existing idioms of lordship and devotion. Company masters secured the loyalty of their subordinates by greatly intensifying hierarchical ties while simultaneously buttressing the power of their subordinates over inhabitants in new ways. Family, Kin and the Kacceri How did everyday familial practices of patrimony constitute the early colonial bureaucracy? Scholars drawing on feminist theory have used the category of the family as a tool of social history, even as they have recovered its heuristic and material constitution through the discourse and practice of colonial law in South Asia.5 But even if the material and ideological restructuring of kin relations by colonial law is widely acknowledged, the importance of kinship to state making in South Asian historiography has been restricted to the pre-colonial period. Historians of the state have for long seen the familial as pre-capitalist/pre-modern, replaced by a rule bound colonial bureaucracy. For this reason, familial or kin relations in the early colonial Company office are understood as a survival of an earlier era that resisted reform. Recent scholarship fundamentally questions these assumptions. In addition to recent attention in South Asian historiography, familial practices have begun to receive attention as crucial to the political expansion of European trading companies...

Penalty Function-Based Two-Level Hybrid Shop Floor Control System

Although several hybrid shop floor control architectures have been proposed in the literature, varying degrees of autonomy of subordinate controllers and their effects on supervisory level performance have not been studied. In this paper, we present a new hybrid control architecture with two levels, where the autonomy of subordinate agents changes adaptively. The penalty function in the architecture represents the degree of negative impact on performance that will result from changing the original schedule. When a disturbance occurs, the disturbance agent invokes rescheduling at the appropriate level depending on the threshold disturbance level. The subordinate agents execute tasks based on the schedule from the supervisory agent in the absence of disturbances; or else revise the original schedule optimally with regard to the supervisory level performance (via penalty function) and the disturbance before executing the tasks. We present math programming formulations, quantitative metrics to indicate the disturbance level and the levels of autonomy. The proposed architecture is illustrated using a job shop problem and is then tested with various other problems. Note to Practitioners-In a hybrid shop floor control system, the objective of global system performance conflicts with the objective of a timely response to local disturbances. For a given disturbance, there is a need to compare the cost (e.g., computation time) associated with supervisory level rescheduling against the potential improvement in system performance that this rescheduling can provide. If the cost of rescheduling is higher, rescheduling at the subordinate level will be appropriate. Otherwise, rescheduling at the supervisory level will be appropriate. In this paper, manufacturing scheduling problems of various complexities are used to discuss the tradeoff between the objectives listed above. We then use these problems to show how to find an appropriate threshold value to trigger subordinate or supervisory level rescheduling. Limited classes of performance metrics are considered. Work is currently being carried out to extend the approach to broader performance metrics. It is believed that our results can be applied to other systems for which hierarchical structures are commonly used. Examples of such systems exist in the transportation, health-care, and defense sectors