Agent Memory Research Articles

Long homogeneous payoff records with the latest strategy promotes the cooperation

In this study, we studied the fraction of cooperators in the public goods game, taking into account the memory effect that affects the strategy updating. Unlike previous studies where an agent learned the opponent's last strategy based on their last payoffs, agents with memory in this study choose to cooperate according to opponents' effective strategies by comparing their effective payoffs based on payoffs and strategies in their memories. The effective payoff of an agent is the weighted average of previous strategies in the agent's memory. The weight is the decay measuring the significance of previous strategies and former payoffs are less significant than latter ones upon the future strategy. And it is the same with the effective strategy. By this means, when the effective payoff and the effective strategy share a same memory length and a same set of decay, the numerical simulation shows increasing the memory length or a homogeneous decay promotes cooperation among agents. However, it is a surprise that the effective payoff and the effective strategy have opposite effects. Homogeneous payoff weights lead to a higher fraction of cooperators, while heterogeneous strategy weights favors the cooperation, especially when agents only consider the latest strategy. Comparing the effect of memorizing payoffs and strategies, the effect of memorizing payoffs plays a dominant role. Furthermore, when the total memory length is limited, agents should memorize as many historical payoffs as possible. In addition the qualitative result above is independent of the rational noise.

Interpreting Tangled Program Graphs Under Partially Observable Dota 2 Invoker Tasks

Interpretable learning agents directly construct models that provide insight into the relationships learnt. Moreover, to date, there has been a lot of emphasis on interpreting reactive models developed for supervised learning tasks. In this work, we consider the case of models developed to address a suite of 6 partially observable tasks defined in the Dota 2 Online Battle Arena game engine. This means that learning agents need to make decisions based on the previous state as developed by the learning agent's memory; in addition to a 310-dimensional state vector provided by the game engine. Interpretability is addressed by adopting the tangled program graph approach to developing learning agents. Thus, decision-making is explicitly divide-and-conquer, with different parts of the resulting graph visited depending on the task context. We demonstrate that programs comprising the tangled program graph approach self-organize such that: (1) small subsets of task features are identified to define conditions under which index memory is written, and; (2) the subset of programs responsible for defining actions typically query indexed memory rather than task features. Particular preferences emerge for different tasks; thus, the blocking (or evasion) tasks result in a preference for specific actions whereas more open-ended tasks assume policies based on combinations of behaviours. In short, the ability to evolve the topology of the learning agent provides insights into how the policies are being constructed for addressing partially observable tasks.

Sample Efficient Reinforcement Learning Using Graph-Based Memory Reconstruction

Reinforcement learning (RL) algorithms typically require orders of magnitude more interactions than humans to learn effective policies. Research on memory in neuroscience suggests that humans' learning efficiency benefits from associating their experiences and reconstructing potential events. Inspired by this finding, we introduce a human brain-like memory structure for agents and build a general learning framework based on this structure to improve the RL sampling efficiency. Since this framework is similar to the memory reconstruction process in psychology, we name the newly proposed RL framework as Graph-Based Memory Reconstruction (GBMR). In particular, GBMR first maintains an attribute graph on the agent's memory and then retrieves its critical nodes to build and update potential paths among these nodes. This novel pipeline drives the RL agent to learn faster with its memory-enhanced value functions and reduces interactions with the environment by reconstructing its valuable paths. Extensive experimental analyses and evaluations in the Grid Maze and some challenging Atari environments demonstrate GBMR's superiority over traditional RL methods. We will release the source code and trained models to facilitate further studies in this research direction.

Emergence of intelligent collective motion in a group of agents with memory.

Intelligent agents collect and process information from their dynamically evolving neighborhood to efficiently navigate through it. However, agent-level intelligence does not guarantee that at the level of a collective; a common example is the jamming we observe in traffic flows. In this study, we ask: how and when do the interactions between intelligent agents translate to desirable or intelligent collective outcomes? To explore this question, we choose a collective consisting of two kinds of agents with opposing desired directions of movement. Agents in this collective are minimally intelligent: they possess only a single facet of intelligence, viz., memory, where the agents remember how well they were able to travel in their desired directions and make up for their non-optimal past. We find that dynamics due to the agent's memory influences the collective, giving rise to diverse outcomes at the level of the group: from those that are undesirable to those that can be called "intelligent." When memory is short term, local rearrangement of agents leads to the formation of symmetrically jammed arrangements that take longer to unjam. However, when agents remember across longer time-scales, their dynamics become sensitive to small differences in their movement history. This gives rise to heterogeneity in the movement that causes agents to unjam more readily and form lanes.

ScanBot: Autonomous Reconstruction via Deep Reinforcement Learning

Autoscanning of an unknown environment is the key to many AR/VR and robotic applications. However, autonomous reconstruction with both high efficiency and quality remains a challenging problem. In this work, we propose a reconstruction-oriented autoscanning approach, called ScanBot, which utilizes hierarchical deep reinforcement learning techniques for global region-of-interest (ROI) planning to improve the scanning efficiency and local next-best-view (NBV) planning to enhance the reconstruction quality. Given the partially reconstructed scene, the global policy designates an ROI with insufficient exploration or reconstruction. The local policy is then applied to refine the reconstruction quality of objects in this region by planning and scanning a series of NBVs. A novel mixed 2D-3D representation is designed for these policies, where a 2D quality map with tailored quality channels encoding the scanning progress is consumed by the global policy, and a coarse-to-fine 3D volumetric representation that embodies both local environment and object completeness is fed to the local policy. These two policies iterate until the whole scene has been completely explored and scanned. To speed up the learning of complex environmental dynamics and enhance the agent's memory for spatial-temporal inference, we further introduce two novel auxiliary learning tasks to guide the training of our global policy. Thorough evaluations and comparisons are carried out to show the feasibility of our proposed approach and its advantages over previous methods. Code and data are available at https://github.com/HezhiCao/Scanbot.

The Tensor Brain: A Unified Theory of Perception, Memory, and Semantic Decoding.

We present a unified computational theory of an agent's perception and memory. In our model, both perception and memory are realized by different operational modes of the oscillating interactions between a symbolic index layer and a subsymbolic representation layer. The two layers form a bilayer tensor network (BTN). The index layer encodes indices for concepts, predicates, and episodic instances. The representation layer broadcasts information and reflects the cognitive brain state; it is our model of what authors have called the "mental canvas" or the "global workspace." As a bridge between perceptual input and the index layer, the representation layer enables the grounding of indices by their subsymbolic embeddings, which are implemented as connection weights linking both layers. The propagation of activation to earlier perceptual processing layers in the brain can lead to embodiments of indices. Perception and memories first create subsymbolic representations, which are subsequently decoded semantically to produce sequences of activated indices that form symbolic triple statements. The brain is a sampling engine: only activated indices are communicated to the remaining parts of the brain. Triple statements are dynamically embedded in the representation layer and embodied in earlier processing layers: the brain speaks to itself. Although memory appears to be about the past, its main purpose is to support the agent in the present and the future. Recent episodic memory provides the agent with a sense of the here and now. Remote episodic memory retrieves relevant past experiences to provide information about possible future scenarios. This aids the agent in decision making. "Future" episodic memory, based on expected future events, guides planning and action. Semantic memory retrieves specific information, which is not delivered by current perception, and defines priors for future observations. We argue that it is important for the agent to encode individual entities, not just classes and attributes. Perception is learning: episodic memories are constantly being formed, and we demonstrate that a form of self-supervised learning can acquire new concepts and refine existing ones. We test our model on a standard benchmark data set, which we expanded to contain richer representations for attributes, classes, and individuals. Our key hypothesis is that obtaining a better understanding of perception and memory is a crucial prerequisite to comprehending human-level intelligence.

Data-Efficient Reinforcement Learning with Data Augmented Episodic Memory

Reinforcement learning has been implemented to model a task by doing the task repeatedly to get the maximum results based on the reward and punishment policy. It has been implemented in the game and agent-based modelling. In the game, the game agent or Non-Players character can be modelled using several techniques to achieve the goal (e.g. reinforcement learning, deep neural network and Monte Carlo Tree Search). Deep neural networks and Monte Carlo Tree Search, two more sophisticated techniques in reinforcement learning algorithms, assisted the present reinforcement learning in resolving more challenging issues. However, this area has two challenges: the minimum number of data to model and generalization to different environments. Determining a minimum number of data required by the architecture to train the model is quite a cumbersome task to be applied to real- world jobs and situations since it demands substantial data to be explicitly provided and trial-and-error re-configuration. This work proposes a Data-Efficient Reinforcement Learning model by augmenting the data and implementing episodic memory. To illustrate the effectiveness of the proposed model, this research compares it to several models, such as the Deep Q-Network (DQN) with episodic memory to the same model with data augmentation and episodic memory. The model adds to the observations before being stored in the agent's memory, causing the agent to use the same logic and take the same action in comparable situations. The outcome demonstrates that the augmented model can surpass the fundamental model in speed (with an improvement of 50% quicker).

Interrogation Room

Interrogation Room

Augmented Memory Replay in Reinforcement Learning With Continuous Control

Online reinforcement learning agents are currently able to process an increasing amount of data by converting it into a higher order value functions. This expansion of the information collected from the environment increases the agent's state space enabling it to scale up to a more complex problems but also increases the risk of forgetting by learning on redundant or conflicting data. To improve the approximation of a large amount of data, a random mini-batch of the past experiences that are stored in the replay memory buffer is often replayed at each learning step. The proposed work takes inspiration from a biological mechanism which act as a protective layer of human brain higher cognitive functions: active memory consolidation mitigates the effect of forgetting of previous memories by dynamically processing the new ones. The similar dynamics are implemented by a proposed augmented memory replay AMR capable of optimizing the replay of the experiences from the agent's memory structure by altering or augmenting their relevance. Experimental results show that an evolved AMR augmentation function capable of increasing the significance of the specific memories is able to further increase the stability and convergence speed of the learning algorithms dealing with the complexity of continuous action domains.

Exploiting the Agent's Memory in Asymptotic and Finite-Time Consensus Over Multi-Agent Networks

This article proposes two average consensus algorithms exploiting the memory of agents. The performance of the proposed as well as of several state-of-the-art consensus algorithms is evaluated considering different communication ranges, and evaluating the impact of transmission errors. To compare asymptotic and finite-time average consensus schemes, the $\varepsilon$ -convergence time is adopted for a fair comparison. A discussion about memory requirements, transmission overhead, a priori information on network topology, and robustness to errors is provided.

Hardware Accelerated Semantic Declarative Memory Systems through CUDA and MapReduce

Declarative memory enables cognitive agents to effectively store and retrieve factual memory in real-time. Increasing the capacity of a real-time agent's declarative memory increases an agent's ability to interact intelligently with its environment but requires a scalable retrieval system. This work represents an extension of the Accelerated Declarative Memory (ADM) system, referred to as Hardware Accelerated Declarative Memory (HADM), to execute retrievals on a GPU. HADM also presents improvements over ADM's CPU execution and considers critical behavior for indefinitely running declarative memories. The negative effects of a constant maximum associative strength are considered, and mitigating solutions are proposed. HADM utilizes a GPU to process the entire semantic network in parallel during retrievals, yielding significantly faster declarative retrievals. The resulting GPU-accelerated retrievals show an average speedup of approximately 70 times over the previous Service Oriented Architecture Declarative Memory (soaDM) implementation and an average speedup of approximately 5 times over ADM. HADM is the first GPU-accelerated declarative memory system in existence.

Quantum machine learning with glow for episodic tasks and decision games

We consider a general class of models, where a reinforcement learning (RL) agent learns from cyclic interactions with an external environment via classical signals. Perceptual inputs are encoded as quantum states, which are subsequently transformed by a quantum channel representing the agent's memory, while the outcomes of measurements performed at the channel's output determine the agent's actions. The learning takes place via stepwise modifications of the channel properties. They are described by an update rule that is inspired by the projective simulation (PS) model and equipped with a glow mechanism that allows for a backpropagation of policy changes, analogous to the eligibility traces in RL and edge glow in PS. In this way, the model combines features of PS with the ability for generalization, offered by its physical embodiment as a quantum system. We apply the agent to various setups of an invasion game and a grid world, which serve as elementary model tasks allowing a direct comparison with a basic classical PS agent.

Forgetfulness can help you win games.

We present a simple game model where agents with different memory lengths compete for finite resources. We show by simulation and analytically that an instability exists at a critical memory length, and as a result, different memory lengths can compete and coexist in a dynamical equilibrium. Our analytical formulation makes a connection to statistical urn models, and we show that temperature is mirrored by the agent's memory. Our simple model of memory may be incorporated into other game models with implications that we briefly discuss.

Applying Short-term Memory to Social Search Agents

Abstract This paper presents about our research in social search. Generally, the research in social search falls into two principal challenges. The first challenge is how to find more relevant answers to the question. The second one is how to increase speed in finding relevant answers. Recently, we had provided two algorithms called Asknext and Question Waves to find more relevant answers compared to the baseline algorithm BFS. But, the search speed of the two proposed algorithms still the subject to improve. In this paper, we introduce the agents’ ability of learning the answers from the interactions with other agents so that they can quickly answer the question of other agents. We model this learning process by implementing the concept of data caching as the short-term memory of each social search agent. The result improvement of the speediness and the reduction of the number of messages used to communicate between agents, after apply agent's short-term memory concept, demonstrates the usefulness of the proposed approach.

Intrinsic motivations and open-ended development in animals, humans, and robots: an overview.

Intrinsic motivations and open-ended development in animals, humans, and robots: an overview.

Unified Route Choice Framework

The route choice system and the traffic control system (TCS) constitute two major approaches to mitigating congestion in urban road networks. The interaction between signal control and route choice is considered from a narrower route choice system perspective, with the focus on route choice models for operational purposes. The goal is to analyze the relative performance of alternative route choice models as different assumptions are made about the type of TCS in use. To this end, an agent-based framework for formulating different route choice models is defined, and this framework is integrated with a microscopic traffic simulation environment. Within the framework, each agent's memory is updated repeatedly (daily) to reflect available prior individual and social experience, and then a route is chosen by a probabilistic sequential decision-making process. Several previously developed route choice models from the literature are implemented with the framework, and their performance, along with some additional hybrid models that are suggested by the modeling framework, is evaluated on two simulated real-world systems: a 32-intersection road network in Pittsburgh, Pennsylvania, running with a SYNCHRO-generated coordinated timing plan and the same road network running with the scalable urban traffic control (SURTRAC) adaptive TCS. The results show that specific route choice models perform differentially when applied in conventional and adaptive traffic control settings and that better overall network performance for all route choice models is achieved in the adaptive control setting. This unified framework also makes it possible to analyze the performance impact of route choice model components and to formulate better-performing hybrid models.

A Functional Analysis of Historical Memory Retrieval Bias in the Word Sense Disambiguation Task

Effective access to knowledge within large declarative memory stores is one challenge in the development and understanding of long-living, generally intelligent agents. We focus on a sub-component of this problem: given a large store of knowledge, how should an agent's task-independent memory mechanism respond to an ambiguous cue, one that pertains to multiple previously encoded memories. A large body of cognitive modeling work suggests that human memory retrievals are biased in part by the recency and frequency of past memory access. In this paper, we evaluate the functional benefit of a set of memory retrieval heuristics that incorporate these biases, in the context of the word sense disambiguation task, in which an agent must identify the most appropriate word meaning in response to an ambiguous linguistic cue. In addition, we develop methods to integrate these retrieval biases within a task-independent declarative memory system implemented in the Soar cognitive architecture and evaluate their effectiveness and efficiency in three commonly used semantic concordances.

Incorporating background knowledge and structured explananda in abductive reasoning: a framework

Abductive reasoning plays a dominant role in a wide variety of cognitive tasks, including diagnosis, language understanding, and learning. The author's work is aimed at developing a computational model of abductive reasoning keeping in view some of the important aspects that cut across specific instances of abductive reasoning. A significant portion of the paper is devoted to the discussion of two of those aspects, namely, those of background knowledge and structured explananda. The former aspect concerns the effect the agent's memory of specific facts has on abductive reasoning. The latter aspect corresponds to dealing with explananda having conceptual structure and contrasts with the approach of taking explananda to be atomic. The paper discusses various issues related to these two aspects and develops an algorithm for abductive reasoning incorporating those aspects.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

A design for the ICARUS architecture

We describe our designs for ICARUS, an integrated architecture for controlling an intelligent agent in a complex physical environment. By navigating between locations and manipulating other objects, the agent influences the world and achieves its goals. The architecture includes components for perceiving the environment, for generating plans to solve problems, and for executing the plans generated. A fourth component manages the agent's long-term memory. Our assessment of the design suggests that it is general, versatile, scales well to larger problems, and is consistent with a variety of psychological results.

Rationality and the structure of human memory

A tacit and highly idealized model of the agent's memory is presupposed in philosophy. The main features of a more psychologically realistic duplex (orn-plex) model are sketched here. It is argued that an adequate understanding of the rationality of an agent's actions is not possible without a satisfactory theory of the agent's memory and of the trade-offs involved in management of the memory, particularly involving “compartmentalization” of the belief set. The discussion identifies some basic constraints on the organization of knowledge representations in general.