Goal-directed Exploration Research Articles

A computational neuroimaging study of reinforcement learning and goal-directed exploration in schizophrenia spectrum disorders.

Prior evidence indicates that negative symptom severity and cognitive deficits, in people with schizophrenia (PSZ), relate to measures of reward-seeking and loss-avoidance behavior (implicating the ventral striatum/VS), as well as uncertainty-driven exploration (reliant on rostrolateral prefrontal cortex/rlPFC). While neural correlates of reward-seeking and loss-avoidance have been examined in PSZ, neural correlates of uncertainty-driven exploration have not. Understanding neural correlates of uncertainty-driven exploration is an important next step that could reveal insights to how this mechanism of cognitive and negative symptoms manifest at a neural level. We acquired fMRI data from 29 PSZ and 36 controls performing the Temporal Utility Integration decision-making task. Computational analyses estimated parameters corresponding to learning rates for both positive and negative reward prediction errors (RPEs) and the degree to which participates relied on representations of relative uncertainty. Trial-wise estimates of expected value, certainty, and RPEs were generated to model fMRI data. Behaviorally, PSZ demonstrated reduced reward-seeking behavior compared to controls, and negative symptoms were positively correlated with loss-avoidance behavior. This finding of a bias toward loss avoidance learning in PSZ is consistent with previous work. Surprisingly, neither behavioral measures of exploration nor neural correlates of uncertainty in the rlPFC differed significantly between groups. However, we showed that trial-wise estimates of relative uncertainty in the rlPFC distinguished participants who engaged in exploratory behavior from those who did not. rlPFC activation was positively associated with intellectual function. These results further elucidate the nature of reinforcement learning and decision-making in PSZ and healthy volunteers.

Goal-Directed Exploration for Learning Vowels and Syllables: A Computational Model of Speech Acquisition

Infants learn to speak rapidly during their first years of life, gradually improving from simple vowel-like sounds to larger consonant-vowel complexes. Learning to control their vocal tract in order to produce meaningful speech sounds is a complex process which requires to learn the relationship between motor and sensory processes. In this paper, a computational framework is proposed that models the problem of learning articulatory control for a physiologically plausible 3-D vocal tract model using a developmentally-inspired approach. The system babbles and explores efficiently in a low-dimensional space of goals that are relevant to the learner in its synthetic environment. The learning process is goal-directed and self-organized, and yields an inverse model of the mapping between sensory space and motor commands. This study provides a unified framework that can be used for learning static as well as dynamic motor representations. The successful learning of vowel and syllable sounds as well as the benefit of active and adaptive learning strategies are demonstrated. Categorical perception is found in the acquired models, suggesting that the framework has the potential to replicate phenomena of human speech acquisition.

Semantic Active Visual Search System Based on Text Information for Large and Unknown Environments.

Different high-level robotics tasks require the robot to manipulate or interact with objects that are in an unexplored part of the environment or not already in its field of view. Although much works rely on searching for objects based on their colour or 3D context, we argue that text information is a useful and functional visual cue to guide the search. In this paper, we study the problem of active visual search (AVS) in large unknown environments. In this paper, we present an AVS system that relies on semantic information inferred from texts found in the environment, which allows the robot to reduce the search costs by avoiding not promising regions for the target object. Our semantic planner reasons over the numbers detected from door signs to decide either perform a goal-directed exploration towards unknown parts of the environment or carefully search in the already known parts. We compared the performance of our semantic AVS system with two other search systems in four simulated environments. First, we developed a greedy search system that does not consider any semantic information, and second, we invited human participants to teleoperate the robot while performing the search. Our results from simulation and real-world experiments show that text is a promising source of information that provides different semantic cues for AVS systems.

Intrinsic motivation and episodic memories for robot exploration of high-dimensional sensory spaces

This work presents an architecture that generates curiosity-driven goal-directed exploration behaviours for an image sensor of a microfarming robot. A combination of deep neural networks for offline unsupervised learning of low-dimensional features from images and of online learning of shallow neural networks representing the inverse and forward kinematics of the system have been used. The artificial curiosity system assigns interest values to a set of pre-defined goals and drives the exploration towards those that are expected to maximise the learning progress. We propose the integration of an episodic memory in intrinsic motivation systems to face catastrophic forgetting issues, typically experienced when performing online updates of artificial neural networks. Our results show that adopting an episodic memory system not only prevents the computational models from quickly forgetting knowledge that has been previously acquired but also provides new avenues for modulating the balance between plasticity and stability of the models.

Computational mechanisms of curiosity and goal-directed exploration.

Successful behaviour depends on the right balance between maximising reward and soliciting information about the world. Here, we show how different types of information-gain emerge when casting behaviour as surprise minimisation. We present two distinct mechanisms for goal-directed exploration that express separable profiles of active sampling to reduce uncertainty. 'Hidden state' exploration motivates agents to sample unambiguous observations to accurately infer the (hidden) state of the world. Conversely, 'model parameter' exploration, compels agents to sample outcomes associated with high uncertainty, if they are informative for their representation of the task structure. We illustrate the emergence of these types of information-gain, termed active inference and active learning, and show how these forms of exploration induce distinct patterns of 'Bayes-optimal' behaviour. Our findings provide a computational framework for understanding how distinct levels of uncertainty systematically affect the exploration-exploitation trade-off in decision-making.

Restoring striatal WAVE-1 improves maze exploration performance of GluN1 knockdown mice.

NMDA receptors are important for cognition and are implicated in neuropsychiatric disorders. GluN1 knockdown (GluN1KD) mice have reduced NMDA receptor levels, striatal spine density deficits, and cognitive impairments. However, how NMDA depletion leads to these effects is unclear. Since Rho GTPases are known to regulate spine density and cognition, we examined the levels of RhoA, Rac1, and Cdc42 signaling proteins. Striatal Rac1-pathway components are reduced in GluN1KD mice, with Rac1 and WAVE-1 deficits at 6 and 12 weeks of age. Concurrently, medium spiny neuron (MSN) spine density deficits are present in mice at these ages. To determine whether WAVE-1 deficits were causal or compensatory in relation to these phenotypes, we intercrossed GluN1KD mice with WAVE-1 overexpressing (WAVE-Tg) mice to restore WAVE-1 levels. GluN1KD-WAVE-Tg hybrids showed rescue of striatal WAVE-1 protein levels and MSN spine density, as well as selective behavioral rescue in the Y-maze and 8-arm radial maze tests. GluN1KD-WAVE-Tg mice expressed normalized WAVE-1 protein levels in the hippocampus, yet spine density of hippocampal CA1 pyramidal neurons was not significantly altered. Our data suggest a nuanced role for WAVE-1 effects on cognition and a delineation of specific cognitive domains served by the striatum. Rescue of striatal WAVE-1 and MSN spine density may be significant for goal-directed exploration and associated long-term memory in mice.

Autonomous exploration of motor skills by skill babbling

Autonomous exploration of motor skills is a key capability of learning robotic systems. Learning motor skills can be formulated as inverse modeling problem, which targets at finding an inverse model that maps desired outcomes in some task space, e.g., via points of a motion, to appropriate actions, e.g., motion control policy parameters. In this paper, autonomous exploration of motor skills is achieved by incrementally learning inverse models starting from an initial demonstration. The algorithm is referred to as skill babbling, features sample-efficient learning, and scales to high-dimensional action spaces. Skill babbling extends ideas of goal-directed exploration, which organizes exploration in the space of goals. The proposed approach provides a modular framework for autonomous skill exploration by separating the learning of the inverse model from the exploration mechanism and a model of achievable targets, i.e. the workspace. The effectiveness of skill babbling is demonstrated for a range of motor tasks comprising the autonomous bootstrapping of inverse kinematics and parameterized motion primitives.

Diagnosis and characterization of mania: Quantifying increased energy and activity in the human behavioral pattern monitor

Increased energy or activity is now an essential feature of the mania of Bipolar Disorder (BD) according to DSM-5. This study examined whether objective measures of increased energy can differentiate manic BD individuals and provide greater diagnostic accuracy compared to rating scales, extending the work of previous studies with smaller samples. We also tested the relationship between objective measures of energy and rating scales. 50 hospitalized manic BD patients were compared to healthy subjects (HCS, n=39) in the human Behavioral Pattern Monitor (hBPM) which quantifies motor activity and goal-directed behavior in an environment containing novel stimuli. Archival hBPM data from 17 schizophrenia patients were used in sensitivity and specificity analyses.Manic BD patients exhibited higher motor activity than HCS and higher novel object interactions. hBPM activity measures were not correlated with observer-rated symptoms, and hBPM activity was more sensitive in accurately classifying hospitalized BD subjects than observer ratings. Although the findings can only be generalized to inpatient populations, they suggest that increased energy, particularly specific and goal-directed exploration, is a distinguishing feature of BD mania and is best quantified by objective measures of motor activity. A better understanding is needed of the biological underpinnings of this cardinal feature.

Incremental Embodied Chaotic Exploration of Self-Organized Motor Behaviors with Proprioceptor Adaptation

This paper presents a general and fully dynamic embodied neural system, which incrementally explores and learns motor behaviours through an integrated combination of chaotic search and reflex learning. The former uses adaptive bifurcation to exploit the intrinsic chaotic dynamics arising from neuro-body-environment interactions, while the latter is based around proprioceptor adaptation. The architecture developed here allows realtime goal-directed exploration and learning of the possible motor patterns (e.g. for locomotion) of embodied systems of arbitrary morphology. Examples of its successful application to a simple biomechanical model and a simulated 3D swimming robot are given. The tractability of the biomechanical systems allows detailed analysis of the overall dynamics of the search process. This analysis points out strong parallels with evolutionary search.

Explorative learning of inverse models: A theoretical perspective

We investigate the role of redundancy for exploratory learning of inverse functions, where an agent learns to achieve goals by performing actions and observing outcomes. We present an analysis of linear redundancy and investigate goal-directed exploration approaches, which are empirically successful, but hardly theorized except negative results for special cases, and prove convergence to the optimal solution. We show that the learning curves of such processes are intrinsically low-dimensional and S-shaped, which explains previous empirical findings, and finally compare our results to non-linear domains.

Active Visual Object Search in Unknown Environments Using Uncertain Semantics

In this paper, we study the problem of active visual search (AVS) in large, unknown, or partially known environments. We argue that by making use of uncertain semantics of the environment, a robot tasked with finding an object can devise efficient search strategies that can locate everyday objects at the scale of an entire building floor, which is previously unknown to the robot. To realize this, we present a probabilistic model of the search environment, which allows for prioritizing the search effort to those parts of the environment that are most promising for a specific object type. Further, we describe a method for reasoning about the unexplored part of the environment for goal-directed exploration with the purpose of object search. We demonstrate the validity of our approach by comparing it with two other search systems in terms of search trajectory length and time. First, we implement a greedy coverage-based search strategy that is found in previous work. Second, we let human participants search for objects as an alternative comparison for our method. Our results show that AVS strategies that exploit uncertain semantics of the environment are a very promising idea, and our method pushes the state-of-the-art forward in AVS.

Active Choice of Teachers, Learning Strategies and Goals for a Socially Guided Intrinsic Motivation Learner

Abstract We present an active learning architecture that allows a robot to actively learn which data collection strategy is most efficient for acquiring motor skills to achieve multiple outcomes, and generalise over its experience to achieve new outcomes. The robot explores its environment both via interactive learning and goal-babbling. It learns at the same time when, who and what to actively imitate from several available teachers, and learns when not to use social guidance but use active goal-oriented self-exploration. This is formalised in the framework of life-long strategic learning. The proposed architecture, called Socially Guided Intrinsic Motivation with Active Choice of Teacher and Strategy (SGIM-ACTS), relies on hierarchical active decisions of what and how to learn driven by empirical evaluation of learning progress for each learning strategy. We illustrate with an experiment where a simulated robot learns to control its arm for realising two kinds of different outcomes. It has to choose actively and hierarchically at each learning episode: 1) what to learn: which outcome is most interesting to select as a goal to focus on for goal-directed exploration; 2) how to learn: which data collection strategy to use among self-exploration, mimicry and emulation; 3) once he has decided when and what to imitate by choosing mimicry or emulation, then he has to choose who to imitate, from a set of different teachers. We show that SGIM-ACTS learns significantly more efficiently than using single learning strategies, and coherently selects the best strategy with respect to the chosen outcome, taking advantage of the available teachers (with different levels of skills).

AUTONOMOUS ONLINE LEARNING OF REACHING BEHAVIOR IN A HUMANOID ROBOT

In this paper we describe an autonomous strategy which enables a humanoid robot to learn how to reach for a visually identified object in the 3D space. The robot is a 22-DOF upper-body humanoid with moving eyes, neck, arm and hand. The robot is bootstrapped with limited a-priori knowledge, sufficient to start the interaction with the environment; this interaction allows the robot to learn different sensorimotor mappings, required for reaching. The arm-head forward kinematic model and a visuo-motor inverse model are learned from sensory experience. Learning is performed purely online (without any separation between training and execution) through a goal-directed exploration of the environment. During the learning the robot is also able to build an internal representation of its reachable space.

Chaotic Exploration and Learning of Locomotion Behaviors

We present a general and fully dynamic neural system, which exploits intrinsic chaotic dynamics, for the real-time goal-directed exploration and learning of the possible locomotion patterns of an articulated robot of an arbitrary morphology in an unknown environment. The controller is modeled as a network of neural oscillators that are initially coupled only through physical embodiment, and goal-directed exploration of coordinated motor patterns is achieved by chaotic search using adaptive bifurcation. The phase space of the indirectly coupled neural-body-environment system contains multiple transient or permanent self-organized dynamics, each of which is a candidate for a locomotion behavior. The adaptive bifurcation enables the system orbit to wander through various phase-coordinated states, using its intrinsic chaotic dynamics as a driving force, and stabilizes on to one of the states matching the given goal criteria. In order to improve the sustainability of useful transient patterns, sensory homeostasis has been introduced, which results in an increased diversity of motor outputs, thus achieving multiscale exploration. A rhythmic pattern discovered by this process is memorized and sustained by changing the wiring between initially disconnected oscillators using an adaptive synchronization method. Our results show that the novel neurorobotic system is able to create and learn multiple locomotion behaviors for a wide range of body configurations and physical environments and can readapt in realtime after sustaining damage.

The evolution of a goal-directed exploration model: effects of information scent and GoBack utility on successful exploration.

We explore the match of a computational information foraging model to participant data on multi-page web search tasks and find its correlation on several important metrics to be too low to be used with confidence in the evaluation of user-interface designs. We examine the points of mismatch to inspire changes to the model in how it calculates information scent scores and how it assesses the utility of backing up from a lower-level page to a higher-level page. The outcome is a new model that qualitatively matches participant behavior better than the original model, has utility equations more appealing to "common sense" than the original equations, and significantly improves the correlation between model and participant data on our metrics.

Intrisically Motivated Goal Space Creation for Autonomous Goal-Directed Exploration in High-Dimensional Unbounded Sensorimotor Spaces

Today's robotic systems are given increasingly complex tasks in an increasing variety of situations such as object or social interaction. Many of those situations cannot be anticipated at design time : autonomous learning capacities are needed to adapt to novel, unexpected conditions. Yet, because of their complex bodies and multiple sensors, robots face highly-dimensional, unbounded, continuous sensorimotor spaces whose semantics are often unknown. Such spaces are too large to be explored exhaustively, an issue even more crucial in robotics given the expensive and slow nature of the physical interactions needed to gather training data. Learning in those spaces also raises other challenges, because robot's sensorimotors spaces are highly heterogeneous and multi-modal, with unreachable areas because of physical constraints, unlearnable areas because the actions of the agent do not have any influence on the sensors values, and yet other area where learning is made difficult by huge noise-to-signal ratios or requires the previous aquisition of other skills (e.g. learning reaching before grasping). This is why efficient explorations techniques are needed, where each interaction maximize the knowledge or competence gained through each interaction. To adress this issue, statistical learning techniques have focused on optimizing exploration policies to maximize various criteria in particular through active learning [1]-[3]. Another approach have stemmed from the field of developmental robotics, where inspiration from psychology and neuroscience research on animal and infant learning [4] [5] [6] have highlighted the importance of curiosity in skill acquisition. Several intrinsically motivated learning techniques have been proposed [10] [11] [12]. In this article, we will build on a particular intrinsically motivated, goal-oriented technique initiated by Baranes and Oudeyer [7], which defines the interest of an area of the sensorimotor space as the progress of the competence in reaching self-assigned goals in this area. This method has yielded excellent results in experiment with motor spaces of high dimension. Yet sensory spaces have remained limited to 2 or 3 dimensions, and the robot had only one type of action to consider. Moreover, the goal space was predefined by hand. We propose a broad expansion of the previous architecture, where the sensory space has 10+ dimensions, and relevant goal space are created and their interest evaluated by the algorithms through novel techniques. Additionally, we considers robotic agents that have several different actions at their disposal that can combine them temporally. To our knowledge, no existing work addresses both those challenges.

Rapid goal-directed exploration of a scene: The choice between a direct and a pragmatic scan path

Rapid goal-directed exploration of a scene: The choice between a direct and a pragmatic scan path

目標指向型探査に基づく多目的強化学習と２足歩行ロボットへの応用

目標指向型探査に基づく多目的強化学習と２足歩行ロボットへの応用

The effect of representation and knowledge on goal-directed exploration with reinforcement-learning algorithms

We analyze the complexity of on-line reinforcement-learning algorithms applied to goal-directed exploration tasks. Previous work had concluded that, even in deterministic state spaces, initially uninformed reinforcement learning was at least exponential for such problems, or that it was of polynomial worst-case time-complexity only if the learning methods were augmented. We prove that, to the contrary, the algorithms are tractable with only a simple change in the reward structure ("penalizing the agent for action executions") or in the initialization of the values that they maintain. In particular, we provide tight complexity bounds for both Watkins' Q-learning and Heger's Q-hat-learning and show how their complexity depends on properties of the state spaces. We also demonstrate how one can decrease the complexity even further by either learning action models or utilizing prior knowledge of the topology of the state spaces. Our results provide guidance for empirical reinforcement-learning researchers on how to distinguish hard reinforcement-learning problems from easy ones and how to represent them in a way that allows them to be solved efficiently.

. Differences in Exploration of Toys By One-Year-Old Children: a Korean and American Comparison

This study compared the play of Korean and American one-year-olds, to determine differences in behaviors reflective of mastery motivation. The two groups of children exhibited similar levels of mastery behavior (goal-directed exploration); however, the American children used general exploration significantly more than did the Korean children. There were significant differences as well in parental behavior during the play sessions. The two groups of parents also differed with respect both to their percep tions of their children's play and to their child-rearing attitudes.