Exploration-exploitation Dilemma Research Articles

Behavioral Bias for Exploration Is Associated with Enhanced Signaling in the Lateral and Medial Frontopolar Cortex.

Should we keep doing what we know works for us, or should we risk trying something new as it could work even better? The exploration-exploitation dilemma is ubiquitous in daily life decision-making, and balancing between the two is crucial for adaptive behavior. Yet, we only have started to unravel the neurocognitive mechanisms that help us to find this balance in practice. Analyzing BOLD signals of healthy young adults during virtual foraging, we could show that a behavioral tendency for prolonged exploitation was associated with weakened signaling during exploration in central node points of the frontoparietal attention network, plus the frontopolar cortex. These results provide an important link between behavioral heuristics that we use to balance between exploitation and exploration and the brain function that supports shifts from one tendency to the other. Importantly, they stress that interindividual differences in behavioral strategies are reflected in differences in brain activity during exploration and should thus be more in the focus of basic research that aims at delineating general laws governing visual attention.

Innovative search and imitation heuristics: an agent-based simulation study

AbstractProminent research in strategic imitation, exploration, exploitation, and organizational learning identifies imitation as a less costly alternative to experimentation. Yet, its role in the exploration–exploitation dilemma remains underexplored in the literature. This study employs an agent-based model to examine how two distinct agent types—those who imitate and those who experiment—interact and influence each other. The model incorporates the concept of “satisficing” derived from the behavioral theory of the firm, along with insights from research on imitative heuristics. The findings reveal that overcrowding affects both agent types negatively. Imitators suffer from diminished performance due to intensified competition, which increases as more imitators join the system. Meanwhile, explorers are hindered in their attempts at radical innovation due to the presence of other explorers and clusters of imitators. This paper contributes to the field as the first to model individual agents as ‘satisficers’ within a competitive exploration–exploitation framework. By incorporating imitation, it provides novel insights into the dynamics of organizational learning and strategic decision-making.

Dysfunctional feedback processing in male methamphetamine abusers: Evidence from neurophysiological and computational approaches

Methamphetamine use disorder (MUD) as a major public health risk is associated with dysfunctional neural feedback processing. Although dysfunctional feedback processing in people who are substance dependent has been explored in several behavioral, computational, and electrocortical studies, this mechanism in MUDs requires to be well understood. Furthermore, the current understanding of latent components of their behavior such as learning speed and exploration-exploitation dilemma is still limited. In addition, the association between the latent cognitive components and the related neural mechanisms also needs to be explored. Therefore, in this study, the underlying neurocognitive mechanisms of feedback processing of such impairment, and age/gender-matched healthy controls are evaluated within a probabilistic learning task with rewards and punishments. Mathematical modeling results based on the Q-learning paradigm suggested that MUDs show less sensitivity in distinguishing optimal options. Additionally, it may be worth noting that MUDs exhibited a slight decrease in their ability to learn from negative feedback compared to healthy controls. Also through the lens of underlying neural mechanisms, MUDs showed lower theta power at the medial-frontal areas while responding to negative feedback. However, other EEG measures of reinforcement learning including feedback-related negativity, parietal-P300, and activity flow from the medial frontal to lateral prefrontal regions, remained intact in MUDs. On the other hand, the elimination of the linkage between value sensitivity and medial-frontal theta activity in MUDs was observed. The observed dysfunction could be due to the adverse effects of methamphetamine on the cortico-striatal dopamine circuit, which is reflected in the anterior cingulate cortex activity as the most likely region responsible for efficient behavior adjustment. These findings could help us to pave the way toward tailored therapeutic approaches.

Continuous Control With Swarm Intelligence Based Value Function Approximation

Value function approximation, such as Q-learning, is widely used in the discrete control rather than the continuous one because the optimal action in the discrete setting is more easily selected. Optimizing the action is a non-convex optimization problem with respect to the complex value function. Some notable studies simplify the non-convex optimization problem by assuming the value function as quadratic in the actions or by discretizing the action space. However, the performance of the output policy will decline if these studies’ premises do not hold. In order to address the problem, we propose a framework that combines swarm intelligence algorithms with value-based Reinforcement Learning, where the swarm intelligence algorithms are employed to search for the optimal action with respect to the state and the value function. To ensure the correctness of this framework, we conditionally claim the convergence rate of swarm intelligence algorithms with high probability. We then implement it by searching the batch optimal actions to various states on the GPU platform for the batch training. Furthermore, we employ the population-based atomic actions for the compatibility with the existing related work about solving discrete control problems. Four classical control models and four robot simulation environments are utilized in the comparisons. According to empirical results, our framework outputs a policy comparable with that of the policy-based algorithms by 10% timesteps in the continuous control. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper is motivated by the exploration-exploitation dilemma of Reinforcement Learning to solve continuous control tasks. To balance the exploration and exploitation, the stochastic exploration and the prioritized exploration are roughly two feasible ways, where the prioritized one is a better choice due to the higher data efficiency than the stochastic one, e.g. <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\varepsilon$</tex-math> </inline-formula> -greedy. Normally, the prioritized exploration works well in the value-based Reinforcement Learning algorithms rather than the policy-based ones; meanwhile, the policy-based algorithms are more suitable to continuous control tasks than the value-based ones. To tackle this conflict, we especially design a particle swarm optimization to maximize the Q-value of action in Q-learning. Our design can be hybridized by various swarm intelligence and value-based Reinforcement Learning algorithms. Also, it can be embedded in most intelligent control systems easily. The aim of this study is to solve the continuous control tasks by value-based algorithms as the first step of applying the prioritized exploration. The simulative results verify the effectiveness and efficiency of our design.

How does apathy impact exploration-exploitation decision-making in older patients with neurocognitive disorders?

Apathy is a pervasive clinical syndrome in neurocognitive disorders, characterized by a quantitative reduction in goal-directed behaviors. The brain structures involved in the physiopathology of apathy have also been connected to the brain structures involved in probabilistic reward learning in the exploration-exploitation dilemma. This dilemma in question involves the challenge of selecting between a familiar option with a more predictable outcome, and another option whose outcome is uncertain and may yield potentially greater rewards compared to the known option. The aim of this study was to combine experimental procedures and computational modeling to examine whether, in older adults with mild neurocognitive disorders, apathy affects performance in the exploration-exploitation dilemma. Through using a four-armed bandit reinforcement-learning task, we showed that apathetic older adults explored more and performed worse than non-apathetic subjects. Moreover, the mental flexibility assessed by the Trail-making test-B was negatively associated with the percentage of exploration. These results suggest that apathy is characterized by an increased explorative behavior and inefficient decision-making, possibly due to weak mental flexibility to switch toward the exploitation of the more rewarding options. Apathetic participants also took longer to make a choice and failed more often to respond in the allotted time, which could reflect the difficulties in action initiation and selection. In conclusion, the present results suggest that apathy in participants with neurocognitive disorders is associated with specific disturbances in the exploration-exploitation trade-off and sheds light on the disturbances in reward processing in patients with apathy.

LOGICS: Learning optimal generative distribution for designing de novo chemical structures

In recent years, the field of computational drug design has made significant strides in the development of artificial intelligence (AI) models for the generation of de novo chemical compounds with desired properties and biological activities, such as enhanced binding affinity to target proteins. These high-affinity compounds have the potential to be developed into more potent therapeutics for a broad spectrum of diseases. Due to the lack of data required for the training of deep generative models, however, some of these approaches have fine-tuned their molecular generators using data obtained from a separate predictor. While these studies show that generative models can produce structures with the desired target properties, it remains unclear whether the diversity of the generated structures and the span of their chemical space align with the distribution of the intended target molecules. In this study, we present a novel generative framework, LOGICS, a framework for Learning Optimal Generative distribution Iteratively for designing target-focused Chemical Structures. We address the exploration—exploitation dilemma, which weighs the choice between exploring new options and exploiting current knowledge. To tackle this issue, we incorporate experience memory and employ a layered tournament selection approach to refine the fine-tuning process. The proposed method was applied to the binding affinity optimization of two target proteins of different protein classes, κ-opioid receptors, and PIK3CA, and the quality and the distribution of the generative molecules were evaluated. The results showed that LOGICS outperforms competing state-of-the-art models and generates more diverse de novo chemical structures with optimized properties. The source code is available at the GitHub repository (https://github.com/GIST-CSBL/LOGICS).

Reinforcement Learning for Mobile Robotics Exploration: A Survey.

Efficient exploration of unknown environments is a fundamental precondition for modern autonomous mobile robot applications. Aiming to design robust and effective robotic exploration strategies, suitable to complex real-world scenarios, the academic community has increasingly investigated the integration of robotics with reinforcement learning (RL) techniques. This survey provides a comprehensive review of recent research works that use RL to design unknown environment exploration strategies for single and multirobots. The primary purpose of this study is to facilitate future research by compiling and analyzing the current state of works that link these two knowledge domains. This survey summarizes: what are the employed RL algorithms and how they compose the so far proposed mobile robot exploration strategies; how robotic exploration solutions are addressing typical RL problems like the exploration-exploitation dilemma, the curse of dimensionality, reward shaping, and slow learning convergence; and what are the performed experiments and software tools used for learning and testing. Achieved progress is described, and a discussion about remaining limitations and future perspectives is presented.

Incentive Mechanism for Spatial Crowdsourcing With Unknown Social-Aware Workers: A Three-Stage Stackelberg Game Approach

In this paper, we investigate the incentive problem in Spatial Crowdsourcing (SC), where mobile social-aware workers have unknown qualities and can share their answers to tasks via social networks. The objectives are to recruit high-quality workers and maximize all parties' utilities simultaneously. However, most existing works assume that the qualities of workers are known in advance or cannot take all parties' utilities into account together, especially having not considered the impact of social networks. Thus, we propose an incentive mechanism based on the multi-armed bandit and three-stage Stackelberg game, called TACT. We first design a greedy arm-pulling scheme to recruit workers, which not only can solve the exploration-exploitation dilemma but also takes workers' social relations into account. Based on the recruitment results, we further design the utility functions incorporating with social benefits for workers, and model the payment computation problem as a three-stage Stackelberg game among all participants. Next, we derive the optimal strategy group so that each party can maximize its own utility to form a multi-win situation. Moreover, we theoretically prove the unique existence of Stackelberg equilibrium and the worst regret bound. Finally, we conduct extensive simulations on a real trace to corroborate the performance of TACT.

High-functioning adults with autism spectrum disorder differ from neurotypical adults under exploration-exploitation dilemma: MEG study

High-functioning adults with autism spectrum disorder differ from neurotypical adults under exploration-exploitation dilemma: MEG study

Reinforcement learning with modified exploration strategy for mobile robot path planning

AbstractDriven by the remarkable developments we have observed in recent years, path planning for mobile robots is a difficult part of robot navigation. Artificial intelligence applied to mobile robotics is also a distinct challenge; reinforcement learning (RL) is one of the most used algorithms in robotics. The exploration-exploitation dilemma is a motivating challenge for the performance of RL algorithms. The problem is balancing exploitation and exploration, as too much exploration leads to a decrease in cumulative reward, while too much exploitation locks the agent in a local optimum. This paper proposes a new path planning method for mobile robot based on Q-learning with an improved exploration strategy. In addition, a comparative study of Boltzmann distribution and $\epsilon$ -greedy politics is presented. Through simulations, the better performance of the proposed method in terms of execution time, path length, and cost function is confirmed.

Habitat quality influences trade-offs in animal movement along the exploration–exploitation continuum

To successfully establish itself in a novel environment, an animal must make an inherent trade-off between knowledge accumulation and exploitation of knowledge gained (i.e., the exploration–exploitation dilemma). To evaluate how habitat quality affects the spatio-temporal scale of switching between exploration and exploitation during home range establishment, we conducted experimental trials comparing resource selection and space-use of translocated animals to those of reference individuals using reciprocal translocations between habitat types of differing quality. We selected wild pigs (Sus scrofa) as a model species to investigate hypotheses related to the movement behavior of translocated individuals because they are globally distributed large mammals that are often translocated within their introduced range to facilitate recreational hunting. Individuals translocated to higher quality habitat (i.e. higher proportions of bottomland hardwood habitats) exhibited smaller exploratory movements and began exploiting resources more quickly than those introduced to lower quality areas, although those in lower-quality areas demonstrated an increased rate of selection for preferred habitat as they gained knowledge of the landscape. Our data demonstrate that habitat quality mediates the spatial and temporal scale at which animals respond behaviorally to novel environments, and how these processes may determine the success of population establishment.

Realistic Actor-Critic: A framework for balance between value overestimation and underestimation.

The value approximation bias is known to lead to suboptimal policies or catastrophic overestimation bias accumulation that prevent the agent from making the right decisions between exploration and exploitation. Algorithms have been proposed to mitigate the above contradiction. However, we still lack an understanding of how the value bias impact performance and a method for efficient exploration while keeping stable updates. This study aims to clarify the effect of the value bias and improve the reinforcement learning algorithms to enhance sample efficiency. This study designs a simple episodic tabular MDP to research value underestimation and overestimation in actor-critic methods. This study proposes a unified framework called Realistic Actor-Critic (RAC), which employs Universal Value Function Approximators (UVFA) to simultaneously learn policies with different value confidence-bound with the same neural network, each with a different under overestimation trade-off. This study highlights that agents could over-explore low-value states due to inflexible under-overestimation trade-off in the fixed hyperparameters setting, which is a particular form of the exploration-exploitation dilemma. And RAC performs directed exploration without over-exploration using the upper bounds while still avoiding overestimation using the lower bounds. Through carefully designed experiments, this study empirically verifies that RAC achieves 10x sample efficiency and 25% performance improvement compared to Soft Actor-Critic in the most challenging Humanoid environment. All the source codes are available at https://github.com/ihuhuhu/RAC. This research not only provides valuable insights for research on the exploration-exploitation trade-off by studying the frequency of policies access to low-value states under different value confidence-bounds guidance, but also proposes a new unified framework that can be combined with current actor-critic methods to improve sample efficiency in the continuous control domain.

Integrating the exploration-exploitation dilemma and bad institutions to the Austrian theory of destructive entrepreneurship: a new perspective

AbstractContemporary Austrian theory has expanded widely on the relationship between entrepreneurship and the structure of production, yet it has never touched on the existence of an exploration-exploitation dilemma within organizations. The objective of the article is to show that the integration of the exploration-exploitation dilemma into the Austrian theory adds new fruitful elements to the function of the destructive entrepreneur, as presented by the Austrian economists of the firm. By showing how an organization's complexity can motivate destructive entrepreneurship on competition, it firstly explains how their analytical tools, which were limited in this area to the infra-organizational field, can also be applied to the catallactic field; it secondly enriches the traditional Austrian vision of the relation between institutions and entrepreneurship, in highlighting a reverse causal relationship, that has not yet been pointed by Austrian literature on entrepreneurship, between bad institutions and the destructive entrepreneur.

Surfing Information: The Challenge of Intelligent Decision-Making

Reinforcement learning (RL) is indispensable for building intelligent decision-making agents. However, current RL algorithms suffer from statistical and computational inefficiencies that render them useless in most real-world applications. We argue that high-value information in the real world is essential for intelligent decision-making; however, it is not addressed by most RL formalisms. Through a closer investigation of high-value information, it becomes evident that, to exploit high-value information, there is a need to formalize intelligent decision-making as bounded-optimal lifelong RL. Thus, the challenge of achieving intelligent decision-making is summarized as effectively surfing information, specifically regarding handling the non-IID (independent and identically distributed) information stream while operating with limited resources. This study discusses the design of an intelligent decision-making agent and examines its primary challenges, which are (a) online learning for non-IID data streams, (b) efficient reasoning with limited resources, and (c) the exploration–exploitation dilemma. We review relevant problems and research in the field of RL literature and conclude that current RL methods are insufficient to address these challenges. We propose that an agent capable of overcoming these challenges could effectively surf the information overload in the real world and achieve sample- and compute-efficient intelligent decision-making.

The Equations for Consciousness: A Reply to “Tracking the Travels,” a Review of Journey of the Mind

The Equations for Consciousness: A Reply to “Tracking the Travels,” a Review of <i>Journey of the Mind</i>

Active inference and the two-step task

Sequential decision problems distill important challenges frequently faced by humans. Through repeated interactions with an uncertain world, unknown statistics need to be learned while balancing exploration and exploitation. Reinforcement learning is a prominent method for modeling such behaviour, with a prevalent application being the two-step task. However, recent studies indicate that the standard reinforcement learning model sometimes describes features of human task behaviour inaccurately and incompletely. We investigated whether active inference, a framework proposing a trade-off to the exploration-exploitation dilemma, could better describe human behaviour. Therefore, we re-analysed four publicly available datasets of the two-step task, performed Bayesian model selection, and compared behavioural model predictions. Two datasets, which revealed more model-based inference and behaviour indicative of directed exploration, were better described by active inference, while the models scored similarly for the remaining datasets. Learning using probability distributions appears to contribute to the improved model fits. Further, approximately half of all participants showed sensitivity to information gain as formulated under active inference, although behavioural exploration effects were not fully captured. These results contribute to the empirical validation of active inference as a model of human behaviour and the study of alternative models for the influential two-step task.

On the strategic learning of signal associations

Abstract Signal detection theory (SDT) has been widely used to identify the optimal response of a receiver to a stimulus when it could be generated by more than one signaler type. While SDT assumes that the receiver adopts the optimal response at the outset, in reality, receivers often have to learn how to respond. We, therefore, recast a simple signal detection problem as a multi-armed bandit (MAB) in which inexperienced receivers chose between accepting a signaler (gaining information and an uncertain payoff) and rejecting it (gaining no information but a certain payoff). An exact solution to this exploration–exploitation dilemma can be identified by solving the relevant dynamic programming equation (DPE). However, to evaluate how the problem is solved in practice, we conducted an experiment. Here humans (n = 135) were repeatedly presented with a four readily discriminable signaler types, some of which were on average profitable, and others unprofitable to accept in the long term. We then compared the performance of SDT, DPE, and three candidate exploration–exploitation models (Softmax, Thompson, and Greedy) in explaining the observed sequences of acceptance and rejection. All of the models predicted volunteer behavior well when signalers were clearly profitable or clearly unprofitable to accept. Overall however, the Softmax and Thompson sampling models, which predict the optimal (SDT) response towards signalers with borderline profitability only after extensive learning, explained the responses of volunteers significantly better. By highlighting the relationship between the MAB and SDT models, we encourage others to evaluate how receivers strategically learn about their environments.

Population changes in a whale breeding ground revealed by citizen science noninvasive genetics

Historical exploitation, and a combination of current anthropogenic impacts, such as climate change and habitat degradation, impact the population dynamics of marine mammalian megafauna. Right whales ( Eubalaena spp.) are large cetaceans recovering from hunting, whose reproductive and population growth rate appear to be impacted by climate change. We apply noninvasive genetic methods to monitor southern right whale ( E. australis , SRW) and test the application of noninvasive genetics to minimise the observer effects on the population. Our aim is to describe population structure, and interdecadal and interannual changes to assess species status in the Great Acceleration period of Anthropocene. As a basis for population genetic analyses, we collected samples from sloughed skin during post-migration epidermal moult. Considering the exploration-exploitation dilemma, we collaborated with whale watching companies, as part of a citizen science approach and to reduce ad hoc logistic operations and biopsy equipment. We used mitochondrial and microsatellite data and population genetic tools. We report for the first time the genetic composition and differentiation of the Namibian portion of the range. Population genetic parameters suggest that South Africa hosts the largest population. This corresponds with higher estimates of current gene flow from Africa compared to older samples. We have observed considerable interannual variation in population density at the breeding ground and an interdecadal shift in genetic variability, evidenced by an increase in the point estimate inbreeding. Clustering analyses confirmed differentiation between the Atlantic and Indo-Pacific, presumably originating during the ice ages. We show that population monitoring of large whales, essential for their conservation management, is feasible using noninvasive sampling within non-scientific platforms. Observed patterns are concurrent to changes of movement ecology and decline in reproductive success of the South African population, probably reflecting a large-scale restructuring of pelagic marine food webs.

Chasing Unknown Bandits: Uncertainty Guidance in Learning and Decision Making

In repeated decision problems for which it is possible to learn from experience, people should actively seek out uncertain options, rather than avoid ambiguity or uncertainty, in order to learn and improve future decisions. Research on human behavior in a variety of multiarmed-bandit tasks supports this prediction. Multiarmed-bandit tasks involve repeated decisions between options with initially unknown reward distributions and require a careful balance between learning about relatively unknown options (exploration) and obtaining high immediate rewards (exploitation). Resolving this exploration-exploitation dilemma optimally requires considering not only the estimated value of each option, but also the uncertainty in these estimations. Bayesian learning naturally quantifies uncertainty and hence provides a principled framework to study how humans resolve this dilemma. On the basis of computational modeling and behavioral results in bandit tasks, I argue that human learning, attention, and exploration are guided by uncertainty. These results support Bayesian theories of cognition and underpin the fundamental role of subjective uncertainty in both learning and decision making.

The role of intolerance of uncertainty when solving the exploration-exploitation dilemma

When making behavioral decisions, individuals need to balance between exploiting known options or exploring new ones. How individuals solve this exploration-exploitation dilemma (EED) is a key research question across psychology, leading to attempting to disentangle the cognitive mechanisms behind it. A potential predictive factor of performance in an EED is intolerance of uncertainty (IU), an individual difference factor referring to the extent to which uncertain situations are reported to be aversive. Here, we present the results of a series of exploratory analyses in which we tested the relationship between IU and performance in an EED task. For this, we compiled data from 3 experiments, in which participants received the opportunity to exploit different movements in order to avoid a painful stimulus and approach rewards. For decomposing performance in this task, we used different computational models previously employed in studies on the EED. Then, the parameters of the winning model were correlated with the scores of participants in the IU scale. Correlational and cluster analyses, within both frequentists and Bayesian frameworks, did not provide strong evidence for a relation between EED and IU, apart from the decay rate and the subscale “tendency to become paralyzed in the face of uncertainty”. Given the theoretical relation between EED and IU, we propose research with different experimental paradigms.