Exploration-exploitation Trade-off Research Articles

Disentangling the roles of dopamine and noradrenaline in the exploration-exploitation tradeoff during human decision-making

Balancing the exploration of new options and the exploitation of known options is a fundamental challenge in decision-making, yet the mechanisms involved in this balance are not fully understood. Here, we aimed to elucidate the distinct roles of dopamine and noradrenaline in the exploration-exploitation tradeoff during human choice. To this end, we used a double-blind, placebo-controlled design in which participants received either a placebo, 400 mg of the D2/D3 receptor antagonist amisulpride, or 40 mg of the β-adrenergic receptor antagonist propranolol before they completed a virtual patch-foraging task probing exploration and exploitation. We systematically varied the rewards associated with choice options, the rate by which rewards decreased over time, and the opportunity costs it took to switch to the next option to disentangle the contributions of dopamine and noradrenaline to specific choice aspects. Our data show that amisulpride increased the sensitivity to all of these three critical choice features, whereas propranolol was associated with a reduced tendency to use value information. Our findings provide novel insights into the specific roles of dopamine and noradrenaline in the regulation of human choice behavior, suggesting a critical involvement of dopamine in directed exploration and a role of noradrenaline in more random exploration.

Manage Inventories with Learning on Demands and Buy-up Substitution Probability

Problem Definition: This paper considers a setting in which an airline company sells seats periodically, and each period consists of two selling phases, an early-bird discount phase and a regular-price phase. In each period, when the early-bird discount seat is stocked out, an early-bird customer who comes for the discounted seat either purchases the regular-price seat as a substitute (called buy-up substitution) or simply leaves. Methodology/Results: The optimal inventory level of the discounted seats reserved for the early-bird sale is a critical decision for the airline company to maximize its revenue. The airline company learns about the demands for both discounted and regular-price seats and the buy-up substitution probability from historical sales data, which, in turn, are affected by past inventory allocation decisions. In this paper, we investigate two information scenarios based on whether lost sales are observable, and we provide the corresponding Bayesian updating mechanism for learning about demand parameters and substitution probability. We then construct a dynamic programming model to derive the Bayesian optimal inventory level decisions in a multiperiod setting. The literature finds that the unobservability of lost sales drives the inventory manager to stock more (i.e., the Bayesian optimal inventory level should be kept higher than the myopic inventory level) to observe and learn more about demand distributions. Here, we show that when the buy-up substitution probability is known, one may stock less, because one can infer some information about the primary demand for the discounted seat from the customer substitution behavior. We also find that to learn about the unknown buy-up substitution probability drives the inventory manager to stock less so as to induce more substitution trials. Finally, we develop a SoftMax algorithm to solve our dynamic programming problem. We show that the obtained stock more (less) result can be utilized to speed up the convergence of the algorithm to the optimal solution. Managerial Implications: Our results shed light on the airline seat protection level decision with learning about demand parameters and buy-up substitution probability. Compared with myopic optimization, Bayesian inventory decisions that consider the exploration-exploitation tradeoff can avoid getting stuck in local optima and improve the revenue. We also identify new driving forces behind the stock more (less) result that complement the Bayesian inventory management literature. Funding: Z. Luo acknowledges the financial support by the Internal Start-up Fund of The Hong Kong Polytechnic University [Grant P0039035]. P. Guo acknowledges the financial support from the Research Grants Council of Hong Kong [Grant 15508518]. Y. Wang’s work was supported by the Research Grants Council of Hong Kong [Grant 15505318] and the National Natural Science Foundation of China [Grant 71971184]. Supplemental Material: The e-companion is available at https://doi.org/10.1287/msom.2022.1169 .

Examining the relationship between substance use and exploration-exploitation behavior in young adults.

Substance use is characterized by reward processing dysregulation and cognitive control deficits. One area of research that remains relatively unexplored is the relationship between substance use and exploration-exploitation trade-offs, which involve a continuum from switching (exploration) to perseverative (exploitation). In dynamic, volatile environments, exploitation of well-known options can lead to habit-driven perseveration, and exploration of new opportunities can produce new information that may enhance one's future state. The primary aim was to investigate the relationship between regular substance use and spontaneous eyeblink rate (EBR) on exploration-exploitation behavior. Young adults (N = 83) aged 18-23 completed a single laboratory session. A dynamic foraging task was used to characterize exploration/exploitation behavior. Substance use was defined using the Externalizing Spectrum Inventory-Brief Form. Dopamine levels were operationalized using spontaneous EBR. The primary outcome was proportion of switch choices on the foraging task, which reflects a continuum of exploitation (low values) to exploration (high values) behavior. A linear mixed-effects regression was conducted to examine the effect of substance use and EBR on the proportion of switch trials. Results demonstrated a significant negative interaction between substance use and EBR on proportion of switch trials (p < .001). Participants with regular substance use and low EBR showed decreased switch choices, indicative of increased exploitation, compared to those with higher EBR. EBR was positively associated with proportion of switch trials (p = .032) and thus greater exploration. The relationship between substance use and increased exploitation in young adults appears specific to those with low EBR. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Efficiently exploring for human robot interaction: partially observable Poisson processes

Consider a mobile robot exploring an office building with the aim of observing as much human activity as possible over several days. It must learn where and when people are to be found, count the observed activities, and revisit popular places at the right time. In this paper we present a series of Bayesian estimators for the levels of human activity that improve on simple counting. We then show how these estimators can be used to drive efficient exploration for human activities. The estimators arise from modelling the human activity counts as a partially observable Poisson process (POPP). This paper presents novel extensions to POPP for the following cases: (i) the robot’s sensors are correlated, (ii) the robot’s sensor model, itself built from data, is also unreliable, (iii) both are combined. It also combines the resulting Bayesian estimators with a simple, but effective solution to the exploration-exploitation trade-off faced by the robot in a real deployment. A series of 15 day robot deployments show how our approach boosts the number of human activities observed by 70% relative to a baseline and produces more accurate estimates of the level of human activity in each place and time.

Feature selection using PRACO method for IDS in cloud environment

Cloud Computing is the distribution of computing resources on demand to the users over Internet. But with virtual existence of data and resources comes the problem of privacy and security. In such environments Intrusion Detection System (IDS) comes in handy. They read huge chunks of data to find out attack patterns. But learning through this huge amount of data is very time consuming. So, data reduction is necessary. Using feature selection methods, number of features can be reduced by eliminating redundant and irrelevant attributes from datasets. In this paper the authors have proposed a Penalty Reward based Ant Colony Optimization (PRACO) method for feature selection. The penalty and reward terms used in this paper help in better exploration-exploitation trade-off by rewarding the useful features and penalizing the other ones. Along with that the concepts of max-relevance and min-redundancy are also used to indicate interactions between selected features. The proposed model is assessed on 10% KDD Cup 99, NSL-KDD and UNSW-NB15 datasets. It was observed that the PRACO method achieved 81.682% and 83.584% accuracy on average during train-test phase using NSL-KDD and UNSW-NB15 datasets. The results provide substantial evidence that the proposed model is effective in finding optimal results and thus provide IDS with increased efficiency.

When to stop searching in a highly uncertain world? A theoretical and experimental investigation of “two-way” sequential search tasks

When to stop exploring is crucial in contexts where learning to manage time and uncertainty is critical for carrying out successful initiatives (e.g., innovation, personnel recruitment, vaccine discovery). We investigate analytically and experimentally the exploration-exploitation trade-offs in such contexts. A “two-way” sequential search task is proposed, where the classical exploration-exploitation trade-off in sequential decisions with finite-horizon is coupled with a further one about discovering the real value of each alternative. The longer the time spent on a specific alternative, the higher the certainty about its expected value but at the higher cost of an under-exploitation of the best alternative so far explored. People learn better when to stop the more certain the information is. A potential behavioral trap in the exploration of “two-way” search tasks is identified that brings towards local optima. We recommend policies that induce people to reduce the time spent exploring the alternatives.

Some performance considerations when using multi-armed bandit algorithms in the presence of missing data.

When comparing the performance of multi-armed bandit algorithms, the potential impact of missing data is often overlooked. In practice, it also affects their implementation where the simplest approach to overcome this is to continue to sample according to the original bandit algorithm, ignoring missing outcomes. We investigate the impact on performance of this approach to deal with missing data for several bandit algorithms through an extensive simulation study assuming the rewards are missing at random. We focus on two-armed bandit algorithms with binary outcomes in the context of patient allocation for clinical trials with relatively small sample sizes. However, our results apply to other applications of bandit algorithms where missing data is expected to occur. We assess the resulting operating characteristics, including the expected reward. Different probabilities of missingness in both arms are considered. The key finding of our work is that when using the simplest strategy of ignoring missing data, the impact on the expected performance of multi-armed bandit strategies varies according to the way these strategies balance the exploration-exploitation trade-off. Algorithms that are geared towards exploration continue to assign samples to the arm with more missing responses (which being perceived as the arm with less observed information is deemed more appealing by the algorithm than it would otherwise be). In contrast, algorithms that are geared towards exploitation would rapidly assign a high value to samples from the arms with a current high mean irrespective of the level observations per arm. Furthermore, for algorithms focusing more on exploration, we illustrate that the problem of missing responses can be alleviated using a simple mean imputation approach.

Intelligent Action Planning for Well Construction Operations Demonstrated for Hole Cleaning Optimization and Automation

Reactive and biased human decision-making during well construction operations can result in problems ranging from minor inefficiencies to events that can have far-reaching negative consequences for safety, environmental compliance and cost. A system that can automatically generate an optimal action sequence from any given state to meet an operation’s objectives is therefore highly desirable. Moreover, an intelligent agent capable of self-learning can offset the computation and memory costs associated with evaluating the action space, which is often vast. This paper details the development of such action planning systems by utilizing reinforcement learning techniques. The concept of self-play used by game AI engines (such as AlphaGo and AlphaZero in Google’s DeepMind group) is adapted here for well construction tasks, wherein a drilling agent learns and improves from scenario simulations performed using digital twins. The first step in building such a system necessitates formulating the given well construction task as a Markov Decision Process (MDP). Planning is then accomplished using Monte Carlo tree search (MCTS), a simulation-based search technique. Simulations, based on the MCTS’s tree and rollout policies, are performed in an episodic manner using a digital twin of the underlying task(s). The results of these episodic simulations are then used for policy improvement. Domain-specific heuristics are included for further policy enhancement, considered factors such as trade-offs between safety and performance, the distance to the goal state, and the feasibility of taking specific actions from specific states. We demonstrate our proposed action planning system for hole cleaning, a task which to date has proven difficult to optimize and automate. Comparing the action sequences generated by our system to real field data, it is shown that it would have resulted in significantly improved hole cleaning performance compared to the action taken in the field, as quantified by the final state reached and the long-term reward. Such intelligent sequential decision-making systems, which use heuristics and exploration–exploitation trade-offs for optimum results, are novel applications in well construction and may pave the way for the automation of tasks that until now have been exclusively controlled by humans.

Learning on the Job: Long-Term Behavioural Adaptation in Human-Robot Interactions

In this work, we propose a framework for allowing autonomous robots deployed for extended periods of time in public spaces to adapt their own behaviour online from user interactions. The robot behaviour planning is embedded in a Reinforcement Learning (RL) framework, where the objective is maximising the level of overall user engagement during the interactions. We use the Upper-Confidence-Bound Value-Iteration (UCBVI) algorithm, which gives a helpful way of managing the exploration-exploitation trade-off for real-time interactions. An engagement model trained end-to-end generates the reward function in real-time during policy execution. We test this approach in a public museum in Lincoln (U.K.), where the robot is deployed as a tour guide for the visitors. Results show that after a couple of months of exploration, the robot policy learned to maintain the engagement of users for longer, with an increase of 22.8% over the initial static policy in the number of items visited during the tour and a 30% increase in the probability of completing the tour. This work is a promising step toward behavioural adaptation in long-term scenarios for robotics applications in social settings.

Developmental Dyslexia: Disorder or Specialization in Exploration?

We raise the new possibility that people diagnosed with developmental dyslexia (DD) are specialized in explorative cognitive search, and rather than having a neurocognitive disorder, play an essential role in human adaptation. Most DD research has studied educational difficulties, with theories framing differences in neurocognitive processes as deficits. However, people with DD are also often proposed to have certain strengths – particularly in realms like discovery, invention, and creativity – that deficit-centered theories cannot explain. We investigate whether these strengths reflect an underlying explorative specialization. We re-examine experimental studies in psychology and neuroscience using the framework of cognitive search, whereby many psychological processes involve a trade-off between exploration and exploitation. We report evidence of an explorative bias in DD-associated cognitive strategies. High DD prevalence and an attendant explorative bias across multiple areas of cognition suggest the existence of explorative specialization. An evolutionary perspective explains the combination of findings and challenges the view that individuals with DD have a disorder. In cooperating groups, individual specialization is favored when features that confer fitness benefits are functionally incompatible. Evidence for search specialization suggests that, as with some other social organisms, humans mediate the exploration–exploitation trade-off by specializing in complementary strategies. The existence of a system of collective cognitive search that emerges through collaboration would help to explain our species’ exceptional adaptiveness. It also aligns with evidence for substantial variability during our evolutionary history and the notion that humans are adapted not to a particular habitat but to variability itself. Specialization creates interdependence and necessitates balancing complementary strategies. Reframing DD therefore underscores the urgency of changing certain cultural practices to ensure we do not inhibit adaptation. Key improvements would remove cultural barriers to exploration and nurture explorative learning in education, academia, and the workplace, as well as emphasize collaboration over competition. Specialization in complementary search abilities represents a meta-adaptation; through collaboration, this likely enables human groups (as a species and as cultural systems) to successfully adapt. Cultural change to support this system of collaborative search may therefore be essential in confronting the challenges humanity now faces.

Energy-Aware Bag-of-Tasks Scheduling in the Cloud Computing System Using Hybrid Oppositional Differential Evolution-Enabled Whale Optimization Algorithm

Bag-of-Tasks (BoT) scheduling over cloud computing resources called Cloud Bag-of-Tasks Scheduling (CBS) problem, which is a well-known NP-hard optimization problem. Whale Optimization Algorithm (WOA) is an effective method for CBS problems, which still requires further improvement in exploration ability, solution diversity, convergence speed, and ensuring adequate exploration–exploitation tradeoff to produce superior scheduling solutions. In order to remove WOA limitations, a hybrid oppositional differential evolution-enabled WOA (called h-DEWOA) approach is introduced to tackle CBS problems to minimize workload makespan and energy consumption. The proposed h-DEWOA incorporates chaotic maps, opposition-based learning (OBL), differential evolution (DE), and a fitness-based balancing mechanism into the standard WOA method, resulting in enhanced exploration, faster convergence, and adequate exploration–exploitation tradeoff throughout the algorithm execution. Besides this, an efficient allocation heuristic is added to the h-DEWOA method to improve resource assignment. CEA-Curie and HPC2N real cloud workloads are used for performance evaluation of scheduling algorithms using the CloudSim simulator. Two series of experiments have been conducted for performance comparison: one with WOA-based heuristics and another with non-WOA-based metaheuristics. Experimental results of the first series of experiments reveal that the h-DEWOA approach results in makespan improvement in the range of 5.79–13.38% (for CEA-Curie workloads), 5.03–13.80% (for HPC2N workloads), and energy consumption in the range of 3.21–14.70% (for CEA-Curie workloads) and 10.84–19.30% (for HPC2N workloads) over well-known WOA-based metaheuristics. Similarly, h-DEWOA also resulted in significant performance in comparison with recent state-of-the-art non-WOA-based metaheuristics in the second series of experiments. Statistical tests and box plots also revealed the robustness of the proposed h-DEWOA algorithm.

Adaptive Sequential Experiments with Unknown Information Arrival Processes

Problem definition: Sequential experiments that are deployed in a broad range of practices are characterized by an exploration-exploitation trade-off that is well understood when in each time period feedback is received only on the action that was selected in that period. However, in many practical settings, additional information may become available between decision epochs. We study the performance that one may achieve when leveraging such auxiliary information and the design of algorithms that effectively do so without prior knowledge of the information arrival process. Methodology/results: Our formulation considers a broad class of distributions that are informative about rewards from actions and allows auxiliary observations from these distributions to arrive according to an arbitrary and a priori unknown process. When it is known how to map auxiliary observations to reward estimates, we characterize the best achievable performance as a function of the information arrival process. In terms of achieving optimal performance, we establish that upper confidence bound and Thompson sampling algorithms possess natural robustness with respect to the information arrival process, which uncovers a novel property of these popular algorithms. When the mappings connecting auxiliary observations and rewards are a priori unknown, we characterize a necessary and sufficient condition under which auxiliary information allows performance improvement and devise an adaptive policy (termed 2UCBs) that guarantees near optimality. We use a data set from a large media site to analyze the value that may be captured by leveraging auxiliary observations in the design of content recommendations. Managerial implications: Our study highlights the importance of utilizing auxiliary information in the design of sequential experiments and characterizes how salient features of the auxiliary information stream impact performance. Our study also emphasizes the risk in processing auxiliary information using nonadaptive approaches that are predicated on correct interpretation of this information, as opposed to deploying flexible, adaptive methods.

Shared mechanisms mediate the explore-exploit tradeoff in macaques and humans

Ancestors of macaques and humans separated into distinct lineages 25 million years ago. Despite this long separation, Hogeveen etal. (2022) show, in this issue of Neuron, that they mediate the explore-exploit tradeoff, which must be managed by any agent adapting to a dynamic environment, using similar computational and neural mechanisms.

An improved binary sparrow search algorithm for feature selection in data classification

Feature Selection (FS) is an important preprocessing step that is involved in machine learning and data mining tasks for preparing data (especially high-dimensional data) by eliminating irrelevant and redundant features, thus reducing the potential curse of dimensionality of a given large dataset. Consequently, FS is arguably a combinatorial NP-hard problem in which the computational time increases exponentially with an increase in problem complexity. To tackle such a problem type, meta-heuristic techniques have been opted by an increasing number of scholars. Herein, a novel meta-heuristic algorithm, called Sparrow Search Algorithm (SSA), is presented. The SSA still performs poorly on exploratory behavior and exploration-exploitation trade-off because it does not duly stimulate the search within feasible regions, and the exploitation process suffers noticeable stagnation. Therefore, we improve SSA by adopting: i) a strategy for Random Re-positioning of Roaming Agents (3RA); and ii) a novel Local Search Algorithm (LSA), which are algorithmically incorporated into the original SSA structure. To the FS problem, SSA is improved and cloned as a binary variant, namely, the improved Binary SSA (iBSSA), which would strive to select the optimal or near-optimal features from a given dataset while keeping the classification accuracy maximized. For binary conversion, the iBSSA was primarily validated against nine common S-shaped and V-shaped Transfer Functions (TFs), thus producing nine iBSSA variants. To verify the robustness of these variants, three well-known classification techniques, including k-Nearest Neighbor (k-NN), Support Vector Machine (SVM), and Random Forest (RF) were adopted as fitness evaluators with the proposed iBSSA approach and many other competing algorithms, on 18 multifaceted, multi-scale benchmark datasets from the University of California Irvine (UCI) data repository. Then, the overall best-performing iBSSA variant for each of the three classifiers was compared with binary variants of 12 different well-known meta-heuristic algorithms, including the original SSA (BSSA), Artificial Bee Colony (BABC), Particle Swarm Optimization (BPSO), Bat Algorithm (BBA), Grey Wolf Optimization (BGWO), Whale Optimization Algorithm (BWOA), Grasshopper Optimization Algorithm (BGOA) SailFish Optimizer (BSFO), Harris Hawks Optimization (BHHO), Bird Swarm Algorithm (BBSA), Atom Search Optimization (BASO), and Henry Gas Solubility Optimization (BHGSO). Based on a Wilcoxon’s non-parametric statistical test (alpha =0.05), the superiority of iBSSA with the three classifiers was very evident against counterparts across the vast majority of the selected datasets, achieving a feature size reduction of up to 92% along with up to 100% classification accuracy on some of those datasets.

The Exploration-Exploitation Trade-off in the Newsvendor Problem

When an inventory manager attempts to construct probabilistic models of demand based on past data, demand samples are almost never available: only sales data can be used. This demand censoring introduces an exploration-exploitation trade-off as the ordering decisions impact the information collected. Much of the literature has sought to understand how operational decisions should be modified to incorporate this trade-off. We ask an even more basic question: When does the exploration-exploitation trade-off matter? To what extent should one deviate from a myopic policy that takes the optimal decision for the current period without consideration for future periods? We analyze these questions in the context of a well-studied stationary multiperiod newsvendor problem in which the decision maker starts with a prior on parameters characterizing the demand distribution. We show that, under very general conditions in both perishable and nonperishable settings, the myopic policy will almost surely learn the optimal decision one would have taken with knowledge of the unknown parameters. Furthermore, in the perishable setting, we analyze finite time performance for a broad family of tractable cases. Through a combination of analytical parametric bounds and exhaustive exact analysis, we show that the myopic optimality gap is negligible for many practical instances. Funding: The third author was partially supported by the National Science Foundation [Grant CMMI-1235023]. Supplemental Material: The online supplement is available at https://doi.org/10.1287/stsy.2022.0093 .

Bayesian optimisation for efficient parameter inference in a cardiac mechanics model of the left ventricle.

We consider parameter inference in cardio‐mechanic models of the left ventricle, in particular the one based on the Holtzapfel‐Ogden (HO) constitutive law, using clinical in vivo data. The equations underlying these models do not admit closed form solutions and hence need to be solved numerically. These numerical procedures are computationally expensive making computational run times associated with numerical optimisation or sampling excessive for the uptake of the models in the clinical practice. To address this issue, we adopt the framework of Bayesian optimisation (BO), which is an efficient statistical technique of global optimisation. BO seeks the optimum of an unknown black‐box function by sequentially training a statistical surrogate‐model and using it to select the next query point by leveraging the associated exploration‐exploitation trade‐off. To guarantee that the estimates based on the in vivo data are realistic also for high‐pressures, unobservable in vivo, we include a penalty term based on a previously published empirical law developed using ex vivo data. Two case studies based on real data demonstrate that the proposed BO procedure outperforms the state‐of‐the‐art inference algorithm for the HO constitutive law.

Individual exploration and selective social learning: balancing exploration-exploitation trade-offs in collective foraging.

Search requires balancing exploring for more options and exploiting the ones previously found. Individuals foraging in a group face another trade-off: whether to engage in social learning to exploit the solutions found by others or to solitarily search for unexplored solutions. Social learning can better exploit learned information and decrease the costs of finding new resources, but excessive social learning can lead to over-exploitation and too little exploration for new solutions. We study how these two trade-offs interact to influence search efficiency in a model of collective foraging under conditions of varying resource abundance, resource density and group size. We modelled individual search strategies as Lévy walks, where a power-law exponent (μ) controlled the trade-off between exploitative and explorative movements in individual search. We modulated the trade-off between individual search and social learning using a selectivity parameter that determined how agents responded to social cues in terms of distance and likely opportunity costs. Our results show that social learning is favoured in rich and clustered environments, but also that the benefits of exploiting social information are maximized by engaging in high levels of individual exploration. We show that selective use of social information can modulate the disadvantages of excessive social learning, especially in larger groups and when individual exploration is limited. Finally, we found that the optimal combination of individual exploration and social learning gave rise to trajectories with μ ≈ 2 and provide support for the general optimality of such patterns in search. Our work sheds light on the interplay between individual search and social learning, and has broader implications for collective search and problem-solving.

Evolution of innate behavioral strategies through competitive population dynamics.

Many organism behaviors are innate or instinctual and have been “hard-coded” through evolution. Current approaches to understanding these behaviors model evolution as an optimization problem in which the traits of organisms are assumed to optimize an objective function representing evolutionary fitness. Here, we use a mechanistic birth-death dynamics approach to study the evolution of innate behavioral strategies in a simulated population of organisms. In particular, we performed agent-based stochastic simulations and mean-field analyses of organisms exploring random environments and competing with each other to find locations with plentiful resources. We find that when organism density is low, the mean-field model allows us to derive an effective objective function, predicting how the most competitive phenotypes depend on the exploration-exploitation trade-off between the scarcity of high-resource sites and the increase in birth rate those sites offer organisms. However, increasing organism density alters the most competitive behavioral strategies and precludes the derivation of a well-defined objective function. Moreover, there exists a range of densities for which the coexistence of many phenotypes persists for evolutionarily long times.

Robust experience replay sampling for multi-agent reinforcement learning

• Propose new algorithms for acquiring suitable experiences from buffer through filtering. • Strengthen exploration strategy by reducing repetitive decisions at a given state. • Improve performance which is higher than or comparable to the baseline algorithms. • Achieve early convergence and improved policy searching compared to the baselines. Learning from the relevant experiences leads to fast convergence if the experiences provide useful information. We present the new and simple yet efficient technique to find suitable samples of experiences to train the agents in a given state of an environment. We intended to increase the number of states visited and unique sequences that efficiently reduce the number of states the agents have to explore or exploit. Our technique implicitly introduces additional strength to the exploration-exploitation trade-off. It filters the samples of experiences that can benefit more than half the number of agents and then utilizes the experiences to extract useful information for decision making. First, we compute the similarities between the observed state and previous states in the experiences to achieve this filtering. Then, we filter the samples using the hyper-parameter, z , to decide which experiences will be suitable. We found out that agents learn quickly and efficiently since sampled experiences provide useful information that speeds up convergence. In every episode, most agents learn or contribute to improve the total expected future return. We further study our approaches’ generalization ability and present different settings to show significant improvements in diverse experiment environments.

A bias toward the unknown: individual and environmental factors influencing exploratory behavior.

With limited resources, exploring new opportunities is crucial for survival. Exploring novel options, however, comes at the cost of uncertainty. Therefore, there is a trade‐off between exploiting options with a known beneficial outcome and exploring novel options with a potentially higher gain. Computational models have suggested that novelty may promote exploratory behavior by inducing a so‐called novelty bonus through reward‐related processes. So far, few studies have provided behavioral evidence for such a novelty bonus. In this study, we aimed to investigate whether spatial novelty can stimulate exploratory behavior (Experiment 1), and whether age, novelty‐seeking, and reduced action radius or social interactions due to COVID‐19 restrictions influenced the exploration–exploitation trade‐off (Experiment 2). In both experiments, we employed a novel paradigm in which participants made binary decisions between food items, while on rare trials, a surprise option was presented. Results from Experiment 1 are in line with a novelty bonus, with spatial novelty promoting exploratory behavior. In Experiment 2, we found that exploratory behavior declined with age, high novelty seekers made more exploratory choices than low novelty seekers, and participants with a smaller action radius made fewer exploratory choices. These findings are consistent with previous findings in animals and predictions from computational models.