Exploration-exploitation Trade-off Research Articles

A bias toward the unknown: individual and environmental factors influencing exploratory behavior.

With limited resources, exploring new opportunities is crucial for survival. Exploring novel options, however, comes at the cost of uncertainty. Therefore, there is a trade‐off between exploiting options with a known beneficial outcome and exploring novel options with a potentially higher gain. Computational models have suggested that novelty may promote exploratory behavior by inducing a so‐called novelty bonus through reward‐related processes. So far, few studies have provided behavioral evidence for such a novelty bonus. In this study, we aimed to investigate whether spatial novelty can stimulate exploratory behavior (Experiment 1), and whether age, novelty‐seeking, and reduced action radius or social interactions due to COVID‐19 restrictions influenced the exploration–exploitation trade‐off (Experiment 2). In both experiments, we employed a novel paradigm in which participants made binary decisions between food items, while on rare trials, a surprise option was presented. Results from Experiment 1 are in line with a novelty bonus, with spatial novelty promoting exploratory behavior. In Experiment 2, we found that exploratory behavior declined with age, high novelty seekers made more exploratory choices than low novelty seekers, and participants with a smaller action radius made fewer exploratory choices. These findings are consistent with previous findings in animals and predictions from computational models.

Reinforcement Learning Model With Dynamic State Space Tested on Target Search Tasks for Monkeys: Self-Determination of Previous States Based on Experience Saturation and Decision Uniqueness.

The real world is essentially an indefinite environment in which the probability space, i. e., what can happen, cannot be specified in advance. Conventional reinforcement learning models that learn under uncertain conditions are given the state space as prior knowledge. Here, we developed a reinforcement learning model with a dynamic state space and tested it on a two-target search task previously used for monkeys. In the task, two out of four neighboring spots were alternately correct, and the valid pair was switched after consecutive correct trials in the exploitation phase. The agent was required to find a new pair during the exploration phase, but it could not obtain the maximum reward by referring only to the single previous one trial; it needed to select an action based on the two previous trials. To adapt to this task structure without prior knowledge, the model expanded its state space so that it referred to more than one trial as the previous state, based on two explicit criteria for appropriateness of state expansion: experience saturation and decision uniqueness of action selection. The model not only performed comparably to the ideal model given prior knowledge of the task structure, but also performed well on a task that was not envisioned when the models were developed. Moreover, it learned how to search rationally without falling into the exploration–exploitation trade-off. For constructing a learning model that can adapt to an indefinite environment, the method of expanding the state space based on experience saturation and decision uniqueness of action selection used by our model is promising.

Tuning the Evaporation Parameter in ACO MANET Routing Using a Satisfaction-Form Game-Theoretic Approach

A Mobile Ad Hoc Network (MANET) is a communication network that links communicating devices (nodes) and does not contain permanent infrastructure. There are no dedicated routing devices in MANETs, where the routing task is assigned to a routing algorithm installed on all communicating nodes. In this work, communicating nodes utilize one of the most widely used algorithms: Ant Colony Optimization (ACO) routing algorithms. The ACO algorithms aim to balance between exploring new routes for the communication packets vs. utilizing the best-known routes discovered during the communication session. Achieving the optimality in this tradeoff is traditionally set manually by assigning many values to some parameters and measuring the network performance after the simulation session. This manual optimality tuning approach depends on human intuition and does not cope with MANET’s dynamic topology. In this research, we introduce a novel method to find an optimal balance for the exploration-exploitation tradeoff during the communication session. We formulate weighing the benefits of exploring new routes vs. exploiting known ones upon the MANET performance as a game between the two semantic players. This equilibrium is reflected as an optimal value for the pheromone evaporation parameter of the ACO algorithm during the communication session. Experimental results show a higher performance of this online tuning algorithm than the traditional offline tuning algorithms.

Naïve individuals promote collective exploration in homing pigeons.

Group-living animals that rely on stable foraging or migratory routes can develop behavioural traditions to pass route information down to inexperienced individuals. Striking a balance between exploitation of social information and exploration for better alternatives is essential to prevent the spread of maladaptive traditions. We investigated this balance during cumulative route development in the homing pigeon Columba livia. We quantified information transfer within pairs of birds in a transmission-chain experiment and determined how birds with different levels of experience contributed to the exploration-exploitation trade-off. Newly introduced naïve individuals were initially more likely to initiate exploration than experienced birds, but the pair soon settled into a pattern of alternating leadership with both birds contributing equally. Experimental pairs showed an oscillating pattern of exploration over generations that might facilitate the discovery of more efficient routes. Our results introduce a new perspective on the roles of leadership and information pooling in the context of collective learning.

Learning Personalized Product Recommendations with Customer Disengagement

Problem definition: We study personalized product recommendations on platforms when customers have unknown preferences. Importantly, customers may disengage when offered poor recommendations. Academic/practical relevance: Online platforms often personalize product recommendations using bandit algorithms, which balance an exploration-exploitation trade-off. However, customer disengagement—a salient feature of platforms in practice—introduces a novel challenge because exploration may cause customers to abandon the platform. We propose a novel algorithm that constrains exploration to improve performance. Methodology: We present evidence of customer disengagement using data from a major airline’s ad campaign; this motivates our model of disengagement, where a customer may abandon the platform when offered irrelevant recommendations. We formulate the customer preference learning problem as a generalized linear bandit, with the notable difference that the customer’s horizon length is a function of past recommendations. Results: We prove that no algorithm can keep all customers engaged. Unfortunately, classical bandit algorithms provably overexplore, causing every customer to eventually disengage. Motivated by the structural properties of the optimal policy in a scalar instance of our problem, we propose modifying bandit learning strategies by constraining the action space up front using an integer program. We prove that this simple modification allows our algorithm to perform well by keeping a significant fraction of customers engaged. Managerial implications: Platforms should be careful to avoid overexploration when learning customer preferences if customers have a high propensity for disengagement. Numerical experiments on movie recommendations data demonstrate that our algorithm can significantly improve customer engagement.

Age-based maintenance under population heterogeneity: Optimal exploration and exploitation

We consider a system with a finite lifespan and a single critical component that is subject to random failures. An age-based replacement policy is applied to preventively replace the component before its failure. The components used for replacement come from either a weak population or a strong population, referred to as population heterogeneity. However, the true population type is unknown to the decision maker. By considering that the decision maker has a belief on the probability of having a weak population, we build a partially observable Markov decision process model with the objective of minimizing the total cost over the lifespan of the system. The resulting optimal policy updates the belief variable in a Bayesian fashion by using the data obtained over the course of the system lifespan, and it denotes when to execute preventive replacement. It optimally balances the trade-off between the cost of learning the true population type (via deliberately delaying the preventive replacement time to better learn the population type) and the cost of maintenance activities. By addressing this so-called exploration-exploitation trade-off, we generate insights on the optimal policy and compare its performance with existing heuristic approaches from the literature. We also characterize a lower bound to the optimal cost, allowing us to determine the value of resolving the uncertainty on the population type.

Hedging the Drift: Learning to Optimize Under Nonstationarity

We introduce data-driven decision-making algorithms that achieve state-of-the-art dynamic regret bounds for a collection of nonstationary stochastic bandit settings. These settings capture applications such as advertisement allocation, dynamic pricing, and traffic network routing in changing environments. We show how the difficulty posed by the (unknown a priori and possibly adversarial) nonstationarity can be overcome by an unconventional marriage between stochastic and adversarial bandit learning algorithms. Beginning with the linear bandit setting, we design and analyze a sliding window-upper confidence bound algorithm that achieves the optimal dynamic regret bound when the underlying variation budget is known. This budget quantifies the total amount of temporal variation of the latent environments. Boosted by the novel bandit-over-bandit framework that adapts to the latent changes, our algorithm can further enjoy nearly optimal dynamic regret bounds in a (surprisingly) parameter-free manner. We extend our results to other related bandit problems, namely the multiarmed bandit, generalized linear bandit, and combinatorial semibandit settings, which model a variety of operations research applications. In addition to the classical exploration-exploitation trade-off, our algorithms leverage the power of the “forgetting principle” in the learning processes, which is vital in changing environments. Extensive numerical experiments with synthetic datasets and a dataset of an online auto-loan company during the severe acute respiratory syndrome (SARS) epidemic period demonstrate that our proposed algorithms achieve superior performance compared with existing algorithms. This paper was accepted by George J. Shanthikumar, Management Science Special Section on Data-Driven Prescriptive Analytics.

Uncertainty quantification and exploration–exploitation trade-off in humans

The main objective of this paper is to outline a theoretical framework to analyse how humans’ decision-making strategies under uncertainty manage the trade-off between information gathering (exploration) and reward seeking (exploitation). A key observation, motivating this line of research, is the awareness that human learners are amazingly fast and effective at adapting to unfamiliar environments and incorporating upcoming knowledge: this is an intriguing behaviour for cognitive sciences as well as an important challenge for Machine Learning. The target problem considered is active learning in a black-box optimization task and more specifically how the exploration/exploitation dilemma can be modelled within Gaussian Process based Bayesian Optimization framework, which is in turn based on uncertainty quantification. The main contribution is to analyse humans’ decisions with respect to Pareto rationality where the two objectives are improvement expected and uncertainty quantification. According to this Pareto rationality model, if a decision set contains a Pareto efficient (dominant) strategy, a rational decision maker should always select the dominant strategy over its dominated alternatives. The distance from the Pareto frontier determines whether a choice is (Pareto) rational (i.e., lays on the frontier) or is associated to “exasperate” exploration. However, since the uncertainty is one of the two objectives defining the Pareto frontier, we have investigated three different uncertainty quantification measures and selected the one resulting more compliant with the Pareto rationality model proposed. The key result is an analytical framework to characterize how deviations from “rationality” depend on uncertainty quantifications and the evolution of the reward seeking process.

History-Aware Online Cache Placement in Fog-Assisted IoT Systems: An Integration of Learning and Control

In fog-assisted Internet-of-Things systems, it is a common practice to cache popular content at the network edge to achieve high quality of service. Due to uncertainties, in practice, such as unknown file popularities, the cache placement scheme design is still an open problem with unresolved challenges: 1) how to maintain time-averaged storage costs under budgets; 2) how to incorporate online learning to aid cache placement to minimize performance loss [also known as (a.k.a.) regret]; and 3) how to exploit offline historical information to further reduce regret. In this article, we formulate the cache placement problem with unknown file popularities as a constrained combinatorial multiarmed bandit problem. To solve the problem, we employ virtual queue techniques to manage time-averaged storage cost constraints, and adopt history-aware bandit learning methods to integrate offline historical information into the online learning procedure to handle the exploration–exploitation tradeoff. With an effective combination of online control and history-aware online learning, we devise a cache placement scheme with history-aware bandit learning called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CPHBL</i> . Our theoretical analysis and simulations show that CPHBL achieves a sublinear time-averaged regret bound. Moreover, the simulation results verify CPHBL’s advantage over the deep reinforcement learning-based approach.

Action Selection for MDPs: Anytime AO* Versus UCT

In the presence of non-admissible heuristics, A* and other best-first algorithms can be converted into anytime optimal algorithms over OR graphs, by simply continuing the search after the first solution is found. The same trick, however, does not work for best-first algorithms over AND/OR graphs, that must be able to expand leaf nodes of the explicit graph that are not necessarily part of the best partial solution. Anytime optimal variants of AO* must thus address an exploration-exploitation tradeoff: they cannot just ”exploit”, they must keep exploring as well. In this work, we develop one such variant of AO* and apply it to finite-horizon MDPs. This Anytime AO* algorithm eventually delivers an optimal policy while using non-admissible random heuristics that can be sampled, as when the heuristic is the cost of a base policy that can be sampled with rollouts. We then test Anytime AO* for action selection over large infinite-horizon MDPs that cannot be solved with existing off-line heuristic search and dynamic programming algorithms, and compare it with UCT.

An Integration of Online Learning and Online Control for Green Offloading in Fog-Assisted IoT Systems

In fog-assisted IoT systems, it is a common practice to offload tasks from IoT devices to their nearby fog nodes to reduce task processing latencies and energy consumptions. However, the design of online energy-efficient scheme is still an open problem because of various uncertainties in system dynamics such as processing capacities and transmission rates. Moreover, the decision-making process is constrained by resource limits on fog nodes and IoT devices, making the design even more complicated. In this paper, we formulate such a task offloading problem with unknown system dynamics as a combinatorial multi-armed bandit (CMAB) problem with time-averaged energy consumption constraints. Through an effective integration of online learning and online control, we propose a Learning-Aided Green Offloading (LAGO) scheme. In LAGO, we employ bandit learning methods to handle the exploitation-exploration tradeoff and utilize virtual queue techniques to deal with the time-averaged constraints. Our theoretical analysis shows that LAGO reduces the average task latency with a tunable sublinear regret bound over a finite time horizon and satisfies the time-averaged energy constraints. We conduct extensive simulations to verify such theoretical results.

An empirical evaluation of active inference in multi-armed bandits

A key feature of sequential decision making under uncertainty is a need to balance between exploiting—choosing the best action according to the current knowledge, and exploring—obtaining information about values of other actions. The multi-armed bandit problem, a classical task that captures this trade-off, served as a vehicle in machine learning for developing bandit algorithms that proved to be useful in numerous industrial applications. The active inference framework, an approach to sequential decision making recently developed in neuroscience for understanding human and animal behaviour, is distinguished by its sophisticated strategy for resolving the exploration–exploitation trade-off. This makes active inference an exciting alternative to already established bandit algorithms. Here we derive an efficient and scalable approximate active inference algorithm and compare it to two state-of-the-art bandit algorithms: Bayesian upper confidence bound and optimistic Thompson sampling. This comparison is done on two types of bandit problems: a stationary and a dynamic switching bandit. Our empirical evaluation shows that the active inference algorithm does not produce efficient long-term behaviour in stationary bandits. However, in the more challenging switching bandit problem active inference performs substantially better than the two state-of-the-art bandit algorithms. The results open exciting venues for further research in theoretical and applied machine learning, as well as lend additional credibility to active inference as a general framework for studying human and animal behaviour.

From Arms to Trees: Opportunity Costs and Path Dependence and the Exploration-Exploitation Tradeoff

The literature on the exploration-exploitation tradeoff has anchored on the n-armed bandit problem as its canonical formal representation. This structure, however, omits a fundamental property of evolutionary dynamics. Contrary to a bandit formulation, foregoing an opportunity may negate the possibility of engaging in that opportunity in the future, not just modifying the beliefs about the attractiveness of engaging in that opportunity. Thus, the bandit structure only incorporates path dependence with respect to beliefs and not with regard to capabilities as our usual conceptions of dynamics of learning and capabilities would suggest. Furthermore, the consideration of opportunity cost is rather static and does not address the dynamic unfolding of opportunity structures. The nature of path dependence and opportunity costs are used to frame many of our existing conceptualizations of search processes and firm dynamics, including bandit models, real options, pivoting, the “secretary problem,” and “island” models of firm diversification. The discussion points to the need to develop canonical models of what evolutionary biologists’ term phylogenetic trees and opens up a set of new questions, such as what is the degree of parallelism of trajectories that is possible within an organization, what is the fecundity of different trajectories in terms of likelihood of branching possibilities arising, and how are these latent branching opportunities accessed?

Seamlessly Unifying Attributes and Items: Conversational Recommendation for Cold-start Users

Static recommendation methods like collaborative filtering suffer from the inherent limitation of performing real-time personalization for cold-start users. Online recommendation, e.g., multi-armed bandit approach, addresses this limitation by interactively exploring user preference online and pursuing the exploration-exploitation (EE) trade-off. However, existing bandit-based methods model recommendation actions homogeneously. Specifically, they only consider the items as the arms, being incapable of handling the item attributes , which naturally provide interpretable information of user’s current demands and can effectively filter out undesired items. In this work, we consider the conversational recommendation for cold-start users, where a system can both ask the attributes from and recommend items to a user interactively. This important scenario was studied in a recent work [54]. However, it employs a hand-crafted function to decide when to ask attributes or make recommendations. Such separate modeling of attributes and items makes the effectiveness of the system highly rely on the choice of the hand-crafted function, thus introducing fragility to the system. To address this limitation, we seamlessly unify attributes and items in the same arm space and achieve their EE trade-offs automatically using the framework of Thompson Sampling. Our Conversational Thompson Sampling (ConTS) model holistically solves all questions in conversational recommendation by choosing the arm with the maximal reward to play. Extensive experiments on three benchmark datasets show that ConTS outperforms the state-of-the-art methods Conversational UCB (ConUCB) [54] and Estimation—Action—Reflection model [27] in both metrics of success rate and average number of conversation turns.

Adaptive exploration policy for exploration–exploitation tradeoff in continuous action control optimization

Adaptive exploration policy for exploration–exploitation tradeoff in continuous action control optimization

Learning a functional control for high-frequency finance

We use a deep neural network to generate controllers for optimal trading on high-frequency data. For the first time, a neural network learns the mapping between the preferences of the trader, i.e. risk aversion parameters, and the optimal controls. An important challenge in learning this mapping is that, in intra-day trading, trader's actions influence price dynamics in closed loop via the market impact. The exploration–exploitation tradeoff generated by the efficient execution is addressed by tuning the trader's preferences to ensure long enough trajectories are produced during the learning phase. The issue of scarcity of financial data is solved by transfer learning: the neural network is first trained on trajectories generated thanks to a Monte-Carlo scheme, leading to a good initialization before training on historical trajectories. Moreover, to answer to genuine requests of financial regulators on the explainability of machine learning generated controls, we project the obtained ‘blackbox controls’ on the space usually spanned by the closed-form solution of the stylized optimal trading problem, leading to a transparent structure. For more realistic loss functions that have no closed-form solution, we show that the average distance between the generated controls and their explainable version remains small. This opens the door to the acceptance of ML-generated controls by financial regulators.

The exploration-exploitation trade-off in a foraging task is affected by mood-related arousal and valence

The exploration-exploitation trade-off shows conceptual, functional, and neural analogies with the persistence-flexibility trade-off. We investigated whether mood, which is known to modulate the persistence-flexibility balance, would similarly affect the exploration-exploitation trade-off in a foraging task. More specifically, we tested whether interindividual differences in foraging behavior can be predicted by mood-related arousal and valence. In 119 participants, we assessed mood-related interindividual differences in exploration-exploitation using a foraging task that included minimal task constraints to reduce paradigm-induced biases of individual control tendencies. We adopted the marginal value theorem as a model-based analysis approach, which approximates optimal foraging behavior by tackling the patch-leaving problem. To assess influences of mood on foraging, participants underwent either a positive or negative mood induction. Throughout the experiment, we assessed arousal and valence levels as predictors for explorative/exploitative behavior. Our mood manipulation affected participants' arousal and valence ratings as expected. Moreover, mood-related arousal was found to predict exploration while valence predicted exploitation, which only partly matched our expectations and thereby the proposed conceptual overlap with flexibility and persistence, respectively. The current study provides a first insight into how processes related to arousal and valence differentially modulate foraging behavior. Our results imply that the relationship between exploration-exploitation and flexibility-persistence is more complicated than the semantic overlap between these terms might suggest, thereby calling for further research on the functional, neural, and neurochemical underpinnings of both trade-offs.

Robust risk-averse multi-armed bandits with application in social engagement behavior of children with autism spectrum disorder while imitating a humanoid robot

The stochastic multi-armed bandit problem is a standard model to solve the exploration–exploitation trade-off in sequential decision problems. In clinical trials, which are sensitive to outlier data, the goal is to learn a risk-averse policy to provide a trade-off between exploration, exploitation, and safety. In this paper, we present a risk-averse multi-armed bandit algorithm to solve a decision-making problem based on the social engagement behaviors of children with Autism Spectrum Disorder (ASD). The algorithm is carried out when children interact with a humanoid robot and imitate a sequence of the robot's movements. The proposed algorithm is based on the Best Empirical Sampled Average algorithm under Entropic Value-at-Risk as a risk measure to decide on the best sequence of movements that can improve the social engagement behaviors of the children with ASD while imitating the robot's movements. We provide a detailed experimental analysis to compare the performance of our proposed algorithm to some well-known risk-averse multi-armed bandit algorithms on some artificial scenarios and our real-world problem. The experimental results report that the proposed algorithm outperforms its competitors in terms of robustness, risk avoidance, and cumulative regret, promoting the social engagement behaviors of children with ASD when imitating a robot's movements.

Reinforced Imitative Graph Representation Learning for Mobile User Profiling: An Adversarial Training Perspective

In this paper, we study the problem of mobile user profiling, which is a critical component for quantifying users' characteristics in the human mobility modeling pipeline. Human mobility is a sequential decision-making process dependent on the users' dynamic interests. With accurate user profiles, the predictive model can perfectly reproduce users' mobility trajectories. In the reverse direction, once the predictive model can imitate users' mobility patterns, the learned user profiles are also optimal. Such intuition motivates us to propose an imitation-based mobile user profiling framework by exploiting reinforcement learning, in which the agent is trained to precisely imitate users' mobility patterns for optimal user profiles. Specifically, the proposed framework includes two modules: (1) representation module, that produces state combining user profiles and spatio-temporal context in real-time; (2) imitation module, where Deep Q-network (DQN) imitates the user behavior (action) based on the state that is produced by the representation module. However, there are two challenges in running the framework effectively. First, epsilon-greedy strategy in DQN makes use of the exploration-exploitation trade-off by randomly pick actions with the epsilon probability. Such randomness feeds back to the representation module, causing the learned user profiles unstable. To solve the problem, we propose an adversarial training strategy to guarantee the robustness of the representation module. Second, the representation module updates users' profiles in an incremental manner, requiring integrating the temporal effects of user profiles. Inspired by Long-short Term Memory (LSTM), we introduce a gated mechanism to incorporate new and old user characteristics into the user profile. In the experiment, we evaluate our proposed framework on real-world datasets. The extensive experimental results validate the superiority of our method comparing to baseline algorithms.

Reward-Biased Maximum Likelihood Estimation for Linear Stochastic Bandits

Modifying the reward-biased maximum likelihood method originally proposed in the adaptive control literature, we propose novel learning algorithms to handle the explore-exploit trade-off in linear bandits problems as well as generalized linear bandits problems. We develop novel index policies that we prove achieve order-optimality, and show that they achieve empirical performance competitive with the state-of-the-art benchmark methods in extensive experiments. The new policies achieve this with low computation time per pull for linear bandits, and thereby resulting in both favorable regret as well as computational efficiency.