Exploration-exploitation Trade-off Research Articles

An Enhanced Multiple Unmanned Aerial Vehicle Swarm Formation Control Using a Novel Fractional Swarming Strategy Approach

This paper addresses the enhancement of multiple Unmanned Aerial Vehicle (UAV) swarm formation control in challenging terrains through the novel fractional memetic computing approach known as fractional-order velocity-pausing particle swarm optimization (FO-VPPSO). Existing particle swarm optimization (PSO) algorithms often suffer from premature convergence and an imbalanced exploration–exploitation trade-off, which limits their effectiveness in complex optimization problems such as UAV swarm control in rugged terrains. To overcome these limitations, FO-VPPSO introduces an adaptive fractional order β and a velocity pausing mechanism, which collectively enhance the algorithm’s adaptability and robustness. This study leverages the advantages of a meta-heuristic computing approach; specifically, fractional-order velocity-pausing particle swarm optimization is utilized to optimize the flying path length, mitigate the mountain terrain costs, and prevent collisions within the UAV swarm. Leveraging fractional-order dynamics, the proposed hybrid algorithm exhibits accelerated convergence rates and improved solution optimality compared to traditional PSO methods. The methodology involves integrating terrain considerations and diverse UAV control parameters. Simulations under varying conditions, including complex terrains and dynamic threats, substantiate the effectiveness of the approach, resulting in superior fitness functions for multi-UAV swarms. To validate the performance and efficiency of the proposed optimizer, it was also applied to 13 benchmark functions, including uni- and multimodal functions in terms of the mean average fitness value over 100 independent trials, and furthermore, an improvement at percentages of 29.05% and 2.26% is also obtained against PSO and VPPSO in the case of the minimum flight length, as well as 16.46% and 1.60% in mountain terrain costs and 55.88% and 31.63% in collision avoidance. This study contributes valuable insights to the optimization challenges in UAV swarm-formation control, particularly in demanding terrains. The FO-VPPSO algorithm showcases potential advancements in swarm intelligence for real-world applications.

Comparing Human Behavior to an Optimal Policy for Innovation

Human learning does not stop at solving a single problem. Instead, we seek new challenges, define new goals, and come up with new ideas. Unlike the classic explore-exploit trade-off between known and unknown options, making new tools or generating new ideas is not about collecting data from existing unknown options, but rather about create new options out of what is currently available. We introduce a discovery game designed to study how rational agents make decisions about pursuing innovations, where discovering new ideas is a process of combining existing ideas in an open-ended compositional space. We derive optimal policies of this decision problem formalized as a Markov decision process, and compare people's behaviors to the model predictions in an online behavioral experiment. We found evidence that people both innovate rationally, guided by potential returns in this discovery game, and under- and over-explore systematically in different settings.

Optimizing Movie Recommendation Systems with Multi-Armed Bandit Algorithms

In the current digital era, with the proliferation of online content, users are often overwhelmed by an abundance of digital advertisements, a consequence of the lower costs associated with digital channels compared to traditional media. Amidst this information deluge, recommendation systems have become pivotal in offering tailored suggestions to users. The Multi-Armed Bandit (MAB) problem, rooted in reinforcement learning, presents an effective approach for addressing the exploration-exploitation trade-offs inherent in recommendation systems. This paper specifically examines the implementation of MAB algorithms within the context of movie recommendation systems. A thorough literature review is conducted, focusing on the synergy between reinforcement learning and MAB algorithms, their role in recommendation systems, and an overview of fundamental MAB algorithms. To assess the efficacy of these algorithms, an experimental approach is designed, leveraging real-world datasets. This approach encompasses the formulation of the problem, data collection methods, selection of hyperparameters, and analysis of empirical results. The findings from these experiments indicate that Thompson Sampling stands out as the most effective MAB algorithm in the context of the studied datasets. The results of this study suggest that MAB algorithms have considerable potential to enhance the effectiveness of movie recommendation systems. Furthermore, the methodology and framework proposed in this paper offer valuable insights for integrating and combining multiple algorithms in complex decision-making scenarios.

Advancements in Reinforcement Learning

Reinforcement Learning (RL) stands at the forefront of machine learning, offering a paradigm where agents learn to navigate complex environments through trial and error interactions. This paper provides a comprehensive overview of RL concepts, algorithms, challenges, and future directions. Key concepts such as agent-environment interaction, Markov Decision Processes (MDPs), value-based and policy-based methods are elucidated. Challenges including the exploration-exploitation trade-off, sample efficiency, and safety concerns are discussed, alongside potential breakthroughs in deep RL, handling continuous action spaces, and dealing with partial observability. The integration of RL with emerging technologies like IoT and its implications for real-world applications such as robotics, autonomous vehicles, and healthcare are explored. Finally, future trends and directions in RL research, including AI engineering and automated feature engineering, are outlined, highlighting the potential for transformative advancements and their impact on various domains. This paper serves as a comprehensive resource for researchers, practitioners, and enthusiasts interested in the evolving landscape of Reinforcement Learning

Socio-economic constraints to low-carbon transitions: insights from Kazakhstan’s Emissions Trading Scheme

ABSTRACT In this research, we examine energy facilities’ (both electricity and heat) responses to emissions trading in Kazakhstan, a fossil-fuel-dependent developing country, to understand why facilities have delayed the adoption of substantial carbon mitigation measures. Our research relies on the frameworks of sociotechnical systems (STS) and firm-level exploration-exploitation trade-offs (EET) because energy transitions are dependent on firm-level characteristics as well as industry – and society-level environments. We find poorly designed and inconsistent regulations, ambivalence to the carbon emissions, challenges from vested interests, energy security regulations and desire for stable socio-economic conditions to be the main society-level factors impeding the effectiveness of the Kazakhstan Emissions Trading Scheme (KazETS). The main industry-level factors hindering the KazETS were found to be the involvement of the state-owned enterprise (SOE) Samruk Energo which accesses government funding for coal-based facility investment, abundant coal reserves that are extracted at low cost, increasing costs and decreasing quality of fuel, relatively very high cost of natural gas, increasing demand for electricity and heat, lack of market competition and no international firms operating in energy generation. Finally, the main firm-level factors restricting investment in low-carbon technologies are low market incentives for investment to replace technologically obsolete assets, the lack of firm-level carbon strategies and very low capacity of workers.

Adaptive feature selection for active trachoma image classification

Trachoma is a neglected tropical eye disease caused by ocular strains of Chlamydia trachomatis, which affects millions of people worldwide. To examine the eye for signs of active trachoma, healthcare providers typically look for clusters of five or more follicles on the conjunctiva of the upper eyelid for the follicular inflammatory trachoma stage. However, it is also possible to find individual follicles scattered throughout the conjunctiva, particularly in mild or early-stage trachoma cases. Additionally, the datasets are photographic images collected in the field that can be high-dimensional and may contain large amounts of redundant information. We propose integrating novel attention-based feature extraction and feature selection techniques to address these challenges. First, we present the Lambda layer within the Convolutional Block Attention Module (L-CBAM) to normalize attention weights and improve the feature extraction process. Second, we introduce an adaptive mechanism, Adaptive Beta Hill Climbing (AβHC) with Social Ski-Driver (SSD), which adjusts the exploration-exploitation trade-off during the search process, allowing for better exploration of the search space and more efficient convergence toward an optimal feature subset. We then use the multilayer perceptron (MLP) classifier to produce final classification results using selected subsets. We evaluated the proposed approach on active trachoma inverted eyelid images and obtained accuracy scores of 93.3% with only 19.7% of the selected features, surpassing many of the algorithms used for comparison. Our proposed method has demonstrated excellent performance compared to recent works utilizing the same datasets. The source code of this work is available at https://github.com/mshitie2/Active_Trachoma.

Optimized Federated Learning Algorithm for Breast Cancer Detection Using the Marine Predators Algorithm

Breast cancer stands as the leading cause of cancer-related fatalities among women worldwide, necessitating effective detection methods. However, the current landscape of breast cancer detection poses several challenges, including high costs, complexity, and suboptimal accuracy rates. In response, this paper presents a pioneering approach to breast cancer detection leveraging federated learning, a paradigm that allows model training across distributed data sources without centralized data sharing. Central to this novel detection model is the integration of a Convolutional Neural Network (CNN) architecture and a meta-heuristic optimization algorithm known as the marine predator's algorithm (MPA). CNNs have emerged as powerful tools for biomedical image analysis, enabling automatic feature extraction crucial for detecting abnormalities in medical images. However, optimizing CNNs for detection tasks requires meticulous tuning of hyperparameters across different layers, a process impractical to perform manually due to its complexity and time-intensive nature. To address this challenge, the paper proposes the utilization of the MPA as an optimization algorithm within the federated learning framework. The MPA, inspired by the foraging behaviour of marine predators, enhances the exploration-exploitation trade-off to prevent the model from getting stuck in local optima during training. By incorporating the MPA into the CNN architecture, the proposed model achieves remarkable testing accuracy, surpassing the performance of traditional DCNN models. During testing, the proposed model attained an impressive accuracy rate of 98.32%, showcasing its superior performance compared to the DCNN model, which achieved a 95.95% accuracy. Moreover, the proposed approach outperforms existing algorithms, achieving an accuracy rate of 98%. These results underscore the effectiveness of the federated learning-based approach in enhancing breast cancer detection accuracy. the proposed breast cancer detection model represents a significant advancement in the field, offering a highly accurate and efficient solution to overcome the challenges associated with traditional detection methods. By leveraging federated learning and innovative optimization techniques, this model holds promise for improving early detection rates, ultimately contributing to better patient outcomes in the fight against breast cancer

Collective incentives reduce over-exploitation of social information in unconstrained human groups

Collective dynamics emerge from countless individual decisions. Yet, we poorly understand the processes governing dynamically-interacting individuals in human collectives under realistic conditions. We present a naturalistic immersive-reality experiment where groups of participants searched for rewards in different environments, studying how individuals weigh personal and social information and how this shapes individual and collective outcomes. Capturing high-resolution visual-spatial data, behavioral analyses revealed individual-level gains—but group-level losses—of high social information use and spatial proximity in environments with concentrated (vs. distributed) resources. Incentivizing participants at the group (vs. individual) level facilitated adaptation to concentrated environments, buffering apparently excessive scrounging. To infer discrete choices from unconstrained interactions and uncover the underlying decision mechanisms, we developed an unsupervised Social Hidden Markov Decision model. Computational results showed that participants were more sensitive to social information in concentrated environments frequently switching to a social relocation state where they approach successful group members. Group-level incentives reduced participants’ overall responsiveness to social information and promoted higher selectivity over time. Finally, mapping group-level spatio-temporal dynamics through time-lagged regressions revealed a collective exploration-exploitation trade-off across different timescales. Our study unravels the processes linking individual-level strategies to emerging collective dynamics, and provides tools to investigate decision-making in freely-interacting collectives.

Graph Bayesian Optimization for Multiplex Influence Maximization

Influence maximization (IM) is the problem of identifying a limited number of initial influential users within a social network to maximize the number of influenced users. However, previous research has mostly focused on individual information propagation, neglecting the simultaneous and interactive dissemination of multiple information items. In reality, when users encounter a piece of information, such as a smartphone product, they often associate it with related products in their minds, such as earphones or computers from the same brand. Additionally, information platforms frequently recommend related content to users, amplifying this cascading effect and leading to multiplex influence diffusion. This paper first formulates the Multiplex Influence Maximization (Multi-IM) problem using multiplex diffusion models with an information association mechanism. In this problem, the seed set is a combination of influential users and information. To effectively manage the combinatorial complexity, we propose Graph Bayesian Optimization for Multi-IM (GBIM). The multiplex diffusion process is thoroughly investigated using a highly effective global kernelized attention message-passing module. This module, in conjunction with Bayesian linear regression (BLR), produces a scalable surrogate model. A data acquisition module incorporating the exploration-exploitation trade-off is developed to optimize the seed set further. Extensive experiments on synthetic and real-world datasets have proven our proposed framework effective. The code is available at https://github.com/zirui-yuan/GBIM.

Dealing with uncertainty: Balancing exploration and exploitation in deep recurrent reinforcement learning

Incomplete knowledge of the environment leads an agent to make decisions under uncertainty. One of the major dilemmas in Reinforcement Learning (RL) where an autonomous agent has to balance two contrasting needs in making its decisions is: exploiting the current knowledge of the environment to maximize the cumulative reward as well as exploring actions that allow improving the knowledge of the environment, hopefully leading to higher reward values (exploration–exploitation trade-off). Concurrently, another relevant issue regards the full observability of the states, which may not be assumed in all applications. For instance, when 2D images are considered as input in an RL approach used for finding the best actions within a 3D simulation environment. In this work, we address these issues by deploying and testing several techniques to balance exploration and exploitation trade-off on partially observable systems for predicting steering wheels in autonomous driving scenarios. More precisely, the final aim is to investigate the effects of using both adaptive and deterministic exploration strategies coupled with a Deep Recurrent Q-Network. Additionally, we adapted and evaluated the impact of a modified quadratic loss function to improve the learning phase of the underlying Convolutional Recurrent Neural Network. We show that adaptive methods better approximate the trade-off between exploration and exploitation and, in general, Softmax and Max-Boltzmann strategies outperform ϵ-greedy techniques.

Dynamic Exploration-Exploitation Trade-Off in Active Learning Regression with Bayesian Hierarchical Modeling

Active learning provides a framework to adaptively query the most informative experiments towards learning an unknown black-box function. Various approaches of active learning have been proposed in the literature, however, they either focus on exploration or exploitation in the design space. Methods that do consider exploration-exploitation simultaneously employ fixed or ad-hoc measures to control the trade-off that may not be optimal. In this paper, we develop a Bayesian hierarchical approach, referred as BHEEM, to dynamically balance the exploration-exploitation trade-off as more data points are queried. To sample from the posterior distribution of the trade-off parameter, We subsequently formulate an approximate Bayesian computation approach based on the linear dependence of queried data in the feature space. Simulated and real-world examples show the proposed approach achieves at least 21 % and 11 % average improvement when compared to pure exploration and exploitation strategies respectively. More importantly, we note that by optimally balancing the trade-off between exploration and exploitation, BHEEM performs better or at least as well as either pure exploration or pure exploitation.

A closer look at reinforcement learning-based automatic speech recognition

Reinforcement learning (RL) has demonstrated effectiveness in improving model performance and robustness for automatic speech recognition (ASR) tasks. Researchers have employed RL-based training strategies to enhance performance beyond conventional supervised or semi-supervised learning. However, existing approaches treat RL as a supplementary tool, leaving the untapped potential of RL training largely unexplored. In this paper, we formulate a novel pure RL setting where an ASR model is trained exclusively through RL via human feedback metrics, e.g., word error rate (WER) or binary reward. This approach promises to significantly simplify the annotation process if we could replace the conventional onerous annotation with a single numeric value in a human–computer interaction (HCI) way. Our experiments demonstrate the feasibility of this new setting and also identify two main inherent issues in conventional RL-based ASR training that may lead to performance degradation: (1) the mismatch issue between the action and reward has been commonly overlooked in Connectionist Temporal Classification (CTC) based models, which is attributed to the inherent CTC alignment mapping issue; (2) the classic exploration–exploitation trade-off still exists in the sampling stage of RL-based ASR, and finding the balance between them becomes a challenge. To address these issues, we first propose a new RL-based approach named CTC-aligned Policy Gradient (CTC-PG), which provides a unified formulation for different sampling strategies and alleviates the mismatch issue of action and reward in CTC-based models. Moreover, we propose Focal Sampling to balance the trade-off between exploration and exploitation with a flexible temperature parameter. Experiment results on LibriSpeech dataset showcase the effectiveness and robustness of our methods by harnessing the full potential of RL in training ASR models.

Adaptive Bayesian contextual hyperband: A novel hyperparameter optimization approach

<p>Hyperparameter tuning plays a significant role when building a machine learning or a deep learning model. The tuning process aims to find the optimal hyperparameter setting for a model or algorithm from a pre-defined search space of the hyperparameters configurations. Several tuning algorithms have been proposed in recent years and there is scope for improvement in achieving a better exploration-exploitation tradeoff of the search space. In this paper, we present a novel hyperparameter tuning algorithm named adaptive Bayesian contextual hyperband (Adaptive BCHB) that incorporates a new sampling approach to identify best regions of the search space and exploit those configurations that produce minimum validation loss by dynamically updating the threshold in every iteration. The proposed algorithm is assessed using benchmark models and datasets on traditional machine learning tasks. The proposed Adaptive BCHB algorithm shows a significant improvement in terms of accuracy and computational time for different types of hyperparameters when compared with state-of-the-art tuning algorithms.</p>

Finding the optimal exploration-exploitation trade-off online through Bayesian risk estimation and minimization

We propose endogenous Bayesian risk minimization (EBRM) over policy sets as an approach to online learning across a wide range of settings. Many real-world online learning problems have complexities such as action- and belief-dependent rewards, time-discounting of reward, and heterogeneous costs for actions and feedback; we find that existing online learning heuristics cannot leverage most problem-specific information, to the detriment of their performance. We introduce a belief-space Markov decision process (BMDP) model that can capture these complexities, and further apply the concepts of aleatoric, epistemic, and process risks to online learning. These risk functions describe the risk inherent to the learning problem, the risk due to the agent's lack of knowledge, and the relative quality of its policy, respectively. We demonstrate how computing and minimizing these risk functions guides the online learning agent towards the optimal exploration-exploitation trade-off in any stochastic online learning problem, constituting the basis of the EBRM approach. We also show how Bayes' risk, the minimization objective in stochastic online learning problems, can be decomposed into the aforementioned aleatoric, epistemic, and process risks.In simulation experiments, EBRM algorithms achieve state-of-the-art performance across various classical online learning problems, including Gaussian and Bernoulli multi-armed bandits, best-arm identification, mixed objectives with action- and belief-dependent rewards, and dynamic pricing, a finite partial monitoring problem. To our knowledge, it is also the first computationally efficient online learning approach that can provide online bounds on an algorithm's Bayes' risk. Finally, because the EBRM approach is parameterized by a set of policy algorithms, it can be extended to incorporate new developments in online learning algorithms, and is thus well-suited as the foundation for developing real-world learning agents.

Batched Neural Bandits

In many sequential decision-making problems, the individuals are split into several batches and the decision-maker is only allowed to change her policy at the end of batches. These batch problems have a large number of applications, ranging from clinical trials to crowdsourcing. Motivated by this, we study the stochastic contextual bandit problem for general reward distributions under the batched setting. We propose the BatchNeuralUCB algorithm which combines neural networks with optimism to address the exploration-exploitation tradeoff while keeping the total number of batches limited. We study BatchNeuralUCB under both fixed and adaptive batch size settings and prove that it achieves the same regret as the fully sequential version while reducing the number of policy updates considerably. We confirm our theoretical results via simulations on both synthetic and real-world datasets.

High-Dimensional Multi-Objective Bayesian Optimization With Block Coordinate Updates: Case Studies in Intelligent Transportation System

Many transportation system problems can be formulated as high-dimensional expensive multi-objective problems. They are challenging for Gaussian process-based Bayesian optimization methods to find the Pareto fronts due to the curse of dimensionality and the boundary issue in the acquisition function optimization. This paper presents a multi-objective Bayesian optimization method with block coordinate updates, Block-MOBO, to solve high-dimensional expensive multi-objective problems. Block-MOBO first partitions the decision variable space into different blocks, each of which includes a low-dimensional multi-objective problem. At each iteration, one block is considered and the decision variables not in this block are approximated by context-vector generation embedded with the Pareto prior knowledge thus promoting convergence. To tackle the boundary issue, we present <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\epsilon$</tex-math> </inline-formula> -greedy acquisition function in a Bayesian and multi-objective fashion, which recommends candidates either from the exploitation-exploration trade-off perspective or with probability <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\epsilon$</tex-math> </inline-formula> from the Pareto dominance relationship perspective. We verify the effectiveness of Block-MOBO by comparing it with other multi-objective Bayesian methods on two real-world optimization problems in transportation system and three multi-objective synthetic test suites. The experimental results show that Block-MOBO can find more evenly distributed and non-dominated solutions in the whole search space with lower complexity compared with other state-of-the-art approaches. Our analyses illustrate that block coordinate updates and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\epsilon$</tex-math> </inline-formula> -greedy acquisition function contribute to computational complexity reduction and convergence-diversity trade-offs, respectively.

Improving Dual-Population Differential Evolution Based on Hierarchical Mutation and Selection Strategy

The dual-population differential evolution (DDE) algorithm is an optimization technique that simultaneously maintains two populations to balance global and local search. It has been demonstrated to outperform single-population differential evolution algorithms. However, existing improvements to dual-population differential evolution algorithms often overlook the importance of selecting appropriate mutation and selection operators to enhance algorithm performance. In this paper, we propose a dual-population differential evolution (DPDE) algorithm based on a hierarchical mutation and selection strategy. We divided the population into elite and normal subpopulations based on fitness values. Information exchange between the two subpopulations was facilitated through a hierarchical mutation strategy, promoting a balanced exploration–exploitation trade-off in the algorithm. Additionally, this paper presents a new hierarchical selection strategy aimed at improving the population’s capacity to avoid local optima. It achieves this by accepting discarded trial vectors differently compared to previous methods. We expect that the newly introduced hierarchical selection and mutation strategies will work in synergy, effectively harnessing their potential to enhance the algorithm’s performance. Extensive experiments were conducted on the CEC 2017 and CEC 2011 test sets. The results showed that the DPDE algorithm offers competitive performance, comparable to six state-of-the-art differential evolution algorithms.

Ensemble Active Learning by Contextual Bandits for AI Incubation in Manufacturing

An Industrial Cyber-physical System (ICPS) provides a digital foundation for data-driven decision-making by artificial intelligence (AI) models. However, the poor data quality (e.g., inconsistent distribution, imbalanced classes) of high-speed, large-volume data streams poses significant challenges to the online deployment of offline-trained AI models. As an alternative, updating AI models online based on streaming data enables continuous improvement and resilient modeling performance. However, for a supervised learning model (i.e., a base learner), it is labor-intensive to annotate all streaming samples to update the model. Hence, a data acquisition method is needed to select the data for annotation to ensure data quality while saving annotation efforts. In the literature, active learning methods have been proposed to acquire informative samples. Different acquisition criteria were developed for exploration of under-represented regions in the input variable space or exploitation of the well-represented regions for optimal estimation of base learners. However, it remains a challenge to balance the exploration-exploitation trade-off under different online annotation scenarios. On the other hand, an acquisition criterion learned by AI adapts itself to a scenario dynamically, but the ambiguous consideration of the trade-off limits its performance in frequently changing manufacturing contexts. To overcome these limitations, we propose an ensemble active learning method by contextual bandits ( CbeAL ). CbeAL incorporates a set of active learning agents (i.e., acquisition criteria) explicitly designed for exploration or exploitation by a weighted combination of their acquisition decisions. The weight of each agent will be dynamically adjusted based on the usefulness of its decisions to improve the performance of the base learner. With adaptive and explicit consideration of both objectives, CbeAL efficiently guides the data acquisition process by selecting informative samples to reduce the human annotation efforts. Furthermore, we characterize the exploration and exploitation capability of the proposed agents theoretically. The evaluation results in a numerical simulation study and a real case study demonstrates the effectiveness and efficiency of CbeAL in manufacturing process modeling of the ICPS.

Exploration–Exploitation Tradeoff in the Adaptive Information Sampling of Unknown Spatial Fields with Mobile Robots

Adaptive information-sampling approaches enable efficient selection of mobile robots’ waypoints through which the accurate sensing and mapping of a physical process, such as the radiation or field intensity, can be obtained. A key parameter in the informative sampling objective function could be optimized balance the need to explore new information where the uncertainty is very high and to exploit the data sampled so far, with which a great deal of the underlying spatial fields can be obtained, such as the source locations or modalities of the physical process. However, works in the literature have either assumed the robot’s energy is unconstrained or used a homogeneous availability of energy capacity among different robots. Therefore, this paper analyzes the impact of the adaptive information-sampling algorithm’s information function used in exploration and exploitation to achieve a tradeoff between balancing the mapping, localization, and energy efficiency objectives. We use Gaussian process regression (GPR) to predict and estimate confidence bounds, thereby determining each point’s informativeness. Through extensive experimental data, we provide a deeper and holistic perspective on the effect of information function parameters on the prediction map’s accuracy (RMSE), confidence bound (variance), energy consumption (distance), and time spent (sample count) in both single- and multi-robot scenarios. The results provide meaningful insights into choosing the appropriate energy-aware information function parameters based on sensing objectives (e.g., source localization or mapping). Based on our analysis, we can conclude that it would be detrimental to give importance only to the uncertainty of the information function (which would explode the energy needs) or to the predictive mean of the information (which would jeopardize the mapping accuracy). By assigning more importance to the information uncertainly with some non-zero importance to the information value (e.g., 75:25 ratio), it is possible to achieve an optimal tradeoff between exploration and exploitation objectives while keeping the energy requirements manageable.

48 Should I Stay or Should I Go? Neural Circuits Underlying Decisions to Explore or Exploit

Objective:Adaptive decision-making is necessary to sustain functional independence. Maladaptive decisions are among the most prevalent features of psychological and neurological disorders. One crucial aspect of decision-making involves arbitrating between exploring new avenues with risky but potentially lucrative outcomes or exploiting prior knowledge and endorsing predictable outcomes. Balancing this dichotomy creates a behavioral tension that shapes all decisions and is termed the exploration-exploitation trade-off. This trade-off has been linked to reward and affective drives and associated neural circuitry as well as neuropsychological dysfunction. However, the neural mechanisms underlying the exploration-exploitation trade-off are still uncertain, due to the scarcity of literature and the heterogeneity of paradigms. This study aimed to systematically quantify and disambiguate neuroanatomical correlates of the exploration-exploitation tradeoff in a normative adult sample. These findings provide a necessary starting point for future investigations of this fundamental aspect of decision-making across clinical populations, with potential implications for assessment and intervention.Participants and Methods:We used the effect-location method of meta-analysis to analyze data from 10 functional neuroimaging studies investigating the exploration-exploitation tradeoff in non-clinical samples. We analyzed the location and frequency of significant neural activations across studies for both explorative and exploitative decisions and characterized them as core and non-core regions. Core activations were defined as those reported in over 50% of studies. Secondary and tertiary activations were defined as those reported in 40% and 30% of studies, respectively. The present review was conducted in accordance with the guidelines of the 2009 Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement.Results:The results revealed that explorative and exploitative choice behaviours differed markedly with respect to associated patterns of task-related brain activity. Exploration was associated with activity in brain regions implicated in externally directed, goal-based attentional processing and reward-related uncertainty, mainly tapping bilateral parietal and frontal circuitry, with relatively high consistency across studies. A core explorative network was revealed, consisting of activity in the frontal polar cortex, the dorsal anterior cingulate cortex, the bilateral medial frontal gyrus, the bilateral precuneus, and the bilateral intraparietal sulcus. Secondary and tertiary regions were also detected, including the bilateral anterior insula, the left precentral gyrus, the bilateral superior frontal gyrus, the right inferior frontal gyrus, the left supplementary motor area, the bilateral superior parietal lobule, and the bilateral thalamus. Exploitation was associated with brain regions implicated in internally directed processes including reward valuation, motivation, and memory. Core exploitative activations included the ventromedial prefrontal cortex, the bilateral anterior cingulate cortex, and the bilateral orbitofrontal cortex. Secondary and tertiary activations included the bilateral hippocampus, the left middle temporal gyrus, the bilateral angular gyrus, the left posterior cingulate cortex, the left superior frontal gyrus, and the bilateral superior temporal gyrus.Conclusions:The exploration-exploitation trade-off provides a novel paradigmatic approach to study adaptive and maladaptive decision-making behaviour in humans. Our findings support the neural dichotomization of exploration and exploitation and illuminate potential neural networks underlying this fundamental feature of decision-making. Understanding these mechanistic networks opens a new avenue of inquiry into decision-making deficits in clinical populations, including neurodegenerative, neurodevelopmental, and neuropsychiatric syndromes.