Exploitation Trade-off Research Articles

Dynamic Auctions with Demand Learning

The minimum bid is a key design variable in retail auctions as it affects the number of posted bids and the clearing price. Retailers often conduct sequential, single-unit auctions online as a revenue generation and inventory clearing tool, and need to dynamically decide the minimum bid in each auction depending on the inventory level at that time. To reduce inventory costs, it may in fact be optimal to scrap some of the inventory rather than holding it until it is auctioned off. In auctions of fashion goods, seasonal products, new products, and where the seller is new to the market, the seller may be uncertain about the market response. Such a seller may want to dynamically learn the demand by observing the number of posted bids. This introduces an additional, exploration versus exploitation trade-off. We formulate a Markov decision process (MDP) to study this dynamic auction-design problem. We consider a Vickrey mechanism, and a random number of bidders with private, independent, and identically distributed valuations. We first develop a clairvoyant model where the seller knows the demand distribution. We prove that it is optimal to scrap all inventory above a certain threshold and then auction the remaining units. We derive a first order necessary condition, which states that, in any inventory level, the bidders' virtual value (as defined by Myerson) at an optimal minimum bid should equal the seller's marginal profit. This is an intuitive generalization of Riley and Samuelson's well-known result from the one single-unit auction case. Under the standard regularity condition that the virtual value is strictly increasing, our first order necessary condition is also sufficient and leads to a structured value iteration algorithm for efficiently computing optimal policies. In that case, we prove that optimal minimum bids (i) decrease with increasing inventory levels, and (ii) are higher than optimal minimum bids in one single-unit auction. We also prove that optimal scrapping thresholds and optimal minimum bids both decrease with increasing inventory costs.We illustrate our results and perform sensitivity analyses using examples with a Poisson-distributed number of bidders and Beta-distributed private valuations.In the second half of the paper, we assume that the number of bidders is Poisson distributed but the seller does not know its mean. The seller uses a mixture-of-Gamma prior on this mean and updates this belief over the sequence of auctions while simultaneously deciding minimum bids and scrapping quantities. This results in a high-dimensional Bayesian MDP whose exact solution is intractable. We therefore implement a certainty equivalent control (CEC) heuristic instead. We extend our clairvoyant structural result to CEC and numerically investigate the effect of initial inventory and mean demand on CEC performance.

Balanced Cartesian Genetic Programming via migration and opposition-based learning: application to symbolic regression

The exploration---exploitation trade-off is an important aspect of evolutionary algorithms which determines the efficiency and accuracy of these algorithms. Cartesian Genetic Programming (CGP) is a generalization of the graph based genetic programming. It is implemented with mutation only and does not have any possibility to share information among solutions. The main goal of this paper is to present an effective method for balancing the exploration and exploitation of CGP referred to as Balanced Cartesian Genetic Programming (BCGP) by incorporating distinctive features from biogeography-based optimization (BBO) and opposition-based learning. To achieve this goal, we apply BBO's migration operator without considering any modifications in the representation of CGP. This operator has good exploitation ability and can be used to share information among individuals in CGP. In addition, in order to improve the exploration ability of CGP, a new mutation operator is integrated into CGP inspired from the concept of opposition-based learning. Experiments have been conducted on symbolic regression. The experimental results show that the proposed BCGP method outperforms the traditional CGP in terms of accuracy and the convergence speed.

Lead users, suppliers, and experts: the exploration and exploitation trade-off in product development

We develop a theoretical framework that explains how lead users, key suppliers, and product experts play into the exploration and exploitation trade-off, and how their changing roles support the development of their ideas into marketable solutions. We explain that close collaboration with product experts, suppliers and lead users supports exploitation, as it tends to enable the development and improvement of existing products. Irregular or non-systematic collaboration with lead users, on the other hand, supports exploration by providing ideas for entirely new products. Furthermore, our theoretical framework argues that the changing role of the external agents increases their understanding of the products they help to develop, supporting a process of exploration and exploitation. This learning process supports the technical abilities of the lead user group in particular, who can then use their newly acquired knowledge to further modify existing products and introduce new ones to meet their needs. At the same time, this process educates internal staff, suppliers, and product experts about the use of the product, which then enhances their ability to develop new and marketable ideas. Our proposed theoretical model has several implications for lead user theory and for product development managers seeking to enhance the development of new products and solutions, which we discuss at the end of the paper.

Exploration–exploitation trade-off features a saltatory search behaviour

Searching experiments conducted in different virtual environments over a gender-balanced group of people revealed a gender irrelevant scale-free spread of searching activity on large spatio-temporal scales. We have suggested and solved analytically a simple statistical model of the coherent-noise type describing the exploration-exploitation trade-off in humans ('should I stay' or 'should I go'). The model exhibits a variety of saltatory behaviours, ranging from Lévy flights occurring under uncertainty to Brownian walks performed by a treasure hunter confident of the eventual success.

On Implications of Demand Censoring in the Newsvendor Problem

We consider a repeated newsvendor problem in which the decision maker (DM) does not have access to the underlying demand distribution. The goal of this paper is to characterize the implications of demand censoring on performance. To that end, we compare the benchmark setting in which the DM has access to demand observations to a setting in which the DM may only rely on sales data. We measure performance in terms of regret: the difference between the cumulative costs of a policy and the optimal cumulative costs with knowledge of the demand distribution. Through upper and lower bounds, we characterize the optimal magnitude of the worst-case regret for the two settings, enabling one to isolate the implications of demand censoring. In particular, the results imply that the exploration–exploitation trade-off introduced by demand censoring is fundamentally different in the continuous and discrete demand cases, and that active exploration plays a much stronger role in the latter case. We further establish that in the discrete demand case, the need for active exploration almost disappears as soon as a lost sales indicator (that records whether demand was censored or not) becomes available, in addition to the censored demand samples. This paper was accepted by Gérard P. Cachon, stochastic models and simulation.

Keeping track of changes: the performance of ant colonies in dynamic environments

Animals living in dynamic foraging environments are subject to trade-offs between exploiting current resources and exploring alternative resources. Social insects face the additional challenge of redistributing an often large foraging force among multiple patches in the absence of leadership or centralized control. We used comparative, field-based techniques to study the dynamic performance of five ant species. We also investigated how the number of ants involved in the decision-making process influenced feeder exploitation and dynamic performance. Colonies were offered two feeders that differed in quality. After 30 min, the feeders changed quality; the challenge for the colonies was to reallocate foraging effort following the swap. We predicted that colonies that invested heavily in exploiting the high-quality resource would have difficulty reallocating foragers following the quality swap. We also predicted that larger foraging groups would have stronger exploitation intensity and better dynamic performance than smaller groups. Overall, we found evidence for a trade-off between exploitation and flexibility such that species that tended to focus strongly on high-quality resources had relatively poor dynamic performance. Contrary to our prediction, we found no evidence that the number of ants influenced dynamic performance. The number of ants did, however, influence exploitation intensity such that colonies with fewer foragers tended to allocate a greater proportion of ants to the higher quality feeder than larger colonies. Our results highlight the factors that influence how species respond to the exploration–exploitation trade-off when faced with short-term fluctuations in the foraging environment.

Rational value of information estimation for measurement selection

Computing value of information VOI is a crucial task in various aspects of decision-making under uncertainty, such as in meta-reasoning for search; in selecting measurements to make, prior to choosing a course of action; and in managing the exploration vs. exploitation tradeoff. Since such applications typically require numerous VOI computations during a single run, it is essential that VOI be computed efficiently. We explore the tradeoff between the accuracy of estimating VOI and computational resources used for the estimation, and extend the known greedy algorithm with selective estimation of VOI based on principles of limited rationality. As a case study, we examine VOI estimation in the measurement selection problem. Empirical evaluation of the proposed extension in this domain shows that computational resources can indeed be significantly reduced, at little cost in expected rewards achieved in the overall decision problem.

A requirement for the mutation operator in continuous optimization

During the recent decades, much effort has been dedicated to revise and improve the evolution operators of evolutionary algorithms. This article aims at minimums of an efficient mutation operator in continuous problems for which reachability, scalability, unbiasedness and isotropy have already been considered necessary. A new requirement, called robust exploration and exploitation trade-off, is introduced which is usually satisfied for common univariate density functions in 1D space, while for higher dimensions, these density functions should be revised, otherwise the exploration or exploitation abilities of sampling turns highly limited. Empirical simulation is carried out to check the validity of the drawn conclusions, which verifies that if modification of the density function is ignored, dramatic performance deterioration can be concluded.

Hardware reuse in modern application-specific processors and accelerators

Effective exploitation of the application-specific parallel patterns and computation operations through their direct implementation in hardware is the base for construction of high-quality application-specific (re-) configurable application specific instruction set processors (ASIPs) and hardware accelerators for modern highly-demanding applications. Although it receives a lot of attention from the researchers and practitioners, a very important problem of hardware reuse in ASIP and accelerator synthesis is clearly underestimated and does not get enough attention in the published research. This paper is an effect of an industry and academic collaborative research. It analyses the problem of hardware sharing, shows its high practical relevance, as well as a big influence of hardware sharing on the major circuit and system parameters, and its importance for the multi-objective optimization and tradeoff exploitation. It also demonstrates that the state-of-the-art synthesis tools do not sufficiently address this problem and gives several guidelines related to enhancement of the hardware reuse.

A learning approach to optimizing exploration–exploitation tradeoff in relevance feedback

Relevance feedback is an effective technique for improving search accuracy in interactive information retrieval. In this paper, we study an interesting optimization problem in interactive feedback that aims at optimizing the tradeoff between presenting search results with the highest immediate utility to a user (but not necessarily most useful for collecting feedback information) and presenting search results with the best potential for collecting useful feedback information (but not necessarily the most useful documents from a user's perspective). Optimizing such an exploration---exploitation tradeoff is key to the optimization of the overall utility of relevance feedback to a user in the entire session of relevance feedback. We formally frame this tradeoff as a problem of optimizing the diversification of search results since relevance judgments on more diversified results have been shown to be more useful for relevance feedback. We propose a machine learning approach to adaptively optimizing the diversification of search results for each query so as to optimize the overall utility in an entire session. Experiment results on three representative retrieval test collections show that the proposed learning approach can effectively optimize the exploration---exploitation tradeoff and outperforms the traditional relevance feedback approach which only does exploitation without exploration.

Pupil Diameter Predicts Changes in the Exploration–Exploitation Trade-off: Evidence for the Adaptive Gain Theory

The adaptive regulation of the balance between exploitation and exploration is critical for the optimization of behavioral performance. Animal research and computational modeling have suggested that changes in exploitative versus exploratory control state in response to changes in task utility are mediated by the neuromodulatory locus coeruleus-norepinephrine (LC-NE) system. Recent studies have suggested that utility-driven changes in control state correlate with pupil diameter, and that pupil diameter can be used as an indirect marker of LC activity. We measured participants' pupil diameter while they performed a gambling task with a gradually changing payoff structure. Each choice in this task can be classified as exploitative or exploratory using a computational model of reinforcement learning. We examined the relationship between pupil diameter, task utility, and choice strategy (exploitation vs. exploration), and found that (i) exploratory choices were preceded by a larger baseline pupil diameter than exploitative choices; (ii) individual differences in baseline pupil diameter were predictive of an individual's tendency to explore; and (iii) changes in pupil diameter surrounding the transition between exploitative and exploratory choices correlated with changes in task utility. These findings provide novel evidence that pupil diameter correlates closely with control state, and are consistent with a role for the LC-NE system in the regulation of the exploration-exploitation trade-off in humans.

The effects of 24-hour sleep deprivation on the exploration–exploitation trade-off

Sleep deprivation has a complex set of neurological effects that go beyond a mere slowing of mental processes. While cognitive and perceptual impairments in sleep deprived individuals are widespread, some abilities remain intact. In an effort to characterize these effects, some have suggested an impairment of complex decision-making ability despite intact ability to follow simple rules. To examine this trade-off, 24-hour total sleep deprived individuals performed two versions of a resource acquisition foraging task, one in which exploration is optimal (to succeed, abandon low value, high saliency options) and another in which exploitation is optimal (to succeed, refrain from switching between options). Sleep deprived subjects exhibited decreased performance on the exploitation task compared to non-sleep deprived controls, yet both groups exhibited increased performance on the exploratory task. These results speak to previous neuropsychological work on cognitive control.

THE OPTIMAL SAMPLING STRATEGY FOR UNFAMILIAR PREY

Precisely how predators solve the problem of sampling unfamiliar prey types is central to our understanding of the evolution of a variety of antipredator defenses, ranging from Müllerian mimicry to polymorphism. When predators encounter a novel prey item then they must decide whether to take a risk and attack it, thereby gaining a potential meal and valuable information, or avoid such prey altogether. Moreover, if predators initially attack the unfamiliar prey, then at some point(s) they should decide to cease sampling if evidence mounts that the type is on average unprofitable to attack. Here, I cast this problem as a "two-armed bandit," the standard metaphor for exploration-exploitation trade-offs. I assume that as predators encounter and attack unfamiliar prey they use Bayesian inference to update both their beliefs as to the likelihood that individuals of this type are chemically defended, and the probability of seeing the prey type in the future. I concurrently use dynamic programming to identify the critical informational states at which predator should cease sampling. The model explains why predators sample more unprofitable prey before complete rejection when the prey type is common and explains why predators exhibit neophobia when the unfamiliar prey type is perceived to be rare.

Two-stage reinforcement-learning-based cognitive radio with exploration control

This study presents a novel two-stage reinforcement-learning-based algorithm for distributed cognitive radio (CR) spectrum sharing. The traditional reinforcement-learning model is modified in order to be applied in a fully distributed CR scenario. CRs are able to discover the best available resources autonomously by utilising learning, which results in significantly improved performance, while reducing the need for spectrum sensing. Instead of sensing all available spectrum arbitrarily, the scheme is designed to share the spectrum based on an optimal spectrum sharing strategy, which is discovered by the CR agents from their trial-and-error interactions with the wireless communication environment. On the other hand, the inherent exploration against exploitation trade-off seen in reinforcement learning is also examined in the context of CR. A ‘warm-up’ stage is proposed to effectively control the exploration phase of the learning process. A better system performance can be expected by carefully balancing the tradeoff between exploration and exploitation. The benefit of applying a warm-up stage is demonstrated. Comparisons of system performance using different warm-up strategies are also given to illustrate their impact on the spectrum sharing process.

Evaluation of basic and alternative breeding programs for Sahiwal cattle genetic resources in Kenya

The Sahiwal cattle breeding program in Kenya has been operational on an interim basis for over 45 years. However, there have been no systematic efforts undertaken to evaluate its suitability, or to examine how competitive it is compared with other alternative programs in terms of genetic and economic merit. The objective of this study was therefore to evaluate the genetic and economic success of the current basic and alternative Sahiwal cattle breeding programs in Kenya. The breeding programs examined were the current closed nucleus with two breeding strategies: a purebreeding (CNPURE) and a crossbreeding system (CNCROSS) involving Sahiwal sires and East African Zebu dams. An open nucleus with a certain proportion of pastoral-born Sahiwal bulls introduced into the nucleus herds to produce cows was simulated as an alternative breeding program. In this program only a purebreeding strategy (ONPURE) was considered. The breeding strategies were evaluated under two breeding objective scenarios that addressed traditional markets where animals are sold on body size/weight basis and the Kenya Meat Commission where payment is based on carcass characteristics. Sensitivity analyses to changes in nucleus size and gene contribution were also performed. The annual monetary genetic gain and profit per cow for all investigated breeding programs varied within breeding objectives. Closed nucleus purebreeding program was the most attractive economically but less competitive in regard to genetic superiority compared with either CNCROSS or ONPURE. Returns and profits were generally higher for the carcass characteristic basis compared with the body size/weight basis for all evaluated breeding strategies. Expansion of the nucleus size was not attractive because of the associated reduction in genetic and economic benefits. However, gradual importation of pastoral-born sires into the nucleus farms at the current nucleus proportion of 14% was both genetically and economically beneficial. The CNCROSS plays a complimentary role of facilitating the exploitation of trade-offs that exist between the Sahiwal and the locally better adapted East African Zebu, it also represents an intermediate phase in the on-going upgrading program.

An adaptive and robust biological network based on the vacant-particle transportation model

A living system reveals local computing by referring to a whole system beyond the exploration–exploitation dilemma. The slime mold, Physarum polycephalum, uses protoplasmic flow to change its own outer shape, which yields the boundary condition and forms an adaptive and robust network. This observation suggests that the whole Physarum can be represented as a local protoplasmic flow system. Here, we show that a system composed of particles, which move and are modified based upon the particle transformation that contains the relationship between the parts and the whole, can emulate the network formed by Physarum. This system balances the exploration–exploitation trade-off and shows a scale-free sub-domain. By decreasing the number of particles, our model, VP-S, can emulate the Physarum adaptive network as it is attracted to a food stimulus. By increasing the number of particles, our model, VP-D, can emulate the pattern of a growing Physarum. The patterns produced by our model were compared with those of the Physarum pattern quantitatively, which showed that both patterns balance exploration with exploitation. This model should have a wide applicability to study biological collective phenomena in general.

Exploration and Exploitation Tradeoff in Fuzzy Reinforcement Learning

Difficulty of making a balance between exploration and exploitation in multiagent environment is a dilemma that does not have a clear answer and there are still different methods for investigation of this problem that all refer to it. In this paper, we provide a method based on fuzzy variables for making exploration and exploitation in multiagent environment. In this method, an effective agent (e in e-greedy method) is obtained which is updated using fuzzy variables in each step to manage tradeoff between exploration and exploitation.The proposed algorithm is investigated for determination an optimized path in the Grid World. In this method, agents effort to reach locations with a highest gain in a cooperative environment. Outcomes of the suggested fuzzy based algorithm compared with the results by conventional e-greedy method. In addition, quality improvement of interaction between exploration and exploitation is discussed.

The enforcement–exploitation trade-off in international cooperation between weak and powerful states

According to the imperative of enforcement, states must threaten defectors with sanctions or reciprocal suspension of international cooperation. I show that in international cooperation between the weak and the powerful, states should nevertheless limit the supply of collective enforcement power. Strong sanctions allow exploitative international agreements, so weak states refuse to engage in negotiations with powerful states in the first place. For powerful states, toothless international agreements are a credible commitment to limit power politics. The result holds even if sanctions can be used only to enforce international agreements and not for coercion. It implies that under power asymmetry, states must accept constraints on the use of power, as opposed to simply maximizing the supply of collective enforcement power. The theory offers a new perspective to international cooperation and a synthesis of the enforcement and managerial schools of international cooperation. It produces precise analytical boundary conditions and generates falsifiable empirical hypotheses.

Censored exploration and the dark pool problem

Dark pools are a recent type of stock exchange in which information about outstanding orders is deliberately hidden in order to minimize the market impact of large-volume trades. The success and proliferation of dark pools have created challenging and interesting problems in algorithmic trading---in particular, the problem of optimizing the allocation of a large trade over multiple competing dark pools. In this work, we formalize this optimization as a problem of multi-venue exploration from censored data, and provide a provably efficient and near-optimal algorithm for its solution. Our algorithm and its analysis have much in common with well-studied algorithms for managing the exploration--exploitation trade-off in reinforcement learning. We also provide an extensive experimental evaluation of our algorithm using dark pool execution data from a large brokerage.

Managing Know-How

We study how firms can use a knowledge management system to optimally leverage employee-generated know-how. In particular, we consider the following practical strategic questions for the manager of a knowledge-intensive firm: Should her firm develop a formal knowledge system? And if so, how should it be managed, particularly in terms of what information to record? We find that firms benefit more from a knowledge system when they are larger, face the same issues more frequently, have higher turnover, and face problems about which there is less general knowledge. In terms of what information to record, a key insight is that recording moderately successful practices can be counterproductive, because doing so may inefficiently reduce employees' incentives to experiment. This “strong-form competency trap” forces firms into an exploration–exploitation trade-off. Firms that value a knowledge system most should also be most selective in recording information. We further find that recording successes is more valuable than recording failures, which supports firms' focus on best practice. Beyond these main principles, we also show that it may be optimal to disseminate know-how on a plant level but not on a firm level, and that recording backup solutions is most valuable at medium levels of environmental change.