Unknown Rewards Research Articles

Too much or too little information: how unknown uncertainty fuels time inconsistency

Under uncertainty, there is considerable heterogeneity in expectations of results, and the outcome of each choice is a reflection of those expectations. This study aims to understand the role of subjective probabilistic inference in updating information for decision-making procedures under uncertainty. We show that adding uncertainty of trade-offs in decision-making criteria induces more inconsistent present preferences. We find that subjective probabilistic inference results in different levels of information acquisition, which plays a central role in many everyday cases of forecasting. The result of forecasting exerts substantive constraints on cognitive processes and shapes a type of restriction or stimulus in decision-making procedures. As uncertainty increases, generated fear of losses turns into an obstacle to the information acquisition process, and especially participants with low probabilistic inference tend to overestimate or underestimate future unknown rewards. In addition, our experiment shows that risk preference does not play a key role in decision-making procedures under unknown uncertainty. This finding is an experimental manifestation of Knight’s argument (Risk, uncertainty, and profit, Houghton Mifflin, Boston, 1921), which explains unknown uncertainty, and shows the relationship between cognitive ability and time inconsistency.

Combination of Auction Theory and Multi-Armed Bandits: Model, Algorithm, and Application

The multi-armed bandit (MAB) models have always received lots of attention from multiple research communities due to their broad application domains. The optimal selection problem with unknown rewards in advance, such as ad recommendation in social networks, spectrum access in the cognitive radio field, etc., can be efficiently solved by using MAB models. In an MAB model, given <inline-formula><tex-math notation="LaTeX">$N$</tex-math></inline-formula> arms whose rewards are unknown in advance, the player selects exactly one arm in each round, and his goal is to maximize the cumulative rewards over a fixed horizon. Further, a more general model called combinatorial MAB (i.e., CMAB), where <inline-formula><tex-math notation="LaTeX">$K$</tex-math></inline-formula> arms can be played simultaneously in each round, is put forward. However, the existing CMAB models neglect the strategic behaviors of the <inline-formula><tex-math notation="LaTeX">$N$</tex-math></inline-formula> arms, which indicates that one arm might report false information to increase its own profits. In fact, in many applications such as user selection in crowdsensing, the arms are not the feelingless machines but the rational individuals. To this end, we combine the upper confidence bound (UCB) with auction theory to develop a new algorithm called auction-based UCB (AUCB). We divide the auction-based CMAB problem into two sub-problems: winning arm selection and payment computation problems. For AUCB, we derive an upper bound on regret and prove the truthfulness in one round, individual rationality, and computational efficiency. In addition, we consider an extended situation that some arms may be unavailable in some rounds and the arms will bid inconsistently in different rounds. We devise another algorithm called eAUCB to solve this problem. Extensive simulations are conducted to show the significant performance of the proposed algorithms.

Technology-Mediated Dangerous Behaviors as Foraging for Social-Hedonic Rewards: The Role of Implied Inequality

Technology-mediated dangerous behaviors (TMDBs), such as posting selfies while driving or posting private information, are prevalent and potentially require interventions. Knowledge about the drivers of these, and specifically the role of information in stimulating such behaviors, is limited. To address this gap, this paper turns to foraging and risk-sensitivity theories. These theories suggest that animals engage in more dangerous behaviors when their perceived need for calories is high. Similarly, humans increase financial risk-taking when they perceive dissatisfaction with what they have. Importantly, inequality information can increase such perceptions and change people’s risk-taking propensity. Adapting these ideas, the paper postulates that TMDBs resemble food-seeking in animals in that they are goal oriented, can be dangerous, and yield unknown (probabilistic) rewards. Therefore, TMDBs are explained using foraging and risk-sensitivity theory angles. Focusing on social media users (Studies 1-4; four experiments; total n = 2,504), I argue that (1) it is reasonable to view users as foraging the “fields of social media” for social-hedonic rewards, (2) it is possible to alter their risk appetite and TMDBs through inequality information and upward comparison mechanisms, (3) this process can be mediated not only through cognitions, but also emotions, and (4) perceived scarcity of rewards and social comparison orientation affect this process. With Study 5, the paper extends the core aspects of this theoretical perspective to the U.S. state level and argues that objective financial inequality can explain differences between states in terms of TMDBs such as texting while driving and relative interest in TMDBs such as prank videos. The findings largely support these assertions. They illuminate the role of information, notably inequality, in driving TMDBs, extend prior research focused on basic needs (e.g., physiological needs in the case of food intake decisions) to an evaluation of higher-order human needs (e.g., needs for belongingness, esteem, and self-actualization) catered to by nonphysiological, social-hedonic rewards, and point to important mechanisms that translate inequality into TMDBs.

Dynamic Task Migration Combining Energy Efficiency and Load Balancing Optimization in Three-Tier UAV-Enabled Mobile Edge Computing System

As transportation becomes more convenient and efficient, users move faster and faster. When a user leaves the service range of the original edge server, the original edge server needs to migrate the tasks offloaded by the user to other edge servers. An effective task migration strategy needs to fully consider the location of users, the load status of edge servers, and energy consumption, which make designing an effective task migration strategy a challenge. In this paper, we innovatively proposed a mobile edge computing (MEC) system architecture consisting of multiple smart mobile devices (SMDs), multiple unmanned aerial vehicle (UAV), and a base station (BS). Moreover, we establish the model of the Markov decision process with unknown rewards (MDPUR) based on the traditional Markov decision process (MDP), which comprehensively considers the three aspects of the migration distance, the residual energy status of the UAVs, and the load status of the UAVs. Based on the MDPUR model, we propose a advantage-based value iteration (ABVI) algorithm to obtain the effective task migration strategy, which can help the UAV group to achieve load balancing and reduce the total energy consumption of the UAV group under the premise of ensuring user service quality. Finally, the results of simulation experiments show that the ABVI algorithm is effective. In particular, the ABVI algorithm has better performance than the traditional value iterative algorithm. And in a dynamic environment, the ABVI algorithm is also very robust.

A Primer on Foraging and the Explore/Exploit Trade-Off for Psychiatry Research.

Foraging is a fundamental behavior, and many types of animals appear to have solved foraging problems using a shared set of mechanisms. Perhaps the most common foraging problem is the choice between exploiting a familiar option for a known reward and exploring unfamiliar options for unknown rewards-the so-called explore/exploit trade-off. This trade-off has been studied extensively in behavioral ecology and computational neuroscience, but is relatively new to the field of psychiatry. Explore/exploit paradigms can offer psychiatry research a new approach to studying motivation, outcome valuation, and effort-related processes, which are disrupted in many mental and emotional disorders. In addition, the explore/exploit trade-off encompasses elements of risk-taking and impulsivity-common behaviors in psychiatric disorders-and provides a novel framework for understanding these behaviors within an ecological context. Here we explain relevant concepts and some common paradigms used to measure explore/exploit decisions in the laboratory, review clinically relevant research on the neurobiology and neuroanatomy of explore/exploit decision making, and discuss how computational psychiatry can benefit from foraging theory.

Neurons in the primate dorsal striatum signal the uncertainty of object-reward associations.

To learn, obtain reward and survive, humans and other animals must monitor, approach and act on objects that are associated with variable or unknown rewards. However, the neuronal mechanisms that mediate behaviours aimed at uncertain objects are poorly understood. Here we demonstrate that a set of neurons in an internal-capsule bordering regions of the primate dorsal striatum, within the putamen and caudate nucleus, signal the uncertainty of object–reward associations. Their uncertainty responses depend on the presence of objects associated with reward uncertainty and evolve rapidly as monkeys learn novel object–reward associations. Therefore, beyond its established role in mediating actions aimed at known or certain rewards, the dorsal striatum also participates in behaviours aimed at reward-uncertain objects.

Implementing the “Wisdom of the Crowd”

We study a novel mechanism design model in which agents each arrive sequentially and choose one action from a set of actions with unknown rewards. The information revealed by the principal affects the incentives of the agents to explore and generate new information. We characterize the optimal disclosure policy of a planner whose goal is to maximize social welfare. One interpretation of our result is the implementation of what is known as the “wisdom of the crowd.” This topic has become increasingly relevant with the rapid spread of the Internet over the past decade.

Sequential Sampling with Economics of Selection Procedures

Sequential sampling problems arise in stochastic simulation and many other applications. Sampling is used to infer the unknown performance of several alternatives before one alternative is selected as best. This paper presents new economically motivated fully sequential sampling procedures to solve such problems, called economics of selection procedures. The optimal procedure is derived for comparing a known standard with one alternative whose unknown reward is inferred with sampling. That result motivates heuristics when multiple alternatives have unknown rewards. The resulting procedures are more effective in numerical experiments than any previously proposed procedure of which we are aware and are easily implemented. The key driver of the improvement is the use of dynamic programming to model sequential sampling as an option to learn before selecting an alternative. It accounts for the expected benefit of adaptive stopping policies for sampling, rather than of one-stage policies, as is common in the literature. This paper was accepted by Assaf Zeevi, stochastic models and simulation.

Optimal selection with alternative information

AbstractThis article examines the problem of optimally selecting from several unknown rewards when there are given alternative, costly sources of information. The optimal rule, indicating the information to be purchased and the reward to be selected, is specified as a function of the decision maker's prior probabilities regarding the value of each alternative. The rule is surprisingly complex, balancing prior beliefs, the “informativeness” of the relevant information system, and the cost of acquiring information.