Experience-weighted Attraction Learning Research Articles

Sharing hydrogen storage capacity planning for multi-microgrid investors with limited rationality: A differential evolution game approach

Bottom on the ripple of the multiplication of sharing economy, hydrogen energy storage (HES) shared calls for novel solutions to ameliorate the cleanness and economy of micro-grids under the dual carbon target in China. The traditional investment method may become impracticable due to conflicting interests, insufficient utilization, and poor interoperability. The practical and prospective design of HES relies on innovation at the joint investment mechanism under incomplete synchronized and the interested equilibrium under limited rationality. This paper studies the capacity planning method for HES by multiple micro-grids from the sharing perspective. The evolution game model is utilized for conflict of interest, which exactly accounts for limited rationality. The dynamic competitive as well as time-varying demand for HES among multiple micro-grids are established by a small-word complex network theory. The dual dynamic evolutionary game of investment decision in multi-micro-grid and capacity planning for sharing HES is realized by a proposed method of differential evolutionary game algorithm based on experience-weighted attraction learning. The numerical simulation is carried out to verify the effectiveness of the proposed method and interpret the development path for sharing HES. The result shows that, the sizing method with limited rationality reduces the cost by 4.75% compared with the traditional game under complete rationality and 9.15% compared with the multi-objective optimization. The willingness of the park to participate in joint investment in shared HES is influenced by the long-term regulatory demand for energy storage and the decisions of other parks in the neighbors.

Reinforcement learning with foregone payoff information in normal form games

This paper studies the reinforcement learning of Erev and Roth with foregone payoff information in normal form games: players observe not only the realised payoffs but also foregone payoffs, the ones which they could have obtained if they had chosen the other actions. We provide conditions under which the reinforcement learning process almost surely converges to a regular quantal response equilibrium (Goeree et al. 2005). We also show that the reinforcement learning model and an adaptive learning model which nests experience-weighted attraction learning, payoff assessment learning and stochastic fictitious play learning models share the same asymptotic behaviour under the linear choice rule of the reinforcement learning model. In addition, we provide conditions under which the reinforcement learning process under the logit choice rule almost surely converges to a Nash equilibrium.

Regional Cooperation in Marine Plastic Waste Cleanup in the South China Sea Region

This paper uses countries in the South China Sea Region (SCSR) as examples to study the level of regional cooperation in the marine plastic waste cleanup initiative. We designed a cooperation model to investigate the “cleanup system” from the Ocean Cleanup initiative to reduce marine plastic pollution. The non-cooperative game theory was applied to regional cooperation. The simulation results indicate that the plastic waste cleanup cooperation in the SCSR is related to the plastic trade network structure, the influence parameters of the Experience-Weighted Attraction learning model, and the economic effects. The results suggest that regional cooperation in the cleanup system in the SCSR is feasible, and it could create a significantly larger investment in the cleanup project than the current project attracts. Therefore, countries in the SCSR should adjust their laws and policies to make a good cooperative environment and to maximize the contribution to the marine plastic waste cleanup.

Estimating Case-Based Learning

We propose a framework in order to econometrically estimate case-based learning and apply it to empirical data from twelve 2 × 2 mixed strategy equilibria experiments. Case-based learning allows agents to explicitly incorporate information available to the experimental subjects in a simple, compact, and arguably natural way. We compare the estimates of case-based learning to other learning models (reinforcement learning and self-tuned experience weighted attraction learning) while using in-sample and out-of-sample measures. We find evidence that case-based learning explains these data better than the other models based on both in-sample and out-of-sample measures. Additionally, the case-based specification estimates how factors determine the salience of past experiences for the agents. We find that, in constant sum games, opposing players’ behavior is more important than recency and, in non-constant sum games, the reverse is true.

An agent-based learning-embedded model (ABM-learning) for urban land use planning: A case study of residential land growth simulation in Shenzhen, China

A forward-looking urban land use plan is crucial to a city’s sustainability, which requires a deep understanding of human-environment interactions between different domains, and modelling them soundly. One of the key challenges of modelling these interactions is to understand and model how human individuals make and develop their location decisions by learning that then shape urban land-use patterns. To investigate this issue, we have constructed an extended experience-weighted attraction learning model to represent the human agents’ learning when they make location decisions. Consequently, we propose and have developed an agent-based learning-embedded model (ABM-learning) for residential land growth simulation that incorporates a learning model, a decision-making model, a land use conversion model and the constraint of urban land use master plan. The proposed model was used for a simulation of the residential land growth in Shenzhen city, China. By validating the model against empirical data, the results showed that the site-specific accuracy of the model has been improved when embedding learning model. The analysis on the simulation accuracies has proved the argument that modelling individual-level learning matters in the agent’s decision model and the agent-based models. We also applied the model to predict residential land growth in Shenzhen from 2015 to 2035, and the result can be a reference for land-use allocation in detailed planning of Shenzhen. The ABM-learning is applicable to studying the past urban growth trajectory, aiding in the formulation of detailed residential land and public service facility planning and assessing the land use planning effectiveness.

Reinforcement Learning with Foregone Payoff Information in Normal Form Games

This paper studies the reinforcement learning of Erev and Roth with foregone payoff information in normal form games: players observe not only the realised payoffs but also foregone payoffs, the ones which they could have obtained if they had chosen the other actions. We provide conditions under which the reinforcement learning process almost surely converges to a regular quantal response equilibrium (Goeree et al. 2005). We also show that the reinforcement learning model and an adaptive learning model which nests experience-weighted attraction learning, payoff assessment learning and stochastic fictitious play learning models share the same asymptotic behaviour under the linear choice rule of the reinforcement learning model. In addition, we provide conditions under which the reinforcement learning process under the logit choice rule almost surely converges to a Nash equilibrium.

The Strategy Evolution in Double Auction Based on the Experience-Weighted Attraction Learning Model

The double auction is a widely applicable trading mechanism used to converge to competitive equilibrium in different markets from which multiple equilibriums and incomplete information may arise. Therefore, different learning models have been applied to facilitate bidding strategies for buyers and sellers in the market. However, due to the existence of problems in double auction markets such as individual bounded rationality and information incompleteness, it is still necessary to explore a more general learning model to depict the learning mechanism in double auction markets and predict the evolution processes of bidding strategies for both sides. Therefore, this paper aims at introducing the use of the experience-weighted attraction (EWA) model for double auction because it combines reinforcement learning with belief learning that then converts EWA in a suitable and interesting learning model for describing and improving individuals’ learning behavior. It can become an effective learning model for bidding strategies in the double auction. In addition to the use of the EWA for strategy evolution in the double auction, the impact of its different bidding strategy performance parameters will be also analyzed and compared with other learning models.

Learning Dynamics and Norm Psychology Supports Human Cooperation in a Large-Scale Prisoner’s Dilemma on Networks

In this work, we explore the role of learning dynamics and social norms in human cooperation on networks. We study the model recently introduced in [Physical Review E, 97, 042321 (2018)] that integrates the well-studied Experience Weighted Attraction learning model with some features characterizing human norm psychology, namely the set of cognitive abilities humans have evolved to deal with social norms. We provide further evidence that this extended model—that we refer to as Experience Weighted Attraction with Norm Psychology—closely reproduces cooperative patterns of behavior observed in large-scale experiments with humans. In particular, we provide additional support for the finding that, when deciding to cooperate, humans balance between the choice that returns higher payoffs with the choice in agreement with social norms. In our experiment, agents play a prisoner’s dilemma game on various network structures: (i) a static lattice where agents have a fixed position; (ii) a regular random network where agents have a fixed position; and (iii) a dynamic lattice where agents are randomly re-positioned at each game iteration. Our results show that the network structure does not affect the dynamics of cooperation, which corroborates results of prior laboratory experiments. However, the network structure does seem to affect how individuals balance between their self-interested and normative choices.

Convergence results on stochastic adaptive learning

We investigate an adaptive learning model which nests several existing learning models such as payoff assessment learning, valuation learning, stochastic fictitious play learning, experience-weighted attraction learning and delta learning with foregone payoff information in normal form games. In particular, we consider adaptive players each of whom assigns payoff assessments to his own actions, chooses the action which has the highest assessment with some perturbations and updates the assessments using observed payoffs, which may include payoffs from unchosen actions. Then, we provide conditions under which the learning process converges to a quantal response equilibrium in normal form games.

Dynamical selection of Nash equilibria using reinforcement learning: Emergence of heterogeneous mixed equilibria.

We study the distribution of strategies in a large game that models how agents choose among different double auction markets. We classify the possible mean field Nash equilibria, which include potentially segregated states where an agent population can split into subpopulations adopting different strategies. As the game is aggregative, the actual equilibrium strategy distributions remain undetermined, however. We therefore compare with the results of a reinforcement learning dynamics inspired by Experience-Weighted Attraction (EWA) learning, which at long times leads to Nash equilibria in the appropriate limits of large intensity of choice, low noise (long agent memory) and perfect imputation of missing scores (fictitious play). The learning dynamics breaks the indeterminacy of the Nash equilibria. Non-trivially, depending on how the relevant limits are taken, more than one type of equilibrium can be selected. These include the standard homogeneous mixed and heterogeneous pure states, but also heterogeneous mixed states where different agents play different strategies that are not all pure. The analysis of the reinforcement learning involves Fokker-Planck modeling combined with large deviation methods. The theoretical results are confirmed by multi-agent simulations.

Carbon Reduction Strategies Based on an NW Small-World Network with a Progressive Carbon Tax

There is an increasingly urgent need to reduce carbon emissions. Devising effective carbon tax policies has become an important research topic. It is necessary to explore carbon reduction strategies based on the design of carbon tax elements. In this study, we explore the effect of a progressive carbon tax policy on carbon emission reductions using the logical deduction method. We apply experience-weighted attraction learning theory to construct an evolutionary game model for enterprises with different levels of energy consumption in an NW small-world network, and study their strategy choices when faced with a progressive carbon tax policy. The findings suggest that enterprises that adopt other energy consumption strategies gradually transform to a low energy consumption strategy, and that this trend eventually spreads to the entire system. With other conditions unchanged, the rate at which enterprises change to a low energy consumption strategy becomes faster as the discount coefficient, the network externality, and the expected adjustment factor increase. Conversely, the rate of change slows as the cost of converting to a low energy consumption strategy increases.

Heterogeneity in generalized reinforcement learning and its relation to cognitive ability

In this paper, we study the connections between working memory capacity (WMC) and learning in the context of economic guessing games. We apply a generalized version of reinforcement learning, popularly known as the experience-weighted attraction (EWA) learning model, which has a connection to specific cognitive constructs, such as memory decay, the depreciation of past experience, counterfactual thinking, and choice intensity. Through the estimates of the model, we examine behavioral differences among individuals due to different levels of WMC. In accordance with ‘Miller’s magic number’, which is the constraint of working memory capacity, we consider two different sizes (granularities) of strategy space: one is larger (finer) and one is smaller (coarser). We find that constraining the EWA models by using levels (granules) within the limits of working memory allows for a better characterization of the data based on individual differences in WMC. Using this level-reinforcement version of EWA learning, also referred to as the EWA rule learning model, we find that working memory capacity can significantly affect learning behavior. Our likelihood ratio test rejects the null that subjects with high WMC and subjects with low WMC follow the same EWA learning model. In addition, the parameter corresponding to ‘counterfactual thinking ability’ is found to be reduced when working memory capacity is low.

Interbank network and regulation policies: an analysis through agent-based simulations with adaptive learning

We develop an agent-based model to study the impacts of a broad range of regulation policies over the banking system. It builds on an iterated version of the \citet{DiamondDybvig1983} framework and resorts to the experience-weighted attraction learning scheme of \citet{CamererHo1999} to model agents' adaptive learning. Thereby, we can capture not only the direct impacts of regulation policies, but also the ones that take part through shifting agents' adaptive strategies. Our results show that the introduction of an interbank clearinghouse is a good instrument to face the risk of contagion; the regulatory guidelines of the Basel Accord are effective in reducing the probability of bank failure; and the adoption of a deposit insurance can be adequate to avoid bank runs. However, we also show that these policies have drawbacks, and can either reduce bank activity or stimulate moral hazard.

Learning in Repeated Public Goods Games - A Meta Analysis

I examine the generalizability of a broad range of prominent learning models in explaining contribution patterns in repeated linear public goods games. Experimental data from twelve previously published papers are considered in testing several learning models in terms of how accurately they describe individuals’ round-by-round choices. The experimental data are split into 18 datasets. Each of these datasets is different from the remaining in at least one of the following aspects: the marginal per capita return, group size, matching protocol, number of rounds, and endowment that determines the number of stage game strategies. Both ex-post descriptive fit of learning models and their ex-ante predictive accuracy are examined. The following learning models are included in the study: reinforcement learning, normalized reinforcement learning, reinforcement average model with loss aversion strategy (REL), stochastic fictitious play, normalized stochastic fictitious play, experience weighted attraction learning (EWA), self-tuning EWA, and Impulse matching learning. REL outperforms all other learning models in both within dataset descriptive fit and out-of-sample dataset predictive accuracy. While all the learning models out-perform the random choice benchmark, only REL performs at least as well as the model that reflects dataset level overall empirical frequencies. The results suggest that learning in repeated linear public goods games is more in line with reinforcement learning than that of belief learning or regret-based learning. Finally, REL also outperforms individual evolutionary learning (IEL) in predicting the full distribution of contributions. Average reinforcement learning that is sensitive to the observed payoff variability and insensitive to the payoff magnitude underlie the success of REL in explaining contributions in repeated public goods games over a broad spectrum of game parameters.

EWA Selection Strategy with Channel Handoff Scheme in Cognitive Radio

Experience-Weighted Attraction (EWA) learning allows Cognitive Radio (CR) to learn radio environment communication channel characteristics online. An improved EWA selection strategy based on channel switch mechanism is proposed in this paper to meet CR application's practical requirement. By accumulating history channel experience, it can predict, select and change the current optimal communication channel, dynamic ensure the quality of communication links and finally reduce system communication outage probability. Sensitivity and stability verification is conducted by focusing on the comparison with Q learning algorithm after establishment of a wireless simulation space with time slot sequences that extremely imitates real CR wireless environments.

Fast Channel Selection Strategy in Cognitive Wireless Sensor Networks

In order to meet the practical requirement for Cognitive Wireless Sensor Networks applications, this paper proposes innovative fast channel selection algorithm to solve the shortcomings of original Experience-Weighted Attraction algorithm's complexity, higher energy consuming, and the nodes’ hardware restrictions of real-time data processing capabilities. Research is conducted by comparing channel selection differences and timeliness with traditional Experience-Weighted Attraction learning. Though not as stable as traditional Experience-Weighted Attraction learning, fast channel selection algorithm has effectively reduced the complexity of the original algorithm and has superior performance than Q learning.

Experiments on individual strategy updating in iterated snowdrift game under random rematching.

How do people actually play the iterated snowdrift games, particularly under random rematching protocol is far from well explored. Two sets of laboratory experiments on snowdrift game were conducted to investigate human strategy updating rules. Four groups of subjects were modeled by experience-weighted attraction learning theory at individual-level. Three out of the four groups (75%) passed model validation. Substantial heterogeneity is observed among the players who update their strategies in four typical types, whereas rare people behave like belief-based learners even under fixed pairing. Most subjects (63.9%) adopt the reinforcement learning (or alike) rules; but, interestingly, the performance of averaged reinforcement learners suffered. It is observed that two factors seem to benefit players in competition, i.e., the sensitivity to their recent experiences and the overall consideration of forgone payoffs. Moreover, subjects with changing opponents tend to learn faster based on their own recent experience, and display more diverse strategy updating rules than they do with fixed opponent. These findings suggest that most of subjects do apply reinforcement learning alike updating rules even under random rematching, although these rules may not improve their performance. The findings help evolutionary biology researchers to understand sophisticated human behavioral strategies in social dilemmas.

Cognitive Radio Channel Selection Strategy Based on Experience-Weighted Attraction Learning

In this paper, an innovative proposed channel selection algorithm based on Experience-Weighted Attraction (EWA) learning allows Cognitive Radio (CR) to learn radio environment communication channel characteristics online. By accumulating the history channel experience, it can predict, select and change the current optimal communication channel, dynamic ensure the quality of communication links and finally reduce system communication outage probability. Validation and reliability have been strictly verified by Matlab simulations. DOI : http://dx.doi.org/10.11591/telkomnika.v12i1.3900

An Adaptive Learning Model with Foregone Payoff Information

AbstractIn this paper, we provide theoretical predictions on the long-run behavior of an adaptive decision maker with foregone payoff information. In the model, the decision maker assigns a subjective payoff assessment to each action based on his past experience and chooses the action that has the highest assessment. After receiving a payoff, the decision maker updates his assessments of actions in an adaptive manner, using not only the objective payoff information but also the foregone payoff information, which may be distorted. The distortion may arise from “the grass is always greener on the other side” effect, pessimism/optimism or envy/gloating; it depends on how the decision maker views the source of the information. We first provide conditions in which the assessment of each action converges, in that the limit assessment is expressed as an average of the expected objective payoff and the expected distorted payoff of the action. Then, we show that the decision maker chooses the optimal action most frequently in the long run if the expected distorted payoff of the action is greater than the ones of the other actions. We also provide conditions, under which this model coincides with the experience-weighted attraction learning, stochastic fictitious play and quantal response equilibrium models, and thus this model provides theoretical predictions for the models in decision problems.

Close Cousins? Bayesian versus Experience-Weighted Attraction Learning Under Tail Risk

Close Cousins? Bayesian versus Experience-Weighted Attraction Learning Under Tail Risk