Greedy Action Research Articles

Two-Stage Optimization Model Based on Neo4j-Dueling Deep Q Network

To alleviate the power flow congestion in active distribution networks (ADNs), this paper proposes a two-stage load transfer optimization model based on Neo4j-Dueling DQN. First, the Neo4j graph model was established as the training environment for Dueling DQN. Meanwhile, the power supply paths from the congestion point to the power source point were obtained using the Cypher language built into Neo4j, forming a load transfer space that served as the action space. Secondly, based on various constraints in the load transfer process, a reward and penalty function was formulated to establish the Dueling DQN training model. Finally, according to the ε−greedy action selection strategy, actions were selected from the action space and interacted with the Neo4j environment, resulting in the optimal load transfer operation sequence. In this paper, Python was used as the programming language, TensorFlow open-source software library was used to form a deep reinforcement network, and Py2neo toolkit was used to complete the linkage between the python platform and Neo4j. We conducted experiments on a real 79-node system, using three power flow congestion scenarios for validation. Under the three power flow congestion scenarios, the time required to obtain the results was 2.87 s, 4.37 s and 3.45 s, respectively. For scenario 1 before and after load transfer, the line loss, voltage deviation and line load rate were reduced by about 56.0%, 76.0% and 55.7%, respectively. For scenario 2 before and after load transfer, the line loss, voltage deviation and line load rate were reduced by 41.7%, 72.9% and 56.7%, respectively. For scenario 3 before and after load transfer, the line loss, voltage deviation and line load rate were reduced by 13.6%, 47.1% and 37.7%, respectively. The experimental results show that the trained model can quickly and accurately derive the optimal load transfer operation sequence under different power flow congestion conditions, thereby validating the effectiveness of the proposed model.

Learning Multi-Agent Cooperation via Considering Actions of Teammates.

Recently value-based centralized training with decentralized execution (CTDE) multi-agent reinforcement learning (MARL) methods have achieved excellent performance in cooperative tasks. However, the most representative method among these methods, Q-network MIXing (QMIX), restricts the joint action Q values to be a monotonic mixing of each agent's utilities. Furthermore, current methods cannot generalize to unseen environments or different agent configurations, which is known as ad hoc team play situation. In this work, we propose a novel Q values decomposition that considers both the return of an agent acting on its own and cooperating with other observable agents to address the nonmonotonic problem. Based on the decomposition, we propose a greedy action searching method that can improve exploration and is not affected by changes in observable agents or changes in the order of agents' actions. In this way, our method can adapt to ad hoc team play situation. Furthermore, we utilize an auxiliary loss related to environmental cognition consistency and a modified prioritized experience replay (PER) buffer to assist training. Our extensive experimental results show that our method achieves significant performance improvements in both challenging monotonic and nonmonotonic domains, and can handle the ad hoc team play situation perfectly.

Energy-balanced routing in wireless sensor networks with reinforcement learning using greedy action chains

Energy-balanced routing in wireless sensor networks with reinforcement learning using greedy action chains

A Quantum States Preparation Method Based on Difference-Driven Reinforcement Learning

Due to the large state space of the two-qubit system, and the adoption of ladder reward function in the existing quantum state preparation methods, the convergence speed is slow and it is difficult to prepare the desired target quantum state with high fidelity under limited conditions. To solve the above problems, a difference-driven reinforcement learning (RL) algorithm for quantum state preparation of two-qubit system is proposed by improving the reward function and action selection strategy. Firstly, a model is constructed for the problem of preparing quantum states of a two-qubit system, with restrictions on the type of quantum gates and the time for quantum state evolution. In the preparation process, a weighted differential dynamic reward function is designed to assist the algorithm quickly obtain the maximum expected cumulative reward. Then, an adaptive [Formula: see text]-greedy action selection strategy is adopted to achieve a balance between exploration and utilization to a certain extent, thereby improving the fidelity of the final quantum state. The simulation results show that the proposed algorithm can prepare quantum state with high fidelity under limited conditions. Compared with other algorithms, it has different degrees of improvement in convergence speed and fidelity of the final quantum state.

Brain stimulation reveals distinct motives underlying reciprocal punishment and reward.

Reciprocal fairness, in the form of punishment and reward, is at the core of human societal order. Its underlying neural mechanisms are, however, not fully understood. We systemize suggestive evidence regarding the involvement of the right dorsolateral prefrontal cortex (rDLPFC) and medial prefrontal cortex (mPFC) in reciprocal fairness in three cognitive mechanisms (cognitive control, domain-general and self-reference). We test them and provide novel insights in a comprehensive behavioural experiment with non-invasive brain stimulation where participants can punish greedy actions and reward generous actions. Brain stimulation of either brain area decreases reward and punishment when reciprocation is costly but unexpectedly increases reward when it is non-costly. None of the hypothesized mechanisms fully accounts for the observed behaviour, and the asymmetric involvement of the investigated brain areas in punishment and reward suggests that different psychological mechanisms are underlying punishing selfishness and rewarding generosity. We propose that, for reciprocal punishment, the rDLPFC and the mPFC process self-relevant information, in terms of both personal cost and personal involvement; for reciprocal reward, these brain regions are involved in controlling selfish and pure reciprocity motives, while simultaneously promoting the enforcement of fairness norms. These insights bear importance for endeavours to build biologically plausible models of human behaviour.

Basic bounds on cluster error using distortion-rate

Clustering is a popular type of unsupervised learning technique that performs natural groupings on samples to generate probably approximately correct groups. These clusters or groups are expected to have high intra-similarity and high inter-distinctiveness. Practical clustering problems may combine varying levels of intra-similarity or quality and distinctiveness or diversity. We propose an information analytic approach to measure the quality and diversity given in terms of the cluster error. Unlike previous models that show information measures derived from probability of samples in cluster, the proposed framework employs distortion-rate approach by first formulating the probability of distortion for multiple-types of samples in the cluster. In this framework, cluster formation is shown as naturally greedy action which leads to showing the average minimum distortion of cluster. We also obtain probabilistic bounds on the cluster error and present case study on use of distortion-rate approach for clustering. For limiting case of binary typical cluster, the bound is shown to resemble Fano inequality. For the first time, analytic performance limits along with notion of bias and variance are formalized for clustering.

An Efficient Algorithm for Deep Stochastic Contextual Bandits

In stochastic contextual bandit (SCB) problems, an agent selects an action based on certain observed context to maximize the cumulative reward over iterations. Recently there have been a few studies using a deep neural network (DNN) to predict the expected reward for an action, and the DNN is trained by a stochastic gradient based method. However, convergence analysis has been greatly ignored to examine whether and where these methods converge. In this work, we formulate the SCB that uses a DNN reward function as a non-convex stochastic optimization problem, and design a stage-wised stochastic gradient descent algorithm to optimize the problem and determine the action policy. We prove that with high probability, the action sequence chosen by our algorithm converges to a greedy action policy respecting a local optimal reward function. Extensive experiments have been performed to demonstrate the effectiveness and efficiency of the proposed algorithm on multiple real-world datasets.

Entangled and correlated photon mixed strategy for social decision making

Collective decision making is important for maximizing total benefits while preserving equality among individuals in the competitive multi-armed bandit (CMAB) problem, wherein multiple players try to gain higher rewards from multiple slot machines. The CMAB problem represents an essential aspect of applications such as resource management in social infrastructure. In a previous study, we theoretically and experimentally demonstrated that entangled photons can physically resolve the difficulty of the CMAB problem. This decision-making strategy completely avoids decision conflicts while ensuring equality. However, decision conflicts can sometimes be beneficial if they yield greater rewards than non-conflicting decisions, indicating that greedy actions may provide positive effects depending on the given environment. In this study, we demonstrate a mixed strategy of entangled- and correlated-photon-based decision-making so that total rewards can be enhanced when compared to the entangled-photon-only decision strategy. We show that an optimal mixture of entangled- and correlated-photon-based strategies exists depending on the dynamics of the reward environment as well as the difficulty of the given problem. This study paves the way for utilizing both quantum and classical aspects of photons in a mixed manner for decision making and provides yet another example of the supremacy of mixed strategies known in game theory, especially in evolutionary game theory.

Transgenic Florae: Frame of Mind an A Dilemmatical Sitch

Throughout the history of mankind, we have developed far more advanced techniques to cope with the challenges we face on the path of progression. No doubts, most of the problems are due to our greedy actions that ultimately led to a series of unforeseen consequences. One of them is the point of view on the use of genetically modified plants. It is a fact that today, we may have been facing a very different scenario if these transgenic plants don’t have existed at all. Most of the world would’ve been starved to death by now due to the daily exponential increase in population. But the ethical and other concerns on the use of these plants are still a major hurdle in the path of development of these beneficial florae. This review summarizes the sum of possible problems and major situations related to transgenic plants according to modern research.

Imitation and Transfer Q-Learning-Based Parameter Identification for Composite Load Modeling

Fast and accurate load parameter identification has a large impact on power systems operation and stability analysis. This article proposes a novel Imitation and Transfer Q-learning (ITQ)-based method to identify parameters of composite constant impedance-current-power (ZIP) and induction motor (IM) load models. Firstly, an imitation learning process is introduced to improve the exploitation and exploration processes. Then, a transfer learning method is employed to overcome the challenge of time-consuming optimization when dealing with new identification tasks. An associative memory is designed to realize dimension reduction, knowledge learning and transfer between different identification tasks. Agents can exploit the optimal knowledge from source tasks to accelerate the search rate in new tasks and improve solution accuracy. A greedy action selection rule is adopted for agents to balance the global and local search. The performance of the proposed ITQ approach has been validated on a 68-bus test system. Simulation results in multi-test cases verify that the proposed method is robust and can estimate load parameters accurately. Comparisons with other methods show that the proposed method has superior convergence rate and stability.

Genetic Algorithm-Optimized Fuzzy Lyapunov Reinforcement Learning for Nonlinear Systems

This paper aims to introduce nonlinear optimization in the fuzzy reinforcement learning (RL) approach through genetic algorithm (GA)-based minimization. In conventional fuzzy RL, an agent attempts to find most optimal action at each stage by choosing an action having the lowest Q value or the greedy action. However, Q function is an unknown function and an attempt to find minima of such a function based on a limited set of values, in our view, is inaccurate and insufficient. A more rigorous approach would be to employ a nonlinear optimization procedure for finding minima of the Q function. We propose to employ genetic algorithm for finding optimal action value in each iteration of the algorithm rather than plain algebraic minimum. For guaranteed stability of the designed controller, we use Lyapunov theory-based fuzzy RL control with GA optimizer. We validate the performance of our controller on three benchmark nonlinear NL control problems: (1) inverted pendulum swing up, (2) cart pole balance, and (3) rotational/translational proof-mass actuator system. We carry out comparative evaluation of our controller against: (1) hybrid Lyapunov fuzzy RL control and (2) fuzzy Q learning control. Results show that our proposed GA-optimized fuzzy Lyapunov RL controller is able to achieve a high success rate with stable and superior tracking performance.

Greedy Action Selection and Pessimistic Q-Value Updating in Multi-Agent Reinforcement Learning with Sparse Interaction

Although multi-agent reinforcement learning (MARL) is a promising method for learning a collaborative action policy, enabling each agent to accomplish specified tasks, MARL has a problem of exponen...

Actual and Potential Harm is Central to Perceptions of Greed

Despite its centrality to public discourse and everyday ethical judgment and behavior, a robust conceptual understanding of greed has been lacking from the management literature (Wang & Murninghan, 2011). This paper focuses on the factors that shape ordinary perceptions of greedy actions and business practices. Supplementing existing accounts that conceptualize greed as a sin of acquisitiveness or insatiability (always wanting more), we find in five studies that real or potential harm to other parties is an additional important determinant of whether actions are seen as greedy. People are sensitive to both the degree and victims of harm, and use both acquisitiveness and harm to evaluate the amount of greed reflected in real-world business practices. Results are discussed in terms of both theoretical and applied implications.

Path planning of multi-agent systems in unknown environment with neural kernel smoothing and reinforcement learning

Path planning is a basic task of robot navigation, especially for autonomous robots. It is more complex and difficult for multi-agent systems. The popular reinforcement learning method cannot solve the path planning problem directly in unknown environment.In this paper, the classical multi-agent reinforcement learning algorithm is modified such that it does not need the unvisited state. The neural networks and kernel smoothing techniques are applied to approximate greedy actions by estimating the unknown environment. Experimental and simulation results show that the proposed algorithms can generate paths in unknown environment for multiple agents.

Using Learning Classifier Systems to Learn Stochastic Decision Policies

To solve reinforcement learning problems, many learning classifier systems (LCSs) are designed to learn state-action value functions through a compact set of maximally general and accurate rules. Most of these systems focus primarily on learning deterministic policies by using a greedy action selection strategy. However, in practice, it may be more flexible and desirable to learn stochastic policies, which can be considered as direct extensions of their deterministic counterparts. In this paper, we aim to achieve this goal by extending each rule with a new policy parameter. Meanwhile, a new method for adaptive learning of stochastic action selection strategies based on a policy gradient framework has also been introduced. Using this method, we have developed two new learning systems, one based on a regular gradient learning technology and the other based on a new natural gradient learning method. Both learning systems have been evaluated on three different types of reinforcement learning problems. The promising performance of the two systems clearly shows that LCSs provide a suitable platform for efficient and reliable learning of stochastic policies.

Approximate ideal multi-objective solution Q(λ) learning for optimal carbon-energy combined-flow in multi-energy power systems

This paper proposes a novel approximate ideal multi-objective solution Q(λ) learning for optimal carbon-energy combined-flow in multi-energy power systems. The carbon emissions, fuel cost, active power loss, voltage deviation and carbon emission loss are chosen as the optimization objectives, which are simultaneously optimized by five different Q-value matrices. The dynamic optimal weight of each objective is calculated online from the entire Q-value matrices such that the greedy action policy can be obtained. Case studies are carried out to evaluate the optimization performance for carbon-energy combined-flow in an IEEE 118-bus system and the regional power grid of southern China.

Avoiding greedy behavior in situations of uncertainty: The role of magical thinking

Previous studies have found evidence for the belief that actions which tempt fate increase the likelihood of negative outcomes. These included actions that presuppose a good outcome, that reflect hubris or that involve excessive risk taking. This paper explores a related form of magical thinking whereby individuals believe that asking for too much in situations of uncertainty may be punished by the universe and may decrease the probability of the desired outcome. It was found that many participants irrationally forgo the “greedy” option under uncertainty, even though it dominates other options and their behavior is not observed. It is suggested that some participants fear being magically punished for greediness and it is shown that the avoidance of greedy actions under uncertainty is related to the belief that one should not tempt fate. This phenomenon may have implications for various types of economic decisions such as charity donation, insurance purchase and bargaining.

Clique-based cooperative multiagent reinforcement learning using factor graphs

In this paper, we propose a clique-based sparse reinforcement learning (RL) algorithm for solving cooperative tasks. The aim is to accelerate the learning speed of the original sparse RL algorithm and to make it applicable for tasks decomposed in a more general manner. First, a transition function is estimated and used to update the Q-value function, which greatly reduces the learning time. Second, it is more reasonable to divide agents into cliques, each of which is only responsible for a specific subtask. In this way, the global Q-value function is decomposed into the sum of several simpler local Q-value functions. Such decomposition is expressed by a factor graph and exploited by the general maxplus algorithm to obtain the greedy joint action. Experimental results show that the proposed approach outperforms others with better performance.

Optimal tracking agent: a new framework of reinforcement learning for multiagent systems

SUMMARYThe curse of dimensionality is a ubiquitous problem for multiagent reinforcement learning, which means the learning and storing space grows exponentially with the number of agents and hinders the application of multiagent reinforcement learning. To relieve this problem, we propose a new framework named as optimal tracking agent (OTA). The OTA views the other agents as part of the environment and uses a reduced form to learn the optimal decision. Although merging other agents into the environment may reduce the dimension of action space, the environment characterized by such form is dynamic and does not satisfy the convergence of reinforcement learning (RL). Thus, we develop an estimator to track the dynamics of the environment. The estimator obtains the dynamic model, and then the model‐based RL can be used to react to the dynamic environment optimally. Because the Q‐function in OTA is also a dynamic process because of other agents’ dynamics, different from traditional RL, in which the learning is a stationary process and the usual action selection mechanisms just suit to such stationary process, we improve the greedy action selection mechanism to adapt to such dynamics. Thus, the OTA will have convergence. An experiment illustrates the validity and efficiency of the OTA.Copyright © 2012 John Wiley & Sons, Ltd.

Stochastic enforced hill-climbing

Enforced hill-climbing is an effective deterministic hill-climbing technique that deals with local optima using breadth-first search (a process called flooding). We propose and evaluate a stochastic generalization of enforced hill-climbing for online use in goal-oriented probabilistic planning problems. We assume a provided heuristic function estimating expected cost to the goal with flaws such as local optima and plateaus that thwart straightforward greedy action choice. While breadth-first search is effective in exploring basins around local optima in deterministic problems, for stochastic problems we dynamically build and solve a heuristic-based Markov decision process (MDP) model of the basin in order to find a good escape policy exiting the local optimum. We note that building this model involves integrating the heuristic into the MDP problem because the local goal is to improve the heuristic. We evaluate our proposal in twenty-four recent probabilistic planning-competition benchmark domains and twelve probabilistically interesting problems from recent literature. For evaluation, we show that stochastic enforced hill-climbing (SEH) produces better policies than greedy heuristic following for value/cost functions derived in two very different ways: one type derived by using deterministic heuristics on a deterministic relaxation and a second type derived by automatic learning of Bellman-error features from domain-specific experience. Using the first type of heuristic, SEH is shown to generally outperform all planners from the first three international probabilistic planning competitions.