Improved Reinforcement Learning Algorithm Research Articles

A Novel Automatic Generation Control Method Based on the Large-Scale Electric Vehicles and Wind Power Integration Into the Grid.

In order to solve the problem of frequency instability of power system due to strong random disturbance caused by large-scale electric vehicles and wind power grid connection, an improved reinforcement learning algorithm, namely, optimistic initialized double Q, is proposed in this article from the perspective of automatic generation control. The proposed algorithm uses the optimistic initialization principle to expand the agent action exploration space, so as to prevent Q-learning from falling into local optimum by greedy strategy; meanwhile, it integrates double Q-learning to solve the problem of overestimation of action value in traditional reinforcement learning based on Q-learning. In the algorithm, the hyperparameter ατ is introduced to improve the learning efficiency, and the reward bτ based on exploration times is introduced to increase the Q value estimation to drive the exploration of the algorithm, so as to obtain the optimal solution. By simulating the two-area load frequency control model integrated with large-scale electric vehicles and the four-area interconnected power grid model integrated with large-scale wind power generation, it is verified that the proposed algorithm can obtain the global optimal solution, thus effectively solvinng the frequency instability caused by strong random disturbance in the grid-connected mode of large-scale wind power generation, and compared with many reinforcement learning algorithms, the proposed algorithm has better control performance.

A multi-agent reinforcement learning algorithm with the action preference selection strategy for massive target cooperative search mission planning

Target search is widely applied in military reconnaissance, geological exploration and personnel search and rescue. Most target search algorithms perform well in single target search but are inefficient or even ineffective in multi-target search. To solve the multi-target search problem in an uncertain environment, this paper constructs a multi-agent massive target cooperative search mission planning model and proposes an improved reinforcement learning algorithm using the action preference selection strategy. Based on the Reinforce algorithm, this algorithm solves the problem of invalid searches within the stochastic strategy by changing the preferred action selection method. The proposed method improves the efficiency of multiple agents in the search for targets without collision using a cooperative mechanism and reward rules based on the odor effect. Simulation experiments are conducted in three aspects to verify the effectiveness and robustness of the improved algorithm and compare it with other reinforcement learning algorithms in the field of multi-agent learning. The results demonstrate that the improved algorithm has obvious advantages in terms of mission success rate, target search rate and average search time, and the movement trajectory of multiple agents is more concise.

Orbit correction based on improved reinforcement learning algorithm

Recently, reinforcement learning (RL) algorithms have been applied to a wide range of control problems in accelerator commissioning. In order to achieve efficient and fast control, these algorithms need to be highly efficient, so as to minimize the online training time. In this paper, we incorporated the beam position monitor trend into the observation space of the twin delayed deep deterministic policy gradient (TD3) algorithm and trained two different structure agents, one based on physical prior knowledge and the other using the original TD3 network architecture. Both of the agents exhibit strong robustness in the simulated environment. The effectiveness of the agent based on physical prior knowledge has been validated in a real accelerator. Results show that the agent can overcome the difference between simulated and real accelerator environments. Once the training is completed in the simulated environment, the agent can be directly applied to the real accelerator without any online training process. The RL agent is deployed to the medium energy beam transport section of China Accelerator Facility for Superheavy Elements. Fast and automatic orbit correction is being tested with up to ten degrees of freedom. The experimental results show that the agents can correct the orbit to within 1 mm. Moreover, due to the strong robustness of the agent, when a trained agent is applied to different lattices of different particles, the orbit correction can still be completed. Since there are no online data collection and training processes, all online corrections are done within 30 s. This paper shows that, as long as the robustness of the RL algorithm is sufficient, the offline learning agents can be directly applied to online correction, which will greatly improve the efficiency of orbit correction. Such an approach to RL may find promising applications in other areas of accelerator commissioning.

Real-Time Pricing Method for Spot Cloud Services with Non-Stationary Excess Capacity

Cloud operators face massive unused excess computing capacity with a stochastic non-stationary nature due to time-varying resource utilization with peaks and troughs. Low-priority spot (pre-emptive) cloud services with real-time pricing have been launched by many cloud operators, which allow them to maximize excess capacity revenue while keeping the right to reclaim capacities when resource scarcity occurs. However, real-time spot pricing with the non-stationarity of excess capacity has two challenges: (1) it faces incomplete peak–trough and pattern shifts in excess capacity, and (2) it suffers time and space inefficiency in optimal spot pricing policy, which needs to search over the large space of history-dependent policies in a non-stationary state. Our objective was to develop a real-time pricing method with a spot pricing scheme to maximize expected cumulative revenue under a non-stationary state. We first formulated the real-time spot pricing problem as a non-stationary Markov decision process. We then developed an improved reinforcement learning algorithm to obtain the optimal solution for real-time pricing problems. Our simulation experiments demonstrate that the profitability of the proposed reinforcement learning algorithm outperforms that of existing solutions. Our study provides both efficient optimization algorithms and valuable insights into cloud operators’ excess capacity management practices.

Research on multi-factory combination optimization based on DOSTAR

With the development of industrial big data, it has become an important research direction to use combinatorial optimization to coordinate multi-objective problems in complex manufacturing scenarios with multiple factories. At present, most of the multi-objective problems are decomposed into single-objective solutions. However, it is difficult to resolve the contradiction between multiple goals. There are many participants in multi-objective problems and complex data types, so there is no suitable research method at present. Based on big data, this paper integrates various aspects of supply chain management of multiple factories, and proposes a DOSTAR combined model. On the one hand, it conducts knowledge discovery based on the fusion of human-cyber-physical ternary data, on the other hand, it conducts multi-objective optimization through knowledge structure. Among them, the most important thing is to establish the six-tuple as the basic model. Then the space weight, time weight and decision weight are obtained through the weight sub-model. Finally, the improved reinforcement learning algorithm is used to extract relevant new knowledge and complete multi-objective coordination. This article takes the supply chain management of Haier water heaters as an example, using the above-mentioned combined model, and the experimental results show that the purpose of improving performance has been achieved. • This article builds a combined optimization model based on knowledge discovery and reinforcement learning. • The combined model mainly includes: six tuples, weight sub-models (AHP and spatiotemporal data), and improved reinforcement learning algorithms. • This research solves the multi-objective optimization problem in the complex manufacturing environment of multi-factories.

Intelligent Energy Management Strategy Based on an Improved Reinforcement Learning Algorithm With Exploration Factor for a Plug-in PHEV

An intelligent energy management strategy (EMS) based on an improved Reinforcement Learning (RL) algorithm is developed to enhance the adaptability of the EMS and to further improve the fuel efficiency of a Plug-in Parallel Hybrid Electric Vehicle (PHEV). Both the numerical model and the energy management strategy of a plug-in PHEV are described. The improved RL with Q-learning algorithm is implemented to acquire the optimal control strategies for improving fuel economy. The Markov Chain is employed to calculate the Transition Probability Matrix of the required power. A Kullback-Leibler (KL) divergence rate is designed to activate the update of EMS, when a new corresponding driving cycle is expected. An Exploration Factor (EF) is proposed to overcome the disadvantages of the normal RL algorithm in convergence rate and reward cost evaluation. The diverse KL divergence rates are examined to seek optimal solutions. The normal-RL strategy, rule-based strategy, and dynamic programming strategy are implemented as benchmark strategies to verify the effectiveness of the proposed strategy. The validation results indicate that the improved RL algorithm with EF makes it possible to promote the EMS capable of significantly improving the energy efficiency of a plug-in PHEV.

Optimizing Constrained Guidance Policy With Minimum Overload Regularization

Using reinforcement learning (RL) algorithm to optimize guidance law can address non-idealities in complex environment. However, the optimization is difficult due to huge state-action space, unstable training, and high requirements on expertise. In this paper, the constrained guidance policy of a neural guidance system is optimized using improved RL algorithm, which is motivated by the idea of traditional model-based guidance method. A novel optimization objective with minimum overload regularization is developed to restrain the guidance policy directly from generating redundant missile maneuver. Moreover, a bi-level curriculum learning is designed to facilitate the policy optimization. Experiment results show that the proposed minimum overload regularization can reduce the vertical overloads of missile significantly, and the bi-level curriculum learning can further accelerate the optimization of guidance policy.

Crash mitigation controller for unavoidable T-bone collisions using reinforcement learning

T-bone collision constitutes an emergency crash scenario that results in casualties and heavy losses; it is an excessively complicated scenario that cannot be handled by conventional control systems. This paper presents an innovative crash mitigation controller for application during unavoidable T-bone collisions to expand the vehicle-maneuverability envelope and minimize crash severity; this controller combines prior knowledge using an optimum expert-behavior policy and drift-operation mechanism based on an improved reinforcement learning algorithm, TD3. Vehicle and tire modeling are performed considering the nonlinear and coupled dynamics characteristics to improve control accuracy. Unlike conventional control systems and other reinforcement learning algorithms, the proposed controller realizes the optimum crash mitigation effect under different scenarios. It is expected to afford autonomous driving technologies with enhanced operating capabilities under extreme conditions.

Improved reinforcement learning algorithm for mobile robot path planning

In order to solve the problem that traditional Q-learning algorithm has a large number of invalid iterations in the early convergence stage of robot path planning, an improved reinforcement learning algorithm is proposed. Firstly, the gravitational potential field in the improved artificial potential field algorithm is introduced when the Q table is initialized to accelerate the convergence. Secondly, the Tent Chaotic Mapping algorithm is added to the initial state determination process of the algorithm, which allows the algorithm to explore the environment more fully. In addition, an ε-greed strategy with the number of iterations changing the ε value becomes the action selection strategy of the algorithm, which improves the performance of the algorithm. Finally, the grid map simulation results based on MATLAB show that the improved Q-learning algorithm has greatly reduced the path planning time and the number of non-convergence iterations compared with the traditional algorithm.

Reliable Path Planning Algorithm Based on Improved Artificial Potential Field Method

In order to solve the “minimum trap” of artificial potential field method and the limitation of traditional path planning algorithm in dynamic obstacle environment, a path planning control algorithm based on improved artificial potential field method is proposed. Firstly, a virtual potential field detection circle model (VPFDCM) with adjustable radius is proposed to detect the “minimum trap” formed by the repulsion field of obstacles in advance. And the motion model of unmanned vehicle is established. Combined with the improved reinforcement learning algorithm based on Long Short-Term Memory(LSTM), the radius of virtual potential field detection circle is adjusted to achieve effective avoidance of dynamic obstacles, The reliable online collision free path planning of unmanned vehicle in semi closed dynamic obstacle environment is realized. Finally, the reliability and robustness of the algorithm are verified by MATLAB simulation. The simulation results show that the improved artificial potential field method can effectively solve the problem of unmanned vehicle falling into the “minimum trap” and improve the reliability of unmanned vehicle movement. Compared with the traditional artificial potential field method, the improved artificial potential field method can achieve more than 90% success rate in obstacle avoidance.

Optimization of English Online Learning Dictionary System Based on Multiagent Architecture

As a universal language in the world, English has become a necessary language communication tool under the globalization of trade. Intelligent, efficient, and reasonable English language-assisted learning system helps to further improve the English ability of language learners. English online learning dictionary, as an important query tool for English learners, is an important part of English online learning. This paper will optimize the design of English online learning dictionary system based on multiagent architecture. Based on the hybrid multiagent cooperative algorithm, this paper will improve the disadvantages of the online English learning dictionary system and propose an appropriate dictionary application evaluation function. At the same time, an improved reinforcement learning algorithm is introduced into the corresponding English online learning dictionary navigation problem so as to improve the efficiency of the online English learning dictionary system. English online learning dictionary is more intelligent and efficient. In this paper, the new online learning dictionary system optimization algorithm is proposed and compared with the traditional system algorithm. The experimental results show that the algorithm proposed in this paper solves the collaborative confusion problem of English learning online dictionary to a certain extent and further solves the corresponding navigation problem so as to improve the efficiency.

An adaptive adjustment strategy for bolt posture errors based on an improved reinforcement learning algorithm

Designing an intelligent and autonomous system remains a great challenge in the assembly field. Most reinforcement learning (RL) methods are applied to experiments with relatively small state spaces. However, the complicated situation and high-dimensional spaces of the assembly environment cause traditional RL methods to behave poorly in terms of their efficiency and accuracy. In this paper, a model-driven adaptive proximal proximity optimization (MAPPO) method was presented to make the assembly system autonomously rectify the bolt posture error. In the MAPPO method, a probabilistic tree and adaptive reward mechanism were used to improve the calculation efficiency and accuracy of the traditional PPO method. The size of the action space was reduced by establishing a hierarchical logical relationship for each parameter with a probabilistic tree. Based on an adaptive reward mechanism, the phenomenon that the algorithm easily falls into local minima could be improved. Finally, the proposed method was verified based on the Unity simulation engine. The advancement and robustness of the proposed model were also validated by comparing different cases in simulations and experiments. The results revealed that MAPPO has better algorithm efficiency and accuracy compared with other state-of-the-art algorithms.

Hierarchical learning optimisation method for the coordination dispatch of the inter‐regional power grid considering the quality of service index

The economic dispatch of the inter-regional power grid with multiple uncertain sources and loads is focused in this study. As the tie-line can transmit the power between regional grids, a coordination mechanism between the tie-line power schedule and the economic dispatch of regional power grids is built to solve the focused dispatch problem under the uncertain environment. Furthermore, the quality of service (QoS) method is introduced in this study to consider and study the service attribute in the power dispatch process, which is quantified by the responding behaviour of the price-sensitive consumers. Then, a model-free hierarchical optimisation method based on the learning technique is designed. The hierarchical structure consists of two levels and multiple agents, where the agents learn knowledge from the interaction between themselves and the environment. An improved reinforcement learning algorithm is adopted to find the optimal dispatch policy for each agent, which realises an online optimisation with the operation samples rather than the support of an accurate system model. Finally, the simulation results are shown to validate the effectiveness of the designed method. Specifically, the optimisation process and the obtained dispatch policies are analysed, and the impact of the QoS index on the optimisation is introduced.

Research on parameter optimisation of dynamic priority scheduling algorithm based on improved reinforcement learning

The dynamic priority scheduling algorithm is one of the real-time scheduling algorithms in a power system. However, it ignores the impact of the weight of each index when selecting indicators that affect scheduling performance. There is no definite objective function relation between weight parameters and scheduling performance. Hence, the heuristic algorithm is difficult to optimise the weight parameters. Aiming to solve this problem, a dynamic priority scheduling algorithm based on improved reinforcement learning (RL) is proposed for parameter optimisation. By learning from each other, the weighting parameters and the deadline miss rate (DMR), the global optimisation of weighting parameters can be achieved, but the learning efficiency of the conventional RL method is low. According to the task scheduling performance (the DMR) and the task characteristics, this study improves the RL action step and reward function, which accelerates the online learning speed and improves the optimisation ability of the RL algorithm. Experimental results show that the improved RL algorithm not only optimises the weight parameters but reduces the DMR, which reduces the number of iterations of RL. A scheduling algorithm optimised by RL can be better applied to industrial control and power system resource scheduling, which not only improves control efficiency but reduces scheduling costs.

An improved reinforcement learning algorithm based on knowledge transfer and applications in autonomous vehicles

Autonomous learning is a crucially important capability of intelligent robots. As one of the most fashionable machine learning techniques, the reinforcement learning (RL) enables an agent taking an optimized action by interacting with the environment so as to maximize some notion of cumulative reward. In this paper, an improved RL algorithm, named as the KT-HA-Q(λ) algorithm, is proposed by resorting to the knowledge transfer of source domain. First, a BP neural network and a liner sensor network are skillfully constructed to perform the knowledge transfer of source task for weight initialization in target task, and the knowledge transfer on actions of case base obtained by source domain, respectively. Then, the novel case base expansion and progressive forgetting criterion, which realize the balance between new experience via online learning and historical experience in the case base, are developed to enhance the learning efficiency and the learning rate. Furthermore, an improved heuristic function is proposed by replacing the action traditionally obtained via a selection strategy by the experience action. This function acts as a crucial role for both the best action selection and its Q value calculation. Finally, the proposed algorithm is utilized in the hill-climbing experiment of unmanned vehicles under a complex 3D scene by transferring the knowledge obtained in a 2D scene. The results of contrast experiments verified the advantages and effectiveness of the proposed algorithm.

Q Value Reinforcement Learning Algorithm Based on Multi Agent System

Q-learning algorithm of multi-agent system is studied in this paper. In order to improve the learning efficiency and convergence speed of the Q algorithm which is a typical learning algorithm of multi agent system, this paper proposes an improved reinforcement learning algorithm of multi agent system based on the existing design experience and surrounding environment information. The Q learning algorithm is effectively extended to the multi agent systems by information sharing among multiple agents in the improved algorithm. The effectiveness of the proposed algorithm is verified by simulation.

An Improved Reinforcement Learning Algorithm for Cooperative Behaviors of Mobile Robots

Reinforcement learning algorithm for multirobot will become very slow when the number of robots is increasing resulting in an exponential increase of state space. A sequentialQ-learning based on knowledge sharing is presented. The rule repository of robots behaviors is firstly initialized in the process of reinforcement learning. Mobile robots obtain present environmental state by sensors. Then the state will be matched to determine if the relevant behavior rule has been stored in the database. If the rule is present, an action will be chosen in accordance with the knowledge and the rules, and the matching weight will be refined. Otherwise the new rule will be appended to the database. The robots learn according to a given sequence and share the behavior database. We examine the algorithm by multirobot following-surrounding behavior, and find that the improved algorithm can effectively accelerate the convergence speed.

Dynamic job-shop scheduling using reinforcement learning agents

Static and dynamic scheduling methods have attracted a lot of attention in recent years. Among these, dynamic scheduling techniques handle scheduling problems where the scheduler does not possess detailed information about the jobs, which may arrive at the shop at any time. In this paper, an intelligent agent based dynamic scheduling system is proposed. It consists of two independent components: the agent and the simulated environment. The agent selects the most appropriate priority rule according to the shop conditions in real time, while simulated environment performs scheduling activities using the rule selected by the agent. The agent is trained by an improved reinforcement learning algorithm through the learning stage and then it successively makes decisions to schedule the operations.