Improved Reinforcement Learning Research Articles

Method for Collaborative Layout Optimization of Ship Equipment and Pipe Based on Improved Multi-Agent Reinforcement Learning and Artificial Fish Swarm Algorithm

The engine room is the core area of a ship, critical to its operation, safety, and efficiency. Currently, many researchers merely address the ship engine room layout design (SERLD) problem using optimization algorithms and independent layout strategies. However, the engine room environment is complex, involving two significantly different challenges: equipment layout and pipe layout. Traditional methods fail to achieve optimal collaborative layout objectives. To address this research gap, this paper proposes a collaborative layout method that combines improved reinforcement learning and heuristic algorithms. For equipment layout, the engine room space is first discretized into a grid, and a Markov decision process (MDP) framework suitable for equipment layout is proposed, including state space, action space, and reward mechanisms suitable for equipment layout. An improved adaptive guided multi-agent Q-learning (AGMAQL) algorithm is employed to train the layout model in a centralized manner, with enhancements made to the agent’s exploration state, exploration action, and learning strategy. For pipe layout, this paper proposes an improved adaptive trajectory artificial fish swarm algorithm (ATAFSA). This algorithm incorporates a hybrid encoding method, adaptive strategy, scouting strategy, and parallel optimization strategy, resulting in enhanced stability, accuracy, and problem adaptability. Subsequently, by comprehensively considering layout objectives and engine room attributes, a collaborative layout method incorporating hierarchical and adaptive weight strategies is proposed. This method optimizes in phases according to the layout objectives and priorities of different stages, achieving multi-level optimal layouts and providing designers with various reference schemes with different focuses. Finally, based on a typical real-world engine room engineering case, various leading algorithms and strategies are tested and compared. The results show that the proposed AGMAQL-ATAFSA (AGMAQL-ATA) exhibits robustness, efficiency, and engineering practicality. Compared to previous research methods and algorithms, the final layout quality improved overall: equipment layout effectiveness increased by over 4.0%, pipe optimization efficiency improved by over 40.4%, and collaborative layout effectiveness enhanced by over 2.2%.

Interpretable multi-graph convolution network integrating spatial-temporal attention and dynamic combination for wind power forecasting

Graph neural networks (GNN) can effectively improve wind power forecasting accuracy due to its ability to capture spatial correlations between wind farms. However, conventional GNN-based forecasting models constructing the graphs from only one specific perspective fail to characterize the spatial correlation patterns comprehensively and accurately. Furthermore, their black-box structure leads to deficient interpretability of the forecasting results, which hampers the quantification of forecasts uncertainty and reduces the credibility of applying these forecasting models in practice. To address the above problems, an interpretable ultra-short-term wind power forecasting method is proposed based on multi-graph convolution network integrating spatial–temporal attention and dynamic graph combination. Four undirected weight graphs are constructed to form a multi-graph scheme by considering various geographical and statistical aspects, and then graph convolution network (GCN) and temporal convolution network (TCN) are used to enhance the forecasting performance by leveraging information in both space and time dimensions. Meanwhile, it consists of temporal attention that deeply mines the temporal dependence information and spatial attention that interprets influence of wind farms on the forecasts. Additionally, to boost the adaptability and generalization capability of the proposed method, an improved reinforcement learning is elaborately designed to combine the results of different graphs by dynamically optimizing their weights to obtain the final forecasts. Experiment results suggest that the proposed method exhibits superior forecasting accuracy and reasonable interpretability compared with other benchmark methods.

RLingua: Improving Reinforcement Learning Sample Efficiency in Robotic Manipulations With Large Language Models

RLingua: Improving Reinforcement Learning Sample Efficiency in Robotic Manipulations With Large Language Models

AUV Obstacle Avoidance Framework Based on Event-Triggered Reinforcement Learning

Autonomous Underwater Vehicles (AUVs), as a member of the unmanned intelligent ocean vehicle group, can replace human beings to complete dangerous tasks in the ocean. It is of great significance to apply reinforcement learning (RL) to AUVs to realize intelligent control. This paper proposes an AUV obstacle avoidance framework based on event-triggered reinforcement learning. Firstly, an environment perception model is designed to judge the relative position relationship between the AUV and all unknown obstacles and known targets. Secondly, considering that the detection range of AUVs is limited, and the proposed method needs to deal with unknown static obstacles and unknown dynamic obstacles at the same time, two different event-triggered mechanisms are designed. Soft actor–critic (SAC) with a non-policy sampling method is used. Then, improved reinforcement learning and the event-triggered mechanism are combined in this paper. Finally, a simulation experiment of the obstacle avoidance task is carried out on the Gazebo simulation platform. Results show that the proposed method can obtain higher rewards and complete tasks successfully. At the same time, the trajectory and the distance between each obstacle confirm that the AUV can reach the target well while maintaining a safe distance from static and dynamic obstacles.

A Novel Automatic Generation Control Method Based on the Large-Scale Electric Vehicles and Wind Power Integration Into the Grid.

In order to solve the problem of frequency instability of power system due to strong random disturbance caused by large-scale electric vehicles and wind power grid connection, an improved reinforcement learning algorithm, namely, optimistic initialized double Q, is proposed in this article from the perspective of automatic generation control. The proposed algorithm uses the optimistic initialization principle to expand the agent action exploration space, so as to prevent Q-learning from falling into local optimum by greedy strategy; meanwhile, it integrates double Q-learning to solve the problem of overestimation of action value in traditional reinforcement learning based on Q-learning. In the algorithm, the hyperparameter ατ is introduced to improve the learning efficiency, and the reward bτ based on exploration times is introduced to increase the Q value estimation to drive the exploration of the algorithm, so as to obtain the optimal solution. By simulating the two-area load frequency control model integrated with large-scale electric vehicles and the four-area interconnected power grid model integrated with large-scale wind power generation, it is verified that the proposed algorithm can obtain the global optimal solution, thus effectively solvinng the frequency instability caused by strong random disturbance in the grid-connected mode of large-scale wind power generation, and compared with many reinforcement learning algorithms, the proposed algorithm has better control performance.

A novel reinforcement learning policy optimization based adaptive VSG control technique for improved frequency stabilization in AC microgrids

In this paper, an improved Reinforcement Learning (RL) based approach for adaptive Virtual Synchronous Generator (VSG) control of grid-forming inverters is proposed. High penetration of inverter-based resources causes recurrent frequency stabilization issues. For this matter, a novel model-free Actor–Critic policy optimization technique is proposed to improve frequency transients metrics such as frequency nadir and Rate of Change of Frequency (RoCoF). Stability analysis of the VSG controlled system is conducted through small signal modeling to determine the stability margins of both the virtual inertia parameter J and damping coefficient Dp. The control problem is formulated in RL terms using a new technique for state, action and reward construction that improves the learning process of the agents. The proposed approach’s efficiency is tested for three RL algorithms, namely, Deep Deterministic Policy Gradient (DDPG), Soft Actor–Critic (SAC), and Twin-Delayed DDPG (TD3) with special attention given to the latter. Various performance metrics such as training time and maximum cumulative episodic reward are used to compare the three algorithms. The resulting agent based VSG adaptive controllers are simulated and then compared for power quality and frequency stabilization capabilities. Finally, a comparison between non-adaptive VSG and TD3 based VSG is presented to conclude on the efficiency improvements achieved by the proposed approach.

Autonomous Shape Decision Making of Morphing Aircraft with Improved Reinforcement Learning

The autonomous shape decision-making problem of a morphing aircraft (MA) with a variable wingspan and sweep angle is studied in this paper. Considering the continuity of state space and action space, a more practical autonomous decision-making algorithm framework of MA is designed based on the deep deterministic policy gradient (DDPG) algorithm. Furthermore, the DDPG with a task classifier (DDPGwTC) algorithm is proposed in combination with the long short-term memory (LSTM) network to improve the convergence speed of the algorithm. The simulation results show that the shape decision-making algorithm based on the DDPGwTC enables MA to adopt the optimal morphing strategy in different task environments with higher autonomy and environmental adaptability, which verifies the effectiveness of the proposed algorithm.

Research on MOOC Curriculum Recommendation Model of Higher Vocational English based on Improved Intensive Learning Network

With the development and maturity of the Internet education industry, more and more vocational colleges have opened English teaching courses based on massive open online courses courses. However, there are many English-related courses on the massive open online courses course platform, and the use of scientific recommendation models can improve the teaching quality of such courses. Therefore, this research attempts to design two improved attention mechanisms and user-based embedded expression using meta-path technology. At the same time, these two are combined with reinforcement learning technology to design an improved massive open online courses English course recommendation model. The test results show that the hit rate of the model designed in this study is 89.84%, 74.28%, 70.81% and 71.35% respectively when the rank number is 20 and the parameter is 10. At this time, the cumulative income of normalized discount is 48.24%, 34.58%, 25.96% and 28.69% respectively. However, when the number of calculated samples reaches the maximum value of 1158609, the calculation time of the improved reinforcement learning recommendation model is 1867 seconds, which is also higher than the comparison model. The experimental results show that the curriculum recommendation accuracy of the massive open online courses recommendation model designed in this study is higher and the recommendation results are more reasonable. The results of this research have a certain application potential in the field of the construction of online education in colleges and universities.

An Improved Reinforcement Learning Method Based on Unsupervised Learning

An Improved Reinforcement Learning Method Based on Unsupervised Learning

Improved Reinforcement Learning for Reliable Routing in Medical Wireless Sensor Networks

Improved Reinforcement Learning for Reliable Routing in Medical Wireless Sensor Networks

Improving reinforcement learning based moving object grasping with trajectory prediction

Improving reinforcement learning based moving object grasping with trajectory prediction

An ensemble learning framework for click-through rate prediction based on a reinforcement learning algorithm with parameterized actions

Click-through rate (CTR) prediction is essential for targeted advertising systems. Although there have been many studies on CTR prediction and forming some representative models, building a CTR prediction model that performs well under various scenarios remains a formidable challenge. Ensemble learning is a class of methods that improve the final performance by aggregating multiple base models. This paper proposes an ensemble learning framework that can predict the click probability for each ad impression by combining the results of one or more CTR prediction models. The framework utilizes an improved Reinforcement Learning (RL) algorithm that supports parametric operations to dynamically determine the models and their weights that participated in predicting CTR for each ad impression. Specifically, for each ad impression, the agent generates a discrete action and a continuous action vector based on predicted click-through rates by all base models. The discrete action determines the number of base models involved in prediction, while each continuous action decides the weight of each base model. We validate the ensemble learning framework on three benchmark datasets, including iPinYou, Criteo, and Avazu. The experimental results demonstrate that, in most cases, the ensemble learning framework outperforms the base models in terms of AUC and LogLoss metrics. Finally, we give two simplified variants of the ensemble learning framework and discuss their applicability.

A multi-agent reinforcement learning algorithm with the action preference selection strategy for massive target cooperative search mission planning

Target search is widely applied in military reconnaissance, geological exploration and personnel search and rescue. Most target search algorithms perform well in single target search but are inefficient or even ineffective in multi-target search. To solve the multi-target search problem in an uncertain environment, this paper constructs a multi-agent massive target cooperative search mission planning model and proposes an improved reinforcement learning algorithm using the action preference selection strategy. Based on the Reinforce algorithm, this algorithm solves the problem of invalid searches within the stochastic strategy by changing the preferred action selection method. The proposed method improves the efficiency of multiple agents in the search for targets without collision using a cooperative mechanism and reward rules based on the odor effect. Simulation experiments are conducted in three aspects to verify the effectiveness and robustness of the improved algorithm and compare it with other reinforcement learning algorithms in the field of multi-agent learning. The results demonstrate that the improved algorithm has obvious advantages in terms of mission success rate, target search rate and average search time, and the movement trajectory of multiple agents is more concise.

Excitatory stimulation of the ventromedial prefrontal cortex reduces cognitive gambling biases via improved feedback learning

Humans are subject to a variety of cognitive biases, such as the framing-effect or the gambler's fallacy, that lead to decisions unfitting of a purely rational agent. Previous studies have shown that the ventromedial prefrontal cortex (vmPFC) plays a key role in making rational decisions and that stronger vmPFC activity is associated with attenuated cognitive biases. Accordingly, dysfunctions of the vmPFC are associated with impulsive decisions and pathological gambling. By applying a gambling paradigm in a between-subjects design with 33 healthy adults, we demonstrate that vmPFC excitation via transcranial direct current stimulation (tDCS) reduces the framing-effect and the gambler's fallacy compared to sham stimulation. Corresponding magnetoencephalographic data suggest improved inhibition of maladaptive options after excitatory vmPFC-tDCS. Our analyses suggest that the underlying mechanism might be improved reinforcement learning, as effects only emerge over time. These findings encourage further investigations of whether excitatory vmPFC-tDCS has clinical utility in treating pathological gambling or other behavioral addictions.

An Efficient Aircraft Conflict Detection and Resolution Method Based on an Improved Reinforcement Learning Framework

With the steady increase of air traffic column, an auxiliary decision tool is required to compensate the operation redundancy deficiency of more sectors of air traffic control. To solve the problem of nonconflict high-density departure and arrival traffic flow, this method is expected to rapidly establish and maintain safe separation with more flexible changing strategies for aircraft heading and speed. This paper proposes an improved reinforcement learning framework to achieve conflict detection and resolution. The proposed framework includes the first development of an air traffic flow model based on a multiagent Markov decision process. The goal reward function was then maximized by improved Monte-Carlo tree search combined with an upper confidence bound tree. Three simulation scenarios were designed for illustrating the improvements of the proposed algorithm, with the results indicating that the algorithm could establish and maintain safe separation between 20 agents in the simplified hexagon-shaped airspace of Huadong, China. Furthermore, the proposed method was demonstrated to reduce the number of conflicts between aircraft agents by up to 26.32% compared to previous research.

Research on Inertial Navigation Technology of Unmanned Aerial Vehicles with Integrated Reinforcement Learning Algorithm

With the continuous expansion of unmanned aerial vehicle (UAV) applications, traditional inertial navigation technology exhibits significant limitations in complex environments. In this study, we integrate improved reinforcement learning (RL) algorithms to enhance existing unmanned aerial vehicle inertial navigation technology and introduce a modulated mechanism (MM) for adjusting the state of the intelligent agent in an innovative manner [1,2]. Through interaction with the environment, the intelligent machine can learn more effective navigation strategies [3]. The ultimate goal is to provide a foundation for autonomous navigation of unmanned aerial vehicles during flight and improve navigation accuracy and robustness. We first define appropriate state representation and action space, and then design an adjustment mechanism based on the actions selected by the intelligent agent. The adjustment mechanism outputs the next state and reward value of the agent. Additionally, the adjustment mechanism calculates the error between the adjusted state and the unadjusted state. Furthermore, the intelligent agent stores the acquired experience samples containing states and reward values in a buffer and replays the experiences during each iteration to learn the dynamic characteristics of the environment. We name the improved algorithm as the DQM algorithm. Experimental results demonstrate that the intelligent agent using our proposed algorithm effectively reduces the accumulated errors of inertial navigation in dynamic environments. Although our research provides a basis for achieving autonomous navigation of unmanned aerial vehicles, there is still room for significant optimization. Further research can include testing unmanned aerial vehicles in simulated environments, testing unmanned aerial vehicles in realworld environments, optimizing the design of reward functions, improving the algorithm workflow to enhance convergence speed and performance, and enhancing the algorithm's generalization ability. It has been proven that by integrating reinforcement learning algorithms, unmanned aerial vehicles can achieve autonomous navigation, thereby improving navigation accuracy and robustness in dynamic and changing environments [4]. Therefore, this research plays an important role in promoting the development and application of unmanned aerial vehicle technology.

Improved reinforcement learning for collision-free local path planning of dynamic obstacle

The rapid development of artificial intelligence has driven the transformation of traditional industry. The proposal for an intelligent ship reflects the development of the shipping industry in the direction of intelligence. Safe maritime transportation is an essential branch of the intelligent ship industry. In this context, applying various intelligent algorithms in ship dynamic obstacle avoidance has attracted the attention and discussion of scholars. Due to the complexity and diversity of ship navigation, environments cannot be described by a definite mathematical model. Reinforcement learning has certain advantages in solving path planning in a complex environment. Traditional mathematical algorithms and swarm intelligence algorithms need mathematical models to constrain boundary conditions when solving complex problems. However, some complex dynamic environments in the real world cannot define mathematical models. This paper optimizes the parameter update and exploration strategies of the actor–critic algorithm in reinforcement learning to improve the algorithm’s performance. Under complex meteorological conditions, improved reinforcement learning is applied to the dynamic ship offshore channel simulation environment to verify the algorithm’s performance.

A traffic light control method based on multi-agent deep reinforcement learning algorithm

Intelligent traffic light control (ITLC) algorithms are very efficient for relieving traffic congestion. Recently, many decentralized multi-agent traffic light control algorithms are proposed. These researches mainly focus on improving reinforcement learning method and coordination method. But, as all the agents need to communicate while coordinating with each other, the communication details should be improved as well. To guarantee communication effectiveness, two aspect should be considered. Firstly, a traffic condition description method need to be designed. By using this method, traffic condition can be described simply and clearly. Secondly, synchronization should be considered. As different intersections have different cycle lengths and message sending event happens at the end of each traffic signal cycle, every agent will receive messages of other agents at different time. So it is hard for an agent to decide which message is the latest one and the most valuable. Apart from communication details, reinforcement learning algorithm used for traffic signal timing should also be improved. In the traditional reinforcement learning based ITLC algorithms, either queue length of congested cars or waiting time of these cars is considered while calculating reward value. But, both of them are very important. So a new reward calculation method is needed. To solve all these problems, in this paper, a new ITLC algorithm is proposed. To improve communication efficiency, this algorithm adopts a new message sending and processing method. Besides, to measure traffic congestion in a more reasonable way, a new reward calculation method is proposed and used. This method takes both waiting time and queue length into consideration.

Improved reinforcement learning path planning algorithm integrating prior knowledge.

In order to realize the optimization of autonomous navigation of mobile robot under the condition of partial environmental knowledge known. An improved Q-learning reinforcement learning algorithm based on prior knowledge is proposed to solve the problem of slow convergence and low learning efficiency in mobile robot path planning. Prior knowledge is used to initialize the Q-value, so as to guide the agent to move toward the target direction with a greater probability from the early stage of the algorithm, eliminating a large number of invalid iterations. The greedy factor ε is dynamically adjusted based on the number of times the agent successfully reaches the target position, so as to better balance exploration and exploitation and accelerate convergence. Simulation results show that the improved Q-learning algorithm has a faster convergence rate and higher learning efficiency than the traditional algorithm. The improved algorithm has practical significance for improving the efficiency of autonomous navigation of mobile robots.

Orbit correction based on improved reinforcement learning algorithm

Recently, reinforcement learning (RL) algorithms have been applied to a wide range of control problems in accelerator commissioning. In order to achieve efficient and fast control, these algorithms need to be highly efficient, so as to minimize the online training time. In this paper, we incorporated the beam position monitor trend into the observation space of the twin delayed deep deterministic policy gradient (TD3) algorithm and trained two different structure agents, one based on physical prior knowledge and the other using the original TD3 network architecture. Both of the agents exhibit strong robustness in the simulated environment. The effectiveness of the agent based on physical prior knowledge has been validated in a real accelerator. Results show that the agent can overcome the difference between simulated and real accelerator environments. Once the training is completed in the simulated environment, the agent can be directly applied to the real accelerator without any online training process. The RL agent is deployed to the medium energy beam transport section of China Accelerator Facility for Superheavy Elements. Fast and automatic orbit correction is being tested with up to ten degrees of freedom. The experimental results show that the agents can correct the orbit to within 1 mm. Moreover, due to the strong robustness of the agent, when a trained agent is applied to different lattices of different particles, the orbit correction can still be completed. Since there are no online data collection and training processes, all online corrections are done within 30 s. This paper shows that, as long as the robustness of the RL algorithm is sufficient, the offline learning agents can be directly applied to online correction, which will greatly improve the efficiency of orbit correction. Such an approach to RL may find promising applications in other areas of accelerator commissioning.