Improved Reinforcement Learning Research Articles

Adaptive hysteresis compensation control of a macro-fiber composite bimorph by improved reinforcement learning

The hysteresis characteristics related to the frequency and amplitude of the control signal seriously affect the precision of the displacement tracking control of the macro-fiber composite (MFC) bimorph. The traditional feedforward compensator tends to exhibit low precision in controlling displacement. Although it enhances the control accuracy to a certain extent by incorporating a feedback controller, the existing feedback controller has a weak adaptive ability. Thus, the Q-learning (QL) algorithm is combined with the Bouc-Wen (BW) feedforward compensator in this study. The output voltage from the BW compensator has a more significant impact on the control accuracy and convergence speed of the QL algorithm. Therefore, this paper proposes an improved QL (IQL) algorithm that leverages the control error. Moreover, a BW-IQL adaptive control method is proposed to realize precise adaptive control of the MFC bimorph. Experimental comparisons are performed with the proposed method and the traditional BW, BW-PID, and BW-fuzzy PID controllers. The BW-IQL controller reduces the average relative errors of the three controllers by 86.9%, 59.9%, and 41.8%, respectively. Meanwhile, the Q̄ table plays a major role in error suppression in the IQL controller. These results verify that the BW-IQL control method has higher adaptability and accuracy.

Two-stage learning scatter search algorithm for the distributed hybrid flow shop scheduling problem with machine breakdown

The distributed hybrid flow shop scheduling problem with machine breakdown is investigated to reduce the negative impact on real production caused by machine breakdown events. (DHFSPMB). DHFSPMB comprises two subproblems: the maintenance problem with machine breakdown and the distributed hybrid flow shop scheduling problem (DHFSP). A rescheduling method is designed to address the maintenance problem. Subsequently, a two-stage learning scatter search (TLSS) algorithm is proposed for optimizing the DHFSP when the machines break down. Firstly, a mixed integer programming model for DHFSPMB is constructed. Secondly, TLSS employs an improved reinforcement learning approach to enhance the capability of exploration by guiding the direction of global search. A two-stage approach is designed to address the lack of knowledge in the early periods of learning. Finally, a hybrid search strategy is devised to enhance the development capability of TLSS. The experimental results demonstrate that the TLSS algorithm outperforms the comparison algorithms in effectively addressing the DHFSPMB.

A HEART FAILURE PREDICTION ALGORITHM BASED ON IMPROVED REINFORCEMENT LEARNING FRAMEWORK

Reducing the occurrence of diseases has become an important research direction in today’s social medicine. Accurate prediction of heart failure can greatly reduce the complication rate, allowing patients to know the condition in advance and receive treatment. To deal with heart failure, this paper proposes a heart failure prediction algorithm based on a reinforcement learning framework. This paper combines and improves the reinforcement learning algorithm and the swarm intelligence (SI) optimization algorithm, aiming to use the reinforcement learning algorithm to improve the global optimization capability of the SI optimization algorithm. In order to better improve the global search ability of the SI optimization algorithm, the penalty function will be replaced by a combination mode of dynamic and static reward. The improved reinforcement learning algorithm framework is significantly better than the previous algorithm framework and has been used to predict heart failure. The experimental results have proven the effectiveness of the algorithm and its accuracy in prediction of heart failure.

A fault reconfiguration strategy based on logical structure and improved reinforcement learning for ship DC regional grid

Ship DC Regional Grid (SDRG) is an advanced power system that divides the power system into different regions, which can improve energy efficiency and stability. However, little research has been done on its fault reconfiguration. In the event of a fault, the switch is controlled by the fault reconfiguration technique to reconfigure the direction of the flow, which effectively improves the reliability of the power supply to the loads. A two-stage fault reconfiguration method based on Logical Structure (LS) and Improved Reinforcement Learning (IRL) is proposed, aiming to maximize the power load and minimize the number of switches. First, LS is utilized to design the load connectivity matrix based on the grid characteristics, and the grid topology is adjusted to achieve fast reconfiguration based on load prioritization. Further, IRL is employed to improve the reconfiguration speed and quality by establishing the optimal solution experience pool and improving the reward and punishment mechanism. The proposed method enables the selection of an appropriate reconfiguration strategy based on fault size, consequently improving convergence time and flexibility. Comparative experimental results in a four-region SDRG show that the proposed method can effectively solve the SDRG fault reconfiguration problem.

Method for Collaborative Layout Optimization of Ship Equipment and Pipe Based on Improved Multi-Agent Reinforcement Learning and Artificial Fish Swarm Algorithm

The engine room is the core area of a ship, critical to its operation, safety, and efficiency. Currently, many researchers merely address the ship engine room layout design (SERLD) problem using optimization algorithms and independent layout strategies. However, the engine room environment is complex, involving two significantly different challenges: equipment layout and pipe layout. Traditional methods fail to achieve optimal collaborative layout objectives. To address this research gap, this paper proposes a collaborative layout method that combines improved reinforcement learning and heuristic algorithms. For equipment layout, the engine room space is first discretized into a grid, and a Markov decision process (MDP) framework suitable for equipment layout is proposed, including state space, action space, and reward mechanisms suitable for equipment layout. An improved adaptive guided multi-agent Q-learning (AGMAQL) algorithm is employed to train the layout model in a centralized manner, with enhancements made to the agent’s exploration state, exploration action, and learning strategy. For pipe layout, this paper proposes an improved adaptive trajectory artificial fish swarm algorithm (ATAFSA). This algorithm incorporates a hybrid encoding method, adaptive strategy, scouting strategy, and parallel optimization strategy, resulting in enhanced stability, accuracy, and problem adaptability. Subsequently, by comprehensively considering layout objectives and engine room attributes, a collaborative layout method incorporating hierarchical and adaptive weight strategies is proposed. This method optimizes in phases according to the layout objectives and priorities of different stages, achieving multi-level optimal layouts and providing designers with various reference schemes with different focuses. Finally, based on a typical real-world engine room engineering case, various leading algorithms and strategies are tested and compared. The results show that the proposed AGMAQL-ATAFSA (AGMAQL-ATA) exhibits robustness, efficiency, and engineering practicality. Compared to previous research methods and algorithms, the final layout quality improved overall: equipment layout effectiveness increased by over 4.0%, pipe optimization efficiency improved by over 40.4%, and collaborative layout effectiveness enhanced by over 2.2%.

Interpretable multi-graph convolution network integrating spatio-temporal attention and dynamic combination for wind power forecasting

Graph neural networks (GNN) can effectively improve wind power forecasting accuracy due to their ability to capture spatial correlations between wind farms. However, conventional GNN-based forecasting models constructing the graphs from only one specific perspective fail to characterize the spatial correlation patterns comprehensively and accurately. Furthermore, their black-box structure leads to deficient interpretability of the forecasting results, which hampers the quantification of forecasting uncertainty and reduces the credibility of applying these forecasting models in practice. To address the above problems, an interpretable ultra-short-term wind power forecasting method is proposed based on multi-graph convolution network integrating spatio-temporal attention and dynamic graph combination. Four undirected weight graphs are constructed to form a multi-graph scheme by considering various geographical and statistical aspects, and then graph convolution network (GCN) and temporal convolution network (TCN) are used to enhance the forecasting performance by leveraging information in both space and time dimensions. Meanwhile, it consists of temporal attention that deeply mines the temporal dependencies and spatial attention that interprets the influence of wind farms on the forecasts. Additionally, to boost the adaptability and generalization capability of the proposed method, an improved reinforcement learning is elaborately designed to combine the results of different graphs by dynamically optimizing their weights to obtain the final forecasts. Experiment results suggest that the proposed method exhibits superior forecasting accuracy and reasonable interpretability compared with other benchmark methods.

Improving Reinforcement Learning Exploration by Autoencoders

Reinforcement learning is a field with massive potential related to solving engineering problems without field knowledge. However, the problem of exploration and exploitation emerges when one tries to balance a system between the learning phase and proper execution. In this paper, a new method is proposed that utilizes autoencoders to manage the exploration rate in an epsilon-greedy exploration algorithm. The error between the real state and the reconstructed state by the autoencoder becomes the base of the exploration-exploitation rate. The proposed method is then examined in two experiments: one benchmark is the cartpole experiment while the other is a gridworld example created for this paper to examine long-term exploration. Both experiments show results such that the proposed method performs better in these scenarios.

RLingua: Improving Reinforcement Learning Sample Efficiency in Robotic Manipulations With Large Language Models

RLingua: Improving Reinforcement Learning Sample Efficiency in Robotic Manipulations With Large Language Models

DiffSkill: Improving Reinforcement Learning through diffusion-based skill denoiser for robotic manipulation

Although Reinforcement Learning (RL) has demonstrated impressive success in various applications, addressing complex robotic manipulation tasks remains a formidable challenge. Recently, Skill-based approaches that extract reusable skills from offline data and encode them into a latent space are proposed to leverage prior knowledge for accelerating robot learning. However, existing skill learning methods predominantly rely on regularization constraints or reversible mappings to guide skill prior generation, lacking explicit control over the trade-off between exploiting offline knowledge and exploring novel skill behaviors. In this paper, we point out that the challenge of skill exploration lies in the noise within skill embeddings and propose a novel denoising-based skill-based RL framework, DiffSkill. Specifically, our DiffSkill integrates a diffusion-based skill denoiser into the hierarchical architecture, effectively bridging the gap between offline knowledge and learned skill prior embeddings through iterative denoising. Nevertheless, incorporating diffusion models into the skill-based RL framework for robot control faces two main challenges: (i) Uncertain noisy levels of skill embeddings and (ii) Action oscillation during skill transitions. In this regard, we propose a cycle anneal scheduler for dynamic timestep adjustment and an online momentum smoothing strategy to effectively mitigate oscillations during skill transitions, resulting in more stable and superior performance. Extensive comparison experiments across six challenging robotic manipulation tasks demonstrate that DiffSkill consistently outperforms state-of-the-art methods by a significant margin in all downstream tasks. Ablation studies and additional discussions further validate the effectiveness of each component and strategy.

AUV Obstacle Avoidance Framework Based on Event-Triggered Reinforcement Learning

Autonomous Underwater Vehicles (AUVs), as a member of the unmanned intelligent ocean vehicle group, can replace human beings to complete dangerous tasks in the ocean. It is of great significance to apply reinforcement learning (RL) to AUVs to realize intelligent control. This paper proposes an AUV obstacle avoidance framework based on event-triggered reinforcement learning. Firstly, an environment perception model is designed to judge the relative position relationship between the AUV and all unknown obstacles and known targets. Secondly, considering that the detection range of AUVs is limited, and the proposed method needs to deal with unknown static obstacles and unknown dynamic obstacles at the same time, two different event-triggered mechanisms are designed. Soft actor–critic (SAC) with a non-policy sampling method is used. Then, improved reinforcement learning and the event-triggered mechanism are combined in this paper. Finally, a simulation experiment of the obstacle avoidance task is carried out on the Gazebo simulation platform. Results show that the proposed method can obtain higher rewards and complete tasks successfully. At the same time, the trajectory and the distance between each obstacle confirm that the AUV can reach the target well while maintaining a safe distance from static and dynamic obstacles.

A Novel Automatic Generation Control Method Based on the Large-Scale Electric Vehicles and Wind Power Integration Into the Grid.

In order to solve the problem of frequency instability of power system due to strong random disturbance caused by large-scale electric vehicles and wind power grid connection, an improved reinforcement learning algorithm, namely, optimistic initialized double Q, is proposed in this article from the perspective of automatic generation control. The proposed algorithm uses the optimistic initialization principle to expand the agent action exploration space, so as to prevent Q-learning from falling into local optimum by greedy strategy; meanwhile, it integrates double Q-learning to solve the problem of overestimation of action value in traditional reinforcement learning based on Q-learning. In the algorithm, the hyperparameter ατ is introduced to improve the learning efficiency, and the reward bτ based on exploration times is introduced to increase the Q value estimation to drive the exploration of the algorithm, so as to obtain the optimal solution. By simulating the two-area load frequency control model integrated with large-scale electric vehicles and the four-area interconnected power grid model integrated with large-scale wind power generation, it is verified that the proposed algorithm can obtain the global optimal solution, thus effectively solvinng the frequency instability caused by strong random disturbance in the grid-connected mode of large-scale wind power generation, and compared with many reinforcement learning algorithms, the proposed algorithm has better control performance.

A novel reinforcement learning policy optimization based adaptive VSG control technique for improved frequency stabilization in AC microgrids

In this paper, an improved Reinforcement Learning (RL) based approach for adaptive Virtual Synchronous Generator (VSG) control of grid-forming inverters is proposed. High penetration of inverter-based resources causes recurrent frequency stabilization issues. For this matter, a novel model-free Actor–Critic policy optimization technique is proposed to improve frequency transients metrics such as frequency nadir and Rate of Change of Frequency (RoCoF). Stability analysis of the VSG controlled system is conducted through small signal modeling to determine the stability margins of both the virtual inertia parameter J and damping coefficient Dp. The control problem is formulated in RL terms using a new technique for state, action and reward construction that improves the learning process of the agents. The proposed approach’s efficiency is tested for three RL algorithms, namely, Deep Deterministic Policy Gradient (DDPG), Soft Actor–Critic (SAC), and Twin-Delayed DDPG (TD3) with special attention given to the latter. Various performance metrics such as training time and maximum cumulative episodic reward are used to compare the three algorithms. The resulting agent based VSG adaptive controllers are simulated and then compared for power quality and frequency stabilization capabilities. Finally, a comparison between non-adaptive VSG and TD3 based VSG is presented to conclude on the efficiency improvements achieved by the proposed approach.

Autonomous Shape Decision Making of Morphing Aircraft with Improved Reinforcement Learning

The autonomous shape decision-making problem of a morphing aircraft (MA) with a variable wingspan and sweep angle is studied in this paper. Considering the continuity of state space and action space, a more practical autonomous decision-making algorithm framework of MA is designed based on the deep deterministic policy gradient (DDPG) algorithm. Furthermore, the DDPG with a task classifier (DDPGwTC) algorithm is proposed in combination with the long short-term memory (LSTM) network to improve the convergence speed of the algorithm. The simulation results show that the shape decision-making algorithm based on the DDPGwTC enables MA to adopt the optimal morphing strategy in different task environments with higher autonomy and environmental adaptability, which verifies the effectiveness of the proposed algorithm.

Research on MOOC Curriculum Recommendation Model of Higher Vocational English based on Improved Intensive Learning Network

With the development and maturity of the Internet education industry, more and more vocational colleges have opened English teaching courses based on massive open online courses courses. However, there are many English-related courses on the massive open online courses course platform, and the use of scientific recommendation models can improve the teaching quality of such courses. Therefore, this research attempts to design two improved attention mechanisms and user-based embedded expression using meta-path technology. At the same time, these two are combined with reinforcement learning technology to design an improved massive open online courses English course recommendation model. The test results show that the hit rate of the model designed in this study is 89.84%, 74.28%, 70.81% and 71.35% respectively when the rank number is 20 and the parameter is 10. At this time, the cumulative income of normalized discount is 48.24%, 34.58%, 25.96% and 28.69% respectively. However, when the number of calculated samples reaches the maximum value of 1158609, the calculation time of the improved reinforcement learning recommendation model is 1867 seconds, which is also higher than the comparison model. The experimental results show that the curriculum recommendation accuracy of the massive open online courses recommendation model designed in this study is higher and the recommendation results are more reasonable. The results of this research have a certain application potential in the field of the construction of online education in colleges and universities.

Optimizing Artificial Neural Network Learning Using Improved Reinforcement Learning in Artificial Bee Colony Algorithm

Artificial neural networks (ANNs) are widely used machine learning techniques with applications in various fields. Heuristic search optimization methods are typically used to minimize the loss function in ANNs. However, these methods can lead the network to become stuck in local optima, limiting performance. To overcome this challenge, this study introduces an improved optimization approach, the improvement of reinforcement learning in the artificial bee colony (improved R‐ABC) algorithm, to enhance the optimization process for ANNs. The proposed method aims to overcome the limitations of heuristic search and improve the efficiency of weight adjustment in ANNs. This new approach enhances the discovery phase of the traditional R‐ABC by including the parameters of neighboring food sources, augmenting the search capabilities for finding the optimal solution. The performance of the improved R‐ABC was compared with ANNs utilizing backpropagation with stochastic gradient descent (SGD) and Adam optimizers, as well as other swarm intelligence (SI) methods such as particle swarm optimization (PSO) and traditional R‐ABC. The results showed that both PSO and R‐ABC continuously improved the solutions across all benchmark datasets. In the iris dataset, all SI approaches consistently achieved F1‐scores exceeding 0.94, outperforming SGD and Adam. For the other datasets, the SI approach generally outperformed the other optimization methods. The results indicate that when the improved R‐ABC is applied to ANNs, it outperforms heuristic search optimization, especially as the network size expands. Although SGD and Adam achieved faster execution times with TensorFlow, the study suggests that using PSO and improved R‐ABC can improve model accuracy and efficiency. Advanced SI methods enhance the optimization process and increase the ability of ANNs to obtain optimal solutions. Enhanced R‐ABC and PSO algorithms can significantly improve ANN training performance and efficiency, especially in complex and high‐dimensional datasets.

An Improved Reinforcement Learning Method Based on Unsupervised Learning

An Improved Reinforcement Learning Method Based on Unsupervised Learning

Improved Reinforcement Learning for Reliable Routing in Medical Wireless Sensor Networks

Improved Reinforcement Learning for Reliable Routing in Medical Wireless Sensor Networks

Improving reinforcement learning based moving object grasping with trajectory prediction

Improving reinforcement learning based moving object grasping with trajectory prediction

An ensemble learning framework for click-through rate prediction based on a reinforcement learning algorithm with parameterized actions

Click-through rate (CTR) prediction is essential for targeted advertising systems. Although there have been many studies on CTR prediction and forming some representative models, building a CTR prediction model that performs well under various scenarios remains a formidable challenge. Ensemble learning is a class of methods that improve the final performance by aggregating multiple base models. This paper proposes an ensemble learning framework that can predict the click probability for each ad impression by combining the results of one or more CTR prediction models. The framework utilizes an improved Reinforcement Learning (RL) algorithm that supports parametric operations to dynamically determine the models and their weights that participated in predicting CTR for each ad impression. Specifically, for each ad impression, the agent generates a discrete action and a continuous action vector based on predicted click-through rates by all base models. The discrete action determines the number of base models involved in prediction, while each continuous action decides the weight of each base model. We validate the ensemble learning framework on three benchmark datasets, including iPinYou, Criteo, and Avazu. The experimental results demonstrate that, in most cases, the ensemble learning framework outperforms the base models in terms of AUC and LogLoss metrics. Finally, we give two simplified variants of the ensemble learning framework and discuss their applicability.

A multi-agent reinforcement learning algorithm with the action preference selection strategy for massive target cooperative search mission planning

Target search is widely applied in military reconnaissance, geological exploration and personnel search and rescue. Most target search algorithms perform well in single target search but are inefficient or even ineffective in multi-target search. To solve the multi-target search problem in an uncertain environment, this paper constructs a multi-agent massive target cooperative search mission planning model and proposes an improved reinforcement learning algorithm using the action preference selection strategy. Based on the Reinforce algorithm, this algorithm solves the problem of invalid searches within the stochastic strategy by changing the preferred action selection method. The proposed method improves the efficiency of multiple agents in the search for targets without collision using a cooperative mechanism and reward rules based on the odor effect. Simulation experiments are conducted in three aspects to verify the effectiveness and robustness of the improved algorithm and compare it with other reinforcement learning algorithms in the field of multi-agent learning. The results demonstrate that the improved algorithm has obvious advantages in terms of mission success rate, target search rate and average search time, and the movement trajectory of multiple agents is more concise.