Application Of Reinforcement Learning Research Articles

Advancing Additive Manufacturing Through Machine Learning Techniques: A State-of-the-Art Review

In the fourth industrial revolution, artificial intelligence and machine learning (ML) have increasingly been applied to manufacturing, particularly additive manufacturing (AM), to enhance processes and production. This study provides a comprehensive review of the state-of-the-art achievements in this domain, highlighting not only the widely discussed supervised learning but also the emerging applications of semi-supervised learning and reinforcement learning. These advanced ML techniques have recently gained significant attention for their potential to further optimize and automate AM processes. The review aims to offer insights into various ML technologies employed in current research projects and to promote the diverse applications of ML in AM. By exploring the latest advancements and trends, this study seeks to foster a deeper understanding of ML’s transformative role in AM, paving the way for future innovations and improvements in manufacturing practices.

Application of Reinforcement Learning in Controlling Quadrotor UAV Flight Actions

Most literature has extensively discussed reinforcement learning (RL) for controlling rotorcraft drones during flight for traversal tasks. However, most studies lack adequate details regarding the design of reward and punishment mechanisms, and there is a limited exploration of the feasibility of applying reinforcement learning in actual flight control following simulation experiments. Consequently, this study focuses on the exploration of reward and punishment design and state input for RL. The simulation environment is constructed using AirSim and Unreal Engine, with onboard camera footage serving as the state input for reinforcement learning. The research investigates three RL algorithms suitable for discrete action training. The Deep Q Network (DQN), Advantage Actor–Critic (A2C), and Proximal Policy Optimization (PPO) were combined with three different reward and punishment design mechanisms for training and testing. The results indicate that employing the PPO algorithm along with a continuous return method as the reward mechanism allows for effective convergence during the training process, achieving a target traversal rate of 71% in the testing environment. Furthermore, this study proposes integrating the YOLOv7-tiny object detection (OD) system to assess the applicability of reinforcement learning in real-world settings. Unifying the state inputs of simulated and OD environments and replacing the original simulated image inputs with a maximum dual-target approach, the experimental simulation achieved a target traversal rate of 52% ultimately. In summary, this research formulates a set of logical frameworks for an RL reward and punishment design deployed with real-time Yolo’s OD implementation synergized as a useful aid for related RL studies.

A novel reinforcement learning-based method for structure optimization

With the rapid development of deep learning technology, Reinforcement Learning (RL) has garnered considerable acclaim within the realm of structural optimization owing to its excellent exploration mechanism. However, the widespread application of RL in this field is limited owing to the excessive number of iterations required to converge and the expensive computational cost it brings. To address these challenges, this article presents a novel RL framework for structural optimization, combining Monte Carlo tree search with the proximal policy optimization method, called LMPOM. The key contributions of LMPOM encompass: (1) an enhanced Monte Carlo tree search strategy for partitioning the hybrid design space; (2) a strategy for adaptively updating surrogate models to reduce simulation costs; and (3) the introduction of a novel termination condition for the RL algorithms. Through tests on three benchmark problems, compared with previous RL algorithms, LMPOM consistently shows fewer iterations and better optimization results.

Enhancing stability and explainability in reinforcement learning with machine learning

Abstract. In the field of reinforcement learning, training agents using machine learning algorithms to learn and perform tasks in complex environments has become a prevalent approach. However, reinforcement learning faces challenges such as training instability and decision opacity, which limit its feasibility in real-world applications. To solve the problems of stability and transparency in reinforcement learning, this project will use advanced algorithms like Proximal Policy Optimization (PPO), Q-DAGGER, and Gradient Boosting Decision Trees to set up reinforcement learning agents in the OpenAI Gymnasium environment. Specifically, the study selected the Atari game Breakout as the testbed, enhancing training efficiency and game performance by refining reward structures and decision-making processes, and integrating interpretable models to provide explanations for agent decisions. This study has successfully developed robust reinforcement learning agents that excel in complex environments. By employing advanced algorithms like PPO, Q-DAGGER, and Gradient Boosting Decision Trees, the study has addressed issues of training instability, and improved game performance through optimized reward structures and decision processes. Additionally, by integrating interpretable models, the study has provided insights into the learned strategies of the agents, thereby enhancing decision transparency. These findings provide crucial support for the broader application of reinforcement learning in real-world scenarios and offer valuable insights for tackling other complex tasks.

Recovering Permuted Sequential Features for effective Reinforcement Learning.

When applying Reinforcement Learning (RL) to the real-world visual tasks, two major challenges necessitate consideration: sample inefficiency and limited generalization. To address the above two challenges, previous works focus primarily on learning semantic information from the visual state for improving sample efficiency, but they do not explicitly learn other valuable aspects, such as spatial information. Moreover, they improve generalization by learning representations that are invariant to alterations of task-irrelevant variables, without considering task-relevant variables. To enhance sample efficiency and generalization of the base RL algorithm in visual tasks, we propose an auxiliary task called Recovering Permuted Sequential Features (RPSF). Our method enhances generalization by learning the spatial structure information of the agent, which can mitigate the effects of changes in both task-relevant and task-irrelevant variables. Moreover, it explicitly learns both semantic and spatial information from the visual state by disordering and subsequently recovering a sequence of features to generate more holistic representations, thereby improving sample efficiency. Extensive experiments demonstrate that our method significantly improves the sample efficiency and generalization of the base RL algorithm and outperforms various state-of-the-art baselines across diverse tasks in unseen environments. Furthermore, our method exhibits compatibility with both CNN and Transformer architectures.

Risk-averse supply chain management via robust reinforcement learning

Classical reinforcement learning (RL) may suffer performance degradation when the environment deviates from training conditions, limiting its application in risk-averse supply chain management. This work explores using robust RL in supply chain operations to hedge against environment inconsistencies and changes. Two robust RL algorithms, Qˆ-learning and β-pessimistic Q-learning, are examined against conventional Q-learning and a baseline order-up-to inventory policy. Furthermore, this work extends RL applications from forward to closed-loop supply chains. Two case studies are conducted using a supply chain simulator developed with agent-based modeling. The results show that Q-learning can outperform the baseline policy under normal conditions, but notably degrades under environment deviations. By comparison, the robust RL models tend to make more conservative inventory decisions to avoid large shortage penalties. Specifically, fine-tuned β-pessimistic Q-learning can achieve good performance under normal conditions and maintain robustness against moderate environment inconsistencies, making it suitable for risk-averse decision-making.

Research on Bionic Robot Motion Control Based on Reinforcement Learning

This research explores the application of reinforcement learning (RL) to enhance the motion control of bio-inspired robots across various environments. Focusing on underwater, terrestrial, and aerial robotic models, the study integrates model-free Q-learning, Deep Q Network (DQN), State-Action-Reward-State-Action (SARSA), and Double Deep Q Network (DDQN) algorithms. These methods are employed to achieve adaptive and efficient movement strategies without relying on pre-defined environmental models. The methodologies range from real-time data capture using live camera feeds for terrestrial robots to simulating aquatic and flight dynamics in controlled environments. Experimental results confirm the efficacy of these RL methods, demonstrating significant improvements in the robots' ability to adapt to dynamic and unknown environments, optimize movement efficiency, and navigate complex scenarios. The findings suggest a promising direction for future robotic applications, emphasizing the need for further research on optimizing these algorithms and expanding their real-world applicability. This study highlights the potential of RL to revolutionize robotic motion control, making robots more versatile and capable in varied and unpredictable settings.

Locating Values in the Space of Possibilities

Abstract Where do values live in thought? A straightforward answer is that we (or our brains) make decisions using explicit value representations which are our values. Recent work applying reinforcement learning to decision-making and planning suggests that, more specifically, we may represent both the instrumental expected value of actions as well as the intrinsic reward of outcomes. In this paper, I argue that identifying value with either of these representations is incomplete. For agents such as humans and other animals, there is another place where reward can be located in thought: the division of the space of possibilities or “state space.”

Reinforcement learning when your life depends on it: A neuro-economic theory of learning.

Synaptic plasticity enables animals to adapt to their environment, but memory formation can require a substantial amount of metabolic energy, potentially impairing survival. Hence, a neuro-economic dilemma arises whether learning is a profitable investment or not, and the brain must therefore judiciously regulate learning. Indeed, in experiments it was observed that during starvation, Drosophila suppress formation of energy-intensive aversive memories. Here we include energy considerations in a reinforcement learning framework. Simulated flies learned to avoid noxious stimuli through synaptic plasticity in either the energy expensive long-term memory (LTM) pathway, or the decaying anesthesia-resistant memory (ARM) pathway. The objective of the flies is to maximize their lifespan, which is calculated with a hazard function. We find that strategies that switch between the LTM and ARM pathways, based on energy reserve and reward prediction error, prolong lifespan. Our study highlights the significance of energy-regulation of memory pathways and dopaminergic control for adaptive learning and survival. It might also benefit engineering applications of reinforcement learning under resources constraints.

Reinforcement Learning Model-Based and Model-Free Paradigms for Optimal Control Problems in Power Systems: Comprehensive Review and Future Directions

This paper reviews recent works related to applications of reinforcement learning in power system optimal control problems. Based on an extensive analysis of works in the recent literature, we attempt to better understand the gap between reinforcement learning methods that rely on complete or incomplete information about the model dynamics and data-driven reinforcement learning approaches. More specifically we ask how such models change based on the application or the algorithm, what the currently open theoretical and numerical challenges are in each of the leading applications, and which reinforcement-based control strategies will rise in the following years. The reviewed research works are divided into “model-based” methods and “model-free” methods in order to highlight the current developments and trends within each of these two groups. The optimal control problems reviewed are energy markets, grid stability and control, energy management in buildings, electrical vehicles, and energy storage.

Experimental Implementation of Reinforcement Learning Applied to Maximise Energy from a Wave Energy Converter

Wave energy has the potential to provide a sustainable solution for global energy demands, particularly in coastal regions. This study explores the use of reinforcement learning (RL), specifically the Q-learning algorithm, to optimise the energy extraction capabilities of a wave energy converter (WEC) using a single-body point absorber with resistive control. Experimental validation demonstrated that Q-learning effectively optimises the power take-off (PTO) damping coefficient, leading to an energy output that closely aligns with theoretical predictions. The stability observed after approximately 40 episodes highlights the capability of Q-learning for real-time optimisation, even under irregular wave conditions. The results also showed an improvement in efficiency of 12% for the theoretical case and 11.3% for the experimental case from the initial to the optimised state, underscoring the effectiveness of the RL strategy. The simplicity of the resistive control strategy makes it a viable solution for practical engineering applications, reducing the complexity and cost of deployment. This study provides a significant step towards bridging the gap between the theoretical modelling and experimental implementation of RL-based WEC systems, contributing to the advancement of sustainable ocean energy technologies.

A Review of Aircraft Engine Fault Prediction and Performance Optimization Based on Deep Learning

This paper is aimed at presenting a systematic review of the recent studies on the use of deep learning techniques in aircraft engine health management for fault detection and performance enhancement. This comes in the light of increased growth in the air transport industry, the efficiency of aircraft engines and the efficiency of their maintenance have emerged as key issues in the safety and efficiency of air transport. This paper also identifies some of the key issues emerging from this area such as the data issues and the need to develop very accurate models for faulting and performance enhancement. Some of the most advanced techniques, namely the Long Short-Term Memory (LSTM) networks and Convolutional Neural Networks (CNNs) are discussed as potential solutions to these challenges. The theoretical descriptions of the working principles, as well as the strengths and weaknesses of the methods in question with regard to the problems of aircraft engine health management, are elaborated. Furthermore, this paper introduces the application of Reinforcement Learning as a new technique in the area of optimal control of engine parameters with the hope of improving the dependability and maintainability of aircraft engines and in the process of achieving sustainable growth in the aviation industry. The issues on model interpretability, model generalization and model deployment are also discussed and research guidelines for the future work in this field are also outlined. In this paper, the state of the art and the importance of deep learning as well as the applications of deep learning in aircraft engine health management are discussed based on the literature and case studies.

Adaptive Multi-Agent Reinforcement Learning for Optimizing Dynamic Electric Vehicle Charging Networks in Thailand

The rapid growth of electric vehicles (EVs) necessitates efficient management of dynamic EV charging networks to optimize resource utilization and enhance service reliability. This paper explores the application of adaptive multi-agent reinforcement learning (MARL) to address the complexities of EV charging infrastructure in Thailand. By employing MARL, multiple autonomous agents learn to optimize charging strategies based on real-time data by adapting to fluctuating demand and varying electricity prices. Building upon previous research that applied MARL to static network configurations, this study extends the application to dynamic and real-world scenarios, integrating real-time data to refine agent learning processes and also evaluating the effectiveness of adaptive MARL in maximizing rewards and improving operational efficiency compared to traditional methods. Experimental results indicate that MARL-based strategies increased efficiency by 20% and reduced energy costs by 15% relative to conventional algorithms. Key findings demonstrate the potential of extending MARL in transforming EV charging network management, highlighting its benefits for stakeholders, including EV owners, operators, and utility providers. This research contributes insights into advancing electric mobility and energy management in Thailand through innovative AI-driven approaches. The implications of this study include significant improvements in the reliability and cost-effectiveness of EV charging networks, fostering greater adoption of electric vehicles and supporting sustainable energy initiatives. Future research directions include enhancing MARL adaptability and scalability as well as integrating predictive analytics for proactive network optimization and sustainability. These advancements promise to further refine the efficacy of EV charging networks, ensuring that they meet the growing demands of Thailand’s evolving electric mobility landscape.

Application of Reinforcement Learning in Network Security in Dynamical Adjusting Defense Strategies

Application of Reinforcement Learning in Network Security in Dynamical Adjusting Defense Strategies

Intelligent CAD Modeling and Reinforcement Learning Application in Environment and Landscape Design

Intelligent CAD Modeling and Reinforcement Learning Application in Environment and Landscape Design

Customer Behavior Analysis and Reinforcement Learning Application in Tourism Service Design

Customer Behavior Analysis and Reinforcement Learning Application in Tourism Service Design

A review of reinforcement learning for natural language processing and applications in healthcare.

Reinforcement learning (RL) represents a pivotal avenue within natural language processing (NLP), offering a potent mechanism for acquiring optimal strategies in task completion. This literature review studies various NLP applications where RL has demonstrated efficacy, with notable applications in healthcare settings. To systematically explore the applications of RL in NLP, focusing on its effectiveness in acquiring optimal strategies, particularly in healthcare settings, and provide a comprehensive understanding of RL's potential in NLP tasks. Adhering to the PRISMA guidelines, an exhaustive literature review was conducted to identify instances where RL has exhibited success in NLP applications, encompassing dialogue systems, machine translation, question-answering, text summarization, and information extraction. Our methodological approach involves closely examining the technical aspects of RL methodologies employed in these applications, analyzing algorithms, states, rewards, actions, datasets, and encoder-decoder architectures. The review of 93 papers yields insights into RL algorithms, prevalent techniques, emergent trends, and the fusion of RL methods in NLP healthcare applications. It clarifies the strategic approaches employed, datasets utilized, and the dynamic terrain of RL-NLP systems, thereby offering a roadmap for research and development in RL and machine learning techniques in healthcare. The review also addresses ethical concerns to ensure equity, transparency, and accountability in the evolution and application of RL-based NLP technologies, particularly within sensitive domains such as healthcare. The findings underscore the promising role of RL in advancing NLP applications, particularly in healthcare, where its potential to optimize decision-making and enhance patient outcomes is significant. However, the ethical challenges and technical complexities associated with RL demand careful consideration and ongoing research to ensure responsible and effective implementation. By systematically exploring RL's applications in NLP and providing insights into technical analysis, ethical implications, and potential advancements, this review contributes to a deeper understanding of RL's role for language processing.

Unlocking optimal performance and flow level control of three-phase separator based on reinforcement learning: A case study in Basra refinery

This research explores the application of a new reinforcement learning (RL-based) controller for a three-phase separator connected to a gas turbine. The control of flow levels within the separator directly impacts fluid flow turbulence, especially when the equipment is linked to waste heat gas from the turbine to improve gas quality. The study introduces the novel RL-based controller and validates its effectiveness in real-world conditions using three-phase separators in Basra, Iraq, and through a review of relevant literature. The controller can adapt to inlet conditions such as pressure, temperature, mass flow rate, and incoming heat from the gas turbine. Waste heat recovery from the gas can enhance gas purity but also increase turbulence in water and oil. Maintaining a calm flow while ensuring high-speed flow over the baffle in the middle of the separator is crucial for optimal performance. The study considers two geometrical configurations of the vessel for redesigning the separator at the Basra refinery. The controller was implemented using the groovyBC utility within the OpenFOAM software. This model was then utilized to simulate real-world scenarios at the Basra refinery, displaying faster convergence, more rapid response, and more accurate tracking of the target fluid level. This study marks the initial effort to apply the deep deterministic policy gradient (DDPG) controller in computational fluid dynamic (CFD) work. The findings demonstrated a significant enhancement in separation efficiency by more than 36%, as well as smoother streamlines through the control and maintenance of pressure and velocity over the baffle.

Trajectory optimization of spacecraft autonomous far-distance rapid rendezvous based on deep reinforcement learning

This paper investigates the application of Deep Reinforcement Learning (DRL) in the trajectory optimization of spacecraft far-distance rapid rendezvous, and uses the most advanced DRL method Proximal Policy Optimization (PPO) to solve the continuous high-thrust minimum-fuel trajectory optimization problem. The space J2 perturbation was considered, its impact on the spacecraft’s on-orbit operation and trajectory design was analyzed, and the effectiveness and accuracy of the proposed method were verified in two far-distance rapid rendezvous missions. In order to ensure the safety of the subsequent close-range operation phase, a safe area reward framework is proposed, and sparse and dense safe area reward functions are designed. The dense safe area reward function significantly improves the training efficiency of the algorithm on the basis of ensuring terminal performance. In addition, the modeling and analysis of possible uncertainties in the spacecraft’s orbit operation, including observation uncertainty, state uncertainty and control uncertainty, is carried out to verify the performance of the proposed method through simulation. For uncertainties, the closed-loop performance of the policy is also evaluated by performing Monte Carlo simulations. The results show that the PPO algorithm can effectively deal with the rendezvous problem in uncertainty environments. These preliminary results demonstrate the great potential of the DRL method in achieving autonomous guidance of spacecraft.

Privacy preservation in deep reinforcement learning: A training perspective

Reinforcement learning (RL) is a principled AI framework for autonomous, experience-driven learning. Deep reinforcement learning (DRL) enhances this by incorporating deep learning models, promoting a higher-level understanding of the visual world. However, privacy concerns are emerging in RL applications that involve vast amounts of private information. Recent studies have demonstrated that DRL can leak private information and be vulnerable to attacks aiming to infer the training environment from an agent’s behaviors without direct access to the environment. To address these privacy concerns, we propose a differentially private DRL approach that obfuscates the agent’s observations from each visited state. This defends against privacy leakage attacks and prevents the inference of the agent’s training environment from its optimized policy. We provide a theoretical analysis and design comprehensive experiments to thoroughly reproduce the privacy leakage attack. Both the theoretical analysis and experimental results demonstrate that our method effectively defends against privacy leakage attacks while maintaining the model utility of the RL agent.