Implementation Of Reinforcement Learning Research Articles

Machine Learning Methods to Personalize Persuasive Strategies in mHealth Interventions That Promote Physical Activity: Scoping Review and Categorization Overview.

Although physical activity (PA) has positive effects on health and well-being, physical inactivity is a worldwide problem. Mobile health interventions have been shown to be effective in promoting PA. Personalizing persuasive strategies improves intervention success and can be conducted using machine learning (ML). For PA, several studies have addressed personalized persuasive strategies without ML, whereas others have included personalization using ML without focusing on persuasive strategies. An overview of studies discussing ML to personalize persuasive strategies in PA-promoting interventions and corresponding categorizations could be helpful for such interventions to be designed in the future but is still missing. First, we aimed to provide an overview of implemented ML techniques to personalize persuasive strategies in mobile health interventions promoting PA. Moreover, we aimed to present a categorization overview as a starting point for applying ML techniques in this field. A scoping review was conducted based on the framework by Arksey and O'Malley and the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) criteria. Scopus, Web of Science, and PubMed were searched for studies that included ML to personalize persuasive strategies in interventions promoting PA. Papers were screened using the ASReview software. From the included papers, categorized by the research project they belonged to, we extracted data regarding general study information, target group, PA intervention, implemented technology, and study details. On the basis of the analysis of these data, a categorization overview was given. In total, 40 papers belonging to 27 different projects were included. These papers could be categorized in 4 groups based on their dimension of personalization. Then, for each dimension, 1 or 2 persuasive strategy categories were found together with a type of ML. The overview resulted in a categorization consisting of 3 levels: dimension of personalization, persuasive strategy, and type of ML. When personalizing the timing of the messages, most projects implemented reinforcement learning to personalize the timing of reminders and supervised learning (SL) to personalize the timing of feedback, monitoring, and goal-setting messages. Regarding the content of the messages, most projects implemented SL to personalize PA suggestions and feedback or educational messages. For personalizing PA suggestions, SL can be implemented either alone or combined with a recommender system. Finally, reinforcement learning was mostly used to personalize the type of feedback messages. The overview of all implemented persuasive strategies and their corresponding ML methods is insightful for this interdisciplinary field. Moreover, it led to a categorization overview that provides insights into the design and development of personalized persuasive strategies to promote PA. In future papers, the categorization overview might be expanded with additional layers to specify ML methods or additional dimensions of personalization and persuasive strategies.

Implementing Reinforcement Learning in a Financial Decision Support System for Stock Market Investment

Implementing Reinforcement Learning in a Financial Decision Support System for Stock Market Investment

Adaptive Control Strategies for Enhanced Integration of Solar Power in Smart Grids Using Reinforcement Learning

Integrating solar power into smart grids is challenging because of the variable nature of solar energy. This study focuses on implementing reinforcement learning (RL) using a Deep Q-Network algorithm to enhance the stability and efficiency of a grid. A custom environment was designed using OpenAI Gym, in which real-time simulation of grid operations was conducted using real-time data on solar power, weather, and other grid metrics. The trained RL agent exhibited high predictability in optimally distributing the load and managing the battery storage, with R-squared = 0.886, mean average error = 1,173,046.55 Wh, and root mean squared error = 2,075,515.10 Wh. The model effectively captured the seasonality and daily variations in solar power generation. Forecasting using the proposed model provides insights into future energy trends and uncertainties. The reward function will be further refined and scaled for more complex energy systems by incorporating additional variables and hybrid approaches. This study highlights the potential of RL-based adaptive control strategies for developing more efficient and resilient integration of renewable energy sources into smart grids.

Embedded ML-based locomotion control for a 12-joint four-legged robot

Reinforcement learning has developed as a promising approach for robot locomotion control, which can save engineering effort compared to conventional approaches. This article presents the implementation of Reinforcement learning on a low-cost, 12 degree of freedom robot known as a quadruped (spider) to optimize locomotion control, enabling the robot to move on different surfaces such as flat surfaces, ramps, speed bumps, and rough terrain. A MATLAB Simulink model is developed as a digital twin of the spider robot. The dynamics of the model were studied and validated using an open-loop algorithm. Then, the model is utilized in a training simulation environment to apply the reinforcement learning algorithm, showing its ability to move along a predefined path as a replacement for conventional motion control systems. Moreover, the work compares the performance and effectiveness of machine learning-based locomotion control with traditional motion control systems regarding navigation accuracy, speed, and adaptability in challenging environments. The minimum hardware requirements are also studied to move the experiment from simulation to reality.

An Exhaustive Analysis of Reinforcement Learning Implementations within Recommendation System Paradigms

This manuscript begins with a detailed exploration of reinforcement learning, presenting a thorough analysis of its core models and algorithms. It contrasts these with traditional machine learning approaches, providing a nuanced comparative study. Following this, the paper dives into the origins and development of recommendation systems. It methodically classifies and explains the unique features of various prevalent recommendation algorithms. The focus of this research is the innovative integration of reinforcement learning in recommendation systems. It carefully examines how reinforcement learning can be merged with these systems, highlighting current challenges and outlining potential directions for future research, supported by empirical examples from practical applications. This comprehensive approach offers a deep insight into the interplay between advanced machine learning techniques and their application in improving recommendation systems. The manuscript aims to contribute significantly to the field by not only elucidating the theoretical underpinnings of these technologies but also by demonstrating their real-world efficacy and potential. Through this, it seeks to pave the way for more sophisticated, efficient, and personalized recommendation systems, driven by the latest advancements in machine learning.

Drug Dosage Control System Using Reinforcement Learning

This project introduces a pioneering approach for optimizing drug dosage control strategies through the utilization of reinforcement learning (RL), a sophisticated subset of machine learning techniques. The core objective is to dynamically adjust drug dosages in real-time based on patient responses, thereby maximizing therapeutic efficacy while minimizing potential adverse effects. By integrating reinforcement learning algorithms, including Q-learning, Deep Q-Networks (DQN), and actor-critic methods, the system learns from patient data to make precise dosage adjustments considering individual patient characteristics, disease progression, and response to treatment. The framework promises to revolutionize personalized medicine by providing tailored drug dosages, enhancing treatment outcomes, and ensuring patient safety. The project's scope covers not only the development and implementation of this innovative RL- based system but also addresses significant challenges such as model interpretability, scalability, and regulatory compliance, ensuring its practical applicability in healthcare settings. Through this work, we aim to bridge the gap between conventional drug prescription methodologies and the potential for personalized, optimized care, making a substantial contribution to the advancement of healthcare systems.

The application of reinforcement learning algorithms in intelligent games

The ongoing advancement of computer hardware has spurred the need for high-quality gaming experiences. The pressing need for intelligent games, the proven efficacy of reinforcement learning in various fields, the advancement of deep reinforcement learning technology, and the desire for customized gaming experiences collectively propel the ongoing exploration and implementation of reinforcement learning in intelligent games.This study use algorithms sourced from the literature review to examine multiple facets of reinforcement learning, including the modeling of game settings, reward design, the utilization of deep learning approaches, and adversarial learning. The utilization of reinforcement learning in intelligent games holds immense importance. Initially, it enhances the gaming encounter and renders intelligent games more demanding and tailored. Furthermore, the game gains complexity and intrigue by the implementation of intelligent adversaries capable of assimilating and adjusting to the player's tactics. This not only fosters the advancement of AI technology through research and development, but also broadens the utilization of reinforcement learning in other domains, like autonomous driving and smart manufacturing. Furthermore, tailored encounters, cutting-edge game design, and utilization in schooling are likewise significant contributions, rendering reinforcement learning a pivotal element in propelling technological advancement and augmenting user experience.

Self-supervised, active learning seismic full-waveform inversion

A novel recursive, self-supervised machine-learning (ML) inversion scheme is developed. It is applied for fast and accurate full-waveform inversion of land seismic data. ML generalization is enhanced by using virtual super gathers (VSGs) of field data for training. These are obtained from midpoint-offset sorting and stacking after applying surface-consistent corrections from the decomposition of the transmitted wavefield. The procedure implements reinforcement learning concepts by adopting an inversion agent to interact with the environment and explore the model space under a data misfit optimization policy. The generated parameter distributions and related forward responses are used as new training samples for supervised learning. The active learning (AL) paradigm is further embedded in the procedure, for which queries on data diversity and uncertainty are used to generate fully informative reduced sets for training. The procedure is recursive. At each cycle, the physics-based inversion is coupled to the ML predictions via penalty terms that promote a long-term data misfit reduction. The resulting self-supervised, AL, physics-driven deep-learning inversion generalizes well with field data. The method is applied to perform full-waveform inversion (FWI) of a complex land seismic data set characterized by transcurrent faulting and related structures. High signal-to-noise VSGs are inverted with a 1.5D Laplace-Fourier FWI scheme. The AL inversion procedure uses a small fraction of data for training while achieving sharper velocity reconstructions and a lower data misfit when compared with previous results. AL FWI is highly generalizable and effective for land seismic velocity model building and for other inversion scenarios.

Deep reinforcement learning implementation on IC engine idle speed control

Efficient control of automotive engine idle speed is crucial for achieving better fuel economy and smoother engine running. This paper presents a comparison between proportional-integral-derivative (PID) control and Reinforcement Learning (RL) using the Deep Q-Network (DQN) algorithm as a high-level control method for minimizing idle speed fluctuations caused by changes in engine irregularities, and the response time and accuracy of the throttle control mechanism. In addition to low-level PID control for the throttle valve position, MATLAB/Simulink was employed to build the simulation environment, incorporating an engine model and an electronic throttle body model, and observing the engine's current speed. The results demonstrated the superiority of RL-based control over PID in reducing idle speed fluctuations and enhancing engine performance in simulations and real-world experiments. This study advances automotive engine control strategies.

Efficient Reinforcement Learning for Real-Time Hardware-Based Energy System Experiments

In the context of urgent climate challenges and the pressing need for rapid technology development, Reinforcement Learning (RL) stands as a compelling data-driven method for controlling real-world physical systems. However, RL implementation often entails time-consuming and computationally intensive data collection and training processes, rendering them inefficient for real-time applications that lack non-real-time models. To address these limitations, real-time emulation techniques have emerged as valuable tools for the lab-scale rapid prototyping of intricate energy systems. While emulated systems offer a bridge between simulation and reality, they too face constraints, hindering comprehensive characterization, testing, and development. In this research, we construct a surrogate model using limited data from simulated systems, enabling an efficient and effective training process for a Double Deep Q-Network (DDQN) agent for future deployment. Our approach is illustrated through a hydropower application, demonstrating the practical impact of our approach on climate-related technology development.

Redesign of Leg Assembly and Implementation of Reinforcement Learning for a Multi-Purpose Rehabilitation Robotic Device (RoboREHAB)

Patients who are suffering from neuromuscular disorders or injuries that impair motor control need to undergo rehabilitation to regain mobility. Gait training is commonly prescribed to patients to regain muscle memory. Automated-walking training devices were created to aid this process; while these devices establish accurate ankle-path trajectories, the knee and hip movements are inaccurate. In this work, a redesign of the leg assembly in a multi-purpose rehabilitation robotic device (RoboREHAB) was explored to improve hip- and knee-movement accuracy by adding an extra link and rollers to the assembly. Motion analysis was employed to test feasibility, reinforcement learning was utilized to train the new leg assembly to walk, and the joint motions achieved with the redesign were compared to those achieved by motion-capture (mocap) data. As a key result, the motion analysis showed an improvement in the knee- and hip-path trajectories due to the added roller/joint segment. The redesigned leg assembly, under the reinforcement-learning policy, showed a 5% deviation from the motion-capture joint trajectories with a maximum deviation of 51.177 mm but maintained a similar profile to the mocap trajectory data. This is an improvement over the original two-segment design, which achieved a maximum deviation of 72.084 mm. These results in the knee- and hip-joint movements more closely reflect the mocap and motion-analysis results, validating the redesign and opening it up to further experimentation and technical improvement.

A spiking binary neuron — detector of causal links

Purpose. Causal relationship recognition is a fundamental operation in neural networks aimed at learning behavior, action planning, and inferring external world dynamics. This operation is particularly crucial for reinforcement learning (RL). In the context of spiking neural networks (SNNs), events are represented as spikes emitted by network neurons or input nodes. Detecting causal relationships within these events is essential for effective RL implementation. Methods. This research paper presents a novel approach to realize causal relationship recognition using a simple spiking binary neuron. The proposed method leverages specially designed synaptic plasticity rules, which are both straightforward and efficient. Notably, our approach accounts for the temporal aspects of detected causal links and accommodates the representation of spiking signals as single spikes or tight spike sequences (bursts), as observed in biological brains. Furthermore, this study places a strong emphasis on the hardware-friendliness of the proposed models, ensuring their efficient implementation on modern and future neuroprocessors. Results. Being compared with precise machine learning techniques, such as decision tree algorithms and convolutional neural networks, our neuron demonstrates satisfactory accuracy despite its simplicity. Conclusion. We introduce a multi-neuron structure capable of operating in more complex environments with enhanced accuracy, making it a promising candidate for the advancement of RL applications in SNNs.

Training reinforcement learning-based controller using performance simulation of the laser remelting process

Surface topography modification is a common post-processing step used to enhance surface quality or to add micro/nano surface structures for various functionalities. Laser remelting (LRM) is a new technology aimed to resolve some of the shortcomings of traditional post-processing technologies. In LRM, a high-energy laser beam melts the top surface and offers control of the molten material flow through the control of laser power, scanning speed, melt pool size, and scan trajectory. By controlling the molten material flow, surface polishing by laser remelting (SP-LRM) can be used to produce finishes characterized by areal arithmetical mean height - Sa < 0.1 µm as well as surface microstructures. LRM is faster than traditional methods, creates no harmful byproducts, and allows for polishing high aspect ratio freeform surfaces without harming surrounding areas. The primary limiting factor for wide-rage industrial use of the LRM is represented by thermodynamic process instabilities that are associated with rapid melting and solidification. For instance, deep trenches or grinding marks in the initial surface can cause small deviations of local surface absorptivity to lead to laser melt pool instabilities. The high-speed nature of the LRM process combined with the complex thermodynamic phenomena makes process instabilities difficult to model, detect, predict, and control. This study aimed to use reinforcement learning (RL) in order to build a ‘self-learning’ controller capable of adapting to any material and control instabilities on-line and in real-time. Along these lines, the study will detail a preliminary application of two RL implementations involving SP-LRM performance simulation to demonstrate the implementation of a double dueling deep Q-network (DDDQN) controller for melt pool temperature distribution control. The results obtained showed that the 3-action implementation achieved 48% less variance and 35% increase in the mean value of the total reward collected compared with the 31-action implementation.

Exploring the power of heterogeneous UAV swarms through reinforcement learning

The object of research is heterogeneous and homogeneous swarms of unmanned aerial vehicles (UAVs). The primary focus of this study is the comparison between heterogeneous and homogeneous UAV swarms, examining their performance in a simulated environment designed using the Python Gym library. The research involves implementing reinforcement learning algorithms, specifically the Proximal Policy Optimization (PPO), to train and evaluate the swarms. The central issue addressed by this research is to determine which type of UAV swarm – heterogeneous or homogeneous – exhibits better performance in a defined task. The chosen task involves searching for groups of objects in an unknown area, emphasizing the ability of the swarm to adapt and efficiently locate objects in dynamic environments. The obtained results reveal an advantage for heterogeneous UAV swarms over their homogeneous counterparts. The heterogeneous swarm has a steeper learning curve and achieves higher rewards in fewer episodes during the training phase. The key finding indicates that the varied skill set within the heterogeneous swarm allows for quicker adaptation to changing environmental conditions. The superior performance of the heterogeneous swarm is attributed to the diversity of capabilities among its UAV agents, enabling them to leverage their individual strengths to achieve better overall performance in the given task. The practical application of these results is contingent upon the task requirements and environmental conditions. In scenarios where tasks demand diverse skills and adaptability to changing conditions, heterogeneous UAV swarms are recommended. The results suggest their efficacy in applications such as search and rescue operations, environmental monitoring, and other dynamic tasks. In conclusion, this research provides valuable insights into optimizing UAV swarm composition for specific tasks. The results contribute both theoretically and practically by highlighting the advantages of heterogeneity in swarm capabilities.

A Novel Approach for Target Attraction and Obstacle Avoidance of a Mobile Robot in Unknown Environments Using a Customized Spiking Neural Network

In recent years, implementing reinforcement learning in autonomous mobile robots (AMRs) has become challenging. Traditional methods face complex trials, long convergence times, and high computational requirements. This paper introduces an innovative strategy using a customized spiking neural network (SNN) for autonomous learning and control of mobile robots (AMR) in unknown environments. The model combines spike-timing-dependent plasticity (STDP) with dopamine modulation for learning. It utilizes the Izhikevich neuron model, leading to biologically inspired and computationally efficient control systems that adapt to changing environments. The performance of the model is evaluated in a simulated environment, replicating real-world scenarios with obstacles. In the initial training phase, the model faces significant challenges. Integrating brain-inspired learning, dopamine, and the Izhikevich neuron model adds complexity. The model achieves an accuracy rate of 33% in reaching its target during this phase. Collisions with obstacles occur 67% of the time, indicating the struggle of the model to adapt to complex obstacles. However, the model’s performance improves as the study progresses to the testing phase after the robot has learned. Its accuracy surges to 94% when reaching the target, and collisions with obstacles reduce it to 6%. This shift demonstrates the adaptability and problem-solving capabilities of the model in the simulated environment, making it more competent for real-world applications.

Review and Evaluation of Reinforcement Learning Frameworks on Smart Grid Applications

With the rise in electricity, gas and oil prices and the persistently high levels of carbon emissions, there is an increasing demand for effective energy management in energy systems, including electrical grids. Recent literature exhibits large potential for optimizing the behavior of such systems towards energy performance, reducing peak loads and exploiting environmentally friendly ways for energy production. However, the primary challenge relies on the optimization of such systems, which introduces significant complexities since they present quite dynamic behavior. Such cyberphysical frameworks usually integrate multiple interconnected components such as power plants, transmission lines, distribution networks and various types of energy-storage systems, while the behavior of these components is affected by various external factors such as user individual requirements, weather conditions, energy demand and market prices. Consequently, traditional optimal control approaches—such as Rule-Based Control (RBC)—prove inadequate to deal with the diverse dynamics which define the behavior of such complicated frameworks. Moreover, even sophisticated techniques—such as Model Predictive Control (MPC)—showcase model-related limitations that hinder the applicability of an optimal control scheme. To this end, AI model-free techniques such as Reinforcement Learning (RL) offer a fruitful potential for embedding efficient optimal control in cases of energy systems. Recent studies present promising results in various fields of engineering, indicating that RL frameworks may prove the key element for delivering efficient optimal control in smart buildings, electric vehicle charging and smart grid applications. The current paper provides a comprehensive review of RL implementations in energy systems frameworks—such as Renewable Energy Sources (RESs), Building Energy-Management Systems (BEMSs) and Electric Vehicle Charging Stations (EVCSs)—illustrating the benefits and the opportunities of such approaches. The work examines more than 80 highly cited papers focusing on recent RL research applications—between 2015 and 2023—and analyzes the model-free RL potential as regards the energy systems’ control optimization in the future.

Trajectory Planning With Deep Reinforcement Learning in High-Level Action Spaces

This paper presents a technique for trajectory planning based on parameterized high-level actions. These high-level actions are sub trajectories that have variable shape and duration. The use of high-level actions can improve the performance of guidance algorithms. Specifically, we show how the use of high-level actions improves the performance of guidance policies that are generated via reinforcement learning (RL). Reinforcement learning has shown great promise for solving complex control, guidance, and coordination problems but can still suffer from long training times and poor performance. This work shows how the use of high-level actions reduces the required number of training steps and increases the path performance of an RL-trained guidance policy. We demonstrate the method on a space-shuttle guidance example. We show the proposed method increases the path performance (latitude range) by 18% percent compared to a baseline RL implementation. Similarly, we show the proposed method achieves steady state during training with approximately 75% fewer training steps. We also show how the guidance policy enables effective performance in an obstacle field. Finally, this paper develops a loss function term for policy-gradient-based Deep RL, which is analogous to an anti-windup mechanism in feedback control. We demonstrate that the inclusion of this term in the underlying optimization increases the average policy return in our numerical example.

Deep Learning Model to Predict Ice Crystal Growth.

The demand for highly specific and complex materials has made the development of controllable manufacturing processes crucial. Among the numerous manufacturing methods, casting is important because it is economical and highly flexible regarding the geometry of manufactured parts. Since solidification is an important stage in the casting process that influences the properties of the final product, the development of a controllable solidification process using modeling methods is necessary to create superior structural properties. However, traditional modeling methods are computationally expensive and require sophisticated mathematical schemes. Therefore, a deep learning model is proposed to predict the morphology of the dendritic crystal growth solidification process, along with a reinforcement learning model to control the solidification process. By training the deep learning model with data generated using the phase field method, the solidification process can be successfully predicted. The crystal growth structures are designed to be altered by adjusting the degree of supercooling in the deep learning model while implementing reinforcement learning to control the dendritic arteries. This research opens new avenues for applying artificial intelligence to the optimization of casting processes, with the potential to utilize it in the processing of advanced materials and to improve the target properties of material design.

Deep Robot Sketching: An application of Deep Q-Learning Networks for human-like sketching

The current success of Reinforcement Learning algorithms for its performance in complex environments has inspired many recent theoretical approaches to cognitive science. Artistic environments are studied within the cognitive science community as rich, natural, multi-sensory, multi-cultural environments. In this work, we propose the introduction of Reinforcement Learning for improving the control of artistic robot applications. Deep Q-learning Neural Networks (DQN) is one of the most successful algorithms for the implementation of Reinforcement Learning in robotics. DQN methods generate complex control policies for the execution of complex robot applications in a wide set of environments. Current art painting robot applications use simple control laws that limits the adaptability of the frameworks to a set of simple environments. In this work, the introduction of DQN within an art painting robot application is proposed. The goal is to study how the introduction of a complex control policy impacts the performance of a basic art painting robot application. The main expected contribution of this work is to serve as a first baseline for future works introducing DQN methods for complex art painting robot frameworks. Experiments consist of real world executions of human drawn sketches using the DQN generated policy and TEO, the humanoid robot. Results are compared in terms of similarity and obtained reward with respect to the reference inputs.

Towards Sustainable Energy Systems Considering Unexpected Sports Event Management: Integrating Machine Learning and Optimization Algorithms

This paper proposes a novel approach for achieving sustainable energy systems in unexpected sports event management by integrating machine learning and optimization algorithms. Specifically, we used reinforcement learning for peak load forecasting and bat evolutionary algorithm for optimization, since the energy management problem in sports events is typically non-linear. Machine learning algorithms, specifically reinforcement learning, are used to analyze historical data and provide accurate peak load forecasts. This information can then be used to optimize energy consumption during the event through the use of algorithms such as the bat evolutionary algorithm, which can effectively solve non-linear optimization problems. The integration of these algorithms in unexpected sports event management can lead to significant improvements in sustainability and cost-effectiveness. This paper presents a case study of the implementation of reinforcement learning and bat evolutionary algorithms in an unexpected sports event management scenario, demonstrating the effectiveness of the proposed approach in achieving sustainable energy systems and reducing overall energy consumption. Overall, this paper provides a roadmap for integrating machine learning and optimization algorithms, such as reinforcement learning and bat evolutionary algorithm, in unexpected sports event management to achieve sustainable energy systems, promoting a more sustainable future for the sports event industry and the planet as a whole.