Concept Of Reinforcement Learning Research Articles

Solving the green reverse logistics problem in e-commerce using a reinforcement learning based genetic algorithm

This study explores the two-phase green reverse logistics problem with time windows and a focus on perishable items that pose a significant challenge in the management of returned goods in e-commerce. We proposed a mixed integer programming model that considers carbon emissions, fuel consumption costs, facility establishment and operating costs, among other factors.We incorporated reinforcement learning concepts to adjust parameters in traditional genetic algorithms, which often have inflexible parameter settings, thereby enhancing both the efficiency and quality of the solutions. The Q-learning algorithm was adopted as the learning method, and various action combinations of reinforcement learning were explored and compared. We further evaluated the performance of different genetic algorithm variations. The results indicate that the proposed algorithm provides high-quality solutions, and that effective parameter configuration significantly impacts the algorithm’s overall performance.

Two-dimensional reinforcement learning model-free fault-tolerant control for batch processes against multi- faults

Aiming at the characteristics of batch process changing along with time and batch directions, the existence of unmodeled dynamics, and the partial failure of actuators or/and sensors, we propose a novel 2D reinforcement learning (RL) fault tolerant control strategy without considering model parameters. Firstly, a two-Dimensional (2D) augmented state space model and 2D Q function-based fault tolerant control (FTC) framework is established. The 2D Bellman equation can be acquired by analyzing the relationship between the 2D value function and the 2D Q function. Based on the extended model and Q-learning concept of RL, a data-driven FTTC independent of model parameters is designed, and a 2D data-driven Q-learning algorithm is proposed. Finally, taking the pressure holding phase in the injection process as the experimental object, the control effect is compared with that of the traditional model-based FTC, and better tracking performance and unbiasedness to the probing noise can be proved.

Teaching approach for deep reinforcement learning of robotic strategies

AbstractThis paper presents the development of a teaching approach for Reinforcement Learning (RL) for students at the Faculty of Electrical Engineering, University of Ljubljana. The approach is designed to introduce students to the basic concepts, approaches, and algorithms of RL through examples and experiments in both simulation environments and on a real robot. The approach includes practical programs written in Python and presents various RL algorithms. The Q‐learning algorithm is introduced and a deep Q network is implemented to introduce the use of neural networks in deep RL. The software is user‐friendly and allows easy modification of learning parameters, reward functions, and algorithms. The approach was tested successfully on a Franka Emika Panda robot, where the robot manipulator learned to move to a randomly generated target position, shoot a real ball into the goal, and push various objects into target position. The goal of the presented teaching approach is to serve as a study aid for future generations of students of robotics to help them better understand the basic concepts of RL and apply them to a wide variety of problems.

Exploring Multi-Armed Bandit (MAB) as an AI Tool for Optimising GMA-WAAM Path Planning

Conventional path-planning strategies for GMA-WAAM may encounter challenges related to geometrical features when printing complex-shaped builds. One alternative to mitigate geometry-related flaws is to use algorithms that optimise trajectory choices—for instance, using heuristics to find the most efficient trajectory. The algorithm can assess several trajectory strategies, such as contour, zigzag, raster, and even space-filling, to search for the best strategy according to the case. However, handling complex geometries by this means poses computational efficiency concerns. This research aimed to explore the potential of machine learning techniques as a solution to increase the computational efficiency of such algorithms. First, reinforcement learning (RL) concepts are introduced and compared with supervised machining learning concepts. The Multi-Armed Bandit (MAB) problem is explained and justified as a choice within the RL techniques. As a case study, a space-filling strategy was chosen to have this machining learning optimisation artifice in its algorithm for GMA-AM printing. Computational and experimental validations were conducted, demonstrating that adding MAB in the algorithm helped to achieve shorter trajectories, using fewer iterations than the original algorithm, potentially reducing printing time. These findings position the RL techniques, particularly MAB, as a promising machining learning solution to address setbacks in the space-filling strategy applied.

Continuous control of structural vibrations using hybrid deep reinforcement learning policy

An efficient active structural control scheme by utilizing the collective concepts of data-driven and physics-inspired reinforcement learning (RL) approaches is being introduced here. The controller is designed to deliver optimal feedback forces based on the full-state information, wherein the training process involves the use of deep neural networks (NNs) and a specially designed gradient descent-based sequence within the RL framework. This integration of algorithms results in a unique active control policy that accelerates the learning process, thereby demands considerably less computational resources to develop an optimal and stable controller as compared to existing data-driven approaches. Most importantly, the mentioned data-driven and physics-inspired approaches encompass deep deterministic policy gradient (DDPG) and an iterative gradient-based state feedback control (SFSC) algorithms, respectively, to establish the architecture of the hybrid control policy, referred to as hybrid RL-controller. Additional advantage of the hybrid RL-controller is its ability to operate in continuous state–action spaces, which allows the designed controller to address diverse structural control problems. The staggered performance of the hybrid RL-controller, both in continuous and discrete time, is evaluated in three case studies that involve structures in linear and nonlinear regimes. The outcomes include a detailed comparison of the hybrid RL-controller with the individually designed DDPG and SFSC RL control strategies, as well as the uncontrolled scenario. Furthermore, this study thoroughly investigates the real-life implementation concerns of control strategies, such as perturbations in model parameters and input forces, and time delays in the feedback loop. Finally, the results from this study corroborate that the designed RL-controller showcases superior performance and faster execution time in the feedback loop, making it suitable for the vibration control of multidimensional complex structures.

Advancements in Reinforcement Learning

Reinforcement Learning (RL) stands at the forefront of machine learning, offering a paradigm where agents learn to navigate complex environments through trial and error interactions. This paper provides a comprehensive overview of RL concepts, algorithms, challenges, and future directions. Key concepts such as agent-environment interaction, Markov Decision Processes (MDPs), value-based and policy-based methods are elucidated. Challenges including the exploration-exploitation trade-off, sample efficiency, and safety concerns are discussed, alongside potential breakthroughs in deep RL, handling continuous action spaces, and dealing with partial observability. The integration of RL with emerging technologies like IoT and its implications for real-world applications such as robotics, autonomous vehicles, and healthcare are explored. Finally, future trends and directions in RL research, including AI engineering and automated feature engineering, are outlined, highlighting the potential for transformative advancements and their impact on various domains. This paper serves as a comprehensive resource for researchers, practitioners, and enthusiasts interested in the evolving landscape of Reinforcement Learning

Research on Multi-agent Sparse Reward Problem

Sparse reward poses a significant challenge in deep reinforcement learning, leading to issues such as low sample utilization, slow agent convergence, and subpar performance of optimal policies. Overcoming these challenges requires tackling the complexity of sparse reward algorithms and addressing the lack of unified understanding. This paper aims to address these issues by introducing the concepts of reinforcement learning and sparse reward, as well as presenting three categories of sparse reward algorithms. Furthermore, the paper conducts an analysis and summary of three key aspects: manual labeling, hierarchical reinforcement learning, and the incorporation of intrinsic rewards. Hierarchical reinforcement learning is further divided into option-based and subgoal-based methods. The implementation principles, advantages, and disadvantages of all algorithms are thoroughly examined. In conclusion, this paper provides a comprehensive review and offers future directions for research in this field.

Self-supervised, active learning seismic full-waveform inversion

A novel recursive, self-supervised machine-learning (ML) inversion scheme is developed. It is applied for fast and accurate full-waveform inversion of land seismic data. ML generalization is enhanced by using virtual super gathers (VSGs) of field data for training. These are obtained from midpoint-offset sorting and stacking after applying surface-consistent corrections from the decomposition of the transmitted wavefield. The procedure implements reinforcement learning concepts by adopting an inversion agent to interact with the environment and explore the model space under a data misfit optimization policy. The generated parameter distributions and related forward responses are used as new training samples for supervised learning. The active learning (AL) paradigm is further embedded in the procedure, for which queries on data diversity and uncertainty are used to generate fully informative reduced sets for training. The procedure is recursive. At each cycle, the physics-based inversion is coupled to the ML predictions via penalty terms that promote a long-term data misfit reduction. The resulting self-supervised, AL, physics-driven deep-learning inversion generalizes well with field data. The method is applied to perform full-waveform inversion (FWI) of a complex land seismic data set characterized by transcurrent faulting and related structures. High signal-to-noise VSGs are inverted with a 1.5D Laplace-Fourier FWI scheme. The AL inversion procedure uses a small fraction of data for training while achieving sharper velocity reconstructions and a lower data misfit when compared with previous results. AL FWI is highly generalizable and effective for land seismic velocity model building and for other inversion scenarios.

An Empirical Study of Different Reinforcement Learning Algorithms for Resource Allocation in Cloud Computing

Regarding the increasing importance of cloud computing in modern IT architecture, it is crucial to create highly efficient resource allocation algorithms. This work conducts an empirical investigation into the utilisation of several reinforcement learning methods for optimising resource allocation in cloud computing environments. Our objective is to assess the efficiency of RL algorithms in a dynamic environment with fluctuating workloads, focusing on resource utilisation, cost effectiveness, and optimality. This research examines the effects of altering cloud settings in order to integrate theoretical reinforcement learning concepts into a practical resource management system. In the literature review, we consider the traditional resource allocation techniques and their inability to accommodate the changing demand. We additionally analyse the available research employing machine learning approaches, paying special attention to RL in cloud computing resource distribution. The methodology describes the research design, detailing the employed RL algorithms Q-learning, Deep Q Networks (DQN), and Proximal Policy Optimization (PPO). We explain the data collection procedure that involves different workloads and also situations to mimic real environments. Performance of each RL algorithm is presented in the experimental results based on the resource utilization, cost efficiency and also system responsiveness. Q-learning, DQN and PPO are being tested which provides a better understanding of their pros and cons. Discussion that follows the Interprets results of these findings bringing to light many challenges along the way as well as possible directions for future inquiry. Therefore, this research fills in the evolving landscape of cloud computing by demonstrating RL algorithms’ adaptability and effectiveness regarding resource allocation challenges under the dynamic environments.

Hidden Reward: Affect and Its Prediction Errors as Windows Into Subjective Value

Scientists increasingly apply concepts from reinforcement learning to affect, but which concepts should apply? And what can their application reveal that we cannot know from directly observable states? An important reinforcement learning concept is the difference between reward expectations and outcomes. Such reward prediction errors have become foundational to research on adaptive behavior in humans, animals, and machines. Owing to historical focus on animal models and observable reward (e.g., food or money), however, relatively little attention has been paid to the fact that humans can additionally report correspondingly expected and experienced affect (e.g., feelings). Reflecting a broader “rise of affectivism,” attention has started to shift, revealing explanatory power of expected and experienced feelings—including prediction errors—above and beyond observable reward. We propose that applying concepts from reinforcement learning to affect holds promise for elucidating subjective value. Simultaneously, we urge scientists to test—rather than inherit—concepts that may not apply directly.

Low carbon dispatch of the park integrated energy system based on the electric vehicles flexible load storage characteristics

The integrated energy system is an efficient way of utilizing energy in industry park. However, with the massive integration of renewable energy and disorganized charging of electric vehicles, the safe operation of this system faces several challenges. To address these issues, we propose a novel dispatch model that incorporates the flexible load characteristics of electric vehicles clusters. Firstly, we elucidate the operational framework for the integrated energy system in parks and establish models for users and microgrid operators incorporating carbon trading mechanisms. These models can effectively portray how an integrated energy system operates within a park setting. Secondly, using charging data from parks, we uncover potential dispatchable charging/discharging capacities for electric vehicles clusters and formulate strategies to utilize electric vehicles as flexible loads in our dispatch operation policy. By appropriately regulating electric vehicles charging/discharging behaviors, demand-supply balance within the system can be better achieved. Subsequently, aiming to maximize benefits for all entities in the park area, we construct a master-slave game model that involves multiple users and microgrid operators. Lastly, employing reinforcement learning concepts, we establish an equivalent power output models for wind turbines, photovoltaic power generation and apply it to an integrated energy system in an industrial park in a specific city. An analysis reveals that our proposed model not only minimizes cost associated with energy storage equipment but also significantly reduces carbon emissions; yielding mutual benefits for both microgrid operators and users.

Introduction of Reinforcement Learning and Its Application Across Different Domain

In the modern era of rapid development in Deep Neural Networks, Reinforcement Learning (RL) has evolved into a pivotal and transformative technology. RL, a learning process where these machine agent interacts with several unknown environment through trial and error. The agent, responsive to the learning machine, go through these interaction, and start receiving feedback in the form of positive rewards or negative rewards like penalties from the environment, and constantly refines its behavior. This research paper offers an in-depth introduction to the foundational concepts of RL, focusing on Markov Decision Processes and various RL algorithms.</p> <p>Machine Learning (ML) is a subset of Artificial Intelligence, which deals with ‘‘the question of how to develop software agents (Machine) that improve automatically with experience’’. The basic three categories of Machine Learning are.</p> <ol> <li>Supervised Learning</li> <li>Unsupervised Learning</li> <li>Reinforcement Learning</li> </ol> <p>RL method is that in any situation the agent has to choose between using its acquired knowledge of the environment i.e. using an action already tried or performed previously or exploring actions never tried before in that situation.</p> <p>In this review paper, we will discuss the most used learning algorithms in games robotics and healthcare, autonomous control as well as communication and networking, natural language processing.[1]</p>

Simulated annealing with reinforcement learning for the set team orienteering problem with time windows

This research investigates the Set Team Orienteering Problem with Time Windows (STOPTW), a new variant of the well-known Team Orienteering Problem with Time Windows and Set Orienteering Problem. In the STOPTW, customers are grouped into clusters. Each cluster is associated with a profit attainable when a customer in the cluster is visited within the customer’s time window. A Mixed Integer Linear Programming model is formulated for STOPTW to maximizing total profit while adhering to time window constraints. Since STOPTW is an NP-hard problem, a Simulated Annealing with Reinforcement Learning (SARL) algorithm is developed. The proposed SARL incorporates the core concepts of reinforcement learning, utilizing the ε-greedy algorithm to learn the fitness values resulting from neighborhood moves. Numerical experiments are conducted to assess the performance of SARL, comparing the results with those obtained by CPLEX and Simulated Annealing (SA). For small instances, both SARL and SA algorithms outperform CPLEX by obtaining eight optimal solutions and 12 better solutions. For large instances, both algorithms obtain better solutions to 28 out of 29 instances within shorter computational times compared to CPLEX. Overall, SARL outperforms SA by resulting in lower gap percentages within the same computational times. Specifically, SARL outperforms SA in solving 13 large STOPTW benchmark instances. Finally, a sensitivity analysis is conducted to derive managerial insights.

An iterative gradient descent-based reinforcement learning policy for active control of structural vibrations

A contemporary policy gradient-based optimization scheme is presented for active structural control by exerting the concept of reinforcement learning (RL). The RL-based control algorithm is demonstrated both in proportional (P) state, and proportional-integral (PI) state-output feedback approaches, wherein, the latter strategy is deliberated based on the theory of servo-mechanism. The search for optimal P and PI controller parameters in a training sequence is attained by engaging an efficient gradient descent-based optimization strategy. The utilization of gradient-based sequence within the RL framework accelerates the learning protocol, ensuring effective dissipation of structural energy, and achieving suboptimal control. The proposed algorithms are validated through numerical experiments on two different structural systems: (i) a quarter car model subjected to periodic road excitation, having a single actuator and presented in continuous time, and (ii) an 8-story building model subjected to random seismic excitation, having multiple actuators and presented in discrete time. Practical implementation concerns of control strategies are thoroughly investigated by considering perturbations in model parameters and input forces. The findings from this study indicate that the RL-based P and PI controllers exhibit high stabilizing performance, and are applicable in both analog and digital domains. Finally, it has affirmed that the proposed RL algorithms exhibit relatively low computational complexity in real-time, and thus, open up a wide range of perspectives for its application in large-scale complex structures.

RL-HAT: A New Framework for Understanding Human-Agent Teaming

This paper presents a novel framework for human-agent teaming grounded in the principles of Reinforcement Learning (RL). Recognizing the need for a unified language across various disciplines, we utilize RL concepts to provide a standard for the understanding and evaluation of diverse teaming strategies. Our framework extends beyond traditional RL constructs, integrating aspects such as belief states, prior knowledge, social considerations, situational awareness, and mental models. A particular focus is placed on the role of ethics and trust in effective teaming. Additionally, we discuss how sensor data, perception models, and actuator modules can be incorporated, emphasizing the adaptability of our framework to a broad range of tasks and environments. We believe this work forms a substantial contribution to the field of human-agent teaming, establishing a solid foundation for future research and application.

Federated Discrete Reinforcement Learning for Automatic Guided Vehicle Control

Under the federated learning paradigm, the agents learn in parallel and combine their knowledge to build a global knowledge model. This new machine learning strategy increases privacy and reduces communication costs, some benefits that can be very useful for industry applications deployed in the edge. Automatic Guided Vehicles (AGVs) can take advantage of this approach since they can be considered intelligent agents, operate in fleets, and are normally managed by a central system that can run in the edge and handles the knowledge of each of them to obtain a global emerging behavioral model. Furthermore, this idea can be combined with the concept of reinforcement learning (RL). This way, the AGVs can interact with the system to learn according to the policy implemented by the RL algorithm in order to follow specified routes, and send their findings to the main system. The centralized system collects this information in a group policy to turn it over to the AGVs. In this work, a novel Federated Discrete Reinforcement Learning (FDRL) approach is implemented to control the trajectories of a fleet of AGVs. Each industrial AGV runs the modules that correspond to an RL system: a state estimator, a rewards calculator, an action selector, and a policy update algorithm. AGVs share their policy variation with the federated server, which combines them into a group policy with a learning aggregation function. To validate the proposal, simulation results of the FDRL control for five hybrid tricycle-differential AGVs and four different trajectories (ellipse, lemniscate, octagon, and a closed 16-polyline) have been obtained and compared with a Proportional Integral Derivative (PID) controller optimized with genetic algorithms. The intelligent control approach shows an average improvement of 78% in mean absolute error, 75% in root mean square error, and 73% in terms of standard deviation. It has been shown that this approach also accelerates the learning up to a 50 % depending on the trajectory, with an average of 36% speed up while allowing precise tracking. The suggested federated-learning based technique outperforms an optimized fuzzy logic controller (FLC) for all of the measured trajectories as well. In addition, different learning aggregation functions have been proposed and evaluated. The influence of the number of vehicles (from 2 to 10) on the path following performance and on network transmission has been analyzed too.

REINFORCEMENT LEARNING EMPOWERED DIGITAL TWINS: PIONEERING SMART CITIES TOWARDS OPTIMAL URBAN DYNAMICS

Smart cities have emerged as a promising solution to address the challenges posed by rapid urbanization and the quest for sustainable urban development. To achieve optimal urban dynamics and enhance the quality of life for citizens, there is a growing need for innovative approaches that integrate cutting-edge technologies. This paper introduces the concept of Reinforcement Learning Empowered Digital Twins as a pioneering strategy for smart cities. By combining the power of digital twin technology with reinforcement learning algorithms, cities can create dynamic, real-time virtual representations that mirror urban systems and interact with the physical world. This integration enables data-driven decision-making, efficient resource management, and optimized traffic flow, ultimately leading to reduced congestion, decreased fuel consumption, and improved air quality. The paper explores the potential applications of reinforcement learning empowered digital twins in various smart city domains, such as intelligent transportation systems, energy management, and urban planning, but mainly with respect to traffic flow and optimisation particularly in the state of Chhattisgarh and its prospects. Moreover, it identifies research gaps and discusses future directions to unlock the full potential of this transformative approach in pioneering smart cities towards optimal urban dynamics. Various scientists have been focusing on this and suggest further investigation and examination as a response and have given their own assessments; this paper essentially discusses, and overviews created by 5 articles, Machine learning approaches for smart city applications: Emergence, challenges and opportunities [1] by Sonam Mehta, Bharat Bhushan & Raghvendra Kumar; Applications of artificial intelligence and machine learning in smart cities [2] by Zaib Ullah, Fadi Al-Turjman, Leonardo Mostarda, Roberto Gagliardi; Enabling cognitive smart cities using big data and machine learning: Approaches and challenges [3] by Mehdi Mohammadi, Ala Al-Fuqaha; A survey on algorithms for intelligent computing and smart city applications [4] by Zhao Tong, Feng Ye, Ming Yan, Hong Liu, Sunitha Basod; The Reversible Lane Network Design Problem (RL-NDP) for Smart Cities with Automated Traffic [5] by L Conceição, GHA Correia, JP Tavares. KEYWORDS: Smart Cities, Digital Twins, Reinforcement Learning, Intelligent Transportation Systems, Urban Dynamics, Traffic Flow Optimization, Energy Consumption Management

A review on reinforcement learning for contact-rich robotic manipulation tasks

Research and application of reinforcement learning in robotics for contact-rich manipulation tasks have exploded in recent years. Its ability to cope with unstructured environments and accomplish hard-to-engineer behaviors has led reinforcement learning agents to be increasingly applied in real-life scenarios. However, there is still a long way ahead for reinforcement learning to become a core element in industrial applications. This paper examines the landscape of reinforcement learning and reviews advances in its application in contact-rich tasks from 2017 to the present. The analysis investigates the main research for the most commonly selected tasks for testing reinforcement learning algorithms in both rigid and deformable object manipulation. Additionally, the trends around reinforcement learning associated with serial manipulators are explored as well as the various technological challenges that this machine learning control technique currently presents. Lastly, based on the state-of-the-art and the commonalities among the studies, a framework relating the main concepts of reinforcement learning in contact-rich manipulation tasks is proposed. The final goal of this review is to support the robotics community in future development of systems commanded by reinforcement learning, discuss the main challenges of this technology and suggest future research directions in the domain.

Particle Swarm Optimization Method Combined with off Policy Reinforcement Learning Algorithm for the Discovery of High Utility Itemset

Mining of High Utility Itemset (HUI) is an area of high importance in data mining that involves numerous methodologies for addressing it effectively. When the diversity of items and size of an item is quite vast in the given dataset, then the problem search space that needs to be solved by conventional exact approaches to High Utility Itemset Mining (HUIM) also increases in terms of exponential. This factual issue has made the researchers to choose alternate yet efficient approaches based on Evolutionary Computation (EC) to solve the HUIM problem. Particle Swarm Optimization (PSO) is an EC-based approach that has drawn the attention of many researchers to unravel different NP-Hard problems in real-time. Variants of PSO techniques have been established in recent years to increase the efficiency of the HUIs mining process. In PSO, the Minimization of execution time and generation of reasonable decent solutions were greatly influenced by the PSO control parameters namely Acceleration Coefficient and and Inertia Weight. The proposed approach is called Adaptive Particle Swarm Optimization using Reinforcement Learning with Off Policy (APSO-RLOFF), which employs the Reinforcement Learning (RL) concept to achieve the adaptive online calibration of PSO control and, in turn, to increase the performance of PSO. The state-of-the-art RL approach called the Q-Learning algorithm is employed in the APSO-RLOFF approach. In RL, state-action utility values are estimated during each episode using Q-Learning. Extensive tests are carried out on four benchmark datasets to evaluate the performance of the suggested technique. An exact approach called HUP-Miner and three EC-based approaches, namely HUPEUMU-GRAM, HUIM-BPSO, and AGA_RLOFF, are used to relate the performance of the anticipated approach. From the outcome, it is inferred that the performance metrics of APSO-RLOFF, namely no of discovered HUIs and execution time, outstrip the previously considered EC computations.

Hosting Capacity Assessment Strategies and Reinforcement Learning Methods for Coordinated Voltage Control in Electricity Distribution Networks: A Review

Increasing connection rates of rooftop photovoltaic (PV) systems to electricity distribution networks has become a major concern for the distribution network service providers (DNSPs) due to the inability of existing network infrastructure to accommodate high levels of PV penetration while maintaining voltage regulation and other operational requirements. The solution to this dilemma is to undertake a hosting capacity (HC) study to identify the maximum penetration limit of rooftop PV generation and take necessary actions to enhance the HC of the network. This paper presents a comprehensive review of two topics: HC assessment strategies and reinforcement learning (RL)-based coordinated voltage control schemes. In this paper, the RL-based coordinated voltage control schemes are identified as a means to enhance the HC of electricity distribution networks. RL-based algorithms have been widely used in many power system applications in recent years due to their precise, efficient and model-free decision-making capabilities. A large portion of this paper is dedicated to reviewing RL concepts and recently published literature on RL-based coordinated voltage control schemes. A non-exhaustive classification of RL algorithms for voltage control is presented and key RL parameters for the voltage control problem are identified. Furthermore, critical challenges and risk factors of adopting RL-based methods for coordinated voltage control are discussed.