SARSA Algorithm Research Articles

Effective machine lifespan management using determined state–action cost estimation for multi-dimensional cost function optimization

ABSTRACT This study introduces a comprehensive framework designed to enhance production efficiency by integrating maintenance strategies, energy costs, and production specifications. This integration is achieved through a novel empirical method for estimating state–action costs, suitable for both machines with measurable and non-measurable states-of-health. We address the challenge of under-determination in state–action cost optimization by employing a k-means clustering approach, ensuring robustness and applicability. Utilizing an adapted SARSA algorithm, our framework optimally controls shop-floor machinery to minimize the global cost function. The efficacy of the state–action cost estimation method is validated using NASA’s C-MAPSS dataset. Additionally, the optimization strategy is further corroborated through its successful implementation in an autonomous mining cart model on the shop floor. Our results highlight the framework’s ability to optimize machine lifetime and production processes effectively, providing tailored solutions that adapt to varying operational conditions without depending on predefined machine degradation models and costs.

Controlling a Single Tank Liquid Level System with Classical Control Methods and Reinforcement Learning Methods

In this study, the control of the single tank liquid level system used in control systems has been carried out. The control of the single tank liquid level system has been performed with the classic PI, modified PI, state feedback with integrator action, and Q learning algorithm and SARSA algorithms, one of the artificial intelligence methods. The tank system to be modelled was carried out using classical physics, namely Newton's laws. Then, the mathematical model obtained of the system that are continuous model in time is acquired. The originality of the study; the non-linear liquid tank system is controlled by classical controllers and reinforcement methods. For this purpose, the system was firstly designed to model the system, then the system has been linearized at a specific point in order to design classic PI, modified PI, and state feedback with integral. After that, agents of the Q Learning algorithm and SARSA algorithms were trained for the system. Then the agents have controlled the single-level tank system. The results of the classic controllers and supervised controllers are contrasted with regard to performance criteria such as rising time, settling time, overshoot and integral square error. Consequently, Q learning method has produced 0.0804-sec rising time, 0.943 sec settling time and 0.574 integral square errors. So, Q learning algorithm has produced and exhibited more thriving and successful results for controlling single liquid tank system than PI, Modified PI, state feedback controllers and SARSA.

An Improved Q-Learning Algorithm for Optimizing Sustainable Remanufacturing Systems

In our modern society, there has been a noticeable increase in pollution due to the trend of post-use handling of items. This necessitates the adoption of recycling and remanufacturing processes, advocating for sustainable resource management. This paper aims to address the issue of disassembly line balancing. Existing disassembly methods largely rely on manual labor, raising concerns regarding safety and sustainability. This paper proposes a human–machine collaborative disassembly approach to enhance safety and optimize resource utilization, aligning with sustainable development goals. A mixed-integer programming model is established, considering various disassembly techniques for hazardous and delicate parts, with the objective of minimizing the total disassembly time. The CPLEX solver is employed to enhance model accuracy. An improvement is made to the Q-learning algorithm in reinforcement learning to tackle the bilateral disassembly line balancing problem in human–machine collaboration. This approach outperforms CPLEX in both solution efficiency and quality, especially for large-scale problems. A comparative analysis with the original Q-learning algorithm and SARSA algorithm validates the superiority of the proposed algorithm in terms of convergence speed and solution quality.

Research on flexible job-shop scheduling problem based on variation-reinforcement learning

The main focus of this paper is to solve the optimization problem of minimizing the maximum completion time in the flexible job-shop scheduling problem. In order to optimize this objective, random sampling is employed to extract a subset of states, and the mutation operator of the genetic algorithm is used to increase the diversity of sample chromosomes. Additionally, 5-tuple are defined as the state space, and a 4-tuple is designed as the action space. A suitable reward function is also developed. To solve the problem, four reinforcement learning algorithms (Double-Q-learning algorithm, Q-learning algorithm, SARS algorithm, and SARSA(λ) algorithm) are utilized. This approach effectively extracts states and avoids the curse of dimensionality problem that occurs when using reinforcement learning algorithms. Finally, experimental results using an international benchmark demonstrate the effectiveness of the proposed solution model.

Research on distributed photovoltaic power prediction method based on reinforcement learning

The distributed photovoltaic power prediction is of great significance for reasonably adjusting and optimizing the power generation plan in the dispatching system. When using machine learning algorithms to construct photovoltaic power prediction models, there are usually problems such as poor prediction accuracy under default hyperparameters and high cost of human experience parameter adjustment experiments. Therefore, a reinforcement learning algorithm based on the long short-term memory network model (SARSA-LSTM) is proposed. Firstly, the reinforcement learning SARSA algorithm is used to tune the hyperparameters of LSTM automatically. Then, the optimal hyperparameter combination is used for regression prediction of photovoltaic power. The experiment compared the model results under default hyperparameters, Bayesian optimization, and grid search optimization hyperparameters. The results showed that the SARSA-LSTM proposed in this paper has better training efficiency and prediction performance compared to other models, which can meet the needs of practical prediction applications.

Integrating meta-heuristics and a Sarsa algorithm for disassembly scheduling problems with cycle time and hazard coefficients

Integrating meta-heuristics and a Sarsa algorithm for disassembly scheduling problems with cycle time and hazard coefficients

Yolo Based Vision-Aided Obstacles Navigation and Avoidance with Path Planning using Sarsa Algorithm for Biped Robot in an Uncertain Hospital Environment

Yolo Based Vision-Aided Obstacles Navigation and Avoidance with Path Planning using Sarsa Algorithm for Biped Robot in an Uncertain Hospital Environment

Correction to: A class integration test order generation approach based on Sarsa algorithm

Correction to: A class integration test order generation approach based on Sarsa algorithm

A Machine Learning Operations (MLOps) Monitoring Model Using BI-LSTM and SARSA Algorithms

A Machine Learning Operations (MLOps) Monitoring Model Using BI-LSTM and SARSA Algorithms

A class integration test order generation approach based on Sarsa algorithm

A class integration test order generation approach based on Sarsa algorithm

Retracted: Cooperation Mode of Soccer Robot Game Based on Improved SARSA Algorithm

Retracted: Cooperation Mode of Soccer Robot Game Based on Improved SARSA Algorithm

Dynamic Regimes for Corporate Human Capital Development Used Reinforcement Learning Methods

Corporate human capital is a critical driver of sustainable economic growth, which is becoming increasingly important in the changing nature of work. Due to the expansion of various areas of human activity, the employee’s profile becomes multifaceted. Therefore, the problem of human capital management based on the individual trajectories of professional development, aimed at increasing the labor efficiency and contributing to the growth of the corporate operational efficiency, is relevant, timely, socially, and economically significant. The paper proposes a methodology for the dynamic regimes for human capital development (DRHC) to design individual trajectories for the employee’s professional development, based on reinforcement learning methods. The DRHC develops an optimal management regime as a set of programs aimed at developing an employee in the professional field, taking into account their individual characteristics (health quality, major and interdisciplinary competencies, motivation, and social capital). The DRHC architecture consists of an environment—an employee model—as a Markov decision-making process and an agent—decision-making center of a company. The DRHC uses DDQN, SARSA, and PRO algorithms to maximize the agent’s utility function. The implementation of the proposed DRHC policy would improve the quality of corporate human capital, increase labor resource efficiency, and ensure the productivity growth of companies.

Reinforcement learning for the traveling salesman problem: Performance comparison of three algorithms

AbstractTravelling salesman problem (TSP) is one of the most famous problems in graph theory, as well as one of the typical nondeterministic polynomial time (NP)‐hard problems in combinatorial optimization. Reinforcement learning (RL) has been widely regarded as an effective tool for solving combinatorial optimization problems. This paper attempts to solve the TSP problem using different reinforcement learning algorithms and evaluated the performance of three RL algorithms (Q‐Learning, SARSA, and Double Q‐Learning) under different reward functions, ε‐greedy decay strategies, and running times. A comprehensive analysis and comparison of the three algorithms mentioned above were conducted in the experiment. First, the experimental results indicate that the Double Q‐Learning algorithm is the best algorithm. Among the eight TSP instances, the Double Q‐Learning algorithm outperforms the other two algorithms in five instances. Additionally, it has shorter runtimes compared to the SARSA algorithm and similar runtimes to the Q‐Learning algorithm across all instances. Second, upon analysing the results, it was found that using the reward strategy contributes to obtaining the best results for all algorithms. Among the 24 combinations of 3 algorithms and 8 instances, 17 combinations achieved the best results when the reward strategy was set to .

Q-learning-based unmanned aerial vehicle path planning with dynamic obstacle avoidance

Recently, unmanned aerial vehicles (UAVs) have shown promising results for autonomous sensing. UAVs have been deployed for multiple applications that include surveillance, mapping, tracking, and search operations. Finding an efficient path between a source and a goal is a critical issue that has been the focus of recent exploration. Many path-planning algorithms are utilized to find an efficient path for a UAV to navigate from a source to a goal with obstacle avoidance. Despite the extensive literature and numerous research proposals for path planning, dynamic obstacle avoidance has not been addressed with machine learning. When the obstacles are dynamic, i.e., they can change their position over time, and the constraints of the path planning algorithm become more challenging. This in turn adds a layer of complexity to the path planning algorithm. To address this challenge, a Q-learning algorithm is proposed in this work to facilitate efficient path planning for UAVs with both static and dynamic obstacle avoidance. We introduced the Shortest Distance Prioritization policy in the learning process which marginally reduces the distance that the UAV has to travel to reach the goal. Further, the proposed Q-learning algorithm adopts a grid-graph-based method to solve the path-planning problem. It learns to maximize the reward based on the agent’s behavior in the environment. Through results, the performance comparison between the proposed approach and state-of-the-art path planning approaches such as A-star, Dijkstra, and Sarsa algorithms are evaluated in terms of learning time and path length. We show through results that the proposed approach results in improved performance when compared to state-of-the-art approaches. Further, the effect of an increased number of obstacles are evaluated on the performance of the proposed approach.

Reinforcement learning for control design of uncertain polytopic systems

This work is concerned with the design of state-feedback, and static output-feedback controllers for uncertain discrete-time systems. The reinforcement learning (RL) method is employed and the controller to be designed is considered as an agent changing the behavior of the plant, which is the environment. A State-Action-Reward-State-Action (SARSA) algorithm is developed to achieve this goal. This is an open problem, as this offline design through the usage of RL is an approach not so well explored in the literature. The gain matrices are used directly as design variables in the SARSA algorithm, and a time-varying incremental step is employed. The method uses a grid in the uncertain parameters to place the poles of the closed-loop system in a disk on the complex plane. In addition, a stability test based on the Lyapunov theory is performed to provide a hard stability certificate for the closed-loop system. Numerical experiments from the literature are used to illustrate the efficacy of the method, through the use of benchmark examples and exhaustive testing.

Safe Reinforcement Learning for Single Train Trajectory Optimization via Shield SARSA

The single train trajectory optimization, also known as speed profile optimization (SPO), is a traditional problem to minimize the traction energy consumption of trains. As a kind of optimal method, reinforcement learning (RL) has been used to solve the SPO problem. In the learning process of a common RL algorithm, a soft constraint (punishment) is always used to keep the agent away from unsafe states. However, a soft constraint can not guarantee and explain the safety of the result. For the SPO problem, it means that the optimized speed profile obtained by a simple RL may break the speed limit which is unacceptable in reality. This paper proposes a protection mechanism called Shield and constructs a Shield SARSA ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${S}$ </tex-math></inline-formula> -SARSA) algorithm to protect the learning process of the high-speed train. Four different reward functions are used to compare the protective efficacy between the proposed algorithm and the soft constraint. The numerical experiments based on the line data from Wuxi East to Suzhou North verify the protective efficacy and effectiveness.

A second-order dynamic and static ship path planning model based on reinforcement learning and heuristic search algorithms

Ship path planning plays an important role in the intelligent decision-making system which can provide important navigation information for ship and coordinate with other ships via wireless networks. However, existing methods still suffer from slow path planning and low security problems. In this paper, we propose a second-order ship path planning model, which consists of two main steps, i.e., first-order static global path planning and second-order dynamic local path planning. Specifically, we first create a raster map using ArcGIS. Second, the global path planning is performed on the raster map based on the Dyna-Sarsa(lambda) model, which integrates the eligibility trace and the Dyna framework on the Sarsa algorithm. Particularly, the eligibility trace has a short-term memory for the trajectory, which can improve the convergence speed of the model. Meanwhile, the Dyna framework obtains simulation experience through simulation training, which can further improve the convergence speed of the model. Then, the improved ship trajectory prediction model based on stacked bidirectional gated recurrent unit is used to identify the risk of ship collision and switch the path planning from the first order to the second order. Finally, the second-order dynamic local path planning is presented based on the FCC-A* algorithm, where the cost function of the traditional path planning A* algorithm is rewritten using the fuzzy collision cost membership function (fuzzy collision cost, FCC) to reduce the collision risk of ships. The proposed model is evaluated on the Baltic Sea geographic information and ship trajectory datasets. The experimental results show that the eligibility trace and the Dyna learning framework in the proposed model can effectively improve the planning efficiency of the ship’s global path planning, and the collision risk membership function can effectively reduce the number of collisions in A* local path planning and thus improve the navigation safety of encountering ships.

Defense scheme against advanced persistent threats in mobile fog computing security

In mobile fog computing (MFC), data processing and applications are concentrated in devices at the network’s edge. This means that data can be processed in local intelligent devices without being sent to the cloud for execution, bringing great convenience to end-users. Fog computing is a semi-virtualised service computing architecture model between cloud computing and personal computing. However, its virtualized environment is vulnerable to advanced persistent threats (APT). In this paper, we propose an APT defense scheme based on MFC’s Double Q-learning (DQL) algorithm. First, we apply prospect theory (PT) to build a static subjective game model between APT attackers and legitimate users. In addition, a dynamic game scheme based on DQL is proposed against APT attacks. Finally, we compare our proposed method with the existing methods, i.e., the Q-learning algorithm, Sarsa algorithm and Greedy algorithm. The experimental results show that the proposed method can effectively suppress the attack motivation of APT attackers, improve the utility of legitimate users, and protect the security of the fog computing environment.

Risk-Aware Travel Path Planning Algorithm Based on Reinforcement Learning during COVID-19

The outbreak of COVID-19 brought great inconvenience to people’s daily travel. In order to provide people with a path planning scheme that takes into account both safety and travel distance, a risk aversion path planning model in urban traffic scenarios was established through reinforcement learning. We have designed a state and action space of agents in a “point-to-point” way. Moreover, we have extracted the road network model and impedance matrix through SUMO simulation, and have designed a Restricted Reinforcement Learning-Artificial Potential Field (RRL-APF) algorithm, which can optimize the Q-table initialization operation before the agent learning and the action selection strategy during learning. The greedy coefficient is dynamically adjusted through the improved greedy strategy. Finally, according to different scenarios, our algorithm is verified by the road network model and epidemic historical data in the surrounding areas of Xinfadi, Beijing, China, and comparisons are made with common Q-Learning, the Sarsa algorithm and the artificial potential field-based reinforcement learning (RLAFP) algorithm. The results indicate that our algorithm improves convergence speed by 35% on average and the travel distance is reduced by 4.3% on average, while avoiding risk areas, compared with the other three algorithms.

Learning-Based Online QoE Optimization in Multi-Agent Video Streaming

Video streaming has become a major usage scenario for the Internet. The growing popularity of new applications, such as 4K and 360-degree videos, mandates that network resources must be carefully apportioned among different users in order to achieve the optimal Quality of Experience (QoE) and fairness objectives. This results in a challenging online optimization problem, as networks grow increasingly complex and the relevant QoE objectives are often nonlinear functions. Recently, data-driven approaches, deep Reinforcement Learning (RL) in particular, have been successfully applied to network optimization problems by modeling them as Markov decision processes. However, existing RL algorithms involving multiple agents fail to address nonlinear objective functions on different agents’ rewards. To this end, we leverage MAPG-finite, a policy gradient algorithm designed for multi-agent learning problems with nonlinear objectives. It allows us to optimize bandwidth distributions among multiple agents and to maximize QoE and fairness objectives on video streaming rewards. Implementing the proposed algorithm, we compare the MAPG-finite strategy with a number of baselines, including static, adaptive, and single-agent learning policies. The numerical results show that MAPG-finite significantly outperforms the baseline strategies with respect to different objective functions and in various settings, including both constant and adaptive bitrate videos. Specifically, our MAPG-finite algorithm maximizes QoE by 15.27% and maximizes fairness by 22.47% compared to the standard SARSA algorithm for a 2000 KB/s link.