Fuzzy Reinforcement Learning Research Articles

Fuzzy Reinforcement Learning and Curriculum Transfer Learning for Micromanagement in Multi-Robot Confrontation

Multi-Robot Confrontation on physics-based simulators is a complex and time-consuming task, but simulators are required to evaluate the performance of the advanced algorithms. Recently, a few advanced algorithms have been able to produce considerably complex levels in the context of the robot confrontation system when the agents are facing multiple opponents. Meanwhile, the current confrontation decision-making system suffers from difficulties in optimization and generalization. In this paper, a fuzzy reinforcement learning (RL) and the curriculum transfer learning are applied to the micromanagement for robot confrontation system. Firstly, an improved Qlearning in the semi-Markov decision-making process is designed to train the agent and an efficient RL model is defined to avoid the curse of dimensionality. Secondly, a multi-agent RL algorithm with parameter sharing is proposed to train the agents. We use a neural network with adaptive momentum acceleration as a function approximator to estimate the state-action function. Then, a method of fuzzy logic is used to regulate the learning rate of RL. Thirdly, a curriculum transfer learning method is used to extend the RL model to more difficult scenarios, which ensures the generalization of the decision-making system. The experimental results show that the proposed method is effective.

Research on Fuzzy Enhanced Learning Model of Multienhanced Signal Learning Automata

With the continuous iteration of information technology, information presents an explosive development trend, so how to effectively classify, retrieve, and summarize information is very important. Data classification and mining technology as the key technology to solve the aforementioned problems is of great significance. At present, in the fields of data processing, information classification and machine learning reinforcement learning, it is usually based on the learning automata, but the traditional learning automata algorithm cannot meet the development of information technology in terms of convergence and learning rate. In order to improve the convergence stability and learning efficiency of the learning automata algorithm in random environment and enhance its learning ability in dealing with large-scale space and action, this paper innovatively proposes an adaptive fuzzy reinforcement learning model of the learning automata, which makes full use of the randomness of gradient information to solve its stability problem, and abandons the original reinforcement learning method. For the linear function, an adaptive fuzzy numerical superposition algorithm is used to further enhance its learning rate and improve its stability. The experimental results show that the proposed learning automata fuzzy reinforcement learning model has practical application value, and can solve the problems of unstable convergence and poor learning efficiency of the current learning automata.

Adaptive Fuzzy Fault-Tolerant Tracking Control for Partially Unknown Systems With Actuator Faults via Integral Reinforcement Learning Method

In this paper, a fuzzy reinforcement learning (RL)-based tracking control algorithm is first proposed for partially unknown systems with actuator faults. Based on Takagi–Sugeno fuzzy model, a novel fuzzy-augmented tracking dynamic is developed and the overall fuzzy control policy with corresponding performance index is designed, where four kinds of actuator faults, including actuator loss of effectiveness and bias fault, are considered. Combining the RL technique and fuzzy-augmented model, the new fuzzy integral RL-based fault-tolerant control algorithm is designed, and it runs in real time for the system with actuator faults. The dynamic matrices can be partially unknown and the online algorithm requires less information transmissions or computational load along with the learning process. Under the overall fuzzy fault-tolerant policy, the tracking objective is achieved and the stability is proven by Lyapunov theory. Finally, the applications in the single-link robot arm system and the complex pitch-rate control problem of F-16 fighter aircraft demonstrate the effectiveness of the proposed method.

Genetic Algorithm-Optimized Fuzzy Lyapunov Reinforcement Learning for Nonlinear Systems

This paper aims to introduce nonlinear optimization in the fuzzy reinforcement learning (RL) approach through genetic algorithm (GA)-based minimization. In conventional fuzzy RL, an agent attempts to find most optimal action at each stage by choosing an action having the lowest Q value or the greedy action. However, Q function is an unknown function and an attempt to find minima of such a function based on a limited set of values, in our view, is inaccurate and insufficient. A more rigorous approach would be to employ a nonlinear optimization procedure for finding minima of the Q function. We propose to employ genetic algorithm for finding optimal action value in each iteration of the algorithm rather than plain algebraic minimum. For guaranteed stability of the designed controller, we use Lyapunov theory-based fuzzy RL control with GA optimizer. We validate the performance of our controller on three benchmark nonlinear NL control problems: (1) inverted pendulum swing up, (2) cart pole balance, and (3) rotational/translational proof-mass actuator system. We carry out comparative evaluation of our controller against: (1) hybrid Lyapunov fuzzy RL control and (2) fuzzy Q learning control. Results show that our proposed GA-optimized fuzzy Lyapunov RL controller is able to achieve a high success rate with stable and superior tracking performance.

Developing Train Station Parking Algorithms: New Frameworks Based on Fuzzy Reinforcement Learning

Train station parking (TSP) accuracy is important to enhance the efficiency of train operation and the safety of passengers for urban rail transit. However, TSP is always subject to a series of uncertain factors such as extreme weather and uncertain conditions of rail track resistances. To increase the parking accuracy, robustness, and self-learning ability, we propose new train station parking frameworks by using the reinforcement learning (RL) theory combined with the information of balises. Three algorithms were developed, involving a stochastic optimal selection algorithm (SOSA), a Q-learning algorithm (QLA), and a fuzzy function based Q-learning algorithm (FQLA) in order to reduce the parking error in urban rail transit. Meanwhile, five braking rates are adopted as the action vector of the three algorithms and some statistical indices are developed to evaluate parking errors. Simulation results based on real-world data show that the parking errors of the three algorithms are all within the ±30cm, which meet the requirement of urban rail transit.

Online Tuning of a PID Controller with a Fuzzy Reinforcement Learning MAS for Flow Rate Control of a Desalination Unit

This paper proposes a hybrid Zeigler-Nichols (Z-N) fuzzy reinforcement learning MAS (Multi-Agent System) approach for online tuning of a Proportional Integral Derivative (PID) controller in order to control the flow rate of a desalination unit. The PID gains are set by the Z-N method and then are adapted online through the fuzzy Q-learning MAS. The fuzzy Q-learning is introduced in each agent in order to confront with the continuous state-action space. The global state of the MAS is defined by the value of the error and the derivative of error. The MAS consists of three agents and the output signal of each agent defines the percentage change of each gain. The increment or the reduction of each gain can be in the range of 0% to 100% of its initial value. The simulation results highlight the performance of the suggested hybrid control strategy through comparison with the conventional PID controller tuned by Z-N.

Experimental evaluation of new navigator of mobile robot using fuzzy Q-learning

In this paper, we propose an approach of fusing the fuzzy control actions of the obstacle avoidance and goal-seeking which utilises fuzzy logic and reinforcement learning for navigation of a mobile robot in unknown environments. The proposed reactive navigator consists of three modules: move to goal, obstacle avoidance, and fuzzy behaviour supervisor. The selection of the actions available in each fuzzy rule is learned through reinforcement learning (Q-learning algorithm). A new and powerful method is used to construct automatically these rules. The experiments carried out on the Pioneer 2P mobile robot have shown that the navigator is able to perform a successful navigation task in various unknown environments with smooth action and exceptionally good robustness.

Fuzzy Reinforcement Learning for Robust Spectrum Access in Dynamic Shared Networks

The persistent increases of wireless terminals have brought about diverse shared networks, where robust and efficient spectrum reuse among heterogeneous users is of critical importance while still remains as a challenging task for practical application. In this paper, we study the problem of robust spectrum access (RSA) in a canonical wireless shared network (WSN) with fully considering the inherent dynamics of the wireless environment. The non-static features of WSNs result in uncertain channel state information (CSI) and complicated coupling interference, which can’t be directly formulated as the well-accepted crisp game model, rendering most existing perfect CSI relied approaches inefficient or even unfeasible. To address this, by interpreting the estimated CSI with uncertainty as fuzzy number, a novel framework referred to as a non-cooperative fuzzy game (NC-FG) is adopted, whereby the user utility is mapped as a fuzzy value via the user-defined fuzzy utility function. Based on the derived property of the NC-FG that fuzzy Nash equilibrium (FNE) exists, a fuzzy-logic inspired reinforcement learning (FLRL) algorithm is proposed to achieve the FNE solutions of the constructed NC-FG to obtain the RSA in dynamic WSN, with which both the iterative learning and decision making procedures are implemented in a fuzzy-space, thus the sensitiveness of our scheme to environmental variations is alleviated. Finally, numerical simulations are provided to demonstrate the convergence, effectiveness, and superiority of our proposed FLRL algorithm in dynamic WSNs.

A PSO-Optimized Fuzzy Reinforcement Learning Method for Making the Minimally Invasive Surgical Arm Cleverer

For robotic-assisted surgery, the preoperative preparation procedure within which the surgical arms need to be adjusted manually to their expected configuration can pose interlocking problems in terms of accuracy, system robustness, and human-robot interaction experience. Previous work about variable impedance/admittance control methods did improve the smoothness of this procedure while individual characteristics of different operating personnel haven't yet been adequately considered. To further improve this process on the basis of existing methods, a novel strategy based on fuzzy Sarsa(λ)-learning algorithm is both proposed and incorporated into the virtual parameter adjustment strategy to achieve physical human-robot interaction (pHRI) so that the robot can acclimatize various handling characteristics of different operators through enough online learning. To also shorten the online training cycle, reduce the undesirable subjective factors and improve the overall training performance, a particle-swarm-optimization-based (PSO-based) algorithm is as well employed for optimizing the partition of state variable space and the distribution of discrete actions. Several groups of experiments have demonstrated the validity of this scheme and the effectiveness of the PSO optimization element.

Solving nonconvex economic thermal power dispatch problem with multiple fuel system and valve point loading effect using fuzzy reinforcement learning

Solving nonconvex economic thermal power dispatch problem with multiple fuel system and valve point loading effect using fuzzy reinforcement learning

Particle swarm optimization for generating interpretable fuzzy reinforcement learning policies

Fuzzy controllers are efficient and interpretable system controllers for continuous state and action spaces. To date, such controllers have been constructed manually or trained automatically either using expert-generated problem-specific cost functions or incorporating detailed knowledge about the optimal control strategy. Both requirements for automatic training processes are not found in most real-world reinforcement learning (RL) problems. In such applications, online learning is often prohibited for safety reasons because it requires exploration of the problem’s dynamics during policy training. We introduce a fuzzy particle swarm reinforcement learning (FPSRL) approach that can construct fuzzy RL policies solely by training parameters on world models that simulate real system dynamics. These world models are created by employing an autonomous machine learning technique that uses previously generated transition samples of a real system. To the best of our knowledge, this approach is the first to relate self-organizing fuzzy controllers to model-based batch RL. FPSRL is intended to solve problems in domains where online learning is prohibited, system dynamics are relatively easy to model from previously generated default policy transition samples, and it is expected that a relatively easily interpretable control policy exists. The efficiency of the proposed approach with problems from such domains is demonstrated using three standard RL benchmarks, i.e., mountain car, cart-pole balancing, and cart-pole swing-up. Our experimental results demonstrate high-performing, interpretable fuzzy policies.

A fuzzy reinforcement learning approach to thermal unit commitment problem

Unit commitment problem (UCP) aims at optimizing generation cost for meeting a given load demand under several operational constraints. We propose to use fuzzy reinforcement learning (RL) approach for efficient and reliable solution to the unit commitment problem. In particular, we cast UCP as a multiagent fuzzy reinforcement learning task wherein individual generators act as players for optimizing the cost to meet a given load over a twenty-four-hour period. Unit commitment task has been fuzzified, and the most optimal unit commitment solution is generated by employing RL on this fuzzy multigenerator setup. Our proposed multiagent RL framework does not assume any a priori task or system knowledge, and the generators gradually learn to produce most optimal output solely based on their collective generation. We look at the UCP as a sequential decision-making task with reward/penalty to reduce the collective generation cost of generators. To the best of our knowledge, ours is a first attempt at solving UCP by employing fuzzy reinforcement learning. We test our approach on a ten-generating-unit system with several equality and inequality constraints. Simulation results and comparisons against several recent UCP solution methods prove superiority and viability of our proposed multiagent fuzzy reinforcement learning technique.

Linguistic Lyapunov reinforcement learning control for robotic manipulators

We propose a Lyapunov theory based linguistic reinforcement learning (RL) framework for stable tracking control of robotic manipulators. In particular, we employ Lyapunov theory to constrain fuzzy rule consequents for ensuring stability of the designed controller. Proposed fuzzy RL controller employs Lyapunov theory dictated rules for discovering an optimal yet stable control strategy for robotic manipulators. Furthermore, our proposed linguistic RL controller handles payload variations and external disturbances quite effectively. We validate linguistic Lyapunov RL controller on two benchmark control problems: (i) a standard two-link robotic arm manipulator, and (ii) a two link selective compliance assembly robotic arm (SCARA). Simulation results and comparison against (a) baseline fuzzy Q learning (FQL) controller, and (b) a recently proposed Lyapunov theory based Markov game controller showcases our controller's superior tracking performance and lower computational complexity. Furthermore, our controller exhibits high stability with disturbances and payload variations.

An Intelligent Strategy Based on Fuzzy Reinforcement Learning for Backing Truck up

An Intelligent Strategy Based on Fuzzy Reinforcement Learning for Backing Truck up

Continuous-time path planning for multi-agents with fuzzy reinforcement learning

Continuous-time path planning for multi-agents with fuzzy reinforcement learning

Variable Admittance Control Based on Fuzzy Reinforcement Learning for Minimally Invasive Surgery Manipulator.

In order to get natural and intuitive physical interaction in the pose adjustment of the minimally invasive surgery manipulator, a hybrid variable admittance model based on Fuzzy Sarsa(λ)-learning is proposed in this paper. The proposed model provides continuous variable virtual damping to the admittance controller to respond to human intentions, and it effectively enhances the comfort level during the task execution by modifying the generated virtual damping dynamically. A fuzzy partition defined over the state space is used to capture the characteristics of the operator in physical human-robot interaction. For the purpose of maximizing the performance index in the long run, according to the identification of the current state input, the virtual damping compensations are determined by a trained strategy which can be learned through the experience generated from interaction with humans, and the influence caused by humans and the changing dynamics in the robot are also considered in the learning process. To evaluate the performance of the proposed model, some comparative experiments in joint space are conducted on our experimental minimally invasive surgical manipulator.

A Residual Gradient Fuzzy Reinforcement Learning Algorithm for Differential Games

In this work, we propose a new fuzzy reinforcement learning algorithm for differential games that have continuous state and action spaces. The proposed algorithm uses function approximation systems whose parameters are updated differently from the updating mechanisms used in the algorithms proposed in the literature. Unlike the algorithms presented in the literature which use the direct algorithms to update the parameters of their function approximation systems, the proposed algorithm uses the residual gradient value iteration algorithm to tune the input and output parameters of its function approximation systems. It has been shown in the literature that the direct algorithms may not converge to an answer in some cases, while the residual gradient algorithms are always guaranteed to converge to a local minimum. The proposed algorithm is called the residual gradient fuzzy actor–critic learning (RGFACL) algorithm. The proposed algorithm is used to learn three different pursuit–evasion differential games. Simulation results show that the performance of the proposed RGFACL algorithm outperforms the performance of the fuzzy actor–critic learning and the Q-learning fuzzy inference system algorithms in terms of convergence and speed of learning.

Fuzzy Lyapunov Reinforcement Learning for Non Linear Systems

We propose a fuzzy reinforcement learning (RL) based controller that generates a stable control action by lyapunov constraining fuzzy linguistic rules. In particular, we attempt at lyapunov constraining the consequent part of fuzzy rules in a fuzzy RL setup. Ours is a first attempt at designing a linguistic RL controller with lyapunov constrained fuzzy consequents to progressively learn a stable optimal policy. The proposed controller does not need system model or desired response and can effectively handle disturbances in continuous state-action space problems. Proposed controller has been employed on the benchmark Inverted Pendulum (IP) and Rotational/Translational Proof-Mass Actuator (RTAC) control problems (with and without disturbances). Simulation results and comparison against a) baseline fuzzy Q learning, b) Lyapunov theory based Actor-Critic, and c) Lyapunov theory based Markov game controller, elucidate stability and viability of the proposed control scheme.

Fuzzy Least Square Policy Iteration and Its Mathematical Analysis

In this paper we present a novel approach to reinforcement learning for continuous state–action control problems. This approach is obtained by combining least square policy iteration (LSPI) with zero-order Takagi–Sugeno fuzzy system, which we call it, “fuzzy least square policy iteration (FLSPI).” FLSPI is a critic-only method and has advantages of both LSPI and fuzzy systems together. We define state–action basis functions based on a fuzzy system while LSPI theorem conditions are satisfied. Our aim is to find the most suitable continuous action in every state using fuzzy rules. This method is learning rate independent and has positive mathematical analysis that defines an error bound for it. We show by simulation that the learning is faster and the quality of results is better than the two previous fuzzy reinforcement learning approaches based on critic-only architecture, i.e., fuzzy Q-learning (FQL) and Fuzzy SARSA Learning (FSL). We test FLSPI on four well-known problems (i.e., boat problem, maze, inverted pendulum and cart–pole balancing) and show the FLSPI higher performance, function of its error bound, its convergence against FQL and FSL divergence and its excellence against the latest proposed methods, respectively.

Coordination of Multiple Robotic Fish With Applications to Underwater Robot Competition

This paper is concerned with the coordination control of multiple biomimetic robotic fish in highly dynamic aquatic environments by building a hybrid centralized system. With the aid of the results of biorobotics and control techniques, a radio-controlled multijoint robotic fish and its locomotion control are developed. To enable a closed control loop, a visual subsystem that is responsible for tracking of multiple moving objects is constructed and implemented in real time. Furthermore, a behavior-based hierarchical architecture in conjunction with fuzzy reinforcement learning is proposed to accomplish effective coordination among multiple swimming robots. Finally, experiments on 2vs2 water polo game are carried out to verify the proposed coordination control scheme. Over the past eight years, this multirobot platform has been successfully applied to international underwater robot competitions to promote innovative research and education in underwater robotics.