Fuzzy Q-learning Algorithm Research Articles

A Novel Method for a Pursuit–Evasion Game Based on Fuzzy Q-Learning and Model-Predictive Control

This paper explores a pursuit–evasion game (PEG) based on quadrotors by combining fuzzy Q-learning (FQL) and model-predictive control (MPC) algorithms. Initially, the FQL algorithm is employed to perceive, make decisions, and predict the trajectory of the evader. Based on the position and velocity information of both players in the game, the pursuer quadrotor determines its action strategy using the FQL algorithm. Subsequently, a state feedback controller is designed using the MPC algorithm, with reference inputs derived from the FQL algorithm. Within each MPC cycle, the FQL algorithm dynamically provides reference inputs to the MPC, thereby enhancing its robust control and dynamic optimization for the quadrotor. Finally, simulation results verify the effectiveness of the proposed algorithm.

Improvement of electric vehicle safety using a new hybrid fuzzy Q-learning algorithm for lithium-ion battery state-of-charge estimation

Improvement of electric vehicle safety using a new hybrid fuzzy Q-learning algorithm for lithium-ion battery state-of-charge estimation

A Bilateral Teleoperation System With Learning-Based Cognitive Guiding Force

This article investigates the problem involving a fuzzy logic system-based bilateral teleoperation system with a unified impedance/admittance structure and learning-based assisted cognitive guiding force, enhancing a smooth transition between augmentation and autonomous motion on the master side. The impedance structure regulates the impedance and admittance via the parameters determined by the fuzzy logic system and, besides the variable impedance during interaction with the environment, the fuzzy Q-learning algorithm generates an assisted cognitive guiding force to guide the trainee operator in free space for reaching the interaction surface smoothly and steadily. The degeneration of tracking performance in the bilateral teleoperation system caused by time delay is analyzed and reduced via a mixed-type error. Experimental results verify the stability analyses, and under the proposed control scheme, the human-robot-environment interaction is smooth and stable.

Simultaneous control of rigidity and hand tremor by adaptive fuzzy Q-learning

Deep brain stimulation (DBS) is a successful treatment for the symptoms of various neurodegenerative diseases, including Parkinson's disease (PD). While DBS has been shown to improve motor symptoms in individuals with PD, it operates mostly in an open-loop mode, meaning that the stimulation settings remain constant regardless of changes in the disease state. To address this issue, the current paper describes the use of a fuzzy Q-learning algorithm combined with a model-free non-linear controller, called FQL-based MFNC, to control both hand tremor and rigidity at the same time. In contrast to previous tuning methods that focus on obtaining the mathematical model of a controlled system, the current technique explores the use of a fuzzy Q-learning agent to achieve the best design for DBS. Initially, a fuzzy reinforcement learning approach is employed to adjust the proposed controller in a simulation setting. Specifically, the FQL algorithm determines the best approach based on a reward function to modify the designed control coefficients, ensuring that the DBS requirements are satisfied despite changes in tremors. Multiple examinations, including Normal, Noise, and Stability tests, along with different indices, have been conducted to validate the efficacy and feasibility of the proposed FQL-based MFNC approach on a dynamic model pertaining to rigidity and tremors.

Discrete-Time Control for Double-Integrator Systems With State Constraint and Learning Gain

This brief investigates a barrier Lyapunov function based discrete-time control with Q-learning based gains for double-integrator systems with state constraint. It is found, from the stability proof, that the high conservatism of analysis for the stabilization of discrete-time system with state constraint is revealed, where the explicit selection of constant high gain is challenging. To address this problem, the fuzzy Q-learning algorithm is employed to search for the nearly optimal control gains for both fast response and low steady-state error in the long view of performance consideration. The numerical and experimental results verify the effectiveness of the proposed method, and varying gains based on fuzzy approximation Q-learning can aid to reduce the steady-state error while fast response to the reference motion trajectories.

Adaptive Fuzzy Q-Learning Control Design and Application to Grid-Tied Nine-Level Packed E-Cell (PEC9) Inverter

This letter applies a multiagent fuzzy Q-learning (FQL) algorithm, incorporated with a model-free nonlinear controller (MFNC), entitled FQL-MFNC for stabilized controlling of a recently introduced grid-connected nine-level Packed E-Cell (PEC9) inverter under dynamical operation. Unlike previous tuning schemes, which concentrate on extracting mathematical formulation of a controlled plant, this letter investigates a fuzzy Q-learning agent for optimal design of PEC9. In first step, the fuzzy reinforcement learning is adopted to tune the MFNC controller in the simulation environment. In fact, the FQL algorithm finds the optimal policy based on a reward function for adjustment of the MFNC control coefficients to guarantee the grid-connectivity requirements under PEC9 dynamical operation are met. The experimental tests are conducted to assure efficiency and practicability of the designed multi-agent FQL-MFNC scheme on the single-phase grid-connected PEC9 inverter.

Multi-objective fuzzy Q-learning to solve continuous state-action problems

Many real world problems are multi-objective. Thus, the need for multi-objective learning and optimization algorithms is inevitable. Although the multi-objective optimization algorithms are well-studied, the multi-objective learning algorithms have attracted less attention. In this paper, a fuzzy multi-objective reinforcement learning algorithm is proposed, and we refer to it as the multi-objective fuzzy Q-learning (MOFQL) algorithm. The algorithm is implemented to solve a bi-objective reach-avoid game. The majority of the multi-objective reinforcement algorithms proposed address solving problems in the discrete state-action domain. However, the MOFQL algorithm can also handle problems in a continuous state-action domain. A fuzzy inference system (FIS) is implemented to estimate the value function for the bi-objective problem. We used a temporal difference (TD) approach to update the fuzzy rules. The proposed method is a multi-policy multi-objective algorithm and can find the non-convex regions of the Pareto front.

A Fuzzy Q-Learning Algorithm for Storage Optimization in Islanding Microgrid

In islanding microgrids, energy storage plays a key role in obtaining flexible power control and operation. The energy storage solves the effects of randomness, intermittency and uncertainty of renewable energy through its peak regulation and frequency modulation. In order to better to improve the economics of the microgrid, this paper proposes a Q-learning algorithm based on fuzzy control. It is a model-free algorithm, without complicated modelling, the new algorithm can handle the state space explosion problem of the Q learning algorithm. This method improves the system's renewable resource utilization rate by controlling the energy flow of the energy storage and adjusting the system's energy scheduling. The results show that compared with the Q-learning, the fuzzy Q-learning algorithm reduced the use of fossil fuels by 33.7%, increases the utilization rate of renewable resources from 85.7 to 93.5%.

A Q Learning and Fuzzy Q Learning Approach for Optimization of Interference Constellations in Femto–Macro Cellular Architecture in Downlink

The reduction in the cost and enhancement in the network coverage and capacity are the main objectives in the establishment of mobile networks. These objectives were the key force behind the idea of femtocells deployment. But, there are many technical issues for the femtocells deployment with already existing macro network. Cross-tier interference is one of the main challenge that must have to be resolved for smooth operation of macro---femto network. This paper gives self-optimizing and self-healing technique that utilizes Q-learning and fuzzy Q-learning algorithm for the objectives of enhancement in the network capacity and spectral efficiency. In our proposed scheme, each macro base station acts as an agent which interacts with its local environment (all the femtocells and mobile stations under its coverage area), gathers the information and takes the suitable actions correspondingly. For the objective of controlling cross-tier interference, macrousers are rescheduled in such an intelligent way that performance of the femtousers, located on the overlapped spectral portion, is not degraded. The simulation results confirm our proposed approach to improve the network capacity and spectral efficiency as well as sharp convergence, which designates its capability to meet the self organizing network requirements set by 3GPP.

Model-Based and Learning-Based Decision Making in Incomplete Information Cournot Games: A State Estimation Approach

In an incomplete information game, a big challenge is to find the best way of exploiting available information for optimal decision making of the agents. In this paper, two decision making methods, namely model-based and learning-based bidding strategies, are proposed and compared, for repeated Cournot competition of the generators in a day-ahead electricity market. The sum of the rivals' offered quantities (SROQ) is considered as the state of the agent and its value is estimated using an adaptive expectation method. In the model-based approach, the convergence of the agents' strategies to the Nash equilibrium point is also studied in two different cases. In the learning-based approach, the optimal bidding strategy is learned through combination of state estimation and a reinforcement learning method. Using the estimated state (SROQ), the optimal decision is learned through a fuzzy Q-learning algorithm. Through a case study, which is performed on the three-bus benchmark Cournot model, the convergence of the generators' bids to the Nash-Cournot equilibrium is examined.

Cooperative game theoretic approach using fuzzy Q-learning for detecting and preventing intrusions in wireless sensor networks

Owing to the distributed nature of denial-of-service attacks, it is tremendously challenging to detect such malicious behavior using traditional intrusion detection systems in Wireless Sensor Networks (WSNs). In the current paper, a game theoretic method is introduced, namely cooperative Game-based Fuzzy Q-learning (G-FQL). G-FQL adopts a combination of both the game theoretic approach and the fuzzy Q-learning algorithm in WSNs. It is a three-player strategy game consisting of sink nodes, a base station, and an attacker. The game performs at any time a victim node in the network receives a flooding packet as a DDoS attack beyond a specific alarm event threshold in WSN. The proposed model implements cooperative defense counter-attack scenarios for the sink node and the base station to operate as rational decision-maker players through a game theory strategy. In order to evaluate the performance of the proposed model, the Low Energy Adaptive Clustering Hierarchy (LEACH) was simulated using NS-2 simulator. The model is subsequently compared against other existing soft computing methods, such as fuzzy logic controller, Q-learning, and fuzzy Q-learning, in terms of detection accuracy, counter-defense, network lifetime and energy consumption, to demonstrate its efficiency and viability. The proposed model׳s attack detection and defense accuracy yield a greater improvement than existing above-mentioned machine learning methods. In contrast to the Markovian game theoretic, the proposed model operates better in terms of successful defense rate.

Anomaly Detection using Fuzzy Q-learning Algorithm

Wireless networks are increasingly overwhelmed by Distributed Denial of Service (DDoS) attacks by generating flooding packets that exhaust critical computing and communication resources of a victim's mobile device within a very short period of time. This must be protected. Effective detection of DDoS attacks requires an adaptive learning classifier, with less computational complexity, and an accurate decision making to stunt such attacks. In this paper, we propose an intrusion detection system called Fuzzy Q- learning (FQL) algorithm to protect wireless nodes within the network and target nodes from DDoS attacks to identify the attack patterns and take appropriate countermeasures. The FQL algorithm was trained and tested to establish its performance by generating attacks from the NSL-KDD and CAIDA DDoS Attack datasets during the simulation experiments. Experimental results show that the proposed FQL IDS has higher accuracy of detection rate than Fuzzy Logic Controller and Q-learning algorithm alone.

Temperature control approach in intelligent office environment

The parameters of inverter air-conditioner are fixed,which cannot adapt to changed intelligent office environment.This paper proposed a new air-conditioner control approach in order to improve the comfort of the ambient temperature.Multi-Agent technology was introduced,and input and output variables and their fuzzy sets were given,and then the fuzzy Q-learning algorithm introduced rewards was proposed,which was executed by reasoning Agent to learn manually adjusting the air-conditioner action,and modify the fuzzy rules,so the fuzzy rules were optimized.Finally the optimization fuzzy rules were applied to control ambient temperature.The experimental result shows that the response time and overshoot of the air-conditioner are less than the conventional fuzzy temperature control approach.

Optimization of load balancing using fuzzy Q-Learning for next generation wireless networks

Load balancing is considered by the 3GPP as an important issue in Self-Organizing Networks due to its effectiveness to increase network capacity. In next generation wireless networks, load balancing can be easily implemented by tuning handover (HO) margins, achieving a decrease in call blocking. However, call dropping can be increased as a negative effect of the HO-based load balancing, because users usually are handed over to cells where the radio conditions are worse. In this work, a Fuzzy Logic Controller (FLC) optimized by the fuzzy Q-Learning algorithm is proposed for the load balancing problem, with the aim of decreasing call blocking in congested cells, while at the same time restricting call dropping in neighboring cells according to the network policy. In particular, two different approaches for the FLC optimization are evaluated in this work, highlighting that one of the proposed methods allows to accurately preserve the call quality constraint during the load balancing, while the other can adapt to network variations. Results show that the optimized FLC provides a notable reduction in call blocking while preserving call dropping under the operator constraint.

Self-Optimization of Coverage and Capacity in LTE Networks Based on Central Control and Decentralized Fuzzy Q-Learning

To reduce capital expenditures (CAPEX) and operational expenditures (OPEX) in network operations, self-organizing network (SON) has been introduced as a key part of long-term-evolution (LTE) system. Self-optimization of coverage and capacity is one of the most important tasks in the context of SON. This paper proposes a central control mechanism that utilizes the fuzzy Q-learning algorithm in a decentralized fashion for this task. In our proposed approach, each eNB is a learning agent that tries to optimize its antenna downtilt automatically using information from its own and its neighboring cells, and the initialization and the termination of the optimization processes of all agents are in the control of the central entity. The simulation results verify that our proposed approach can achieve remarkable performance enhancement as well as fast convergence, indicating that it is able to meet different levels of demands defined by 3GPP for coverage and capacity optimization.

Executable design and control framework using reconfigurable manufacturing holons and its evolution algorithm

This paper provides an executable design framework for modelling and controlling holonic manufacturing processes and systems. This framework is based on reconfigurable manufacturing holonic modules. These holonic modules consist of holons represented using parameterised Petri Net modules. As the suggested modelling framework shares advantages of IEC 61499 and FCPN-std, it supports the design of complex holonic manufacturing processes/system and allows simulation and analysis under general manufacturing uncertainties. Self-organisation activity in a holonic manufacturing system is explained using an evolution concept. A holon's inner evolution is executed through cascading parameterised Petri Net modules. This evolution is used for capturing the dynamic characteristics of a manufacturing environment. Routeing and reconfiguration is fulfilled through configuration evolution. It is achieved using the suggested constraint-based fuzzy Q-learning algorithm. The provided framework and two evolution algorithms are considered an executable design framework for modelling and controlling holonic manufacturing system.

Fuzzy-Q-learning-based autonomic management of macro-diversity algorithm in UMTS networks

Third generationUmts network has come with significant high-quality services that considerably increase the complexity of its management. Autonomic management has been introduced to alleviate these complex lengthy tasks. In this paper, we propose an autonomic management of macro-diversity algorithm inUmts networks. The new approach allows to dynamically adapt macro-diversity parameters to varying network situations. The online adaptation of these parameters is made by an intelligent controller calledfuzzy-Q-Learning. The combination of Fuzzy Inference System (Fis) and Q-learning algorithm allows to determine the best on-line parameterization of base stations and to deal with large number of continuous states and actions. The proposed scheme improves the system capacity up to 30% compared to a classical network with fixed parameters, balances the load between base stations and minimizes human interventions in the network management. However, the reactivity of the controller should be chosen with a special care since it impacts the frequency of active set updates and hence signalling messages in the radio interface as well as in the core network.

Enhancements Of Fuzzy Q-Learning Algorithm

Fuzzy Q-Learning algorithm combines reinforcement learning techniques with fuzzy modelling. It provides a flexible solution for automatic discovery of rules for fuzzy systems inthe process of reinforcement learning. In this paper we propose several enhancements tothe original algorithm to make it more performant and more suitable for problems withcontinuous-input continuous-output space. Presented improvements involve generalizationof the set of possible rule conclusions. The aim is not only to automatically discover anappropriate rule-conclusions assignment, but also to automatically define the actual conclusions set given the all possible rules conclusions. To improve algorithm performance whendealing with environments with inertness, a special rule selection policy is proposed.