Adaptive Q-learning Research Articles

An adaptive Q-learning based particle swarm optimization for multi-UAV path planning

An adaptive Q-learning based particle swarm optimization for multi-UAV path planning

Optimizing data aggregation and clustering in Internet of things networks using principal component analysis and Q-learning

The Internet of things (IoT) is a wireless network designed to perform specific tasks and plays a crucial role in various fields such as environmental monitoring, surveillance, and healthcare. To address the limitations imposed by inadequate resources, energy, and network scalability, this type of network relies heavily on data aggregation and clustering algorithms. Although various conventional studies have aimed to enhance the lifespan of a network through robust systems, they do not always provide optimal efficiency for real-time applications. This paper presents an approach based on state-of-the-art machine-learning methods. In this study, we employed a novel approach that combines an extended version of principal component analysis (PCA) and a reinforcement learning algorithm to achieve efficient clustering and data reduction. The primary objectives of this study are to enhance the service life of a network, reduce energy usage, and improve data aggregation efficiency. We evaluated the proposed methodology using data collected from sensors deployed in agricultural fields for crop monitoring. Our proposed approach (PQL) was compared to previous studies that utilized adaptive Q-learning (AQL) and regional energy-aware clustering (REAC). Our study outperformed in terms of both network longevity and energy consumption and established a fault-tolerant network.

Reinforcement learning-based hybrid differential evolution for global optimization of interplanetary trajectory design

The global optimization and design of interplanetary trajectories is one of the most important tasks in deep space exploration. Its search space is often characterized by multi-constraints, extreme non-linearity and sensitivity to initial conditions. To cope with these difficulties, an improved reinforcement learning-based hybrid differential evolution (DE) named RL_HDE is proposed in this paper. In RL_HDE, a novel multi-mutation strategy LSHADE based on an adaptive Q-Learning framework is proposed for global exploration. To further balance the global exploration and local exploitation of RL_HDE, a new parameter adaptive strategy based on Q-Learning is designed to control two trigger parameters. The performance of RL_HDE is verified by the well-known Global Trajectory Optimization Problems (GTOP), which are developed by Advanced Concepts Team of European Space Agency (ESA-ACT). A comparison of RL_HDE with three sets of algorithms, including ten state-of-the-art hybrid evolutionary algorithms, four interplanetary trajectory design algorithms, and seven reinforcement learning-based hybridizations. Experimental statistical results demonstrate that RL_HDE outperforms other competitors in terms of convergence efficiency and accuracy. RL_HDE has better performance for solving complex interplanetary trajectory design problems such as Cassini2 and Messenger-full.

Control with adaptive Q-learning: A comparison for two classical control problems

This paper evaluates adaptive Q-learning (AQL) and single-partition adaptive Q-learning (SPAQL), two algorithms for efficient model-free episodic reinforcement learning (RL), in two classical control problems (Pendulum and CartPole). AQL adaptively partitions the state–action space of a Markov decision process (MDP), while learning the control policy, i.e., the mapping from states to actions. The main difference between AQL and SPAQL is that the latter learns time-invariant policies, where the mapping from states to actions does not depend explicitly on the time step. This paper also proposes the SPAQL with terminal state (SPAQL-TS), an improved version of SPAQL tailored for the design of regulators for control problems. The time-invariant policies are shown to result in a better performance than the time-variant ones in both problems studied. These algorithms are particularly fitted to RL problems where the action space is finite, as is the case with the CartPole problem. SPAQL-TS solves the OpenAI Gym CartPole problem, while also displaying a higher sample efficiency than trust region policy optimization (TRPO), a standard RL algorithm for solving control tasks. Moreover, the policies learned by SPAQL are interpretable, while TRPO policies are typically encoded as neural networks, and therefore hard to interpret. Yielding interpretable policies while being sample-efficient are the major advantages of SPAQL.The code for the experiments is available at https://github.com/jaraujo98/SinglePartitionAdaptiveQLearning.

Battery energy storage control using a reinforcement learning approach with cyclic time-dependent Markov process

Scheduling efficient energy management system operations to respond to the unstable customer demand, electricity prices, and weather increases the complexity of the control systems and requires a flexible and cost-effective control policy. This study develops an intelligent and real-time battery energy storage control based on a reinforcement learning model focused on residential houses connected to the grid and equipped with solar photovoltaic panels and a battery energy storage system. Because the reinforcement learning’s performance is very dependent on the design of the underlying Markov decision process, a cyclic time-dependent Markov Process is uniquely designed to capture existing daily cyclic patterns in demand, electricity price, and solar energy. The Markov Process is successfully used in the Q-learning algorithm, resulting in more efficient battery energy control and saving electricity costs. The proposed Q-learning algorithm is compared with benchmark models of a deterministic equivalent solution and a One-step Roll-out algorithm. Numerical experiments show the gap between the deterministic equivalent solution and Q-learning approaches for one-month electricity cost decreased from 7.99% to 3.63% for house 27 and 6.91% to 3.26% for house 387 when the discrete size of demand, solar energy, price, and battery energy level adjusted to 20. Accordingly, the better performance of the proposed Q-learning is demonstrated compared to the One-step Roll-out algorithm. Moreover, the effect of discrete size of state-space parameters on the adaptive Q-learning performance and computational time are investigated. Variations in the electricity price significantly affect the Q-learning algorithm’s performance more than other parameters.

Online adaptive Q-learning method for fully cooperative linear quadratic dynamic games

A model-based offline policy iteration (PI) algorithm and a model-free online Q-learning algorithm are proposed for solving fully cooperative linear quadratic dynamic games. The PI-based adaptive Q-learning methodcan learn the feedback Nash equilibrium online using the state samples generated by behavior policies, without sending inquiries to the system model. Unlike the existing Q-learning methods, this novel Q-learning algorithm executes both policy evaluation and policy improvement in an adaptive manner. We prove the convergence of the offline PI algorithm by proving its equivalence to Newtons method while solving the game algebraic Riccati equation (GARE). Furthermore, we prove that the proposed Q-learning method will converge to the Nash equilibrium under a small learning rate if the method satisfies certain persistence of excitation conditions, which can be easily met by suitable behavior policies. Our simulation results demonstrate the good performance of the proposed online adaptive Q-learning algorithm.

Optimizing the network energy of cloud assisted internet of things by using the adaptive neural learning approach in wireless sensor networks

Cloud-assisted internet of things (CIoT) is backboned by the wireless sensor network (WSN) architecture. A sensor network is an autonomous self-resource constraint collection of internet of things (IoT) sensor nodes. The nodes communicate in an ad-hoc fashion to transfer cloud information over the virtual environment. Clustering in WSNs helps to improve the quality of the network by controlling energy consumption and improving data gathering accuracy. This improves the service rates of CIoT. Optimizing IoT sensor networks through energy and overhead management requires complex clustering algorithms. A simple clustering scheme cannot achieve the desired performance enhancement during transmission in a virtual environment. This research attempts to propose a reinforcement-based learning technique, adaptive Q-learning (AQL) to improve network performance with minimum energy–overhead tradeoff in a sensor network-aided CIoT. AQL operates in two distinct phases for cluster head selection and forwarder selection. The decision-making system is used to qualify nodes based on their past behavior over transmission. AQL improves both inter- and intra-cluster communication optimization through adaptive forwarder and header selection conditions. The simulation results prove the consistency of the proposed AQL by retaining the live node counts in the network and their persistent energy despite the reduced overheads in the sensor network. With the achievement of constructive features in the sensor networks, the performance of CIoT is considerably improved. The experimental results illustrate the effectiveness of the proposed learning technique by improving network lifetime with a high request–response rate and by minimizing delay, overhead, and request failures.