Q-learning Reinforcement Learning Research Articles

Coordinated control of yaw and roll stability in heavy vehicles considering dynamic safety requirements

In the field of heavy vehicle stability research, traditional safety requirements are often based on static scenario settings. However, the complexity and variability of actual road environments require safety control strategies that can be adapted to different driving conditions and environmental changes in real-time. To address this challenge, the paper proposes a coordinated control strategy for yaw and roll stability that considers the dynamic safety requirements. First, a quantitative analysis method for vehicle stability is proposed based on the dissipated energy theory, taking into account the lateral-vertical dynamics coupling characteristics. Additionally, a dynamic safety requirements identification model is developed by integrating the vehicle's future driving risk prediction algorithm. To meet dynamic safety requirements, a dynamic weight model predictive control method based on randomized ensembled double Q-learning reinforcement learning is designed. This method adjusts the control weights of yaw and roll stability online to flexibly address various destabilization risks, aiming to achieve more precise and efficient stability control. Through simulation and experimental verification, the results demonstrate that the proposed coordinated control strategy can effectively enhance the stability and safety of heavy vehicles in complex and dynamic driving environments.

Optimisation tool: Q-learning and its application in various fields

Q-learning in reinforcement learning can serve as a useful tool in many fields. The one popular technology one can hear lately is artificial intelligence, which can be found in almost any field and area. However, artificial intelligence is a combination of different technologies and methods that are most of the time ignored or overshadowed. Q-learning is a method in which each action or step is analyzed and provided by feedback. Depending on the feedback the method continues its actions by choosing the best possible path. In this paper, Q-learning will be briefly expanded with the examples. The main focus of this work will be on Its application in different fields. The following fields will be covered: healthcare, education, gaming, manufacturing, and finances.

AI-Based Q-Learning Approach for Performance Optimization in MIMO-NOMA Wireless Communication Systems

In this paper, we investigate the performance enhancement of Multiple Input, Multiple Output, and Non-Orthogonal Multiple Access (MIMO-NOMA) wireless communication systems using an Artificial Intelligence (AI) based Q-Learning reinforcement learning approach. The primary challenge addressed is the optimization of power allocation in a MIMO-NOMA system, a complex task given the non-convex nature of the problem. Our proposed Q-Learning approach adaptively adjusts power allocation strategy for proximal and distant users, optimizing the trade-off between various conflicting metrics and significantly improving the system’s performance. Compared to traditional power allocation strategies, our approach showed superior performance across three principal parameters: spectral efficiency, achievable sum rate, and energy efficiency. Specifically, our methodology achieved approximately a 140% increase in the achievable sum rate and about 93% improvement in energy efficiency at a transmitted power of 20 dB while also enhancing spectral efficiency by approximately 88.6% at 30 dB transmitted Power. These results underscore the potential of reinforcement learning techniques, particularly Q-Learning, as practical solutions for complex optimization problems in wireless communication systems. Future research may investigate the inclusion of enhanced channel simulations and network limitations into the machine learning framework to assess the feasibility and resilience of such intelligent approaches.

Deep Q-Network Approach for Train Timetable Rescheduling Based on Alternative Graph

The disturbance of local areas with complex railway networks and high traffic density not only impedes the efficient use of rail networks in those areas, but also propagates delays to the entire railway network. This has motivated research on train rescheduling problems in high-density local areas to minimize train delays by modifying their planned arrival and departure times. In this paper, we present a train rescheduling method based on Q-learning in reinforcement learning. More specifically, we used deep neural networks to approximate the action-value function, and the underlying Markov decision process (MDP) is based on the alternative graph formulation for the train rescheduling problem. In the proposed MDP formulation, the status of the alternative graph corresponding to the current schedule is defined as the state, and the alternative arc corresponds to the action the agent can take. The MDP is approximately solved via deep Q-learning in which deep neural networks are used to approximate the action-value function in Q-learning. Although the size of the alternative graph depends on the number of trains, our MDP formulation is independent of the number of trains, which makes the proposed method more scalable. The evaluation of the method was performed on a simple railway network and a real-world example in Seoul, South Korea, with randomly generated initial train schedules and train delays. The experimental result showed that the proposed method is comparable to the mixed-integer-linear-programming (MILP)-based exact approach with respect to the quality of the solution.

Positioning an electric wheelchair in 2D grid map based on natural landmarks for navigation using Q-learning

Self-mobility electric wheelchairs are very useful for people with disabilities, so they can move without help in indoor environments. To create one selfmobility electric wheelchair, modern methods for control such as computer vision and machine learning can be applied. In particular, this electric wheelchair can move from any position in the indoor environment to the desired destination. For accuracy, natural landmarks are used and the navigation of the wheelchair is determined using a Q-learning reinforcement learning algorithm. In particular, this algorithm is applied to find the best path for the wheelchair to reach the destination. The article proposes a method to build one 2D grid map for wheelchair movement based on natural landmarks in the indoor environment. The new point of this method is that the position of the wheelchair can be accurately determined from a certain landmark instead of many landmarks applied in traditional methods. Some practical experiments were performed to illustrate the effectiveness of the proposed method in the indoor environment. This proposed method can be developed in more complex environments with natural landmarks.

Improved reinforcement learning path planning algorithm integrating prior knowledge.

In order to realize the optimization of autonomous navigation of mobile robot under the condition of partial environmental knowledge known. An improved Q-learning reinforcement learning algorithm based on prior knowledge is proposed to solve the problem of slow convergence and low learning efficiency in mobile robot path planning. Prior knowledge is used to initialize the Q-value, so as to guide the agent to move toward the target direction with a greater probability from the early stage of the algorithm, eliminating a large number of invalid iterations. The greedy factor ε is dynamically adjusted based on the number of times the agent successfully reaches the target position, so as to better balance exploration and exploitation and accelerate convergence. Simulation results show that the improved Q-learning algorithm has a faster convergence rate and higher learning efficiency than the traditional algorithm. The improved algorithm has practical significance for improving the efficiency of autonomous navigation of mobile robots.

Parallel hyper heuristic algorithm based on reinforcement learning for the corridor allocation problem and parallel row ordering problem

Hyper heuristics is a relatively new optimisation algorithm. Numerous studies have reported that hyper heuristics are well applied in combinatorial optimisation problems. As a classic combinatorial optimisation problem, the row layout problem has not been publicly reported on applying hyper heuristics to its various sub-problems. To fill this gap, this study proposes a parallel hyper-heuristic approach based on reinforcement learning for corridor allocation problems and parallel row ordering problems. For the proposed algorithm, an outer layer parallel computing framework was constructed based on the encoding of the problem. The simulated annealing, tabu search, and variable neighbourhood algorithms were used in the algorithm as low-level heuristic operations, and Q-learning in reinforcement learning was used as a high-level strategy. A state space containing sequences and fitness values was designed. The algorithm performance was then evaluated for benchmark instances of the corridor allocation problem (37 groups) and parallel row ordering problem (80 groups). The results showed that, in most cases, the proposed algorithm provided a better solution than the best-known solutions in the literature. Finally, the meta-heuristic algorithm applied to three low-level heuristic operations is taken as three independent algorithms and compared with the proposed hyper-heuristic algorithm on four groups of parallel row ordering problem instances. The effectiveness of Q-learning in selection is illustrated by analysing the comparison results of the four algorithms and the number of calls of the three low-level heuristic operations in the proposed method.

A reinforcement learning based computational intelligence approach for binary optimization problems: The case of the set-union knapsack problem

As today’s one of the hottest topics, machine learning brings about opportunities in various research areas. Moreover, computational intelligence and metaheuristics open up new strategies, which are shown to be efficient in solving optimization problems. However, studies bringing such remarkable approaches together are still lacking. In this context, the present paper introduces a Q-learning reinforcement learning strategy for binary optimization problems. The developed algorithm works as a reinforcement and recommendation system that evaluates the used algorithms, assigns rewards, promotes or demotes them. Thus, it invokes more promising optimizers more frequently. The proposed Q-learning algorithm uses Particle Swarm Optimization (PSO), Genetic Algorithm and a hybrid of these algorithms, namely, genetic-based PSO (gbPSO) as optimizers. Therefore, it is aimed to avoid local optima by using various optimizers and gathering additional statistical data. Secondarily, all optimizers are further enhanced by adopting an initial solution generation technique and triggered random immigrants mechanism to preserve swarm diversity. In addition to these procedures, a mutation procedure that decreases the diversity is adopted. Thus, more intensified search is encouraged towards the end of search. Moreover, while PSO requires for transfer functions in order to perform in binary spaces, the adopted and further improved gbPSO does not necessarily need such auxiliary procedures. Finally, the performances of all used algorithms are analysed on a recently caught on binary problem, namely, the set-union knapsack problem, which has a wide range of real-life applications. As demonstrated by the comprehensive experimental study and appropriate statistical tests, promising improvements are achieved.

Lightweight Task Coordination of LEO Satellite Cluster based on Distributed Reinforcement Learning

Low earth orbit (LEO) mega-constellation is in the stage of rapid construction, which is expected to serve more regions and apply to more scenarios, such as regional navigation enhancement, emergency communication, collaborative detection, etc. For such typical regional tasks, the construction of temporary LEO satellite clusters is an effective way to improve the quality of temporary services. However, LEO satellites move too fast, and relying on ground-based manual control can no longer meet the mission requirements. The cluster task coordination based on reinforcement learning can effectively improve the autonomous performance of the cluster, but it is difficult to apply to satellites, especially LEO satellites with extremely limited resources. Based on this, this paper proposes a lightweight collaboration method for LEO satellite constellation based on distributed reinforcement learning, which optimizes resources on the premise of ensuring the independent realization of task requirements. In this paper, the Q-learning reinforcement learning algorithm is used to compare the performance of the centralized Q-learning algorithm and distributed AC algorithm. The experiment of the star cluster cooperative detection task shows that the convergence time of the proposed algorithm is 81.5% less than that of the centralized algorithm, and it can run on embedded raspberry pi. It shows that the algorithm is expected to be applied to cluster task coordination of LEO satellites.

Embedded Learning Approaches in the Whale Optimizer to Solve Coverage Combinatorial Problems

When we face real problems using computational resources, we understand that it is common to find combinatorial problems in binary domains. Moreover, we have to take into account a large number of possible candidate solutions, since these can be numerous and make it complicated for classical algorithmic techniques to address them. When this happens, in most cases, it becomes a problem due to the high resource cost they generate, so it is of utmost importance to solve these problems efficiently. To cope with this problem, we can apply other methods, such as metaheuristics. There are some metaheuristics that allow operation in discrete search spaces; however, in the case of continuous swarm intelligence metaheuristics, it is necessary to adapt them to operate in discrete domains. To perform this adaptation, it is necessary to use a binary scheme to take advantage of the original moves of the metaheuristics designed for continuous problems. In this work, we propose to hybridize the whale optimization algorithm metaheuristic with the Q-learning reinforcement learning technique, which we call (the QBWOA). By using this technique, we are able to realize an smart and fully online binarization scheme selector, the results have been statistically promising thanks to the respective tables and graphs.

Frequency matching optimization model of ultrasonic scalpel transducer based on neural network and reinforcement learning

Aiming at the problem that the excitation frequency and resonant frequency of the transducer cannot keep synchronous, the output amplitude decreases and the vibration is unstable. In this study, the working principle of piezoelectric transducers is firstly analyzed by the equivalent circuit method and the instantaneous characteristic variables (installing preload, assembly preload, load, and other factors) that affect the frequency matching obtained to establish the equivalent relationship with the resonant frequency. Secondly, in order to make the synchronization between excitation frequency and resonance frequency, the key instantaneous characteristic variables are extracted based on Pearson correlation coefficient. Thirdly, the mathematical model of the mapping relationships between instantaneous characteristic variables and resonance frequency is established with the radial basis function neural network (RBFNN). Fourthly, for the purpose of the adaptation to the characteristics of dynamic load and real-time frequency modulation in the operation of ultrasonic scalpels, the reinforcement learning (Q-learning algorithm) and the weight vector of RBFNN are used to define the eligibility trace, which is used to dynamically adjust the RBFNN frequency matching optimization model in real time and maintain the “constant” amplitude output and stable harmonic response process. Finally, the experimental results show that, compared with the traditional methods, the frequency matching optimization model of ultrasonic scalpel transducer based on RBF neural network and Q-Learning reinforcement learning is effective, and the vibration amplitude of the transducer is increased by 15.25μm. The amplitude fluctuation is stable at 0.92μm. It can provide decision-making guidance for relevant engineering fields.

Hyperparameter Optimization for the LSTM Method of AUV Model Identification Based on Q-Learning

An accurate mathematical model is a basis for controlling and estimating the state of an Autonomous underwater vehicle (AUV) system, so how to improve its accuracy is a fundamental problem in the field of automatic control. However, AUV systems are complex, uncertain, and highly non-linear, and it is not easy to obtain through traditional modeling methods. We fit an accurate dynamic AUV model in this study using the long short-term memory (LSTM) neural network approach. As hyper-parameter values have a significant impact on LSTM performance, it is important to select the optimal combination of hyper-parameters. The present research uses the improved Q-learning reinforcement learning algorithm to achieve this aim by improving its recognition accuracy on the verification dataset. To improve the efficiency of action exploration, we improve the Q-learning algorithm and choose the optimal initial state according to the Q table in each round of learning. It can effectively avoid the ineffective exploration of the reinforcement learning agent between the poor-performing hyperparameter combinations. Finally, the experiments based on simulated or actual trial data demonstrate that the proposed model identification method can effectively predict kinematic motion data, and more importantly, the modified Q-Learning approach can optimize the network hyperparameters in the LSTM.

Discrete Event Modeling and Simulation for Reinforcement Learning System Design

Discrete event modeling and simulation and reinforcement learning are two frameworks suited for cyberphysical system design, which, when combined, can give powerful tools for system optimization or decision making process for example. This paper describes how discrete event modeling and simulation could be integrated into reinforcement learning concepts and tools in order to assist in the realization of reinforcement learning systems, more specially considering the temporal, hierarchical, and multi-agent aspects. An overview of these different improvements are given based on the implementation of the Q-Learning reinforcement learning algorithm in the framework of the Discrete Event system Specification (DEVS) and System Entity Structure (SES) formalisms.

Constrained evolutionary optimization based on reinforcement learning using the objective function and constraints

Solving constrained optimization problems (COPs) with evolutionary algorithms is highly active in the evolutionary computation community. Combining evolutionary algorithms with the learning techniques is an efficient way to obtain promising performance for the COPs. Based on this consideration, we propose a differential evolution assisted by reinforcement learning (RL), namely RL-CORCO, to effectively solve the COPs. The proposed method can be featured as (i) the Q-learning in RL is used for adaptive operator selection; (ii) the hierarchical population is set as a state to find the feasible optimal solution; and (iii) the correlation between constraints and the objective function is utilized. The RL-CORCO is tested on 18 benchmark problems in the CEC 2010 competition and 28 benchmark problems in the CEC 2017 competition. Experimental results show that in CEC2017, RL-CORCO performed better than others on 12 problems in 50 dimensions and 14 problems in 100 dimensions. The results of the Friedman’s test demonstrate the efficacy of the algorithm, which is able to obtain highly competitive results compared with other related methods.

CLASSQ-L: A Q-Learning Algorithm for Adversarial Real-Time Strategy Games

We present CLASSQ-L (for: class Q-learning) an application of the Q-learning reinforcement learning algorithm to play complete Wargus games. Wargus is a real-time strategy game where players control armies consisting of units of different classes (e.g., archers, knights). CLASSQ-L uses a single table for each class of unit so that each unit is controlled and updates its class’ Q-table. This enables rapid learning as in Wargus there are many units of the same class. We present initial results of CLASSQ-L against a variety of opponents.

Integrating Production Planning with Truck-Dispatching Decisions through Reinforcement Learning While Managing Uncertainty

This paper presents a new truck dispatching policy approach that is adaptive given different mining complex configurations in order to deliver supply material extracted by the shovels to the processors. The method aims to improve adherence to the operational plan and fleet utilization in a mining complex context. Several sources of operational uncertainty arising from the loading, hauling and dumping activities can influence the dispatching strategy. Given a fixed sequence of extraction of the mining blocks provided by the short-term plan, a discrete event simulator model emulates the interaction arising from these mining operations. The continuous repetition of this simulator and a reward function, associating a score value to each dispatching decision, generate sample experiences to train a deep Q-learning reinforcement learning model. The model learns from past dispatching experience, such that when a new task is required, a well-informed decision can be quickly taken. The approach is tested at a copper–gold mining complex, characterized by uncertainties in equipment performance and geological attributes, and the results show improvements in terms of production targets, metal production, and fleet management.

Optimization of thermal comfort, indoor quality, and energy-saving in campus classroom through deep Q learning

This study develops a control algorithm for optimization the energy consumptions of air-conditioning and exhaust fans through Deep Q-Learning in reinforcement learning. The proposed agent is able to balance indoor air quality (CO2), thermal comfort, and energy consumption. The algorithm was first trained in a similar environment simulation, and was then applied and tested in a classroom with maximum 72 occupants. Tests were conducted in one month during summer. The effects of outdoor environments and class conditions on the energy-saving and indoor air quality are examined in details. Via agent control, optimization of indoor air quality, thermal comfort, and energy consumption of air-conditioning units and exhaust fans can be achieved. With the same thermal comfort, the agent can offer energy-saving up to 43% when compared to air-conditioning with a fixed temperature of 25 °C, and on average the agent offers about 19% less of the energy consumption. Yet the corresponding CO2 level is reduced by about 24% with the agent control. Similarly, when compared with a fixed temperature of 26 °C, the agent can offer about 15% lower energy consumption on average and the concentration of carbon dioxide can be reduced by 13% in average.

Multi-AUV Collaborative Target Recognition Based on Transfer-Reinforcement Learning

Due to the existence of unfavorable factors such as turbid water quality and target occlusion, it is difficult to obtain valid data of target features. Due to the repeated calculation of similar data, the real-time performance of the algorithm is poor. In view of the above problems, this paper proposes a multi-AUV collaborative target recognition method based on transfer-reinforcement learning. The features of the target information which is collected by multi-AUV are fused based on wavelet transformation and affine invariance. The similarity of features is calculated by Mahalanobis distance and the learning model is selected autonomously based on the similarity threshold. Based on the Q-learning reinforcement learning model, the target information under the interference environment is trained intensively, and the effective features are extracted and stored in the source domain, which can reduce the impact of the environmental interference on the target recognition. The feature transfer learning model based on deep confidence network transfers the feature data of the source domain to the target domain, reducing the repeated calculation of similar data, and then ensuring the real-time performance of the algorithm. Simulation experiments are conducted in the SUN dataset under five underwater environments (turbid water, target occlusion, insufficient light, complex background, and overlapping targets), and the results demonstrate that the proposed model achieves better performance.

Crop Yield Prediction Using Deep Reinforcement Learning Model for Sustainable Agrarian Applications

Predicting crop yield based on the environmental, soil, water and crop parameters has been a potential research topic. Deep-learning-based models are broadly used to extract significant crop features for prediction. Though these methods could resolve the yield prediction problem there exist the following inadequacies: Unable to create a direct non-linear or linear mapping between the raw data and crop yield values; and the performance of those models highly relies on the quality of the extracted features. Deep reinforcement learning provides direction and motivation for the aforementioned shortcomings. Combining the intelligence of reinforcement learning and deep learning, deep reinforcement learning builds a complete crop yield prediction framework that can map the raw data to the crop prediction values. The proposed work constructs a Deep Recurrent Q-Network model which is a Recurrent Neural Network deep learning algorithm over the Q-Learning reinforcement learning algorithm to forecast the crop yield. The sequentially stacked layers of Recurrent Neural network is fed by the data parameters. The Q- learning network constructs a crop yield prediction environment based on the input parameters. A linear layer maps the Recurrent Neural Network output values to the Q-values. The reinforcement learning agent incorporates a combination of parametric features with the threshold that assist in predicting crop yield. Finally, the agent receives an aggregate score for the actions performed by minimizing the error and maximizing the forecast accuracy. The proposed model efficiently predicts the crop yield outperforming existing models by preserving the original data distribution with an accuracy of 93.7%.

Comparison of conventional and rapid-acting antidepressants in a rodent probabilistic reversal learning task.

Deficits in reward processing are a central feature of major depressive disorder with patients exhibiting decreased reward learning and altered feedback sensitivity in probabilistic reversal learning tasks. Methods to quantify probabilistic learning in both rodents and humans have been developed, providing translational paradigms for depression research. We have utilised a probabilistic reversal learning task to investigate potential differences between conventional and rapid-acting antidepressants on reward learning and feedback sensitivity. We trained 12 rats in a touchscreen probabilistic reversal learning task before investigating the effect of acute administration of citalopram, venlafaxine, reboxetine, ketamine or scopolamine. Data were also analysed using a Q-learning reinforcement learning model to understand the effects of antidepressant treatment on underlying reward processing parameters. Citalopram administration decreased trials taken to learn the first rule and increased win-stay probability. Reboxetine decreased win-stay behaviour while also decreasing the number of rule changes animals performed in a session. Venlafaxine had no effect. Ketamine and scopolamine both decreased win-stay probability, number of rule changes performed and motivation in the task. Insights from the reinforcement learning model suggested that reboxetine led animals to choose a less optimal strategy, while ketamine decreased the model-free learning rate. These results suggest that reward learning and feedback sensitivity are not differentially modulated by conventional and rapid-acting antidepressant treatment in the probabilistic reversal learning task.