Reinforcement Q-learning Research Articles

Multi-AUV Collaborative Target Recognition Based on Transfer-Reinforcement Learning

Due to the existence of unfavorable factors such as turbid water quality and target occlusion, it is difficult to obtain valid data of target features. Due to the repeated calculation of similar data, the real-time performance of the algorithm is poor. In view of the above problems, this paper proposes a multi-AUV collaborative target recognition method based on transfer-reinforcement learning. The features of the target information which is collected by multi-AUV are fused based on wavelet transformation and affine invariance. The similarity of features is calculated by Mahalanobis distance and the learning model is selected autonomously based on the similarity threshold. Based on the Q-learning reinforcement learning model, the target information under the interference environment is trained intensively, and the effective features are extracted and stored in the source domain, which can reduce the impact of the environmental interference on the target recognition. The feature transfer learning model based on deep confidence network transfers the feature data of the source domain to the target domain, reducing the repeated calculation of similar data, and then ensuring the real-time performance of the algorithm. Simulation experiments are conducted in the SUN dataset under five underwater environments (turbid water, target occlusion, insufficient light, complex background, and overlapping targets), and the results demonstrate that the proposed model achieves better performance.

A Generic approach for Pronominal Anaphora and Zero Anaphora resolution in Arabic language

Abstract This paper deals with the resolution of pronominal anaphora and zero anaphora in Arabic language. While researchers have treated the two phenomena separately, we propose a generic approach for both of them. Our resolution system combines a Q-learning reinforcement method and Word Embedding models. The Q-learning method uses syntactic criteria as preference factors to select candidate antecedents. It reinforces the best combination criteria for evaluating candidate antecedents. The Word Embedding models provide semantic similarity measures that help to validate the best antecedent. Our approach is evaluated on different type of Arabic texts and the obtained precision can reach79.37%.

Crop Yield Prediction Using Deep Reinforcement Learning Model for Sustainable Agrarian Applications

Predicting crop yield based on the environmental, soil, water and crop parameters has been a potential research topic. Deep-learning-based models are broadly used to extract significant crop features for prediction. Though these methods could resolve the yield prediction problem there exist the following inadequacies: Unable to create a direct non-linear or linear mapping between the raw data and crop yield values; and the performance of those models highly relies on the quality of the extracted features. Deep reinforcement learning provides direction and motivation for the aforementioned shortcomings. Combining the intelligence of reinforcement learning and deep learning, deep reinforcement learning builds a complete crop yield prediction framework that can map the raw data to the crop prediction values. The proposed work constructs a Deep Recurrent Q-Network model which is a Recurrent Neural Network deep learning algorithm over the Q-Learning reinforcement learning algorithm to forecast the crop yield. The sequentially stacked layers of Recurrent Neural network is fed by the data parameters. The Q- learning network constructs a crop yield prediction environment based on the input parameters. A linear layer maps the Recurrent Neural Network output values to the Q-values. The reinforcement learning agent incorporates a combination of parametric features with the threshold that assist in predicting crop yield. Finally, the agent receives an aggregate score for the actions performed by minimizing the error and maximizing the forecast accuracy. The proposed model efficiently predicts the crop yield outperforming existing models by preserving the original data distribution with an accuracy of 93.7%.

Comparison of conventional and rapid-acting antidepressants in a rodent probabilistic reversal learning task.

Deficits in reward processing are a central feature of major depressive disorder with patients exhibiting decreased reward learning and altered feedback sensitivity in probabilistic reversal learning tasks. Methods to quantify probabilistic learning in both rodents and humans have been developed, providing translational paradigms for depression research. We have utilised a probabilistic reversal learning task to investigate potential differences between conventional and rapid-acting antidepressants on reward learning and feedback sensitivity. We trained 12 rats in a touchscreen probabilistic reversal learning task before investigating the effect of acute administration of citalopram, venlafaxine, reboxetine, ketamine or scopolamine. Data were also analysed using a Q-learning reinforcement learning model to understand the effects of antidepressant treatment on underlying reward processing parameters. Citalopram administration decreased trials taken to learn the first rule and increased win-stay probability. Reboxetine decreased win-stay behaviour while also decreasing the number of rule changes animals performed in a session. Venlafaxine had no effect. Ketamine and scopolamine both decreased win-stay probability, number of rule changes performed and motivation in the task. Insights from the reinforcement learning model suggested that reboxetine led animals to choose a less optimal strategy, while ketamine decreased the model-free learning rate. These results suggest that reward learning and feedback sensitivity are not differentially modulated by conventional and rapid-acting antidepressant treatment in the probabilistic reversal learning task.

An intelligent wireless channel allocation in HAPS 5G communication system based on reinforcement learning

Channel allocation is the prerequisite for the HAPS (high-altitude platform station) 5G communication network to transmit information. An intelligent wireless channel allocation algorithm for HAPS 5G massive MIMO (multiple-input multiple-output) system based on reinforcement learning was proposed. Q-learning reinforcement learning algorithm and the back-propagation neural network were combined, which enabled HAPS 5G massive MIMO systems autonomous learn according to the environment and allocate channel resources of the system efficiently. Agents perceive the state information in the channel environment through continuous interaction with the channel environment, learn from the environment state to the action mapping, and use the back-propagation neural network for learning training, use the neural network instead of the Q value table, and train the network with each Q update as a training example to update the evaluation function. It effectively improves the overall performance of the system.

Decision-making method for vehicle longitudinal automatic driving based on reinforcement Q-learning

In the development of autonomous driving, decision-making has become one of the technical difficulties. Traditional rule-based decision-making methods lack adaptive capacity when dealing with unfamiliar and complex traffic conditions. However, reinforcement learning shows the potential to solve sequential decision problems. In this article, an independent decision-making method based on reinforcement Q-learning is proposed. First, a Markov decision process model is established by analysis of car-following. Then, the state set and action set are designed by the synthesized consideration of driving simulator experimental results and driving risk principles. Furthermore, the reinforcement Q-learning algorithm is developed mainly based on the reward function and update function. Finally, the feasibility is verified through random simulation tests, and the improvement is made by comparative analysis with a traditional method.

Context aware Q-Learning-based model for decision support in the negotiation of energy contracts

Automated negotiation plays a crucial role in the decision support for bilateral energy transactions. In fact, an adequate analysis of past actions of opposing negotiators can improve the decision-making process of market players, allowing them to choose the most appropriate parties to negotiate with in order to increase their outcomes. This paper proposes a new model to estimate the expected prices that can be achieved in bilateral contracts under a specific context, enabling adequate risk management in the negotiation process. The proposed approach is based on an adaptation of the Q-Learning reinforcement learning algorithm to choose the best scenario (set of forecast contract prices) from a set of possible scenarios that are determined using several forecasting and estimation methods. The learning process assesses the probability of occurrence of each scenario, by comparing each expected scenario with the real scenario. The final chosen scenario is the one that presents the higher expected utility value. Besides, the learning method can determine which is the best scenario for each context, since the behaviour of players can change according to the negotiation environment. Consequently, these conditions influence the final contract price of negotiations. This approach allows the supported player to be prepared for the negotiation scenario that is the most probable to represent a reliable approximation of the actual negotiation environment.

Reinforcement Q-learning based on Multirate Generalized Policy Iteration and Its Application to a 2-DOF Helicopter

In this paper, we propose a novel Q-learning method based on multirate generalized policy iteration (MGPI) for unknown discrete-time (DT) linear quadratic regulation (LQR) problems. Q-learning is an effective scheme for unknown dynamical systems because it does not require any knowledge of the system dynamics to solve optimal control problems. By applying the MGPI concept, which is an extension of basic GPI with multirate time horizon steps, a new Q-learning algorithm is proposed for solving the LQR problem. Further, it is proven that the proposed algorithm converges to an optimal solution i.e., it learns the optimal control policy iteratively using the states and the control-input information. Finally, we employ the two degree-of-freedom helicopter model to verify the effectiveness of the proposed method and investigate its convergence properties.

Data-driven model-free slip control of anti-lock braking systems using reinforcement Q-learning

Abstract This paper proposes the design and implementation of a model-free tire slip control for a fast and highly nonlinear Anti-lock Braking System (ABS). A reinforcement Q-learning optimal control approach is inserted in a batch neural fitted scheme using two neural networks to approximate the value function and the controller, respectively. The transition samples required for learning high performance control can be collected by interacting with the process either by online exploiting the current iteration controller (or policy) under an e-greedy exploration strategy, or by using data collected under any other controller that is capable of ensuring efficient exploration of the action-state space. Both approaches are highlighted in the paper. Fortunately, the ABS process fits this type of learning-by-interaction because it does not need an initial stabilizing controller. The validation case studies conducted on a real laboratory setup reveal that high control system performance can be achieved using the proposed approaches. Insightful comments on the observed control behavior are offered along with performance comparisons with several types of model-based and model-free controllers including relay, model-based optimal PI, an original model-free neural network state-feedback VRFT controller and a model-free neural network adaptive actor-critic one. With the ability to improve control performance starting from different supervisory controllers or to learn high performance controllers from scratch, the proposed Q-learning optimal control approach proves its performance in a wide operating range and is therefore recommended to its industrial application on ABS.

Model-Free control performance improvement using virtual reference feedback tuning and reinforcement Q-learning

ABSTRACTThis paper proposes the combination of two model-free controller tuning techniques, namely linear virtual reference feedback tuning (VRFT) and nonlinear state-feedback Q-learning, referred to as a new mixed VRFT-Q learning approach. VRFT is first used to find stabilising feedback controller using input-output experimental data from the process in a model reference tracking setting. Reinforcement Q-learning is next applied in the same setting using input-state experimental data collected under perturbed VRFT to ensure good exploration. The Q-learning controller learned with a batch fitted Q iteration algorithm uses two neural networks, one for the Q-function estimator and one for the controller, respectively. The VRFT-Q learning approach is validated on position control of a two-degrees-of-motion open-loop stable multi input-multi output (MIMO) aerodynamic system (AS). Extensive simulations for the two independent control channels of the MIMO AS show that the Q-learning controllers clearly improve performance over the VRFT controllers.

Development of a Smart Home Context-aware Application: A Machine Learning based Approach

Context-awareness is an important characteristic of smart home. Several methods are used in context-aware application to provide services. The main target of smart home is to predict the demand of home users and proactively provide the proper services by computing user’s context information. In this paper, we present a context-aware application which can provide service according to predefined choice of user. It uses Mahalanobis distance based k nearest neighbors classifier technique for inference of predefined service. We combine the features of supervised and unsupervised machine learning in the proposed application. This application can also adapt itself when the choice of user is changed by using Q-learning reinforcement learning algorithm.

Reinforcement learning-based prediction approach for distributed Dynamic Data-Driven Application Systems

With the current advances in cloud and distributed system technology, data have become ubiquitous and their dynamics has increased. It is an extreme challenge to find the interdependencies among distributed data in order to dynamically manage and predict the trend within large amounts of data sources. This paper proposes a new distributed dynamic data-driven model and strategy to direct and evaluate the interlinked data sets in Dynamic Data-Driven Application Systems (DDDAS). The underlying technique involves the introduction of a reinforcement Q-Learning approach which includes search strategies to determine how to drill and drive a series of highly dependent data in order to enhance prediction accuracy and efficiency. It can tackle dynamic data issues in a real-time, dynamic and resource-bounded environment. The proposed framework is a comprehensive skeleton for modeling complex, flexible and dynamic tasks in a distributed environment for solving DDDAS problems. In simulation, the new model utilizes individual sensors, distributed databases and predictors in Dynamic Data Stream Nodes with multiple dimensional variables which can be instantiated to explore the search space, thereby improving the search convergence. This study shows the effectiveness and applicability of using the technique in the analysis of typhoon rainfall data. The result shows that the proposed approach performed better than traditional linear regression approaches, reducing the error rate by 36.34 %.

Obstacle avoidance of redundant manipulators using neural networks based reinforcement learning

This paper proposes a new approach for solving the problem of obstacle avoidance during manipulation tasks performed by redundant manipulators. The developed solution is based on a double neural network that uses Q-learning reinforcement technique. Q-learning has been applied in robotics for attaining obstacle free navigation or computing path planning problems. Most studies solve inverse kinematics and obstacle avoidance problems using variations of the classical Jacobian matrix approach, or by minimizing redundancy resolution of manipulators operating in known environments. Researchers who tried to use neural networks for solving inverse kinematics often dealt with only one obstacle present in the working field. This paper focuses on calculating inverse kinematics and obstacle avoidance for complex unknown environments, with multiple obstacles in the working field. Q-learning is used together with neural networks in order to plan and execute arm movements at each time instant. The algorithm developed for general redundant kinematic link chains has been tested on the particular case of PowerCube manipulator. Before implementing the solution on the real robot, the simulation was integrated in an immersive virtual environment for better movement analysis and safer testing. The study results show that the proposed approach has a good average speed and a satisfying target reaching success rate.

Mechanisms Underlying Dopamine-Mediated Reward Bias in Compulsive Behaviors

Pathological behaviors such as problem gambling or shopping are characterized by compulsive choice despite alternative options and negative costs. Reinforcement learning algorithms allow a computation of prediction error, a comparison of actual and expected outcomes, which updates our predictions and influences our subsequent choices. Using a reinforcement learning model, we show data consistent with the idea that dopamine agonists in susceptible individuals with Parkinson's disease increase the rate of learning from gain outcomes. Dopamine agonists also increase striatal prediction error activity, thus signifying a "better than expected" outcome. Thus, our findings are consistent with a model whereby a distorted estimation of the gain cue underpins a choice bias toward gains.

FUZZY STATE AGGREGATION AND POLICY HILL CLIMBING FOR STOCHASTIC ENVIRONMENTS

Reinforcement learning is one of the more attractive machine learning technologies, due to its unsupervised learning structure and ability to continually learn even as the operating environment changes. Additionally, by applying reinforcement learning to multiple cooperative software agents (a multi-agent system) not only allows each individual agent to learn from its own experience, but also opens up the opportunity for the individual agents to learn from the other agents in the system, thus accelerating the rate of learning. This research presents the novel use of fuzzy state aggregation, as the means of function approximation, combined with the fastest policy hill climbing methods of Win or Lose Fast (WoLF) and policy-dynamics based WoLF (PD-WoLF). The combination of fast policy hill climbing and fuzzy state aggregation function approximation is tested in two stochastic environments: Tileworld and the simulated robot soccer domain, RoboCup. The Tileworld results demonstrate that a single agent using the combination of FSA and PHC learns quicker and performs better than combined fuzzy state aggregation and Q-learning reinforcement learning alone. Results from the multi-agent RoboCup domain again illustrate that the policy hill climbing algorithms perform better than Q-learning alone in a multi-agent environment. The learning is further enhanced by allowing the agents to share their experience through a weighted strategy sharing.