Simple Reinforcement Learning Research Articles

Reinforcement learning as a robotics-inspired framework for insect navigation: from spatial representations to neural implementation

Bees are among the master navigators of the insect world. Despite impressive advances in robot navigation research, the performance of these insects is still unrivaled by any artificial system in terms of training efficiency and generalization capabilities, particularly considering the limited computational capacity. On the other hand, computational principles underlying these extraordinary feats are still only partially understood. The theoretical framework of reinforcement learning (RL) provides an ideal focal point to bring the two fields together for mutual benefit. In particular, we analyze and compare representations of space in robot and insect navigation models through the lens of RL, as the efficiency of insect navigation is likely rooted in an efficient and robust internal representation, linking retinotopic (egocentric) visual input with the geometry of the environment. While RL has long been at the core of robot navigation research, current computational theories of insect navigation are not commonly formulated within this framework, but largely as an associative learning process implemented in the insect brain, especially in the mushroom body (MB). Here we propose specific hypothetical components of the MB circuit that would enable the implementation of a certain class of relatively simple RL algorithms, capable of integrating distinct components of a navigation task, reminiscent of hierarchical RL models used in robot navigation. We discuss how current models of insect and robot navigation are exploring representations beyond classical, complete map-like representations, with spatial information being embedded in the respective latent representations to varying degrees.

Reinforcement learning based fractional fuzzy controller for photovoltaic systems

This article presents an innovative artificial nonlinear control technique for photovoltaic (PV) systems. A boost converter extracts maximum power from the PV system under different weather conditions. A fuzzy logic maximum power point (MPP) finding algorithm is used to determine the reference voltage of the PV system under various atmospheric conditions. It is assumed that the nonlinear dynamic model, parameters, and uncertainties of the converter are unknown. To estimate a precise model that contains nonlinearities, model uncertainties, and disturbances of the converter, proximal policy optimization (PPO) with a special reward function is applied. PPO is optimal in policy learning and sampling, fast to implement, and is a simple reinforcement learning (RL) algorithm. Then, estimations of dynamics and uncertainties are used to enhance the robustness of the suggested fractional order sliding mode controller. Furthermore, the controller coefficients are optimally adjusted by the PPO algorithm. The Lyapunov theorem is used for proving the control system's closed-loop stability. Also, the proposed methods can be generalized and used for other control systems. The simulation is conducted by MATLAB to evaluate the suggested controller's performance effectiveness. Robustness against the presence of parameter uncertainty is investigated. Moreover, the suggested technique is also compared with sliding mode controller.

A novel technique for delineating the effect of variation in the learning rate on the neural correlates of reward prediction errors in model-based fMRI.

Computational models play an increasingly important role in describing variation in neural activation in human neuroimaging experiments, including evaluating individual differences in the context of psychiatric neuroimaging. In particular, reinforcement learning (RL) techniques have been widely adopted to examine neural responses to reward prediction errors and stimulus or action values, and how these might vary as a function of clinical status. However, there is a lack of consensus around the importance of the precision of free parameter estimation for these methods, particularly with regard to the learning rate. In the present study, I introduce a novel technique which may be used within a general linear model (GLM) to model the effect of mis-estimation of the learning rate on reward prediction error (RPE)-related neural responses. Simulations employed a simple RL algorithm, which was used to generate hypothetical neural activations that would be expected to be observed in functional magnetic resonance imaging (fMRI) studies of RL. Similar RL models were incorporated within a GLM-based analysis method including derivatives, with individual differences in the resulting GLM-derived beta parameters being evaluated with respect to the free parameters of the RL model or being submitted to other validation analyses. Initial simulations demonstrated that the conventional approach to fitting RL models to RPE responses is more likely to reflect individual differences in a reinforcement efficacy construct (lambda) rather than learning rate (alpha). The proposed method, adding a derivative regressor to the GLM, provides a second regressor which reflects the learning rate. Validation analyses were performed including examining another comparable method which yielded highly similar results, and a demonstration of sensitivity of the method in presence of fMRI-like noise. Overall, the findings underscore the importance of the lambda parameter for interpreting individual differences in RPE-coupled neural activity, and validate a novel neural metric of the modulation of such activity by individual differences in the learning rate. The method is expected to find application in understanding aberrant reinforcement learning across different psychiatric patient groups including major depression and substance use disorder.

Intelligent Vehicle Decision-Making and Trajectory Planning Method Based on Deep Reinforcement Learning in the Frenet Space.

The complexity inherent in navigating intricate traffic environments poses substantial hurdles for intelligent driving technology. The continual progress in mapping and sensor technologies has equipped vehicles with the capability to intricately perceive their exact position and the intricate interplay among surrounding traffic elements. Building upon this foundation, this paper introduces a deep reinforcement learning method to solve the decision-making and trajectory planning problem of intelligent vehicles. The method employs a deep learning framework for feature extraction, utilizing a grid map generated from a blend of static environmental markers such as road centerlines and lane demarcations, in addition to dynamic environmental cues including vehicle positions across varied lanes, all harmonized within the Frenet coordinate system. The grid map serves as the input for the state space, and the input for the action space comprises a vector encompassing lane change timing, velocity, and vertical displacement at the lane change endpoint. To optimize the action strategy, a reinforcement learning approach is employed. The feasibility, stability, and efficiency of the proposed method are substantiated via experiments conducted in the CARLA simulator across diverse driving scenarios, and the proposed method can increase the average success rate of lane change by 6.8% and 13.1% compared with the traditional planning control algorithm and the simple reinforcement learning method.

A Hierarchical Framework for Quadruped Robots Gait Planning Based on DDPG.

In recent years, significant progress has been made in employing reinforcement learning for controlling legged robots. However, a major challenge arises with quadruped robots due to their continuous states and vast action space, making optimal control using simple reinforcement learning controllers particularly challenging. This paper introduces a hierarchical reinforcement learning framework based on the Deep Deterministic Policy Gradient (DDPG) algorithm to achieve optimal motion control for quadruped robots. The framework consists of a high-level planner responsible for generating ideal motion parameters, a low-level controller using model predictive control (MPC), and a trajectory generator. The agents within the high-level planner are trained to provide the ideal motion parameters for the low-level controller. The low-level controller uses MPC and PD controllers to generate the foot-end force and calculates the joint motor torque through inverse kinematics. The simulation results show that the motion performance of the trained hierarchical framework is superior to that obtained using only the DDPG method.

Reinforcement Learning Under Uncertainty: Expected Versus Unexpected Uncertainty and State Versus Reward Uncertainty

Two prominent types of uncertainty that have been studied extensively are expected and unexpected uncertainty. Studies suggest that humans are capable of learning from reward under both expected and unexpected uncertainty when the source of variability is the reward. How do people learn when the source of uncertainty is the environment’s state and the rewards themselves are deterministic? How does their learning compare with the case of reward uncertainty? The present study addressed these questions using behavioural experimentation and computational modelling. Experiment 1 showed that human subjects were generally able to use reward feedback to successfully learn the task rules under state uncertainty, and were able to detect a non-signalled reversal of stimulus-response contingencies. Experiment 2, which combined all four types of uncertainties—expected versus unexpected uncertainty, and state versus reward uncertainty—highlighted key similarities and differences in learning between state and reward uncertainties. We found that subjects performed significantly better in the state uncertainty condition, primarily because they explored less and improved their state disambiguation. We also show that a simple reinforcement learning mechanism that ignores state uncertainty and updates the state-action value of only the identified state accounted for the behavioural data better than both a Bayesian reinforcement learning model that keeps track of belief states and a model that acts based on sampling from past experiences. Our findings suggest a common mechanism supports reward-based learning under state and reward uncertainty.

Reinforcement learning of a multi-link swimmer at low Reynolds numbers

The use of machine learning techniques in the development of microscopic swimmers has drawn considerable attention in recent years. In particular, reinforcement learning has been shown useful in enabling swimmers to learn effective propulsion strategies through its interactions with the surroundings. In this work, we apply a reinforcement learning approach to identify swimming gaits of a multi-link model swimmer. The swimmer consists of multiple rigid links connected serially with hinges, which can rotate freely to change the relative angles between neighboring links. Purcell [“Life at low Reynolds number,” Am. J. Phys. 45, 3 (1977)] demonstrated how the particular case of a three-link swimmer (now known as Purcell's swimmer) can perform a prescribed sequence of hinge rotation to generate self-propulsion in the absence of inertia. Here, without relying on any prior knowledge of low-Reynolds-number locomotion, we first demonstrate the use of reinforcement learning in identifying the classical swimming gaits of Purcell's swimmer for case of three links. We next examine the new swimming gaits acquired by the learning process as the number of links increases. We also consider the scenarios when only a single hinge is allowed to rotate at a time and when simultaneous rotation of multiple hinges is allowed. We contrast the difference in the locomotory gaits learned by the swimmers in these scenarios and discuss their propulsion performance. Taken together, our results demonstrate how a simple reinforcement learning technique can be applied to identify both classical and new swimming gaits at low Reynolds numbers.

Safe Reinforcement Learning for Single Train Trajectory Optimization via Shield SARSA

The single train trajectory optimization, also known as speed profile optimization (SPO), is a traditional problem to minimize the traction energy consumption of trains. As a kind of optimal method, reinforcement learning (RL) has been used to solve the SPO problem. In the learning process of a common RL algorithm, a soft constraint (punishment) is always used to keep the agent away from unsafe states. However, a soft constraint can not guarantee and explain the safety of the result. For the SPO problem, it means that the optimized speed profile obtained by a simple RL may break the speed limit which is unacceptable in reality. This paper proposes a protection mechanism called Shield and constructs a Shield SARSA ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${S}$ </tex-math></inline-formula> -SARSA) algorithm to protect the learning process of the high-speed train. Four different reward functions are used to compare the protective efficacy between the proposed algorithm and the soft constraint. The numerical experiments based on the line data from Wuxi East to Suzhou North verify the protective efficacy and effectiveness.

Cholinergic activity reflects reward expectations and predicts behavioral responses

Basal forebrain cholinergic neurons (BFCNs) play an important role in associative learning, suggesting that BFCNs may participate in processing stimuli that predict future outcomes. However, the impact of outcome probabilities on BFCN activity remained elusive. Therefore, we performed bulk calcium imaging and recorded spiking of identified cholinergic neurons from the basal forebrain of mice performing a probabilistic Pavlovian cued outcome task. BFCNs responded more to sensory cues that were often paired with reward. Reward delivery also activated BFCNs, with surprising rewards eliciting a stronger response, whereas punishments evoked uniform positive-going responses. We propose that BFCNs differentially weigh predictions of positive and negative reinforcement, reflecting divergent relative salience of forecasting appetitive and aversive outcomes, partially explained by a simple reinforcement learning model of a valence-weighed unsigned prediction error. Finally, the extent of cue-driven cholinergic activation predicted subsequent decision speed, suggesting that the expectation-gated cholinergic firing is instructive to reward-seeking behaviors.

The computational roots of positivity and confirmation biases in reinforcement learning.

Humans do not integrate new information objectively: outcomes carrying a positive affective value and evidence confirming one's own prior belief are overweighed. Until recently, theoretical and empirical accounts of the positivity and confirmation biases assumed them to be specific to 'high-level' belief updates. We present evidence against this account. Learning rates in reinforcement learning (RL) tasks, estimated across different contexts and species, generally present the same characteristic asymmetry, suggesting that belief and value updating processes share key computational principles and distortions. This bias generates over-optimistic expectations about the probability of making the right choices and, consequently, generates over-optimistic reward expectations. We discuss the normative and neurobiological roots of these RL biases and their position within the greater picture of behavioral decision-making theories.

Reduced Primacy Bias in Autism during Early Sensory Processing.

Recent theories of autism propose that a core deficit in autism would be a less context-sensitive weighting of prediction errors. There is also first support for this hypothesis on an early sensory level. However, an open question is whether this decreased context sensitivity is caused by faster updating of one's model of the world (i.e., higher weighting of new information), proposed by predictive coding theories, or slower model updating. Here, we differentiated between these two hypotheses by investigating how first impressions shape the mismatch negativity (MMN), reflecting early sensory prediction error processing. An autism and matched control group of human adults (both n = 27, 8 female) were compared on the multi-timescale MMN paradigm, in which tones were presented that were either standard (frequently occurring) or deviant (rare), and these roles reversed every block. A well-replicated observation is that the initial model (i.e., the standard and deviant sound in the first block) influences MMN amplitudes in later blocks. If autism is characterized by faster model updating, and thus a smaller primacy bias, we hypothesized (and demonstrate using a simple reinforcement learning model) that their MMN amplitudes should be less influenced by the initial context. In line with this hypothesis, we found that MMN responses in the autism group did not differ between the initial deviant and initial standard sounds as they did in the control group. These findings are consistent with the idea that autism is characterized by faster model updating during early sensory processing, as proposed by predictive coding accounts of autism.SIGNIFICANCE STATEMENT Recent theories of autism propose that a core deficit in autism is that they are faster to update their models of the world based on new sensory information. Here, we tested this hypothesis by investigating how first impressions shape brain responses during early sensory processing, and hypothesized that individuals with autism would be less influenced by these first impressions. In line with earlier studies, our results show that early sensory processing was influenced by first impressions in a control group. However, this was not the case in an autism group. This suggests that individuals with autism are faster to abandon their initial model, and is consistent with the proposal that they are faster to update their models of the world.

Model-based learning retrospectively updates model-free values

Reinforcement learning (RL) is widely regarded as divisible into two distinct computational strategies. Model-free learning is a simple RL process in which a value is associated with actions, whereas model-based learning relies on the formation of internal models of the environment to maximise reward. Recently, theoretical and animal work has suggested that such models might be used to train model-free behaviour, reducing the burden of costly forward planning. Here we devised a way to probe this possibility in human behaviour. We adapted a two-stage decision task and found evidence that model-based processes at the time of learning can alter model-free valuation in healthy individuals. We asked people to rate subjective value of an irrelevant feature that was seen at the time a model-based decision would have been made. These irrelevant feature value ratings were updated by rewards, but in a way that accounted for whether the selected action retrospectively ought to have been taken. This model-based influence on model-free value ratings was best accounted for by a reward prediction error that was calculated relative to the decision path that would most likely have led to the reward. This effect occurred independently of attention and was not present when participants were not explicitly told about the structure of the environment. These findings suggest that current conceptions of model-based and model-free learning require updating in favour of a more integrated approach. Our task provides an empirical handle for further study of the dialogue between these two learning systems in the future.

Asymmetric reinforcement learning facilitates human inference of transitive relations

Humans and other animals are capable of inferring never-experienced relations (for example, A > C) from other relational observations (for example, A > B and B > C). The processes behind such transitive inference are subject to intense research. Here we demonstrate a new aspect of relational learning, building on previous evidence that transitive inference can be accomplished through simple reinforcement learning mechanisms. We show in simulations that inference of novel relations benefits from an asymmetric learning policy, where observers update only their belief about the winner (or loser) in a pair. Across four experiments (n = 145), we find substantial empirical support for such asymmetries in inferential learning. The learning policy favoured by our simulations and experiments gives rise to a compression of values that is routinely observed in psychophysics and behavioural economics. In other words, a seemingly biased learning strategy that yields well-known cognitive distortions can be beneficial for transitive inferential judgements.

EasyRL: A Simple and Extensible Reinforcement Learning Framework

In recent years, Reinforcement Learning (RL), has become a popular field of study as well as a tool for enterprises working on cutting-edge artificial intelligence research. To this end, many researchers have built RL frameworks such as openAI Gym, and KerasRL for ease of use. While these works have made great strides towards bringing down the barrier of entry for those new to RL, we propose a much simpler framework called EasyRL, by providing an interactive graphical user interface for users to train and evaluate RL agents. As it is entirely graphical, EasyRL does not require programming knowledge for training and testing simple built-in RL agents. EasyRL also supports custom RL agents and environments, which can be highly beneficial for RL researchers in evaluating and comparing their RL models.

Rigid reduced successor representation as a potential mechanism for addiction.

Difficulty in cessation of drinking, smoking, or gambling has been widely recognized. Conventional theories proposed relative dominance of habitual over goal‐directed control, but human studies have not convincingly supported them. Referring to the recently suggested “successor representation (SR)” of states that enables partially goal‐directed control, we propose a dopamine‐related mechanism that makes resistance to habitual reward‐obtaining particularly difficult. We considered that long‐standing behavior towards a certain reward without resisting temptation can (but not always) lead to a formation of rigid dimension‐reduced SR based on the goal state, which cannot be updated. Then, in our model assuming such rigid reduced SR, whereas no reward prediction error (RPE) is generated at the goal while no resistance is made, a sustained large positive RPE is generated upon goal reaching once the person starts resisting temptation. Such sustained RPE is somewhat similar to the hypothesized sustained fictitious RPE caused by drug‐induced dopamine. In contrast, if rigid reduced SR is not formed and states are represented individually as in simple reinforcement learning models, no sustained RPE is generated at the goal. Formation of rigid reduced SR also attenuates the resistance‐dependent decrease in the value of the cue for behavior, makes subsequent introduction of punishment after the goal ineffective, and potentially enhances the propensity of nonresistance through the influence of RPEs via the spiral striatum‐midbrain circuit. These results suggest that formation of rigid reduced SR makes cessation of habitual reward‐obtaining particularly difficult and can thus be a mechanism for addiction, common to substance and nonsubstance reward.

Advancements in 5G New Radio TDD Cross Link Interference Mitigation

This article presents enhancements that further improve the performance of 5G New Radio time division duplex (TDD) operation and performance. The focus is on mechanisms to mitigate co-channel cross link interference (CLI) between neighboring cells, providing pointers also to recent advances in remote interference mitigation between cells separated by tens to hundreds of kilometers. A sophisticated framework where User Equipment's (UEs) are used as sensors detecting CLI problems is outlined, and the underlying design rationales are presented. Solutions for simple coordination of TDD radio frame configurations between network elements are described. Solutions based on simple reinforcement learning algorithms for the network to adjust the TDD radio frame configuration are proposed, and validated for both macro and indoor factory deployments, considering both ultra-reliable low latency communication (URLLC) and enhanced mobile broadband (eMBB) traffic. Results from extensive dynamic system-level simulations confirm that such solutions offer attractive benefits.

Application of Machine Learning Algorithms for Solving Problems in the Oil and Gas Sector

The paper examines the life cycle of field development, analyzes the processes of the field development design stage for the application of machine learning methods. For each process, relevant problems are highlighted, existing solutions based on machine learning methods, ideas and problems are proposed that could be effectively solved by machine learning methods. For the main part of the processes, examples of solutions are briefly described; the advantages and disadvantages of the approaches are identified. The most common solution method is feed-forward neural networks. Subject to preliminary normalization of the input data, this is the most versatile algorithm for regression and classification problems. However, in the problem of selecting wells for hydraulic fracturing, a whole ensemble of machine learning models was used, where, in addition to a neural network, there was a random forest, gradient boosting and linear regression. For the problem of optimizing the placement of a grid of oil wells, the disadvantages of existing solutions based on a neural network and a simple reinforcement learning approach based on Markov decision-making process are identified. A deep reinforcement learning algorithm called Alpha Zero is proposed, which has previously shown significant results in the role of artificial intelligence for games. This algorithm is a decision tree search that directs the neural network: only those branches that have received the best estimates from the neural network are considered more thoroughly. The paper highlights the similarities between the tasks for which Alpha Zero was previously used, and the task of optimizing the placement of a grid of oil producing wells. Conclusions are made about the possibility of using and modifying the algorithm of the optimization problem being solved. Аn approach is proposed to take into account symmetric states in a Monte Carlo tree to reduce the number of required simulations.

Models of benthic bipedalism.

Walking is a common bipedal and quadrupedal gait and is often associated with terrestrial and aquatic organisms. Inspired by recent evidence of the neural underpinnings of primitive aquatic walking in the little skate Leucoraja erinacea, we introduce a theoretical model of aquatic walking that reveals robust and efficient gaits with modest requirements for body morphology and control. The model predicts undulatory behaviour of the system body with a regular foot placement pattern, which is also observed in the animal, and additionally predicts the existence of gait bistability between two states, one with a large energetic cost for locomotion and another associated with almost no energetic cost. We show that these can be discovered using a simple reinforcement learning scheme. To test these theoretical frameworks, we built a bipedal robot and show that its behaviours are similar to those of our minimal model: its gait is also periodic and exhibits bistability, with a low efficiency mode separated from a high efficiency mode by a 'jump' transition. Overall, our study highlights the physical constraints on the evolution of walking and provides a guide for the design of efficient biomimetic robots.

Coordinated crawling via reinforcement learning.

Rectilinear crawling locomotion is a primitive and common mode of locomotion in slender soft-bodied animals. It requires coordinated contractions that propagate along a body that interacts frictionally with its environment. We propose a simple approach to understand how this coordination arises in a neuromechanical model of a segmented, soft-bodied crawler via an iterative process that might have both biological antecedents and technological relevance. Using a simple reinforcement learning algorithm, we show that an initial all-to-all neural coupling converges to a simple nearest-neighbour neural wiring that allows the crawler to move forward using a localized wave of contraction that is qualitatively similar to what is observed in Drosophila melanogaster larvae and used in many biomimetic solutions. The resulting solution is a function of how we weight gait regularization in the reward, with a trade-off between speed and robustness to proprioceptive noise. Overall, our results, which embed the brain-body-environment triad in a learning scheme, have relevance for soft robotics while shedding light on the evolution and development of locomotion.

Operation scheduling in a solar thermal system: A reinforcement learning-based framework

Reinforcement learning (RL) provides an alternative method for designing condition-based decision making in engineering systems. In this study, a simple and flexible RL tabular Q-learning framework is employed to identify the optimal operation schedules for a solar hot water system according to action–reward feedback. The system is simulated in TRNSYS software. Three energy sources must supply a building’s hot-water demand: low-cost heat from solar thermal collectors and a heat-recovery chiller, coupled to a conventional heat pump. Key performance indicators are used as rewards for balancing the system’s performance with regard to energy efficiency, heat-load delivery, and operational costs. A sensitivity analysis is performed for different reward functions and meteorological conditions. Optimal schedules are obtained for selected scenarios in January, April, July, and October, according to the dynamic conditions of the system. The results indicate that when solar radiation is widely available (October through April), the nominal operation schedule frequently yields the highest performance. However, the obtained schedule differs when the solar radiation is reduced, for instance, in July. On average, with prioritization of the efficient use of both low-cost energy sources, the performance in July can be on average 21% higher than under nominal schedule-based operation.