Reinforcement Learning Process Research Articles

Reinforcement learning coupled with finite element modeling for facial motion learning

Background and ObjectiveFacial palsy patients or patients with facial transplantation have abnormal facial motion due to altered facial muscle functions and nerve damage. Computer-aided system and physics-based models have been developed to provide objective and quantitative information. However, the predictive capacity of these solutions is still limited to explore the facial motion patterns with emerging properties. The present study aims to couple the reinforcement learning and the finite element modeling for facial motion learning and prediction. MethodsA novel modeling workflow for learning facial motion was developed. A physically-based model of the face within the Artisynth modeling platform was used. Information exchange protocol was proposed to link reinforcement learning and rigid multi-bodies dynamics outcomes. Two reinforcement learning algorithms (deep deterministic policy gradient (DDPG) and Twin-delayed DDPG (TD3)) were used and implemented to drive the simulations of symmetry-oriented and smile movements. Numerical outcomes were compared to experimental observations (Bosphorus database) for evaluation and validation purposes. ResultsAs result, after more than 100 episodes of exploring the environment, the agent starts to learn from previous trials and can find the optimal policy after more than 300 episodes of training. Regarding the symmetry-oriented motion, the muscle excitations predicted by the trained agent help to increase the value of reward from R = -2.06 to R = -0.23, which counts for ∼89% improvement of the symmetry value of the face. For smile-oriented motion, two points at the edge of the mouth move up 0.35 cm, which is within the range of movements estimated from the Bosphorus database (0.4 ± 0.32 cm). ConclusionsThe present study explored the muscle excitation patterns by coupling reinforcement learning with a detailed finite element model of the face. We developed, for the first time, a novel coupling scheme to integrate the finite element simulation into the reinforcement learning process for facial motion learning. As perspectives, this present workflow will be applied for facial palsy and facial transplantation patients to guide and optimize the functional rehabilitation program.

Computational Mechanisms of Osmoregulation: A Reinforcement Learning Model for Sodium Appetite.

Homeostatic control with oral nutrient intake is a vital complex system involving the orderly interactions between the external and internal senses, behavioral control, reward learning, and decision-making. Sodium appetite is a representative system and has been intensively investigated in animal models of homeostatic systems and oral nutrient intake. However, the system-level mechanisms for regulating sodium intake behavior and homeostatic control remain unclear. In the current study, we attempted to provide a mechanistic understanding of sodium appetite behavior by using a computational model, the homeostatic reinforcement learning model, in which homeostatic behaviors are interpreted as reinforcement learning processes. Through simulation experiments, we confirmed that our homeostatic reinforcement learning model successfully reproduced homeostatic behaviors by regulating sodium appetite. These behaviors include the approach and avoidance behaviors to sodium according to the internal states of individuals. In addition, based on the assumption that the sense of taste is a predictor of changes in the internal state, the homeostatic reinforcement learning model successfully reproduced the previous paradoxical observations of the intragastric infusion test, which cannot be explained by the classical drive reduction theory. Moreover, we extended the homeostatic reinforcement learning model to multimodal data, and successfully reproduced the behavioral tests in which water and sodium appetite were mediated by each other. Finally, through an experimental simulation of chemical manipulation in a specific neural population in the brain stem, we proposed a testable hypothesis for the function of neural circuits involving sodium appetite behavior. The study results support the idea that osmoregulation via sodium appetitive behavior can be understood as a reinforcement learning process, and provide a mechanistic explanation for the underlying neural mechanisms of decision-making related to sodium appetite and homeostatic behavior.

Endocannabinoid regulation of behavior in response to negative affective states associated with alcohol abstinence

Alcohol use disorder (AUD) is characterized by elevated levels of alcohol intake followed by negative affective states when individuals halt alcohol consumption. In individuals suffering from AUD, alcohol taking and seeking is often hypothesized to be motivated by negative reinforcement, where individuals continue consuming alcohol in order to avoid the negative affective states that are triggered by abstinence. Females are particularly vulnerable to the effects of AUD and are more likely to drink to regulate this negative affect produced by abstinence. The endogenous cannabinoid (eCB) signaling system is heavily implicated in the mediation of both reward and reinforcement, and alcohol consumption has been shown to be positively correlated with release of the eCB ligand 2‐arachidonoylglycerol (2‐AG) within the reward neurocircuitry. Taking this into consideration, we took a multifaceted approach to investigate the mechanisms by which individuals avoid negative affective symptoms associated with prolonged alcohol abstinence, one of the leading causes of relapse. Using a mouse model of chronic alcohol consumption followed by prolonged forced abstinence (CDFA) and pharmacological enhancement of eCB with JZL184, we were able to examine how altered eCB signaling can affect how females respond to avoid negative outcomes. First, by using negative reinforcement operant conditioning, we show that increased eCB signaling via a single injection of JZL184 causes an increase in acquisition rate as compared to ethanol‐treated and control animals. After reaching acquisition, a difference in behavioral pattern is noted; when given the chance to either escape or avoid the foot shock, JZL‐treated animals consistently chose to escape the shock rather than fully avoid it, possibly indicating a reduced sensitivity to the aversive stimulus. Next, when comparing latency to first bite in the novelty‐suppressed feeding test (NSFT), a common protocol used to measure depression and anxiety, with acquisition rate, we found that, in ethanol‐treated animals, decreased latency to eat is negatively correlated with days to acquisition. Moreover, we are able to reverse this effect with a single dose of JZL184. Together, these data indicate that a single dose of JZL184 has the ability to cause long‐lasting effects on the eCB signaling system and, consequently, the reward circuitry. Moving forward it will be critical to directly link these effects between reinforcement learning and reward processing in order to develop a pharmacological intervention to reduce the negative affective states following alcohol abstinence.

Adaptive fuzzy sliding-mode consensus control of nonlinear under-actuated agents in a near-optimal reinforcement learning framework

This study presents a new framework for merging the Adaptive Fuzzy Sliding-Mode Control (AFSMC) with an off-policy Reinforcement Learning (RL) algorithm to control nonlinear under-actuated agents. In particular, a near-optimal leader-follower consensus is considered, and a new method is proposed using the framework of graphical games. In the proposed technique, the sliding variables’ coefficients are considered adaptively tuned policies to achieve an optimal compromise between the satisfactory tracking performance and the allowable control efforts. Contrary to the conventional off-policy RL algorithms for consensus control of multi-agent systems, the proposed method does not require partial knowledge of the system dynamics to initialize the RL process. Furthermore, an actor-critic fuzzy methodology is employed to approximate optimal policies using the measured input/output data. Therefore, using the tuned sliding vector, the control input for each agent is generated which includes a fuzzy term, a robust term, and a saturation compensating term. In particular, the fuzzy system approximates a nonlinear function, and the robust part of the input compensates for any possible mismatches. Furthermore, the saturation compensating gain prevents instability due to any possible actuator saturation. Based on the local sliding variables, the fuzzy singletons, the bounds of the approximation errors, and the compensating gains are adaptively tuned. Closed-loop asymptotic stability is proved using the second Lyapunov theorem and Barbalat's lemma. The method's efficacy is verified by consensus control of multiple REMUS AUVs in the vertical plane.

Satellite-Assisted UAV Trajectory Control in Hostile Jamming Environments

Satellite and unmanned aerial vehicle (UAV) networks have been introduced as enhanced approaches to provide dynamic control, massive connections and global coverage for future wireless communication systems. This paper considers a coordinated satellite-UAV communication system, where the UAV performs the environmental reconnaissance task with the assistance of satellites in a hostile jamming environment. To fulfill this task, the UAV needs to realize autonomous trajectory control and upload the collected data to the satellite. With the aid of the uploading data, the satellite builds the environment situation map integrating the beam quality, jamming status, and traffic distribution. Accordingly, we propose a closed-loop anti-jamming dynamic trajectory optimization approach, which is divided into three stages. Firstly, a coarse trajectory planning is made according to the limited prior information and preset points. Secondly, the flight control between two adjacent preset points is formulated as a Markov decision process, and reinforcement learning (RL) based automatic flying control algorithms are proposed to explore the unknown hostile environment and realize autonomous and precise trajectory control. Thirdly, based on the collected data during the UAV’s flight, the satellite utilizes an environment situation estimating algorithm to build an environment situation map, which is used to reselect the preset points for the first stage and provide better initialization for the RL process in the second stage. Simulation results verify the validity and superiority of the proposed approach.

Deep-Learning-Based Predictive Control of Battery Management for Frequency Regulation

This paper proposes a deep-learning-based optimal battery management scheme for frequency regulation (FR) by integrating model predictive control (MPC), supervised learning (SL), reinforcement learning (RL), and high-fidelity battery models. By taking advantage of deep neural networks (DNNs), the derived DNN-approximated policy is computationally efficient in online implementation. The design procedure of the proposed scheme consists of two sequential processes: (1) the SL process, in which we first run a simulation with an MPC embedding a low-fidelity battery model to generate a training data set, and then, based on the generated data set, we optimize a DNN-approximated policy using SL algorithms; (2) the RL process, in which we utilize RL algorithms to improve the performance of the DNN-approximated policy by balancing short-term economic incentives and long-term battery degradation. The SL process speeds up the subsequent RL process by providing a good initialization. By utilizing RL algorithms, one prominent property of the proposed scheme is that it can learn from the data generated by simulating the FR policy on the high-fidelity battery simulator to adjust the DNN-approximated policy, which is originally initialized using a low-fidelity battery model. A case study using real-world data of FR signals and prices is performed. Simulation results show that, compared to conventional MPC schemes, the proposed deep-learning-based scheme can effectively achieve higher economic benefits of FR participation while maintaining lower online computational cost.

A self-learning cognitive architecture exploiting causality from rewards

Inspired by the human vision system and learning, we propose a novel cognitive architecture that understands the content of raw videos in terms of objects without using labels. The architecture achieves four objectives: (1) Decomposing raw frames in objects by exploiting foveal vision and memory. (2) Describing the world by projecting objects on an internal canvas. (3) Extracting relevant objects from the canvas by analyzing the causal relation between objects and rewards. (4) Exploiting the information of relevant objects to facilitate the reinforcement learning (RL) process. In order to speed up learning, and better identify objects that produce rewards, the architecture implements learning by causality from the perspective of Wiener and Granger using object trajectories stored in working memory and the time series of external rewards. A novel non-parametric estimator of directed information using Renyi’s entropy is designed and tested. Experiments on three environments show that our architecture extracts most of relevant objects. It can be thought of as ‘understanding’ the world in an object-oriented way. As a consequence, our architecture outperforms state-of-the-art deep reinforcement learning in terms of training speed and transfer learning.

PSTO: Learning Energy-Efficient Locomotion for Quadruped Robots

Energy efficiency is critical for the locomotion of quadruped robots. However, energy efficiency values found in simulations do not transfer adequately to the real world. To address this issue, we present a novel method, named Policy Search Transfer Optimization (PSTO), which combines deep reinforcement learning and optimization to create energy-efficient locomotion for quadruped robots in the real world. The deep reinforcement learning and policy search process are performed by the TD3 algorithm and the policy is transferred to the open-loop control trajectory further optimized by numerical methods, and conducted on the robot in the real world. In order to ensure the high uniformity of the simulation results and the behavior of the hardware platform, we introduce and validate the accurate model in simulation including consistent size and fine-tuning parameters. We then validate those results with real-world experiments on the quadruped robot Ant by executing dynamic walking gaits with different leg lengths and numbers of amplifications. We analyze the results and show that our methods can outperform the control method provided by the state-of-the-art policy search algorithm TD3 and sinusoid function on both energy efficiency and speed.

Efficient Resource Allocation for Multi-Beam Satellite-Terrestrial Vehicular Networks: A Multi-Agent Actor-Critic Method With Attention Mechanism

With the rapid development of intelligent transportation systems, there is an increasing demand for a variety of vehicular services, such as automated driving assistance, emergency alert, infotainment, etc. However, in some situations (e.g., remote areas or maritime scenarios), the terrestrial networks alone cannot serve the vehicular applications very well due to the infrastructure deployment and maintenance issues. Satellite networks have become an effective supplement to terrestrial networks, which complement well in terms of coverage, flexibility, reliability, and availability. In this paper, we consider the low orbit multi-beam satellite-terrestrial networks to serve for vehicles. We model this problem as a cooperative multi-agent reinforcement learning process, where each beam acts as an agent, and the global bandwidth is cooperatively shared among all the agents. A multi-agent actor-critic method with attention mechanism is proposed to allocate resources for vehicles with strict delay requirements and minimum bandwidth consumption. When allocating bandwidth, the channel efficiency, the angle of the beams and the priorities of requests in different regions are also considered. Centralized training and distributed execution is performed in the training of the agents. Extensive simulation results verify the effectiveness of our proposed method, where all the agents can well cooperative to achieve efficient resource allocation on-demand for the vehicles under strictly limited bandwidth resources.

A neural network account of memory replay and knowledge consolidation.

Replay can consolidate memories through offline neural reactivation related to past experiences. Category knowledge is learned across multiple experiences, and its subsequent generalization is promoted by consolidation and replay during rest and sleep. However, aspects of replay are difficult to determine from neuroimaging studies. We provided insights into category knowledge replay by simulating these processes in a neural network which approximated the roles of the human ventral visual stream and hippocampus. Generative replay, akin to imagining new category instances, facilitated generalization to new experiences. Consolidation-related replay may therefore help to prepare us for the future as much as remember the past. Generative replay was more effective in later network layers functionally similar to the lateral occipital cortex than layers corresponding to early visual cortex, drawing a distinction between neural replay and its relevance to consolidation. Category replay was most beneficial for newly acquired knowledge, suggesting replay helps us adapt to changes in our environment. Finally, we present a novel mechanism for the observation that the brain selectively consolidates weaker information, namely a reinforcement learning process in which categories were replayed according to their contribution to network performance. This reinforces the idea of consolidation-related replay as an active rather than passive process.

Automatic Verification Flow Shop Scheduling of Electric Energy Meters Based on an Improved Q-Learning Algorithm

Considering the engineering problem of electric energy meter automatic verification and scheduling, this paper proposes a novel scheduling scheme based on an improved Q-learning algorithm. First, by introducing the state variables and behavior variables, the ranking problem of combinatorial optimization is transformed into a sequential decision problem. Then, a novel reward function is proposed to evaluate the pros and cons of the different strategies. In particular, this paper considers adopting the reinforcement learning algorithm to efficiently solve the problem. In addition, this paper also considers the ratio of exploration and utilization in the reinforcement learning process, and then provides reasonable exploration and utilization through an iterative updating scheme. Meanwhile, a decoupling strategy is introduced to address the restriction of over estimation. Finally, real time data from a provincial electric energy meter automatic verification center are used to verify the effectiveness of the proposed algorithm.

Model-based learning retrospectively updates model-free values

Reinforcement learning (RL) is widely regarded as divisible into two distinct computational strategies. Model-free learning is a simple RL process in which a value is associated with actions, whereas model-based learning relies on the formation of internal models of the environment to maximise reward. Recently, theoretical and animal work has suggested that such models might be used to train model-free behaviour, reducing the burden of costly forward planning. Here we devised a way to probe this possibility in human behaviour. We adapted a two-stage decision task and found evidence that model-based processes at the time of learning can alter model-free valuation in healthy individuals. We asked people to rate subjective value of an irrelevant feature that was seen at the time a model-based decision would have been made. These irrelevant feature value ratings were updated by rewards, but in a way that accounted for whether the selected action retrospectively ought to have been taken. This model-based influence on model-free value ratings was best accounted for by a reward prediction error that was calculated relative to the decision path that would most likely have led to the reward. This effect occurred independently of attention and was not present when participants were not explicitly told about the structure of the environment. These findings suggest that current conceptions of model-based and model-free learning require updating in favour of a more integrated approach. Our task provides an empirical handle for further study of the dialogue between these two learning systems in the future.

A Deep Reinforcement Learning Strategy Combining Expert Experience Guidance for a Fruit-Picking Manipulator

When using deep reinforcement learning algorithms for path planning of a multi-DOF fruit-picking manipulator in unstructured environments, it is much too difficult for the multi-DOF manipulator to obtain high-value samples at the beginning of training, resulting in low learning and convergence efficiency. Aiming to reduce the inefficient exploration in unstructured environments, a reinforcement learning strategy combining expert experience guidance was first proposed in this paper. The ratios of expert experience to newly generated samples and the frequency of return visits to expert experience were studied by the simulation experiments. Some conclusions were that the ratio of expert experience, which declined from 0.45 to 0.35, was more effective in improving learning efficiency of the model than the constant ratio. Compared to an expert experience ratio of 0.35, the success rate increased by 1.26%, and compared to an expert experience ratio of 0.45, the success rate increased by 20.37%. The highest success rate was achieved when the frequency of return visits was 15 in 50 episodes, an improvement of 31.77%. The results showed that the proposed method can effectively improve the model performance and enhance the learning efficiency at the beginning of training in unstructured environments. This training method has implications for the training process of reinforcement learning in other domains.

Active Exploration by Chance-Constrained Optimization for Voltage Regulation with Reinforcement Learning

Voltage regulation in distribution networks encounters a challenge of handling uncertainties caused by the high penetration of photovoltaics (PV). This research proposes an active exploration (AE) method based on reinforcement learning (RL) to respond to the uncertainties by regulating the voltage of a distribution network with battery energy storage systems (BESS). The proposed method integrates engineering knowledge to accelerate the training process of RL. The engineering knowledge is the chance-constrained optimization. We formulate the problem in a chance-constrained optimization with a linear load flow approximation. The optimization results are used to guide the action selection of the exploration for improving training efficiency and reducing the conserveness characteristic. The comparison of methods focuses on how BESSs are used, training efficiency, and robustness under varying uncertainties and BESS sizes. We implement the proposed algorithm, a chance-constrained optimization, and a traditional Q-learning in the IEEE 13 Node Test Feeder. Our evaluation shows that the proposed AE method has a better response to the training efficiency compared to traditional Q-learning. Meanwhile, the proposed method has advantages in BESS usage in conserveness compared to the chance-constrained optimization.

Human engagement providing evaluative and informative advice for interactive reinforcement learning

Interactive reinforcement learning proposes the use of externally sourced information in order to speed up the learning process. When interacting with a learner agent, humans may provide either evaluative or informative advice. Prior research has focused on the effect of human-sourced advice by including real-time feedback on the interactive reinforcement learning process, specifically aiming to improve the learning speed of the agent, while minimising the time demands on the human. This work focuses on answering which of two approaches, evaluative or informative, is the preferred instructional approach for humans. Moreover, this work presents an experimental setup for a human trial designed to compare the methods people use to deliver advice in terms of human engagement. The results obtained show that users giving informative advice to the learner agents provide more accurate advice, are willing to assist the learner agent for a longer time, and provide more advice per episode. Additionally, self-evaluation from participants using the informative approach has indicated that the agent’s ability to follow the advice is higher, and therefore, they feel their own advice to be of higher accuracy when compared to people providing evaluative advice.

An intelligent environment for preventing medication errors in home treatment

This article presents an assistance system based on home sensors, Ambient Intelligence, and Artificial Intelligence, which helps the elderly during their medical treatment at home to reduce medication errors. The sort of medication errors we address are those due to medication omission, wrong dosage or timing, drug–drug interactions. Since the patient may have some physical and/or cognitive disabilities, the proposed solution provides advanced features of self-adaptation and exploits the most cutting edge Artificial Intelligence technologies such as Reinforcement Learning, Deep Learning and Natural Language Processing (NLP) to remind and monitor adherence to the prescribed treatment. In particular, the system offers functions for (i) personalised reminds; i.e. an intelligent agent – called Tutor – self-learns (via Reinforcement Learning) the best way to communicate with the patient; (ii) feedback about the medication the patient is going to take; i.e., another intelligent agent – called Checker- identifies the pillbox that the patient is handling before taking the pill (via Deep Neural Network, Optical Character Recognition, and Barcode Reading); and, (iii) alerts in case of known drug–drug interactions; i.e., an intelligent service – called Advisor – searches for the active principles of the medication (via NLP and Unified Medical Language System (UMLS) RxNorm resources) identified by the Checker and known interactions with other medications of the treatment. The final objective is to remind effectively when and what medication is to be taken, to check that the patient is going to take the correct medication, and to alert if possible drug–drug interactions are identified, remotely reporting about the adherence to the therapy or anomalies to the caregivers and/or doctors. Experimental evaluations show encouraging results in terms of drug recognition and drug–drug interactions identification.

Alleviating parameter-tuning burden in reinforcement learning for large-scale process control

Modern process controllers necessitate high quality models and remedial system re-identification upon performance degradation. Reinforcement Learning (RL) can be a promising replacement for those laborious manual procedures. However, in realistic scenarios time is limited, algorithms that can robustly learn with reduced human-agent interactions or self-exploration e.g. parameter tuning are desired. In practice, a great portion of time in setting up an RL algorithm to properly work is spent on those trial-and-error interactions. To reduce the interaction time, we propose a principled framework to ensure monotonic policy improvement even with underperforming parameters, enhancing the robustness of RL process against parameter setting. We incorporate key ingredients such as random features and factorial policy into monotonic improvement mechanism for learning cautiously in large-scale process control problems. We demonstrate in challenging control problems on the simulated vinyl acetate monomer process that the proposed method robustly learns meaningful policy within a short, fixed learning horizon given various parameter configurations that simulate the interactions, comparing to the other method that can only show good performance specific to a narrow range of parameters.

Hermit Crab Optimizer (HCO): A Novel Meta-heuristic Algorithm

This paper proposes a novel meta-heuristic algorithm (MA), called hermit crab optimizer (HCO), which simulates the swarm intelligence of hermit crabs in nature in finding shells protecting and letting them grow in their lifetime. HCO guides search agents separately and in parallel using new solitary and social search operators. It acts similar to a reinforcement learning process, in which the successful agents and failed ones are treated differently, inspired by the group behavior of hermit crabs and environmental characteristics. Computational experiments with well-known test problems confirm HCO's validity, accuracy, robustness, ability to escape local optima, and balance exploration-exploitation.

Reinforcement Learning-based control using Q-learning and gravitational search algorithm with experimental validation on a nonlinear servo system

This paper presents a novel Reinforcement Learning (RL)-based control approach that uses a combination of a Deep Q-Learning (DQL) algorithm and a metaheuristic Gravitational Search Algorithm (GSA). The GSA is employed to initialize the weights and the biases of the Neural Network (NN) involved in DQL in order to avoid the instability, which is the main drawback of the traditional randomly initialized NNs. The quality of a particular set of weights and biases is measured at each iteration of the GSA-based initialization using a fitness function aiming to achieve the predefined optimal control or learning objective. The data generated during the RL process is used in training a NN-based controller that will be able to autonomously achieve the optimal reference tracking control objective. The proposed approach is compared with other similar techniques which use different algorithms in the initialization step, namely the traditional random algorithm, the Grey Wolf Optimizer algorithm, and the Particle Swarm Optimization algorithm. The NN-based controllers based on each of these techniques are compared using performance indices specific to optimal control as settling time, rise time, peak time, overshoot, and minimum cost function value. Real-time experiments are conducted in order to validate and test the proposed new approach in the framework of the optimal reference tracking control of a nonlinear position servo system. The experimental results show the superiority of this approach versus the other three competing approaches.

Reinforcement Learning for Compensating Power Excursions in Amplified WDM Systems

Wavelength-dependent power excursions in gain-controlled erbium doped fiber amplifiers (EDFA) is a challenging issue in optical networks. We investigate a launch channel power control method using reinforcement learning (RL) to mitigate the power excursions of EDFA systems. A machine learning engine is developed, trained and evaluated with four different policy-gradient RL algorithms that are compared according to two main criteria: achieved power excursion reduction and learning time. Different scenarios are considered with 12-, 24-, 40- active channels at fixed wavelengths and with variable number of active channels (between 12 and 64) assigned randomly at different wavelengths during RL process. We show 62% power excursion reduction in the 40-channel scenario and 28% in the variable scenario, which demonstrates the promising role of online RL approach for controlling power excursion in EDFA systems.