Reinforcement Learning Process Research Articles

Food reinforcement architecture: A framework for impulsive and compulsive overeating and food abuse.

Few reward-based theories address key drivers of susceptibility to food cues and consumption beyond fullness. Decision-making and habit formation are governed by reinforcement-based learning processes that, when overstimulated, can drive unregulated hedonically motivated overeating. Here, a model food reinforcement architecture is proposed that uses fundamental concepts in reinforcement and decision-making to identify maladaptive eating habits that can lead to obesity. This model is unique in that it identifies metabolic drivers of reward and incorporates neuroscience, computational decision-making, and psychology to map overeating and obesity. Food reinforcement architecture identifies two paths to overeating: a propensity for hedonic targeting of food cues contributing to impulsive overeating and lack of satiation that contributes to compulsive overeating. A combination of those paths will result in a conscious and subconscious drive to overeat independent of negative consequences, leading to food abuse and/or obesity. Use of this model to identify aberrant reinforcement learning processes and decision-making systems that can serve as markers of overeating risk may provide an opportunity for early intervention in obesity.

FmLRE: A Low-Resource Relation Extraction Model Based on Feature Mapping Similarity Calculation

Low-resource relation extraction (LRE) aims to extract relations from limited labeled corpora. Existing work takes advantages of self-training or distant supervision to expand the limited labeled data in the data-driven approaches, while the selection bias of pseudo labels may cause the error accumulation in subsequent relation classification. To address this issue, this paper proposes fmLRE, an iterative feedback method based on feature mapping similarity calculation to improve the accuracy of pseudo labels. First, it calculates the similarities between pseudo-label and real-label data of the same category in a feature mapping space based on semantic features of labeled dataset after feature projection. Then, it fine-tunes initial model according to the iterative process of reinforcement learning. Finally, the similarity is used as a threshold for screening high-precision pseudo-labels and the basis for setting different rewards, which also acts as a penalty term for the loss function of relation classifier. Experimental results demonstrate that fmLRE achieves the state-of-the-art performance compared with strong baselines on two public datasets.

Adaptive closed-loop maneuver planning for low-thrust spacecraft using reinforcement learning

Autonomy is an increasingly essential component of future space missions, and new technologies are necessary to accommodate off-nominal occurrences onboard that may pose risks to mission success. Identifying a suitable maneuver plan for onboard, low-thrust mission applications in cislunar space remains challenging, particularly in the case of unanticipated events. This investigation addresses this challenge by demonstrating artificial neural networks as promising tools for accurately estimating startup solutions for a conventional targeting-based iterative guidance and control algorithm. This blended approach produces a robust ‘hybrid’ architecture that simultaneously benefits from the computational simplicity of neural networks and the robustness of differential corrections to satisfy mission requirements. In this paradigm, a multiple shooting scheme is incorporated directly into a reinforcement learning process, tasking the resulting neural network controller with convergence basin identification for trajectory recovery. Rapid low-thrust maneuver planning is demonstrated in a ‘runaway’ spacecraft scenario, where deviation over time from a planned near rectilinear halo orbit path renders stationkeeping ineffective and demands an alternative approach to determine an effective recovery plan. The proposed Neural Network-Initialized Targeting (NNIT) algorithm significantly extends the recovery window compared to a crossing-control orbit maintenance approach and exhibits promising results in identifying both the timing and control components for low-thrust maneuver plans.

Extension of constraint-procedural logic-generated environments for deep Q-learning agent training and benchmarking

Abstract Autonomous robots can be employed in exploring unknown environments and performing many tasks, such as, e.g. detecting areas of interest, collecting target objects, etc. Deep reinforcement learning (RL) is often used to train this kind of robot. However, concerning the artificial environments aimed at testing the robot, there is a lack of available data sets and a long time is needed to create them from scratch. A good data set is in fact usually produced with high effort in terms of cost and human work to satisfy the constraints imposed by the expected results. In the first part of this paper, we focus on the specification of the properties of the solutions needed to build a data set, making the case of environment exploration. In the proposed approach, rather than using imperative programming, we explore the possibility of generating data sets using constraint programming in Prolog. In this phase, geometric predicates describe a virtual environment according to inter-space requirements. The second part of the paper is focused on testing the generated data set in an AI gym via space search techniques. We developed a Neuro-Symbolic agent built from the following: (i) A deep Q-learning component implemented in Python, able to address via RL a search problem in the virtual space; the agent has the goal to explore a generated virtual environment to seek for a target, improving its performance through a RL process. (ii) A symbolic component able to re-address the search when the Q-learning component gets stuck in a part of the virtual environment; these components stimulate the agent to move to and explore other parts of the environment. Wide experimentation has been performed, with promising results, and is reported, to demonstrate the effectiveness of the approach.

Intelligent routing algorithm for wireless sensor networks dynamically guided by distributed neural networks

Using reinforcement learning to adjust the power balance of sensor nodes dynamically is an essential approach for extending the lifetime of wireless sensor networks (WSNs), which makes sensor nodes adapt to changing environments by calculating optimal decision solutions in real time. However, as WSN grows, error propagation cannot keep up with topological changes in the networks, which prevents the sensor nodes’ routing decisions from being timely optimized. In this paper, we propose the DNN-HR algorithm, which optimizes the reinforcement learning process of WSN in three ways. First, this paper constructs a two-level learning structure. Discrete nodes perform reinforcement learning during the interaction with their neighboring nodes. Simultaneously, the neural networks that fit the V values guide the underlying nodes’ learning. Second, the algorithm divides the sensor networks into two-level subspaces according to their geographic locations, and cluster heads are elected in each subspace to form weight incremental propagation trees. Third, each sensor node sends the weight increments it learned to its first-level cluster head. At the first-level cluster head, the weight increments sent by multiple nodes are received and accumulated, and then the results are sent to the second-level cluster head. The base station acquires the weight increments of all the second-level cluster heads and then broadcasts the updated neural network weights to all the WSN nodes. Experiments determine the appropriate heuristic parameters. Compared to other algorithms, the experiments show that our proposed scheme significantly improves the packet delivery rate, node survival rate, and energy standard deviation of nodes.

Artificial Intelligence Technologies in Cardiology

As the world produces exabytes of data, there is a growing need to find new methods that are more suitable for dealing with complex datasets. Artificial intelligence (AI) has significant potential to impact the healthcare industry, which is already on the road to change with the digital transformation of vast quantities of information. The implementation of AI has already achieved success in the domains of molecular chemistry and drug discoveries. The reduction in costs and in the time needed for experiments to predict the pharmacological activities of new molecules is a milestone in science. These successful applications of AI algorithms provide hope for a revolution in healthcare systems. A significant part of artificial intelligence is machine learning (ML), of which there are three main types-supervised learning, unsupervised learning, and reinforcement learning. In this review, the full scope of the AI workflow is presented, with explanations of the most-often-used ML algorithms and descriptions of performance metrics for both regression and classification. A brief introduction to explainable artificial intelligence (XAI) is provided, with examples of technologies that have developed for XAI. We review important AI implementations in cardiology for supervised, unsupervised, and reinforcement learning and natural language processing, emphasizing the used algorithm. Finally, we discuss the need to establish legal, ethical, and methodical requirements for the deployment of AI models in medicine.

Reinforcement learning-based real-time intelligent energy management for hybrid electric vehicles in a model predictive control framework

—This paper proposes a real-time energy management strategy (EMS) for hybrid electric vehicles by incorporating reinforcement learning (RL) in a model predictive control (MPC) framework, which avoids the inherent drawbacks of RL—the excessive learning time and lack of adaptability—and remarkably enhances the real-time performance of MPC. First, the MPC framework for the energy management problem is formulated. In that, a novel long short-term memory (LSTM) neural network is utilized to construct the velocity predictor for a more accurate prediction, and its prediction capability is verified by a comparative analysis. Then, the HEV prediction model and the velocity predictor are regarded as the RL model with which the RL agent can interact. On this basis, the optimal control sequence in the prediction horizon can be learned through model-based RL, but only the first element is actually executed, and the RL process begins anew after the prediction horizon moves forward. In the simulation, the algorithm's convergence is analyzed and the influence of the prediction horizon length is evaluated. Then, the proposed EMS is compared with DP, conventional MPC, and RL method, the results of which demonstrate its performance and adaptability. As last, a hardware-in-the-loop test validates its actual applicability.

A reinforcement learning‐based approach for near‐optimal sliding‐mode control of output‐constrained uncertain nonlinear systems

AbstractThis study presents a near‐optimal learning‐based approach for sliding‐mode control of uncertain nonsquare nonlinear systems subject to output constraints. To achieve a compromise between safety and optimality, a reinforcement learning algorithm is proposed to compute the near‐optimal values of the sliding manifold coefficients. In the reinforcement learning algorithm, only measured input‐output data obtained by experiments or simulations are employed. Furthermore, the presented method does not require partial knowledge of the system dynamics to initialize the reinforcement learning process. By employing the tuned sliding vector, an adaptive fuzzy sliding‐mode control (AFSMC) input, including a fuzzy term and a robust term, is generated. The fuzzy term is used to approximate an unknown nonlinear function and the robust term is designed for mismatch compensation. To guarantee that the output constraints are not violated, the adaptation laws for obtaining the fuzzy singletons and the bounds of the approximation errors are designed based on the barrier Lyapunov functions (BLF) theorem. The closed loop asymptotic stability is theoretically analyzed in the article, while the effectiveness of the method is assessed in simulation.

Multiagent Manuvering with the Use of Reinforcement Learning

This paper presents an approach for defining, solving, and implementing dynamic cooperative maneuver problems in autonomous driving applications. The formulation of these problems considers a set of cooperating cars as part of a multiagent system. A reinforcement learning technique is applied to find a suboptimal policy. The key role in the presented approach is a multiagent maneuvering environment that allows for the simulation of car-like agents within an obstacle-constrained space. Each of the agents is tasked with reaching an individual goal, defined as a specific location in space. The policy is determined during the reinforcement learning process to reach a predetermined goal position for each of the simulated cars. In the experiments, three road scenarios—zipper, bottleneck, and crossroads—were used. The trained policy has been successful in solving the cooperation problem in all scenarios and the positive effects of applying shared rewards between agents have been presented and studied. The results obtained in this work provide a window of opportunity for various automotive applications.

Combining behavioral insights with artificial intelligence: New perspectives for technology assessment

Policy decisions concerning technology applications can have far-reaching societal consequences. Rationality-enhancing procedures are thus essential to ensure that such decisions are in the best interest of society. We propose a novel framework addressing this challenge. It combines a structured approach to decision-making, the mediating assessments protocol (MAP), with artificial intelligence (AI) methods to mitigate human bias and handle uncertainty in a normative manner. We introduce the steps for implementing MAP and discuss how it can be complemented and improved by AI methods such as dynamic programming, reinforcement learning and natural language processing. As a potential practical application, we consider the construction of a new wind park in a community and highlight critical aspects warranting special caution.

Neural Index of Reinforcement Learning Predicts Improved Stimulus-Response Retention under High Working Memory Load.

Human learning and decision-making are supported by multiple systems operating in parallel. Recent studies isolating the contributions of reinforcement learning (RL) and working memory (WM) have revealed a trade-off between the two. An interactive WM/RL computational model predicts that although high WM load slows behavioral acquisition, it also induces larger prediction errors in the RL system that enhance robustness and retention of learned behaviors. Here, we tested this account by parametrically manipulating WM load during RL in conjunction with EEG in both male and female participants and administered two surprise memory tests. We further leveraged single-trial decoding of EEG signatures of RL and WM to determine whether their interaction predicted robust retention. Consistent with the model, behavioral learning was slower for associations acquired under higher load but showed parametrically improved future retention. This paradoxical result was mirrored by EEG indices of RL, which were strengthened under higher WM loads and predictive of more robust future behavioral retention of learned stimulus-response contingencies. We further tested whether stress alters the ability to shift between the two systems strategically to maximize immediate learning versus retention of information and found that induced stress had only a limited effect on this trade-off. The present results offer a deeper understanding of the cooperative interaction between WM and RL and show that relying on WM can benefit the rapid acquisition of choice behavior during learning but impairs retention.SIGNIFICANCE STATEMENT Successful learning is achieved by the joint contribution of the dopaminergic RL system and WM. The cooperative WM/RL model was productive in improving our understanding of the interplay between the two systems during learning, demonstrating that reliance on RL computations is modulated by WM load. However, the role of WM/RL systems in the retention of learned stimulus-response associations remained unestablished. Our results show that increased neural signatures of learning, indicative of greater RL computation, under high WM load also predicted better stimulus-response retention. This result supports a trade-off between the two systems, where degraded WM increases RL processing, which improves retention. Notably, we show that this cooperative interplay remains largely unaffected by acute stress.

Stimulating human prefrontal cortex increases reward learning

Work in computational psychiatry suggests that mood disorders may stem from aberrant reinforcement learning processes. Specifically, it has been proposed that depressed individuals believe that negative events are more informative than positive events, resulting in higher learning rates from negative outcomes (Pulcu and Browning, 2019). In this proof-of-concept study, we investigated whether transcranial direct current stimulation (tDCS) applied to dorsolateral prefrontal cortex, as commonly used in depression treatment trials, might change learning rates for affective outcomes. Healthy adults completed an established reinforcement learning task (Pulcu and Browning, 2017) in which the information content of reward and loss outcomes was manipulated by varying the volatility of stimulus-outcome associations. Learning rates on the tasks were quantified using computational models. Stimulation over dorsolateral prefrontal cortex (DLPFC) but not motor cortex (M1) increased learning rates specifically for reward outcomes. The effects of prefrontal tDCS were cognitive state-dependent: tDCS applied during task performance increased learning rates for wins; tDCS applied before task performance decreased both win and loss learning rates. A replication study confirmed the key finding that tDCS to DLPFC during task performance increased learning rates specifically for rewards. Taken together, these findings demonstrate the potential of tDCS for modulating computational parameters of reinforcement learning that are relevant to mood disorders.

Integrating Two-Level Reinforcement Learning Process for Enhancing Task Scheduling Efficiency in a Complex Problem-Solving Environment

Task scheduling scenarios require the system designers to have complete information about the resources and their capabilities, along with the tasks and their application-specific requirements. An effective task-to-resource mapping strategy will maximize resource utilization under constraints, while minimizing the task waiting time, which will in-turn maximize the task execution efficiency. In this work, a two-level reinforcement learning algorithm for task scheduling is proposed. The algorithm utilizes a deep-intensive learning stage to generate a deployable strategy for task-to-resource mapping. This mapping is re-evaluated at specific execution breakpoints, and the strategy is re-evaluated based on the incremental learning from these breakpoints. In order to perform incremental learning, real-time parametric checking is done on the resources and the tasks; and a new strategy is devised during execution. The mean task waiting time is reduced by 20% when compared with standard algorithms like Dynamic and Integrated Resource Scheduling, Improved Differential Evolution, and Q-learning-based Improved Differential Evolution; while the resource utilization is improved by more than 15%. The algorithm is evaluated on datasets from different domains like Coronavirus disease (COVID-19) datasets of public domain, National Aeronautics and Space Administration (NASA) datasets and others. The proposed method performs consistently on all the datasets.

Imitation Learning-Based Implicit Semantic-Aware Communication Networks: Multi-Layer Representation and Collaborative Reasoning

Semantic communication has recently attracted significant interest from both industry and academia due to its potential to transform the existing data-focused communication architecture towards a more generally intelligent and goal-oriented semantic-aware networking system. Despite its promising potential, semantic communications and semantic-aware networking are still in their infancy. Most existing works focus on transporting and delivering the explicit semantic information, e.g., labels or features of objects, that can be directly identified from the source signal. The original definition of semantics as well as recent results in cognitive neuroscience suggest that it is the implicit semantic information, in particular the hidden relations connecting different concepts and feature items that play the fundamental role in recognizing, communicating, and delivering the real semantic meanings of messages. Motivated by this observation, we propose a novel reasoning-based implicit semantic-aware communication network architecture that allows destination users to directly learn a reasoning mechanism that can automatically generate complex implicit semantic information based on a limited clue information sent by the source users. Our proposed architecture can be implemented in a multi-tier cloud/edge computing networks in which multiple tiers of cloud data center (CDC) and edge servers can collaborate and support efficient semantic encoding, decoding, and implicit semantic interpretation for multiple end-users. We introduce a new multi-layer representation of semantic information taking into consideration both the hierarchical structure of implicit semantics as well as the personalized inference preference of individual users. We model the semantic reasoning process as a reinforcement learning process and then propose an imitation-based semantic reasoning mechanism learning (iRML) solution to learning a reasoning policy that imitates the inference behavior of the source user. A federated graph convolutional network (GCN)-based collaborative reasoning solution is proposed to allow multiple edge servers to jointly construct a shared semantic interpretation model based on decentralized semantic message samples. Extensive experiments have been conducted based on real-world datasets to evaluate the performance of our proposed architecture. Numerical results confirm that iRML offers up to 25.8 dB improvement on the semantic symbol error rate, compared to the semantic-irrelevant communication solutions.

Data-based [formula omitted]gain optimal control for discrete-time system with unknown dynamics

This paper considers the L2gain optimal problem for a class of discrete-time linear time-invariant with state-disturbance feedback controller and unknown system dynamic. Firstly, for a given stabilizing control policy, we establish the relation between the L2gain and a sequence of lower triangle Toeplitz matrices. Meanwhile, we show that the upper bound of optimal L2gain is proportional to the linear correlation degree between the input and disturbance matrices. Secondly, to overcome the obstacle arising from the unknown system dynamics, a data-based reinforcement learning scheme is developed for the optimal control policy by using linear matrix inequality technique and Q-learning with policy iteration. Under certain conditions, we prove that either the reinforcement learning process ends in a finite number of iterations, or the L2gain sequence is strictly monotonically convergent along the iteration axis provided that the disturbance data set can fully activate the closed-loop system. Finally, simulations are given to illustrate the effectiveness of our findings.

Homeostatic Reinforcement Theory Accounts for Sodium Appetitive State- and Taste-Dependent Dopamine Responding

Seeking and consuming nutrients is essential to survival and the maintenance of life. Dynamic and volatile environments require that animals learn complex behavioral strategies to obtain the necessary nutritive substances. While this has been classically viewed in terms of homeostatic regulation, recent theoretical work proposed that such strategies result from reinforcement learning processes. This theory proposed that phasic dopamine (DA) signals play a key role in signaling potentially need-fulfilling outcomes. To examine links between homeostatic and reinforcement learning processes, we focus on sodium appetite as sodium depletion triggers state- and taste-dependent changes in behavior and DA signaling evoked by sodium-related stimuli. We find that both the behavior and the dynamics of DA signaling underlying sodium appetite can be accounted for by a homeostatically regulated reinforcement learning framework (HRRL). We first optimized HRRL-based agents to sodium-seeking behavior measured in rodents. Agents successfully reproduced the state and the taste dependence of behavioral responding for sodium as well as for lithium and potassium salts. We then showed that these same agents account for the regulation of DA signals evoked by sodium tastants in a taste- and state-dependent manner. Our models quantitatively describe how DA signals evoked by sodium decrease with satiety and increase with deprivation. Lastly, our HRRL agents assigned equal preference for sodium versus the lithium containing salts, accounting for similar behavioral and neurophysiological observations in rodents. We propose that animals use orosensory signals as predictors of the internal impact of the consumed good and our results pose clear targets for future experiments. In sum, this work suggests that appetite-driven behavior may be driven by reinforcement learning mechanisms that are dynamically tuned by homeostatic need.

On-Orbit Verification of RL-Based APC Calibrations for Micrometre Level Microwave Ranging System

Micrometre level ranging accuracy between satellites on-orbit relies on the high-precision calibration of the antenna phase center (APC), which is accomplished through properly designed calibration maneuvers batch estimation algorithms currently. However, the unmodeled perturbations of the space dynamic and sensor-induced uncertainty complicated the situation in reality; ranging accuracy especially deteriorated outside the antenna main-lobe when maneuvers performed. This paper proposes an on-orbit APC calibration method that uses a reinforcement learning (RL) process, aiming to provide the high accuracy ranging datum for onboard instruments with micrometre level. The RL process used here is an improved Temporal Difference advantage actor critic algorithm (TDAAC), which mainly focuses on two neural networks (NN) for critic and actor function. The output of the TDAAC algorithm will autonomously balance the APC calibration maneuvers amplitude and APC-observed sensitivity with an object of maximal APC estimation accuracy. The RL-based APC calibration method proposed here is fully tested in software and on-ground experiments, with an APC calibration accuracy of less than 2 mrad, and the on-orbit maneuver data from 11–12 April 2022, which achieved 1–1.5 mrad calibration accuracy after RL training. The proposed RL-based APC algorithm may extend to prove mass calibration scenes with actions feedback to attitude determination and control system (ADCS), showing flexibility of spacecraft payload applications in the future.

Valence precedes value in neural encoding of prediction error.

Event-related potentials that follow feedback in reinforcement learning tasks have been proposed to reflect neural encoding of prediction errors. Prior research has shown that in the interval of 240-340 ms multiple different prediction error encodings appear to co-occur, including a value signal carrying signed quantitative prediction error and a valence signal merely carrying sign. The effects used to identify these two encoders, respectively a sign main effect and a sign × size interaction, do not reliably discriminate them. A full discrimination is made possible by comparing tasks in which the reinforcer available on a given trial is set to be either appetitive or aversive against tasks where a trial allows the possibility of either. This study presents a meta-analysis of reinforcement learning experiments, the majority of which presented the possibility of winning or losing money. Value and valence encodings were identified by conventional difference wave methodology but additionally by an analysis of their predicted behavior using a Bayesian analysis that incorporated nulls into the evidence for each encoder. The results suggest that a valence encoding, sensitive only to the available outcomes on the trial at hand precedes a later value encoding sensitive to the outcomes available in the wider experimental context. The implications of this for modeling computational processes of reinforcement learning in humans are discussed.

A deep reinforcement learning-based approach to onboard trajectory generation for hypersonic vehicles

AbstractAn onboard three-dimensional (3D) trajectory generation approach based on the reinforcement learning (RL) algorithm and deep neural network (DNN) is proposed for hypersonic vehicles in glide phase. Multiple trajectory samples are generated offline through the convex optimisation method. The deep learning (DL) is employed to pre-train the DNN for initialising the actor network and accelerating the RL process. Based on the offline deep policy deterministic actor-critic algorithm, a flight target-oriented reward function with path constraints is designed. The actor network is optimised by the end-to-end RL and policy gradients of the critic network until the reward function converges to the maximum. The actor network is considered as the onboard trajectory generator to compute optimal control values online based on the real-time motion states. The simulation results show that the single-step online planning time meets the real-time requirements of onboard trajectory generation. The significant improvement in terminal accuracy of the online trajectory and the better generalisation under biased initial states for hypersonic vehicles in glide phase is observed.

Intrinsic fluctuations of reinforcement learning promote cooperation

In this work, we ask for and answer what makes classical temporal-difference reinforcement learning with epsilon-greedy strategies cooperative. Cooperating in social dilemma situations is vital for animals, humans, and machines. While evolutionary theory revealed a range of mechanisms promoting cooperation, the conditions under which agents learn to cooperate are contested. Here, we demonstrate which and how individual elements of the multi-agent learning setting lead to cooperation. We use the iterated Prisoner’s dilemma with one-period memory as a testbed. Each of the two learning agents learns a strategy that conditions the following action choices on both agents’ action choices of the last round. We find that next to a high caring for future rewards, a low exploration rate, and a small learning rate, it is primarily intrinsic stochastic fluctuations of the reinforcement learning process which double the final rate of cooperation to up to 80%. Thus, inherent noise is not a necessary evil of the iterative learning process. It is a critical asset for the learning of cooperation. However, we also point out the trade-off between a high likelihood of cooperative behavior and achieving this in a reasonable amount of time. Our findings are relevant for purposefully designing cooperative algorithms and regulating undesired collusive effects.