Generalized Reinforcement Learning Research Articles

A Sub-Second Method for SAR Image Registration Based on Hierarchical Episodic Control

For Synthetic Aperture Radar (SAR) image registration, successive processes following feature extraction are required by both the traditional feature-based method and the deep learning method. Among these processes, the feature matching process—whose time and space complexity are related to the number of feature points extracted from sensed and reference images, as well as the dimension of feature descriptors—proves to be particularly time consuming. Additionally, the successive processes introduce data sharing and memory occupancy issues, requiring an elaborate design to prevent memory leaks. To address these challenges, this paper introduces the OptionEM-based reinforcement learning framework to achieve end-to-end SAR image registration. This framework outputs registered images directly without requiring feature matching and the calculation of the transformation matrix, leading to significant processing time savings. The Transformer architecture is employed to learn image features, while a correlation network is introduced to learn the correlation and transformation matrix between image pairs. Reinforcement learning, as a decision process, can dynamically correct errors, making it more-efficient and -robust compared to supervised learning mechanisms such as deep learning. We present a hierarchical reinforcement learning framework combined with Episodic Memory to mitigate the inherent problem of invalid exploration in generalized reinforcement learning algorithms. This approach effectively combines coarse and fine registration, further enhancing training efficiency. Experiments conducted on three sets of SAR images, acquired by TerraSAR-X and Sentinel-1A, demonstrated that the proposed method’s average runtime is sub-second, achieving subpixel registration accuracy.

Soft Contrastive Learning With Q-Irrelevance Abstraction for Reinforcement Learning

The difference between training and testing environments is a huge challenge to generalizing reinforcement learning (RL) algorithms. We propose a Soft Contrastive learning with a coarser approximate Q-irrelevance abstraction for Reinforcement Learning (SCQRL) to increase RL generalization. Specifically, we specify the coarser approximate Q-irrelevance abstraction as the feature of the state with a theoretical analysis for better generalization ability. We construct a positive and negative sample selection mechanism based on the Q value for contrastive learning to achieve efficient representation learning. Considering the selection error of positive and negative samples, we design soft contrastive learning and combine it with reinforcement learning in the form of an auxiliary task to propose SCQRL. The generalization experiments on several Procgen environments demonstrate that SCQRL outperforms the excellent generalized RL algorithm.

A generalized reinforcement learning based deep neural network agent model for diverse cognitive constructs

Human cognition is characterized by a wide range of capabilities including goal-oriented selective attention, distractor suppression, decision making, response inhibition, and working memory. Much research has focused on studying these individual components of cognition in isolation, whereas in several translational applications for cognitive impairment, multiple cognitive functions are altered in a given individual. Hence it is important to study multiple cognitive abilities in the same subject or, in computational terms, model them using a single model. To this end, we propose a unified, reinforcement learning-based agent model comprising of systems for representation, memory, value computation and exploration. We successfully modeled the aforementioned cognitive tasks and show how individual performance can be mapped to model meta-parameters. This model has the potential to serve as a proxy for cognitively impaired conditions, and can be used as a clinical testbench on which therapeutic interventions can be simulated first before delivering to human subjects.

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3T and 7T.

The model‐free algorithms of “reinforcement learning” (RL) have gained clout across disciplines, but so too have model‐based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This “generalized reinforcement learning” (GRL) model, a frugal extension of RL, parsimoniously retains the single reward‐prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal‐learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high‐resolution high‐field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value‐based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

Generalized reinforcement learning for building control using Behavioral Cloning

Advanced building control methods such as model predictive control (MPC) offer significant benefits to both consumers and grid operators, but high computational requirements have acted as barriers to more widespread adoption. Local control computation requires installation of expensive computational hardware, while cloud computing introduces data security and privacy concerns. In this paper, we drastically reduce the local computational requirements of advanced building control through a reinforcement learning (RL)-based approach called Behavioral Cloning, which represents the MPC policy as a neural network that can be locally implemented and quickly computed on a low-cost programmable logic controller. While previous RL and approximate MPC methods must be specifically trained for each building, our key improvement is that the proposed controller can generalize to many buildings, electricity rates, and thermostat setpoint schedules without additional, effort-intensive retraining. To provide this versatility, we have adapted the traditional Behavioral Cloning approach through two innovations: (1) a constraint-informed parameter grouping (CIPG) method that provides a more efficient representation of the training data and (2) a new deep learning model-structure called reverse-time recurrent neural networks (RT-RNN) that allows future information to flow backward in time to more effectively interpret the temporal information in disturbance predictions. The result is an easy-to-deploy, generalized behavioral clone of MPC that can be implemented on a programmable logic controller and requires little building-specific controller tuning, reducing the effort and costs associated with implementing smart residential heat pump control.

Intelligent User-Centric Networks: Learning-Based Downlink CoMP Region Breathing

In the presence of irregular transmission/reception point (TRP) topologies and non-uniform user distribution, the user-to-node association optimization is a rather challenging process in real user-centric networks, especially for the joint transmission aided coordinated multipoint (CoMP) technique. The grade of challenge further escalates, when taking the dynamic user scheduling process into account in order to enhance the system capacity attained. To tackle the above-mentioned problem, we holistically optimize the system by conceiving joint user scheduling and user-to-node association. Then, for the sake of striking a significantly better balance between the network capacity and coverage quality, we propose a generalized reinforcement learning assisted framework intrinsically amalgamated both with neural-fitted Q-iteration as well as with ensemble learning and transfer learning techniques. Consequently, a powerful policy can be found for dynamically adjusting the set of TRPs participating in the joint transmission, thus allowing the CoMP-region to breathe, depending on both the temporal and geographical distribution of the tele-traffic load across the network. To facilitate the prompt learning of the global policy supporting flexible scalability, the overall network optimization process is decoupled into multiple local optimization phases associated with a number of TRP clusters relying on iterative information exchange among them. Our simulation results show that the proposed scheme is capable of producing a policy achieving a network-edge throughput gain of up to 140%$ and a network capacity gain of up to 190% under the challenging scenario of having a non-uniform geographical UE distribution and bursty traffic.

A generalized reinforcement learning scheme for random neural networks

We propose a reinforcement learning scheme for random neural networks (RNN) that update network weights based on the relative reward obtained by past actions. This new learning scheme achieves improved performance in random environments under certain conditions, in particular, when the number of actions is small. We also explore hybrid learning structures, where a RNN functions as a meta decision unit that learns how to best delegate the action decision to a second layer of decision units. An extensive simulation study demonstrates the performance of these learning methods applied to minimizing response times in a distributed task assignment scenario. Results are compared to the cognitive packet networks’ reinforcement learning scheme, a reward-only variation, and several non-learning methods. The study reveals that no single method can produce the best performance results in all cases. However, both the generalized reinforcement learning scheme and the application of meta decision units can help to achieve improved performance in specific cases.

Equilibrium selection in the stag hunt game under generalized reinforcement learning

We apply the generalized reinforcement (GR) learning protocol to the stag hunt game. GR learning combines positive and negative reinforcement. The GR learning rule generates the GR dynamic, which governs the evolution of the mixed strategy of agents in the population. We identify conditions under which the GR dynamic converges globally to one of the two pure strategy Nash equilibria of the game.

Heterogeneity in generalized reinforcement learning and its relation to cognitive ability

In this paper, we study the connections between working memory capacity (WMC) and learning in the context of economic guessing games. We apply a generalized version of reinforcement learning, popularly known as the experience-weighted attraction (EWA) learning model, which has a connection to specific cognitive constructs, such as memory decay, the depreciation of past experience, counterfactual thinking, and choice intensity. Through the estimates of the model, we examine behavioral differences among individuals due to different levels of WMC. In accordance with ‘Miller’s magic number’, which is the constraint of working memory capacity, we consider two different sizes (granularities) of strategy space: one is larger (finer) and one is smaller (coarser). We find that constraining the EWA models by using levels (granules) within the limits of working memory allows for a better characterization of the data based on individual differences in WMC. Using this level-reinforcement version of EWA learning, also referred to as the EWA rule learning model, we find that working memory capacity can significantly affect learning behavior. Our likelihood ratio test rejects the null that subjects with high WMC and subjects with low WMC follow the same EWA learning model. In addition, the parameter corresponding to ‘counterfactual thinking ability’ is found to be reduced when working memory capacity is low.

Generalized reinforcement learning in perfect-information games

This paper studies reinforcement learning in which players base their action choice on valuations they have for the actions. We identify two general conditions on valuation updating rules that together guarantee that the probability of playing a subgame perfect Nash equilibrium (SPNE) converges to one in games where no player is indifferent between two outcomes without every other player being also indifferent. The same conditions guarantee that the fraction of times a SPNE is played converges to one almost surely. We also show that for additively separable valuations, in which valuations are the sum of empirical and error terms, the conditions guaranteeing convergence can be made more intuitive. In addition, we give four examples of valuations that satisfy our conditions. These examples represent different degrees of sophistication in learning behavior and include well-known examples of reinforcement learning.

The dynamics of generalized reinforcement learning

We consider reinforcement learning in games with both positive and negative payoffs. The Cross rule is the prototypical reinforcement learning rule in games that have only positive payoffs. We extend this rule to incorporate negative payoffs to obtain the generalized reinforcement learning rule. Applying this rule to a population game, we obtain the generalized reinforcement dynamic which describes the evolution of mixed strategies in the population. We apply the dynamic to the class of Rock–Scissor–Paper (RSP) games to establish local convergence to the interior rest point in all such games, including the bad RSP game.

An intelligent agent for optimal river‐reservoir system management

A generalized software package is presented for developing an intelligent agent for stochastic optimization of complex river‐reservoir system management and operations. Reinforcement learning is an approach to artificial intelligence for developing a decision‐making agent that learns the best operational policies without the need for explicit probabilistic models of hydrologic system behavior. The agent learns these strategies experientially in a Markov decision process through observational interaction with the environment and simulation of the river‐reservoir system using well‐calibrated models. The graphical user interface for the reinforcement learning process controller includes numerous learning method options and dynamic displays for visualizing the adaptive behavior of the agent. As a case study, the generalized reinforcement learning software is applied to developing an intelligent agent for optimal management of water stored in the Truckee river‐reservoir system of California and Nevada for the purpose of streamflow augmentation for water quality enhancement. The intelligent agent successfully learns long‐term reservoir operational policies that specifically focus on mitigating water temperature extremes during persistent drought periods that jeopardize the survival of threatened and endangered fish species.

Neuroevolutionary reinforcement learning for generalized control of simulated helicopters.

This article presents an extended case study in the application of neuroevolution to generalized simulated helicopter hovering, an important challenge problem for reinforcement learning. While neuroevolution is well suited to coping with the domain’s complex transition dynamics and high-dimensional state and action spaces, the need to explore efficiently and learn on-line poses unusual challenges. We propose and evaluate several methods for three increasingly challenging variations of the task, including the method that won first place in the 2008 Reinforcement Learning Competition. The results demonstrate that (1) neuroevolution can be effective for complex on-line reinforcement learning tasks such as generalized helicopter hovering, (2) neuroevolution excels at finding effective helicopter hovering policies but not at learning helicopter models, (3) due to the difficulty of learning reliable models, model-based approaches to helicopter hovering are feasible only when domain expertise is available to aid the design of a suitable model representation and (4) recent advances in efficient resampling can enable neuroevolution to tackle more aggressively generalized reinforcement learning tasks.