Reinforcement Learning Model Research Articles

Development of AI-assisted microscopy frameworks through realistic simulation with pySTED

The integration of artificial intelligence into microscopy systems significantly enhances performance, optimizing both image acquisition and analysis phases. Development of artificial intelligence-assisted super-resolution microscopy is often limited by access to large biological datasets, as well as by difficulties to benchmark and compare approaches on heterogeneous samples. We demonstrate the benefits of a realistic stimulated emission depletion microscopy simulation platform, pySTED, for the development and deployment of artificial intelligence strategies for super-resolution microscopy. pySTED integrates theoretically and empirically validated models for photobleaching and point spread function generation in stimulated emission depletion microscopy, as well as simulating realistic point-scanning dynamics and using a deep learning model to replicate the underlying structures of real images. This simulation environment can be used for data augmentation to train deep neural networks, for the development of online optimization strategies and to train reinforcement learning models. Using pySTED as a training environment allows the reinforcement learning models to bridge the gap between simulation and reality, as showcased by its successful deployment on a real microscope system without fine tuning.

Two-dimensional reinforcement learning model-free fault-tolerant control for batch processes against multi- faults

Aiming at the characteristics of batch process changing along with time and batch directions, the existence of unmodeled dynamics, and the partial failure of actuators or/and sensors, we propose a novel 2D reinforcement learning (RL) fault tolerant control strategy without considering model parameters. Firstly, a two-Dimensional (2D) augmented state space model and 2D Q function-based fault tolerant control (FTC) framework is established. The 2D Bellman equation can be acquired by analyzing the relationship between the 2D value function and the 2D Q function. Based on the extended model and Q-learning concept of RL, a data-driven FTTC independent of model parameters is designed, and a 2D data-driven Q-learning algorithm is proposed. Finally, taking the pressure holding phase in the injection process as the experimental object, the control effect is compared with that of the traditional model-based FTC, and better tracking performance and unbiasedness to the probing noise can be proved.

Integrated Intelligent Control of Redundant Degrees-of-Freedom Manipulators via the Fusion of Deep Reinforcement Learning and Forward Kinematics Models

Redundant degree-of-freedom (DOF) manipulators offer increased flexibility and are better suited for obstacle avoidance, yet precise control of these systems remains a significant challenge. This paper addresses the issues of slow training convergence and suboptimal stability that plague current deep reinforcement learning (DRL)-based control strategies for redundant DOF manipulators. We propose a novel DRL-based intelligent control strategy, FK-DRL, which integrates the manipulator’s forward kinematics (FK) model into the control framework. Initially, we conceptualize the control task as a Markov decision process (MDP) and construct the FK model for the manipulator. Subsequently, we expound on the integration principles and training procedures for amalgamating the FK model with existing DRL algorithms. Our experimental analysis, applied to 7-DOF and 4-DOF manipulators in simulated and real-world environments, evaluates the FK-DRL strategy’s performance. The results indicate that compared to classical DRL algorithms, the FK-DDPG, FK-TD3, and FK-SAC algorithms improved the success rates of intelligent control tasks for the 7-DOF manipulator by 21%, 87%, and 64%, respectively, and the training convergence speeds increased by 21%, 18%, and 68%, respectively. These outcomes validate the proposed algorithm’s effectiveness and advantages in redundant manipulator control using DRL and FK models.

Energy-aware bio-inspired spiking reinforcement learning system architecture for real-time autonomous edge applications.

Mobile, low-cost, and energy-aware operation of Artificial Intelligence (AI) computations in smart circuits and autonomous robots will play an important role in the next industrial leap in intelligent automation and assistive devices. Neuromorphic hardware with spiking neural network (SNN) architecture utilizes insights from biological phenomena to offer encouraging solutions. Previous studies have proposed reinforcement learning (RL) models for SNN responses in the rat hippocampus to an environment where rewards depend on the context. The scale of these models matches the scope and capacity of small embedded systems in the framework of Internet-of-Bodies (IoB), autonomous sensor nodes, and other edge applications. Addressing energy-efficient artificial learning problems in such systems enables smart micro-systems with edge intelligence. A novel bio-inspired RL system architecture is presented in this work, leading to significant energy consumption benefits without foregoing real-time autonomous processing and accuracy requirements of the context-dependent task. The hardware architecture successfully models features analogous to synaptic tagging, changes in the exploration schemes, synapse saturation, and spatially localized task-based activation observed in the brain. The design has been synthesized, simulated, and tested on Intel MAX10 Field-Programmable Gate Array (FPGA). The problem-based bio-inspired approach to SNN edge architectural design results in 25X reduction in average power compared to the state-of-the-art for a test with real-time context learning and 30 trials. Furthermore, 940x lower energy consumption is achieved due to improvement in the execution time.

A hierarchical reinforcement learning model explains individual differences in attentional set shifting

Attentional set shifting refers to the ease with which the focus of attention is directed and switched. Cognitive tasks, such as the widely used CANTAB IED, reveal great variation in set shifting ability in the general population, with notable impairments in those with psychiatric diagnoses. The attentional and learning processes underlying this cognitive ability and how they lead to the observed variation remain unknown. To directly test this, we used a modelling approach on two independent large-scale online general-population samples performing CANTAB IED, with one including additional psychiatric symptom assessment. We found a hierarchical model that learnt both feature values and dimension attention best explained the data and that compulsive symptoms were associated with slower learning and higher attentional bias to the first relevant stimulus dimension. These data showcase a new methodology to analyse data from the CANTAB IED task, as well as suggest a possible mechanistic explanation for the variation in set shifting performance, and its relationship to compulsive symptoms.

Reference RL: Reinforcement learning with reference mechanism and its application in traffic signal control

This paper addresses the challenges of deploying reinforcement learning (RL) models for traffic signal control (TSC) in real-world environments. Real-world training can prevent mismatches between simulation environments and the actual traffic conditions, thereby achieving better performance of agent upon deployment. However, free explorations by agents during real-world training can disrupt traffic operations. To mitigate this, this paper proposes a reference mechanism to guide the decision-making process within the RL framework. A reference timing policy, typically a model-based signal strategy, is integrated into the learning process to provide agents with reference actions. Specifically, an additional Q-value function is introduced to evaluate both the agent’s actions and those of the reference policy, allowing for adjustments before the actions are executed in real traffic system. Numerical results indicate that the reference mechanism effectively enhances system performance early in the training process, thus accelerating learning. We also combine the reference RL method with a pretraining procedure and a jump-start algorithm, respectively. Experimental results demonstrate their effectiveness in further enhancing system performance and facilitating real-world training.

Assessing social anhedonia in a transdiagnostic sample: Insights from a computational psychiatry lens.

Anhedonia, a reduced ability to experience positive affect and seek rewards, is present across many psychiatric disorders, notably among individuals who experienced trauma. Within the social domain, anhedonia manifests as an altered sense of belonging and social isolation and is associated with poorer clinical outcomes. Yet, mechanistic operationalizations of social anhedonia are lacking, limiting our understanding of the relationship between these mechanisms and affective symptoms. To address these questions, we developed a social reward exploration task which was administered to a transdiagnostic sample of trauma-exposed Veterans (N = 33) while they underwent functional magnetic resonance imaging. The goal was to maximize compliments from two unknown partners, as participants were told these partners selected compliments based on seeing their picture. A Bayesian reinforcement learning modeling approach was used to extract cognitive and neural markers of compliment (reward) exploration. To address these questions, we developed a social reward exploration task which was administered to a transdiagnostic sample of trauma-exposed Veterans (N = 33) while they underwent functional magnetic resonance imaging. The goal was to maximize compliments from two unknown partners, as participants were told these partners selected compliments based on seeing their picture. A Bayesian reinforcement learning modeling approach was used to extract cognitive and neural markers of compliment (reward) exploration. Higher social connectedness (β = 0.51; 95 % CI=[0.11,0.94]) and anxiety (β = 0.57; 95 % CI=[0.13,1.00]) were independently associated with more model-based choices of the partner they anticipated to be most complimenting. In the dorsal anterior cingulate cortex (ACC; z = 3.89, p .001) and left inferior parietal lobule (z = 3.96, p .001), neural responses to reward prediction errors (RPE) were more positive in response to compliment relative to non-compliment outcomes. Greater positive RPE ACC activation was associated with lower anxiety (β = −0.51; 95 % CI=[−0.99,−0.10]. Computational approaches to social reinforcement learning can help identify important neurocognitive differences in social reward sensitivity among individuals with complex affective profiles, such as trauma-exposed individuals. Understanding these differences may help develop new prediction and treatment tools for social anhedonia.

Deep contextual reinforcement learning algorithm for scalable power scheduling

This article introduces a deep contextual reinforcement learning (DCRL) optimization algorithm to tackle the NP-hard power scheduling problem. The algorithm is based on a multi-agent simulation environment that decomposes the power scheduling problem into sequential Markov decision processes (MDPs). The simulation environment inherently corrects incorrect decisions and adjusts supply capacities so that the MDPs adhere to the optimization constraints. A deep reinforcement learning (DRL) model is trained on these MDPs to provide optimal solutions. Demonstrating its applicability and effectiveness, the proposed method is evaluated on various test systems with different unit numbers, constraints, and production cost functions. The experimental results show that the proposed method has better performance relative to alternative methods, such as binary alternative moth-flame optimization (BAMFO), binary particle swarm optimization (BPSO), teaching learning-based optimization (TLBO), new binary particle swarm optimization (NBPSO), binary learning particle swarm optimization (BLPSO), quasi-oppositional teaching learning-based algorithm (QOTLBO), binary-real-coded genetic algorithm (BRCGA), hybrid genetic-imperialist competitive algorithm (HGICA), three-stage priority list (TSPL), and hybridized evolutionary algorithm and sequential quadratic programming (HEPSQP). The novelties of the proposed method lie in its adaptability to longer planning horizons and linear dimensionality complexity, rendering it suitable for large-scale power systems.

Autonomous countertraction for secure field of view in laparoscopic surgery using deep reinforcement learning.

Countertraction is a vital technique in laparoscopic surgery, stretching the tissue surface for incision and dissection. Due to the technical challenges and frequency of countertraction, autonomous countertraction has the potential to significantly reduce surgeons' workload. Despite several methods proposed for automation, achieving optimal tissue visibility and tension for incision remains unrealized. Therefore, we propose a method for autonomous countertraction that enhances tissue surface planarity and visibility. We constructed a neural network that integrates a point cloud convolutional neural network (CNN) with a deep reinforcement learning (RL) model. This network continuously controls the forceps position based on the surface shape observed by a camera and the forceps position. RL is conducted in a physical simulation environment, with verification experiments performed in both simulation and phantom environments. The evaluation was performed based on plane error, representing the average distance between the tissue surface and its least-squares plane, and angle error, indicating the angle between the tissue surface vector and the camera's optical axis vector. The plane error decreased under all conditions both simulation and phantom environments, with 93.3% of case showing a reduction in angle error. In simulations, the plane error decreased from to , and the angle error from to . In the phantom environment, the plane error decreased from to , and the angle error from to . The proposed neural network was validated in both simulation and phantom experimental settings, confirming that traction control improved tissue planarity and visibility. These results demonstrate the feasibility of automating countertraction using the proposed model.

Correction: Yalçın, S.; Herdem, M.S. Optimizing EV Battery Management: Advanced Hybrid Reinforcement Learning Models for Efficient Charging and Discharging. Energies 2024, 17, 2883

Correction: Yalçın, S.; Herdem, M.S. Optimizing EV Battery Management: Advanced Hybrid Reinforcement Learning Models for Efficient Charging and Discharging. Energies 2024, 17, 2883

Multi-agent Dual Level Reinforcement Learning of Strategy and Tactics in Competitive Games

Reinforcement learning has been used extensively to learn the low-level tactical choices during gameplay; however, less effort is invested in the strategic decisions governing the effective engagement of a diverse set of opponents. In this paper, a two-tier reinforcement learning model is created to play competitive games and effectively engage in matches with different opponents to maximize earnings. The multi-agent environment has four types of learners, which vary in their ability to learn gameplay directly (tactics) and their ability to learn to bet or withdraw from gameplay (strategy). The players are tested in three different competitive games: Connect 4, Dots and Boxes, and Tic-Tac-Toe. Analyzing the behavior of players as they progress from naivety to game mastery reveals some interesting features: (1) learners who optimize strategy and tactics outperform all learners, (2) learners who initially optimize their strategy to engage in matches outperform those who focus on optimizing tactical gameplay, and (3) the advantage of strategy optimization versus tactical gameplay optimization diminishes as more games are played. A reinforcement learning model with a dual learning scheme presents possible applications in adversarial scenarios where both strategic and tactical learning are critical. We present detailed results in a systematic manner, providing strong support for our claim.

Towards online training for RL-based query optimizer

AbstractJoin query optimization aims to find the best join order for tables in a query, which is critical for query processing performance. Recently, reinforcement learning models have been proposed to solve the challenges existing with query processing and join query optimization. However, changes in the data distribution can turn the trained reinforcement learning models into obsolete models, resulting in longer execution times. In this paper, we propose a new training strategy in order to extend the existing reinforcement learning models and improve their adaptation when the data distribution changes. The experiments show that the proposed strategy has a significant benefit in decreasing training time for the models given the changes in the data distribution.

Analyzing deep reinforcement learning model decisions with Shapley additive explanations for counter drone operations

As the use of drones continues to increase, their capabilities pose a threat to airspace safety when they are misused. Deploying AI models for intercepting these unwanted drones becomes crucial. However, these AI models, such as deep learning models, often operate as “black boxes”, making it hard to trust their decision-making system. This also affects end-users’ confidence in these AI systems. In this paper, the explainability of deep reinforcement learning is investigated and a deep reinforcement learning (DRL) method, double deep Q-network with dueling architecture and prioritized experience replay is applied to train the AI models. To make the AI model decisions more transparent and to understand the reasoning behind the AI decisions for counter-drone systems, Shapley Additive Explanations (SHAP) method is implemented. After training the DRL agent, experience replay is visualized, and the absolute SHAP values are calculated to explain the key factors that influence the deep reinforcement learning agent’s choices. The integration of DRL with explainable AI methods such as SHAP demonstrates significant potential for the advancement of robust and efficient counter-drone systems.

Cognitive Mapping of Reachable Space in Humans

Reinforcement learning illustrates how people represent and learn large-scale environments for spatial navigation. However, it is unclear how “reachable” spaces are represented when performing manual tasks like chopping vegetables. To analyze the mechanisms underlying learning reachable space, we used inspiration from a recent study (de Cothi et al., 2022) to develop a haptic maze task where human participants reached to a target while avoiding invisible haptic obstacles in the environment. Participants reached with a robotic handle, generating contact forces to simulate the maze boundary, obstacle walls, and the floor, which supported the hand. Participants were blind to their hand position, while the target and maze boundary remained visible. Two conditions were implemented for each experiment variation; maze obstacles were invisible in the first condition, then became visible in the second. Audio feedback was given upon finding the target. We tested participants in 25 unique mazes, performing 10 trials within each maze. Two experimental procedures were employed: one with a fixed target and varying starting locations (18 participants), while the other had a fixed starting location and variable targets (10 participants). We simulated and compared the likelihoods of 3 different reinforcement learning models: model-based (MB), model-free (MF) and successor representation (SR). MB-simulated agents learned a map of the maze, planning the shortest route to the target. MF-simulated agents cached and updated the expected total future reward for each action using experience, pursuing the highest-reward actions. SR-simulated agents generated a “cognitive” (predictive) map between grids (states), integrating it with the target to plan actions. Results suggested that humans use a combination of MB and MF to learn reachable spaces. MB agents had higher likelihoods (better represented human behaviour) than MF agents for earlier trials, but MF had greater likelihoods for later trials. Meanwhile, SR learning was significantly worse at predicting participant behaviour.

Reinforcement learning as a robotics-inspired framework for insect navigation: from spatial representations to neural implementation.

Bees are among the master navigators of the insect world. Despite impressive advances in robot navigation research, the performance of these insects is still unrivaled by any artificial system in terms of training efficiency and generalization capabilities, particularly considering the limited computational capacity. On the other hand, computational principles underlying these extraordinary feats are still only partially understood. The theoretical framework of reinforcement learning (RL) provides an ideal focal point to bring the two fields together for mutual benefit. In particular, we analyze and compare representations of space in robot and insect navigation models through the lens of RL, as the efficiency of insect navigation is likely rooted in an efficient and robust internal representation, linking retinotopic (egocentric) visual input with the geometry of the environment. While RL has long been at the core of robot navigation research, current computational theories of insect navigation are not commonly formulated within this framework, but largely as an associative learning process implemented in the insect brain, especially in the mushroom body (MB). Here we propose specific hypothetical components of the MB circuit that would enable the implementation of a certain class of relatively simple RL algorithms, capable of integrating distinct components of a navigation task, reminiscent of hierarchical RL models used in robot navigation. We discuss how current models of insect and robot navigation are exploring representations beyond classical, complete map-like representations, with spatial information being embedded in the respective latent representations to varying degrees.

HMM for discovering decision-making dynamics using reinforcement learning experiments.

Major depressive disorder (MDD), a leading cause of years of life lived with disability, presents challenges in diagnosis and treatment due to its complex and heterogeneous nature. Emerging evidence indicates that reward processing abnormalities may serve as a behavioral marker for MDD. To measure reward processing, patients perform computer-based behavioral tasks that involve making choices or responding to stimulants that are associated with different outcomes, such as gains or losses in the laboratory. Reinforcement learning (RL) models are fitted to extract parameters that measure various aspects of reward processing (e.g. reward sensitivity) to characterize how patients make decisions in behavioral tasks. Recent findings suggest the inadequacy of characterizing reward learning solely based on a single RL model; instead, there may be a switching of decision-making processes between multiple strategies. An important scientific question is how the dynamics of strategies in decision-making affect the reward learning ability of individuals with MDD. Motivated by the probabilistic reward task within the Establishing Moderators and Biosignatures of Antidepressant Response in Clinical Care (EMBARC) study, we propose a novel RL-HMM (hidden Markov model) framework for analyzing reward-based decision-making. Our model accommodates decision-making strategy switching between two distinct approaches under an HMM: subjects making decisions based on the RL model or opting for random choices. We account for continuous RL state space and allow time-varying transition probabilities in the HMM. We introduce a computationally efficient Expectation-maximization (EM) algorithm for parameter estimation and use a nonparametric bootstrap for inference. Extensive simulation studies validate the finite-sample performance of our method. We apply our approach to the EMBARC study to show that MDD patients are less engaged in RL compared to the healthy controls, and engagement is associated with brain activities in the negative affect circuitry during an emotional conflict task.

A novel Soft Actor–Critic framework with disjunctive graph embedding and autoencoder mechanism for Job Shop Scheduling Problems

The Job-Shop Scheduling Problem (JSSP) is a well-established and classic NP-hard combinatorial optimization issue. The quality of its scheduling scheme directly affects the operational efficiency of manufacturing systems. Priority Dispatching Rules (PDRs) are often utilized to address JSSP in real-world contexts, but the process of creating effective PDRs can be daunting and time-consuming. It also necessitates comprehensive domain knowledge, typically resulting in mediocre performance. In this paper, we introduce a novel reinforcement learning (RL) model called Disjunctive Graph Embedding with Autoencoder Mechanism for Job Shop Scheduling Problems (DGEAM-JSSP), designed to automate PDRs learning. Our proposed model confronts the issue using a Graph Neural Network (GNN) to learn node features that encapsulate the spatial structure of the JSSP graph representation. The ensuing policy network is size-agnostic, enabling effective generalization on larger-scale instances. Additionally, we employ a transformer encoder, incorporating parallel encoding and a self-attention mechanism, to successfully recognize long-term dependencies among operations in large-scale scheduling problems. We also implemented an end-to-end training approach using the Soft Actor–Critic (SAC) algorithm to instruct the two modules. Computational experiment results reveal that, with a single training, our agent successfully learns a superior dispatching policy, surpassing PDRs and state-of-the-art RL frameworks specifically tailored for each JSSP instance size in solution quality, as well as OR-Tools in execution speed. Moreover, results from random and benchmark instances illustrate that the uniquely-modeled learned policies have impressive generalization performance on real-world instances and significantly larger-scale scenarios involving up to 2000 operations.

A Proposal of Score Distribution Predictive Model in Self-Play Deep Reinforcement Learning

A Proposal of Score Distribution Predictive Model in Self-Play Deep Reinforcement Learning

BaziGooshi: A Hybrid Model of Reinforcement Learning for Generalization in Gameplay

BaziGooshi: A Hybrid Model of Reinforcement Learning for Generalization in Gameplay

Enhanced Fault Diagnosis in IoT: Uniting Data Fusion with Deep Multi-Scale Fusion Neural Network

One of the biggest challenges is figuring out when industrial IoT devices break. Notwithstanding these difficulties, one of the many advantages of using real-time sensor data from the Industrial Internet of Things (IIoT) is the ability to monitor events and respond promptly. The IIoT's sensor numbers may be analysed, combined with data from other sources, and applied quickly to make decisions. This manuscript proposes Enhanced Fault Diagnosis in IoT, Uniting Data Fusion with Deep Multi-Scale Fusion Neural Network (FD-IoT-DMSFNN). Initially, input sensor data are taken from the CWRU Dataset. Then, sensor data is normalised using Multivariate Fast Iterative Filtering. Then, the normalised sensor data are extensively dispersed and diverse; hence, the data outlier detection is performed using the Deep isolation forest (DIF) approach. Therefore, this manuscript proposes a Mexican Axolotl Optimization (MAO) for tuning the weight parameter of DMSFNN, which exhibits an enhanced fault detection process. Python is used to implement the recommended strategy. The proposed method's performance yields a lower completion time of 6.5%, 1.8%, and 4.13%; higher prediction accuracy of 2.26%, 6.71%, and 8.32% compared with existing approaches such as towards deep domain adaptability training methods for industrial IoT edge device soft real-time fault diagnosis (FD-IoT-LSTM), Intelligent Fault Quantitative Identification for the Industrial Internet of Things (IIoT) utilising a particular deep dual reinforcement learning model with insufficient samples (FD-IoT-DRL) and IIoT- based fault identification model for industrial use (FD-IoT-ML-LSTM) respectively.