Learning Agent Research Articles

Anatomical Landmark Detection in 3d MRI Scan using Deep Neuro-Dynamic Programming

Automatic and precise biometric measures are crucial for diagnosis and supporting important decisions in medical imaging. For several applications of medical image analysis, automatically identifying anatomical landmarks is an important step. In this study, we determine the effectiveness of deep-neurodynamic programming methods for developing agents that can consistently and accurately recognize ground truth landmarks in medical imaging. Through interaction with an environment of three-dimensional (3D) scan ultrasound images, a simulated reinforcement learning (RL) agent learns to identify the optimum path leading to the landmark. Deep Q-Network (DQN) algorithm is applied to magnetic resonance imaging (MRI) images of the human brain. The model’s effectiveness and localization error are measured in terms of the mean Euclidean distance between the detected and ground truth landmarks’ position. We achieved a low localization error of 3.55mm during our experiments with the ADNI dataset.

Structure-aware dual-target drug design through collaborative learning of pharmacophore combination and molecular simulation.

Dual-target drug design has gained significant attention in the treatment of complex diseases, such as cancers and autoimmune disorders. A widely employed design strategy is combining pharmacophores to leverage the knowledge of structure-activity relationships of both targets. Unfortunately, pharmacophore combination often struggles with long and expensive trial and error, because the protein pockets of the two targets impose complex structural constraints. In this study, we propose AIxFuse, a structure-aware dual-target drug design method that learns pharmacophore fusion patterns to satisfy the dual-target structural constraints simulated by molecular docking. AIxFuse employs two self-play reinforcement learning (RL) agents to learn pharmacophore selection and fusion by comprehensive feedback including dual-target molecular docking scores. Collaboratively, the molecular docking scores are learned by active learning (AL). Through collaborative RL and AL, AIxFuse learns to generate molecules with multiple desired properties. AIxFuse is shown to outperform state-of-the-art methods in generating dual-target drugs against glycogen synthase kinase-3 beta (GSK3β) and c-Jun N-terminal kinase 3 (JNK3). When applied to another task against retinoic acid receptor-related orphan receptor γ-t (RORγt) and dihydroorotate dehydrogenase (DHODH), AIxFuse exhibits consistent performance while compared methods suffer from performance drops, leading to a 5 times higher performance in success rate. Docking studies demonstrate that AIxFuse can generate molecules concurrently satisfying the binding mode required by both targets. Further free energy perturbation calculation indicates that the generated candidates have promising binding free energies against both targets.

A Generalist Reinforcement Learning Agent for Compressing Convolutional Neural Networks

A Generalist Reinforcement Learning Agent for Compressing Convolutional Neural Networks

Forecasting Future Behavior: Agents in Board Game Strategy

This paper presents findings on machine learning agent behavior prediction in a board game application developed by a group of students. The goal of this research is to create a model facilitating collaboration between a user and an AI to play together in the board game using a Human-in-the-Loop architecture. By injecting explainability, the aim is to enhance communication and understanding between the user and the AI agent. Featuring a competitive Artificial Intelligence (AI) based on the Proximal Policy Optimization model, this research explores methods to make AI decisions transparent for enhanced player understanding. Two predictive models, a Decision Tree (DT) and a Deep Learning (DL) classifier, were developed and compared. The results show that the DT model is effective for short-term predictions but limited in broader applications, while the DL classifier shows potential for long-term prediction without requiring direct access to the game's AI. This study contributes to understanding human-AI interaction in gaming and offers insights into AI decision-making processes.

WrapperRL: Reinforcement Learning Agent for Feature Selection in High-Dimensional Industrial Data

WrapperRL: Reinforcement Learning Agent for Feature Selection in High-Dimensional Industrial Data

PandORA: Automated Design and Comprehensive Evaluation of Deep Reinforcement Learning Agents for Open RAN

PandORA: Automated Design and Comprehensive Evaluation of Deep Reinforcement Learning Agents for Open RAN

Enhancing Medical Training Through Learning From Mistakes by Interacting With an Ill-Trained Reinforcement Learning Agent.

This study presents a 3D medical simulation that employs reinforcement learning (RL) and interactive reinforcement learning (IRL) to teach and assess the procedure of donning and doffing personal protective equipment (PPE). The simulation is motivated by the need for effective, safe, and remote training techniques in medicine, particularly in light of the COVID-19 pandemic. The simulation has two modes: a tutorial mode and an assessment mode. In the tutorial mode, a computer-based, ill-trained RL agent utilizes RL to learn the correct sequence of donning the PPE by trial and error. This allows students to experience many outlier cases they might not encounter in an in-class educational model. In the assessment mode, an IRL-based method is used to evaluate how effective the participant is at correcting the mistakes performed by the RL agent. Each time the RL agent interacts with the environment and performs an action, the participants provide positive or negative feedback regarding the action taken. Following the assessment, participants receive a score based on the accuracy of their feedback and the time taken for the RL agent to learn the correct sequence. An experiment was conducted using two groups, each consisting of 10 participants. The first group received RL-assisted training for donning PPE, followed by an IRL-based assessment. Meanwhile, the second group observed a video featuring the RL agent demonstrating only the correct donning order without outlier cases, replicating traditional training, before undergoing the same assessment as the first group. Results showed that RL-assisted training with many outlier cases was more effective than traditional training with only regular cases. Moreover, combining RL with IRL significantly enhanced the participants' performance. Notably, 90% of the participants finished the assessment with perfect scores within three iterations. In contrast, only 10% of those who did not engage in RL-assisted training finished the assessment with a perfect score, highlighting the substantial impact of RL and IRL integration on participants' overall achievement.

Identifying Transfer Learning in the Reshaping of Inductive Biases.

Transfer learning, the reuse of newly acquired knowledge under novel circumstances, is a critical hallmark of human intelligence that has frequently been pitted against the capacities of artificial learning agents. Yet, the computations relevant to transfer learning have been little investigated in humans. The benefit of efficient inductive biases (meta-level constraints that shape learning, often referred as priors in the Bayesian learning approach), has been both theoretically and experimentally established. Efficiency of inductive biases depends on their capacity to generalize earlier experiences. We argue that successful transfer learning upon task acquisition is ensured by updating inductive biases and transfer of knowledge hinges upon capturing the structure of the task in the inductive bias that can be reused in novel tasks. To explore this, we trained participants on a non-trivial visual stimulus sequence task (Alternating Serial Response Times, ASRT); during the Training phase, participants were exposed to one specific sequence for multiple days, then on the Transfer phase, the sequence changed, while the underlying structure of the task remained the same. Our results show that beyond the acquisition of the stimulus sequence, our participants were also able to update their inductive biases. Acquisition of the new sequence was considerably sped up by earlier exposure but this enhancement was specific to individuals showing signatures of abandoning initial inductive biases. Enhancement of learning was reflected in the development of a new internal model. Additionally, our findings highlight the ability of participants to construct an inventory of internal models and alternate between them based on environmental demands. Further, investigation of the behavior during transfer revealed that it is the subjective internal model of individuals that can predict the transfer across tasks. Our results demonstrate that even imperfect learning in a challenging environment helps learning in a new context by reusing the subjective and partial knowledge about environmental regularities.

Filter-Based Fully Distributed Output Regulation of Heterogeneous Learning Agents

Filter-Based Fully Distributed Output Regulation of Heterogeneous Learning Agents

Reinforced Computer-Aided Framework for Diagnosing Thyroid Cancer.

Thyroid cancer is the most pervasive disease in the endocrine system and is getting extensive attention. The most prevalent method for an early check is ultrasound examination. Traditional research mainly concentrates on promoting the performance of processing a single ultrasound image using deep learning. However, the complex situation of patients and nodules often makes the model dissatisfactory in terms of accuracy and generalization. Imitating the diagnosis process in reality, a practical diagnosis-oriented computer-aided diagnosis (CAD) framework towards thyroid nodules is proposed, using collaborative deep learning and reinforcement learning. Under the framework, the deep learning model is trained collaboratively with multiparty data; afterward classification results are fused by a reinforcement learning agent to decide the final diagnosis result. Within the architecture, multiparty collaborative learning with privacy-preserving on large-scale medical data brings robustness and generalization, and diagnostic information is modeled as a Markov decision process (MDP) to get final precise diagnosis results. Moreover, the framework is scalable and capable of containing more diagnostic information and multiple sources to pursue a precise diagnosis. A practical dataset of two thousand thyroid ultrasound images is collected and labeled for collaborative training on classification tasks. The simulated experiments have shown the advancement of the framework in promising performance.

Perspectives on Practice: Student-Driven Individualized Education Program Practice: Collaborating with Young Students as Literacy Learning Agents

Providing student-facing templates, this article offers protocols for teachers to engage students with their Individualized Education Program goals during literacy lessons.

A Feasible Study of a Deep Learning Model Supporting Human–Machine Collaborative Learning of Object-Oriented Programming

Due to the development of deep learning technology, its application in education has received increasing attention from researchers. Intelligent agents based on deep learning technology can perform higher-order intellectual tasks than ever. However, the high deployment cost of deep learning models has hindered their widespread application in education. In addition, there needs to be more research on applying deep learning technology in education. In this study, we develop an intelligent agent using a Performer-based encoder-decoder neural model to classify object-oriented programming (OOP) errors in student code and generate hint feedback in natural language to help students correct the code. This study investigates the feasibility of deploying this agent in an educational setting to support the learning of OOP. This study first examines the low-speed inference problem of the deep learning model. A fast inference algorithm is proposed for the model, which achieves a speedup of eighty times. This study further explores integrating a human-machine collaborative learning process with the deep learning agent. Students were surveyed about their perceptions of the agent in supporting learning. Student responses are interpreted within the learning partnerships model (LPM) framework to show how the agent's technical automation and autonomy features support student-agent learning partnerships. Finally, implications and suggestions for educational application and research of deep learning technology are presented.

SMARLA: A Safety Monitoring Approach for Deep Reinforcement Learning Agents

SMARLA: A Safety Monitoring Approach for Deep Reinforcement Learning Agents

Reinforced Metapath Optimization in Heterogeneous Information Networks for Drug-Target Interaction Prediction.

Graph neural networks offer an effective avenue for predicting drug-target interactions. In this domain, researchers have found that constructing heterogeneous information networks based on metapaths using diverse biological datasets enhances prediction performance. However, the performance of such methods is closely tied to the selection of metapaths and the compatibility between metapath subgraphs and graph neural networks. Most existing approaches still rely on fixed strategies for selecting metapaths and often fail to fully exploit node information along the metapaths, limiting the improvement in model performance. This paper introduces a novel method for predicting drug-target interactions by optimizing metapaths in heterogeneous information networks. On one hand, the method formulates the metapath optimization problem as a Markov decision process, using the enhancement of downstream network performance as a reward signal. Through iterative training of a reinforcement learning agent, a high-quality set of metapaths is learned. On the other hand, to fully leverage node information along the metapaths, the paper constructs subgraphs based on nodes along the metapaths. Different depths of subgraphs are processed using different graph convolutional neural network. The proposed method is validated using standard heterogeneous biological benchmark datasets. Experimental results on standard datasets show significant advantages over traditional methods.

Mitigating the Impact of Photovoltaic Power Ramps on Intraday Economic Dispatch Using Reinforcement Forecasting

The rapid growth of large-scale photovoltaic (PV) power generation has received considerable attention, where a major challenge is that PV power ramp events can adversely affect the operations of integrated power systems. Therefore, effective forecast and dispatch methodologies are critical. However, it is relatively difficult to predict an accurate PV power ramp rate. Another issue is that, since forecast and dispatch are typically considered in research as two separate problems, forecast modeling does not involve the knowledge of dispatch. In this case, unbiased PV forecast is unpractical for PV ramp events, increasing their negative effects. Besides, such a unified optimization with both forecast and dispatch cannot be solved easily since the problem is strongly nonlinear. Motivated by these challenges, a reinforcement forecasting framework is proposed for intraday economic dispatch in this study. First, the PV power ramp rate forecasting is categorized into two tasks of multi-classification and class expectation prediction, where the class continuity is considered in order to improve the forecast distribution. Next, a reinforcement learning agent is established for connecting PV forecast and system dispatch, which adjusts PV power generation schedules based on real-time system feedbacks. Comparative studies are conducted on an intraday system dispatch environment with energy storage. The results of the studies indicate that the proposed framework can provide higher economic benefits than those under benchmark schemes.

Deep Reinforcement Learning for Path-Following Control of an Autonomous Surface Vehicle using Domain Randomization

In this paper, we propose a path-following controller for an autonomous surface vehicle (ASV) that is based on model-free deep reinforcement learning. To make the learning agent more robust, we investigate domain randomization for sim-to-real transfer. We provide a comparison between three different algorithms: Deep Deterministic Policy Gradient (DDPG), Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO). The trained models are evaluated on the small-scale ASV Altus-LSA Niriis in the maritime test basin at DFKI RIC, Germany. Our results show that applying domain randomization leads to a significant performance improvement compared to no domain randomization, when tested on real hardware.

The Time-Sensitive Networking Scheduling Algorithm Based on Q-learning

Abstract Time-Sensitive Networking (TSN) occupies a vital position in modern communication domains, with the 802.1Qbv standard being an important network technology designed to meet real-time requirements. This standard requires network traffic to be transmitted within strict time windows, presenting challenges in network planning, necessitating efficient resource allocation and scheduling strategies. This paper addresses the 802.1Qbv planning problem through the introduction of reinforcement learning algorithms, offering an automated and intelligent solution. We have designed a reinforcement learning agent capable of perceiving network status, learning optimal resource allocation strategies, and dynamically adjusting in real-time environments. Through simulation and experimentation, we have validated the effectiveness of our proposed method, comparing it with traditional planning approaches. The contribution of this study lies in introducing a novel solution to the 802.1Qbv planning problem for time-sensitive networks, enhancing network resource utilization and performance. This approach offers strong support for the development and enhancement of TSN-like networks, holding significant importance for meeting the growing demands of real-time applications.

To Boost Zero-Shot Generalization for Embodied Reasoning With Vision-Language Pre-Training.

Recently, there exists an increased research interest in embodied artificial intelligence (EAI), which involves an agent learning to perform a specific task when dynamically interacting with the surrounding 3D environment. There into, a new challenge is that many unseen objects may appear due to the increased number of object categories in 3D scenes. It makes developing models with strong zero-shot generalization ability to new objects necessary. Existing work tries to achieve this goal by providing embodied agents with massive high-quality human annotations closely related to the task to be learned, while it is too costly in practice. Inspired by recent advances in pre-trained models in 2D visual tasks, we attempt to boost zero-shot generalization for embodied reasoning with vision-language pre-training that can encode common sense as general prior knowledge. To further improve its performance on a specific task, we rectify the pre-trained representation through masked scene graph modeling (MSGM) in a self-supervised manner, where the task-specific knowledge is learned from iterative message passing. Our method can improve a variety of representative embodied reasoning tasks by a large margin (e.g., over 5.0% w.r.t. answer accuracy on MP3D-EQA dataset that consists of many real-world scenes with a large number of new objects during testing), and achieve the new state-of-the-art performance.

Equitable access to agency-supportive early schooling contexts for young children of color

Over the past thirty years, developmental and learning sciences have started to illuminate the need for better access to anti-racist, culturally sustaining early childhood education and care programs. While physical access to programs continues to be an urgent need in the U.S., many communities of color continue to demand access to early childhood education spaces that are safe for their children to learn - programs that are working hard to root out racism and White supremacy from their policies and practices. In this study, we draw upon over 350 hours of data from teachers of young children ages 3-8 years old to offer a detailed, qualitative exploration of how racism impacts children's early learning experiences across early childhood bilingual, inclusion, and multiracial classrooms that serve majority children of color. We examine the policy implications of racism on children's access to agentic learning experiences at school. We also discuss how the findings from our study suggest guidance for programs and policies that want to increase access to agency-supportive environments for young children of color.

Object Detection for Reinforcement Learning Agents

In traditional reinforcement learning applications with images as input, the observation for the agent to learn from, is an image. In these models, a Convolutional Neural Network (CNN) is typically used to extract the features before for the learning process, in order to maximize the cumulative reward. In this paper, a different approach for pre-processing the input for reinforcement learning agents is considered. The proposed approach uses object detectors instead instead of CNNs, and converts each input image into bounding boxes and object locations for the agent to learn from.