Learning Agent Research Articles

Estimating reaction barriers with deep reinforcement learning1

Stable states in complex systems correspond to local minima on the associated potential energy surface. Transitions between these local minima govern the dynamics of such systems. Precisely determining the transition pathways in complex and high-dimensional systems is challenging because these transitions are rare events, and isolating the relevant species in experiments is difficult. Most of the time, the system remains near a local minimum, with rare, large fluctuations leading to transitions between minima. The probability of such transitions decreases exponentially with the height of the energy barrier, making the system’s dynamics highly sensitive to the calculated energy barriers. This work aims to formulate the problem of finding the minimum energy barrier between two stable states in the system’s state space as a cost-minimization problem. It is proposed to solve this problem using reinforcement learning algorithms. The exploratory nature of reinforcement learning agents enables efficient sampling and determination of the minimum energy barrier for transitions.

A federated advisory teacher–student framework with simultaneous learning agents

Multi-agent reinforcement learning requires numerous interactions with the environment and other agents to learn an optimal policy. The teacher–student framework is one paradigm that can enhance the learning performance of reinforcement learning by allowing agents to seek advice from one another. However, recent studies show limitations in knowledge sharing between agents, as advisory learning is only peer-to-peer at a given time. Some methods enable a student to accept multiple pieces of advice, but they typically rely on pre-trained and/or policy-fixed teachers, rendering them unsuitable for agents with simultaneous advisory learning. Simultaneous learning with multiple pieces of advice has not been thoroughly investigated. Furthermore, most research has concentrated on the sharing of knowledge samples, a practice vulnerable to security breaches that could allow attackers to deduce details about the environment. To address these challenges, we propose a federated advisory framework that uses a federated learning structure to aggregate multiple sources of advice with deep reinforcement learning, ensuring that the shared advice is not sample-based. Our experimental comparisons with leading advisory learning techniques confirm that our approach significantly enhances learning performance.

Transformation of the Values of Agape Love as a Learning Agent for Moral Change in the Context of Theology Student

Transformation of the Values of Agape Love as a Learning Agent for Moral Change in the Context of Theology Student

Deep reinforcement learning augmented decision model for intelligent driving cars

In view of the fact that the state machine decision model cannot effectively handle the rich contextual information and the influence of uncertain factors in ice and snow environments, a deep reinforcement learning agent based on the deep Q network algorithm (DQN) was constructed. The motion planner was used to augment the agent, and the rule-based decision planning module and the deep reinforcement learning model were integrated to establish the DQN-planner model, thereby improving the convergence speed and driving ability of the reinforcement learning agent. Finally, based on the CARLA simulation platform, a comparative experiment was conducted on the driving ability of the DQN model and the DQN-planner model on low-adhesion ice and snow roads, and the training process and verification results were analyzed respectively.

A deep reinforcement learning framework for achieving super-human hazard perception in autonomous driving agents

Modeling driver behavior can significantly improve autonomous vehicles and intelligent driver assistance/training systems. Hazard perception (HP) is an essential driving skill for preventing driving accidents in potentially dangerous traffic situations. HP is not a Markov decision process, so modeling HP with reinforcement learning techniques is challenging. We introduce a framework for modeling HP scenarios using the soft actor-critic approach. Then, we use this framework to model HP in the emergence of a critical lateral hazard onto the road. The dual environments – the mirror environment for training and the target environment for usage – accelerate the learning process. The agent is employed in the target environment using an interpreter algorithm. The results show that the agent learns HP and outperforms the best human results by 2.6%. The training is implemented 40 times faster in the mirror environment. Evaluation tests show a driving experience similar to real-world scenarios rather than gaming simulations.

Toward a synergistic framework of investment and motivation: Bridging imagined identities and English learning of Chinese university students

AbstractImagined identity, which opens up a variety of future possibilities for language learners, has been theorized from seemingly incompatible psychological and sociological perspectives in applied linguistics. Researchers have thus appealed for a transdisciplinary theoretical combination of these perspectives for a more nuanced understanding of language learning in context. This article makes the case for a synergistic theoretical framework of investment and motivation to theorize the idiosyncratic relationship between individuals’ imagined identity and language learning behavior. Empirical findings from a multiple‐case study conducted in Yunnan, in the south‐western corner of China, are presented to demonstrate how this framework enables a nuanced analysis of individuals’ language learning. Findings suggest that the interconnectedness of imagined identity, motivation, and investment is informed by both learner agency and contexts. This paper proposes that learners’ development of imagined identities and ways of investing in learning could be individually different, even under similar circumstances, which indicates the need for personalized teaching to improve learning outcomes. This synergistic framework thus provides implications for pedagogical practices and a need for personalized teaching.

Adaptive Multi-Agent Reinforcement Learning for Optimizing Dynamic Electric Vehicle Charging Networks in Thailand

The rapid growth of electric vehicles (EVs) necessitates efficient management of dynamic EV charging networks to optimize resource utilization and enhance service reliability. This paper explores the application of adaptive multi-agent reinforcement learning (MARL) to address the complexities of EV charging infrastructure in Thailand. By employing MARL, multiple autonomous agents learn to optimize charging strategies based on real-time data by adapting to fluctuating demand and varying electricity prices. Building upon previous research that applied MARL to static network configurations, this study extends the application to dynamic and real-world scenarios, integrating real-time data to refine agent learning processes and also evaluating the effectiveness of adaptive MARL in maximizing rewards and improving operational efficiency compared to traditional methods. Experimental results indicate that MARL-based strategies increased efficiency by 20% and reduced energy costs by 15% relative to conventional algorithms. Key findings demonstrate the potential of extending MARL in transforming EV charging network management, highlighting its benefits for stakeholders, including EV owners, operators, and utility providers. This research contributes insights into advancing electric mobility and energy management in Thailand through innovative AI-driven approaches. The implications of this study include significant improvements in the reliability and cost-effectiveness of EV charging networks, fostering greater adoption of electric vehicles and supporting sustainable energy initiatives. Future research directions include enhancing MARL adaptability and scalability as well as integrating predictive analytics for proactive network optimization and sustainability. These advancements promise to further refine the efficacy of EV charging networks, ensuring that they meet the growing demands of Thailand’s evolving electric mobility landscape.

A novel reinforcement learning agent for rotating machinery fault diagnosis with data augmentation

The problem of data imbalance and limited sample poses a prominent challenge to fault diagnosis in industry, especially in rotating machinery fault diagnosis due to the constraints of sampling and labeling. Reinforcement learning is known for its auto-optimized learning ability but it is severely limited by the quality and quantity of data it is able to collect. This paper applies the Equilibrium Deep Q-Network (Equilibrium-DQN) based agent with the Variational Autoencoder with Wasserstein Generative Adversarial Network and Gradient Penalty (VAE-WGAN-GP) for rotating machinery fault diagnosis. The Equilibrium-DQN possesses the property of leveraging multiple target networks to improve training stability and accuracy. The VAE-WGAN-GP excels in resolving data scarcity problems in fault diagnosis, especially in enriching labeled data with more typical samples while keeping the original data features. Herein, a novel framework is presented that integrates both data augmentation techniques with reinforcement learning agent for fault diagnosis. The proposed framework improves the weakness of insufficient samples in fault diagnosis and reveals its potential in complex industrial applications. The experiments, both on the test bench dataset and on the real locomotive dataset, confirm the advantage of the proposed framework.

Hybrid lane change strategy of autonomous vehicles based on SOAR cognitive architecture and deep reinforcement learning

Research on lane change strategies for autonomous vehicles holds paramount importance in optimizing traffic flow efficiency, enhancing driving safety, and adapting to complex traffic environments. While numerous rule-based or machine-learning approaches have been explored to tackle the challenge of lane change on highways, they frequently exhibit limited performance owing to the complexity of driving environments. This study proposes a novel lane change strategy for autonomous vehicles, which utilizes a hybrid framework integrating the SOAR cognitive architecture and deep reinforcement learning (DRL) to address the lane change challenge on highways. First, we introduce a rule extraction algorithm, the RuleCOSI+, which is based on tree ensemble algorithms, designed to extract concise lane change rules from large-scale human driving data. These straightforward rules, together with traffic regulations and safety rules, constitute the long-term memory of the SOAR cognitive architecture, enabling transparent decision-making processes. Next, by analyzing the clipping mechanism of the proximal policy optimization (PPO) algorithm, we propose an Adaptive Clipping PPO (ACPPO) algorithm which is based on the importance of samples. This algorithm adopts different clipping strategies for SOAR samples and ACPPO samples during the training process, enabling the algorithm to more effectively utilize samples with different levels of importance. Then, we propose a hybrid decision-making algorithm: SOAR-ACPPO, which combines the SOAR cognitive architecture with the ACPPO algorithm. This algorithm leverages SOAR’s prior knowledge to effectively and safely guide agent learning. Finally, by selecting appropriate intervention probability and weaning strategy, the system avoids inappropriate knowledge intervention and ensures adequate environment exploration. Simulation experiments conducted using the CARLA simulator illustrate that the proposed strategy not only improves model learning efficiency but also enhances driving efficiency and safety. Additionally, it demonstrates a certain degree of human-like characteristics and interpretability.

Inverse reinforcement learning methods for linear differential games

In this study, we considered the problem of inverse reinforcement learning or estimating the cost function of expert players in multi-player differential games. We proposed two online data-driven solutions for linear–quadratic games that are applicable to systems that fulfill a specific dimension criterion or whose unknown matrices in the cost function conform to a diagonal condition. The first method, which is partially model-free, utilizes the trajectories of expert agents to solve the problem. The second method is entirely model-free and employs the trajectories of both expert and learner agents. We determined the conditions under which the solutions are applicable and identified the necessary requirements for the collected data. We conducted numerical simulations to establish the effectiveness of the proposed methods.

Optimizing bucket-filling strategies for wheel loaders inside a dream environment

Reinforcement Learning (RL) requires many interactions with the environment to converge to an optimal strategy, which makes it unfeasible to apply to wheel loaders and the bucket filling problem without using simulators. However, it is difficult to model the pile dynamics in the simulator because of unknown parameters, which results in poor transferability from the simulation to the real environment. Instead, this paper uses world models, serving as a fast surrogate simulator, creating a dream environment where a reinforcement learning (RL) agent explores and optimizes its bucket-filling behavior. The trained agent is then deployed on a full-size wheel loader without modifications, demonstrating its ability to outperform the previous benchmark controller, which was synthesized using imitation learning. Additionally, the same performance was observed as that of a controller pre-trained with imitation learning and optimized on the test pile using RL.

Optimization of pressure management strategies for geological CO2 storage using surrogate model-based reinforcement learning

Injecting greenhouse gas (e.g. CO2) into deep underground reservoirs for permanent storage can inadvertently lead to fault reactivation, caprock fracturing and greenhouse gas leakage when the injection-induced stress exceeds the critical threshold. It is essential to monitor the evolution of pressure and the movement of the CO2 plume closely during the injection to allow for timely remediation actions or rapid adjustments to the storage design. Extraction of pre-existing fluids at various stages of the injection process, referred as pressure management, can mitigate associated risks and lessen environmental impact. However, identifying optimal pressure management strategies typically requires thousands of simulations, making the process computationally prohibitive. This paper introduces a novel surrogate model-based reinforcement learning method for devising optimal pressure management strategies for geological CO2 sequestration efficiently. Our approach comprises of two steps. The first step involves developing a surrogate model using the embed to control method, which employs an encoder-transition-decoder structure to learn dynamics in a latent or reduced space. The second step, leveraging this proxy model, utilizes reinforcement learning to find an optimal strategy that maximizes economic benefits while satisfying various control constraints. The reinforcement learning agent receives the latent state representation and immediate reward tailored for CO2 sequestration and choose real-time controls which are subject to predefined engineering constraints in order to maximize the long-term cumulative rewards. To demonstrate its effectiveness, this framework is applied to a compositional simulation model where CO2 is injected into saline aquifer. The results reveal that our surrogate model-based reinforcement learning approach significantly optimizes CO2 sequestration strategies, leading to notable economic gains compared to baseline scenarios.

Personal Agency and Career Development of Adolescent Learners

Personal Agency and Career Development of Adolescent Learners

Learning placement order for constructive floorplanning

Floorplanning is an early and essential task of physical design. Recently, there has been a surge in the application of learning-based methods to tackle floorplanning problem. A prevalent approach involves training a reinforcement learning (RL) agent to sequentially place blocks on a chip canvas. However, existing methods mainly focus on learning block placement, relying on heuristic rules for placement order determination. In contrast to previous approaches, we propose an RL-based method to determine the placement order. Based on block features and states, an agent is trained to select the block for placement. Once a block is selected, we enumerate all potential relative positions captured by sequence pairs and select the optimal placement. After establishing the layout topology, we further optimize wirelength through linear programming. Experimental results demonstrate the effectiveness of our proposed method. On the original-outline MCNC benchmarks, our method achieves a notable 25.2% average improvement in wirelength compared to a recent learning-based method. Additionally, when applied to rescaled-outline benchmarks from MCNC and GSRC, our method outperforms state-of-the-art results, resulting in an average wirelength reduction of 12.5%.

Towards multi-agent system for learning object recommendation

The rapid increase of online educational content has made it harder for students to find specific information. E-learning recommender systems help students easily find the learning objects they require, improving the learning experience. The effectiveness of these systems is further improved by integrating deep learning with multi-agent systems. Multi-agent systems facilitate adaptable interactions within the system's various parts, and deep learning processes extensive data to understand learners' preferences. This collaboration results in custom-made suggestions that cater to individual learners. Our research introduces a multi-agent system tailored for suggesting learning objects in line with learners' knowledge levels and learning styles. This system uniquely comprises four agents: the learner agent, the tutor agent, the learning object agent, and the recommendation agent. It applies the Felder and Silverman model to pinpoint various student learning styles and organizes educational content based on the newest IEEE Learning Object Metadata standard. The system uses advanced techniques, such as Convolutional Neural Networks (CNN) and Multilayer Perceptrons (MLP), to propose learning objects. In terms of creating personalized learning experiences, this system is a considerable step forward. It effectively suggests learning objects that closely match each learner's personal profile, greatly enhancing student engagement and making the learning process more efficient.

Online probabilistic knowledge distillation on cryptocurrency trading using Deep Reinforcement Learning

Leveraging Deep Reinforcement Learning (DRL) for training agents for financial trading has gained significant attention in recent years. However, training these agents in noisy financial environments remains challenging and unstable, significantly impacting their performance as trading agents, as the recent literature has also showcased. This paper introduces a novel distillation method for DRL agents, aiming to improve the training stability of DRL agents. The proposed method transfers knowledge from a teacher ensemble to a student model, incorporating both the action probability distribution knowledge from the output layer, as well as the knowledge from the intermediate layers of the teacher’s network. Furthermore, the proposed method also works in an online fashion, allowing for eliminating the separate teacher training process typically involved in many DRL distillation pipelines, simplifying the distillation process. The proposed method is extensively evaluated on a large-scale cryptocurrency trading setup, demonstrating its ability to both lead to significant improvements in trading accuracy and obtained profit, as well as increase the stability of the training process.

Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualization.

A key challenge of continual reinforcement learning (CRL) in dynamic environments is to promptly adapt the reinforcement learning (RL) agent's behavior as the environment changes over its lifetime while minimizing the catastrophic forgetting of the learned information. To address this challenge, in this article, we propose DaCoRL, that is, dynamics-adaptive continual RL. DaCoRL learns a context-conditioned policy using progressive contextualization, which incrementally clusters a stream of stationary tasks in the dynamic environment into a series of contexts and opts for an expandable multihead neural network to approximate the policy. Specifically, we define a set of tasks with similar dynamics as an environmental context and formalize context inference as a procedure of online Bayesian infinite Gaussian mixture clustering on environment features, resorting to online Bayesian inference to infer the posterior distribution over contexts. Under the assumption of a Chinese restaurant process (CRP) prior, this technique can accurately classify the current task as a previously seen context or instantiate a new context as needed without relying on any external indicator to signal environmental changes in advance. Furthermore, we employ an expandable multihead neural network whose output layer is synchronously expanded with the newly instantiated context and a knowledge distillation regularization term for retaining the performance on learned tasks. As a general framework that can be coupled with various deep RL algorithms, DaCoRL features consistent superiority over existing methods in terms of stability, overall performance, and generalization ability, as verified by extensive experiments on several robot navigation and MuJoCo locomotion tasks.

Policy learning for policy change on energy efficiency in European companies

The EU energy efficiency directive (EED) includes provisions to stimulate increased energy efficiency in companies. Mandatory provisions were first introduced in 2012 and recast in 2023. Policy learning has been suggested as an important route to policy change. This paper analyses how and why policy learning helped revising EU legislation to enhance energy efficiency in companies, using provisions of mandatory energy audits as a case. Negative experience from member states’ governments with the original provisions were voiced shortly after the adoption of EED. A complex process going back and forth between member state and EU levels led by a learning agent facilitated collective learning and change of beliefs, first in member states then in the Council. Several cognitive biases among individuals in the European Commission led to non-learning at the individual level and blocked learning at the collective level. This further blocked policy learning in the EU when EED was amended in 2018. However, external crises and the entering office of a new Commission College in 2019 made the Commission to reconsider its beliefs. Political leadership opened a window for individual and collective learning in the Commission and policy learning in the EU when provisions were changed with the recast of EED in 2023. This suggests that individual and collective learning in the EC is key for policy change to happen. Without new beliefs in the EC, it is hard to get a topic onto the policy agenda. The paper proposes policy recommendations on how to facilitate policy learning and suggests areas for further research.

Too hot to print, too slow to handle; finding optimal path characteristics for WAAM

Wire + Arc Additive Manufacturing (WAAM) is recognized as a highly capable metal additive process. However, the quality of manufactured parts is often compromised by defects stemming from process-induced temperature fields and transient weld-bead shapes, largely due to suboptimal path planning. Recent efforts have predominantly focused on optimizing either the thermal aspects or the productivity of WAAM, but the interplay between these factors and the final geometric accuracy of the parts has not been thoroughly investigated. This study introduces an automated framework for path planning optimization in WAAM, effectively bridging the gap between temperature control, geometric accuracy, and productivity. The approach utilizes a reinforcement learning agent to generate paths that minimize deposition time. These paths are subsequently segmented and optimized for their temperature response using a Monte Carlo tree search algorithm, with the thermal effects efficiently approximated through a reduced-order model. The paper presents a comparative analysis of various deposition strategies, offering insights and recommendations to enhance both productivity and part quality in WAAM path planning.

Machine-directed Gravitational-wave Counterpart Discovery

Joint observations in electromagnetic and gravitational waves shed light on the physics of objects and surrounding environments with extreme gravity that are otherwise unreachable via siloed observations in each messenger. However, such detections remain challenging due to the rapid and faint nature of counterparts. Protocols for discovery and inference still rely on human experts manually inspecting survey alert streams and intuiting optimal usage of limited follow-up resources. Strategizing an optimal follow-up program requires adaptive sequential decision-making given evolving light curve data that (i) maximizes a global objective despite incomplete information and (ii) is robust to stochasticity introduced by detectors/observing conditions. Reinforcement learning (RL) approaches allow agents to implicitly learn the physics or detector dynamics and the behavior policy that maximizes a designated objective through experience. To demonstrate the utility of such an approach for the kilonova follow-up problem, we train a toy RL agent with the goal of maximizing follow-up photometry for the true kilonova among several contaminant transient light curves. In a simulated environment where the agent learns online, it achieves 3× higher accuracy compared to a random strategy. However, it is surpassed by human agents by up to a factor of 2. This is likely because our hypothesis function (Q that is linear in state-action features) is an insufficient representation of the optimal behavior policy. More complex agents could perform at par or surpass human experts. Agents like these could pave the way for machine-directed software infrastructure to efficiently respond to next generation detectors, for conducting science inference and optimally planning expensive follow-up observations, scalably and with demonstrable performance guarantees.