Learning Agent Research Articles

Examining and Visualizing the Effects of Pedagogical Agents on Learning Outcomes

While there has been a growing interest in the use of animated, interactive, and embodied agents in education, there is a lack of comprehensive understanding of their impact on learning outcomes. To comprehensively examine their impact on learning outcomes, this study utilized VOSviewer and CitNetExplorer for a bibliometric analysis, examining the research trends in the field of pedagogical agents for learning. The top 10 cited authors, sources, organizations, and countries in this field were identified. The findings indicate that animated, interactive, and embodied pedagogical agents may enhance learning outcomes. Furthermore, ethnicity and prior knowledge are significant factors in influencing learning outcomes. The integration of image, audio, visuals, and narration within these agents can positively impact learning outcomes, creating a more immersive and engaging learning experience. The ethical and social implications of pedagogical agents cannot be overlooked in the future research.

Enhancing inclusive teaching. A teacher professional development research grounded in UDL principles

The international scientific literature emphasizes the importance of inclusive education, which places the valorization of all diversities at the core of every learning environment. It also highlights the Universal Design for Learning (UDL) as an educational framework that promotes accessibility in learning, enabling all students to fully participate regardless of their abilities by designing flexible and customizable educational material. In light of this, the UDL can serve as a model for crafting educational pathways aimed at supporting the academic success of all individuals while promoting full participation in creating a new democratic educational culture oriented towards equity, universality, and belonging. Against this background, the paper focuses on the implementation phases of a Teacher Professional Development Research (TPDR) pathway involving a group of in-service teachers in UDL approaches within lower secondary education. Its objective is to assess whether this pathway leads to an increase in inclusive practices in the classroom, a positive change in teachers' perception of their professional agency, and a positive impact on students' perception of their learner agency. The research aims, on the one hand, to contribute to the scientific literature on teacher professional development and inclusive education and, on the other hand, to promote the co-construction of knowledge within the teachers' community of practice, which is significant as it is deeply rooted in the real context, fostering transformative change in teaching practices and inclusive processes from an emancipatory perspective.

The agency development of teachers of an additional language via immersive VR telecollaboration

Teacher agency is essential for professional growth and teachers’ ability to support transformative student learning. This study looks into the potential of virtual reality (VR) as a joint practice teaching and reflective discussion environment for achieving language teacher agency. A group of teachers of an additional language (LX) participated in a telecollaborative project. They were provided with VR devices and language learning software, Immerse, for practice teaching and reflective discussions after each VR teaching. The data of this qualitative study were video recordings of the teaching and discussion sessions and the participants’ answers to an end-of-project questionnaire based on the Cognitive Affective Model of Immersive Learning (CAMIL) by Makransky and Petersen in 2021. The video recording analysis results showed that the participants used almost all the teaching facilitation functions (e.g., tools that allowed teachers to present content, conduct classroom management, and monitor students) provided by Immerse. The themes that emerged from the participants’ reflective discussions analysis also indicated mutual support for teacher agency development. Their responses to the questionnaire revealed that taking advantage of the teaching facilitation functions reflected the factors affecting learner agency as outlined in the CAMIL model. The findings of this study provided empirical evidence that VR environments can be a viable choice for teacher intercultural collaboration in which teachers of different backgrounds and experiences could together explore the potentiality of VR for teaching an additional language, sustain one another’s growth, and achieve teacher agency.

Training Reinforcement Learning Agents to React to an Ambush for Military Simulations

Training Reinforcement Learning Agents to React to an Ambush for Military Simulations

Reinforcement Learning Agents with Generalizing Behavior

Reinforcement Learning Agents with Generalizing Behavior

A learning analytics‐based collaborative conversational agent to foster productive dialogue in inquiry learning

AbstractBackgroundSustaining productive student–student dialogue in online collaborative inquiry learning is challenging, and teacher support is limited when needed in multiple groups simultaneously. Collaborative conversational agents (CCAs) have been used in the past to support student dialogue. Yet, research is needed to reveal the characteristics and effectiveness of such agents.ObjectivesTo investigate the extent to which our analytics‐based Collaborative Learning Agent for Interactive Reasoning (Clair) can improve the productivity of student dialogue, we assessed both the levels at which students shared thoughts, listened to each other, deepened reasoning, and engaged with peer's reasoning, as well as their perceived productivity in terms of their learning community, accurate knowledge, and rigorous thinking.MethodIn two separate studies, 19 and 27 dyads of secondary school students from Brazil and the Netherlands, respectively, participated in digital inquiry‐based science lessons. The dyads were assigned to two conditions: with Clair present (treatment) or absent (control) in the chat. Sequential pattern mining of chat logs and the student's responses to a questionnaire were used to evaluate Clair's impact.ResultsAnalysis revealed that in both studies, Clair's presence resulted in dyads sharing their thoughts at a higher frequency compared to dyads that did not have Clair. Additionally, in the Netherlands' study, Clair's presence led to a higher frequency of students engaging with each other's reasoning. No differences were observed in students' perceived productivity.ConclusionThis work deepens our understanding of how CCAs impact student dialogue and illustrates the importance of a multidimensional perspective in analysing the role of CCAs in guiding student dialogue.

An Ecological Perspective on Agency: L2 Learners’ Sociopragmatic Interpretations and Strategies in a Study Abroad Context

In applied linguistics, there has been a growing body of L2 pragmatics studies that investigate the intricate relationship between language learners’ subjective pragmatic choices and various contextual factors. The current study contributes to the understanding of language learner agency by illustrating the complex processes through which language learners enact their agency in response to varying contextual factors when making their sociopragmatic interpretations and strategies. To capture the ecological nature of agency, this study conceptualizes agency as contextually, interpersonally, intrapersonally, spatially, and temporally embedded. Through the in-depth examination of the accounts of three L2 learners of Japanese regarding their interactions with religious group members during their study abroad in Japan, this study demonstrates language learners’ divergent ways of enacting their agency, stemming from their orientations to the unique configurations of various contextual attributes in the L2 interactions. Such differences were rooted in and guided by their past experiences, present environments, and future aspirations that impacted their sociopragmatic perceptions, expectations, and choices. This study provides a complicated picture of language learner agency as a dialogic and reflexive process in which learners interact with contextual factors and adapt their sociopragmatic choices. Thus, it calls for an ecological, processual, and holistic approach to language learner agency through the close examination of the ways in which various contextual factors come together in L2 interactions, the process of how language learners orient to dynamic configurations of contextual factors, and what guides such orientations.

Multi-hop temporal knowledge graph reasoning with multi-agent reinforcement learning

Knowledge graph (KG) is a key component of artificial intelligence. In recent years, many large-scale knowledge graphs have been produced and put into practical applications. At present, researchers have proposed many methods to reason facts that do not exist in knowledge graphs using the existing information. However, most traditional reasoning methods lack interpretability, and cannot get the reasoning paths. Therefore, the multi-hop path reasoning method of knowledge graph has gradually become a hot spot. This method finds the next relations through the reinforcement learning agent, gets the whole path, and scores the path. Although these multi-hop path reasoning methods make the reasoning method interpretable, these multi-hop path reasoning methods focus on the choice of relations, ignoring the importance of entities in the path reasoning process. Moreover, with the development of the Temporal Knowledge Graph (TKG), traditional multi-hop path reasoning methods cannot effectively process time information. To solve these problems, a multi-hop path reasoning method of temporal knowledge graph based on multi-agent reinforcement learning is proposed, which is named MA-TPath. MA-TPath uses two agents to perform relation selection and entity selection iteratively. Meanwhile, considering the diversity of temporal reasoning paths, we propose a new type of reward function. In MA-TPath, two agents employ the Long Short-Term Memory Networks (LSTM) to capture current results from the environment and output the corresponding action vectors to the environment through activation functions. Experimentally, MA-TPath outperforms all existing models in 4 out of 4 indicators on the YAGO dataset, 3 out of 4 indicators on the GDETL-5 dataset and the ICEWS-250 dataset, and 1 out of 4 indicators on the ICEWS05–15 dataset. Analysis of the temporal reasoning path indicates that MA-TPath not only surpasses other state-of-the-art reasoning methods over four public temporal knowledge graph datasets but provides rationality for results.

Measuring the effects of unconventional monetary policy tools under adaptive learning

We compare the economic effects of forward guidance and quantitative easing utilizing the four-equation New Keynesian model of Sims et al. (2023) with agents forming expectations via an adaptive learning rule. The results indicate forward guidance can have a greater influence on macroeconomic variables compared to quantitative easing. Adaptive learning agents estimate a higher effect of forward guidance on the economy leading to a greater impact on expectations, and thus, contemporaneous inflation. However, the performance gap between forward guidance and quantitative easing can change. If quantitative easing includes anticipated shocks, more households finance consumption through long-term borrowing, and the central bank provides a greater percentage of liquidity in the long-term borrowing market, the performance of quantitative easing can increase, and at times, outperform forward guidance.

Online energy management strategy for ammonia-hydrogen hybrid electric vehicles harnessing deep reinforcement learning

Powertrain electrification and fuel decarbonization play pivotal roles in the pursuit of carbon peak and carbon neutrality within the realm of transportation. Ammonia and hydrogen are essential carbon-neutral fuels suited for deployment in heavy-duty, long-haul transportation applications. The technological complementarity of ammonia and hydrogen can be realized in the current context by combining various energy conversion devices. Energy management strategy (EMS) is one of the critical technologies in hybrid systems. In this study, a novel EMS based on deep reinforcement learning (DRL) is proposed for a heavy-duty automotive hybrid system containing an ammonia-hydrogen internal combustion engine (ICE), a fuel cell system (FCS), an ammonia electrolysis cell (AEC), and Li-ion batteries (LB). To address the protracted training times and convergence challenges intrinsic to DRL, a fuzzy logic control (FLC) is introduced to provide expert demonstrations during the DRL agent's learning process. The incorporation of FLC not only facilitates the convergence speed of the DRL algorithm, but also strikes a good balance between the operation efficiency improvement and battery SOC maintenance. The simulation results show that the proposed EMS improves average efficiency by 2 % with a modest 0.75 % SOC reduction across four operational scenarios compared to the FLC-based EMS. After thorough validation of the proposed EMS, a parametric study is conducted to examine three key parameters: charge mode threshold, FCS power change rate limit, and hybrid mode threshold. The study aims to offer valuable insights for formulating the EMS of the ammonia-hydrogen hybrid system.

Adaptive reinforcement learning-based control using proximal policy optimization and slime mould algorithm with experimental tower crane system validation

This paper presents a novel optimal reference tracking control approach resulted from the combination of a popular policy gradient Reinforcement Learning (RL) algorithm, namely Proximal Policy Optimization (PPO), and a metaheuristic Slime Mould Algorithm (SMA). One of the most important parameters in the PPO-based RL process is the learning rate, which has a big impact on how the parameters of the actor neural network (NN) are iteratively updated. In every episode of the RL process, the weights and the biases of the actor NN are multiplied with the learning rate, determining how much the learning agent will step into a certain direction computed based on previous experiences. The classical PPO algorithm usually relies on fixed values for the learning rates which rarely change, or not at all, during the learning process. However, its main drawback is that the learning agent cannot take advantage of positive momentum in the learning process by accelerating towards good learning experiences or slow down and quickly change the direction in the case of consecutive negative learning experiences. The main objective of the combination proposed in this paper is to create an adaptive SMA-based PPO approach applied to control systems, which instead of using fixed learning rate values, it uses the SMA to compute optimal values of the learning rates in each time step of the learning process based on the progress of the learning agent. This paper investigates if the adaptive SMA-based PPO control approach can be considered as an alternative to the classical PPO version, which employs fixed values of the learning rate. A comparison is carried out using control system performance indices gathered while performing an optimal reference tracking control task on tower crane system laboratory equipment.

Sequential Label Enhancement.

Label distribution learning (LDL) is a novel machine learning paradigm for solving ambiguous tasks, where the degree to which each label describing the instance is ambiguous. However, obtaining the label distribution is high cost and the description degree is difficult to quantify. Most existing research works focus on designing an objective function to obtain the whole description degrees at once but seldom care about the sequentiality in the process of recovering the label distribution. In this article, we formulate the label distribution recovering task as a sequential decision process called sequential label enhancement (Seq_LE), which is more consistent with the process of annotating the label distribution in human brains. Specifically, the discrete label and its description degree are serially mapped by the reinforcement learning (RL) agent. Besides, we carefully design a joint reward function to drive the agent to fully learn the optimal decision policy. Extensive experiments on 16 LDL datasets are conducted under various evaluation metrics. The experimental results demonstrate convincingly that the proposed sequential label enhancement (LE) leads to better performance over the state-of-the-art methods.

Reinforcement Learning for Efficient Power Systems Planning: A Review of Operational and Expansion Strategies

The efficient planning of electric power systems is essential to meet both the current and future energy demands. In this context, reinforcement learning (RL) has emerged as a promising tool for control problems modeled as Markov decision processes (MDPs). Recently, its application has been extended to the planning and operation of power systems. This study provides a systematic review of advances in the application of RL and deep reinforcement learning (DRL) in this field. The problems are classified into two main categories: Operation planning including optimal power flow (OPF), economic dispatch (ED), and unit commitment (UC) and expansion planning, focusing on transmission network expansion planning (TNEP) and distribution network expansion planning (DNEP). The theoretical foundations of RL and DRL are explored, followed by a detailed analysis of their implementation in each planning area. This includes the identification of learning algorithms, function approximators, action policies, agent types, performance metrics, reward functions, and pertinent case studies. Our review reveals that RL and DRL algorithms outperform conventional methods, especially in terms of efficiency in computational time. These results highlight the transformative potential of RL and DRL in addressing complex challenges within power systems.

Composing Synergistic Macro Actions for Reinforcement Learning Agents.

Macro actions have been demonstrated to be beneficial for the learning processes of an agent and have encouraged a variety of techniques to be developed for constructing more effective ones. However, previous techniques usually do not further consider combining macro actions to form a synergistic macro action ensemble, in which synergism exhibits when the constituent macro actions are favorable to be jointly used by an agent during evaluation. Such a synergistic macro action ensemble may potentially allow an agent to perform even better than the individual macro actions within it. Motivated by the recent advances of neural architecture search (NAS), in this brief, we formulate the construction of a synergistic macro action ensemble as a Markov decision process (MDP) and evaluate the constructed macro action ensemble as a whole. Such a problem formulation enables synergism to be taken into account by the proposed evaluation procedure. Our experimental results demonstrate that the proposed framework is able to discover the synergistic macro action ensembles. Furthermore, we also highlight the benefits of these macro action ensembles through a set of analytical cases.

Learner Agency: Multiple Case Studies of English Speaking Learning of English Majors in China

Objectives This article delves into the ways in which three English major students, each possessing varying levels of proficiency in spoken English―high, moderate, and low―express their agency throughout the process of acquiring spoken language. This exploration encompasses their experiences both within and beyond the classroom, all situated within a broader contextual framework. Methods Qualitative research was conducted during one fall semester from September to December 2023. The researchers observed an English speaking course instructed by a Chinese female teacher. Data were collected through classroom observations, semi-structured interviews, course documents and learners’ artifacts. Data analysis was conducted with three levels of coding (open coding, axial coding, and selective coding) and the credibility was ensured with data triangulation. Results The findings revealed that English major students, when encountering difficulties and challenges, such as pronunciation problems and limited vocabulary etc, exercised different forms of agency to address and overcome obstacles in the process of learning spoken English. Specifically, the agency forms of English major students in spoken English learning is manifested through action engagement, action adjustment, and action resistance. Conclusions The findings and the analysis of the current study revealed that learner agency had a significant role to play in English as a foreign language classroom in a Chinese college context. Based on the thematic data analysis, this study offered some pedagogical implications for educators and teachers both at Chinese universities and beyond.

Sim-to-Real Application of Reinforcement Learning Agents for Autonomous, Real Vehicle Drifting

Enhancing the safety of passengers by venturing beyond the limits of a human driver is one of the main ideas behind autonomous vehicles. While drifting is mostly witnessed in motorsports as an advanced driving technique, it could provide many possibilities for improving traffic safety by avoiding accidents in extreme traffic situations. The purpose of the research presented in this article is to provide a machine learning-based solution to autonomous drifting as a proof of concept for vehicle control at the limits of handling. To achieve this, reinforcement learning (RL) agents were trained for the task in a MATLAB/Simulink-based simulation environment, using the state-of-the-art Soft Actor–Critic (SAC) algorithm. The trained agents were tested in reality at the ZalaZONE proving ground on a series production sports car with zero-shot transfer. Based on the test results, the simulation environment was improved through domain randomization, until the agent could perform the task both in simulation and in reality on a real test car.

Deep Learning Pricing of Processing Firms in Agricultural Markets

The pricing behavior of agricultural processing firms in input markets has large impacts on farmers’ and processors’ prosperity as well as the overall market structure. Despite analytical approaches to food processors’ pricing in agricultural input markets, the need for models to represent complex market features is urgent. Agent-based models (ABMs) serve as computational laboratories to understand complex markets emerging from autonomously interacting agents. Yet, individual agents within ABMs must be equipped with intelligent learning algorithms. In this paper, we propose supervised and unsupervised learning agents to simulate the pricing behavior of firms in agricultural markets’ ABMs. Supervised learning firms are pre-trained to accurately best respond to their competitors and are deemed to result in the market Nash equilibria. Unsupervised learning firms play a course of pricing interaction with their competitors without any pre-knowledge but based on deep reinforcement learning. The simulation results show that unsupervised deep learning firms are capable of approximating the pricing equilibria obtained by the supervised firms in different spatial market settings. Optimal discriminatory and uniform delivery pricing emerges in agricultural input markets with the high and intermediary importance placed on space. Free on board pricing emerges in agricultural input markets with small importance placed on space.

Comprehensible Extraction of Knowledge Bases for Learning Agents in Games

AbstractThis dissertation abstract summarizes results of the thesis “Comprehensible Knowledge Base Extraction for Learning Agents - Practical Challenges and Applications in Games” (accepted as dissertation at the Department of Computer Science of TU Dortmund University, Germany). The thesis presents approaches that allow for the automated creation of knowledge bases from agent behavior learned in the context of games. The aims are twofold: (1) The creation of human-readable knowledge that can provide insights into what an agent learned, and (2) the investigation of how learning agents themselves can benefit from incorporating these approaches into their learning processes. Applications are presented, e.g., in the context of general video game playing. Moreover, an outlook on the InteKRator toolbox is provided which implements the most essential approaches in a more general context for the potential use in other domains (e.g. in medical informatics).

A framework for designing Reinforcement Learning agents with Dynamic Difficulty Adjustment in single-player action video games

Dynamic Difficulty Adjustment (DDA) within video games aims to avoid frustration or boredom. This paper provides the first framework for designing a Reinforcement Learning (RL) agent with DDA for single-player action video games. The framework includes the definitions of states, actions, and rewards. We propose a 2Q-table system that can provide a better winning/losing ratio and extend the duration of the rounds. We apply the framework to a use case study. We address the challenges that the design and implementation of RL agents with DDA for single-player action video games might present, such as (i) large and/or continuous action–state spaces, (ii) an appropriate definition of the rewards for achieving a correct DDA, (iii) learning from each player online from limited samples and (iv) in an arcade shooter video game. The two evaluations performed (with computer-driven and human players) show that the paper’s goals are met since players face personalized challenges according to their playing skills.

Enhancing reinforcement learning based adversarial malware generation to evade static detection

The anti-detection capabilities of adversarial malware examples have drawn the attention of antivirus vendors and researchers. In black-box scenarios where internal information of the target model cannot be accessed, with the ability of the reinforcement learning (RL) agents to adjust strategies based on the feedback from the environment, existing RL-based methods enable evasion of Windows PE malware detectors. However, obtaining evasion rewards as positive feedback in the black-box setting proves challenging, resulting in low training efficiency. To address this issue, we introduce the intrinsic curiosity reward into the framework to motivate the agent to explore unknown state spaces and learn effective evasion strategies. Additionally, we employ the generative adversarial network (GAN) to obtain varying synthetic data, which replaces random or benign bytes as adversarial payloads for the agent's action content, improving attack capabilities and reducing the risk of hard-coded adversarial perturbations being anchored. We compare the attack performance with other RL-based baseline methods, and experimental results show that our framework is more flexible and effective, achieving a 63%-85% attack success rate against EMBER, FireEye and MalConv. Even if defensive measures are taken, the proposed method still has a certain attack capability, and the success rate remains between 48%-67%.