Learning Agent Research Articles

Finding the optimal exploration-exploitation trade-off online through Bayesian risk estimation and minimization

We propose endogenous Bayesian risk minimization (EBRM) over policy sets as an approach to online learning across a wide range of settings. Many real-world online learning problems have complexities such as action- and belief-dependent rewards, time-discounting of reward, and heterogeneous costs for actions and feedback; we find that existing online learning heuristics cannot leverage most problem-specific information, to the detriment of their performance. We introduce a belief-space Markov decision process (BMDP) model that can capture these complexities, and further apply the concepts of aleatoric, epistemic, and process risks to online learning. These risk functions describe the risk inherent to the learning problem, the risk due to the agent's lack of knowledge, and the relative quality of its policy, respectively. We demonstrate how computing and minimizing these risk functions guides the online learning agent towards the optimal exploration-exploitation trade-off in any stochastic online learning problem, constituting the basis of the EBRM approach. We also show how Bayes' risk, the minimization objective in stochastic online learning problems, can be decomposed into the aforementioned aleatoric, epistemic, and process risks.In simulation experiments, EBRM algorithms achieve state-of-the-art performance across various classical online learning problems, including Gaussian and Bernoulli multi-armed bandits, best-arm identification, mixed objectives with action- and belief-dependent rewards, and dynamic pricing, a finite partial monitoring problem. To our knowledge, it is also the first computationally efficient online learning approach that can provide online bounds on an algorithm's Bayes' risk. Finally, because the EBRM approach is parameterized by a set of policy algorithms, it can be extended to incorporate new developments in online learning algorithms, and is thus well-suited as the foundation for developing real-world learning agents.

An innovative heterogeneous transfer learning framework to enhance the scalability of deep reinforcement learning controllers in buildings with integrated energy systems

AbstractDeep Reinforcement Learning (DRL)-based control shows enhanced performance in the management of integrated energy systems when compared with Rule-Based Controllers (RBCs), but it still lacks scalability and generalisation due to the necessity of using tailored models for the training process. Transfer Learning (TL) is a potential solution to address this limitation. However, existing TL applications in building control have been mostly tested among buildings with similar features, not addressing the need to scale up advanced control in real-world scenarios with diverse energy systems. This paper assesses the performance of an online heterogeneous TL strategy, comparing it with RBC and offline and online DRL controllers in a simulation setup using EnergyPlus and Python. The study tests the transfer in both transductive and inductive settings of a DRL policy designed to manage a chiller coupled with a Thermal Energy Storage (TES). The control policy is pre-trained on a source building and transferred to various target buildings characterised by an integrated energy system including photovoltaic and battery energy storage systems, different building envelope features, occupancy schedule and boundary conditions (e.g., weather and price signal). The TL approach incorporates model slicing, imitation learning and fine-tuning to handle diverse state spaces and reward functions between source and target buildings. Results show that the proposed methodology leads to a reduction of 10% in electricity cost and between 10% and 40% in the mean value of the daily average temperature violation rate compared to RBC and online DRL controllers. Moreover, online TL maximises self-sufficiency and self-consumption by 9% and 11% with respect to RBC. Conversely, online TL achieves worse performance compared to offline DRL in either transductive or inductive settings. However, offline Deep Reinforcement Learning (DRL) agents should be trained at least for 15 episodes to reach the same level of performance as the online TL. Therefore, the proposed online TL methodology is effective, completely model-free and it can be directly implemented in real buildings with satisfying performance.

MFRLMO: Model-free reinforcement learning for multi-objective optimization of apache spark

Hyperparameter optimization (HO) is a must to figure out to what extent can a specific configuration of hyperparameters contribute to the performance of a machine learning task. The hardware and MLlib library of Apache Spark have the potential to improve big data processing performance when a tuning operation is combined with the exploitation of hyperparameters. To the best of our knowledge, the most of existing studies employ a black-box approach that results in misleading results due to ignoring the interior dynamics of big data processing. They suffer from one or more drawbacks including high computational cost, large search space, and sensitivity to the dimension of multi-objective functions. To address the issues above, this work proposes a new model-free reinforcement learning for multi-objective optimization of Apache Spark, thereby leveraging reinforcement learning (RL) agents to uncover the internal dynamics of Apache Spark in HO. To bridge the gap between multi-objective optimization and interior constraints of Apache Spark, our method runs a lot of iterations to update each cell of the RL grid. The proposed model-free learning mechanism achieves a tradeoff between three objective functions comprising time, memory, and accuracy. To this end, optimal values of the hyperparameters are obtained via an ensemble technique that analyzes the individual results yielded by each objective function. The results of the experiments show that the number of cores has not a direct effect on $speedup$. Further, although grid size has an impact on the time passed between two adjoining iterations, it is negligible in the computational burden. Dispersion and risk values of model-free RL differ when the size of the data is small. On average, MFRLMO produced $speedup$ that is 37% better than those of the competitors. Last, our approach is very competitive in terms of converging to a high accuracy when optimizing Convolutional Neural networks (CNN).

Optimization of binding affinities in chemical space with generative pre-trained transformer and deep reinforcement learning.

The key challenge in drug discovery is to discover novel compounds with desirable properties. Among the properties, binding affinity to a target is one of the prerequisites and usually evaluated by molecular docking or quantitative structure activity relationship (QSAR) models. In this study, we developed SGPT-RL, which uses a generative pre-trained transformer (GPT) as the policy network of the reinforcement learning (RL) agent to optimize the binding affinity to a target. SGPT-RL was evaluated on the Moses distribution learning benchmark and two goal-directed generation tasks, with Dopamine Receptor D2 (DRD2) and Angiotensin-Converting Enzyme 2 (ACE2) as the targets. Both QSAR model and molecular docking were implemented as the optimization goals in the tasks. The popular Reinvent method was used as the baseline for comparison. The results on the Moses benchmark showed that SGPT-RL learned good property distributions and generated molecules with high validity and novelty. On the two goal-directed generation tasks, both SGPT-RL and Reinvent were able to generate valid molecules with improved target scores. The SGPT-RL method achieved better results than Reinvent on the ACE2 task, where molecular docking was used as the optimization goal. Further analysis shows that SGPT-RL learned conserved scaffold patterns during exploration. The superior performance of SGPT-RL in the ACE2 task indicates that it can be applied to the virtual screening process where molecular docking is widely used as the criteria. Besides, the scaffold patterns learned by SGPT-RL during the exploration process can assist chemists to better design and discover novel lead candidates.

Agentic citizenship learning in China: official curriculum and student agency

Agentic citizenship learning in China: official curriculum and student agency

Human-in-the-Loop: The Future of Machine Learning in Automated Electron Microscopy

Abstract Machine learning (ML) methods are progressively gaining acceptance in the electron microscopy community for de-noising, semantic segmentation, and dimensionality reduction of data post-acquisition. The introduction of the application programming interfaces (APIs) by major instrument manufacturers now allows the deployment of ML workflows in microscopes, not only for data analytics but also for real-time decision-making and feedback for microscope operation. However, the number of use cases for real-time ML remains remarkably small. Here, we discuss some considerations in designing ML-based active experiments and pose that the likely strategy for the next several years will be human-in-the-loop automated experiments (hAE). In this paradigm, the ML learning agent directly controls beam position and image and spectroscopy acquisition functions, and a human operator monitors experiment progression in real and feature space of the system and tunes the policies of the ML agent to steer the experiment toward specific objectives.

Image Identification and Error Correction Method for Test Report Based on Deep Reinforcement Learning and IoT Platform in Smart Laboratory

In order to solve the problems that most models are complex, time-consuming, and have difficulty in identifying image errors, an image identification and error correction method of test report based on deep reinforcement learning and the internet of things platform in the smart lab was proposed. Firstly, a smart lab architecture was designed based on the internet of things platform, achieving efficient operation of the laboratory through cloud edge collaboration. Then, the depth separable convolution improved convolutional neural network is used to extract image features, and the features are input into bidirectional recurrent neural networks (BiLSTM) for analysis to complete image recognition. Finally, the ICNN-BiLSTM model is used as the agent of reinforcement learning, and image error correction is completed by identifying the distance between the image and the key points of the reference image. Based on the Python platform, the proposed method was experimentally demonstrated, and the results showed that its average error correction accuracy reached 96.75%, with a processing time of 15.37s.

Scheduling for the Flexible Job-Shop Problem with a Dynamic Number of Machines Using Deep Reinforcement Learning

The dynamic flexible job-shop problem (DFJSP) is a realistic and challenging problem that many production plants face. As the product line becomes more complex, the machines may suddenly break down or resume service, so we need a dynamic scheduling framework to cope with the changing number of machines over time. This issue has been rarely addressed in the literature. In this paper, we propose an improved learning-to-dispatch (L2D) model to generate a reasonable and good schedule to minimize the makespan. We formulate a DFJSP as a disjunctive graph and use graph neural networks (GINs) to embed the disjunctive graph into states for the agent to learn. The use of GINs enables the model to handle the dynamic number of machines and to effectively generalize to large-scale instances. The learning agent is a multi-layer feedforward network trained with a reinforcement learning algorithm, called proximal policy optimization. We trained the model on small-sized problems and tested it on various-sized problems. The experimental results show that our model outperforms the existing best priority dispatching rule algorithms, such as shortest processing time, most work remaining, flow due date per most work remaining, and most operations remaining. The results verify that the model has a good generalization capability and, thus, demonstrate its effectiveness.

QESRL: Exploring Selfish Reinforcement Learning for Repeated Quantum Games

A novel unified learning algorithm that seamlessly applies to both classical and quantum non-zero sum games is presented. Building upon the exploring selfish reinforcement learning (ESRL) framework previously proposed in the context of classical games, we extend this approach to handle quantum games with imperfect information. A comparison is made between performance and fairness among agents learning using plain QRL vs. QESRL. The latter enables agents to explore and learn periodic policy strategies in quantum games, leveraging the quantization of games to uncover fairer results. By addressing the challenges posed by the expanded strategy space in quantum games, we test the algorithm’s scalability by increasing the number of agents. Empirical evidence is provided to showcase its performance and to compare classical and quantum game scenarios. The proposed learning algorithm represents a significant step towards understanding the convergence and optimality of strategies in non-zero sum games across classical and quantum settings, bringing us closer to harnessing the potential of independent reinforcement learning and quantum computing in game theory applications.

A reinforcement learning agent for head and neck intensity-modulated radiation therapy

Head and neck (HN) cancers pose a difficult problem in the planning of intensity-modulated radiation therapy (IMRT) treatment. The primary tumor can be large and asymmetrical, and multiple organs at risk (OARs) with varying dose-sparing goals lie close to the target volume. Currently, there is no systematic way of automating the generation of IMRT plans, and the manual options face planning quality and long planning time challenges. In this article, we present a reinforcement learning (RL) model for the purposes of providing automated treatment planning to reduce clinical workflow time as well as providing a better starting point for human planners to modify and build upon. Several models with progressing complexity are presented, including the relevant plan dosimetry analysis and model interpretations of the resulting strategies learned by the auto-planning agent. Models were trained on a set of 40 patients and validated on a set of 20 patients. The presented models are shown to be consistent with the requirements of an RL model to be underpinned by a Markov decision process (MDP). In-depth interpretability of the models is presented by examination of the decision space using action hyperplanes. The auto-planning agent was able to generate plans with superior reduction in the mean dose of the left and right parotid glands by approximately 7 Gy ± 2.5 Gy (p < 0.01) over a starting, static template plan with only pre-defined general prescription information. RL plans were comparable to a human expert’s clinical plans for the primary (44 Gy), boost (26 Gy) , and the summed plans (70 Gy) with p-values of 0.43, 0.72, and 0.67, respectively, for the dosimetric endpoints and uniform target coverage normalization. The RL planning agent was able to produce the plans used in validation in an average of 13.58 min, with a minimum and a maximum planning time of 2.27 and 44.82 min, respectively.

Explaining the Behaviour of Reinforcement Learning Agents in a Multi-Agent Cooperative Environment Using Policy Graphs

The adoption of algorithms based on Artificial Intelligence (AI) has been rapidly increasing during the last few years. However, some aspects of AI techniques are under heavy scrutiny. For instance, in many use cases, it is not clear whether the decisions of an algorithm are well informed and conforming to human understanding. Having ways to address these concerns is crucial in many domains, especially whenever humans and intelligent (physical or virtual) agents must cooperate in a shared environment. In this paper, we apply an explainability method based on the creation of a Policy Graph (PG) based on discrete predicates that represent and explain a trained agent’s behaviour in a multi-agent cooperative environment. We show that from these policy graphs, policies for surrogate interpretable agents can be automatically generated. These policies can be used to measure the reliability of the explanations enabled by the PGs through a fair behavioural comparison between the original opaque agent and the surrogate one. The contributions of this paper represent the first use case of policy graphs in the context of explaining agent behaviour in cooperative multi-agent scenarios and present experimental results that sets this kind of scenario apart from previous implementations in single-agent scenarios: when requiring cooperative behaviour, predicates that allow representing observations about the other agents are crucial to replicate the opaque agent’s behaviour and increase the reliability of explanations.

RETRACTED ARTICLE: Optimizing IoT-enabled WSN routing strategies using whale optimization-driven multi-criterion correlation approach employs the reinforcement learning agent

RETRACTED ARTICLE: Optimizing IoT-enabled WSN routing strategies using whale optimization-driven multi-criterion correlation approach employs the reinforcement learning agent

Vocabulary learning in Finnish early foreign language learning textbooks

Abstract Vocabulary acquisition of young foreign language learners is an understudied topic. This study examines how early foreign language textbooks in Finland present and practise vocabulary. Moreover, studying textbooks is essential as they guide teachers’ actions and pupils’ perceptions of what is central in learning and how one should learn. We examined five different textbooks used in Finland for early foreign language learning. The languages covered were English, French, German, Swedish and Spanish. We analysed how the textbooks presented vocabulary using Thornbury’s (2002) classification as a basis. In particular, we looked at how the books took into account pupils’ existing linguistic resources, how they introduced and attempted to consolidate new vocabulary, what aspects of vocabulary knowledge were emphasised, what the role of formulaic sequences was, and if there were differences between the textbooks of different languages. The results showed that even though the textbooks had versatile exercises, there was a lack of progression and consolidating activities. Moreover, it seems that the textbooks do not largely aim at learner agency or translanguaging for vocabulary learning. Therefore, the examined textbooks for early foreign language learning in Finland are not fully in line with current understanding of vocabulary learning and language learning in general.

How pervasive is joint attention? Mother-child dyads from a Wichi community reveal a different form of "togetherness".

Theories of early development have emphasized the power of caregivers as active agents in infant socialization and learning. However, there is variability, across communities, in the tendency of caregivers to engage with their infants directly. This raises the possibility that infants and children in some communities spend more time engaged in solitary activities than in dyadic or triadic interactions. Here, we focus on one such community (indigenous Wichi living in Argentina's Chaco Forest) to test this possibility. We examine naturally occurring attentional activity involving the mother and child among the Wichi and among Eurodescendant Spanish-speaking families living in Argentina. We engaged 16 families-8 Wichi and 8 Eurodescendant-in an observational study of interactions between caregivers and their 1- to 2-year-olds. A mixed-analytic approach revealed no differences between communities in the proportion of time infants spent alone, or in mother-child interaction. What does differ, however, is how mothers engage in these interactions: Wichi mothers spend a greater proportion of their time observing their infants than do Eurodescendant mothers. Moreover, when infants in both groups are alone, they focus their 'solitary' activities differently: Wichi infants engaged primarily in observation alone, whereas Eurodescendant infants were more focused on the object. Finally, all mother-child pairs engaged in dyadic and triadic (object-infant-caregiver) patterns of attention, but the triadic patterns differed considerably between cultures: Among Wichi, mothers actively "watched" infants as they engaged with objects, whereas Eurodescendant mothers actively engaged with their infants in joint attentional episodes. This work illustrates how attention and socialization, key mechanisms of early development, are culturally organized. A video abstract of this article can be viewed at https://www.youtube.com/watch?v=SwsOCLXubKQ. RESEARCH HIGHLIGHTS: Longitudinal, observational investigation of mother-infant interaction in two distinct Argentine cultural groups (Wichi and Eurodescendants) reveals both commonalities and clear community-based differences in interactions between mothers and their 1- to 2-year-olds. Wichi mother-infant dyads engaged primarily in visual observation of one another, but their Eurodescendant counterparts tended to engage in more verbal or physical interaction. We identify a new form of triadic interaction-lateral joint attention-among the Wichi dyads. This work underscores that attention and socialization, key mechanisms of early development, are culturally organized.

An improved fuzzy inference strategy using reinforcement learning for trajectory-tracking of a mobile robot under a varying slip ratio

AbstractIn this study, a fuzzy reinforcement learning control (FRLC) is proposed to achieve trajectory tracking of a differential drive mobile robot (DDMR). The proposed FRLC approach designs fuzzy membership functions to fuzzify the relative position and heading between the current position and a prescribed trajectory. Instead of fuzzy inference rules, the relationship between the fuzzy inputs and actuator voltage outputs is built using a reinforcement learning (RL) agent. Herein, the deep deterministic policy gradient (DDPG) methodology consisted of actor and critic neural networks is employed in the RL agent. Simulations are conducted with considering varying slip ratio disturbances, different initial positions, and two different trajectories in the testing environment. In the meantime, a comparison with the classical DDPG model is presented. The results show that the proposed FRLC is capable of successfully tracking different trajectories under varying slip ratio disturbances as well as having performance superiority to the classical DDPG model. Moreover, experimental results validate that the proposed FRLC is also applicable to real mobile robots.

Digital game-based language learning for vocabulary development

Current literature suggests that one of the impressive benefits of digital game-based language learning is its potential to create engaging vocabulary learning experiences. However, this literature is dominated by game-play approaches, rather than game-making approaches. Drawing upon constructivist, motivation, dual-coding, and constructionist theories, this study examined elementary English language learners’ vocabulary learning experiences in a game-based learning context and investigated their experience as they built their own games based on their understanding of different non-fiction texts. This mixed-methods case study involved novel use of network analysis to examine student learning, investigate the relationships between various aspects of learner experiences, and evaluate the strengths and weaknesses of the program. Findings indicated that a complex set of interdependencies between enactments of theoretical principles including generativity, agency, tinkering, intrinsic motivation, and contextualized learning lead to enjoyment and powerful language learning experiences. Design principles for future digital game-based language learning include optimizing visual aspects, spending more time on technical skill development, increasing learner agency, and focusing more on generative and joyful learning experiences.

A systematic review of the use of WeChat in teaching Chinese as a foreign language

Abstract Over the years, social media have gained popularity as tools for language teaching. The social media platform WeChat has been employed for teaching Chinese as a foreign language (TCFL). While studies have investigated the use of WeChat for TCFL, no systematic reviews have been conducted to provide an overview of existing research. This review article examines empirical studies on WeChat in TCFL, summarising their main characteristics, findings, and critically assessing the quality and relevance of the available evidence. Scopus and Web of Science were used as primary databases for this review. Relevant studies were identified by using keyword sets such as “WeChat Chinese language”, “WeChat Mandarin”, “WeChat Hanyu” and “WeChat Putonghua”. The review identified the contexts, research methods and themes of the selected articles. The main themes include language learning, social interactions, learner agency and motivation. The paper also discusses the implications of using WeChat for TCFL and highlights areas for future research. It is hoped that the insights provided here will inform the design and implementation of future courses that use WeChat or other technology-assisted applications for language teaching.

From “Lab Rats” to “Mad Scientists”: Cultivating Agentic Learning Through Student-Led Simulation Redesign

ABSTRACTSimulations are increasingly prominent in classrooms, and research shows that such activities provide an array of benefits. In general, simulations tend to situate students as “lab rats” (Asal 2005) who get to discuss the experiment. We call for bringing students into a simulation not as lab rats but instead as “mad scientists.” We specifically propose using a simulation’s debriefing step as an opportunity for students to imagine how they might revise the simulation that they experienced and to defend their revisions using concepts from class. We argue that such student-led revisions cultivate agentic engagement, a form of engagement in which students recognize themselves as co-constructors of knowledge with and for their peers. To test this notion, we modified the “Isle of Ted” with an expanded post-simulation debriefing. We found that our approach cultivated student engagement and buy-in, required students to think of concepts dynamically, and was relatively simple to implement. Our study contributes to the literature on simulations as effective and engaging classroom tools, and it also emphasizes the potential for simulations to spark agentic learning.

Addressing Challenges in Dynamic Modeling of Stewart Platform using Reinforcement Learning-Based Control Approach

In this paper, we focus on enhancing the performance of the controller utilized in the Stewart platform by investigating the dynamics of the platform. Dynamic modeling is crucial for control and simulation, yet challenging for parallel robots like the Stewart platform due to closed-loop kinematics. We explore classical methods to solve its inverse dynamical model, but conventional approaches face difficulties, often resulting in simplified and inaccurate models. To overcome this limitation, we propose a novel approach by replacing the classical feedforward inverse dynamic block with a reinforcement learning (RL) agent, which, to our knowledge, has not been tried yet in the context of the Stewart platform control. Our proposed methodology utilizes a hybrid control topology that combines RL with existing classical control topologies and inverse kinematic modeling. We leverage three deep reinforcement learning (DRL) algorithms and two model-based RL algorithms to achieve improved control performance, highlighting the versatility of the proposed approach. By incorporating the learned feedforward control topology into the existing PID controller, we demonstrate enhancements in the overall control performance of the Stewart platform. Notably, our approach eliminates the need for explicit derivation and solving of the inverse dynamic model, overcoming the drawbacks associated with inaccurate and simplified models. Through several simulations and experiments, we validate the effectiveness of our reinforcement learning-based control approach for the dynamic modeling of the Stewart platform. The results highlight the potential of RL techniques in overcoming the challenges associated with dynamic modeling in parallel robot systems, promising improved control performance. This enhances accuracy and reduces the development time of control algorithms in real-world applications. Nonetheless, it requires a simulation step before practical implementations.

Coordination as inference in multi-agent reinforcement learning

The Centralized Training and Decentralized Execution (CTDE) paradigm, where a centralized critic is allowed to access global information during the training phase while maintaining the learned policies executed with only local information in a decentralized way, has achieved great progress in recent years. Despite the progress, CTDE may suffer from the issue of Centralized–Decentralized Mismatch (CDM): the suboptimality of one agent’s policy can exacerbate policy learning of other agents through the centralized joint critic. In contrast to centralized learning, the cooperative model that most closely resembles the way humans cooperate in nature is fully decentralized, i.e. Independent Learning (IL). However, there are still two issues that need to be addressed before agents coordinate through IL: (1) how agents are aware of the presence of other agents, and (2) how to coordinate with other agents to improve joint policy under IL. In this paper, we propose an inference-based coordinated MARL method: Deep Motor System (DMS). DMS first presents the idea of individual intention inference where agents are allowed to disentangle other agents from their environment. Secondly, causal inference was introduced to enhance coordination by reasoning each agent’s effect on others’ behavior. The proposed model was extensively experimented on a series of Multi-Agent MuJoCo and StarCraftII tasks. Results show that the proposed method outperforms independent learning algorithms and the coordination behavior among agents can be learned even without the CTDE paradigm compared to the state-of-the-art baselines including IPPO and HAPPO.