Learning Agent Research Articles

Unsupervised Training Sequence Design: Efficient and Generalizable Agent Training

To train generalizable Reinforcement Learning (RL) agents, researchers recently proposed the Unsupervised Environment Design (UED) framework, in which a teacher agent creates a very large number of training environments and a student agent trains on the experiences in these environments to be robust against unseen testing scenarios. For example, to train a student to master the “stepping over stumps” task, the teacher will create numerous training environments with varying stump heights and shapes. In this paper, we argue that UED neglects training efficiency and its need for very large number of environments (henceforth referred to as infinite horizon training) makes it less suitable to training robots and non-expert humans. In real-world applications where either creating new training scenarios is expensive or training efficiency is of critical importance, we want to maximize both the learning efficiency and learning outcome of the student. To achieve efficient finite horizon training, we propose a novel Markov Decision Process (MDP) formulation for the teacher agent, referred to as Unsupervised Training Sequence Design (UTSD). Specifically, we encode salient information from the student policy (e.g., behaviors and learning progress) into the teacher's state space, enabling the teacher to closely track the student's learning progress and consequently discover the optimal training sequences with finite lengths. Additionally, we explore the teacher's efficient adaptation to unseen students at test time by employing the context-based meta-learning approach, which leverages the teacher's past experiences with various students. Finally, we empirically demonstrate our teacher's capability to design efficient and effective training sequences for students with varying capabilities.

Explaining Reinforcement Learning Agents through Counterfactual Action Outcomes

Explainable reinforcement learning (XRL) methods aim to help elucidate agent policies and decision-making processes. The majority of XRL approaches focus on local explanations, seeking to shed light on the reasons an agent acts the way it does at a specific world state. While such explanations are both useful and necessary, they typically do not portray the outcomes of the agent's selected choice of action. In this work, we propose ``COViz'', a new local explanation method that visually compares the outcome of an agent's chosen action to a counterfactual one. In contrast to most local explanations that provide state-limited observations of the agent's motivation, our method depicts alternative trajectories the agent could have taken from the given state and their outcomes. We evaluated the usefulness of COViz in supporting people's understanding of agents' preferences and compare it with reward decomposition, a local explanation method that describes an agent's expected utility for different actions by decomposing it into meaningful reward types. Furthermore, we examine the complementary benefits of integrating both methods. Our results show that such integration significantly improved participants' performance.

Scaling Offline Evaluation of Reinforcement Learning Agents through Abstraction

A critical challenge for the widescale adoption of reinforcement learning (RL) is the need to give domain experts assurance that learned policies will improve decision-making -- and not lead to unacceptable behavior. To meet this challenge, my work aims to develop new methods for offline policy evaluation in real world RL domains. There has been much recent interest in offline evaluation and many advances. However, recent benchmarking efforts have also shown that there remains a substantial gap between current state-of-the-art methods and real world domains such as robotics. Towards scalable offline evaluation, my group is investigating the use of methods for abstraction and representation learning. In this new faculty highlight, I will present our recent results that show the promise of this direction for scaling offline evaluation in RL domains. I will then describe future directions in this line of that work which will further realize the promise of offline policy evaluation for increasing confidence in deployed RL.

Coordination of Emergent Demand Changes via Value-Based Negotiation for Supply Chain Management (Student Abstract)

We propose an automated negotiation for a reinforcement learning agent to adapt the agent to unexpected situations such as demand changes in supply chain management (SCM). Existing studies that consider reinforcement learning and SCM assume a centralized environment where the coordination of chain components is hierarchical rather than through negotiations between agents. This study focused on a negotiation agent that considered the value function of reinforcement learning for SCM as its utility function in automated negotiation. We demonstrated that the proposed approach could avoid inventory shortages under increased demand requests from the terminal customer.

Improving Knowledge Extraction from LLMs for Task Learning through Agent Analysis

Large language models (LLMs) offer significant promise as a knowledge source for task learning. Prompt engineering has been shown to be effective for eliciting knowledge from an LLM, but alone it is insufficient for acquiring relevant, situationally grounded knowledge for an embodied agent learning novel tasks. We describe a cognitive-agent approach, STARS, that extends and complements prompt engineering, mitigating its limitations and thus enabling an agent to acquire new task knowledge matched to its native language capabilities, embodiment, environment, and user preferences. The STARS approach is to increase the response space of LLMs and deploy general strategies, embedded within the autonomous agent, to evaluate, repair, and select among candidate responses produced by the LLM. We describe the approach and experiments that show how an agent, by retrieving and evaluating a breadth of responses from the LLM, can achieve 77-94% task completion in one-shot learning without user oversight. The approach achieves 100% task completion when human oversight (such as an indication of preference) is provided. Further, the type of oversight largely shifts from explicit, natural language instruction to simple confirmation/discomfirmation of high-quality responses that have been vetted by the agent before presentation to a user.

RLfOLD: Reinforcement Learning from Online Demonstrations in Urban Autonomous Driving

Reinforcement Learning from Demonstrations (RLfD) has emerged as an effective method by fusing expert demonstrations into Reinforcement Learning (RL) training, harnessing the strengths of both Imitation Learning (IL) and RL. However, existing algorithms rely on offline demonstrations, which can introduce a distribution gap between the demonstrations and the actual training environment, limiting their performance. In this paper, we propose a novel approach, Reinforcement Learning from Online Demonstrations (RLfOLD), that leverages online demonstrations to address this limitation, ensuring the agent learns from relevant and up-to-date scenarios, thus effectively bridging the distribution gap. Unlike conventional policy networks used in typical actor-critic algorithms, RLfOLD introduces a policy network that outputs two standard deviations: one for exploration and the other for IL training. This novel design allows the agent to adapt to varying levels of uncertainty inherent in both RL and IL. Furthermore, we introduce an exploration process guided by an online expert, incorporating an uncertainty-based technique. Our experiments on the CARLA NoCrash benchmark demonstrate the effectiveness and efficiency of RLfOLD. Notably, even with a significantly smaller encoder and a single camera setup, RLfOLD surpasses state-of-the-art methods in this evaluation. These results, achieved with limited resources, highlight RLfOLD as a highly promising solution for real-world applications.

Stability of Multi-Agent Learning in Competitive Networks: Delaying the Onset of Chaos

The behaviour of multi agent learning in competitive network games is often studied within the context of zero sum games, in which convergence guarantees may be obtained. However, outside of this class the behaviour of learning is known to display complex behaviours and convergence cannot be always guaranteed. Nonetheless, in order to develop a complete picture of the behaviour of multi agent learning in competitive settings, the zero sum assumption must be lifted. Motivated by this we study the Q Learning dynamics, a popular model of exploration and exploitation in multi agent learning, in competitive network games. We determine how the degree of competition, exploration rate and network connectivity impact the convergence of Q Learning. To study generic competitive games, we parameterise network games in terms of correlations between agent payoffs and study the average behaviour of the Q Learning dynamics across all games drawn from a choice of this parameter. This statistical approach establishes choices of parameters for which Q Learning dynamics converge to a stable fixed point. Differently to previous works, we find that the stability of Q Learning is explicitly dependent only on the network connectivity rather than the total number of agents. Our experiments validate these findings and show that, under certain network structures, the total number of agents can be increased without increasing the likelihood of unstable or chaotic behaviours.

Enhancing Retailer Satisfaction: Strategies and Best Practices

Within the energetic and competitive scene of the retail industry, the fulfillment of retailers plays a essential part in deciding the victory and supportability of supplier-retailer associations. This long theoretical investigates comprehensive methodologies and best hones pointed at upgrading retailer fulfillment, in this manner cultivating more grounded and more affluent connections between providers and retailers. Understanding the different needs and inclinations of retailers is foundational to any exertion pointed at upgrading fulfillment. Providers ought to utilize different strategies, counting standard overviews, input components, and advertise examination, to pick up bits of knowledge into retailer desires, requests, and showcase patterns. By understanding these components, providers can tailor their offerings and administrations to meet the particular prerequisites of retailers, hence upgrading fulfillment. Consistency in item quality is vital in keeping up retailer fulfillment. Providers must actualize strong quality control measures all through the generation and supply chain to guarantee that items reliably meet or surpass the desires of retailers and their clients. Clear details, rules for item dealing with and capacity, and responsive back for quality-related issues contribute to building believe and certainty among retailers. Proficiency in requesting and conveyance forms is another basic angle of upgrading retailer fulfillment. Providers ought to contribute in user-friendly online requesting stages, streamlined coordinations, and adaptable conveyance choices to disentangle the obtainment prepare for retailers. Opportune and exact conveyances are fundamental for minimizing disturbances to retailers’ operations and keeping up believe and unwavering quality. Remarkable client bolster is irreplaceable in tending to retailers’ request, concerns, and issues expeditiously and viably. Providers ought to keep up devoted back channels staffed with learned and responsive agents. Standard preparing sessions can enable bolster staff with item information and communication aptitudes, empowering them to supply personalized help and investigating direction to retailers. Building solid and collaborative connections is crucial to cultivating retailer fulfillment. Providers ought to develop a organization mentality based on believe, straightforwardness, and shared regard. Organizing occasions, workshops, and motivation programs can encourage interaction and collaboration between providers and retailers, cultivating a sense of community and shared objectives Nonstop advancement and advancement are key to remaining ahead within the competitive retail scene. Providers ought to request progressing input from retailers to distinguish zones for enhancement and contribute in investigate and improvement to present modern items, highlights, and administrations custom fitted to retailer needs and inclinations. Checking and measuring fulfillment measurements are basic for assessing execution and identifying areas for improvement. Key execution markers such as arrange precision, on-time conveyance, and client criticism scores can give profitable bits of knowledge into the adequacy of supplier-retailer intuitive. Leveraging information analytics devices empowers providers to pick up more profound experiences into retailer behavior and inclinations, encouraging educated decision-making and methodology refinement. In conclusion, upgrading retailer fulfillment requires a multifaceted approach that envelops understanding retailer needs, guaranteeing item quality and unwavering quality, streamlining forms, giving remarkable bolster, building solid connections, cultivating innovation, and checking execution. By prioritizing these methodologies and best hones, providers can develop more grounded associations with retailers, driving common victory and competitive advantage within the retail industry.

Stabilizing reinforcement learning control: A modular framework for optimizing over all stable behavior

We propose a framework for the design of feedback controllers that combines the optimization-driven and model-free advantages of deep reinforcement learning with the stability guarantees provided by using the Youla-Kučera parameterization to define the search domain. Recent advances in behavioral systems allow us to construct a data-driven internal model; this enables an alternative realization of the Youla-Kučera parameterization based entirely on input–output exploration data. Perhaps of independent interest, we formulate and analyze the stability of such data-driven models in the presence of noise. The Youla-Kučera approach requires a stable “parameter” for controller design. For the training of reinforcement learning agents, the set of all stable linear operators is given explicitly through a matrix factorization approach. Moreover, a nonlinear extension is given using a neural network to express a parameterized set of stable operators, which enables seamless integration with standard deep learning libraries. Finally, we show how these ideas can also be applied to tune fixed-structure controllers

A novel approach for self-driving car in partially observable environment using life long reinforcement learning

Despite ground-breaking advancements in robotics, gaming, and other challenging domains, reinforcement learning still faces significant challenges in solving dynamic, open-world problems. Since reinforcement learning algorithms usually perform poorly when exposed to new tasks outside of their data distribution, continuous learning algorithms have drawn significant attention. In parallel with work on lifelong learning algorithms, there is a need for challenging environments, properly planned trials, and metrics to measure research success. In this context, a Deep Asynchronous Autonomous Learning System (DAALS) is proposed in this paper for training a self-driving car in a partially observable environment for real-world scenarios in a continuous state-action space. To cater to three different use cases, three different algorithms were used. To train their agents for learning and upgrading discrete state policies, DAALS used the Asynchronous Advantage Stager Reviewer (AASR) algorithm. To train its agent for continuous state spaces, DAALS also uses an Extensive Deterministic Policy Gradient (EDPG) algorithm. To train the agent in a lifelong form of learning for partially observable environments, DAALS uses a Deep Deterministic Policy Gradient Novel Lifelong Learning Algorithm (DDPGNLLA). The system offers flexibility to the user to train the agents for both discrete and continuous state-action spaces. Compared to previous models in continuous state-action spaces, Deep deterministic policy gradient lifelong learning algorithm outperforms previous models by 46.09%. Furthermore, the Deep Asynchronous Autonomous System tends to outperform all previous reinforcement learning algorithms, making our proposed approach a real-world solution. As DAALS has tested on number of different environments it provides the insights on how modern Artificial Intelligence (AI) solutions can be generalized making it one of the better solutions for AI general domain problems.

Reinforcement Learning-Based Multi-Objective Optimization for Generation Scheduling in Power Systems

Multi-objective power scheduling (MOPS) aims to address the simultaneous minimization of economic costs and different types of environmental emissions during electricity generation. Recognizing it as an NP-hard problem, this article proposes a novel multi-agent deep reinforcement learning (MADRL)-based optimization algorithm. Within a custom multi-agent simulation environment, representing power-generating units as collaborative types of reinforcement learning (RL) agents, the MOPS problem is decomposed into sequential Markov decision processes (MDPs). The MDPs are then utilized for training an MADRL model, which subsequently offers the optimal solution to the optimization problem. The practical viability of the proposed method is evaluated across several experimental test systems consisting of up to 100 units featuring bi-objective and tri-objective problems. The results demonstrate that the proposed MADRL algorithm has better performance compared to established methods, such as teaching learning-based optimization (TLBO), real coded grey wolf optimization (RCGWO), evolutionary algorithm based on decomposition (EAD), non-dominated sorting algorithm II (NSGA-II), and non-dominated sorting algorithm III (NSGA-III).

Creating valid adversarial examples of malware

Because of its world-class results, machine learning (ML) is becoming increasingly popular as a go-to solution for many tasks. As a result, antivirus developers are incorporating ML models into their toolchains. While these models improve malware detection capabilities, they also carry the disadvantage of being susceptible to adversarial attacks. Although this vulnerability has been demonstrated for many models in white-box settings, a black-box scenario is more applicable in practice for the domain of malware detection. We present a method of creating adversarial malware examples using reinforcement learning algorithms. The reinforcement learning agents utilize a set of functionality-preserving modifications, thus creating valid adversarial examples. Using the proximal policy optimization (PPO) algorithm, we achieved an evasion rate of 53.84% against the gradient-boosted decision tree (GBDT) detector. The PPO agent previously trained against the GBDT classifier scored an evasion rate of 11.41% against the neural network-based classifier MalConv and an average evasion rate of 2.31% against top antivirus programs. Furthermore, we discovered that random application of our functionality-preserving portable executable modifications successfully evades leading antivirus engines, with an average evasion rate of 11.65%. These findings indicate that ML-based models used in malware detection systems are sensitive to adversarial attacks and that better safeguards need to be taken to protect these systems.

Reinforcement-Learning-Based Virtual Inertia Controller for Frequency Support in Islanded Microgrids

As the world grapples with the energy crisis, integrating renewable energy sources into the power grid has become increasingly crucial. Microgrids have emerged as a vital solution to this challenge. However, the reliance on renewable energy sources in microgrids often leads to low inertia. Renewable energy sources interfaced with the network through interlinking converters lack the inertia of conventional synchronous generators, and hence, need to provide frequency support through virtual inertia techniques. This paper presents a new control algorithm that utilizes the reinforcement learning agents Twin Delayed Deep Deterministic Policy Gradient (TD3) and Deep Deterministic Policy Gradient (DDPG) to support the frequency in low-inertia microgrids. The RL agents are trained using the system-linearized model and then extended to the nonlinear model to reduce the computational burden. The proposed system consists of an AC–DC microgrid comprising a renewable energy source on the DC microgrid, along with constant and resistive loads. On the AC microgrid side, a synchronous generator is utilized to represent the low inertia of the grid, which is accompanied by dynamic and static loads. The model of the system is developed and verified using Matlab/Simulink and the reinforcement learning toolbox. The system performance with the proposed AI-based methods is compared to conventional low-pass and high-pass filter (LPF and HPF) controllers.

Testing of Deep Reinforcement Learning Agents with Surrogate Models

Deep Reinforcement Learning (DRL) has received a lot of attention from the research community in recent years. As the technology moves away from game playing to practical contexts, such as autonomous vehicles and robotics, it is crucial to evaluate the quality of DRL agents. In this article, we propose a search-based approach to test such agents. Our approach, implemented in a tool called Indago , trains a classifier on failure and non-failure environment (i.e., pass) configurations resulting from the DRL training process. The classifier is used at testing time as a surrogate model for the DRL agent execution in the environment, predicting the extent to which a given environment configuration induces a failure of the DRL agent under test. The failure prediction acts as a fitness function, guiding the generation towards failure environment configurations, while saving computation time by deferring the execution of the DRL agent in the environment to those configurations that are more likely to expose failures. Experimental results show that our search-based approach finds 50% more failures of the DRL agent than state-of-the-art techniques. Moreover, such failures are, on average, 78% more diverse; similarly, the behaviors of the DRL agent induced by failure configurations are 74% more diverse.

On the optimality of quantum circuit initial mapping using reinforcement learning

Quantum circuit optimization is an inevitable task with the current noisy quantum backends. This task is considered non-trivial due to the varying circuits’ complexities in addition to hardware-specific noise, topology, and limited connectivity. The currently available methods either rely on heuristics for circuit optimization tasks or reinforcement learning with complex unscalable neural networks such as transformers. In this paper, we are concerned with optimizing the initial logical-to-physical mapping selection. Specifically, we investigate whether a reinforcement learning agent with simple scalable neural network is capable of finding a near-optimal logical-to-physical mapping, that would decrease as much as possible additional CNOT gates, only from a fixed-length feature vector. To answer this question, we train a Maskable Proximal Policy Optimization agent to progressively take steps towards a near-optimal logical-to-physical mapping on a 20-qubit hardware architecture. Our results show that our agent coupled with a simple routing evaluation is capable of outperforming other available reinforcement learning and heuristics approaches on 12 out of 19 test benchmarks, achieving geometric mean improvements of 2.2% and 15% over the best available related work and two heuristics approaches, respectively. Additionally, our neural network model scales linearly as the number of qubits increases.

Deep Deterministic Policy Gradient (DDPG) Agent-Based Sliding Mode Control for Quadrotor Attitudes

A novel reinforcement deep learning deterministic policy gradient agent-based sliding mode control (DDPG-SMC) approach is proposed to suppress the chattering phenomenon in attitude control for quadrotors, in the presence of external disturbances. First, the attitude dynamics model of the quadrotor under study is derived, and the attitude control problem is described using formulas. Second, a sliding mode controller, including its sliding mode surface and reaching law, is chosen for the nonlinear dynamic system. The stability of the designed SMC system is validated through the Lyapunov stability theorem. Third, a reinforcement learning (RL) agent based on deep deterministic policy gradient (DDPG) is trained to adaptively adjust the switching control gain. During the training process, the input signals for the agent are the actual and desired attitude angles, while the output action is the time-varying control gain. Finally, the trained agent mentioned above is utilized in the SMC as a parameter regulator to facilitate the adaptive adjustment of the switching control gain associated with the reaching law. The simulation results validate the robustness and effectiveness of the proposed DDPG-SMC method.

Satellite fault tolerant attitude control based on expert guided exploration of reinforcement learning agent

ABSTRACT This research provides a method that accelerates learning and avoids local minima to improve the policy gradient algorithm’s learning process. Reinforcement learning has the advantage of not requiring a model. Consequently, it can improve control performance, mainly when a model is generally unavailable, such as when an error occurs. The proposed method efficiently and expeditiously investigates the action space. First, it quantifies the resemblance between agents’ and traditional controllers’ actions. Then, the principal reward function is modified to reflect this similarity. This reward-shaping mechanism guides the agent to maximize its return via an attractive force during the gradient ascent. To validate our concept, we establish a satellite attitude control environment with a similarity subsystem. The outcomes demonstrate the effectiveness and robustness of our method.

ChatGPT: Enhancing Learner Agency and Multilingual Practices in English Writing Class

ChatGPT is a newly developed artificial intelligence (AI) model that has the potential to improve student motivation and engagement in English writing classrooms. By using ChatGPT, teachers can create more engaging and effective writing instruction that fosters learner agency. This study explored how AI tools, such as ChatGPT, can be used to enhance learner agency and multilingual practices in English writing classrooms. The study successfully integrated learner agency theory and power dynamics theory into the specific and innovative context of ESL classrooms. The study investigated students' attitudes and experiences with the use of AI in writing classrooms. The results showed that most students had positive attitudes towards AI writing tools and believed that they could improve writing efficiency and quality.

Hybrid Elastic Scaling Strategy for Container Cloud based on Load Prediction and Reinforcement Learning

To harness the advantages of both proactive and responsive scaling, adapting to various workload scenarios, this paper introduces a container hybrid scaling strategy called HyPredRL, rooted in load prediction and reinforcement learning. Within the proactive scaling module RL-PM, a load prediction model, MSC-LSTM, predict workloads and, in conjunction with current workload states, leverages reinforcement learning agents for intelligent scaling decisions. The responsive scaling strategy, SLA-HPA, enhances Kubernetes’ native scaling strategy, which primarily considers resource utilization, by incorporating response time metrics. Ultimately, a hybrid scaling controller is designed, applying the principles of “rapid scaling out” and “balanced conflicts” to coordinate proactive and responsive scaling. Experimental results demonstrate that HyPredRL outperforms existing methods in SLA violation rate, resource utilization, and request response time, effectively improving application performance and scalability.

Learner engagement with instructor‐generated video

AbstractEffective video resources are assumed to promote learner engagement, but the extent to which this occurs is unclear. This study examines learners' engagement with instructor‐generated video. It contributes an analytical synthesis of qualitative and quantitative data that provides the basis for investigating the extent to which, and how, learners engage with video resources provided in their courses. Specifically, three dimensions of learner engagement with video are studied: behavioural, cognitive and affective. The study contributes to educational technology research by identifying distinctive patterns of learner control over the use of video which diverges from instructors' assumptions. It shows the complex and nuanced features of cognitive and affective engagement. Videos can have positive effects, but inappropriate use of technical features results in learner disengagement. This study contributes a novel use of signalling theory, suggesting the importance of instructor signals concerning relevance, focus and utility as a feature of video generation, as a prerequisite of cognitive engagement. A research and theory development agenda is developed, locating video‐based learning in student contexts as a basis for explaining both engagement and disengagement with video technology affordances. Practitioner notesWhat is already known about this topic Higher education tutors are increasingly expected to generate video content as a feature of their course delivery. Although effective video resources are assumed to promote learner engagement, the extent to which this occurs is unclear. Video technology affordances rarely feature in academic professional development programmes leaving tutors ill‐equipped to understand how and in what ways students engage with video resources as a part of their courses. What this paper adds A novel use of signalling theory to explain different patterns of learner engagement and disengagement. Learner agency and control results in video technology affordances being fitted and ‘flexed’ around factors that instructors rarely consider in the video generation process. A new model to describe factors that affect student engagement with instructor‐generated video. Implications for practice and/or policy Raise awareness for instructors about patterns of student engagement with video technology affordances that affect learning experience and outcomes. Implement professional development programmes to equip instructors to design and generate video content that takes account of patterns of behavioural, cognitive and affective engagement and signals relevance, focus and utility. Develop clear policies regarding technical features of video generation to avoid learner disengagement.