Meta-reinforcement Learning Research Articles

Impact of Autoscaling on Application Performance in Cloud Environments

The current paper studies the impact of Autoscaling on application performance in Cloud computing environments. Cloud computing is one of the most encouraging innovations due to its vast applications. Predictive autoscaling is an advanced technique that aims to address the challenges in the autoscaling trends for large-scale systems. To account for the Quality of Service to the customer, features like load balancing according to the workload demand, understanding resource allocation and utilization, and dynamic decision-making are vital to any cloud computing application. This paper interprets these challenges and reviews a meta-reinforcement learning approach for predictive autoscaling in cloud environments. A novel RL-based predictive autoscaling approach on a popular large-scale digital payment platform system, Alipay, is compared with the existing models such as Autopilot and FIRM. The aim is to conduct a detailed analysis of performance metrics before and after autoscaling actions, aiming to identify optimal scaling strategies that minimize response time and maximize resource utilization without over- provisioning.

Context-Based Meta-Reinforcement Learning With Bayesian Nonparametric Models.

Deep reinforcement learning agents usually need to collect a large number of interactions to solve a single task. In contrast, meta-reinforcement learning (meta-RL) aims to quickly adapt to new tasks using a small amount of experience by leveraging the knowledge from training on a set of similar tasks. State-of-the-art context-based meta-RL algorithms use the context to encode the task information and train a policy conditioned on the inferred latent task encoding. However, most recent works are limited to parametric tasks, where a handful of variables control the full variation in the task distribution, and also failed to work in non-stationary environments due to the few-shot adaptation setting. To address those limitations, we propose MEta-reinforcement Learning with Task Self-discovery (MELTS), which adaptively learns qualitatively different nonparametric tasks and adapts to new tasks in a zero-shot manner. We introduce a novel deep clustering framework (DPMM-VAE) based on an infinite mixture of Gaussians, which combines the Dirichlet process mixture model (DPMM) and the variational autoencoder (VAE), to simultaneously learn task representations and cluster the tasks in a self-adaptive way. Integrating DPMM-VAE into MELTS enables it to adaptively discover the multi-modal structure of the nonparametric task distribution, which previous methods using isotropic Gaussian random variables cannot model. In addition, we propose a zero-shot adaptation mechanism and a recurrence-based context encoding strategy to improve the data efficiency and make our algorithm applicable in non-stationary environments. On various continuous control tasks with both parametric and nonparametric variations, our algorithm produces a more structured and self-adaptive task latent space and also achieves superior sample efficiency and asymptotic performance compared with state-of-the-art meta-RL algorithms.

Meta-Reinforcement Learning in Nonstationary and Nonparametric Environments.

Recent state-of-the-art artificial agents lack the ability to adapt rapidly to new tasks, as they are trained exclusively for specific objectives and require massive amounts of interaction to learn new skills. Meta-reinforcement learning (meta-RL) addresses this challenge by leveraging knowledge learned from training tasks to perform well in previously unseen tasks. However, current meta-RL approaches limit themselves to narrow parametric and stationary task distributions, ignoring qualitative differences and nonstationary changes between tasks that occur in the real world. In this article, we introduce a Task-Inference-based meta-RL algorithm using explicitly parameterized Gaussian variational autoencoders (VAEs) and gated Recurrent units (TIGR), designed for nonparametric and nonstationary environments. We employ a generative model involving a VAE to capture the multimodality of the tasks. We decouple the policy training from the task-inference learning and efficiently train the inference mechanism on the basis of an unsupervised reconstruction objective. We establish a zero-shot adaptation procedure to enable the agent to adapt to nonstationary task changes. We provide a benchmark with qualitatively distinct tasks based on the half-cheetah environment and demonstrate the superior performance of TIGR compared with state-of-the-art meta-RL approaches in terms of sample efficiency (three to ten times faster), asymptotic performance, and applicability in nonparametric and nonstationary environments with zero-shot adaptation. Videos can be viewed at https://videoviewsite.wixsite.com/tigr.

Multi-objective two-stage robust optimization of wind/PV/thermal power system based on meta multi-agent reinforcement learning

The integration of renewable energy into the power grid poses significant challenges for optimization and scheduling of the power system. In recent years, methods based on deep reinforcement learning have surpassed traditional methods on the high complexity and long-term decision-making of power system optimization and scheduling. However, faced with the inherent uncertainty of renewable energy generation and the different optimization objectives in power system, the deep reinforcement learning methods are unable to effectively address them. This paper proposes a method that combines meta reinforcement learning with multi-agent reinforcement learning to solve the multi-objective two-stage robust optimization of wind/PV/thermal power system. We conducts optimization and scheduling experiments on the IEEE39 bus system. The results indicate that our method not only enhances the robustness of the scheduling strategy, but also outperforms baseline methods in terms of convergence, diversity, and uniformity of the Pareto frontier.

Unsupervised Meta-Testing With Conditional Neural Processes for Hybrid Meta-Reinforcement Learning

Unsupervised Meta-Testing With Conditional Neural Processes for Hybrid Meta-Reinforcement Learning

Learning and Fast Adaptation for Air Combat Decision with Improved Deep Meta-reinforcement Learning

Learning and Fast Adaptation for Air Combat Decision with Improved Deep Meta-reinforcement Learning

Meta-Reinforcement Learning in Time-Varying UAV Communications: Adaptive Anti-Jamming Channel Selection

Meta-Reinforcement Learning in Time-Varying UAV Communications: Adaptive Anti-Jamming Channel Selection

Meta-Reinforcement Learning for Spacecraft Proximity Operations Guidance and Control in Cislunar Space

Innovative lightweight and model-free guidance algorithms are essential to achieve full autonomy of spacecraft and address future space exploration challenges. In the near future, this type of technology will be essential for cislunar space proximity operations, such as NASA’s Artemis program and its Lunar Gateway project. In this scenario, a meta-reinforcement learning algorithm is employed to address the real-time optimal guidance problem of a spacecraft in the cislunar environment. Non-Keplerian orbits pose complex dynamics, where classic control theory may be less adaptable and more computationally expensive compared to machine learning approaches. Meta-RL stands out for its ability to learn how to learn through experience, training on various tasks to enhance efficiency, and effectiveness in tackling new ones. By modeling a stochastic optimal control problem within the circular restricted three-body problem framework as a Markov decision process, an LSTM-based network agent is trained using proximal policy optimization, considering operational constraints and stochastic effects for safety and robustness evaluation. Additionally, an MLP-based agent and an optimal control pseudo-spectral solution are assessed for comparison. The resulting tool autonomously guides spacecraft in cislunar proximity operations, approximating the optimal control solution with a versatile algorithmic framework. This ensures both robustness and computational efficiency.

Contrastive meta-reinforcement learning for heterogeneous graph neural architecture search

Heterogeneous Graph Neural Networks (HGNNs) have demonstrated significant success in capturing complex interactions within heterogeneous graphs to learn graph representations. However, designing effective HGNN architecture is a challenging task that demands substantial domain knowledge and human effort. Fortunately, the advent of Heterogeneous Graph Architecture Search (HGNAS) has automated this process and yielded networks surpassing those designed by humans. Nonetheless, conventional HGNAS approaches are limited in their ability to generalize to unseen or diverse task types. To address these limitations, we propose an efficient method called Contrastive Meta-reinforcement learning-based Heterogeneous Graph Neural Architecture Search (CM-HGNAS). Our approach aims to produce high-performance networks tailored to different types of tasks. Firstly, we use gradient-based meta-learning to rapidly adapt to new tasks by leveraging the knowledge acquired from meta-training tasks. Secondly, since our meta-training tasks encompass various types of tasks, including node classification and link prediction, we employ contrastive learning to unify the evaluation metric across all tasks. This unification enhances the generalization capabilities of our model across diverse tasks. Experimental results demonstrate that our method successfully generalizes to unseen tasks and surpasses existing HGNN baselines and HGNAS techniques in terms of performance.

Meta-reinforcement learning for active visual tracking about space non-cooperative object

Meta-reinforcement learning for active visual tracking about space non-cooperative object

Human-Inspired Meta-Reinforcement Learning Using Bayesian Knowledge and Enhanced Deep Q-Network

Over the last decades, there has been growing interest in research in multiple and interdisciplinary fields of human-AI computing. In particular, approaches integrating human’s perspective and design with reinforcement learning (RL) have received more attention. However, the current research on RL may need to consider its enhancement from human-inspired approaches further. In this work, we focus on enabling a meta-reinforcement learning (meta-RL) agent to achieve adaptation and generalization, according to modeling Markov Decision Processes (MDP) using Bayesian knowledge and analysis. By introducing a novel framework called human-inspired meta-RL (HMRL), we incorporate the agent performing resilient actions to leverage the dynamic dense reward based on the knowledge and prediction of a Bayesian analysis. The proposed framework can make the agent learn generalization and prevent the agent from failing catastrophically. The experimental results show that our approach helps the agent reduce computational costs with learning adaptation. In addition to the system design, we have also extended further algorithmic improvement based on learning within a deep Q-network (DQN) implementations for more complicated future tasks, which compared replay buffers to possibly enhance the optimization process. Finally, we conclude and anticipate that integrating human-inspired meta-RL can enable learning more formulations relating to robustness and scalability, leading to promising directions and more complex AI goals in the future.

Meta Reinforcement Learning for Resource Allocation in Aerial Active-RIS-Assisted Networks With Rate-Splitting Multiple Access

Meta Reinforcement Learning for Resource Allocation in Aerial Active-RIS-Assisted Networks With Rate-Splitting Multiple Access

Offline Meta-Reinforcement Learning for Active Pantograph Control in High-Speed Railways

Offline Meta-Reinforcement Learning for Active Pantograph Control in High-Speed Railways

Brain-Inspired Deep Meta-Reinforcement Learning for Active Coordinated Fault-Tolerant Load Frequency Control of Multi-Area Grids

This paper proposes an active coordinated fault tolerance load frequency control (AFCT-LFC) method, which effectively prevents sudden frequency changes caused by unit actuator failures or unplanned decommissioning in a multi-area interconnected grid subject to the performance-based frequency regulation market mechanism. It can also reduce regulation mileage payments and achieve multi-objective active fault-tolerant control. In addition, this paper proposes a brain-Inspired deep meta-deterministic policy gradient algorithm (BIMA-DMDPG), which adopts multi-agent centralized training, equates the controller of each area as an agent capable of independent decision making, and implements distributed training by dividing the environment into multiple environments. In addition, meta-reinforcement learning is employed to realize multi-task collaborative learning. The optimal policy is actively selected under different fault conditions to achieve active fault-tolerant control. The superior performance of the method is verified in a four-area LFC model of the China Southern Grid (CSG), in which it is tested alongside a selection of existing algorithms. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —AFCT-LFC is based on advanced artificial intelligence algorithm, which can effectively identify any fault in multi-area grids and make rapid response to achieve active fault-tolerant control. Compared with the existing model-based fault-tolerant control methods, the BIMA-DMDPG algorithm proposed in this paper does not need to rely on accurate mathematical models, and can be applied to practice through simple training, which is very suitable for practical applications. Therefore, AFCT-LFC is an advanced adaptive active fault-tolerant control method that can be truly applied in practice because of its fast-decision-making ability and performance.

Adaptive meta-reinforcement learning for AUVs 3D guidance and control under unknown ocean currents

Given the challenges posed by underwater communication constraints, Autonomous Underwater Vehicles (AUVs) must operate with enhanced autonomy while balancing operational safety and efficiency within complex marine settings. AUVs face a 3D guidance and control challenge, requiring complex adjustments to unknown ocean currents that affect their stability, maneuverability, and Guidance-Navigation-Control system efficiency. This study introduces and implements an off-policy meta-reinforcement learning framework to address AUVs’ 3D guidance and control challenges under unknown ocean currents. Our method, APE-SAC, utilizes an inference network to identify latent variables, streamlining the adaptation to unknown ocean currents by optimizing their strength and direction. Consequently, this framework redefines the issue of guidance and control under unknown current disturbances into a manageable learning task across multiple predefined scenarios. Utilizing the 3D dynamics of the REMUS AUV, we construct meta-Markov Decision Processes to detail the comprehensive state space, action space, and robust reward function tailored to the task’s specific demands. Furthermore, we develop a highly realistic simulation environment specifically designed for comprehensive experiments. Extensive results demonstrate that, compared to deep reinforcement learning baselines, APE-SAC exhibits outstanding performance in guiding and controlling AUVs under unknown ocean currents.

Self-Adaptive Server Anomaly Detection Using Ensemble Meta-Reinforcement Learning

As the user’s behavior changes at any time with cloud computing and network services, abnormal server resource utilization traffic will lead to severe service crashes and system downtime. The traditional single anomaly detection model cannot handle the rapid failure prediction ahead. Therefore, this study proposed ensemble learning combined with model-agnostic meta-reinforcement learning called ensemble meta-reinforcement learning (EMRL) to implement self-adaptive server anomaly detection rapidly and precisely, according to the time series of server resource utilization. The proposed ensemble approach combines hidden Markov model (HMM), variational autoencoder (VAE), temporal convolutional autoencoder (TCN-AE), and bidirectional long short-term memory (BLSTM). The EMRL algorithm trains this combination with several tasks to learn the implicit representation of various anomalous traffic, where each task executes trust region policy optimization (TRPO) to quickly adapt the time-varying data distribution and make rapid decisions precisely for an agent response. As a result, our proposed approach can improve the precision of anomaly prediction by 2.4 times and reduce the model deployment speed by 5.8 times on average because a meta-learner can immediately be applied to new tasks.

UAS Visual Navigation in Large and Unseen Environments via a Meta Agent

Abstract. The aim of this work is to develop an approach that enables Unmanned Aerial System (UAS) to efficiently learn to navigate in large-scale urban environments and transfer their acquired expertise to novel environments. To achieve this, we propose a metacurriculum training scheme. First, meta-training allows the agent to learn a master policy to generalize across tasks. The resulting model is then fine-tuned on the downstream tasks. We organize the training curriculum in a hierarchical manner such that the agent is guided from coarse to fine towards the target task. In addition, we introduce Incremental Self-Adaptive Reinforcement learning (ISAR), an algorithm that combines the ideas of incremental learning and meta-reinforcement learning (MRL). In contrast to traditional reinforcement learning (RL), which focuses on acquiring a policy for a specific task, MRL aims to learn a policy with fast transfer ability to novel tasks. However, the MRL training process is time consuming, whereas our proposed ISAR algorithm achieves faster convergence than the conventional MRL algorithm. We evaluate the proposed methodologies in simulated environments and demonstrate that using this training philosophy in conjunction with the ISAR algorithm significantly improves the convergence speed for navigation in large-scale cities and the adaptation proficiency in novel environments. The project page is publicly available at https://superhan2611.github.io/isar_nav/.

Energy-efficient tool path generation and expansion optimisation for five-axis flank milling with meta-reinforcement learning

Energy-efficient tool path generation and expansion optimisation for five-axis flank milling with meta-reinforcement learning

A recurrent network model of planning explains hippocampal replay and human behavior

When faced with a novel situation, people often spend substantial periods of time contemplating possible futures. For such planning to be rational, the benefits to behavior must compensate for the time spent thinking. Here, we capture these features of behavior by developing a neural network model where planning itself is controlled by the prefrontal cortex. This model consists of a meta-reinforcement learning agent augmented with the ability to plan by sampling imagined action sequences from its own policy, which we call ‘rollouts’. In a spatial navigation task, the agent learns to plan when it is beneficial, which provides a normative explanation for empirical variability in human thinking times. Additionally, the patterns of policy rollouts used by the artificial agent closely resemble patterns of rodent hippocampal replays. Our work provides a theory of how the brain could implement planning through prefrontal–hippocampal interactions, where hippocampal replays are triggered by—and adaptively affect—prefrontal dynamics.

Federated Meta Reinforcement Learning for Personalized Tasks

Federated Meta Reinforcement Learning for Personalized Tasks