High-dimensional Action Spaces Research Articles

Physics-Informed Graph Neural Operator for Mean Field Games on Graph: A Scalable Learning Approach

Mean-field games (MFGs) are developed to model the decision-making processes of a large number of interacting agents in multi-agent systems. This paper studies mean-field games on graphs (G-MFGs). The equilibria of G-MFGs, namely, mean-field equilibria (MFE), are challenging to solve for their high-dimensional action space because each agent has to make decisions when they are at junction nodes or on edges. Furthermore, when the initial population state varies on graphs, we have to recompute MFE, which could be computationally challenging and memory-demanding. To improve the scalability and avoid repeatedly solving G-MFGs every time their initial state changes, this paper proposes physics-informed graph neural operators (PIGNO). The PIGNO utilizes a graph neural operator to generate population dynamics, given initial population distributions. To better train the neural operator, it leverages physics knowledge to propagate population state transitions on graphs. A learning algorithm is developed, and its performance is evaluated on autonomous driving games on road networks. Our results demonstrate that the PIGNO is scalable and generalizable when tested under unseen initial conditions.

Sample efficient reinforcement learning with active learning for molecular design

Active learning accelerates the design of molecules during generative reinforcement learning by creating surrogate models of expensive reward functions, obtaining a 4- to 64-fold reduction in computational effort per hit.

Joint Task Offloading and Resource Allocation for Intelligent Reflecting Surface-Aided Integrated Sensing and Communication Systems Using Deep Reinforcement Learning Algorithm.

This paper investigates an intelligent reflecting surface (IRS)-aided integrated sensing and communication (ISAC) framework to cope with the problem of spectrum scarcity and poor wireless environment. The main goal of the proposed framework in this work is to optimize the overall performance of the system, including sensing, communication, and computational offloading. We aim to achieve the trade-off between system performance and overhead by optimizing spectrum and computing resource allocation. On the one hand, the joint design of transmit beamforming and phase shift matrices can enhance the radar sensing quality and increase the communication data rate. On the other hand, task offloading and computation resource allocation optimize energy consumption and delay. Due to the coupled and high dimension optimization variables, the optimization problem is non-convex and NP-hard. Meanwhile, given the dynamic wireless channel condition, we formulate the optimization design as a Markov decision process. To tackle this complex optimization problem, we proposed two innovative deep reinforcement learning (DRL)-based schemes. Specifically, a deep deterministic policy gradient (DDPG) method is proposed to address the continuous high-dimensional action space, and the prioritized experience replay is adopted to speed up the convergence process. Then, a twin delayed DDPG algorithm is designed based on this DRL framework. Numerical results confirm the effectiveness of proposed schemes compared with the benchmark methods.

Soft imitation reinforcement learning with value decomposition for portfolio management

Imitation learning has been recognized as a method to accelerate the training process of deep reinforcement learning agents in search of optimal strategies. Nevertheless, existing imitation learning algorithms have limitations in effectively leveraging expert demonstrations when confronted with dynamic environments, as the behavior cloning loss weight cannot be adaptively updated. To overcome this challenge, we propose a novel approach called Soft Imitation Reinforcement Learning (SIRL), which combines imitation learning and reinforcement learning to guide the training of reinforcement learning agents in an adaptive manner. Additionally, we addressed the challenge of high-dimensional action spaces for reinforcement learning in portfolio management with value decomposition, and provide theoretical proof of convergence for this method. To validate the effectiveness of the SIRL algorithm, we conduct extensive experiments using stock market data from emerging (China) and developed (the US) countries. Our experiments indicate the versatility of the proposed SIRL across different types of trading data, encompassing both high-frequency (5-minute interval) and low-frequency (daily and weekly) data.

Adaptive Gait Generation for Hexapod Robots Based on Reinforcement Learning and Hierarchical Framework

Gait plays a decisive role in the performance of hexapod robot walking; this paper focuses on adaptive gait generation with reinforcement learning for a hexapod robot. Moreover, the hexapod robot has a high-dimensional action space and therefore it is a great challenge to use reinforcement learning to directly train the robot’s joint angles. As a result, a hierarchical and modular framework and learning details are proposed in this paper, using only seven-dimensional vectors to denote the agent actions. In addition, we conduct experiments and deploy the proposed framework using a real hexapod robot. The experimental results show that superior reinforcement learning algorithms can converge in our framework, such as SAC, PPO, DDPG and TD3. Specifically, the gait policy trained in our framework can generate new adaptive hexapod gait on flat terrain, which is stable and has lower transportation cost than rhythmic gaits.

Deep Reinforcement Learning for Autonomous Ground Vehicle Exploration Without A-Priori Maps

Autonomous Ground Vehicles (AGVs) are essential tools for a wide range of applications stemming from their ability to operate in hazardous environments with minimal human operator input. Effective motion planning is paramount for successful operation of AGVs. Conventional motion planning algorithms are dependent on prior knowledge of environment characteristics and offer limited utility in information poor, dynamically altering environments such as areas where emergency hazards like fire and earthquake occur, and unexplored subterranean environments such as tunnels and lava tubes on Mars. We propose a Deep Reinforcement Learning (DRL) framework for intelligent AGV exploration without a-priori maps utilizing Actor-Critic DRL algorithms to learn policies in continuous and high-dimensional action spaces directly from raw sensor data. The DRL architecture comprises feedforward neural networks for the critic and actor representations in which the actor network strategizes linear and angular velocity control actions given current state inputs, that are evaluated by the critic network which learns and estimates Q-values to maximize an accumulated reward. Three off-policy DRL algorithms, DDPG, TD3 and SAC, are trained and compared in two environments of varying complexity, and further evaluated in a third with no prior training or knowledge of map characteristics. The agent is shown to learn optimal policies at the end of each training period to chart quick, collision-free exploration trajectories, and is extensible, capable of adapting to an unknown environment without changes to network architecture or hyperparameters. The best algorithm is further evaluated in a realistic 3D environment.

IQoR-LSE: An Intelligent QoS On-Demand Routing Algorithm With Link State Estimation

With the rapid development of Internet applications, diversified Quality of Service (QoS) has been required in packet routing to meet the demand of various types of applications. In this article, an Intelligent QoS on-demand Routing (IQoR) framework has been proposed to support multiclass QoS Provisioning for packet forwarding. In addition, we present an IQoR with Link State Estimation (IQoR-LSE) algorithm with the assistance of link congestion inference to guide the exploration of action space in deep reinforcement learning, to seek optimal routing policies. The IQoR-LSE algorithm is proposed to solve the non-convergence problem at exploring the high-dimensional action space by jointly estimating the link congestion. Extensive simulations show that IQoR-LSE algorithm outperforms other benchmark routing algorithms with efficient learning and a significant reduction in average delay, jitter and packet loss.

An IoT enabled smart healthcare system using deep reinforcement learning

AbstractSmart healthcare systems do exist with a variety of architectures. However, the hunt for better smart healthcare systems is more predominant. The cutting‐edge field of IoT (internet of things) and technological developments provide better solutions for smart healthcare systems using Sensor–Body Area Networks. Thus, the patient's sensor data can be collected, stored, analyzed, and suitable treatments can be offered, over the inter‐network, anytime, anywhere. The most complex part in such systems is the physician analysis of the huge volume of patient's data, to handle and prepare suitable diagnose and treatment for humanity. This article reveals a methodology of Deep Reinforcement Learning for smart healthcare decisions in an IoT interfaced Smart Healthcare–intelligent monitoring system. The system incorporates four layers, patient data collection, Edge computing, patient data transmission and Cloud computing. IoT is employed for automatic collection of Patient's data and for transmission of data, to data centers. Artificial intelligence techniques are used to analyze these data to provide suitable decisions, diagnosis, and treatment for those patients and humanity. Deep Reinforcement Learning provides the platform for smart decisions, diagnosis, and treatment. The investigation was experimented with synthetic simulated data of various BAN sensors. We developed a data set of size 286, which contains 21 different health parameters. After pre‐processing, these data were stored in the Amazon web services (AWS) cloud server using (message queue telemetry and transport) MQTT–IoT protocol. Initially, the Deep Q‐Network (DQN) was imposed to the training algorithm. The methodology was examined in PyTorch using a single GTX 1080 Ti X GPU with the training data sizes from 27 to 1536. The training time was about 10,000 to 90,000 s for training 500 epochs. In the high dimensional action space environment, the algorithm responded slowly to analyze, explore, and determine effective healthcare strategies. The systems convergence response of estimated hidden health state (g') and the actual health state (g) for the 21 different health parameters were estimated, whose values range from 0 to 1. The system responded with smart decisive interventions, which were good and close to that of a Physician's decision. The proposed methodology is definitely a promising solution for a smart and economic telemedicine.

A practical deep reinforcement learning framework for multivariate occupant-centric control in buildings

Reinforcement learning (RL) has been shown to have the potential for optimal control of heating, ventilation, and air conditioning (HVAC) systems. Although research on RL-based building control has received extensive attention in recent years, there is limited real-world implementation to evaluate its performance while keeping occupants in the loop. Additionally, many HVAC systems consist of multiple subsystems, but conventional RL algorithms face significant challenges when dealing with high-dimensional action spaces. This study proposes a practical deep reinforcement learning (DRL) based multivariate occupant-centric control framework that considers personalized thermal comfort and occupant presence. Specifically, Branching Dueling Q-network (BDQ) is leveraged as the learning agent to efficiently solve the multi-dimensional control task, and a tabular-based personal comfort modeling method is applied that is naturally integrated into human-in-the-loop operations. The BDQ agent is pre-trained in a virtual environment, followed by online deployment in a real office space for 5-dimensional action control. Based on the actual deployment and real-time comfort votes, our results showed a 14% reduction in cooling energy and an 11% improvement in total thermal acceptability.

Artificial Intelligence Powered Mobile Networks: From Cognition to Decision

Mobile networks (MNs) are anticipated to provide unprecedented opportunities to enable a new world of connected experiences and radically shift the way people interact with everything. MNs are becoming more and more complex, driven by ever increasing complicated configuration issues and blossoming new service requirements. This complexity poses significant challenges in deployment, management, operation, optimization, and maintenance, since they require complete understanding and cognition of MNs. Artificial intelligence (AI), which deals with the simulation of intelligent behavior in computers, has demonstrated enormous success in many application domains, suggesting its potential in cognizing the state of an MN and making intelligent decisions. In this article, we first propose an AI-powered MN architecture and discuss challenges in terms of cognition complexity, decisions with high-dimensional action space, and self-adaptation to system dynamics. Then potential solutions associated with AI are discussed. Finally, we propose a deep learning approach that directly maps the state of an MN to perceived QoS, integrating cognition with the decision. Our proposed approach helps operators to make more intelligent decisions to guarantee QoS. Meanwhile, the effectiveness and advantages of our proposed approach are demonstrated on a real-world dataset involving 31,261 users over 77 stations within 5 days.

Network Selection Based on Evolutionary Game and Deep Reinforcement Learning in Space-Air-Ground Integrated Network

In next generation communication system, space-air-ground integrated network (SAGIN) would be utilized to provide ubiquitous and unlimited wireless connectivity with large coverage, high throughput, and strong resilience. In this integrated network, there are multiple heterogeneous network options to satisfy service requirements, where an efficient network selection strategy is required to improve resource utilization and achieve load balance. In this paper, we propose a system model of network selection in SAGIN and formulate a corresponding evolutionary game. A network selection algorithm based on evolutionary game is proposed to study the autonomous decision-making process of network selection as a supplement. We also propose a deep deterministic policy gradient (DDPG)-based network selection algorithm to handle continuous and high-dimensional action spaces. A particular case is studied for further simulation and analysis. The evolutionary game obtains the selection strategy with the highest payoff at the evolutionary equilibrium point, and the stability of evolutionary equilibrium is proved by varying relative factors. The DDPG-based network selection algorithm obtains the same strategy with the highest reward at convergence at a slower speed. In comprehensive comparison, our proposed algorithms perform better than the repeated stochastic game approach and proximal policy optimization (PPO) algorithm.

Deep Learning for Wireless Coded Caching With Unknown and Time-Variant Content Popularity

Coded caching is effective in leveraging the accumulated storage size in wireless networks by distributing different coded segments of each file in multiple cache nodes. This paper aims to find a wireless coded caching policy to minimize the total discounted network cost, which involves both transmission delay and cache replacement cost, using tools from deep learning. The problem is known to be challenging due to the unknown, time-variant content popularity as well as the continuous, high-dimensional action space. We first propose a clustering based long short-term memory (C-LTSM) approach to predict the number of content requests using historical request information. This approach exploits the correlation of the historical request information between different files through clustering. Based on the predicted results, we then propose a supervised deep deterministic policy gradient (SDDPG) approach. This approach, on one hand, can learn the caching policy in continuous action space by using the actor-critic architecture. On the other hand, it accelerates the learning process by pre-training the actor network based on the solution of an approximate problem that minimizes the per-slot cost. Real-world trace-based numerical results show that the proposed prediction and caching policy using deep learning outperform the considered existing methods.

Deep Deterministic Policy Gradient (DDPG)-Based Resource Allocation Scheme for NOMA Vehicular Communications

This paper investigates the resource allocation problem in vehicular communications based on multi-agent Deep Deterministic Policy Gradient (DDPG), in which each Vehicle-to-Vehicle (V2V) communication acts as agent and adopts Non-Orthogonal Multiple Access (NOMA) technology to share the frequency spectrum that pre-allocated to Vehicle-to-Infrastructure (V2I) communications. Different with conventional D2D communications, the fast varying channel condition due to the high mobility in vehicular environment causes the difficulty of collecting instantaneous Channel State Information (CSI) at base station. Meanwhile, one tremendous challenge faced by vehicular communications is how to maximize the sum-rate of V2I communications simultaneously guaranteeing the latency and reliability requirements for the transmission of safety-critical information in V2V communications. In response, we formulate the resource allocation problem as a decentralized Discrete-time and Finite-state Markov Decision Process (DFMDP), in which allocation decisions are made by multiple agents that do not have complete and global network information. Due to the complexity of the problem, we propose a DDPG algorithm which is capable of handling continuous high dimensional action spaces to find the optimal allocation strategy. Numerical results verify that each agent can effectively learn from the environment by means of the proposed DDPG algorithm to maximize the sum-rate of V2I communications while satisfying the stringent latency and reliability constraints of V2V communications.

Online Deep Reinforcement Learning for Computation Offloading in Blockchain-Empowered Mobile Edge Computing

Offloading computation-intensive tasks (e.g., blockchain consensus processes and data processing tasks) to the edge/cloud is a promising solution for blockchain-empowered mobile edge computing. However, the traditional offloading approaches (e.g., auction-based and game-theory approaches) fail to adjust the policy according to the changing environment and cannot achieve long-term performance. Moreover, the existing deep reinforcement learning-based offloading approaches suffer from the slow convergence caused by high-dimensional action space. In this paper, we propose a new model-free deep reinforcement learning-based online computation offloading approach for blockchain-empowered mobile edge computing in which both mining tasks and data processing tasks are considered. First, we formulate the online offloading problem as a Markov decision process by considering both the blockchain mining tasks and data processing tasks. Then, to maximize long-term offloading performance, we leverage deep reinforcement learning to accommodate highly dynamic environments and address the computational complexity. Furthermore, we introduce an adaptive genetic algorithm into the exploration of deep reinforcement learning to effectively avoid useless exploration and speed up the convergence without reducing performance. Finally, our experimental results demonstrate that our algorithm can converge quickly and outperform three benchmark policies.

Autonomous exploration of motor skills by skill babbling

Autonomous exploration of motor skills is a key capability of learning robotic systems. Learning motor skills can be formulated as inverse modeling problem, which targets at finding an inverse model that maps desired outcomes in some task space, e.g., via points of a motion, to appropriate actions, e.g., motion control policy parameters. In this paper, autonomous exploration of motor skills is achieved by incrementally learning inverse models starting from an initial demonstration. The algorithm is referred to as skill babbling, features sample-efficient learning, and scales to high-dimensional action spaces. Skill babbling extends ideas of goal-directed exploration, which organizes exploration in the space of goals. The proposed approach provides a modular framework for autonomous skill exploration by separating the learning of the inverse model from the exploration mechanism and a model of achievable targets, i.e. the workspace. The effectiveness of skill babbling is demonstrated for a range of motor tasks comprising the autonomous bootstrapping of inverse kinematics and parameterized motion primitives.

Human Reinforcement Learning Subdivides Structured Action Spaces by Learning Effector-Specific Values

Humans and animals are endowed with a large number of effectors. Although this enables great behavioral flexibility, it presents an equally formidable reinforcement learning problem of discovering which actions are most valuable because of the high dimensionality of the action space. An unresolved question is how neural systems for reinforcement learning-such as prediction error signals for action valuation associated with dopamine and the striatum-can cope with this "curse of dimensionality." We propose a reinforcement learning framework that allows for learned action valuations to be decomposed into effector-specific components when appropriate to a task, and test it by studying to what extent human behavior and blood oxygen level-dependent (BOLD) activity can exploit such a decomposition in a multieffector choice task. Subjects made simultaneous decisions with their left and right hands and received separate reward feedback for each hand movement. We found that choice behavior was better described by a learning model that decomposed the values of bimanual movements into separate values for each effector, rather than a traditional model that treated the bimanual actions as unitary with a single value. A decomposition of value into effector-specific components was also observed in value-related BOLD signaling, in the form of lateralized biases in striatal correlates of prediction error and anticipatory value correlates in the intraparietal sulcus. These results suggest that the human brain can use decomposed value representations to "divide and conquer" reinforcement learning over high-dimensional action spaces.

Autonomous mental development in high dimensional context and action spaces

Autonomous Mental Development (AMD) of robots opened a new paradigm for developing machine intelligence, using neural network type of techniques and it fundamentally changed the way an intelligent machine is developed from manual to autonomous. The work presented here is a part of SAIL (Self-Organizing Autonomous Incremental Learner) project which deals with autonomous development of humanoid robot with vision, audition, manipulation and locomotion. The major issue addressed here is the challenge of high dimensional action space (5–10) in addition to the high dimensional context space (hundreds to thousands and beyond), typically required by an AMD machine. This is the first work that studies a high dimensional (numeric) action space in conjunction with a high dimensional perception (context state) space, under the AMD mode. Two new learning algorithms, Direct Update on Direction Cosines (DUDC) and High-Dimensional Conjugate Gradient Search (HCGS), are developed, implemented and tested. The convergence properties of both the algorithms and their targeted applications are discussed. Autonomous learning of speech production under reinforcement learning is studied as an example.