Size Of Action Space Research Articles

Component-wise Markov decision process for solving condition-based maintenance of large multi-component systems with economic dependence

Condition-Based Maintenance (CBM) of multi-component systems is a prevalent engineering problem due to its effectiveness in reducing the operational and maintenance costs of a system. However, developing the exact optimal maintenance decisions for a large multi-component system is computationally challenging, even not feasible, due to the exponential growth in system state and action space size with the number of components in the system. To address the scalability issue in CBM of large multi-component systems, we propose a Component-Wise Markov Decision Process(CW-MDP) and an Adjusted Component-Wise Markov Decision Process (ACW-MDP) to obtain an approximation of the optimal system-level CBM decision policy for large systems with heterogeneous components. We propose using an extended single-component action space to model the impact of system-level setup cost on a component-level solution. The theoretical gap between the proposed approach and system-level optima is also derived. Additionally, theoretical convergence and the relationship between ACW-MDP and CW-MDP are derived. The study further shows extensive numerical studies to demonstrate the effectiveness of component-wise solutions for solving large multi-component systems.

Double RISs assisted task offloading for NOMA-MEC with action-constrained deep reinforcement learning

Reconfigurable intelligent surface (RIS) is expected to enhance task offloading performance in non-line-of-sight mobile edge computing (MEC) scenarios. This paper aims at reducing the long-term energy consumption for task offloading in a MEC system assisted by double RISs while considering non-orthogonal multiple access (NOMA). Due to the non-convexity of the formulated problem, we propose an action-constrained deep reinforcement learning (DRL) framework based on twin delayed deep deterministic policy gradient (TD3) algorithm that includes inner and outer optimization processes, which greatly reduces the action space size of the DRL agent, making it easy to implement and achieve fast convergence. During the inner optimization phase, by utilizing theoretical derivations, we propose a low-complexity method to optimally derive the transmit power, the local computing frequency, and the computation resource allocation in the base station. In the outer optimization phase, building on the solution derived from the inner optimization, we further use the TD3 algorithm to jointly determine the phase shifting of RIS elements, the offloading ratio, and the transmission time for each time slot. Experiment results demonstrate that the proposed algorithm achieves rapid convergence performance. In comparison to single RIS assisted offloading, the double RISs assisted offloading scheme proposed in this paper can reduce energy consumption by 42.8% on average.

ADRLO: Adaptive deep reinforcement learning-based offloading for edge computing

Computation offloading, as an efficient and promising computing paradigm for mobile edge computing (MEC), provides optimal computation allocation between MEC servers and Internet of Things (IoT) devices. However, in modern wireless communications, most of the computational tasks are latency-sensitive, presenting an efficiency challenge for offloading services. Moreover, to make the best use of computation resources, it remains an issue to ensure stable optimal offloading for the allocation of computation resources. In this paper, a novel adaptive deep reinforcement learning-based offloading (ADRLO) framework is developed to achieve a stable optimal computation revenue with low execution latency for MEC offloading. Structurally, we implement a residual connected deep neural network (DNN) to generate a series of offloading strategies and use them to obtain the global system computation rate. Especially, the strategy that achieves the maximum computation rate is then selected as experience to support the DNN training, forming a closed-loop deep reinforcement learning (DRL) mechanism. To further enhance the efficiency and stability of the MEC offloading, an adaptive regulator is presented for the framework parameters including the size of action space and the DNN learning rate. Numerical results show that, compared to existing state-of-the-art approaches, our proposal achieves optimal computation revenue, exhibits better convergence performance, and significantly reduces the execution latency by almost 50%.

Computing Offloading With Fairness Guarantee: A Deep Reinforcement Learning Method

Edge computing can reduce service latency and save backhaul bandwidth by completing services at network edges, providing support for diverse computation-intensive and delay-sensitive services. However, it is not practical to support all services at edge nodes due to the limited network resources. The decision that which services can be provided locally and which services should been offloaded to cloud significantly impacts the user experience. Cloud-edge computing offloading becomes an important issue in edge computing. In this paper, we take the fairness into the optimization objective of computing offloading problem, and consider both computing capacity and storage space as problem constraints. The problem is formulated as a long-term average optimization problem to maximize the α-fair utility function of saved time, and further translated as a Markov decision process. As the optimization problem with fairness guarantee and huge action space, we cannot solve it with traditional methods. Therefore, an innovative multi-update deep reinforcement learning algorithm is proposed which can optimize the objective with α-fair utility function and reduce dramatically the size of action space. We also prove the convergence of our algorithm theoretically. To our best knowledge, the long-term average optimization of computing offloading with fairness guarantee is rarely seen in literature. Extensive simulation experiments show that our algorithm can converge quickly and has better performance in terms of service delay and fairness.

Affordance-Based Human–Robot Interaction With Reinforcement Learning

Planning precise manipulation in robotics to perform grasp and release-related operations, while interacting with humans is a challenging problem. Reinforcement learning (RL) has the potential to make robots attain this capability. In this paper, we propose an affordance-based human-robot interaction (HRI) framework, aiming to reduce the action space size that would considerably impede the exploration efficiency of the agent. The framework is based on a new algorithm called Contextual Q-learning (CQL). We first show that the proposed algorithm trains in a reduced amount of time (2.7 seconds) and reaches an 84% of success rate. This suits the robot’s learning efficiency to observe the current scenario configuration and learn to solve it. Then, we empirically validate the framework for implementation in HRI real-world scenarios. During the HRI, the robot uses semantic information from the state and the optimal policy of the last training step to search for relevant changes in the environment that may trigger the generation of a new policy.

Reinforcement learning-based cost-efficient service function chaining with CoMP zero-forcing beamforming in edge networks

As two promising paradigms in emerging 6G wireless systems, service function chaining (SFC) and mobile edge computing (MEC) have attracted insensitive attentions from both industry and academia, and would bring more close-proximity services to 6G users with communication, computing and caching (3C) resources, yet also faced with challenges arising in time-varying channel conditions and resource dynamics. In this work, boosted by recent advents in artificial intelligence and reinforcement learning, we investigate the on-line SFC deployment in the edge of 6G wireless systems via the actor–critic learning framework. First, one long-run cost-efficient SFC deployment problem is investigated, and the coordinated multiple points (CoMP)-based zero-forcing beamforming is utilized to cancel the interference across SFCs. Then, by exploiting the Markov decision processes (MDP) property of long-run SFC deployment, one natural gradient-based actor–critic framework is proposed to characterize edge network dynamics, and meanwhile facilitates the training of neural networks to the global optimum. Next, to lower the size of action space, we follow the principle that a subproblem is embedded into each state–action pair’s critic to solve the reward function, and then utilize both the ℓp(0<p<1) norm-based successive convex approximation (SCA) and proximal center-based dual decomposition to approach the global optimum and accelerate the convergence. Finally, numerical results are used to validate proposed actor–critic approach, showing that the communication resource management deserves special attentions in the SFC deployment in the edge of 6G wireless systems.

Energy-Efficient Ultra-Dense Network With Deep Reinforcement Learning

With the explosive growth in mobile data traffic, ultra-dense network (UDN) where a large number of small cells are densely deployed on top of macro cells has received a great deal of attention in recent years. While UDN offers a number of benefits, an upsurge of energy consumption in UDN due to the intensive deployment of small cells has now become a major bottleneck in achieving the primary goals viz., 100-fold increase in the throughput in 5G+ and 6G. In recent years, an approach to reduce the energy consumption of base stations (BSs) by selectively turning off the lightly-loaded BSs, referred to as the sleep mode technique, has been suggested. However, determining an appropriate active/sleep modes of BSs is a difficult task due to the huge computational overhead and inefficiency caused by the frequent BS mode conversion. An aim of this paper is to propose a deep reinforcement learning (DRL)-based approach to achieve a reduction of energy consumption in UDN. Key ingredient of the proposed scheme is to use decision selection network to reduce the size of action space. Numerical results show that the proposed scheme can significantly reduce the energy consumption of UDN while ensuring the rate requirement of network.

Heterogeneous Markov Decision Process Model for Joint Resource Allocation and Task Scheduling in Network Slicing Enabled Internet of Vehicles

In the Internet of Vehicles (IoV) with network slicing functions, both the inter-slice resource allocation and the intra-slice task scheduling have received extensive attention. However, the joint optimization of these two problems has not been fully studied. This letter focuses on the complexity issue of the joint resource allocation and task scheduling problem in the IoV with both ultra-reliable low-latency communication (URLLC) and enhanced mobile broadband (eMBB) slices. A heterogeneous Markov decision process (HMDP) which models the resource allocation process and the task scheduling process in two different layers is proposed. Benefiting from the two-layer structure of the HMDP, the size of action space is significantly reduced, and the asynchronous decisions of different sub-MDPs are enabled. The corresponding layered deep reinforcement learning (DRL) architecture is also designed to solve the HMDP-based optimization problem. Simulation results show that the layered DRL-based algorithm outperforms other methods.

EC-SAGINs: Edge-Computing-Enhanced Space–Air–Ground-Integrated Networks for Internet of Vehicles

Edge-computing-enhanced Internet of Vehicles (EC-IoV) enables ubiquitous data processing and content sharing among vehicles and terrestrial edge computing (TEC) infrastructures (e.g., 5G base stations and roadside units) with little or no human intervention, and plays a key role in the intelligent transportation systems. However, EC-IoV is heavily dependent on the connections and interactions between vehicles and TEC infrastructures, thus will break down in some remote areas where TEC infrastructures are unavailable (e.g., desert, isolated islands, and disaster-stricken areas). Driven by the ubiquitous connections and global-area coverage, space–air–ground-integrated networks (SAGINs) efficiently support seamless coverage and efficient resource management, and represent the next frontier for edge computing. In light of this, we first review the state-of-the-art edge computing research for SAGINs in this article. After discussing several existing orbital and aerial edge computing architectures, we propose a framework of edge computing-enabled SAGINs to support various Internet of Vehicles (EC-IoV) services for the vehicles in remote areas. The main objective of the framework is to minimize the task completion time and satellite resource usage. To this end, a preclassification scheme is presented to reduce the size of action space, and a deep imitation learning-driven offloading and caching algorithm is proposed to achieve real-time decision making. The simulation results show the effectiveness of our proposed scheme. Finally, we also discuss some technology challenges and future directions.

An $L^2$ Analysis of Reinforcement Learning in High Dimensions with Kernel and Neural Network Approximation

Reinforcement learning (RL) algorithms based on high-dimensional function approximation have achieved tremendous empirical success in large-scale problems with an enormous number of states. However, most analysis of such algorithms gives rise to error bounds that involve either the number of states or the number of features. This paper considers the situation where the function approximation is made either using the kernel method or the two-layer neural network model, in the context of a fitted Q-iteration algorithm with explicit regularization. We establish an $\tilde{O}(H^3|\mathcal {A}|^{\frac14}n^{-\frac14})$ bound for the optimal policy with $Hn$ samples, where $H$ is the length of each episode and $|\mathcal {A}|$ is the size of action space. Our analysis hinges on analyzing the $L^2$ error of the approximated Q-function using $n$ data points. Even though this result still requires a finite-sized action space, the error bound is independent of the dimensionality of the state space.

Optimal control in dynamic food supply chains using big data

We consider a dynamic food supply chain with multiple process steps where the decisions at each step include supplier selection and settings for their process parameters. We assume that the resulting quality level of the product at every step is stochastic. We also consider that this quality level at each supply chain step has a minimum accepted level and any product that does not meet the minimum accepted level is rejected. We propose an integrated approach that uses Big Data mining techniques to study the effect of these decisions on the quality of the final product and determine our state transition matrix. From there, stochastic optimization methods are implemented to find an optimal policy describing the preferred suppliers and settings for their process parameters. The goal of this policy is to maximize the supply chain’s expected profit and reduce rejected product. We show how this technique outperforms traditional techniques in running time as the problem’s complexity (state and action space size) grows. We also conduct experiments to explore how the optimal policy changes as the quality threshold for acceptance becomes more restrictive and exhibits a convex behavior.

Sequential Reinforced 360-Degree Video Adaptive Streaming With Cross-User Attentive Network

In the tile-based 360-degree video streaming, predicting user's future viewpoints and developing adaptive bitrate (ABR) algorithms are essential for optimizing user's quality of experience (QoE). Traditional single-user based viewpoint prediction methods fail to achieve good performance in long-term prediction, and the recently proposed reinforcement learning (RL) based ABR schemes applied in traditional video streaming can not be directly applied in the tile-based 360-degree video streaming due to the exponential action space. Therefore, we propose a sequential reinforced 360-degree video streaming scheme with cross-user attentive network. Firstly, considering different users may have the similar viewing preference on the same video, we propose a cross-user attentive network (CUAN), boosting the performance of long-term viewpoint prediction by selectively utilizing cross-user information. Secondly, we propose a sequential RL-based (360SRL) ABR approach, transforming action space size of each decision step from exponential to linear via introducing a sequential decision structure. We evaluate the proposed CUAN and 360SRL using trace-driven experiments and experimental results demonstrate that CUAN and 360SRL outperform existing viewpoint prediction and ABR approaches with a noticeable margin.

A Turbo Q-Learning (TQL) for Energy Efficiency Optimization in Heterogeneous Networks

In order to maximize energy efficiency in heterogeneous networks (HetNets), a turbo Q-Learning (TQL) combined with multistage decision process and tabular Q-Learning is proposed to optimize the resource configuration. For the large dimensions of action space, the problem of energy efficiency optimization is designed as a multistage decision process in this paper, according to the resource allocation of optimization objectives, the initial problem is divided into several subproblems which are solved by tabular Q-Learning, and the traditional exponential increasing size of action space is decomposed into linear increase. By iterating the solutions of subproblems, the initial problem is solved. The simple stability analysis of the algorithm is given in this paper. As to the large dimension of state space, we use a deep neural network (DNN) to classify states where the optimization policy of novel Q-Learning is set to label samples. Thus far, the dimensions of action and state space have been solved. The simulation results show that our approach is convergent, improves the convergence speed by 60% while maintaining almost the same energy efficiency and having the characteristics of system adjustment.

Application of Deep Reinforcement Learning in Traffic Signal Control: An Overview and Impact of Open Traffic Data

Persistent congestions which are varying in strength and duration in the dense traffic networks are the most prominent obstacle towards sustainable mobility. Those types of congestions cannot be adequately resolved by the traditional Adaptive Traffic Signal Control (ATSC). The introduction of Reinforcement Learning (RL) in ATSC as tackled those types of congestions by using on-line learning, which is based on the trial and error approach. Furthermore, RL is prone to the dimensionality curse related to the state–action space size based on which a non-linear quality function is derived. The Deep Reinforcement Learning (DRL) framework uses Deep Neural Networks (DNN) to digest raw traffic data to approximate the quality function of RL. This paper provides a comprehensive analysis of the most recent DRL approaches used for the ATSC algorithm design. Special emphasis is set to overview of the traffic state representation and multi-agent DRL frameworks applied for the large traffic networks. Best practices are provided for choosing the adequate DRL model, hyper-parameters tuning, and model architecture design. Finally, this paper provides a discussion about the importance of the open traffic data concept for the extensive application of DRL in the real world ATSC.

A Deep Reinforcement Learning-Based Transcoder Selection Framework for Blockchain-Enabled Wireless D2D Transcoding

The boom of video streaming industry has resulted in the increasing demands for transcoding services from heterogeneous users. Recent advances of blockchain technology allow some startups to realize decentralized collaborative transcoding through device-to-device (D2D) networks, where a group of transcoders are selected to perform transcoding cooperatively. For the blockchain-enabled D2D transcoding systems, it's imperative to jointly design transcoder selection, task scheduling and resource allocation schemes in order to provide efficient and trustworthy transcoding services. In this paper, viewing the involved multi-dimensional complex factors and channel fluctuation, we propose a novel deep reinforcement learning (DRL) based transcoder selection framework for blockchain enabled D2D transcoding systems where both the platform dynamics and channel statistics are captured. To reduce the action space size, we adopt a two-stage decision approach to first select the transcoders through a normal DRL based framework and then obtain the optimal task scheduling, power control, and resource allocation scheme by solving a stochastic optimization problem with the constrained stochastic successive convex approximation (CSSCA) approach. Simulation results show that our proposed framework can achieve high transcoding revenue while meeting the quality of service (QoS) requirements, and it can well handle dynamic cases.

On-Demand Channel Bonding in Heterogeneous WLANs: A Multi-Agent Deep Reinforcement Learning Approach.

In order to meet the ever-increasing traffic demand of Wireless Local Area Networks (WLANs), channel bonding is introduced in IEEE 802.11 standards. Although channel bonding effectively increases the transmission rate, the wider channel reduces the number of non-overlapping channels and is more susceptible to interference. Meanwhile, the traffic load differs from one access point (AP) to another and changes significantly depending on the time of day. Therefore, the primary channel and channel bonding bandwidth should be carefully selected to meet traffic demand and guarantee the performance gain. In this paper, we proposed an On-Demand Channel Bonding (O-DCB) algorithm based on Deep Reinforcement Learning (DRL) for heterogeneous WLANs to reduce transmission delay, where the APs have different channel bonding capabilities. In this problem, the state space is continuous and the action space is discrete. However, the size of action space increases exponentially with the number of APs by using single-agent DRL, which severely affects the learning rate. To accelerate learning, Multi-Agent Deep Deterministic Policy Gradient (MADDPG) is used to train O-DCB. Real traffic traces collected from a campus WLAN are used to train and test O-DCB. Simulation results reveal that the proposed algorithm has good convergence and lower delay than other algorithms.

On Value Function Representation of Long Horizon Problems

In Reinforcement Learning, an intelligent agent has to make a sequence of decisions to accomplish a goal. If this sequence is long, then the agent has to plan over a long horizon. While learning the optimal policy and its value function is a well studied problem in Reinforcement Learning, this paper focuses on the structure of the optimal value function and how hard it is to represent the optimal value function. We show that the generalized Rademacher complexity of the hypothesis space of all optimal value functions is dependent on the planning horizon and independent of the state and action space size. Further, we present bounds on the action-gaps of action value functions and show that they can collapse if a long planning horizon is used. The theoretical results are verified empirically on randomly generated MDPs and on a grid-world fruit collection task using deep value function approximation. Our theoretical results highlight a connection between value function approximation and the Options framework and suggest that value functions should be decomposed along bottlenecks of the MDP's transition dynamics.

Every team deserves a second chance: an extended study on predicting team performance

Voting among different agents is a powerful tool in problem solving, and it has been widely applied to improve the performance in finding the correct answer to complex problems. We present a novel benefit of voting, that has not been observed before: we can use the voting patterns to assess the performance of a team and predict their final outcome. This prediction can be executed at any moment during problem-solving and it is completely domain independent. Hence, it can be used to identify when a team is failing, allowing an operator to take remedial procedures (such as changing team members, the voting rule, or increasing the allocation of resources). We present three main theoretical results: (1) we show a theoretical explanation of why our prediction method works; (2) contrary to what would be expected based on a simpler explanation using classical voting models, we show that we can make accurate predictions irrespective of the strength (i.e., performance) of the teams, and that in fact, the prediction can work better for diverse teams composed of different agents than uniform teams made of copies of the best agent; (3) we show that the quality of our prediction increases with the size of the action space. We perform extensive experimentation in two different domains: Computer Go and Ensemble Learning. In Computer Go, we obtain high quality predictions about the final outcome of games. We analyze the prediction accuracy for three different teams with different levels of diversity and strength, and show that the prediction works significantly better for a diverse team. Additionally, we show that our method still works well when trained with games against one adversary, but tested with games against another, showing the generality of the learned functions. Moreover, we evaluate four different board sizes, and experimentally confirm better predictions in larger board sizes. We analyze in detail the learned prediction functions, and how they change according to each team and action space size. In order to show that our method is domain independent, we also present results in Ensemble Learning, where we make online predictions about the performance of a team of classifiers, while they are voting to classify sets of items. We study a set of classical classification algorithms from machine learning, in a data-set of hand-written digits, and we are able to make high-quality predictions about the final performance of two different teams. Since our approach is domain independent, it can be easily applied to a variety of other domains.

Optimal Policy of Cross-Layer Design for Channel Access and Transmission Rate Adaptation in Cognitive Radio Networks

In this paper, we investigate the cross-layer design of joint channel access and transmission rate adaptation in CR networks with multiple channels for both centralized and decentralized cases. Our target is to maximize the throughput of CR network under transmission power constraint by taking spectrum sensing errors into account. In centralized case, this problem is formulated as a special constrained Markov decision process (CMDP), which can be solved by standard linear programming (LP) method. As the complexity of finding the optimal policy by LP increases exponentially with the size of action space and state space, we further apply action set reduction and state aggregation to reduce the complexity without loss of optimality. Meanwhile, for the convenience of implementation, we also consider the pure policy design and analyze the corresponding characteristics. In decentralized case, where only local information is available and there is no coordination among the CR users, we prove the existence of the constrained Nash equilibrium and obtain the optimal decentralized policy. Finally, in the case that the traffic load parameters of the licensed users are unknown for the CR users, we propose two methods to estimate the parameters for two different cases. Numerical results validate the theoretic analysis.