Optimal Joint Policy Research Articles

Multi-Agent Deep Reinforcement Learning for Trajectory Design and Power Allocation in Multi-UAV Networks

Unmanned aerial vehicle (UAV) is regarded as an effective technology in future wireless networks. However, due to the non-convexity feature of joint trajectory design and power allocation (JTDPA) issue, it is challenging to attain the optimal joint policy in multi-UAV networks. In this article, a multi-agent deep reinforcement learning-based approach is presented to achieve the maximum long-term network utility while satisfying the user equipments' quality of service requirements. Moreover, considering that the utility of each UAV is determined based on the network environment and other UAVs' actions, the JTDPA problem is modeled as a stochastic game. Due to the high computational complexity caused by the continuous action space and large state space, a multi-agent deep deterministic policy gradient method is proposed to obtain the optimal policy for the JTDPA issue. Numerical results indicate that our method can obtain the higher network utility and system capacity than other optimization methods in multi-UAV networks with lower computational complexity.

Communications, Caching, and Computing for Mobile Virtual Reality: Modeling and Tradeoff

Virtual reality (VR) over wireless is emerging as an important use case of 5G networks. Immersive VR experience requires the delivery of huge data at ultra-low latency, thus demanding ultra-high transmission rate. This challenge can be largely addressed by the recent network architecture known as mobile edge computing (MEC), which enables caching and computing capabilities at the edge of wireless networks. This paper presents a novel MEC-based mobile VR delivery framework that is able to cache parts of the field of views (FOVs) in advance and run certain post-processing procedures at the mobile VR device. To optimize resource allocation at the mobile VR device, we formulate a joint caching and computing decision problem to minimize the average required transmission rate while meeting a given latency constraint. When FOVs are homogeneous, we obtain a closed-form expression for the optimal joint policy which reveals interesting communications-caching-computing tradeoffs. When FOVs are heterogeneous, we obtain a local optima of the problem by transforming it into a linearly constrained indefinite quadratic problem then applying concave convex procedure. Numerical results demonstrate great promises of the proposed mobile VR delivery framework in saving communication bandwidth while meeting low latency requirement.

Joint Sponsor Scheduling in Cellular and Edge Caching Networks for Mobile Video Delivery

The explosive growth of mobile video traffic introduces new challenges for the network infrastructure. Edge caching, as one of the key technologies in 5G wireless networks, has shown great potential to improve the quality of mobile video services by reducing the transmission overhead over backhaul links. With edge caching, content providers (CPs) need to decide not only the traditional data sponsoring strategy on cellular networks (where CPs cover part or all of the mobile users’ cellular data cost), but also a novel cache sponsoring strategy on the edge caching networks (where CPs place part of contents on edge networks in advance). In this paper, we study the joint optimization of both sponsors on cellular and edge caching networks for a single CP, aiming at maximizing the CP's revenue. Specifically, we formulate the joint optimization problem as a two-stage sequential decision problem. In stage I, the CP determines the edge caching policy (for a relatively long time period). In stage II, the CP decides the real-time data sponsoring strategy for each content request within the period. We analyze this two-stage decision problem systematically. First, we propose an online sponsoring strategy in stage II based on Lyapunov optimization framework. Then, we propose an edge caching strategy in stage I via predicting the number of aggregate user requests. Simulations on real data traces show that such a joint optimization policy can increase the CP's revenue by 124%–154%, comparing with the traditional data sponsoring policy (i.e., without edge caching). Moreover, the proposed online strategy can achieve 90% of the maximum revenue in the offline benchmark.

A Proactive Decision Support System for Online Event Streams

In-stream big data processing is an important part of big data processing. Proactive decision support systems can predict future system states and execute some actions to avoid unwanted states. In this paper, we propose a proactive decision support system for online event streams. Based on Complex Event Processing (CEP) technology, this method uses structure varying dynamic Bayesian network to predict future events and system states. Different Bayesian network structures are learned and used according to different event context. A networked distributed Markov decision processes model with predicting states is proposed as sequential decision making model. A Q-learning method is investigated for this model to find optimal joint policy. The experimental evaluations show that this method works well for congestion control in transportation system.

Joint Optimal Policy for Subsidy on Electric Vehicles and Infrastructure Construction in Highway Network

The promotion of the battery electric vehicle has become a worldwide problem for governments due to its short endurance range and slow charging rate. Besides an appropriate network of charging facilities, a subsidy has proved to be an effective way to increase the market share of battery electric vehicles. In this paper, we investigate the joint optimal policy for a subsidy on electric vehicles and infrastructure construction in a highway network, where the impact of siting and sizing of fast charging stations and the impact of subsidy on the potential electric vehicle flows is considered. A new specified local search (LS)-based algorithm is developed to maximize the overall number of available battery electric vehicles in the network, which can get provide better solutions in most situations when compared with existed algorithms. Moreover, we firstly combined the existing algorithms to establish a multi-stage optimization method, which can obtain better solutions than all existed algorithms. A practical case from the highway network in Hunan, China, is studied to analyze the factors that impact the choice of subsidy and the deployment of charging stations. The results prove that the joint policy for subsidy and infrastructure construction can be effectively improved with the optimization model and the algorithms we developed. The managerial analysis indicates that the improvement on the capacity of charging facility can increase the proportion of construction fees in the total budget, while the improvement in the endurance range of battery electric vehicles is more efficient in expanding battery electric vehicle adoption in the highway network. A more detailed formulation of the battery electric vehicle flow demand and equilibrium situation will be studied in the future.

Energy-efficient proactive edge caching with sleep scheduling for green networks

Proactive content caching in small cells (SCs) (i.e., proactive edge caching) can significantly reduce energy consumption of networks. However, the sleep mode of SCs can make the cached contents in SCs unavailable. Therefore, a joint optimization of edge caching strategy and sleep scheduling should be conducted to maximize the effectiveness of edge caching. In this paper, we propose an energy-efficient proactive edge caching with sleep scheduling (E3CS) where the controller jointly adjusts the caching strategy and the sleep scheduling of SCs by considering the content popularity dynamics. To optimize the performance of E3CS (i.e., minimize the overall energy consumption), we formulate a Markov decision process (MDP) and the joint optimal policy on the caching strategy and sleep scheduling is obtained by a value iteration algorithm. Evaluation results demonstrate that E3CS with the optimal policy outperforms the comparison schemes in terms of the overall energy consumption.

Energy-Efficient Mode Switching Mechanism With Flexible Functional Splitting in Energy Harvesting Cloud Radio Access Networks

In energy harvesting cloud radio access networks, the sleep mode switching of small radio remote head (SRRH) and the functional splitting between SRRH and baseband unit are two important challenges for improving energy efficiency and throughput. In this paper, we propose an energy-efficient mode switching mechanism (E2MSM) with flexible functional splitting in which a controller determines the modes of SRRHs and functional splitting level by considering the renewable energy levels and populations of SRRHs. To optimize the performance of E2MSM, a constraint Markov decision process problem is formulated and a joint optimal policy on the sleep scheduling and the functional splitting level is obtained by a linear programming. Evaluation results demonstrate that E2MSM with the optimal policy can achieve comparable throughput to the throughput-oriented scheme while consuming no non-renewable energy.

Dynamic optimal joint policies for tourism promotion and environmental restoration

Policies regarding tourism management can be separated into (1) tourism promotion policies and (2) environmental restoration policies. Because of the close relationship between the number of visitors and environmental quality, both policy types should be jointly designed and implemented. This paper studies the optimal use of these two policies in tourism management. In the case of Khao Laem Ya–Mu Koh Samet National Park in Thailand, the optimal policy balance requires the substantially higher restoration expenditure than the promotion expenditure at all time. With optimal policy balance, the number of visitors at the steady state increases more than twice, whereas the water quality remains unchanged. The implementation of the optimal policy balance would yield an additional benefit in terms of net present value of approximately $120 million over 50 years. The results from sensitivity analysis affirm the importance of restoration over promotion policy and suggest that improving effectiveness of restoration expenditure and reducing visitor impact on environmental quality can significantly increase benefits from tourism.

Joint optimization of lot sizing and condition-based maintenance for multi-component production systems

This paper presents a joint optimization model of production lot sizing and condition-based maintenance for a multi-component production system which produces the products to meet the demand in a finite time horizon. The components in the system deteriorate gradually with usage and age. To evaluate the conditions of components, inspections are carried out after each production run. The condition-based maintenance decision-making rules are based not only on the predictive reliability but also on the structural importance of each component. Moreover, the economic dependency among components is also considered when performing corrective maintenance actions. The preventive maintenance cost, the corrective maintenance cost, the setup cost, the inspection cost, the inventory holding cost and the shortage cost are considered in this paper. The aim is to minimize the total cost by jointly optimizing two decision variables: the production lot size and the preventive maintenance threshold. The optimal joint policy is obtained by coupling simulation model and genetic algorithm. Finally, the use and advantages of our model are illustrated through a case study of a cluster tool.

A joint optimal policy of block preventive replacement and spare part inventory

Maintenance and spare part should be considered jointly as they are related activities and maintenance generates the demand of spare parts. In this study, a joint optimization policy of block replacement and spare parts inventory for multi-components system is proposed by simulation optimization approach using Monte Carlo simulation. Through analyzing the process of maintenance replacement and spare inventory within replacement interval T, the total expected cost can be derived. The expected cost per unit time is the objective function to optimize the block replacement interval T and maximum inventory level S. Monte Carlo method is used to simulate the operate process and simulation algorithm is presented to derive the approximate optimal solution. Finally, a numerical example is presented to illustrate the effectiveness of the proposed model. Sensitivity analysis and the effect of relevant parameters on the optimal decisions are discussed.

Optimal ordering and pricing decisions for a company issuing product-specific gift cards

In this paper, we investigate gift card's influence on retailers’ ordering decisions and analyse the benefits of issuing product-specific gift cards. We propose an optimal ordering model for retailers’ decision problems with gift cards being issued. We also solve the problem with the analytical forms of optimal order quantities and expected profits. By comparing the results with the classical newsvendor problem, we classify the benefits of issuing gift cards into three categories: (1) the demand stimulating; (2) the pre-payment, and (3) the non-redemption. We step further to explore a retailer's problem on how to determine the optimal discount for gift cards, which is characterized by a joint optimal policy on the ordering quantity and the discount. We derive the optimal condition on this joint policy. Numerical examples are conducted to illustrate the model results and analyse the influences of parameters. A sample average approximation method is also been introduced to solve the optimization model.

Modeling condition-based maintenance and replacement strategies for an imperfect production-inventory system

In this article, we consider an imperfect production-inventory system which produces a single type of product to meet the constant demand. The system deteriorates stochastically with usage and the deterioration process is modeled by a non-stationary gamma process. The production process is imperfect which means that the system produces some non-conforming items and the product quality depends on the degradation level of the production system. To prevent the system from deteriorating worse and improve the product quality, preventive maintenance is performed when the level of the system degradation reaches a certain threshold. However, the preventive maintenance is imperfect which cannot restore the system as good as new. Hence, the aging system will be replaced by a new one after some production cycles. The preventive maintenance cost, the replacement cost, the production cost, the inventory holding cost and the penalty cost of lost sales are considered in this article. The objective is to minimize the total cost per unit item which depends on two decision variables: the preventive maintenance threshold and the time at which the system is replaced. We derive the explicit expression of the total cost per unit item and the optimal joint policy can be obtained numerically. An illustrative example and sensitivity analysis are given to demonstrate the proposed model.

Optimal multi-level classification and preventive maintenance policy for highly reliable products

Highly reliable products are widely used in aerospace, automotive, integrated manufacturing and other fields. With increasing market demand and competition, product classification for different segment market segments has become more and more critical. Leading manufacturers are always searching and designing classification policies for highly reliable products. On the other hand, preventive maintenance can improve the operation efficiency of the product, extend the service life and reduce enormous losses brought by failures. These two factors are taken into account by many large enterprises when making sound economical and operational decisions. Therefore, this research proposes a joint multi-level classification and preventive maintenance model (JMCPM model) under age-based maintenance. Different preventive maintenance policies are developed for corresponding level units. Accordingly, the optimal joint policy of multi-level classification and preventive maintenance can be obtained by JMCPM. In this model, degradation-based burn-in is utilised to eliminate defective units and collect degradation data. The degradation data are the basis of classification and can be used to estimate the residual life. Then, for making full use of these data, linear discriminant analysis is employed to design classification rules. The objective of the JMCPM model is to minimise the average cost per unit time by properly choosing the settings of classification and preventive maintenance intervals simultaneously. Finally, a simulation study is carried out for evaluating the performance of the JMCPM model. For an illustration of the proposed model and the methods of inference developed here, a real case involving degradation data from electrical connectors is analysed.

Maximizing Energy Efficiency in Multiuser Multicarrier Broadband Wireless Systems: Convex Relaxation and Global Optimization Techniques

A key challenge toward green communications is how to maximize energy efficiency by optimally allocating wireless resources in large-scale multiuser multicarrier orthogonal frequency-division multiple-access (OFDMA) systems. The quality-of-service (QoS)-constrained energy efficiency maximization problem is generally hard to solve due to the inverse transposition of the optimization operands in the optimization objective. We apply convex relaxation to make the problem quasiconcave with respect to power and concave with respect to the subcarrier indexing coefficients. The Karush–Kuhn–Tucker (KKT) optimality conditions lead to transcendental functions, where existing solutions are only numerically tractable. Different from the existing approaches, we apply the Maclaurin series expansion technique to transform the complex transcendental functions into simple polynomial expressions that allow us to obtain the global optimum in fast polynomial time, with the tractable upper bound of truncation error. With the new solution method, we propose a joint optimal allocation policy for both adaptive power and dynamic subcarrier allocations. We gain insight on the optimality, feasibility, and computational complexity of the joint optimal solution to show that the proposed scheme is theoretically and practically sound with fast convergence toward near-optimal solutions with an explicitly tractable truncation error. The simulation results confirm that the proposed scheme achieves a much higher energy efficiency performance with the guaranteed QoS and much lower complexity than existing approaches in the literature.

Optimal Inventory Control and Pricing of Perishable Items Without Shortages

In this paper, we consider a joint pricing and dynamic production policy for perishable items without shortages. A unimodal time-varying demand function is assumed to be convex nonincreasing in sales price. We propose a dynamic optimization model to maximize total profit by allocating a limited production capacity and setting a suitable sales price. For a given sales price, the optimal production rate can be obtained by solving a dynamic optimization problem with Pontryagin’s maximum principle. By virtue of the relationships between price intervals and the corresponding optimal production policies, an effective algorithm is designed to generate the joint policy for a system with a general nonlinear demand function of price. For a linear demand function, it is shown that the objective function is concave in price and that there exists a unique optimal joint policy. Numerical examples are presented to illustrate the validity of the theoretical results, and several managerial implications in terms of the production and pricing of perishable items are provided.

Solving Transition-Independent Multi-Agent MDPs with Sparse Interactions

In cooperative multi-agent sequential decision making under uncertainty, agents must coordinate to find an optimal joint policy that maximises joint value. Typical algorithms exploit additive structure in the value function, but in the fully-observable multi-agent MDP (MMDP) setting such structure is not present. We propose a new optimal solver for transition-independent MMDPs, in which agents can only affect their own state but their reward depends on joint transitions. We represent these dependencies compactly in conditional return graphs (CRGs). Using CRGs the value of a joint policy and the bounds on partially specified joint policies can be efficiently computed. We propose CoRe, a novel branch-and-bound policy search algorithm building on CRGs. CoRe typically requires less runtime than available alternatives and finds solutions to previously unsolvable problems.

Geometric process repair model for an unreliable production system with an intermediate buffer

In this paper, we consider an unreliable production system consisting of two machines (M1 and M2) in which M1 produces a single product type to satisfy a constant and continuous demand of M2 and it is subjected to random failures. In order to palliate perturbations caused by failures, a buffer stock is built up to satisfy the demand during the production unavailability of M1. A traditional assumption made in the previous research is that repairs can restore the failed machines to as good as new state. To develop a more realistic mathematical model of the system, we relax this assumption by assuming that the working times of M1 after repairs are geometrically decreasing, which means M1 cannot be repaired as good as new. Undergoing a specified number of repairs, M1 will be replaced by an identical new one. A bivariate policy [Formula: see text] is considered, where S is the buffer stock level and N is the number of failures at which M1 is replaced. We derive the long-run average cost rate [Formula: see text] used as the basis for optimal determination of the bivariate policy. The optimal policies [Formula: see text] and [Formula: see text] are derived, respectively. Then, an algorithm is presented to find the optimal joint policy [Formula: see text]. Finally, an illustrative example is given to validate the proposed model. Sensitivity analyses are also carried out to illustrate the effectiveness and robustness of the proposed methodology.

Dynamic joint pricing and production policy for perishable products

AbstractA joint dynamic pricing and production problem for perishable products without shortages is considered. The demand rate is price‐dependent and time‐varying. This paper constructs an optimal control model to maximize the total profit under a general nonlinear production cost function. The feature of the optimal joint dynamic pricing and production policy is analyzed by solving the corresponding optimal control problem on the basis of improved Pontryagin's maximum principle. Then, an effective algorithm is designed to obtain the optimal joint policy. The case of the joint static optimal policy is also investigated and compared with the dynamic one. Finally, numerical examples are presented to illustrate the effectiveness of the proposed methods, and some managerial implications are provided for the management of perishable items.

Joint optimal lot sizing and preventive maintenance policy for a production facility subject to condition monitoring

In this paper, we consider the joint optimization of economic manufacturing quantity (EMQ) and preventive maintenance (PM) policy for a production facility subject to deterioration and condition monitoring (CM). Unlike the previous joint models of EMQ and maintenance policy which used traditional maintenance approaches, we propose the proportional hazards model (PHM) to consider CM information as well as the age of the production facility. The deterioration process is determined by the age and covariate values and the covariate process is modeled as a continuous-time Markov process. The condition information is available at each inspection epoch, which is the end of each production run. The hazard rate is estimated after obtaining the new information through CM. The problem is formulated and solved in the semi-Markov decision process (SMDP) framework. The objective is to minimize the long-run expected average cost per unit time. Also, the mean residual life (MRL) of the production facility is calculated as an important statistic for practical applications. A numerical example is provided and a comparison with the age-based policy shows an outstanding performance of the new model and the control policy proposed in this paper.

Learning Multirobot Hose Transportation and Deployment by Distributed Round-Robin Q-Learning

Multi-Agent Reinforcement Learning (MARL) algorithms face two main difficulties: the curse of dimensionality, and environment non-stationarity due to the independent learning processes carried out by the agents concurrently. In this paper we formalize and prove the convergence of a Distributed Round Robin Q-learning (D-RR-QL) algorithm for cooperative systems. The computational complexity of this algorithm increases linearly with the number of agents. Moreover, it eliminates environment non sta tionarity by carrying a round-robin scheduling of the action selection and execution. That this learning scheme allows the implementation of Modular State-Action Vetoes (MSAV) in cooperative multi-agent systems, which speeds up learning convergence in over-constrained systems by vetoing state-action pairs which lead to undesired termination states (UTS) in the relevant state-action subspace. Each agent’s local state-action value function learning is an independent process, including the MSAV policies. Coordination of locally optimal policies to obtain the global optimal joint policy is achieved by a greedy selection procedure using message passing. We show that D-RR-QL improves over state-of-the-art approaches, such as Distributed Q-Learning, Team Q-Learning and Coordinated Reinforcement Learning in a paradigmatic Linked Multi-Component Robotic System (L-MCRS) control problem: the hose transportation task. L-MCRS are over-constrained systems with many UTS induced by the interaction of the passive linking element and the active mobile robots.