Optimal Joint Policy Research Articles

Edge caching and computing of video chunks in multi-tier wireless networks

This paper proposes a video caching and transcoding strategy for delay-constrained content delivery in multi-tier wireless networks. Particularly for multimedia services whose content can be encoded into multiple quality versions and consists of multiple chunks, we present an approach of caching chunks of an identical file separately in different network layers with different qualities. Given this caching policy, a complete file can be retrieved by receiving its chunks from different network tiers; therefore, a video transcoding and delivery method is jointly designed with caching to maximize the expected content quality with a strict delay constraint. Motivated by that caching is proactively conducted in advance of many user requests, the joint optimization problem is decomposed into caching and transcoding subproblems. The optimal transcoding policy and its transcoded qualities at different network tiers are analytically found by linear programming for a given cache state. Using the closed-form transcoded qualities, the caching policy is optimized to maximize the expected content quality with respect to randomness of user locations and channel gains. Due to the difficulty of expectation, the caching policy of each network layer is independently obtained by water-filling first, and an alternating optimization method is used to find the joint optimal caching policy satisfying the cache size constraints. Simulation results show that the proposed scheme succeeds in delivering content of 5% to 10% higher expected quality to users than comparison schemes which do not use partitioning for caching. Simultaneously, our approach meets the delay requirement and effectively manages the trade-offs arising from the varying storage and computational capacities within hierarchies, as well as video diversity.

Agent-based modeling for an end-of-life power battery cross-regional recycling system and subregional policy analysis: A case study in China

Informal recycling of end-of-life electric vehicle batteries has caused severe environmental pollution and tax losses. Linear models and static strategies have difficulty capturing complex relationships and regional heterogeneity. Therefore, we aim to suggest an optimal subregional joint policy under fiscal constraints. This study uses agent-based modeling with geographic information systems to construct a cross-regional recycling system model that assesses the economic and environmental effects of policies. Based on actual data, the model considers four subsystems, multiple region types, two battery types, and six stakeholders. The results show that: (1) The optimal subregional joint policy can increase the average formal recycling rate from 28.44 % to 48.58 % and achieve an emission reduction rate of 19.81 %; (2) In underdeveloped regions, allocating 70–80 % of fiscal funds to supervision can achieve optimal economic and environmental benefits; and (3) For the local governments with few formal sectors, investing in a unit subsidy exceeding $1.4/kWh will diminish economic efficiency. Provincial governments in economically developed areas with formal business clusters should provide higher subsidies, while other regions should allocate more fiscal expenditure for supervision. This cross-regional recycling system model can improve solid waste management in countries with large areas and decentralized facilities.

Pricing and dynamic service policy for an imperfect production system: Extended Pontryagin’s maximum principle for interval control problems

Pontryagin’s maximum principle is one of the most important topics of control theory. It plays a crucial role in solving variational problems in an alternative way. In this paper, Pontryagin’s maximum principle is extended in order to solve interval valued control problems (IVCPs). To serve this purpose, firstly all the formal definitions (viz. IVCP, interval ranking and optimizer of IVCP etc.) are presumed. Further, the necessary and sufficient condition, i.e., Pontryagin’s principles are extended in interval environment in order to solve an IVCP. The second part of this work presents an application of the proposed extension of Pontryagin’s maximum principle for IVCPs. Then, an imperfect production inventory problem is considered by assuming flexibility of all the involved parameters in interval environment. The proposed production problem mainly addresses a joint optimal policy for pricing, dynamic service investment and reworked/salvaged under interval flexibility. The IVCP related to the proposed model is solved using extended Pontryagin’s maximum principle and the interval optimization problem corresponding to the model is solved using improved c-r optimization technique. Further, a numerical example is taken and solved using the sparrow search algorithm (SSA) in order to illustrate the proposed theoretical results and validate the proposed model. Finally, sensitivity analyses are carried out w. r. to various system parameters in order to evaluate some implications to the manager.

LJIR: Learning Joint-Action Intrinsic Reward in cooperative multi-agent reinforcement learning

Effective exploration is the key to achieving high returns for reinforcement learning. Agents must explore jointly in multi-agent systems to find the optimal joint policy. Due to the exploration problem and the shared reward, the policy-based multi-agent reinforcement learning algorithms face policy overfitting, which may lead to the joint policy falling into a local optimum. This paper introduces a novel general framework called Learning Joint-Action Intrinsic Reward (LJIR) for improving multi-agent reinforcement learners’ joint exploration ability and performance. LJIR observes agents’ state and joint actions to learn to construct an intrinsic reward online that can guide effective joint exploration. With the novel combination of Transformer and random network distillation, LJIR selects the novel states to give more intrinsic rewards, which help agents find the best joint actions. LJIR can dynamically adjust the weight of exploration and exploitation during training and keep the policy invariance finally. To ensure LJIR seamlessly adopts existing MARL algorithms, we also provide a flexible combination method for intrinsic and external rewards. Empirical results on the SMAC benchmark show that the proposed method achieves state-of-the-art performance in challenging tasks.

Joint optimization of condition‐based maintenance and condition‐based reallocation for a system with multiple degrading components

AbstractCondition‐based maintenance has been studied for managing the degradation of components in a system. However, components in different positions may undergo different workload, leading to imbalanced components degradation. Component reallocation is an effective method to balance the degradation of components in different positions with different degradation rates. Thus, this paper investigates the joint optimization of condition‐based maintenance and condition‐based reallocation for a non‐repairable system with multiple identical and functionally‐exchangeable components that undergo different continuous degradation processes due to different workload. We model this problem through Markov decision process, which is solved by a dynamic programming algorithm, and the objective is to provide the optimal maintenance and reallocation actions exactly for each system state, that is, the joint optimal policy, to minimize the discounted long‐run total cost. Especially, our joint policy is on component‐level and proved to be optimal. Finally, the numerical analysis shows the structural insights and effectiveness of the proposed joint policy, for which the sensitivity analysis on the degradation parameters, setup cost, reallocation cost, and downtime cost is performed. We also propose a new reallocation strategy and show the results and computation time for systems with different scales, which indicates that our model can handle systems with reasonable size.

Optimal Joint Inspection and Mission Abort Policies for Degenerative Systems

This article investigates the optimal joint inspection and mission abort policies for deteriorating systems which execute a mission continuously. The system degradation is modelled by gamma process and the information on the system state can only be obtained via inspections. To enhance the system survivability, the mission can be aborted and the rescue procedure will be initiated immediately. Different from previous research, the decision to abort or to continue the mission depends on the predictive reliability other than the fixed degradation level, which is proved to be more effective in cost saving. The inspection period and the mission abort threshold jointly form a two-dimensional policy. Under this policy, the explicit expressions for mission success probability and system survivability are derived analytically. The optimal joint policy is investigated to balance the tradeoff between mission success probability and system survivability. It aims to determine the inspection period and the reliability threshold simultaneously such that the expected total cost including mission failure cost, system loss cost and inspection cost is minimized. Some structural properties of the optimal policy are obtained theoretically. Finally, numerical examples are provided to demonstrate the model.

Joint condition-based maintenance and spare provisioning policy for a K-out-of-N system with failures during inspection intervals

Recent advances in dedicated sensors and surveillance technology have given rise to condition-based maintenance. In addition to the degradation condition of the system, maintenance schedules should take into account the availability of spare parts. By incorporating spare part order information, planners can make more precise arrangements for both maintenance and spare parts replenishment, leading to a reduction in operating cost. This paper investigates the joint optimization of condition-based maintenance and spare provisioning for a K-out-of-N system considering failures during inspection intervals. The degradation states of the system are revealed at periodic inspection epochs where planners have the opportunity to replace components and order spare parts. The resulting sequential decision problem is formulated as a Markov decision process that integrates the comprehensive information of the components and spare parts orders into decision making. We propose an optimal joint policy by using the value iteration algorithm and compare it with two separated policies that consider part of the information. Sensitivity experiments are conducted to analyze the impact of the parameters on the effect of the proposed policy with the comprehensive information. The results show that the comprehensive information is more important to the ordering decisions than the replacing decisions.

Modeling on Energy-Efficiency Computation Offloading Using Probabilistic Action Generating

Wireless-powered mobile-edge computing (MEC) emerges as a crucial component in the Internet of Things (IoTs). It can cope with the fundamental performance limitations of low-power networks, such as wireless sensor networks or mobile networks. Although computation offloading and resource allocation in MEC have been studied with different optimization objectives, performance optimization in larger-scale systems still needs to be further improved. More importantly, energy efficiency is also a key issue as well as computation offloading and resource allocation for wireless-powered MEC. In this article, we investigate the joint optimization of computation rate and energy consumption under limited resources, and propose an online offloading model to search for the asymptotically optimal offloading and resource allocation strategy. First, the joint optimization problem is modeled as a mixed integer programming (MIP) problem. Second, a deep reinforcement learning (DRL)-based method, energy efficiency computation offloading using probabilistic action generating (ECOPG), is designed to generate the joint optimization policy for computation offloading and resource allocation. Finally, to avoid the curse of dimensionality in large network scales, an action exploration mechanism based on probability is introduced to accelerate the convergence rate by targeted sampling and dynamic experience replay. The experimental results demonstrate that the proposed methods significantly outperform other DRL-based methods in energy consumption, and gain better computation rate and execution efficiency at the same time. With the expansion of the network scale, the improvements become more apparent.

Joint optimization of inspection-based and age-based preventive maintenance and spare ordering policies for single-unit systems

This paper presents a joint optimization policy of preventive maintenance (PM) and spare ordering for single-unit systems, which deteriorate subject to the delay-time concept with three deterioration stages. PM activities that combine a nonperiodic inspection scheme with age-replacement are implemented. When the system is detected to be in the minor defective stage by an inspection for the first time, place an order and shorten the inspection interval. If the system has deteriorated to a severe defective stage, it is either repaired imperfectly or replaced by a new spare. However, an immediate replacement is required once the system fails, the maximal number of imperfect maintenance (IPM) is satisfied or its age reaches to a pre-specified threshold. In consideration of the spare's availability as needed, there are three types of decisions, i.e., an immediate or a delayed replacement by a regular ordered spare, an immediate replacement by an expedited ordered spare with a relative higher cost. Then, some mutually independent and exclusive renewal events at the end of a renewal cycle are discussed, and the optimization model of such a joint policy is further developed by minimizing the long-run expected cost rate to find the optimal inspection and age-replacement intervals, and the maximum number of IPM. A Monte-Carlo based integration method is also designed to solve the proposed model. Finally, a numerical example is given to illustrate the proposed joint optimization policy and the performance of the Monte-Carlo based integration method.

Horrible trade-offs in a pandemic: Poverty, fiscal space, policy, and welfare

We analyze how poverty and a country’s fiscal space impact policy and welfare in times of a pandemic. We introduce a subsistence level of consumption into a tractable heterogeneous agent framework, and use this framework to characterize optimal joint policies of a lockdown and transfer payments. In our model, a more stringent lockdown helps fighting the pandemic, but it also deepens the recession, which implies that poorer parts of society find it harder to subsist. This reduces their compliance with the lockdown, and may cause deprivation of the very poor, giving rise to an excruciating trade-off between saving lives from the pandemic and from deprivation. Transfer payments help mitigate this trade-off. We show that, ceteris paribus, the optimal lockdown is stricter in richer countries and the aggregate death burden and welfare losses smaller. We then consider a government borrowing constraint and show that limited fiscal space lowers the optimal lockdown and welfare, and increases the aggregate death burden during the pandemic. This is particularly true in societies where a larger fraction of the population is in poverty. We discuss evidence from the literature and provide reduced-form regressions that support the relevance of our main mechanisms. We finally discuss distributional consequences and the political economy of fighting a pandemic.

Pricing and Channel Coordination in Online-to-Offline Supply Chain Considering Corporate Environmental Responsibility and Lateral Inventory Transshipment

In this study, we investigate pricing policy and coordination conditions in an online-to-offline supply chain considering corporate environmental responsibility and lateral inventory transshipment. First, we provide demand functions to capture effects of price, corporate environmental responsibility level, and preference degree of the consumer to online channel. Then, we build profit functions and develop three joint pricing and corporate environmental responsibility-level decision models for centralized decision (Scenario CD), retailer Stackelberg game (Scenario RS), and manufacturer Stackelberg game (Scenario MS). Furthermore, we determine the optimal decision policies by solving developed models, and conduct sensitivity analysis of significant factors. Finally, we use a revenue-sharing contract to realize supply chain coordination and find coordination conditions for Scenario RS and MS, and further show the impacts of revenue-sharing rate and investment cost sensitivity on the conditions using numerical studies. We find that optimal joint decision policies can be affected by significant factors to a varying degree. In certain conditions, the revenue-sharing contract can coordinate online-to-offline supply chains considering corporate environmental responsibility and lateral inventory transshipment. Our study proposes a new decision problem, constructs new joint decision models, determines new optimal joint policies, conducts new coordination analysis, and thus contributes to the research on supply chain operations considering corporate environmental responsibility and lateral inventory transshipment.

Influence-Based Policy Abstraction for Weakly-Coupled Dec-POMDPs

Decentralized POMDPs are powerful theoretical models for coordinating agents’ decisions in uncertain environments, but the generally-intractable complexity of optimal joint policy construction presents a significant obstacle in applying Dec-POMDPs to problems where many agents face many policy choices. Here, we argue that when most agent choices are independent of other agents’ choices, much of this complexity can be avoided: instead of coordinating full policies, agents need only coordinate policy abstractions that explicitly convey the essential interaction influences. To this end, we develop a novel framework for influence-based policy abstraction for weakly-coupled transition-dependent Dec-POMDP problems that subsumes several existing approaches. In addition to formally characterizing the space of transition-dependent influences, we provide a method for computing optimal and approximately-optimal joint policies. We present an initial empirical analysis, over problems with commonly-studied flavors of transition-dependent influences, that demonstrates the potential computational benefits of influence-based abstraction over state-of-the-art optimal policy search methods.

Solving Common-Payoff Games with Approximate Policy Iteration

For artificially intelligent learning systems to have widespread applicability in real-world settings, it is important that they be able to operate decentrally. Unfortunately, decentralized control is difficult---computing even an epsilon-optimal joint policy is a NEXP complete problem. Nevertheless, a recently rediscovered insight---that a team of agents can coordinate via common knowledge---has given rise to algorithms capable of finding optimal joint policies in small common-payoff games. The Bayesian action decoder (BAD) leverages this insight and deep reinforcement learning to scale to games as large as two-player Hanabi. However, the approximations it uses to do so prevent it from discovering optimal joint policies even in games small enough to brute force optimal solutions. This work proposes CAPI, a novel algorithm which, like BAD, combines common knowledge with deep reinforcement learning. However, unlike BAD, CAPI prioritizes the propensity to discover optimal joint policies over scalability. While this choice precludes CAPI from scaling to games as large as Hanabi, empirical results demonstrate that, on the games to which CAPI does scale, it is capable of discovering optimal joint policies even when other modern multi-agent reinforcement learning algorithms are unable to do so.

Joint optimal pricing and inventory management policy and its sensitivity analysis for perishable products: Lost sale case

<p style='text-indent:20px;'>In the real world, the demand cannot be depicted exactly because of customer behavior cannot be forecasted without error. In this paper, we study the effect of the error of the estimated price-demand parameters by analyzing the sensitivity of the optimal joint pricing and ordering policy on the price-demand parameters based on a periodic-review, multi-period and lost sale inventory model for perishable products with constant quantity decay rate and price-sensitive demand. Firstly, we formulate the joint pricing and inventory control problem and find the optimal ordering quantity and the optimal price for deterministic price-demand function. The optimal solutions show that the retailer tends to set a lower price in early periods of each ordering cycle in order to reduce the inventory holding costs. Furthermore, the sensitivity of the optimal joint pricing and inventory control system with respect to the price-demand parameters is examined analytically and evaluated numerically. The sensitivity analysis reveals that compared to the optimal ordering quantity, the optimal prices are less sensitive in the demand-price parameters. Finally, according to the findings of the sensitivity analysis, a heuristic method of regulating the estimated demand-price parameters is employed to improve the average profit. <b>185</b> words.</p>

Joint optimization of condition-based and age-based replacement policy and inventory policy for a two-unit series system

This paper proposes a joint optimization problem of the condition-based and age-based replacement policy and spare parts inventory policy for a series system with two non-identical units. Unit 1 is subject to soft failure and condition-based maintenance actions are adopted. Unit 2 is subject to hard failure and maintenance actions are performed based its usage. Besides the replacement policies, the (0,1) inventory policy is used to order the spare part of unit 1 and the (s,S) inventory policy is employed for unit 2. Unlike the previous studies where the maintenance policy and inventory policy are optimized separately, the replacement and order activities are simultaneously determined at decision epochs in this paper. Moreover, the system state transition probabilities, the expected maintenance costs and expected sojourn times would be derived by utilizing the semi-Markov decision process. The optimal preventive replacement thresholds of both units and the order level of unit 2 are simultaneously determined to minimize the system average maintenance cost per unit time. An illustrative example of a two-unit wind turbine system is used to demonstrate the priority of the proposed joint optimization policy.

On optimal joint replacement policy for deteriorating series systems

In this paper, a general repair-replacement model for a k-component series system is proposed. Components of the system may break down randomly and can be repaired immediately. Due to the imperfect repair, the successive operating times decrease stochastically while the consecutive repair times increase stochastically. A joint replacement policy is adopted based on the failure number of each component. The explicit expression of the long-run average cost rate is derived. Furthermore, we investigate the optimal policies of series system and single-component system respectively, the explicit relationship between them is determined in the form of a theorem for the first time. It shows that the optimal replacement policy of a component in a series system is not less than that when it forms a single-component system, and the former is non-decreasing with the number of components. Some other analytical results are also obtained. Based on these useful results, an efficient algorithm is devised to search the optimal joint policy which leads to an elegant, mathematical appealing analysis that is easy to use in practice. Finally, illustrative examples are provided to demonstrate the proposed model and the method.

Reinforcement Learning With Task Decomposition for Cooperative Multiagent Systems.

In this article, we study cooperative multiagent systems (MASs) with multiple tasks by using reinforcement learning (RL)-based algorithms. The target for a single-agent RL system is represented by its scalar reward signals. However, for an MAS with multiple cooperative tasks, the holistic reward signal consists of multiple parts to represent the tasks, which makes the problem complicated. Existing multiagent RL algorithms search distributed policies with holistic reward signals directly, making it difficult to obtain an optimal policy for each task. This article provides efficient learning-based algorithms such that each agent can learn a joint optimal policy to accomplish these multiple tasks cooperatively with other agents. The main idea of the algorithms is to decompose the holistic reward signal for each agent into multiple parts according to the subtasks, and then the proposed algorithms learn multiple value functions with the decomposed reward signals and update the policy with the sum of distributed value functions. In addition, this article presents a theoretical analysis of the proposed approach. Finally, the simulation results for both discrete decision-making and continuous control problems have demonstrated the effectiveness of the proposed algorithms.

Joint production control and maintenance policy for a serial system with quality deterioration and stochastic demand

We consider a serial production system subject to stochastic deterioration with usage. Machines in the system will suffer quality failures and produce non-conforming items when they reach the out-of-control state. Non-conforming items transferred into the downstream machines may accelerate their deterioration process. Inspection is performed on the system to detect the state of all machines before starting each production lot. The predictive maintenance policy based on the predictive failure probability of each machine and the production control policy based on the target service level are proposed to meet the dynamic stochastic demand every period. Moreover, an opportunistic maintenance policy is considered to address the stochastic dependency and economic dependency in the system. An integrated model of production, quality and maintenance is developed to minimize the total cost. The optimal joint policy of the model is obtained by a simulation-based optimization method. Numerical experiments validate the advantages of our model in terms of cost savings and service improvement.

Adaptive Learning: A New Decentralized Reinforcement Learning Approach for Cooperative Multiagent Systems

Multiagent systems (MASs) have received extensive attention in a variety of domains, such as robotics and distributed control. This paper focuses on how independent learners (ILs, structures used in decentralized reinforcement learning) decide on their individual behaviors to achieve coherent joint behavior. To date, Reinforcement learning(RL) approaches for ILs have not guaranteed convergence to the optimal joint policy in scenarios in which communication is difficult. Especially in a decentralized algorithm, the proportion of credit for a single agent’s action in a multiagent system is not distinguished, which can lead to miscoordination of joint actions. Therefore, it is highly significant to study the mechanisms of coordination between agents in MASs. Most previous coordination mechanisms have been carried out by modeling the communication mechanism and other agent policies. These methods are applicable only to a particular system, so such algorithms do not offer generalizability, especially when there are dozens or more agents. Therefore, this paper mainly focuses on the MAS contains more than a dozen agents. By combining the method of parallel computation, the experimental environment is closer to the application scene. By studying the paradigm of centralized training and decentralized execution(CTDE), a multi-agent reinforcement learning algorithm for implicit coordination based on TD error is proposed. The new algorithm can dynamically adjust the learning rate by deeply analyzing the dissonance problem in the matrix game and combining it with a multiagent environment. By adjusting the dynamic learning rate between agents, coordination of the agents’ strategies can be achieved. Experimental results show that the proposed algorithm can effectively improve the coordination ability of a MAS. Moreover, the variance of the training results is more stable than that of the hysteretic Q learning(HQL) algorithm. Hence, the problem of miscoordination in a MAS can be avoided to some extent without additional communication. Our work provides a new way to solve the miscoordination problem for reinforcement learning algorithms in the scale of dozens or more number of agents. As a new IL structure algorithm, our results should be extended and further studied.

Horrible Trade-offs in a Pandemic: Lockdowns, Transfers, Fiscal Space, and Compliance

In this paper, we develop a heterogeneous agent general equilibrium framework to analyze optimal joint policies of a lockdown and transfer payments in times of a pandemic. In our model, the effectiveness of a lockdown in mitigating the pandemic depends on endogenous compliance. A more stringent lockdown deepens the recession which implies that poorer parts of society find it harder to subsist. This reduces their compliance with the lockdown, and may cause deprivation of the very poor, giving rise to an excruciating trade-off between saving lives from the pandemic and from deprivation. Lump-sum transfers help mitigate this trade-off. We identify and discuss key trade-offs involved and provide comparative statics for optimal policy. We show that, ceteris paribus, the optimal lockdown is stricter for more severe pandemics and in richer countries. We then consider a government borrowing constraint and show that limited fiscal space lowers the optimal lockdown and welfare, and increases the aggregate death burden during the pandemic. We finally discuss distributional consequences and the political economy of fighting a pandemic.