Optimal Policy Research Articles

Is Pessimism Provably Efficient for Offline Reinforcement Learning?

We study offline reinforcement learning (RL), which aims to learn an optimal policy based on a data set collected a priori. Because of the lack of further interactions with the environment, offline RL suffers from the insufficient coverage of the data set, which eludes most existing theoretical analysis. In this paper, we propose a pessimistic variant of the value iteration algorithm (PEVI), which incorporates an uncertainty quantifier as the penalty function. Such a penalty function simply flips the sign of the bonus function for promoting exploration in online RL, which makes it easily implementable and compatible with general function approximators. Without assuming the sufficient coverage of the data set (e.g., finite concentratability coefficients or uniformly lower-bounded densities of visitation measures), we establish a data-dependent upper bound on the suboptimality of PEVI for general Markov decision processes (MDPs). When specialized to linear MDPs, it matches the information-theoretic lower bound up to multiplicative factors of the dimension and horizon. In other words, pessimism is not only provably efficient but also, minimax optimal. In particular, given the data set, the learned policy serves as the “best effort” among all policies as no other policies can do better. Our theoretical analysis identifies the critical role of pessimism in eliminating a notion of spurious correlation, which arises from the “irrelevant” trajectories that are less covered by the data set and not informative for the optimal policy. Funding: Z. Yang acknowledges the Simons Institute (Theory of Reinforcement Learning). Z. Wang acknowledges the National Science Foundation [Awards 2048075, 2008827, 2015568, and 1934931], the Simons Institute (Theory of Reinforcement Learning), Amazon, J. P. Morgan, and Two bSigma.

On the design of an optimal immigration policy

AbstractThis paper constructs a model for the study of optimal immigration from the perspective of natives. They have preferences over redistributive transfers, a public good subject to congestion, and over the level of family‐based migration. Border enforcement is costly. The model derives conditions for an optimal immigration policy, which balances skill‐based against family‐based migration, equalizes the fiscal opportunity cost of unauthorized immigration with the marginal cost of enforcement, and balances the effects on transfers against the implied congestion effects. A parameterized version of the model studies the 1994–2008 period and finds legislation projects on immigration consistent with the model's predictions.

Robustness meets co-jumps: optimal consumption and portfolio choice with derivatives

In this paper, we study a robust, dynamic, continuous-time optimal consumption and portfolio allocation problem for investors with recursive preferences who have access to both stock and derivatives markets. We assume the stock price process follows a stochastic volatility model, with instantaneous precision as the unique state variable, allowing for discontinuities in all the dynamics. We obtain a closed-form approximate solution up to a system of ODEs to the optimization problem for a non-unitary value of the elasticity of intertemporal substitution of consumption, being able to derive an exact solution as a particular case. Our theoretical findings show that the optimal policies are remarkably affected by the ambiguity-aversion parameters to diffusive and jump risks. A detailed numerical analysis confirms the effectiveness of our theoretical results on real data. Finally, we prove that investors who do not believe in ambiguity may suffer considerable wealth losses.

Exploring the Impact of Institutional Innovation in Free Trade Zones and Free Trade Ports on Cultural Exchange and Soft Power Construction

This paper explores how institutional innovations in Free Trade Zones (FTZ) and Free Trade Ports (FTP) promote cultural exchange and soft power construction. Through a systematic literature review and policy analysis, it is found that institutional innovations in FTZs and FTPs, including relaxed restrictions on foreign investment in the cultural industry, streamlined approval processes for cultural exchange activities, the establishment of diverse cultural exchange platforms, and optimization of cultural talent introduction policies, effectively enhance the depth and breadth of international cultural exchange. These efforts have significantly improved China’s cultural soft power. A case study of the Hainan Free Trade Port demonstrates how proactive cultural policy innovations have led China to achieve notable success in international cultural exchanges, offering new ideas for global soft power competition. This study provides reference recommendations for future policy formulation and academic research, especially in optimizing cultural policies in FTZs and FTPs to further enhance cultural soft power.

A scoping review of the design and characteristics of e-bike financial incentives

ABSTRACT E-bikes are recognised as a sustainable mode of transportation with an unmet potential for widespread adoption. However, despite a decade of global implementation, research gaps persist regarding the design and characteristics of incentive programmes for e-bikes and their effectiveness. This review examines different design elements of implemented financial incentive programmes for e-bike uptake in OECD countries. The findings reveal three main components common to these schemes: (1) target cohort, with the majority of programmes focusing on the local population; (2) eligible e-bike types, with regular e-bikes being the most frequently chosen; and (3) financial incentive structures aimed at maximising uptake among the target cohort, with post-purchase rebates being the most prevalent. Another significant aspect identified was the allocation process, predominantly following a “first come, first served” structure. Very few (n = 4) studies included in this review assessed the effectiveness of financial incentives to purchase an e-bike. The variety of designs in these schemes, coupled with a lack of effectiveness assessments and limited evidence, highlights challenges in determining optimal transport policies. In addition to highlighting the knowledge and research gaps, this review synthesises global insights on design elements of financial incentive schemes to boost e-bike uptake, providing a guide of available components of such schemes for programme administrators in designing executive programmes.

Mergers and Licensing With Horizontal Differentiation

ABSTRACTWe consider a research laboratory that owns a patented process innovation and two firms producing differentiating goods in a Bertrand setting. The laboratory considers the possibility to license the innovation as an outsider patentee or to merge with one of the firms in the industry, becoming an incumbent patentee. Licensing takes place through observable two‐part tariff contracts. We show that the merger is profitable only for small innovations and increases social welfare for both small and large innovations. Even though we allow the royalty to be higher than the size of the innovation, and opposite to the result in a Cournot setting, we find a region where the merger is both profitable and welfare improving. This occurs only for small innovations and sufficiently differentiated goods. The same result arises for consumer surplus which allows us to derive the optimal merger policy: compared with Cournot competition, a Bertrand setting calls for a more lenient merger policy.

An approach for regime-switching stochastic control problems with memory and terminal conditions

In this research article, we focus on a stochastic optimal control problem with two types of terminal constraints. These specific conditions provide real-valued and stochastic Lagrange multipliers. Our model evolves according to a Markov regime-switching jump diffusion model with memory. In this context, the memory is represented by a Stochastic Differential Delay Equation. We present two theorems for each constraint within the general formulation of stochastic optimal control theory in a Lagrangian environment. We approach to this task from a theoretical perspective and provide mild technical assumptions, which make our theorems applicable for a broad class of stochastic control problems as well as for a wide range of disciplines such as engineering, biology, operations research, medicine, computer science and economics. In this work, we apply Stochastic Maximum Principle to demonstrate an optimal dividend policy corresponding to a time-delayed wealth process of a company. Moreover, we determine the real-valued Lagrange multiplier of this control problem explicitly.

Optimization of High-Frequency Trading Strategies Using Deep Reinforcement Learning

This study presents a new method for optimising high-risk trading (HFT) strategies using deep learning (DRL). We propose a multi-time DRL framework integrating advanced neural network architectures with sophisticated business data processing techniques. The framework employs a combination of convolutional neural networks for manual order analysis, short-term memory networks for time series processing, and a multi-head listening mechanism for body fusion. We formulate the HFT problem based on Markov Decision Processes and use the Proximal Policy Optimization algorithm for training. The model is evaluated using tick-by-tick data from the NASDAQ exchange, including ten liquid stocks in 6 months. The experimental results show the superiority of our method, achieving a Sharpe ratio of 3.42, outperforming the learning model and machine learning based on benchmarks up to 33%. The proposed strategy has demonstrated strong performance across a wide range of regulatory markets and has shown potential for strategic objectives. Sensitivity analysis confirms the model's stability across a range of hyperparameters. Our findings suggest that the DRL-based approach can improve HFT performance and provide better market adaptation and risk management. This research leads to the continuous evolution of algorithmic trading strategies and shows the potential of AI-driven approaches in financial markets.

COPSA: a computation offloading strategy based on PPO algorithm and self-attention mechanism in MEC-empowered smart factories

With the dawn of Industry 5.0 upon us, the smart factory emerges as a pivotal element, playing a crucial role in the realm of intelligent manufacturing. Meanwhile, mobile edge computing is proposed to alleviate the computational burden presented by substantial workloads in smart factories. Nonetheless, it is very challenging to effectively incorporate edge computing resources to improve the efficiency of resource deployment in smart factories. Accordingly, we devise a novel approach based on Proximal Policy Optimization algorithm with the Self-Attention Mechanism to implement computing resource allocation in MEC-Empowered Smart Factories. More specifically, the self-attention mechanism is incorporated to enable dynamic focus on state information, accelerates convergence and facilitates global control. A great number of experiments conducted on both simulated and real datasets have verified the superiority of our proposed approach compared to the state-of-the-art baselines.

Simulation and Optimization of Automated Guided Vehicle Charging Strategy for U-Shaped Automated Container Terminal Based on Improved Proximal Policy Optimization

As the decarbonization strategies of automated container terminals (ACTs) continue to advance, electrically powered Battery-Automated Guided Vehicles (B-AGVs) are being widely adopted in ACTs. The U-shaped ACT, as a novel layout, faces higher AGV energy consumption due to its deep yard characteristics. A key issue is how to adopt charging strategies suited to varying conditions to reduce the operational capacity loss caused by charging. This paper proposes a simulation-based optimization method for AGV charging strategies in U-shaped ACTs based on an improved Proximal Policy Optimization (PPO) algorithm. Firstly, Gated Recurrent Unit (GRU) structures are incorporated into the PPO to capture temporal correlations in state information. To effectively limit policy update magnitudes in the PPO, we improve the clipping function. Secondly, a simulation model is established by mimicking the operational process of the U-shaped ACTs. Lastly, iterative training of the proposed method is conducted based on the simulation model. The experimental results indicate that the proposed method converges faster than standard PPO and Deep Q-network (DQN). When comparing the proposed method-based charging threshold with a fixed charging threshold strategy across six different scenarios with varying charging rates, the proposed charging strategy demonstrates better adaptability to terminal condition variations in two-thirds of the scenarios.

Deephive: A Reinforcement Learning Approach for Automated Discovery of Swarm-Based Optimization Policies

We present an approach for designing swarm-based optimizers for the global optimization of expensive black-box functions. In the proposed approach, the problem of finding efficient optimizers is framed as a reinforcement learning problem, where the goal is to find optimization policies that require a few function evaluations to converge to the global optimum. The state of each particle within the swarm is defined as its current position and function value within a design space, and the particles learn to take favorable actions that maximize the reward, which is based on the final value of the objective function. The proposed approach is tested on 50 benchmark optimization functions and compared to the performance of other global optimization strategies. Furthermore, the generalization capabilities of the trained particles on the four categories of optimization benchmark functions are investigated. The results show superior performance compared to the other optimizers, desired scaling when the dimension of the functions is varied, and acceptable performance even when applied to unseen functions. On a broader scale, the results show promise for the rapid development of domain-specific optimizers.

NHPP Software Reliability Model with Rayleigh Fault Detection Rate and Optimal Release Time for Operating Environment Uncertainty

Software is used in diverse settings and depends on development and testing environments. Software development should improve the reliability, quality, cost, and stability of software, making the software testing period crucial. We proposed a software reliability model (SRM) that considers the uncertainty of software environments and the fault detection rate function as a Rayleigh distribution, with an explicit mean value function solution in the model. The goodness-of-fit of the proposed model relative to several existing nonhomogeneous Poisson process (NHPP) SRMs is presented based on three software application failure datasets. Further, a cost model is also presented that addresses the error-removal risk level and required time. The optimal testing release policy for minimizing the expected total cost (ETC) is also determined for NHPP SRMs. The impact of the software environment is studied by varying it, and the optimal release times and minimum ETCs are compared. The goodness-of-fit comparison confirmed that the proposed model has more accurate prediction values than other models. Further, whereas the existing models applied to the cost model do not change after a certain operation period, the proposed model yields changes in release time even for long operating periods.

Joint Learning of Volume Scheduling and Order Placement Policies for Optimal Order Execution

Order execution is an extremely important problem in the financial domain, and recently, more and more researchers have tried to employ reinforcement learning (RL) techniques to solve this challenging problem. There are a lot of difficulties for conventional RL methods to tackle the order execution problem, such as the large action space including price and quantity, and the long-horizon property. As naturally order execution is composed of a low-frequency volume scheduling stage and a high-frequency order placement stage, most existing RL-based order execution methods treat these stages as two distinct tasks and offer a partial solution by addressing either one individually. However, the current literature fails to model the non-negligible mutual influence between these two tasks, leading to impractical order execution solutions. To address these limitations, we propose a novel automatic order execution approach based on the hierarchical RL framework (OEHRL), which jointly learns the policies for volume scheduling and order placement. OEHRL first extracts the state embeddings at both the macro and micro levels with a sequential variational auto-encoder model. Based on the effective embeddings, OEHRL generates a hindsight expert dataset, which is used to train a hierarchical order execution policy. In the hierarchical structure, the high-level policy is in charge of the target volume and the low-level learns to determine the prices for a series of the allocated sub-orders from the high level. These two levels collaborate seamlessly and contribute to the optimal order execution policy. Extensive experiment results on 200 stocks across the US and China A-share markets validate the effectiveness of the proposed approach.

Evaluation of green development of energy consumption in the Pearl River Delta urban agglomeration based on radial basis function neural network

AbstractThe study utilizes the RBF neural network model to evaluate the green development of the Pearl River Delta urban agglomeration, revealing a consistent increase in development levels from 2005 to 2020 despite notable inter‐city variations. The model adeptly handles complex interactions within urban green development, offering insights into comprehensive impacts and aiding in policy evaluation and optimization. It provides urban planners and decision‐makers with data‐driven support, enhancing the scientific basis and stability of policy‐making.

UAVs Revolutionizing Efficiency: Integrating DRL and Random Walrus for Enhanced Energy and Spectral Resource Management

The improvement of energy efficiency and spectral efficiency in networks is achieved by seamlessly integrating energy harvesting, cognitive radio technologies, and Non-Orthogonal Multiple Access (NOMA) techniques. These complementary strategies optimize resource usage and address challenges related to energy consumption. Additionally, the adaptability and versatility of Unmanned Aerial Vehicles (UAVs) offer innovative solutions for enhancing coverage performance, thereby improving connectivity, efficiency, and reliability. We introduce a novel approach called the Deep Reinforcement Learning-Random Walrus (DRL-RW) algorithm, which combines Deep Reinforcement Learning (DRL) with the Random Walrus optimization (RWO) technique for efficient spectrum resource allocation and energy harvesting management in dynamic environments. The DRL-RW algorithm enables UAVs to learn optimal spectrum-sharing strategies and energy harvesting policies, while the RWO enhances the algorithm's adaptability and speed in exploring diverse solutions. Simulation results demonstrate the effectiveness of the DRL-RW algorithm, showing significant improvements in several performance metrics, including reduced energy consumption, enhanced computation time, improved convergence, increased signal-to-noise ratio, higher throughput, extended network lifetime, and increased harvested energy. These findings underscore the efficacy of the DRL-RW approach in addressing challenges associated with energy management in cognitive radio networks. The integration of UAVs, NOMA networks, and this novel algorithm represents a promising direction for developing energy-efficient communication systems.

On Optimal Scheduling

We consider a decision-maker sequentially choosing among alternatives when periodic payoffs depend on both chosen and unchosen alternatives in that period. We show that when flow payoffs are the sum or product of payoffs from chosen and unchosen alternatives, the optimal policy is an index policy. We characterize key properties of the optimal dynamics and present an algorithm for computing the indices explicitly. Furthermore, we use the results to generalize Weitzman’s (1979) classic “Pandora’s boxes” problem to allow for complementarities. We illustrate the framework’s usefulness through applications, including decision problems with disappearing alternatives, repeated bargaining, dynamic supervision, and dynamic occupational choice. (JEL C78, D82, M11)

An Optimal Two-Dimensional Maintenance Policy for Self-Service Systems with Multi-Task Demands and Subject to Competing Sudden and Deterioration-Induced Failures

Self-service systems, such as electric vehicle charging piles (EVCPs), are typically deployed without on-site personnel. While frequent maintenance ensures high service revenue, it also leads to significant maintenance setup costs. Therefore, balancing service revenue and maintenance costs is essential for profit maximization. In this paper, we develop a maintenance policy optimization framework to maximize the profit rate of a fleet of self-service systems. First, we propose a maintenance policy that ensures sufficient functional systems while preventing high corrective maintenance costs. Next, we model the fleet's state transition process and its profit rate by characterizing two unique failure-induced demand-and-system interactions: demand switching and stepwise demand arrival rates, where the demands involve multiple tasks and systems are subject to multiple failure modes with non-constant occurrence rates. We develop a Tabu-search algorithm with random exploration to optimize the maintenance policy. Building on this, we investigate the impacts of key model parameters through a case study of thirteen EVCPs in Hong Kong and draw implications for profit maximization.

Optimal refund policy design for ship berthing appointment mechanism

Optimal refund policy design for ship berthing appointment mechanism

Dynamic policy in the presence of social norms

AbstractIndividual actions can depend on prevailing social norms. We investigate how optimal policy to promote pro‐social action should exploit the underlying social dynamics. We develop a dynamic model of prosocial action in which conformist consumers repeatedly choose whether to engage in some prosocial activity. Whereas individual behavior is not observed, the overall participation rate in the previous period is common knowledge. We demonstrate how conformity can lead to multiple steady states and how their selection depends on starting conditions and discount factors. We further show that the optimal subsidy path can be non‐monotonic and can decrease before reaching the steady state‐level. Our model thus provides a rationale for introductory subsidies to promote environmentally friendly behavior from a behavioral perspective.

Incorporating a machine learning approach for commodity price prediction to generate a cut-off grade optimisation policy under grade uncertainty

ABSTRACT The economic importance of cut-off grade (COG) is that it differentiates ore from waste. Therefore, when COG is optimised, the economic value of a mining operation can be maximised. COG is optimised by considering several factors that include economic, geological, and operational parameters. Since uncertainty is inherent in some of these parameters, this paper used grade–tonnage distributions to incorporate grade uncertainty and a machine learning approach for commodity price prediction to account for price volatility when determining COGs. The paper demonstrates how uncertainty in these parameters can be incorporated in developing a dynamic COG policy over the life of mine.