Optimal Policy Research Articles

Interpretable deep learning models for the inference and classification of LHC data

The Shower Deconstruction methodology is pivotal in distinguishing signal and background jets, leveraging the detailed information from perturbative parton showers. Rooted in the Neyman-Pearson lemma, this method is theoretically designed to differentiate between signal and background processes optimally in high-energy physics experiments. A key challenge, however, arises from the combinatorial growth associated with increasing jet constituents, which hampers its computational feasibility. We address this by demonstrating that the likelihood derived from comparing the most probable signal and background shower histories is equally effective for discrimination as the conventional approach of summing over all potential histories in top quark versus Quantum Chromodynamics (QCD) scenarios. We propose a novel approach by conceptualising the identification of the most probable shower history as a Markov Decision Process (MDP). Utilising a sophisticated modular point-transformer architecture, our method efficiently learns the optimal policy for this task. The developed neural agent excels in constructing the most likely shower history and demonstrates robust generalisation capabilities on unencountered test data. Remarkably, our approach mitigates the complexity inherent in the inference process, achieving a linear scaling relationship with the number of jet constituents. This offers a computationally viable and theoretically sound method for signal-background differentiation, paving the way for more effective data analysis in particle physics.

Trustworthy autonomous driving via defense-aware robust reinforcement learning against worst-case observational perturbations

Despite the substantial advancements in reinforcement learning (RL) in recent years, ensuring trustworthiness remains a formidable challenge when applying this technology to safety-critical autonomous driving domains. One pivotal bottleneck is that well-trained driving policy models may be particularly vulnerable to observational perturbations or perceptual uncertainties, potentially leading to severe failures. In view of this, we present a novel defense-aware robust RL approach tailored for ensuring the robustness and safety of autonomous vehicles in the face of worst-case attacks on observations. The proposed paradigm primarily comprises two crucial modules: an adversarial attacker and a robust defender. Specifically, the adversarial attacker is devised to approximate the worst-case observational perturbations that attempt to induce safety violations (e.g., collisions) in the RL-driven autonomous vehicle. Additionally, the robust defender is developed to facilitate the safe RL agent to learn robust optimal policies that maximize the return while constraining the policy and cost perturbed by the adversarial attacker within specified bounds. Finally, the proposed technique is assessed across three distinct traffic scenarios: highway, on-ramp, and intersection. The simulation and experimental results indicate that our scheme enables the agent to execute trustworthy driving policies, even in the presence of the worst-case observational perturbations.

Reinforcement learning-driven proximal policy optimization-based voltage control for PV and WT integrated power system

The depletion of conventional fuel reserves and the high carbon emissions have alarmed the energy sector drivers. The integration of renewable energy resources (RERs) towards power system transformation is an obvious approach for a new-zero future. However, this integration also brings challenges in terms of voltage stability and power losses due to the variable output of RERs. To address these challenges, this work proposes a novel approach utilizing photovoltaic (PV) inverters and static var compensators (SVCs) for reactive power control in power distribution networks (PDNs). It enhances voltage stability and minimizes power losses. The proposed study deploys a proximal policy optimization (PPO) algorithm for real-time communication and control between reactive power devices. Performance evaluation was made on an IEEE-33 Bus system to demonstrate the effectiveness of the proposed scheme in integrating RER-based distributed generators (DGs). The proposed system achieved 68 % voltage control, keeping the voltage within a certain range of ±5 % while minimizing power losses. The proposed system also reduced the voltage out-of-control ratio to 0.044, which indicates minimum voltage deviation from the standard value. The proposed study provides a promising solution for controlling the voltage of DGs integrated PDN, which can potentially enhance the efficiency and reliability of the power system.

Macroprudential policy with earnings-based borrowing constraints

A large literature has studied optimal regulatory policy in macroeconomic models with asset-based collateral constraints. A common conclusion is that agents ‘over-borrow’ and optimal policy reduces debt positions through taxes. The reason is that agents do not internalize the effects of their choices on asset prices. However, recent empirical evidence shows that firms primarily borrow against their earnings rather than their assets. This paper studies optimal macroprudential policy with earnings-based borrowing constraints, both in closed and open economies. We reach the opposite conclusion to the previous literature. Agents ‘over-save’ (and ‘under-borrow’) relative to the social optimum, as they do not internalize changes in wages, which in turn affect firms’ earnings. A numerical model exercise demonstrates that incorrectly rolling out a tax policy derived under the assumption of asset-based constraints in an economy where firms actually borrow based on earnings leads to a consumption equivalent welfare loss of up to 2.55%. Optimal macroprudential policy thus critically depends on the specific form of financial constraints.

Stochastic credibility and optimal monetary policy

AbstractIn this paper, I study optimal monetary policy in a simple New Keynesian model with loose commitment and stochastic credibility. The loose commitment framework breaks the commitment‐discretion dichotomy in optimal monetary policy problems and allows for intermediate cases between commitment and discretion. Under this framework, the central bank is imperfectly credible, meaning that it occasionally reneges on promised policy plans. I contribute to the literature by introducing time‐variation in the central bank's credibility. I model credibility as an exogenous two‐state Markov chain and use a recursive saddlepoint functional equation to solve the model. I find that greater persistence and frequency of credibility losses increase welfare losses.

Monetary Policy, Employment Shortfalls, and the Natural Rate Hypothesis

Activity shortfalls are more costly than strong activity. I consider optimal monetary policy under discretion with an asymmetric (activity shortfalls) loss function. The model satisfies the natural rate hypothesis. The asymmetric loss function and resulting optimal monetary policy exacerbates shortfalls in activity. The additional frequency of activity shortfalls arises from the adjustment of expectations implied by the natural rate hypothesis. The shortfalls asymmetry leads to an inflationary bias, similar to results in the time-consistency literature. Mandating a central bank objective with greater symmetry than the social loss function improves outcomes. Greater symmetry lowers the magnitude of activity shortfalls. Greater symmetry also reduces inflation bias. The model also implies that an optimal monetary policy does not accommodate fluctuations from aggregate demand shocks, as is standard in such models. As a result, the analysis implies that monetary accommodation of strength in economic activity likely requires justifications other than asymmetric costs of shortfalls.

Deep reinforcement learning for maintenance optimization of a scrap-based steel production line

This paper presents a Deep Reinforcement Learning (DRL)-based optimization approach for determining the optimal inspection and maintenance planning of a scrap-based steel production line. The DRL-based optimization maintenance recommends the adequate time for inspections and maintenance activities based on the monitoring conditions of the production line, such as machine productivity, buffer level, and production demand. Some practical aspects of the system, such as such uncertainty of the maintenance duration and the variable production rate of the machines, were considered. A scrap-based steel production line was modeled as a multi-component system considering components dependencies. A simulation model was developed to simulate the dynamics of the system and assist with the development of the DRL maintenance approach. The proposed DRL-based maintenance is compared with traditional maintenance policies, such reactive maintenance, time-based maintenance, and condition-based maintenance. In addition, different DRL algorithms such as PPO (Proximal Policy Optimization), TRPO (Trust Region Policy Optimization), and DQN (Deep Q-Network) are investigated in the case-based scenario. The findings indicated the potential for significant financial savings. Therefore, the proposed maintenance approach demonstrates system adaptability and has the potential to be a powerful tool for industrial competitiveness.

Policy Optimization Over Submanifolds for Linearly Constrained Feedback Synthesis

Policy Optimization Over Submanifolds for Linearly Constrained Feedback Synthesis

Exact evaluation and optimization of integer-ratio policies for stochastic one-warehouse multiple-retailer inventory systems

This paper considers one-warehouse multiple-retailer inventory systems. The retailers are non-identical and face Poisson demand processes. We consider an integer-ratio policy in which each retailer orders in a fixed interval that is either an integer or integer-ratio multiple of the warehouse's reorder interval. Base-stock policy and virtual allocation are assumed. We show that cost evaluations at the retailers are completely separable, leading to a simple approach to develop an optimal integer-ratio policy for a two-level distribution system with heterogeneous retailers.

Adapting reservoir operations for optimal water management under varying climate and demand scenarios using metaheuristic algorithms

The operational rule curve is significant for proper reservoir operations and water resources management of the dam. The optimisation of the release policy is rather a complicated and stochastic problem. Various types of algorithms have been used to develop the release policy for the Klang Gate Dam (KGD), but alas, climate parameters associated with climate change, be it regional or globally, e.g., temperature factors, had not been considered then. In this study, the consideration of maximum and minimum temperature factors is given emphasis, while water demand is evaluated in the optimisation problem using simulation-optimisation approaches. The support vector regression simulation model (SVRS), and the multilayer perceptron simulation model (MLPS) were used to simulate the demand for the scenario cases. SVRS utilises the hyperplane theory to study the relationship between the input and output data, whereas MLPS employs the human neural system to weight and process the inputs via multiple layers of neurons into a targeted output. Four distinct types of simulation-optimisation combination model, including the SVRS-firefly algorithm (SVRS-FA), the SVRS-particle swarm optimisation (SVRS-PSO), the MLPS-firefly algorithm (MLPS-FA), and the MLPS-particle swarm optimisation (MLPS-PSO), were applied and investigated. The ultimate goal of this study was to minimise the water deficit. The PSO optimisation models overall have outperformed the FA optimisation models in all aspects. The periodic reliability of the MLPS-PSO was found to be the highest when the minimum temperature scenario was considered, and is 0.6491, with the least shortage period of 80 months. The MLPS-PSO is also the most resilient model in the same scenario, with a resilience value of 0.6750. Whereas the SVRS-PSO was the least vulnerable model in the minimum temperature scenario with the lowest shortage index of 0.0047 and a volumetric deficiency of about 2.44 MCM in total. The SVRS-PSO model showed excellent performance for high and low inflow conditions, while the MLPS-PSO model was better during medium inflows. In short, each of the models investigated has its particular field of advancement, but since the maximum temperatures sounded critical, additional research for the maximum temperature case could be pursued further in depth.

Sequential Label Enhancement.

Label distribution learning (LDL) is a novel machine learning paradigm for solving ambiguous tasks, where the degree to which each label describing the instance is ambiguous. However, obtaining the label distribution is high cost and the description degree is difficult to quantify. Most existing research works focus on designing an objective function to obtain the whole description degrees at once but seldom care about the sequentiality in the process of recovering the label distribution. In this article, we formulate the label distribution recovering task as a sequential decision process called sequential label enhancement (Seq_LE), which is more consistent with the process of annotating the label distribution in human brains. Specifically, the discrete label and its description degree are serially mapped by the reinforcement learning (RL) agent. Besides, we carefully design a joint reward function to drive the agent to fully learn the optimal decision policy. Extensive experiments on 16 LDL datasets are conducted under various evaluation metrics. The experimental results demonstrate convincingly that the proposed sequential label enhancement (LE) leads to better performance over the state-of-the-art methods.

Non-additive network pricing with non-cooperative mobility service providers

This study addresses a mobility network pricing problem in a competitive environment. We consider a multimodal transportation network where the links are operated by multiple profit-maximizing, mobility service providers (MSPs). We take the perspective of a network regulator that aims to increase ridership in a target mobility network by providing non-additive, path-based subsidies to travelers. We model paths’ attractiveness using generalized cost functions that combine path travel time and path cost, and we use linear elastic travel demand functions to capture the proportion of demand served by a path. MSPs are non-cooperative and adjust link fares according to the subsidy policy implemented by the regulator. The goal of the network regulator is to solve a budget-constrained mobility network pricing problem under MSP competition. This game-theoretical framework is modeled as a single-leader multi-follower game (SLMFG) wherein the leader player represents the network regulator and multiple follower players represent the MSPs. We conduct a theoretical analysis of this SLMFG by identifying necessary and sufficient conditions for the existence of solutions to the parameterized generalized Nash equilibrium problem (GNEP) that is played amongst MSPs. We show that this GNEP is jointly convex and we use this property to develop an exact numerical approach to solve the SLMFG based on customized branch-and-bound algorithms. Numerical results reveal the impact of MSP competition in this mobility network pricing problem and shed novel insights into the design of optimal path-based subsidy policies.

Vital signs monitoring in Australasian emergency departments: Development of a consensus statement from ACEM and CENA

BackgroundEmergency Department (ED) care is provided for a diverse range of patients, clinical acuity and conditions. This diversity often calls for different vital signs monitoring requirements. Requirements often change depending on the circumstances that patients experience during episodes of ED care. AimTo describe expert consensus on vital signs monitoring during ED care in the Australasian setting to inform the content of a joint Australasian College for Emergency Medicine (ACEM) and College of Emergency Nursing Australasia (CENA) position statement on vital signs monitoring in the ED. MethodA 4-hour online nominal group technique workshop with follow up surveys. ResultsTwelve expert ED nurses and doctors from adult, paediatric and mixed metropolitan and regional ED and research facilities spanning four Australian states participated in the workshop and follow up surveys. Consensus building generated 14 statements about vital signs monitoring in ED. Good consensus was reached on whether vital signs should be assessed for 15 of 19 circumstances that patients may experience. ConclusionThis study informed the creation of a joint position statement on vital signs monitoring in the Australasian ED setting, endorsed by CENA and ACEM. Empirical evidence is needed for optimal, safe and achievable policy on this fundamental practice.

Everybody’s got to learn sometime? A causal machine learning evaluation of training programmes for jobseekers in France

This paper estimates the heterogeneous impact of three types of vocational training- preparation, qualifying, and combined – on jobseekers’ return to employment using the Modified Causal Forest method. Analysing data from 33,699 individuals over 24 months, it reveals a short-term negative lock-in effect for all programmes, persisting in the medium term for combined training. Only qualifying training shows a positive medium-term effect. Seniors, low-skilled, foreign-born, and those with poor job histories benefit most, while youth and higher education levels benefit less. Targeting foreign-born individuals could significantly enhance programme effectiveness, as indicated by the clustering analysis and optimal policy trees.

Curiosity model policy optimization for robotic manipulator tracking control with input saturation in uncertain environment

In uncertain environments with robot input saturation, both model-based reinforcement learning (MBRL) and traditional controllers struggle to perform control tasks optimally. In this study, an algorithmic framework of Curiosity Model Policy Optimization (CMPO) is proposed by combining curiosity and model-based approach, where tracking errors are reduced via training agents on control gains for traditional model-free controllers. To begin with, a metric for judging positive and negative curiosity is proposed. Constrained optimization is employed to update the curiosity ratio, which improves the efficiency of agent training. Next, the novelty distance buffer ratio is defined to reduce bias between the environment and the model. Finally, CMPO is simulated with traditional controllers and baseline MBRL algorithms in the robotic environment designed with non-linear rewards. The experimental results illustrate that the algorithm achieves superior tracking performance and generalization capabilities.

Satellite-Based ITS Data Offloading & Computation in 6G Networks: A Cooperative Multi-Agent Proximal Policy Optimization DRL With Attention Approach

Satellite-Based ITS Data Offloading & Computation in 6G Networks: A Cooperative Multi-Agent Proximal Policy Optimization DRL With Attention Approach

An adaptive dual-level reinforcement learning approach for optimal trade execution

The purpose of this research is to devise a tactic that can closely track the daily cumulative volume-weighted average price (VWAP) using reinforcement learning while minimizing the deviation from the VWAP. Previous studies often choose a relatively short trading horizon to implement their models, making it difficult to accurately track the daily cumulative VWAP since the stock price movement is often insignificant within the short trading horizon. On the other hand, training reinforcement learning models directly over a longer, daily horizon is burdensome due to extensive sequence length. Hence, there is a need for a method that can divide the long daily horizon into smaller, more manageable segments. We propose a method that leverages the U-shaped pattern of intraday stock trade volumes and uses Proximal Policy Optimization (PPO) as the learning algorithm. Our method follows a dual-level approach: a Transformer model that captures the overall (global) distribution of daily volumes in a U-shape, and a LSTM model that handles the distribution of orders within smaller (local) time intervals. The results from our experiments suggest that this dual-level architecture improves cumulative VWAP tracking accuracy compared to previous reinforcement learning approaches. The key finding is that explicitly accounting for the U-shaped intraday volume pattern leads to better performance in approximating the cumulative daily VWAP. This has implications for developing trading strategies that need to efficiently track VWAP over a full trading day.

Prism Blockchain Enabled Internet of Things with Deep Reinforcement Learning

This paper presents a Deep Reinforcement Learning based IoT enabled Prism BLockchain. The recent advancements in the field of Internet of Things (IoT) motivate the development of a secure infrastructure for storing and sharing vast amounts of data. Blockchain, a distributed and immutable ledger, is best known as a potential solution to data security and privacy for IoT. The scalability of blockchain, which should optimize the throughput and handle the dynamics of the IoT environment, becomes a challenge due to the enormous amount of IoT data. The critical challenge in scaling blockchain is to guarantee decentralization, latency, and security of the system while optimizing the transaction throughput. This paper presents a deep reinforcement learning (DRL)-based performance optimization for blockchain-enabled IoT. We consider one of the recent promising blockchains, Prism as the underlying blockchain system because of its good performance guarantees. We integrate the IoT data to Prism Blockchain and optimize the performance of the system by leveraging Proximal Policy Optimization (PPO) method. The DRL method helps to optimize the blockchain parameters like mining rate and mined blocks to adapt to the environment dynamics of the IoT system. Our results show that the proposed method can improve the throughput of Prism blockchain based IoT systems while preserving Prism performance guarantees. Our scheme can achieve 1.5 times more system rewards than IoT integrated Prism. In our experimental setup, the proposed scheme could improve the average throughput of the system by about 6,000 transactions per sec compared to Prism.

Stochastic Integrated Actor-Critic for Deep Reinforcement Learning.

We propose a deep stochastic actor-critic algorithm with an integrated network architecture and fewer parameters. We address stabilization of the learning procedure via an adaptive objective to the critic's loss and a smaller learning rate for the shared parameters between the actor and the critic. Moreover, we propose a mixed on-off policy exploration strategy to speed up learning. Experiments illustrate that our algorithm reduces the sample complexity by 50%-93% compared with the state-of-the-art deep reinforcement learning (RL) algorithms twin delayed deep deterministic policy gradient (TD3), soft actor-critic (SAC), proximal policy optimization (PPO), advantage actor-critic (A2C), and interpolated policy gradient (IPG) over continuous control tasks LunarLander, BipedalWalker, BipedalWalkerHardCore, Ant, and Minitaur in the OpenAI Gym.

A Reinforcement Learning-Based Pantograph Control Strategy for Improving Current Collection Quality in High-Speed Railways.

In high-speed railways, the pantograph-catenary system (PCS) is a critical subsystem of the train power supply system. In particular, when the double-PCS (DPCS) is in operation, the passing of the leading pantograph (LP) causes the contact force of the trailing pantograph (TP) to fluctuate violently, affecting the power collection quality of the electric multiple units (EMUs). The actively controlled pantograph is the most promising technique for reducing the pantograph-catenary contact force (PCCF) fluctuation and improving the current collection quality. Based on the Nash equilibrium framework, this study proposes a multiagent reinforcement learning (MARL) algorithm for active pantograph control called cooperative proximity policy optimization (Coo-PPO). In the algorithm implementation, the heterogeneous agents play a unique role in a cooperative environment guided by the global value function. Then, a novel reward propagation channel is proposed to reveal implicit associations between agents. Furthermore, a curriculum learning approach is adopted to strike a balance between reward maximization and rational movement patterns. An existing MARL algorithm and a traditional control strategy are compared in the same scenario to validate the proposed control strategy's performance. The experimental results show that the Coo-PPO algorithm obtains more rewards, significantly suppresses the fluctuation in PCCF (up to 41.55%), and dramatically decreases the TP's offline rate (up to 10.77%). This study adopts MARL technology for the first time to address the coordinated control of double pantographs in DPCS.