Optimal Policy Research Articles

Markov models for clinical decision-making in radiation oncology: A systematic review.

The intrinsic stochasticity of patients' response to treatment is a major consideration for clinical decision-making in radiation therapy. Markov models are powerful tools to capture this stochasticity and render effective treatment decisions. This paper provides an overview of the Markov models for clinical decision analysis in radiation oncology. A comprehensive literature search was conducted within MEDLINE using PubMed, following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Only studies published from 2000 to 2023 were considered. Selected publications were summarized in two categories: (i) studies that compare two (or more) fixed treatment policies using Monte Carlo simulation and (ii) studies that seek an optimal treatment policy through Markov Decision Processes (MDPs). Relevant to the scope of this study, 61 publications were selected for detailed review. The majority of these publications (n = 56) focused on comparative analysis of two or more fixed treatment policies using Monte Carlo simulation. Classifications based on cancer site, utility measures and the type of sensitivity analysis are presented. Five publications considered MDPs with the aim of computing an optimal treatment policy; a detailed statement of the analysis and results is provided for each work. As an extension of Markov model-based simulation analysis, MDP offers a flexible framework to identify an optimal treatment policy among a possibly large set of treatment policies. However, the applications of MDPs to oncological decision-making have been understudied, and the full capacity of this framework to render complex optimal treatment decisions warrants further consideration.

Optimizing Internet-Wide Port Scanning for IoT Security and Network Resilience: A Reinforcement Learning-Based Approach in WLANs with IEEE 802.11ah

ABSTRACT The rapid proliferation of Internet of Things (IoT) devices has spurred the need for robust security mechanisms within wireless local area network (WLAN) environments. Specifically, the IEEE 802.11ah (WiFi-HaLow) technology offers low-power, long-range communication capabilities ideally suited for IoT devices, but these devices face security threats due to constrained computational resources. This research addresses the challenge of optimizing Internet-wide port scanning to enhance IoT security while minimizing disruptions to network performance. In this paper, we propose a novel reinforcement learning-based approach to achieve this optimization. Our approach harnesses the power of the Proximal Policy Optimization (PPO) algorithm to guide decision-making for IoT security and network performance enhancement. The proposed solution entails training an agent to dynamically adjust the port scanning rate, ensuring the delicate balance between security improvement and network responsiveness. We establish a comprehensive system model, encompassing WiFi-HaLow infrastructure, IPv6-enabled IoT devices, and the integration of Internet-security (IPSec) protocols. Mathematical formulations encapsulate constraints such as resource limitations and security-performance trade-offs. Our experimental setup utilizes a carefully curated dataset and a simulation environment to rigorously evaluate the proposed solution’s effectiveness. The proposed approach demonstrates remarkable security enhancement through the prevention of unauthorized access attempts, data breaches, and improved intrusion detection. Moreover, network performance is optimized, with reductions in latency, improvements in throughput, and energy-efficient operation. By presenting a plethora of comparison scenarios, we validate the superiority of our solution against various benchmarks. This research contributes a comprehensive framework that not only advances IoT security within WLAN environments but also highlights the intricate relationship between security and network performance optimization.

Policy Instrument Preferences and Optimization Strategies: Based Text Analysis of Provincial-Level Education Digitalization Policy from China

Policy Instrument Preferences and Optimization Strategies: Based Text Analysis of Provincial-Level Education Digitalization Policy from China

Safe Autonomous Driving with Latent Dynamics and State-Wise Constraints

Autonomous driving has the potential to revolutionize transportation, but developing safe and reliable systems remains a significant challenge. Reinforcement learning (RL) has emerged as a promising approach for learning optimal control policies in complex driving environments. However, existing RL-based methods often suffer from low sample efficiency and lack explicit safety constraints, leading to unsafe behaviors. In this paper, we propose a novel framework for safe reinforcement learning in autonomous driving that addresses these limitations. Our approach incorporates a latent dynamic model that learns the underlying dynamics of the environment from bird’s-eye view images, enabling efficient learning and reducing the risk of safety violations by generating synthetic data. Furthermore, we introduce state-wise safety constraints through a barrier function, ensuring safety at each state by encoding constraints directly into the learning process. Experimental results in the CARLA simulator demonstrate that our framework significantly outperforms baseline methods in terms of both driving performance and safety. Our work advances the development of safe and efficient autonomous driving systems by leveraging the power of reinforcement learning with explicit safety considerations.

Decentralized Navigation with Optimality for Multiple Holonomic Agents in Simply Connected Workspaces

Multi-agent systems are utilized more often in the research community and industry, as they can complete tasks faster and more efficiently than single-agent systems. Therefore, in this paper, we are going to present an optimal approach to the multi-agent navigation problem in simply connected workspaces. The task involves each agent reaching its destination starting from an initial position and following an optimal collision-free trajectory. To achieve this, we design a decentralized control protocol, defined by a navigation function, where each agent is equipped with a navigation controller that resolves imminent safety conflicts with the others, as well as the workspace boundary, without requesting knowledge about the goal position of the other agents. Our approach is rendered sub-optimal, since each agent owns a predetermined optimal policy calculated by a novel off-policy iterative method. We use this method because the computational complexity of learning-based methods needed to calculate the global optimal solution becomes unrealistic as the number of agents increases. To achieve our goal, we examine how much the yielded sub-optimal trajectory deviates from the optimal one and how much time the multi-agent system needs to accomplish its task as we increase the number of agents. Finally, we compare our method results with a discrete centralized policy method, also known as a Multi-Agent Poli-RRT* algorithm, to demonstrate the validity of our method when it is attached to other research algorithms.

Beyond Static Obstacles: Integrating Kalman Filter with Reinforcement Learning for Drone Navigation

Autonomous drones offer immense potential in dynamic environments, but their navigation systems often struggle with moving obstacles. This paper presents a novel approach for drone trajectory planning in such scenarios, combining the Interactive Multiple Model (IMM) Kalman filter with Proximal Policy Optimization (PPO) reinforcement learning (RL). The IMM Kalman filter addresses state estimation challenges by modeling the potential motion patterns of moving objects. This enables accurate prediction of future object positions, even in uncertain environments. The PPO reinforcement learning algorithm then leverages these predictions to optimize the drone’s real-time trajectory. Additionally, the capability of PPO to work with continuous action spaces makes it ideal for the smooth control adjustments required for safe navigation. Our simulation results demonstrate the effectiveness of this combined approach. The drone successfully navigates complex dynamic environments, achieving collision avoidance and goal-oriented behavior. This work highlights the potential of integrating advanced state estimation and reinforcement learning techniques to enhance autonomous drone capabilities in unpredictable settings.

On the optimality of stepwise policies for managing capacity, inventory and backorders

We consider the problem of simultaneously managing capacity, inventory and backorders in a multi-mode production environment modelled via Brownian motion. The presence of more than two production modes adds an additional level of complexity: not just when to change modes, but also which mode to change to. We show that, under the two assumptions that demand is the overriding source of variability in the process and that the cost to change from one production mode to another is proportional to the difference in the production capacities, a policy that moves stepwise among the modes minimizes the long-run average cost. Examples demonstrate that if either assumption is violated no policy that moves stepwise among the modes may be optimal. To focus on the complexity of identifying when to change modes and which mode to change to, we restrict our model to simple convex holding and backorder costs and linear processing costs and costs for rejecting demand and idling capacity. We adopt the economic average cost model that allows the manager to reject demand or idle capacity at any time. We demonstrate that under our two assumptions we may impose an ordering assumption on the relative value functions that dramatically simplifies the classic Hamilton-Jacobi-Bellman equations. We show that under the economic average cost model a production mode that is initially unattractive may later become attractive as new modes are added. Our arguments are essentially constructive and lead to a practical algorithm for finding an optimal policy.

Analisis Kebijakan Sertifikasi Guru dalam Meningkatkan Mutu Pendidikan di SMPN 47 Surabaya

This study examines the effectiveness of the teacher certification policy aimed at improving educational standards at SMP Negeri 47 in Surabaya, with a focus on how it affects teacher’s competence, welfare, and skills. The certification policy is hypothesized to play a pivotal role in elevating the quality of education offered. Employing a qualitative approach, this study examines the implementation and outcomes of the policy since its introduction. Data was collected through various qualitative methodologies and analyzed to ascertain the policy’s contribution to enhancing teacher welfare, competence, and professionalism. The findings reveal that the teacher certification policy at SMPN 47 Surabaya has been significantly instrumental in advancing educational quality aligning with the policy’s prime objectives. These insights hold wide implications for educational policy optimization at a broader level.

Managing Hybrid Manufacturing/Remanufacturing Inventory Systems With Random Production Capacities

In this article, we consider hybrid manufacturing/remanufacturing inventory systems that produce a single product to satisfy demands over a finite planning horizon. In each period, the firm receives random demand and returns of end-of-life products. A serviceable product can be manufactured from ample raw materials or remanufactured from a returned product. The two operations possess random dedicated capacities. The firm’s objective is to minimize the expected total discounted cost over the planning horizon. We partially characterize the firm’s optimal inventory policy when the two capacities are positively dependent and completely characterize it when only one capacity is random. When there is ample manufacturing capacity, we connect the model with an auxiliary dual-sourcing inventory model and derive a more detailed structure of the optimal policy. Finally, our numerical study provides actionable insights into the effects of random capacities. Among others, we find that approximating a slightly/moderately variable remanufacturing capacity as its deterministic mean capacity or ignoring the correlation between two random capacities under a multi-period setting incurs a limited cost to the firm.

The Algorithmic Assignment of Incentive Schemes

The assignment of individuals with different observable characteristics to different treatments is a central question in designing optimal policies. We study this question in the context of increasing workers’ performance via targeted incentives using machine learning algorithms with worker demographics, personality traits, and preferences as input. Running two large-scale experiments, we show that (i) performance can be predicted by accurately measured worker characteristics, (ii) a machine learning algorithm can detect heterogeneity in responses to different schemes, (iii) a targeted assignment of schemes to individuals increases performance significantly above the level of the single best scheme, and (iv) algorithmic assignment is more effective for workers who have a high likelihood to repeatedly interact with the employer or who provide more consistent survey answers. This paper was accepted by Yan Chen, behavioral economics and decision analysis. Funding: Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy [Grant EXC 2126/1-390838866]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2022.03362 .

Impact of warranty and green level of the product with nonlinear demand via optimal control theory and Artificial Hummingbird Algorithm

Due to the current environmental situation and human health, a green manufacturing system is very essential in the manufacturing world. Several researchers have developed various types of green manufacturing models by considering green products, green investments, carbon emission taxes, etc. Motivated by this topic, a green production model is formulated by considering selling price, time, warranty period and green level dependent demand with a carbon emission tax policy. Also, the production rate of the system is an unknown function of time. Per unit production cost of the products is taken as increasing function of production rate and green level of the products. In our proposed model, carbon emission rate is taken as linear function of time. Then, an optimization problem of the production model is constructed. To validate of our proposed model, a numerical example is considered and solved it by AHA. Further, other five metaheuristics algorithms (AEFA, FA, GWOA, WOA and EOA) are taken to compare the results obtained from AHA. Also, concavity of the average profit function and convergence graph of different metaheuristics algorithms are presented. Finally, a sensitivity analysis is carried out to investigate the impact of different system parameters on our optimal policy and reach a fruitful conclusion from this study.

Spatiotemporal Evolution and Factors Influencing Regional Ecological Land in a Multidimensional Perspective: A Case Study of the Beijing–Tianjin–Hebei Region

A systematic analysis of the spatiotemporal evolution patterns and factors influencing ecological land (EL) can support the optimization of EL protection policies and ensure the stability of regional ecosystems. Based on remote sensing data, using the gravity center shift model, the landscape pattern index, and the equivalent factor method, the characteristics of EL evolution in the Beijing–Tianjin–Hebei (BTH) region from 1980 to 2020 were analyzed. A fixed-effects model was used to quantitatively explore the factors influencing EL evolution and heterogeneity analysis. The results are as follows: (1) The EL area exhibited a trend of initial decrease followed by a subsequent increase during the study period. The most significant area transfer occurred between cropland and EL, but, after the 21st century, the proportion of area transfer between construction land and EL noticeably increased. (2) The compactness and fragmentation of EL showed a certain spatiotemporal stability, but the spatial distribution of compactness and fragmentation hot and cold spots exhibited significant differences. The fragmentation hot spots mainly displayed a strip distribution, while those of compactness showed a clustered distribution. (3) Although the ecosystem service value in the BTH region demonstrated dynamic “M”-shaped changes, the distribution of hot and cold spots still exhibited spatial stability. Regulating services consistently occupied a higher proportion of the sub-service functions, while cultural services still needed further enhancement. (4) Factors influencing the evolution of areas and values demonstrated similarities. The landscape was significantly influenced by construction land, showing a non-linear “U”-shaped relationship with fragmentation. Different economic development gradients and altitudes exhibited differentiated characteristics in terms of their influencing factors. This study provides scientific support for dynamically and precisely adjusting governmental EL management policies, contributing to the sustainable development of regional socio-economics.

Control of Qubit Dynamics Using Reinforcement Learning

The progress in machine learning during the last decade has had a considerable impact on many areas of science and technology, including quantum technology. This work explores the application of reinforcement learning (RL) methods to the quantum control problem of state transfer in a single qubit. The goal is to create an RL agent that learns an optimal policy and thus discovers optimal pulses to control the qubit. The most crucial step is to mathematically formulate the problem of interest as a Markov decision process (MDP). This enables the use of RL algorithms to solve the quantum control problem. Deep learning and the use of deep neural networks provide the freedom to employ continuous action and state spaces, offering the expressivity and generalization of the process. This flexibility helps to formulate the quantum state transfer problem as an MDP in several different ways. All the developed methodologies are applied to the fundamental problem of population inversion in a qubit. In most cases, the derived optimal pulses achieve fidelity equal to or higher than 0.9999, as required by quantum computing applications. The present methods can be easily extended to quantum systems with more energy levels and may be used for the efficient control of collections of qubits and to counteract the effect of noise, which are important topics for quantum sensing applications.

Optimal Scheme Models with Two Kinds of Errors

In this paper, we propose stochastic scheme models with two kinds of errors for a computer system. A highly reliable computer system needs greatly to be backed up successfully in case of computer system crashes due to hardware failures and malicious software infections. We aim to determine appropriate backup times in computer processing of constant and random tasks. For this purpose, the trade-off between backup times and data losses due to errors is considered. The total execution times until the computing processing completes are obtained, and optimal backup policies to minimize them are derived analytically. Furthermore, we take up extended models of random tasks with [Formula: see text] sub-tasks and analyze them.

Urbanization and economic growth in China: empirical evidence based on a SIVQR approach

PurposeWe seek to quantify the relationship between urbanization and economic growth in China using recent advances in econometric techniques.Design/methodology/approachThis study adopts a smoothed instrumental variables quantile regression (SIVQR) estimator to obtain consistent estimates of the effects of urbanization on economic growth in China. Our approach accounts for the differential impacts of urbanization across the conditional distribution of economic growth while allowing for an identification strategy that addresses the endogeneity of urbanization. Our main findings reveal that ignoring urbanization endogeneity leads to inconsistent estimates of urbanization effects. Further, we find a positive relationship between urbanization and growth resembling an inverted U-shape. This supports the hypothesis that the beneficial effects of urbanization intensify at initial stages while diminishing beyond a certain threshold, due perhaps to weakening scale economies.FindingsOur main results indicate that the individual productivity gains brought by urbanization outweigh the negative effects thereof that impede productivity, thus contributing to the economic growth in China. Further, we find that ignoring differential impacts of urbanization underestimates the beneficial effects of urbanization for provinces whose quality of governance is in the vicinity of the center of quality distribution. Ignoring the endogeneity of urbanization generates inconsistent estimates of the elasticity of economic growth with respect to urbanization. Finally, we estimate an inverted U-shape resembling relationship between urbanization and growth.Research limitations/implicationsFirst, future studies would benefit from incorporating more data as provinces further east on the mainland become more urbanized and urbanization runs its course. Second, controlling the barriers to rural-urban mobility would contribute to the robustness of the estimated relationship between urbanization and growth once such data became available. Unveiling the impact of government-imposed barriers is key to designing optimal policies that help fuel economic growth in the country. Finally, future research could benefit from information on urbanization sources not considered here such as inter-provincial migration, as such data become publicly available.Practical implicationsQuantifying the beneficial effects of urbanization on economic growth can help guide the government in China to further fuel the growth through a set of relevant policy tools that promote urbanization.Social implicationsRural-urban migration in China lays the groundwork for economic advancement in recipient cities and economies, as it may induce scale economies. This can benefit both the economy at large and the migrants.Originality/valueThe SIVQR estimator accounts for potential heterogeneous effects of urbanization across the entire conditional distribution of growth while allowing for an identification strategy that addresses the endogeneity of urbanization. An additional distinguishing feature of the current study is our use of the most recent novel, provincial-level data obtained from the National Bureau of Statistics of China. Our focus on a single country allows sidestepping issues arising from the inconsistency of the definition of urban across different countries while accounting for intra-country urbanization drivers intrinsic to China, such as natural features and geographic characteristics. Therefore, our approach has the potential to sidestep the bias resulting from the differences in mechanisms behind urbanization-growth relationships across different countries.

Performance evaluation of national and international kidney exchange programmes with the ENCKEP simulator

AbstractIn this paper we present simulations for international kidney exchange programmes (KEPs). KEPs are organised in more than ten countries in Europe to facilitate the exchanges of immunologically incompatible donors. The matching runs are typically conducted in every three months for finding optimal exchanges using hierarchical optimisation with integer programming techniques. In recent years several European countries started to organise international exchanges using different collaboration policies. In this paper we conduct simulations for estimating the benefits of such collaborations with a simulator developed by the team of the ENCKEP COST Action. We conduct our simulations on generated datasets mimicking the practice of the three largest KEPs in Europe, the UK, Spanish and the Dutch programmes. Our main performance measure is the number of transplants compared to the number of registrations to the KEP pools over a 5-year period, however, as a novelty we also analyse how the optimisation criteria play a role in the lexicographic and weighted optimisation policies for these countries. Besides analysing the performances on a single instance, we also conduct large number of simulations to obtain robust findings on the performance of specific national programmes and on the possible benefits of international collaborations.

Deep Reinforcement Learning for Optimizing Restricted Access Window in IEEE 802.11ah MAC Layer

The IEEE 802.11ah standard is introduced to address the growing scale of internet of things (IoT) applications. To reduce contention and enhance energy efficiency in the system, the restricted access window (RAW) mechanism is introduced in the medium access control (MAC) layer to manage the significant number of stations accessing the network. However, to achieve optimized network performance, it is necessary to appropriately determine the RAW parameters, including the number of RAW groups, the number of slots in each RAW, and the duration of each slot. In this paper, we optimize the configuration of RAW parameters in the uplink IEEE 802.11ah-based IoT network. To improve network throughput, we analyze and establish a RAW parameters optimization problem. To effectively cope with the complex and dynamic network conditions, we propose a deep reinforcement learning (DRL) approach to determine the preferable RAW parameters to optimize network throughput. To enhance learning efficiency and stability, we employ the proximal policy optimization (PPO) algorithm. We construct network environments with periodic and random traffic in an NS-3 simulator to validate the performance of the proposed PPO-based RAW parameters optimization algorithm. The simulation results reveal that using the PPO-based DRL algorithm, optimized RAW parameters can be obtained under different network conditions, and network throughput can be improved significantly.

Deep-reinforcement-learning-based RMSCA for space division multiplexing networks with multi-core fibers [Invited Tutorial

The escalating demands for network capacities catalyze the adoption of space division multiplexing (SDM) technologies. With continuous advances in multi-core fiber (MCF) fabrication, MCF-based SDM networks are positioned as a viable and promising solution to achieve higher transmission capacities in multi-dimensional optical networks. However, with the extensive network resources offered by MCF-based SDM networks comes the challenge of traditional routing, modulation, spectrum, and core allocation (RMSCA) methods to achieve appropriate performance. This paper proposes an RMSCA approach based on deep reinforcement learning (DRL) for MCF-based elastic optical networks (MCF-EONs). Within the solution, a novel state representation with essential network information and a fragmentation-aware reward function were designed to direct the agent in learning effective RMSCA policies. Additionally, we adopted a proximal policy optimization algorithm featuring an action mask to enhance the sampling efficiency of the DRL agent and speed up the training process. The performance of the proposed algorithm was evaluated with two different network topologies with varying traffic loads and fibers with different numbers of cores. The results confirmed that the proposed algorithm outperforms the heuristics and the state-of-the-art DRL-based RMSCA algorithm in reducing the service blocking probability by around 83% and 51%, respectively. Moreover, the proposed algorithm can be applied to networks with and without core switching capability and has an inference complexity compatible with real-world deployment requirements.

Clinical and Genomic-Based Decision Support System to Define the Optimal Timing of Allogeneic Hematopoietic Stem-Cell Transplantation in Patients With Myelodysplastic Syndromes

PURPOSE Allogeneic hematopoietic stem-cell transplantation (HSCT) is the only potentially curative treatment for patients with myelodysplastic syndromes (MDS). Several issues must be considered when evaluating the benefits and risks of HSCT for patients with MDS, with the timing of transplantation being a crucial question. Here, we aimed to develop and validate a decision support system to define the optimal timing of HSCT for patients with MDS on the basis of clinical and genomic information as provided by the Molecular International Prognostic Scoring System (IPSS-M). PATIENTS AND METHODS We studied a retrospective population of 7,118 patients, stratified into training and validation cohorts. A decision strategy was built to estimate the average survival over an 8-year time horizon (restricted mean survival time [RMST]) for each combination of clinical and genomic covariates and to determine the optimal transplantation policy by comparing different strategies. RESULTS Under an IPSS-M based policy, patients with either low and moderate-low risk benefited from a delayed transplantation policy, whereas in those belonging to moderately high-, high- and very high-risk categories, immediate transplantation was associated with a prolonged life expectancy (RMST). Modeling decision analysis on IPSS-M versus conventional Revised IPSS (IPSS-R) changed the transplantation policy in a significant proportion of patients (15% of patient candidate to be immediately transplanted under an IPSS-R–based policy would benefit from a delayed strategy by IPSS-M, whereas 19% of candidates to delayed transplantation by IPSS-R would benefit from immediate HSCT by IPSS-M), resulting in a significant gain-in-life expectancy under an IPSS-M–based policy ( P = .001). CONCLUSION These results provide evidence for the clinical relevance of including genomic features into the transplantation decision making process, allowing personalizing the hazards and effectiveness of HSCT in patients with MDS.

Fair Pension Policies with Occupation-Specific Ageing

Abstract We study the optimal design of a fair public pension system in a multi-period overlapping generations model with occupation-specific morbidity and mortality that depends on the retirement age. The fairness constraint acts as institutional device ensuring that lifetime returns to contributions are equal across occupational groups. We consider group-specific replacement rates and a calculatory interest rate for early contributions as policy instruments. Calibrating the model to Germany, we find that the transition to optimal fair pension policies may induce early retirement of blue-collar workers and significantly raises their lifetime pension benefits and welfare. Aggregate welfare increases in all fair pension scenarios.