Online Policy Research Articles

Impact of department chair gender on paid parental leave across American anaesthesiology residencies

BackgroundResidency training and parenthood are conflicting pursuits for many residents, as both often occur during similar years of life. Online policy about paid parental leave for residents is important for...

A knowledge-guided reinforcement learning method for lateral path tracking

Lateral Control algorithms in autonomous vehicles often necessitates an online fine-tuning procedure in the real world. While reinforcement learning (RL) enables vehicles to learn and improve the lateral control performance through repeated trial and error interactions with a dynamic environment, applying RL directly to safety-critical applications in real physical world is challenging because ensuring safety during the learning process remains difficult. To enable safe learning, a promising direction is to make use of previously gathered offline data, which is frequently accessible in engineering applications. In this context, this paper presents a set of knowledge-guided RL algorithms that can not only fully leverage the prior collected offline data without the need of a physics-based simulator, but also allow further online policy improvement in a smooth, safe and efficient manner. To evaluate the effectiveness of the proposed algorithms on a real controller, a hardware-in-the-loop and a miniature vehicle platform are built. Compared with the vanilla RL, behavior cloning and the existing controller, the proposed algorithms realize a closed-loop solution for lateral control problems from offline training to online fine-tuning, making it attractive for future similar RL-based controller to build upon.

A review of Peyronie's disease insurance coverage.

Peyronie's disease (PD) is a fibroproliferative disorder that causes an abnormal curve of the penis resulting in pain, discomfort, and erectile dysfunction with management options including correctional surgery, penile external/internal devices, shock wave therapy (SWT), intralesional Verapamil (IV), and collagenase Clostridium histolyticum injections. To investigate the insurance coverage of these treatment options. The authors performed a cross-sectional analysis of the top US insurance policies for coverage of PD. Companies were chosen based on their market share and enrollment. Their policies were identified through a Web-based search and telephone interviews, and the companies' medical necessity criteria were defined. The online policies were then re-examined for interval change 3 years later. There are significant discrepancies for in coverage for the different modalities of Peyronie's treatment. Of the 100 companies examined, only 54% of companies had a policy that directly addressed the treatment coverage for PD. The most covered treatment was CCH injections with 37 companies providing unanimous coverage (n = 37, 100%). Within this category, all companies required a palpable plaque as a requirement. Additionally, external/internal devices were unanimous covered by 18 companies (n = 18, 100%). Surgical treatment was covered by eight companies with six companies denying coverage (n = 8 vs. n = 6, 57.1% vs. 42.9%). IV was covered by two companies. The least covered treatment option was SWT which was universally denied by 19 companies (n = 19, 100%). Insurance coverage of PD should be aligned with current medical literature to better increase access to care. This study is limited by the updated policies of insurance companies and future applicability. Additionally, this study assumes that a written policy will provide coverage and may overestimate the actual extent of coverage. Finally, this study only addressed some of the common treatment options of PD and did not expand on all possible treatment options. Differing rates for modalities of treatment may present a barrier for patients to receive individualized and adequate treatment for PD.

SIGACT News Online Algorithms Column 42

For this issue, I am pleased to present a tutorial by Will Ma on using randomized rounding to design online policies. Randomized rounding has of course been used extensively in the design of approximation algorithms and also to approximately solve sequential stochastic optimization problems. Will describes several examples that show how randomized rounding can be used in this context. As always, I would like to invite more contributions to this column, be it surveys, conference reports, or technical articles related to online algorithms and competitive analysis. If you are considering becoming a guest writer, don't hesitate to mail me at rob.vanstee@uni-siegen.de.

Exploring the cross-sectional association between the strength of school vaping policies and student vaping behaviours using data from the 2021-2022 COMPASS Study.

Youth vaping is a concern in Canada. While school-level policies influence student behaviours, few studies have investigated the association between school vaping policies and student vaping. This study reviewed and scored the comprehensiveness of school vaping policies and investigated the association between school vaping policy scores and student vaping. Online policy documents from n = 39 schools in Ontario, Alberta, and British Columbia, Canada, participating in the 2021-2022 wave of the COMPASS study were collected, reviewed, and scored for comprehensiveness (/39) using the School Tobacco Policy Index (STPI) rating form. The mean and range of scores for each domain of the STPI were calculated. School policy scores were linked to student vaping data from the COMPASS study. Multilevel logistic regression analyses identified the association between school vaping policy score and student lifetime and current (past 30-day) vaping. The mean total policy score was 10.2/39 (range 0‒24), and 28% of schools scored 0/39. The majority of school policies did not identify enforcement approaches or available preventive or cessation resources. Increasing STPI score was not associated with the odds of student lifetime or current vaping in multilevel logistic regression analyses. The STPI quickly identified components of school vaping policies that were missing. The overall score of most school vaping policies in our sample was low and most school vaping policies lacked many important components. Future studies should explore factors associated with adolescent vaping and identify effective prevention measures.

Technical Note—Near-Optimal Bayesian Online Assortment of Reusable Resources

Near-Optimal Bayesian Online Assortment of Reusable Resources Motivated by rental services in e-commerce, we consider revenue maximization in the online assortment of reusable resources for different types of arriving consumers. We design competitive online algorithms compared with the optimal online policy in the Bayesian setting, where consumer types are drawn independently from known heterogeneous distributions over time. In scenarios with large initial inventories, our main result is a near-optimal competitive algorithm for reusable resources. Our algorithm relies on an expected linear programming (LP) benchmark, solves this LP, and simulates the solution through independent randomized rounding. The main challenge is achieving inventory feasibility efficiently using these simulation-based algorithms. To address this, we design discarding policies for each resource, balancing inventory feasibility and revenue loss. Discarding a unit of a resource impacts future consumption of other resources, so we introduce postprocessing assortment procedures to design and analyze our discarding policies. Additionally, we present an improved competitive algorithm for nonreusable resources and evaluate our algorithms using numerical simulations on synthetic data.

Vaping posts on Instagram violate federal and online policies, research finds

Vaping posts on Instagram violate federal and online policies, research finds

Dynamic vehicle routing problem with drone resupply for same-day delivery

We study a same-day delivery problem where customer orders arrive dynamically throughout the day and the service operator must determine, in real-time, whether to accept the orders and how to adjust the ongoing distribution plan. We develop a route-based Markov Decision Process and an efficient online policy to dynamically route a truck that can receive newly arrived orders along its route via drones dispatched from a depot. Numerical experiments show that our online policy has an average fill rate decrease of at most 20% over the perfect-information counterpart. Further, this online policy has a fill rate increase of up to 8% over a naïve greedy policy. We also show that drone resupply increases fill rates by up to 21% compared to a conventional truck-only resupply system. Computational times to make each decision are in the hundredths of a second, thus allowing real-time feedback to customers regarding their eligibility for same-day delivery.

Offline Planning and Online Learning Under Recovering Rewards

Motivated by emerging applications, such as live-streaming e-commerce, promotions, and recommendations, we introduce and solve a general class of nonstationary multi-armed bandit problems that have the following two features: (i) the decision maker can pull and collect rewards from up to [Formula: see text] out of N different arms in each time period and (ii) the expected reward of an arm immediately drops after it is pulled and then nonparametrically recovers as the arm’s idle time increases. With the objective of maximizing the expected cumulative reward over T time periods, we design a class of purely periodic policies that jointly set a period to pull each arm. For the proposed policies, we prove performance guarantees for both the offline and the online problems. For the offline problem when all model parameters are known, the proposed periodic policy obtains a long-run approximation ratio that is at the order of [Formula: see text], which is asymptotically optimal when K grows to infinity. For the online problem when the model parameters are unknown and need to be dynamically learned, we integrate the offline periodic policy with the upper confidence bound procedure to construct on online policy. The proposed online policy is proved to approximately have [Formula: see text] regret against the offline benchmark. Our framework and policy design may shed light on broader offline planning and online learning applications with nonstationary and recovering rewards. This paper was accepted by J. George Shanthikumar, data science. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2021.04202 .

Get a Head Start: On-Demand Pedagogical Policy Selection in Intelligent Tutoring

Reinforcement learning (RL) is broadly employed in human-involved systems to enhance human outcomes. Off-policy evaluation (OPE) has been pivotal for RL in those realms since online policy learning and evaluation can be high-stake. Intelligent tutoring has raised tremendous attentions as highly challenging when applying OPE to human-involved systems, due to that students' subgroups can favor different pedagogical policies and the costly procedure that policies have to be induced fully offline and then directly deployed to the upcoming semester. In this work, we formulate on-demand pedagogical policy selection (ODPS) to tackle the challenges for OPE in intelligent tutoring. We propose a pipeline, EduPlanner, as a concrete solution for ODPS. Our pipeline results in an theoretically unbiased estimator, and enables efficient and customized policy selection by identifying subgroups over both historical data and on-arrival initial logs. We evaluate our approach on the Probability ITS that has been used in real classrooms for over eight years. Our study shows significant improvement on learning outcomes of students with EduPlanner, especially for the ones associated with low-performing subgroups.

Network Revenue Management With Demand Learning and Fair Resource-Consumption Balancing

In addition to maximizing the total revenue, decision-makers in lots of industries would like to guarantee balanced consumption across different resources. For instance, in the retailing industry, ensuring a balanced consumption of resources from different suppliers enhances fairness and helps maintain a healthy channel relationship; in the cloud computing industry, resource-consumption balance helps increase customer satisfaction and reduce operational costs. Motivated by these practical needs, this paper studies the price-based network revenue management (NRM) problem with both demand learning and fair resource-consumption balancing. We introduce the regularized revenue, that is, the total revenue with a balancing regularization, as our objective to incorporate fair resource-consumption balancing into the revenue maximization goal. We propose a primal-dual-type online policy with the upper-confidence-bound demand learning method to maximize the regularized revenue. We adopt several innovative techniques to make our algorithm a unified and computationally efficient framework for the continuous price set and a wide class of balancing regularizers. Our algorithm achieves a worst-case regret of [Formula: see text], where [Formula: see text] denotes the number of products and [Formula: see text] denotes the number of time periods. Numerical experiments in a few NRM examples demonstrate the effectiveness of our algorithm in simultaneously achieving revenue maximization and fair resource-consumption balancing.

Online Policy Learning-Based Output-Feedback Optimal Control of Continuous-Time Systems

Online Policy Learning-Based Output-Feedback Optimal Control of Continuous-Time Systems

Online Adaptive Dynamic Programming for Optimal Self-Learning Control of VTOL Aircraft Systems With Disturbances

In this paper, a novel online adaptive dynamic programming control algorithm is developed to solve optimal control problems for Vertical Take Off and Landing (VTOL) aircraft systems. Considering the input disturbances of the VTOL systems and adding perturbation interference term to the value function, a robust policy iteration-based ADP method will be introduced to obtain the optimal controller with an approximate optimal control approach. The convergence property is developed to ensure the value function converges to a finite neighborhood of the optimal one. The uniform ultimate boundedness (UUB) of the iterative errors will be strictly proved by Lyapunov stability theory. Finally, we will give mathematical simulation results with the developed method. Compared with sliding mode control (SMC) and linear quadratic regulator (LQR) methods, the simulation results illustrate better overall performance of the novel method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —As an effective tool to solve the optimal control problem of complex nonlinear systems, adaptive dynamic programming (ADP) can effectively overcome the computational problems caused by “curse of dimensionality”. VTOL aircraft system is a typical strongly coupled, underactuate, non-minimum phase system, and the sudden random disturbances will make the control task more challenging. Aiming at the particularity of VTOL system, a new online policy iteration-based ADP algorithm is developed to obtain optimal control with an approximate optimal control strategy in this paper. Decoupling and disturbances can be effectively avoided. convergence will be analyzed to make sure the value function converge to a finite neighborhood of the optimal value function. The UUB property of the system will be proved by the Lyapunov approach.

Event-Triggered Robust Adaptive Dynamic Programming for Multiplayer Stackelberg-Nash Games of Uncertain Nonlinear Systems.

In this article, an event-triggered robust adaptive dynamic programming (ETRADP) algorithm is developed to solve a class of multiplayer Stackelberg-Nash games (MSNGs) for uncertain nonlinear continuous-time systems. Considering the different roles of players in the MSNG, the hierarchical decision-making process is described as the designed value functions for the leader and all followers, which assist to transform the robust control problem of the uncertain nonlinear system into an optimal regulation problem of the nominal system. Then, an online policy iteration algorithm is formulated to solve the derived coupled Hamilton-Jacobi equation. Meanwhile, an event-triggered mechanism is designed to alleviate computational and communication burdens. Moreover, critic neural networks (NNs) are constructed to obtain the event-triggered approximate optimal control polices for all players, which constitute the Stackelberg-Nash equilibrium of the MSNG. By using Lyapunov's direct method, the stability of the closed-loop uncertain nonlinear system is guaranteed under the ETRADP-based control scheme in the sense of uniform ultimate boundedness. Finally, a numerical simulation is provided to demonstrate the effectiveness of the present ETRADP-based control scheme.

Policy Iteration Q -Learning for Linear Itô Stochastic Systems With Markovian Jumps and Its Application to Power Systems.

This article addresses the solution of continuous-time linear Itô stochastic systems with Markovian jumps using an online policy iteration (PI) approach grounded in Q -learning. Initially, a model-dependent offline algorithm, structured according to traditional optimal control strategies, is designed to solve the algebraic Riccati equation (ARE). Employing Lyapunov theory, we rigorously derive the convergence of the offline PI algorithm and the admissibility of the iterative control law through mathematical analysis. This article represents the first attempt to tackle these technical challenges. Subsequently, to address the limitations inherent in the offline algorithm, we introduce a novel online Q -learning algorithm tailored for Itô stochastic systems with Markovian jumps. The proposed Q -learning algorithm obviates the need for transition probabilities and system matrices. We provide a thorough stability analysis of the closed-loop system. Finally, the effectiveness and applicability of the proposed algorithms are demonstrated through a simulation example, underpinned by the theorems established herein.

NVIF: Neighboring Variational Information Flow for Cooperative Large-Scale Multiagent Reinforcement Learning.

Communication-based multiagent reinforcement learning (MARL) has shown promising results in promoting cooperation by enabling agents to exchange information. However, the existing methods have limitations in large-scale multiagent systems due to high information redundancy, and they tend to overlook the unstable training process caused by the online-trained communication protocol. In this work, we propose a novel method called neighboring variational information flow (NVIF), which enhances communication among neighboring agents by providing them with the maximum information set (MIS) containing more information than the existing methods. NVIF compresses the MIS into a compact latent state while adopting neighboring communication. To stabilize the overall training process, we introduce a two-stage training mechanism. We first pretrain the NVIF module using a randomly sampled offline dataset to create a task-agnostic and stable communication protocol, and then use the pretrained protocol to perform online policy training with RL algorithms. Our theoretical analysis indicates that NVIF-proximal policy optimization (PPO), which combines NVIF with PPO, has the potential to promote cooperation with agent-specific rewards. Experiment results demonstrate the superiority of our method in both heterogeneous and homogeneous settings. Additional experiment results also demonstrate the potential of our method for multitask learning.

Sequential learning based re-optimization approaches for less model-based dynamic pick-up routing problem

We address a lessmodel-based dynamic routing problem arising from home parcel pick-up service, where lessmodel-based means existing customers who dynamically request services independently following Poisson process with a stochastic rate. Overall, through an extended application of re-optimization (RO) strategy, a Markov decision process formulation and approximation dynamic programming- and Bayes' theorem-based solution approaches are proposed. Specifically, first a pool of basic policies corresponding to all possible values of the rate are developed offline via approximate value iteration. Then, Bayes' theorem-based sequential learning is designed that can sequentially update the belief about the rate's probability distribution over its possible values. Third, coupled with the updated belief, basic policies are collectively implemented in two different ways, resulting in RO approaches which involve constructions of two different online policies, i.e. a belief-weighted deterministic policy and a belief-based random policy, and their re-optimizations at decision epochs. In the numerical study, through comparison with model-based (i.e. using full knowledge of the rate) and model-free heuristics, our approaches are examined and valuable insights are obtained. Important insights include that (i) the belief-weighted deterministic policy outperforms the belief-based random policy, and further, (ii) the former is better than the latter at preserving the improvement resulting from the improved model-based policy.

Parental leave policies in general surgery residencies

BackgroundProspective residents use program websites to glean information regarding parental leave policies. This study investigates the online availability and content of parental leave policies for general surgery residency programs. MethodsParental leave policy information was collected from general surgery residency program and Graduate Medical Education (GME) websites. Descriptive statistics and multivariable logistic regression were used for analysis. ResultsOf the 344 general surgery residency programs, parental leave policies were found on 6% of program and 52% of GME websites. Family Medical Leave Act policies were reported the most, followed by maternity, then paternity, and then adoption/other clauses. Academic programs, program location in the Southeastern US and larger program size were all significant predictors of online policy availability. ConclusionsGeneral surgery parental leave policies vary and are not readily available online. These findings identify a significant opportunity for surgery residency programs to improve the disclosure of parental leave policy information.

Coping with Teacher Professional Identity Tensions: Using TPACK During Global Crisis Time

ABSTRACT Sudden teachers’ transition to an online context has remarkably influenced their professional identity construction. However, little is known about the tensions English Language Teachers (ELTs) experience during this shift. The present study examined ELTs’ identity tensions in online courses, the relevant coping strategies, teachers’ TPACK level and its relationship with their professional identity development. The data were collected through a TPACK/teacher identity questionnaire and semi-structured interviews. Descriptive data analysis revealed that teachers had a high level of TPACK with moderate technological knowledge. The teacher identity analysis indicated that the teachers reported institutional, pedagogical, and sociocultural tensions with varying degrees. Likewise, the qualitative analysis revealed similar tensions, including low access to basic infrastructure and professional support, greater responsibility, top-down online policies, etc. The teachers also reported various coping strategies for these tensions. The study concluded with pedagogical implications for policymakers, administrators, and teacher educators. To support English language teachers’ professionalism, there is a need to rethink policies, facilities, and contextual factors and promote effective technology integration into course content and pedagogies.

The Causes and Measures of Online Deviance

The Internet as a virtual space facilitates the phenomenon of people’s deviant behavior. Due to its enormous scale and versatility, it plays an indispensable role in modern society. It is important to maintain online environment and cybersecurity is now a major challenge to society and traditional law enforcement. How is the Internet fueled deviant behavior? What’s the difficulties of dealing with online deviance? How to implement intervention without damaging the benefits from the network? This study explains the causes of virtual deviance and gives measures, through the nature of the Internet to answer these questions. Online deviance is influenced by factors like online disinhibition, including benign and toxic disinhibition, which can lead to aggressive behavior due to the absence of immediate feedback. The Internet's characteristics, such as dissociative anonymity and minimized status and authority, worsen online disinhibition, resulting in behaviors like online harassment. Low costs and lenient enforcement contribute to the proliferation of online deviance. Accessible tools and tutorials for cyber deviance, along with online anonymity, further encourage such behavior, creating a self-perpetuating cycle. Defining online deviance is challenging due to its global nature and constantly evolving forms. Laws and definitions vary among countries, and the anonymous nature of online activities hampers identification and classification. Gathering evidence for cyber deviance cases is also complex, as electronic evidence can be tampered with or deleted. It suggests that law enforcement should pay more attention to deal with the reports from Internet users, by implementing targeted warnings and cautions in real world to stop potential criminals, as well as help potential victims be more effective in defending their rights. It also mentions an intervention from family regulatory layer can protect adolescents against deviant behavior from online community. The ideal control effect may be expected to differ, depends on local agencies’ capacity and different online policies. This study aims to elucidate the phenomenon of virtual deviance and provide feasible methods to improve online environment. The problems in online society need to be taken seriously.