Switching Policy Research Articles

A Multi-Agent Reinforcement Learning Method for Omnidirectional Walking of Bipedal Robots.

Achieving omnidirectional walking for bipedal robots is considered one of the most challenging tasks in robotics technology. Reinforcement learning (RL) methods have proved effective in bipedal walking tasks. However, most existing methods use state machines to switch between multiple policies and achieve omnidirectional gait, which results in shaking during the policy switching process for bipedal robots. To achieve a seamless transition between omnidirectional gait and transient motion for full-size bipedal robots, we propose a novel multi-agent RL method. Firstly, a multi-agent RL algorithm based on the actor-critic framework is designed, and policy entropy is introduced to improve exploration efficiency. By learning agents with parallel initial state distributions, we minimize reliance on gait planner effectiveness in the Robot Operating System (ROS). Additionally, we design a novel heterogeneous policy experience replay mechanism based on Euclidean distance. Secondly, considering the periodicity of bipedal robot walking, we develop a new periodic gait function. Including periodic objectives in the policy can accelerate the convergence speed of training periodic gait functions. Finally, to enhance the robustness of the policy, we construct a novel curriculum learning method by discretizing Gaussian distribution and incorporate it into the robot's training task. Our method is validated in a simulation environment, and the results show that our method can achieve multiple gaits through a policy network and achieve smooth transitions between different gaits.

Disturbance Observer-based Pointing Control of Leighton Chajnantor Telescope

Leighton Chajnantor Telescope (LCT), i.e., the former Caltech Submillimeter Observatory telescope, will be refurbished at the new site in Chajnantor Plateau, Chile in 2023. The environment of LCT will change significantly after its relocation, and the telescope will be exposed to large wind disturbances directly because its enclosure will be completely open during observation. The wind disturbance is expected to be a challenge for LCT’s pointing control since the existing control method cannot reject this disturbance very well. Therefore, it is very necessary to develop a new pointing control method with good capability of disturbance rejection. In this research, a disturbance observer—based composite position controller (DOB-CPC) is designed, in which an H∞ feedback controller is employed to compress the disturbance, and a feedforward linear quadratic regulator is employed to compensate the disturbance precisely based on the estimated disturbance signal. Moreover, a controller switching policy is adopted, which applies the proportional controller to the transient process to achieve a quick response and applies the DOB-CPC to the steady state to achieve a small position error. Numerical experiments are conducted to verify the good performance of the proposed pointing controller (i.e., DOB-CPC) for rejecting the disturbance acting on LCT.

A novel condition‐based maintenance policy considering imperfect switching for warm standby systems

AbstractThis paper proposes a novel condition‐based maintenance model for a two‐component warm standby system considering an imperfect switching policy. Unlike the conventional models, joint optimization of preventive and failure switching strategies is investigated. The standby component can be switched preventively when the component reaches the preventive maintenance (PM) threshold before the online one fails. The failure switching is triggered when the online component reaches the corrective maintenance (CM) threshold. In addition, the switching mechanism between the two components is related to the order fulfillment rate. The optimal PM threshold and order fulfillment rate can be derived by minimizing the long‐term average cost rate. A numerical example is provided to demonstrate the robustness of the decision model under different scenarios through sensitivity analysis. Ultimately, the effectiveness of the proposed model in this article was verified by comparing the models.

The dynamic effect of public debt on economic growth in the era of Macroprudential policy regime: a Bayesian approach

Purpose This study aims to test the dynamic impact of public debt and economic growth on newly democratized African countries (South Africa and Namibia) and compare the findings with those of newly democratized European countries (Germany and Ukraine) during the period 1990–2022. Design/methodology/approach The methodology involves three stages: identifying the appropriate transition variable, assessing the linearity between public debt and economic growth and selecting the order m of the transition function. The linearity test helps identify the nature of relationships between public debt and economic growth. The wild cluster bootstrap-Lagrange Multiplier test is used to evaluate the model’s appropriateness. All these tests would be executed using the Lagrange Multiplier type of test. Findings The results signify the policy switch, as the authors find that the relationship between public debt and economic growth is characterized by two transitions that symbolize that the current stage of the relationship is beyond the U-shape; however, an S-shape. The results show that for newly democratized African countries, the threshold during the first waves was 50% of GDP, represented by a U-shape, which then transits to an inverted U-shape with a threshold of 65% of GDP. Then, for the European case, it was 60% of GDP, which is now 72% of GDP. Originality/value The findings suggest that an escalating level of public debt has a negative impact on economic growth; therefore, it is important to implement fiscal discipline, prioritize government spending and reduce reliance on debt financing. This can be achieved by focusing on revenue generation, implementing effective taxation policies, reducing wasteful expenditures and promoting investment and productivity-enhancing measures.

Rational social distancing policy during epidemics with limited healthcare capacity.

Epidemics of infectious diseases posing a serious risk to human health have occurred throughout history. During recent epidemics there has been much debate about policy, including how and when to impose restrictions on behaviour. Policymakers must balance a complex spectrum of objectives, suggesting a need for quantitative tools. Whether health services might be 'overwhelmed' has emerged as a key consideration. Here we show how costly interventions, such as taxes or subsidies on behaviour, can be used to exactly align individuals' decision making with government preferences even when these are not aligned. In order to achieve this, we develop a nested optimisation algorithm of both the government intervention strategy and the resulting equilibrium behaviour of individuals. We focus on a situation in which the capacity of the healthcare system to treat patients is limited and identify conditions under which the disease dynamics respect the capacity limit. We find an extremely sharp drop in peak infections at a critical maximum infection cost in the government's objective function. This is in marked contrast to the gradual reduction of infections if individuals make decisions without government intervention. We find optimal interventions vary less strongly in time when interventions are costly to the government and that the critical cost of the policy switch depends on how costly interventions are.

Optimal switching policy for batch servers

We consider a queueing system with two Poisson arrival queues and two batch servers with finite capacities within a pre-specified time horizon. The problem is to decide a switching time of the two servers in order to maximize the total expected number of customers served in the end. We propose an optimal time threshold-based switching policy and analyze the properties of the switching policy. We also study the asymptotic benefit of having the switching option. Finally, we show the superior performance of the switching policy by numerical experiments.

Towards a model of human-cyber–physical automata and a synthesis framework for control policies

Advances in research and increasing applications of Cyber–Physical Systems (CPSs) show the need to consider factors of humans in the loop. This has led to the growing research focus on Human-Cyber–Physical Systems (HCPSs). In general, humans in an HCPS interact with both the cyber and physical systems, as well as among the humans themselves. For a better understanding, correct design, development, operation, and maintenance of HCPSs, a computational theory based on a computational model is required. This paper presents our initial work towards a model of human-cyber–physical automata (HCPA). We consider an HCPS as a combination of a human-physical system (HPS) and a CPS in which the control switches between the humans and the machines. We define an HCPA by connecting the automaton of the HPS and the automaton of the CPS through a switch control automaton. The switch control automaton makes switching decision in some critical states shared by the HPS and the CPS. Our theorem shows that the control switching between the HPS and the CPS increases the probability of satisfying a given property. We model the behaviour of a human in specified applications or even in carry out specific tasks, instead of general human intelligence. Therefore, a human can make mistakes to decision making and thus it is a probabilistic automaton with learning ability. The switching between the human and the machine is modelled by an oracle. The oracle learns about the human behaviour, the machine behaviour, as well as the environment to make the control decisions. To generate the control policies of the human and the oracle, we propose a synthesis framework to maximize the probability of the satisfaction of a property specified in Linear Temporal Logic (LTL) by the HCPA. We present a prototype implementation of the framework by extending the model-free reinforcement learning (RL) algorithm and model-free deep-RL algorithm, and our experiment shows that our synthesis framework is effective in obtaining switch policies.

Reinforcement Learning Model for Managing Noninvasive Ventilation Switching Policy.

Noninvasive ventilation (NIV) has been recognized as a first-line treatment for respiratory failure in patients with chronic obstructive pulmonary disease (COPD) and hypercapnia respiratory failure, which can reduce mortality and burden of intubation. However, during the long-term NIV process, failure to respond to NIV may cause overtreatment or delayed intubation, which is associated with increased mortality or costs. Optimal strategies for switching regime in the course of NIV treatment remain to be explored.For the goal of reducing 28-day mortality of the patients undergoing NIV, Double Dueling Deep Q Network (D3QN) of offline-reinforcement learning algorithm was adopted to develop an optimal regime model for making treatment decisions of discontinuing ventilation, continuing NIV, or intubation. The model was trained and tested using the data from Multi-Parameter Intelligent Monitoring in Intensive Care III (MIMIC-III) and evaluated by the practical strategies. Furthermore, the applicability of the model in majority disease subgroups (Catalogued by International Classification of Diseases, ICD) was investigated. Compared with physician's strategies, the proposed model achieved a higher expected return score (4.25 vs. 2.68) and its recommended treatments reduced the expected mortality from 27.82% to 25.44% in all NIV cases. In particular, for these patients finally received intubation in practice, if the model also supported the regime, it would warn of switching to intubation 13.36 hours earlier than clinicians (8.64 vs. 22 hours after the NIV treatment), granting a 21.7% reduction in estimated mortality. In addition, the model was applicable across various disease groups with distinguished achievement in dealing with respiratory disorders. The proposed model is promising to dynamically provide personalized optimal NIV switching regime for patients undergoing NIV with the potential of improving treatment outcomes.

An antenna subset selection algorithm in distributed MIMO radar for target localization via convolutional neural network

AbstractSince the distributed MIMO radar (DMR) has widely spread transmitters and receivers, it can provide higher target detection probability, as well as superior target tracking and localization performance than the monostatic/bistatic radar systems. An effective radar resource allocation scheme can optimise the DMR system parameter and obtain better system performance. In this paper, a critical but limited system resource is optimised, that is, select a subset of the active radar antennas. In this scenario, the convolutional neural network of the antenna subset selection for target localization (LCNN‐ASS) algorithm in the DMR is proposed based on two free switching policies. The proposed algorithm is immune to the failure of a single policy and selects antenna subsets from the entire sets with a remarkable computational speed. Therefore, the proposed algorithm increases the flexibility of resource scheduling over traditional algorithms. Simulation experiments and performance analysis demonstrate the localization performance and flexibility of the LCNN‐ASS algorithm.

Hierarchical Maneuver Decision Method Based on PG-Option for UAV Pursuit-Evasion Game

Aiming at the autonomous decision-making problem in an Unmanned aerial vehicle (UAV) pursuit-evasion game, this paper proposes a hierarchical maneuver decision method based on the PG-option. Firstly, considering various situations of the relationship of both sides comprehensively, this paper designs four maneuver decision options: advantage game, quick escape, situation change and quick pursuit, and the four options are trained by Soft Actor-Critic (SAC) to obtain the corresponding meta-policy. In addition, to avoid high dimensions in the state space in the hierarchical model, this paper combines the policy gradient (PG) algorithm with the traditional hierarchical reinforcement learning algorithm based on the option. The PG algorithm is used to train the policy selector as the top-level strategy. Finally, to solve the problem of frequent switching of meta-policies, this paper sets the delay selection of the policy selector and introduces the expert experience to design the termination function of the meta-policies, which improves the flexibility of switching policies. Simulation experiments show that the PG-option algorithm has a good effect on UAV pursuit-evasion game and adapts to various environments by switching corresponding meta-policies according to current situation.

Dissipativity for Discrete-Time Switched Positive Delay Systems: A Dwell-Time-Dependent Linear Copositive Storage Functional Method

This article addresses the problems of dissipativity analysis and feedback dissipativity of discrete-time switched positive systems with time-varying delay. On the basis of the positivity property of the investigated system, the definition of linear <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(q,r)$</tex-math> </inline-formula> - <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\alpha$</tex-math> </inline-formula> -dissipativity, utilizing the linear copositive storage functionals with linear supply rates, is first introduced. The switching among subsystems is orchestrated by a state-dependent switching policy that obeys prespecified dwell-time constraints. Second, in order to reduce the conservatism brought by the identical summation term functional for all positive subsystems, we propose a dwell-time-dependent linear copositive storage functionals involving summation terms which are diverse appearing in the functional. Then, less conservative criteria are developed to guarantee the system is strict <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(q,r)$</tex-math> </inline-formula> - <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\alpha$</tex-math> </inline-formula> -dissipativity by resorting to the switching policy or the joint implementation of the switching policy and time-varying controllers for subsystems. Finally, the validity of the proposed techniques is illustrated through two examples, and the relationship among the minimum dwell time, performance index, and time-varying delay is also revealed.

On Single-Input Dual-Output (SIDO) DC/DC Multi-Port Converters for DC Microgrid Applications

This paper proposes the use of linear-assisted switching power converters in the context of single-inductor dual-output (SIDO) for microgrids (MG) applications. By combining a DC/DC ripple-controlled switching power converter with the respective voltage linear regulators at each output, improved performance in terms of load and line regulations is obtained. To achieve that aim, a current-steering switching policy is proposed, together with a resource-aware circuit implementation. The ripple-based hysteretic control results in variable switching frequency to guarantee critical conduction mode (boundary of CCM and DCM).

P21 Promoting antimicrobial stewardship and sustainability utilizing a ‘Go Green’ IV to oral switch (IVOST) policy and electronic prescribing prompts in antimicrobials with high oral bioavailability

Abstract Background Antimicrobial stewardship (AS) is crucial in reducing antimicrobial resistance (AMR) and the sustainability impact of inappropriate antimicrobial use.1 Promotion of the oral route and IV to oral switch (IVOST) are key AS initiatives providing benefits for patients, staff and the environment including reduced risk of line infections and AMR, reduced workload and costs, reduced plastic waste and improved sustainability.2 A recent quality improvement (QI) initiative to promote oral metronidazole (bioavailability &amp;gt; 90%) in eligible surgical patients at the Royal Alexandra Hospital (RAH) resulted in a change in prescribing behaviour and a 45% median reduction in IV metronidazole use.3 Objectives To scale and spread promotion of the oral route in antimicrobials with high oral bioavailability with the aim of reducing collective IV use by 20% by February 2023 at RAH. Methods A QI approach was used to promote the oral route where appropriate in the following agents: metronidazole, levofloxacin, fluconazole, ciprofloxacin, co-trimoxazole, clindamycin, linezolid and rifampicin. Techniques included the following: ‘Go Green IVOST’ posters in key ward areas highlighting benefits of the oral route; targeted pharmacist and pharmacy technician prospective audit and feedback using the electronic prescribing system; presentations to ward and pharmacy teams; staff champions and electronic IVOST prompts displayed when prescribing and administering the targeted agents IV. To measure improvement, oral and IV usage data were measured at baseline and for 24 months post change. Data were displayed on run charts to determine if the QI target was achieved. Results There was a 44% median reduction in IV defined daily doses (DDDs) in the targeted antimicrobials, surpassing the 20% target (Figure 1). There was a median 10% shift in the proportion of these agents given IV as a percentage of total (IV plus oral) use (Figure 2) The greatest reduction (60%) was in IV metronidazole where usage declined from a monthly baseline of 667 to 267 DDDs. This equates to a median reduction of 1200 IV metronidazole administrations and nearly 600 hours nursing time saved per month. IV fluconazole and IV levofloxacin usage declined from baseline by a median of 35% and 24%, respectively. Conclusions A QI approach to promote appropriate use of the oral route in antimicrobials with high oral bioavailability resulted in a change in prescribing behaviour and a sustained reduction in IV prescribing of these agents. This is important in terms of improved AS, patient care and safety, healthcare cost and efficiency and environmental sustainability.

EE426 A Comparison of the Cost-Saving Impact of Biosimilar Switching Policies across Canadian Provinces: A Case-Study for a Ranibizumab Biosimilar

This examined the impact of differing ophthalmology biosimilar switching policies on cost-savings for Canadian payors. The anticipated budget impact of a ranibizumab biosimilar was estimated across ten provincial drug plans. Resulting analysis identified possible opportunities for increased drug plan cost-savings associated with adopting proactive biosimilar switching policies.

Adaptive Context Caching for IoT-Based Applications: A Reinforcement Learning Approach.

Making internet-of-things (IoT)-based applications context-aware demands large amounts of raw data to be collected, interpreted, stored, and reused or repurposed if needed from many domains and applications. Context is transient but interpreted data can be distinguished from IoT data in many aspects. Managing context in cache is a novel area of research that has been given very little attention. Performance metric-driven adaptive context caching (ACOCA) can have a profound impact on the performance and cost efficiency of context-management platforms (CMPs) when responding to context queries in realtime. Our paper proposes an ACOCA mechanism to maximize both the cost and performance efficiency of a CMP in near realtime. Our novel mechanism encompasses the entire context-management life cycle. This, in turn, distinctively addresses the problems of efficiently selecting context for caching and managing the additional costs of context management in the cache. We demonstrate that our mechanism results in long-term efficiencies for the CMP that have not been observed in any previous study. The mechanism employs a novel, scalable, and selective context-caching agent implemented using the twin delayed deep deterministic policy gradient method. It further incorporates an adaptive context-refresh switching policy, a time-aware eviction policy, and a latent caching decision management policy. We point out in our findings that the additional complexity of adaptation introduced to the CMP through ACOCA is significantly justified, considering the cost and performance gains achieved. Our algorithm is evaluated using a real-world inspired heterogeneous context-query load and a data set based on parking-related traffic in Melbourne, Australia. This paper presents and benchmarks the proposed scheme against traditional and context-aware caching policies. We demonstrate that ACOCA outperforms the benchmarks in both cost and performance efficiency, i.e., up to 68.6%, 84.7%, and 67% more cost efficient compared to traditional data caching policies to cache context, redirector mode, and context-aware adaptive data caching under real-world-like circumstances.

Switch-Off Policies in Job Shop Controlled by Workload Control Concept

The reduction in emissions and the increase in energy costs push companies to identify solutions to reduce energy consumption in production systems. One of the approaches proposed in the literature is the shutdown of machines to reduce energy consumption in the idle state. This solution does not affect production processes and can be applied in various manufacturing fields. This paper proposes switch-off policies in manufacturing systems under a workload control system. The shutdown policies developed consider the number of items in the queue and the calculation derived from the workload control mechanism. Simulation models have been developed to test the proposed policies using the case always on as a benchmark, considering different levels of absorbed power in the inactivity and warm-up states and different warm-up times. The results highlight how the switch policies that include the workload evaluation drastically reduce the number of on/off activities, assuring lower energy consumption.

Workload analysis of a two-queue fluid polling model

AbstractIn this paper, we analyze a two-queue random time-limited Markov-modulated polling model. In the first part of the paper, we investigate the fluid version: fluid arrives at the two queues as two independent flows with deterministic rate. There is a single server that serves both queues at constant speeds. The server spends an exponentially distributed amount of time in each queue. After the completion of such a visit time to one queue, the server instantly switches to the other queue, i.e., there is no switch-over time.For this model, we first derive the Laplace–Stieltjes transform (LST) of the stationary marginal fluid content/workload at each queue. Subsequently, we derive a functional equation for the LST of the two-dimensional workload distribution that leads to a Riemann–Hilbert boundary value problem (BVP). After taking a heavy-traffic limit, and restricting ourselves to the symmetric case, the BVP simplifies and can be solved explicitly.In the second part of the paper, allowing for more general (Lévy) input processes and server switching policies, we investigate the transient process limit of the joint workload in heavy traffic. Again solving a BVP, we determine the stationary distribution of the limiting process. We show that, in the symmetric case, this distribution coincides with our earlier solution of the BVP, implying that in this case the two limits (stationarity and heavy traffic) commute.

On the Capacity Region of Bipartite and Tripartite Entanglement Switching

We study a quantum entanglement distribution switch serving a set of users in a star topology with equal-length links. The quantum switch, much like a quantum repeater, can perform entanglement swapping to extend entanglement across longer distances. Additionally, the switch is equipped with entanglement switching logic, enabling it to implement switching policies to better serve the needs of the network. In this work, the function of the switch is to create bipartite or tripartite entangled states among users at the highest possible rates at a fixed ratio. Using Markov chains, we model a set of randomized switching policies. Discovering that some are better than others, we present analytical results for the case where the switch stores one qubit per user, and find that the best policies outperform a time division multiplexing policy for sharing the switch between bipartite and tripartite state generation. This performance improvement decreases as the number of users grows. The model is easily augmented to study the capacity region in the presence of quantum state decoherence and associated cut-off times for qubit storage, obtaining similar results. Moreover, decoherence-associated quantum storage cut-off times appear to have little effect on capacity in our identical-link system. We also study a smaller class of policies when the switch stores two qubits per user.

Uptake and Spending on Biosimilar Infliximab and Etanercept After New Start and Switching Policies in Canada: An Interrupted Time Series Analysis.

Uptake of biosimilars has been suboptimal in North America. This study was undertaken to quantify the impact of various policy interventions (namely, new start and switching policies) on uptake and spending on biosimilar infliximab and etanercept in British Columbia (BC), Canada. We used administrative claims data to identify BC residents ≥18 years of age with rheumatoid arthritis, ankylosing spondylitis, psoriatic arthritis, and/or plaque psoriasis who qualified for public drug coverage from January 2013 to November 2020. Using interrupted time series analysis, we studied the change in proportion spent on and prescriptions dispensed of biosimilar infliximab and etanercept out of the total amount per agent after new start and biosimilar switching policies were implemented. Our study included 208,984 individuals living with rheumatoid arthritis, ankylosing spondylitis, plaque psoriasis, and/or psoriatic arthritis, corresponding to 5,884 patients taking infliximab and etanercept. After the new start policy, we detected a small gradual increase in the proportion of dispensed biosimilar etanercept prescriptions of 0.65% per month (95% confidence interval [95% CI] 0.44, 0.85). The trend related to the proportion of total spending on biosimilar etanercept also increased (0.51% [95% CI 0.28, 0.73]). After the switching policy, there was a sustained increase in the proportion of dispensed biosimilar etanercept and infliximab prescriptions of 76.98% (95% CI 75.56, 78.41) and 58.43% (95% CI 52.11, 64.75), respectively. Similarly, there was a persistent increase in monthly spending on biosimilar etanercept and infliximab of 78.22% (95% CI 76.65, 79.79) and 71.23% (95% CI 66.82, 75.65), respectively. We found that mandatory switching policies were much more effective than new starting policies for increasing the use of biosimilar medications.

Energy-Efficient Opportunistic Spectrum Access in Multichannel Cognitive Radio Networks

We study an opportunistic spectrum access problem in multichannel Cognitive Radio Networks in which the secondary user (SU) has an energy cost when switching across channels. Channel switching from one RF chain to another also involves a reconfiguration delay that results in a shorter duration of transmission opportunity for the SU. For an imperfect sensing, we formulate the spectrum access problem in the framework of Partially Observable Markov Decision Process from which we obtain an average energy optimal policy. Also, we propose Efficacy - Efficient channel access with switching policy, for channel sensing and access based on the structure of the optimal policy. We compare the optimum policy with Efficacy, and show that Efficacy has a near optimal average energy performance.