Dynamic Control Policy Research Articles

Data-driven-based sliding-mode dynamic event-triggered control of unknown nonlinear systems via reinforcement learning

This paper investigates a data-driven-based dynamic event-triggered control problem for continuous-time unknown nonlinear systems using the reinforcement learning method and the sliding-mode surface technique. Initially, by constructing a cost function associated with sliding-mode surface variables for the nominal system, the original control problem is equivalently transformed into a problem of designing a dynamic event-triggered optimal control policy. To handle the unknown issue of system dynamics, a data-driven model is established to reconstruct the system dynamics. Then, under the framework of reinforcement learning, a critic network is employed to solve the event-triggered Hamilton–Jacobi–Bellman equation. The weight vector in the critic network is updated through the current data and historical data, such that the persistence of excitation condition is no longer needed. After that, it is strictly proven via Lyapunov stability theory that all the signals of the considered system are bounded in the sense of uniformly ultimately boundedness. Finally, the effectiveness of the developed control method is demonstrated by two simulation examples.

Simple heuristics for the joint inventory and pricing models with fixed replenishment costs

This study considers the joint inventory and pricing problem of a firm selling a single item over a multi-period planning horizon. A fixed replenishment cost is incurred whenever the replenishment decision is made. Period demands are considered to be non-stationary, stochastic, and price-dependent. The literature shows that the optimal policy in such a system is of an (s, S, p)-type. The implementation of this type of policy might be challenging in practice due to the prevailing computational difficulty in determining the optimal policy parameters. This study, therefore, aims to develop computationally efficient heuristic approaches for the joint inventory and pricing problem. The proposed heuristics provide different levels of flexibility in making inventory and pricing decisions. Furthermore, they are essentially built on the concept of the replenishment cycle and do not require solving a stochastic dynamic program. Our numerical study demonstrates the profit efficiency of the proposed heuristic approaches against the optimal policy. The results also indicate that firms lacking the technical ability to employ a dynamic pricing strategy might still gain significant profit improvement by only using a dynamic inventory control policy.

N-Level Hierarchy-Based Optimal Control to Develop Therapeutic Strategies for Ecological Evolutionary Dynamics Systems.

This article mainly proposes an evolutionary algorithm and its first application to develop therapeutic strategies for ecological evolutionary dynamics systems (EEDS), obtaining the balance between tumor cells and immune cells by rationally arranging chemotherapeutic drugs and immune drugs. First, an EEDS nonlinear kinetic model is constructed to describe the relationship between tumor cells, immune cells, dose, and drug concentration. Second, the N-level hierarchy optimization (NLHO) algorithm is designed and compared with five algorithms on 20 benchmark functions, which proves the feasibility and effectiveness of NLHO. Finally, we apply NLHO into EEDS to give a dynamic adaptive optimal control policy and develop therapeutic strategies to reduce tumor cells, while minimizing the harm of chemotherapy drugs and immune drugs to the human body. The experimental results prove the validity of the research method.

A novel routing method for dynamic control in distributed computing power networks

Driven by diverse intelligent applications, computing capability is moving from the central cloud to the edge of the network in the form of small cloud nodes, forming a distributed computing power network. Tasked with both packet transmission and data processing, it requires joint optimization of communications and computing. Considering the diverse requirements of applications, we develop a dynamic control policy of routing to determine both paths and computing nodes in a distributed computing power network. Different from traditional routing protocols, additional metrics related to computing are taken into consideration in the proposed policy. Based on the multi-attribute decision theory and the fuzzy logic theory, we propose two routing selection algorithms, the Fuzzy Logic-Based Routing (FLBR) algorithm and the low-complexity Pairwise Multi-Attribute Decision-Making (lPMADM) algorithm. Simulation results show that the proposed policy could achieve better performance in average processing delay, user satisfaction, and load balancing compared with existing works.

QoS-Driven Slicing Management for Vehicular Communications

Network slicing is introduced for elastically instantiating logical network infrastructure isolation to support different application types with diversified quality of service (QoS) class indicators. In particular, vehicular communications are a trending area that consists of massive mission-critical applications in the range of safety-critical, intelligent transport systems, and on-board infotainment. Slicing management can be achieved if the network infrastructure has computing sufficiency, a dynamic control policy, elastic resource virtualization, and cross-tier orchestration. To support the functionality of slicing management, incorporating core network infrastructure with deep learning and reinforcement learning has become a hot topic for researchers and practitioners in analyzing vehicular traffic/resource patterns before orchestrating the steering policies. In this paper, we propose QoS-driven management by considering (edge) resource block utilization, scheduling, and slice instantiation in a three-tier resource placement, namely, small base stations/access points, macro base stations, and core networks. The proposed scheme integrates recurrent neural networks to trigger hidden states of resource availability and predict the output of QoS. The intelligent agent and slice controller, namely, RDQ3N, gathers the resource states from three-tier observations and optimizes the action on allocation and scheduling algorithms. Experiments are conducted on both physical and virtual representational vehicle-to-everything (V2X) environments; furthermore, service requests are set to massive thresholds for rendering V2X congestion flow entries.

A Dynamic Model for Managing Volunteer Engagement

Managing Volunteer Engagement Non-profit organizations that provide food, shelter, and other services to people in need, rely on volunteers to deliver their services. Unlike paid labor, non-profit organizations have less control over unpaid volunteers’ schedules, efforts, and reliability. However, these organizations can invest in volunteer engagement activities to strive for a steady and adequate supply of volunteer labor. The paper, “A Dynamic Model for Managing Volunteer Engagement”, models the volunteer management process. The paper derives a dynamic control policy that judiciously uses the non-profit’s resources for volunteer engagement activities depending on explicit congestion thresholds, thereby lowering the overall cost of the non-profit operation.

Observer-based dynamic event-triggered control for distributed parameter systems over mobile sensor-plus-actuator networks

Abstract The objective of this paper is to develop a policy of observer-based dynamic event-triggered state feedback control for distributed parameter systems over a mobile sensor-plus-actuator network. It is assumed that the mobile sensing devices that provide spatially averaged state measurements can be used to improve state estimation in the network. For the purpose of decreasing the update frequency of controller and unnecessary sampled data transmission, an efficient dynamic event-triggered control policy is constructed. In an event-triggered system, when an error signal exceeding a specified time-varying threshold it indicates the occurrence of a typical event. The global asymptotic stability of the event-triggered closed-loop system and the boundedness of the minimum inter-event time can be guaranteed. Based on the linear quadratic optimal regulator, the actuator selects the optimal displacement only when an event occurs. A simulation example is finally used to verify the effectiveness of such a control strategy can enhance the system performance.

Walking, the body, and the pandemic: the public value of walking art in China

ABSTRACT In December 2022, the dynamic zero-COVID control policy came to an end, marking the conclusion of a three-year pandemic that affected 1.4 billion Chinese people. The pandemic and related policies created a unique, temporary, and historic social ecosystem where walking became more crucial than ever before. The pandemic not only severely restricted people’s movement in public spaces but also exposed the longstanding contradictions between human bodies, modern mobility, and urban space. Over the three years of the pandemic, walking became an aesthetic survival attempt by Chinese people to cope with their limited freedoms under the pandemic. As the pandemic stagnated and worsened over time, walking-dominant activities gradually became a widespread social phenomenon that encouraged urban residents to participate in rebuilding society across various fields such as politics, art, nature, etc. The development of walking as an artistic form during this period represents a new aesthetic strategy and political awakening while reflecting humans’ need to reconnect with land, social space, and their own bodies. This paper reviews how walking art has evolved historically through three periods – before, during, and after the pandemic – aiming to highlight both the public value of walking art and challenges within China’s social ecosystem.

Multi-agent deep reinforcement learning with traffic flow for traffic signal control

Multi-agent Reinforcement Learning (MARL) has become one of the best methods in Adaptive Traffic Signal Control (ATSC). Traffic flow is a very regular traffic volume, which is highly critical to signal control policy. However, dynamic control policies will directly affect traffic flow formation, and it is impossible to provide observation through the original traffic flow prediction. This paper proposes a method for estimating traffic flow according to the time window in Reinforcement Learning (RL) training. Therefore, it is verified on both the regular road network and the real road network. Our method further reduces the intersection delay and queue length compared with the original method.

Event-Triggered Deep Reinforcement Learning Using Parallel Control: A Case Study in Autonomous Driving

This paper utilizes parallel control to investigate the problem of event-triggered deep reinforcement learning and develops an event-triggered deep Q-network (ETDQN) for decision-making of autonomous driving, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">without training an explicit triggering condition</i> . Based on the framework of parallel control, the developed ETDQN incorporates information of actions into the feedback and constructs a dynamic control policy. First, in the realization of the dynamic control policy, we integrate the current state and the previous action to construct the augmented state as well as the augmented Markov decision process. Meanwhile, it is shown theoretically that the goal of the developed dynamic control policy is to learn the variation rate of the action. The augmented state contains information of the current state and the previous action, which enables the developed ETDQN to directly design the immediate reward considering communication loss. Then, based on dueling double deep Q-network (dueling DDQN), we establish the augmented action-value, value, and advantage functions to directly learn the optimal event-triggered decision-making policy of autonomous driving without an explicit triggering condition. It is worth noticing that the developed ETDQN applies to various deep Q-networks (DQNs). Empirical results demonstrate that, in event-triggered control, the developed ETDQN outperforms dueling DDQN and reduces communication loss effectively. Source code of this paper is available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/lujingweihh/Event-triggered-deep-Q-network</uri> https://github.com/lujingweihh/Event-triggered-deep-Q-network.

Observer-Based Dynamic Event-Triggered Adaptive Control of Distributed Networked Systems With Application to Ground Vehicles

In this paper, a distributed observer-based event-based practical consensus control for networked systems subject to adaptive nonlinear couplings is investigated. Considering that the node's state is not always measurable and the finite communication bandwidth exists in engineering systems, two distributed adaptive event-triggered control protocols subject to nonlinear couplings are proposed under observer-based dynamic output feedback. Then, distributed dynamic event-triggered control policies corresponding to two above mentioned adaptive controllers are proposed, respectively. The proposed nonlinear coupled event-triggered control schemes not only overcome continuous communication among nodes, but also relax the requirement of relying on global information for achieving a distributed practical consensus. Finally, the theoretical analysis is verified through a team of practical networked ground vehicles implemented thorough Matlab/Similink and Robot Operating Systems (ROS) environment.

Optimizing Transportation Management with a Hybrid GNN-RL Approach

The process of rapid urbanization in contemporary cities has resulted in a rise in traffic congestion, accidents, and pollution, leading to negative impacts on transportation efficiency and urban development. In response, Intelligent Transportation Systems (ITS) have emerged as a crucial field of research and development, with a goal to leverage technological advancements to enhance transportation safety, efficiency, and sustainability. However, traditional expert systems and traffic control methods have displayed limitations in handling the complexities of urban transportation. To address these challenges in Advanced Traffic Management Systems (ATMSs), this paper pro- poses a new approach utilizing Graph Neural Network-Reinforcement Learning (GNN-RL). By using GNNs for pre-training and information aggregation from the entire grid, our method enables downstream RL tasks, allowing for fine-tuning of effective global information aggregation and dynamic traffic signal control policy adjustment. The GNN-RL approach offers enhanced adapt- ability and performance in managing long-term dependencies, rendering it a promising alternative for traffic control in contemporary urban transportation systems.

End-to-End Deep Reinforcement Learning Control for HVAC Systems in Office Buildings

The heating, ventilation, and air conditioning (HVAC) system is a major energy consumer in office buildings, and its operation is critical for indoor thermal comfort. While previous studies have indicated that reinforcement learning control can improve HVAC energy efficiency, they did not provide enough information about end-to-end control (i.e., from raw observations to ready-to-implement control signals) for centralized HVAC systems in multizone buildings due to the limitations of reinforcement learning methods or the test buildings being single zones with independent HVAC systems. This study developed a model-free end-to-end dynamic HVAC control method based on a recently proposed deep reinforcement learning framework to control the centralized HVAC system of a multizone office building. By using the deep neural network, the proposed control method could directly take measurable parameters, including weather and indoor environment conditions, as inputs and control indoor temperature setpoints at a supervisory level. In some test cases, the proposed control method could successfully learn a dynamic control policy to reduce HVAC energy consumption by 12.8% compared with the baseline case using conventional control methods, without compromising thermal comfort. However, an over-fitting problem was noted, indicating that future work should first focus on the generalization of deep reinforcement learning.

A parallel algorithm based on quantum annealing and double-elite spiral search for mixed-integer optimal control problems in engineering

Many engineering processes can be summarized as mixed-integer optimal control problems (MIOCPs) owing to the needs for optimizing mixed-integer dynamic control policies. However, MIOCP is a challenging NP-hard problem with great computational complexity, resulting in slow convergence or premature convergence by most current heuristics. Accordingly, this study explores a new and effective hybrid Quantum Annealing-Double-Elite Spiral Search (QA-DESS) algorithm to address this issue. To be specific, QA algorithm specializes in solving integer optimization with high efficiency due to the unique quantum-tunnelling-based annealing mechanism. For the optimization of continuous decisions, a DESS algorithm which adopts adaptive Cauchy mutation and double-elite evolutionary mechanism to enhance global searching is designed. The hybrid QA-DESS algorithm integrates the strengths of such algorithms to better balance the exploration and exploitation abilities. The overall evolution performs to find the optimal mixed-integer decisions by interactive parallel computing of QA and DESS. Simulation results on benchmark functions and practical engineering MIOCPs verify that the proposed optimization algorithm is more excel at finding promising results than other 7 heuristics (including traditional and state-of-the-art algorithms).

Stochastic Sequential Allocations for Creative Crowdsourcing

Creative crowdsourcing is an innovative online business model in which a platform marshals independent professionals (e.g., designers) to conduct creative work projects. Typically, clients submit project requests stochastically to a platform which possesses a pool of registered designers. For each submitted project, designers decide whether to participate and attempt to submit a design, and the client either chooses a winner among all submissions, or rejects them all, based on subjective criteria. In general, platforms cannot control individual designers directly because of the nature of the freelance market, incurring possible mismatches between designers and arriving projects. To tackle the problem, we present a dynamic control policy applied to the maximum number of participants for each arrived project. Our study reveals that the optimal policy follows an inverted‐ U‐shaped function of the project value, highlighting the importance of applying a stronger restriction on the number of participants for some sufficiently high‐valued projects. In addition, the optimal policy we have developed allows the platform to gain higher rewards judiciously even when the market is more volatile. Furthermore, extensive numerical studies have been conducted to glean managerial insights. Specifically, the optimal policy becomes more beneficial even when more designers are available, which is counter‐intuitive to the common‐sense notion that control should be more valuable when designers are scarcer; and the optimal policy is further shown to be robust when the objective is changed from maximizing the total reward to maximizing the total number of successful projects.

Reconfigurable manufacturing system scheduling: a deep reinforcement learning approach

Reconfigurable Manufacturing Systems (RMS) bring new possibilities toward meeting demand fluctuations while, at the same time, challenges scheduling efficiency. This paper presents a novel approach that, for the scheduling problem of RMS on multiple products, finds a dynamic control policy via a group of deep reinforcement learning agents. These teamed agents, embedded with a shared value decomposition network, aim on minimising the make-span of a constant updating order group by guiding a group of automated guided vehicles to move modules of machine, raw materials, and finished products inside the system.

Infection control operations of a large hospital in Taiwan to prevent nosocomial outbreaks during COVID-19 pandemic

BackgroundSevere acute respiratory syndrome coronavirus-2 (SARS-CoV-2) that causes coronavirus disease 2019 (COVID-19) is highly contagious, with a potential to cause large nosocomial outbreaks in the hospital setting. We report the advance deployment of comprehensive, multi-level infection control measures in a 3,700-bed large hospital to prevent nosocomial outbreaks of COVID-19 during the pandemic. MethodsWe implemented a series of dynamic infection control policies during the pandemic. A confirmed COVID-19 case was defined by positive real-time reverse transcriptase–polymerase chain reaction (RT-PCR) assay. All healthcare worker (HCW) having symptoms or close contact with the confirmed case received the RT-PCR test. ResultsA total of 5,722 patients were tested in our hospital from January to May 2020. Twenty-five patients were confirmed COVID-19, including two inpatients. A cluster of 4 HCWs with COVID-19 associated with the 2nd inpatient was identified in the early stage of epidemic. Our enhanced traffic control bundling, mask wearing, hand hygiene and environmental cleaning were reinforced after the outbreak. All other confirmed cases were identified at our outdoor quarantine station or epidemic clinic afterwards, and the results of testing for 146 symptomatic HCWs were all negative. ConclusionsIntegrated teamwork, advance deployment of infection control measures and efficient diagnostic testing and response protected HCW and facilities from large SARS-CoV-2 outbreaks and preserved the capacity and function of the health care system during the pandemic.

Optimal Control of Distributed Computing Networks With Mixed-Cast Traffic Flows

Distributed computing networks, tasked with both packet transmission and processing, require the joint optimization of communication and computation resources. We develop a dynamic control policy that determines both routes and processing locations for packets upon their arrival at a distributed computing network. The proposed policy, referred to as Universal Computing Network Control (UCNC), guarantees that packets i) are processed by a specified chain of service functions, ii) follow cycle-free routes between consecutive functions, and iii) are delivered to their corresponding set of destinations via proper packet duplications. UCNC is shown to be throughput-optimal for any mix of unicast and multicast traffic, and is the first throughput-optimal policy for non-unicast traffic in distributed computing networks with both communication and computation constraints. Moreover, simulation results suggest that UCNC yields substantially lower average packet delay compared with existing control policies for unicast traffic.

Integrated production, maintenance and quality control policy for unreliable manufacturing systems under dynamic inspection

This paper proposes a new joint production, maintenance, and dynamic sampling inspection control policy, for failure-prone manufacturing systems. The integrated policy help decision-makers to find both production rate and maintenance frequency, as well as the size of the sample to be inspected. The inspection policy is based on a dynamic sampling plan, which adapts the sample size according to the system degradation and takes into account the interactions between production, maintenance, and quality control. To optimize jointly the integrated policy's control settings, a comprehensive and thorough mathematical model is developed to capture the complex dynamic and stochastic behavior of the manufacturing system. The proposed model relaxes several simplifying assumptions so that reliability and quality degradations are operation-dependent, duration and cost of inspection are non-negligible, no restriction on random distributions and failures occur at any time, as in the real manufacturing world. Besides, a new signal-based simulation model is developed and a detailed sensitivity analysis is carried out to validate analytical results. The results show the relevance of the analytical approach and help us to prove the importance of adapting the inspection effort to the inner quality in the production. An extended comparative study is provided and shows that the proposed integrated policy, based on dynamic inspection, outperforms those based on classical inspection strategies considered in the literature and practice.

An Approach for Mitigation of Beam Blockage in mmWave-Based Indoor Networks

The rapidly evolving scenario around cellular networks, Internet of Things (IoT), and the associated services ecosystem has generated a wide range of expectations for the future networks. However, the unavailability of sufficient bandwidth and severe interference prohibit the existing commercially used radio-frequency (RF) bands from supporting some of these services. In this respect, the millimeter wave (mmWave)-based systems are seen as promising alternatives for supporting various high data rate enhanced mobile broadband (eMBB) services. Nevertheless, the adverse wave propagation characteristics in these bands and the resulting media access control (MAC) layer problems of beam blockage restrict these systems from operating in a reliable manner. Therefore, in this article, we propose a MAC layer-based approach of control delegation to tackle the problem of beam blockage. The mechanism uses policy-based relinquishment of access point (AP) control from the default AP to other AP capable member nodes to improve node visibility and alleviate the blockage scenario without requiring any additional network infrastructure. To account for the dynamic behavior of the network, a discrete-time Markov decision process (MDP)-based model is proposed. It uses a dynamic optimization methodology to accommodate multiple reconfigurable performance metrics, which can then be adjusted to cater to different performance criteria. Using computer simulations, we evaluate the effectiveness of our proposed solution for some alternate static and dynamic control policies. Our simulation results show that our proposed solution can significantly reduce the blockage in a network while satisfying a variety of performance objectives over different classes of indoor IoT scenarios.