Optimal Control Framework Research Articles

Online Q-learning for stochastic linear systems with state and control dependent noise

For continuous-time (CT) systems that are characterized by stochastic differential equations (SDEs) with completely unknown dynamics parameters, a reinforcement learning (RL)-based optimal control framework is presented in this paper. To obtain the near-optimal control policy, an online Q-learning algorithm is proposed by learning the data sampled from the system state trajectory, while an integral reinforcement learning (IRL) approach is developed in stochastic situation so as to formulate the Q-learning iterative algorithm. In particular, an actor/critic neural network (NN) structure is applied in iteration, where a critic approximator is used for estimating the designed Q-function while an actor approximator is for estimating the optimal control policy online. To ensure the convergence of iteration, the tuning laws of two neural networks are designed, respectively, by a gradient descent scheme. Moreover, the mean-square stability of the closed-loop system is proved through rigorous analysis, and the convergence to optimal solution is guaranteed as well.

Enhanced and robust contrast in CEST MRI: Saturation pulse shape design via optimal control.

To employ optimal control for the numerical design of Chemical Exchange Saturation Transfer (CEST) saturation pulses to maximize contrast and stability against inhomogeneities. We applied an optimal control framework for the design pulse shapes for CEST saturation pulse trains. The cost functional minimized both the pulse energy and the discrepancy between the corresponding CEST spectrum and the target spectrum based on a continuous radiofrequency (RF) pulse. The optimization is subject to hardware limitations. In measurements on a 7 T preclinical scanner, the optimal control pulses were compared to continuous-wave and Gaussian saturation methods. We conducted a comparison of the optimal control pulses with Gaussian, block pulse trains, and adiabatic spin-lock pulses. The optimal control pulse train demonstrated saturation levels comparable to continuous-wave saturation and surpassed Gaussian saturation by up to 50 % in phantom measurements. In phantom measurements at 3 T the optimized pulses not only showcased the highest CEST contrast, but also the highest stability against field inhomogeneities. In contrast, block pulse saturation resulted in severe artifacts. Dynamic Bloch-McConnell simulations were employed to identify the source of these artifacts, and underscore the robustness of the optimized pulses. In this work, it was shown that a substantial improvement in pulsed saturation CEST imaging can be achieved by using Optimal Control design principles. It is possible to overcome the sensitivity of saturation to B0 inhomogeneities while achieving CEST contrast close to continuous wave saturation.

The inverse problem of identifying complex hyperbolic equation source terms in electromagnetic propagation

Abstract This article investigates the inverse problem of determining the source term of the hyperbolic equation for electromagnetic propagation using terminal data. This study is an important method for identifying propagation sources in electromagnetics. Unlike wave equations, the complexity of the underlying equations can make theoretical analysis quite difficult. Firstly, the uniqueness of the inverse problem was proved using the energy method. Then, based on the optimal control framework, the inverse problem was transformed into an optimal control problem, and the existence of the optimal solution and its necessary conditions were established. Secondly, the global uniqueness and stability of the optimal solution have been proven, which is a completely new conclusion. This has laid a solid theoretical foundation for numerical algorithms. Finally, it is proposed to apply the Landweber iteration method and conjugate gradient method to this problem, and some numerical examples are provided to demonstrate the effectiveness and convergence speed of these two algorithms.

Intelligent Control Framework for Improving Energy System Stability Through Deep Learning-Based Modal Optimization Scheme

Ensuring the stability of power systems is essential to promote energy sustainability. The integrated operation of these systems is critical in sustaining modern societies and economies, responding to the increasing demand for electricity and curbing environmental consequences. This study focuses on the optimization of energy system stability through the coordination of power system stabilizers (PSSs) and power oscillation dampers (PODs) in a single-machine infinite bus energy grid configuration that has flexible AC alternating current transmission system (FACTS) devices. Intelligent control strategies using PSS and POD techniques are suggested to increase power system stability and generate supplementary control signals for both the generator excitation system and FACTS device switching control. An intelligent optimal modal control framework equipped with deep learning methods is introduced to control the generator excitation system and thyristor-controlled series capacitor (TCSC). By optimally choosing the weighting matrix Q and implementing close-loop pole shifting, an optimal modal control approach is formulated. To harness its adaptive potential in fine-tuning controller parameters, an auxiliary deep learning-based optimization algorithm with actor–critic architecture is implemented. This comprehensive technique provides a promising path to effectively reduce electromechanical oscillations, thereby enhancing voltage regulation and transient stability in power systems.

Towards using active learning methods for human‐seat interactions to generate realistic occupant motion

AbstractIn the context of developing new vehicle concepts, especially autonomous vehicles with novel seating arrangements and occupant activities, predicting occupant motion can be a tool for ensuring safety and comfort. In this study, a data‐driven surrogate contact model integrated into an optimal control framework to predict human occupant behavior during driving maneuvers is presented. High‐fidelity finite element simulations are utilized to generate a dataset of interaction forces and moments for various human body configurations and velocities. To automate the generation of training data, an active learning approach is introduced, which iteratively queries the high‐fidelity finite element simulation for an additional dataset. The feasibility and effectiveness of the proposed method are demonstrated through a case study of a head interaction with an automotive headrest, showing promising results in accurately replicating contact forces and moments while reducing manual effort.

Multi-agent distributed control of integrated process networks using an adaptive community detection approach

This paper focuses on developing an adaptive system decomposition approach for multi-agent distributed model predictive control (DMPC) of integrated process networks. The proposed system decomposition employs a refined spectral community detection method to construct an optimal distributed control framework based on the weighted graph representation of the state space process model. The resulting distributed architecture assigns controlled outputs and manipulated inputs to controller agents and delineates their interactions. The decomposition evolves as the process network undergoes various operating conditions, enabling adjustments in the distributed architecture and DMPC design. This adaptive architecture enhances the closed-loop performance and robustness of DMPC systems. The effectiveness of the multi-agent distributed control approach is investigated for a benchmark benzene alkylation process under two distinct operating conditions characterized by medium and low recycle ratios. Simulation results demonstrate that adaptive decompositions derived through spectral community detection, utilizing weighted graph representations, outperform the commonly employed unweighted hierarchical community detection-based system decompositions in terms of closed-loop performance and computational efficiency.

The optimal spatially-dependent control measures to effectively and economically eliminate emerging infectious diseases.

Non-pharmaceutical interventions (NPIs) are effective in mitigating infections during the early stages of an infectious disease outbreak. However, these measures incur significant economic and livelihood costs. To address this, we developed an optimal control framework aimed at identifying strategies that minimize such costs while ensuring full control of a cross-regional outbreak of emerging infectious diseases. Our approach uses a spatial SEIR model with interventions for the epidemic process, and incorporates population flow in a gravity model dependent on gross domestic product (GDP) and geographical distance. We applied this framework to identify an optimal control strategy for the COVID-19 outbreak caused by the Delta variant in Xi'an City, Shaanxi, China, between December 2021 and January 2022. The model was parameterized by fitting it to daily case data from each district of Xi'an City. Our findings indicate that an increase in the basic reproduction number, the latent period or the infectious period leads to a prolonged outbreak and a larger final size. This indicates that diseases with greater transmissibility are more challenging and costly to control, and so it is important for governments to quickly identify cases and implement control strategies. Indeed, the optimal control strategy we identified suggests that more costly control measures should be implemented as soon as they are deemed necessary. Our results demonstrate that optimal control regimes exhibit spatial, economic, and population heterogeneity. More populated and economically developed regions require a robust regular surveillance mechanism to ensure timely detection and control of imported infections. Regions with higher GDP tend to experience larger-scale epidemics and, consequently, require higher control costs. Notably, our proposed optimal strategy significantly reduced costs compared to the actual expenditures for the Xi'an outbreak.

Machine learning model‐based optimal tracking control of nonlinear affine systems with safety constraints

AbstractThis work focuses on the development of a machine learning (ML) model‐based framework for safe optimal tracking control of a class of nonlinear control‐affine systems to ensure simultaneous closed‐loop stability and safety. Specifically, a novel multilayer feedforward neural network (FNN) with a control‐affine architecture is designed to model nonlinear dynamic systems. Subsequently, a model‐based reinforcement learning (RL) framework is presented, utilizing a novel cost function with Control Lyapunov‐Barrier Function (CLBF) properties, to learn both the control policy and the optimal value function for an infinite‐horizon optimal tracking control problem for nonlinear systems with safety constraints. The efficacy of the proposed methodology is demonstrated through simulations of a one‐link robot manipulator and a chemical process example.

The Nexus of Fiscal Policy and Growth in the Optimal Control Framework

This research investigates the effect of fiscal policy on economic growth, focusing on the intertwined relationship between public goods provision, taxation policies, and their impact on economic dynamics and growth. The study aims to contribute to this research domain by introducing an economic model within the optimal control framework that integrates taxation policies into the social constraint and incorporates public goods into the social utility function, addressing recent limitations in this research area. Through numerical analysis, the study examines the potential effects of taxes and public goods provision on consumption and economic growth, as reflected by the level of capital. Results indicate that increasing government spending on public goods can decrease private goods consumption without significant economic growth benefits. At the same time, high tax rates could potentially hinder economic growth by overly relying on government intervention. The research findings highlight the importance of an optimal approach to fiscal policy, with policy implications including the need to carefully evaluate public funds' allocation, enhance public spending efficiency, and implement optimal tax to foster economic growth.

Inverse problem of reconstructing source term for a class of non‐divergence parabolic equations

This paper explores an inverse problem pertaining to the determination of a source function in non‐divergence parabolic equations, where the solution is known at a discrete set of points. Being different from other ordinary inverse source problems, which are often dependent on only one variable, the unknown coefficient in this paper not only depends on the space variable but also depends on the time . On the basis of the optimal control framework, the existence of the optimal solution of the control function is proved. The necessary conditions to be satisfied by the optimal solution are given. The convergence of the optimal solution when the mesh parameters tend to zero is obtained. The conjugate gradient method is applied to the inverse problem and some numerical results are presented for various typical test examples.

Towards a modelling, optimization and predictive control framework for smart irrigation

Smart irrigation scheduling is a promising approach for improving the efficiency and sustainability of agricultural water use, especially in arid and semi-arid lands. In this paper, we present a design and simulation of a model predictive controller for smart irrigation scheduling. The proposed controller is based on a mathematical model of the irrigation system, which is used to predict the future states of the system and determine the optimal irrigation schedule for a given crop and field. This study further evaluates the impact of the predictive control framework on crop yield, water use efficiency (WUE), and water savings in tomato production. Three control strategies, including manual control, open-loop control, and model predictive control (MPC), were compared using a Completely Randomized Design (CRD) methodology. Results indicate that MPC outperforms the other strategies, yielding the highest average crop yield of 20 t/ha and demonstrating superior WUE at 10.4 kg/m3. Additionally, MPC significantly reduces water consumption, achieving a 29 % and 8 % savings compared to manual and open-loop control, respectively. These findings underscore the efficacy of MPC in optimizing crop yield, conserving water resources, and promoting sustainable agriculture practices. The proposed controller has the potential to address the global water scarcity challenge and contribute to the sustainability of agriculture in arid and semi-arid lands.

Intelligent optimal control of furnace temperature for the municipal solid waste incineration process using multi-loop controller and particle swarm optimization

Furnace temperature (FT) is a key variable in municipal solid waste incineration (MSWI) processes, influenced by many manipulated variables and directly impacting pollutant concentrations in exhaust gas. Domain experts cannot achieve the optimal FT setpoint value under manual control, leading to abnormal pollutant emission concentrations. To address this, we propose an intelligent optimal control framework for FT aiming to minimize pollutant emission concentration. First, the FT controlled object model is established using the Tikhonov regularization-least regression decision tree (TR-LRDT) algorithm. Then, based on the experience of domain experts, a multi-loop controller is developed using an improved single neuron adaptive PID (ISNA-PID) algorithm to stabilize FT. Next, after establishing NOx and CO2 indicator models, the particle swarm optimization (PSO) algorithm is employed to determine the FT setpoint value in terms of minimum pollutant emission concentration. Finally, the FT intelligent optimal control framework is verified. Experimental results indicate that the optimal FT setpoint value can reduce NOx and CO2 emission concentrations by 19.93 % and 6.99 %, respectively.

A learning-based nearly optimal control framework for trajectory tracking of a flexible-link manipulator system with actuator fault

In this paper, a learning-based nearly optimal control framework with fault-tolerant capability is designed to tackle the tracking control problem of a flexible-link manipulator in the presence of actuator fault and model uncertainties. Initially, the optimal control law is obtained by adopting the dynamic programming and a critic structure as the solution of Hamilton–Jacobi–Bellman equation for the nominal model. Then, by implementing an integral sliding mode control, the robustness against actuator fault and model uncertainty is guaranteed. The adaptive laws are constructed based on radial basis functions neural networks to estimate the upper bound of uncertainty and the actuator bias fault, satisfying both optimal performance and chattering reduction of the sliding surface. Furthermore, the actuator effectiveness loss is handled. The stability of the closed-loop system is analytically proven, and the performance of the proposed framework is investigated against several practical operating conditions. This incorporates the fidelity assessment of tracking precision and trackability of control signal using performance indices such as the integral absolute error and root-mean-square error. The results of extensive simulation studies confirm the effectiveness and robustness of the proposed control framework.

Vaccination and transportation intervention strategies for effective pandemic control

The COVID-19 pandemic highlighted the pivotal role of transportation-related controls and vaccination in mitigating widespread infections. However, the substantial costs associated with these interventions necessitate a nuanced equilibrium between control measures and infection risks. This study, driven by the recognition of spatial heterogeneities in mobility networks and epidemic vulnerability, presents an optimal control framework for two control measures—vaccine distribution and transportation restrictions—within a metapopulation structure. The overarching goal is to minimize the total costs derived from the implementation of control measures and health and opportunity loss from infection. Our findings highlight the effectiveness of transportation control and vaccine distribution in reducing both total costs and infection rates. Notably, the efficacy of these control measures exhibits regional variations, and the simultaneous implementation of both measures emerges as the most effective and economically viable strategy. The synergy effect between vaccine distribution and transportation restrictions is also a key observation in our simulations, showcasing their complementary roles in pandemic control. By considering the spatial nuances of mobility networks and infection vulnerability, this framework provides a flexible and region-specific strategy to optimize the allocation of resources for pandemic control, ultimately striking a balance between efficacy and economic feasibility.

A distributed economic model predictive control-based FPPT scheme for large-scale solar farm

Increasing solar-photovoltaic-power penetration necessitates the implementation of flexible power point tracking (FPPT) in solar farms. However, coordinating multiple solar photovoltaic (PV) power generation systems (SPVPGSs) poses a significant challenge due to their inherent intermittency and varying dynamic characteristics, complicating FPPT implementation. To overcome this challenge, this paper proposes a distributed economic model predictive control (DEMPC) scheme to achieve FPPT, while simultaneously enhancing overall economic performance in solar farms. By integrating solar farm control and local control into one optimal control framework, this scheme eliminates the need for power allocation, PV voltage reference calculation, and pulse width modulation. Leveraging the SPVPGS model and soft power constraint, DEMPC controllers are designed to achieve the economic targets of solar farms, which cooperate through a communication network to realize FPPT and economic optimization. Additionally, the strong nonlinearity of SPVPGS causes a non-convex mixed integer nonlinear programming (MINLP) problem, solved by a MINLP algorithm using finite converter switching states. Simulations under step-changed irradiance and power reference, as well as urgent maintenance conditions, demonstrate that the DEMPC-based FPPT scheme significantly outperforms the traditional hierarchical model predictive control-based FPPT scheme, presenting both superior dynamic response and enhanced economic performance in FPPT implementation.

Human navigation strategies and their errors result from dynamic interactions of spatial uncertainties

Goal-directed navigation requires continuously integrating uncertain self-motion and landmark cues into an internal sense of location and direction, concurrently planning future paths, and sequentially executing motor actions. Here, we provide a unified account of these processes with a computational model of probabilistic path planning in the framework of optimal feedback control under uncertainty. This model gives rise to diverse human navigational strategies previously believed to be distinct behaviors and predicts quantitatively both the errors and the variability of navigation across numerous experiments. This furthermore explains how sequential egocentric landmark observations form an uncertain allocentric cognitive map, how this internal map is used both in route planning and during execution of movements, and reconciles seemingly contradictory results about cue-integration behavior in navigation. Taken together, the present work provides a parsimonious explanation of how patterns of human goal-directed navigation behavior arise from the continuous and dynamic interactions of spatial uncertainties in perception, cognition, and action.

Reinforcement learning control of multi-spectral LED systems for rendering optimal lighting performance in indoor environment

Light-emitting diodes (LEDs) have superseded traditional light sources due to their superior qualities. However, achieving desired spectral distribution in indoor environment still remains a challenge, particularly for optimizing multiple lighting parameters, which are nonlinear functions of the light spectra, including correlated color temperature (CCT), color rendering index (CRI) and circadian action factor (CAF). Existing studies have been mainly devoted to seeking optimal solutions, by means of various optimization methods, e.g. genetic algorithms. However, these methods can only result in fixed recipes to achieve pre-assigned reference parameters, and cannot meet the need of dynamic varying lighting environment. Therefore, this paper proposes a novel framework for optimal spectrum design and control in multichannel LED lighting systems. It surpasses traditional methods by employing reinforcement learning (RL) to generate an entire optimal spectral power distribution (SPD) model, comprehensively considering CCT, CRI, CAF, and other relevant parameters. This framework integrates the RL-based optimal SPD generator with a corresponding real-time controller, bridging the gap between simulations and real-world applications. The effectiveness of this new framework has been verified by an experimental prototype, showcasing both the optimal SPD and the achieved control performance. This work paves the way for high-quality, user-centric lighting with dynamic CCT and spectral control, and enables the creation of lighting environments that optimize human well-being, productivity, and overall experience.

Performance-prescribed optimal neural control for hypersonic vehicles considering disturbances: An adaptive dynamic programming approach

The precise trajectory tracking control of hypersonic vehicles plays a crucial role in secure flight. To address this challenge, this article develops an optimal neural control framework with prescribed performance constraints for height-velocity control tasks with disturbances. To this end, the control-oriented error model is firstly established and transformed into an affine nonlinear form to provide a foundation for the design of controller. Then, in order to balance performance constraints and convergence speed, a performance-constraint function is introduced to convert the constrained tracking error into an unconstrained one. After that, an improved sliding mode compensation observer is proposed by combining low-fidelity approximation data with high-fidelity compensation model for eliminating disturbances. The developed control scheme is organized by an adaptive dynamic programming (ADP) algorithm based on the actor-critic policy structure to achieve the near-optimal control strategy. Meanwhile, the designed neural network (NN) weight updating laws are provided to obtain the solution of Hamiltonian-Jacobi-Bellman (HJB) equation rapidly without high calculation consumption. Finally, the stability conditions of the closed-loop system are performed and ensure the entire framework is semi-global uniformly ultimately bounded (SGUUB), two simulation cases verify the effectiveness and superiority of the proposed approach.

Optimal Control of Collective Electrotaxis in Epithelial Monolayers

Epithelial monolayers are some of the best-studied models for collective cell migration due to their abundance in multicellular systems and their tractability. Experimentally, the collective migration of epithelial monolayers can be robustly steered e.g. using electric fields, via a process termed electrotaxis. Theoretically, however, the question of how to design an electric field to achieve a desired spatiotemporal movement pattern is underexplored. In this work, we construct and calibrate an ordinary differential equation model to predict the average velocity of the centre of mass of a cellular monolayer in response to stimulation with an electric field. We use this model, in conjunction with optimal control theory, to derive physically realistic optimal electric field designs to achieve a variety of aims, including maximising the total distance travelled by the monolayer, maximising the monolayer velocity, and keeping the monolayer velocity constant during stimulation. Together, this work is the first to present a unified framework for optimal control of collective monolayer electrotaxis and provides a blueprint to optimally steer collective migration using other external cues.

Fast generation of collision-free low-thrust trajectories for asteroid landing using deep neural networks

This study presents a fast generation method of time-optimal low-thrust trajectories based on deep neural networks (DNN) for the collision-free asteroid landings. An equivalent unconstrained differentiable problem is transformed via the introduction of smooth penalty function, so that it can be addressed with the optimal control framework. To efficiently solve the equivalent problem to generate sufficient dataset for training DNNs, a double homotopy method is developed. Specifically, the equivalent problem is connected to the time-optimal control problem without anti-collision through two consecutive simplifications, and a solving process with double backward continuation is also formulated. Besides, two DNN models are constructed to provide high-quality predictions for the minimum glide-slope constraint angle of an initial state, and the initial costates and landing time for time-optimal low-thrust trajectories with anti-collision constraints, respectively. As such, the efficiency of collision-free low-thrust landing trajectory generation is improved significantly. For the cases within the range of training dataset or within a certain extension, the low-thrust landing trajectories with a very small terminal landing error can be generated using the predicted initial costates. Meanwhile, with the high-quality initial costates, the high-precision landing trajectories can be obtained by the double homotopy method in only a few iterative steps. Finally, the promising results in a Bennu collision-free asteroid landing scenario validate the efficiency of the proposed method.