Optimal Control Protocols Research Articles

Learning protocols for the fast and efficient control of active matter

Exact analytic calculation shows that optimal control protocols for passive molecular systems often involve rapid variations and discontinuities. However, similar analytic baselines are not generally available for active-matter systems, because it is more difficult to treat active systems exactly. Here we use machine learning to derive efficient control protocols for active-matter systems, and find that they are characterized by sharp features similar to those seen in passive systems. We show that it is possible to learn protocols that effect fast and efficient state-to-state transformations in simulation models of active particles by encoding the protocol in the form of a neural network. We use evolutionary methods to identify protocols that take active particles from one steady state to another, as quickly as possible or with as little energy expended as possible. Our results show that protocols identified by a flexible neural-network ansatz, which allows the optimization of multiple control parameters and the emergence of sharp features, are more efficient than protocols derived recently by constrained analytical methods. Our learning scheme is straightforward to use in experiment, suggesting a way of designing protocols for the efficient manipulation of active matter in the laboratory.

Quantum superpositions of current states in Rydberg-atom networks

Quantum simulation of many-body quantum systems using Rydberg-atom platforms has become of extreme interest in the last years. The possibility to realize spin Hamiltonians and the accurate control at the single atom level paved the way for the study of quantum phases of matter and dynamics. Here, we propose a quantum optimal control protocol to engineer current states: quantum states characterized by Rydberg excitations propagating in a given spatially closed tweezer networks. Indeed, current states with different winding numbers can be generated on demand. Besides those ones with single winding number, superposition of quantum current states characterized by more winding numbers can be obtained. The single current states are eigenstates of the current operator that therefore can define an observable that remains persistent at any time. In particular, the features of the excitations dynamics reflects the nature of current states, a fact that in principle can be used to characterize the nature of the flow experimentally without the need of accessing high order correlators. Published by the American Physical Society 2024

A Novel Parallel Control Method for Optimal Consensus of Nonlinear Multiagent Systems.

This work concentrates on the initial introduction of parallel control to investigate an optimal consensus control strategy for continuous-time nonlinear multiagent systems (MASs) via adaptive dynamic programming (ADP). First, the control input is integrated into the feedback system for parallel control, facilitating an augmented system's optimal consensus control with an appropriate augmented performance index function to be established, which is identical to the original system's suboptimal control with a conventional performance index. Second, the feasibility of the proposed control scheme is evaluated based on the policy iteration algorithm, and the convergence of the algorithm is demonstrated. Then, an online learning algorithm becomes available to implement the ADP-based optimal parallel consensus control protocol without prior knowledge of the system. The Lyapunov approach is employed to indicate that the signals are convergent. Ultimately, the experimental data support the theoretical results.

Optimal synchronization with [formula omitted]-gain performance: An adaptive dynamic programming approach

This paper studies an optimal synchronous control protocol design for nonlinear multi-agent systems under partially known dynamics and uncertain external disturbance. Under some mild assumptions, Hamilton–Jacobi–Isaacs equation is derived by the performance index function and system dynamics, which serves as an equivalent formulation. Distributed policy iteration adaptive dynamic programming is developed to obtain the numerical solution to the Hamilton–Jacobi–Isaacs equation. Three theoretical results are given about the proposed algorithm. First, the iterative variables is proved to converge to the solution to Hamilton–Jacobi–Isaacs equation. Second, the L2-gain performance of the closed loop system is achieved. As a special case, the origin of the nominal system is asymptotically stable. Third, the obtained control protocol constitutes an Nash equilibrium solution. Neural network-based implementation is designed following the main results. Finally, two numerical examples are provided to verify the effectiveness of the proposed method.

Optimal distributed time-varying formation control for second-order multiagent systems: LQR-based method

The optimal leaderless and leader-following time-varying formation (TVF) control problems for second-order multiagent systems (MASs) are investigated, where two optimal TVF control protocols are proposed to achieve the desired formations as well as minimize the comprehensive optimization function that contain the cooperative performance index and the control energy index. For leaderless case, the optimal formation control problem is reformulated as an infinite-time state regulator problem by employing the state space decomposition method, which is subject to specified constraints on energy and performance indices, and the analytic criterion for optimal TVF achievability is subsequently proposed. Then, the results of optimal leaderless TVF control are extended to the leader-following case with switching topologies, where the main challenge is changed to find the optimal controller rather than the optimal gain matrix, and the optimal value of the comprehensive index is accurately determined. Finally, two simulation cases are proposed to validate the effectiveness of the theoretical results, and comparisons with previous works are presented to expound the optimality of the proposed formation control method.

Distributed Minimum-Energy Containment Control of Continuous-Time Multi-Agent Systems by Inverse Optimal Control

Dear Editor, This letter focuses on the distributed optimal containment control of continuous-time multi-agent systems (CTMASs) with respect to the minimum-energy performance index over fixed topology. To achieve this, we firstly investigate the optimal containment control problem using the inverse optimal control method, where all states of followers asymptotically converge to the convex hull spanned by the leaders while some quadratic performance indexes get minimized. A sufficient condition for existence of the distributed optimal containment control protocol is derived. By introducing the parametric algebraic Riccati equation (PARE), it is strictly proved that the global performance index can be used to approximate the standard minimum-energy performance index as the parameters tends to infinity. In consequence, the standard minimum-energy cooperative containment control can be solved by local steady state feedback protocols. Finally, numerical examples are given to demonstrate the effectiveness of theoretical results.

Optimal Voltage Recovery Learning Control for Microgrids with N-Distributed Generations via Hybrid Iteration Algorithm

Considering that the nonlinearity and uncertainty of the microgrid model complicate the derivation and design of the optimal controller, an adaptive dynamic programming (ADP) algorithm is designed to solve the model-free non-zero-sum game. By combining the advantages of policy iteration and value iteration, an optimal learning control scheme based on hybrid iteration is constructed to provide stringent real power sharing for the nonlinear and coupled microgrid systems with N-distributed generations. First, using non-zero-sum differential game strategy, a novel distributed secondary voltage recovery consensus optimal control protocol is built using a hybrid iteration method to realize the voltage recovery of microgrids. Then, the data of the system state and input are gathered along a dynamic system trajectory and a data-driven optimal controller learns the game solution without microgrid physics information, enhancing convenience and efficiency in practical applications. Furthermore, the convergence analysis is given in detail, and it is proved that the control protocol can converge to the optimal solution so as to ensure the stability of the voltage recovery of the microgrid system. Convergence analysis proves the convergence of the the protocol to the optimal solution, ensuring voltage recovery stability. Simulation results validate the feasibility and effectiveness of the proposed scheme.

Universal Symmetry of Optimal Control at the Microscale

Optimizing the energy efficiency of driving processes provides valuable insights into the underlying physics and is of crucial importance for numerous applications, from biological processes to the design of machines and robots. Knowledge of optimal driving protocols is particularly valuable at the microscale, where energy supply is often limited. Here, we experimentally and theoretically investigate the paradigmatic optimization problem of moving a potential carrying a load through a fluid, in a finite time and over a given distance, in such a way that the required work is minimized. An important step towards more realistic systems is the consideration of memory effects in the surrounding fluid, which are ubiquitous in real-world applications. Therefore, our experiments were performed in viscous and viscoelastic media, which are typical environments for synthetic and biological processes on the microscale. Despite marked differences between the protocols in both fluids, we find that the optimal control protocol and the corresponding average particle trajectory always obey a time-reversal symmetry. We show that this symmetry, which surprisingly applies here to a class of processes far from thermal equilibrium, holds universally for various systems, including active, granular, and long-range correlated media in their linear regimes. The uncovered symmetry provides a rigorous and versatile criterion for optimal control that greatly facilitates the search for energy-efficient transport strategies in a wide range of systems. Using a machine learning algorithm, we demonstrate that the algorithmic exploitation of time-reversal symmetry can significantly enhance the performance of numerical optimization algorithms. Published by the American Physical Society 2024

Robust optimal tracking control of multiple autonomous underwater vehicles subject to uncertain disturbances

AbstractThis paper considers the problem of robust optimal tracking control of multiple autonomous underwater Vehicles (AUVs) subject to uncertain external disturbances. First, the Takagi‐Sugeno (T‐S) fuzzy based technique is utilized to convert the high‐order nonlinear multi‐AUV system into a series of linearized subsystems. Second, a novel fully distributed sliding mode control (FDSMC) strategy is proposed to attenuate the disturbances. Meanwhile, the leader‐following consensus and the nearly optimization of the energy‐cost function for the multi‐AUV system can be achieved simultaneously through the designed optimal nominal control protocol. Moreover, the proposed control strategy has more mild constraints on the communication topologies. Finally, the effectiveness of the proposed FDSMC strategy is verified by numerical simulation studies.

Deterministic generation of highly squeezed GKP states in ultracold atoms

We demonstrate a method for encoding Gottesman–Kitaev–Preskill (GKP) error-correcting qubits with single ultracold atoms trapped in individual sites of a deep optical lattice. Using quantum optimal control protocols, we demonstrate the generation of GKP qubit states with 10 dB squeezing, which is the current minimum allowable squeezing level for use in surface code error correction. States are encoded in the vibrational levels of the individual lattice sites and generated via phase modulation of the lattice potential. Finally, we provide a feasible experimental protocol for the realization of these states. Our protocol opens up possibilities for generating large arrays of atomic GKP states for continuous-variable quantum information.

Universally Robust Quantum Control.

We study the robustness of the evolution of a quantum system against small uncontrolled variations in parameters in the Hamiltonian. We show that the fidelity susceptibility, which quantifies the perturbative error to leading order, can be expressed in superoperator form and use this to derive control pulses that are robust to any class of systematic unknown errors. The proposed optimal control protocol is equivalent to searching for a sequence of unitaries that mimics the first-order moments of the Haar distribution, i.e., it constitutes a 1-design. We highlight the power of our results for error-resistant single- and two-qubit gates.

Model‐free distributed optimal control for general discrete‐time linear systems using reinforcement learning

AbstractThis article proposes a novel data‐driven framework of distributed optimal consensus for discrete‐time linear multi‐agent systems under general digraphs. A fully distributed control protocol is proposed by using linear quadratic regulator approach, which is proved to be a sufficient and necessary condition for optimal control of multi‐agent systems through dynamic programming and minimum principle. Moreover, the control protocol can be constructed by using local information with the aid of the solution of the algebraic Riccati equation (ARE). Based on the Q‐learning method, a reinforcement learning framework is presented to find the solution of the ARE in a data‐driven way, in which we only need to collect information from an arbitrary follower to learn the feedback gain matrix. Thus, the multi‐agent system can achieve distributed optimal consensus when system dynamics and global information are completely unavailable. For output feedback cases, accurate state information estimation is established such that optimal consensus control is realized. Moreover, the data‐driven optimal consensus method designed in this article is applicable to general digraph that contains a directed spanning tree. Finally, numerical simulations verify the validity of the proposed optimal control protocols and data‐driven framework.

Data-Driven Optimal Formation Control for Multiple Nonlinear Quadrotors With Switching Topologies

This paper addresses the optimal distributed formation control of multiple quadrotor vehicles under switching topologies, where each vehicle suffers from significant nonlinearity and uncertainty, simultaneously. A distributed observer is constructed to generate the trajectory references for each vehicle. An optimal leader-following formation control protocol is then developed through the reinforcement learning method from the sampled vehicle data. Simulation results from a multiple quadrotors example under switching topologies are presented to verify the obtained results.

Particle swarm optimization based leader-follower cooperative control in multi-agent systems

Multi-agent systems (MAS) have attracted significant attention in recent years due to their wide applications in cooperative control, formation control, synchronization of complex networks, and distributed coordination. A fundamental problem in MAS is the leader-follower consensus or cooperative tracking, where the followers are required to track the state trajectory of the leader agent. To solve the leader-follower consensus problem, we propose a novel evolutionary computation approach to design the optimal distributed control protocols for leader-follower MAS. First, we formulate the design of distributed control gains for leader-follower consensus as an optimization problem to minimize tracking errors. Then, we leverage particle swarm optimization as an efficient evolutionary technique for distributed gain optimization in multi-agent networks. Finally, we guarantee stability for the closed-loop dynamics under directed communication topologies based on algebraic graph theory. The simulation results indicate that the proposed method yields a diminished tracking error, expedites the convergence process, and minimizes the requisite control effort while enhancing computational efficiency. Furthermore, these results exemplify the method's versatility when applied to nonlinear dynamic scenarios, directed network topologies, fluctuating disturbances, and optimization across multiple domains.

Reinforcement learning-based optimal adaptive fuzzy control for nonlinear multi-agent systems with prescribed performance

In this paper, the problem of optimal adaptive consensus tracking control for nonlinear multi-agent systems with prescribed performance is investigated. To address the issue of satisfying the initial value conditions in existing results, an improved performance function is employed as the prescribed performance boundary, effectively resolving this problem. Then, by employing the error transformation function, the constrained system is converted into an unconstrained one. Furthermore, fuzzy logic systems are employed to identify unknown system parts. By applying the dynamic surface technique, the problem of "differential explosion", which often occurs in backstepping, is solved. Moreover, a distributed optimal adaptive fuzzy control protocol based on the reinforcement learning actor-critic algorithm is proposed. Under the proposed control scheme, it is proved that all the signals within the closed-loop system are bounded, and the consensus tracking errors have remained within the predefined bounds. Finally, the numerical simulation results demonstrate the effectiveness of the proposed scheme.

Nonzero-Sum Game-Based Voltage Recovery Consensus Optimal Control for Nonlinear Microgrids System.

Since most of the existing models based on the microgrids (MGs) are nonlinear, which could cause the controller oscillate, resulting in the excessive line loss, and the nonlinear could also lead to the controller design difficulty of MGs system. Therefore, this article researches the distributed voltage recovery consensus optimal control problem for the nonlinear MGs system with N -distributed generations (DGs), in the case of providing stringent real power sharing. First, based on the distributed cooperative control concept of multiagent systems and the critic neural networks (NNs), a novel distributed secondary voltage recovery consensus optimal control protocol is constructed via applying the backstepping technique and nonzero-sum (NZS) differential game strategy to realize the voltage recovery of island MGs. Meanwhile, the model identifier is established to reconstruct the unknown NZS games systems based on a three-layer NN. Then, a critic NN weight adaptive adjustment tuning law is proposed to ensure the convergence of the cost functions and the stability of the closed-loop system. Furthermore, according to Lyapunov stability theory, it is proven that all signals are uniform ultimate boundedness in the closed loop system and the voltage recovery synchronization error converges to an arbitrarily small neighborhood of the origin near. Finally, some simulation results in MATLAB illustrate the validity of the proposed control strategy.

Dynamic Coordinated Control for Multiconverter Systems via a Multistep Prediction Scheme

Modular DC-DC converters are popular in DC power systems such as demagnetization systems due to their advantages of high efficiency, high reliability, and low current stress on components. However, the dynamic consensus of output currents of the multiple converter degaussing systems (MCDSs) has not been well solved yet, especially when communication delay is considered. In this paper, a novel coordinated control method based on multi-step state predictions with active compensation of delay is proposed to address the poor dynamic consensus performance of currents in MCDSs. The implementation of the scheme consists of two main phases: the design of a multi-step state predictor for MCDSs and the optimization of the coordinating cost of currents. The multi-step state predictor can estimate immeasurable future states of converters over a large horizon length, which presents a novel approach to the current regulator design. The optimal control protocol is derived by minimizing a distributed coordinating performance index function (PIF) for the output currents. This optimization of the PIF minimizes the consensus error between output currents for converters and compensates for communication constraints. Further, the conditions for simultaneous stability and consensus of the closed-loop degaussing system with the distributed multi-step state prediction controller are given. Finally, the performance of the proposed scheme is verified by numerous case studies in an FPGA-based OPAL-RT experimental testbed.

Enhancing Associative Learning in Rats With a Computationally Designed Training Protocol

BackgroundLearning requires the activation of protein kinases with distinct temporal dynamics. In Aplysia, nonassociative learning can be enhanced by a computationally designed learning protocol with intertrial intervals (ITIs) that maximize the interaction between fast-activated PKA (protein kinase A) and slow-activated ERK (extracellular signal-regulated kinase). Whether a similar strategy can enhance associative learning in mammals is unknown. MethodsWe simulated 1000 training protocols with varying ITIs to predict an optimal protocol based on empirical data for PKA and ERK dynamics in rat hippocampus. Adult male rats received the optimal protocol or control protocols in auditory fear conditioning and fear extinction experiments. Immunohistochemistry was performed to evaluate pCREB (phosphorylated cAMP response element binding)\\protein levels in brain regions that have been implicated in fear acquisition. ResultsRats exposed to the optimal conditioning protocol with irregular ITIs exhibited impaired extinction memory acquisition within the session using a standard footshock intensity, and stronger fear memory retrieval and spontaneous recovery with a weaker footshock intensity, compared with rats that received massed or spaced conditioning protocols with fixed ITIs. Rats exposed to the optimal extinction protocol displayed improved extinction of contextual fear memory and reduced spontaneous recovery compared with rats that received standard extinction protocols. Moreover, the optimal conditioning protocol increased pCREB levels in the dentate gyrus of the dorsal hippocampus, suggesting enhanced induction of long-term potentiation. ConclusionsThese findings demonstrate that a computational model–driven behavioral intervention can enhance associative learning in mammals and may provide insight into strategies to improve cognition in humans.

Optimal performance of the stochastic thermodynamic engine with a periodic heat bath

Stochastic thermodynamics provides a conceptual framework for describing the fluctuating behavior of small systems like colloids or biomolecules far from thermodynamic equilibriums but still contacted with a heat bath. In contrast to most literature focusing on the classical paradigm of Carnot engines, we herein study the optimal performance of the thermodynamic heat engine with a heat bath that periodically changes temperature, which is outside controllable by a time-dependent harmonic potential. Under reasonable assumptions on the control actuation, we derive the achievable upper bound for the maximal power and also the optimal control protocol. In addition, we also obtain the corresponding efficiency at maximal power, which only depends on the ratio of the minimal and maximal value of the temperature profile.

Input‐constrained optimal output synchronization of heterogeneous multiagent systems via observer‐based model‐free reinforcement learning

AbstractThis paper presents a new formulation of input‐constrained optimal output synchronization problem and proposes an observer‐based distributed optimal control protocol for discrete‐time heterogeneous multiagent systems with input constraints via model‐free reinforcement learning. First, distributed adaptive observers are designed for all agents to estimate the leader's trajectory without requiring its dynamics knowledge. Subsequently, the optimal control input associated with the optimal value function is derived based on the solution to the tracking Hamilton‐Jacobi‐Bellman equation, which is always difficult to solve analytically. To this end, motivated by reinforcement learning technique, a model‐free Q‐learning policy iteration algorithm is proposed, and the actor‐critic neural network structure is implemented to iteratively find the optimal tracking control input without knowing system dynamics. Moreover, inputs of all agents are constrained in the permitted bounds by inserting a nonquadratic function into the performance function, where input constraints are encoded into the optimization problem. Finally, a numerical simulation example is provided to illustrate the effectiveness of the proposed theoretical results.