Learning Control Research Articles

Optimized Proportional-derivative Feedback-assisted Iterative Learning Control for Manipulator Trajectory Tracking

Optimized Proportional-derivative Feedback-assisted Iterative Learning Control for Manipulator Trajectory Tracking

Iterative Learning Tracking Control for Nonlinear Systems Based on Data Driven Control

This paper introduces a novel methodology for tracking control of general nonlinear systems. Unlike traditional methods, which rely on linearization or nonlinear cancellation. This article starts from the iterative domain and utilizes the concept of iterative learning control to improve the control performance of the system. It employs a data-driven approach for model-free adaptive control, addressing the challenges posed by strong system nonlinearity, difficulties in control, and even modeling issues caused by disturbances and other factors. It enables precise trajectory tracking without the need for complex computations, making it suitable for real-time control applications. The effectiveness of the methodology is demonstrated through examples.

Experiences of the nursing student in learning HAI prevention and control in Asian countries through the use of scenario-based simulation: an explorative qualitative study

Experiences of the nursing student in learning HAI prevention and control in Asian countries through the use of scenario-based simulation: an explorative qualitative study

Iterative-learning-based tracking control of a two-wheeled mobile robot with model uncertainties and unknown periodic disturbances

In this paper, we develop an adaptive iterative learning approach to investigate the trajectory tracking control issue for a class of two-wheeled mobile robots subject to model uncertainties and unknown disturbances. First, we derive the nonlinear velocity error dynamics. Then a parameterization-based adaptive iterative learning control scheme is adopted to achieve precise tracking, along with logic-based update law for the estimated period and bound of the disturbance. Moreover, the boundness of all the closed-loop signals is rigorously analyzed based on the Lyapunov stability theory to provide the theoretical foundation for the proposed method. The experimental results show the efficacy and viability of our results.

State Identification Via Symbolic Time Series Analysis for Reinforcement Learning Control

Abstract This technical brief makes use of the concept of symbolic time-series analysis (STSA) for identifying discrete states from the nonlinear time response of a chaotic dynamical system for model-free reinforcement learning (RL) control. Along this line, a projection-based method is adopted to construct probabilistic finite state automata (PFSA) for identification of the current state (i.e., operational regime) of the Lorenz system; and a simple Q-map-based (and model-free) RL control strategy is formulated to reach the target state from the (identified) current state. A synergistic combination of PFSA-based state identification and RL control is demonstrated by the simulation of a numeric model of the Lorenz system, which yields very satisfactory performance to reach the target states from the current states in real-time.

Novel quantized iterative learning control based on spherical polar coordinates

AbstractThis study investigates the iterative learning control method for discrete‐time systems with data quantization, and it employs a quantizer based on spherical polar coordinates. The quantizer uses spherical polar coordinates to transform an unquantized signal into one with a predetermined quantization level. The quantization capability is dependent on the size of the support sphere, which is dynamically adjustable. In the scenarios of system output quantization, general quantization, and tracking error quantization, the learning control schemes are developed with consideration of the smallest sphere quantization features. The radii of the support spheres are designed independently based on the characteristics of the quantized signals (system output, control input, tracking error). The findings suggest that the system can achieve a bounded error convergence in the scenarios of system output and general quantization. In the context of quantization in tracking error, it is achievable to attain tracking performance with zero error. The example of a permanent magnet synchronous motor is provided to illustrate the findings of the study.

Position control of a digital electrohydraulic system with limited sensory data using double deep q-network controller

In this study, we propose a reinforcement learning controller for position control of a digital electrohydraulic system with independent metering valve structure. Our study addresses the inherent challenges of traditional control methods, such as the need for continuous pressure measurement and complex control structures. As a solution, a double deep q-network (DDQN) controller which relies solely on position feedback from D-EHS is suggested. A simulation environment was developed to train the DDQN controller, wherein the artificial neural network structure, training hyperparameters, and reward function were optimized through preliminary studies. The DDQN controller was trained using a sinusoidal position reference signal of a single frequency and evaluated experimentally for its position control performance. Tracking performance was assessed against various position reference signals, including ramp and sinusoidal signal with different frequencies. Robustness against external disturbances was also tested by increasing the nominal supply pressure by 20% and 40%. The results demonstrate that the DDQN controller successfully controls the digital electrohydraulic system and is achieving good tracking performance for the training position reference signal. Furthermore, the controller exhibits satisfactory tracking performance for non-training position reference signals, highlighting its ability to learn the digital electrohydraulic system dynamics with limited training data. Even under varying supply pressure conditions, the DDQN controller maintains an acceptable level of control performance. Our findings emphasize the generalizability and reliability of the DDQN reinforcement learning controller, making it a suitable candidate for application in digital electrohydraulic systems and contributing to the advancement of control theory research.

Adaptive neural network iterative learning control of long-stroke hybrid robots with initial errors and full state constraints

Research on initial errors and constraint restrictions is one of the main research directions in the field of control of uncertain robotic systems. An adaptive iterative learning control (AILC) method based on radial basis function (RBF) neural network is proposed to address the trajectory tracking problem of the long-stroke hybrid robot system with random initial errors and full state constraints. The RBF neural network is used to approximate the unknown nonlinear terms, and the network weights are updated using an iterative learning law that incorporates a projection mechanism. Additionally, a robust learning strategy is used to compensate for both the approximation error of the neural network and the external disturbances that vary with each iteration. To relax the requirement of traditional iterative learning control (ILC) for identical initial condition, an equivalent error function is constructed based on the time-varying boundary layer. The tangent-type barrier Lyapunov function (BLF) is designed to ensure that the joint position and speed of the robot system are bounded within a predetermined range. Through stability analysis based on barrier composite energy function (BCEF), it can be proved that the boundedness of all signals in the closed-loop system and the tracking error of the robot system will converge to an adjustable residual set asymptotically. Finally, through simulation experiments conducted on the MATLAB platform, the results demonstrate that the method overcomes the random initial errors of the system effectively, ensures that the system satisfies the full-state constraints, and realizes high-precision trajectory tracking.

Finite-time composite learning control for nonlinear teleoperation systems under networked time-varying delays

Finite-time composite learning control for nonlinear teleoperation systems under networked time-varying delays

No evidence for cross-paradigm transfer of abstract task knowledge in adults and school-aged children.

Cognitive control is a hallmark of human cognition. A large number of studies have focused on the plasticity of cognitive control and examined how repeated task experience leads to the improvement of cognitive control in novel task environments. However, it has been demonstrated that training-induced changes are very selective and that transfer occurs within one task paradigm but not across different task paradigms. The current study tested the possibility that cross-paradigm transfer would occur if a common cognitive control strategy is employed across different task paradigms. Specifically, we examined whether prior experience of using reactive control in one task paradigm (i.e., either the cued task-switching paradigm or the AX-CPT) makes adults (N = 137) and 9- to 10-year-olds (N = 126) respond in a reactive way in a subsequent condition of another task paradigm in which proactive control could have been engaged. Bayesian generalized mixed-effects modelsrevealed clear evidence of an absence of cross-paradigm transfer of reactive control in both adults and school-aged children. Based on these findings, we discuss to what extent learned control could be transferred across different task contexts and the task-specificity of proactive/reactive control strategies.

Critic learning control via zero-sum differential game for affine formation maneuver of multi-agent systems with cyber-attacks

The affine formation maneuver (AFM) control problem using the zero-sum differential game is studied for a second-order multi-agent system (MAS) against cyber-injection attacks on the actuators. First and foremost, the distributed control signals and the cyber-injected attack signals are viewed as two competing teams. Then, the leader-follower tracking error system is established where the followers aim to track the AFMs of the leaders. The leader-follower distributed optimal control policies and the cyber-attack attenuation policies are derived from the Nash equilibrium solution of the Hamilton-Jacobi-Isaac (HJI) equation. The solution of the HJI equation is estimated with the aid of the critic neural network. The weights of the critic system are learned online based on the adaptive dynamic programming (ADP) scheme. The uniformly ultimately bounded stability of the closed-loop system under the formulated zero-sum differential game control strategy has been proven by the Lyapunov stability theorem. A simulation example illustrates that the presented control method can attenuate cyber-injection attacks on the actuators and maintain the AFMs of the agents.

Model-Free Nonstationary Reinforcement Learning: Near-Optimal Regret and Applications in Multiagent Reinforcement Learning and Inventory Control

We consider model-free reinforcement learning (RL) in nonstationary Markov decision processes. Both the reward functions and the state transition functions are allowed to vary arbitrarily over time as long as their cumulative variations do not exceed certain variation budgets. We propose Restarted Q-Learning with Upper Confidence Bounds (RestartQ-UCB), the first model-free algorithm for nonstationary RL, and show that it outperforms existing solutions in terms of dynamic regret. Specifically, RestartQ-UCB with Freedman-type bonus terms achieves a dynamic regret bound of [Formula: see text], where S and A are the numbers of states and actions, respectively, [Formula: see text] is the variation budget, H is the number of time steps per episode, and T is the total number of time steps. We further present a parameter-free algorithm named Double-Restart Q-UCB that does not require prior knowledge of the variation budget. We show that our algorithms are nearly optimal by establishing an information-theoretical lower bound of [Formula: see text], the first lower bound in nonstationary RL. Numerical experiments validate the advantages of RestartQ-UCB in terms of both cumulative rewards and computational efficiency. We demonstrate the power of our results in examples of multiagent RL and inventory control across related products. This paper was accepted by Omar Besbes, revenue management and market analytics. Funding: The research of D. Simchi-Levi and R. Zhu was supported by the MIT Data Science Laboratory. The research of W. Mao, K. Zhang, and T. Başar was supported in part by the U.S. Army Research Laboratory (ARL) Cooperative Agreement W911NF-17-2-0196, in part by the Office of Naval Research (ONR) [MURI Grant N00014-16-1-2710], and in part by the Air Force Office of Scientific Research (AFOSR) [Grant FA9550-19-1-0353]. K. Zhang also acknowledges support from U.S. Army Research Laboratory (ARL) [Grant W911NF-24-1-0085]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2022.02533 .

Dissipative imitation learning for discrete dynamic output feedback control with sparse data sets

AbstractImitation learning enables synthesis of controllers for systems with complex objectives and uncertain plant models. However, ensuring an imitation learned controller is stable requires copious amounts of data and/or a known plant model. In this paper, we explore an input–output (IO) stability approach to imitation learning, which achieves stability with sparse data sets while only requiring coarse knowledge of the energy characteristics of the plant. A constrained optimization problem is developed, in which the controller learns to mimic expert data while maintaining stabilizing energy characteristics induced by the plant. While the learning objective is nonconvex, iterative convex overbounding (ICO) and projected gradient descent (PGD) are explored as methods to learn the controller. In numerical examples, it is shown that with little knowledge of the plant model and a small data set, the dissipativity constrained learned controller achieves closed loop stability and successfully mimics the behavior of the expert controller, while other methods often fail to maintain stability and achieve good performance.

Large language model-based interpretable machine learning control in building energy systems

The potential of Machine Learning Control (MLC) in HVAC systems is hindered by its opaque nature and inference mechanisms, which is challenging for users and modelers to fully comprehend, ultimately leading to a lack of trust in MLC-based decision-making. To address this challenge, this paper investigates and explores Interpretable Machine Learning (IML), a branch of Machine Learning (ML) that enhances transparency and understanding of models and their inferences, to improve the credibility of MLC and its industrial application in HVAC systems. Specifically, we developed an innovative framework that combines the principles of Shapley values and the in-context learning feature of Large Language Models (LLMs). While the Shapley values are instrumental in dissecting the contributions of various features in ML models, LLM provides an in-depth understanding of the non-data-driven or rule-based elements in MLC; combining them, LLM further packages these insights into a coherent, human-understandable narrative. The paper presents a case study to demonstrate the feasibility of the developed IML framework for model predictive control-based precooling under demand response events in a virtual testbed. The results indicate that the developed framework generates and explains the control signals in accordance with the rule-based rationale.

An autonomous stair climbing method for the crawler robot based on a hybrid reinforcement learning controller

In response to problems in using deep reinforcement learning to perform the stair climbing task for the crawler robot, such as the lack of theoretical values for stair size range, the inaccurate force on the model using wheel to simulate track and difficulty in training, we propose an autonomous stair climbing method for the crawler robot based on a hybrid reinforcement learning controller. We propose a dynamics parameter settings method for fast simulation model by studying the distribution model of the ground pressure of the track, which improves the accuracy of the force on the fast simulation model in the stair climbing task. Our method calculates the climbing ability of the crawler robot by studying the kinematics and dynamics models of the real crawler robot in each stage of the stair climbing task, and combining with the control flow of the hybrid reinforcement learning controller. Further, we develop the training course of the hybrid reinforcement learning controller. The agent is trained in ROS-Gazebo. The course reduces the impact of poor trajectory on training, achieves higher success rates and faster convergence speed compared to random training, and effectively improves the generalization ability of the strategy.

The quantum cartpole: A benchmark environment for non-linear reinforcement learning

Feedback-based control is the de-facto standard when it comes to controlling classical stochastic systems and processes. However, standard feedback-based control methods are challenged by quantum systems due to measurement induced backaction and partial observability. Here we remedy this by using weak quantum measurements and model-free reinforcement learning agents to perform quantum control. By comparing control algorithms with and without state estimators to stabilize a quantum particle in an unstable state near a local potential energy maximum, we show how a trade-off between state estimation and controllability arises. For the scenario where the classical analogue is highly nonlinear, the reinforcement learned controller has an advantage over the standard controller. Additionally, we demonstrate the feasibility of using transfer learning to develop a quantum control agent trained via reinforcement learning on a classical surrogate of the quantum control problem. Finally, we present results showing how the reinforcement learning control strategy differs from the classical controller in the non-linear scenarios.

A gradient-descent iterative learning control algorithm for a non-linear system

Original iterative learning control (OILC) has been proved a powerful tool in dealing with the model-free control problems by repetitively corrections based on the control error. However, the steady-state error under widely-used proportional-type original iterative learning control (P-type OILC) is highly corresponded to the proportional learning gain, making the algorithm parameter-determined. Therefore, a new gradient-descent iterative learning control (GDILC) algorithm is proposed to achieve a parameter-free approach by simulating the gradient-descent process. First, GDILC problem is formulated mathematically. Next, the idea of the algorithm is proposed, the analyses of the convergence and the steady-state error are conducted and the algorithm is implemented. GDILC will generate a random correction with a gradient-descent upper bound, rather than a correction proportional to the error in P-type OILC. Finally, illustrative and application simulations are conducted to validate the algorithm. Results show that the algorithm will be convergent after adequate iterations under proper corrections. The steady-state error will be less affected by the algorithm parameters under GDILC than that under OILC.

Long-term neurological and neurocognitive impairments after tick-borne encephalitis in Lithuania - a prospective study

Background The aim of this study was to characterise long-term neurological and neurocognitive sequelae after tick-borne encephalitis (TBE) in adults. Methods 98 prospective consecutive TBE patients, classified by disease severity, were included. Immediate outcomes were evaluated with Glasgow Outcome Scale (GOS) and Rankin Scale (RS). After 6 and 18 months, long-term disability was evaluated using Modified Rankin Scale (MRS) and neurocognitive assessment was performed with Matrics Consensus Cognitive Battery (MCCB), measuring processing speed, attention/vigilance, working memory, verbal learning, visual learning, reasoning/problem solving and social cognition. The MCCB results were compared to healthy age, gender and education-matched controls. Results Mild, moderate, and severe TBE was diagnosed in 53.1%, 38.8%, and 8.2% of cases, respectively. At discharge, 25.5% of the patients had major or moderate impairments (GOS) and various levels of disability in 34.7% (RS). Up to 18 months from the onset of TBE, over 20% remained with slight to moderate disability (MRS). GOS, RS and MRS scores correlated with disease severity. At 6 months after the onset, TBE patients scored significantly lower than controls in processing speed, verbal, and visual learning. Two latter domains were significantly more impaired in patients with mild TBE. Patients aged 18-39 performed significantly worse in attention/vigilance and working memory, whereas aged 60+ in verbal learning. A year later, significant improvement was observed in six of seven cognitive domains. Conclusions Long-term neurological sequelae persist in a substantial proportion of TBE patients with significant impairment in several cognitive domains, especially in younger patients and even after mild TBE.

SWOAM: Swarm optimized agents for energy management in grid-interactive connected buildings

With increasing complexity of smart city energy systems and rising energy demand, effective energy management solutions are crucial. Buildings now incorporate renewable energy sources and battery storage for efficient energy utilization, making optimal control strategies important. Compared to rule-based controllers and model-based methods, swarm and evolutionary algorithms have the advantages of providing cost-effective, stable, and scalable alternatives. However, their potential in data-rich environments and multi-building energy systems remains underexplored. This research bridges this gap by demonstrating the cooperative capabilities of population-based optimization agents for efficient energy management. Specifically, novel algorithm and framework, named Swarm Optimized Agents for Sequential Decision Making (SWOAM) is proposed. It combines an online k-means classifier and a swarm optimizer for optimal control of multi-building energy systems. By designing an online K-means learning strategy and an agent-based sequential episodic sampling approach, it is feasible to train a swarm of agents find an optimal energy management policy for buildings in a district. For training and evaluation, we use the standardized CityLearn building energy management environment. The agents are evaluated on three key metrics: electricity cost, carbon dioxide emissions and grid ramping. SWOAM delivers state-of-the-art performance and outperforms modern reinforcement learning and rule-based controllers.

A homotopy-based reinforcement learning scheme to optimal control for Markov switched interconnected systems

ABSTRACT In this paper, a homotopy-based reinforcement learning optimal control method is developed for Markov switched interconnected systems with unknown system dynamics. By utilising the subsystem decomposition method and parallel learning control method, the solution of the game coupled algebraic Riccati equations with jumping parameters is approximated. To dispense with the requirement of initial stability, a homotopy-based policy iteration is introduced, which can place unstable poles into a stable plane. In this regard, a model-free reinforcement learning method is presented to design the optimal controller for Markov switched interconnected systems. Finally, the effectiveness of the proposed method is verified by a numerical example and a practical example.