Closed-loop Learning Research Articles

Nonlinear robust integral based actor–critic reinforcement learning control for a perturbed three-wheeled mobile robot with mecanum wheels

In this article, a novel Robust Integral of the Sign of the Error (RISE)-based Actor/Critic reinforcement learning control structure is established, which addresses the trajectory tracking control problem, optimality performance and observer effectiveness of a three mecanum wheeled mobile robot to be subject to slipping effect. The Actor–Critic Reinforcement Learning algorithm with a discount factor is introduced in integration with the Nonlinear RISE feedback term, which is designated to eliminate the dynamic uncertainties/disturbances from the affine nominal system. On the other hand, the persistence of excitation (PE) condition can be ignored due to the presence of RISE term. Stability analyses in two proposed theorems demonstrate all the signals in the closed-loop system and learning weights would be Uniformly Ultimate Boundedness (UUB) and the consideration of the system under the impact of RISE that can promote the tracking effectiveness. In conclusion, simulation results are shown in conjunction with the comparison to illustrate the powerful capability as well as the economy in control resources of the proposed algorithm.

Forced convection heat transfer control for cylinder via closed-loop continuous goal-oriented reinforcement learning

Forced convection heat transfer control offers considerable engineering value. This study focuses on a two-dimensional rapid temperature control problem in a heat exchange system, where a cylindrical heat source is immersed in a narrow cavity. First, a closed-loop continuous deep reinforcement learning (DRL) framework based on the deep deterministic policy gradient (DDPG) algorithm is developed. This framework swiftly achieves the target temperature with a temperature variance of 0.0116, which is only 5.7% of discrete frameworks. Particle tracking technology is used to analyze the evolution of flow and heat transfer under different control strategies. Due to the broader action space for exploration, continuous algorithms inherently excel in addressing delicate control issues. Furthermore, to address the deficiency that traditional DRL-based active flow control (AFC) frameworks require retraining with each goal changes and cost substantial computational resources to develop strategies for varied goals, the goal information is directly embedded into the agent, and the hindsight experience replay (HER) is employed to improve the training stability and sample efficiency. Then, a closed-loop continuous goal-oriented reinforcement learning (GoRL) framework based on the HER-DDPG algorithm is first proposed to perform real-time rapid temperature transition control and address multiple goals without retraining. Generalization tests show the proposed GoRL framework accomplishes multi-goal tasks with a temperature variance of 0.0121, which is only 5.8% of discrete frameworks, and consumes merely 11% of the computational resources compared with frameworks without goal-oriented capability. The GoRL framework greatly enhances the ability of AFC systems to handle multiple targets and time-varying goals.

An iterative learning-based integrated motion planning and control method for autonomous patrolling of unmanned surface vehicles

Abstract The minimization of maritime patrol durations and the optimization of patrol trajectories are still a prevailing challenge for autonomous navigation of unmanned surface vehicles (USVs). In this paper, we propose an integrated trajectory planning and control method to achieve time-optimal autonomous patrol of USVs by a data-efficient iterative learning-based predictive control algorithm. We build the optimization problem of the algorithm by introducing a local cost and a local safe constraint set for closed-loop efficient data-driven learning. To guarantee recursive feasibility, stability and convergence properties of the optimization algorithm at each iteration, the local cost and the local constraint set are designed and update iteratively using the historical vehicle states at previous iterations as a dataset. Different from traditional optimization methods, the proposed method does not require a reference path. The cost function of the optimization method decreases monotonically and converge to obtain a time-optimal control law for trajectory planning and tracking of USVs only after several iterations. The proposed approach is validated on the robotic operation system in typical maritime patrol scenarios. The results show that the proposed method implement a 5% reduction of patrol time, fewer lateral track errors, and smoother trajectories compared to the traditional MPC algorithm under water flow conditions. In addition, Enhancements in patrol operation efficiency of USVs are achieved by the algorithm, even under the constraints of varied patrol paths, attesting to its versatility.

DiffTune-MPC: Closed-Loop Learning for Model Predictive Control

DiffTune-MPC: Closed-Loop Learning for Model Predictive Control

The End is the Beginning is the End: The closed-loop learning analytics framework

This article provides a comprehensive review of current practices and methodologies within the field of learning analytics, structured around a dedicated closed-loop framework. This framework effectively integrates various aspects of learning analytics into a cohesive framework, emphasizing the interplay between data collection, processing and analysis, as well as adaptivity and personalization, all connected by the learners involved and underpinned by educational and psychological theory. In reviewing each step of the closed loop, the article delves into the advancements in data collection, exploring how technological progress has expanded data collection methods, particularly focusing on the potential of multimodal data acquisition and how theory can inform this step. The processing and analysis step is thoroughly reviewed, highlighting a range of methods including machine learning and AI, and discussing the critical balance between prediction accuracy and interpretability. The adaptivity and personalization step examines the current state of research, underscoring significant gaps and the necessity for theory-informed, personalized learning interventions. Overall, the article underscores the importance of interdisciplinarity in learning analytics, advocating for the integration of insights from various fields to address challenges such as ethical data usage and the creation of quality learning experiences. This framework and review aim to guide future research and practice in learning analytics, promoting the development of effective, learner-centric educational environments driven by balancing data-driven insights and theoretical understanding.

Iterative Learning with Adaptive Sliding Mode Control for Trajectory Tracking of Fast Tool Servo Systems

To address the tracking control problem of the periodic motion fast tool servo system (FTS), we propose a control method that combines adaptive sliding mode control with closed-loop iterative learning control. Adaptive sliding mode control enhances the system’s robustness to external non-repetitive disturbances, and exponential gain iterative learning control compensates for the influence of periodic disturbances such as cutting force. The experimental results show that the proposed iterative learning controller based on adaptive sliding mode control can effectively eliminate the influence of various interference factors, achieve accurate tracking of the FTS system’s motion trajectory within a limited number of iterations, and ensure the stability of the system, which has the advantages of a fast convergence speed, high tracking accuracy, and strong robustness.

Interactions between circuit architecture and plasticity in a closed-loop cerebellar system.

Determining the sites and directions of plasticity underlying changes in neural activity and behavior is critical for understanding mechanisms of learning. Identifying such plasticity from neural recording data can be challenging due to feedback pathways that impede reasoning about cause and effect. We studied interactions between feedback, neural activity, and plasticity in the context of a closed-loop motor learning task for which there is disagreement about the loci and directions of plasticity: vestibulo-ocular reflex learning. We constructed a set of circuit models that differed in the strength of their recurrent feedback, from no feedback to very strong feedback. Despite these differences, each model successfully fit a large set of neural and behavioral data. However, the patterns of plasticity predicted by the models fundamentally differed, with the direction of plasticity at a key site changing from depression to potentiation as feedback strength increased. Guided by our analysis, we suggest how such models can be experimentally disambiguated. Our results address a long-standing debate regarding cerebellum-dependent motor learning, suggesting a reconciliation in which learning-related changes in the strength of synaptic inputs to Purkinje cells are compatible with seemingly oppositely directed changes in Purkinje cell spiking activity. More broadly, these results demonstrate how changes in neural activity over learning can appear to contradict the sign of the underlying plasticity when either internal feedback or feedback through the environment is present.

Precise tracking control via iterative learning for one-sided Lipschitz Caputo fractional-order systems.

This paper investigates iterative learning control for Caputo fractional-order systems with one-sided Lipschitz nonlinearity. Both open- and closed-loop P-type learning algorithms are proposed to achieve perfect tracking for the desired trajectory, and their convergence conditions are established. It is shown that the algorithms can make the output tracking error converge to zero along the iteration axis. A simulation example illustrates the application of the theoretical findings, and shows the effectiveness of the proposed approach.

Federated Closed-Loop Learning for Cross-modal Generation of Tactile Friction in Tactile Internet

Federated Closed-Loop Learning for Cross-modal Generation of Tactile Friction in Tactile Internet

Stability-informed Bayesian Optimization for MPC Cost Function Learning

Designing predictive controllers towards optimal closed-loop performance while maintaining safety and stability is challenging. This work explores closed-loop learning for predictive control parameters under imperfect information while considering closed-loop stability. We employ constrained Bayesian optimization to learn a model predictive controller's (MPC) cost function parametrized as a feedforward neural network, optimizing closed-loop behavior as well as minimizing model-plant mismatch. Doing so offers a high degree of freedom and, thus, the opportunity for efficient and global optimization towards the desired and optimal closed-loop behavior. We extend this framework by stability constraints on the learned controller parameters, exploiting the optimal value function of the underlying MPC as a Lyapunov candidate. The effectiveness of the proposed approach is underlined in simulations, highlighting its performance and safety capabilities.

Machine-learning-accelerated simulations to enable automatic surface reconstruction.

Understanding material surfaces and interfaces is vital in applications such as catalysis or electronics. By combining energies from electronic structure with statistical mechanics, ab initio simulations can, in principle, predict the structure of material surfaces as a function of thermodynamic variables. However, accurate energy simulations are prohibitive when coupled to the vast phase space that must be statistically sampled. Here we present a bi-faceted computational loop to predict surface phase diagrams of multicomponent materials that accelerates both the energy scoring and statistical sampling methods. Fast, scalable and data-efficient machine learning interatomic potentials are trained on high-throughput density-functional-theory calculations through closed-loop active learning. Markov chain Monte Carlo sampling in the semigrand canonical ensemble is enabled by using virtual surface sites. The predicted surfaces for GaN(0001), Si(111) and SrTiO3(001) are in agreement with past work and indicate that the proposed strategy can model complex material surfaces and discover previously unreported surface terminations.

ADRLO: Adaptive deep reinforcement learning-based offloading for edge computing

Computation offloading, as an efficient and promising computing paradigm for mobile edge computing (MEC), provides optimal computation allocation between MEC servers and Internet of Things (IoT) devices. However, in modern wireless communications, most of the computational tasks are latency-sensitive, presenting an efficiency challenge for offloading services. Moreover, to make the best use of computation resources, it remains an issue to ensure stable optimal offloading for the allocation of computation resources. In this paper, a novel adaptive deep reinforcement learning-based offloading (ADRLO) framework is developed to achieve a stable optimal computation revenue with low execution latency for MEC offloading. Structurally, we implement a residual connected deep neural network (DNN) to generate a series of offloading strategies and use them to obtain the global system computation rate. Especially, the strategy that achieves the maximum computation rate is then selected as experience to support the DNN training, forming a closed-loop deep reinforcement learning (DRL) mechanism. To further enhance the efficiency and stability of the MEC offloading, an adaptive regulator is presented for the framework parameters including the size of action space and the DNN learning rate. Numerical results show that, compared to existing state-of-the-art approaches, our proposal achieves optimal computation revenue, exhibits better convergence performance, and significantly reduces the execution latency by almost 50%.

Closed-loop superconducting materials discovery

Discovery of novel materials is slow but necessary for societal progress. Here, we demonstrate a closed-loop machine learning (ML) approach to rapidly explore a large materials search space, accelerating the intentional discovery of superconducting compounds. By experimentally validating the results of the ML-generated superconductivity predictions and feeding those data back into the ML model to refine, we demonstrate that success rates for superconductor discovery can be more than doubled. Through four closed-loop cycles, we report discovery of a superconductor in the Zr-In-Ni system, re-discovery of five superconductors unknown in the training datasets, and identification of two additional phase diagrams of interest for new superconducting materials. Our work demonstrates the critical role experimental feedback provides in ML-driven discovery, and provides a blueprint for how to accelerate materials progress.

A NARX network optimized with an adaptive weighted square-root cubature Kalman filter for the dynamic state of charge estimation of lithium-ion batteries

Due to the high nonlinearities and unstable working conditions, accurately estimating the state of charge (SOC) by the battery management system (BMS) is a major challenge in ensuring the safety and reliability of lithium-ion batteries in electric vehicles. This paper presents a deep learning network, a nonlinear autoregressive model with exogenous inputs (NARX) network with a closed-loop architecture and transfer learning mechanism, which is optimized using a proposed adaptive weighted square-root cubature Kalman filter (AWSCKF) with a moving sliding window and an adaptive weighing coefficient for SOC estimation of lithium-ion batteries. The proposed AWSCKF method is established through square-root and cubature updates to optimize the statistical value of the state estimate, error covariance, and measurement noise covariance matrices, with the ability to incorporate high nonlinearities to filter out the noise, stabilize, and optimize the final SOC. To evaluate the effectiveness of the optimized NARX network and verify the proposed AWSCKF method, battery tests are carried out using a lithium cobalt oxide battery at various charge-discharge rates and a lithium nickel cobalt manganese oxide battery at temperatures of 0 and 45 °C under five complex working conditions. The SOC accuracy of lithium-ion batteries is enhanced by the hybrid method estimation process, which is based on sensitivity analysis and adaptation to various working conditions. The comprehensive results show that the proposed NARX-AWSCKF model achieves the overall best mean absolute error, root mean square error, and mean absolute percentage error values of 0.07293%, 0.0912%, and 0.40356%, respectively, under various complex conditions. By effectively utilizing battery domain knowledge for real-world BMS applications, the proposed model outperforms other existing methods in terms of high effectiveness, robustness, and potential to boost the NARX performance.

Ultra-Short-Term Wind Power Prediction Based on LSTM with Loss Shrinkage Adam

With the rapid increase in wind power, its strong randomness has brought great challenges to power system operation. Accurate and timely ultra-short-term wind power prediction is essential for the stable operation of power systems. In this paper, an LsAdam–LSTM model is proposed for ultra-short-term wind power prediction, which is obtained by accelerating the long short-term memory (LSTM) network using an improved Adam optimizer with loss shrinkage (LsAdam). For a specific network topology, training progress heavily depends on the learning rate. To make the training loss of LSTM shrink faster with standard Adam, we use the past training loss-changing information to finely tune the next learning rate. Therefore, we design a gain coefficient according to the loss change to adjust the global learning rate in every epoch. In this way, the loss change in the training process can be incorporated into the learning progress and a closed-loop adaptive learning rate tuning mechanism can be constructed. Drastic changes in network parameters will deteriorate learning progress and even make the model non-converging, so the gain coefficient is designed based on the arctangent function with self-limiting properties. Because the learning rate is iteratively tuned with past loss-changing information, the trained model will have better performance. The test results on a wind turbine show that the LsAdam–LSTM model can obtain higher prediction accuracy with much fewer training epochs compared with Adam–LSTM, and the prediction accuracy has significant improvements compared with BP and SVR models.

Guaranteed Closed-Loop Learning in Model Predictive Control

In this article, we propose a novel learning-based model predictive control framework for nonlinear systems which is able to guarantee closed-loop learning of the controlled system. We consider a cost function that combines a general economic cost with a user-defined learning cost function that aims at incentivizing learning of the unknown system. In particular, due to the finite horizon of the MPC scheme and to the presence of disturbances, the open-loop trajectory usually differs from the closed-loop one. Such a mismatch causes existing learning-based MPC schemes to only show a learning phase in the open-loop prediction, without providing any formal guarantee on the actual closed-loop learning. In this article, we show how existing MPC schemes can be easily modified in order to guarantee closed-loop learning of the system by including a suitable discount factor in the chosen learning cost function, and implementing an additional constraint in the original MPC scheme. We show that various techniques for online learning the system dynamics, such as kinky inference methods, Gaussian processes, or parametric approaches, can be used within the proposed general framework.

Robust Data-Driven Iterative Learning Control for Linear-Time-Invariant and Hammerstein-Wiener Systems.

Iterative learning control (ILC) relies on a finite-time interval output predictor to determine the output trajectory in each trial. Robust ILCs intend to model the uncertainties in the predictor and to guarantee the convergence of the learning process subject to such model errors. Despite the vast literature in ILCs, parameterizing the uncertainties with the stochastic errors in the predictor parameters identified from system I/O data and thus robustifying the ILC have not yet been targeted. This work is devoted to solving such problems in a data-driven fashion. The main contributions are two-fold. First, a data-driven ILC method is developed for LTI systems. The relationship is established between the errors in the predictor matrix and the stochastic disturbances to the system. Its robust monotonic convergence (RMC) is then linked with the closed-loop learning gain matrix that contains the predictor uncertainties and is analyzed based on a closed-form expectation of this gain matrix multiplied with its own transpose, that is, in a mean-square sense (MS-RMC). Second, the data-driven ILC and MS-RMC analysis are extended to nonlinear Hammerstein-Wiener (H-W) systems. The advantages of the proposed methods are finally verified via extensive simulations in terms of their convergence and uncorrelated tracking performance with the stochastic parametric uncertainties.

Multi-Objective Surrogate Modeling Through Transfer Learning for Telescopic Boom Forklift

Simulation and optimization methods have been widely used in forklift design due to their cost-effectiveness. However, this type of method involves challenges such as the accuracy of the simulation model and the simulation solution time. These challenges reduce the stability and precision of the surrogate model and hence generate further optimization errors. In this paper, a multi-objective surrogate modeling (MSM) method for telescopic boom forklifts based on closed-loop transfer learning is proposed in order to solve these challenges. The MSM consists of the following two steps: to pre-train an initial deep neural network model (deep model) with a large amount of existing simulation data from the same type of forklift and to transfer the model with a small amount of measurement data collected on the current forklift. A general framework for deep neural network (DNN) training is introduced to improve the approximation ability of the initial model. Moreover, a novel uncertainty-analysis-based sampling method is suggested for measurement data development, and combined with transfer learning to form a closed-loop mode to improve the stability of the final model. The superiority of MSM is demonstrated through comparative studies with the fine-tuning method on a telescopic boom forklift with two objectives. The experimental results show that the Correlation coefficient (R) of the deep model can reach 0.9971 by using only 80 sets of training data. In addition, it can also achieve an improvement of at least a 13.25% reduction in Root Mean Squared Error (RMSE) and a 9.19% reduction on average in Maximum Absolute Error (MAE), as well as stronger robustness compared to the benchmarks. Furthermore, it will provide a valuable reference for the simulation optimization of complex electromechanical products.

Iterative learning algorithms for boundary tracing problems of nonlinear fractional diffusion equations

<abstract><p>In this paper, the iterative learning control technique is extended to distributed parameter systems governed by nonlinear fractional diffusion equations. Based on $ P $-type and $ PI^{\theta} $-type iterative learning control methods, sufficient conditions for the convergences of systems are given. Finally, numerical examples are presented to illustrate the efficiency of the proposed iterative schemes. The numerical results show that the closed-loop iterative learning control scheme converges faster than the open-loop iterative learning control scheme and the $ PI^{\theta} $-type iterative learning control scheme converges faster than the $ P $-type and the $ PI $-type iterative learning control scheme.</p></abstract>

Closed-Loop Learning for Accuracy Improvement of Nonintrusive Load Monitoring in Smart Homes

Closed-Loop Learning for Accuracy Improvement of Nonintrusive Load Monitoring in Smart Homes