Learning Control Research Articles

Improving the Quality of Industrial Robot Control Using an Iterative Learning Method with Online Optimal Learning and Intelligent Online Learning Function Parameters

It is inevitable that the characteristics of a robot system change inaccurately or cannot be accurately determined during movement and are affected by external disturbances. There are many adaptive control methods, such as the exact linearization method, sliding control, or neural control, to improve the quality of trajectory tracking for a robot’s motion system. However, those methods require a great deal of computation to solve the constrained nonlinear optimization problem. This article first presents some techniques for determining the online learning function parameters of an intelligent controller, including two circuits: the inner circuit is an uncertain function component estimator to compensate for the robot’s input, and the outer circuit is an iterative learning controller and does not use a mathematical model of the robot with optimal online learning function parameters. The optimal condition is based on the model in the time domain to determine the learning function parameters that change adaptively according to the sum of squared tracking errors of each loop. As for the intelligent online learning function parameters, they closely follow the general model to stabilize the robot system, based on the principle of intelligent estimation of the uncertainty component and total noise. This method is built on Taylor series analysis for the state vector and does not use a mathematical model of the system at all. It allows feedback linearization, as well as intelligent stabilization of the system. This article’s content uses a 2-DOF flat robot implemented on MatlabR2022b software to verify the theory. These findings indicate that superior tracking performance is achievable.

Distributed control of partial differential equations using convolutional reinforcement learning

We present a convolutional framework which significantly reduces the complexity and thus, the computational effort for distributed reinforcement learning control of dynamical systems governed by partial differential equations (PDEs). Exploiting translational equivariances, the high-dimensional distributed control problem can be transformed into a multi-agent control problem with many identical, uncoupled agents. Furthermore, using the fact that information is transported with finite velocity in many cases, the dimension of the agents’ environment can be drastically reduced using a convolution operation over the state space of the PDE, by which we effectively tackle the curse of dimensionality otherwise present in deep reinforcement learning. In this setting, the complexity can be flexibly adjusted via the kernel width or by using a stride greater than one (meaning that we do not place an actuator at each sensor location). Moreover, scaling from smaller to larger domains – or the transfer between different domains – becomes a straightforward task requiring little effort. We demonstrate the performance of the proposed framework using several PDE examples with increasing complexity, where stabilization is achieved by training a low-dimensional deep deterministic policy gradient agent using minimal computing resources.

PD-Based Iterative Learning Control for the Nonlinear Low-Speed-Jitter Vibration of a Wind Turbine in Yaw Motion

Aiming at the nonlinear low-speed-jitter (LSJ) vibration suppression for a yaw system of a megawatt wind turbine, a kinematics mechanism of the yaw system is investigated from the perspective of tribology, and a kinematics model of the yaw system based on an equilibrium position is established. On the basis of the dynamic modeling of the yaw system, a nonlinear mathematical model of the LSJ system is deduced. Based on the two lead motors’ driving of the conventional yaw motion, an innovative design with a special installation of two auxiliary motors for yaw transmission is carried out, which is integrated with a matching centralized lubrication system (CLS). Based on open-loop proportional-derivative (PD) control and the iterative learning control methods of the time-varying continuous system, the stability control and jitter amplitude suppression of the yaw system are realized by using a combined driving torque provided by the lead and auxiliary gears. From the stability and convergence of the time-domain response and the convergence of the iterative error, the effectiveness of the iterative learning control method with the PD-based regulation is verified, and its advantages for engineering applications are shown based on the algorithm solver improvement. The feasibility of the physical realization and engineering application of the control methodology is verified by using controller-hardware-in-the-loop (C-HITL) simulation technology.

Digital, Self-Regulated Vocabulary Learning and Device Control In Out-Of-Class, Higher Education Settings

Self-regulation of learning behaviour is particularly important when it comes to vocabulary learning for academic purposes in a second language because it often needs to be done on a regular and consistent basis and mostly in out-of-class, self-directed settings to be successful. Self-regulation is also vital when this learning takes place using digital activities on smartphones because these are now ubiquitous devices and deeply embedded in both daily life and higher education settings. Features such as notifications from social media applications can end up distracting students from their academic tasks unless they have the capacity to manage and control their behaviour. This naturalistic, mixed methods study conducted with students on an academic English foundation course in a higher education context aimed to measure their capacity for self-regulated vocabulary learning through technology before and after 10 weeks of intentional digital vocabulary learning in out-of-class settings and to see if there was any difference between learning on a laptop and a smartphone. The purpose of this study was to find out if device control was a relevant dimension of self-regulation, which is an under-researched area. The study collected quantitative data through a recently developed self-report survey tool, and differences in scores were measured using Wilcoxon signed-rank tests. Qualitative data was also collected from students through paired-depth interviews, and this was analysed using typological analysis. The results revealed that the students’ self-reported capacity for self-regulated vocabulary learning through laptops was significantly higher than their capacity for self-regulated vocabulary learning through smartphones. In addition, commitment regulation when using a smartphone decreased significantly over the 10-week period primarily due to distractions from social media notifications. At the same time, students were aware of when to use each device for different types of learning activities and under different temporal and spatial conditions. Overall, this study showed that device control should be considered an additional dimension of a model of digital, self-regulated vocabulary learning and should also be incorporated into future research in the field of e-learning. In addition, students in higher education need to be given more guidance about the benefits and drawbacks of different devices and how to develop their capacity and strategies for greater device self-regulation.

Segment-wise learning control for trajectory tracking of robot manipulators under iteration-dependent periods

Segment-wise learning control for trajectory tracking of robot manipulators under iteration-dependent periods

An innovative heterogeneous transfer learning framework to enhance the scalability of deep reinforcement learning controllers in buildings with integrated energy systems

AbstractDeep Reinforcement Learning (DRL)-based control shows enhanced performance in the management of integrated energy systems when compared with Rule-Based Controllers (RBCs), but it still lacks scalability and generalisation due to the necessity of using tailored models for the training process. Transfer Learning (TL) is a potential solution to address this limitation. However, existing TL applications in building control have been mostly tested among buildings with similar features, not addressing the need to scale up advanced control in real-world scenarios with diverse energy systems. This paper assesses the performance of an online heterogeneous TL strategy, comparing it with RBC and offline and online DRL controllers in a simulation setup using EnergyPlus and Python. The study tests the transfer in both transductive and inductive settings of a DRL policy designed to manage a chiller coupled with a Thermal Energy Storage (TES). The control policy is pre-trained on a source building and transferred to various target buildings characterised by an integrated energy system including photovoltaic and battery energy storage systems, different building envelope features, occupancy schedule and boundary conditions (e.g., weather and price signal). The TL approach incorporates model slicing, imitation learning and fine-tuning to handle diverse state spaces and reward functions between source and target buildings. Results show that the proposed methodology leads to a reduction of 10% in electricity cost and between 10% and 40% in the mean value of the daily average temperature violation rate compared to RBC and online DRL controllers. Moreover, online TL maximises self-sufficiency and self-consumption by 9% and 11% with respect to RBC. Conversely, online TL achieves worse performance compared to offline DRL in either transductive or inductive settings. However, offline Deep Reinforcement Learning (DRL) agents should be trained at least for 15 episodes to reach the same level of performance as the online TL. Therefore, the proposed online TL methodology is effective, completely model-free and it can be directly implemented in real buildings with satisfying performance.

Sampling-based event-triggered iterative learning control in nonlinear hyperbolic distributed parameter systems

This article explores a sampling-based event-triggered iterative learning control (SETILC) method for achieving precise control in nonlinear hyperbolic distributed parameter systems. Firstly, by designing a Lyapunov function regarding the system output error, reasonable triggering conditions are derived to selectively update the system control input. Subsequently, a sampling-based event-triggered iterative learning control algorithm is proposed to further enhance the control performance of the system while avoiding the occurrence of Zeno phenomena. A condition sufficient for ensuring the convergence of the system’s tracking error to zero is presented, and its convergence is proven. Finally, the proposed method’s effectiveness is verified through numerical simulations.

Multiagent Reinforcement Learning for Active Guidance Control of Railway Vehicles with Independently Rotating Wheels

This paper presents a novel data-driven multiagent reinforcement learning (MARL) controller for enhancing the running stability of independently rotating wheels (IRW) and reducing wheel–rail wear. We base our active guidance controller on the multiagent deep deterministic policy gradient (MADDPG) algorithm. In this framework, each IRW controller is treated as an independent agent, facilitating localized control of individual wheelsets and reducing the complexity of the required observations. Furthermore, we enhance the MADDPG algorithm with prioritized experience replay (PER), resulting in the PER-MADDPG algorithm, which optimizes training convergence and stability by prioritizing informative experience samples. In this paper, we compare the PER-MADDPG algorithm against existing controllers, demonstrating the superior simulation performance of the proposed algorithm, particularly in terms of self-centering capability and curve-negotiation behavior, effectively reducing the wear number. We also develop a scaled IRW vehicle for active guidance experiments. The experimental results validate the enhanced running performance of IRW vehicles using our proposed controller.

The impact of using reinforcement learning to personalize communication on medication adherence: findings from the REINFORCE trial

Text messaging can promote healthy behaviors, like adherence to medication, yet its effectiveness remains modest, in part because message content is rarely personalized. Reinforcement learning has been used in consumer technology to personalize content but with limited application in healthcare. We tested a reinforcement learning program that identifies individual responsiveness (“adherence”) to text message content and personalizes messaging accordingly. We randomized 60 individuals with diabetes and glycated hemoglobin A1c [HbA1c] ≥ 7.5% to reinforcement learning intervention or control (no messages). Both arms received electronic pill bottles to measure adherence. The intervention improved absolute adjusted adherence by 13.6% (95%CI: 1.7%–27.1%) versus control and was more effective in patients with HbA1c 7.5- < 9.0% (36.6%, 95%CI: 25.1%–48.2%, interaction p < 0.001). We also explored whether individual patient characteristics were associated with differential response to tested behavioral factors and unique clusters of responsiveness. Reinforcement learning may be a promising approach to improve adherence and personalize communication at scale.

Ripple force compensation based on ILC–ASMC for linear servo system

The ripple force, which displays intricate dynamic characteristics and is fundamentally nonlinear in linear servo systems, poses challenges in modeling and compensation. To achieve superior suppression of the ripple force, this paper proposes a new scheme called the iterative learning control (ILC)–adaptive sliding-mode control (ASMC), which combines ASMC and ILC. Initially, the servo system model is established, followed by an analysis of the ripple force mechanism and obtaining a simplified model. Then, an adaptive sliding-mode structure with a feed-forward component is proposed to enhance the servo dynamic response based on the design of a sliding-mode surface. In addition, a new ripple force compensator based on the ILC is designed, allowing for iterative learning from the previous period to accurately estimate the ripple force online. It is worth noting that the process is divided into the 0th iteration and the ith ( i ≥ 1) iteration to avoid the selection for initial values. The 0th iteration ensures initial system stability and provides the initial iteration values, while subsequent iterations approximate the real ripple force through iterative learning. Importantly, the controller and the compensator are combined through the same sliding-mode surface, obviously simplifying the system structure. Finally, experimental results validate the effectiveness of the proposed scheme.

No pupillometric evidence for effortful proactive control in the proportion-congruent Stroop paradigm.

Cognitive control is the ability to allocate attention away from stimuli that are irrelevant to achieving a goal, towards stimuli that are. When conflict is anticipated, attention is biased in a global, top-down manner called proactive control and this effortful type of cognitive control is engaged before stimulus onset. The list-wise congruency proportion (LWPC) effect, where the Stroop congruency effect is reduced when there are more incongruent than congruent trials compared to vice versa, has been viewed as one of the prime signatures of this type of cognitive control. However, there has been recent debate about the extent to which this effect should be attributed to proactive control instead of alternative explanations such as simpler associative learning or reactive control. Thus, by using pupillometry (i.e., an indicator of cognitive effort), the present study investigated the extent to which LWPC effects result from effortful proactive control. Experiment 1 employed a classic proportion congruency manipulation, while Experiment 2 replaced congruent trials with neutral trials to control for potential effects of associative learning. While in line with past findings, proportion congruency effects were obtained in response times of both experiments and pupillometry showed both proportion congruency and Stroop effects after stimulus onset, no differences in pupil sizes were found during the preparatory phase. Therefore, these results do not support the idea that the observed LWPC effects are due to participants engaging in effortful proactive control.

Acceleration and deceleration control algorithm of CNC system based on deep neural network reinforcement learning and MCU

Acceleration and deceleration control, as one of the key technologies in high-speed CNC system, directly affects the machining efficiency, the stability of machining process and the error of machining follow. Therefore, it is necessary to study and explore new acceleration and deceleration control methods in high-speed CNC system to ensure the smooth feed and improve the machining accuracy. Therefore, the acceleration and deceleration control algorithm of NC system based on deep reinforcement learning and single chip microcomputer is studied. The theoretical basis of deep reinforcement learning is analyzed, and the acceleration of acceleration and deceleration is calculated based on the linear acceleration and deceleration control. The whole integer operation of single chip microcomputer is used to estimate the value range of each step. The variation characteristics of trajectory motion are predicted so that acceleration and deceleration can be processed across program segments. The velocity of transfer point is calculated by the rate of change of feed velocity vector, and the speed of multi-program is smoothed by adjusting the allowable contour error. Based on the proximal strategy optimization algorithm in deep reinforcement learning, the acceleration and deceleration control model of CNC system is established to realize the acceleration and deceleration control. The experimental results show that the proposed algorithm has better control effect, shorter time and smaller interpolation error, which can ensure the NC system to feed smoothly at high speed.

Precise control of magnetic soft microrobot in flowing environment

Microrobots have demonstrated attractive advantages for biomedical use, ranging from cargo delivery to invasive therapies. However, it remains a challenge to develop a universal control method for various microrobots to guarantee the precision of microrobot-based executions and further spread their applications. Herein, we propose an iterative learning control (ILC) method that can achieve precise position control for microrobots in a bionic flowing environment. Although the kinematics system of the microrobots and flowing environment has severe nonlinearities and uncertainties, the model- free control method, ILC, can optimize the input gradually to accurately accomplish microrobots’ arrangements, including position control and drug release. Oil-droplet-based magnetic soft microrobots are utilized as control objects to prove the advantages of the proposed control method. Two electro- magnets are set up as actuation devices, and a microchannel with a flowing medium mimics the vessel environment. By rigorous mathematical analysis and numerous experiments, the proposed method is verified to be effective, and the results show that the microrobots can be precisely controlled to the desired position in the flowing environment. These results pave a novel and accessible path to achieve precise position control for various microrobots.

Iterative learning–based model-free adaptive precise heading following of an autonomous underwater vehicle with unknown disturbances

Due to the nonlinearity, strong coupling, and uncertain parameters of autonomous underwater vehicle (AUV), it is difficult to build an accurate dynamic model, which makes precise control of AUV extremely challenging. To handle the precise heading-following problem of AUV, this paper proposes an iterative learning-based redefine model-free adaptive heading control (IL-RMFAC) method for the underactuated AUV with unknown disturbances based on data driven. The control scheme consists of a learning control algorithm, a parameter iterative update algorithm, and a parameter reset algorithm. It is designed using only the input and output (I/O) data of the controlled system and is a data-driven control method. The pseudo partial derivative (PPD) can be iteratively calculated through the parameter iterative update algorithm and reset algorithm to adjust the learning gain, solving the problem of strictly limited initial position of the traditional fixed learning gain iterative learning control (ILC). A linear combination of angle and angular velocity is introduced in the kinematic layer to avoid overshooting of the expected following target, and an iterative learning method is introduced in the dynamics to improve the accuracy. As the number of iterations increases, the steady-state error is gradually decreased. Finally, by comparing traditional proportional–integral–derivative (PID) simulations, the proposed algorithm’s effectiveness and outstanding performance for the AUV heading tracking are confirmed.

Optimal state filters for networked iterative learning control systems with data losses and noises

SummaryData losses and noises in both forward and feedback channels significantly impact the convergence of networked iterative learning control (ILC) systems. To address this issue, this article considers a class of linear time‐invariant objects controlled by proportional ILC controllers, an optimal state filter is then designed at the ILC controller side that aims to guarantee the convergence of the input transmitted by ILC controllers. First, two data transmission processes are introduced to account for the effects of data losses and noises. Second, a filtering model is established utilizing only the object information and the aforementioned data transmission processes. Third, the optimal state filter is designed on the basis of the orthogonal projection principle. This filtered state facilitates the acquisition of actual output errors, thus improving the convergence of the input transmitted by ILC controllers. Simulation results demonstrate the effectiveness of the proposed state filters.

Grey wolf optimization algorithm-based robust neural learning control of passive torque simulators with predetermined performance

Grey wolf optimization algorithm-based robust neural learning control of passive torque simulators with predetermined performance

Machine learning controller for data rate management in science DMZ networks

This article presents a Machine Learning Controller (MLC) supported by a P4 switch for improving rate control in non-dedicated Science Demilitarized Zone (Science DMZ) cyberinfrastructures. The proposed scheme utilizes passive data plane measurements such as Round Trip Time (RTT), throughput, queuing delay, and active flow count to regulate campus network output and achieve a desired Data Transfer Node (DTN) target rate. We evaluated our solution through a testbed using a bare-metal data plane switch, legacy router, and emulated hosts. Results show that including a rate controller based on data plane programmable devices on a non-dedicated Science DMZ cyberinfrastructure can effectively improve the completion time of scientific big data flows, while having a low impact on the campus network traffic and bottleneck link utilization. Specifically, the proposed controller achieved an average improvement of 21.72% in Flow Completion Time (FCT) compared to a trivial fixed-rate solution when the DTN uses BBR2 as a Congestion Control Algorithm (CCA). The results highlight the potential of machine learning techniques in conjunction with data plane measurements for optimizing the performance of non-dedicated networks.

Learning-Based Parameter Optimization for a Class of Orbital Tracking Control Laws

This paper presents a learning algorithm for tuning the parameters of a family of stabilizing nonlinear controllers for orbital tracking, in order to minimize a cost function which combines convergence time and fuel consumption. The main feature of the proposed approach is that it achieves performance optimization while guaranteeing closed-loop stability of the resulting controller. This property is exploited also to restrict the class of admissible controllers and hence to expedite the training process. The learning algorithm is tested on three case studies: two different orbital transfers and a rendezvous mission. Numerical simulations show that the learned control parameters lead to a significant improvement of the considered performance measure.

Monitoring carbon emissions using deep learning and statistical process control: a strategy for impact assessment of governments’ carbon reduction policies

Across the globe, governments are developing policies and strategies to reduce carbon emissions to address climate change. Monitoring the impact of governments’ carbon reduction policies can significantly enhance our ability to combat climate change and meet emissions reduction targets. One promising area in this regard is the role of artificial intelligence (AI) in carbon reduction policy and strategy monitoring. While researchers have explored applications of AI on data from various sources, including sensors, satellites, and social media, to identify areas for carbon emissions reduction, AI applications in tracking the effect of governments’ carbon reduction plans have been limited. This study presents an AI framework based on long short-term memory (LSTM) and statistical process control (SPC) for the monitoring of variations in carbon emissions, using UK annual CO2 emission (per capita) data, covering a period between 1750 and 2021. This paper used LSTM to develop a surrogate model for the UK’s carbon emissions characteristics and behaviours. As observed in our experiments, LSTM has better predictive abilities than ARIMA, Exponential Smoothing and feedforward artificial neural networks (ANN) in predicting CO2 emissions on a yearly prediction horizon. Using the deviation of the recorded emission data from the surrogate process, the variations and trends in these behaviours are then analysed using SPC, specifically Shewhart individual/moving range control charts. The result shows several assignable variations between the mid-1990s and 2021, which correlate with some notable UK government commitments to lower carbon emissions within this period. The framework presented in this paper can help identify periods of significant deviations from a country’s normal CO2 emissions, which can potentially result from the government’s carbon reduction policies or activities that can alter the amount of CO2 emissions.

Iterative learning control for differential inclusion systems with random fading channels by varying average technique.

The aim of this paper is to study iterative learning control for differential inclusion systems with random fading channels between the plant and the controller. In reality, the phenomenon of fading will inevitably occur in network transmission, which will greatly affect the tracking ability of output trajectory. This study discusses the impact of fading channel on tracking performance at the input and output sides, respectively. First, a set-valued mapping in a differential inclusion system with uncertainty is converted into a single-valued mapping by means of a Steiner-type selector. Then, to offset the effect of the fading channel and improve the tracking ability, a variable local average operator is constructed. The convergence of the learning control algorithm designed by the average operator is proved. The results show that the parameters in the varying local average operator can be adjusted to trade-off between the learning rate and the fading offset rate. Finally, the theoretical results are verified by numerical simulation of the switched reluctance motors.