Self-learning Control Research Articles

Autonomous Cooperative Hunting with Rule-Based and Self-Learning Control for Multiagent Systems

This paper considers the problem of autonomous cooperative hunting in an unknown dynamic environment, where a group of mobile agents collaborate to capture a moving target. Due to the decentralized decision-making nature of multi-agent systems and the presence of real-world constraints, it is a challenging task. To solve this problem, an artificial rule based hunting algorithm (AR-HA) is firstly developed based on the principles of attraction and repulsion with heading adjustment, and each agent is controlled by the designed rules. Then, to further enhance the stability of cooperative hunting, a self-learning algorithm based on Twin Delayed Deep Deterministic policy gradient (SL-TD3) is proposed. Each agent is governed by its own SL-TD3 controller and learns independently from its interaction with the environment, taking advantage of the reward function designed based on the control rules of AR-HA. Besides, in order to improve training efficiency, imitation learning is employed to initialize the actor network. Experiments on both virtual and real robots demonstrate the effectiveness of the proposed algorithms for autonomous cooperative hunting.

Gait self-learning control based on reference trajectory generation online for an asymmetric limb rehabilitation exoskeleton

Lower limb exoskeleton (LEX) are widely used to assist stoke survivors with walking dysfunction, which is lack of a more flexible trajectory and fails to address the control challenge posed by gait variability and asymmetry in rehabilitation training. This paper introduces an asymmetric self-learning lower exoskeleton (AS-LEX) based on reference trajectory generation for the affected side. Motor intent of the unaffected limb based on thresholds was identified to classify the gait phase of stance and swing. A parameterized gait trajectory was generated online, namely a combination of circular trajectory in the stance phase and an elliptical trajectory in the swing phase. Gait self-learning control is presented to make the affected limb adaptively learn the gait parameters generated by the unaffected limb. Feasibility of the AS-LEX is demonstrated experimentally using three healthy subjects. Resuls demonstrate that overground walking in a more natural speed (with a stride length 600 mm and 700 mm) make subjects more actively learn gait of the affected side from the unaffected side. Additionally, experiments of the fatigue level of the affected limb and human-robot interaction torques were carried out, and the results indicate a more natural gait and reduced interaction forces with the AS-LEX.

Research on Self-Learning Control Method of Reusable Launch Vehicle Based on Neural Network Architecture Search

Research on Self-Learning Control Method of Reusable Launch Vehicle Based on Neural Network Architecture Search

Self-learning for translational control of elliptical orbit spacecraft formations

PurposeThis paper aims to investigate the relative translational control for multiple spacecraft formation flying. This paper proposes an engineering-friendly, structurally simple, fast and model-free control algorithm.Design/methodology/approachThis paper proposes a tanh-type self-learning control (SLC) approach with variable learning intensity (VLI) to guarantee global convergence of the tracking error. This control algorithm utilizes the controller's previous control information in addition to the current system state information and avoids complicating the control structure.FindingsThe proposed approach is model-free and can obtain the control law without accurate modeling of the spacecraft formation dynamics. The tanh function can tune the magnitude of the learning intensity to reduce the control saturation behavior when the tracking error is large.Practical implicationsThis algorithm is model-free, robust to perturbations such as disturbances and system uncertainties, and has a simple structure that is very conducive to engineering applications.Originality/valueThis paper verified the control performance of the proposed algorithm for spacecraft formation in the presence of disturbances by simulation and achieved high steady-state accuracy and response speed over comparisons.

Self-learning Controller Design for DC–DC Power Converters with Enhanced Dynamic Performance

Self-learning Controller Design for DC–DC Power Converters with Enhanced Dynamic Performance

Assessing the Effectiveness of an Innovative Thermal Energy Storage System Installed in a Building in a Moderate Continental Climatic Zone

In the present work, the operating results from an innovative, renewable, energy-based space-heating and domestic hot water (DHW) system are shown. The system used solar thermal energy as its primary source and was assisted by a shallow geothermal application in order to accommodate the space-heating and DHW needs of a domestic building in Austria. The system incorporated phase-change materials (PCMs) in specially designed containers to function as heat-storage modules and provide an energy storage capability for both the space-heating and DHW subsystems. This system was designed, implemented, and tested under real operating conditions in a building for a period of one year. The operating and energy results for the system are demonstrated in this work. The system was compared with a conventional one, and a reduction in the primary energy consumption equal to 84.3% was achieved. The maintenance and operating costs of the system were reduced by 79.7% compared to the conventional system, thus significantly contributing to the NZEB target of the building. The newly proposed system, although presenting an increased operating complexity, utilizes an innovative self-learning control system that manages all of its operations. The combination of a solar thermal energy source with thermal energy storage increases the use of renewable energy by extending the capacity of the system beyond the solar hours and using excess solar energy for space-heating needs. The thermal energy storage unit also increases the energy and economic efficiency of the geothermal heat pump by operating it during the hours of a reduced electricity tariff and using the stored energy during hours of a high electricity demand. The cost for the installation of such a system is higher than a conventional one, but due to the significantly decreased operating costs, the pay-back period was calculated to be 8.7 years.

Online Adaptive Dynamic Programming for Optimal Self-Learning Control of VTOL Aircraft Systems With Disturbances

In this paper, a novel online adaptive dynamic programming control algorithm is developed to solve optimal control problems for Vertical Take Off and Landing (VTOL) aircraft systems. Considering the input disturbances of the VTOL systems and adding perturbation interference term to the value function, a robust policy iteration-based ADP method will be introduced to obtain the optimal controller with an approximate optimal control approach. The convergence property is developed to ensure the value function converges to a finite neighborhood of the optimal one. The uniform ultimate boundedness (UUB) of the iterative errors will be strictly proved by Lyapunov stability theory. Finally, we will give mathematical simulation results with the developed method. Compared with sliding mode control (SMC) and linear quadratic regulator (LQR) methods, the simulation results illustrate better overall performance of the novel method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —As an effective tool to solve the optimal control problem of complex nonlinear systems, adaptive dynamic programming (ADP) can effectively overcome the computational problems caused by “curse of dimensionality”. VTOL aircraft system is a typical strongly coupled, underactuate, non-minimum phase system, and the sudden random disturbances will make the control task more challenging. Aiming at the particularity of VTOL system, a new online policy iteration-based ADP algorithm is developed to obtain optimal control with an approximate optimal control strategy in this paper. Decoupling and disturbances can be effectively avoided. convergence will be analyzed to make sure the value function converge to a finite neighborhood of the optimal value function. The UUB property of the system will be proved by the Lyapunov approach.

Optimal operational self-learning control for multi-time scale industrial processes with signal compensations

Since the multi-time scales and strong nonlinearity are objectively existed in the practical industrial processes, it is extremely challenging to design control policy for optimizing the industrial operation. Therefore, this paper focuses on the multi-time scale optimal operational control (OOC) problem with unmodeled dynamics. Under the framework of reinforcement learning (RL) technology, a self-learning model-free method by integrating zero-sum game and singular perturbation (SP) theories is proposed in this paper, such that composite controllers with signal compensation can be found under which the suboptimal operation of the whole process can be achieved using only measured data. The remarkable highlight in this paper is the novel design method of the composite controllers with signal compensations for unknown nonlinear systems with multi-rate operations. Finally, the effectiveness of the proposed method is verified by a practical example and a numerical example.

Q-Learning-Based Multi-Rate Optimal Control for Process Industries

This paper studies the multi-rate optimal control problem for a class of industrial processes, whose controlling rate will be set faster than the sampling rate sometimes. This multi-rate phenomenon makes the accurate modeling of control systems challenging and difficult. In this brief, we present a model-free self-learning control scheme for the real-time solution of this problem, combining the lifting technology and Q-learning. For the asynchronous periods, the lifting system is established first to reconstruct the input and output by stacking the control and sampling signals to a frame period, maintaining the original dynamic information. Then, Q-learning is adopted to learn the optimal control policy with the real-time data and the convergence analysis of the proposed algorithm is derived. In this way, the control actions are executed at a faster rate to obtain the better dynamic performance. Finally, a hardware-in-loop (HIL) simulation study for process industries is carried out, showing that the proposed approach has high tracking and real-time performance.

Dual Heuristic Dynamic Programming Enables Trajectory Tracking Control

_ This article, written by JPT Technology Editor Chris Carpenter, contains highlights of paper SPE 200271, “Dual Heuristic Dynamic Programming in the Oil and Gas Industry for Trajectory Tracking Control,” by Seaar Al-Dabooni, Basra Oil Company; Alaa Azeez Tawiq, Technical Institute of Basra; and Hussen Alshehab, Basra Oil Company. The paper has not been peer reviewed. _ The complete paper presents an artificial intelligence (AI) algorithm the authors call dual heuristic dynamic programming (DHDP) that is used to solve optimization-control problems. Fast, self-learning control based on DHDP is illustrated for trajectory-tracking levels on a quadruple tank system (QTS) consisting of four tanks and two electrical pumps with two pressure-control valves. Two artificial neural networks are constructed for the DHDP approach: the critic network (the provider of a critique or evaluated signals) and the actor network or controller (the provider of control signals). The DHDP controller is learned without human intervention. Approximate Dynamic Programming (ADP) Recently, many different types of artificial algorithms have been applied in petroleum fields to solve optimization problems. This complete paper introduces a new field of AI applicable to oil and gas, ADP. ADP is a useful tool to overcome the behavior of nonlinear systems and is a special algorithm of reinforcement learning (RL). The authors write that ADP can be viewed as consisting of three categories: heuristic dynamic programming (HDP), DHDP, and globalized HDP. ADP features two neural networks—an actor and a critic—to provide an optimal control signal and long-cost value, respectively. ADP has numerous applications. The complete paper references work that discusses control on turbo-generator and swarm-robot problems by use of DHDP and that illustrates that action-dependent HDP can obtain an optimal path by multirobot navigation. QTS is frequently used in the oil and gas industry. DHDP is used to control the voltage of the two pumps to follow the desired level (set-point-level value) of the tanks, an approach that can learn by itself (a self-learning controller). The complete paper devotes several pages to equations and parameters that describe HDP. In ADP, optimal control problems are solved, thereby allowing agents to select an optimal action to minimize a long-term cost value through solution of Bellman’s equation. RL and ADP are used to train the actor neural network to provide optimal actions based on minimizing the cost-to-go value produced from the critic network. The actor function approximator is denoted for the actor neural network. After full training of these networks, the optimal action values are obtained from the actor network. System Functionality The equipment receives the system states of the process through sensors, and the algorithm maximizes the reward by selecting the correct optimal action (control signal) to feed the equipment. The simulation results for applying DHDP with QTS as a benchmark test problem were obtained using MATLAB. QTS is illustrated as an example in the paper because QTS is widely used in most petroleum exploration or production fields as an entire system or in parts. Another reason for the authors’ choice of QTS as a test problem is that QTS features a difficult model to control that has a limited zone of operating parameters to be stable. The multi-input/multioutput (MIMO) model of QTS was a similar model as most MIMO devices in the oil and gas field. The overall learning-control-system performance was tested and compared with HDP and a well-known industrial controller, a proportional integral derivative (PID) using MATLAB programming. The simulation results of DHDP provide enhanced performance compared with the PID approach, with a 98.9002% improvement.

Adaptive self-learning controllers with disturbance compensation for automatic track guidance of industrial trucks

This paper presents an extended control concept for automatic track guidance of industrial trucks in intralogistic systems. It is based on Reinforcement Learning (RL), a method of Artificial Intelligence (AI). The presented approach is able to adapt itself to different industrial truck variants and to the associated specific vehicle parameters. In order to avoid starting the whole training of the controller for each truck variant from scratch, the training process is divided into two steps. In the first step, the controller is trained on a simplified linear model using parameters of a nominal vehicle variant. Based on this, the control parameters are only fine-tuned in the second step using a more complex nonlinear model, representing the real industrial truck. In this way, the controller is adapted to the actual truck variant and the corresponding parameter values. By using the nonlinear model, it can be ensured that the forklift's dynamic is approximated within the entire operating range, even at high steering angles. Moreover, the influence of the disturbance variable of the system (path curvature) is compensated by considering this a priori knowledge within the control design. Therefore, the Artificial Neural Networks (ANN) of the RL controller and the observation vector are suitably adjusted. In this way, the occurring path curvatures can be considered in both training steps and the control parameters can be optimized accordingly. Thus, the influence of the disturbance variable can be compensated, which significantly improves the control quality. In order to demonstrate this, the new approach is compared to an RL control concept, which is not considering the disturbance variable and to a classical two-degrees-of-freedom (2DoF) control approach.

A possible approach to the creation of self-learning control systems for autonomous robots

Unmanned systems often function in new, non-predetermined environment. This demands flexible and simultaneously stable function of these systems. This can be implemented by their adaptation and, as a final goal, self-learning or self-organization in control systems. Autonomous functioning in extreme environment requires self-learning drastically. Such environment can be hostile towards unmanned systems, it may contain sophisticated communication conditions. All these are deteriorated by poor on-board control algorithms. In some of such cases unmanned systems may become inefficient or, even more, useless in some cases. This brings to front necessity to create adaptive self-learning control systems for robots. These should be capable to generate efficient, or even optimal decisions in extreme environment. This paper presents on of possible approaches to create self-learning control systems based on so called decision along analogues.

Data-Driven Self-Learning Controller Design Approach for Power-Aware IoT Devices based on Double Q-Learning Strategy

Data-Driven Self-Learning Controller Design Approach for Power-Aware IoT Devices based on Double Q-Learning Strategy

Towards self-learning control of HVAC systems with the consideration of dynamic occupancy patterns: Application of model-free deep reinforcement learning

This study proposes a self-learning control system that aims to learn occupancy profiles, building energy consumption patterns, and lag-time of the heating, ventilation, and air-conditioning (HVAC) systems. The control system learns by interacting with the environment with no need to develop building models and occupancy prediction models. The controller is developed based on a double deep Q-networks (DDQN) algorithm, as a model-free reinforcement learning method. The system's performance is evaluated and compared with that of a model predictive control (MPC) system under two scenarios of perfect and actual occupancy predictions based on occupancy data collected from 20 residential units. The MPC is assisted by a genetic algorithm and supervised learning models for predicting future occupancy patterns, indoor operative temperature, and building energy consumption. The results show that in the case of using perfect occupancy prediction, the self-learning controller operates almost as well as the MPC while not requiring any models. When occupancy prediction uncertainty is added to the problem, the proposed method outperforms the MPC in terms of thermal comfort by increasing the average temperature deviation and deviation period by 0.24 °C and 7.87%, respectively. However, the DDQN agent causes significant thermal comfort violations during the initial training period. The system causes up to a 2.8% longer deviation period and a 0.32 °C higher average temperature deviation, compared with the performance of the fully-trained system.

Model predictive control of DC/DC boost converter with reinforcement learning

Power electronics is seeing an increase in the use of sophisticated self-learning controllers as single board computers and microcontrollers progress faster. Traditional controllers, such as PI controllers, suffer from transient instability difficulties. The duty cycle and output voltage of a DC/DC converter are not linear. Due to this non-linearity, the PI controller generates variable levels of voltage fluctuations depending on the operating region of the converter. In some cases, non-linear controllers outperform PI controllers. The non-linear model of a non-linear controller is determined by data availability. So, a self-calibrating controller that collects data and optimizes itself as the operation goes on is necessary. Iteration and oscillation can be minimized with a well-trained reinforcement learning model utilizing a non-linear policy. A boost converter's output power supply capacity changes with a change in load, due to which the maximum duty cycle limit of a converter also changes. A support vector calibrated by reinforcement learning can dynamically change the duty cycle limit of a converter under variable load. This research highlights how reinforcement learning-based non-linear controllers can improve control and efficiency over standard controllers. The proposed concept is based on a microgrid system. Simulation and experimental analysis have been conducted on how reinforcement learning-based controller works for DC-DC boost converter.

Self-regulated learning in online classes: A comparative study between Malaysian and Indonesian students

Self-regulated learning (SRL) becomes a crucial part for the success of online classroom activities during the COVID-19 pandemic. To date, studies exploring students’ SRL in the contexts of Malaysian and Indonesian universities are still lacking and remain more empirical evidence. This study aims to examine and compare both university students’ SRL toward online learning activities at the time of global pandemic. The required data are quantitatively collected from 103 undergraduates from University A in Malaysia and 119 college students from University B in Indonesia by using an online questionnaire on students’ self-regulated learning adapted from Motivated Strategies for Learning Questionnaire (MSLQ). The sudden shifting from face-to-face interaction to online class instruction is a new learning style and could be challenging, but the finding of this study shows that the students from both universities achieve high mean scores on the SRL questionnaire. This finding indicates that the students have a positive attitude, high academic motivation levels, and good self-learning control over themselves during the online classes. This study contributes to the extension of the current literature about self-regulated learning in higher education context. This study further suggests that teachers could assist students’ self-regulated learning to improve their academic performances by keeping their motivation and engagement, particularly in the time of online classes during COVID-19 pandemic era. Hence, future studies are recommended to investigate the impact of students’ SRL on learning achievement as it can help teachers to enhance students’ performance in online classes.

Self-learning control of model uncertain active suspension systems with observer–critic structure

This paper presents a self-learning control algorithm for model uncertain suspension systems using single network adaptive critic (SNAC) approach. First, a differential neural network (DNN) observer in conjunction with the weight updating law is established to observe the uncertain dynamic. Then, the nominal optimal value function is approximated by a critic NN whose weight is updated by a novel design learning law driven by the filtered parameter error. The online self-learning control policy is thus derived by approximately solving the Hamilton–Jacobi–Bellman (HJB) equation based on SNAC technique. The Lyapunov approach is synthesized to ensure the convergent characteristics of the entire closed-loop system composed of the DNN observer and the self-learning control policy. Computer simulation of a quarter car suspension system is established to verify the effectiveness of the proposed approach. Simulation results illustrated that the designed method can ensure the good performance in terms with the road hold and ride quality. In addition, independent of model and online self-learning characteristics make it possible to design a high-performance vehicle active suspension controller.

Latent Thermal Energy Storage Application in a Residential Building at a Mediterranean Climate

An innovative thermal energy storage system (TESSe2b) was retrofitted in a residential building in Cyprus with a typical Mediterranean climate. The system comprises flat-plate solar collectors, thermal energy storage tanks filled with organic phase change material, a geothermal installation consisting of borehole heat exchangers with and without phase change material and a ground source heat pump, an advanced self-learning control system, backup devices and several other auxiliary components. The thermal energy storage tanks cover the building’s needs at certain temperature ranges (10–17 °C for cooling, 38–45 °C for heating and 50–60 °C for domestic hot water). A performance evaluation was conducted by comparing the TESSe2b system with the existing conventional heating and cooling system. The systems were simulated using commercial software, and the performance of the systems and the building’s energy needs were calculated. Based on the energy quantities, an economic analysis followed. The equivalent annual primary energy consumption with the conventional system resulted in being 43335 kWh, while for the storage system, it was only 8398 kWh. The payback period for the storage system was calculated to be equal to 9.76 years. The operation of the installed storage system provided data for calculations of the seasonal performance factor and storage performance. The seasonal performance factor values were very high during June, July and August, since the TESSe2b system works very efficiently in cooling mode due to the very high temperatures that dominate in Cyprus. The measured stored thermal energy for cooling, heating and domestic hot water resulted in being 14.5, 21.9 and 6.2 kWh, respectively. Moreover, the total volume of the phase change material thermal energy storage tanks for heating and domestic hot water was calculated to be roughly several times smaller than the volume of a tank with water as a storage medium.

Data-Driven Self-Learning Controller for Power-Aware Mobile Monitoring IoT Devices

Nowadays, there is a significant need for maintenance free modern Internet of things (IoT) devices which can monitor an environment. IoT devices such as these are mobile embedded devices which provide data to the internet via Low Power Wide Area Network (LPWAN). LPWAN is a promising communications technology which allows machine to machine (M2M) communication and is suitable for small mobile embedded devices. The paper presents a novel data-driven self-learning (DDSL) controller algorithm which is dedicated to controlling small mobile maintenance-free embedded IoT devices. The DDSL algorithm is based on a modified Q-learning algorithm which allows energy efficient data-driven behavior of mobile embedded IoT devices. The aim of the DDSL algorithm is to dynamically set operation duty cycles according to the estimation of future collected data values, leading to effective operation of power-aware systems. The presented novel solution was tested on a historical data set and compared with a fixed duty cycle reference algorithm. The root mean square error (RMSE) and measurements parameters considered for the DDSL algorithm were compared to a reference algorithm and two independent criteria (the performance score parameter and normalized geometric distance) were used for overall evaluation and comparison. The experiments showed that the novel DDSL method reaches significantly lower RMSE while the number of transmitted data count is less than or equal to the fixed duty cycle algorithm. The overall criteria performance score is 40% higher than the reference algorithm base on static confirmation settings.

Automatic track guidance of industrial trucks with time-variant vehicle parameters using AI-based controllers

This paper presents an extension of a self-learning control concept for automatic track guidance of industrial trucks in intralogistic systems. The presented approach is based on Reinforcement Learning (RL), a method of Artificial Intelligence (AI) and is able to adapt itself to different industrial truck variants and the associated specific vehicle parameters. Moreover, time-variant parameters during operation, such as the vehicle's velocity are taken into account. In order to consider the existing a priori knowledge of the controlled system and to avoid starting the whole training process of the controller for each truck variant from scratch, the training process is divided into two steps. In the first step, the controller is trained on a model using parameters of a nominal vehicle variant. Based on this, the control parameters are only fine-tuned in the second step. In this way the controller is adapted to the actual truck variant and the corresponding parameter values. In order to take into account the time-variant vehicle parameters during operation, the Artificial Neural Networks (ANN) of the RL controller and the observation vector are suitably extended. In this way, the varying speed can be considered in both training steps and the control parameters can be optimized accordingly. Thus, in case of the investigated scenarios a stable control loop behavior can be guaranteed for the entire speed range of industrial trucks. In order to demonstrate this, the new approach is compared with a RL control concept, not considering time-variant parameters.