Dynamic Programming Structure Research Articles

Optimal Spin Polarization Control for the Spin-Exchange Relaxation-Free System Using Adaptive Dynamic Programming.

This work is the first to solve the 3-D spin polarization control (3DSPC) problem of atomic ensembles, which controls the spin polarization to achieve arbitrary states with the cooperation of multiphysics fields. First, a novel adaptive dynamic programming (ADP) structure is proposed based on the developed multicritic multiaction neural network (MCMANN) structure with nonquadratic performance functions, as a way to solve the multiplayer nonzero-sum game (MP-NZSG) problem in 3DSPC under the constraints of asymmetric saturation inputs. Then, we utilize the MCMANNs to implement the multicritic multiaction ADP (MCMA-ADP) algorithm, whose convergence is proven by the compression mapping principle. Finally, the MCMA-ADP is deployed in the spin-exchange relaxation-free (SERF) system to provide a set of control laws in 3DSPC that fully exploits the multiphysics fields to achieve arbitrary spin polarization states. Numerical simulations support the theoretical results.

A Novel Approximate Dynamic Programming Structure for Optimal Control of Discrete-Time Time-Varying Nonlinear Systems

A Novel Approximate Dynamic Programming Structure for Optimal Control of Discrete-Time Time-Varying Nonlinear Systems

Vibration Suppression of a Flexible Beam Structure Coupled with Liquid Sloshing via ADP Control Based on FBG Strain Measurement

In this study, an adaptive dynamic programming (ADP) control strategy based on the strain measurement of a fiber Bragg grating (FGB) sensor array is proposed for the vibration suppression of a complicated flexible-sloshing coupled system, which usually exists in aerospace engineering, such as launch vehicles with a large amount of liquid propellant as well as a flexible beam structure. To simplify the flexible-sloshing coupled dynamics model, the equivalent spring-mass-damper (SMD) model of liquid sloshing is employed, and a finite-element method (FEM) dynamic model for the beam structure coupled with the liquid sloshing is mathematically established. Then, a strain-based vibration dynamic model is derived by employing a transformation matrix based on the relationship between displacement and strain of the beam structure. To facilitate the design of a strain-based control, a tracking differentiator is designed to provide the strains’ derivative signals as partial states’ estimations. Feeding the system with the strain measurements and their derivatives’ estimations, an ADP controller with an action-dependent heuristic dynamic programming structure is proposed to suppress the vibration of the flexible-sloshing coupled system, and the corresponding Lyapunov stability of the closed-loop system is theoretically guaranteed. Numerical results show the proposed method can effectively suppress coupled vibration depending on limited strain measurements irrespective of external disturbances.

Two Kadane Algorithms for the Maximum Sum Subarray Problem

The maximum sum subarray problem is to find a contiguous subarray with the largest sum. The history of algorithms to address this problem is recounted, culminating in what is known as Kadane’s algorithm. However, that algorithm is not the algorithm Kadane intended. Nonetheless, the algorithm known as Kadane’s has found many uses, some of which are recounted here. The algorithm Kadane intended is reported here, and compared to the algorithm attributed to Kadane. They are both linear in time, employ just a few words of memory, and use a dynamic programming structure. The results proved here show that these two algorithms differ only in the case of an input consisting of only negative numbers. In that case, the algorithm Kadane intended is more informative than the algorithm attributed to him.

Output‐feedback stochastic model predictive control of chance‐constrained nonlinear systems

AbstractThis study covers the output‐feedback model predictive control (MPC) of nonlinear systems subjected to stochastic disturbances and state chance constraints. The stochastic optimal control problem is solved in a stochastic dynamic programming fashion, and the output‐feedback control is performed with the extended Kalman filter. The information state is summarized as a dynamic Gaussian belief model. Thus, the stochastic Bellman equation is transformed into a deterministic equation using this model. The resulting constrained Bellman equation is solved using the proposed constrained, approximate dynamic programming algorithm. The algorithm is proved to have a Q‐superlinear local convergence rate. Numerical experiments show that the proposed algorithm can attain good control performance and reasonable chance‐constraint satisfaction and is computationally efficient owing to its dynamic programming structure.

Optimal design of a driver assistance controller based on surrounding vehicle’s social behavior game model

Driver assistance control in the cut-in scenarios is challenging, since the controller needs to ensure driving safety and avoid unnecessary intervention, while considering the interaction with surrounding vehicles. This paper proposes an optimal driver assistance controller considering the social behaviors of the surrounding vehicles to assist the drivers in the cut-in scenarios. To model the social behavior of the surrounding vehicle, we first formulate the interaction between the semi-autonomous vehicle and the surrounding vehicle as a misinformation game, which is achieved by assuming the surrounding vehicle is interacting with a hypothetical vehicle under the framework of non-cooperative game. Then, adaptive dynamic programming theory is utilized to find the Nash equilibrium, which is represented by deep neural networks and solved iteratively. Based on the established social behavior model and the nonlinear driver-vehicle dynamic model, an affine input nonlinear system model is obtained for the design of driver assistance controller, and the optimal assistance control strategy is also derived under the structure of adaptive dynamic programming. Several numerical simulations and driver-in-the-loop simulator experiments are conducted for validation. Results show that the proposed strategy can assist the driver in the cut-in scenario while addressing different social interactions with the surrounding vehicles. Importantly, by taking into account the surrounding vehicle’s social behavior through our established social behavior model, our proposed strategy significantly outperforms the no-interaction strategy both in terms of driving safety and intervention degree, validating its effectiveness and superiority.

Dynamic Programming Structure Learning Algorithm of Bayesian Network Integrating MWST and Improved MMPC

Dynamic programming is difficult to apply to large-scale Bayesian network structure learning. In view of this, this article proposes a BN structure learning algorithm based on dynamic programming, which integrates improved MMPC (maximum-minimum parents and children) and MWST (maximum weight spanning tree). First, we use the maximum weight spanning tree to obtain the maximum number of parent nodes of the network node. Second, the MMPC algorithm is improved by the symmetric relationship to reduce false-positive nodes and obtain the set of candidate parent-child nodes. Finally, with the maximum number of parent nodes and the set of candidate parent nodes as constraints, we prune the parent graph of dynamic programming to reduce the number of scoring calculations and the complexity of the algorithm. Experiments have proved that when an appropriate significance level α is selected, the MMPCDP algorithm can greatly reduce the number of scoring calculations and running time while ensuring its accuracy.

Model-Free Adaptive Optimal Control for Unknown Nonlinear Multiplayer Nonzero-Sum Game.

In this article, an online adaptive optimal control algorithm based on adaptive dynamic programming is developed to solve the multiplayer nonzero-sum game (MP-NZSG) for discrete-time unknown nonlinear systems. First, a model-free coupled globalized dual-heuristic dynamic programming (GDHP) structure is designed to solve the MP-NZSG problem, in which there is no model network or identifier. Second, in order to relax the requirement of systems dynamics, an online adaptive learning algorithm is developed to solve the Hamilton-Jacobi equation using the system states of two adjacent time steps. Third, a series of critic networks and action networks are used to approximate value functions and optimal policies for all players. All the neural network (NN) weights are updated online based on real-time system states. Fourth, the uniformly ultimate boundedness analysis of the NN approximation errors is proved based on the Lyapunov approach. Finally, simulation results are given to demonstrate the effectiveness of the developed scheme.

A heuristics for HTTP traffic identification in measuring user dissimilarity

The prevalence of HTTP web traffic on the Internet has long transcended the layer 7 classification, to layers such as layer 5 of the OSI model stack. This coupled with the integration-diversity of other layers and application layer protocols has made identification of user-initiated HTTP web traffic complex, thus increasing user anonymity on the Internet. This study reveals that, with the current complex nature of Internet and HTTP traffic, browser complexity, dynamic web programming structure, the surge in network delay, and unstable user behavior in network interaction, user-initiated requests can be accurately determined. The study utilizes HTTP request method of GET filtering, to develop a heuristic algorithm to identify user-initiated requests. The algorithm was experimentally tested on a group of users, to ascertain the certainty of identifying user-initiated requests. The result demonstrates that user-initiated HTTP requests can be reliably identified with a recall rate at 0.94 and F-measure at 0.969. Additionally, this study extends the paradigm of user identification based on the intrinsic characteristics of users, exhibited in network traffic. The application of these research findings finds relevance in user identification for insider investigation, e-commerce, and e-learning system as well as in network planning and management. Further, the findings from the study are relevant in web usage mining, where user-initiated action comprises the fundamental unit of measurement.

Neuro-Optimal Tracking Control for Continuous Stirred Tank Reactor With Input Constraints

This paper proposes a novel data-based optimal control algorithm for continuous stirred tank reactor (CSTR) system based on adaptive dynamic programming (ADP). To overcome the challenge of establishing an accurate mathematical model for the CSTR system, neural networks are employed to reconstruct the dynamics of the CSTR system using the production data of the system. A new nonquadratic form performance index function is provided, where the control input is constrained in order not to exceed the bound of the actuator. Then, the operational optimal control problem of CSTR is formulated. Furthermore, an iterative ADP (IADP) algorithm is developed to obtain the optimal tracking controller for the CSTR system with control constraints. In particular, the convergence analysis of the IADP algorithm is developed. The proposed IADP algorithm is implemented via the dual heuristic dynamic programming structure. Finally, the proposed approach is applied to the real CSTR system to verify the effectiveness and performance.

An efficient hybrid CS‐DP application for generation expansion planning problem

This article proposes a new hybrid approach for generation expansion planning (GEP) based on cuckoo search (CS) and dynamic programming (DP). Generally, the GEP problem is known as having very large search space and highly nonlinear complex combinatorial optimization. The global optimal solution is to solve with the full enumeration DP. However, the computation time grows exponentially when the planning horizon and candidate options increase, which is called ‘curse of dimensionality’. To solve this problem, a hybrid CS‐DP approach is proposed with three techniques for solution quality and convergence characteristic enhancement. First, the CS is integrated into the DP structure for the optimum solution of the very large search space problem. Second, the feasible search concept can eliminate infeasible solutions from the searching process. Finally, the memory exploration search is applied to prevent repeated searches. To measure its effectiveness, the proposed method is applied to 15 existing power plants, with 5 candidate options and 2 test cases: 14 year and 20 year study periods. The experimental results are compared with classical and metaheuristic optimizations. The test results indicate that the proposed hybrid CS‐DP achieves higher solution quality and superior convergence characteristics than other methods. © 2019 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.

Data-Based Optimal Tracking Control for Natural Gas Desulfurization System

Desulfurization control of natural gas has long been a challenging industrial issue owing to its inherent difficulty in establishing accurate mathematical model for the nonlinear and strong coupling process. In this paper, a data-based adaptive dynamic programming (ADP) algorithm is presented to solve optimal control for natural gas desulfurization. First, neural network (NN) is used to reconstruct the dynamics of the desulfurization system via the input and output production data. Then, an improved unscented Kalman filter (IUKF) aided ADP method is presented to solve optimal control problem for desulfurization system, where IUKF algorithm is developed as a new weight-updating strategy for the action network and the critic network. The IUKF aided algorithm can improve the convergence speed as well as the anti-interference ability of the ADP controller. Furthermore, the proposed IUKF-ADP algorithm is implemented using the heuristic dynamic programming (HDP) structure. Finally, the effectiveness of the proposed IUKF-ADP algorithm is demonstrated through experiments of the natural gas desulfurization system.

Ascent trajectory tracking method using time-varying quadratic adaptive dynamic programming

Ascent trajectory tracking in the longitudinal plane is a class of nonaffine noncascade nonlinear system control problems with single input and multiple output, which is difficult to control by nonlinear method directly. Adaptive dynamic programming algorithm has the advantages of precise control and adaptability for general nonlinear control problem, and the time-varying quadratic adaptive dynamic programming algorithm proposed in this paper promotes the convergence and calculation speed of the traditional adaptive dynamic programming algorithm. To implement the algorithm effectively, the independent variables of ascent model are substituted in the launching coordinate, and then the model is treated as a discrete nominal trajectory tracking problem. Besides, the heuristic dynamic programming structure is used to train the processed model, and thus only the time-varying weight in the designed evaluation network needs to be updated. Simulation shows that the proposed algorithm can update the control variable online after predicting the cost function offline with the dynamic equations, which is faster than the general adaptive dynamic programming algorithm. In addition, the proposed algorithm can effectively and accurately track the nominal trajectory under the uncertainty of parameters compared with the linear quadratic regulators algorithm.

An adaptive longitudinal control method for autonomous follow driving based on neural dynamic programming and internal model structure

Autonomous vehicles are considered to have great potentials in improving transportation safety and efficiency. Autonomous follow driving is one of the highly probable application forms of autonomou...

Online H∞ control for completely unknown nonlinear systems via an identifier–critic-based ADP structure

ABSTRACTIn this paper, we propose an identifier–critic-based approximate dynamic programming (ADP) structure to online solve H∞ control problem of nonlinear continuous-time systems without knowing precise system dynamics, where the actor neural network (NN) that has been widely used in the standard ADP learning structure is avoided. We first use an identifier NN to approximate the completely unknown nonlinear system dynamics and disturbances. Then, another critic NN is proposed to approximate the solution of the induced optimal equation. The H∞ control pair is obtained by using the proposed identifier–critic ADP structure. A recently developed adaptation algorithm is used to online directly estimate the unknown NN weights simultaneously, where the convergence to the optimal solution can be rigorously guaranteed, and the stability of the closed-loop system is analysed. Thus, this new ADP scheme can improve the computational efficiency of H∞ control implementation. Finally, simulation results confirm the effectiveness of the proposed methods.

Novel iterative neural dynamic programming for data-based approximate optimal control design

As a powerful method of solving the nonlinear optimal control problem, the iterative adaptive dynamic programming (IADP) is usually established on the known controlled system model and is particular for affine nonlinear systems. Since most nonlinear systems are complicated to establish accurate mathematical models, this paper provides a novel data-based approximate optimal control algorithm, named iterative neural dynamic programming (INDP) for affine and non-affine nonlinear systems by using system data rather than accurate system models. The INDP strategy is built within the framework of IADP, where the convergence guarantee of the iteration is provided. The INDP algorithm is implemented based on the model-based heuristic dynamic programming (HDP) structure, where model, action and critic neural networks are employed to approximate the system dynamics, the control law and the iterative cost function, respectively. During the back-propagation of action and critic networks, the approach of directly minimizing the iterative cost function is developed to eliminate the requirement of establishing system models. The neural network implementation of the INDP algorithm is presented in detail and the associated stability is also analyzed. Simulation studies are conducted on affine and non-affine nonlinear systems, and further on the manipulator system, where all results have demonstrated the effectiveness of the proposed data-based approximate optimal control method.

Efficient dynamic programming algorithm with prior knowledge for protein β-strand alignment

One of the main tasks towards the prediction of protein β-sheet structure is to predict the native alignment of β-strands. The alignment of two β-strands defines similar regions that may reflect functional, structural, or evolutionary relationships between them. Therefore, any improvement in β-strands alignment not only reduces the computational search space but also improves β-sheet structure prediction accuracy. To define the alignment scores, previous studies utilized predicted residue-residue contacts (contact maps). However, there are two serious problems using them. First, the precision of contact map prediction techniques, especially for long-range contacts (i.e., β-residues), is still not satisfactory. Second, the residue-residue contact predictors usually utilize general properties of amino acids and disregard the structural features of β-residues. In this paper, we consider β-structure information, which is estimated from protein β-sheet data sets, as alignment scores. However, the predicted contact maps are used as a prior knowledge about residues. They are used for strengthening or weakening the alignment scores in our algorithm. Thus, we can utilize both β-residues and β-structure information in alignment of β-strands. The structure of dynamic programming of the alignment algorithm is changed in order to work with our prior knowledge. Moreover, the Four Russians method is applied to the proposed alignment algorithm in order to reduce the time complexity of the problem. For evaluating the proposed method, we applied it to the state-of-the-art β-sheet structure prediction methods. The experimental results on the BetaSheet916 data set showed significant improvements in the execution time, the accuracy of β-strands’ alignment and consequently β-sheet structure prediction accuracy. The results are available at http://conceptsgate.com/BetaSheet.

基于近似动态规划的三轴卫星姿态最优控制

应用近似动态规划方法解决三轴卫星姿态最优轨迹规划问题，首先使用三轴卫星的动力学和运动学模型，对于给定的始末姿态，选取姿态机动能量消耗最少作为待优化的性能指标。文中根据自适应动态规划结构，分别利用评价网络来近似性能指标函数和执行网络来逼近控制变量，龙格库塔法求解状态变量，并给出了适合该类问题的一种效用函数的具体表达式。仿真结果表明应用近似动态规划解得的三轴卫星最优轨迹，能够较好地满足各种约束条件，而且计算精度高、速度快，具有很好的实时性。 The optimal attitude trajectory planning of three-axis satellite using approximate dynamic pro-gramming (ADP) method is discussed. Firstly, the dynamic and kinematic equations of the three-axis satellite are used, and for given initial and final attitudes, the performance to be opti-mized is selected as minimizing the rest-to-rest maneuver energy. On grounds of adaptive dynamic programming structure, critic network and action network are used to approximate performance index function and control variables respectively, and Runge-Kutta method to solve the state variables. Besides, a concrete expression of the utility function is provided which is suitable for this kind of problem. The simulation results show that the proposed algorithm satisfies the constraints well and can be used on-line with its small computational amount and low computational complexity.

Event-Based Robust Control for Uncertain Nonlinear Systems Using Adaptive Dynamic Programming.

In this paper, the robust control problem for a class of continuous-time nonlinear system with unmatched uncertainties is investigated using an event-based control method. First, the robust control problem is transformed into a corresponding optimal control problem with an augmented control and an appropriate cost function. Under the event-based mechanism, we prove that the solution of the optimal control problem can asymptotically stabilize the uncertain system with an adaptive triggering condition. That is, the designed event-based controller is robust to the original uncertain system. Note that the event-based controller is updated only when the triggering condition is satisfied, which can save the communication resources between the plant and the controller. Then, a single network adaptive dynamic programming structure with experience replay technique is constructed to approach the optimal control policies. The stability of the closed-loop system with the event-based control policy and the augmented control policy is analyzed using the Lyapunov approach. Furthermore, we prove that the minimal intersample time is bounded by a nonzero positive constant, which excludes Zeno behavior during the learning process. Finally, two simulation examples are provided to demonstrate the effectiveness of the proposed control scheme.

Maximum wind power tracking based on cloud RBF neural network

Based on the mathematical model of Permanent magnet synchronous generator (PMSG), maximum wind power tracking control strategy without wind speed detection is analyzed and a controller based on cloud RBF neural network and approximate dynamic programming is designed to track the maximum wind power point. Optimal power-speed curve and vector control principles are used to control the electromagnetic torque by approximate dynamic programming controller to adjust the voltage of stator, so the speed of wind turbine can be operated at the optimal speed corresponding to the best power point. Cloud RBF neural network is adopted as the function approximation structure of approximate dynamic programming, and it has the advantage of the fuzziness and randomness of cloud model. Simulation results show that the method can solve the optimal control problem of complex nonlinear system such as wind generation and track the maximum wind power point accurately.