Theory Of Dynamic Programming Research Articles

Adaptive neural optimal tracking control for uncertain unmanned surface vehicle

This article proposes a novel adaptive neural optimal control scheme for the uncertain unmanned surface vehicle (USV) system. The proposed adaptive neural optimal control scheme is composed of a neural feedforward controller and a neural optimal feedback controller. The neural feedforward controller is designed by using neural networks (NNs) and the backstepping control design technique to make the unknown nonlinear dynamics of USV system to be stabilized. The neural optimal feedback controller is designed based on adaptive dynamic programming (ADP) theory. It is proved that the formulated adaptive neural optimal control scheme can achieve all signals in USV system are uniformly ultimately bounded (UUB) and the cost function is minimized. The simulation and comparison results demonstrate the effectiveness of the proposed optimal control scheme.

Data mining based on computer game search algorithm in Japanese E-learning and multi context translation

With the continuous expansion of the translation market, users’ demand for translation tools such as computer software is also increasing, resulting in various machine translation technologies and methods. Japanese translation, as one of the important components of Chinese publicity work, has strong professionalism and complexity. In practice, how to effectively improve the translation quality of Japanese translation models using computer technology has always been a research issue of great concern. Based on this, we propose a game search algorithm and use it as a prototype to improve the algorithm. Then, a data mining model was established that can effectively improve search performance. This model solves the above problems by introducing genetic algorithms and dynamic programming theory. At the same time, in order to further improve the efficiency of existing data mining algorithms, we need to explore more efficient methods and strategies. After experimental verification, the four data mining models proposed in this article perform the best in numerical fitting. Overall, in Japanese multi context translation systems, the use of data mining techniques based on computer game search algorithms can provide more accurate and matching related translations. Compared to traditional Japanese translation software, it has higher quality and efficiency.

Improvement of the method of time rationing for assembling car groups on one track

Purpose. To improve the method for standardizing the duration of the shunting operation of assembling cars on one track. This can be achieved as a result of solving the following research problems: development of a method for searching the optimal order of assembling cars on one track; distribution parameters estimation of the random value of the duration of shunting operation of assembling cars on one track based on calculation experiments. Methodology. During the research, the methods of theory of railway operation, dynamic programming and mathematical statistics were used. Findings. Research on the assembling process of car group to one track established the distribution parameters of the random variable of time spent for shunting. In the course of the research, the problem of choosing the optimal order of shunting operations during car assembling was formalized and solved as a problem of dynamic programming. The time spent for shunting work was chosen as the optimality criterion. The paper considers the possibility of approximating the data of calculation experiments by analytical dependencies. It was found out that the use of linear polynomials with interaction allows obtaining dependencies describing time standards with a relative accuracy of ±5 %. Originality. The method is improved for developing the time standards for shunting work, which, unlike the existing one, is based on the performance of a series of calculation experiments, each of which solves the optimization problem of finding such an order of assembling cars that ensures minimum time consumption for shunting. Practical value. The methods developed in the work and the dependencies obtained allow improving the quality of decisions made when developing technology and designing railway stations and sidings of industrial enterprises.

A Fully Distributed ADP Algorithm for Real-Time Economic Dispatch of Microgrid

This paper proposes a fully distributed approximate dynamic programming (FD-ADP) algorithm framework for real-time economic dispatch of a microgrid (MG). Firstly, a modified distributed consensus-based algorithm (DCBA) framework with power restriction factors is proposed to distributedly obtain the real-time ED decision of MG containing distributed energy resources with linear cost functions, which improves the convergence stability compared with the traditional DCBA framework. Then, combined with the approximate dynamic programming (ADP) theory, a FD-ADP algorithm is proposed to deal with data uncertainties. To this end, the piece-wise linear (PWL) function is applied to approximate the value function, and a distributed slope update approach is proposed to embed empirical knowledge into the value function while ensuring the distributed property in off-line training. With the well-trained PWL function assisting in decision, the distributed real-time ED strategy of MG with approximate global optimality is obtained by the proposed FD-ADP algorithm during the entire schedule horizon (e.g., 24 hours). Finally, case studies on the modified IEEE 6-bus MG system and IEEE 123-bus MG system demonstrate the superiority of solution convergence, validity of distributed training, and optimality of distributed schedule of the proposed FD-ADP algorithm.

Study on the building zoning of regional energy planning based on a dynamic programming theory

Building zoning is crucial for regional energy planning. Appropriate building zoning can effectively optimize the peak load, daily load rate and load standard deviation of buildings. However, a clear quantitative standard for building zoning is currently lacking, and zoning methods based on minimum distance or load fluctuation are often employed, which often fail to achieve optimal zoning and energy-saving effects. To clarify building zoning in regional energy planning, this study proposes a dynamic programming approach and employs the analytic hierarchy process (AHP) to improve the zoning method. This is achieved by using the daily load rate, load standard deviation and building distance of each building group as the criterion layer, and assigns each peripheral building group to the appropriate building group. Taking a university campus as an example, this method reduces the peak heating load of 6136.61 kW and the peak cooling load of 15507.88 kW. The standard deviation of heating and cooling load of the building group has been reduced by 1272.53 kW and 2352.54 kW, respectively. Furthermore, the daily load rate of the building has been stabilized, and the staggered peak has been adjusted, resulting in a more uniform distribution frequency of the building load.

Event-Triggered Adaptive Fuzzy Optimal Control for a Class of Strict-Feedback Nonlinear Systems With External Disturbances

In this paper, the problem of the event-triggered optimal control for a class of strict-feedback nonlinear systems with external disturbances is investigated. Firstly, by introducing proper low-pass filters to the steps of backstepping design, the problem of “jump of the error surfaces” is avoided. What's more, it facilitates the design of a fuzzy hybrid-mode-triggered feedforward controller, which aims to transform the original controlled nonlinear strict-feedback system into an equivalent nonlinear system in affine form. Secondly, by utilizing the adaptive dynamic programming theory, an event-triggered feedback controller is developed for the equivalent affine nonlinear system. Differing from the existing methods, the real control input consists of two event-triggered terms, which can be updated synchronously under the proposed composite error-based triggering condition. Moreover, it is proven that all the closed-loop signals are bounded. At last, by taking a second-order nonlinear system and missile integrated guidance and control system as examples, simulation results demonstrate the effectiveness of the proposed approach.

Optimization of track distribution of industrial railway stations between car designations

Purpose. To improve the method for distributing the tracks of industrial railway stations for the accumulation of car groups between separate assignments. The optimization problem is to find such a distribution of classification work between the industrial marshalling yard and freight stations, as well as such a distribution of marshalling yard tracks between individual destinations, which ensures the minimum time expenditures on shunting work. Methodology. The studies were carried out using the methods of the theory of railway operation, simulation modeling and dynamic programming. Findings. The optimization problem of distribution of sorting work between the marshalling yard and freight stations of an industrial enterprise, as well as searching for such a distribution of marshalling yard tracks between individual destinations, which ensures the minimum time expenditures for shunting work, has been solved. Originality. The paper proposes a method for formalizing and solving the problem of distributing the tracks of industrial railway stations for the accumulation of car groups between individual destinations as a dynamic programming problem. Unlike the existing methods, where the number of tracks is considered as a constraint, in the proposed method, the number of tracks is an objective function argument of minimizing the duration of shunting operations, which improves the quality of the solutions obtained. Practical value. The method proposed in the paper, due to the rational distribution of the existing track arrangement of railway stations, makes it possible to reduce the time expenditures on the making- and breaking-up the trains and shunting transfers and, due to this, increase the carrying capacity of the stations, as well as reduce the cost of production of enterprises.

An adaptive dynamic programming in cooperative target tracking for energy acquisition in wireless sensor networks

In order to further study the application of energy acquisition model, based on the relevant theory of adaptive dynamic programming, the wireless sensor is tracked by modifying the energy acquisition model, and the relevant data and parameters are quantitatively analyzed according to the test results. Finally, the parameters of different samples are tracked and identified. The results show that the evaluation analysis curve with four different parameters can be divided into two stages according to the different variation trends: fluctuation stage and stable stage. It indicates that the increase of time can effectively improve the stability corresponding to the convergence of neural network. The trajectory under the three different parameter states shows a general trend of rapid decline at first and then tends to be stable. There is a certain disturbance in the rapid decline of the curve, but when the time is between 1 and 2, it corresponds to the drop of the curve, and the curve gradually flattens after rapid decline. Parameters in the adaptive dynamic programming model are shown as follows: parameter W decreases slowly to the lowest point and then rises slowly with the increase of iteration steps; The proportion of parameter W1 in the model shows a trend of slow increase. Relevant research can provide theoretical basis for the application of energy acquisition model based on adaptive dynamic programming and analysis in other fields.

Dynamic Operational Strategies Incorporating Consumer Reference Price Effects and Enterprise Behavior: A Differential Game Approach

The continuous evolution of consumer behavior in the modern era of consumption has prompted enterprises to explore the underlying behavioral factors of consumers and cater to their particular needs. Moreover, developing a rational operational behavior model and responding effectively to the dynamic market environment have become critical concerns for businesses. This study examines the impact of consumer reference price effects and enterprise short-sighted behavior on strategic selection and performance, employing differential game theory to construct a game model between manufacturers and retailers. Utilizing Behrman's continuous dynamic programming theory, analytical solutions for various models are derived, followed by comparative analyses and numerical examples. The research reveals that: (1) manufacturers' behavior patterns are found to be dominant, favoring far-sighted behavior, which not only enhances profits but also enables consumers to access higher quality and cost-effective products; retailers should opt for collaboration with far-sighted manufacturers and exhibit a preference for short-sighted behavior. (2) In terms of overall system profit, the FM model emerges as the optimal combination. (3) When the reference price effect has a small impact on market demand, enterprises can make use of the reference price effect to actively promote marketing and gain profit; as the influence increases, intensifying the degree of influence effectively augments profits.

Dynamic Task Offloading for Mobile Edge Computing with Hybrid Energy Supply

Mobile edge computing (MEC), as a new distributed computing model, satisfies the low energy consumption and low latency requirements of computation-intensive services. The task offloading of MEC has become an important research hotspot, as it solves the problems of insufficient computing capability and battery capacity of Internet of things (IoT) devices. This study investigates task offloading scheduling in a dynamic MEC system. By integrating energy harvesting technology into IoT devices, we propose a hybrid energy supply model. We jointly optimize local computing, offloading duration, and edge computing decisions to minimize system cost. On the basis of stochastic optimization theory, we design an online dynamic task offloading algorithm for MEC with a hybrid energy supply called DTOME. DTOME can make task offloading decisions by weighing system cost and queue stability. We quote dynamic programming theory to obtain the optimal task offloading strategy. Simulation results verify the effectiveness of DTOME, and show that DTOME entails lower system cost than two baseline task offloading strategies.

Emergency repair strategy of distribution network under wind disaster based on multidimensional importance evaluation

In order to enhance the power supply reliability of distribution network under wind disasters and provide assistance for distribution network operators to formulate post disaster repair strategies, this paper proposes an optimization method for key nodes and lines evaluation and distribution network repair strategy under wind disaster. Firstly, based on the complex network theory, we considered the status of nodes and lines in the topology and working state. The multi-dimensional evaluation indexes are proposed including improved node degree, node cohesion degree, node intermediate number, weighted line intermediate number, line transfer power flow entropy and line failure rate, etc. Then, the entropy weight method is used to objectively weight the six indicators, and TOPSIS algorithm is used to rank the comprehensive importance of nodes and lines in the distribution network. On this basis, the dynamic programming theory is used to optimize the emergency repair strategy of the fault distribution network after the wind disaster. And the problem of restoring as many key nodes and lines as possible in a short time is proposed. Finally, the method is applied to the IEEE 39 bus system network to verify the feasibility.

Operation Strategies of Green Supply Chain Members with Short-Sighted and Far-Sighted Behavior: A Differential Game Theory Approach

The concept of “green supply chain” has gained increasing attention in the pursuit of sustainable development by enterprises globally, leading to the optimization of supply chain management. However, the behavior patterns of some members in green supply chains exhibit differences. Hence, analyzing the choices of different behavior patterns among green supply chain members is of practical significance in improving supply chain governance and achieving sustainable development. In this study, a differential game model is constructed using differential game theory to analyze the behavior patterns among members of a green supply chain. Analytical solutions under different models are obtained with the aid of Bellman continuous dynamic programming theory, and comparative analysis and numerical examples are carried out to explore the operation strategies of green supply chain members and the choice of short-sighted and far-sighted behavior patterns. Our findings indicate that suppliers playing a leading role in the green supply chain should exhibit far-sighted behavior as decision makers, while retailers should also display far-sighted behavior. This study adopts a dynamic perspective and is innovative and cutting-edge, providing targeted recommendations for enterprises and serving as a reference for the future development of green supply chain management.

An Improved Dynamic Programming-Based Decentralized Algorithm for AC-OPF of Radially Connected Multi-Level Networks

In this paper, an improved dynamic programming (IDP)-based decentralized algorithm is proposed for the AC-optimal power flow (AC-OPF) of radially connected transmission-distribution-microgrid networks (RCTDMNs). Most of the existing decentralized algorithms need to iterate among networks, which may lead to additional computational and information exchange burden. Different from these algorithms, we decompose the original AC-OPF problem into small-scale subproblems based on dynamic programming (DP) theory, and then solve them in a decentralized manner without iteration among networks. In order to improve the computational performance of DP, its value functions are directly obtained by solving Galerkin projection equations (GPEs) instead of traversing the whole state space. Moreover, a parallel technique is proposed to further accelerate the proposed IDP algorithm when dealing with the AC-OPF problem of RCTDMNs. Case studies on various test systems demonstrate the effectiveness of the proposed method.

Green Agricultural Products Supply Chain Subsidy Scheme with Green Traceability and Data-Driven Marketing of the Platform

Government subsidies have played an important role in the development of green agriculture. In addition, the Internet platform is becoming a new channel to realize green traceability and promote the sale of agricultural products. In this context, we consider a two-level green agricultural products supply chain (GAPSC) consisting of one supplier and one Internet platform. The supplier makes green R&D investments to produce green agricultural products along with conventional agricultural products, and the platform implements green traceability and data-driven marketing. The differential game models are established under four government subsidy scenarios: no subsidy (NS), consumer subsidy (CS), supplier subsidy (SS), and supplier subsidy with green traceability cost-sharing (TSS). Then, the optimal feedback strategies under each subsidy scenario are derived using Bellman's continuous dynamic programming theory. The comparative static analyses of key parameters are given, and the comparisons among different subsidy scenarios are conducted. Numerical examples are employed to obtain more management insights. The results show that the CS strategy is effective only if the competition intensity between two types of products is below a certain threshold. Compared to the NS scenario, the SS strategy can always improve the supplier's green R&D level, the greenness level, market demand for green agricultural products, and the system's utility. The TSS strategy can build on the SS strategy to further enhance the green traceability level of the platform and the greenness level and demand for green agricultural products due to the advantage of the cost-sharing mechanism. Accordingly, a win-win situation for both parties can be realized under the TSS strategy. However, the positive effect of the cost-sharing mechanism will be weakened as the supplier subsidy increases. Moreover, compared to three other scenarios, the increase in the environmental concern of the platform has a more significant negative impact on the TSS strategy.

Optimal design of a driver assistance controller based on surrounding vehicle’s social behavior game model

Driver assistance control in the cut-in scenarios is challenging, since the controller needs to ensure driving safety and avoid unnecessary intervention, while considering the interaction with surrounding vehicles. This paper proposes an optimal driver assistance controller considering the social behaviors of the surrounding vehicles to assist the drivers in the cut-in scenarios. To model the social behavior of the surrounding vehicle, we first formulate the interaction between the semi-autonomous vehicle and the surrounding vehicle as a misinformation game, which is achieved by assuming the surrounding vehicle is interacting with a hypothetical vehicle under the framework of non-cooperative game. Then, adaptive dynamic programming theory is utilized to find the Nash equilibrium, which is represented by deep neural networks and solved iteratively. Based on the established social behavior model and the nonlinear driver-vehicle dynamic model, an affine input nonlinear system model is obtained for the design of driver assistance controller, and the optimal assistance control strategy is also derived under the structure of adaptive dynamic programming. Several numerical simulations and driver-in-the-loop simulator experiments are conducted for validation. Results show that the proposed strategy can assist the driver in the cut-in scenario while addressing different social interactions with the surrounding vehicles. Importantly, by taking into account the surrounding vehicle’s social behavior through our established social behavior model, our proposed strategy significantly outperforms the no-interaction strategy both in terms of driving safety and intervention degree, validating its effectiveness and superiority.

Research on Supply Chain Quality Decision Model Considering Reference Effect and Competition under Different Decision-Making Modes

The reference effect of consumers’ product quality and service quality is an important behavioral factor that affects consumers’ purchase decisions. In this paper, considering the inherent dynamics of the operation mode, the two were combined into a product-service supply chain composed of a manufacturer and two competitive retailers, and the service reference effect was further divided into horizontal and vertical service reference effects. Differential game models between the manufacturer and the retailers were constructed. Using the continuous dynamic programming theory, the manufacturer’s quality level strategy, retailer’s service level strategy and supply chain system performance under the modes of no cost sharing decision, cost sharing decision and centralized decision were analyzed and compared. Main findings: The optimal strategy of product quality under centralized decision is greater than the corresponding value under cost sharing contract decision. Compared with decentralized decision making of no cost sharing, cost sharing contract cannot promote the manufacturer to improve product quality. Different from previous studies, the service quality under centralized decision making is not higher than other decisions. When the horizontal reference effect of service quality meets a certain condition, the supply chain profit under centralized decision making is lower than the corresponding value under decentralized decision making.

Two-dimensional dynamic time warping algorithm for matrices similarity

Dynamic Time Warping (DTW algorithm) provides an effective method to obtain the similarity between unequal-sized signals. However, it cannot directly deal with high-dimensional samples such as matrices. Expanding a matrix to one dimensional vector as the input data of DTW will decrease the measure accuracy because of the losing of position information in the matrix. Aiming at this problem, a two-dimensional dynamic time warping algorithm (2D-DTW) is proposed in this paper to directly measure the similarity between matrices. In 2D-DTW algorithm, a three dimensional distance-cuboid is constructed, and its mapped distance matrix is defined by cutting and compressing the distance-cuboid. By introducing the dynamic programming theory to search the shortest warping path in the mapped matrix, the corresponding shortest distance can be obtained as the expected similarity measure. The experimental results suggest that the performance of 2D-DTW distance is superior to the traditional Euclidean distance and can improve the similarity accuracy between matrices by introducing the warping alignment mechanisms. 2D-DTW algorithm extends the application ranges of traditional DTW and is especially suitable for high-dimensional data.

Recycling Model Selection for Electronic Products Considering Platform Power and Blockchain Empowerment

This paper considers a dynamic platform-based, closed-loop supply chain consisting of a manufacturer and an online platform. As an online distributor of the manufacturer, the platform expands the market scale by exerting the platform power. At the same time, to solve the problem of inconsistency between the actual recycling amount and the theoretical recycling amount in the recycling process of waste electronic products, the whole-process supervision of waste products is carried out with the help of blockchain technology, which is difficult to tamper with and is traceable. With the help of differential game theory, four differential game models of manufacturer recycling and platform recycling with and without blockchain are established. The state feedback strategies are derived from Bellman’s continuous dynamic programming theory. Through analytical results and comparative analysis, the adoption conditions of blockchain and the impact of blockchain on the selection of recycling models are obtained. The results illustrated that the introduction of blockchain technology effectively improves the real recycling rate of waste electronics, building trust in consumers, which benefits corporations in certain conditions. However, it amplifies the double marginal effect of the CLSC. Nevertheless, the implementation of blockchain is still beneficial to consumers, as the adverse impact of the double marginal effect is compensated by the improvement in consumer surplus. In addition, the study shows that the implementation of the blockchain incentivizes members, who benefit on the same recycling model when the fixed cost of the blockchain and the share ratio of the residual value of waste electronics are between certain thresholds. That is, both the manufacturer and the platform are better off in a manufacturer recycling model enabled by blockchain. Moreover, in this model, the social welfare and the recycling rate of waste electronics are increased, which enable the CLSC to achieve benefits related to economy, environment, and society.

An approach to solving optimal control problems of nonlinear systems by introducing detail-reward mechanism in deep reinforcement learning

In recent years, dynamic programming and reinforcement learning theory have been widely used to solve the nonlinear control system (NCS). Among them, many achievements have been made in the construction of network model and system stability analysis, but there is little research on establishing control strategy based on the detailed requirements of control process. Spurred by this trend, this paper proposes a detail-reward mechanism (DRM) by constructing the reward function composed of the individual detail evaluation functions in order to replace the utility function in the Hamilton-Jacobi-Bellman (HJB) equation. And this method is introduced into a wider range of deep reinforcement learning algorithms to solve optimization problems in NCS. After the mathematical description of the relevant characteristics of NCS, the stability of iterative control law is proved by Lyapunov function. With the inverted pendulum system as the experiment object, the dynamic environment is designed and the reward function is established by using the DRM. Finally, three deep reinforcement learning algorithm models are designed in the dynamic environment, which are based on Deep Q-Networks, policy gradient and actor-critic. The effects of different reward functions on the experimental accuracy are compared. The experimental results show that in NCS, using the DRM to replace the utility function in the HJB equation is more in line with the detailed requirements of the designer for the whole control process. By observing the characteristics of the system, designing the reward function and selecting the appropriate deep reinforcement learning algorithm model, the optimization problem of NCS can be solved.

Optimal harvesting under marine reserves and uncertain environment

Persistence in the literature is the perception of an inherent tradeoff between ecological conservation and economic harvesting goals. Overexploitation may lead to resource destruction, including extinction. Conservation measures should be decided and implemented. A standard ecological response is to impose no-take areas or marine reserves. In this paper, our objective is to study a harvesting management problem under the constraints of a no-take area, which we formulate mathematically as a bi-dimensional singular stochastic control problem. Using dynamic programming theory, we characterise our value function as the unique solution to a HJB equation. We also describe the optimal harvesting strategy by identifying the harvesting and non-harvesting regions. We show that setting up reserve area not only leads to a better ecological conservation but may also increase the economic benefit for fishing industry in the long run. We further enrich our studies with some numerical analysis, enabling us to get some insightful understanding on the size of the reserve area where no-take policy should be implemented.