Dynamic Programming Scheme Research Articles

Adaptive control of plug-in hybrid electric vehicles based on energy management strategy and dynamic programming algorithm

This study mainly analyses the fuel consumption of plug-in hybrid vehicles during operation. A new control method for automobiles based on energy management strategy and dynamic programming algorithm is proposed. The new method plans and analyses the minimum electricity consumption, and then uses dynamic programming algorithms to analyse this parameter. The research results indicated that the vehicle state was constantly changing with the variation of SOC value during driving. The energy mobilization of the vehicle was more obvious after adding dynamic programming strategy. The efficiency of the vehicle was relatively high in driving state 1, with a minimum value of 70%, which was about 20% higher than in driving state 4. The average fuel consumption in driving state 2 was 1.8L higher than in other driving states. The overall efficiency of automobiles after incorporating dynamic programming was improved, with a shorter time to reach the lowest efficiency point compared with not incorporating dynamic programming algorithms. The highest efficiency value was 7.86% higher than that of not incorporating dynamic programming models. The new control method can reduce energy consumption and improve the energy management and control effect. The study provides a better research direction for energy management and control of hybrid electric vehicles in the future.

Learning‐based tracking control of AUV: Mixed policy improvement and game‐based disturbance rejection

AbstractA mixed adaptive dynamic programming (ADP) scheme based on zero‐sum game theory is developed to address optimal control problems of autonomous underwater vehicle (AUV) systems subject to disturbances and safe constraints. By combining prior dynamic knowledge and actual sampled data, the proposed approach effectively mitigates the defect caused by the inaccurate dynamic model and significantly improves the training speed of the ADP algorithm. Initially, the dataset is enriched with sufficient reference data collected based on a nominal model without considering modelling bias. Also, the control object interacts with the real environment and continuously gathers adequate sampled data in the dataset. To comprehensively leverage the advantages of model‐based and model‐free methods during training, an adaptive tuning factor is introduced based on the dataset that possesses model‐referenced information and conforms to the distribution of the real‐world environment, which balances the influence of model‐based control law and data‐driven policy gradient on the direction of policy improvement. As a result, the proposed approach accelerates the learning speed compared to data‐driven methods, concurrently also enhancing the tracking performance in comparison to model‐based control methods. Moreover, the optimal control problem under disturbances is formulated as a zero‐sum game, and the actor‐critic‐disturbance framework is introduced to approximate the optimal control input, cost function, and disturbance policy, respectively. Furthermore, the convergence property of the proposed algorithm based on the value iteration method is analysed. Finally, an example of AUV path following based on the improved line‐of‐sight guidance is presented to demonstrate the effectiveness of the proposed method.

Discrete‐Time Optimal Control of State‐Constrained Nonlinear Systems Using Approximate Dynamic Programming

ABSTRACTThis article investigates the optimal control problem (OCP) for a class of discrete‐time nonlinear systems with state constraints. First, to overcome the challenge caused by the constraints, the original constrained OCP is transformed into an unconstrained OCP by utilizing the system transformation technique. Second, a new cost function is designed to alleviate the effect of system transformation on the optimality of the original system. Further, a novel off‐policy deterministic approximate dynamic programming (ADP) scheme is developed to obtain a near‐optimal solution for the transformed OCP. Compared to existing off‐policy deterministic ADP schemes, the developed scheme relaxes the requirement on the learning data and saves computing resources from the perspective of training neural networks. Third, considering approximation errors, we analyze the convergence and stability of the developed ADP scheme. Finally, the developed ADP with the designed cost function is tested in two numerical cases, and simulation results confirm its effectiveness.

MDSA: A Dynamic and Greedy Approach to Solve the Minimum Dominating Set Problem

The graph theory is one of the fundamental structures in computer science used to model various scientific and engineering problems. Many problems within the graph theory are categorized as NP-hard and NP-complete. One such problem is the minimum dominating set (MDS) problem, which seeks to identify the minimum possible subsets in a graph such that every other node in the subset is directly connected to a node in this subset. Due to its inherent complexity, developing an efficient polynomial-time method to address the MDS problem remains a significant challenge in graph theory. This paper introduces a novel algorithm that utilizes a centrality measure known as the Malatya Centrality to effectively address the MDS problem. The proposed algorithm, called the Malatya Dominating Set Algorithm (MDSA), leverages centrality values to identify dominating sets within a graph. It extends the Malatya centrality by incorporating a second-level centrality measure, which enhances the identification of dominating nodes. Through a systematic and algorithmic approach, these centrality values are employed to pinpoint the elements of the dominating set. The MDSA uniquely integrates greedy and dynamic programming strategies. At each step, the algorithm selects the most optimal (or near-optimal) node based on the centrality values (greedy approach) while updating the neighboring nodes’ criteria to influence subsequent decisions (dynamic programming). The proposed algorithm demonstrates efficient performance, particularly in large-scale graphs, with time and space requirements scaling proportionally with the size of the graph and its average degree. Experimental results indicate that our algorithm outperforms existing methods, especially in terms of time complexity when applied to large datasets, showcasing its effectiveness in addressing the MDS problem.

A novel EMS design framework for SPHTs based on instantaneous layer, driving event layer, and driving cycle layer

To effectively harness the energy-saving potential of series–parallel hybrid transmissions (SPHTs) with multiple gears and modes and to enhance the driving cycle adaptability of the rule-based energy management strategy (RB EMS), this study establishes a novel EMS design framework for SPHTs at three levels: the instantaneous, the driving event, and the driving cycle layers. Firstly, an equivalent fuel factor based on the principles of energy conservation and calorific value measurements is constructed, the energy consumption of different working modes and torque combinations under different instantaneous power units is determined, and the decision conditions of working modes are determined. Secondly, the energy consumption of engine and motor torque combination under constant speed, acceleration and deceleration driving events is compared, and the torque distribution feature of multi-power sources is determined. Thirdly, a driving cycle distribution factor is introduced and calculated using vehicle navigation map information, which can adjust the gear switching condition of parallel driving mode online. Based on above, an RB EMS adjusted with driving cycles (ADC-RB EMS) is proposed. The effectiveness of the proposed strategy is then verified through real-world vehicle tests. The fuel consumption performance of this strategy falls between the RB EMS and the dynamic programming (DP) strategy. According to the fuel consumption results, the RB EMS consumes an additional 0.35 L/100 km of fuel, while the DP strategy saves only 3.66% more energy compared to this strategy. The application potential of driving cycle distribution factor in instantaneous optimal control is tested. Compared with ADC-RB and CD-CS, the fuel saving rate is 1.71% and 5.67% respectively, which proves the energy saving performance of A-ECMS. This study provides a development direction for improving the energy saving effect of the RB EMS.

Optimal Rule-Interposing Reinforcement Learning-Based Energy Management of Series—Parallel-Connected Hybrid Electric Vehicles

P2–P3 series–parallel hybrid electric vehicles exhibit complex configurations with multiple power sources and operational modes, presenting a difficulty in developing efficient energy management strategies. This paper takes a P2–P3 series–parallel hybrid power system-KunTye 2DHT system as the research object and proposes a deep reinforcement learning framework based on pre-optimized energy management to improve the energy consumption performance of the hybrid electric vehicles. Firstly, a control-oriented model is established based on its system configuration and characteristics. Then, the optimal distribution of the motor energy under different operating modes is pre-optimized, which aims to reduce the energy management task’s dimensionality by equating two motors as an equivalent motor. Subsequently, based on real-time traffic information under connected conditions, deep reinforcement learning is utilized to optimize the optimal operating modes of the hybrid system and the optimal distribution between the engine and equivalent motors. Combining the pre-optimized results, the optimal energy distribution between the engine and the two motors in the system is achieved. Finally, performance comparisons are made between the predictive control and the traditional Dynamic Programming and Adaptive Equivalent Consumption Minimization Strategy, revealing the proposed optimization algorithm’s promising potential in reducing fuel consumption.

Human-machine cooperation: Optimizing layout design and drug retrieval sequencing in automated drug dispensing systems

Automated drug dispensing systems (ADDSs) are increasingly in demand in today’s pharmacies, primarily driven by the growing aging population. Recognizing the practical challenges faced by pharmacies implementing ADDSs, this study aims to optimize the layout design and retrieval sequencing issues to enhance the order picking efficiency of ADDSs within a human–machine cooperation environment. In ADDSs, the robotic arm retrieves the drug bin and after that pharmacists need to sort the drug at the Input/output (I/O) points for order picking, which is a human–machine cooperation way. We consider the pharmacists’ stochastic sorting time in the order picking problem. For the various layout designs in ADDSs, we develop dual command retrieval sequencing models with one I/O point layout design and two I/O points layout designs. The successive order arrival and the dual command mode operations make the process intertwined. The proposed models are tested using the pharmacy data. We incorporate the stochastic sorting time of pharmacists to analyze its impact on ADDS performance. Through experimental comparisons of average picking times for prescription orders under various parameters, we demonstrate that the system layout design with two I/O points significantly enhances the efficiency of prescription order fulfillment within a human–machine cooperation environment. Furthermore, our proposed retrieval sequencing method outperforms dynamic programming, greedy, and random strategies in terms of improving prescription order-picking efficiency. Specifically, a comparison is made with the current situation in hospitals as well. By addressing the layout design and sequencing challenges, our research contributes to the field of intelligent warehousing, particularly in smart pharmacies.

An energy management strategy based on dynamic programming for fuel cell hybrid trucks in ports

The deterioration of environment and the proposal of carbon neutrality are causing the transformation of port freight transportation. Fuel cell hybrid trucks (FCHTs) may be a promising solution for port logistic with the characteristics of low speed, heavy duty and frequent start-stops. How to achieve efficient energy management for FCHTs in ports is a key issue, which is still in the initial stage. The objective of this study is to investigate the effects of dynamic programming (DP) strategy on energy management of FCHTs. With the intention of calculating power demands, the FCHT structure and the models of each component are established. In the DP method, the state of charge (SOC) of the battery, the power supplied by the battery, and hydrogen fuel consumption are regarded as the state variable, the decision variable, and the objective function, respectively. The simulation results under two different typical working conditions in ports show that the proposed DP outperforms existing genetic algorithm (GA) and rule-based strategy for reducing hydrogen fuel consumption by over 10%. Moreover, a set of instances among several FCHTs with different parameters and conditions suggest different influences of key feature parameters on hydrogen fuel consumption and SOC. Nearly 50% cost reduction of the FCHT in the "quay crane-yard " (QCY) cycle compared with diesel trucks at the current price indicates wide application prospect of FCHTs. At last, the results under different conditions reconfirm that DP is an excellent strategy to tackle energy management of FCHTs in ports with frequent start-stops and heavy-load scenarios.

Energy management strategy of series–parallel hybrid transmission integrating map information and personalized driving characteristics

The integration of multi-source intelligent and connected information during a driving trip, along with its online application to globally optimized energy management strategies, has emerged as a crucial technical approach for enhancing the energy-saving effectiveness of hybrid transmissions. However, the action mode of such information and the optimization calculation efficiency of existing dynamic programming (DP) methods limit the online application of the aforementioned strategies with global optimization capabilities. To address these problems, the present study proposes a hierarchical energy management strategy that follows the reference trajectory of the battery state of charge (SoC) and comprehensively considers the multi-source information on the driving trip. First, a global speed prediction model based on personalized driving characteristics is proposed to obtain an accurate driving cycle input for the space-domain DP method. Second, the aforementioned tasks as well as the working-mode decision of the hybrid transmission and the multi-power-source torque distribution calculation tasks are deployed in the dual-core controller. Finally, the hierarchical energy management strategy is verified via vehicle testing. Compared with the DP strategy, the proposed strategy has an energy-saving potential of 4.17% that is yet to be realized. Furthermore, compared with the charge-depleting and charge-sustaining (CD–CS) strategy, the proposed strategy reduces fuel consumption by 0.38 L per 100 km, and its energy-saving effect is significant. This study is the first to apply the DP method to the vehicle controller, thereby facilitating the online application of energy management strategies with global optimization capabilities.

A comparative analysis of power control strategies for a plug-in extended-range FCHEV with battery and PEMFC durability considerations

ABSTRACT Fuel cell hybrid electric vehicles employ fuel cells along with batteries or ultracapacitors as power sources, and energy management systems can control their power split. This study investigates energy management strategies for lightweight plug-in extended-range fuel cell hybrid electric vehicles to optimise fuel economy, extend component lifespans, and enhance vehicle efficiency. Comparisons between power follower and dynamic programming-based strategies are conducted, focusing on incorporating PEMFC health constraints to prolong stack durability via on-off binary mode operation. Using the New European Driving Cycle, performance evaluation is conducted on a parallel architecture featuring a 16-kW motor, regenerative braking system, eight 420-Wh batteries in series, and six 1-kW fuel cell stacks. Dynamic programming strategies demonstrated potential improvements, with savings in hydrogen consumption of 62–72 L per driving cycle and SOC levels of 3.9%. Moreover, optimised battery lifespan enabled the system to achieve approximately 18,087 -18,745 charge/discharge cycles. Implementing PEMFC health constraints facilitated reductions in average accumulated active time to 205 seconds per driving cycle and startup/shutdown cycle frequency to 36.5 cycles per hour, promoting enhanced system durability and operational efficiency.

An energy management strategy with considering ultracapacitor ideal state of charge for fuel cell/battery/ultracapacitor vehicle

The hybridization of energy storage systems consisting of fuel cells, batteries, and ultracapacitors has tremendous potential in fuel cell hybrid electric vehicles. However, the utilization of ultracapacitors in existing energy management strategies is insufficient, preventing complete exploitation of the energy storage system's benefits. To strengthen the utilization of ultracapacitors, this paper proposes an energy management strategy for fuel cell vehicles with three energy sources, considering the ideal state of charge (SOC) of ultracapacitors. The proposed method disassembles the motor power demand, with the ultracapacitor primarily supplying the acceleration portion. It calculates the ideal SOC of the ultracapacitor, and controlling the actual SOC of the ultracapacitor to match the ideal SOC. Furthermore, a fuzzy logic controller is designed to modify the fuel cell's output power, reducing power fluctuation. Additionally, according to the variation of the SOC of the battery, the fuel cell output power is modified to enhance the overall performance of the composite energy storage system. Simulation and hardware-in-loop results demonstrate that, compared to the adaptive filtering strategy with fuel cell power constraints and dynamic programming strategy, the strategy proposed in this paper reduces operational costs by 14.56 % and 4.99 %, respectively.

Demand side management strategy for smart building using multi-objective hybrid optimization technique

This study proposes a home energy management system that uses the load-shifting technique for demand-side management as a way to improve the energy consumption patterns of a smart house. This system's goal is to optimize the energy of household appliances in order to effectively regulate load demand, with the end result being a reduction in the peak-to-average ratio (PAR) and a consequent minimization of electricity costs. This is accomplished while also keeping user comfort as a priority. Load scheduling based on both a next-day and real-time basis is what is used to meet the load demand requested by energy customers. In addition to providing a fitness criterion, utilizing a multi-objective hybrid optimization technique makes it easier to achieve an equitable distribution of workload between on-peak and off-peak hours. Moreover, the idea of developing coordination among home appliances in order to achieve real-time rescheduling is now being studied as a concept. Because of the inherent parallels between the two problems, the real-time rescheduling issue is framed as a knapsack problem and is solved using a dynamic programming strategy. The performance of the suggested methodology is evaluated in this study in relation to real-time pricing (RTP), time-of-use pricing (ToU), and crucial peak pricing (CPP). The simulation findings, which were assessed using a confidence interval that was set at 95 %, provide proof of the relevance that has been shown to be associated with the proposed optimization method. During scheduling RTP signal showcases a minimum PAR of 2.22 and a cost reduction of 24.06 % for HAG compared to the unscheduled case. Under the TOU tariff, HAG manages to reduce PAR by 46.14 % and cost by 20.44 %. Similarly, in the case of CPP, HAG outperforms by reducing PAR by up to 29.5 % and cost by up to 31.47 %.

Critic learning control via zero-sum differential game for affine formation maneuver of multi-agent systems with cyber-attacks

The affine formation maneuver (AFM) control problem using the zero-sum differential game is studied for a second-order multi-agent system (MAS) against cyber-injection attacks on the actuators. First and foremost, the distributed control signals and the cyber-injected attack signals are viewed as two competing teams. Then, the leader-follower tracking error system is established where the followers aim to track the AFMs of the leaders. The leader-follower distributed optimal control policies and the cyber-attack attenuation policies are derived from the Nash equilibrium solution of the Hamilton-Jacobi-Isaac (HJI) equation. The solution of the HJI equation is estimated with the aid of the critic neural network. The weights of the critic system are learned online based on the adaptive dynamic programming (ADP) scheme. The uniformly ultimately bounded stability of the closed-loop system under the formulated zero-sum differential game control strategy has been proven by the Lyapunov stability theorem. A simulation example illustrates that the presented control method can attenuate cyber-injection attacks on the actuators and maintain the AFMs of the agents.

Dynamic programming strategy in optimal controller design for a wind turbine system

Dynamic programming strategy in optimal controller design for a wind turbine system

双电机驱动电动拖拉机能量管理策略的研究

At present, electric tractors experience significant battery energy loss during operation, resulting in a short continuous running time. Therefore, in order to reduce the power consumption of the tractor drive system, minimize battery energy loss, and extend the operating time under various conditions, this paper presents a method for driving an electric tractor based on dual-motor coupling. Based on the characteristics of the transmission structure, an online torque distribution strategy for dual-motor coupling-driven electric tractors using a fuzzy control approach is proposed. First, an enhanced genetic algorithm is utilized to optimize the fuzzy rule table. Simultaneously, it is compared with the offline optimization strategy of dynamic programming. Subsequently, a method that integrates test data models and theoretical models is employed to establish an efficiency model of key components of the electric tractor drive system and a longitudinal dynamics model of the entire machine. The performance of the entire vehicle was simulated and analyzed under plowing conditions. Finally, on the experimental bench, conduct steady-state load tests and dynamic performance tests on the dual-motor coupled drive system. The results show that the State of Charge (SOC) change trends of the fuzzy control strategy based on the improved genetic algorithm and the dynamic programming strategy are similar. The SOC change values are close, which enhances the adaptability of the electric tractor in various operating conditions. Compared with the fuzzy control strategy, the improved strategy reduced average power consumption by 8.8%, demonstrating that the fuzzy control energy management strategy based on the enhanced genetic algorithm is both economical and superior. The bench experiment demonstrated that the dual-motor drive system can adapt to load changes to achieve power distribution between the two motors, meeting the required workload while reducing power consumption.

A Computational Evaluation of Minimum Feature Size in Projection Two-Photon Lithography for Rapid Sub-100 nm Additive Manufacturing.

Two-photon lithography (TPL) is a laser-based additive manufacturing technique that enables the printing of arbitrarily complex cm-scale polymeric 3D structures with sub-micron features. Although various approaches have been investigated to enable the printing of fine features in TPL, it is still challenging to achieve rapid sub-100 nm 3D printing. A key limitation is that the physical phenomena that govern the theoretical and practical limits of the minimum feature size are not well known. Here, we investigate these limits in the projection TPL (P-PTL) process, which is a high-throughput variant of TPL, wherein entire 2D layers are printed at once. We quantify the effects of the projected feature size, optical power, exposure time, and photoinitiator concentration on the printed feature size through finite element modeling of photopolymerization. Simulations are performed rapidly over a vast parameter set exceeding 10,000 combinations through a dynamic programming scheme, which is implemented on high-performance computing resources. We demonstrate that there is no physics-based limit to the minimum feature sizes achievable with a precise and well-calibrated P-TPL system, despite the discrete nature of illumination. However, the practically achievable minimum feature size is limited by the increased sensitivity of the degree of polymer conversion to the processing parameters in the sub-100 nm regime. The insights generated here can serve as a roadmap towards fast, precise, and predictable sub-100 nm 3D printing.

Backstepping method tracking control for underactuated AUV with unknown dynamics based on action-critic networks based ADP

This work investigates trajectory-tacking control problem for underactuated autonomous underwater vehicles (AUV) with unknown dynamics. Due to the unknown dynamics, an action-critic networks based adaptive dynamic programming (ADP) scheme combined with backstepping approach is designed, which can achieve high-level system stability and tracking control accuracy. Firstly, the backstepping approach is introduced into the kinematic model of underactuated AUV and produces a virtual velocity control which is taken as the desired velocity input of the dynamic model of underactuated AUV. Secondly, the error tracking system is constructed according to the dynamic model of underactuated AUV. Thirdly, the critic neural network and the action neural network are employed to transform the trajectory-tracking control problem into optimal control problem based on policy iteration algorithm. At last simulation results are given to verify the effectiveness of the proposed control scheme.

Costate-Supplement ADP for Model-Free Optimal Control of Discrete-Time Nonlinear Systems.

In this article, an adaptive dynamic programming (ADP) scheme utilizing a costate function is proposed for optimal control of unknown discrete-time nonlinear systems. The state-action data are obtained by interacting with the environment under the iterative scheme without any model information. In contrast with the traditional ADP scheme, the collected data in the proposed algorithm are generated with different policies, which improves data utilization in the learning process. In order to approximate the cost function more accurately and to achieve a better policy improvement direction in the case of insufficient data, a separate costate network is introduced to approximate the costate function under the actor-critic framework, and the costate is utilized as supplement information to estimate the cost function more precisely. Furthermore, convergence properties of the proposed algorithm are analyzed to demonstrate that the costate function plays a positive role in the convergence process of the cost function based on the alternate iteration mode of the costate function and cost function under a mild assumption. The uniformly ultimately bounded (UUB) property of all the variables is proven by using the Lyapunov approach. Finally, two numerical examples are presented to demonstrate the effectiveness and computation efficiency of the proposed method.

Parallel and Flexible Dynamic Programming via the Mini-Batch Bellman Operator

The Bellman operator constitutes the foundation of dynamic programming (DP). An alternative is presented by the Gauss-Seidel operator, whose evaluation, differently from that of the Bellman operator where the states are all processed at once, updates one state at a time, while incorporating into the computation the interim results. The provably better convergence rate of DP methods based on the Gauss-Seidel operator comes at the price of an inherent sequentiality, which prevents the exploitation of modern multi-core systems. In this work we propose a new operator for dynamic programming, namely, the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mini-batch Bellman operator</i> , which aims at realizing the trade-off between the better convergence rate of the methods based on the Gauss-Seidel operator and the parallelization capability offered by the Bellman operator. After the introduction of the new operator, a theoretical analysis for validating its fundamental properties is conducted. Such properties allow one to successfully deploy the new operator in the main dynamic programming schemes, such as value iteration and modified policy iteration. We compare the convergence of the DP algorithm based on the new operator with its earlier counterparts, shedding light on the algorithmic advantages of the new formulation and the impact of the batch-size parameter on the convergence. Finally, an extensive numerical evaluation of the newly introduced operator is conducted. In accordance with the theoretical derivations, the numerical results show the competitive performance of the proposed operator and its superior flexibility, which allows one to adapt the efficiency of its iterations to different structures of MDPs and hardware setups.

A comparative study of energy-oriented driving strategy for connected electric vehicles on freeways with varying slopes

—This paper proposes two real-time energy-oriented driving strategies to minimize the energy consumption for electric vehicles on highways with varying slopes. First, a novel strategy, called normalized-energy consumption minimization strategy (NCMS), adopts a designed kinetic energy conversion factor to convert the vehicle kinetic energy change into the equivalent battery energy consumption. By minimizing the total normalized energy consumption, the energy-orientated vehicle control sequence is calculated. In addition, a logic car-following algorithm is developed to enhance NCMS for avoiding collisions with the potential preceding vehicle on the journey. Second, an improved model predictive control (MPC) is developed with a hierarchical framework, which achieves a balance between optimization and computational efficiency. In the upper level, a global, coarse-grained, iterative dynamic programming is employed to penalize the MPC terminal state, while the lower level performs online rolling optimization of the vehicle within a moderate time step. Thirdly, the performance of the proposed driving strategies is verified through a traffic simulation to evaluate the energy efficiency improvement and processor computation time compared to dynamic programming and constant speed strategy. Finally, a vehicle-in-the-loop test is carried out to validate the feasibility of the proposed two novel driving strategies.