Dynamic Programming Optimization Research Articles

Optimization for fuel consumption and TCO of a heavy-duty truck with electricity-propelled trailer

As renewable energy grows in the heavy-duty truck sector, exploration of diverse solutions with varied energy types and powertrain configurations becomes a necessity. While most studies focus on powertrain configurations in tractors, this study takes a different approach: propose and evaluate new plug-in hybrid configurations with multiple power sources for heavy-duty trucks. The tractor retains its engine powertrain, while the trailer is equipped with electric axle(s), forming a fuel-electricity hybrid propulsion system. The specifications of the electric propulsion system are optimized, using a multi-objective particle swarm optimization algorithm and dynamic programming control algorithm to evaluate the energy consumption and total cost of ownership (TCO). The results highlight the potential of the configuration with three electric axles and single reduction gear in regard to vehicle energy consumption. Conversely, the configuration with one electric axle and a two-speed transmission exhibits the lowest TCO. Under the China World Transient Vehicle Cycle, the fuel consumption of the diesel truck is reduced by up to 44.59 % with the use of the optimized hybrid configuration. The truck's TCO can be then reduced by up to 832 thousand CNY in one-million-kilometer operation. For an actual road operation, these figures change to 34.79 % and 284 thousand CNY, respectively.

Stochastic measurement-based multi-cloud consolidation for efficient resource distribution

In distributed cloud systems, due to high computational complexity, existing methods cannot infinitely reduce the measurement interval to obtain the mean demand. However, considering both the unpredictability of demand and the diversity of pricing adds significant complexity to the problem. In response, we introduce an efficient algorithm called Stochastic Demand-oriented Resource consolidation based on Meta-heuristics (SDRM). The algorithm introduces the stochastic demand model and formulates a cross-cloud demand consolidation and resource scheduling problem, and tries to solve it efficiently by combining differential evolution algorithm with dynamic programming optimization. The results are provided for experiments utilizing both simulated and real-world data. It demonstrates that compared with traditional algorithms, increasing gain by up to 53%, SDRM just incurs an affordable time expense (i.e., 1.10-2.51 times that of existing algorithms). Therefore, SDRM stands as a compelling solution for resource distribution in heterogeneous cloud environments.

Efficient Pump Scheduling for Large‐Scale Multiproduct Pipelines Using Deep Reinforcement Learning

ABSTRACTOptimizing pump scheduling in multiproduct pipelines can significantly reduce energy consumption and carbon emissions. For pump scheduling in multiproduct pipelines, to describe hydraulic losses more accurately, the model needs to adopt shorter discrete time intervals, which will lead to longer decision‐making time. It combined with the large solution space of large‐scale pipelines, will lead to low solution efficiency with dynamic programming methods and poor solution quality with heuristic optimization algorithms. Given that, this article develops a multiproduct refined oil transmission simulation system and employs the enhanced Proximal Policy Optimization (PPO) algorithm with action space shaping trick to optimize pump scheduling for large‐scale multiproduct pipelines. The method of converting discrete action space to multi‐discrete action space through action shaping can address PPO's low convergence efficiency issue resulting from the large discrete action space challenge common in large‐scale multiproduct pipelines. The experimental results indicate that the proposed method, that is, PPO algorithm with multidiscrete action space exhibits significant advantages in terms of efficiency and robustness in large‐scale pipelines compared to mainstream methods for pump scheduling such as dynamic programming (DP), genetic algorithms (GA), and ant colony optimization (ACO). Furthermore, we demonstrate the effectiveness of action space shaping in large‐scale pipelines from the perspectives of exploration and exploitation.

The principle of optimality in dynamic programming: A pedagogical note

The principle of optimality is a fundamental aspect of dynamic programming, which states that the optimal solution to a dynamic optimization problem can be found by combining the optimal solutions to its sub-problems. While this principle is generally applicable, it is often only taught for problems with finite or countable state spaces in order to sidestep measure-theoretic complexities. Therefore, it cannot be applied to classic models such as inventory management and dynamic pricing models that have continuous state spaces, and students may not be aware of the possible challenges involved in studying dynamic programming models with general state spaces. To address this, we provide conditions and a self-contained simple proof that establish when the principle of optimality for discounted dynamic programming is valid. These conditions shed light on the difficulties that may arise in the general state space case. We provide examples from the literature that include the relatively involved case of universally measurable dynamic programming and the simple case of finite dynamic programming where our main result can be applied to show that the principle of optimality holds.

An improved data-driven predictive optimal control approach for designing hybrid electric vehicle energy management strategies

This paper proposes an improved “prediction + optimal control” method for energy management in hybrid electric vehicles equipped with planetary gears. A differentiable predictor and a differentiable optimal controller are developed using supervised learning and reinforcement learning approaches, respectively. Three training steps are performed for the initial predictor, the optimal controller, and the final predictor. This method improves the traditional energy management predictive optimal control approach by incorporating an additional step of retraining the differentiable predictor. This adjustment ensures that the predictor does not blindly improve its performance based on evaluation criterion irrelevant to energy management, which was commonly used in previous studies. Instead, it focuses on enhancing the overall performance of energy management under the “prediction + optimal control” framework. The approach introduced in this paper is compared with the globally optimal dynamic programming results and traditional predictive optimal control methods on the Next Generation Simulation (NGSIM) data. Our method outperforms traditional approaches in energy management on both the training dataset and the test dataset. This further illustrates that the conventional practice of presumptuously optimizing predictors in “prediction + optimal control” methods can be improved using the proposed method.

Threshold-awareness in adaptive cancer therapy.

Although adaptive cancer therapy shows promise in integrating evolutionary dynamics into treatment scheduling, the stochastic nature of cancer evolution has seldom been taken into account. Various sources of random perturbations can impact the evolution of heterogeneous tumors, making performance metrics of any treatment policy random as well. In this paper, we propose an efficient method for selecting optimal adaptive treatment policies under randomly evolving tumor dynamics. The goal is to improve the cumulative "cost" of treatment, a combination of the total amount of drugs used and the total treatment time. As this cost also becomes random in any stochastic setting, we maximize the probability of reaching the treatment goals (tumor stabilization or eradication) without exceeding a pre-specified cost threshold (or a "budget"). We use a novel Stochastic Optimal Control formulation and Dynamic Programming to find such "threshold-aware" optimal treatment policies. Our approach enables an efficient algorithm to compute these policies for a range of threshold values simultaneously. Compared to treatment plans shown to be optimal in a deterministic setting, the new "threshold-aware" policies significantly improve the chances of the therapy succeeding under the budget, which is correlated with a lower general drug usage. We illustrate this method using two specific examples, but our approach is far more general and provides a new tool for optimizing adaptive therapies based on a broad range of stochastic cancer models.

Research on energy management strategy for fuel cell hybrid electric vehicles based on improved dynamic programming and air supply optimization

It is crucial to accurately calculate the cost function of the energy management strategy (EMS) of the hybrid powertrain to improve the hydrogen economy of the system. This paper proposes an EMS for fuel cell hybrid electric vehicles (FCHEV) based on improved dynamic programming (DP) and air supply optimization to improve economy and reliability. Taking the maximum net power output of the FC system as the target, the optimal oxygen excess ratio (OER) and cathode pressure of the FC system under different current densities are solved by using PSO. A velocity prediction method based on Bi-LSTM is developed to predict short-term velocity changes in real time. The DP algorithm is introduced and the EMS of the DP algorithm based on short-term velocity prediction is developed for real-time hybrid powertrain optimization and management. Based on the results of energy allocation and optimal gas supply conditions of FCs, the cost function of EMS is modified to reallocate the power of the FC system and battery. The results demonstrate that the proposed method achieves the lowest hydrogen consumption compared to the other two algorithms. Remarkably, it reduces the fuel cost by up to 8.85 % compared to the commonly used online DP algorithm.

A review of dynamic optimization in aquaculture production economics

AbstractDynamic optimization (DO) has been applied to aquaculture industry to determine optimal management strategies of aquaculture production systems. Optimal control of feeding, optimal stocking successions, and harvesting time among other factors has been studied. This work shows a review of the application of optimal control theory (OCT) and dynamic programming in economics and management of aquaculture. The Pontriagyn's maximum principle and the Bellman's optimality are the most commonly used mechanisms to solve control problems that optimize the producer's benefit or cost. Recently, model‐based predictive control has also been applied. Such tools have been used in aspects involving the design, planning, and monitoring of variables relevant to the optimal management of the culture system. It was found that in aquaculture there is low scientific productivity in the application of DO in bioeconomic models. In this review, an example of OCT applied to the control of feeding of farmed tilapia using a bioeconomic model is shown. DO represents a useful tool for optimal decision‐making and this review discusses the implications of the use of DO in aquaculture and recommendations for its use in the future.

Research on Energy Hierarchical Management and Optimal Control of Compound Power Electric Vehicle

In response to the challenges posed by the low energy utilization of single-power pure electric vehicles and the limited lifespan of power batteries, this study focuses on the development of a compound power system. This study constructs a composite power system, analyzes the coupling characteristics of multiple systems, and investigates the energy management and optimal control mechanisms. Firstly, a power transmission scheme is designed for a hybrid electric vehicle. Then, a multi-state model is established to assess the electric vehicle’s performance under complex working conditions and explore how these conditions impact system coupling. Next, load power is redistributed using the Haar wavelet theory. The super capacitor is employed to stabilize chaotic and transient components in the required power, with low-frequency components serving as input variables for the controller. Further, power distribution is determined through the application of fuzzy logic theory. Input parameters include the system’s power requirements, power battery status, and super capacitor state of charge. The result is the output of a composite power supply distribution factor. To fully exploit the composite power supply’s potential and optimize the overall system performance, a global optimization control strategy using the dynamic programming algorithm is explored. The optimization objective is to minimize power loss within the composite power system, and the optimal control is calculated through interpolation using the interp function. Finally, a comparative simulation experiment is conducted under UDDS cycle conditions. The results show that the composite power system improved the battery discharge efficiency and reduced the number of discharge cycles and discharge current of the power battery. Under the cyclic working condition of 1369 s, the state of charge of the power battery in the hybrid power system decreases from 0.9 to 0.69, representing a 12.5% increase compared to the single power system. The peak current of the power battery in the hybrid power system decreases by approximately 20 A compared with that in the single power system. Based on dynamic programming optimization, the state of charge of the power battery decreases from 0.9 to 0.724. Compared with that of the single power system, the power consumption of the proposed system increases by 25%, that of the hybrid power fuzzy control system increases by 14.2%, and that of the vehicle decreases by 14.7% after dynamic programming optimization. The multimode energy shunt relationship is solved through efficient and reasonable energy management and optimization strategies. The performance and advantages of the composite energy storage system are fully utilized. This approach provides a new idea for the energy storage scheme of new energy vehicles.

A real-time energy management strategy for fuel cell vehicle based on Pontryagin’s minimum principle

This paper proposes a Pontryagin's minimum principle (PMP) energy management strategy (EMS) based on driving cycle recognition for fuel cell vehicle powertrains, aiming to minimize hydrogen consumption and fuel cell degradation. Firstly, the neural network-based driving cycle recognizer is optimized using the tuna swarm optimization (TSO) algorithm and trained under four typical driving cycles. Then, the optimal co-state variables for the four driving cycles are obtained by iteration. Finally, the co-state variables are dynamically updated based on real-time driving cycle recognition results. Comparative analysis demonstrates that the PMP-DCR effectively improves fuel cell lifetime and vehicle economy under short-distance driving cycles. Based on the combined driving cycle, the proposed PMP-DCR EMS exhibits similar economy performance to optimal dynamic programming (DP) EMS, reducing equivalent hydrogen consumption by 13.8% and 9.2%, and decreasing fuel cell degradation rates by 93% and 8.7% in comparison to the conventional power-following and PMP EMS, respectively.

Development of a deep Q-learning energy management system for a hybrid electric vehicle

In recent years, Machine Learning (ML) techniques have gained increasing popularity in several fields thanks to their ability to find hidden and complex relationships between data. Their capabilities for solving complex optimization tasks have made them extremely attractive also for the design of the Energy Management System (EMS) of electrified vehicles. Among the plethora of existing techniques, Reinforcement Learning (RL) algorithms have unprecedented potential since they can self-learn by directly interacting with the external environment through a trial-and-error procedure. In this paper, a Deep Q-Learning (DQL) agent, which exploits Deep Neural Networks (DNNs) to map the state-action pair to its value, was trained to reduce the CO2 emissions of a state-of-the-art diesel Plug-in Hybrid Electric Vehicle (PHEV) available on the European market. The proposed methodology was tested on a virtual test rig of the investigated vehicle while operating on a charge-sustaining logic. A sensitivity analysis was performed on the reward to test the capabilities of different penalty functions to improve the fuel economy while guaranteeing the battery charge sustainability. The potential of the proposed control strategy was firstly assessed on the Worldwide harmonized Light-duty vehicles Test Cycle (WLTC) and benchmarked against a Dynamic Programming (DP) optimization to evaluate each reward. Then the best agent was tested on a wide range of type-approval and Read Driving Emission (RDE) scenarios. The results show that the best-performing agent can reach performance close to the DP reference, with a limited gap (7 %) in terms of CO2 emissions.

Which Link Matters? Maintaining Connectivity of Uncertain Networks Under Adversarial Attack

This paper studies connectivity maintenance in uncertain networks under adversarial attack, where a defender conceals crucial links to prevent the largest connected component from being decomposed by an attacker. In contrast with its static counterpart, connectivity maintenance in uncertain networks involves additional probing on links to determine their existence. Therefore, by modeling an uncertain network as a random graph with each link associated with an existence probability and a probing cost, our goal is to design a defensive strategy for link selection that maximizes the expected size of the largest remaining connected component with the minimum expected probing cost, and moreover, the strategy should be independent of the attacking patterns. To this end, we first unravel the computational complexity of the problem by proving its NP-hardness, and then propose optimal defensive strategies based on dynamic programming and multi-objective optimization. Due to the prohibitive computational cost of optimality, two approximate defensive strategies are further designed to pursue decent performance with quasilinear complexity, in which the first one is a heuristic approach that quantifies the link vulnerability through an analogy from the degree centrality of a vertex in static networks to the connectivity weight of a link in uncertain networks, and the second one is an adaptive greedy policy incorporating the minimax rule from game theory, which minimizes the possible loss suffered by the defender in a worst-case scenario and has a constant approximation ratio. Extensive experiments on both synthetic and real-world network datasets under diverse attacking patterns demonstrate the superiority of the proposed strategies over baselines.

Distributed Sequential Detection for Cooperative Spectrum Sensing in Cognitive Internet of Things.

The rapid development of wireless communication technology has led to an increasing number of internet of thing (IoT) devices, and the demand for spectrum for these devices and their related applications is also increasing. However, spectrum scarcity has become an increasingly serious problem. Therefore, we introduce a collaborative spectrum sensing (CSS) framework in this paper to identify available spectrum resources so that IoT devices can access them and, meanwhile, avoid causing harmful interference to the normal communication of the primary user (PU). However, in the process of sensing the PUs signal in IoT devices, the issue of sensing time and decision cost (the cost of determining whether the signal state of the PU is correct or incorrect) arises. To this end, we propose a distributed cognitive IoT model, which includes two IoT devices independently using sequential decision rules to detect the PU. On this basis, we define the sensing time and cost functions for IoT devices and formulate an average cost optimization problem in CSS. To solve this problem, we further regard the optimal sensing time problem as a finite horizon problem and solve the threshold of the optimal decision rule by person-by-person optimization (PBPO) methodology and dynamic programming. At last, numerical simulation results demonstrate the correctness of our proposal in terms of the global false alarm and miss detection probability, and it always achieves minimal average cost under various costs of each observation taken and thresholds.

A controllable neural network-based method for optimal energy management of fuel cell hybrid electric vehicles

Neural Networks (NNs) can be used for energy management of hybrid vehicles, but they are hard to tune in inference to adapt to different driving conditions. To make the NN-based energy management strategy more flexible, this paper proposes a controllable NN for optimal energy management of fuel cell hybrid electric vehicles. Inspired by the equivalent factor in the Equivalent Consumption Minimization Strategy (ECMS), we introduce an adjustable target variable for the final state as an input to the NN-based strategy. During training, classification and regression networks with single-step and multi-step inputs are considered. An efficient shooting method and an adaptive method are then introduced to realize the precise control of the final state and online parameter adaptation. Simulations of the proposed method and the benchmarking method are carried out in different battery discharge modes. Results demonstrate that the proposed shooting neural classifier can achieve 99.7% fuel optimality of dynamic programming in a similar computational time to the shooting ECMS, and the proposed adaptive neural classifier can adapt to different driving conditions and has better fuel economy than the adaptive ECMS.

A dynamic programming based method for optimal control of a cascaded heat pump system with thermal energy storage

AbstractThe residential heating and cooling sector has been increasingly electrifying, predominantly using electrically driven heat pumps (HP) in combination with thermal/electrical energy storage systems. While these developments contribute to increased renewable and low carbon energy shares in the sector, exploiting the full potential of the technology requires a smart control of these systems that can account for predicted renewable energy availability in the future and the corresponding HP system performance. However, modelling a system featuring complex internal dynamics, in a way that is suitable for smart control, is challenging. Models need to be sophisticated enough to accurately capture the system's nonlinearities and intricacies while at the same time fast enough to enable a thorough search of the solutions space, in suitable computational time. Dynamic programming (DP) is a promising approach to smart controls, as it combines the ability to use complex, non-linear models while being an exhaustive search algorithm, guaranteeing that the global optimum is found. This paper presents an innovative modelling framework that entails reduced order models (ROM) of an HP substation's main components (i.e., HP and thermal energy storage—TES), elaborated in a fashion suitable for use in DP; these have been shaped as to include significant physical operating constraints (e.g., HP compressor variable speed, non-linear coefficient of performance—COP—dependency on outdoor and distribution temperature) affecting the system performance, while at the same time minimising the amount of state variables (i.e., TES temperatures, HP thermal and electric capacity) the optimizer needs to handle. In an application to an exemplary HP system, our system models compare remarkably well to detailed TRNSYS counterparts, used as a reference ground truth. The system achieves significant cost-saving enabled by the dynamic programming optimization approach, facilitating a 13% decrease in power consumption compared to conventional rule-based control.

The dispersive art gallery problem

We introduce a new variant of the art gallery problem that comes from safety issues. In this variant we are not interested in guard sets of smallest cardinality, but in guard sets with largest possible distances between these guards. To the best of our knowledge, this variant has not been considered before. We call it the Dispersive Art Gallery Problem. In particular, in the dispersive art gallery problem we are given a polygon P and a real number ℓ, and want to decide whether P has a guard set such that every pair of guards in this set is at least a distance of ℓ apart.In this paper, we study the vertex guard variant of this problem for the class of polyominoes. We consider rectangular visibility and distances as geodesics in the L1-metric. Our results are as follows. We give a (simple) thin polyomino such that every guard set has minimum pairwise distances of at most 3. On the positive side, we describe an algorithm that computes guard sets for simple polyominoes that match this upper bound, i.e., the algorithm constructs worst-case optimal solutions. We also study the computational complexity of computing guard sets that maximize the smallest distance between all pairs of guards within the guard sets. We prove that deciding whether there exists a guard set realizing a minimum pairwise distance for all pairs of guards of at least 5 in a given polyomino is NP-complete.We also present an optimal dynamic programming approach that computes a guard set that maximizes the minimum pairwise distance between guards in tree-shaped polyominoes, i.e., computes optimal solutions. Because the shapes constructed in the NP-hardness reduction are thin as well (but have holes), this result completes the case for thin polyominoes.

Multi-objective Floorplanning optimization engaging dynamic programming for system on chip

The quest for optimal VLSI Floorplanning (FP) addressing noteworthy challenges namely Area, Wirelength, and Temperature remains an ongoing field of research. Prevailing FP designs individually optimize these constraints that increase the model complexity with reduced efficiency. Given this motivation, this study provides an optimum FP aimed at minimizing the overall chip area by adhering to the standard design requirements. Accordingly, the divide-and-conquer model of Dynamic Programming (DP) adopted in this work simultaneously optimizes the chip area by devising separate analytical cost functions related to the physical design parameters that are then collated to yield the compact Floor plan. The formulated DP optimization coined as Weighted Aggregated DP-Based FP (WADPFP) adaptively tunes the cost function using rank-sum scaled weights and the discount factor thereby reducing congestion and hotspots. This objective is met using two novel recursive functions ensuring temperature and area scalability. Simulation and synthesis of the proposed FP on the Microelectronic Centre of North Carolina (MCNC) and Gigascale Systems Research Center (GSRC) Benchmark circuits demonstrated the viability of the automated FP design by registering 6.05%, 13%, and 4.35% reduction in area, wirelength, and temperature respectively in comparison with the traditional and recent peers. Additional analysis on the AMI49_X benchmark circuits emphasizes the scalable nature of the introduced WADPFP.

Resilient healthcare network for simultaneous product allocations during supply chain disruptions

ABSTRACT COVID-19 is responsible for medical emergencies, and uncertainty in healthcare supply chain (HSC). The HSC is struggling against the massive disruption due to the pandemic; thus, this alarming trend reinforces the need for a resilient HSC and a unique dynamic, responsive plan. The pandemic has strained hospitals and quarantine centres, necessitating crisis management for resource allocation. Addressing HSC issues, we've meticulously analysed vehicle routing problems (VRP) and devised solutions. The proposed methodology is exploring the dynamic programming (DP) for resilient HSC. The study offers a single-vehicle routing strategy with simultaneous loading and unloading of items between the depot and N nodes (hospitals and quarantine centres) to alleviate the vulnerability of commodities and recovery of items. The implication of this work is to provide dynamics of special purpose vehicle (SPV) with two storage facilities, one dedicated to fresh healthcare products (depot to nodes) and the second dedicated to old/recyclable/disposable healthcare products (nodes to depot). Finally, utilising the DP and particle swarm optimisation introduces an innovative strategy that compares efficient routing and allocation decisions for resilient HSC. It offers practitioners guidance on optimal operational sequences, optimal cost, and computational times for SPV, enhancing HSC response capabilities during crises.

Joint Offloading Decision and Trajectory Design for UAV-Enabled Edge Computing With Task Dependency

In this paper, we investigate the joint problem of task offloading, Unmanned Aerial Vehicle (UAV) trajectory design, and resource allocation for UAV-enabled edge computing, considering and highlighting the dependency among different tasks. The corresponding optimization problem, which is a mixed-integer problem, is formulated. To solve this problem, we propose an iterative method based on Block Coordinate Descent (BCD) to decompose the original problem into two subproblems. Given the offloading decision and resource allocation, the subproblem of UAV trajectory optimization is solved by convex optimization methods. Then, given the UAV trajectory, the subproblem of task offloading decision and the corresponding resource allocation is solved by dynamic programming and convex optimization methods. Simulation results show that our proposed method can significantly reduce energy consumption compared to the benchmark schemes.

Mobile Platform with Dynamic Optimization of the Pattern in Education in Colleges Through the Perspective of Network Informatization

The combination of mobile learning platforms and network informatization offers numerous benefits to learners, educators, and institutions. Learners can take control of their learning journey, accessing educational materials at their convenience and engaging in collaborative learning activities with peers from diverse backgrounds. This paper aims to explore the integration of mobile learning platforms and network informatization, examining their impact on educational practices, learner engagement, and the overall learning experience. The network informatization is assessed and monitored with Dynamic Programming Optimization (DPO) to compute the feature in reverse osmosis in English education. The attributes and features in the English language are computed and estimated for the periodic information update within the system. The DPO process is implemented along with the mandhani fuzzy set for the estimation of features in English education in colleges and universities. The information processed is updated in the mobile learning platform for the computation of the features in the English language and classification is performed with the deep learning model. Simulation analysis stated that constructed model is effective for the estimation and computation of the features and patterns in English language teaching in colleges and universities.