Bellman Equation Research Articles

Finite Differences on Sparse Grids for Continuous-Time Heterogeneous Agent Models

We present a finite difference method working on sparse grids to solve higher dimensional heterogeneous agent models. If one wants to solve the arising Hamilton–Jacobi–Bellman equation on a standard full grid, one faces the problem that the number of grid points grows exponentially with the number of dimensions. Discretizations on sparse grids only involve O(N(logN)d−1) degrees of freedom in comparison to the O(Nd) degrees of freedom of conventional methods, where N denotes the number of grid points in one coordinate direction and d is the dimension of the problem. While one can show convergence for the used finite difference method on full grids by using the theory introduced by Barles and Souganidis, we explain why one cannot simply use their results for sparse grids. Our numerical studies show that our method converges to the full grid solution for a two-dimensional model. We analyze the convergence behavior for higher dimensional models and experiment with different sparse grid adaptivity types.

An eco-physiological model of forest photosynthesis and transpiration under combined nitrogen and water limitation.

Although the separate effects of water and nitrogen (N) limitations on forest growth are well known, the question of how to predict their combined effects remains a challenge for modeling of climate change impacts on forests. Here, we address this challenge by developing a new eco-physiological model that accounts for plasticity in stomatal conductance and leaf N concentration. Based on optimality principle, our model determines stomatal conductance and leaf N concentration by balancing carbon uptake maximization, hydraulic risk and cost of maintaining photosynthetic capacity. We demonstrate the accuracy of the model predictions by comparing them against gross primary production estimates from eddy covariance flux measurements and sap-flow measurement scaled canopy transpiration in a long-term fertilized and an unfertilized Scots pine (Pinus sylvestris L.) forest in northern Sweden. The model also explains the response to N fertilization as a consequence of (i) reduced carbon cost of N uptake and (ii) increased leaf area per hydraulic conductance. The results suggest that leaves optimally coordinate N concentration and stomatal conductance both on short (weekly) time scales in response to weather conditions and on longer time scales in response to soil water and N availabilities.

Prediction of pressure-induced superconductivity in the ternary systems CaYH2n (n = 3–6) at moderate pressures

The high-pressure phase structures of CaYH2n (n = 3–6) ternary hydrides were systematically explored employing a combination of particle swarm optimization and first principles calculations.

Thermal performance optimization principle on absorption heat exchanger utilizing LiBr/H2O solution as working fluids: integration, interpretation and improvement

Thermal performance optimization principle on absorption heat exchanger utilizing LiBr/H2O solution as working fluids: integration, interpretation and improvement

Optimal and near-optimal control of a capacitated assemble-to-order system with component commonality and backordered demands

We investigate the optimal policy of a capacitated assemble-to-order (ATO) system with component commonality and backordered demands. Using a Markov decision process framework, we model the problem with the objective of minimising both the expected total discounted cost and the expected average cost rate over an infinite-horizon. To characterise the structure of the optimal policy, we adopt a two-step approach. First, we demonstrate that the value function satisfies a set of preliminary properties, indicating that it is optimal to allocate the common component’s inventory to the product with a larger backorder penalty, if its demand can be immediately fulfilled. This insight simplifies the Bellman optimality equation, allowing for a lower-dimensional state representation, which we then applied to fully characterise the optimal policy. Building on the structure of the optimal policy, we develop four heuristic policies. Extensive numerical experiments demonstrate that these heuristic policies perform well. In particular, heuristics H2 and H3 exhibit, on average, only 2% and 1% performance loss compared to the optimal policy. Our results offer practical implications for addressing larger ATO systems.

Comparative Analysis of OAR and PTV Radiation Dose Achievements for Planning 3D-CRT and IMRT Radiotherapy Techniques for Nasopharyngeal Cancer

A comparison has been conducted between the 3D-CRT technique and the IMRT technique on the radiation dose achievement of Planning Target Volume (PTV) and Organ at Risk (OAR) in nasopharyngeal cancer patients. This study used secondary data in the form of Dose Volume Histogram (DVH) graphs obtained from the results of the Treatment Planning System (TPS). Data analysis was carried out on the dose achievement at PTV with reference to The International Commission on Radiation Units and Measurements (ICRU) Report 62 in 1999, and the dose achievement at OAR with the guidelines of Quantitative Analysis of Normal. Tissue Effects in the Clinic (QUANTEC). The analysis was carried out on 15 nasopharyngeal cancer patients using the 3D-CRT technique and 15 nasopharyngeal cancer patients using the IMRT technique at Ken Saras Hospital, Semarang Regency. The results of the analysis showed that, in the IMRT radiation technique, the percentage of patients who received optimal radiation doses at PTV was 88.9% while for OAR, 11.85% of patients received radiation doses exceeding the QUANTEC tolerance limit. In the 3D-CRT radiation technique, the percentage of patients who received optimal radiation doses on the PTV was only 20% while for OAR, 28.9% of patients received radiation doses exceeding the QUANTEC tolerance limit. The results of the study showed that the IMRT radiation technique is more efficient for nasopharyngeal cancer therapy, because the dose achieved on the PTV is more optimal and the dose on the OAR can be minimized so that the principles of optimization and radiation limitation can be met properly. Keywords: nasopharyngeal cancer, 3D-CRT, IMRT, PTV, OAR, DVH

Two stackelberg games in life insurance: Mean-variance criterion

Abstract We study two continuous-time Stackelberg games between a life insurance buyer and seller over a random time horizon. The buyer invests in a risky asset and purchases life insurance, and she maximizes a mean-variance criterion applied to her wealth at death. The seller chooses the insurance premium rate to maximize its expected wealth at the buyer’s random time of death. We consider two life insurance games: one with term life insurance and the other with whole life insurance—the latter with pre-commitment of the constant investment strategy. In the term life insurance game, the buyer chooses her life insurance death benefit and investment strategy continuously from a time-consistent perspective. We find the buyer’s equilibrium control strategy explicitly, along with her value function, for the term life insurance game by solving the extended Hamilton–Jacobi–Bellman equations. By contrast, in the whole life insurance game, the buyer pre-commits to a constant life insurance death benefit and a constant amount to invest in the risky asset. To solve the whole life insurance problem, we first obtain the buyer’s objective function and then we maximize that objective function over constant controls. Under both models, the seller maximizes its expected wealth at the buyer’s time of death, and we use the resulting optimal life insurance premia to find the Stackelberg equilibria of the two life insurance games. We also analyze the effects of the parameters on the Stackelberg equilibria, and we present some numerical examples to illustrate our results.

A counterexample and a corrective to the vector extension of the Bellman equations of a Markov decision process

AbstractUnder the expected total reward criterion, the optimal value of a finite-horizon Markov decision process can be determined by solving the Bellman equations. The equations were extended by White to processes with vector rewards. Using a counterexample, we show that the assumptions underlying this extension fail to guarantee its validity. Analysis of the counterexample enables us to articulate a sufficient condition for White’s functional equations to be valid. The condition is shown to be true when the policy space has been refined to include a special class of non-Markovian policies, when the dynamics of the model are deterministic, and when the decision making horizon does not exceed two time steps. The paper demonstrates that in general, the solutions to White’s equations are sets of Pareto efficient policy returns over the refined policy space. Our results are illustrated with an example.

Strategic Traffic Management in Mixed Traffic Road Networks: A Methodological Approach Integrating Game Theory, Bilevel Optimization, and C-ITS

The integration of Connected Vehicles into conventional traffic systems presents significant challenges due to the diverse behaviors and objectives of different drivers. Conventional vehicle drivers typically follow User Equilibrium principles, aiming to minimize their individual travel times without considering the overall network impact. In contrast, Connected Vehicle drivers, guided by real-time information from central authorities or private service providers, can adopt System Optimum strategies or Cournot-Nash oligopoly behaviors, respectively. The coexistence of these distinct player classes in mixed-traffic environments complicates the task of achieving optimal traffic flow and network performance. This paper presents a comprehensive framework for optimizing mixed-traffic road networks through a multiclass traffic assignment model. The framework integrates three distinct types of players: conventional vehicle drivers adhering to User Equilibrium principles, Connected Vehicle drivers following System Optimum principles under a central governing authority, and Connected Vehicle drivers operating under Cournot-Nash oligopoly conditions with access to services from private companies. The methodology includes defining a model to achieve optimal mixed equilibria, designing an algorithm for multiclass traffic assignment, formulating strategic games to analyze player interactions, and establishing key performance indicators to evaluate network efficiency and effectiveness. The framework is applied to a real-world road network, validating its practicality and effectiveness through computational results. The extraction and analysis of computational results are used to propose optimal traffic management policies for mixed-traffic environments. The findings provide significant insights into the dynamics of mixed traffic networks and offer practical recommendations for improving traffic management in increasingly complex urban transportation systems.

An Operational Framework for a Low-carbon, Green Growth Economy: CO-STIRPAT Dynamic System

<p class="MsoNormal" style="margin-top: 12.0pt;"><span style="font-family: arial, helvetica, sans-serif;"><span lang="EN-US">This paper presents an operational framework for assessing the trajectories of production, energy, emissions, and capital accumulation to ensure the implementation of Nationally Determined Contributions (NDCs). The framework combines widely used methodologies (STIRPAT, system dynamics, and optimization) to simulate the pathways of variables </span><span lang="EN-US">until</span> <span lang="EN-US">a target year. The CO-STIRPAT dynamic system allows us to identify the spillover pathways from carbon policy to economic growth based on output optimization principles; to conduct a more systematic analysis of the interconnections between the main drivers that determine carbon emissions; to develop a cost-effective climate policy mix that is a backbone for the right combination of carbon pricing, energy efficiency, and carbon intensity; and to assess NDC targets with respect to ambition gaps, implementation gaps, and feasibility.</span></span></p>

Optimization of Energy Involved in Rolling Stock of a Sub-Urban Rail Transport System

Objectives: This research is based on the real-world study of Mumbai Metro, which focuses on enhancing the efficiency of metro rail systems by optimizing the timetable to synchronize train movements. The primary goal is to minimize total energy consumption by maximizing the exchange of regenerative-braking energy. Methods: Mumbai Metro Rolling Stock is having regenerative braking systems, allowing surplus energy regenerated during braking to be transferred to other trains in the same electrical rail section. To achieve this optimization, a graphical method is used to coordinate the time synchronization of the braking train arriving at a station with the accelerating train departing from the same or different station. This coordination ensures that regenerative-braking energy is effectively utilized and distributed among trains within the system. By quantifying the energy savings resulting from synchronization, this model provides an incentive for implementing more efficient train scheduling practices. The application of this model was demonstrated through the design of schedules for the Mumbai Metro. The implementation resulted in measurable energy savings, showcasing a clear correlation between train movement synchronization and reduced energy consumption. Importantly, this optimization strategy does not compromise overall passenger service quality and can be implemented with relatively low investment costs. In conclusion, adjusting the timetable in force based on principles of time synchronization and energy optimization has the potential to yield considerable energy savings for metro rail systems. Findings: This research emphasizes the importance of strategic train scheduling in maximizing the efficiency and sustainability of public transportation networks using regeneration of electrical energy. By implementing the optimized timetable, could lead to approximately a 5.03 % increase in regenerative energy utilization which would save 2.2 % in total energy consumption. Novelty: This study introduces a novel method to assess energy cooperation between trains using timetable optimization because this is the first real-world study of its kind conducted on any Indian metro system in which if one train is in braking mode then another train in the same electrical section should be in motoring mode so as to avoid the losses of the regenerated energy. Keywords: Optimization, Headway, Dwell time, Coasting period, Traction energy

Fixed-time reinforcement learning and optimal control design

This paper presents a fixed-time optimal control design approach using reinforcement learning (RL) that guarantees not only fixed-time convergence of the learning algorithm to an optimal controller but also fixed-time stability of the learned control solution. To ensure the former, zero-finding capabilities of the zeroing neural networks (ZNNs) are leveraged, and novel adaptive laws are presented accordingly. To ensure the latter, conditions on the cost function are provided for which its corresponding optimal controller assures the fixed-time stability of the closed-loop system. It is also shown that imposing a fixed-time stability constraint on the infinite-horizon optimal control solution actually solves the classical fixed-final-time (FFTM) finite-horizon optimal control problem. The Hamilton–Jacobi–Bellman (HJB) equation for the FFTM optimal control problem is time-varying, which makes it hard or even impossible to learn it directly online using RL. The presented approach bypasses this difficulty by developing an online solution for infinite-horizon optimal control problems under fixed-time stability constraints and with fixed-time convergent tuning laws. This approach makes learning and closed-loop system settling times predictable, tunable, and bounded. Simulation results for fixed-time optimal adaptive stabilization of a torsional pendulum system clarify this new design for nonlinear optimal control theory.

Core elements of the perioperative medicine for older people undergoing surgery (POPS) model of care.

Hospital surgical services that utilise the approach of the perioperative medicine for older people undergoing surgery (POPS) model of care improve outcomes for older people contemplating and undergoing surgery. Complex models of care like POPS may be difficult to implement without understanding the elements that comprise that model of care. Logic models can be used to aid implementation by visually depicting theoretical relationships between the elements of the model of care. Our objective was to understand the core elements of the POPS model of care at health services other than where it was first developed. A qualitative case study at three contextually different health services in England with POPS models of care of varying implementation maturity was undertaken. We conducted semi-structured interviews with clinicians and managers involved in POPS (n = 56). The interviews were analysed using inductive and deductive methods. We developed a logic model with seven domains and themes that described the core elements of the POPS model of care compared and contrasted across the three health services. We found POPS could be adapted to 'fit' the local contexts of our study and still achieve its desired outcomes if it remained true to the principles of comprehensive geriatric assessment and optimisation and was delivered by staff with expert skills and attitudes. Our logic model provides potentially generalisable information about the core elements of the POPS service in three health services. This information can be used to aid the implementation of the POPS model of care in healthcare settings similar to our study. Further research may be required to test the logic model in other healthcare contexts.

Study on the Impact of Operating Room Nursing Process Optimization on Surgical Safety

This study aims to explore the impact of optimizing the operating room nursing process on surgical safety. In the current healthcare environment, surgical safety is a core aspect of clinical work, and optimizing nursing processes is a key factor in enhancing surgical safety. By analyzing the current situation of nursing processes in Chinas operating rooms, this study identifies certain issues and deficiencies in the existing processes. Based on optimization principles, a scientifically designed nursing process optimization plan was developed and empirically tested. The results of the study indicate that the optimized nursing process significantly improved safety indicators, nursing efficiency, and patient satisfaction. This research provides theoretical support and practical guidance for the optimization of operating room nursing processes, which has significant practical implications for enhancing surgical safety.

Pensions and protestants: or why everything in retirement can’t be optimized

Abstract A common narrative among insurance actuaries and business economists is that national or regional pension systems can be finetuned, optimized, and improved simply by tinkering with demographic and financial parameters; all within the context of the “right” mathematical model. Indeed, recent papers in the actuarial literature have offered technical fixes around savings rates, retirement ages, decumulation strategies as well as more refined mortality and interest rate models. But alas, not everything in the world of pensions and retirement can be optimized, in particular as it relates to the history, background culture, or religion of the underlying population. This paper documents a statistically significant relationship between a region’s pension plan “health status” and the fraction of the region’s population identifying as Protestant Christians (PC). We begin the analysis at the national level using a well-known pension quality index and then obtain similar results for the actuarial funded status of U.S. state pension plans. Overall, this work is within the sphere of recent literature that indicates historical religious beliefs, values, and culture matter for financial economic outcomes; a factor which obviously can’t be optimized within a mathematical Hamilton–Jacobi–Bellman (HJB) equation. In other words, some things in retirement are truly beyond control.

A simple method for common-mode voltage optimization in three-level three-phase four-leg inverters

Abstract In response to the problem of large common-mode (CM) voltage magnitude and high-frequency variation under traditional modulation strategies for three-level three-phase four-leg (3L3P4L) inverters, this paper proposes a modulation strategy based on carrier combination and modulation wave splitting. Firstly, this paper establishes a CM model for the 3L3P4L inverter and derives mathematical expressions of the system’s CM voltage in terms of switch states and system parameters. Building upon this foundation, this paper explains the reasons for high-frequency variations in CM voltage under traditional schemes and provides a detailed introduction to the optimization process and principles of the proposed strategy. Finally, a simulation model is developed by using Matlab/Simulink software. Through comparison with traditional approaches, the effectiveness of the proposed strategy is validated.

Phase transition in a kinetic mean-field game model of inertial self-propelled agents

The framework of mean-field games (MFGs) is used for modeling the collective dynamics of large populations of non-cooperative decision-making agents. We formulate and analyze a kinetic MFG model for an interacting system of non-cooperative motile agents with inertial dynamics and finite-range interactions, where each agent is minimizing a biologically inspired cost function. By analyzing the associated coupled forward–backward in a time system of nonlinear Fokker–Planck and Hamilton–Jacobi–Bellman equations, we obtain conditions for closed-loop linear stability of the spatially homogeneous MFG equilibrium that corresponds to an ordered state with non-zero mean speed. Using a combination of analysis and numerical simulations, we show that when energetic cost of control is reduced below a critical value, this equilibrium loses stability, and the system transitions to a traveling wave solution. Our work provides a game-theoretic perspective to the problem of collective motion in non-equilibrium biological and bio-inspired systems.

The Projected Bellman Equation in Reinforcement Learning

The Projected Bellman Equation in Reinforcement Learning

The temporal scale of energy maximization explains allometric variations in movement decisions of large herbivores

AbstractEmpirical testing of energy maximization models has been used to clarify the drivers of resource partitioning among large herbivores. Most studies, however, have not considered that predictions of optimal diet depend on the temporal scale of maximization. This omission can hinder the effectiveness of optimality principles in elucidating animal distributions, dietary choices, and the dynamics of species coexistence. We used movement analysis and scale‐dependent energy gain modeling to study how three large herbivores share resources: red hartebeest (Alcelaphus buselaphus), a 120‐kg grazing ruminant; zebra (Equus quagga), a 300‐kg grazing nonruminant; and eland (Tragelaphus oryx), a 460‐kg ruminant, mixed feeder. We found that resource partitioning was achieved through a synergy of spatial segregation and interspecies differences in habitat selection and in the temporal scale of energy maximization. Radio‐collared individuals of the three species spent 95% of their time >850 m from one another. Hierarchical movement analysis revealed that red hartebeest and zebra selected grasslands within which they selected patches maximizing their daily energy gains. Selection was particularly strong for red hartebeest, as expected for a ruminant of relatively small size. Unlike the other species, eland avoided grasslands; when they ventured into grasslands, they selected patches offering high short‐term energy gains at the expense of daily gains. This selection for rapid energy gain could reflect relatively high missed opportunity costs when foraging in grasslands due to the broad range of feeding opportunities for this large mixed feeder. This finding is also consistent with the notion that larger herbivores tend to face stronger constraints from resources availability than digestibility. Overall, differences in selection strength and foraging currencies among these large herbivores are consistent with allometric theory. Our study illuminates the drivers of resource partitioning that can promote the coexistence of large herbivore species, while also showing that, to provide a useful and robust basis to explain animal movement and resource partitioning, energetic models should be based on a relevant scale of energy maximization.

In-Depth Analysis of MEC Resource Optimization and Reliability Under 5G Empowerment

<p>The swift proliferation of 5G networks underscores the pivotal role of Mobile Edge Computing (MEC) in catering to the ever-evolving service demands. This study introduces a dynamic resource orchestration framework tailored for MEC within 5G standalone cellular architectures. This framework addresses the intricate challenges of computational resource provisioning and power management in a holistic manner. Leveraging convex optimization principles, we devise an optimal resource allocation strategy that incorporates reliability metrics to ensure robust performance. Furthermore, we employ Deep Reinforcement Learning (DRL) techniques to tackle a formulated Markov Decision Process (MDP), aimed at optimizing resource distribution for latency minimization. Our simulation outcomes demonstrate the efficacy of the proposed approach in mitigating task delays and enhancing the system’s adaptability across diverse workload profiles and network environments, thereby contributing novel insights into the field with minimal overlap in existing literature.</p> <p> </p>