Bellman Equation Research Articles

Integrating product design principles to design a smart assistive cane

This paper presents the design of a smart assistive cane using the principles of product design and quality assurance. The smart cane is aimed at improving the safety and independence of individuals with walking difficulties. The proposed cane incorporates advanced features such as GPS navigation, fall detection, adjustable height, and integrated storage. A comprehensive design process was employed, including a thorough analysis of user needs through a Kano analysis, followed by the selection of appropriate materials and manufacturing methods. The cane's structural integrity and functionality were assessed through rigorous testing and finite element analysis. This research contributes to the development of assistive technologies that can incorporate user’s necessities and builds quality into the product with the design process, significantly enhancing the quality of life for individuals with mobility challenges. It serves as a potential demonstration to designers and researchers to incorporate product design, optimization, and quality assurance principles in design of similar assistive technologies and products.

Machine‐Learning‐Assisted Process Optimization for High‐Performance Organic Thermoelectrics

AbstractOptimizing the processing conditions of conjugated polymers is a key step in improving the performance of various organic electronic devices, including organic thermoelectric (TE) devices that are capable of harvesting energy from waste heat. However, the inherent structural and energetic disorders in these materials and their unpredictable property changes after processing complicate the optimization principles, requiring numerous trial‐and‐error experiments to maximize the TE performance. Here, a machine‐learning (ML)‐based design of experiments approach is introduced to address these challenges, and its effectiveness is demonstrated using a representative thiophene‐based doped polymer system for organic TE. This approach not only helps to quantitatively understand how each processing parameter affects the TE properties of the polymer but also facilitates the prediction of optimal processing conditions, leading to achieve maximum TE performance with minimal experiments within a large parameter space. Furthermore, the ML‐based approach is validated by revealing the origins of the power factor maximization at the ML‐predicted processing conditions through analysis of the morphology and electronic states of the doped polymer films. The proposed methodology is applicable to the majority of polymer systems for organic thermoelectric energy conversion and will provide guidelines for determining optimal process conditions and improving TE performance with minimal effort.

Distributed optimal consensus control for discrete-time multi-agent systems with composite switching topologies via an iterative Q-learning method

This paper aims to achieve distributed optimal consensus control for discrete-time multi-agent systems (MASs) under composite switching topologies through utilising the iterative Q-learning approach. First, the optimal consensus control strategy, independent of system dynamics information, is developed by using the designed iterative Q-learning approach to guarantee that all follower states can synchronise to the desired trajectory formulated by the leader. Compared with the invariable algebraic or edge-fixed topology structures, the designed Q-learning control algorithm in this paper is implemented based on the composite switching topologies consisting of periodic and stochastic mechanisms. Then, an error dynamic system is established, which transforms the optimal consensus control issue into an optimal regulation issue and removes the influence of the two types of switching topologies. To conquer the problems of lack of model information and the inaccessible analytical solution of Hamilton-Jacobi-Isaacs (HJI) equation, an iterative Q-learning approach is designed according to the reconstructed Q-function Bellman equation. Through further iterative learning, the optimal control strategy can be acquired so as to realise consensus control and ensure the stability of the closed-loop systems. Finally, two simulation examples are provided to demonstrate the effectiveness of the proposed distributed optimal control scheme.

Decentralized Robust Tracking Control of Interconnected Nonlinear‐Constrained Systems: A Dynamic Event‐Sampled Method

ABSTRACTThis paper presents a decentralized robust dynamic event‐sampled tracking (EST) control law for interconnected nonlinear‐constrained systems. The core of developing such a control law is to convert the original EST control problem into the event‐sampled decentralized stabilization problem of augmented interconnected systems. To address the transformed decentralized stabilization problem, an indirect approach relying on the optimal control methodology is proposed. Initially, a group of cost functions are constructed for the nominal subsystems related to the augmented interconnected systems. Then, the dynamic event‐sampling mechanisms are introduced for lessening the computational burden. Meanwhile, the event‐sampled Hamilton–Jacobi–Bellman equations (ES‐HJBEs) are proposed for the augmented interconnected systems. To approximately solve the ES‐HJBEs, the critic approximators are used with their parameters tuned under the reinforcement learning framework. After that, the uniform ultimate boundedness of the tracking errors and the approximators' parameter estimation errors are assured based on the Lyapunov theorem. Finally, a nonlinear plant is provided to validate the decentralized robust dynamic EST control law.

Predicting Wheat Potential Yield in China Based on Eco-Evolutionary Optimality Principles

Accurately predicting the wheat potential yield (PY) is crucial for enhancing agricultural management and improving resilience to climate change. However, most existing crop models for wheat PY rely on type-specific parameters that describe wheat traits, which often require calibration and, in turn, reduce prediction confidence when applied across different spatial or temporal scales. In this study, we integrated eco-evolutionary optimality (EEO) principles with a universal productivity model, the Pmodel, to propose a comprehensive full-chain method for predicting wheat PY. Using this approach, we forecasted wheat PY across China under typical shared socioeconomic pathways (SSPs). Our findings highlight the following: (1) Incorporating EEO theory improves PY prediction performance compared to current parameter-based crop models. (2) In the absence of phenological responses, rising atmospheric CO2 concentrations universally benefit wheat growth and PY, while increasing temperatures have predominantly negative effects across most regions. (3) Warmer temperatures expand the window for selecting sowing dates, leading to a national trend toward earlier sowing. (4) By simultaneously considering climate impacts on wheat growth and sowing dates, we predict that PY in China’s main producing regions will significantly increase from 2020 to 2060 and remain stable under SSP126. However, under SSP370, while there is no significant trend in PY during 2020–2060, increases are expected thereafter. These results provide valuable insights for policymakers navigating the complexities of climate change and optimizing wheat production to ensure food security.

Optimal Consumption and Investment with Income Adjustment and Borrowing Constraints

In this paper, we address the utility maximization problem of an infinitely lived agent who has the option to increase their income. The agent can increase their income at any time, but doing so incurs a wealth cost proportional to the amount of the increase. To prevent the agent from infinitely increasing their income and borrowing against future income, we additionally consider a non-negative wealth constraint that prohibits borrowing based on future income. This utility maximization problem is a mixture of stochastic control, where the agent chooses consumption and investment, and singular control, where the agent chooses a non-decreasing income process. To solve this non-trivial and challenging problem, we derive the Hamilton–Jacobi–Bellman (HJB) equation with a gradient constraint using the dynamic programming principle (DPP). Then, using the guess-and-verify method and a linearization technique, we obtain a closed-form solution to the HJB equation and, based on this, find the optimal strategy.

Relaxed optimal control problem for a finite horizon G-SDE with delay and its application in economics

This paper investigates the existence of a G-relaxed optimal control of a controlled stochastic differential delay equation driven by G-Brownian motion (G-SDDE in short). First, we show that optimal control of G-SDDE exists for the finite horizon case. We present as an application of our result an economic model, which is represented by a G-SDDE, where we studied the optimisation of this model. We connected the corresponding Hamilton–Jacobi–Bellman equation of our controlled system to a decoupled G-forward backward stochastic differential delay equation (G-FBSDDE in short). Finally, we simulate this G-FBSDDE to get the optimal strategy and cost.

New aspects of the development of Kantorovich’s one-product dynamic model of replacement of production funds

In the context of the development of scientific and technological progress and the growth of the capital stock, an important task is modeling and optimization of the terms of operation of production funds. The development of technologies and automation of production lead to a reduction in the workforce employed in production. In the article, for the previously developed single-product dynamic model of the replacement of production assets, taking into account the inertial properties of the funds being introduced, the case of a reduction in labor resources under the condition of an increase in output and capital investment is investigated. A solution was obtained that allows to determine the optimal strategy for the withdrawal of obsolete funds and the introduction of the new ones in case of decrease in labor resources. As the optimization criterion the principle of differential optimization is used. The theorem of equivalence of a trajectory satisfying the principle of differential optimization, the obtained solution is given. A system of functional differential equations is presented, which should be satisfied by variable models both in terms of growth and reduction of labor resources. Numerical methods are used to solve such a system. The variants of the system development are considered under various assumptions regarding changes in the total amount of labor resources. The results of numerical implementation of the model are presented.

ANALYSIS OF OPTIMAL PORTFOLIO ON FINITE AND SMALL-TIME HORIZONS FOR A STOCHASTIC VOLATILITY MODEL WITH MULTIPLE CORRELATED ASSETS

In this paper, we consider the portfolio optimization problem in a financial market where the underlying stochastic volatility model is driven by [Formula: see text]-dimensional Brownian motions. At first, we derive a Hamilton–Jacobi–Bellman equation including the correlations among the standard Brownian motions. We use an approximation method for the optimization of portfolios. With such approximation, the value function is analyzed using the first-order terms of expansion of the utility function in the powers of time to the horizon. The error of this approximation is controlled using the second-order terms of expansion of the utility function. It is also shown that the one-dimensional version of this analysis corresponds to a known result in the literature. We also generate a close-to-optimal portfolio near the time to horizon using the first-order approximation of the utility function. It is shown that the error is controlled by the square of the time to the horizon. Finally, we provide an approximation scheme to the value function for all times and generate a close-to-optimal portfolio.

Optimal Inspection and Maintenance Policy: Integrating a Continuous-Time Markov Chain into a Homing Problem

The state of a machine is modeled as a controlled continuous-time Markov chain. Moreover, the machine is being serviced at random times. The aim is to maximize the time until the machine must be repaired, while taking the maintenance costs into account. The dynamic programming equation satisfied by the value function is derived, enabling optimal decision-making regarding inspection rates, and special problems are solved explicitly. This approach minimizes direct maintenance costs along with potential failure expenses, establishing a robust methodology for determining inspection frequencies in reliability-centered maintenance. The results contribute to the advancement of maintenance strategies and provide explicit solutions for particular cases, offering ideas for application in reliability engineering.

Optimal Model-Free Mean-Square Consensus for Multi-Agents with Markov Switching Topology

Due to the real applications, optimal consensus reinforcement learning with switching topology is still challenging due to the complexity of topological changes. This paper investigates the optimal consensus control problem for discrete multi-agent systems under Markov switching topologies. The goal is to design an appropriate algorithm to find the optimal control policies that minimize the performance index while achieving consensus among the agents. The concept of mean-square consensus is introduced, and the relationship between consensus error and tracking error to achieve mean-square consensus is studied. A performance function for each agent under switching topologies is established and a policy iteration algorithm using system data is proposed based on the Bellman optimality principle. The theoretical analysis shows that the consensus error realizes mean-square consensus and the performance function is optimized. The efficacy of the suggested approach is confirmed by numerical simulation using an actor–critic neural network. As a result, the value function is the optimum and the mean-square consensus can be reached using this technique.

Feedback Stabilization of Chain-Like MDOF Nonlinear Structural Systems Under Purely Parametric Gaussian White Noises

A feedback control method is proposed to asymptotically stabilize the chain-like multi-degree-of-freedom (MDOF) nonlinear structural system under purely parametric Gaussian white noises. First, the motion equation of a chain-like nonlinear structural system with [Formula: see text] coupled elements and an undetermined control law is derived. It is reduced to a one-dimensional partially averaged Itô stochastic differential equation of the controlled Hamiltonian by applying the stochastic averaging method, which bypasses the challenge of calculating multiple Lyapunov exponents. Next, based on the dynamical programming principle, the dynamical programming equation of the averaged system with an undetermined cost function is established and addressed to obtain the optimal control law. Then, the Lyapunov exponent of the completely averaged Itô equation is derived. For the given requirement of stabilizing the system, the Lyapunov exponent is entirely fixed and used to analyze the stability of the originally chain-like MDOF nonlinear structural system. A controlled chain-like 8-DOF nonlinear structural system is taken as an example to illustrate the application and effectiveness of the proposed procedure and the effect of control forces on stabilizing the system.

On penalized goal-reaching probability minimization under borrowing and short-selling constraints

Abstract We consider a robust optimal investment–reinsurance problem to minimize the goal-reaching probability that the value of the wealth process reaches a low barrier before a high goal for an ambiguity-averse insurer. The insurer invests its surplus in a constrained financial market, where the proportion of borrowed amount to the current wealth level is no more than a given constant, and short-selling is prohibited. We assume that the insurer purchases per-claim reinsurance to transfer its risk exposure to a reinsurer whose premium is computed according to the mean–variance premium principle. Using the stochastic control approach based on the Hamilton–Jacobi–Bellman equation, we derive robust optimal investment–reinsurance strategies and the corresponding value functions. We conclude that the behavior of borrowing typically occurs with a lower wealth level. Finally, numerical examples are given to illustrate our results.

Informational and Methodical Support of Out-Of-School Education Institutions: Basic Principles and Current State

The article reveals the essence of informational and methodical support of out-of-school education institutions. It was found that informational and methodological support is a system of necessary and sufficient conditions for the organization of the educational process, which includes a set of informational, educational, methodological and scientific materials to ensure the need for information sources. It was determined that when developing educational and methodological materials, certain principles should be followed: scientificity, availability and reliability of information, systematicity, relevance, adaptability and innovativeness of educational materials. The main principles of informational and methodical provision of out-of-school education institutions are characterized, in particular, the principle of legality and compliance of the regulatory and legal field; the principle of unity of purpose and goals; the principle of information security; the principle of functional structuring; optimization principle; principle of availability; the principle of sufficiency of informational and methodological support. An analysis of the current state of informational and methodological support of out-of-school education institutions in wartime conditions was carried out by interviewing leaders of clubs and other creative associations from all over Ukraine. The exception was Zaporizhzhia, Kharkiv and Kherson regions, which, due to the hostilities, could not participate. The level of provision of out-of-school education institutions with the necessary teaching and methodical materials for conducting classes has been determined; the quality of extracurricular education programs was analyzed; forms of improvement of the professional level of leaders of circles and other creative associations during the last five years have been clarified; the level of use of innovative forms and methods of work by teachers and which innovations are mainly used by teachers of extra-curricular education institutions is established. It has been proven that informational and methodological support plays a key role in the system of extracurricular education, contributing to the improvement of the quality of the educational process and the effectiveness of pedagogical activities.

Snow Resource Reutilization: Design of Snow Collection and Compression Equipment Based on Functional Analysis Method

High-latitude regions of the Earth are rich in natural snow resources; however, owing to their negative impact on daily life, they have not been effectively utilized for a long time and are instead viewed as obstacles that require substantial resources for clearance. This waste of resources contradicts the principles of sustainable development. With the rapid development of the ice and snow industry, the social and economic value of snow resources is gradually becoming apparent. Therefore, to promote sustainable social and economic development, this study explores new methods for processing snow resources to achieve their recycling and high-value transformation. This study employs the functional analysis method to analyze and solve the functions of the design system, utilizing the Analytic Hierarchy Process (AHP) and fuzzy comprehensive evaluation for the objective assessment and selection of schemes. The obtained relative optimal principles are utilized to guide the parameter setting and innovative design of the scheme in terms of functional structure. After the final scheme is output, its feasibility is verified through finite element simulation. Ultimately, to address the issue of snow resource clearing and recycling, this study designs a product scheme capable of collecting and compressing snow on road surfaces, proposing the use of snow in the form of standardized compressed snow blocks for large-scale snow sculpture construction and other fields. This method significantly reduces the cost of snow sculpture production, enhances efficiency, realizes the comprehensive utilization and high-value transformation of snow resources, and provides a reference for the sustainable development of the low-altitude ice and snow tourism industry.

Multi-objective optimization of a bistable curved shell with controllable thickness based on machine learning

Bistable curved shells have become a promising low-cost application in energy absorption fields owing to recent advances in material and technology. Significant research has been conducted to improve their energy absorption effect through forward prediction and single-objective optimization. However, these approaches may not fully explore their functional potential. In this study, we propose a multi-objective optimization framework based on the principle of main objective optimization that combines neural networks and genetic algorithms. The energy absorption effect and backward snapping force of the bistable curved shell are improved synchronously. Meanwhile, a reverse design algorithm is developed to generate the preset load-displacement curve, which further expands the application of machine learning methods in the field of multi-objective optimization. The combination of machine learning and multi-objective optimization is highly effective for building meta-structures with specific performance requirements and has potential applications in solving complex optimization tasks in various fields.

Enhanced and robust contrast in CEST MRI: Saturation pulse shape design via optimal control.

To employ optimal control for the numerical design of Chemical Exchange Saturation Transfer (CEST) saturation pulses to maximize contrast and stability against inhomogeneities. We applied an optimal control framework for the design pulse shapes for CEST saturation pulse trains. The cost functional minimized both the pulse energy and the discrepancy between the corresponding CEST spectrum and the target spectrum based on a continuous radiofrequency (RF) pulse. The optimization is subject to hardware limitations. In measurements on a 7 T preclinical scanner, the optimal control pulses were compared to continuous-wave and Gaussian saturation methods. We conducted a comparison of the optimal control pulses with Gaussian, block pulse trains, and adiabatic spin-lock pulses. The optimal control pulse train demonstrated saturation levels comparable to continuous-wave saturation and surpassed Gaussian saturation by up to 50 % in phantom measurements. In phantom measurements at 3 T the optimized pulses not only showcased the highest CEST contrast, but also the highest stability against field inhomogeneities. In contrast, block pulse saturation resulted in severe artifacts. Dynamic Bloch-McConnell simulations were employed to identify the source of these artifacts, and underscore the robustness of the optimized pulses. In this work, it was shown that a substantial improvement in pulsed saturation CEST imaging can be achieved by using Optimal Control design principles. It is possible to overcome the sensitivity of saturation to B0 inhomogeneities while achieving CEST contrast close to continuous wave saturation.

Policy Iteration-Based Learning Design for Linear Continuous-Time Systems Under Initial Stabilizing OPFB Policy.

Policy iteration (PI), an iterative method in reinforcement learning, has the merit of interactions with a little-known environment to learn a decision law through policy evaluation and improvement. However, the existing PI-based results for output-feedback (OPFB) continuous-time systems relied heavily on an initial stabilizing full state-feedback (FSFB) policy. It thus raises the question of violating the OPFB principle. This article addresses such a question and establishes the PI under an initial stabilizing OPFB policy. We prove that an off-policy Bellman equation can transform any OPFB policy into an FSFB policy. Based on this transformation property, we revise the traditional PI by appending an additional iteration, which turns out to be efficient in approximating the optimal control under the initial OPFB policy. We show the effectiveness of the proposed learning methods through theoretical analysis and a case study.

A dedicated branch-price-and-cut algorithm for advance patient planning and surgeon scheduling

In this paper, we study the patient planning and surgeon scheduling in the operating room theatre. The problem considers the simultaneous planning of patients and the assignment of time blocks to surgeons so that they can perform the surgery of their patients. The timing and length of the allotted time blocks depend on the patient characteristics on the surgeons’ waiting lists. Solving this problem in an exact manner is challenging due to the large number of rooms, surgeons, and patients involved. To overcome this challenge, we propose an efficient branch-price-and-cut algorithm to find an optimal solution in an acceptable time span. For that purpose, we include different dedicated mechanisms to accelerate the solution-finding process. In this regard, the branch-price-and-cut tree is set up using an intelligent branching scheme, the nodes are searched in order of the lowest number of fractional variables, and improved bounds are computed to prune nodes earlier. To tighten the convex hull of the linear programming relaxation in each node, the algorithm relies on a row generation mechanism for adding valid inequalities. We conducted various computational experiments to demonstrate the performance of our algorithm and validate for each component the contribution of the implemented optimisation principles. Additionally, we show the superior performance of the proposed algorithm to alternative optimisation procedures.

Dynamic programming principle for optimal control of uncertain random differential equations and its application to optimal portfolio selection

This study aimed to examine an uncertain stochastic optimal control problem premised on an uncertain stochastic process. The proposed approach is used to solve an optimal portfolio selection problem. This paper’s research is relevant because it outlines the procedure for solving optimal control problems in uncertain random environments. We implement Bellman’s principle of optimality method in dynamic programming to derive the principle of optimality. Then the resulting Hamilton-Jacobi-Bellman equation (the equation of optimality in uncertain stochastic optimal control) is used to solve a proposed portfolio selection problem. The results of this study show that the dynamic programming principle for optimal control of uncertain stochastic differential equations can be applied in optimal portfolio selection. Also, the study results indicate that the optimal fraction of investment is independent of wealth. The main conclusion of this study is that, in Itô-Liu financial markets, the dynamic programming principle for optimal control of uncertain stochastic differential equations can be applied in solving the optimal portfolio selection problem.