Bellman Equation Research Articles

Portfolio and reinsurance optimization under unknown market price of risk

We investigate the optimal investment-and-reinsurance problem for insurance company with partial information on the market price of the risk. Through the use of filtering techniques, we convert the original optimization problem involving different filtrations into an equivalent stochastic control problem under the observation filtration, i.e. the so-called separated problem. The Markovian structure of the separated problem allows us to apply a classical approach to stochastic optimization based on the Hamilton–Jacobi–Bellman equation, and to provide explicit formulas for the value function and the optimal investment-and-reinsurance strategy. We finally discuss some comparisons between the optimal strategies pursued by a partially informed insurer and that followed by a fully informed insurer, and we evaluate the value of information using the idea of indifference pricing. These results are also supported by numerical experiments.

Vibration suppression of series elastic actuator used for robotic grinding based on reconstructed hybrid optimized input shaper

Vibration suppression of the series elastic actuator (SEA) used for robotic compliant grinding is significant for guaranteeing force control accuracy and stability, which is important for material removal accuracy and surface quality improvement. This paper establishes a multi-feedforward control system to achieve vibration suppression for the SEA. The hybrid optimized input shaper (HOIS), instruction reconstructor, and nonlinear friction compensation model are seamlessly integrated into the multi-feedforward control method. Combination optimization principles are employed to formulate the HOIS, ensuring selectable performances on vibration suppression, response speed, and model stability when addressing the residual vibration caused by elastic elements. Base on the inherent delay characteristics of the shaper, a head/tail compression and intermediate shift instruction reconstruction (CSIR) method is introduced to address the tracking error induced by the time-delay of the shaper. Additionally, the friction compensator is established based on the Stribeck model and integrated into the feedforward control system to mitigate the vibration caused by the low-speed viscous friction. The effectiveness of the proposed method is verified by experimental tests. Results show that the HOIS has rich optional performance and can effectively suppress vibration. The tracking error caused by the shaper is effectively mitigated through the reconstruction of instructions. Grinding tests on curved parts are conducted on the robotic grinding system with SEA used as the force-control system. Compared with the benchmark method, the developed reconstructed hybrid optimized input shaper reduces the average force tracking error, the average absolute grinding depth error, and average roughness by 26.6 %, 22.5 %, and 21.5 % respectively. At the natural frequency of the grinding system, the amplitude of the frequency spectrum with the developed multi-feedforward method decreases by 48.13 %.

Online optimal tracking control of unknown nonlinear singularly perturbed systems using single network adaptive critic with improved learning

This study is the first time devoted to seek an online optimal tracking solution for unknown nonlinear singularly perturbed systems based on single network adaptive critic (SNAC) design. Firstly, a novel identifier with more efficient parametric multi-time scales differential neural network (PMTSDNN) is developed to obtain the unknown system dynamics. Then, based on the identification results, the online optimal tracking controller consists of an adaptive steady control term and an optimal feedback control term is developed by using SNAC to solve the Hamilton–Jacobi–Bellman (HJB) equation online. New learning law considering filtered parameter identification error is developed for the PMTSDNN identifier and the SNAC, which can realize online synchronous learning and fast convergence. The Lyapunov approach is synthesized to ensure the convergence characteristics of the overall closed loop system consisting of the PMTSDNN identifier, the SNAC and the optimal tracking control policy. Three examples are provided to illustrate the effectiveness of the investigated method.

Power utility maximization with expert opinions at fixed arrival times in a market with hidden Gaussian drift

AbstractIn this paper we study optimal trading strategies in a financial market in which stock returns depend on a hidden Gaussian mean reverting drift process. Investors obtain information on that drift by observing stock returns. Moreover, expert opinions in the form of signals about the current state of the drift arriving at fixed and known dates are included in the analysis. Drift estimates are based on Kalman filter techniques. They are used to transform a power utility maximization problem under partial information into an optimization problem under full information where the state variable is the filter of the drift. The dynamic programming equation for this problem is studied and closed-form solutions for the value function and the optimal trading strategy of an investor are derived. They allow to quantify the monetary value of information delivered by the expert opinions. We illustrate our theoretical findings by results of extensive numerical experiments.

A learning-based nearly optimal control framework for trajectory tracking of a flexible-link manipulator system with actuator fault

In this paper, a learning-based nearly optimal control framework with fault-tolerant capability is designed to tackle the tracking control problem of a flexible-link manipulator in the presence of actuator fault and model uncertainties. Initially, the optimal control law is obtained by adopting the dynamic programming and a critic structure as the solution of Hamilton–Jacobi–Bellman equation for the nominal model. Then, by implementing an integral sliding mode control, the robustness against actuator fault and model uncertainty is guaranteed. The adaptive laws are constructed based on radial basis functions neural networks to estimate the upper bound of uncertainty and the actuator bias fault, satisfying both optimal performance and chattering reduction of the sliding surface. Furthermore, the actuator effectiveness loss is handled. The stability of the closed-loop system is analytically proven, and the performance of the proposed framework is investigated against several practical operating conditions. This incorporates the fidelity assessment of tracking precision and trackability of control signal using performance indices such as the integral absolute error and root-mean-square error. The results of extensive simulation studies confirm the effectiveness and robustness of the proposed control framework.

A review of the trajectory planning of Industrial robots

With the continuous extension of the application scenarios of industrial robots, industrial robots have made great contributions to increase productivity and automation in the industrial field. And trajectory planning of industrial robots is the key link for its successful completion of tasks. In order to deeply analyze the mechanism of trajectory planning of industrial robots, this paper firstly introduces the principle of trajectory optimization. And then it reviews the relevant theories and methods of trajectory planning. Finally, it summarizes the development trend of trajectory planning of industrial robots.

Dining in danger: Resolving adaptive fish behavior increases realism of modeled ecosystem dynamics.

Animals occupying higher trophic levels can have disproportionately large influence on ecosystem structure and functioning, owning to intricate behavioral responses to their environment, but the effects of behavioral adaptations on aquatic ecosystem dynamics are underrepresented, especially in model studies. Here, we explore how adaptive behavior of fish can affect the dynamics of aquatics ecosystems. We frame fish behavior in the context of the central trade-off between feeding and predation, calculating the optimal level of feeding determined by ambient food availability and predation risk. To explore whole-ecosystem consequences of fish behavior, we embed our behavioral model within the Water Ecosystems Tool (WET), a contemporary end-to-end aquatic ecosystem model. The principle of optimality provides a robust and mechanistic framework for representing animal behavior that is relevant for complex models, and can provide a stabilizing effect on model dynamics. The model predicts an emergent functional response similar to Holling type III, but with richer dynamics and a more rigorous theoretical foundation. We show how adaptive fish behavior works to stabilize food web dynamics compared to a control model with no optimal behavior, and how changing the strength of the underlying trade-off has profound effects on trophic control and food web structure. Furthermore, we demonstrate how including fish behavior allows for an overall more realistic response of the model system to environmental perturbation in the form of nutrient enhancement. We discuss the structuring effects of behavioral adaptations in real ecosystems, and how approaches like this one may benefit aquatic ecological modeling. Our study further highlights how a mechanistic approach based on concepts from theoretical ecology can be successfully implemented in complex operational models resulting in improved dynamics and descriptive power.

Multi-Objective Optimization of the Ultrasonic Scalpel Rod and Tip with Improved Performance: Vibration Frequency, Amplitude, and Service Life

Ultrasonic scalpel design for minimally invasive surgical procedures is mainly focused on optimizing cutting performance. However, an important issue is the low fatigue life of traditional ultrasonic scalpels, which affects their long-term reliability and effectiveness and creates hidden dangers for surgery. In this study, a multi-objective optimal design for the cutting performance and fatigue life of ultrasonic scalpels was proposed using finite element analysis and fatigue simulation. The optimal design parameters of resonance frequency and amplitude were determined. By setting the transition fillet and keeping the gain structure away from the node position to enable the scalpel to have a high service life with excellent cutting performance. The frequency modulation method of setting the vibration node bosses at the node position and setting the vibration antinode grooves at the antinode position was compared. Then, the mechanism of the influence of various design elements, such as tip, shank, node position, and antinode position, on the resonance frequency, amplitude, and fatigue life of the ultrasonic scalpel was analyzed, and the optimal design principles of the ultrasonic scalpel were obtained. The proposed ultrasonic scalpel design was confirmed by simulations, impedance measurements, and liver tissue cutting experiments, demonstrating its feasibility and enhanced performance. This research introduces innovative design strategies to improve the fatigue life and performance of ultrasonic scalpels to address an important issue in minimally invasive surgery.

Kullback-Leibler Control in Boolean Control Networks.

This article addresses the Kullback-Leibler (KL) control problem in Boolean control networks. In the considered problem, an extended stage cost function depending on the control inputs is introduced; in contrast to a stage cost of the conventional KL control problems in the Markov decision process cannot take into consideration the control inputs. An associated Bellman equation and a matrix-based iteration algorithm are presented. The theoretical analysis shows that the proposed KL control results in an approximated form of conventional dynamic programming (DP). Furthermore, the convergence analysis is presented, with the weight parameter converging to zero and diverging to infinity. In practical application examples, a comparison of the conventional DP and proposed KL control is illustrated.

A life insurance model with asymmetric time preferences

We build a life insurance model in the tradition of Richard (1975) and Pliska and Ye (2007). Two agents purchase life insurance by continuously paying two premiums. At the random time of death of an agent, the life insurance payment is added to the household wealth to be used by the other agent. We allow for the agents to discount future utilities at different rates, which implies that the household has inconsistent time preferences. To solve the model, we employ the equilibrium of Ekeland and Lazrak (2010), and we derive a new dynamic programming equation which is designed to find this equilibrium for our model. The most important contribution of the paper is to combine the issue of inconsistent time preferences with the presence of several agents. We also investigate the sensitivity of the behaviors of the agents to the parameters of the model by using numeric analysis. We find, among other things, that while the purchase of life insurance of one agent increases in her own discount rate, it decreases in the discount rate of the other agent.

Developing Optimal Decision Strategies with Markovian Decision Process

The use of Markovian Decision Processes (MDPs) in creating the best possible decision strategies is examined in this research. When outcomes are partly controlled by a decision-maker and partially random, MDPs offer a mathematical framework for simulating decision-making. Finding strategies that optimize cumulative benefits over time while accounting for decision-making's immediate and long-term effects is the main goal. We go over the fundamental ideas of MDPs, such as states, actions, transitions, rewards, and the Bellman equation, which serves as the foundation for figuring out the best course of action. Practical aspects are also looked at, such as computer techniques for solving MDPs and situations in which MDPs work especially well. Decision-makers can systematically examine complicated decision situations by utilizing MDPs, which can result in well-informed and effective decision-making techniques across a variety of areas. Creating methods and algorithms for determining the best policies or decision-making processes within the context of MDPs is another goal. This entails formulating techniques to calculate ideal policies that traverse a series of states and acts in a way that maximizes long-term rewards. The study provides some information about MPs as well as specific conclusions pertaining to martingales. We show the obtained results on a general finite-dimensional filter. This work presents some findings on a general finite dimensional filter, semi-martingale decomposition, Zakai recursion, and related findings about smoothers in different contexts. After deriving the log-likelihood function, it is shown to be both increasing and convergent. Furthermore, the MLE of the parameter has been found.

Observer‐based adaptive neural networks optimal control for spacecraft proximity maneuver with state constraints

AbstractThis article proposes an adaptive neural network (NN) optimal control approach for autonomous relative motion control of non‐cooperative spacecraft in proximity. The proposed method aims to minimize fuel consumption under various challenges including model uncertainty, state constraints, external disturbances, and input saturation. To account for uncertain parameters of non‐cooperative target and external disturbances, we start by designing a NN disturbance observer. Subsequently, a novel optimal control index function is presented. An adaptive NN based on the actor‐critic (A‐C) framework and backstepping theory is then utilized to approximate the solution of Hamilton–Jacobi–Bellman (HJB) equation and obtain an optimal control law. The Lyapunov framework is leveraged to establish the stability of the closed‐loop control system. Finally, numerical simulations are conducted to assess the feasibility and effectiveness of the proposed control scheme in comparison with an existing approach.

Structural modal parameter identification with the Power-Exponential window function

In view of the demand for accurate modal identification, and based on the characteristics of free vibration response, this paper introduces a new window function for Fourier Transform called the Power–Exponential window. The Power–Exponential window addresses the characteristics of free vibration response. It significantly enhances the accuracy of modal identification by improving the spectral properties of structural response. The proposed window function consists of exponential and power terms. This study focuses on the additional damping and frequency-domain differentiation introduced by the Power–Exponential window function. The exponential term weakens the boundary effect related to the time-domain truncation and suppresses the spectral leakage. Moreover, it can be interpreted in clear physical terms as providing additional damping to the signal. The power term in the window function corresponds to frequency domain differentiation, and it alleviates the spectral broadening that arises due to the additional damping. Furthermore, the analytical expression for the response spectrum confirms that the Power–Exponential window not only aligns the peak response frequency with the damped natural frequency but also establishes an explicit linear relationship between the actual structural damping ratio and the identification result from the half power bandwidth method. Both contribute to an improved accuracy and usability of certain frequency-domain modal identification methods. The influence of the Power–Exponential window parameters on modal parameter identification is analyzed, and the optimal selection principle and suggested parameter values are proposed. Finally, numerical simulations and an experimental frame model test are conducted to verify the accuracy and validity of modal parameter identification based on the Power–Exponential window.

Species Richness Net Primary Productivity and the Water Balance Problem.

Species energy theory suggests that, because of limitations on reproduction efficiency, a minimum density of plant individuals per viable species exists and that this minimum correlates the total number of plant individuals N with the number of species S. The simplest assumption is that the mean energy input per individual plant is independent of the number of individuals, making N, and thus S as well, proportional to the total energy input into the system. The primary energy input to a plant-dominated ecosystem is estimated as its Net Primary Productivity (NPP). Thus, species energy theory draws a direct correspondence from NPP to S. Although investigations have verified a strong connection between S and NPP, strong influences of other factors, such as topography, ecological processes such as competition, and historical contingencies, are also at play. The lack of a simple model of NPP expressed in terms of the principal climate variables, precipitation P, and potential evapotranspiration, PET, introduces unnecessary uncertainty to the understanding of species richness across scales. Recent research combines percolation theory with the principle of ecological optimality to derive an expression for NPP(P, PET). Consistent with assuming S is proportional to NPP, we show here that the new expression for NPP(P, PET) predicts the number of plant species S in an ecosystem as a function of P and PET. As already demonstrated elsewhere, the results are consistent with some additional variation due to non-climatic inputs. We suggest that it may be easier to infer specific deviations from species energy predictions with increased accuracy and generality of the prediction of NPP(P, PET).

Multi-stage type-α constrained minimum-cost consensus for linguistic Z-number large-scale decision-making

As an extension of group decision-making in terms of scale and uncertainty, linguistic Z-number large-scale decision-making (LZ-LSDM) is emerging as a prominent research topic in the field of decision science. The unique structure of LZ-LSDM presents new challenges for both clustering analysis and consensus building. Minimum-cost consensus (MCC) based on the optimization principle is widely recognized as an effective tool for managing the consensus-reaching process. However, there is a scarcity of literature that addresses the study of MCC within the context of LZ-LSDM, as well as the application of MCC in the identification and treatment of non-cooperative behaviors. To this end, this study proposes a punishment strategy-driven multi-stage type-α constrained MCC model for LZ-LSDM problems. First, a similarity constraint-based clustering method with linguistic Z-numbers is proposed. Given the clustering results, a type-α constrained MCC (α-CMCC) model with personalized feedback constraints is designed to provide a personalized solution for visualizing opinion adjustment and preventing over-adjustment. Based on the optimal solution obtained by α-CMCC, the identification rule for non-cooperative behaviors is proposed. We conclude three punishment strategies—namely, pure, mixed, and cross—to address non-cooperative behaviors by arranging and combining commonly used punishment approaches. Finally, we illustrate the feasibility and validity of the model through a case study designed to facilitate consensus among an online patient community on knowledge-based recommendations. An exhaustive comparative analysis reveals the advantages and features of the proposed consensus model.

The Impacts of Centralized Control on Mixed Traffic Network Performance: A Strategic Games Analysis

Cooperative Intelligent Transport Systems (C-ITS) address contemporary transportation challenges, as Connected Vehicles (CVs) can play a pivotal role in enhancing efficiency and safety. The role of central governing authorities in shaping traffic management policies for CVs influences decision-making processes and system performance. In this work, the role of central governing authorities in the traffic management of a mixed traffic network is examined, integrating System Optimum principles with game theory. More specifically, we introduce and develop a framework that models and analyses the strategic interactions between different stakeholders in a mixed traffic environment, considering central governing authorities with varying levels of control. The results indicate how the various levels of control of a central governing authority may have an impact on the network in terms of traffic measures. Through a strategic games analysis, the trade-offs associated with centralized control mechanisms are demonstrated and recommendations are offered for policymakers and practitioners to optimize traffic management strategies.

Analysis and optimization of a Stochastic Petri Net for air-rail intermodal transportation.

Air-rail intermodal transportation (ARIT) plays a crucial role in China's intermodal transportation system. This study aims to model and optimize issues such as inefficiency and complexity in China's ARIT freight transportation using Business Process Reengineering and Stochastic Petri Nets theories. The Petri Net (PN) model for incoming freight transport in ARIT is based on actual operations, employing a new method involving Stochastic Petri Nets and isomorphic Markov Chains theory for performance analysis. Performance analysis results help intuitively identify areas needing optimization. Based on optimization principles, elements such as railway container packaging are improved, resulting in an optimized PN model for ARIT. Finally, data analysis shows that the optimized ARIT model reduces total delay by 7.7% compared to the original. This demonstrates that the new method, combining Markov Chain performance analysis and optimization principles, is feasible and effective for ARIT optimization.

The Bellman equation and optimal local flipping strategies for kinetic Ising models

There is a deep connection between thermodynamics, information and work extraction. Ever since the birth of thermodynamics, various types of Maxwell demons have been introduced in order to deepen our understanding of the second law. Thanks to them, it has been shown that there is a deep connection between thermodynamics and information, and between information and work in a thermal system. In this paper we study the problem of energy extraction from a thermodynamic system satisfying detailed balance by an agent with perfect information, e.g. one that has an optimal strategy, given by the solution of the Bellman equation, in the context of Ising models. We call these agents kobolds, in contrast to Maxwell’s demons which do not necessarily need to satisfy detailed balance. This is in stark contrast with typical Monte Carlo algorithms, which choose an action at random at each time step. It is thus natural to compare the behavior of these kobolds to a Metropolis algorithm. For various Ising models, we study numerically and analytically the properties of the optimal strategies, showing that there is a transition in the behavior of the kobold as a function of the parameter characterizing its strategy.

Design-informed generative modelling of skeletal structures using structural optimization

Although various structural optimization techniques have a sound mathematical basis, the structural robustness and practical constructability of optimal designs pose a great challenge in the manufacturing stage. This paper presents an automated novel approach stemming from structural optimization and engineering principles, where discrete members of the structurally optimized designs are driven towards optimal utilization. The developed workflow unifies topology, layout and size optimization in a single parametric platform, which subsequently outputs a ready-to-manufacture CAD skeletal model which can be manufactured either additively or by assembly. All such outputs are checked and validated for structural requirements; strength, stiffness and stability in accordance with standard codes of practice. In the implementations, first, a topology-optimal model is generated and converted to a one-pixel-wide chain model using skeletonization. Herein, this paper uses a novel efficient method to extract the skeleton by using pixel-padding near the domain borders. Secondly, a spatial frame is extracted from the skeleton for its member size and layout optimization. Finally, the CAD model is generated using constructive solid geometry trees and the structural integrity of each member is assessed to ensure structural robustness prior to manufacturing. Various examples presented in the paper showcase the validity of the presented workflow across various structural engineering applications.

Is the output growth rate in NIPA a welfare measure?

Abstract This paper shows that, in a dynamic general equilibrium framework, the rate of output growth as measured by national income and product accounts reflects changes in welfare in the precise sense of equivalent variation. The main argument is straightforward. In a two-sector dynamic general equilibrium model of heterogeneous households, recursive preferences and quasi-concave technology, the Bellman equation provides a representation of household preferences over current consumption and investment. When applied to this representation of preferences, a Fisher–Shell true quantity index turns out to be equal to the Divisia index, closely approximated by the Fisher ideal chain index used in national income and product accounts.