Bellman Equation Research Articles

A learning-based nearly optimal control framework for trajectory tracking of a flexible-link manipulator system with actuator fault

In this paper, a learning-based nearly optimal control framework with fault-tolerant capability is designed to tackle the tracking control problem of a flexible-link manipulator in the presence of actuator fault and model uncertainties. Initially, the optimal control law is obtained by adopting the dynamic programming and a critic structure as the solution of Hamilton–Jacobi–Bellman equation for the nominal model. Then, by implementing an integral sliding mode control, the robustness against actuator fault and model uncertainty is guaranteed. The adaptive laws are constructed based on radial basis functions neural networks to estimate the upper bound of uncertainty and the actuator bias fault, satisfying both optimal performance and chattering reduction of the sliding surface. Furthermore, the actuator effectiveness loss is handled. The stability of the closed-loop system is analytically proven, and the performance of the proposed framework is investigated against several practical operating conditions. This incorporates the fidelity assessment of tracking precision and trackability of control signal using performance indices such as the integral absolute error and root-mean-square error. The results of extensive simulation studies confirm the effectiveness and robustness of the proposed control framework.

A review of the trajectory planning of Industrial robots

With the continuous extension of the application scenarios of industrial robots, industrial robots have made great contributions to increase productivity and automation in the industrial field. And trajectory planning of industrial robots is the key link for its successful completion of tasks. In order to deeply analyze the mechanism of trajectory planning of industrial robots, this paper firstly introduces the principle of trajectory optimization. And then it reviews the relevant theories and methods of trajectory planning. Finally, it summarizes the development trend of trajectory planning of industrial robots.

Dining in danger: Resolving adaptive fish behavior increases realism of modeled ecosystem dynamics.

Animals occupying higher trophic levels can have disproportionately large influence on ecosystem structure and functioning, owning to intricate behavioral responses to their environment, but the effects of behavioral adaptations on aquatic ecosystem dynamics are underrepresented, especially in model studies. Here, we explore how adaptive behavior of fish can affect the dynamics of aquatics ecosystems. We frame fish behavior in the context of the central trade-off between feeding and predation, calculating the optimal level of feeding determined by ambient food availability and predation risk. To explore whole-ecosystem consequences of fish behavior, we embed our behavioral model within the Water Ecosystems Tool (WET), a contemporary end-to-end aquatic ecosystem model. The principle of optimality provides a robust and mechanistic framework for representing animal behavior that is relevant for complex models, and can provide a stabilizing effect on model dynamics. The model predicts an emergent functional response similar to Holling type III, but with richer dynamics and a more rigorous theoretical foundation. We show how adaptive fish behavior works to stabilize food web dynamics compared to a control model with no optimal behavior, and how changing the strength of the underlying trade-off has profound effects on trophic control and food web structure. Furthermore, we demonstrate how including fish behavior allows for an overall more realistic response of the model system to environmental perturbation in the form of nutrient enhancement. We discuss the structuring effects of behavioral adaptations in real ecosystems, and how approaches like this one may benefit aquatic ecological modeling. Our study further highlights how a mechanistic approach based on concepts from theoretical ecology can be successfully implemented in complex operational models resulting in improved dynamics and descriptive power.

Multi-Objective Optimization of the Ultrasonic Scalpel Rod and Tip with Improved Performance: Vibration Frequency, Amplitude, and Service Life

Ultrasonic scalpel design for minimally invasive surgical procedures is mainly focused on optimizing cutting performance. However, an important issue is the low fatigue life of traditional ultrasonic scalpels, which affects their long-term reliability and effectiveness and creates hidden dangers for surgery. In this study, a multi-objective optimal design for the cutting performance and fatigue life of ultrasonic scalpels was proposed using finite element analysis and fatigue simulation. The optimal design parameters of resonance frequency and amplitude were determined. By setting the transition fillet and keeping the gain structure away from the node position to enable the scalpel to have a high service life with excellent cutting performance. The frequency modulation method of setting the vibration node bosses at the node position and setting the vibration antinode grooves at the antinode position was compared. Then, the mechanism of the influence of various design elements, such as tip, shank, node position, and antinode position, on the resonance frequency, amplitude, and fatigue life of the ultrasonic scalpel was analyzed, and the optimal design principles of the ultrasonic scalpel were obtained. The proposed ultrasonic scalpel design was confirmed by simulations, impedance measurements, and liver tissue cutting experiments, demonstrating its feasibility and enhanced performance. This research introduces innovative design strategies to improve the fatigue life and performance of ultrasonic scalpels to address an important issue in minimally invasive surgery.

Kullback-Leibler Control in Boolean Control Networks.

This article addresses the Kullback-Leibler (KL) control problem in Boolean control networks. In the considered problem, an extended stage cost function depending on the control inputs is introduced; in contrast to a stage cost of the conventional KL control problems in the Markov decision process cannot take into consideration the control inputs. An associated Bellman equation and a matrix-based iteration algorithm are presented. The theoretical analysis shows that the proposed KL control results in an approximated form of conventional dynamic programming (DP). Furthermore, the convergence analysis is presented, with the weight parameter converging to zero and diverging to infinity. In practical application examples, a comparison of the conventional DP and proposed KL control is illustrated.

Developing Optimal Decision Strategies with Markovian Decision Process

The use of Markovian Decision Processes (MDPs) in creating the best possible decision strategies is examined in this research. When outcomes are partly controlled by a decision-maker and partially random, MDPs offer a mathematical framework for simulating decision-making. Finding strategies that optimize cumulative benefits over time while accounting for decision-making's immediate and long-term effects is the main goal. We go over the fundamental ideas of MDPs, such as states, actions, transitions, rewards, and the Bellman equation, which serves as the foundation for figuring out the best course of action. Practical aspects are also looked at, such as computer techniques for solving MDPs and situations in which MDPs work especially well. Decision-makers can systematically examine complicated decision situations by utilizing MDPs, which can result in well-informed and effective decision-making techniques across a variety of areas. Creating methods and algorithms for determining the best policies or decision-making processes within the context of MDPs is another goal. This entails formulating techniques to calculate ideal policies that traverse a series of states and acts in a way that maximizes long-term rewards. The study provides some information about MPs as well as specific conclusions pertaining to martingales. We show the obtained results on a general finite-dimensional filter. This work presents some findings on a general finite dimensional filter, semi-martingale decomposition, Zakai recursion, and related findings about smoothers in different contexts. After deriving the log-likelihood function, it is shown to be both increasing and convergent. Furthermore, the MLE of the parameter has been found.

Observer‐based adaptive neural networks optimal control for spacecraft proximity maneuver with state constraints

AbstractThis article proposes an adaptive neural network (NN) optimal control approach for autonomous relative motion control of non‐cooperative spacecraft in proximity. The proposed method aims to minimize fuel consumption under various challenges including model uncertainty, state constraints, external disturbances, and input saturation. To account for uncertain parameters of non‐cooperative target and external disturbances, we start by designing a NN disturbance observer. Subsequently, a novel optimal control index function is presented. An adaptive NN based on the actor‐critic (A‐C) framework and backstepping theory is then utilized to approximate the solution of Hamilton–Jacobi–Bellman (HJB) equation and obtain an optimal control law. The Lyapunov framework is leveraged to establish the stability of the closed‐loop control system. Finally, numerical simulations are conducted to assess the feasibility and effectiveness of the proposed control scheme in comparison with an existing approach.

Structural modal parameter identification with the Power-Exponential window function

In view of the demand for accurate modal identification, and based on the characteristics of free vibration response, this paper introduces a new window function for Fourier Transform called the Power–Exponential window. The Power–Exponential window addresses the characteristics of free vibration response. It significantly enhances the accuracy of modal identification by improving the spectral properties of structural response. The proposed window function consists of exponential and power terms. This study focuses on the additional damping and frequency-domain differentiation introduced by the Power–Exponential window function. The exponential term weakens the boundary effect related to the time-domain truncation and suppresses the spectral leakage. Moreover, it can be interpreted in clear physical terms as providing additional damping to the signal. The power term in the window function corresponds to frequency domain differentiation, and it alleviates the spectral broadening that arises due to the additional damping. Furthermore, the analytical expression for the response spectrum confirms that the Power–Exponential window not only aligns the peak response frequency with the damped natural frequency but also establishes an explicit linear relationship between the actual structural damping ratio and the identification result from the half power bandwidth method. Both contribute to an improved accuracy and usability of certain frequency-domain modal identification methods. The influence of the Power–Exponential window parameters on modal parameter identification is analyzed, and the optimal selection principle and suggested parameter values are proposed. Finally, numerical simulations and an experimental frame model test are conducted to verify the accuracy and validity of modal parameter identification based on the Power–Exponential window.

Species Richness Net Primary Productivity and the Water Balance Problem.

Species energy theory suggests that, because of limitations on reproduction efficiency, a minimum density of plant individuals per viable species exists and that this minimum correlates the total number of plant individuals N with the number of species S. The simplest assumption is that the mean energy input per individual plant is independent of the number of individuals, making N, and thus S as well, proportional to the total energy input into the system. The primary energy input to a plant-dominated ecosystem is estimated as its Net Primary Productivity (NPP). Thus, species energy theory draws a direct correspondence from NPP to S. Although investigations have verified a strong connection between S and NPP, strong influences of other factors, such as topography, ecological processes such as competition, and historical contingencies, are also at play. The lack of a simple model of NPP expressed in terms of the principal climate variables, precipitation P, and potential evapotranspiration, PET, introduces unnecessary uncertainty to the understanding of species richness across scales. Recent research combines percolation theory with the principle of ecological optimality to derive an expression for NPP(P, PET). Consistent with assuming S is proportional to NPP, we show here that the new expression for NPP(P, PET) predicts the number of plant species S in an ecosystem as a function of P and PET. As already demonstrated elsewhere, the results are consistent with some additional variation due to non-climatic inputs. We suggest that it may be easier to infer specific deviations from species energy predictions with increased accuracy and generality of the prediction of NPP(P, PET).

Multi-stage type-α constrained minimum-cost consensus for linguistic Z-number large-scale decision-making

As an extension of group decision-making in terms of scale and uncertainty, linguistic Z-number large-scale decision-making (LZ-LSDM) is emerging as a prominent research topic in the field of decision science. The unique structure of LZ-LSDM presents new challenges for both clustering analysis and consensus building. Minimum-cost consensus (MCC) based on the optimization principle is widely recognized as an effective tool for managing the consensus-reaching process. However, there is a scarcity of literature that addresses the study of MCC within the context of LZ-LSDM, as well as the application of MCC in the identification and treatment of non-cooperative behaviors. To this end, this study proposes a punishment strategy-driven multi-stage type-α constrained MCC model for LZ-LSDM problems. First, a similarity constraint-based clustering method with linguistic Z-numbers is proposed. Given the clustering results, a type-α constrained MCC (α-CMCC) model with personalized feedback constraints is designed to provide a personalized solution for visualizing opinion adjustment and preventing over-adjustment. Based on the optimal solution obtained by α-CMCC, the identification rule for non-cooperative behaviors is proposed. We conclude three punishment strategies—namely, pure, mixed, and cross—to address non-cooperative behaviors by arranging and combining commonly used punishment approaches. Finally, we illustrate the feasibility and validity of the model through a case study designed to facilitate consensus among an online patient community on knowledge-based recommendations. An exhaustive comparative analysis reveals the advantages and features of the proposed consensus model.

The Impacts of Centralized Control on Mixed Traffic Network Performance: A Strategic Games Analysis

Cooperative Intelligent Transport Systems (C-ITS) address contemporary transportation challenges, as Connected Vehicles (CVs) can play a pivotal role in enhancing efficiency and safety. The role of central governing authorities in shaping traffic management policies for CVs influences decision-making processes and system performance. In this work, the role of central governing authorities in the traffic management of a mixed traffic network is examined, integrating System Optimum principles with game theory. More specifically, we introduce and develop a framework that models and analyses the strategic interactions between different stakeholders in a mixed traffic environment, considering central governing authorities with varying levels of control. The results indicate how the various levels of control of a central governing authority may have an impact on the network in terms of traffic measures. Through a strategic games analysis, the trade-offs associated with centralized control mechanisms are demonstrated and recommendations are offered for policymakers and practitioners to optimize traffic management strategies.

Analysis and optimization of a Stochastic Petri Net for air-rail intermodal transportation.

Air-rail intermodal transportation (ARIT) plays a crucial role in China's intermodal transportation system. This study aims to model and optimize issues such as inefficiency and complexity in China's ARIT freight transportation using Business Process Reengineering and Stochastic Petri Nets theories. The Petri Net (PN) model for incoming freight transport in ARIT is based on actual operations, employing a new method involving Stochastic Petri Nets and isomorphic Markov Chains theory for performance analysis. Performance analysis results help intuitively identify areas needing optimization. Based on optimization principles, elements such as railway container packaging are improved, resulting in an optimized PN model for ARIT. Finally, data analysis shows that the optimized ARIT model reduces total delay by 7.7% compared to the original. This demonstrates that the new method, combining Markov Chain performance analysis and optimization principles, is feasible and effective for ARIT optimization.

The Bellman equation and optimal local flipping strategies for kinetic Ising models

There is a deep connection between thermodynamics, information and work extraction. Ever since the birth of thermodynamics, various types of Maxwell demons have been introduced in order to deepen our understanding of the second law. Thanks to them, it has been shown that there is a deep connection between thermodynamics and information, and between information and work in a thermal system. In this paper we study the problem of energy extraction from a thermodynamic system satisfying detailed balance by an agent with perfect information, e.g. one that has an optimal strategy, given by the solution of the Bellman equation, in the context of Ising models. We call these agents kobolds, in contrast to Maxwell’s demons which do not necessarily need to satisfy detailed balance. This is in stark contrast with typical Monte Carlo algorithms, which choose an action at random at each time step. It is thus natural to compare the behavior of these kobolds to a Metropolis algorithm. For various Ising models, we study numerically and analytically the properties of the optimal strategies, showing that there is a transition in the behavior of the kobold as a function of the parameter characterizing its strategy.

Design-informed generative modelling of skeletal structures using structural optimization

Although various structural optimization techniques have a sound mathematical basis, the structural robustness and practical constructability of optimal designs pose a great challenge in the manufacturing stage. This paper presents an automated novel approach stemming from structural optimization and engineering principles, where discrete members of the structurally optimized designs are driven towards optimal utilization. The developed workflow unifies topology, layout and size optimization in a single parametric platform, which subsequently outputs a ready-to-manufacture CAD skeletal model which can be manufactured either additively or by assembly. All such outputs are checked and validated for structural requirements; strength, stiffness and stability in accordance with standard codes of practice. In the implementations, first, a topology-optimal model is generated and converted to a one-pixel-wide chain model using skeletonization. Herein, this paper uses a novel efficient method to extract the skeleton by using pixel-padding near the domain borders. Secondly, a spatial frame is extracted from the skeleton for its member size and layout optimization. Finally, the CAD model is generated using constructive solid geometry trees and the structural integrity of each member is assessed to ensure structural robustness prior to manufacturing. Various examples presented in the paper showcase the validity of the presented workflow across various structural engineering applications.

Is the output growth rate in NIPA a welfare measure?

Abstract This paper shows that, in a dynamic general equilibrium framework, the rate of output growth as measured by national income and product accounts reflects changes in welfare in the precise sense of equivalent variation. The main argument is straightforward. In a two-sector dynamic general equilibrium model of heterogeneous households, recursive preferences and quasi-concave technology, the Bellman equation provides a representation of household preferences over current consumption and investment. When applied to this representation of preferences, a Fisher–Shell true quantity index turns out to be equal to the Divisia index, closely approximated by the Fisher ideal chain index used in national income and product accounts.

Surgical Strategies for Renal Transplantation: A Pictorial Essay.

This pictorial essay aims to navigate through the complexities and challenges of renal transplantation (RT), by weaving together visual imagery with clinical insights within a comprehensive illustrative surgical guide. Herein, we provide a detailed visual exploration of the intricate anatomy and surgical processes necessary for both renal graft retrieval from the donor and also for an adequate implantation in the recipient. Regarding graft retrieval, after reviewing the relevant retroperitoneal surgical anatomy, and donor nephrectomy techniques, graft preservation and optimal backbench graft dissection principles were meticulously analyzed. Thereafter, the recipient surgical strategy for graft implantation was addressed, focusing on preoperative preparations, the site of implantation selection, exposure, operative bed dissection, graft revascularization, and urinary tract reconstruction. Careful donor and recipient selection, meticulous surgical execution, and rigorous postoperative management clearly hold a pivotal role in optimizing patient outcomes. Fostering a deeper understanding of the surgical nuances and clinical management practices that contribute to successful results post-RT, we hope to provide a useful practical tool for clinicians about to embark on the treacherous road of RT surgery. Innovative technologies and surgical practices that have already significantly improved the safety and effectiveness of RT stand testament to the importance of further scientific inquiry, conceptual developments, and clinical integration. Moving forward, it is essential that the medical community continues to refine these strategies and advocate for equitable access to transplantation, ensuring that advancements in the field translate into real-world benefits for all patients grappling with ESRD. The collaborative efforts of multidisciplinary teams are essential in addressing the complex clinical challenges associated with RT, with the ultimate goal of improving patient survival, enhancing graft longevity, and reducing healthcare disparities.

Principles of Creating Multi-objective Quality Models for Software Systems

This article proposes an original approach at a mathematical level to the creation of multi- objective quality models for software systems. The research is based on the study and generalization of quality modeling trends of software systems and user needs in order to determine optimal principles for constructing such models. The article provides mathematical explanations that play a key role in identifying and formalizing the principles of creating multi-objective quality models of software. An important aspect is the consideration of quality model construction principles at the mathematical level, allowing for a more precise assessment and analysis of various aspects of software quality. The research results indicate that incorporating mathematical principles into the creation of multi-objective quality models for software systems can have a significant practical impact. It is established that on a practical level of developing multi-objective quality models for software systems, consideration of the principles of creating multi-objective quality models can have numerous practical implications. Specifically, the application of metrics within established principles allows for a comprehensive view of software quality and identifies areas requiring attention and improvement. This helps developers and software quality engineers make informed decisions regarding system improvement and optimization. The research has shown that creating high-quality models of quality requires attention to various aspects, from user needs to testing and continuous improvement, as well as the use of mathematical methods for their formalization and analysis. The developed principles of creating multi-objective quality models at the level of mathematical models allow the use of these models to assess and analyze various aspects of software quality, representing each model using a corresponding function that determines the relationship between quality metrics and the quality of the software itself. It is expected that further development and implementation of these principles will contribute to improving software development processes and ensure high quality of the resulting software.

A collaborative neurodynamic approach with two-timescale projection neural networks designed via majorization-minimization for global optimization and distributed global optimization

In this paper, two two-timescale projection neural networks are proposed based on the majorization-minimization principle for nonconvex optimization and distributed nonconvex optimization. They are proved to be globally convergent to Karush–Kuhn–Tucker points. A collaborative neurodynamic approach leverages multiple two-timescale projection neural networks repeatedly re-initialized using a meta-heuristic rule for global optimization and distributed global optimization. Two numerical examples are elaborated to demonstrate the efficacy of the proposed approaches.

Dynamic event-based cooperative control of virtually coupled high-speed trains: an ADP-based optimal control scheme

This paper is concerned with the cooperative control issue of virtually coupled high-speed trains (VCHSTs) under an adaptive dynamic programming (ADP) framework, where a dynamic event-triggered mechanism is employed to govern the communication between sensors and observers. A learning-based observer, equipped with a robust term, is exploited to mitigate the effect of the unknown nonlinear dynamics of the VCHSTs. Furthermore, to realise the distributed design, an auxiliary control signal is employed to simply the closed-loop system and also integrated into the cost function to handle the coupled nature of the investigated systems. The Hamilton–Jacobi–Bellman (HJB) equation is approximated by actor and critic networks in the ADP framework, which iteratively acquires the desired control strategy. By resorting to the common Lyapunov stability, some sufficient conditions are derived to achieve the desired observer gain and learning rates of NNs. Ultimately, the control scheme's efficiency is confirmed by simulations of VCHSTs.

Data-driven-based sliding-mode dynamic event-triggered control of unknown nonlinear systems via reinforcement learning

This paper investigates a data-driven-based dynamic event-triggered control problem for continuous-time unknown nonlinear systems using the reinforcement learning method and the sliding-mode surface technique. Initially, by constructing a cost function associated with sliding-mode surface variables for the nominal system, the original control problem is equivalently transformed into a problem of designing a dynamic event-triggered optimal control policy. To handle the unknown issue of system dynamics, a data-driven model is established to reconstruct the system dynamics. Then, under the framework of reinforcement learning, a critic network is employed to solve the event-triggered Hamilton–Jacobi–Bellman equation. The weight vector in the critic network is updated through the current data and historical data, such that the persistence of excitation condition is no longer needed. After that, it is strictly proven via Lyapunov stability theory that all the signals of the considered system are bounded in the sense of uniformly ultimately boundedness. Finally, the effectiveness of the developed control method is demonstrated by two simulation examples.