Value Function Research Articles

Robust Design Optimization of Electric Machines with Isogeometric Analysis

In electric machine design, efficient methods for the optimization of the geometry and associated parameters are essential. Nowadays, it is necessary to address the uncertainty caused by manufacturing or material tolerances. This work presents a robust optimization strategy to address uncertainty in the design of a three-phase, six-pole permanent magnet synchronous motor (PMSM). The geometry is constructed in a two-dimensional framework within MATLAB®, employing isogeometric analysis (IGA) to enable flexible shape optimization. The main contributions of this research are twofold. First, we integrate shape optimization with parameter optimization to enhance the performance of PMSM designs. Second, we use robust optimization, which creates a min–max problem, to ensure that the motor maintains its performance when facing uncertainties. To solve this bilevel problem, we work with the maximal value functions of the lower-level maximization problems and apply a version of Danskin’s theorem for the computation of generalized derivatives. Additionally, the adjoint method is employed to efficiently solve the lower-level problems with gradient-based optimization. The paper concludes by presenting numerical results showcasing the efficacy of the proposed robust optimization framework. The results indicate that the optimized PMSM designs not only perform competitively compared to their non-robust counterparts but also show resilience to operational and manufacturing uncertainties, making them attractive for industrial applications.

An interactive method based on multi-objective optimization for limited-angle CT reconstruction

Objective. Limited-angle x-ray computed tomography (CT) is a typical ill-posed inverse problem, leading to artifacts in the reconstructed image due to the incomplete projection data. Most iteration CT reconstruction methods involve optimization for a single object. This paper explores a multi-objective optimization model and an interactive method based on multi-objective optimization to suppress the artifacts of limited-angle CT. Approach. The model includes two objective functions on the dual domain within the data consistency constraint. In the interactive method, the structural similarity index measure (SSIM) is regarded as the value function of the decision maker (DM) firstly. Secondly, the DM arranges the objective functions of the multi-objective optimization model to be optimized according to their absolute importance. Finally, the SSIM and the simulated annealing (SA) method help the DM choose the desirable reconstruction image by improving the SSIM value during the iteration process. Main results. Simulation and real data experiments demonstrate that the artifacts can be suppressed by the proposed method, and the results were superior to those reconstructed by the other three reconstruction methods in preserving the edge structure of the image. Significance. The proposed interactive method based on multi-objective optimization shows some potential advantages over classical single object optimization methods.

An Overestimation Reduction Method Based on the Multi-step Weighted Double Estimation Using Value-Decomposition Multi-agent Reinforcement Learning

The joint action-value function (JAVF) plays a key role in the centralized training of multi-agent deep reinforcement learning (MADRL)-based algorithms using the value function decomposition (VFD) and in the generating process of a collaborative policy between agents. However, under the influence of multiple factors such as environmental noise, inadequate exploration and iterative updating mechanism, estimation bias is inevitably introduced, causing its overestimation problem, which in turn prevents agents from obtaining accurate reward signals during the learning process, and fails to correctly approximate the optimal policy. To address this problem, this paper first analyzes the causes of joint action-value function overestimation, gives the corresponding mathematical proofs and theoretical derivations, and obtains the lower bound of the overestimation error; then, a MADRL overestimation reduction method based on the multi-step weighted double estimation named λWD QMIX is proposed. Specifically, the λWD QMIX method effectively achieves more stable and accurate JAVF estimation results using the bias correction estimation mechanisms based on the weighted double estimation and multi-step updating based on eligibility trace backup, without additionally adding or changing any network structure. The results of a series of experiments on the StarCraft II micromanipulation benchmark show that the proposed λWD QMIX algorithm can effectively improve the final performance and learning efficiency of the baseline algorithm, and can be seamlessly integrated with the partially MADRL algorithms based on communication learning.

Mechanism design for public projects via three machine learning based approaches

We study mechanism design for nonexcludable and excludable binary public project problems. Our aim is to maximize the expected number of consumers and the expected agents’ welfare. We first show that for the nonexcludable public project model, there is no need for machine learning based mechanism design. We identify a sufficient condition on the prior distribution for the existing conservative equal costs mechanism to be the optimal strategy-proof and individually rational mechanism. For general distributions, we propose a dynamic program that solves for the optimal mechanism. For the excludable public project model, we identify a similar sufficient condition for the existing serial cost sharing mechanism to be optimal for 2 and 3 agents. We derive a numerical upper bound and use it to show that for several common distributions, the serial cost sharing mechanism is close to optimality. The serial cost sharing mechanism is not optimal in general. We propose three machine learning based approaches for designing better performing mechanisms. We focus on the family of largest unanimous mechanisms, which characterizes all strategy-proof and individually rational mechanisms for the excludable public project model. A largest unanimous mechanism describes an iterative mechanism, which is defined by an exponential number of mechanism parameters. Our first approach describes the largest unanimous mechanism family using a neural network and training is carried out by minimizing a cost function that combines the mechanism design objective and the constraint violation penalty. We interpret the largest unanimous mechanisms as price-oriented rationing-free (PORF) mechanisms, which enables us to move the mechanisms’ iterative decision making off the neural network, to a separate simulation process, therefore avoiding the vanishing gradient problem. We also feed the prior distribution’s analytical form into the cost function to achieve high-quality gradients for efficient training. Our second approach treats the mechanism design task as a Markov Decision Process with an exponential number of states. During the Markov decision process, the non-consumers are gradually removed from the system. We train multiple neural networks, each for a different number of remaining agents, to learn the optimal value function on the states. Training is carried out by supervised learning toward a set of manually prepared base cases and the Bellman equation. Our third approach is based on reinforcement learning for a Partially Observable Markov Decision Process. Each RL episode randomly draws a type profile, which is hidden from the RL agent (mechanism designer). The RL agent only observes which cost share offers have been accepted under the largest unanimous mechanism under discussion. We use a continuous action space reinforcement learning approach to adjust the offer policy (i.e., adjust mechanism parameters). Lastly, our first two approaches use “supervision to manual mechanisms” as a systematic way for network initialization, which is potentially valuable for machine learning based mechanism design in general.

A novel method-based reinforcement learning with deep temporal difference network for flexible double shop scheduling problem

This paper studies the flexible double shop scheduling problem (FDSSP) that considers simultaneously job shop and assembly shop. It brings about the problem of scheduling association of the related tasks. To this end, a reinforcement learning algorithm with a deep temporal difference network is proposed to minimize the makespan. Firstly, the FDSSP is defined as the mathematical model of the flexible job-shop scheduling problem joined to the assembly constraint level. It is translated into a Markov decision process that directly selects behavioral strategies according to historical machining state data. Secondly, the proposed ten generic state features are input into the deep neural network model to fit the state value function. Similarly, eight simple constructive heuristics are used as candidate actions for scheduling decisions. From the greedy mechanism, optimally combined actions of all machines are obtained for each decision step. Finally, a deep temporal difference reinforcement learning framework is established, and a large number of comparative experiments are designed to analyze the basic performance of this algorithm. The results showed that the proposed algorithm was better than most other methods, which contributed to solving the practical production problem of the manufacturing industry.

Reinforcement learning-based moving-target enclosing control for an unmanned surface vehicle in multi-obstacle environments

This paper investigates the moving-target enclosing problem in multi-obstacle environments for an unmanned surface vehicle (USV) with complex unknown factors, including target dynamics, vehicle dynamics, and disturbances. A reinforcement learning-based moving-target enclosing control scheme is proposed to ensure collision-free behavior and bolster the enclosing capability. Specifically, an extended state observer is deployed to estimate the target dynamics. Leveraging the estimated data and control obstacle functions, a virtual safety control law is formulated to dynamically harmonize obstacle avoidance and target enclosing control. Then, a novel controller is constructed to track this control law utilizing a Nussbaum-type function in conjunction with actor–critic neural networks (NNs). The actor and critic NNs are employed to approximate unknown dynamics encapsulating vehicle dynamics and disturbances, and value function, respectively. The Nussbaum-type function is embedded to adaptively identify unknown inertia mass, regulating the control input of the USV with the actor NN online. The proposed scheme effectively decouples obstacle avoidance and target enclosing control and only relies on the measurable variables. A rigorous theoretical analysis is further employed to ensure the closed-loop stability of the USV system. Eventually, Simulations are demonstrated to validate the effectiveness and superiority of the proposed scheme for the USV in multi-obstacle environments.

Multi-omics and chemical profiling approaches to understand the material foundation and pharmacological mechanism of sophorae tonkinensis radix et rhizome-induced liver injury in mice

Ethnopharmacological relevanceSophorae tonkinensis Radix et Rhizoma (STR) is an extensively applied traditional Chinese medicine (TCM) in southwest China. However, its clinical application is relatively limited due to its hepatotoxicity effects. Aim of the studyTo understand the material foundation and liver injury mechanism of STR. Materials and methodsChemical compositions in STR and its prototypes in mice were profiled by ultra-performance liquid chromatography coupled quadrupole-time of flight mass spectrometry (UPLC-Q/TOF MS). STR-induced liver injury (SILI) was comprehensively evaluated by STR-treated mice mode. The histopathologic and biochemical analyses were performed to evaluate liver injury levels. Subsequently, network pharmacology and multi-omics were used to analyze the potential mechanism of SILI in vivo. And the target genes were further verified by Western blot. ResultsA total of 152 compounds were identified or tentatively characterized in STR, including 29 alkaloids, 21 organic acids, 75 flavonoids, 1 quinone, and 26 other types. Among them, 19 components were presented in STR-medicated serum. The histopathologic and biochemical analysis revealed that hepatic injury occurred after 4 weeks of intragastric administration of STR. Network pharmacology analysis revealed that IL6, TNF, STAT3, etc. were the main core targets, and the bile secretion might play a key role in SILI. The metabolic pathways such as taurine and hypotaurine metabolism, purine metabolism, and vitamin B6 metabolism were identified in the STR exposed groups. Among them, taurine, hypotaurine, hypoxanthine, pyridoxal, and 4-pyridoxate were selected based on their high impact value and potential biological function in the process of liver injury post STR treatment. ConclusionsThe mechanism and material foundation of SILI were revealed and profiled by a multi-omics strategy combined with network pharmacology and chemical profiling. Meanwhile, new insights were taken into understand the pathological mechanism of SILI.

Optimal investment and reinsurance to reach a bequest goal with random time solvency regulation

ABSTRACT This paper studies optimal investment and proportional reinsurance policies for an insurer with Markov regime-switching model and random time solvency regulation. The goal of the insurer is to maximise the probability that its wealth exceeds a predetermined level l before the regulatory time arrives. By constructing an auxiliary control problem without regime switching, together with the classical results on Hamilton-Jacobi-Bellman (HJB) equation and fixed-point method, we prove the regularity of the value function. When the current state of the Markov chain is given, we found that the optimal policies for an insurer in a multiple-regime market is the same as those in a single-regime market. Explicit optimal policies can be derived when the premium is calculated by the expectation principle. For more general cases, numerical schemes for value functions and feedback optimal policies are given by the Markov chain approximating method.

Robust asset-liability management games for n players under multivariate stochastic covariance models

This paper investigates a non-zero-sum stochastic differential game among n competitive CARA asset-liability managers, who are concerned about the potential model ambiguity and aim to seek the robust investment strategies. The ambiguity-averse managers are subject to uncontrollable and idiosyncratic random liabilities driven by generalized drifted Brownian motions and have access to an incomplete financial market consisting of a risk-free asset, a market index and a stock under a multivariate stochastic covariance model. The market dynamics permit not only stochastic correlation between the risky assets but also path-dependent and time-varying risk premium and volatility, depending on two affine-diffusion factor processes. The objective of each manager is to maximize the expected exponential utility of his terminal surplus relative to the average among his competitors under the worst-case scenario of the alternative measures. We manage to solve this robust non-Markovian stochastic differential game by using a backward stochastic differential equation approach. Explicit expressions for the robust Nash equilibrium investment policies, the density generator processes under the well-defined worst-case probability measures and the corresponding value functions are derived. Conditions for the admissibility of the robust equilibrium strategies are provided. Finally, we perform some numerical examples to illustrate the influence of model parameters on the equilibrium investment strategies and draw some economic interpretations from these results.

A flow‐based ascending auction to compute buyer‐optimal Walrasian prices

AbstractWe consider a market where a set of objects is sold to a set of buyers, each equipped with a valuation function for the objects. The goal of the auctioneer is to determine reasonable prices together with a stable allocation. One definition of “reasonable” and “stable” is a Walrasian equilibrium, which is a tuple consisting of a price vector together with an allocation satisfying the following desirable properties: (i) the allocation is market‐clearing in the sense that as much as possible is sold, and (ii) the allocation is stable in the sense that every buyer ends up with an optimal set with respect to the given prices. Moreover, “buyer‐optimal” means that the prices are smallest possible among all Walrasian prices. In this paper, we present a combinatorial network flow algorithm to compute buyer‐optimal Walrasian prices in a multi‐unit matching market with truncated additive valuation functions. The algorithm can be seen as a generalization of the classical housing market auction and mimics the very natural procedure of an ascending auction. We use our structural insights to prove monotonicity of the buyer‐optimal Walrasian prices with respect to changes in supply or demand.

Demand response for residential building heating: Effective Monte Carlo Tree Search control based on physics-informed neural networks

To reduce global carbon emissions and limit climate change, controlling energy consumption in buildings is an important piece of the puzzle. Here, we specifically focus on using a demand response (DR) algorithm to limit the energy consumption of a residential building's heating system while respecting user's thermal comfort. In that domain, Reinforcement learning (RL) methods have been shown to be quite effective. One such RL method is Monte Carlo Tree Search (MCTS), which has achieved impressive success in playing board games (go, chess). A particular advantage of MCTS is that its decision tree structure naturally allows to integrate exogenous constraints (e.g., by trimming branches that violate them), while conventional RL solutions need more elaborate techniques (e.g., indirectly by adding penalties in the cost/reward function, or through a backup controller that corrects constraint-violating actions). The main aim of this paper is to study the adoption of MCTS for building control, since this (to the best of our knowledge) has remained largely unexplored. A specific property of MCTS is that it needs a simulator component that can predict subsequent system states, based on actions taken. A straightforward data-driven solution is to use black-box neural networks (NNs). We will however extend a Physics-informed Neural Network (PiNN) model to deliver multi-timestep predictions, and show the benefit it offers in terms of lower prediction errors (−32% MAE) as well as better MCTS performance (−4% energy cost, +7% thermal comfort) compared to a black-box NN. A second contribution will be to extend a vanilla MCTS version to adopt the ideas applied in AlphaZero (i.e., using learned prior and value functions and an action selection heuristic) to obtain lower computational costs while maintaining control performance.

A comparison principle for semilinear Hamilton–Jacobi–Bellman equations in the Wasserstein space

The goal of this paper is to prove a comparison principle for viscosity solutions of semilinear Hamilton–Jacobi equations in the space of probability measures. The method involves leveraging differentiability properties of the 2-Wasserstein distance in the doubling of variables argument, which is done by introducing a further entropy penalization that ensures that the relevant optima are achieved at positive, Lipschitz continuous densities with finite Fischer information. This allows to prove uniqueness and stability of viscosity solutions in the class of bounded Lipschitz continuous (with respect to the 1-Wasserstein distance) functions. The result does not appeal to a mean field control formulation of the equation, and, as such, applies to equations with nonconvex Hamiltonians and measure-dependent volatility. For convex Hamiltonians that derive from a potential, we prove that the value function associated with a suitable mean-field optimal control problem with nondegenerate idiosyncratic noise is indeed the unique viscosity solution.

What Makes a Business Model Sustainable? Activities, Design Themes, and Value Functions

What makes a business model sustainable? To answer this question, we conducted a systematic review of 390 journal articles on business models for sustainability (BMfS). Building on the activity system perspective, we engaged in an active categorization process, in which we identified 26 activity groups focused on 12 design themes. These activities and design themes are associated with organizations’ potential to use their business models to contribute to sustainable value creation. Our analysis also revealed that the identified activities and design themes can be related to three overarching value functions of BMfS: maintaining, unlocking, and sharing value. Our findings indicate that these value functions play a pivotal role in creating sustainable value through business models. The identified design themes can serve as guiding principles for organizations seeking to make their business models sustainable, while the identified value functions can provide a foundation for theorizing on sustainable value creation through business models.

Accelerating value function approximations for dynamic dial-a-ride problems via dimensionality reductions

As the success of ride-sharing mobility service providers shows, customer demand for shared mobility services is increasing. The availability of mobile devices enables the constant accessibility of mobility apps and the immediate placement of transport requests. To provide such a dynamic dial-a-ride service, an effective control of the fleet is necessary. One promising solution approach is the value function approximation (VFA), which on the one hand convinces through good performance, but on the other hand also stands out through fast response times for a request. Training a VFA can be a challenging task since, among other things, the dimensionality of the state space plays a decisive role. If many variables to describe a state are used, a high amount of information can produce good performance after completion of the learning process. If the state space is too high-dimensional, there is also a risk that the method will not be able to find a reasonable solution. In contrast, if the number of variables is reduced, the learning speed can be accelerated, but the eventual performance may suffer from the associated loss of information. Furthermore, not all variables are equally relevant, as they contain different amounts of information. This paper presents a hybrid strategy, temporarily lowering the dimensionality of the problem using dimension reduction methods and subsequently increasing it by mapping the lower-dimensional state representations back onto a high-dimensional state space in order to exploit the advantages of both space dimensionalities. VFA in itself results in competitive performance for the dynamic dial-a-ride problem with shared rides. The proposed hybrid state representation can outperform the reference state representations by 3%, which corresponds to a meaningful acceleration in VFA learning speed.

Prediction Horizon-Varying Model Predictive Control (MPC) for Autonomous Vehicle Control

The prediction horizon is a key parameter in model predictive control (MPC), which is related to the effectiveness and stability of model predictive control. In vehicle control, the selection of a prediction horizon is influenced by factors such as speed, path curvature, and target point density. To accommodate varying conditions such as road curvature and vehicle speed, we proposed a control strategy using the proximal policy optimization (PPO) algorithm to adjust the prediction horizon, enabling MPC to achieve optimal performance, and called it PPO-MPC. We established a state space related to the path information and vehicle state, regarded the prediction horizon as actions, and designed a reward function to optimize the policy and value function. We conducted simulation verifications at various speeds and compared them with an MPC with fixed prediction horizons. The simulation demonstrates that the PPO-MPC proposed in this article exhibits strong adaptability and trajectory tracking capability.

Pythagorean Z-linguistic VIKOR method based on modified value function under Pythagorean Z-linguistic environment and its application on smart healthcare

Pythagorean Z-linguistic VIKOR method based on modified value function under Pythagorean Z-linguistic environment and its application on smart healthcare

Study of Road-based Versus Pattern-based Game Strategies in Connect6

Computer game is an important branch in the field of artificial intelligence research nowadays, and the in-depth study of computer game can not only promote the development of game theory, but also contribute to the development of artificial intelligence. In this paper, for the game search strategy of connect6, according to the two most widely used search strategies and valuation function design, one is the valuation strategy based on the road, and the other is the valuation method based on the chess type, under the PVS search algorithm, the principle and implementation of these two search strategies are introduced in detail, and at the same time, the advantages and disadvantages of the two methods are compared, so as to provide the decision-making policy and the design of valuation function for the game of connect6 a more reasonable reference. Through comparison, it can be seen that in hexachrome, due to the complexity of the evaluation function and game strategy based on chess shapes, it requires the game programme to have a strong recognition of chess shapes, and the complexity of accurately evaluating the advantages and disadvantages of the position is higher. The road-based game strategy has better accuracy in evaluating and predicting the game, and can effectively reduce the game failure due to positional judgement errors.

Viscosity solutions approach to finite-horizon continuous-time Markov decision process

This paper investigates the optimal control problems for the finite-horizon continuous-time Markov decision processes with delay-dependent control policies. We develop compactification methods in decision processes and show that the existence of optimal policies. Subsequently, through the dynamic programming principle of the delay-dependent control policies, the differential-difference Hamilton-Jacobi-Bellman (HJB) equation in the setting of discrete space is established. Under certain conditions, we give the comparison principle and further prove that the value function is the unique viscosity solution to this HJB equation. Based on this, we show that among the class of delay-dependent control policies, there is an optimal one which is Markovian.

On the sigma-mu stochastic multicriteria analysis: Exact solutions for common particular cases

The sigma-mu approach is one of the recent innovations in the field of multicriteria decision aiding and composite indicators, extending the stochastic multicriteria acceptability analysis (SMAA) toolbox. The initial stage of this method involves computing the mean (mu) and standard deviation (sigma) for the composite value of the units under evaluation, considering a stochastic distribution on a set of admissible weights. This work develops closed-form formulas to obtain exact values for mu and sigma without needing approximations via Monte-Carlo simulations, which can be applied in some cases that are quite common. In terms of aggregation, these cases are characterized by an additive model, such as a weighted sum, a multiattribute value function, or PROMETHEE II. In terms of stochastic distributions, these cases include uniformly distributed unconstrained vectors of weights, rank-ordered vectors of weights, or lower-bounded weights. The developed formulas are applied to a didactic example and some open problems for future research are suggested.

Markov selections and Feller properties of nonlinear diffusions

In this paper we study a family of nonlinear (conditional) expectations that can be understood as a diffusion with uncertain local characteristics. Here, the differential characteristics are prescribed by a set-valued function. We establish its Feller properties and examine how to linearize the associated sublinear Markovian semigroup. In particular, we observe a novel smoothing effect of sublinear semigroups in frameworks which carry enough randomness. Furthermore, we link the value function corresponding to the semigroup to a nonlinear Kolmogorov equation. This provides a connection to the so-called Nisio semigroup.