Generalized Value Function Research Articles

Intelligent laser welding through representation, prediction, and control learning: An architecture with deep neural networks and reinforcement learning

Laser welding is a widely used but complex industrial process. In this work, we propose the use of an integrated machine intelligence architecture to help address the significant control difficulties that prevent laser welding from seeing its full potential in process engineering and production. This architecture combines three contemporary machine learning techniques to allow a laser welding controller to learn and improve in a self-directed manner. As a first contribution of this work, we show how a deep, auto-encoding neural network is capable of extracting salient, low-dimensional features from real high-dimensional laser welding data. As a second contribution and novel integration step, these features are then used as input to a temporal-difference learning algorithm (in this case a general-value-function learner) to acquire important real-time information about the process of laser welding; temporally extended predictions are used in combination with deep learning to directly map sensor data to the final quality of a welding seam. As a third contribution and final part of our proposed architecture, we suggest that deep learning features and general-value-function predictions can be beneficially combined with actor–critic reinforcement learning to learn context-appropriate control policies to govern welding power in real time. Preliminary control results are demonstrated using multiple runs with a laser-welding simulator. The proposed intelligent laser-welding architecture combines representation, prediction, and control learning: three of the main hallmarks of an intelligent system. As such, we suggest that an integration approach like the one described in this work has the capacity to improve laser welding performance without ongoing and time-intensive human assistance. Our architecture therefore promises to address several key requirements of modern industry. To our knowledge, this architecture is the first demonstrated combination of deep learning and general value functions. It also represents the first use of deep learning for laser welding specifically and production engineering in general. We believe that it would be straightforward to adapt our architecture for use in other industrial and production engineering settings.

Value functions in the Wasserstein spaces: finite time horizons

We study analogs of value functions arising in classical mechanics in the space of probability measures endowed with the Wasserstein metric Wp, for 1<p<∞. Our main result is that each of these generalized value functions is a type of viscosity solution of an appropriate Hamilton–Jacobi equation, completing a program initiated by Gangbo, Nguyen, and Tudorascu. Of particular interest is a formula we derive for a generalized value function when the associated potential energy is of the form V(μ)=∫RdV(x)dμ(x). This formula allows us to make rigorous a well known heuristic connection between Euler–Poisson equations and classical Hamilton–Jacobi equations. Further results are presented which suggest there is a rich theory to be developed of deterministic control in the Wasserstein spaces.

Learning Value Functions in Interactive Evolutionary Multiobjective Optimization

This paper proposes an interactive multiobjective evolutionary algorithm (MOEA) that attempts to learn a value function capturing the users’ true preferences. At regular intervals, the user is asked to rank a single pair of solutions. This information is used to update the algorithm’s internal value function model, and the model is used in subsequent generations to rank solutions incomparable according to dominance. This speeds up evolution toward the region of the Pareto front that is most desirable to the user. We take into account the most general additive value function as a preference model and we empirically compare different ways to identify the value function that seems to be the most representative with respect to the given preference information, different types of user preferences, and different ways to use the learned value function in the MOEA. Results on a number of different scenarios suggest that the proposed algorithm works well over a range of benchmark problems and types of user preferences.

Optimal control of nonlinear discrete time-varying systems using a new neural network approximation structure

In this paper motivated by recently discovered neurocognitive models of mechanisms in the brain, a new reinforcement learning (RL) method is presented based on a novel critic neural network (NN) structure to solve the optimal tracking problem of a nonlinear discrete time-varying system in an online manner. A multiple-model approach combined with an adaptive self-organizing map (ASOM) neural network is used to detect changes in the dynamics of the system. The number of sub-models is determined adaptively and grows once a mismatch between the stored sub-models and the new data is detected. By using the ASOM neural network, a novel value function approximation (VFA) scheme is presented. Each sub-model contributes into the value function based on a responsibility signal obtained by the ASOM. The responsibility signal determines how much each sub-model contributes to the general value function. Novel policy iteration and the value iteration algorithms are presented to find the optimal control for the partially-unknown nonlinear discrete time-varying systems in an online manner. Simulation results demonstrate the effectiveness of the proposed control scheme.

Infinite horizon value functions in the Wasserstein spaces

We perform a systematic study of optimization problems in the Wasserstein spaces that are analogs of infinite horizon, deterministic control problems. We derive necessary conditions on action minimizing paths and present a sufficient condition for their existence. We also verify that the corresponding generalized value functions are a type of viscosity solution of a time independent, Hamilton–Jacobi equation in the space of probability measures. Finally, we prove a special case of a conjecture involving the subdifferential of generalized value functions and their relation to action minimizing paths.

Robust ordinal regression for value functions handling interacting criteria

We present a new method called UTAGMS–INT for ranking a finite set of alternatives evaluated on multiple criteria. It belongs to the family of Robust Ordinal Regression (ROR) methods which build a set of preference models compatible with preference information elicited by the Decision Maker (DM). The preference model used by UTAGMS–INT is a general additive value function augmented by two types of components corresponding to “bonus” or “penalty” values for positively or negatively interacting pairs of criteria, respectively. When calculating value of a particular alternative, a bonus is added to the additive component of the value function if a given pair of criteria is in a positive synergy for performances of this alternative on the two criteria. Similarly, a penalty is subtracted from the additive component of the value function if a given pair of criteria is in a negative synergy for performances of the considered alternative on the two criteria. The preference information elicited by the DM is composed of pairwise comparisons of some reference alternatives, as well as of comparisons of some pairs of reference alternatives with respect to intensity of preference, either comprehensively or on a particular criterion. In UTAGMS–INT, ROR starts with identification of pairs of interacting criteria for given preference information by solving a mixed-integer linear program. Once the interacting pairs are validated by the DM, ROR continues calculations with the whole set of compatible value functions handling the interacting criteria, to get necessary and possible preference relations in the considered set of alternatives. A single representative value function can be calculated to attribute specific scores to alternatives. It also gives values to bonuses and penalties. UTAGMS–INT handles quite general interactions among criteria and provides an interesting alternative to the Choquet integral.

First Steps Towards an Intelligent Laser Welding Architecture Using Deep Neural Networks and Reinforcement Learning

Abstract To address control difficulties in laser welding, we propose the idea of a self-learning and self-improving laser welding system that combines three modern machine learning techniques. We first show the ability of a deep neural network to extract meaningful, low-dimensional features from high-dimensional laser-welding camera data. These features are then used by a temporal-difference learning algorithm to predict and anticipate important aspects of the system's sensor data. The third part of our proposed architecture suggests using these features and predictions to learn to deliver situation-appropriate welding power; preliminary control results are demonstrated using a laser-welding simulator. The intelligent laser-welding architecture introduced in this work has the capacity to improve its performance without further human assistance and therefore addresses key requirements of modern industry. To our knowledge, it is the first demonstrated combination of deep learning and Nexting with general value functions and also the first usage of deep learning for laser welding specifically and production engineering in general. This work also provides a unique example of how predictions can be explicitly learned using reinforcement learning to support laser welding. We believe that it would be straightforward to adapt our approach to other production engineering applications.

Integration-segregation decisions under general value functions: 'Create your own bundle--choose 1, 2 or all 3!'

Whether to keep products segregated (e.g., unbundled) or integrate some or all of them (e.g., bundle) has been a problem of profound interest in areas such as portfolio theory in finance, risk capital allocations in insurance, and marketing of consumer products. Such decisions are inherently complex and depend on factors such as the underlying product values and consumer preferences, the latter being frequently described using value functions, also known as utility functions in economics. In this paper, we develop decision rules for multiple products, which we generally call 'exposure units' to naturally cover manifold scenarios spanning well beyond 'products'. Our findings show, for example, that the celebrated Thaler's principles of mental accounting hold as originally postulated when the values of all exposure units are positive (i.e., all are gains) or all negative (i.e., all are losses). In the case of exposure units mixed-sign values, decision rules are much more complex and rely on cataloging the Bell-number of cases that grow very fast depending on the number of exposure units. Consequently, in the present paper we provide detailed rules for the integration and segregation decisions in the case up to three exposure units, and partial rules for the arbitrary number of units.

Selection of a representative value function in robust multiple criteria ranking and choice

We introduce the concept of a representative value function in robust ordinal regression applied to multiple criteria ranking and choice problems. The proposed method can be seen as a new interactive UTA-like procedure, which extends the UTA GMS and GRIP methods. The preference information supplied by the decision maker (DM) is composed of a partial preorder and intensities of preference on a subset of reference alternatives. Robust ordinal regression builds a set of general additive value functions which are compatible with the preference information, and returns two binary preference relations: necessary and possible. They identify recommendations which are compatible with all or at least one compatible value function, respectively. In this paper, we propose a general framework for selection of a representative value function from among the set of compatibles ones. There are a few targets which build on results of robust ordinal regression, and could be attained by a representative value function. In general, according to the interactively elicited preferences of the DM, the representative value function may emphasize the advantage of some alternatives over the others when all compatible value functions acknowledge this advantage, or reduce the ambiguity in the advantage of some alternatives over the others when some compatible value functions acknowledge an advantage and other ones acknowledge a disadvantage. The basic procedure is refined by few extensions. They enable emphasizing the advantage of alternatives that could be considered as potential best options, accounting for intensities of preference, or obtaining a desired type of the marginal value functions.

Cumulative prospect theory challenges traditional expected utility theory

The Cumulative Prospect Theory (CPT) uses piecewise value functions instead of consumer utility and provides alternative assumptions for investment behaviour approximated by power value function. In this study, our aim to find a generalized value function that will make the value function introduced by Kahneman–Tversky (1992) a special case. This functional form of the value function determine the appropriate parameter of the values function. We believe that if one can approximate the original CPT value function by other types of functions, the optimization problem and the many other implications can be compared to choose the best model depending on the focus of the problems. This, eventually, could result in improving the theory in both theoretical and empirical points of views.

Building a set of additive value functions representing a reference preorder and intensities of preference: GRIP method

We present a method called Generalized Regression with Intensities of Preference (GRIP) for ranking a finite set of actions evaluated on multiple criteria. GRIP builds a set of additive value functions compatible with preference information composed of a partial preorder and required intensities of preference on a subset of actions, called reference actions. It constructs not only the preference relation in the considered set of actions, but it also gives information about intensities of preference for pairs of actions from this set for a given decision maker (DM). Distinguishing necessary and possible consequences of preference information on the considered set of actions, GRIP answers questions of robustness analysis. The proposed methodology can be seen as an extension of the UTA method based on ordinal regression. GRIP can also be compared to the AHP method, which requires pairwise comparison of all actions and criteria, and yields a priority ranking of actions. As for the preference information being used, GRIP can be compared, moreover, to the MACBETH method which also takes into account a preference order of actions and intensity of preference for pairs of actions. The preference information used in GRIP does not need, however, to be complete: the DM is asked to provide comparisons of only those pairs of reference actions on particular criteria for which his/her judgment is sufficiently certain. This is an important advantage comparing to methods which, instead, require comparison of all possible pairs of actions on all the considered criteria. Moreover, GRIP works with a set of general additive value functions compatible with the preference information, while other methods use a single and less general value function, such as the weighted-sum.

On the multi-attribute stopping problem with general value functions

We examine a sequential selection problem in which a single option must be selected. Each option's value is a function of its attributes, whose precise values can be ascertained at a given cost. We prove the optimality of a threshold stopping rule for a general class of objective functions.

A generalized target-driven cache replacement policy for mobile environments

Caching frequently accessed data items on the client side is an effective technique to improve the system performance in wireless networks. Due to cache size limitations, cache replacement algorithms are used to find a suitable subset of items for eviction from the cache. Many existing cache replacement algorithms employ a value function of different factors such as time since last access, entry time of the item in the cache, transfer time, item expiration time and so on. However, most of the existing algorithms are designed for WWW environment under weak consistency model. Their choices of value functions are based on experience and on a value function which only works for a specific performance metric. In this paper, we propose a generalized value function for cache replacement algorithms for wireless networks under a strong consistency model. The distinctive feature of our value function is that it is generalized and can be used for various performance metrics by making the necessary changes. Further, we prove that the proposed value function can optimize the access cost in our system model. To demonstrate the practical effectiveness of the generalized value function, we derive two specific functions and evaluate them by setting up two different targets: minimizing the query delay and minimizing the downlink traffic. Compared to previous schemes, our algorithm significantly improves the performance in terms of query delay or in terms of bandwidth utilization depending on the specified target.

A Generalized Sufficient Condition for Time-Optimal Control

This paper presents a sufficient optimality condition for time-optimal feedback control, which generalizes the existing sufficient conditions, i.e., the HJB equation and a Lyapunov-based condition derived in [12]. The new condition is satisfied by a class of piecewise C1 continuous functions, termed generalized value functions. It admits a larger class of functions, such as bounded functions, than the existing conditions as demonstrated in an example. The generalized condition might, therefore, be useful in solving for the time-optimal feedback control of nonlinear systems such as robotic manipulators.

Pricing and Capacity Decisions for a Service Facility: Stability and Multiple Local Optima

We consider a model for optimal pricing and capacity for a service facility. The problem is formulated as one of optimal design of a single-server queueing system, in which the design variables are the service rate and the arrival rate (equivalently, the price charged for admission). The model is a variant of one introduced by Dewan and Mendelson. We allow for an upper bound on the arrival rate and consider slightly more general user value functions. We show that an optimal solution may not lie in the interior of the feasible region and thus may not be characterized by the first-order differential conditions. Moreover, the first-order conditions typically have several solutions, some of which may be relative minima and produce a negative value of the objective function (customer value minus the sum of expected delay cost and capacity cost per unit time). We also examine the stability of the equilibrium arrival rate and the convergence of a dynamic adaptive algorithm for finding the optimal service rate, in the context of a model in which the distribution of customers' value of service is uniform. We show that the equilibrium arrival rate is stable if and only if the service rate is above a threshold value, which depends on the price charged for admission and the parameters of the uniform distribution of value of service. The dynamic, adaptive algorithm always converges to a relative maximum of the objective function if the service rate can be adjusted every time the arrival rate changes. Otherwise, the algorithm will start to diverge if and when the service rate ever falls below the threshold value associated with stability of the equilibrium arrival rate.

Structure of Pareto optima in an infinite-horizon economy where agents have recursive preferences

This article generalizes the one-agent growth theory with discounting to the case of several agents with recursive preferences. In a multi-consumption goods world, we show that, under some regularity conditions, any Pareto optimum can be viewed as a function of a trajectory of a dynamical system. The state space can be chosen to be the product of the space of capitals and the unit simplex. We define and study the properties of generalized value functions.