State Evaluation Function Research Articles

Towards Design Automation of Microfluidic Mixers: Leveraging Reinforcement Learning and Artificial Neural Networks.

Microfluidic mixers, a pivotal application of microfluidic technology, are primarily utilized for the rapid amalgamation of diverse samples within microscale devices. Given the intricacy of their design processes and the substantial expertise required from designers, the intelligent automation of microfluidic mixer design has garnered significant attention. This paper discusses an approach that integrates artificial neural networks (ANNs) with reinforcement learning techniques to automate the dimensional parameter design of microfluidic mixers. In this study, we selected two typical microfluidic mixer structures for testing and trained two neural network models, both highly precise and cost-efficient, as alternatives to traditional, time-consuming finite-element simulations using up to 10,000 sets of COMSOL simulation data. By defining effective state evaluation functions for the reinforcement learning agents, we utilized the trained agents to successfully validate the automated design of dimensional parameters for these mixer structures. The tests demonstrated that the first mixer model could be automatically optimized in just 0.129 s, and the second in 0.169 s, significantly reducing the time compared to manual design. The simulation results validated the potential of reinforcement learning techniques in the automated design of microfluidic mixers, offering a new solution in this field.

Heuristic search strategy based on probabilistic and geostatistical simulation approach for simultaneous identification of groundwater contaminant source and simulation model parameters

In this study, a heuristic search strategy based on probabilistic and geostatistical simulation approach is developed for simultaneous identification of groundwater contaminant source and simulation model parameters. The numerical simulation model, which is repeatedly invoked to evaluate the likelihood in Bayesian formula, can be substituted by the surrogate system to reduce the huge computational load. To improve the approximation accuracy of the surrogate system to the simulation model, we employ Entropy Weight method as a novel method to establish a combined surrogate system by combining Gaussian process, support vector regression, and kernel extreme learning machine. A state evaluation (heuristic) function based on Bayesian formula and the surrogate system is introduced to quantify the approximation degree of variables in current state to true values for contaminant sources and simulation model parameters. Thereafter, a heuristic search iterative process related to artificial intelligence is designed for simultaneous identification, which takes full advantage of the guidance and correction role of actual field monitoring data. A multi-vector and variable-step size random walk method is proposed to select the candidate point. A Metropolis formula based on the state evaluation function is constructed, and the result is used as the judging criterion for the state transition. Finally, simultaneous identification results are obtained when the iteration reaches the convergence criteria. The proposed approaches are tested with a numerical case study. The results indicate that the heuristic search strategy can assist in identifying groundwater contaminant source and simulation model parameters simultaneously with high accuracy and efficiency.

Improving RTS Game AI by Supervised Policy Learning, Tactical Search, and Deep Reinforcement Learning

Constructing strong AI systems for video games is difficult due to enormous state and action spaces and the lack of good state evaluation functions and high-level action abstractions. In spite of recent research progress in popular video game genres such as Atari 2600 console games and multiplayer online battle arena (MOBA) games, to this day strong human players can still defeat the best AI systems in adversarial video games. In this paper, we propose to use a deep Convolutional Neural Network (CNN) to select among a limited set of abstract action choices in Real-Time Strategy (RTS) games, and to utilize the remaining computation time for game tree search to improve low-level tactics. The CNN is trained by supervised learning on game states labeled by Puppet Search, a strategic search algorithm that uses action abstractions. Replacing Puppet Search by a CNN frees up time that can be used for improving units' tactical behavior while executing the strategic plan. Experiments in the μRTS game show that the combined algorithm results in higher win-rates than either of its two independent components and other state-of-the-art μRTS agents. We then present a case-study that investigates how deep Reinforcement Learning (RL) can be used in modern video games, such as Total War: Warhammer, to improve tactical multi-agent AI modules. We use popular RL algorithms such as Deep-Q Networks (DQN) and Asynchronous AdvantageActor Critic (A3C), basic network architectures and minimal hyper-parameter tuning to learn complex cooperative behaviors that defeat the highest difficulty built-in AI in mediumscale scenarios.

Vector Control-Based Singularity Avoidance/Escape Steering Law for Single Gimbal Control Moment Gyros

The single-gimbal control moment gyros (SGCMGs) steering law has been a standing topic in the field of spacecraft attitude control for several decades. A practical steering law should meet the following requirements simultaneously: powerful singularity avoidance ability, high precision torque output ability and fast singularity escape ability. Moreover, the angular velocity commands generated by the steering law should not jitter sharply. In this paper, a vector control-based singularity avoidance and escape steering law for SGCMGs is proposed to satisfy these goals. In addition, a system angular momentum state evaluation function for command torque is defined for the first time, which can be used to effectively evaluate the remaining angular momentum resources for the command torque. We demonstrate the effectiveness and superiority of the proposed steering law with detailed comparisons to existing efforts.

Bayesian Optimization with Resource Constraints and Production

In this paper, we aim to take a step toward a tighter integration of automated planning and Bayesian Optimization (BO). BO is an approach for optimizing costly-to-evaluate functions by selecting a limited number of experiments that each evaluate the function at a specified input. Typical BO formulations assume that experiments are selected one at a time, or in fixed batches, and that experiments can be executed immediately upon request. This setup fails to capture many real-world domains where the execution of an experiment requires setup and preparation time. In this paper, we define a novel BO problem formulation that models the resources and activities needed to prepare and run experiments. We then present a planning approach, based on finite-horizon tree search, for scheduling the potentially concurrent experimental activities with the aim of best optimizing the function within a limited time horizon. A key element of the approach is a novel state evaluation function for evaluating leaves of the search tree, for which we prove approximate guarantees. We evaluate the approach on a number of diverse benchmark problems and show that it produces high-quality results compared to a number of natural baselines.

RoboCupサッカーシミュレーションリーグ2Dにおける リターンパス：chain action上の状態予測と局面評価関数

RoboCupサッカーシミュレーションリーグ2Dにおける リターンパス：chain action上の状態予測と局面評価関数

An Automatically Generated Evaluation Function in General Game Playing

General game-playing (GGP) competitions provide a framework for building multigame-playing agents. In this paper, we describe an attempt at the implementation of such an agent. It relies heavily on our knowledge-free method of automatic construction of an approximate state evaluation function, based on game rules only. This function is then employed by one of the two game tree search methods: MTD (f) or guided upper confidence bounds applied to trees (GUCT), the latter being our proposal of an algorithm combining UCT with the usage of an evaluation function. The performance of our agent is very satisfactory when compared to a baseline UCT implementation.

A knowledge-based problem solving method in GIS application

Model design for theme analysis is one of the biggest challenges in GIS. Many real applications in GIS require functioning not only in data management and visualization, but also in analysis and decision-making. Confronted with an application of planning a new metro line in a city, a typical GIS is unable to accomplish the task in the absence of human experts or artificial intelligence technologies. Apart from being models for analyzing in different themes, some applications are also instances of problem solving in AI. Therefore, in order to strengthen its ability in automatic analysis, many theories and technologies from AI can be embedded in the GIS. In this paper, a state space is defined to formalize the metro line planning problem. By utilizing the defined state evaluation function, knowledge-based rules and strategies, a heuristic searching method is developed to optimize the solutions iteratively. Experiments are implemented to illuminate the validity of this AI-enhanced automatic analysis model of GIS.

Car-driving assistance using organization measurement of reactive multi-agent system

This work presents an approach to the obstacle avoidance problem, applicable in the frame of driver assistance. A decision, expressed as a proposed acceleration vector for the vehicle, is elaborated from the evaluation of a set of indicators characterizing the global state of a system of reactive agents (RMAS). Those agents evolve in a virtual environment produced on the base of vehicle’s perceptions of the material environment around it. Agent-to-agent and agent-to-environment interactions are defined in order to produce a distribution of agents over the virtual environment. This distribution, taken as the global state of the system, is analyzed by applying a set of indicators inspired from statistical physics, to calculate a new vehicle’s acceleration vector. This work presents the details of the RMAS model and its interaction laws, together with the global state evaluation functions. The approach has been applied to experimentation with a laboratory vehicle. Some experimental results are presented here.

A pulse neural network reinforcement learning algorithm for partially observable Markov decision processes

AbstractThis paper considers learning by a pulse neural network and proposes a new reinforcement learning algorithm focusing on the ability of pulse neuron elements to process time series. The conventional integrator neuron element is modeled in terms of the average firing rate of the biological neuron. But the pulse neuron is a modeling of the input–output relation of the time‐series pulse (spike) and the decay of the internal state (internal potential). The application of such neural networks has been considered in recent engineering studies. It is known in particular that a pulse neuron with a high decay rate acts as a coincidence detector. The proposed model combines pulse neuron elements with different decay rates, which facilitates the processing of the time‐series input information and the discrimination of fuzzy states in a partially observable Markov decision process. The proposed network is a four‐layered feedforward network in which the pulse neuron elements forming the two hidden layers provide a pseudo‐representation of the state in the environment. The elements generate a secondary reinforcement signal which results in learning similar to the conventional reinforcement scheme based on the state evaluation function. A computer experiment verifies that the proposed model works effectively in an environment which is strongly partially observable. © 2005 Wiley Periodicals, Inc. Syst Comp Jpn, 36(3): 42–52, 2005; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.10645

Stochastic income and wealth

As is well known, the deterministic version of the maximum principle allows us to state rigorously the relationship between deterministic income (essentially the Hamiltonian) and deterministic wealth (essentially the state evaluation function). As a background motivating point of departure, consider the simple story of an infinitely long-lived individual whose sole wealth consists of a bank deposit paying a constant rate of interest. For this simple parable, whether we define and measure income in the spirit of Fisher as being the return on wealth, or in the spirit of Lindahl as being consumption plus net value of investment, or in the spirit of Hicks as being the largest permanently maintainable level of consumption, we get the same answer to the question ‘‘what is income?’’. The theory of the deterministic maximum principle shows is that this fundamental identity of the three seemingly different definitions of income is much broader and goes much deeper than the simple bank-account parable. We now seek to extend the investigation to cover uncertainty by dealing with the case where net capital accumulation is described by a stochastic diffusion equation in place of a deterministic differential equation. Stochastic diffusion equations introduce mathematical subtleties and complexities, which we will only deal with casually in this treatment. The introduction of uncertainty ratchets up the required level of analysis another several notches on the scale of mathematical complexity. To treat the subject of this paper fully rigorously and also at the level of generality of a multi-dimensional economic growth problem could easily turn this paper into a book and would be out of all proportion to its www.elsevier.com/locate/econbase Japan and the World Economy 16 (2004) 277–301

ALPHA-BETA SEARCH ENHANCEMENTS WITH A REAL-VALUE GAME-STATE EVALUATION FUNCTION

The note presents results of applying several Alpha-Beta search enhancements in the game of give-away checkers with a real-value state evaluation function. In particular, the MTD-bi (bisection) algorithm is tested and compared (1) with MTD(f ) and (2

State evaluation functions and Lyapunov functions for neural networks

For a network of binary state elements, consider functions from the set of state configurations into the set of real numbers. We first characterize the existence of such state evaluation functions through the properties on their difference functions. A method to restore the original state evaluation function from their difference functions is also shown. A state evaluation function is a Lyapunov function for some network if the function value decreases as the system undergoes state changes. Then we apply the results for networks of McCulloch-Pitts type model neurons to see when there can be Lyapunov functions. In the simplest linear analysis, the weight matrix W of the network has non-negative diagonal elements and must be quasi-symmetric with respect to a positive diagonal matrix C, that is, CW must be symmetric. We have also derived, as another example, more complicated conditions under which neural networks have Lyapunov functions.

An Autonomous Artificial Designer of Thermal Energy Systems: Part 1—Theoretical Considerations

A knowledge-based approach to automated conceptual design (flowsheet synthesis) of thermal energy systems with strong interactions between heat/power/chemical transformations is presented. The approach is based on a computer-oriented state-space search guided by specially developed heuristics, and makes use of Second Law (exergetic) analysis, rather than mimicking the strategy of a human designer. The original design problem, formulated in terms of an equipment units network, is decomposed and reduced to a level of a network of elementary processes, with a resulting reduction in the search space. A special form of fundamental equations for steady-state open thermodynamic systems, based on a “temperature interval” approach, allows one to determine the effects of work, heat and chemical interactions within the system on the magnitude of Second-Law infeasibility, and on the overall exergy loss over any particular temperature interval, prior to the completion of the design. Based on this treatment, a set of generalized transforming operators, a plausible move generator, and a state evaluation function are formulated. The search algorithm is discussed in detail.

An Autonomous Artificial Designer of Thermal Energy Systems: Part 2—Solution Algorithm

A knowledge-based approach to automated conceptual design (flowsheet synthesis) of thermal energy systems with strong interactions between heat/power/chemical transformations is presented. In Part 1, formulation of a thermal design problem is stated in terms of input/output specification, component interaction, feasibility constraints, and penalty function. The problem is then decomposed in inner problems that deal with a set of elementary processes, and outer problems that find a network of components approximating the optimum set of elementary processes. A design state is evaluated using a special form of fundamental equation for steady-state open thermodynamic systems based on a “temperature interval” concept. In Part 2 of this paper, an algorithm is presented. The algorithm makes use of the state evaluation function, transformation operators, and the plausible move operator to search through a space of the design states. A simple closed-cycle gas turbine is employed to illustrate the behavior of the “artificial designer” as it advances from a certain given design to more sophisticated schemes.

Duality Theory for Infinite Horizon Convex Models

Often it is desirable to formulate certain decision problems without specifying a cut-off date and terminal conditions (which are sometimes felt to be arbitrary). This paper examines the duality theory that goes along with the kind of open-ended convex programming models frequently encountered in mathematical economics and operations research. Under a set of general axioms, duality conditions necessary and sufficient for infinite horizon optimality are derived. The proof emphasizes the close connection between duality theory for infinite horizon convex models and dynamic programming. Dual prices with the required properties are inductively constructed in each period as supports to the state evaluation function.