On-line Control Problem Research Articles

Online Markov Decision Processes With Kullback–Leibler Control Cost

This paper considers an online (real-time) control problem that involves an agent performing a discrete-time random walk over a finite state space. The agent's action at each time step is to specify the probability distribution for the next state given the current state. Following the set-up of Todorov, the state-action cost at each time step is a sum of a state cost and a control cost given by the Kullback-Leibler (KL) divergence between the agent's next-state distribution and that determined by some fixed passive dynamics. The online aspect of the problem is due to the fact that the state cost functions are generated by a dynamic environment, and the agent learns the current state cost only after selecting an action. An explicit construction of a computationally efficient strategy with small regret (i.e., expected difference between its actual total cost and the smallest cost attainable using noncausal knowledge of the state costs) under mild regularity conditions is presented, along with a demonstration of the performance of the proposed strategy on a simulated target tracking problem. A number of new results on Markov decision processes with KL control cost are also obtained.

Approximation-Based Robust Adaptive Automatic Train Control: An Approach for Actuator Saturation

This paper addresses an on-line approximation-based robust adaptive control problem for the automatic train operation (ATO) system under actuator saturation caused by constraints from serving motors. A robust adaptive control law is proposed, which is proved capable of on-line estimating of the unknown system parameters and stabilizing the closed-loop system. To cope with actuator saturation, another robust adaptive control is proposed for the ATO system, by explicitly considering the actuator saturation nonlinearity other than unknown system parameters, which is also proved capable of stabilizing the closed-loop system. Simulation results are presented to verify the effectiveness of the two proposed control laws.

Deterministic on-line call control in cellular networks

We study an on-line call control problem in cellular networks that are based on the Frequency Division Multiplexing (FDM) technology. In such networks, interference may occur when the same frequency is assigned to two different calls emanating from the same cell or its neighboring cells. The number of frequencies supporting the networks is limited. The goal is to maximize the number of calls served without causing any interference. We focus on the case that the number of frequencies is sufficiently large and the calls stay forever. We give a deterministic on-line algorithm with asymptotic competitive ratio of 2.5 and show a general lower bound of 2. For the special case of linear cellular networks, we achieve a best possible deterministic on-line algorithm with asymptotic competitive ratio of 3/2.

Dynamic system methods for solving mixed linear matrix inequalities and linear vector inequalities and equalities

A novel idea is proposed for solving a system of mixed linear matrix inequalities and linear vector inequalities and equalities. First, the problem is converted into an unconstrained minimization problem with a continuously differentiable convex objective function. Then, a continuous-time dynamic system and a discrete-time dynamic system are proposed for solving it. Under some mild conditions, the proposed dynamic systems are shown to be globally convergent to a solution of the problem. The merits of the methods refer to their simple numerical implementations and capability for handling nonstrict LMIs easily. In addition, the methods are promising in neural circuits realization, and therefore have potential applications in many online control problems. Several numerical examples are presented to illustrate the performance of the methods and substantiate the theoretical results.

Hybridizing evolutionary computation and reinforcement learning for the design of almost universal controllers for autonomous robots

In this paper a hybrid approach to the autonomous motion control of robots in cluttered environments with unknown obstacles is introduced. It is shown the efficiency of a hybrid solution by combining the optimization power of evolutionary algorithms and at the same time the efficiency of reinforcement learning in real-time and on-line situations. Experimental results concerning the navigation of a L-shaped robot in a cluttered environment with unknown obstacles are also presented. In such environments there appear real-time and on-line constraints well-suited to RL algorithms and, at the same time, there exists an extremely high dimension of the state space usually unpractical for RL algorithms but well-suited to evolutionary algorithms. The experimental results confirm the validity of the hybrid approach to solve hard real-time, on-line and high dimensional robot motion planning and control problems, where the RL approach shows some difficulties.

Effective economic dispatch model and algorithm

Under the electric power market environment, economic dispatch (ED) as part of the on-line Automatic Generation Control (AGC) problem considers unit, network and system constraints. For the ED problem, it is important to establish an effective model and algorithm. This paper examines the decoupled conditions that affect the solution optimality to this problem. It proposes an effective model and solution method. Based on the look-ahead technique, it finds the number of time intervals to guarantee the solution optimality. Next, an efficient technique for finding the optimal solution via the Interior-Point Method (IPM) based linear programming is described. Test cases, which include dispatching fourteen units over 10 time intervals on the test system with line flows, spinning reserve and ramp constraints are presented. Results indicate that as problem size increases the computational effort as measured by execution time increases dramatically if the optimal look-ahead time interval is not used.

On-line lower-order modeling via neural networks

Abstract This paper presents a novel method to determine the parameters of a first-order plus dead-time model using neural networks. The outputs of the neural networks are the gain, dominant time constant, and apparent time delay. By combining this algorithm with a conventional PI or PID controller, we also present an adaptive controller which requires very little a priori knowledge about the plant under control. The simplicity of the scheme for real-time control provides a new approach for implementing neural network applications for a variety of on-line industrial control problems. Simulation and experimental results demonstrate the feasibility and adaptive property of the proposed scheme.

Comparison of methods for estimating real process derivatives in on-line optimization

The problem of derivative information estimation for the on-line optimization control problem has been widely recognized. This paper investigates methods and techniques for the estimation of these derivatives, to be used within the well-known integrated system optimization and parameter estimation algorithm. Methods of finite difference approximation, dual control optimization, Broydon’s method and a dynamic identification method, with a linear and non-linear models, are discussed and applied, under simulation, to a cascade continuous stirred tank reactor system.

A New On-Line PID Tuning Method Using Neural Networks

Abstract This paper presents an on-line PID tuning control method, based on the parameters of a first-order plus dead-time (FOPDT) model, which are obtained by using Neural Networks (NN). The outputs of the neural networks are the three parameters of the FOPDT model. By combining this algorithm with a conventional PID controller, an adaptive controller is obtained which requires very little a priori knowledge about the plant under control. The simplicity and feasibility of the scheme for real-time control provide a new approach for implementing neural network applications for a variety of on-line industrial control problems. Simulation results demonstrate the feasibility and adaptive property of the proposed scheme.

On-line production control using a genetic algorithm

This paper discusses the application of a genetic algorithm as an off-line optimization tool in an on-line production control problem. The on-line control problem is replaced by a sequence of the corresponding off-line problems and solved using a genetic algorithm. The effectiveness of the method is compared with that of a traditional optimization technique (Gradient Projection Method) using simulation experiments. The results indicate the usefulness of this approach.

Adaptive neuro-control for spacecraft attitude control

Spacecraft attitude control is approached as a nonlinear adaptive control problem and neuro-control, which combines concepts from artificial neural networks and adaptive control, is investigated as an alternative to linear control approaches. Three capabilities of neuro-controllers are demonstrated using a nonlinear model of the Space Station Freedom. These capabilities are: (a) synthesis of robust nonlinear controllers using neural networks; (b) copying an existing control law using neural networks; and (c) adaptively modifying neuro-controller characteristics for varying inertia characteristics. The main components of the adaptive neuro-controllers are an identification network and a controller network. Both these networks are trained using the back-propagation of error learning paradigm. To ensure robustness of the neuro-controller, optimally connected neural networks are synthesized for the identification and the controller networks. For the on-line adaptive control problem, a backpropagation of error technique using a linear adaptive critic is introduced in place of the backpropagation through time technique. Performances of the nonlinear neuro-controllers for the three cases listed above are verified using a nonlinear simulation of the Space Station. Results presented substantiate the feasibility of using neural networks in robust nonlinear adaptive control of spacecraft.

An on-line controller for production systems with seasonal demands

This paper investigated a hirarchical production planning and control model for flexible manufacturing systems that simultaneously make multiple parts subject to periodic demands. The off-line planning and on-line control strategy is utilized in the aggregate planning model. First, the optimal inventory levels are determined by the off-line planning model. The problem is formulated as a boundary-free discrete optimal control problem. When the actual inventory levels deviate from the optimal levels due to random interruptions, an on-line control problem is solved. The problem is formulated as a boundary-fixed discrete optimal control problem. The resulting large-scale problem is broken into a network flow problem and several single part aggregate production planning problems. The network flow problem determines the alternative routings among cells for given production rates. Single part production planning problems determine the production rates for each part in each period. Numerical examples illustrate the algorithm's efficiency.

A game-theoretic control approach for job shops in the presence of disruptions

A methodology inspired by a game-theoretic view of the on-line control problem for job-shops is developed which allows the use of static off-line schedules in uncertain environments, and the explicit incorporation of deterministic and stochastic information concerning future disturbances. A discrete event dynamic system representation is used to formulate the control problem. The control objectives are to minimize expected makespan and deviations from an off-line schedule. Computational tractability is achieved through a graph-theoretic decomposition of the job-shop scheduling problem, the development of fast rescheduling heuristics, and efficient sampling of future events. A heuristic search algorithm is developed for problem resolution. Experimental results show that the methodology significantly outperforms existing control methods such as ‘total rescheduling’ and ‘right-shift.’ Most importantly, the control methodology demonstrates consistent performance and small CPU time requirements throughout the tests.

A Measurement Oriented Formulation of the Dynamics of Natural and Robotic Systems

A measurement oriented formulation of rigid body dynamics is applied to a special class of planar linkage systems. It leads to a diagonal moment of inertia matrix, and thus simplifies the feedforward computations in the controller. On the other hand, if measurements are not available or measurements are not allowed, the computational burden shifts to the feedback computation of the generalized or internal forces. This formulation may find applications in off-line digital computer simulations and on-line control of the rigid body systems via the inverse dynamics methods. It may also underly computations in natural and biological systems that are rich with sensory modalities and processes. The computational burden in this model is shifted from the inverse of moment of inertia matrix to the derivation of the forces of constraint, contact, connection, internal or generalized. When these forces are available from measurements, the computations are indeed reduced and consequently on-line control problems are rendered easier. As a consequence of this representation the problems of transmission delay, and predictive compensation become important as is shown via an example. The investigation of the range of intermediate representations where computations and measurements are combined remains a fertile subject.

Receding horizon control of nonlinear systems

The receding horizon control strategy provides a relatively simple method for determining feedback control for linear or nonlinear systems. The method is especially useful for the control of slow nonlinear systems, such as chemical batch processes, where it is possible to solve, sequentially, open-loop fixed-horizon, optimal control problems online. The method has been shown to yield a stable closed-loop system when applied to time-invariant or time-varying linear systems. It is shown that the method also yields a stable closed-loop system when applied to nonlinear systems. >

Statement of the On-line Control Problem in Flexible Manufacturing Systems

Abstract The paper presents the formalized statement of the on-line control problem in flexible manufacturing systems (FMS), featuring two components - a calendar planning, i.e. working - out the jobs schedule for a planned period and on-line planning on the level of FMS working units as the means of influencing the manufacturing process with the view to changing the deviations from the calendar plan.

Control of Emulsion Polymerization Reactors

An overview of some of the research projects on the modelling, optimization and control of emulsion polymerization reactors being carried out at the McMaster Institute for Polymer Production Technology will be presented. The objectives are to optimize production rates and to achieve control over 'product quality variables such as polymer particle size distribution, particle morphology, copolymer composition, sequence length, molecular weights, and long chain branching.The polymer systems under investigation include poly(vinyl acetate) and vinyl acetate copolymers (acrylic acid, butyl acrylate and 2-ethyl hexyl acrylate comonomers) and styrene copolymers (acrylonitrile and butadiene comonomers).One project in particular on the control of oscillations in conversion and latex properties in continuous emulsion polymerization in a CSTR train is discussed in more detail. It is shown that the oscillation phenomenon, which occurs in essentially all Case I emulsion polymerization systems (e.g. vinyl acetate and vinyl chloride emulsion polymerization), is conveniently eliminated with a specially designed reactor train. This design was found with the use of a dynamic model for a latex reactor.The design consists of a train of CSTR's in which the first reactor is a very small nucleating (seed) reactor and the feed streams are split between the first two reactors. The on-line control problem then reduces to the much simpler problem of controlling the "steady-state" generation of particles in the small seed reactor in order to achieve any desired final conversion and particle size. It is shown that this can easily be accomplished in this new reactor configuration by manipulating the split of monomer and water feed between the reactors, and the feedrate of initiator to the first reactor.

A recursive least-squares approach to the on-line adaptive control problem†

A practical study of the application of recursive least squares methods to the adaptive control of a multivariable process is described. The process has a number of inputs and an equal number of state variables are required to be controlled. Investigations are carried out both off-line and in real-time using a hybrid computer. Results show the techniques to be ideally suited for on-line application.