Dynamic Programming-based Approach Research Articles

Approximate Dynamic Programming-Based Approaches for Data-Driven Control of Hydraulic Looper System

Approximate Dynamic Programming-Based Approaches for Data-Driven Control of Hydraulic Looper System

On Routing and Channel Selection in Cognitive Radio Mesh Networks

Secondary users in a cognitive radio mesh network may select from a set of available channels, provided that they do not disrupt communications among primary users. This ability can improve the overall network performance but introduces the question of how to best use the channels. This paper first considers the problem of selecting the channels to use given a routing path such that the end-to-end throughput along the path is maximized. We show that a dynamic programming-based approach can optimally solve the problem and, if the path satisfies a natural condition, in time, be linear in the length (hop count) of the path. In addition, the algorithm can easily be implemented in a distributed fashion. We also examine the harder joint problem of finding the best routing path and channel selection that maximizes the end-to-end throughput. We prove that obtaining a (2/3 + ε) approximation to the joint problem is NP-hard. We then present a heuristic algorithm for the joint problem and a second heuristic channel-aware routing-only algorithm. Numerical results are provided to demonstrate the effectiveness of the methods on several experimental scenarios.

Scheduling for End-to-End Deadline-Constrained Traffic With Reliability Requirements in Multihop Networks

We attack the challenging problem of designing a scheduling policy for end-to-end deadline-constrained traffic with reliability requirements in a multihop network. It is well known that the end-to-end delay performance for a multihop flow has a complex dependence on the high-order statistics of the arrival process and the algorithm itself. Thus, neither the earlier optimization-based approaches that aim to meet the long-term throughput demands nor the solutions that focus on a similar problem for single-hop flows directly apply. Moreover, a dynamic programming-based approach becomes intractable for such multi-timescale quality-of-service (QoS)-constrained traffic in a multihop environment. This motivates us in this paper to develop a useful architecture that enables us to exploit the degree of freedom in choosing appropriate service discipline. Based on the new architecture, we propose three different approaches, each leading to an original algorithm. We study the performance of these algorithms in different scenarios to show both optimality characteristics and to demonstrate the favorable service discipline characteristics they possess. We provide extensive numerical results to compare the performance of all of these solutions to throughput-optimal back-pressure-type schedulers and to longest waiting-time-based schedulers that have provably optimal asymptotic performance characteristics. Our results reveal that the dynamic choice of service discipline of our proposed solutions yields substantial performance improvements compared to both of these types of traditional solutions under nonasymptotic conditions.

Robust policies for the transformer acquisition and allocation problem

As the electric transmission system in the U.S. ages, mitigating the risk of high-voltage transformer failures becomes an increasingly important issue for transmission owners and operators. This paper addresses this issue by presenting an in-depth study of different policies to manage the acquisition and allocation of high-voltage transformers over time. We analyze PJM’s transmission system by simulating and evaluating different policies under various assumptions and parameter settings. This allows us to characterize different policies and more generally learn about the behavior of the system. Our findings conclude that the approximate dynamic programming-based approach introduced in Enders et al. (Technical report, Department of Operations Research and Financial Engineering, Princeton University, 2008) outperforms the policy that PJM currently uses based on several performance measures. We analyze issues of practical importance to PJM such as the effect of system wide optimization vs. optimization by transmission owner. Our findings suggest that system wide decision making considerably reduces the costs associated with transformer failures and replacements.

Optimal Neuro-Fuzzy External Controller for a STATCOM in the 12-Bus Benchmark Power System

An optimal neuro-fuzzy external controller is designed in this paper for a static compensator (STATCOM) in the 12-bus benchmark power system. The controller provides an auxiliary reference signal for the STATCOM in such a way that it improves the damping of the rotor speed deviations of its neighboring generators. A Mamdani fuzzy rule base constitutes the core of the controller. A heuristic dynamic programming-based approach is used to further train the controller and enable it to provide nonlinear optimal control at different operating conditions of the power system. Simulation results are provided that indicate the proposed neuro-fuzzy external controller is more effective than a linear external controller for damping out the speed deviations of the generators. In addition, the two controllers are compared in terms of the control effort generated by each one during various disturbances and the proposed neuro-fuzzy controller proves to be more effective with smaller control effort.

Approximate dynamic programming-based approaches for input–output data-driven control of nonlinear processes

We propose two approximate dynamic programming (ADP)-based strategies for control of nonlinear processes using input–output data. In the first strategy, which we term ‘ J -learning,’ one builds an empirical nonlinear model using closed-loop test data and performs dynamic programming with it to derive an improved control policy. In the second strategy, called ‘ Q -learning,’ one tries to learn an improved control policy in a model-less manner. Compared to the conventional model predictive control approach, the new approach offers some practical advantages in using nonlinear empirical models for process control. Besides the potential reduction in the on-line computational burden, it offers a convenient way to control the degree of model extrapolation in the calculation of optimal control moves. One major difficulty associated with using an empirical model within the multi-step predictive control setting is that the model can be excessively extrapolated into regions of the state space where identification data were scarce or nonexistent, leading to performances far worse than predicted by the model. Within the proposed ADP-based strategies, this problem is handled by imposing a penalty term designed on the basis of local data distribution. A CSTR example is provided to illustrate the proposed approaches.

Question selection for multi-attribute decision-aiding

A decision-maker (DM), or a group of DMs, must select a most preferred alternative from a finite set of alternatives. Each alternative has a single consequence. There are multiple and conflicting attributes. All value scores are known precisely. However, all that is known about the vector of trade-off weights is that it is a member of a given set, the trade-off weight set. A facilitator can ask questions and use the responses to update the trade-off weight set and hence the set of non-dominated alternatives, i.e., those alternatives that are candidates for being a most preferred alternative. The DM can terminate this process at any point. Termination typically occurs when the non-dominated set contains a sufficiently small number of alternatives. The facilitator’s role is to ask questions that efficiently lead the DM to a most preferred alternative. The objective of this research is to aid the facilitator in selecting the best question to ask next. We model the question-response process as a sequential decision-making problem under uncertainty and develop a dynamic programming-based approach that guarantees a finite, and hence potentially computable, representation of the expected optimal cost-to-go function. An example serves to illustrate the approach.

Short-term generation scheduling of thermal units with voltage security and environmental constraints

A dynamic and linear programming-based approach is investigated for the short-term scheduling of thermal power plant units considering the security and environmental constraints of electric power systems. In addition to the basic constraints (i.e. real power limits of the generators, real power demand, spinning reserve, minimum up and down lime of units) a comprehensive set of constraints is considered including: transmission line capacity limits; generators dynamic ramp rate; voltage security margin; and environmental issues. An optimal multiplier continuation method is adopted to effectively calculate the voltage security margin. The constraint relating to the voltage security margin is considered as a transition constraint in the dynamic programming process. The environmental issue is considered as a restriction on the hourly SO/sub 2/ emission allowances and other constraints are considered in the dynamic/linear programming procedures. Results of the application of the proposed algorithm to a test system are presented to illustrate the effectiveness of the approach.