Uniform Convergence Of Empirical Means Research Articles

A convex analytic approach to system identification

This paper introduces a new concept for system identification in order to account for random and nonrandom(deterministic/set-membership) uncertainties. While, random/stochastic models are natural for modeling measurement errors, nonrandom uncertainties are well-suited for modeling parametric and nonparametric components. The new concept introduced is distinct from earlier concepts in many respects. First, inspired by the concept of uniform convergence of empirical means developed in machine learning theory, we seek a stronger notion of convergence in that the objective is to obtain probabilistic uniform convergence of model estimates to the minimum possible radius of uncertainty. Second, the formulation lends itself to convex analysis leading to description of optimal algorithms, which turn out to be well-known instrument-variable methods for many of the problems. Third, we characterize conditions on inputs in terms of second-order sample path properties required to achieve the minimum radius of uncertainty. Finally, we present fundamental bounds and optimal algorithms for system identification for a wide variety of standard as well as nonstandard problems that include special structures such as unmodeled dynamics, positive real conditions, bounded sets and linear fractional maps.

Exact rates in Vapnik–Chervonenkis bounds

Vapnik–Chervonenkis bounds on rates of uniform convergence of empirical means to their expectations have been continuously improved over the years since the precursory work in [26]. The result obtained by Talagrand in 1994 [21] seems to provide the final word as far as universal bounds are concerned. However, in the case where there are some additional assumptions on the underlying probability distribution, the exponential rate of convergence can be fairly improved. Alexander [1] and Massart [15] have found better exponential rates (similar to those in Bennett–Bernstein inequalities) under the assumption of a control on the variance of the empirical process. In this paper, the case of a particular distribution is considered for the empirical process indexed by a family of sets, and we provide the exact exponential rate based on large deviations theorems, as predicted by Azencott [2].

Rates of uniform convergence of empirical means with mixing processes

It has been shown previously by Nobel and Dembo (Stat. Probab. Lett. 17 (1993) 169) that, if a family of functions F has the property that empirical means based on an i.i.d. process converge uniformly to their values as the number of samples approaches infinity, then F continues to have the same property if the i.i.d. process is replaced by a β-mixing process. In this note, this result is extended to the case where the underlying probability is itself not fixed, but varies over a family of measures. Further, explicit upper bounds are derived on the rate at which the empirical means converge to their true values, when the underlying process is β-mixing. These bounds are less conservative than those derived by Yu (Ann. Probab. 22 (1994) 94).

Randomized algorithms for robust controller synthesis using statistical learning theory

By now it is known that several problems in the robustness analysis and synthesis of control systems are NP-complete or NP-hard. These negative results force us to modify our notion of “solving” a given problem. An approach that is recently gaining popularity is that of using randomized algorithms, which can be used to solve a problem approximately, most of the time. We begin with the premise that many problems in robustness analysis and synthesis can be formulated as the minimization of an objective function with respect to the controller parameters. It is argued that, in order to assess the performance of a controller as the plant varies over a prespecified family, it is better to use the average performance of the controller as the objective function to be minimized, rather than its worst-case performance, as the worst-case objective function usually leads to rather conservative designs. Then it is shown that a property from statistical learning theory known as uniform convergence of empirical means (UCEM) plays an important role in allowing us to construct efficient randomized algorithms for a wide variety of controller synthesis problems. In particular, whenever the UCEM property holds, there exists an efficient (i.e., polynomial-time) randomized algorithm. Using very recent results in statistical learning theory, it is shown that the UCEM property holds in any problem in which the satisfaction of a performance constraint can be expressed in terms of a finite number of polynomial inequalities. In particular, several problems such as robust stabilization and weighted H 2/ H ∞-norm minimization are amenable to the randomized approach.

Closure properties of uniform convergence of empirical means and PAC learnability under a family of probability measures

Open Problem No. 12.2 of (Vidyasagar, A Theory of Learning and Generalization: with Application to Neural Networks and Control Systems, Springer, London, 1997) asks: “Are the properties of uniform convergence of empirical means, and learnability preserved when the family of probability measures is replaced by its closure?” In this note, the question is answered in the affirmative. Further, it is shown that these properties are not preserved in general if the family of probability measures is replaced by its convex closure. An open question is posed as to whether it is possible to replace the family of probability measures by its convex closure in case the family is compact.

Randomized Algorithms for Robust Controller Synthesis Using Statistical Learning Theory: A Tutorial Overview

By now it is well known that several problems in the robustness analysis and synthesis of control systems are NP-complete or NP-hard. These negative results force us to modify our notion of “solving”a given problem. If we cannot solve a problem exactly because it is NP-hard, then we must settle for solving it approximately. If we cannot solve all instances of a problem, we must settle for solving “almost all” instances of a problem. An approach that is recently gaining popularity is that of using randomized algorithms. The notion of a randomized algorithm as defined here is somewhat different from that in the computer science literature, and enlarges the class of problems that can be efficiently solved. We begin with the premise that many problems in robustness analysis and synthesis can be formulated as the minimization of an objective function with respect to the controller parameters. It is argued that, in order to assess the performance of a controller as the plant varies over a prespecified family, it is better to use the average performance of the controller as the objective function to be minimized, rather than its worst-case performance, as the worstcase objective function usually leads to rather conservative designs. Then it is shown that a property from statistical learning theory known as uniform convergence of empirical means (UCEM) plays an important role in allowing us to construct efficient randomized algorithms for a wide variety of controller synthesis problems. In particular, whenever the UCEM property holds, there exists an efficient (i.e., polynomial-time) randomized algorithm. Using very recent results in statistical learning theory, it is shown that the UCEM property holds in several problems such as robust stabilization and weighted H2/H∞-norm minimization. Hence it is possible to solve such problems efficiently using randomized algorithms.