Data-driven first-order methods for misspecified convex optimization problems: Global convergence and Rate estimates

Abstract

We consider a misspecified optimization problem that requires minimizing of a convex function f(x; θ*) in x over a closed and convex set X where θ* is an unknown vector of parameters. Suppose θ* may be learnt by a parallel learning process that generates a sequence of estimators θ k , each of which is an increasingly accurate approximation of θ*. In this context, we examine the development of coupled schemes that generate iterates (x k , θ k ) such that as the iteration index k → ∞, then x k → x*, a minimizer of f(x; θ*) over X and θ k → θ*.We make two sets of contributions along this direction. First, we consider the use of gradient methods and proceed to show that such techniques are globally convergent. In addition, such schemes show a quantifiable degradation in the linear rate of convergence observed for strongly convex optimization problems. When strong convexity assumptions are weakened, we see a modification in the convergence rate in function values of O(1/K) by an additive factor ≑θ 0 −θ*≑O(qK g +1/K) where ≑θ 0 −θ*≑ represents the initial misspecification in θ* and qg denotes the contractive factor associated with the learning process. Second, we present an averaging-based subgradient scheme and show that the optimal constant steplength leads to a modification in the rate by ≑θ 0 −θ*≑O(qK g +1/K), implying no effect on the standard rate of O(1/√K).