Stochastic Oracle Research Articles

ORCCA: Optimal Randomized Canonical Correlation Analysis.

Random features approach has been widely used for kernel approximation in large-scale machine learning. A number of recent studies have explored data-dependent sampling of features, modifying the stochastic oracle from which random features are sampled. While proposed techniques in this realm improve the approximation, their suitability is often verified on a single learning task. In this article, we propose a task-specific scoring rule for selecting random features, which can be employed for different applications with some adjustments. We restrict our attention to canonical correlation analysis (CCA) and provide a novel, principled guide for finding the score function maximizing the canonical correlations. We prove that this method, called optimal randomized CCA (ORCCA), can outperform (in expectation) the corresponding kernel CCA with a default kernel. Numerical experiments verify that ORCCA is significantly superior to other approximation techniques in the CCA task.

Stochastic Composition Optimization of Functions Without Lipschitz Continuous Gradient

In this paper, we study stochastic optimization of two-level composition of functions without Lipschitz continuous gradient. The smoothness property is generalized by the notion of relative smoothness which provokes the Bregman gradient method. We propose three stochastic composition Bregman gradient algorithms for the three possible relatively smooth compositional scenarios and provide their sample complexities to achieve an $$\epsilon $$ -approximate stationary point. For the smooth of relatively smooth composition, the first algorithm requires $$\mathcal {O}(\epsilon ^{-2})$$ calls to the stochastic oracles of the inner function value and gradient as well as the outer function gradient. When both functions are relatively smooth, the second algorithm requires $$\mathcal {O}(\epsilon ^{-3})$$ calls to the inner function value stochastic oracle and $$\mathcal {O}(\epsilon ^{-2})$$ calls to the inner and outer functions gradients stochastic oracles. We further improve the second algorithm by variance reduction for the setting where just the inner function is smooth. The resulting algorithm requires $$\mathcal {O}(\epsilon ^{-5/2})$$ calls to the inner function value stochastic oracle, $$\mathcal {O}(\epsilon ^{-3/2})$$ calls to the inner function gradient, and $$\mathcal {O}(\epsilon ^{-2})$$ calls to the outer function gradient stochastic oracles. Finally, we numerically evaluate the performance of these three algorithms over two different examples.

New First-Order Algorithms for Stochastic Variational Inequalities

In this paper, we propose two new solution schemes to solve the stochastic strongly monotone variational inequality (VI) problems: the stochastic extra-point solution scheme and the stochastic extra-momentum solution scheme. The first one is a general scheme based on updating the iterative sequence and an auxiliary extra-point sequence. In the case of a deterministic VI model, this approach includes several state-of-the-art first-order methods as its special cases. The second scheme combines two momentum-based directions: the so-called heavy-ball direction and the optimism direction, where only one projection per iteration is required in its updating process. We show that if the variance of the stochastic oracle is appropriately controlled, then both schemes can be made to achieve optimal iteration complexity of $\mathcal{O}\left(\kappa\ln\left(\frac{1}{\epsilon}\right)\right)$ to reach an $\epsilon$-solution for a strongly monotone VI problem with condition number $\kappa$. As a specific application to stochastic VI, we demonstrate how to incorporate a zeroth-order approach for solving stochastic minimax saddle-point problems in our schemes, where only noisy and biased samples of the objective can be obtained, with a total sample complexity of $\mathcal{O}\left(\frac{\kappa}{\epsilon}\right)$.

Mini-Batch Stochastic Three-Operator Splitting for Distributed Optimization

We consider a network of agents, each with its own private cost consisting of a sum of two possibly nonsmooth convex functions, one of which is composed with a linear operator. At every iteration each agent performs local calculations and can only communicate with its neighbors. The challenging aspect of our study is that the smooth part of the private cost function is given as an expected value and agents only have access to this part of the problem formulation via a heavy-tailed stochastic oracle. To tackle such sampling-based optimization problems, we propose a stochastic extension of the triangular pre-conditioned primal-dual algorithm. We demonstrate almost sure convergence of the scheme and validate the performance of the method via numerical experiments.

Minibatch Forward-Backward-Forward Methods for Solving Stochastic Variational Inequalities

We develop a new stochastic algorithm for solving pseudomonotone stochastic variational inequalities. Our method builds on Tseng’s forward-backward-forward algorithm, which is known in the deterministic literature to be a valuable alternative to Korpelevich’s extragradient method when solving variational inequalities over a convex and closed set governed by pseudomonotone Lipschitz continuous operators. The main computational advantage of Tseng’s algorithm is that it relies only on a single projection step and two independent queries of a stochastic oracle. Our algorithm incorporates a minibatch sampling mechanism and leads to almost sure convergence to an optimal solution. To the best of our knowledge, this is the first stochastic look-ahead algorithm achieving this by using only a single projection at each iteration.

Decentralized and parallel primal and dual accelerated methods for stochastic convex programming problems

Abstract We introduce primal and dual stochastic gradient oracle methods for decentralized convex optimization problems. Both for primal and dual oracles, the proposed methods are optimal in terms of the number of communication steps. However, for all classes of the objective, the optimality in terms of the number of oracle calls per node takes place only up to a logarithmic factor and the notion of smoothness. By using mini-batching technique, we show that the proposed methods with stochastic oracle can be additionally parallelized at each node. The considered algorithms can be applied to many data science problems and inverse problems.

Variable Smoothing for Convex Optimization Problems Using Stochastic Gradients

We aim to solve a structured convex optimization problem, where a nonsmooth function is composed with a linear operator. When opting for full splitting schemes, usually, primal–dual type methods are employed as they are effective and also well studied. However, under the additional assumption of Lipschitz continuity of the nonsmooth function which is composed with the linear operator we can derive novel algorithms through regularization via the Moreau envelope. Furthermore, we tackle large scale problems by means of stochastic oracle calls, very similar to stochastic gradient techniques. Applications to total variational denoising and deblurring, and matrix factorization are provided.

Variance-Based Modified Backward-Forward Algorithm with Line Search for Stochastic Variational Inequality Problems and Its Applications

We propose a variance-based modified backward-forward algorithm with a stochastic approximation version of Armijo’s line search, which is robust with respect to an unknown Lipschitz constant, for solving a class of stochastic variational inequality problems. A salient feature of the proposed algorithm is to compute only one projection and two independent queries of a stochastic oracle at each iteration. We analyze the proposed algorithm for its asymptotic convergence, sublinear convergence rate in terms of the mean natural residual function, and optimal oracle complexity under moderate conditions. We also discuss the linear convergence rate with finite computational budget for the proposed algorithm without strong monotonicity. Preliminary numerical experiments indicate that the proposed algorithm is competitive with some existing algorithms. Furthermore, we consider an application in dealing with an equilibrium problem in stochastic natural gas trading market.

Convergent Noisy forward-backward-forward algorithms in non-monotone variational inequalities

Abstract We develop a new stochastic algorithm with variance reduction for solving pseudo-monotone stochastic variational inequalities. Our method builds on Tseng’s forward-backward-forward algorithm, which is known in the deterministic literature to be a valuable alternative to Korpelevich’s extragradient method when solving variational inequalities over a convex and closed set governed with pseudo-monotone and Lipschitz continuous operators. The main computational advantage of Tseng’s algorithm is that it relies only on a single projection step, and two independent queries of a stochastic oracle. Our algorithm incorporates a variance reduction mechanism, and leads to a.s. convergence to solutions of a merely pseudo-monotone stochastic variational inequality problem. To the best of our knowledge, this is the first stochastic algorithm achieving this by using only a single projection at each iteration.

Stochastic intermediate gradient method for convex optimization problems

New first-order methods are introduced for solving convex optimization problems from a fairly broad class. For composite optimization problems with an inexact stochastic oracle, a stochastic intermediate gradient method is proposed that allows using an arbitrary norm in the space of variables and a prox-function. The mean rate of convergence of this method and the probability of large deviations from this rate are estimated. For problems with a strongly convex objective function, a modification of this method is proposed and its rate of convergence is estimated. The resulting estimates coincide, up to a multiplicative constant, with lower complexity bounds for the class of composite optimization problems with an inexact stochastic oracle and for all usually considered subclasses of this class.

Uncertainty in Time

The relation between scatters of time and energy is well known in Classical and Quantum Physics, but the interpretations of such relation are not well understood. What is the meaning of a time scatter in a measurement? What is the meaning of the correlated energy gap? Why should they combine to give a value greater or equal to a theoretical lower bound? In this paper, we discuss uncertainty in measurement from the computational point of view and introduce the archetype of a generic stochastic oracle to a Turing machine.

A sparsity preserving stochastic gradient methods for sparse regression

We propose a new stochastic first-order algorithm for solving sparse regression problems. In each iteration, our algorithm utilizes a stochastic oracle of the subgradient of the objective function. Our algorithm is based on a stochastic version of the estimate sequence technique introduced by Nesterov (Introductory lectures on convex optimization: a basic course, Kluwer, Amsterdam, 2003). The convergence rate of our algorithm depends continuously on the noise level of the gradient. In particular, in the limiting case of noiseless gradient, the convergence rate of our algorithm is the same as that of optimal deterministic gradient algorithms. We also establish some large deviation properties of our algorithm. Unlike existing stochastic gradient methods with optimal convergence rates, our algorithm has the advantage of readily enforcing sparsity at all iterations, which is a critical property for applications of sparse regressions.

Convergence and application of online active sampling using orthogonal pillar vectors

The analysis of convergence and its application is shown for the Active Sampling-at-the-Boundary method applied to multidimensional space using orthogonal pillar vectors. Active learning method facilitates identifying an optimal decision boundary for pattern classification in machine learning. The result of this method is compared with the standard active learning method that uses random sampling on the decision boundary hyperplane. The comparison is done through simulation and application to the real-world data from the UCI benchmark data set. The boundary is modeled as a nonseparable linear decision hyperplane in multidimensional space with a stochastic oracle.

On-line learning for active pattern recognition

An adaptive on-line learning method is presented to facilitate pattern classification using active sampling to identify the optimal decision boundary for a stochastic oracle with a minimum number of training samples. The strategy of sampling at the current estimate of the decision boundary is shown to be optimal compared to random sampling in the sense that the probability of convergence toward the true decision boundary at each step is maximized, offering theoretical justification on the popular strategy of category boundary sampling used by many query learning algorithms.