Stochastic Setting Research Articles

Constrained regret minimization for multi-criterion multi-armed bandits

We consider a stochastic multi-armed bandit setting and study the problem of constrained regret minimization over a given time horizon. Each arm is associated with an unknown, possibly multi-dimensional distribution, and the merit of an arm is determined by several, possibly conflicting attributes. The aim is to optimize a ‘primary’ attribute subject to user-provided constraints on other ‘secondary’ attributes. We assume that the attributes can be estimated using samples from the arms’ distributions, and that the estimators enjoy suitable concentration properties. We propose an algorithm called Con-LCB that guarantees a logarithmic regret, i.e., the average number of plays of all non-optimal arms is at most logarithmic in the horizon. The algorithm also outputs a boolean flag that correctly identifies, with high probability, whether the given instance is feasible/infeasible with respect to the constraints. We also show that Con-LCB is optimal within a universal constant, i.e., that more sophisticated algorithms cannot do much better universally. Finally, we establish a fundamental trade-off between regret minimization and feasibility identification. Our framework finds natural applications, for instance, in financial portfolio optimization, where risk constrained maximization of expected return is meaningful.

Uncertainty-aware data-driven predictive control in a stochastic setting

Data-Driven Predictive Control (DDPC) has been recently proposed as an effective alternative to traditional Model Predictive Control (MPC), in that the same constrained optimization problem can be addressed without the need to explicitly identify a full model of the plant. However, DDPC is built upon input/output trajectories. Therefore, the finite sample effect of stochastic data, due to, e.g., measurement noise, may have a detrimental impact on closed-loop performance. Exploiting a formal statistical analysis of the prediction error, in this paper we propose the first systematic approach to deal with uncertainty due to finite sample effects. To this end, we introduce two regularization strategies for which, differently from existing regularization-based DDPC techniques, we propose a tuning rationale allowing us to select the regularization hyper-parameters before closing the loop and without additional experiments. Simulation results confirm the potential of the proposed strategy when closing the loop.

The data-driven time-dependent orienteering problem with soft time windows

In this paper, we study an extension of the orienteering problem where travel times are random and time-dependent and service times are random. Additionally, the service at each selected customer is subject to a soft time window; that is, violation of the window is allowed but subject to a penalty that increases in the delay. A solution is a tour determined before the vehicle departs from the depot. The objective is to maximize the sum of the collected prizes net of the expected penalty. The randomness of the travel and service times is modeled by a set of scenarios based on historical data that can be collected from public geographical information services. We present an exact solution method for the problem based on a branch-and-bound algorithm enhanced by a local search procedure at the nodes. A numerical experiment demonstrates the merits of the proposed solution approach. This study is the first to consider an orienteering problem with stochastic travel times and soft time windows, which are more relevant than hard time windows in stochastic settings.

Set-Based Intent-Expressive Trajectory Planning and Intent Estimation

This letter proposes a novel approach to intent-expressive motion planning and intent estimation for agents/robots with uncertain discrete-time affine dynamics. In contrast to the more commonly considered stochastic settings, our intent-expressive trajectory planning approach is set-based and leverages the active model discrimination framework for designing optimal inputs to attain certain target/goal states, while allowing an observer/teammate to clearly infer the intended goal based only on observations of a partial trajectory before it has reached its target/goal state, despite worst-case uncertainties. Further, in tandem with the planning algorithm, we also propose an intent estimation algorithm that can be used by the observer/teammate to determine the intended goal from observations of a noisy, partial trajectory.

Stochastic Learning Rate With Memory: Optimization in the Stochastic Approximation and Online Learning Settings

In this letter, multiplicative stochasticity is applied to the learning rate of stochastic optimization algorithms, giving rise to stochastic learning-rate schemes. In-expectation theoretical convergence results of Stochastic Gradient Descent equipped with this novel learning rate scheme under the stochastic setting, as well as convergence results under the online optimization settings are provided. Empirical results consider the case of an adaptively uniformly distributed multiplicative stochasticity and include not only Stochastic Gradient Descent, but also other popular algorithms equipped with a stochastic learning rate. They demonstrate noticeable optimization performance gains with respect to their deterministic-learning-rate versions.

AEGD: adaptive gradient descent with energy

We propose AEGD, a new algorithm for optimization of non-convex objective functions, based on a dynamically updated 'energy' variable. The method is shown to be unconditionally energy stable, irrespective of the base step size. We prove energy-dependent convergence rates of AEGD for both non-convex and convex objectives, which for a suitably small step size recovers desired convergence rates for the batch gradient descent. We also provide an energy-dependent bound on the stationary convergence of AEGD in the stochastic non-convex setting. The method is straightforward to implement and requires little tuning of hyper-parameters. Experimental results demonstrate that AEGD works well for a large variety of optimization problems. Specifically, it is robust with respect to initial data, capable of making rapid initial progress. The stochastic AEGD shows comparable and often better generalization performance than SGD with momentum for deep neural networks. The code is available at https://github.com/txping/AEGD.

Finite-Data Error Bounds for Koopman-Based Prediction and Control

The Koopman operator has become an essential tool for data-driven approximation of dynamical (control) systems, e.g., via extended dynamic mode decomposition. Despite its popularity, convergence results and, in particular, error bounds are still scarce. In this paper, we derive probabilistic bounds for the approximation error and the prediction error depending on the number of training data points, for both ordinary and stochastic differential equations while using either ergodic trajectories or i.i.d. samples. We illustrate these bounds by means of an example with the Ornstein–Uhlenbeck process. Moreover, we extend our analysis to (stochastic) nonlinear control-affine systems. We prove error estimates for a previously proposed approach that exploits the linearity of the Koopman generator to obtain a bilinear surrogate control system and, thus, circumvents the curse of dimensionality since the system is not autonomized by augmenting the state by the control inputs. To the best of our knowledge, this is the first finite-data error analysis in the stochastic and/or control setting. Finally, we demonstrate the effectiveness of the bilinear approach by comparing it with state-of-the-art techniques showing its superiority whenever state and control are coupled.

The integrated lot-sizing and cutting stock problem under demand uncertainty

Interest in integrating lot-sizing and cutting stock problems has been increasing over the years. This integrated problem has been applied in many industries, such as paper, textile and furniture. Yet, there are only a few studies that acknowledge the importance of uncertainty to optimise these integrated decisions. This work aims to address this gap by incorporating demand uncertainty through stochastic programming and robust optimisation approaches. Both robust and stochastic models were specifically conceived to be solved by a column generation method. In addition, both models are embedded in a rolling-horizon procedure in order to incorporate dynamic reaction to demand realisation and adapt the models to a multistage stochastic setting. Computational experiments are proposed to test the efficiency of the column generation method and include a Monte Carlo simulation to assess both stochastic programming and robust optimisation for the integrated problem. Results suggest that acknowledging uncertainty can cut costs by up to 39.7%, while maintaining or reducing variability at the same time.

Mean square stability conditions for platoons with lossy inter-vehicle communication channels

This paper studies the mean-square stability of heterogeneous LTI vehicular platoons with inter-vehicle communication channels affected by random data loss. We consider a discrete-time platoon system with predecessor following topology and a constant time-headway spacing policy. Lossy channels are modeled by Bernoulli processes and allowed to be correlated in space. We make use of a class of compensation strategies to reduce the effect of data loss. Necessary and sufficient conditions are derived to guarantee the convergence of the mean and variance of the tracking errors, which depend not only on the controller design but also on the compensation strategy and the probabilities of successful transmission. We illustrate the theoretical results through numerical simulations, describing different platoon behaviors. We also provide insights on the mean-square stability as a necessary condition for string stability in this stochastic setting.

Computational characterization of recombinase circuits for periodic behaviors

Recombinases are site-specific proteins found in nature that are capable of rearranging DNA. This function has made them promising gene editing tools in synthetic biology, as well as key elements in complex artificial gene circuits implementing Boolean logic. However, since DNA rearrangement is irreversible, it is still unclear how to use recombinases to build dynamic circuits like oscillators. In addition, this goal is challenging because a few molecules of recombinase are enough for promoter inversion, generating inherent stochasticity at low copy number. Here, we propose six different circuit designs for recombinase-based oscillators operating at a single copy number. We model them in a stochastic setting, leveraging the Gillespie algorithm for extensive simulations, and show that they can yield coherent periodic behaviors. Our results support the experimental realization of recombinase-based oscillators and, more generally, the use of recombinases to generate dynamic behaviors in synthetic biology.

Nonuniqueness in law for stochastic hypodissipative Navier–Stokes equations

We study the incompressible hypodissipative Navier–Stokes equations with dissipation exponent 0<α<12 on the three-dimensional torus perturbed by an additive Wiener noise term and prove the existence of an initial condition for which distinct probabilistic weak solutions exist. To this end, we employ convex integration methods to construct a pathwise probabilistically strong solution, which violates a pathwise energy inequality up to a suitable stopping time. This paper seems to be the first in which such solutions are constructed via Beltrami waves instead of intermittent jets or flows in a stochastic setting.

Optimal budget allocation policy for tabu search in stochastic simulation optimization

Tabu search (TS) is a powerful method for solving combinatorial optimization problems. However, when TS is adopted for stochastic simulation optimization, the simulation noises may mislead the search direction and prevent TS from converging to high-quality solutions. This issue can be mitigated by increasing the number of simulation samples used for solution evaluation, however, real-world applications are generally constrained by a finite computing budget. Therefore, it is critical to reducing the noise effect by an efficient simulation budget allocation, i.e., splitting the total number of samples on solutions. Studies on the budget allocation problem of TS are sparse. Most of the works employ equal allocation or simple policies which are not optimal. Based on the large deviations framework, we propose a new budget allocation policy to enhance the performance of TS under stochastic settings. The proposed allocation policy, provided in closed-form formulas, is asymptotically optimal for maximizing the probability that TS performs the correct move in a single iteration. The efficiency of the proposed method is validated by solving different problems from manufacturing and healthcare scenarios, including an inventory control problem, a throughput maximization problem for the production line, and a physician scheduling problem for the radiotherapy center. The numerical results show that, with the proposed method, TS can obtain better results using the same amount of computational efforts.

Scattering for the cubic Schrödinger equation in 3D with randomized radial initial data

We obtain almost-sure scattering for the Schrödinger equation with a defocusing cubic nonlinearity in the Euclidean space R 3 \mathbb {R}^3 , with randomized radially-symmetric initial data at some supercritical regularity scales. Since we make no smallness assumption, our result generalizes the work of Bényi, Oh and Pocovnicu [Trans. Amer. Math. Soc. Ser. B 2 (2015), pp. 1–50]. It also extends the results of Dodson, Lührmann and Mendelson [Adv. Math. 347 (2019), pp. 619–676] on the energy-critical equation in R 4 \mathbb {R}^4 , to the energy-subcritical equation in R 3 \mathbb {R}^3 . In this latter setting, even if the nonlinear Duhamel term enjoys a stochastic smoothing effect which makes it subcritical, it still has infinite energy. In the present work, we first develop a stability theory from the deterministic scattering results below the energy space, due to Colliander, Keel, Staffilani, Takaoka and Tao. Then, we propose a globalization argument in which we set up the I I -method with a Morawetz bootstrap in a stochastic setting. To our knowledge, this is the first almost-sure scattering result for an energy-subcritical Schrödinger equation outside the small data regime.

Adaptive Algorithms for Meta-Induction

Work in online learning traditionally considered induction-friendly (e.g. stochastic with a fixed distribution) and induction-hostile (adversarial) settings separately. While algorithms like Exp3 that have been developed for the adversarial setting are applicable to the stochastic setting as well, the guarantees that can be obtained are usually worse than those that are available for algorithms that are specifically designed for stochastic settings. Only recently, there is an increasing interest in algorithms that give (near-)optimal guarantees with respect to the underlying setting, even in case its nature is unknown to the learner. In this paper, we review various online learning algorithms that are able to adapt to the hardness of the underlying problem setting. While our focus lies on the application of adaptive algorithms as meta-inductive methods that combine given base methods, concerning theoretical properties we are also interested in guarantees that go beyond a comparison to the best fixed base learner.

Optimizing Age-of-Information in Adversarial and Stochastic Environments

We design efficient online scheduling policies to maximize the freshness of information delivered to the users in a cellular network under both adversarial and stochastic channel and mobility assumptions. The information freshness achieved by a policy is investigated through the lens of a recently proposed metric - <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Age-of-Information</i> (AoI). We show that a natural greedy scheduling policy is competitive against any optimal offline policy in minimizing the AoI in the adversarial setting. We also derive universal lower bounds to the competitive ratio achievable by any online policy in the adversarial framework. In the stochastic setting, we show that a simple index policy is near-optimal for minimizing the average AoI in two different mobility scenarios. Further, we prove that the greedy scheduling policy minimizes the peak AoI for static users in the stochastic setting. Simulation results show that the proposed policies perform well under realistic conditions.

Inventory decisions for ameliorating products under consideration of stochastic demand

Product quality is a key factor in the food and drinks industry, allowing companies to distinguish themselves from their competitors and achieve higher profit margins. However, the quality of a product may be subject to change over time and thus complicate planning and decision-making processes. While most of the literature in this area focuses on quality change in the form of deterioration, there are also products that improve in quality over time, resulting in unique challenges with regard to production and inventory management. In this context, this paper introduces an inventory planning problem for ameliorating products under consideration of demand uncertainty and product loss due to evaporation over time. This problem is solved using a MILP model within a rolling horizon approach. The proposed model is illustrated on two case studies from the cheese and whisky industry and investigates different scenario settings. The general findings show that the differences in profit margins between the quality categories influence the optimal strategy of the manufacturer and that the optimal aging and storage time should be kept to a minimum. Moreover, the results show that the rolling horizon approach is well-suited to handle the stochastic setting and that high demand fulfilment can be achieved as long as an appropriate planning horizon for the different products and product categories is considered • Stochastic problem formulation to study inventory decisions for ameliorating products. • Illustration based on two case studies from the cheese and whisky industry. • Analysis of different scenario settings related to uncertainty and other aspects. • Findings show the impact of profit margins on the optimal strategy of the producer.

Regularized quasi-monotone method for stochastic optimization

We adapt the quasi-monotone method, an algorithm characterized by uniquely having convergence quality guarantees for the last iterate, for composite convex minimization in the stochastic setting. For the proposed numerical scheme we derive the optimal convergence rate of $$\text{ O }\left( \frac{1}{\sqrt{k+1}}\right)$$ in terms of the last iterate, rather than on average as it is standard for subgradient methods. The theoretical guarantee for individual convergence of the regularized quasi-monotone method is confirmed by numerical experiments on $$\ell _1$$ -regularized robust linear regression.

Bisimulations of Probabilistic Boolean Networks

A probabilistic Boolean network (PBN) is a collection of Boolean networks endowed with a probability structure describing the likelihood with which a constituent network is active at each time step. This paper proposes a definition of bisimulation for PBNs. The notion is inspired by the analogous notions for probabilistic chains and for stochastic linear systems. Necessary and sufficient conditions to check the proposed notion are derived, model reduction of PBNs via bisimulation is addressed, and a discussion on the use of probabilistic bisimulation for optimal control design is given. The present results extend the theory of bisimulation known for deterministic Boolean networks to a stochastic setting.

Properties in Stage-Structured Population Models with Deterministic and Stochastic Resource Growth

Modelling population dynamics in ecological systems reveals properties that are difficult to find by empirical means, such as the probability that a population will go extinct when it is exposed to harvesting. To study these properties, we use an aquatic ecological system containing one fish species and an underlying resource as our models. In particular, we study a class of stage-structured population systems with and without starvation. In these models, we study the resilience, the recovery potential, and the probability of extinction and show how these properties are affected by different harvesting rates, both in a deterministic and stochastic setting. In the stochastic setting, we develop methods for deriving estimates of these properties. We estimate the expected outcome of emergent population properties in our models, as well as measures of dispersion. In particular, two different approaches for estimating the probability of extinction are developed. We also construct a method to determine the recovery potential of a species that is introduced in a virgin environment.

Technical Note—An Improved Analysis of LP-Based Control for Revenue Management

Bounded Regret for LP-Based Revenue-Management ProblemsIn “An Improved Analysis of LP-Based Control for Revenue Management,” Chen, Li, and Ye study a class of quantity-based network revenue-management problems. The authors consider a stochastic setting where all the orders are i.i.d. sampled and the customers are of finite type. They focus on the classic LP-based adaptive algorithm and consider regret as the performance measure. They found that when the underlying LP is nondegenerate, the algorithm achieves a problem-dependent regret upper bound that is independent of the horizon/number of time periods T; when the underlying LP is degenerate, the algorithm achieves a tight regret upper bound that scales on the order of T log(T) and matches the lower bound up to a logarithmic order.